Slashdot Mirror
How Do You Protect Servers From a Rogue Admin?
Treborto writes "I work with a non-profit that has an extensive collection of photos and videos. These are used in publications and on the web. We have several levels of privileges: read-only of small, watermarked images; read-only of large, clean images; edit of the site; and admins who can confer privileges. It has happened that people leave the organization in anger. So far, no Admin has done so. Is there a back-up, site mirroring, privilege, or other strategy you'd recommend so we have protection from an Admin gone bad?"
FS snapshotting and backups are the only way, but make sure your backups are protected (locked up) etc.
If people routinely leave your non-profit organization in anger, then the organization's leaders probably need to address a more fundamental problem than server administrative rights.
Create a encrpyted password protected snapshot archive of your server and name it something catchy like angie jolie secret sextape 1-29-2011 and upload it to piratebay. Safe secure lifetime backup retention online.
And usually that's the admins. Most admins gone bad would be smart enough to bone the backups if they were going to do deliberate damage. The best way to protect yourself is an off-site DVD backup, but that's a lot of work to keep current.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
rsnapshot on a regular basis to a off-site service, that's read-only to the organization. I run that kind of service for several organizations for exactly that reason.
Rogue admins are extremely rare. So rare that there are many other more likely threats you will encounter, such as hackers or data breach. Worry about those first.
The reality is that most people work in a spirit of cooperation and don't want the black mark on their reputation. They would rather walk away without burning bridges.
That being said, bad admins (and employees in general) spring from two causes: bad treatment and pre-existing jerks.
The best way to handle both situations is to talk to your employees regularly, and find out how they feel. If you know that some policy or other is bothering them, you can avert a crisis very easily if you know about it beforehand.
Some people are just jerks. Don't let these people continue in your organization, even if they are brilliant and highly capable, and even if you don't have an equally brilliant replacement. A mediocre replacement who can work well with others will be much more productive.
(Often said: About 15% of your productivity comes from innate ability, 85% from working with others.)
That having been said, if you're really worried about someone doing you in, make sure you have regular backups and that you personally have access to the backup system. Reformatting a disk and copying data is easy - position yourself so that you can recover completely from the maximum damage they can do.
Hi, This is one of the classic questions of insider misuse mitigation "who watches the guards". One way to deal with this is to use very good logging using a third audit party. Traditional audit/logging engines are not well suited to this task. You might like to take a look at LUARM (http://luarm.sourceforge.net/). It is an effort to provide very fine grained logging into your systems. The idea is you setup engines like that and your logs are then placed off-site and managed by a third party auditor, away from a potentially rogue sysadmin. Thus, if something happens, you have the means to prove what your bad techie did. Preventing this to happen is another story. Some people say that the knowledge of being monitored deters people from doing stuff. I do not support that view. Simply, my experience in dealing with sysadmins is that they are often underpaid, not appreciated and take all sorts of crap for other people. Make sure you pay them well, support them and listen to what they have to say. (a sysadmin) :-)
How do you protect servers from rogue admins, they same way you protect passengers jets from rogue pilots, they say way you protect ships from rogue captains, the same way you protect buses from rogue drivers, the same way you protect trains from rogue engineers and even the same way you protect patients from rogue doctors.. You don't, any protection you put in place to protect a server from a rogue administrator will be broken by that rogue administrator if they are in any way competent. I suppose you could always seek to hire the most incompetent admin you can find a person who lacks the expertise to break the servers but somehow that seems rather pointless. So how do you protecct your servers from rogue admins, don't hire them in the first place. Consider a full psych evaluation (stay away from the anal types), pay a food salary and, make them part of the executive team.
Chaos - everything, everywhere, everywhen
Don't let clueless PHB's run IT.
Don't make so there only 1 guy doing the network admin
Don't ask for admin password over a conference call
What is you backup method. Many more things can happen than a rogue admin messing up files. Disks fail, equipment gets stolen, users accidentally delete items - all of which point to having a robust, redundant backup strategy. Absent that, rogue admins are the least of your worries.
We've kept rolling backups - i.e several weeks worth, on duplicate media. On-site for fast access and off site for ensuring its availability if something happens on-site. I know others that mirror the entire operation to another secure location.
My suggestion - figure out how much data needs to be backed up, how often does it change, and then develop a redundant backup strategy with teh ability to roll back several generations.
You can't protect against any and all employee actions, but at least you can make it hard to totally destroy your data.
Also - as others pointed out - find out why people leave mad and fix the underlying cause.
I'm a consultant - I convert gibberish into cash-flow.
No matter what solutions you use for backups, the admin will be able to corrupt or bypass them in some way given enough thought and motivation.
However, for sane though disgruntled people it would be sufficient for them to have the common sense understanding that malicious actions will have strict consequences - people generally don't risk going to jail just to annoy a manager or company. And in the cases where someone would really be prepared to risk that, I'd rather worry about them coming to office with a gun, not tampering with a pile of pictures.
What was the aftermath of the previous cases you say of people leaving in anger and presumably doing something damaging? Your previous reaction in these cases forms the expectations in your admins about what they can get away with when leaving in anger.
Even better, set both your system and sudo so that nothing ever goes root... Using system user accounts instead of root mean that even if someone goes berserk, he won't have full access on the system; and restrict sudo to only run some commands as other users, instead of using ALL everywhere...
If you truly are concerned about the trustworthiness of your systems administrator; you definitely don't have the right person in place and you need to take steps NOW to ensure the continuity of your systems. Start implementing strict documentation standards for everything - passwords, system maintenance procedures, run books, network diagrams, etc... This information then needs to be stored in location accessible by senior executives and audited by an external firm to ensure completeness and validity. You have to be careful about this though, because it can be a tip off that the administrator's tenure is coming to a close shortly. It can be very costly to have your admin walk off the job with all the passwords. Your systems will be unmanageable and if the passwords can not be recovered by a forensics firm, you'll have to wipe and re-implement the affected systems. Better to have a discussion with all employees and say that the company has come under regulatory scrutiny, or some other excuse, and that all departments must now thoroughly document everything they do. Then everybody is on an equal playing field and employees are less likely to think more into it.
As far as backups go, bring in an external firm to configure, perform, monitor, and audit the backups. The best system would be an off-site mirror of your data center managed by this firm. But tape archives can be effective as well. Either way, your administrator would be discouraged from tampering with the backups, as an audit would immediately show any attempts at sabotage. But even with backups, you could be talking about days of downtime before all systems could be restored, so best to fix the human problem first before you even get to this point.
I went into a local community college with a team of talented system engineers after they were forced to fire their hands-on IT manager. They neglected to get typed and validated documentation from him before they kicked him out, and it took us days to crack all of the passwords and document all of the systems and networks. I estimate it probably cost the college at least $20,000 for this forensics work because they didn't handle the situation properly.
Don't worry about your infrastructure so much. Having been in this position, I noticed that companies seem to worry quite a lot of it.
But it seems to me that it's an unlikely situation. Let's suppose there's an admin really pissed off at you for some reason. What could they do to your photo collection?
All those options are pointless and ultimately suicidal for the admin involved. All you need to do is to have readonly off-site backups (which you should have anyway, what if the building gets flooded or burns down?). If properly done the rogue admin can't screw that up, and while the things above might hurt, they'll be perfectly survivable. Even the torrent isn't a big deal. A serious publication isn't going to touch an illegal collection with a 10 foot pole. As a public organization they're an easy and profitable target.
However, those things are terribly stupid and suicidal for the rogue admin. Who will be the first suspect in line when any of the above happens? The recently fired angry admin. Law enforcement treats such things harshly, and word of mouth gets around and it's unlikely they'll get another job after that.
All the admins I've seen leave (and I took note and did it myself when leaving a job) tried to leave in an as non-threatening way as possible. For instance on my last day on one job I discussed with a coworker what I had been doing, where the files were, what was unfinished, the lists of passwords and access control methods to be changed, etc. I did everything I could to make sure that nothing in my departure could be interpreted in a "screw you" of any kind, and to make sure my successor could take over.
Now, what should you be worried about? The legal ways an ex-employee can screw you over. For instance, the BSA. It's easy to report to them. From what I hear they're most eager to show up, offer rewards to the reporter, and it's very hard to deny them entry. And I hear that their visits can be very expensive. So make extra sure you're in perfect licensing compliance (which is pretty hard), or switch to Free Software.
At a small company I used to work for ("used to" being the key phrase here), the bosses, who both insisted on full admin rights, had a bit of a difference with each other. One of the bosses came in one Saturday night, killed the backup (they never took my advice of having multiple backups, including one off-site), and ran off with the server.
I tried recovering the backup, but he did a remarkable job in killing it.
The company didn't exist for more than a week after that.
Does it make you happy you're so strange?
First, you need to stop drinking the coolaid. You are paying the sys-admin to keep your systems up and running. They do have "the keys to the kingdom", because you are paying that person to hold them. If you don't trust that person to hold the keys, then you shouldn't have hired them in the first place.
The ways you mitigate the issue of "rogue" admins, is vet them, listen to what they are saying in terms of technology, don't micro-manage them, and pay them well. The good ones without a doubt will know the technology better than their manager/management structure will ever know it. The reason the admin says something about the setup/configuration/technology is almost always because it is needed change. If you can't afford to make those changes, then you need to explain that is the reason, don't make up some BS about how you want things to stay the way the are, or you want to change the organization/structure to something else, because they will "call" you on it. Again, they know the technology better than you ever will.
The other thing to do is to pay them appropriately. You are trusting them with running some of the most complex systems in your entire company, as well as safe-guarding your data, your processes, and your daily operations. The reason why you don't see many rogue CEO's is because he/she is being paid well to run the company, choose its path, and steer the ship, so to say. The system admins in today's information based businesses are the guys keeping your entire company running. If your servers/data were all destroyed, and your business would not survive, then you might want to consider paying the people who keep that data/servers a more appropriate amount of compensation since they are so vital to your business.
Again, there are very few admins who go rogue, and even fewer who did not do so after being mistreated by their bosses/management. If people want to point out at the case of Terry Childs, they need to get a clue. Were mistakes made, sure. Did Terry have some issues? Yes. Did he actually go rogue? No. In his eyes, he was protecting the network from idiots and incompetents, and following the rules as currently defined. He wouldn't give out the passwords in a room of strangers, over the phone, or via email where it can easily be intercepted and then misused, as well as be cause for firing him because policy stated not to do any of those things. So he was placed into a situation where he would be fired if he handed out the passwords, or fired if he didn't. And once fired, he really had no obligation at all to give it out anymore, why? Because he didn't work there. Same as if you fired your top salesman, or stock broker, or process manager. They don't have any obligation to tell you anything about the contacts/client relationships/methods for picking stocks/how things work. If you fired them before you obtained that information, then you should have been fired. In the Childs case, were they trying to obtain that information, sure. But in the wrong way according to policy. They should have taken Terry into a one on one conversation, in a private room, with no one the phone and asked in that setting. Even then, he might have refused to have the manager have the password because the manager didn't have the knowledge or skill to know how to properly vet someone as being capable of having the password. The only thing that would happen is that it will cause someone to screw up the settings and create work for Terry since he will be the one called in to fix it, and most likely not paid for that extra time he had to spend fixing someone else's screw up.
Again, it comes down to properly compensating the admins, listening to them, and not trying to play office politics with them. You treat them well, and they will do whatever it takes to keep the systems running because they take pride in their work. You treat them like crap, blindly disregard their expertise in terms of operating the servers/network because "you know better than they do", you are asking for th
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
Ask the sys admins there to come up with a method; most folks working non-profit do it for the work not the pay, and many techs like the responsibility and challenge. By asking them to help solve the problem, you reduce the stress that would otherwise make them think they are the bad guy, and give them the merit that they do know what they are doing. Even if they cant come up with a reasonable solution, if you pick a third party, they wont be so miffed about it.
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
Ultimately, you cannot be sure you won't get screwed, ever. Not even by hackers outside of your organization, let alone ones inside. It is possible to -- reasonably -- secure a system using methods described above (offsite backups managed by a third party commercial affair, onsite backups under lock and key, careful logging and so on). However, in nearly any network there is one toplevel admin that doles out the privileges and so on, that set the system up, that works on the system many times more often and at a much higher level than the people that typically have permission to do a few things enabled by sudo. There, no matter what, you will be vulnerable.
/. and it is going to cost you money). Basically, the more you try to secure things on the cheap, the more likely it will be that you have a setup with holes you can drive a truck through given the root password and access.
This is a classic problem: Quid custodes custode (who will guard the guardians)?
Paradoxically, you are probably slightly safer if your admins are not uberkinder supergeeks. If I, or any of a dozen people I know, were your toplevel sysadmin and was not the completely honest and trustworthy person that I am, there is no measure you could take for protection that I could not suborn in such a way as to cause you great pain and loss. After all, who would be implementing the measures? Log files are pointless ways to reveal the activities of the person who set up the logging system. Subtly corrupting the backups for long enough to roll over the offsite images (which could be as simple a measure as installing an encrypted filesystem "for security reasons" and making sure that I'm the only person that has the real key). An amateur (or less skilled professional) is less likely to know enough to do dirt and hide their tracks.
There is no real protection against hiring people to do mission critical work of any sort who have a serious personality disorder. So your best protection of all is to hire toplevel systems staff who are, as far as you can tell looking hard, completely ethical and personality disorder free, and then treating them with respect.
Good advice for keeping ordinary employees from going postal, good advice for any organization or task, actually.
There is one more solution -- the NSA sort. Throw an enormous amount of money at it, and hope that the people you hire aren't smarter than the (unknown) one you are defending against and that they leave no holes in what they set up. Hiring ten top sysadmins all tasked with watching each other is good. Having commercial consultants who know what they are doing help you set up a system is good (in other words, if you have to ask the question you need to get an answer somewhere other than
rgb
Even when the experts all agree, they may well be mistaken. --- Bertrand Russell.
A rogue admin will create a back door before they leave. Often they will do this midway in their career to try and ensure continued employment, but that would never work out. Eventually they will be found out. All "Good" admins realise this, so it shouldn't be an issue. Just try to ensure you hire "Good" admins. Personality tests may help in that venue, but history of previous actions taken during "stressful" times may prove to be a better indicator of how they will behave in the future. People often repeat bad mistakes if they don't realise that they are the ones making the mistakes.
If an admin deletes all the files.... how quickly do you need every single photo back?
The longer your business can live without them, the less expensive a solution you will require, and the more reliable a solution you can pick.
Some of the least expensive solutions are.... burn every new file to DVD and backup every new file to tape or traditional film. Translation: backup as you go.
If you want your collection to survive thermonuclear war and EMPs, then record every frame to film using a Film recorder; have the roll fixed and developed, and lock the film up in an underground bunker, in an airtight safe with minimal humidity.
For faster recovery, you will need regular full backups. Lock them up in different places. Make sure to never ever reuse a tape. Always use fresh media for every new backup.
I worked for a bank auditing company for a while, and installing anything (or any administrative work) was a pure PITA. There was a mandatory "four eyes" principle in effect. Logging in without a second person (every admin login caused a text message to go to all admins, just in case you're wondering whether nobody did it "stealthily") was grounds for instant firing. You would grab a fellow admin (or, if nobody was around, anyone who could "supervise"), fill out a form that you and him are going to log in, then you started a protocol (pencil and paper type) of what you are going to do. Every keystroke, every click of the mouse, was to be written down, then executed. Installing a program or an update by protocol could well take an hour or two, and certainly not 'cause the machines were slow. Termination was told to you the moment you were let go, the same moment two admins were sent with high priority order to revoke your admin privs. On the upside, you were let go instantly, i.e. take your stuff, do not log in, you may spend the rest of your working days at home (i.e. effectively another 1 month of paid vacation). If you had to clean up anything on your machine, two admins did it for you.
This is a level of security and paranoia that borders on insane. Personally, I'd say it's a wee bit beyond insane already. But it gives you an idea that banks tend to take security and the threat of rogue admins VERY serious.
But there is one thing you should definitely do when firing an admin: Revoke his admin privs INSTANTLY the moment he learns that he is gone and send him home. Even if laws demand that you have to tell him 2 weeks before firing him, send him home on 2 weeks of paid vacation. It's cheaper than the threat of having him do something to retaliate at you.
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
To be serious about security, you have to eliminate every last single point of failure. Although I seriously doubt a non-profit would have the cash to justify paying rather than simply trusting, if they were serious about limiting the damage an admin could do, they would outsource the backup, requiring that the backup be regularly monitored for suspicious changes and tested both by the outsource and by someone within the company.