Searching For Backdoors From Rogue IT Staff

WHiTe VaMPiRe writes "When IT staff are terminated under duress, there is often justification for a complete infrastructure audit to reduce future risk to a company. Here is an exploration of the steps necessary to maintain security." Of course the first piece of advice is to basically assume you've been rooted. Ouch.

Comments

terminated under duress

[cmiller173]

Seems like it would make sense to simply terminate "with extreme prejudice" when getting rid of potential security threats....

Re:terminated under duress

[arth1]

Yeah, that will really solve the problem of time bombs and dead man's switches...

How about not disgruntling the employee in the first place?

Re:terminated under duress

[Mongoose Disciple]

How about not disgruntling the employee in the first place?

It's a good policy and should be encouraged, because it does solve most problems. However, believing that will solve all your problems rests on the assumption that your employees are basically rational and won't do anything crazy just because. This won't always be true.

Relatively current events counterexample A: Terry Childs.

Re:terminated under duress

[Monkeedude1212]

I know what you're thinking, but not every company has a nuke stationed in orbit. Let's try to be practical here.

Re:terminated under duress

[duguk]

How about not disgruntling the employee in the first place?

Relatively current events counterexample A: Terry Childs.

I would argue that Terry Childs was disgruntled, being as he had an ongoing disciplinary case.

Re:terminated under duress

[mysidia]

How about not disgruntling the employee in the first place?

I suppose this could be used against Terrorists and suicide bombers as well.

Don't make people dissatisfied with your country. Oh wait, you only have limited control of that, oh well....

Re:terminated under duress

[cjb658]

Reminds me of a speech Ian Angell gave at Defcon. I guess a CEO of a bank there terminated and outsourced the entire IT department. A couple days later, it surfaced that he had all kinds of pr0n on his computer.

Re:terminated under duress

[Surt]

So the solution, clearly, is never to hire anyone who in the future might cause you to have to resort to disciplinary action.

Re:terminated under duress

[bill_mcgonigle]

Relatively current events counterexample A: Terry Childs

He may have bucked the chain of command, but if his employer had sat him down, said, "look, Terry, we think you'd be better off somewhere else - we're going to keep you on until you find a better opportunity, and we're going to help you do that," he would have probably said, "yeah, but you have nobody else here who can handle this thing. You're going to need to hire a firm to manage this or get some better talent on staff," which seemed to be his motivating concern. And so they probably would have done that, and nobody would have gone to jail.

Instead it seemed like a "give us the passwords and um, no you don't need to clean out your desk, why?" kind of scenario. I'm not meaning to absolve Childs of incorrect behavior, but a little Golden Rule would have gone a long way there. I think this is what the GP meant by not disgruntling the employees.

Re:terminated under duress

[rickb928]

What makes you think they are any better at terminating employees with 'extreme prejudice' than they are at anything else?

And if they let me out the door, I figure they got what they need.

But I would not be leaving behind any easter eggs or backdoors. Life is too short, and no job I've ever had was worth what was so inelegantly termed 'pound-in-the-ass-prison'. Net admins and such get that. Their big bosses and CEOs get Club Fed and conjugal visits with their wife and mistress.

Re:terminated under duress

[Cramer]

Negative. I doubt there was any other way to get him to give up control of the network he created (and f'ing copyrighted.) Proper management was the answer, but because his manager allowed his ego to run free, we have a nice cautionary tale.

Re:terminated under duress

[Briden]

if you are an employee who is happy with your job, does that make you, gruntled?

Re:terminated under duress

[kevinNCSU]

That's generally a bad idea for anyone in a sensitive position. You certainly wouldn't want nuclear power plants for example to take that tack. A better proposition would be: "Look, Terry, this isn't working so we're going to have to ask you turn hand over the passwords and go home. If your willing to be available over the phone or come in for some 2 hour meetings to help with the transition we'll extend your pay for another month while you look for a new job." But he shouldn't be allowed near the equipment anymore, especially if he was already having attitude problems.

Re:terminated under duress

[John Hasler]

> How about not disgruntling the employee in the first place?

Some employees disgruntle themselves.

Re:terminated under duress

[DDLKermit007]

Or ya know...just treat your IT staff with some dignity & respect and it'll go so far to not leaving bad blood. Plus, giving them two weeks notice so they can get things in order for your company with a 3-6 month severance is always a way to make sure you don't run into problems. Oh, and don't ransack their office going happy go lucky trying to figure out what is & isn't their property unless they have been stealing from the company. Experienced plenty of good ways, and bad ways to be let go. Always a sign as to if your former managers will be able to handle it. Whats really funny though is when they end up having to bring back the person they just laid off at 5x their original hourly rate when things go south because they think they can bring someone else in at half the original cost who doesn't have a clue. It certainly is sweet sweet karmic justice when those events occur, and they almost always do with regularity. /rant

Re:terminated under duress

[OnePumpChump]

Umm...actually, that's pretty good advice for countries as well. Asking how to deal with disgruntled employees when you're treating people like shit is like asking for advice on stopping yellowjackets from stinging you when you're in the habit of throwing rocks and their nests.

Re:terminated under duress

[duguk]

So the solution, clearly, is never to hire anyone who in the future might cause you to have to resort to disciplinary action.

Or perhaps, a more sane solution to take away is to not rely on just one person for security, especially when you're about to discipline them; and disgruntling your only administrator without any form of reprise is possibly a very stupid idea.

Though, feel free to try your Magic-Eight Ball version if you like.

Re:terminated under duress

[Vectormatic]

Bull, a company treating you poorly is perfect justification for leaving ASAP, it does not justify active sabotage and other crap

If you get to the point where you want to install deadman switches and such, FIND A NEW JOB, and just normally leave. Life is too short to continue working at shit jobs, and sabotaging stuff most likely will land you in much more trouble anyway

Re:terminated under duress

[h00manist]

So the solution, clearly, is never to hire anyone who in the future might cause you to have to resort to disciplinary action.

Yep. Nobody said it was easy, but the alternative is worse.

Re:terminated under duress

[h00manist]

if you are an employee who is happy with your job, does that make you, gruntled?

A grunt.

Re:terminated under duress

[tehcyder]

I guess a CEO of a bank there terminated and outsourced the entire IT department. A couple days later, it surfaced that he had all kinds of pr0n on his computer.

So the entire IT department was staffed by fourteen year olds?

Re:terminated under duress

[BitZtream]

So its fairly clear you have no fucking clue what Terry Childs did.

He started his egotistical bullshit because he was being transferred to another department. If you bothered to find out any details of his case rather than warshiping him blindly like the son of god, you'd know that there was never a point where he was doing the right thing, and it becomes clear that everything he did was a power play where he was trying to show everyone else he was in control and they couldn't do anything to him.

It in no way sounded like he was being fired, he was in fact going to be responsible for a lot of the exact same equipment. The fact that he was ignoring city policy (by not putting the information in the cities configuration and password management system) is what got him fucked when he was asked to actually do so.

Either stop talking about Childs or get a fucking clue about what actually happened, you'll look like far less of a moron that way.

the work involved..

[Nick]

to audit your system under the assumption you've been rooted should happen once a year at a minimum anyway, not just when you suspect a rogue employee left on bad terms. I've worked at places that never changed passwords and I found former employee logins enabled from months ago..

Re:the work involved..

[arth1]

It's fairly impossible to audit all systems to the extent needed. You can easily burn enormous amounts of money and time doing that, and the remedies can disrupt production more than the damage the disgruntled employee would do.

There are so many ways to hide what you're doing that even rebuilding all systems isn't enough. Dangers can hide not only in backdoors, but dead man switches built in to compilers, stored procedures in databases, backups, or the Boss' PC, for that matter.

So instead of sending good money after bad, it can be immensely sensible to let things be and instead try to ensure that the employees don't leave disgruntled.

Re:the work involved..

[bloodhawk]

That would be nice but is in reality completely impractical. The time and money to do such an audit properly would be more expensive than just rebuilding your entire environment from the ground up. I could effectively hide a rooted box or backdoor on windows or *nix systems I look after that unless you are going to strip the boxes and mount the drives on seperate boxes to check the binaries you are simply not going to find the holes.

The ONLY way to handle a suspected rooting is a rebuild, anything less is always an assumption that your smarter at finding the exploit than they are at hiding it.

Re:the work involved..

[techno-vampire]

It's fairly impossible to audit all systems to the extent needed.

If the back door is as well hidden as the one Ken Thompson hid in an early version of Unix, a complete audit of the source code and complete recompile of everything won't be enough to get rid of it. Of course, not many people are capable of pulling that kind of stunt off.

Re:the work involved..

[nine-times]

So instead of sending good money after bad, it can be immensely sensible to let things be and instead try to ensure that the employees don't leave disgruntled.

Well step #1 is to try to hire some IT people who are generally trustworthy. Don't hire the shifty guy because he's a little cheaper. *Then* you try to make sure that your IT staff is gruntled.

It's also good if you can hire 2 people to work together. Though it might not be a viable option because of a shortage of work to go around, a person's honesty generally goes up if they're working with a partner. If you have 2 network admins, each will be more hesitant to do something shady for fear of the other one catching on. If one does do something shady, the other might also notice and alert you to it.

So that's the prevention aspect. After the fact, there are things you can do (check firewall rules, rotate all passwords, etc), but to some extent you're going to rely on your ex-employee's lack of cleverness, foresight, and malice. Either that, or you're going to be rebuilding your entire network.

Re:the work involved..

[Anonymous Coward]

to audit your system under the assumption you've been rooted should happen once a year at a minimum anyway ....

I'll get right on that in my copious amounts of free time....

Re:the work involved..

[arth1]

Well step #1 is to try to hire some IT people who are generally trustworthy. Don't hire the shifty guy because he's a little cheaper.

Do you have any data that shows a correlation between looking shifty and betrayal of trust?

Re:the work involved..

[nine-times]

I didn't say "shifty-looking". Anyway, "Don't hire the shifty guy because he's a little cheaper," was just an example. You can't be 100% sure, but I've heard employers express a lack of confidence in their IT people only to say something like, "but he was cheaper than the guy we wanted."

Re:the work involved..

[TheGratefulNet]

OT: but was there ever the source posted that contained this ken thompson back door? I guess it would have to be the gcc sources. is this confirmable, directly? anyone know? I've heard the story many times but never SAW any code.

Re:the work involved..

[techno-vampire]

The whole point of the hack was that it wasn't in the source code, so that auditing the code wouldn't reveal it. It's not mentioned, but I'd bet that people checked it out after his talk and verified that he'd done it.

Re:the work involved..

[pentalive]

Actually, wouldn't have been in the original code for the cc command in an early version of Unix? not Linux?

Re:the work involved..

[Mr. Freeman]

"It's fairly impossible to audit all systems to the extent needed. You can easily burn enormous amounts of money and time doing that, and the remedies can disrupt production more than the damage the disgruntled employee would do."

I'll deal with the second point first. A disgruntled employee could potentially have access to EVERYTHING. Every server, every backup. I'm not an IT professional but I know that companies failing to have secure offsite backups is relatively common. If an admin who had root access to everything (apparently, somewhat common) wanted to, he could conceivably wipe everything out over a weekend if he knew what he was doing.
If your auditing procedures are more expensive than loosing every single piece of data then something is seriously wrong.

As for it being impossible to audit all systems to this extent. Seems to me that something is seriously wrong if it is literally IMPOSSIBLE to audit these systems. This shows a huge lack of planning. What the hell did you plan to do if an admin went rogue? If you can't do a routine security audit then it's really just a matter of time before something goes horribly wrong.

Re:the work involved..

[Mr. Freeman]

"The whole point of the hack was that it wasn't in the source code"

That's impossible. You can't hard code something into a piece of software without it being in the source code. The hack would have been in the source code for the compiler.

Re:the work involved..

[Mr. Freeman]

Why is this modded informative? A lack of free time might prevent proper security procedures from being followed, but it doesn't remove the risk of not doing so. These security audits should happen, if you don't have time to do them then you have a security problem.

In fact, any admin that works somewhere that doesn't have the time to implement these security procedures will probably be more likely to attempt something than an admin that works at a place with proper security procedures. In the latter case, the admin knows he will likely be caught and brought up on criminal charges for vandalism and destruction of property. In the former, the admin knows there's no way he'll be caught even if he's sloppy.

Re:the work involved..

[techno-vampire]

I take it, then, that you didn't follow the link I gave, because the whole point of the hack was that none of it was in the source code. The compiler was hacked to add code to login when it compiled it, and to add code to itself (if it were recompiling itself) to do the work.

Re:the work involved..

[Anonymous Coward]

One company's executives demanded that more women be hired in IT. Then time for cutbacks came and one woman was let go. Now this particular executive was elsewhere because it was HR's responsibility for the cutbacks and was not there to veto her firing. Huge files kept appearing and when looked into it was found that she was e-mailing company information to her outside e-mail account but it was getting rejected because the files were larger than 10mb. I guess this proved they picked the right member of staff to let go.

The moral of this story is to hire trustworthy people and also not to hire staff on a fashionable whim.

Re:the work involved..

[Mr. Freeman]

From the link you posted: "The C compiler contained code that would recognise when the "login" command was being recompiled"

It says right there that he added code to the C compiler. The backdoor itself did not exist in source code until someone compiled the login command. At that point, the compiler added code to make a backdoor. The original hack (messing with the C compiler) WAS IN SOURCE CODE FROM THE BEGINNING.

Next time, read your own article, moron.

Re:the work involved..

[Anonymous Coward]

Even that does not help if the malicious code was put e.g. into the BIOS. And yes, there are quite a few people around who hack BIOSes as a hobby...
And always someone comes along with such great statements as "Seems to me that something is seriously wrong if it is literally IMPOSSIBLE to audit these systems. This shows a huge lack of planning.". Congratulations for the lack of imagination, it makes it a lot easier for people to feel safe while there's actually a huge load of backdoors still around their nicely planned audits did not cover because their imagination is so low they couldn't even conceive them.

Re:the work involved..

[jimicus]

It says right there that he added code to the C compiler. The backdoor itself did not exist in source code until someone compiled the login command. At that point, the compiler added code to make a backdoor. The original hack (messing with the C compiler) WAS IN SOURCE CODE FROM THE BEGINNING.

Yes, but the compiler was also hacked to make sure any recompiled C compiler did the same thing. Think of it as a compiler-level virus, if you like.

Thompson was taking advantage of one simple thing : once you've bootstrapped yourself a C compiler totally from scratch - without an existing C compiler to start with - you don't generally do it twice. Chances are the source code to the compiler that actually gets used is itself written in C and so requires a working C compiler in order to compile.

So you can safely remove your backdoors once they've been compiled into the system compiler and they'll be all but undetectable.

Re:the work involved..

[Craig Ringer]

Personally, I've given up removing employee logins here. I rarely even lock them. We seem to have a revolving door, where employees leave and return all the time. Full timers leave, then return as casuals years later. Casuals leave, and years later re-appear as full time staff. The people responsible don't think this is in any way noteworthy, and don't give any warning when someone who's ostensibly left the company will be re-appearing. They just expect the user's login to still work. In that sort of environment, there's not much you can do exept sigh, sent a "not my problem" written warning to the boss, and forge ahead as best you can.

Re:the work involved..

[Chris Mattern]

The hack would have been in the source code for the compiler.

Originally, yes. But you don't have that source code. You have sanitized source code--which when you compile it, your subverted compiler binary will insert the trojan back in. So recompiling your "clean" compiler source code still gets you another subverted compiler.

Re:the work involved..

[arth1]

As for it being impossible to audit all systems to this extent. Seems to me that something is seriously wrong if it is literally IMPOSSIBLE to audit these systems. This shows a huge lack of planning. What the hell did you plan to do if an admin went rogue? If you can't do a routine security audit then it's really just a matter of time before something goes horribly wrong

Do you have extensive plans for what to do if a secretary goes rogue? Or a CFO? With full post-termination audits when they quit? Do you have full audit plans for your vehicle fleet in case a mechanic goes rogue?

One problem with a pre-planned audit is that it may have been written or re-written by the same persons you are afraid of having compromised the system. Another is that for it to be of any value whatsoever, it can't be full of gaping holes that the admin would also know about. It would also have to be examined and if necessary modified for every change done to any system. I'm not sure you understand how monumental a task this would be.

Then there's the auditing itself. As a 20+ year sysadmin and 10+ year IT security professional, I would require several days per system to do a meaningful audit, if I were given full access to any and all source code and documentation, and full cooperation of all managers. Having an incomplete procedure in place would not make this go any faster. Multiply by number of systems, no matter what it's used for. Add audits of things you don't normally think of, like printer and router firmware and stored procedures in databases -- both prime places to hide malware.
Then there's weeding out all the false positives you will get. Lots of small fixes, kludges and updates that never get recorded even if there's a policy for it. Never mind apps that change the name or directory of where they save internal files. Or the impact an IP change can have on systems.

A cron job that deletes a whole bunch of files is more likely to do a benign housekeeping job than anything else, but you don't know that until you go through the executable and the data it manipulates with a very fine comb. You can't leave it alone, because it may be doing a benign job and a bad one. You can't trust time stamps. You can't even trust compilers. Or the kernel.

Yes, an IT professional has a potential for doing a lot of damage. But slapping on incomplete audits isn't going to prevent that. At best, they could find enough to raise a red flag that one or more systems are compromised, to an unknown extent, at an unknown time, and by unknown people. But things might very well slip through the cracks. And you're guaranteed to disrupt production while doing the audit, else it isn't an audit at all.

Is the price worth it, to alleviate some of your fears? Cause fear is all it's about in most cases -- without a credible threat, there's really no reason to suspect the leaving IT guy of sabotage any more than there's reason to suspect a leaving plane mechanic of sabotage. Yet do you recall and audit every plane the mechanic has worked on when you let her go?

Re:the work involved..

[Kidbro]

Of course there was source for the hack at some point. However, this source "disappeared" (i.e. was reverted) after having been compiled once. Subsequent recompiles (of login, or the compiler itself) by an already contaminated compiler propagated the hack.
In practice, there was no way to get rid of it without compiling the compiler with a compiler that was known to be uncontaminated - something you had no easy way of verifying (or even suspect that you would need to verify).
Remember that at some point, you need to start with a binary (compiler) that you simply have trust (well, at least in practice - in theory you can build your own computer from the scratch with twigs and bubble gum), and unless you're God himself, that binary was probably built by Ken.

Re:the work involved..

[BitZtream]

Yes, because the fact that no one was actually trying to find a back door back then might have something to do with it.

Hiding something in plain site is only impressive once people realize their supposed to be trying to find it.

It wasn't till after that event that people actually stopped thinking of all code as 'trusted' code and started actually looking at what things did.

Ken Thompsons Sendmail back door wasn't impressive, no one was looking for it.

Re:the work involved..

[cerberusss]

That does seem a bit strange to me unless you do not allow any external access like VPN, and the building is physically secured. If not, is the company data worth anything anyway?

Re:the work involved..

[Craig Ringer]

External access is - thankfully - locked down fairly tight. Otherwise I wouldn't mention it in any public forum, it'd be a big flashing "hack me, I'm stupid".

Users have IMAP and SMTP (both TLS, both requiring client certificates) and key-only sftp. That's it. There's no password-only auth for external access, it's all based on crypto.

Because users who "leave" the company tend to and doing patches of work for the company every now and then - and often do so remotely - I can't even assume that a user who's quit won't be working from home as a short-term casual six months down the track.

I find it's a remarkable incentive for keeping really good backups.

Re:the work involved..

[techno-vampire]

The original hack (messing with the C compiler) WAS IN SOURCE CODE FROM THE BEGINNING.

The point of the hack, you stupid little twit, was that after creating the hack and compiling it, he removed the hack from the source code because it wasn't needed any longer. That means that there was nothing left in the source code to show that the hack existed.

Next time, read your own article, moron.

I did, you slimy pustule on the face of Slashdot and, unlike you, I understood what it said. Schmuck!

Re:the work involved..

[TheRaven64]

But only if you compile the compiler with the compromised system compiler. A simple way around this is to compile the compiler with two different compilers, then compile it with both of the compilers that you've just compiled. If the resulting binary is not the same, one of the original compilers was compromised.

It's been a good few years since I read the trusting-trust paper, but I seem to recall that this was the technique that it recommended. It's not so important when writing a new compiler, because it's incredibly difficult to write a compiler that will correctly modify any compiler that you compile with it. Even detecting that something is a C compiler is pretty difficult to do automatically. Something that correctly corrupted pcc, gcc, and llvm+clang would be amazingly hard to write, especially if it needed to corrupt them in such a way that they'd then pass on the corruption. Doing it in pcc in code that pre-dated LLVM would be next to impossible.

Duh...

[Captain Centropyge]

Considering many IT staff have full control over your infrastructure, it's good to take the cautious route of assuming they've planted a back door or some other problem. After all, they likely had the root passwords to your systems. Better get those changed ASAP!

Three words

[pjt33]

Dead man's switch.

Re:Three words

[frinkacheese]

It's great for a bit of extra consultancy work when you have been made redundant too.. Walk out and guess what, a week later things break and you're on $1000 a day fixing it ;-)

But really, the best thing to do is to treat your IT staff properly in the first place.

Re:Three words

[Ironhandx]

This.

I've worked in a highly stressful environment before where I didn't know if I was going to still have a job the next day or not. I had everything set up sufficiently complex but still for good reasons, that if they had fired me getting someone else to fix it would have been a nightmare and cost them a fortune, which they would find out as soon as they tried to get someone else to go in and fix it.

Since I left on good terms I overhauled everything before I left and took out most of the non bog standard bits I had implemented. They ended up with a slightly worse but fixable in a pinch system.

Had the work environment been less stressful I wouldn't have felt it necessary to go through all of the trouble, but they decided to make it that way, so I decided to build some security into my job that was otherwise nonexistant.

Re:Three words

[CharlyFoxtrot]

But really, the best thing to do is to treat your IT staff properly in the first place.

This. I don't understand why it's so hard to grasp for some organizations. Pissing off IT is like telling your mechanic he's an asshole while he's working on your brakes. Sure most are consummate professionals but sooner or later you'll hit on one that isn't and then there'll be hell to pay.

Re:Three words

[blair1q]

But really, the best thing to do is to treat your IT staff properly in the first place.

This. I don't understand why it's so hard to grasp for some organizations.

Organizations learn slowly, and often by having their cost-saving measures (aka laziness) blow up in their face, then they overcompensate and kill efficiency.

The correct answer is "trust but verify,", aka "internal controls." You don't let one of your accountants sign your checks, so don't let your admins do anything without cognizance and review from another admin. Then it takes two people conspiring to screw you over, and if they both know it's better for them to catch the other screwing you over, you win.

Re:Three words

[Anonymous Coward]

This.

I've worked in a highly stressful environment before where I didn't know if I was going to still have a job the next day or not. I had everything set up sufficiently complex but still for good reasons, that if they had fired me getting someone else to fix it would have been a nightmare and cost them a fortune, which they would find out as soon as they tried to get someone else to go in and fix it.

Since I left on good terms I overhauled everything before I left and took out most of the non bog standard bits I had implemented. They ended up with a slightly worse but fixable in a pinch system.

Had the work environment been less stressful I wouldn't have felt it necessary to go through all of the trouble, but they decided to make it that way, so I decided to build some security into my job that was otherwise nonexistant.

This is still an extremely unprofessional thing to do. What if it breaks while you are on vacation? What if something happens to you? What if you get mono and can't work for three months? What if you get in a car accident and are in the hospital for months? What if your code gets audited and you get called out for writing shit code?

Re:Three words

[Cramer]

Except some (ex-)employers would just as soon sue you instead. Terry Childs aside, this happens far more than you read about in the news.

Re:Three words

[Anonymous Coward]

Only in Russia

http://news.slashdot.org/story/10/08/24/225247/UVB-76-Broadcasts-New-Voice-Message?from=rss

Re:Three words

[Cramer]

I'm sorry, but that's the a**hole way of running a network... make the place unnecessarily complex so you're the only one who knows how any of it works so "they don't dare fire me." That rarely works out well -- and often encourages firings. Having been the replacement and consultant called in to sort it all out, I support the death penalty for such people.

Re:Three words

[HungryHobo]

No matter how much code auditing you do a determined employee can bury something nasty in any large and complex system.

In my last workplace there was no shortage of legacy systems with no owners.
Anyone really determined could have embedded something into one of them during a breakfix and there would be no reasonable chance of anyone finding it.

I found enough problems caused by some old bits of code looking for a server that had been end of lifed that I doubt anyone would bat an eyelid if one screwed up a major database or grabbed all the disk space on some really critical server and caused some extremely expensive failures.

come to think of it they could have had many annoyed past employees and we never considered it could be intentionally malicious.

Re:Three words

[drsmithy]

I don't understand why it's so hard to grasp for some organizations.

Because even after multiple demonstrations otherwise, upper and executive management cling tightly to the fantasy that experienced mid-level+ IT (and other) staff are generic and can be disposed of and replaced at will, with essentially no loss to productivity.

Re:Three words

[PitaBred]

If they cared about that shit happening to him, they would have treated him better. What goes around, comes around. They aren't treating him well enough to care.

Re:Three words

[nickdwaters]

If you are that paranoid about keeping your job, find another job. Life is too short.

Re:Three words

[Mr. Freeman]

Everywhere I've been inserting complexity to ensure job security is the number one (or at least in the top 5) way to find yourself without a job. Making something intentionally complex to the point that only you can fix it is unprofessional and, at least in the case of engineers, unethical. The only reason these firings are done without cause as opposed to for cause is because it's more paperwork if you're actually fired for being unprofessional.

Re:Three words

[hot soldering iron]

A friend of mine once did that. He created a program back in the 80's that turned a 2 hour job into a 2 minute job, complete with making it easy to update as forms changed. This job was done millions of times a year, and under the then-current rules he would get a percentage of the money saved over 7 years. Did I mention that this was for the U.S. DOD? He was happy as hell, he was going to be a millionaire!

Until some bean counter said no, they were just going to take it "for the best interests of the country".

His DMS was pretty much undetectable. It wasn't completely in any one program. It had part of it embedded in one programs "dead code" that never executed, and then chained into some other program which called a couple generic libraries, and eventually would wind up deleting major databases and password lists.

He definitely qualified as "disgruntled", but not stupid. Before the trigger date rolled around, he sent it the "permanently dis-arm and dis-assemble" code.

Really happened? Maybe. Good story? Definitely. Will any PHBs learn anything from it? Doubtful, except maybe "screw your programmers mercilessly, they'll chicken out of retribution."

Re:Three words

[Anonymous Coward]

That's why the Bobs gave him a promotion!

Re:Three words

[Evil Shabazz]

Indeed. In my experience, the folks who talk about making systems "so complex only they know how to fix them" don't actually really know what they're doing anyway. The real truth is usually that they've got things set up so batshit crazy trying to hide their mediocrity in this "you can't fire me now!" excuse.

Re:Three words

[helixcode123]

If you are that paranoid about keeping your job, find another job. Life is too short.

Besides, it's exactly the opposite approach to being a successful consultant. Any decent consultant provides their client with a "here's how you fire me" file with all of the information they need to access and maintain the system(s) you've built. The idea here is to do such a good job for your client that they want more, not less, of you. If you can't do this you have no business being a consultant (or general employee, for that matter).

Re:Three words

[Anonymous Coward]

Hmm. I once setup a special logout that would remove every trace of me from their system; I know for a fact that they had illegal copies of expensive software and some questionable data. I didn't remove it. I simply removed any connection between me and them.

When I was sacked, security escorted me back to my desk to pack up and I was allowed to logout. And only that, no other typing, I was watched like a hawk. Luckily, I had already replaced that command with mine, which immediately put the correct one back while it was busy doing other things.

Zap. No data deleted - just every trace of me. Since it was only traces of me that were gone, it would trigger no alarms.

My only reason was to protect myself and disassociate myself from them. Not revenge. Revenge is an indulgence for the wealthy, powerful, and petty. I'm none of those things.

Was that unethical?

Re:Three words

[Anonymous Coward]

If they cared about that shit happening to him, they would have treated him better. What goes around, comes around. They aren't treating him well enough to care.

How would you know, you only heard one side to that story?

Its quite intriguing to see how most people with mod points here seem to easily side with the sob story of an IT guy ('they treated me bad!') without even wondering about the details.

Let me spell it out: They treated me bad! I ask for a vacation, and they always deny it at first. FOR 3 YEARS!. SO I got back at them. Question: "So when did you apply for your vacations?". Unknown to us: "Always 1 day before I want to leave, why?". OR: "As it should; at least 4 weeks up front, I always apply 2 months before".

If this is an indication of common IT staff I can see a lot of problems...

Re:Three words

[kiwimate]

Wow...

I've worked in a highly stressful environment before where I didn't know if I was going to still have a job the next day or not.

Life is too short to put up with that amount of stress. You should've been job hunting.

I had everything set up sufficiently complex but still for good reasons, that if they had fired me getting someone else to fix it would have been a nightmare and cost them a fortune, which they would find out as soon as they tried to get someone else to go in and fix it.

Wow, again. So the client is really screwed if you end up in hospital with pneumonia for two weeks (I pick that example because it happened unexpectedly with one of our developers within the past 12 months). A professional sets things up so they are easy to maintain and trusts in his ability and skill to get jobs, based partly on that.

Since I left on good terms I overhauled everything before I left and took out most of the non bog standard bits I had implemented. They ended up with a slightly worse but fixable in a pinch system.

So out of the generosity of your heart, and because you left on good terms, you decided to magnaminously grant them a bad system rather than an utterly broken one. Wow...yet again.

Had the work environment been less stressful I wouldn't have felt it necessary to go through all of the trouble, but they decided to make it that way, so I decided to build some security into my job that was otherwise nonexistant.

Next time, don't go through all that trouble to sabotage a client's systems. If it's that bad, just do your job properly and take "all that trouble" to instead look for another job. And try building some security into your job by being professional and really good at what you do.

You are the kind of consultant who gives consultants a bad name. Thanks for nothing.

Re:Three words

[lrichardson]

Yes and no. I've done so flashing-star, how-the-heck-did-you-get-that programming, mostly because of a unique position that straddled various corporate silos.

Two killers, i.e. 'making them so complex only ...'

1/ Not having the time to clean stuff up. If it works, management generally wants you to move on to the next fire.

2/ Documentation oversights and assumptions. "Check the syslog for errors" doesn't cover what to do when errors arise. I'd reached the point of coding the automated sending of e-mails on errors - with the fix included - to the person running a job, on dozens of issues. Things that one just assumes after years of experience are complete show-stoppers to someone who doesn't have that same experience. And it only shows up when someone else does try and run something, per the documentation.

&, of course, 1.5, not having the time to do any documentation ...

I like automating the heck out of stuff, handing it off to some poor schlub to run as needed/scheduled, and moving on to the next problem. But I also recognize that it's done me out of a job a couple of times. Which really, truly sucks.

The best advice I received from a friend was "Don't make yourself indispensible. You won't get vacations."

It's a trade-off. I think I prefer being viewed as a valuable asset, getting new challenges, rather than the only guy who knows how to fix something.

Re:Three words

[dbIII]

What about actually applying some reading comprehension skills to the portion quoted? Take note that things were not deliberately complicated but ended up that way to solve problems.
Arcane performance tweaks by people that know the stetup backwards are quick while well documented proceedures designed for newbies take time to develop. You can aim to get there in the end, but the above post appears to be about what would have happened if things were stopped part way through.

Re:Three words

[Tuidjy]

Amen to this.

I still have my job, and have never bothered to install back doors. But I am think about moving to a different position/geographical location, and am trying to get rid of all my hacks and cludges so that my replacement can have an easier time. Let me tell, with both of us working on this 2-3 hours each week, we are nowhere close to getting rid of all the crap.

Just a simple example, of which we got rid last week. In 1997 when I had been just hired, my company was in the process of changing its ERP software. The problem was that they had a front end to it that had been written by an outside contractor whom they had fired. He did not put backdoors or anything, but no one had realized that the front end would not work with the new SQL based solution.

Because the problem was dropped in my lap, I ended up hacking together a really ugly, brute force solution - watching the front end server process for disk access requests, putting it to sleep, and creating the old style file on the fly. Thirteen years later, the company owner and two of his close friends who head two of our 50 warehouses refuse to use any other front end. So until last week, I had a compiled program with full access to the main ERP database, to the payroll's server physical disk and to a modem. Good luck finding that.

And yes, I realize that it was a terrible thing to leave active for more than a decade. But seriously, who remembers to go back and work on something like this unless it breaks? The only reason it's gone is that I am trying to tidy things up before I move... and if I was not moving within the company, I doubt I would be so nice.

Re:Three words

[tsm_sf]

If you can't do this you have no business being a consultant (or general employee, for that matter).

That's a best-case scenario, and you should know it. There are plenty of jobs or projects out there where you will never be given the time it takes to "do it right." If you're the kind of person who's willing to spend their own time documenting systems then more power to you, but most of us don't want to work for free.

Look, just ask yourself if the unbillable time you're spending is making someone else money. That's the metric you need to keep in your head all the time.

Re:Three words

[Anonymous Coward]

Nah. Look for procedural exploits you can use later.

For instance, I know of one major retailer where the cash registers and the inventory/ordering system don't talk very much to one another. I'd be possible to order a bunch of items, then not pay for them. Instead, buy the items off the shelf. Call back a week later say "Hey, I never got my order!" Manager looks in Inventory/ordering system, sees order. Manager looks in Register system, sees purchase. Manager ASSumes the register transaction was for the order, and manually forces the order through. A few days later, you get the ordered items, and return the ones you bought.

No logic bombs or dead-mans switched needed. Just a broke-ass procedural system that, quite frankly, won't be fixed anytime soon.

Re:Three words

[Anonymous Coward]

its not even a very good deterrent because the employer was, at least i assumed, unaware of this fact....

else they would have most likely terminated him/her

Re:Three words

[Antique Geekmeister]

You've left out number 3:

Being completely forbidden by your manager, or the client, from doing it the faster, cheaper, and simpler way in favor of some approach they're more familiar with, and having to work around the crazy in-house architecture they've already deployed and lack willingness or political capital to throw out.

Re:Three words

[jasonwalls]

Most business owners/managers have a better relationship with their mechanic than with their IT people. And why not, the Mercedes (insert any other prestige vehicle here if desired) parked in the MD's parking spot is considered a far more valuable asset to the business than IT. At least that's my exerience.

Re:Three words

[CrashandDie]

Look, just ask yourself if the unbillable time you're spending is making someone else money.

Sure it is, but if you've worked out a good relationship with your boss, or if you negotiated your package right, all that should swing back in your bucket. That's how my previous gig was (infosec consultant); I would work insane weeks, over 90 hours a week in the worst cases, but I either got it back in double as holidays, or healthy financial bonuses.

My bonuses equaled my salary at the end of the first year, at the end of the second year, my bonus were 3 times as high as my salary.

There's working like an idiot, and then there's knowing how much your work is worth.

Re:Three words

[Anonymous Coward]

You have clearly never worked for someone who wants everything for free and doesn't negotiate.

Extra hours? 'You're on a salary, it's expected'

On Call? 'It goes with the territory'

Call out? 'If we paid you call out then we would have to pay everyone call out'

Pay rise? 'Given the current financial conditions I'm afraid there is no pay this year/last year/next year'

I no longer work over time, answer my phone to my boss outside of the hours I was contacted for etc etc.

I still do a good, professional job when I am there - just don't see why I should go above and beyond anymore...

Re:Three words

[Krneki]

Exactly, if you don't give a shit about your employers, don't expect any love in return.

Re:Three words

[Bert64]

In which case they terminate your contract, and replace you with a cheap monkey...

Re:Three words

[Lanir]

You're correct, this is unprofessional.

However... professionalism is basically another word for respect. People talk about giving it out for free like it's this magical thing that makes everything better and exists in a vacuum. The reality check here is that respect and professionalism largely amount to the same thing in business dealings and while you can get some on loan, the interest if you default on it is going to be pretty steep. Think "rip off credit card" here not "bank loan".

You can never really force someone to respect you. Even trying to do so basically tells them you don't give a damn about what they may really think or feel about you and so denies them respect in return. And neglecting to respect someone else is the surest way to make certain they won't respect you in turn.

In the real world, corporate professionalism is a lot like a fad diet. It does nothing, goes nowhere, and if it's not full of bullshit to start with, it will be by the end. It's mostly about looking like part of the herd.

Re:Three words

[h00manist]

The correct answer is "trust but verify,", aka "internal controls." You don't let one of your accountants sign your checks, so don't let your admins do anything without cognizance and review from another admin. Then it takes two people conspiring to screw you over, and if they both know it's better for them to catch the other screwing you over, you win.

That works to an extent. You are not needing to trust anyone, it may become an environment where nobody trusts anyone. Plus it will double or triple labor costs. I prefer finding good people, creating a good environment, and assuming I can trust people. Just keeping a good nose for bad apples that eventually appear.

Re:Three words

[Ironhandx]

Yes, I actually agree. I likely wouldn't have left it that way even if they had fired me but as I said, there were actually legitimate reasons for all of it. The thing was this company was so F&^Ked up that they would on occasion attempt to have me replaced while I was still there, even though I wasn't doing anything wrong, keeping everything in tip top shape, being polite etc, but they would try to replace me with someone that wasn't going to cost them as much money, and we're not talking like even a $10/hour savings, just a $3-5/hour savings.

Sometimes sysadmins have no choice. When you're getting dicked around you have to fight dickery with more dickery.

IT is probably the least appreciated field in the world as well, and its getting worse. Sure some folks get to work for good companies, but when your job security is basically on the whim of the boss that day, not because they don't need you, but because they think they don't(and end up paying double in external fees after they can you, then try to hire you back(didn't happen to me, but did happen to a friend of mine))... you're almost forced into it.

Re:Three words

[Ironhandx]

Nothing was purposely sabotaged, it was just complicated enough that they couldn't replace me with your average straight out of college/university student. Most networks reach this complexity pretty fast anyways.

As for this situation, its never going to happen again. Some of it happened because I didn't have time for documentation etc... some of it needed a solution to work around some old internal software they were using. In the end just to get everything to a point where someone could take over relatively easily I had to leave some of my custom code in place. The particular piece of software was running clients on VT220 emulators, was written in BASIC and had 3 layers of emulator on the server when I started there just to get it to work. Sure it was rock stable, which was why they hadn't replaced it, but it was a bit of a pain in the ass. When I left the whole situation was much better.

As I said, there were legitimate reasons. One of them being I didn't have time to fix and move everything over from the ARCnet that was installed to ethernet.

Basically, the whole thing was complex enough by necessity of them not wanting to change out that piece of software that in order to make it fixable I essentially just made them a "recovery cd" that will format the current server, or another replacement server(to a point, in 5-10 years the replacement if its new would have driver issues) and set everything up as it was, grabbing data from a usb drive to catch their system up to the most recent backup. I simplified how the server was running as well so that if the recovery disc doesn't work they're looking at a much smaller bill than they originally would have been.

Re:Three words

[ihatejobs]

A lot of people here are IT staff in one sense or another, its pretty common for other readers to side with the IT workers like themselves instead of the bosses / company. You can't make a proper judgement without both sides of the story, but that's never stopped anyone from passing judgement anyway.

Re:Three words

[tehcyder]

But really, the best thing to do is to treat your IT staff properly in the first place.

Short of giving everyone jobs for life, this is impossible.

Re:Three words

[tehcyder]

It's people like you that mean companies have to adopt hardline attitudes to staff.

If your work environment is too stressful, do something about it (like getting another job), don't plan on sabotaging the company. Even ignoring professional ethics, it may rebound on you even though you think you left no traces.

Re:Three words

[ultranova]

This is still an extremely unprofessional thing to do.

Professionalism goes both ways. If you keep your employees guessing whether they'll still have a job tomorrow, they'll keep you guessing whether you still have a system tomorrow. Why would you expect to get more than you give?

Re:Three words

[Ironhandx]

I couldn't have left any traces, most of my trouble involved working with what was already there rather than going a step above my required duties and replacing it. They did get a much much more easily maintained system when I left than what they had when I got there.

Even if they had fired me halfway through and I didn't have time to fix up a few things before I left they would have had less of a mess than when I started. There just wasn't much in the way of documentation for some patches.

Did I mention they had a custom piece of software that was critical that was written in BASIC? oh, and partly in a french version of BASIC and partly in english? running in a BASIC emulator that the english coders dug up somewhere that would support both.

Re:Three words

[ultranova]

Because even after multiple demonstrations otherwise, upper and executive management cling tightly to the fantasy that experienced mid-level+ IT (and other) staff are generic and can be disposed of and replaced at will, with essentially no loss to productivity.

Upper and executive management cling thightly to the fantasy that everyone below them is serfs and they, the Chosen Ones, have been divinely appointed as nobility. They cling to this fantasy because how would they justify laying off half the low-level employees to rise their bonuses otherwise?

This is the basic problem of all organizations: at some point those who are at the top will start thinking they deserve to be at the top, at which point they start throwing their weight around, then inventing more or less idiotic justifications for that. It typically ends with catastrophic failure of the organization.

Re:Three words

[Custard Horse]

Paid subscriptions to World of Warcraft?

Re:Three words

[hesaigo999ca]

I have to agree here, in my experience, especially lately, we have so many legacy systems that once replaced with new .net architechture ends up running faster and usually less down time...if they take the time to do it...but all the decision making is made by the owners, and if it ain't broke, why fix it....well they never see the gains they COULD make, even if i tell them 15 times in a row, they still do not want to spend money on where they should....yet they can make 15 million profit at the end of the year and pat themselves on the back...

Is it broken if they were able to make 15million, at this point i have learned to just suggest then move on, if ever they come back to say hey, you never told us, i keep a log of all emails i sent, and also if in the off chance they ever agree with updating the system, i have kept a trail of all links referencing needed materials for that update...

Re:Three words

[Evil Shabazz]

I found the best way to survive the scary lay-off periods is to position yourself as the go-to guy/gal for fixing other peoples' batshit crazy setups. If your management thinks of you as capable of getting into unfamiliar systems, figuring them out, and repairing/improving them - they're not letting you go anywhere, except on vacation to keep you happy.

Re:Three words

[GameboyRMH]

Congratulations, you're a scumbag. Also, news flash: Nobody has good job security these days. Deal with it.

Re:Three words

[GameboyRMH]

Doing a half-assed job when your employer is screwing you is understandable, but going out of your way to make things difficult if you're fired is all kinds of wrong.

Re:Three words

[GameboyRMH]

Nothing was purposely sabotaged, it was just complicated enough that they couldn't replace me with your average straight out of college/university student. Most networks reach this complexity pretty fast anyways.

Of course they'll still try, and that student will have to try to fix your mess on shit pay with no job hunting options.

Re:Three words

[Ironhandx]

Of course they will. My mess is 100x better than the mess that was originally there. The whole setup is already running smoothly though and the student would be able to fix it easily if it wasn't for that one piece of crap I had to keep working.

At least the kid won't have to go buy a BNC crimper and connectors like I did at first. He may however accidentally make a bad enough tangle of the system that they have to upgrade it, which is a net positive in my opinion.

Whats left there in the way of a mess is the fault of the business owners not wanting to change, it has very little to do with me.

Re:Three words

[TheRaven64]

Even if it keeps you in a job, it also has the effect of keeping you in the same job that you're currently doing. When management is looking for someone to promote, they're not going to promote the person who is indispensable in his current job...

Re:Three words

[blair1q]

Sure it costs more. But less than the alternative, which is losing all your data and the confidence of your customers and prospects.

Re:Three words

[kalirion]

While it may stop normal folks from becoming assholes, it won't prevent existing assholes from adding some revenge code.

Re:Three words

[infinite9]

You cannot create an adversarial relationship with your employees, then expect them to treat you like anything other than an adversary.

Re:Three words

[infinite9]

My bonuses equaled my salary at the end of the first year, at the end of the second year, my bonus were 3 times as high as my salary.

After spending 20 years in IT, I would argue that this is the exception and not the rule. I've been promised bonuses 9 times in my career and received it once. Always the excuse was company performance. I did great... it just wasn't in the budget this year. sorry.

Re:Three words

[Anonymous Coward]

Sometimes sysadmins have no choice. When you're getting dicked around you have to fight dickery with more dickery.

You always have a choice. You may not like it, but you can always leave. There is never a valid reason to be a dick. You are only making a mess for yourself, your coworker, and anyone who comes along after you. Unnecessary complexity is not job security; the company will fire you when they want to fire you. They don't understand what you do, so why would you expect them realize you're the only one who can deal with the mess?

IT is under appreciated because it is mostly not understood. If you do your job right, no one knows you've done anything at all. It's only after you're not there doing the things they don't know you do that they learn what isn't getting done anymore.

I'd say treat it like a DR drill

[BobMcD]

If you're seriously considering this as a possibility, I'd say treat it like a DR drill. Burn everything down to bare metal and restore only the data. It's the only way to be sure...

However, before taking my advice, I'd suggest you get your boss to sign off on it, whichever way. Present a list of options from 'ignore it' to 'burn everything' and have them pick. This way, whatever happens, you're covered.

Re:I'd say treat it like a DR drill

[Locke2005]

"I say we take off and nuke the entire site from orbit. It's the only way to be sure."

Re:I'd say treat it like a DR drill

[Meshach]

If you're seriously considering this as a possibility, I'd say treat it like a DR drill. Burn everything down to bare metal and restore only the data. It's the only way to be sure...

That seems a bit risky. I cannot see any manager worth his salt giving authorization to purposely destroying data "to see if the backup works".

Re:I'd say treat it like a DR drill

[BobMcD]

If you're seriously considering this as a possibility, I'd say treat it like a DR drill. Burn everything down to bare metal and restore only the data. It's the only way to be sure...

That seems a bit risky. I cannot see any manager worth his salt giving authorization to purposely destroying data "to see if the backup works".

That's because the order of operations is out of whack.

Rebuild, then cut over. Same result, less risk.

Sorry for glossing that over.

Re:I'd say treat it like a DR drill

[BobMcD]

Just the movie I had in mind, yep.

Re:I'd say treat it like a DR drill

[Fulcrum of Evil]

You don't start with 'burn the building down'. You start with restoring to a backup set of hardware and doing basic validation, then work up to milton style DR by steps. Besides, backups are never the problem - it's the restores.

Re:I'd say treat it like a DR drill

[DigiShaman]

Backups are easy, restores are not.

1. How do you know everything was backed up properly to begin with? Are you sure those logs are being interpreted correctly?

2. Almost every small to medium size businesses do NOT perform disaster recovery drills. Running a test restore of small data is one thing, but a full scale DR drill is quite another task entirely. If they do, it's because they have extra hardware to test them on and/or can spool up a VM. Again, were not talking about fortune 500 companies here with a dedicated IT department.

3. If the restore fails and data is lost, it's your ass on the line. Do you want to come in as a rookie (new hire) and suggest restoring servers from backup? Bad idea regardless what your boss thinks. As IT staff, we don't look after the network but rather the entire operation of the company. The moment the mail server goes down, that bat-line of yours will ring off the hook. Would the entire company and staff be willing to take that risk?

Re:I'd say treat it like a DR drill

[fishbowl]

If your last IT manager had to be fired, you may have months or years of work to do before you can actually do that DR-bare-metal drill.

Re:I'd say treat it like a DR drill

[BobMcD]

You're not suggesting any alternative that I can see. I'll just assume you're advocating the 'head in sand' approach, and assert that the new IT guy cannot afford that risk either. The boss makes the call, or you walk. Better to be job hunting than to be sacrificed when the ousted guy attacks the network, in my opinion.

To answer your concerns, however:

1) It needs testing, period. Further I'm absolutely not advocating recovering everything. Data only. Reinstall the apps and platform by hand.

2) They do not, but they should. Most, if not all, are required by Federal law to do so. Also your false dichotomy where only fortune 500 companies have IT departments is upsetting. If for no other reason, are we not discussing a company with a dedicated IT person being dismissed and replaced? The stock trading angle is moot, and you ought to know it.

3) As others have point out, I left out the necessary order of operations. Build new servers, and THEN burn the old ones. I should have been more clear... And finally, when the ENTIRE PLATFORM GOES DOWN, due to the attack we're assuming will happen, will that same phone not also ring? Outages are what we're hoping to prevent...

Re:I'd say treat it like a DR drill

[Jason_D_Berg]

It's absolutely the best option to tear everything out and start over again. It's really hard to make a business case to spend the time and labor when all you have is a hunch. The reason I asked this question to begin with is because most C level staff don't see the justification. Active Directory, Exchange, Sharepoint...they're all pretty big beasts. If risk can be mitigated and proper backups are in place, I don't see a reason to tear apart the IT infrastructure.

Re:I'd say treat it like a DR drill

[fishbowl]

>That seems a bit risky. I cannot see any manager worth his salt giving authorization to purposely destroying data "to
>see if the backup works".

We do it routinely, but it's not chaotic or risky like your choice of words makes it sound. OTOH we have invested a lot of money and brainpower into getting the redundant system we need to have in order to fail over a production system, tear one down, build it up again, verify it and put it back into production. That costs money... and probably not something the IT manager that had to be "fired under duress" actually accomplished.

Unless you can deploy your standard configuration with nothing but the LTO tape from Iron Mountain and a charge account at your server vendor, you don't have a Disaster Recovery plan. (A fire in our facility probably takes out 4 city blocks. We seriously take this under consideration, and we do drill for it.)

Re:I'd say treat it like a DR drill

[BobMcD]

Maybe, maybe not. Hard to say. What if it he or she tripped a zero-tolerance?

Re:I'd say treat it like a DR drill

[BobMcD]

Well, if the C's won't support it, then you're off the hook. But it still needs to be on the table.

If you can't see the reason, then you've never rooted a system... A proper back door can get you the resources you need to do just about anything at all. And even complex systems need a mechanism by which they can be restored. Maybe you don't do it all in a day, sure, but you ought to spend time knocking them out one-by-one at the very least.

Re:I'd say treat it like a DR drill

[fishbowl]

>Do you want to come in as a rookie (new hire) and suggest restoring servers from backup?

I did :-)

My premise was correct: The backups that they had, were mostly useless.

One of the very first things I did was to establish a backup regime, including offsite storage, nearline rotations and so on, and every new hire in IT learns how it works, how to verify what is backed up, how it is retained, how it is restored, and how to recover a server or a workstation with one of several standard configurations.

Lost data for a single day could expose the company to regulatory liabilities, and could severely impact customer business to the tune of grounded aircraft (commercial and military) so it's a pretty big deal.

Coming in as a non-management role, or as a new guy in a shop where the procedures are already very sound and operating well, that's different. Then I have to wonder why the last guy got fired...

Re:I'd say treat it like a DR drill

[Lehk228]

and the data being restored contains a buffer overflow exploit that reroots the new system

Re:I'd say treat it like a DR drill

[CharlyFoxtrot]

If you're seriously considering this as a possibility, I'd say treat it like a DR drill. Burn everything down to bare metal and restore only the data. It's the only way to be sure...

However, before taking my advice, I'd suggest you get your boss to sign off on it, whichever way. Present a list of options from 'ignore it' to 'burn everything' and have them pick. This way, whatever happens, you're covered.

That takes care of backdoors in the OS assuming you run everything stock. But if you're running custom in-house software that might have backdoors too so it'll still need to be audited. I wouldn't recommend this approach except for the most extreme cases anyway. Best is just to keep a log of all installs and changes from stock and have outside auditors come in regularly to check for anything that can't be accounted for as well as audits of all installations. Not to mention having a strictly enforced change management process.

Re:I'd say treat it like a DR drill

[fishbowl]

>Maybe, maybe not. Hard to say. What if it he or she tripped a zero-tolerance?

No doubt. Or just didn't take the offer after an acquisition (I've _been_ that guy.)

Re:I'd say treat it like a DR drill

[DigiShaman]

Your answers to these questions are good and I agree. I should clarify however that I'm viewing this scenario from the standpoint of an MSP (managed service provider) that provides outsourced IT services.

In my field, I can't tell you the number of times a company will call looking for a network admin to replace a previous MSP or that employee fulfilling the IT role in-house. Of those companies, it's SMBs that mainly look to MSP outsourced IT staff. Knowing that, the idea of getting called in the nth hour to replace rouge IT staff is very frustrating. We literally go into an environment with little to no documentation and zero understand of the layout. Prior to making any changes, the entire system would need to be scoped out and audited. Simply performing a BMR from backup would be out of the question unless no other option exists or they specifically request it. However, reviewing backups via test restores of data is perfectly acceptable and encouraged. Another angle is cost. Our time is billed at the hourly rate. How serious are they about security and will they match it with said funding?

So what's my solution in all this? Simple really. I would explain to the client the kind of security issues they face, how much it may cost at minimum and what future precautions they can take at mitigating being broadsided in the future by rogue IT staff. It's up to them internalize and make the risk assessment to cost. In the end, they may choose to do nothing about or not involve our services (we cost too much...etc). Sad, but true.

Re:I'd say treat it like a DR drill

[Mr. Freeman]

Standard practice mandates that backups should be tested regularly in such a way that will not interfere with current operations. Only a moron would delete everything BEFORE seeing if the backup works.

Re:I'd say treat it like a DR drill

[BobMcD]

I would explain to the client the kind of security issues they face, how much it may cost at minimum and what future precautions they can take at mitigating being broadsided in the future by rogue IT staff. It's up to them internalize and make the risk assessment to cost.

This...

Re:I'd say treat it like a DR drill

[elmick]

I'd say treat it like a DR drill. Burn everything down to bare metal and restore only the data..

The budget for DR testing much be HUGE in your company.

Re:I'd say treat it like a DR drill

[BobMcD]

Why? You can conduct such a drill on old-ish hardware with existing staff. And further, you NEED to test your DR anyway, or you in essence simply do not have a working plan.

little OT....

[Anonymous Coward]

One of many reasons CEOs are given golden parachutes are to keep them quiet about trade secrets and certain contacts. Whether or not that happens is debatable, but discretion is basically paid for.

Why not give similar parachutes to IT admins to follow these unwritten practices? If the CEOs are the frontmens, ITs are the infrastructure of the organization. Treat them like gatekeepers instead of disposable footmen. They have the keys to the castle. And all the secret entrances.

Re:little OT....

[CharlyFoxtrot]

One of many reasons CEOs are given golden parachutes are to keep them quiet about trade secrets and certain contacts. Whether or not that happens is debatable, but discretion is basically paid for.

Why not give similar parachutes to IT admins to follow these unwritten practices? If the CEOs are the frontmens, ITs are the infrastructure of the organization. Treat them like gatekeepers instead of disposable footmen. They have the keys to the castle. And all the secret entrances.

The janitor has all the keys to the building and the cook could poison everyone if he wanted but those people aren't afforded the respect they deserve either. CEO's are given golden parachutes by their buddies who they'll see at the golf club and who they can maybe return the favor later on the board of some other company. We're just staff and staff don't get golden parachutes, they get concrete shoes.

Re:little OT....

[mysidia]

But George, don't we have the firewall for that?

The Janitor has the keys to your building too, and your most sensitive offices.

But they are just disposable, what you do when they leave is really simple.... you make them hand in their badge, so they can no longer get through the front gate.

The IT admin equivallent is the changing of the passwords to the firewall.

And manually verifying the firewall is indeed properly configured so no access to the inside network is possible, except through authorized personnel's VPN credentials.

Then you force all VPN users to change their VPN passwords immediately.

Re:little OT....

[b4upoo]

Giving benefits to people according to the potential harm that they could do is not right according to me. Bosses that take that attitude might want to consider what harm some low level employee could do with a bomb or guns. The lowest guy in the food chain could easily kill off upper management. So who gives the floor cleaner or the gal Friday a golden parachute? Or is it only financial loss that must be prevented?

Re:little OT....

[antifoidulus]

The reason golden parachutes exist is because the sec has repeatedly dismantled shareholder rights in order to guarantee political kickbacks from said CEOs. The CEO now essentially owns the company and spends more time figuring out ways to enhance their own salary than they do actually trying to do any real work. Stocks have gone nowhere in the past decade while CEO salaries have skyrocketed.

Re:little OT....

[rsborg]

One of many reasons CEOs are given golden parachutes are to keep them quiet about trade secrets and certain contacts. Whether or not that happens is debatable, but discretion is basically paid for.

No, with your example, lots of individual contributor types would wield a lot of power and get golden parachutes... the reality is that the CEO is powerful enough to command respect from the company, and that's the only reason (s)he gets the golden parachute.

Re:little OT....

[Lehk228]

and the extra ethernet wire running to a cantenna'd WAP lets the bad guy back in

or the main database server accepts a normal looking connection from a normal employee workstation, that isn't a workstation but actually is a wall wart computer mounted inside the wall masquerading to the network as a workstation.

Re:little OT....

[Monkeedude1212]

The fundamental difference between IT and most of the other Staff positions is that often IT is also the ones who deal with security. We set up the cameras, we check the logs, the punch-in punch-out are sometimes our responsibility. We have to deal with nearly everyone on an almost personal level, whereas the janitor might only deal with 1 building, work after hours, and not have access to the computers. The cook couldn't possibly get away with murder.

On top of that, IT is not something everyone is versed in, so in order to catch an IT criminal it requires an IT expert. Meanwhile, Janitorial duties can be shovelled off to just about anyone (I've seen it happen to receptionists), and catching other vulnerabilities are usually as easy as looking at the camera records.

With a higher up IT member, he can set off Logic Timebombs, Dead man switches, sniff email traffic, root critical systems, tunnel the firewall, create admin accounts, reset passwords, etc etc etc - There's a whole smorgasborg of options open to an IT member that simply can't be reversed by anyone other than another person in IT, who has to not only fix the problem but also determine its causes in order for any kind of evidence to point towards the originally terminated member. This is if you even determine that something went wrong, Like I mentioned with an email sniffer, or if they get access to the database, they can be selling secrets and data for pennies to any of your competitors, while you operate under the impression that everything is running smoothly.

Yeah, IT is usually treated like Staff - and then you get cases like Terry Childs popping up.

Re:little OT....

[mysidia]

and the extra ethernet wire running to a cantenna'd WAP lets the bad guy back in

That's why enterprises install Wireless LAN Controllers which detect and kill rogue APs.

or the main database server accepts a normal looking connection from a normal employee workstation, that isn't a workstation but actually is a wall wart computer mounted inside the wall masquerading to the network as a workstation.

That's why switches have 802.1x support, and only actual workstations in AD have a SSL certificate can authenticate, send packets through the switch to the intranet, and have the IPsec packets to the DB server accepted by the Windows 2008 Network Access Protection policies.

Thanks to NAP, things that aren't a member of the domain, can't pretend to be a workstation and talk to any of the servers, even if physically plugged into the LAN.

Re:little OT....

[Anonymous Coward]

The golden (or at least bronze) parachute would be a great idea. It would save a lot of headaches. I know of at least one company that REALLY wishes they had done it.

Re:little OT....

[mjwx]

The janitor has all the keys to the building and the cook could poison everyone if he wanted but those people aren't afforded the respect they deserve either. CEO's are given golden parachutes by their buddies who they'll see at the golf club and who they can maybe return the favor later on the board of some other company. We're just staff and staff don't get golden parachutes, they get concrete shoes.

Redundancy packages are golden parachutes for ordinary employee's. In Australia if an employer outsourced a department they are required to give X days per month working at the company (I think it's about 7 or 10). But if something untoward happens to the company due to a dead man switch or back door then the ex-employee can be sued for that redundancy package and some. Perhaps even criminal charges. Companies can take the dodgy route and fire everyone for incompetence by giving them an impossible task and sacking them for failing to complete it but this will generally nullify the companies right to claim against workers and open them up to unfair dismissal claims.

Re:little OT....

[Anonymous Coward]

But it depends on the business. If you're a storage warehouse, the janitor will the the important person (or the point of failure). The IT admin at an online business will be the critical point. But if you just have an IT admin at a business that relies mostly on salesmen and marketing (in other words, can rely on *paper* docs as backup) then IT won't be as crucial. In other businesses, the CFO is more important than the CEO.

I'm not saying that IT or janitors are of equal importance everywhere, but it depends on the business.

And not to be nitpicky here, but I stress on the *admin*, not the IT assistant or the guy that fixes the secretary's email font problems.

Re:little OT....

[hedwards]

If you think that's fucked up, imagine private security. Spend you're time being screwed over and reporting to pretty much everybody else including janitorial, then leaving. With the amount we see it's a wonder there isn't a lot more interest in securing our loyalty rather than expecting to scare us into being silent. My former employer definitely can't count on my silence now that I'm no longer on the pay roll, and I definitely said things to a few people on the way out that was pretty embarrassing to the company. But, I have documentation so they can pretty much just kiss my ass. My unemployment insurance application was quite the embarrassment to them as well.

The trick though is to document the problems and document some more. That way if they try to retaliate against you at least you've got some evidence to cover you're backside.

Re:little OT....

[Mr. Freeman]

"One of many reasons CEOs are given golden parachutes are to keep them quiet about trade secrets and certain contacts."

No. The only people with more money and power than the CEOs are the companies that the CEOs left. If a CEO were to leak trade secrets then the previous company's legal team would tear him to shreds, rebuild him, and then do it again. The reason they're given golden parachutes is so that they'll leave without being fired (and because they expect to be paid for simply taking a shit). No CEO is ever fired. They resign due to "personal reasons" or because "they thought that it would be best for the company", etc. And they're paid millions of dollars for it. That way they can get another job being the CEO of another company and the process repeats every few months.

Re:little OT....

[Mr. Freeman]

"that relies mostly on salesmen and marketing (in other words, can rely on *paper* docs as backup)"

Are you insane? Let's assume for even a second that they can actually rely on paper for everything that happens from the IT meltdown forward. There's still no way they can access the previous records. Has this customer paid? Did this customer order anything? Someone just called in, they said that their package hasn't arrived, what was the tracking number? Email is managed through IT, which the sales people can't access. This can be critical to communicating with customers and it does not look good when your sales inquiries start getting bounced back because their email server is down. In some places even the phones are voip and can be taken out by a rogue admin. Business can still grind to a halt when the computers go down even if the company isn't web based.

Re:little OT....

[Mr. Freeman]

It doesn't matter how well documented it is. If you disclosed something like trade secrets or something that an NDA forbid you from disclosing then your ass is in the fire regardless of what documentation you have.

Re:little OT....

[Legion303]

"They resign due to "personal reasons" or because "they thought that it would be best for the company", etc. And they're paid millions of dollars for it. That way they can get another job being the CEO of another company and the process repeats every few months."

See also: Carly Fiorina.

Re:little OT....

[tehcyder]

We're just staff and staff don't get golden parachutes, they get concrete shoes.

This thread seems to be full of people who work for organised crime.

Re:little OT....

[CharlyFoxtrot]

This thread seems to be full of people who work for organised crime.

I work for a bank, which I guess is close enough.

Re:little OT....

[TheRaven64]

$20m to get rid of Carly was very good value for HP.

Multiple Backdoors

[Bryansix]

I usually put in multiple backdoors. Not out of malicious intent but because I support customers who are so far away that I don't want to drive out there all the time. Now this might include software or even out of band management, VPN, etc. Basically, if you put yourself in a position where you have to fire your IT staff then you are a moron. Always do background checks because you are going to be giving these people the keys to the city.

Re:Multiple Backdoors

[Kozar_The_Malignant]

Basically, if you put yourself in a position where you have to fire your IT staff then you are a moron. Always do background checks because you are going to be giving these people the keys to the city.

• Not every problem employee comes with "Crazy MF With Drug Habit" tattooed on his forehead.
• Sometimes people lie when you do background checks. They want their problem to become your problem.
• Your IT guy might be just fine until his wife leaves him for a younger woman who also works for your company.
• Or, like my experience, the first thing you have to do in your new job is fire the sadistic moron that your predecessor tolerated for years.

The point being, you don't always "put yourself" in that position. Sometimes shit happens.

Re:Multiple Backdoors

[Belial6]

Thinking that a background check is going to protect you is naive at best.

Re:Multiple Backdoors

[Monkeedude1212]

All of those problems could be handled in a variety of ways with a competant HR department.

Re:Multiple Backdoors

[Anonymous Coward]

Your IT guy might be just fine until his wife leaves him for a younger woman who also works for your company.

And then the IT guy frantically spends all his time trying to work himself into a threesome?

Re:Multiple Backdoors

[greenbird]

All of those problems could be handled in a variety of ways with a competant HR department.

Isn't that an oxymoron, even if it was spelled correctly.

Re:Multiple Backdoors

[Anonymous Coward]

Your HR can make the IT guys wife not leave him? That is truly competant.

Re:Multiple Backdoors

[Anonymous Coward]

Basically, if you put yourself in a position where you have to fire your IT staff then you are a moron. Always do background checks because you are going to be giving these people the keys to the city.

You really can't think of a reason why sometimes you have to let people go? What an ideal world you must live in. You also assume that a problematic person will have exhibited problematic behavior before you hire him. Truly a utopia. Mind if I come by for some tea and biscuits to admire this wonderful world you live in? Surely, the garden of Eden must pale in comparison.

Re:Multiple Backdoors

[mysidia]

The difference between OOB management and a malicious backdoor, is OOB management is documented very clearly, as essential documentation for accessing the system.

And it's on the "list of security sensitive services" that need to have creds changed when an admin leaves

If the backdoor is not documented, and nobody else is told about it, then you have a problem (potential liability on your part).

Re:Multiple Backdoors

[ben_kelley]

I usually put in multiple backdoors.

IANAL but I hope you have a good lawyer and this is in the contract with your customers.

Re:Multiple Backdoors

[Anonymous Coward]

The parent definitely needs some funny mods.

Re:Multiple Backdoors

[khb]

Sadly this sort of thinking even survives into shipping products.

There are appropriate two factor authentication solutions. http://www.phonefactor.com/ as an example. I've never worked with that company, but a quick search hit them first.

There are better ways to arrange for remote access than hardcoded backdoors. It's one thing to use the technique for Grandma's wireless accesspoint and another to leave it in a production IT environment.

Re:Multiple Backdoors

[dissy]

I usually put in multiple backdoors. Not out of malicious intent but because I support customers who are so far away that I don't want to drive out there all the time. ,/quote>

My logic is, as long as my boss (The owner) knows about this, then there is no problem.

If I was to do it on my own initiative, and suddenly pop up later with "Oh don't worry, I can take care of that from here!" and it would be a surprise to him, then there is a huge problem.

As things have turned out, I seem to hold myself to higher standards than anyone else at the company holds me, but none the less I've always been straight forward and honest with my boss of my intentions, so by definition nothing I have done will ever look shady in the future to the only people whom matter.

A 'backdoor' that the owner doesn't know the details of but does know of its existence is not really a backdoor, it's just an additional front door with an alternate security path.

Re:Multiple Backdoors

[dissy]

Bah, apologies. My shift key dove away from my finger at the last minute.
Reposted as to actually be readable:

I usually put in multiple backdoors. Not out of malicious intent but because I support customers who are so far away that I don't want to drive out there all the time.

My logic is, as long as my boss (The owner) knows about this, then there is no problem.

If I was to do it on my own initiative, and suddenly pop up later with "Oh don't worry, I can take care of that from here!" and it would be a surprise to him, then there is a huge problem.

As things have turned out, I seem to hold myself to higher standards than anyone else at the company holds me, but none the less I've always been straight forward and honest with my boss of my intentions, so by definition nothing I have done will ever look shady in the future to the only people whom matter.

A 'backdoor' that the owner doesn't know the details of but does know of its existence is not really a backdoor, it's just an additional front door with an alternate security path.

Re:Multiple Backdoors

[Anonymous Coward]

Is there such a thing?

Re:Multiple Backdoors

[sco08y]

Your IT guy might be just fine until his wife leaves him for a younger woman who also works for your company.

And then the IT guy frantically spends all his time trying to work himself into a threesome?

Someone's got to take care of the "Multiple Backdoors".

Re:Multiple Backdoors

[Anonymous Coward]

So when organised crime walks into the system using YOUR backdoors, you are responsible. Clever.

Remind me not to hire you.

Re:Multiple Backdoors

[Bryansix]

Two factor authentication is awesome. However its just not implemented very frequently in turnkey systems. Nable for instance is an enterprise level hosted solution that provides remote access and monitoring. They don't implement two factor authentication that I know of. However they do use two passwords. One for the Nable account and one for the actual system you are remoting into.

Re:Multiple Backdoors

[hedwards]

Not really, HR is generally highly competent, just not at what you think they're there for. Most companies have HR employees specifically so that they can be useless and make it as hard as possible for employees to get there benefits, preferably quitting before they're eligible. Sure it's a dumb way to run a business, but it happens. Usually if there's any corruption in a company it's found in HR first and spreads elsewhere.

Re:Multiple Backdoors

[InfiniteZero]

Sometimes it's just part of a "strategic" business move that nobody has any control over.

I'm involved with an investment firm that has a wholly owned subsidiary company (one of many). This coming Monday, they are going to fire EVERYONE in the company, including IT, the CFO, and the CEO, and move assets and operation to another city. Pool performance is the leading cause, plus other factors.

Here is the fun part. While the big shots at the investment firm have been meticulously planning this for months, nobody at the subsidiary company has a clue about their imminent termination, not even the CEO. (Especially the CEO. He will be literally escorted out of the facility Monday morning, while everyone else will be offered to stay for another month to make the transition.)

And I'm supposed to move in with a team Friday evening, bypass everyone including IT, and (hopefully) secure a copy of all data on the network -- before they make the official announcement.

Boy, is it going to be a fun ride. The upside is that it's a small company (~30 people), so it's not unmanageable, and there will be a severance package for everyone. Nevertheless I'm going to shop for a bullet-proof vest in the next couple of days. Dead serious.

Bottom line is, will I trust anything on that network after Friday? No freaking way.

P.S. I'm posting under this account because it's work-safe. Feel free to dig around my profile if you are paranoid (that it might be YOUR company), but you are not going to find anything. It's too late to worry anyway. (And I didn't even mention which country/countries this is happening in.)

Re:Multiple Backdoors

[Anonymous Coward]

... until his wife leaves him for a younger woman ...

Um ...

*feeling blood rushing to head*

Re:Multiple Backdoors

[tehcyder]

Most companies have HR employees specifically so that they can be useless and make it as hard as possible for employees to get there benefits

You are a clueless twat who probably does really, really badly at interviews, and has accumulated an enormous chip on their shoulder from being told politely by HR people to fuck off and practise some people skills.

Or I may be just making shit up, just like you did.

Re:Multiple Backdoors

[Anonymous Coward]

I am sure you are careful with this, but I have seen people get into deep caca by leaving themselves backdoors. Usually one of three things happen, if the backdoor isn't well documented to the client:

1: Someone else finds the backdoor and uses it as an entry point for an attack. Forensics comes in and blames it on the consultant. Result, possible criminal charges because the investigators will not be looking for a hacker, they will be focusing all their time on digging up anything convict-able on the consultant [1].

2: Another consultant comes in, finds the backdoors goes schizo and slanders the previous admin, alleging that the backdoors are there for nefarious reasons.

[1]: You know Joe Sixpack, the guy who didn't have install media for his Windows box, and who bought a new computer because he didn't know or care about reinstalling clean? The same guy who doesn't understand the difference between the Internet, E-mail, and the World Wide Web? Now picture 12 of these guys, all pissed as hell because they are all losing money, trying your fate. This is why I don't consider a consulting job complete until I have a pile of documentation handed to the client about everything done. If I want a way back in, I document it well, down to the SSH RSA key ID.

Re:Multiple Backdoors

[Bryansix]

Organized got the passwords or broke the encryption how exactly? I see why you post as anonymous.

Re:Multiple Backdoors

[hedwards]

Nope, sorry, but what other explanation is there for HR employees to lie, make shit up, cheat workers out of their promised benefits? They're basically there to cover up for whatever the company wants covered up.

Well...

[Anonymous Coward]

Of course the first piece of advice is to basically assume you've been rooted. Ouch.

That's only painful if you didn't have well thought out policies in place beforehand.*

*for everything but the edge cases, of course.

Make your list prresent it to your stupid boss

[shoehornjob]

who doesn't have a clue what you're telling him and watch him veto this because his budget would take a hit. Make notes of what you discussed save emails etc for evidence when said evil admin hacks in and trashes your servers, domain etc. In other words cover your ass.

Re:Make your list prresent it to your stupid boss

[boristdog]

Just make sure to CC your boss's boss when you do this.

THEN your ass is covered!

Re:Make your list prresent it to your stupid boss

[shoehornjob]

lol so true

If they used Macs

[Anonymous Coward]

It should be pretty easy to find and explore their backdoors.

Alot of software opens holds due to poor codeing a

[Joe The Dragon]

Alot of software opens holds due to poor coding as well.

And look at printers and Vender pc's running RIP software likely on a os that lagging behind on updates but the Vender does let you / says we will void the printer contract over messing with the software / os on the RIP PC.

1. Drink Heavily

[jgtg32a]

2. ???
3. Profit

Use different HW, don't changing working HW

[perpenso]

If you're seriously considering this as a possibility, I'd say treat it like a DR drill. Burn everything down to bare metal and restore only the data. It's the only way to be sure...

To elaborate on this idea I would emphasize that the existing and working hardware is not touched, ideally at least. Use a new/different system (your backup/spare hardware - which should be tested anyway and isn't this a good test?) or maybe a new virtual machine. Once the OS and apps are restored from trusted sources, the data is restored, and its verified that all is well then replace the original hardware. Maybe the original hardware now becomes the back/spare for the next machine to go through this process.

Re:Use different HW, don't changing working HW

[BobMcD]

Yes, definitely.

Re:Use different HW, don't changing working HW

[arth1]

A big problem is that production has to be halted from the start of making the backup until the restore is complete. And no, you can't use replicating hot standby machines to avoid this, because you can't trust the replication software.

Then, repeat for each system. With just a few dozen boxes, this can become very expensive.

Never mind all that will break because of custom executables and scripts that the systems depend on, and which won't survive a data-only restore. And that's the good case, where the former employee hasn't done anything wrong.
In a worse case, you may have encrypted data, and a heavily obfuscated front-end binary that contains both the on-the-fly encryption/decryption as well as a dead man's switch.

Re:Use different HW, don't changing working HW

[perpenso]

A big problem is that production has to be halted from the start of making the backup until the restore is complete. And no, you can't use replicating hot standby machines to avoid this, because you can't trust the replication software.

A system may or may not need to be halted during the backup process. Since the originals are unmodified they can be restarted if some problems or delays occur during the restore so the down time seems manageable. Also some of the work on the replacements can be done in advance, for example installing operating systems, applications and local tools/scripts under version control. It seems that data is largely what needs to be restored during the downtime, maybe. It may be that the data is from network storage outside the system(s) in question so restoring the data may not even be an issue. YMMV, It all depends on the overall architecture of the environment.

Then, repeat for each system. With just a few dozen boxes, this can become very expensive.

Never mind all that will break because of custom executables and scripts that the systems depend on, and which won't survive a data-only restore. And that's the good case, where the former employee hasn't done anything wrong. In a worse case, you may have encrypted data, and a heavily obfuscated front-end binary that contains both the on-the-fly encryption/decryption as well as a dead man's switch.

The problems you describe justify the expense of replicating a working environment on different hardware. A data recovery process needs to be debugged just like anything else. The point of the test is to make sure that you can recover from backups *when* the day comes that hardware does fail and the restore is the real thing rather than a test. If custom software and scripts exist this needs to be discovered and the source code and scripts need to be under version control and be part of the backup.

Re:Use different HW, don't changing working HW

[arth1]

A system may or may not need to be halted during the backup process.

Whether the system can be kept running during a backup is largely irrelevant for this purpose -- any work that includes writing or creating data on the system has to halt, because it won't be part of the backup that is restored.

Re:Use different HW, don't changing working HW

[DamnStupidElf]

It depends on the rate of change. If I have a 1 TB database that only makes a gigabyte of transaction logs every day, I can pretty easily back up the entire database, restore it to a new machine and replay the transaction log, then bring the production system down, dump and apply the latest transaction log to the restored system and bring everything up on the new machine in a shot period of time. Any highly available system will have concepts similar to this. Back up the main data and the differential changes separately so that production downtime is only necessary during the period when the very last differences are being applied to the main data set.

Re:Use different HW, don't changing working HW

[arth1]

I thought I had already addressed this, and stated why I believe it isn't valid: you can't trust that the replication system hasn't been compromised, so if you do that, you risk propagating not only data but the malware you want to get rid of too. Like time bombs in stored procedures.

Re:Use different HW, don't changing working HW

[DamnStupidElf]

There are a couple scenarios. If your existing data backups are compromised, then there's really no point in continuing. You need to hire someone to clean everything up for you. This applies even if you only suspect the latest day or two has been compromised; you need to stop production processing until you can verify that the existing data matches a known good backup and that changes since that trusted backup can be verified.

If you trust that the data in your system hasn't been modified (and there's no real reason to believe this if you're worried about logic bombs or dead man switches), then you can trust the differential changes as well.

Ultimately, there is no way to certify that everything in a data set is correct and the result of an authorized action without full review. If your system is designed with this liability in mind, then it should be possible to assign a level of trust to each transaction with the use of cryptographic signatures in the data. Then the data can be restored using the fastest/best method from a compromised system and verified cryptographically on the newly built trusted system. That moves the liability to the cryptographic system which is arguably the right place to deal with matters of authentication and authorization. Even if IT can control all the servers and databases, they should never have control of the signing keys of individual employees. With HSMs, you can make that a reality. Obviously most places don't need this level of security and couldn't afford it, but it is possible.

Non evil stuff may look like logic bombs and if yo

[Joe The Dragon]

Non evil stuff may look like logic bombs and if you don't keep track of all of it. How knows what hacks and work around that you will fine and taking them out may just lead to have to call old guy back just to find out how some of the stuff works.

how meny times do you have have the old come back at X2 X3 X4 times the pay to just to work out stuff that only the people who got layed off know about?

Two words

[Sycraft-fu]

Prison sentence.

Seriously trying to do something like install a dead man switch to fuck over your employer would be the height of stupidity. Wonderful way to end up with a sentence that make the Child's thing look lenient. While I realize that pedantic geeks think they could cover their tracks that isn't the case. They don't have to prove it was you beyond any and all doubt, they just have to prove it was you beyond a reasonable doubt. If they can show means, motive, and opportunity, they've gone a long way to that.

Sounds like the real answer if for companies to get rid of egomaniac assholes in IT before they are in a position to cause trouble.

Re:Two words

[DigiShaman]

I would recommend subjecting all IT staff to a psychological evaluation test. Myself included. Who wants to work with egotistical assholes? I sure don't. I love working in a non-abusive collaborative team environment.

Re:Two words

[Anonymous Coward]

You are seriously delusional if you believe you can.....

A) Find it. Before or after activation.
B) Atribute it to a specific employee.
C) Even recognize that it was malicious and not just a bug, glitch, human error or outside attack when done properly.

You lack imagination.

Re:Two words

[Nrrqshrr]

With all due respect. Someone working in IT and with a reasonnable level of sense and intelligence would probably know how to make the results look a bit more... satisfying? Psychology tests are the most rigged things made to date. They work with illeterate people and rather badely rised kids, for the others, they don't work.

Re:Two words

[blair1q]

But that's where we put the egotistical assholes to keep them out of the rest of the building...

Re:Two words

[Peach Rings]

You could easily just badly document or fail to document passwords and configuration info and stuff. As long as you're around and working with the systems daily, everything runs smoothly. If you get fired, there's confusion with the new guy and your memory fades... it's not like they can really tell exactly what isn't a matter of the new guy not being up to speed for weeks. And you're not responsible for giving them consulting services for free after they fire you. If they can't figure out the non-standard port numbers you used, then that's their problem.

Childs took an idiotic stand where he admitted he knew the passwords and refused to hand them over. That's not the most lenient case, that's the worst case I can think of other than destroying data.

Re:Two words

[timeOday]

Sure, it would be dumb to do. But it does happen, thus it is a legitimate security concern.

As for preventing problems by firing anybody who's going to do something wrong before they do it, good luck. Even Stalin wasn't 100%, and not for lack of trying.

Re:Two words

[Requiem18th]

Did you hear *woosh* over your head? That's the sound of missing that he was proposing revenge for being terminated with extreme prejudice. If you are dead, you don't have to worry about being jailed.

If they fire you without firing AT you, that's good reason to kindly warn them to remove the DMS.
All of this of course, as a joke.

Re:Two words

[Znork]

Sounds like the real answer if for companies to get rid of egomaniac assholes in IT before they are in a position to cause trouble.

Just be careful that the companies policy for getting rid of egomaniac assholes doesn't mean fast-tracking them for management.

Of course, the downside with that might mean missing out on the next Bill/Steve/Larry level CEO material...

Re:Two words

[Anonymous Coward]

That's a personality test you're talking about. I don't know about you, but there are some people I instantly dislike, as soon as I lay my eyes on them I can't stand them, sure I try to behave nicely, but I simply can't stand being around them. Others feel the same, around others or around myself, it's not something you can control, and I don't think it's a reason to not hire or fire someone.

As for the backdoors and such, if I would be in a position to do something like that I would resign long before I would get a chance to find out how things work. But I'll give you a few suggestions, there are certain things you do at work that happen only every few months or at the end of the year or like that, you sabotage those, insert a rm -rf /a/very/specific/file or add a script to change the date to and back again at certain intervals, so reports get messed up. Little things that make life hell, but are nearly impossible to find because you don't know they exist in the first place. Then there are always the backups, force them to restore from backup, with more treasures packed inside, or whenever they backup data you make sure it gets corrupted, not enough to be noticed, but enough to make them useless. All you need is a vivid imagination, and disregard for other peoples work and feelings. If you knew he wouldn't fit in, then why the fuck did you hire him in the first place? Oh right, he worked for less than the other guys ... well, you got what you paid for.

Re:Two words

[dangitman]

But psychological evaluation tests are almost completely worthless, so what would that achieve? Even if the tests did work, humans are not machines. People who are completely stable and sane today can easily become completely insane next week. The human mind is a very fragile thing.

Re:Two words

[X0563511]

You know what a dead-man's switch is, right? The joke he was replying to was that it was better to kill the employee than to fire.

The response was to build a dead-man's switch.

Hard to go to prison after a 9mm to the brainstem...

Re:Two words

[IICV]

Sounds like the real answer if for companies to get rid of egomaniac assholes in CxO positions before they are in a position to cause trouble.

There, fixed that for you. I bet you anything that my preventative will result in far less IT troubles than yours.

As another poster said, these twisted IT systems generally come about because IT doesn't know if they'll have a job tomorrow and need to have everything done yesterday. Get rid of those pressures, let them do their jobs, and you can let IT go without worrying quite as much.

Re:Two words

[fluffy99]

You could easily just badly document or fail to document passwords and configuration info and stuff. As long as you're around and working with the systems daily, everything runs smoothly. If you get fired, there's confusion with the new guy and your memory fades... it's not like they can really tell exactly what isn't a matter of the new guy not being up to speed for weeks. And you're not responsible for giving them consulting services for free after they fire you. If they can't figure out the non-standard port numbers you used, then that's their problem.

Childs took an idiotic stand where he admitted he knew the passwords and refused to hand them over. That's not the most lenient case, that's the worst case I can think of other than destroying data.

Even worse, he deliberately setup the routers so he'd have to manually reconfigure them if/when they rebooted - in other words a deadmans switch.

Re:Two words

[sabt-pestnu]

And as the Childs case points out quite clearly: if you're an IT admin, means and opportunity are part of the job description; and you only have to look at motive.

Which means that as an exiting IT admin in a hostile atmosphere, you need to document the state of the system with witnesses to have evidence proving that the logic bomb wasn't yours. Once you leave, you lose the access to evidence that would prove your innocence.

Jury: Hmm... 48hour shadow, rumpled clothes, yep, he's a villain.

Re:Two words

[Anonymous Coward]

Here's a fun little story, and only slightly relevant, too!

My company's HR head also controlled access to the network. We outsource our IT and the head of HR was the point of contact (the head of HR was also assistant to the CEO. It was one of the few positions in the company that wasn't procedurally isolated from other responsibilities).

Anyway, the CEO and President decide to clean house. The head of HR is fired, and the CEO goes in and changes passwords. The CEO, however, didn't have direct day-to-day focus on network issues (passwords, accounts, etc), and as a result didn't get every single account the ex-head of HR had access to. A day or so later, "someone" accesses the network using the one account that would have still be accessible the ex-employee. Using this account, "someone" went in and deleted a bunch of data from our servers, including data we were maintaining for over a year on a lawsuit that we were very tangentially connected to.

As far as anyone knows, no action or investigation will take place. I suspect the decision-makers just want this one to go away.

Re:Two words

[Anonymous Coward]

Sounds like the real answer if for companies to get rid of egomaniac assholes in IT before they are in a position to cause trouble.

... or maybe it sounds like the real answer is for companies to quit breeding such assholes out of their employees in the first place.

Re:Two words

[sco08y]

I would recommend subjecting all IT staff to a psychological evaluation test. Myself included. Who wants to work with egotistical assholes? I sure don't. I love working in a non-abusive collaborative team environment.

I tend to agree, but egotistical assholes come in handy when dealing with the someone else's egotistical assholes.

And I don't trust some psych eval to weed out abusive people, mostly because there are so many ways to be abusive. I especially don't trust some test to make a good determination for borderline cases, where someone is a little hard to work with but produces better than average code.

Re:Two words

[Anonymous Coward]

::sigh::

http://www.pcworld.com/businesscenter/article/149159-4/sorting_facts_from_fiction_in_the_terry_childs_case.html

Common Practices Portrayed as Nefarious

The documents filed by the city in opposition to Childs' bail reduction contained many vague references and claims of nefarious actions. But to those with experience in network administration, these activities seem like common practice.
For example, the documents portrayed the fact that Childs had configured some number of routers to disable password recovery as a subversive action, when it's common to use that function to secure routers and switches that cannot be physically secured.

Re:Two words

[zaphod777]

You guys didn't have backups? You should have weekly, monthly, and yearly backups going off site.

Why?

[antirelic]

Why assume that the employee is a criminal? Many people get terminated because of bad relationships with their managers every single day. Very few of those people resort to criminal activities against their previous employers, even if they have the ability to do so. I suppose everyone should suspect secretaries of publishing address books, bank statements, inventories, employee social security numbers, etc., all over the internet because they had access to that information all along. How about janitors? They go through garbage. How many things dont get shredded? Perhaps every business should conduct documentation accounting practices because who knows what the janitor might know.

Seriously. This is a bit over exaggerated. Most IT professionals have invested tens of thousands of dollars in their education and training, as well as years into a profession that doesnt really have any value outside of their relevant field. Treating every employee who gets fired as a potential criminal is stupid, and is a good sign that you do not want to work for that business. Everyone who ever works for a company has potential to cause damage to some degree... some employees more than others. But to treat your network as if that person has "rooted and back doored" it is just bad business (fairly disruptive too, considering in many cases its best to take some systems off line if you believe they've been compromised).

But to each their own.

Re:Why?

[Anonymous Coward]

well maybe you don't have to assume its rooted but its nice to make sure there are no bugs or inactive accounts. It's just proper security. Sure you can live your life and not lock your door to your car but one day you might find yourself carjacked.

If I left I would expect a competent admin to revoke all my access. That's just standard procedure.

Re:Why?

[Captain Centropyge]

The person may not be a criminal, but they may not have implemented good security practices. So it's good to look over what they've put in place anyway. There may be accounts they forgot about lingering, awaiting deletion. Extra equipment that could be put to use. Extra equipment you don't need. And changing passwords is just plain common sense.

It depends on why they've been canned. But we're assuming it was "under duress", so it's not a happy parting. If their relationship with the company was going south for some time, it's good to investigate a bit to ensure they didn't leave anything behind that might be a problem.

Re:Why?

[peacefinder]

There's no need of an assumption of criminality to realize that some basic precautions are in order.

When a janitor separates from a company they don't get to keep their keys to all the doors, because there's no reason for them to have it any more and no authorized use to which they could put it. (If it was okay for anyone to enter at any time, you wouldn't have a lock at all.)

When an accountant separates, they don't get to keep the company checkbook. Again, there is no reason for them to keep it, and no authorized use to which they could put it. (If it was okay for anyone to access corporate funds at any time, there's be a big box of money laying about.)

When an IT administrator leaves a company, they do get to keep the passwords, or at least whatever is in their memory. What they don't get to keep is access. Again, there's no reason for them to have it an no authorized use case. In practice that means changing all the passwords and certificates, because there is no physical object to be verifiably* returned.

[*: Of course there is a bit of a flaw in this: the janitor may well have taken an impression of the master key and could make himself a new one; the accountant could have written down all the relevant banking information. But there exist other controls - such as alarm systems and bank audits - to mitigate those threats.]
[: I typed in a WPA key for my previous employer's WiFi so often that now, fifteen months after they fired me, I believe I could still reconstruct it from memory. Which is irrelevant, as it turns out, because I had occasion to visit there a few months ago and my phone connected right up. They hadn't ever bothered to change the damn thing. Good thing I'm a nice guy, or I'd have some idea what *other* passwords they haven't changed. For instance, I'd say the chances are excellent they haven't changed root on their ESX hosts...]

Re:Why?

[Mr. Freeman]

"I suppose everyone should suspect secretaries of publishing address books, bank statements, inventories, employee social security numbers, etc., all over the internet because they had access to that information all along. How about janitors? They go through garbage. How many things dont get shredded? Perhaps every business should conduct documentation accounting practices because who knows what the janitor might know."

If you want to be secure, then yes. There is ALWAYS a risk when giving information to ANYONE. Anyone that knows information COULD disclose that information. This is why you have to go through lots of background checks and investigation in order to work on certain projects related to the government, so that you are found to be unlikely to disclose this information.
In a lot of places, there are policies to prevent employees from seeing information that they don't need to see. For example, there's no reason that a secretary needs to see the social security numbers of anyone. There's no reason that the accounting department needs to see the latest designs for some new product that's being launched, etc. In the past, companies have been compromised because janitors have stolen the backup tapes that were left where they could get them.

Now, the article isn't saying that EVERY admin that gets fired is a criminal. They're saying that IF you have reason to suspect that he is then you need to take some serious measures, which is true. If you have reason to suspect that your secretary/janitor/accountant is leaking information then you need to put a stop to it.

logic bombs on a timer

[ei4anb]

The worst timed logic bomb I have had to deal with was by an intern who was looking for more pay. He had written a statistical analysis program that would have started to introduce subtle errors several weeks after he had left. If I had not found it then our stats would have become useless after a few months of that mangling. I assume he was hoping we would notice data errors, panic and re-hire him to fix it without realizing that he had caused the errors. I became suspicious when the timestamp on the Java source was newer than the class file so I did some reverse engineering. He had edited the logic bomb out of the source after compiling.

Re:logic bombs on a timer

[jjohnson]

That's a really good catch. Well done.

Re:logic bombs on a timer

[grahamsaa]

He knew how to program a logic bomb and how to cover his tracks by removing it from the source, but he didn't have the smarts to change the source file's time stamp? Sounds like an obvious step to take -- not that I'd ever do anything like that, but seriously, changing a time stamp isn't rocket science.

Re:logic bombs on a timer

[Anonymous Coward]

You get interns that can code? I wouldn't trust the ones HR dumps on us anywhere near a computer.

But seriously, we would never let an intern work on a business critical system. Contractors are much the same.... none of them are allowed access to the code signing keys, and all code, written by employees or not, is reviewed by a security team. Our competitors did not see the value of doing this and are now dealing with internal theft of a couple million credit card numbers.

You can pay for security now, or you can pay for not having it later.

(name withheld by request)

Re:logic bombs on a timer

[twebb72]

The worst logic bomb I had to deal with was written similarly by an underpaid (debatable) programmer. He set it up so that when money was exchanged between accounts the program would then truncate the remainder. This, in fact, was only a fraction of a cent. Then he took that remainder (once it had accumulated a bit) and transfer it out into a bank account of his own. As it turns out, it was relatively easy to install.

We were so far behind for the Y2K updates, most people simply didn't notice. A couple days later the building burned down.

Re:logic bombs on a timer

[mysidia]

So where would you be if he had "touch"ed the .class file to have a believable timestamp, I wonder?

I am surprised a "make clean" or your internal equivalent and full rebuild wasn't done if frequently code was still being updated and fixed...

Re:logic bombs on a timer

[Anonymous Coward]

Wasn't that done in a Superman movie? And back in the 80's?

Do you get conjugal visits in pound-you-in-the-ass prison?

Re:logic bombs on a timer

[Anonymous Coward]

Dude I remember that guy. There was some dude who obsessed about his stapler, too.

Re:logic bombs on a timer

[Anonymous Coward]

(name withheld by request)

I didn't realize that as an AC I was expected to request that my name be withheld. I've been doing it wrong this whole time!

Re:logic bombs on a timer

[omglolbah]

Smart people are usually busted by stupid things ;)

Re:logic bombs on a timer

[blueg3]

Be even easier to modify the class file's timestamp. Wait a couple seconds, then touch it.

But yeah, people are generally caught by stupid things, because covering all of your tracks turns out to be trickier than it sounds.

Re:logic bombs on a timer

[Legion303]

That sounds suspiciously like the plot to Superman 3, where Richard Pryor steals a red stapler.

More like not keeping people who'd do that

[Sycraft-fu]

Seriously, it takes a rather large amount of egomania and lack of respect for others to consider doing something like that. Most non-sociopathic types just wouldn't do it. They wouldn't rig up something to damage their employer just on the off chance they ever got mad. Anyone who seems to be that kind of person, well show them the door before they have the ability to cause trouble.

While I fully agree employers should be nice to their employees treating it like a hostage situation where you can never do anything to disgruntle them, which in some cases means let them do whatever the fuck they want, isn't realistic.

Re:More like not keeping people who'd do that

[cjb658]

As an (ex-)employee, it would be to your advantage to maintain good relations with your previous employer anyway, unless you don't plan on ever using them as a reference.

Re:More like not keeping people who'd do that

[arth1]

Anyone who seems to be that kind of person,

And you have psychic powers that can tell you who that is?

What seems common for most people who "lose it" is that nobody saw it coming, and people who are asked tend to say how that was the last person they'd expect to do anything.

The only way I see you can judge a person to not do anything bad is to hire an imbecile incapable of doing anything bad. But then he or she would be incapable of doing the job too. Still, it doesn't seem to deter all companies. :-P

Re:More like not keeping people who'd do that

[nobodylocalhost]

If you are fired, i doubt you would ever want to use them as a reference anyway

Re:More like not keeping people who'd do that

[phantomcircuit]

Most employers will only confirm the dates you worked for them now, for fear of lawsuits.

Re:More like not keeping people who'd do that

[mjwx]

As an (ex-)employee, it would be to your advantage to maintain good relations with your previous employer anyway, unless you don't plan on ever using them as a reference.

If it gets to the point that you're installing back doors, you've passed the point of burning bridges already.

As a sysadmin, my own ethics are responsible for ensuring this doesn't happen. Unfortunately a lot of non-sysadmins that get high levels of access are not as scrupulous. Sysadmins have to fix the problems that occur from departure bombs and dead man switches, once you've done that you develop an extreme dislike of them.

Re:More like not keeping people who'd do that

[Anonymous Coward]

Or that you are looking for a job in another field where talk of your lack of professionalism may not have spread. What goes around comes around, and really, when you leave a job, what do you have? Your skill and reputation and reputation is all a potential employer can thoroughly check.

Re:More like not keeping people who'd do that

[InfiniteWisdom]

That's not necessarily true. Terminations can happen for a lot of reasons. Someone might have the technical skills but just not fit into the team. Or they might be good at some things, but not all of the things you need at the position they're in... they might do better at a different job more focused on the things they're good at. In a larger company you might just be able to move them to a different position, but that might not be an option at a small company.

Being fired hurts, and there might be a temporary emotional payoff to "sticking it to them", but unless there is some serious wrongdoing on their part, you'd probably want to leave on good terms, and burning your bridges would definitely be shooting yourself in the foot.

Re:More like not keeping people who'd do that

[butlerm]

Sysadmins have to fix the problems that occur from departure bombs and dead man switches, once you've done that you develop an extreme dislike of them.

I am personally aware of one circumstance where this happened, and all I can say is that a sysadmin who does something like this deserves three to five years in the nearest federal penitentiary, if not more.

Re:More like not keeping people who'd do that

[helix2301]

Yes but weather you are fired or quite you can still be subject to legal action for doing damage to there network.

pray he hasn't read Thompson

[ei4anb]

Some backdoors are hard to get rid of

Reflections on Trusting Trust http://cm.bell-labs.com/who/ken/trust.html

Re:pray he hasn't read Thompson

[trb]

Yep, exactly what I thought of when I saw this "backdoors" article. "Trusting Trust" was Ken's acceptance speech for the ACM's 1983 Turing award, and described hacking that he had contemplated before then (i.e., more than 25 years ago).

More golden parachutes probably a bad idea

[perpenso]

One of many reasons CEOs are given golden parachutes are to keep them quiet about trade secrets and certain contacts. Whether or not that happens is debatable, but discretion is basically paid for.

Why not give similar parachutes to IT admins to follow these unwritten practices?

Since golden parachutes have been a source of abuse and unintended consequences maybe the concept should not be more widely used?

FWIW golden parachutes are not really about keeping quiet regarding trade secrets, contracts and other material non-public information. Contracts, non-disclosure agreements and other legal tools already cover this area.

Re:More golden parachutes probably a bad idea

[dbitter1]

Golden parachutes can be effective if reasonably written.

For example, cutting all the legalese out of mine it waters down to "your non-compete is as long as your severance package of normal salary". Thus, they give me a year's pay of severance, I don't show up at my competitors door for a year. If the checks bounce, I'm there, and the NDAs say I can do it free and clear.

Having pissed off sysadmins because your employer is an ass is one thing, and I agree there is no reason to torment the keepers of the keys. However, the parachutes come into play when you work in an industry full of hostile and semi-hostile takeovers. It gives me security regardless if my boss is someone I trust as well as [satan|$evil_diety]. Someone wants to buy us out and kill our product because it is killing theirs, fine. I'll take my ball and play elsewhere. :P

Re:More golden parachutes probably a bad idea

[perpenso]

Golden parachutes can be effective if reasonably written.

For example, cutting all the legalese out of mine it waters down to "your non-compete is as long as your severance package of normal salary". Thus, they give me a year's pay of severance, I don't show up at my competitors door for a year. If the checks bounce, I'm there, and the NDAs say I can do it free and clear.

A no-compete clause is one thing but disclosure of trade secrets and other material confidential information is something quite different. The no-compete addresses your experience and the skills you have developed. Many states don't allow no-competes below the executive level, hence the severance pay tie-in to get you to do so voluntarily. However the proprietary knowledge you have in your head is never disclosable.

the zeroeth piece of advice...

[pedantic bore]

... don't hire sysadmins who act unprofessionally or criminally under duress, and then treat them like professionals, like everyone else.

I haven't seen any reason to think that IT staff would be more likely to do such harm than anyone else. Sure, maybe they have easier means to effect harm than your average employee, but they have no more motivation nor mind to do so.

My accidental SSH backdoor...

[Anonymous Coward]

I had to administer a system when the vendor's software would fail on the rollover for the day. So it would fail at 5 am, and I would have to be the one to come in to fix it. As it happens at least once every two weeks I started to SSH in to fix it rather than rush to work and have to work an extra three hours that day (and not be compensated for it). The policy that I fought to implement at work was to do a quick audit, change any passwords/keys for any remote entry and to actually create passwords for many of the accounts that did not have passwords. So done and done I thought.

To continue: I had many problems with upper management, one of which was their wanting me to 'tweak' time sheet accounting so that new entry level minimum wage employees were paid for as little as 75% of their legitimate hours worked. I thought this was particularly dickish as they fired employees on a project basis and anyone was usually fired within two weeks. So I quit and tried to get myself as good as a parachute as I could.

Well two weeks after I left I found out the newbie replacement didn't perform the audit when I accidentally clicked on a bookmark at home (Putty) and I was suddenly in a server from my old job. I logged out and didn't feel particularly compelled to tell them that my keys were still trusted. About a month later I made the same mistake. The hole was no longer there. I thought to myself, "Good for him. I guess he's not so incompetent at all."

But curiousity a la Facebook and Twitter revealed that a server had actually gone down that day. Apparently there was a 'rm -rf' oopsy!!!

The story continues, but the end result is that he managed to destroy three servers within a month of my leaving. If I had been malicious I don't think I could have caused that much destruction...

Re:My accidental SSH backdoor...

[frosty_tsm]

That's pretty awesome. And I can't think of a more deserving company who bypasses minimum wages like that.

Re:My accidental SSH backdoor...

[sco08y]

But curiousity a la Facebook and Twitter revealed that a server had actually gone down that day. Apparently there was a 'rm -rf' oopsy!!!

The story continues, but the end result is that he managed to destroy three servers within a month of my leaving. If I had been malicious I don't think I could have caused that much destruction...

The alternate ending is one where a data recovery company does an audit and finds your SSH keys, and the next day the cops have a warrant to search your computers.

When I leave a company, I want *nothing* pointing back to me.

Re:My accidental SSH backdoor...

[omglolbah]

Glad I'm not the only one thinking in such a way...

To quote "Ivanova" in Babylon 5...

"How russian of you sir, I approve"

Re:My accidental SSH backdoor...

[Anonymous Coward]

If they had logs of successful SSH connections, maybe. If all they had is evidence that no one had revoked my credentials after I left and they tried to search my computers, they'd be the ones in trouble. The fact that my successor failed to revoke my credentials is *not* reasonable suspicion of an attempt on my part to gain unauthorized access to the computer system in question. Plus if your private key is encrypted there's no way to verify that you actually have the private key needed to authenticate against the public key they think belongs to you, even if your computer was searched. Not that such a search wouldn't be terribly inconvenient, just that it wouldn't turn up an useful evidence.

Re:My accidental SSH backdoor...

[sco08y]

If all they had is evidence that no one had revoked my credentials after I left and they tried to search my computers, they'd be the ones in trouble. The fact that my successor failed to revoke my credentials is *not* reasonable suspicion...

Yeah, people often talk like that.

My feeling, though, is that if people are unreasonable and corrupt to try to charge you, why do you expect that you're going to slam down some evidence and cite some laws and all of a sudden they're going to acquire common sense?

I'm sure you'd win, but only after both sides have spent a lot of money on lawyers. Winning in court is never more than a Pyhrric victory.

Re:My accidental SSH backdoor...

[GreggBz]

I work for an ISP. I came up with this.

We have a dial in clapper that activates power to a cheap switch connected to a public facing SSH relay host. So, I call a number. It turns on the switch for 1 hour. I then log into this relay host on some port other than 22 with a pretty strong password. From there, I can ssh to our other servers. It's one point of access for most everything.

My boss is fully briefed on the system. If I leave, unplug the phone line, switch it to another number change the password and watch the logs.

Re:My accidental SSH backdoor...

[illumin8]

o continue: I had many problems with upper management, one of which was their wanting me to 'tweak' time sheet accounting so that new entry level minimum wage employees were paid for as little as 75% of their legitimate hours worked.

That's actually a criminal offense (payroll fraud, or something similar) and you should have called your state labor commission and reported them right then and there.

Well two weeks after I left I found out the newbie replacement didn't perform the audit when I accidentally clicked on a bookmark at home (Putty) and I was suddenly in a server from my old job. I logged out and didn't feel particularly compelled to tell them that my keys were still trusted. About a month later I made the same mistake. The hole was no longer there. I thought to myself, "Good for him. I guess he's not so incompetent at all."

But curiousity a la Facebook and Twitter revealed that a server had actually gone down that day. Apparently there was a 'rm -rf' oopsy!!!

I hate to say this, but you could have ended up in jail for this little "oopsy". If for any reason they thought the "rm -rf" wasn't an oopsy, and was malicious, and someone did forensics on the server and determined you had logged on after your termination date, you probably would have had the FBI knocking on your door.

Not only is it the employer's responsibility to shut down access for terminated or resigning employees, it is the employee's responsibility to destroy any company data they might still have. This includes ssh private keys that might give them access to company systems. It's always a good idea to cover your ass, especially when dealing with former employers that are unethical enough to short their own employee paychecks.

Re:My accidental SSH backdoor...

[MooseTick]

I wonder how much of that he blamed on you and how much they believed it.

Except in S.F.

[Anonymous Coward]

Don't so this if you work IT for the City of San Fransisco. They will seem to want to prosecute you for this.

Not to stand in the way of healthy paranoia...

[Target Practice]

... but if you go around assuming you've been rooted by everyone your company has let go, pretty soon your cycles will be consumed by constant self-evaluation. The result would likely be catastrophic money and time loss, akin to the South Park episode where San Francisco disappeared entirely up its own asshole.

So what is the advice

[bugs2squash]

for those that are terminated and have no intention of connecting back in ? After all, if I am let go, the last thing I want is for my old credentials to be used by someone to trash something and have suspicion fall on me.

Re:So what is the advice

[PPH]

Yeah. When I left Boeing, they kept a couple of my logins active for several years. I guess they figured I was coming back. Although I had no way of getting back through the firewall to use them, if something had happened, who do you think they'd come looking for?

Re:So what is the advice

[Anonymous Coward]

Have a colleague watch as you lock yourself out.

Re:So what is the advice

[mysidia]

I wonder, that... if you had no way of getting back through the firewall... I wonder how you could know the credentials weren't deleted? :)

Re:So what is the advice

[sabt-pestnu]

Wrote the answer to that above, before I saw your post here. To repeat: if it's a hostile environment, you need your own CYA audit, with witnesses. Your replacement could be Evil, or simply Incompetent. And either way, you don't want the blame falling on you.

Re:So what is the advice

[Penguinshit]

I set up a checklist with HR to go in the employee file. One checklist with the steps to create an account, and another to remove one. If all the steps weren't completed I wouldn't sign off until my remaining team completed it. When I left I had my #2 sign off as well as the HR manager.

Re:So what is the advice

[PPH]

I wonder how you could know the credentials weren't deleted?

My Boeing e-mail address was on a number of mailing lists. It took a few years for messages to begin bouncing. People would tell me that my address worked one month but not the next nd I had a pretty good idea when my account was dropped.

Boeing's computing security isn't too bright. They shouldn't be bouncing bad e-mail addresses. It lets spys probe the organizational structure. One can also send a message to a valid employee using the first.last@boeing.com format with a return receipt request and examine the headers to see where it was delivered and the internal domain name structure (which tracks the organizational structure).

Re:So what is the advice

[julian_t]

for those that are terminated and have no intention of connecting back in ? After all, if I am let go, the last thing I want is for my old credentials to be used by someone to trash something and have suspicion fall on me.

The same thing can happen if you change job within a company and no longer need access to certain systems. At one large financial company I work with, it is now *your* responsibility if someone uses your credentials to access a system once you've left or changed job. Not sure if that's a regulatory requirements or just policy for that company.

If you can't remove yourself and your company doesn't have a set procedure, I'd send an email or even hardcopy memo to the sysadmins requesting that your access be revoked. Make sure it is copied to your private email, to your boss and to the IT security guys, so that everyone knows you took your responsibility seriously. Then if someone does get in using your old credentials later on, you should have a first line of defense, at least.

Re:So what is the advice

[Legion303]

After I was laid off from $VLTCWWN (very large American telecom company which wasn't Nortel), I enjoyed reading the various internal emails which were still hitting my .forward file there and redirecting to my personal account for another year. These emails frequently contained passwords for sensitive systems, like switch software repositories. And switches. Of course I wasn't surprised at the lack of an email password policy or account cleanup for departed employees, given the stupidity of the unix admin in the department that worked out of the same cube farm we did. He didn't know how to add user accounts in Solaris, so he gave the root password to everyone else in his department.

I knew this because one of my ex-coworkers moonlighted with this other department for extra overtime pay. And when someone fat-fingered something and locked the root account out, he called me asking how they could fix it. This was after I'd been laid off. This was some time ago, but to the best of my recollection my response was something like this: "BWAHAHAHAHAHAHA (*snort*) AAAAAAAAHHAHAHAHAHAHA!"

Re:So what is the advice

[kieran]

1. Erase all saved passwords and other saved data on your work PC/laptop/mobile device/etc before you hand them back.

2. Hand over a list of all credentials you have and advise they be disabled or changed.

Before changing all the passwords

[gmuslera]

Verify that no keylogger is installed in any computer used to login to other systems

Punish them

[dukerobinson]

I say if you fire an employee unjustly or lay them off to hire some workers for less money then you deserve punishment. While there are not technological solutions to capitalist exploitation currently (only political solutions exist to my knowledge), in the mean time I hope you IT staff who are unjustly terminated bring the pain and cover your tracks.

Re:Punish them

[Jaime2]

If you change you car insurance to a cheaper policy, do you deserve punishment? If you fire the guy who cuts your grass and start doing it yourself, should that be a crime?

If the cheaper employees are really worse than the former employees, then the punishment will happen naturally. If the cheaper employees are just as good, then the more expensive ones were simply overpriced.

BTW, I'm the highest paid employee where I work. I'm positive that if I got replaced by two cheaper people, less work would get done. I don't deserve my pay because I already have it, I deserve my pay because I earn it. At a previous job, we downsized from 120 consultants to 7, then the business closed (Internet bubble). I jumped ship two months before the end. I was the highest paid there, but 112 cheaper people were downsized around me before I fled.

I've missed you all these years. How's it been?

[Anonymous Coward]

I see nothing's changed. I've been on 4chan, Skaldi, Stormfront, and a few other places. you?

easy solution

[roman_mir]

it's OK, no problem, just rewrite everything from scratch, guarantees you won't have backdoors from the previous guy.

Re:Non evil stuff may look like logic bombs and if

[Anonymous Coward]

Good thinking, I think. Maybe instead of logic bombs, they could just pay you to write the documentation. :D

Re:Alot of software opens holds due to poor codein

[crontabminusell]

Alot of software...

http://hyperboleandahalf.blogspot.com/2010/04/alot-is-better-than-you-at-everything.html

I'm not a grammar Nazi, but I think you (and everyone else) can appreciate the humor in the link. "Alot" is actually two words: "a" and "lot". ;)

Separation of rights and duties

[damn_registrars]

A good IT department for a sizable company should have some technicians and some administrators. There is rarely - if ever - reason for technicians to have root access to servers and other administrative rights. Your admins should themselves be vetted well enough to not have to worry about them compromising your network after the fact.

Has to be said

[Dunbal]

You get what you pay for. You hire for the lowest possible salary and treat your professionals like unskilled laborers, well, don't be surprised. A professional would never dream of doing something like this - but then again a professional would not work for peanuts either.

Treat people humanely?

[happyhamster]

How about a radical idea of treating employees as people, with respect and dignity, and they will treat you likewise in return? I know I'm stepping a little above the topic, as you asked what to do when you do fire people suddenly without a cause. Please bear with me and don't "escort me out" yet. The way employees are treated in the U.S nowadays is despicable. It would be unacceptable just a few decades ago in this very country, and it is still unacceptable in many parts of the world. An executive firing employees without good cause would and should be roughed up good after work to freshen their understanding of "immoral". American society should make it socially unacceptable, with after-work consequences, to fire people without a good cause, regardless of "laws' bought by corporations in the last decades.

Re:Treat people humanely?

[SolidAltar]

Why would you bother firing someone without a good reason?

Do you mean a good reason to you?

Re:Treat people humanely?

[Fulcrum of Evil]

Good reason in this instance could mean 'we can get the remaining people to do the same work and look good for the quarter' while ignoring the whole 'dead company in 5 years' part.

Re:Treat people humanely?

[Anonymous Coward]

Mod parent up. Way up. In fact, put those exact words on a plaque. I'm not kidding.

Then beat the money/power-worship members of the management class over the head with it. Metaphorically speaking. (Or not.)

In all seriousness, these exact social mores need to be upheld by those who can. They should at least be accepted as humanity's ideal. I do what *I* can, whenever I can. I just can't see interacting with society in any other way. There are just way too many people who, by nature or nurture, didn't get the same message. The quality of an individual can be measured by how well they treat the people that they don't *have* to treat well.

To use another aphorism: "No good deed goes unpunished" is too often the case. So much so, in fact, that -- in the IT industry at least -- the people in power who have treated me with respect are the minority. Fortunately, it's my experiences with these people that remind me that humanity *does* still exist, somewhere.

Will this ideal become the standard? Sadly, I don't think western society is evolved enough for that to happen. Sociopathy is rewarded with power. Capitalism is profoundly imperfect, but it's the best we've got so far.

It seems that all one can do is remain a skeptical idealist -- and hope that our evolution outpaces our de-evolution.

*Sigh*

EDIT: My captcha upon posting this message was "SERVANTS". It's nice to know that the universe has a sense of humor, at least.

it's amazing

[Anonymous Coward]

Reading this brings to mind the number of companies I've worked for that had absolutely no procedures for dealing with the dismissal of IT personnel. Of at least a dozen jobs, only one HR department contacted and co-ordinated with IT around terminations. And yes... they even removed IT decision makers without variance.

Centralized authentication and authorization for all internal applications that handled sales/inventory/hr/etc made it easy to suspend access to all systems by changing the user's password... which was then given to their supervisor who had a limited amount of time to go through the ex-employees files and email to salvage any of their work if possible.

Why the nastiness ?

[redelm]

Nastiness is usually a sign of guilt: "It is human nature to hate those we have wronged [sic]" Tacitus.

If the corp is nasty, it will attract further nasties and have to cope with the results. The nice people leave.

If a nice corp has to fire someone for gross malfeasance and such yet cannot charge them, then perhaps send in a trusted senior specialist to check things out quietly. A big investigative purge will just tell everyone there you don't trust them. Then why should they trust you? Thieves have the best locks. Lots of moves in this chessgame.

Re:Why the nastiness ?

[Anonymous Coward]

also lots of analogys

Pointless

[codepunk]

So do you really think you are going to find the buffer overflow he injected while compiling that php module running on your servers?

Re:Pointless

[mysidia]

No, but it will go away when you upgrade PHP and all your PHP modules, which should be at least twice a month.

More unwarranted badmouthing of IT personnel

[Anonymous Coward]

There was no incident. Someone is stirring up shit. Name the person, the time, the date, the charges, the verdict and the sentence. PROVE the damages.

Some asshole hiding behind the name "White Vampire' is building strawmen and talking hypotheticals.

His is shitting on techies, spreading FUD, giving his industry a bad name.

Shame on you white vampire.

Re:Non evil stuff may look like logic bombs and if

[Anonymous Coward]

how meny times do you have have the old come back at X2 X3 X4 times the pay to just to work out stuff that only the people who got layed off know about?

Been there, done that. :D

A former employer of mine decided that there were too many problems with our web application (which was in the early stages of being rewritten to compensate for years of changes). They closed down our branch office and hired an external company to develop a version of our software that's more to their liking.

Long story short: Their new system is already over half a year overdue (with the additional cost that entails), still far from completion and playing freelance troubleshooter for our old system has earned me three times as much as I'd have earned had they not closed our branch. I figure, at their speed the suits' decision will easily make me another sixty grand before their new toy can even enter beta stage.

The best security protection

[petes_PoV]

is to hire lazy employees. None of the sysadmins here could be arsed to install a backdoor (even if they knew how, which is very unlikely). Plus, anyone who has the foresight to make a hole in the security system, and thinks they would have a reason to use one has probably already stashed all the company secrets on thumb drives and DVDs in an unknown location long before they ever get fired.

Re:The best security protection

[couchslug]

"The best security protection is to hire lazy employees"

In that case, put the national gold reserves in my workplace and close Fort Knox.

We won't need it any more.

grow up

[luis_a_espinal]

Yeah, that will really solve the problem of time bombs and dead man's switches...

How about not disgruntling the employee in the first place?

Oh, grow the hell up and welcome the nature of life.

Though there are work places that indeed are festering, pedantic shit holes, my experience has been that people who are disgruntled enough to commit a stupidity don't necessarily work in a place causing them to be so disgruntled in the first place. They are simply stupid assholes who either have a sense of victim-hood or are too arrogant and socially incompetent so as to pop a vein at the slightest work-related discomfort.

Work is work, it's not supposed to be pleasant all the time. We get paid to do work that has a certain level of difficulty, both technological and sociological. It has always been so, it will always be so. Half of the time the fault of being disgruntled is in you. How you handle that shit is ultimately one's responsibility.

If you are a mature person with a sense of, oh I dunno, fucking professionalism, you will never get *that* disgruntled no matter the working conditions. If you are not a mature professional and you cannot tell professionalism from shit flinging monkey riding a banana-shaped tricycle, then you'll inevitably construe any slightest difficulty into an affront, building each one of this up, turning you into an arrogant, festering boil of disgruntled human suckage and social incompetence.

And for those who truly voted that post as insightful, man, grow up, really.

Re:grow up

[Anonymous Coward]

I got to the last few paragraphs of that post and all I could hear was...

Chimpanzee riding on a Segway, Chimpanzee riding on a Segway, Chimpanzee riding on a Segway, Bam ba ba bam ba bam
Chimpanzee riding on a Segway, Chimpanzee riding on a Segway, Chimpanzee riding on a Segway, Bam ba ba bam ba bam

repeat until post goes away

Re:grow up

[Anonymous Coward]

Boss? Is that you?

personally

[oliverthered]

I'd just use the errors I know in the existing system to break in.

Let's just say this depends on the former sysad

[mysidia]

This is why companies need to have an IT-savvy IT manager and know their employees well, and have multiple IT workers watching each other, much like accountants and finance officers are supposed to watch each other and have separated powers.

Know your employees, their abilities, and their personality. Without knowing the person, it's difficult to assess the risk as to whether or not they might or might not do or attempt to do certain things. And what things are even possible for them to attempt.

The easiest way to avoid running around in circles is to know what they are capable of exactly. If their personality is psychopathic super-programmer, you might have good reason to look for hand-coded hidden kernel drivers, or little binary blobs in a proprietary tampered-with program, containing custom logic bombs, and exploits for bugs in other programs (automated privilege elevation and exploitation).

If not, well, more mundane audits should be fine.

If the person is familiar with scripting, then, well, you'll have to check all the scripts extra carefully. Even if not, they might have found something on the web, and it doesn't take rocket science to cron "rm -rf". Which should not be that much a concern if you have solid frequent backups and take additional precautions to secure those.

(Probably) the worst case scenario is they are conspiring with skilled outside hackers, who are providing expertise and assistance.

Once the outsiders have enough information, they may get the IT admin to "run some code" from somewhere obscure, which will lay the playing field, and then later the outsiders will infiltrate the network.

However, that implies premeditation. If an IT admin is going to forcibly lose their job for serious disciplinary reasons, and anything is suspected to be a risk, they should be escorted by security and not allowed to touch any computers until they are gone for good.

Make them stay on premise during working hours, and have them use pen and paper to fill out some paperwork and answer questions.

This way they will not have a lot of "free time" until all your new IT admins' audits and password changes are established.

Re:Let's just say this depends on the former sysad

[Anonymous Coward]

rm -rf is so unimaginative. Think "silent, slow data corruption" and you'll be getting closer to the nastiest thing that could happen... if it takes the remaining employees months to find out, your backups may be worthless.

*shudder*

I once got in to a missile system useing Joshua as

[Joe The Dragon]

I once got in to a missile system useing Joshua as the login. No password needed.

In '01

[Anonymous Coward]

We had a bad bit of this sort of thing, though accidental and on a lower-tech scale, at our high school. One of our student volunteers that helped with basic network administration tasks had asked us to give him temporary privileges to add a mouse driver to all of our dells. As he explained, students were complaining that the 200ish identical machines all had non-working scroll wheels, but that he had a driver that he had tested on our testbed that would solve the issue. We granted him access for the day, and all was good.

Until about 2 months later, when the machines started letting the smoke out 1 after another due to graphics card failures, until winter came along and they started to get a "second wind" and our failure rates plummeted. We contacted our buyer about the issue, but we were informed that none of the other schools had been having issues.

We finally discovered that our enterprising student had also taken the liberty of upgrading the graphics drivers while doing the mice. In doing so, he had broken the fan controls on the cards, which were locked in at 5% duty cycle.

We ended up replacing them with another series with integrated graphics (the original purchase decision certainly wasn't our idea) but it made for a very interesting 8 months.

I'd already subverted the system.

[mrmeval]

We paid a police officer to be present as security when we tossed out the two people who made up our IT department one who was a co-owner of the computer repair department and his employee who also maintained our PCs at a company I worked at. With permission I'd already gotten all of their passwords, subverted the network and database system so running a command from any computer I was logged into removed their access and made me administrator. At the same time it locked every user out of the system. My boss knew they'd embezzled 100 grand and had proof we just wanted them to take action with witnesses when they tried to damage the system. The one auspie geek actually tried to nuke the system after he was told he could not touch a computer. They were allowed to remove some things and he jumped on a computer. The cop cuffed him and made him sit till the other person was done. They were escorted to the door. The police officer wrote up a report and gave it to my boss. He submitted that and all of the book keeping information we could recover. The local prosecutor charged them and took a plea. The co-owner took the brunt of it. We only say a fraction of what he stole but he does have a felony conviction. The auspie geek wasn't charged and went on to embezzle money from a local ISP and did get caught after new owners did an audit.

Corporations DEFENSELESS against determined foes

[Anonymous Coward]

I was CIO for a fairly large company. I reported to the CFO. The first few years were great. Then the good CFO retires and is replaced by the bad CFO. This guy was looking for excuses to get rid of me from day one. It took him several years, but eventually he fired me.

It seems he hired some type of security consultant to lock down the place just before I was fired. Some of my staff members were forced into cooperating with this little adventure. Evidently, the bad CFO thought I would launch some kind of high-tech retaliation. This was actually a fair prediction, based on how I was treated. If you treat people badly, you should EXPECT trouble.

While I am in the process of getting the bad news from HR, it occurs to me that I really want revenge. But a high tech attack would be illegal, unethical, and they're probably expecting it. Therefore I will NEVER attempt anything related to IT. I'm not sure exactly how to proceed, but I decide to wait and give it some thought. My staff members knew from day one that I would do something and whatever it was, it was going to be big. The funny part is that the company's security consultant did everything recommended in TFA and then some. And yet he STILL left a gaping hole that I helpfully reported to the company after the fact. So much for the security audit.

Good things come to those who wait. I stumbled across an idea that had nothing to do with IT. I REALLY REALLY want to write about what I did, but details of the operation must remain classified. Everything was 100% legal and ethical. The results were absolutely spectacular. I might turn this into a movie script someday. It was that good. The amount of pain I inflicted exceeds anything that I could have done with computers.

Three important lessons here: (1) Security audits are seldom 100% effective and a determined opponent is going to get in anyway. (2) A really determined foe is not limited to computers. (3) Treating people poorly leads to unintended consequences; see (1) and (2).

Let me correct that

[drolli]

The assumption should be that you have been rooted by somebody who knows exactly what things are logged in your systems, possibly with continuous influence on what is being logged and how long, maybe even with the power to alter log files. IMHO one of the important things is to use several servers just for logs, to whom only a single admin has access. If one of them is going in a bad way, then you have at least the logs on the other machine. If you are paranoid, transfer the md5 checksums of the files on your servers to these machines and use git on the etc directory, backing the etc directories up on these machines. and force the it staff to make builds of custom SW automated.

This means you have
a) logs of what has happened (at least you know what you know)
b) a possibility to determine which files changed
c) a documentation about which configuration changes have been done for which purpose.
d) a backup of the configuration, enabling you to reinstall the machine
e) a way to rebuild programs added to the system easily.

Re:Let me correct that

[profplump]

You can do one better and send your logs off-site to a third party that guarantees read-only access. It's just as easy to setup and unless your admins can bribe the third-party it's awfully hard to alter the logs. They could probably add fake messages, or possibly stop the logs from going out, but the former is possible with just about any logging system, and the later should only be possible after they had done something that first generated a log.

Re:Let me correct that

[xenobyte]

With Unix-family systems it's easy to stream syslog to another server, and that other server should be used for nothing else. Firewall it so it seems down from everywhere (except perhaps a monitoring server) and so that you only access it in two ways: Inbound udp on port 514 (syslog streaming) and inbound ssh on a different port than 22 only from a single access point (another server, a workstation or similar) using a key not stored on that access point and not used anywhere else.

I'd say that it is extremely difficult for someone to compromise another server (webserver typically) and then gain access to the logging server (name or IP evident in /etc/syslog.conf) to erase his tracks there as well.

Re:Let me correct that

[drolli]

I'd say -> I'd hope....

I don't think these servers need anything else by logins on the console....

It's so unnecessary

[bzipitidoo]

Let me add there's no need to intentionally make things more complicated and failure prone than they already are naturally. If they really needed a position filled, and don't replace an employee they just fired, they'll learn that the next time they need some work done or a fire starts, and there's no one around who can do the job or put out the fire. That can happen sooner than you might imagine.

They may even try to rehire the ex-employee. I've seen that happen. Manager A is an idiot who fires people for no good reason, just enjoys the power trip and takes irrational dislikes to random people, but manager B has more sense. But if you've burned your bridge, manager B won't touch you with a 10 foot pole. If they're all fools, and they probably are if they just up and fire people they really need, the last thing a former employee needs is to leave behind a smoking gun that in any way justifies any part of their warped thinking, hands them an excuse to blame everything they can get away with on the former employee. Move on, and let them go down in flames all by themselves. If they're that bad, they will.

On the other hand, if the position wasn't necessary after all, well, I for one would rather not occupy it.

The real problem is that getting another job is such a miserable endeavor, dealing with rejection after rejection. If it wasn't so hard to get another decent job, employees wouldn't feel so desperate about losing one, wouldn't feel that they have little to lose and be tempted to get revenge, wouldn't take it all so personally and hard no matter how viciously the company handled the parting of ways. All this advancement and progress we've had, particularly in the last 200 years, seems to have largely passed by employment relations. Most companies are still run in very autocratic and harsh ways, and get away with appalling behavior. It's even justified by the nasty thinking that employees as a rule are lazy slackers who must be forced to work, and wouldn't if not for fear of being fired. I hear the entire game software industry is all ugly sweatshops. The 40 hour work week seems to have become 45 (lunch somehow became employee time-- whatever happened to "9 to 5"?) and that is a floor, not an average. And vacation is for losers and crybabies. Most of all I can't understand why departments that handle these matters sometime in the 1970s or 80s took the name Human Resources, as if employees are like coal mines, to be worked until they're played out and exhausted. It used to be called Personnel, what was wrong with that name?

clients from hell

[Anonymous Coward]

Based on my experiences and the experiences related in Clients From Hell, every freelancer ought to leave a hidden timebomb to be deactivated only when the client pays up. Cheapskates deserve every minute of downtime and every dollar extracted from them by cleanup crews.

These should be documented and understood

[Anonymous Coward]

i.e. They are failsafe management interfaces. Not backdoors.
 

Kill yourself.

[Alex Belits]

I believe, I have expressed a more popular than usual (for my posts) opinion in this.

Three other words

[h00manist]

But really, the best thing to do is to treat your IT staff properly in the first place.

The golden rule. It got that name with good reason.

Anti-social.

[RichiH]

Poor you, really.

Designing things to be shitty is bad style, anti-social and a heap of other things. I just hope you will get job after job fixing the crap other people like you pulled off. Again under high stress. Hopefully you people just keep on fixing each other's crap and stay out of the way of the rest of us.

The professionals amongst us strive to make things easier for ourselves and each other.

The childish, spine-less power-trippers act like you did.

Re:Anti-social.

[Ironhandx]

The "trouble" I had to go to was actually leaving things mostly as they were and fixing within the mess that was there. There were legitimate reasons, I didn't actually have time to overhaul everything until I was getting ready to hand in my notice and I TOOK the time.

I did my job, and I did it well, just not as well as I could have as I would have put myself out of a job in short order.

Re:Anti-social.

[TheRaven64]

Don't be so hostile. Developers with the grandparent's attitude make me happy - I get to charge ten times their rate to fix their code when management decides that they're not nearly as indispensable as they thought. It's amazingly satisfying to take a mess of horrible code and turn it into something a tenth the size, which runs faster and is easy to maintain in the future.

Re:Anti-social.

[RichiH]

Easy to maintain? Just wait for GP's kin to come around, cussing all the time about your code while morphing it back into a monster. Relational integrity in databases be damned!

WHy don't you grow up

[Viol8]

"f you are a mature person with a sense of, oh I dunno, fucking professionalism, you will never get *that* disgruntled no matter the working conditions."

Oh please, and you're telling OTHER people to grow up? Sounds to me like you've hardly had any work experience in the real world. It doesn't matter how professional you are - everyone has certain buttons that can be pushed and in a long working career believe me , someone WILL push them eventually.

Also you might disguise your young age a bit better if you didn't swear every paragraph.

Re:WHy don't you grow up

[BitZtream]

Also you might disguise your young age a bit better if you didn't swear every paragraph.

Or he is Irish, or he just likes to cuss.

I'm 33 and cuss in writing all the time, does that make me immature for doing it ... or you immature for focusing on the cussing and missing the point of the message?

Seems to me if you were so mature you could simply ignore the cussing and move on.

Ironic isn't it ... most of your post telling him what to do is advice you should take yourself. You guys would make a cute couple.

Re:WHy don't you grow up

[Critical Facilities]

Also you might disguise your young age a bit better if you didn't swear every paragraph

In the interest of full disclosure, I must say that your comment seems pretty clearly to be a case of the pot calling the kettle black . It is your opinion that this person hasn't had enough work experience to comment on the issue, but to make that assumption based on his use of profanity on Slashdot (not exactly known for its "family friendly" comments) is a little petty. More importantly, the issue isn't whether one's "buttons will be pushed" in his/her respective career (they will be), it's how one reacts to such button-pushing that determines success.

Re:WHy don't you grow up

[Viol8]

"I'm 33 and cuss in writing all the time, does that make me immature for doing it "

Do you really want an answer to that?

Re:WHy don't you grow up

[Viol8]

Well the first post was talking about lost lives including my cousins so I think swearing is a bit more appropriate there than talking about work. And point out where I was swearing in the second - I think you'll find I was quoting someone else and making a joke about it.

"it's how one reacts to such button-pushing that determines success."

I'm talking about working in the real world over the course of decades, not what they teach you in MBA class.

Re:WHy don't you grow up

[Dragoness Eclectic]

Or he might be ex-Navy/ex-Army/other ex-military. I had a polite vocabulary *before* my 4-year enlistment in the good old U.S.N. Now I have to make a conscious effort NOT to "swear like a sailor". It's a cultural thing, and has been since humans started organizing militias.

Re:WHy don't you grow up

[Critical Facilities]

I'm talking about working in the real world over the course of decades, not what they teach you in MBA class.

So am I. It seemed pretty plain that the poster to whom you were replying was as well.

The point is, a lot of people manage to convince themselves that they should be put on some sort of pedestal by their respective employer. Sure, it's nice to get some recognition, but one isn't entitled to it. The point I believe the poster was trying to make is that one shouldn't expect to be treated as some essential cog nor should one try to conflate various inconveniences/unpleasantness with poor treatment or "abuse". Every job has its lousy parts (as even you pointed out), so we must all learn to deal with them. That's not MBA speak (especially since I do not have/need/desire an MBA), that's just how it is in the real world, I'm sure you'd agree.

Slightly off-topic, my sincere condolences for the loss of your cousins.

Re:WHy don't you grow up

[Viol8]

"that's just how it is in the real world, I'm sure you'd agree"

I do. And all I was trying to point out is that it doesn't matter how professional you are, at some point over the course of your career you're going to have a bad day or illness or some personal problem combined with delaing with some obnoxious manager/customer/whatever and you're going to lose your cool. It happens to everyone eventually. We're not robots.

Big talk you bullshitter

[Anonymous Coward]

I hope you get fired soon (if you even have a job) and that they burn your ass in terminating you, so that you get to see how well a big talking little pud whacker like you feels when you get the high hard one right up your ass.

Bahahaha

[Atrox666]

It's 2 L8 we r in ur interwebz.

IT staff are adept at exploring backdoors...

[Anonymous Coward]

I'm talking about analingus and anal fisting. That's the scuttlebutt around the office, anyway.

Re:Corporations DEFENSELESS against determined foe

[Dragoness Eclectic]

Let me guess, you founded a new company that outcompeted your old one and drove them out of business? Or you married his ex-wife after the messy divorce, after she got all the money?

Dang, I could have a lot of fun with this.

Re:Corporations DEFENSELESS against determined foe

[Anonymous Coward]

Neither, although I would not have passed up those opportunities if they existed. If I told you, your chin would hit the floor. Someday, the operation will be declassified; you might just see it in a movie.

Re:Corporations DEFENSELESS against determined foe

[Anonymous Coward]

No, although I would not have passed up your ideas if they were practical for me to implement. If I told you, your chin would hit the floor.