Slashdot Mirror
Security Plans for When Your Senior Developer Leaves?
An anonymous reader asks: "Our CTO, responsible for all hardware and networking setup, who also coincidentally happens to be our senior (and only) developer, has just resigned to go work for the competition. We are not a software company, but he's written proprietary code that we use on a daily basis to work. What interim measures should we be taking to ensure a smooth transition to the next person hired to take over? What can we do about security, since this person designed and implemented all current security procedures? What about ensuring that we have all the intellectual property to which we're entitled? As one co-worker put it: 'His resignation was a surprise to us, but it definitely wasn't a surprise to him.' If he wanted to leave some hard-to-find malicious timed-release back-door-opening code running, it's certainly within his means. We don't expect any malicious action, and can rely on a reasonable level of co-operation and documentation before he goes, but I want to get a sense of what others have done in this situation."
Get your lawyers lubed up and ready to go.
There are no trails. There are no trees out here.
You should phrase the question "what should we do the next time"
This is a legal problem. I assume that you have all of the agreements in place (signed NDAs, Non-Competes, etc.). So from there just monitor what is going on. Frankly if you are loosing your CTO, only developer, etc.you are screwed as a company anyway, so maybe it is time to update your resume and get a head start on the new job search that is most likely coming your way
I have mod points and I am not afraid to use them
When you hire the person, make them sign something saying that any proprietary code they develop for the company becomes the property of the company. also, insert clauses with penalties for intentional security breaches, etc. it's all a matter of planning. when you hire someone, you want to bring them on, but you should also look at it from the perspective that they can do real damage to your company. you should have them sign something to the effect that they shouldn't do that damage, and if they do, they will be held responsible for any intentional damage. NDA's while not always enforceable if they are unreasonable are a good deterrent as well.
I'd stick to gouging out his eyes and cutting out his tounge, lest he bring a plauge upon your house.
Or, you know, change passwords, and stuff. I hear that works too.
What were you expecting?
In the heyday of the bubble, jumping ships was a practice that everyone knew about it, and often tried. So, relating from my experience of that...
I think your requirements for a replacement CTO should start with securing the system. Hire consultants until the right guy is found to document what's going on - NOT for more development.
Although I have no info about the politics, your lack of insight into managing your technology is stunningly poor. I hope you pick the best of those consultants and hire them to spread this risk in the future.
Above all this, prepare for your competition to now exploit any weakness you have in your market. No more BusinessAsUsual. If you didn't care about what this guy until he left, perhaps you should re-evaluate what you use your technology for, and take it a bit more seriously.
mug
but I want to get a sense of what others have done in this situation."
Ask him if you could go with him to the new corporation.
"Provided by the management for your protection."
You say you're not a software company, so obviously the code that he's written isn't your product. Is it utilities, or something that manages your workflow and process? If so, it doesn't seem like you've got that much of a problem. I guess a lot of it depends on what terms he left as -- going to the competition doesn't necessarily imply a bad breakup, but the tone of your posts seems to. Well, anyways...
The easiest source of information is going to be him, himself. It doesn't sound like he's left on the worst terms, and, really, the truth of the matter is he's got all the cards now. If he wanted to screw you over with a malicious time bomb, he could, and there's very little you could do about it. So I would just take what he gives you in terms of documentation and all, and, unless evidence proves otherwise, assume that he's on the up and up. You have little choice, and the other options (like lawyers) are going to make him very uncooperative. Most programmers I know don't get malicious unless they feel that they've been royally screwed over. YMMY.
But, to the future! The best way to avoid exactly this kind of thing is to not have a new guy, but two (or more) new guys. Even if its a senior-level and a junior-level, having someone who can be your backup is invaluable. At worst (depending upon the software), you could get an intern or other low-paid peon to serve as the backup on the cheap. Some of them are clods, but some can be quite smart. Code review reduces not only bugs, but logic bombs and backdoors, and it leaves someone who at least has a clue about the system if one of the two leaves.
As for security: Make sure you have a firewall, and the rules are set to the bare minimum allowed in (but you should have this already, right?) Change the root/administrator passwords. If you have a competent sysadmin, have him monitor for unusual activity...but these are all things that should be going on all the time. In other words, nothing out of the norm.
If you think they are the type of person who may do something like that, you probably shouldn't have put them in charge of security.
Nah just kidding.
I would suggest making two complete backups of all data on all machines. That way if there were a problem then the backups could be used for forensics. Second, monitor any connections to the network from remote access modems or internet connections using an intrusion detection system. Then just relax. If the guy is leaving on good terms then you probably have nothing to worry about.
We don't expect any malicious action
Well then you shouldn't have made life so difficult for your CTO. I mean, everyone has their price, PAY IT!
Oh wait, you want team players. Well then who's idea was it to cut his pay, deny funding to the latest project, or take photos at his last "business trip". Certainly not his...
Antiquated competence won't be a job skill forever.
First, hire a security team to secure your systems.
Make sure they remove all existing accounts on all systems, and start with new ones, with very secure passwords. This is a good time to require a password rotation policy, and password length & strenght requirements as well.
No non-secure connections to non-public systems from outside the company, period. Or at all, if you can get away with it. No connections from dynamic-IP connections to internal systems, either. (make sure all allowable connections to internal systems are from a list of known IPs)
Make sure PHYSICAL access is secured! Lots of ex-employees keep security cards, keys, etc, and can often get back in after the fact.
Make sure your people know about 'social engineering'!
Don't use inherently-insecure technology from companies who don't give a rat's ass about your security. No bonus points for correctly guessing which company I'm talking about. This becomes stupendously more important if you're the sort of silly-ass company that only has one techie on staff at a time. Lots of updates are to be applied, no matter what platform you go with.
Now's the time to separate systems if you host stuff. Hosting stuff should go in a co-lo facility (since you obviously don't have the staffing resources to handle your own data center), and you should have separate systems for business needs, like e-mail, etc., in case your website gets DOS'd, it won't impact your e-mail, etc.
Have regular security reviews by external security companies. Rotate which company you use each time.
Make sure your insurance covers all your computing infrastructure and eventualities (fire, flood, theft, cracking, etc.).
Make regular backups.
TEST your backups.
Make sure you have off-site backups.
Make sure you have a disaster preparedness plan and the appropriate people know how to implement it. What happens to your business if the building burns down? If the phones go out? If the Net connection goes down? What if there's a major terrorist attack in your city and noone can get to work? Welcome to the real world.
Make sure you have onsite spare parts for your computers, at least for the critical ones.
Make sure noone saves important documents ONLY on their own machine - either make them start saving to shared drives which get backed up daily, or have each machine backed up daily. Say you lose the business plan you're showing to investors tomorrow? What do you do? WHAT DO YOU DO?!
Don't get locked into proprietary file formats, or you may never be able to switch. Plus you may get hit with 'requests' (ie threats) to inventory every piece of software on your site.
Definitely have more than one techie & programmer (2 of each, at least) at your company. That's flat-out ridiculous, as you are probably aware by now.
Okay, that's all I can think of off the top of my head right now. Have a day.
I'm guessing you've already hired someone to take over at this point. I say this because hopefully anything technical that may need to be done to insure a smooth transition won't be performed by your former CTO. This also leads to a less hostile work environment where the CTO doesn't feel you're worried about him doing something damaging.
Assuming that you already have some sort of data backup performed on a semi-regular basis, my first step would be to keep a static copy somewhere in storage. A snapshot like that might prove useful later should something be 'waiting' for him to leave.
As for the proprietary code, if you haven't already worked out the legal ownership issues involved with it, you're a bit late. The less you have in writing already regarding that, the more you should be considering a replacement setup. At the least, you should be requesting documentation for everything that doesn't have it already.
Get your lawyers lubed up and ready to go.
I think what you meant to say was "Get lubed up and get your lawyers ready to go." They are lawyers, after all. >.>
This statement is false.
We make DSL equipment. Shortly after a layoff last year, all of our test stations at several contract manufacturers stopped working almost simultaneously. It seems one of our test engineers had programmed them to phone home to his PC at headquarters to make sure everything was ok. Thank goodness it wasn't one of the linecard software guys or we could have had thousands of lines out of service.
Trouble? Yes, we've had our Phil.
For the most part, if they were really malicious, you are boned anyhow.... The good news is development is really a small community - even if they don't get the book thrown at them, I know folks that were more or less excommunicated because of bridge burning and other stupid departure tricks. More than ever, jobs are had by personal recommendation rather than some recruiter pushing your resume. You may not like your job, your peers, etc - but I've seen prospects burned before they got in the door because of what they did a company or three back. Odds are, if this guy was a senior level developer, he has more at stake than you. I know I made sure everything was checked in, documented where possible, and asked IT to change my passwords - I also never checked to see if they did...
+++ UGUCAUCGUAUUUCU
Then, once he's gone, audit the code. Maybe you'll need to hire an outside consultant to do it. Anyway, once the source code is audited, you still aren't in the clear. It could be that he put a backdoor in the binaries, leaving the backdoor out of the copy of the source code he pointed you toward. Thus, once you are done auditing the code, compile it. Do a file compare of the current binaries and the newly compiled binaries.
In Windows, the command line is fc /b filename1 filename2.
If there are any differences, that doesn't necessarily mean anything significant. Move the current binaries to a temp directory or someplace out of the way. Don't delete them, as they could be important later. Copy the newly compiled binaries in. Test the whole system to make sure it works.
As for ensuring your intellectual property is protected, I don't know how you can truly do that from a technical standpoint. You should notify your corporate legal counsel of your concern. If you don't know who that is, bring it to your CEO's attention.
Good luck.
first, treat the person leaving with respect. if this person is mature then they won't burn bridges - neither should you.
don't accuse him of things he might not have done. don't screw around with his career. shake hands, wish him well and generally be professional. it's business. cope.
second, solve your problems. the person who is leaving has his own issues - poor communication, poor loyalty, excessive greed, whatever. those are his problems. let him work on those, they're not your problem.
the main reason for your discomfort is that you put all your eggs in one basket. and now your basket has left. so in the future hire two people, not just one.
and when you have these two people on board, talk to them more often. find out how they feel. you were taken by surprise by this person leaving, that suggests poor communication - on his part or your part.
finally, you seem to have no idea what code this person wrote even though your business seems to depend on it. does the code go in a source code control system? do you have a release procedure? can you get the previous releases?
you need to answer yes to all three of those. if you don't answer yes to all of those now, make sure you can in the future.
US Citizen living abroad? Register to vote!
Quite honestly, your company needs to get their ducks in a row. Here is what you are up against :
...
Your company sounds small enough that they had very few 'computer guys' but big enough that the computer infrastructure is fairly complex.
The guy in charge (your soon to be ex-CTO) probably designed and built the existing systems from the ground up. As he didn't have anybody watching over his back, do not be surprised if there is some jury-rigging in there. He probably shared some of the quirks with some of the other computer guys, but not all.
He may be an important part of the wet-ware in your system. An easy to understand example would be a bowling alley - if your company has to bowl a strike every time the ball gets thrown, he was the guy that walked down the alley continually making minor adjustments to the path of the ball. This could be custom reporting on your data, swapping out the backup tapes, deleting temporary files, cleaning out the log files so they don't fill the hard drive, or booting the servers in a particular order so as not to overload the UPS. It has become routine that he takes for granted and probably doesn't even think about them any more so when he doesn't mention it (and they don't get done)
You have some pretty important apps that he may be the only guy that understands how they work.
Today is the day of truth, you are on the cusp of finding out if he is disgruntled or not. If he is disgruntled, the LAST people you want talking to him is HR. They will either piss him off more than he is, or try to bully him - you need to get his favorite tech to take him off-site, dinner or to a strip club, and off the record find out why he is leaving, what his primary concerns are, what he would honestly have changed given the chance, what he anticipates the hot-spots being after he is gone, and most importantly : does he have any recommendations for a good replacement.
This last one is key. There are lots of paper tigers out there (MCS* certs), lots of guys that are good at network administration, lots of guys that can code language A or B or C++, lots of guys that can diagnose and maintain an SQL Server, and lots of guys that can operate in the role of CTO to work as manager and liason between the IT department and upper manglement. You are going to find precious few people that can do ALL of the above(*), and unfortunately that is exactly what you need to do - and find a guy that enjoys doing it because the first few months are going to be rough. Doubly rough if your CTO is disgruntled, so if one of trusted colleages was in there he might hesitate before setting off some time bombs that his pal is going to have to clean up.
The penalty for getting this wrong is going to be pretty severe.
(*) I would be perfect for the job, but I am pretty happy where I am.
Glonoinha the MebiByte Slayer
This is -exactly- what I was talking about when I said the last people you want dealing with this guy is the HR department.
Might as well steal the toys off his desk, decline the steak dinner on his expense report, deduct some personal long distance calls from his paycheck, and key his car while you are at it.
Glonoinha the MebiByte Slayer
Yeah, the lawyers and HR would love that but anyone worth their pay would run the instant yuo suggested it.
"Intentional security breaches," for instance. Okay, no problem, none of us want intentional security breaches and since Outlook and MSIE are both responsible for a large number of breaches they're history. What, I can't do that - you're telling me that you're holding my feet to the fire yet denying me the authority to do anything about it? See ya!
Ditto all of your other suggestions. Of course any code written for the job, at work, for pay, etc., belongs to the company. It may or may not be proprietary, in the sense that I may extend GPL code to fix a problem. It's perfectly legal unless the company wants to distribute the code to others (which doesn't sound like the case here), in which case you need to say so upfront so I can budget about 10x as much resources to duplicate the prior work. But the stuff I do at home, on my own time, is mine.
I could go on, but it shouldn't be necessary. Anyone with real experience has been burned by somebody with such a list, or had a friend burned, and no matter how bad the economy is they know that unemployment is better than being the target of a lawyer trying to prove that their client's incompetence is really your fault.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Another suggestion - depends on how important your uptime is, but ... as you walk him out the building the last time hand him his bonus - if the systems maintain their existing uptime percentages (nobody is 100% uptime, but pretty close) for one year - with his occasional cooperation if necessary (assistance via phone or VPN access or whatever) - then give him his usual bonus for that year, or some arbitary amount of cash ($5,000? $10,000?)
Consider it a very important support contract with a limited lifespan (a year should suffice.) Unless he is really, really pissed or his new company is paying him double what he was making with your company - he will go for it and be a pretty eager helper when the chips are down.
If he declines that offer you guys are hosed, because he declined it for a reason.
Glonoinha the MebiByte Slayer
Seriously, it doesn't hurt you and maybe then you won't have to worry about him...
I have been the guy in charge of everything technical, and left to go work for a competitor. (by that I mean in charge of IT, in charge of engineering, for a sick period of time, in charge of the web site, answering to the board, and oh by the way writing mission critical software.)
Where's about what I told them. (I wish I could find the original letter, but I can't.)
Disable all accounts listed on attachment (a). Better yet, monitor activity on them. Look for ones I've forgotten/failed to list. (We had a horribly fragmented pile of crap for authentication/authorization.)
Review all changes I have made or have caused to be made by others on externally facing systems for the last N months, where N is something I obviously cannot recommend. (I was a nazi about change logging.)
Review all executive reports I've made for accuracy.
Randomly interview my employees and fish for things I might have done, might have wanted to do, etc.
Review this list of important things I've done over the last while, and think of how I could go about damaging you with underhanded techniques.
Review your infrastructure. Hire outside people if needed.
Review your trade secrets. Hire outside people if needed.
Note problems with either of the above. Wait for signs that I'm exploiting them.
Think about my tenure here, what I've done, and what I'm legitimately taking to my next workplace. If you believe I've been underhanded, please call me to task. I'd prefer a non-judicial approach first, obviously.
Don't trust me, think of how, in my shoes and with malicious intent, I'd be sneaky. Please assume I'm not doing so (because I'm not), but verify.
***
There was more, some of it company specific, some of it items I've forgotten. My approach was to put myself beyond reproach. I don't do underhanded things. There was concern that I could, and I wanted to explain the tools they could use to convince themselves of that. Of course, I could have done all of that just to try to trick them... I was hoping (and have been proven correct) that my prior track record reinforced faith in me, even when I was moving to a competitor. Sometimes, being a good person makes sense.
I also had some heated sessions with a corporate attorney at my new role over that letter. It came down to, "well, then I already fucked up. Fire me." They didn't.
(Afterseveral years: the two companies have moved in different directions in the same general field, there haven't been any problems between them, and we still trade employees from time to time. And I'm back to having dinner with my former CEO.)
-obviously, anonymous
Just follow the termination procedure. You do have one, right?
This isn't a termination, of course, but should be handled the same way.
non-compete clause...
California, where no-competes are unenforceable. Note that you could still get the soon-to-be-ex-CTO with inevitable disclose of company secrets, but you have to go to court for that one.
It's Linux, damnit! Pay no attention to renaming attempts by self-aggrandizing blowhards.
The way to address this is to hire good people, and treat them well, especially for key positions like this one. That way when the person leaves to move on in their career, it will be with regret and good wishes. They will be eager to help you in way they can help you recover from their departure, rather than looking for ways to harm you.
Yeah make him think twice about it. But what if he wasn't even about to think of it at all? Thought never occurred to him? "Don't think of the potato" sort of thing...
If you're going to break up, why not remain friends if you can? Not go to "If you take my alarm clock, I'm going to sue you, and remember the time you left the toilet seat up etc". Doh.
After all the questioner said: "We don't expect any malicious action, and can rely on a reasonable level of co-operation and documentation before he goes,"
This pieces of paper should only appear when he first started working or during general policy updates.
I doubt people were talking about kissing the guy's ass. It's more of not burning bridges.
Remember you MIGHT need to call the guy up for HELP.
You're proposing a piece of paper that says if you do something bad we'll send you to jail.
But that stupid piece of paper sure does not incline him to help if something bad happens for other reasons (nothing to do with him) OR you need to change things in your systems around - nothing wrong, business needs often change.
He's less likely to help people who treat him as if he's the enemy. Goodwill often takes years to build up, and can vanish almost immediately with something like this.
If you play nice, you are more likely to get free phone/email advice. People in my prev workplace have asked for advice a number of times after I left them, and I've given it free to them.
I might have still given advice if they had given me such a piece of paper. But unfriendly and risky (threat of jail) environments might mean I'd charge or just hang up.
Nowadays it's a small world. People are just a few seconds away. Whether for good or for bad.
1) Proprietary code written, but wasn't a software company.
2) Non-compete clauses are dumb ideas. You don't want a skilled baker to work in a rival bakery? What's he going to do then? Work as a beautician? You going to pay him to do nothing? Same for programmers.
In fact some companies do pay key people to do a year or so of "gardening".
BTW if someone is really crap... e.g. Mr Anderson, you suck, we'll help your CV look good, just go work for the competition. And if you sign this "do compete clause", you'd get a USD5K bonus. j/k.
It would probably sink the company.
The reason you have someone escorted out is because you believe they may cause some sort of damage.
If he wanted to do any damage, he would have done the dirty work in the period between deciding to leave, and telling everyone else that he was leaving.
At this point, the most damaging thing he could do is leave the building and not share any of the business-critical information inside his head about how the IT infrastructure works. Why would you want to force him to do that? You'd be shooting yourself in the foot.
"Patriotism is your conviction that this country is superior to all other countries because you were born in it." -- GBS