Slashdot Mirror
Ask Slashdot: IT Staff Handovers -- How To Take Over From an Outgoing Sys Admin?
Solar1ze writes "I've just started a role in an IT services firm. I'm required to take over from an incumbent who has been in the position for three years. What are some of the best practices for knowledge transfer you have used when you've taken over from another IT staff member? How do you digest the thousands of hosts, networks and associated software systems in a week, especially when some documentation exists, but much of it is still in the mind of the former worker?"
Run like hell....
Primarily, you'll want to build an honest rapport with the other person. Get inside their head a little and allow them to brag A LOT. Ask how they found the place and what they did to change it. You'll want to breeze through all of the high level and important documentation first so you'll have a baseline. Take as much notes as you can. Ask what websites/resources they use to make it easier to follow in their tracks. Explain your situation to them. It will humanize yourself in their mind and you might be able to engage their compassion for you. Perhaps they would be available to answer questions after they leave! Is there budget money for them to be used as a compensated resource? Hopefully they like the idea of helping others and putting some scratch in their pocket.
Bon chance!
Steve
--- rapper/producer/bachelorette party stripper
I would start by writing your own manuals and have the outgoing person review them.
Some people die at 25 and aren't buried until 75. -Benjamin Franklin
Make sure you have thick gloves and sanitize everything. Check for booby traps. Never push any buttons till you've traced the wires back to their origins.
http://bofh.ntk.net/BOFH/
1) Need passwords... immediatly change them.Exiting person should have no futher access except through you.
2) Require exiting person to produce network diagram. Make it their last duty if one doesn't exist.
3) Now starts the pain... audit devices and systems for rogue accounts.
4) document as you go.
5) turn in passwords to supervisor.
Good Luck
Hope he is leaving on a good note, and not holding grudges.
Then systematically go through each machine for which he has a password and have him record these in some secure password vault application of your choice. And also any root passwords he has. Passwords to routers, print-servers, off site corporate backups, corporate accounts (supplier's web sites etc), certificates owned, domain names, email accounts, etc. (You'd be surprised how many small to mid sized businesses wake up two years hence to find their website unreachable because the renewal went to some gone-guy's inbox and/or bounced).
Go over the system layout (map of the network, interconnects, lans, NAS's, servers, etc), and for EACH NODE, ask if anything has been changed since it was created. If you ask if the document is up to date, he'll just say "pretty much" but if you go over it one router at a time, he will remember things that don't appear in the notes for one reason or another.
But mostly pray he's leaving happy, and not pissed.
Sig Battery depleted. Reverting to safe mode.
The first thing is to figure out what are the Most Mission Critical systems, and cover them in order of priority, really try to press the criticalcality of the system.
Top Priority: Systems where there is a Downtime has an immediate impact. There is NO Work Around, it needs to run
High Priority: Systems where there is downtime work around and they can tolerate it down for a few hours while you mess with it
Medium Priority: Systems that can be down for a Day
Low Priority: Systems that can be down longer then a day
Try to get the passwords, or make sure you have a passwords and rights to all the systems work in order of priorities.
Create a network map, inventory every system, switch and router... Make sure you have access to them.
Find the Power Users in the area, they may be able to help you out later on, they may not know everything the sysadmin does but they know their little section and sometimes has tips and tricks that don't get passed on. If there is an issue after he leaves you have contacts.
Get the vendor support numbers if available.
Working in order or priority find the custom stuff programs/scripts etc... Do an overview on what they do, what language affect what systems...
On the second to last day, shadow the old admin, on the Last day do everything, he should only mentor.
After he leaves. CHANGE ALL THE PASSWORDS he knows, and check for back doors in the network to prevent him from entering the system.
Due to short time of transition you will probably stumble a bit, but you should have enough to hit the ground running.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
If he is leaving happy, get his contact info and ask if you can check in with him in the future if you have more questions.
Most of the issues I've run into over the years did not center around HOW something was done but WHY that particular design was chosen. Usually there's one or two weird items at every site that the rest of the system has be designed to accommodate.
Yep, whenever you change core IT people, you have them do a sloppy braindump if possible before they leave, and you clear the new guy from almost every task save updating the documentation and diagrams, with a few mundane tasks thrown in to get procedures down. This means you postpone your big projects if you have a staff change instead of expecting the new guy to shoulder that. Skillsets are not the same as in-situ knowlege.
Someone had to do it.
You might get a couple of freebies with his contact info but I suspect it'd be better policy for an installation this size to set up a paid arrangement with the outgoing sysadmin. I'm not in IT so I don't know what precedents there are around this, but relying on him to reply for free just seems against human nature.
That is basically with every person you need to replace.
If this is an issue at IT level, it will be an issue at every level.
If it is an issue, then the 'hit by a bus' problem will also exist.
At every company I have worked, I see it at some level. Only one person responsible for some task. Sales rep who is the only person who has contacts with specific customers. HR person the only one who does salary. Accountant the only one who knows the password to critical files.
The first thing I try to do when I get at a company is to get away from the 'This is my problem/customer/whatever' and go to 'This is the companies problem/customer/whatever'. This will not be easy for people who feel insecure.
Try to ask 'what department is responsible for ...' instead of 'who is responsible for ...'. And remember : Graveyards are full of irreplaceable people.
Don't fight for your country, if your country does not fight for you.
Some other bits:
First, oddball configs - that is, take notes on any custom settings and processes.
Nothing is more irritating than to troubleshoot something, only to find that the configs are some goofball way-out-of-the-ordinary rigging that somehow works in spite of itself. Or worse, discovering that what looks like a straightforward deal becomes a messy multi-day-outage when you try to fix it according to best practices.
Sure, you can build-up a replacement that has far better/standard configs, or put together better processes... But that doesn't help you out when $system is down and your users want it back *right now*. It's better to at least get some insight into why it was set up the way it was, and you can then plan of rectifying that before it goes down (and as a bonus, knowing how and why it's rigged like it is, making t-shooting a lot easier to do.)
Also, I'd get some insight into what projects he had planned and in process - those will give you some insight into what you yourself will really want to pay attention to. For instance, if there's a backup improvement project planned, it may well be because the existing backup solution either sucks balls, fails any integrity checks you may have, or is about to collapse any day.
Finally, sit the admin down and go over all vendor-supplied services and service contracts (service, certificates, etc), and find out what's about to expire. It would kinda suck if you have your SAN (or worse, core switch, Oracle DB product, etc) crap out, then discover that the platinum 4-hour service contract attached to it expired a week after that guy left... per-hour charges are brutal, parts are moreso, and if your company does the whole PO thing? It's gonna suck.
Overall though - wring that guy's brain out, and record it to audio if you can. It'll save you a lot of headaches down the road.
Quo usque tandem abutere, Nimbus, patientia nostra?
The big difference here is that some filing clerk or HR drone, or Sales exec leaving, pissed or not, does not put your entire infrastructure at risk.
A pissed sales exec might try and take his customers with him. The HR drone won't be missed, they are a dime a dozen.
But the Sys Admin, leaving pissed, can put you in a world of hurt by just changing his phone number, not doing any skulduggery.
A vindictive ex-sysadmin can put your company down for the count months or years in the future, when you least expect it, from a cafe in Puerto Viarta.
Sig Battery depleted. Reverting to safe mode.
If he's leaving happy, ask him (and your boss) to work out an hourly consulting rate so you can reach out to him for the next few months and he'll be properly compensated for it.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
One more thing that I would add, something non-technical, is ask the outgoing guy who in the organization has caused him the most problems. It might just be the idiot CFO who thinks the Sys Admin is the one that needs to fix his laptop when the latest version of AOL has hosed it, or the branch manager whose answer to every network issue is to yank the power plug on the router to reboot it, but sometimes it can be more troublesome, like the mainframe admins who deliberately try to obstruct projects carried out by the Windows admins. Get a really good handle on the workflow, ticket tracking, and reporting requirements as well (I didn't and am still floundering sometimes).
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
Some companies can't afford 2 sysadmin people. It's not that they are deliberately gambling, they are doing the best they can with limited money.
I don't agree. If admin is critical to the future of the business, either they're cheaping out or they shouldn't be in the business in the first place as they're incapable of estimating the real cost of doing that business.
If something fails when I'm home sick and the business suffers, they should be wearing a "Kick me!" sign on their back. They've no right to blame anyone but themselves. I'm human, not a perfect machine or a robot. Expecting otherwise is just wishful thinking on their part. They deserve the consequences.
"Tongue tied and twisted, just an Earth bound misfit
I would argue that the best way to ensure they leave happy is to pre-pay some token amount of that contract. Nothing ownerous, but say 2 days salary/16 hours up front would be an excellent way to grease the wheels. If you are of sufficient size, the roughly $500-$1000 parting gift is a small price to pay for an enterprise phone-a-friend life-line.
Hopefully, they are leaving as an advancement not out of recourse. As someone in the incumbent situation right now with evenly mixed feelings, a small olive branch saying "we know we will be at a disadvantage without you and would like to buy 8-24 hours of your time when you have it available over the next 6 months, keep it anyway if we don't," would go an awful long way to helping me answer a phone call or any other question that isn't oh yea the password is 'RumSkittles3242#$@%_god'. That said, without that, if anyone besides my supervisor calls when (not if) the project they are working on fails, I'm going to say "HAHA, told you so!" and hang up.
I was a programmer for a small firm when I gave my two weeks. I offered them to come by now and again on on Saturdays or answer questions they may have had. Although they didn't call me often and I gladly went over there a few times, I did have to put my foot down and ask them to stop calling me after 6 months. I thought that was enough time for a transition and I only offered my services to be nice... not as a permanent solution to their inability to hire enough people to read and parse my code. The company didn't really want to look at my code or study it or become familiar with it until they needed a change and then they called me up so I could explain things to them. After reflection, I think most companies would either abuse my kind of offer or never call. Would I do it all over again? Yes. I'm a nice guy at heart and I'd make the same offer to the same people. They were a good bunch to work with.
I put this out here as a tidbit of info for others thinking about doing this.