Ask Slashdot: IT Staff Handovers -- How To Take Over From an Outgoing Sys Admin?

Solar1ze writes "I've just started a role in an IT services firm. I'm required to take over from an incumbent who has been in the position for three years. What are some of the best practices for knowledge transfer you have used when you've taken over from another IT staff member? How do you digest the thousands of hosts, networks and associated software systems in a week, especially when some documentation exists, but much of it is still in the mind of the former worker?"

Comments

There is only one way...

[Anonymous Coward]

Hope to Christ he took good notes.

Re:There is only one way...

[icebike]

Hope he is leaving on a good note, and not holding grudges.

Then systematically go through each machine for which he has a password and have him record these in some secure password vault application of your choice. And also any root passwords he has. Passwords to routers, print-servers, off site corporate backups, corporate accounts (supplier's web sites etc), certificates owned, domain names, email accounts, etc. (You'd be surprised how many small to mid sized businesses wake up two years hence to find their website unreachable because the renewal went to some gone-guy's inbox and/or bounced).

Go over the system layout (map of the network, interconnects, lans, NAS's, servers, etc), and for EACH NODE, ask if anything has been changed since it was created. If you ask if the document is up to date, he'll just say "pretty much" but if you go over it one router at a time, he will remember things that don't appear in the notes for one reason or another.

But mostly pray he's leaving happy, and not pissed.

Re:There is only one way...

[khasim]

If he is leaving happy, get his contact info and ask if you can check in with him in the future if you have more questions.

Most of the issues I've run into over the years did not center around HOW something was done but WHY that particular design was chosen. Usually there's one or two weird items at every site that the rest of the system has be designed to accommodate.

Re:There is only one way...

[thereitis]

You might get a couple of freebies with his contact info but I suspect it'd be better policy for an installation this size to set up a paid arrangement with the outgoing sysadmin. I'm not in IT so I don't know what precedents there are around this, but relying on him to reply for free just seems against human nature.

Re:There is only one way...

[houghi]

But mostly pray he's leaving happy, and not pissed.

That is basically with every person you need to replace.
If this is an issue at IT level, it will be an issue at every level.

If it is an issue, then the 'hit by a bus' problem will also exist.

At every company I have worked, I see it at some level. Only one person responsible for some task. Sales rep who is the only person who has contacts with specific customers. HR person the only one who does salary. Accountant the only one who knows the password to critical files.

The first thing I try to do when I get at a company is to get away from the 'This is my problem/customer/whatever' and go to 'This is the companies problem/customer/whatever'. This will not be easy for people who feel insecure.

Try to ask 'what department is responsible for ...' instead of 'who is responsible for ...'. And remember : Graveyards are full of irreplaceable people.

Re:There is only one way...

[Penguinisto]

Some other bits:

First, oddball configs - that is, take notes on any custom settings and processes.

Nothing is more irritating than to troubleshoot something, only to find that the configs are some goofball way-out-of-the-ordinary rigging that somehow works in spite of itself. Or worse, discovering that what looks like a straightforward deal becomes a messy multi-day-outage when you try to fix it according to best practices.

Sure, you can build-up a replacement that has far better/standard configs, or put together better processes... But that doesn't help you out when $system is down and your users want it back *right now*. It's better to at least get some insight into why it was set up the way it was, and you can then plan of rectifying that before it goes down (and as a bonus, knowing how and why it's rigged like it is, making t-shooting a lot easier to do.)

Also, I'd get some insight into what projects he had planned and in process - those will give you some insight into what you yourself will really want to pay attention to. For instance, if there's a backup improvement project planned, it may well be because the existing backup solution either sucks balls, fails any integrity checks you may have, or is about to collapse any day.

Finally, sit the admin down and go over all vendor-supplied services and service contracts (service, certificates, etc), and find out what's about to expire. It would kinda suck if you have your SAN (or worse, core switch, Oracle DB product, etc) crap out, then discover that the platinum 4-hour service contract attached to it expired a week after that guy left... per-hour charges are brutal, parts are moreso, and if your company does the whole PO thing? It's gonna suck.

Overall though - wring that guy's brain out, and record it to audio if you can. It'll save you a lot of headaches down the road.

Re: There is only one way...

[Anonymous Coward]

While I take pride in my work, all my efforts have to be compensated. I am just like any other business.

Re:There is only one way...

[icebike]

The big difference here is that some filing clerk or HR drone, or Sales exec leaving, pissed or not, does not put your entire infrastructure at risk.
A pissed sales exec might try and take his customers with him. The HR drone won't be missed, they are a dime a dozen.

But the Sys Admin, leaving pissed, can put you in a world of hurt by just changing his phone number, not doing any skulduggery.
A vindictive ex-sysadmin can put your company down for the count months or years in the future, when you least expect it, from a cafe in Puerto Viarta.

Re:There is only one way...

[egamma]

You might get a couple of freebies with his contact info but I suspect it'd be better policy for an installation this size to set up a paid arrangement with the outgoing sysadmin. I'm not in IT so I don't know what precedents there are around this, but relying on him to reply for free just seems against human nature.

This is great advice. A two-week after-hours contract with the admin for after he leaves is a great investment.

Re:There is only one way...

[Spazmania]

If he's leaving happy, ask him (and your boss) to work out an hourly consulting rate so you can reach out to him for the next few months and he'll be properly compensated for it.

Re:There is only one way...

[cusco]

One more thing that I would add, something non-technical, is ask the outgoing guy who in the organization has caused him the most problems. It might just be the idiot CFO who thinks the Sys Admin is the one that needs to fix his laptop when the latest version of AOL has hosed it, or the branch manager whose answer to every network issue is to yank the power plug on the router to reboot it, but sometimes it can be more troublesome, like the mainframe admins who deliberately try to obstruct projects carried out by the Windows admins. Get a really good handle on the workflow, ticket tracking, and reporting requirements as well (I didn't and am still floundering sometimes).

Re:There is only one way...

[rickb928]

Assuming you both can find or figure out or guess each device's existence.

Without a current inventory, you are probably doomed. Something will be missed.

I've left a couple of positions on bad terms, but never, never refused any information. I've inherited several positions, and at least half of them were give to me with the incumbent angry as hell. Anyone remember the Emergency Boot CD? I used EBCDs during the NT era lots of times to hijack the Administrator's account on PDCs and BDCs after hostile dismissals. Windows Servers presented bigger challenges. Novell Servers were lots o fun, but doable.

Cisco stuff was the worst to get into in the absence of the previous admin. Most of the time I got lucky and had running.config files hanging around, and I got so I would scan for common config files right up front. Wireshark and other protocol analyzers came in handy to literally scope out networks and links to see what was going where. Once I rummaged through purchase histories to find out what was actually at a coloc and get prepared for going in to figure it out.

But I've never held up a former client. I've taken calls 2+ years after a gig and answered questions. It's unprofessional to do otherwise.

Re: There is only one way...

[Jeremiah Cornelius]

There IS only ONE WAY

FOIA Request to the NSA.

Re:There is only one way...

[ICLKennyG]

I would argue that the best way to ensure they leave happy is to pre-pay some token amount of that contract. Nothing ownerous, but say 2 days salary/16 hours up front would be an excellent way to grease the wheels. If you are of sufficient size, the roughly $500-$1000 parting gift is a small price to pay for an enterprise phone-a-friend life-line.

Hopefully, they are leaving as an advancement not out of recourse. As someone in the incumbent situation right now with evenly mixed feelings, a small olive branch saying "we know we will be at a disadvantage without you and would like to buy 8-24 hours of your time when you have it available over the next 6 months, keep it anyway if we don't," would go an awful long way to helping me answer a phone call or any other question that isn't oh yea the password is 'RumSkittles3242#$@%_god'. That said, without that, if anyone besides my supervisor calls when (not if) the project they are working on fails, I'm going to say "HAHA, told you so!" and hang up.

Re: There is only one way...

[alanshot]

Then you good sir: are an asshole.

As a poorly paid IT admin that manages thousands of devices across every province in Canada, I'm happy to answer questions about anything I've worked on YEARS later.

I agree... to a point. If its more than a quick 10-15 minute call they can expect to be billed for my time. Well, that is unless I was terminated, then the consulting clock starts ticking the minute the phone rings.

There is a fine line between an ongoing piece-meal brain dump, and raping you by asking you to come in for an hour or more to consult. Dont want to pay me for my time? Figure it out for yourself. *shrug*

I've been in both situations... happily answered the phone several times a day several times a week to answer quick questions. But when they wanted me to come in and spend several hours of MY time and gallons of gasoline (note its no longer THEIR time so I deserve compensation for something I am giving to them) damn skippy they can pay me just like they do the other consultants that come and go.

Felt sorry for one poor soul I used to work with. Was the only one in the company that could do what he did, and it was needed at least 3x a year. He let them walk all over him after they terminated him for no good reason. When the time rolled around to do the recurring programming tweak/task/report, they asked him to come in and he did... FOR FREE! *SMH*

Re:There is only one way...

[Common Joe]

I was a programmer for a small firm when I gave my two weeks. I offered them to come by now and again on on Saturdays or answer questions they may have had. Although they didn't call me often and I gladly went over there a few times, I did have to put my foot down and ask them to stop calling me after 6 months. I thought that was enough time for a transition and I only offered my services to be nice... not as a permanent solution to their inability to hire enough people to read and parse my code. The company didn't really want to look at my code or study it or become familiar with it until they needed a change and then they called me up so I could explain things to them. After reflection, I think most companies would either abuse my kind of offer or never call. Would I do it all over again? Yes. I'm a nice guy at heart and I'd make the same offer to the same people. They were a good bunch to work with.

I put this out here as a tidbit of info for others thinking about doing this.

Only one week?

[Anonymous Coward]

Run like hell....

Start with Foundational Systems: Network, DNS,

[Anonymous Coward]

Did this recently. Started with core network topology documentation, moved on to DNS. Foundational stuff. Documenting subnets, figuring out what documentation and systems should be deprecated. Made lots of diagrams. Reviewed monitoring tools. Prioritized systems by importance to review for best practices. Got a network security audit to find holes. Bam.

Re:Start with Foundational Systems: Network, DNS,

[skids]

Yep, whenever you change core IT people, you have them do a sloppy braindump if possible before they leave, and you clear the new guy from almost every task save updating the documentation and diagrams, with a few mundane tasks thrown in to get procedures down. This means you postpone your big projects if you have a staff change instead of expecting the new guy to shoulder that. Skillsets are not the same as in-situ knowlege.

Ideas

[funky49]

Primarily, you'll want to build an honest rapport with the other person. Get inside their head a little and allow them to brag A LOT. Ask how they found the place and what they did to change it. You'll want to breeze through all of the high level and important documentation first so you'll have a baseline. Take as much notes as you can. Ask what websites/resources they use to make it easier to follow in their tracks. Explain your situation to them. It will humanize yourself in their mind and you might be able to engage their compassion for you. Perhaps they would be available to answer questions after they leave! Is there budget money for them to be used as a compensated resource? Hopefully they like the idea of helping others and putting some scratch in their pocket.

Bon chance!

Steve

Re:Ideas

[cbelt3]

Good approach. Making a mortal enemy of the outgoing sysop, or simply his object of scorn, will screw you badly.

Other than to say "you're screwed", the big step is to also ramp up the professionalism and start building a better system governance policy and documentation process. The best way to explain that to management is to ask "Do you fail the Hit By a Bus Test? ".... If your key administrators are hit by a bus, will your systems go dark ?

Create your own documentation

[schneidafunk]

I would start by writing your own manuals and have the outgoing person review them.

Questions

[whitelabrat]

Take over right away. Don't let him do anything. Ask lots and lots of questions. Take notes.

1. Get da passwords. Verify them.
2. Support contracts.
3. "What are common problems"
4. "Can I get your email" :)

Things to do

[Anonymous Coward]

Make sure you have thick gloves and sanitize everything. Check for booby traps. Never push any buttons till you've traced the wires back to their origins.

http://bofh.ntk.net/BOFH/

happend to me this year

[sdinfoserv]

1) Need passwords... immediatly change them.Exiting person should have no futher access except through you.
2) Require exiting person to produce network diagram. Make it their last duty if one doesn't exist.
3) Now starts the pain... audit devices and systems for rogue accounts.
4) document as you go.
5) turn in passwords to supervisor.
Good Luck

Keep them on consulting

[David Gerard]

Keep them on as a consultant, and *pay* that $$$ per hour when you need to.

(This assumes they are quality folk in the first place, of course.)

Re:Is upper management the real problem ?

[David Gerard]

Who the hell manages to become responsible for 1000s of systems and networks without being forced to document them as part of their job ?

Lots of people. Welcome to system administration! Here's your accordion.

(I spent three years documenting furiously. We finally got a third sysadmin. He found my notes incomprehensible. Sigh.)

Get most important stuff first.

[jellomizer]

The first thing is to figure out what are the Most Mission Critical systems, and cover them in order of priority, really try to press the criticalcality of the system.

Top Priority: Systems where there is a Downtime has an immediate impact. There is NO Work Around, it needs to run
High Priority: Systems where there is downtime work around and they can tolerate it down for a few hours while you mess with it
Medium Priority: Systems that can be down for a Day
Low Priority: Systems that can be down longer then a day

Try to get the passwords, or make sure you have a passwords and rights to all the systems work in order of priorities.
Create a network map, inventory every system, switch and router... Make sure you have access to them.
Find the Power Users in the area, they may be able to help you out later on, they may not know everything the sysadmin does but they know their little section and sometimes has tips and tricks that don't get passed on. If there is an issue after he leaves you have contacts.
Get the vendor support numbers if available.
Working in order or priority find the custom stuff programs/scripts etc... Do an overview on what they do, what language affect what systems...
On the second to last day, shadow the old admin, on the Last day do everything, he should only mentor.

After he leaves. CHANGE ALL THE PASSWORDS he knows, and check for back doors in the network to prevent him from entering the system.

Due to short time of transition you will probably stumble a bit, but you should have enough to hit the ground running.

 

beer

[NikeHerc]

Make sure the person leaving knows you'll buy him or her a beer or two when you have a question you can't figure out on your own in reasonable time.

Re:shadow while you can and guesswork there after

[tqk]

Then document as if you might be killed in a car accident on the way home to work and your manager has to take over.

I've never understood why any admin would care about this. If the employer is too cheap to realise they need support in depth to actually be supported, why should I care about the operation going tits up if I get taken out by a bus? They gambled knowing the risks and lost. Suck it up.

Real support is more than one over-worked wizard who knows and controls everything (cf. San Francisco). I want to be training a PNG into the position who can learn, who I can bounce ideas off, who comes in with a different perspective and history from mine, helps with the drudge work, and takes over when I'm not there (sick, recovering from an outage, holidays, bus error).

Any employer who can't see this can go fsck themselves. You get what you pay for. You gamble wrong, you ought to lose your shirt.

Re:shadow while you can and guesswork there after

[tqk]

Some companies can't afford 2 sysadmin people. It's not that they are deliberately gambling, they are doing the best they can with limited money.

I don't agree. If admin is critical to the future of the business, either they're cheaping out or they shouldn't be in the business in the first place as they're incapable of estimating the real cost of doing that business.

If something fails when I'm home sick and the business suffers, they should be wearing a "Kick me!" sign on their back. They've no right to blame anyone but themselves. I'm human, not a perfect machine or a robot. Expecting otherwise is just wishful thinking on their part. They deserve the consequences.