Distributing Unix Knowledge Among Admins?
chadworthman asks: "I work in a server support role with 6 other sys admins. We are all responsible for 10 to 25 servers each (various flavours of Unix), mostly grouped by project. The person who is responsible for a server is called prime. We also identify a sys admin as secondary. This system is not working out well. Most sys admins are only familiar with the environments that they are prime for, and when a prime contact is not in the office or leaves the company, the rest of us try to figure out the environment. We are currently trying to figure out the best way to transfer knowledge of environments between sys admins. We have considered a plan that would involve partnering with another co-worker while you trade knowledge, then after a certain number of months, trade with someone else. I was wondering what other techniques for knowledge transfer between sys admins Slashdotters have encountered."
Set up a knowledgebase of system information, make it versionable, and perhaps commentable in blogs-style. Make it publish to a departmental web server, and have everyone document the hell out of everything there. Things that go there:
Invetories of systems and of software
Licenses and whatnot
Purchase info
Common practices docs (disk layout procedures, installation procedures, patching procedures, downtime procedures, etc)...
etc...etc..
You get the idea. The company shouldn't be reliant on an employee's brain as part of their business plan - document everythign in such a way that if the whole staff went missing, a new staff of competent unix professionals could take over and do somethign useful based on your web docs.
11*43+456^2
It's a bird!
It's a plane!
It's a flock of binders!
But seriously, documentation is key to anything like this. I know most sysadmins wail and moan and gnash their teeth at the very thought, but good documentation is almost as important as good backups!
YMMV but it might actually be worth picking somebody as the "doc-meister" to learn ALL of the systems and have the other admins submit config changes etc. to this person on an on-going basis.
This also helps prevent the common admin trick of just printing out tons of scripts to fill the binder and saying "See, it's all documented, right here" - except it doesn't actually help anybody understand anything!
This way if the documentation lead can't understand it then you know a replacement admin won't either and changes can be made before it's really needed.
The solution presented in the parent post is the correct one. Documentation is fine and dandy, but it doesn't come close to experience. Great thing about rotating primes is that the "true blue expert" is still around, but he is off learning something new too. Everyone puts their ego on hold and pitches in to help. This will generate well rounded techs that can handle a broader array of issues, as a group and independantly.