Slashdot Mirror
Distributing Unix Knowledge Among Admins?
chadworthman asks: "I work in a server support role with 6 other sys admins. We are all responsible for 10 to 25 servers each (various flavours of Unix), mostly grouped by project. The person who is responsible for a server is called prime. We also identify a sys admin as secondary. This system is not working out well. Most sys admins are only familiar with the environments that they are prime for, and when a prime contact is not in the office or leaves the company, the rest of us try to figure out the environment. We are currently trying to figure out the best way to transfer knowledge of environments between sys admins. We have considered a plan that would involve partnering with another co-worker while you trade knowledge, then after a certain number of months, trade with someone else. I was wondering what other techniques for knowledge transfer between sys admins Slashdotters have encountered."
You need to try to adapt each environment towards a single standard that everyone then becomes familiar with. Yes, this will sacrifice some features of each platform, but that is the price you pay for greater scalability and flexibility. This is the kind of thing that made different flavors of Windows so popular with sys admins, and it's high time the Unix world followed suit.
Karma: Good (despite my invention of the Karma: sig)
Documentation. It's what you need. Some standardization would probably help too.
Set up a knowledgebase of system information, make it versionable, and perhaps commentable in blogs-style. Make it publish to a departmental web server, and have everyone document the hell out of everything there. Things that go there:
Invetories of systems and of software
Licenses and whatnot
Purchase info
Common practices docs (disk layout procedures, installation procedures, patching procedures, downtime procedures, etc)...
etc...etc..
You get the idea. The company shouldn't be reliant on an employee's brain as part of their business plan - document everythign in such a way that if the whole staff went missing, a new staff of competent unix professionals could take over and do somethign useful based on your web docs.
11*43+456^2
Every two months, the primes all rotate. After a year, you will all be experts on all systems.
And the men who hold high places must be the ones who start
To mold a new reality... closer to the heart
It's a bird!
It's a plane!
It's a flock of binders!
But seriously, documentation is key to anything like this. I know most sysadmins wail and moan and gnash their teeth at the very thought, but good documentation is almost as important as good backups!
YMMV but it might actually be worth picking somebody as the "doc-meister" to learn ALL of the systems and have the other admins submit config changes etc. to this person on an on-going basis.
This also helps prevent the common admin trick of just printing out tons of scripts to fill the binder and saying "See, it's all documented, right here" - except it doesn't actually help anybody understand anything!
This way if the documentation lead can't understand it then you know a replacement admin won't either and changes can be made before it's really needed.
So make them act like a team. All admins are responsible for all servers. I am assuming that you the group doesn't have a lot of time to document (most groups don't), but there are still practical ways to make it work, with minimal time taken in advance:
/home, all apps in /apps, all admin-only stuff in /admin (or whatever standards you want to use) /admin/local/README or whatever) and on your team webserver - if you don't have one, get one /admin/local and on the webserver, that describes any changes that aren't in someone's home directory and which survive a reboot - who did them, when and why
1. common file system layouts (for example, all users in
2. one person (team lead) owns all of the licenses, and keeps them up to date, as well as scheduling non-reactive work
3. if you're not responsible for the applications on the system, then everyone should be able to handle any machine, since no specialized knowledge is needed
4. of course, specialized knowledge is still needed, because some systems have quirks. Document the quirks only (not standard routines for the whole team) both on the machine (in
5. keep a change log for each machine, in
6. make sure than standing orders (that is to say, procedures to always be followed, like how to notify clients of an outage) are posted on each machine and on the website
7. use a common root password, known by the team lead and his manager. everyone else uses sudo su - to get to root, or some similar means. give them the root password if they need it (reinstall system, for example), then change it the next day. ideally, set up a system so that each admin saves to a different history file, so that you can tell who did what if you need to (tracking down mysterious file disappearances and such) - this isn't a tool for discipline, it's a tool for troubleshooting
That should solve most of the problems.
-- Two men say they're Jesus. One of them must be wrong. - Dire Straits
Isn't this a little sad and the amount of comments for this "ask slashdot"? Considering that 1/2 the readers here are probably admins themselves you think we could come up with some more feedback then this.
For the last couple of years I have worked in just such an environment. Our biggest push the last year has been standardization of the documentation - what is in there, how it is organized, common issues each system has (does one webserver have a rogue java that requires periodic restarts, etc.), and everything in a central SECURE location that all the admins can get to at need.
/apps filesystem). Content in a standard place (ie. /apps/content) and so on. Startup scripts, and all those other fun stuff standardized. Infrastructure, DNS and bastion hosts, all that fun stuff needs to be built.
Prior to that, the system setups had to be standardized. The applications in thier own directories, running as non-root user (ie. under an
So, first - a standardized (yes, there will be minor differences between systems) image for administration, second - documentation. For documentation, any admin should be able to pick up the documentation, and with as little effort as possible, do what is needed.
And last, the issue of rotation - Yes. On a bi-annual basis, move people's assignments. Secondary becomes primary, primary becomes secondary on another project, etc. With in a year or two, everybody should have had experience on multiple systems, and the one person knowing everything syndrome should be gone.
1. Document what you've got. Make the doumentation standard.
2. Move the 'primes' around every couple of months so you all get exposure.
3. Common install base. make sure you can automagically install from scratch the O/S's and applications (ge jumpstart on SOlaris, HPUX and AIX have their variants). If at any stage you need to type anything you've failed.
4. Read, digest and implement "The Practice of System and Network Administration" by Limoncelli and Hogan ISBN: 0201702711. This is a great book for any admin and for me is the K&R of its subject.
(various flavours of Unix)
That's where your problem lies. Pick one, and get everyone comfortable with that one.
Tarsnap: Online backups for the truly paranoid
You're kidding, right?
They'll compromise a real weak server/app, and then hit the rest of your network up
Essentially, you have a learning problem: how does one efficiently propagate detailed knowledge within a group?
:-)
Answer: Sex. Continually pair off, sharing genes or, in this case, memes.
I was brought into a company a few months ago for six weeks to finish a demo of a networked, multithreaded peer-to-peer system written in C++. Even though I hadn't much experience with networking or threading, I was up to speed in 2-3 days because we were doing extreme programming. Our pairs swapped every few hours, insuring that I saw a lot of the system in a short amount of time.
In your case, I would set up double-rotations. Separate out a handful of logical groups of systems, so that you can have at least two pairs working on a particular group (if you have a ratio of 20 systems to admin and 320 systems, you could have 4 groups). They pair swap at least twice a day.
Every so often (e.g. 2 weeks, 1 month, 2 months), have group swaps such that half the members in a group are swapped.
The result is that when someone leaves, there are at least 3 other people in the affected group that have more-or-less the same knowledge as the person who departed. Specific knowledge travels into different groups when you have a group swap, but only so often, which means you don't have a loss in productivity by folks getting up-to-speed all the time. Over time, best practices will be learned by everyone and a positive team culture will develop.
Not to mention that you'll make fewer mistakes since everything happens under the watchful gaze of a partner...
(not that mistakes will disappear entirely
Pairing just works. Why not try it out with half of the staff and see how people like it? Make sure to get caught up on how extreme programming works and figure out what you want to adopt and what isn't feasible for your particular situation.
Jon
Why not a Vulcan mind meld?
Should effectively transfer all the needed knowledge, and a little of each persons personality, but that might not be such a bad thing.
I know Solaris/HP-UX/Linux/*BSD and Win2K and Some Cisco. AND I happen to want a new job...
Hire me please........
Simply make it known that the next time a sysadmin is called upon to support a box for which he is designated as 'secondary', and his response is inadequate due to lack of knowledge, he will be fired.
After the first sysadmin is fired, I guarantee you that the remaining ones, plus the new guy's successor, will very quickly come up with a system or systems which allows the proper level of knowledge-transfer.
Everyone is very technically competent in each operating system. The problem is the "quirks" of each environment (Oracle, apache, Tuxedo, Tivoli, WebLogic, etc...).
Not to rain on your job-seeking parade but I know all of the above plus Netware 2.x through 6.x, NT all the way back to 3.50, AT&T Sys-V (not that anybody cares any more) and DG's AOS-VS and it STILL took me a year to find a new job. :(
:)
Admittedly I have been pretty picky, I probably could have gotten a job in a month or so if I just took the first thing that came along, but who wants to do tier-one tech support?!?
Thank god I've never been laid off...
(Well, okay, I'll admit it, I was once - but I was only 12 years old.
In the company I work for I ported all the scripts to Linux and Sun from HP. Thus we have 'sameness'. you want to build a database. It does not matter what platform you are on the command is the same. You want to create a new dev env. It does not matter what platform you are on the script is the same on ALL platforms.
By creating a level of 'sameness' across all your platforms it will not matter weather the server is Sun, HP, Linux, BSD, whatever the scripts will all be the same. Since you are talking about being an admin I'd suggest all scripts in perl or sh. The problem you may run into with perl is that perl rarely gets installed in the same place on all platforms. Thus the start of a script with /usr/bin/perl may not work, where /bin/sh will. Yes and there are coding ways around the perl issue as well.
Granted you will have different machine that do different jobs, this is where documentation comes in. Make sure that all your stuff is documented. If someone sets up a server they need to be required to describe how this server was set up. Using the principle of sameness this cuts down on the need for lots of docs, and thus anyone can set up the server.
Shells.. standardize on a shell. Standard login shell. More importantly is the standardization of what shell people use. I go with tcsh, as I like it better than ksh csh and sh, and it is available everywhere (Sun, HP, BSD, Linux, etc). It is also feature rich. You can standardize on any shell, but make sure it is everywhere you need it to be.
Once you have standardized on a shell, use a standard login env. Thus when you login to your BSD box it feels like your Sun box, which feels like your Linux box, etc.
If people want to add to this have a process in place to make it happen.
Except for system tools like Sam on HP, and Redhats sysadmin tools, there is no reason that many other tasks cannot be done in scripts that are standard. You can even standardize on what a database server setup should include, what a web server setup should include, and have standards that are the same or different (I prefer sameness unless performsance is an issue) for each flavor of UNIX.
Only 'flamers' flame!
The job market sucks BIGTIME right now. I am at a dot.com that may soon turn into a dot.bomb.
Hopefully I will find something before that happens. I have been looking for hard for about 3 months now.
I could have had 3 or 4 positions for security IF I wanted contract work.
Any good suggestions where to look? The job boards are pretty much worthless. I appreciate any suggestions.
I have found this to be very effective.
Healthcare article at Kuro5hin
I am assuming you have heard the standardization tips throughout this conversation. However, if you are unable to standardize the environment (maybe because its a client's environment and you have no control over it), you can try to adapt one of the concepts of Extreme Programming.
Pair programming is a great way of doing knowledge transfer.
Basically always work as a pair when doing problems. Also, rotate the pairs around so not everyone is stuck with the same job. Also make sure you rotate the "primes" around so that the "sys admins" can take over the prime role for a short while. (Just make sure the "prime" can be contacted if there is something really major).
Archie - CIO-for-hire
As mentioned by other posters, documentation and job rotation is good.
To make sure the documentation is up-to-date, accessible etc. you can make the support calls from the secondary admins to the "prime" non-paid. After a few support calls in the night, on your holiday etc. you will make sure that the documentation is up-to-date, accessible, people know where it is, etc.
Yes, I am evil :-)
Your company needs to do two things.
First, fire the manager who got you into this situation. If you've "been doing it this way for years," fire the manager who left this system in place. (If that manager just left and the new guy realizes there's a problem and that's why you're asking this, then obviously there's no action taken against him.)
I'm not being bloodthirsty here - anything short of this will leave people doubtful that upper management is serious that this is *the* biggest problem your company faces today, and people will continue to do what they've been doing for anything but the most trivial problems. Senior management needs to send an unambiguous signal that the status quo is unacceptable.
Second, rotate the primes and secondaries as others have suggested, but with a twist. Rotate the secondaries first, and their sole responsibility is to write a list of questions - a long list of questions - about everything that "surprises" them or that needs to be documented somewhere. (An example of the latter is "what are the partitions, what are their sizes, and how was this size determined?"
They turn over their questions to the primes who spend a few weeks documenting the answers while the secondaries cover for their old prime, and this documentation is provided to the next set of secondaries rotated in to ask questions. Lather, rinse, repeat.
By the third time around (maybe 3 months?) you'll have documentation that actually covers almost everything someone will need to get up to speed on the peculiarities of a particular project, and the primaries can start rotating while the secondaries answer any remaining questions.
Finally, I'm deliberately putting the emphasis on the secondaries here because one of the classic problems with your old setup is that it can cause the secondaries's skills to stagnate if the prime handles all of the "hard" or "interesting" problems. You need to give the secondaries room to grow, even if it increases your turnover rate because they're competent enough to be hired as primes at other companies.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Put the pertinent information in a WikiWikiWeb, then any of your admins can access or add info from any web browser that can see the Wiki server. Plus the whole idea of Wiki is so neat that they'll want to use it and watch it grow.
Downmodding is the refuge of the weak. Don't downmod, make a better argument!
I had to laugh when I read your response - I'm taking night classes for my MBA right now!
But to answer your question I found FlipDog to be, hands down, the best job board!
E.g. on Monster/Dice et al I search for "network" in the Seattle area and I get about 5 to 15 hits, most of them pretty lousy entry-level positions.
The same search on FlipDog netted me about 150 positions, half of which were actually semi-relevant!
The only problem I had was cycle time. FlipDog actively goes out and searches company websites for positions so a "new" position listing may actually be a week or so old and/or already filled (esp. in this economy).
I did, however, find my new job there! (When even good local head-hunters who have worked wonders in the past couldn't.)
Good luck!
Often, when writing documentation we leave out things because we take them for granted or "well, it's just like that." Other people without that experience don't know that, and the gaps become apparent QUICK.
Documentation AND experience. One can't take up the slack for the other being weak.
--critical skills cross training is the special forces criteria. It works for them fairly well. NO ONE knows just one job. You have at least three jobs you can do well, two that you are a specialist at. Whomever is the primary in a skill on the team can be instantly replaced by his secondary if he becomes an ineffective.
Same idea. Documentation is good, hands on automatic task-accomplishment is even better.
Watching really competent people work is a great way to learn your skill - like playing with really good musicians, your ability will improve as you learn all manner of cool tricks.
It used to be that you could learn a great deal by building software. That can still be true, but it's mostly now a ./configure-and-forget kind of
world (building TeX from Knuth source is excruciatingly painful. TeTeX saved TeX from
self-immolation... but I digress).
A strong leader can (and should) be very liberal with assigning projects that he knows the victim is unfamiliar with. That sort of where-do-I-begin anxiety usually leads to the right sort of questions, and gets communication flowing between your team members.
We shoot script(1) up everytime an admin procedure gets started so we document every line that appears on a terminal. Later we add commentaries for each of these lines, explaining it's purpose, and archive the hole file.
With this you can infer what an specific environment looks like, how installations were dealt with or problems were solved, amongst other administrative duties.
I guess the only disadvantage is that all your administrators will have to learn how to get away without emacs and vi, since they usually don't do well with script(1). Of course, there's always ed(1).
Utinam logica falsa tuam philosophiam totam suffodiant!
A lot of suggestions in this thread basically consist of "tell the admins to document everything." Which would be fine, execpt for the fact that your average admin isn't really very good at writing documentation. Depending on how many admins/servers/projects you have going on, you could do well to either hire a *nix-savvy tech writer on contract or hiring a full-time tech writer.
Too many companies try to get by with sub-standard docs produced by people who aren't qualified to be writing documentation thinking that they're saving money by putting the job of documenting on the backs of people who are already overworked and will do whatever they can do to avoid writing documentation. In the long run, said company ends up wasting money by having people recreating the wheel or spending time having to fix problems that would never have arose if good documentation had been available in the first place.