Slashdot Mirror
How Do You Interview A Sysadmin Candidate?
benedict writes: "The article No Shortage of Programmers? sparked a really interesting thread about how to interview programmers. Being a systems administrator, I am curious about the Slashdot community's collective wisdom on how to interview sysadmins. I have come up with a few questions of my own to prime the pump. 'What is tcpdump? What is it good for?' 'How about truss/ktrace/strace? What are they good for?' 'What's the largest number of machines you've maintained? What have you done to make it easier on yourself (e.g. what types of automation, file distribution, etc.)' 'Do you use source code control? What for?' I would also present a couple of 'hypothetical' situations from my own experience and ask how people would approach them. How about you: what kinds of questions would you ask, what situations would you describe, what kinds of answers would you look for?"
Depending on their answer, slashdot, activewin, theregister, betanews, macworld, or the msnbc technology section!
Here is a list of some other 'inappropriate' interview questions: http://www.sunfeatures.com/inapprop.htm
I do not deploy Linux. Ever.
A few thoughts on interviewing in general:
- Try to ask the same questions to all candidates, but feel free to customize your follow-ups when warranted. The single biggest mistake I've seen is a scenario where candidates are asked very different questions, and no real comparison can be made.
- Ask questions that relate to the job at hand. Don't get theoretical unless the job involves heavy theory. A good interviewer will have enough understanding of the job at hand to be able to ask and understand answers to the question at hand.
- Don't ask leading, "I know the correct answer and you have to figure out what I'm thinking," questions. I've seen this happen as well; an interviewer asks a leading question, the candidate gives a reasonable (but perhaps incorrect) answer, or at least exposes some interesting and appropriate thoughts on the matter, and the interviewer says, "Wrong!"
- Don't assume that there is a piece of knowledge that is an absolute marker of proficiency or understanding of a topic. Sure, any experienced network admin would probably know about tcpdump, but there are certainly sysadmins who don't have extensive networking experience.
- Prepare for the interview process by assessing what knowledge you require of a candidate, and what you can teach. For example, you shouldn't expect candidates to have knowledge of the specific oddball hw/sw that you are using (e.g. eGate, Citrix, f5). Sure you might some who do, but don't assume that because they have experience with this one area that they are the ideal candidates.
- Others have mentioned it, but it bears repeating; ask _open-ended_ questions to get a sense of the candidate's thought process and problem-solving skills. It is impossible to know "everything there is to know" about any tech/IT job, so you must have someone who can think and solve problems.
I am really worried that any of you guys might end up interviewing me someday. Really. Interviews are a great way of succinctly assessing ability and knowledge, but only if you're ready to get to the meat of what you want.
Here's the thing -- I'm an instructional technologist, so I'm always thinking about how people learn about technology, and how best to move information around regarding technology. So maybe it's easier for me to get to the heart of interview questions by breaking down possible questions into categories . . .
First: Identify what it is the candidate will really be doing for the organization. Is this someone who should be able to turn everything around for the better? Is this person just supposed to fill the shoes of someone who's leaving? Is it a new position? Is this someone you're setting up to take the blame when that domain controller you've been nursing along for months finally tanks? Ask questions with the context in mind.
Second: Consider the level of knowledge that is required for the job, and ask the highest level questions first. Here's how learning works --
** The lowest level would be facts (list the components of OSI model)
** Next would be concepts (what are the characteristics of ethernet versus FDDI)
** Then there are relational rules (how do you know when to back up your account database) and procedural rules (list the steps you follow for a Win2K deployment)
** And finally there is problem solving (apply what you know about disaster recovery and backup plans in telling us how you would react to a long-term power outage.)
So when you're interviewing a potential sysadmin, start with some of the how-well-will-you-fit-in questions, and then follow up with questions appropriate to the level of knowledge you expect them to use on the job . . . and, oh by the way, you might be prepared to answer all of those questions yourself. Makes it easier to turn an interview into a conversation, and I really think conversations will tell you more about a technologist than your standard grilling of the suspect.
Don't ask them little trivia questions like "What does this utility do?". A sysadmin is more than just a person who does routine tasks every day, they have to solve new problems as they arise. You should interview a person by asking them problems and how they would solve them. Ask them about big problems they encountered in the past and their solution to them, or problems they weren't able to solve.
What you want to find is someone who is interested in what they do and can learn new things as new problems arise. You don't want someone who just memorized a book and some man pages, because what will they do if something happens that wasn't in the book they read?...
Outdoor digital photography, mostly in New Engl
I just went through this: being interviewed for a sysadmin job. It's a pretty prestigous place and they were getting 60 resumes/day. My boss for a while there was phone interviewing 20 people/day.
There were a few that made it to the in interviews done in person...and that had to have been teh roughest gauntlet I've done. 9 hours and 14 people. Even lunch was an interview. They asked anything from C coding minutae to very simple sysadmin to favourite accomplishments to my favourite hack.
They also encouraged me to ask questions. THAT would be an excellent way of telling about your candidate. What does he/she ask about? Watch that and you can get some peaks. Sysadmins shouldn't be timid! Nor should they be overbearing.
The other thing that they seem to be noticing is whether or not they are are salary hoppers or not. They've been quite purposefully screening out those that change jobs every 6 months...partially due to the fact that they are going to invest a lot of money in training and such. Loyalty is important too.
Finally, ask things they don't know...as someone else pointed out. How they respond matters quite a bit.
*chuckles* I got the job and I'm quite pleased with it. It was also the roughest interview I've ever had.
Do you know why the road less traveled by is littered with the bones of the unwary?
The Best Questions are Obviously...
Define Red Shift and Wake Turbulance
Than make them recite the Distance Formula for any 2 points on a graph.
If they know the answers do not hire them. Everyone else qualifies.
Part of our technical interviewing involves sitting down with the candidate and a laptop that has BSD loaded on it. There are little problems that we throw at them to see not only if they diagnose them correctly, but can find a workaround. Then we can see problem solving skills in action.
:)
.login. This was causing new shells to be invoked until the user had no further processes. Follow up question was: Why would a user do that? (Trying to change his shell in a not so smart way. :-)
Example, we have them cd to a directory where one of the files is named '-' and have them remove it. rm -. Ooops. Now, how to get around the fact that rm is parsing that as the first part of a flag instead of a filename? The goal isn't to completely stump them, but to give them small cases to display some problem solving skills.
Another question asked during "the laptop interview." "What OS is the machine running?" Not everyone knows the 'uname' command. For those that don't, there are other options! Like the header of the man pages. Or the log in screen. I personally didn't know the uname command during that interview (yes, I took it) but remembered machines advertise the operating system at the login prompt, so I logged out. Not conventional, but they weren't looking for conventional. They were looking for problem solving under pressure. (Being on the spot in an interview like that is pressure. Especially if you don't know the obvious answer and you know you don't know the obvious answer.
Another one they threw at me was having me log into an account. When the prompt appeared the first thing I saw was "no processes" followed by the prompt. If I tried to run any commands I got 'no processes'. What was going on? I tracked the problem down the the user having 'tcsh' in their
I liked that particular problem as it involved diagnosing the problem, and then why the problem occurred, which involved getting into the "virtual" head of a confused user.
Most of the problems in the laptop interview are pulled from real-world examples of problems submitted to sys admins by users in our own environment or others.
"I don't know the answer to that question... and yet I am an excellent sysadmin."
Of course you do, it's: "I don't know the answer to that question, but I know where to find it, and how." That is a much better answer than going forward and pretending you do. In fact, it's the difference between possibly getting the job, and being immediately ruled out if your interviewing with me! Of course, if you can't tell me how to go about finding the info, the interview is definately over!
Guns don't kill people; Physics kills people! - John Lithgow as Dick Solomon on Third Rock From The Sun
I (as a software engineer, not a sysadmin) was in a similar situation in my last round of interviewing. Guy asked some programming problem "How would you fix XYZ" I replied with "By doing ABC then DEF" "Well, think a bit different, what about Y again?" "Er, I suppose I could run LMN on Y and see what it showed" "Right, but if Z got in the way..." This went on for about 5 minutes, until I figured out he was looking for a SPECIFIC answer, not just a correct answer.
People skills! Programmers can hide in their
caves and snarl when thrown their ration of caffeine and carbohydrates. Sysadmins gotta be able to smile while being asked: 1) to do the impossible, 2) Yet Another FAQ, 3) add another user, 4) restore from backup, 5) say no to some bigwig who wants the security policy violated for his/her personal whim, *) you get the idea.
Be good to your sysadmin, even if it isn't his/her day.
technical information can be learned reasonably easily but some things cannot be. when i was interviewing people i looked for people who had a genuine interest in IT too and how easily they could pick stuff up, and how well they could manage a number of concurrent tasks.
1) Tell me about the things that you have done at your last job or two. Tell me about how you got 10 different types of boxes to work together or implemented a system to do something or whatever. I don't really care if you know all the options to tcpdump by heart, you can look them up when you need them. I would much rather know why you have done the things you have done and how.
And 2) Do you have any questions about working here?
Erlang Developer and podcaster
I tend to start with lower level questions and work up until I find out where the candidate's knowledge reaches (or that it surpasses my own :) I start with things like "name the fields in the password file" and "how do you tell how much disk space is being used?", then progress to networking: "how do you determine the number of network interfaces and their IP addresses" and "what is a default route, how do you determine it and how do you set it".
:)
If they're still going strong, I'll get into volume management, system tuning, bare-metal restore, and troubleshooting scenarios. So far, I've only had to interview for admin positions for one OS (Solaris, AIX, HPUX) but if a candidate is strong in one OS, I'll usually give them some leeway towards being able to apply that knowledge to the new commands and syntax without much difficulty.
If there's time left over, I'll ask about scripting (emphasizing perl and ksh) and/or other programming skills (can't be a sysadmin if you don't script!
Excellent! That's a good idea; I always hit them with an un-plugged box, just to see if they know Rule Number One: "CHECK THE PLUGS!" but a bad cable is even better.
I'm a big believer in practical exams like that. Sit the candidate down at a box, ask them to perform some simple tasks or give you some information that should be relatively easy to find. I'd rather have someone that can sit down and work with their hands on a problem than spit up textbook answers all day long. You can read up on esoteric commands and techniques all you like, but if you can't translate it into results on a screen, you're worthless. This is also a good way to see how they perform under pressure, and whether or not their afraid to look for help elsewhere. I'll take the guy who fires up the web browser and finds the answer in thirty seconds off Google a long time before the schmoe who spends ten minutes slogging through machine settings and racking his brain for it.
No relation to Happy Monkey
One of my friends went for a sysadmin interview, they sat him down and started. The head guy said "what does lsof do?". "I don't know". "You just failed". And that was that (I'm not making this up, that was his interview).
A few jobs back I ended up doing a lot of the sysadmin work for a good part of the company. As a joke, my boss dropped a "sysadmin" magazine on my desk one day. It turned out there was a timely article in it... The article talked about how well programming tests work in a dev organization. Some folks did a study where they applied a how _bunch_ of tests to candidates, but did not _do_ anything with the results; they just kept the data around. Two years later they looked at peer feed back, performance reviews, etc, etc, to look for "key" people and tried to find a correlation. The result was interesting: nothing for the programming test part, etc. BUT, there was a strong connection between "interesting" hobbies and key people. I.E., if you did something like gourmet cooking or sky-diving, chances were good the person would end up being cool. So, now when I interview, I always ask what peoples hobbies are. It seems to work pretty well! --stuart
Then came the technical questions. The killer one was What are the file type on Unix?. Most guys answered "normal ones, directories, and hidden" (I'm not making this up), at which point I was thanking them for attendance. If the "filetype test" was passed, I asked them about common tasks like adding users (How do you add a user without using the adduser command), installing software, familiarity with packaging systems (the company used Redhat/Immunix and Debian), how do you patch and compile a kernel (a second common point of failure - esp. the patching part), how do you install software that is not packaged, what to do if this not compiles out of the box etc.
There was the networking part where the killer question were What is the CIDR? and Are the non-continous netmasks legal?
And believe me, it was rare that any candidate would pass this part.
Alex
As a programmer, my goal is to hire someone at at least as competent as myself. As an expert, its easy for me to spot such a person. However, most organizations I have worked in have had only one or two sysadmins. You may not have an expert sysadmin around to evaluate if a candidate's response to a tcpdump question is valid.
So, my quesion: How can a non-sysadmin interview a potential sysadmin? How does a startup find a good sysadmin, or a good lead programmer if no one there is an expert in the field.
One possible solution is to have the last person to hold that position involved in the interview process. But if you are getting rid of a poor sysadmin, you are likely to get another poor sysadmin.
We usually interview in teams. Each member of the team comes up with 10 questions or so ranging in difficulty from very easy to impossible (or at least improbable). We try to avoid "memorization/certification style" questions (eg options to ls) and go for problem oriented questions (although there are some OS specific questions as part of the process).
...
At some point after the "get acquainted" part of the interview we start the technical part. The team members will start asking questions that escalate up the difficulty chain until they simply can't answer the questions anymore.
Purpose number one is to find an approximate level of technical skill. The second major purpose is to find out how they handle stress. Sysadmins need to be able to operate efficiently under very high stress loads. If during the interview they remain clam and composed even when they don't know the answer, they at least have a chance at handling the normal work. If they fold during the interview they probably won't handle a raving user very gracefully.
For these purposes, the particular questions are don't matter much; anything that will stump them will do. The other thing we really ask about is their home system and what sort of work they've done on it in the past. People who are sysadmin material will usually expound at great length about all the tweaks they've made to their systems. Sysadmins are compulsive tinkerers; even those with little experience or those who have left the field can't resist the urge to make the system behave as it's *supposed to*. If the candidate turns it on and forgets about it
Finally we throw some real life "difficult user" problems at them to see how they deal with the situations. This tells us how quickly they think on their feet when there's no technical answers to the questions, and how far they'll go before looking for help. This is one of the realities of the job, so it's a good idea to check for it during the interview.
This process has generally worked pretty well. It's still possible for people to sneak through, but you at least eliminate the candidates who are obviously not right for the job.
We start in the cubicle and have them talk about their current and past job. The kind of things they did. The kinds of machines they worked with. The kind of environment it is.
Then, we go into the technical questions. Things they should be able to explain. Like when you use 'uptime' (or 'w'), what do the three numbers after 'load average' represent? Here. Run a command on this box and tell me how many processors it has, what speed, and how much memory. I run 'top' on a busy production box and have them describe what they see. That kind of thing.
Also, I give them a tour of the datacenter and, while we're in there, have them identify some simple things like cards. Or I'll poing to a Sun E4000 and ask them to tell me, in general terms, about the E4000 server and what it is capable of. I might ask what the difference between the E4000 and E4500 is.
Yes, I also see how they respond to questions they don't know the answer to. But a lot of what I look for is personality. How well they're going to get along with the group and others.