How Do You Interview A Sysadmin Candidate?

benedict writes: "The article No Shortage of Programmers? sparked a really interesting thread about how to interview programmers. Being a systems administrator, I am curious about the Slashdot community's collective wisdom on how to interview sysadmins. I have come up with a few questions of my own to prime the pump. 'What is tcpdump? What is it good for?' 'How about truss/ktrace/strace? What are they good for?' 'What's the largest number of machines you've maintained? What have you done to make it easier on yourself (e.g. what types of automation, file distribution, etc.)' 'Do you use source code control? What for?' I would also present a couple of 'hypothetical' situations from my own experience and ask how people would approach them. How about you: what kinds of questions would you ask, what situations would you describe, what kinds of answers would you look for?"

Comments

Here's a working link...

[krugdm]

...to the No Shortage of Programmers article.

Our standard 20 Questions

[Pogie]

At my office, we've gotten real tired of headhunters and placement firms sending us people who call themselves "Unix Admins" and have taken Solaris 101, or AIX 101, or whatever and think they know what they're doing. Here are questions that we ask every candidate. If they answer a couple wrong, no biggie, but some of them are gotchas. BTW, this is UNIX specific (frankly, WinX admins aren't system administrators. They're WinX admins. A system administrator supporting a WinX box is a very unhappy person.)

1. What are the fields in the passwd file? What do they mean, what do they do?
2. How does enabling shadow passwd's change the functionality of user authentication?
3. Pick your favorite Unix flavor. Pick your favorite HW in that flavor. Walk me through a system load, the whiteboard is over there.
4. What are Unix runlevels and how do they work? Flavor-specific answers are acceptable.
5. What's a socket? What's a port? How do you reserve them? how are they related? How do you find out what's going on with them on a system? (again, pick your flavor)
6. Do you know Perl?
7. If the answer to 6 is No, then why the hell not?
8. How would you grab the middle 300 lines of a 1000 line file, grab the second field of every line in that range, and sort the result alphabetically using only piped commands from the toolbox?
9. Do you know vi?
10. Pick your favorite Unix. Great, now tell me how you would a) mirror the rootdisk, b) grow an existing filesystem, or c) modify the partition table for a disk on that OS.
11. What are the various levels of RAID and what differentiates them?
12. What do you know about NIS? (or NIS+, or LDAP).
13. What do you know about DNS?
14. What do you know about NFS? Automounting?
15. What do you know about Firewalls?
16. Describe the various metrics and procedures you would use to evaluate the performance and system utilization of a Unix machine.
17. What's your biggest fuck up, and how did you fix it?
18. If you didn't know how to do something, how would you go about figuring out how to do it?
19. Do you smoke? Drink Coffee? Drink?
And finally:
20. Ok, pretend for a moment that the entire network crashes and the CEO is in your office wanting to know why. What do you say?

:-)

Pogie
(The only good NT server is a down NT server)

Interviewing sysadmin candiates [unix only]

[GoNINzo]

I've interviewed quite a few, and there are 'sets' of questions you can ask. You don't want to follow a work book, but there are a bunch of formulaic questions you might ask:

• Describe the process in which your favorite version of comes boots up from a cold state. Please use as much detail as possible. The advantage of this question is that there are TONS of sideroads to check. Also, you find out how interested in the underlying part they are. Also, you can see what run control scripts they hit, and you can hit those applications later... Or better yet, they can tell you things like what run level 4 on Solaris is, etc. (ie, trick questions)
• OSI layers? BSD vs. SysV? This tests if they are well rounded. You see if they've touched networking, you can see if they even know the book learning on the different OS's, and get general 'you need to read a book to know this' type stuff. Also, asking the differences between things in the simplest possible terms is another good test to see if the candiate has the ability to talk to managers. `8r)
• Favorite OS and why. Any good unix candiate belives in 'the right tool for the job'. Anyone who says that 'Linux is the answer to everything' is fooling themselves. All the different Unix OS's have their advantages, and the key to having them explain theirs. They don't have to agree with you! that's the key. But they should at least make sense. But don't hold it against them if the answer is 'Because I know that OS the best'. It's a common one. But do NOT let them just say 'Oh, AIX sucks' etc. If they can't back that statement up with facts, they obviously havn't looked at it close enough.
• How would you rate yourself on DNS? Ah, an expert, eh? What are the different types of records? What are some limitations of MX ones? Get deep into at least one major unix process. Sendmail, NFS, NIS, and file systems are all very good parts to go into detail on. By asking how they rate themselves, they show either a) they know what they're talking about and rated themselves appropriately and b) They are rating themselves guru-level when they have trouble remembering even the names of the parts of the program.
• So I had this really hard problem... I was seeing this kind of behavior... What sort of things would you check to solve the problem? No, I tried using This calls into all their troubleshooting skills. You see how deep they go, what they go to next, and why. There are a couple sendmail and NFS problems that can run the gamut.

One more thing... Don't expect excrutiating detail on a process that you don't know either. IE, don't ask a person questions that you don't know the answer to either! And if you do feel inclined to ask about something you don't know, make it clear that you're coming at it from a newbie's point of view.

If people have further questions, i'd be happy to answer them.

Random thoughts

[andy@petdance.com]

Random thoughts from my past, in sysadmin and elsewhere:

• Last programmer I hired was because I saw her reading Webmaster In A Nutshell on a commuter train. "You wouldn't happen to be looking for a job, would you?" Bing-bang-boom.
• I always ask "How do you keep up?" Good answers: "I read Slashdot", "I have a $2K/year book budget", "I'm trying to get a complete collection of O'Reilly books". More importantly is how they reply. If they have to think about it, then they don't see it as a priority.
• Culture in general is key. Any mention of Slashdot or similar fora is almost a requirement.
• Immersion questions are as important as specifics. Someone who crammed through Teach Yourself Perl In 72 Seconds may well know the three basic data types, but only an experienced programmer will know who Larry, Randal and Tom are.
• Ask him to DO something. I've given printouts of suboptimal web pages to the candidate and said "OK, how would you fix this?" (Tell him it's an old version of the page, so he doesn't have to worry about offending anyone with his comments) Read Nick Corcodilos' excellent Ask The Headhunter for more about this sort of interviewing.
• It's critical that the person actually care and/or be interested in the business that you're in. My company is in the education business, and in my eyes, it's important that they see the industry itself as valuable.
• Ask about what kind of machine they have at home.
• DON'T ask the standard bullshit questions, like "Where are you going to be in five years?", "What are your greatest strengths?", etc etc unless you actually CARE about those things. Chances are, you don't, and you can't even answer those yourself.

Have you ever

[Sir_Real]

Have you ever worked with a mixed platform network? Done a network backup on such a network? (I only mention this because I'm a big fan of backups. Well, I am now that I've found that the hassle has a payoff. A large, large payoff.)

Andrew

Outline of an interview...

[sclatter]

I've done more sysad interviews than I can count, so I've come up with a bit of a "method".

The first question I always ask is "Can you describe a project you've worked on that you are really proud of, what you did, and why?" This gives them a chance to brag a little, and puts them at ease. I really try to draw them out on this question, to get them to go into detail.

The second question I ask is "Now tell me about a project you worked on that you would do differently if you could do it again, and why?" Anyone who has been a sysad for more than five minutes has made mistakes. You want someone who has learned from them, and can explain what they have learned.

Then I move into the "clue check" portion of the interview. Generally questions one and two have given me an idea of how technical this person is so I can ask appropriately difficult questions. As others have mentioned, I find it's better to ask "how to" questions that are answerable in the context of any flavor unix. Asking about flags to certain commands is just useless.

One question I nearly always ask is "What is the difference between a symbolic link and a hard link?" or the corollary "Why can't you hard link across filesystems?" It's *amazing* how many people who claim to be sysads can't answer that.

I then move into troubleshooting, which is really the meat of the interview. I paint scenarios and ask the interviewee what steps they'd take to diagnose and resolve the problem. "The performance on the mail server is unacceptably slow. What do you do?" "The load on the ftp server is going up to 3000 and then the machine hangs. What could be wrong?" I try to get the person to tell me what commands they'd run, what output they'd expect if it turned out to be X problem versus Y problem. I explain that I'm not looking for a specific answer, I just want to see what their mental process is.

I end with "fit" questions. What hours they like to work, if they prefer to work alone or in groups, etc. This lets the person relax again and allows you to end on a good note. The last thing I do is ask if the person has any questions for me.

This system has worked very reliably for me. I've yet to recommend someone that turned out to be a dud. :-)

Not "what it does", ask "how to do"

[Syberghost]

Don't ask "what is tcpdump?".

Instead, ask "what would you use to view the contents of TCP packets on the network?"

We start with the basics "what would you use to list the contents of a directory?" and work up from there, to gauge the level of knowledge.

Also, technical folks conduct that part of the interview over the phone, and the person doesn't get a face-to-face with a manager about non-technical issues until AFTER we've made our recommendations.

Thank you!

[20000hitpoints]

THANK YOU.

The same, incidentally, goes for hiring programmers. Requiring interviewees to answer questions like "what are the arguments of the 'exportObject' method in the java.rmi.UnicastRemoteObject class?" will result in you hiring programmers who have maybe done a lot of programming, or maybe a lot of memorizing, but who have not (necessarily) done a whole lot of thinking, learning, or architecting systems. Additional minor details that questions like this don't test for are understanding computers, computer science, or algorithms. Remember, we're supposed to be ENGINEERS -- not typists!

just got done with this.

[SuperQ]

I just got done with a couple weeks of interviewing for a new admin here. Here's a bunch of stuff from my notes:

Describe your previous experience with:

Unix: (Linux, IRIX, Solaris, Others)
What I was looking for here was a good history of their experience.. I didn't want to know "I know linux".. I was looking for something more descriptive.. Interviewing is like telling a story about yourself. "I've been using linux since 'XX. I started out doing slackware, but have moved to debian to make system maintence on a large scale easier"

Since this was for a unix admin position, I was looking for some stuff in the Other category.. I want to see some variety in their experience.. Do they understand basic unix fundamentals, SYSV, BSD?

Windows:
I had to ask, just because we still have a few of these around.. but it's good to see a cringe on their face when you ask the question. Working is not all technical skills.. I had to drop a guy's resume because I just couldn't stand the guy's attitude.

Networking:
Have you worked with a managed network.. describe some of the networks you have setup/designed/worked on. I want to find out if the person has a grasp of network topology, switching, vlans, routing, trunking, etc.

Describe your support experience:
I wanted to see what kind of variety of users you've had to work with.. because when it all comes down to it.. they're the most important part of the job. I want to hear storys about user groups.. have you had to support developers? windows desktop users? students? home users?

Audio/Video:
Somtimes the admin has to wear the presentation hat.. make the VCR work. it's just one of those things.

Are you familiar with Linux Package Development.
We have to maintain our own distribution, and customizations.. it's good to have a fundamental understanding of atleast RPM, or dpkg.

What scripting languages are you familiar with:
I want to see a good base of atleast some bash, tcsh, or other shell stuff. Quantify your experiene.. "I created a easy set of scripts to do user maintence of a NIS server"

The next part I had to do for comparative uses. buzzord bingo.

Name the acronym, and it's use.
NFS
NIS
DNS
DDP
TCP/IP
TeX
PHP
SQL

stuff like that.. I'm trying to get out key phrases to quantify the persons experince, and from their reaction, I can easily guage their knowlage.

example.. lots of people have worked with NFS, but the people that only just mount filesystems, and pretend it's all seamless are missing out on some fundamental understanding.. the guy's that mention NFSv2 NFSv3, and having to beat a solaris server over the head so it can talk to linux are the guys that are really going to know what their doing. it's all about the details.

Describe and list the daemons used for:

SMTP:

HTTP:

FTP:

I'm trying to find out what kind of variety a person has.. and this section will also pull out religious tendancies.. like sendmail bigots, qmail bigots, etc etc.. When working in either a closed source unix, or an open source unix.. it's good to know where you're code came from.. and if the guy doesn't know the difference between wu-ftpd and proftpd.. you might run into trouble.

the last section.. (of the technical side of the interview) is to present a situaion, and guage their knowlage of business processes. here's what I asked.

The department needs to purchase 12 identical linux workstations, for general use on the network.. what factors do you consier in how to spend the money.

A lot of the people I interviewed went strait into technical considerations.. some went into "who's it for".. and only one that I remember asked "what are the existing vendor relations like" there is a lot to consider when you're expanding a network infrastructure.. and diving into buying parts for you're favorite platform is the _worst_ idea.

the other half of interviewing is just trying to pull the guy's work habbits out of him.. being able to pull the bullshiting from the real attitude can be tough.. but the right set of standard interview questions will get you there. just be honest.. if you hate getting up in the morning, but will make the extra effort to be at work on time.. say it. let your overall qualifications stand for themselvs.

Re:SAGE have an excellent booklet on this

[Anonymous Coward]

you idiot... thats just a preview of the book.. to get it you have to be a member... didn't you take have a minute to read that...

go jump off of a cliff and save the world some clean air

SAGE have an excellent booklet on this

[ader]

Don't sort through 300 random Slashdot trolls. Join the System Administrators Guild and get their booklet on Hiring System Administrators. That should answer all your questions in one hit.

Ade_
/