Tough Tests Flunk Good Programming Job Candidates

snydeq writes "Fatal Exception's Neil McAllister discusses the use of quizzes and brain-teasers in evaluating potential software development hires, a practice that seems to be on the rise. 'The company best known for this is Google. Past applicants tell tales of a head-spinning battery of coding problems, riddles, and brain teasers, many of which seem only tangential to the task of software development. Other large companies have similar practices — Facebook and Microsoft being two examples,' McAllister writes. 'You'll need to assess an applicant's skill in one way or another, but it's also possible to take the whole interview-testing concept too far. Here are a few thoughts to keep in mind when crafting your test questions, to avoid slamming the door on candidates unnecessarily.'"

Comments

the way to go

[lecoupdejarnac]

Radical idea: have them write code for a few hours to solve a given problem - then see how their solution looks. This goes a long way towards judging their fit for the job. You can even give them a couple of data structures and algorithms references - on the job we use references all the time, and being able to implement something from a reference and apply it to a problem is a real skill.

Re:the way to go

[Amouth]

When i do it i make sure to use an environment that has the same capabilities but they are not familiar with.

Example for a graphic artist when they list Photoshop i give them gimp (unless they have both then i give them painter). or give them OpenOffice instead of MS.. as for programming normally i will have them debug something that is a language they didn't put on their application, making sure the bug is an obvious one (not a syntax problem).

i look for how they solve the problem - some give up at the start, others try and figure it out, others will ask questions. when they are working and/or asking questions you can tell if they know what they need to do and are just confused by the environment or if they are completely lost. it is also a good way of testing their critical thinking skills.

i'd rather hire and train someone who will work out side of their comfort zone, has critical thinking skills, and has the ability to learn. then to hire and spin someone up who might be an good at something but that is all they can or will be able to do.

On a side note i'm not that great with people skills so while i write the tests i no longer do the interviews - last too applicants ended up crying during the tests.

Re:the way to go

[CannonballHead]

I interviewed at a company, some sort of advertising thing. Relatively small - I was fresh out of college and talked to some heads of something-or-other. Anyways, there were two of us that happened to interview at the same time; they had two test stations setup. There were a few multiple choice sorts of questions, and then an actual programming test dealing with extracting ads out of an XML page, as I recall, in PHP. Well, I'd never dealt with XML in PHP at all, so it was totally new to me. I found various things online (XML parser, etc). After a couple hours and making some progress, but not completing it - and having the other guy already finish roughly an hour or so earlier and whatever - I said I wasn't going to be able to complete it very quickly and asked if I should leave or not.

The head hiring people asked if they could talk to me, so I said sure. Asked if I thought I could complete it, given time - I said sure, it wasn't that difficult, it's just that I had never worked with XML parsing before and was having to learn it as I went. Then they pretty much startled me. They said, basically, the following: the other guy interviewing had all the skills, he was able to churn out the problem, and his test scores were good (as were mine, incidentally). But they didn't like his attitude (somewhat arrogant, loudly proclaiming "oh yeah, it was easy!" and that sort of thing). They said that even though I couldn't complete it within an hour or two, they liked my attitude, they liked that I stayed at it for about two hours (without being asked really), they liked that I was honest to say I would not be able to complete it by the end of the business day (my interview was at like 2:30 and it was now 4:30). They basically said they'd offer me a Jr. Dev position based on the work ethic, honesty, and generally good attitude/personality (along with knowledge of general programming stuff based on test scores, resume, etc).

I didn't end up accepting as I got a much better offer elsewhere - also, primarily, based on work ethic, personality, general knowledge, etc.... it was a "we'll train you to do what we want you to do, because we can see you're versatile enough and willing to learn" - but the experience was pretty eye-opening.

Re:the way to go

[timeOday]

Well, they want to see your raw abilities, not your development environments abilities.

After migrating from C++ in emacs to Java in Netbeans only recently (and maybe a bit grudgingly), I have been spending some time looking over the shoulders of IDE whizzes and found something to admire. One guy I watched basically writes software backwards, starting with a small kernel and "refactoring" it up to a program by referring to classes, methods, and parameters that don't exist, and having the IDE create the skeletons with a few hotkeys. (Not to mention more obvious things like auto-complete and creating imports). I was surprised how fundamentally the tools affect the workflow of somebody who first learned to program using them.

Less False Positives

[Tsarnon]

Sure this technique has tons of false negatives, but I think it has fewer false positives than many other interview techniques. False positives are a much bigger problem then false negatives when hiring.

Re:Works well if done right

[swillden]

I've interviewed lots of people using puzzles of this sort, and I find they work really really well for picking out the better programmers. You need to understand how to do it correctly, though. You're not looking for whether they "get the right answer". You're looking to see how they approach the problem and what sorts of solutions they try (even if they end up not working).

Mod parent up.

I work for Google, and interview engineering candidates at Google (which doesn't make me special -- all Google engineers are expected to participate in interviewing), and the above describes what these puzzle questions are really intended to do.

Evaluating candidates is a really, really hard thing to do. Honestly, it's darned near impossible to do accurately unless you can bring someone in for a few weeks, put them to work on real problems and see how they really perform. Even then you're just scratching the surface. Trying to get a realistic read on someone's capabilities in just a single day? Yeah, right. But hiring decisions must be made, and Google gets hundreds of thousands of applicants per year.

So, the question becomes not how to truly, accurately evaluate each candidate, but how to fill the positions and ensure that no egregiously bad hires are made. It's sad to reject someone who is really good... but it causes tremendous grief to hire someone who just can't do the job, because firing people is really painful for everyone involved, and quite expensive to the employer.

With that in mind, here are my perspectives on the article's points, based not only on what Google does, but also on the dozens (hundreds?) of job interviews I did at previous employers.

1. Recognize that tests are artificial scenarios

This is inarguably true. The interview is a completely artificial environment which bears no relationship to the actual job. But there's no way to avoid that, so the question become what is the best way to extract as much useful information about a candidate's abilities and characteristics as possible in the brief time allowed. And the time has to be very brief, because getting any kind of an accurate reading depends on getting multiple points of view, and fitting four or five interviews into a day without killing the candidate means that each interviewer really only has about 45-50 minutes.

The article comments that the problems are hypothetical, without context, etc., but that's because there's no way to present real-world problems in the timeframe given. And, actually, underspecifying problems is deliberate, because it's a good way to see how the candidate reacts to inadequate requirements specifications, which is perhaps the one part of the interview which accurately models the real world at every place I've worked in 20+ years.

So, the goal is to construct an artificial environment which tests the things that can be realistically tested. Google is perhaps a little different than most companies in that Google doesn't really care what the candidate knows. The infrastructure is so different from everywhere else that it's really not important; there will be so much to learn anyway. Well, basic CS knowledge is crucial, but specific tools and techniques? Nah. What really matters is how quickly the candidate can understand problems, how effectively they can reason their way through the solution space, how well they can communicate their solutions -- verbally and in code -- and how well they can analyze their solutions and explore alternatives. It's all about problem solving and the only way to understand someone's problem solving abilities is to watch them solve a problem they haven't seen before.

The "they haven't seen before" is crucial... and it's also pretty easy to verify. Here's a word of advice: If you go into an interview and are presented with a problem you already know the solution to, tell the interviewer up front that you've seen it before. Odds are the interviewer will figure that