Tough Tests Flunk Good Programming Job Candidates

snydeq writes "Fatal Exception's Neil McAllister discusses the use of quizzes and brain-teasers in evaluating potential software development hires, a practice that seems to be on the rise. 'The company best known for this is Google. Past applicants tell tales of a head-spinning battery of coding problems, riddles, and brain teasers, many of which seem only tangential to the task of software development. Other large companies have similar practices — Facebook and Microsoft being two examples,' McAllister writes. 'You'll need to assess an applicant's skill in one way or another, but it's also possible to take the whole interview-testing concept too far. Here are a few thoughts to keep in mind when crafting your test questions, to avoid slamming the door on candidates unnecessarily.'"

Comments

the way to go

[lecoupdejarnac]

Radical idea: have them write code for a few hours to solve a given problem - then see how their solution looks. This goes a long way towards judging their fit for the job. You can even give them a couple of data structures and algorithms references - on the job we use references all the time, and being able to implement something from a reference and apply it to a problem is a real skill.

Re:the way to go

[anomaly256]

This would be preferable to what one company I applied for a job with did recently. Gave me a fairly straight forward maths problem involving modulus, gave me about *5* seconds to solve it using real code and not just pseudocode. Sure, that was fine. Then they added the caveat 'What if % is an expensive operation? how would you work around it?'. Turns out it was a trick question. They were expecting you to statically store the result explicitly instead of finding different maths that achieved the same result dynamically but more efficiently. less than 2 seconds later the interviewer interjected with the answer before I had a chance to even say or do anything right *or* wrong.

I don't see how *this* particular kind of quizzing *can* weed good candidates from bad, it's stacked against everyone equally. It's hostile. I'm pretty sure they didn't ever find the 'right' candidate.

I'm all for puzzles and quizzes to test someone's experience and ability and problem solving skills during job applications, but they MUST a) be unambiguous otherwise you're just being a jerk, and b) must be given a reasonable amount of time to actually perform them otherwise, again, you're just being a jerk.

Re:the way to go

[anomaly256]

In hindsight though, I'm really glad I didn't get that position. I'd hate to work for someone so near-sighted and irrational. :)

Re:the way to go

[acak]

Too often the interviewer's questions are loaded in two different ways - there is no one right answer and not understanding this or dismissing anything outside the narrow set of answers he/she is looking for shows a lack of maturity on the interviewers side of the table.

Re:the way to go

[CannonballHead]

I interviewed at a company, some sort of advertising thing. Relatively small - I was fresh out of college and talked to some heads of something-or-other. Anyways, there were two of us that happened to interview at the same time; they had two test stations setup. There were a few multiple choice sorts of questions, and then an actual programming test dealing with extracting ads out of an XML page, as I recall, in PHP. Well, I'd never dealt with XML in PHP at all, so it was totally new to me. I found various things online (XML parser, etc). After a couple hours and making some progress, but not completing it - and having the other guy already finish roughly an hour or so earlier and whatever - I said I wasn't going to be able to complete it very quickly and asked if I should leave or not.

The head hiring people asked if they could talk to me, so I said sure. Asked if I thought I could complete it, given time - I said sure, it wasn't that difficult, it's just that I had never worked with XML parsing before and was having to learn it as I went. Then they pretty much startled me. They said, basically, the following: the other guy interviewing had all the skills, he was able to churn out the problem, and his test scores were good (as were mine, incidentally). But they didn't like his attitude (somewhat arrogant, loudly proclaiming "oh yeah, it was easy!" and that sort of thing). They said that even though I couldn't complete it within an hour or two, they liked my attitude, they liked that I stayed at it for about two hours (without being asked really), they liked that I was honest to say I would not be able to complete it by the end of the business day (my interview was at like 2:30 and it was now 4:30). They basically said they'd offer me a Jr. Dev position based on the work ethic, honesty, and generally good attitude/personality (along with knowledge of general programming stuff based on test scores, resume, etc).

I didn't end up accepting as I got a much better offer elsewhere - also, primarily, based on work ethic, personality, general knowledge, etc.... it was a "we'll train you to do what we want you to do, because we can see you're versatile enough and willing to learn" - but the experience was pretty eye-opening.

Re:the way to go

[beelsebob]

Do they? Why? You're going to be working in a development environment. What good is a guy who can solve trivial problems on a white board when you need a guy who can solve complex problems in a dev environment with a bunch of reference material.

Re:the way to go

[mhotchin]

The saddest thing is - the 'no right answer" questions are (in my mind) one of the *best* ways of evaluating a prospective hire if the interviewer then follows up with "Why did you do it that way?".

Rather than finding out if the hire can find the answer, I would want some insight into *how* the answer was achieved. Was it something they memorized? Did they evaluate trade-offs? Did the even *see* the trade-offs? Can they evaluate their own answer if (as happens in the real world!) new constraints are placed on the problem?

An 'inefficient' answer is *just fine* if the code wouldn't be used much, and better if it's more maintainable, for example.

Re:the way to go

[Ethanol-fueled]

On a side note i'm not that great with people skills so while i write the tests i no longer do the interviews - last too applicants ended up crying during the tests.

Huh, you might work for my company. An entire software department in a company I may know about has jumped ship wholesale, with one person literally getting up and walking out with no notice, and chatter of the remainder leaving, because its run by an insecure touretty-aspie asshole.

The funny thing is, he (you?) asks all new hires how they can "think outside the box" and make the department better. When people actually bring up useful suggestions, he gets all butthurt and snappy and puts them on his shit-list for making him feel stupid, harassing them at every opportunity. He's certainly one of you, raised in an affluent household and given everything he wanted while mommy made excuses for his rotten behavior. Working for the company is a double-edged sword - it is impossible to be fired for anything short of murder. It's hell when you get stuck with the wrong boss.

But Ethanol, isn't harassment illegal?

In corporate Amerika, harassment is not illegal unless the harassment falls under a protected category like race or gender. That means your boss yelling at you daily for inadvertently making him look stupid is not really harassment.

But Ethanol, why doesn't anybody complain?

Because the economy sucks and some people have families to feed. The ones who don't would not dare jeopardize their good references within the company, because nobody likes a complainer. You know those 3-page exit surveys you get? HR laughs at them and tosses them in the trash anyway.

They forgot what tests are for

[eparker05]

Tests can be a good measurement of quality when the test is material that can be studied for. In school you have a test at the end of a class. For certifications, tests are meant to measure knowledge gained during training. In graduate school, qualifying exams are done to second year students who have time to prepare and hone their skills.

Testing somebody from a cold start, on subjects they have no practical way to prepare for seems like a good way to hire a trivia expert, but the productivity of an employee should be evaluated by his resume and portfolio.

Re:They forgot what tests are for

[frosty_tsm]

Or their objective isn't specifically verifying the individual in front of them.

Picture this: a hiring manager at some medium company reads an article about the brain teasers asked by Microsoft and Google interviewers. He wants his application process to seem more like theirs so they can say they are doing MS / Google style interviews (which sounds good to other managers and executives and theoretically impresses applicants).

Re:They forgot what tests are for

[Tawnos]

Which is bogus, because at least here at Microsoft, those brain teaser questions haven't been used in many years, after it was determined they were ineffective and dumb.

Mix it up

[Tablizer]

Teams with diverse thinkers are often the most effective. The one who is not good at math puzzles may instead be good at understanding the customer's needs or the intuitiveness of user interface designs in the eyes of non-techies, and vice verse. They each can focus on their specialty, or at least help each other out in their weak spots.

The industry has been trashed by offshoring.

[Anonymous Coward]

The offshoring of software development over the past 15 years hasn't just trashed the quality of the software that many American businesses use, it has also trashed the ability of software developers to become managers.

The best software development managers were formerly software developers themselves. They know that experience is what counts. They know that bullshit HR tests don't work. But these kind of managers are now retiring or getting promoted to executive positions outside of software development. There's nothing but a huge void following them, since there have been very, very few software developers in America over the past 15 years.

The people we have following them often have no software development experience. Many of them are MBAs who don't even know of any programming languages beyond JavaScript, and they only know of JavaScript because they read about it once in some article that was hyping it. The worst are the "professional project managers" who don't even have any relevant college-level training in any useful field (yes, that's right, sociology majors don't know how to be software development managers).

We don't find good managers in the places where the software development was offshored to, either. Skilled management was never a factor there to begin with, and thus the void has always been present over there.

Offshoring software development has been one of the biggest economic mistakes that any civilized nation has ever made.

Too many candidates to choose from

[acidradio]

It seems like every job posting now has around 50-100 people who apply. To weed out this many people en masse they will make you do just about anything - tests that have little application to the job that you are applying for, bark like a dog, sing the interviewer's favorite Barbra Streisand song, paint a painting of a nice wilderness scene, tune the carburetor on the interviewer's old Triumph motorcycle... Many of the people are well-qualified and even over-qualified! To weed them out on that alone would go nowhere.

If I had to tell you how many times I've been asked something stupid and cliche like "Tell me about a time when you experienced change" or "Tell me about a time when you faced challenge" I might go postal. It's almost like HR people invent these questions to pad their interviews because they don't really understand what or who they are interviewing for. I long for the days when a hiring manager or, god forbid, the company owner/proprietor calls and asks you "So, tell me what you are about and tell me why you think I should hire you."

They can treat applicants like total bastards and get away with it. With this kind of market what is really to stop them?

Less False Positives

[Tsarnon]

Sure this technique has tons of false negatives, but I think it has fewer false positives than many other interview techniques. False positives are a much bigger problem then false negatives when hiring.

No thanks asshole

[Fujisawa Sensei]

Sounds like you're trying to weed the good artists. I have hired graphic artists as well. And it was a done deal pretty much before I even talked to him. There was one thing I looked at and that was portfolio, the interview was just a matter of finding out if they were an idiot or not. It doesn't matter what application a good artist uses to produce the artwork, as long as they produce good artwork. You can train on a piece of technology, as long as the person isn't stupid. But if you think you can actually train a good graphic artist, you're the fucking stupid one. Becoming a good graphic artist takes years of training and practice.

But I can see why you no longer do interviews, you actually fucking suck at them.

For instance if I say I know C# and you want me to bang out some code in Python, we're done. Because whether or not you realize it, I'm also interviewing you. And I have stopped the interview process in the middle because the interviewers didn't have their shit together. Seriously if you're going to fuck with me during the interview process you probably don't you have your shit together as much as you think you do. So things are really going to be fucked up once I get started, and I have start looking again. No thanks; find somebody else more gullible.

Re:Works well if done right

[swillden]

I've interviewed lots of people using puzzles of this sort, and I find they work really really well for picking out the better programmers. You need to understand how to do it correctly, though. You're not looking for whether they "get the right answer". You're looking to see how they approach the problem and what sorts of solutions they try (even if they end up not working).

Mod parent up.

I work for Google, and interview engineering candidates at Google (which doesn't make me special -- all Google engineers are expected to participate in interviewing), and the above describes what these puzzle questions are really intended to do.

Evaluating candidates is a really, really hard thing to do. Honestly, it's darned near impossible to do accurately unless you can bring someone in for a few weeks, put them to work on real problems and see how they really perform. Even then you're just scratching the surface. Trying to get a realistic read on someone's capabilities in just a single day? Yeah, right. But hiring decisions must be made, and Google gets hundreds of thousands of applicants per year.

So, the question becomes not how to truly, accurately evaluate each candidate, but how to fill the positions and ensure that no egregiously bad hires are made. It's sad to reject someone who is really good... but it causes tremendous grief to hire someone who just can't do the job, because firing people is really painful for everyone involved, and quite expensive to the employer.

With that in mind, here are my perspectives on the article's points, based not only on what Google does, but also on the dozens (hundreds?) of job interviews I did at previous employers.

1. Recognize that tests are artificial scenarios

This is inarguably true. The interview is a completely artificial environment which bears no relationship to the actual job. But there's no way to avoid that, so the question become what is the best way to extract as much useful information about a candidate's abilities and characteristics as possible in the brief time allowed. And the time has to be very brief, because getting any kind of an accurate reading depends on getting multiple points of view, and fitting four or five interviews into a day without killing the candidate means that each interviewer really only has about 45-50 minutes.

The article comments that the problems are hypothetical, without context, etc., but that's because there's no way to present real-world problems in the timeframe given. And, actually, underspecifying problems is deliberate, because it's a good way to see how the candidate reacts to inadequate requirements specifications, which is perhaps the one part of the interview which accurately models the real world at every place I've worked in 20+ years.

So, the goal is to construct an artificial environment which tests the things that can be realistically tested. Google is perhaps a little different than most companies in that Google doesn't really care what the candidate knows. The infrastructure is so different from everywhere else that it's really not important; there will be so much to learn anyway. Well, basic CS knowledge is crucial, but specific tools and techniques? Nah. What really matters is how quickly the candidate can understand problems, how effectively they can reason their way through the solution space, how well they can communicate their solutions -- verbally and in code -- and how well they can analyze their solutions and explore alternatives. It's all about problem solving and the only way to understand someone's problem solving abilities is to watch them solve a problem they haven't seen before.

The "they haven't seen before" is crucial... and it's also pretty easy to verify. Here's a word of advice: If you go into an interview and are presented with a problem you already know the solution to, tell the interviewer up front that you've seen it before. Odds are the interviewer will figure that