Are Brain Teasers Good Hiring Criteria?

theodp writes "Your brain teaser prowess may win you a job at Google, but the folks at 37signals don't hire programmers based on puzzles, API quizzes, math riddles, or other parlor tricks. 'The only reliable gauge I've found for future programmer success,' explains 37signals' David Heinemeier Hansson, 'is looking at real code they've written, talking through bigger picture issues, and, if all that is swell, trying them out for size.'" Those of you who have hired employees: have you seen correlation between interview puzzle success and job competency? How should an interviewee best handle these questions?

Comments

Well, they're a good indicator of intelligence

[elrous0]

If someone is giving you one, they're probably not very intelligent.

Re:Well, they're a good indicator of intelligence

[ILongForDarkness]

Perhaps. The article says "looking at real code" is better. Again perhaps. For example the problem there is: did they really write the code, if so how long did it take? Did someone else suggest fixes etc? You don't know. I mean 300 lines of beautiful C is all fine and dandy but if it took you 3 months to write it and half of it is cut and pasted from the web how good is it really?

What brain teasers hopefully do is take a problem close to the types of things you see in the job. Even though it is all programming different companies either due to industry or existing infrastructure/policies tend to have different types of coding "puzzles" that come up again and again. Hopefully this test problem is one you haven't seen before and they get to see how you approach something you don't already know how to solve, how close to a good design do you get on the first interation, if they point out a problem how you go about fixing it etc. All real world important stuff to know about someone.

Re:Well, they're a good indicator of intelligence

[leonbloy]

In my experience: They're a moderately good indicator of a special kind of intelligence; which is not a very useful indicator in the typical hiring process.

Puzzles help to distinguish programmers from lawyers. Not to discriminate good programmers from bad programmers.

Re:Well, they're a good indicator of intelligence

[jimbolauski]

I agree. But I would prefer a puzzle to questions like "where do you see yourself in 5 years" and "what are your goals". I want to answer "My goal is to get hired. Why else would I answer such stupid questions?"

Those questions are more pertinent to MBA's where your ability to pile on BS in a believable way is an important skill.

Re:Well, they're a good indicator of intelligence

[w_dragon]

The 5 years question is really just asking if you intend to move up the management or technical ladder. If I'm looking for someone who may eventually become a software architect and a candidate tells me they want to be a VP in 5 years I may think twice before hiring them.

Re:Well, they're a good indicator of intelligence

[Whorhay]

Is using code snippets from the internet really an issue? Is there a good reason to re-invent the wheel, pulley, block and tackle, lever, wrench, lightbulb and whatnot every time?

Time constraints could certainly be a problem if it takes longer for someone to lookup a solution and implement it than if they just come up with the solution themselves and implement it. And there are legal issues with copying large chunks of code, but I wouldn't think that banning it wholesale is a very productive way to go.

Re:Well, they're a good indicator of intelligence

[asliarun]

If someone is giving you one, they're probably not very intelligent.

I completely disagree, or at least, your statement is so broad it is untrue.

Brain teasers are just like any other interviewing tool - what matters is how you use the tool.
As an interviewer, if you use brain teasers to determine *how* the candidate is attempting to solve the problem, you are probably doing it right.
If you are using the brain teaser to tick a box in your checklist based on the answer, you're probably doing it wrong.

The really neat thing about brain teasers or puzzles or the bizarre questions you sometimes encounter like "How many pigeons are there is Manhattan" is that they are a very good way to judge someone's unstructured problem solving ability. How someone approaches this kind of a problem is a good proxy for their ability to debug hard technical issues or their problem solving ability in general.

Making a statement like "hire a programmer based on their programming ability" is also an obvious statement to make, apart from being a bit grandiose (look at us , we are cool because we are contrarians and we swim against the tide). The reason why many interviewers resort to other techniques is two fold - one, lack of time or other constraints that prevent the interviewer from directly testing a programmer's programming ability, and secondly, judge the non-programming aspects of the candidate like how they react to an ill-defined problem or a fuzzy situation, how well they will get along with others, how much of a self-starter they are etc.

Or, if I put it another way, if you are not hiring a programmer on the, to quote, "code they have written", what are you doing, hiring candidates on their baking skills? I get what 37signals is saying and all this got messed up to begin with when HR took over the interviewing process from programmers (especially in large companies). However, the other statements that are flying around about how *any* non-programming related question is stupid is also frankly, over the top.

Re:Well, they're a good indicator of intelligence

[August_zero]

Puzzles with subjective answers can show a number of traits in a potential employee. People with overly rigid communication styles or those that fail in the ability to see through analogies are going to be a pain in your ass because they often can't get along well with or understand others in a work environment.

Honestly how many autistic spectrum IT people do you really need?

Re:Well, they're a good indicator of intelligence

[Anonymous Coward]

I ask those questions, and it is exactly to weed out people like you (and in fact the answers have actually made me say no to candidates in the past).

I have a passion for what I do and in my experience, the best people for jobs where I work are ones who share that passion. We already have people who treat it as a 9-5 job and pretty much all of them are "ok". The ones who stand out are ones who think of it as more than just a job to pay their bills. If you intend to work just to get by, that is all you will ever do at the job.

Re:Well, they're a good indicator of intelligence

[Anonymous Brave Guy]

The 5 years question is really just asking if you intend to move up the management or technical ladder.

Then why not just ask that directly? It might lead to a mutually interesting conversation, and at the very least you're either both going to be more confident in your fit or both going to know immediately that it's not going to work out.

These are my personal interview red flags for a software developer job, starting with the most likely to result in me throwing the interview/leaving early/declining any offer:

1. Hiring is based on brain teasers, $-based certifications, or other similarly irrelevant criteria.
2. Wants to know my current salary. (Bonus point: Doesn't give any indication at the same time of the likely compensation if I'm offered the job. Extra bonus point: Makes clear that the salary range given in the job ad was wildly optimistic.)
3. Won't show me an example of their production code, real documentation, etc. when given a reasonable opportunity to do so.

In each case, someone is skirting what matters, instead of finding out as fast as possible whether we are really a good fit and a mutually satisfactory hire might result.

Re:Well, they're a good indicator of intelligence

[delinear]

It helps if you have 3 months salary in reserve for emergencies like you should so you don't end up entering a bad situation out of desperation.

This is the best advice out there. If you enter any job interview from a position of needing the job rather than wanting it, you risk not being happy in your role. And don't be afraid to turn the question back around - if someone is asking you where you see yourself in five years, why are they asking that? Is it because they want someone who will grow with the company, do they have a specific path mapped out that they'd want you to follow, are they hoping for someone who will stick in the one position forever, etc - ask them, if you got the job, where they'd see your role in five years, because it depends more on them than you at the end of the day (it doesn't matter if you see yourself as CEO if they only see you in a junior role).

Re:Well, they're a good indicator of intelligence

[Anonymous Coward]

Same for 'why do you want to work here." "Because I need a paycheck?" Seriously? Look, we all laugh at Wally in the Dilbert strips, but I'm sure as hell not going to knowingly hire one.

You can delude yourself all you want, but ultimately people get jobs because they need the money. Most of us would love just to lounge around with no real responsibilities in life if we could, but we need to eat, clothe, and house ourselves. So hence, the job.

That isn't to say that some jobs are more fun or better/worse than others, but don't pretend that economic incentive isn't the primary reason why we look for jobs.

Re:Well, they're a good indicator of intelligence

[SecurityGuy]

The really neat thing about brain teasers or puzzles or the bizarre questions you sometimes encounter like "How many pigeons are there is Manhattan" is that they are a very good way to judge someone's unstructured problem solving ability. How someone approaches this kind of a problem is a good proxy for their ability to debug hard technical issues or their problem solving ability in general.

This is really not true. My response to questions like that improved dramatically when I read an article that explained questions way out of left field like that are intended to test your problem solving ability, so do your best to estimate an answer and explain your thought process. Reading that article didn't make me better at debugging hard technical issues, but made me dramatically better at handling off-the-wall interview questions nimbly. You're not measuring what you think you're measuring.

Re:Well, they're a good indicator of intelligence

[Splab]

My problem with "own" code is I have none. Sure I have the stuff I did back at the University, but thats 5 years ago - I never program outside the job, which means anything I've done belongs to someone else; in fact, I don't even own a computer these days.

(Also note that if I did in fact program outside working hours, it would most likely still belong to the company, so again, I'd have nothing to show).

Re:Well, they're a good indicator of intelligence

[tompaulco]

Contrary to what other management type people on here have said in their posts, your answer of "Because I need a paycheck" is the right one. No, that is not the answer that they want to hear. They want to hear how excited you are to work at their company and how you want to grow and make a career out of it, but the real truth is that they don't know what they want. They want someone who will say "Because I need a paycheck". This is because five years from now, they will still have you sitting there writing code for them at approximately the same salary. That IS why they are hiring you after all. They don't want to have you move up the ladder and then have to go hire another person to do your job. So, give them the answer they want to hear, if that helps get you the job.

Re:Well, they're a good indicator of intelligence

[shentino]

The fact that you're willing to pay the guy you're hiring means "to get a paycheck" is implied.

The question is a test of your ability to think ahead.

And if it's an annoying question it's also a handy test of your ability to remain diplomatic and RESPECTFUL.

If I ask an irritating question and they blow their top, I've just saved myself from hiring a ticking time bomb ready to blow.

Re:Well, they're a good indicator of intelligence

[IAmGarethAdams]

Sure, but the question isn't "Why do you want to *work*?", it's "Why do you want to work *here*?"

The answer isn't designed to find out whether you need money at the moment, but why that particular position is the one you're after. Now in most cases that just changes your answer from "Because I need a paycheck" to "Because you guys were the first people who offered me an interview", and that's fine.

But if you're an interviewer and you have to choose between two people, one who answers "Because you're the only people who'll talk to me" and one who answers "Because your output seems interesting and your public image seems like somewhere I'd fit in" - who do you think would be more likely to be productive?

Re:Well, they're a good indicator of intelligence

[marcosdumay]

Plans != drive

But HR drones won't ever think about that. Sometimes smart people ask that question, but let's be honest here, most of the times it is asked just because it is on the "script", and the interviewer can't think for himself.

Re:Well, they're a good indicator of intelligence

[maple_shaft]

Wow, you really hate whiteboarding don't you? I am so completely the opposite of you that I must respect your opinion because I can't relate to it at all.

Whiteboarding conveys a few things, a high level of spatial intelligence required for diagramming and modeling complex problems visually. It also is an accessory to communication about a complex design or process that a group of colleagues or lay people wish to know more about. If a candidate is whiteboarding a process for me and he silently doodles on the board then that is a problem. You are supposed to talk through the problem primarily and cement your ideas in on the board so that everybody can see a visual summary of your explanation.

The fact that you despise it means that you completely fail to understand why somebody wants an engineer who can whiteboard. It is a sign of an individual who can communicate and discuss problems on multiple facets, while somebody who just wants to demonstrate skill by typing in a text editor tells me that this person doesn't care about communicating or discussing complex ideas, they just want to showcase their skill.

You may be extremely productive writing software or some such engineering activity, but you seem like an extremely low-level task oriented person and that is not what most companies want. We want critical thinkers who engage in higher level design, thought, and communication. They don't want cowboy coders. They don't want a lone wolf.

Re:Well, they're a good indicator of intelligence

[Gorobei]

Wow. I don't care if a candidate says "I need a paycheck," but I'd never hire someone if I didn't feel they had the potential to double their salary within a few years.

I only want to hire people who will grow into bigger roles (and yes, I expect them to hire the person who will do their old job.) The bigger role doesn't need to be management, writing code that is more strategic works just as well.

If I want someone to be in the same role for 5 years, I'd hire a consultant. But, ideally, I'd hire someone to fix the underlying problem that we need a body in a pointless role in the first place.

Re:Well, they're a good indicator of intelligence

[HornWumpus]

If it didn't suck a little they wouldn't have to pay you to do it.

How many job openings are there for 'hot chick pussy eater'?

That said many employers are forced to be assholes because they have employees that are assholes.

For every employer that times your bathroom breaks their are a dozen employees holding their shit on the drive in so they can crap on the clock.

Use the rule of threes. If you've had three employers in a row with the same problem, it's not their problem, it's yours. Works for girlfriends too.

Re:Well, they're a good indicator of intelligence

[s73v3r]

The line between "We want to see how you work so we can gauge your technical abilities" and "We want a bit of cheap and/or free labor" is very, very thin.

Re:Well, they're a good indicator of intelligence

[luis_a_espinal]

I'm trying to work out whether you are being deliberately patronising/accusatory/insulting, or whether it's a function of culture or perhaps not speaking English as a first language.

Well, yes, English is not my first language, but I've been speaking it for 22 years (the equal time I've been away from my culture). Whether that has any effect in how I communicate effectively (or not) that would be an interesting conundrum. Independently of cultural mannerisms and language quirks, I am deliberately and unequivocally stating what I said before: a question such as "where do you see yourself in 5 years" is direct and is clear for any college-educated adult.

Barring an ulterior motive by the interviewer (wanting to torpedo you in favor of another applicant), what other rational and reasonable interpretation in the rational and reasonable general case can there be? You want to remain the same or climb up? And if so, want to climb up technically or managerially? They are interviewing you for a job, and all jobs, specially those involving a college education, they have a very specific, self-evident context. And I've never seen an indication that such a context is beyond the reasonable comprehension of a college-educated person.

Whether you find that patronizing, accusatory or insulting, that's a interpretative choice you have to do you on your own, and which you are entitled to.

In any case, I have never encountered a productive answer to the generic question or a particularly interesting follow-up conversation, either when someone has asked me or when someone on an interview panel I've been on has asked an applicant.

But that's you. If that works for you, that's great. But you have to understand then, that this is entirely subjective from your part. Not finding a productive answer is hardly an objective measure; your are measuring productivity on a personal level (which is inherently subjective.) I also think it is potentially counter productive. And I'm not going to try to convince you otherwise as we are all adults here capable of making our own choices.

For me, quite the opposite: such a question opens up a lot of interesting (and very important) follow-up questions that help you gauge your potential employer: "It is hard to answer that at this stage. What type of opportunities does your company provides?". or "I would like to remain technical" or "Depends. Are there opportunities for move into management" or, simply be honest and see where that gets you: "I'm solely interesting in coding at this moment and for the foreseeable future". Use your communications skills to elaborate an answer according to the situation and according to what you want out of it. Anything else is being passive or unreasonably demanding in an interview.

Be honest or be conniving, depending own your own goals. That on itself is interesting. But you cannot get there without first exploiting out the interesting bits out of anything (there are always interesting bits).That on itself is useful. To find something interesting or informative in such a social, subjective setting, that is completely a function of one's predisposition. Furthermore, it is a matter of rhetorical and communicative skills that should be cultivated (and exercised) with one's education. If you find this statement insulting or patronizing, hey, that's your interpretation.

I guess I (and every other interviewer and applicant involved) must be unthinking, lacking in drive, and unable to interpolate things out.

Not every other. Only those that engage in such behavioral patterns. And not only those behavioral traits that you describe. It could also be that the person in question is simply unaware of the importance of finding value, and the choice of being proactive in extracting value out of things, to steer even the most inane questions according to your own designs, that that choice is subjecti

Re:Well, they're a good indicator of intelligence

[luis_a_espinal]

Plans != drive

But HR drones won't ever think about that.

This expression is so cliche, it has lost any meaning.

Sometimes smart people ask that question, but let's be honest here, most of the times it is asked just because it is on the "script",

And what's wrong with the opening interviews to be on script? How do you pretend interviewers (HR and otherwise) to filter 200+ applicants. You filter by script first. Then you leave the meat for a 2nd round of interviews with the actual people involved. I've never been hired directly by HR. It's either HR followed by tech leads and management, or directly by the later.

The script process is not perfect, and there are a lot of things to complain about HR interview processes. Script interviews aren't among them.

and the interviewer can't think for himself.

Here you are simply building a subjective, generalizing strawman with which to prop your argument.

It's important to understand

[AdrianKemp]

Google isn't giving brain teasers to find good programmers. They're giving brain teasers to find creative technical people who can come up with the next big ideas.

Re:It's important to understand

[jythie]

Actually, I would wager they are giving brain teasers as an amusing way to cut down the number of applicants. I think programmers and hiring managers tend to like the questions because they are fun to give, but they are also a quick way to sort through people who look pretty similar on paper. When you have that kind of application volume, figuring out how to reduce it becomes pretty important for one's sanity, but like any other HR technique, it is designed to reduce the pile, not find the best applicants.

Weeds out weak candidates

[Mannfred]

Every organisation has different needs, but even so "looking at real code they've written, talking through bigger picture issues" takes time, and an initial interview with more basic questions should probably be there to weed out the weakest candidates (unless the people in charge of recruitment interviews have nothing else to do and want to look busy, of course).

Puzzles aren't to test programming skills

[joshamania]

The brainteasers are there to see if someone is smart. Could the employee escape from a paper bag if necessary? I'd say these puzzles are important for finding creative problem solving and would be just as useful if not more useful in a manufacturing/fabrication/maker type of job.

Think of that American drilling team that drilled the hole to free the Chilean miners. That engineer's rig wasn't meant to do what he did with it. Can't aim it? He aimed it with a hack. Hole's plugged? Fixed it with a hack? Don't have a 28" drill head for this rig? Let's hack one together in a week. If that guy with the big brain didn't pick up the phone and say "hey"...those 33 guys would probably have been entombed for half a year if not forever.

Dude did it in one month with a toolbox full of hacks. Fucking brilliant.

Ask them to program something specific

[brucmack]

I guess we combine the two approaches: we send our candidates small coding problems to solve. So we see real code they create and have a standardized way of comparing it to what other candidates have provided.

It works really well at filtering out people we don't want to waste time talking to, and gives us a starting point for the technical interview. It isn't useful for deciding whether or not a candidate should be hired, since there are many other factors that come into play.

Re:Bring puzzles as an applicant

[w_dragon]

HR shouldn't be interviewing for technical positions beyond a basic initial interview to ensure that the candidate actually exists and wants to work for the company.

You're asking the wrong question.

[NReitzel]

Over 30 years of hiring programming staff, what I have found is the converse of your proposition.

People who are not good at solving brain teasers are poor at being good programmers. If they are good at solving brain teasers, that really doesn't say anything about whether or not they will be effective as programmers.

Also, I'd be remiss if I didn't add a few words about "being effective." It's not ability to produce code that counts, it's ability to produce code that can be maintained later by other, less effective programmers. A good solid foundation is absolutely necessary, as is sufficient commentary so that someone who is stone cold on the code can dig into it and fix things, or change parameters. The long term cost of code is not creation of code, it is maintenance of code.

Looking at code is an important hiring criterion, but it is also something that is simply and totally out of the ability of an HR person to achieve. Perhaps the idea of using brain teasers is simply because it is a screening process that can be carried out by HR.

Re:climbing up the technical stool

[Medievalist]

Same here. But honestly you probably don't want to be in the top 1% for income; those people are always the first up against the wall when the revolution comes.

http://amultiverse.com/2011/10/24/eat-the-rich/

Shoot for somewhere in the top 5% and you might not end up wearing a bad sweater, eh?