Slashdot Mirror
Tough Tests Flunk Good Programming Job Candidates
snydeq writes "Fatal Exception's Neil McAllister discusses the use of quizzes and brain-teasers in evaluating potential software development hires, a practice that seems to be on the rise. 'The company best known for this is Google. Past applicants tell tales of a head-spinning battery of coding problems, riddles, and brain teasers, many of which seem only tangential to the task of software development. Other large companies have similar practices — Facebook and Microsoft being two examples,' McAllister writes. 'You'll need to assess an applicant's skill in one way or another, but it's also possible to take the whole interview-testing concept too far. Here are a few thoughts to keep in mind when crafting your test questions, to avoid slamming the door on candidates unnecessarily.'"
Radical idea: have them write code for a few hours to solve a given problem - then see how their solution looks. This goes a long way towards judging their fit for the job. You can even give them a couple of data structures and algorithms references - on the job we use references all the time, and being able to implement something from a reference and apply it to a problem is a real skill.
Tests can be a good measurement of quality when the test is material that can be studied for. In school you have a test at the end of a class. For certifications, tests are meant to measure knowledge gained during training. In graduate school, qualifying exams are done to second year students who have time to prepare and hone their skills.
Testing somebody from a cold start, on subjects they have no practical way to prepare for seems like a good way to hire a trivia expert, but the productivity of an employee should be evaluated by his resume and portfolio.
Whatever happened to tests like drinking the interview panel under the table?
Now that is a skill needed on the job.
I am anarch of all I survey.
Teams with diverse thinkers are often the most effective. The one who is not good at math puzzles may instead be good at understanding the customer's needs or the intuitiveness of user interface designs in the eyes of non-techies, and vice verse. They each can focus on their specialty, or at least help each other out in their weak spots.
Table-ized A.I.
google is not full with EXCEPTIONAL developers, it is filled with EXCEPTIONALLY clever people; you get what you select for. As D&D should have taught you, INT is not equals to WIS !
Jehovah be praised, Oracle was not selected
They did not invent the search engine, but the way how to best evaluate the site's hit count,interest,etc... Everything else is one big farm, very very big farm.
Lucky thing it's not a spelling test.
The offshoring of software development over the past 15 years hasn't just trashed the quality of the software that many American businesses use, it has also trashed the ability of software developers to become managers.
The best software development managers were formerly software developers themselves. They know that experience is what counts. They know that bullshit HR tests don't work. But these kind of managers are now retiring or getting promoted to executive positions outside of software development. There's nothing but a huge void following them, since there have been very, very few software developers in America over the past 15 years.
The people we have following them often have no software development experience. Many of them are MBAs who don't even know of any programming languages beyond JavaScript, and they only know of JavaScript because they read about it once in some article that was hyping it. The worst are the "professional project managers" who don't even have any relevant college-level training in any useful field (yes, that's right, sociology majors don't know how to be software development managers).
We don't find good managers in the places where the software development was offshored to, either. Skilled management was never a factor there to begin with, and thus the void has always been present over there.
Offshoring software development has been one of the biggest economic mistakes that any civilized nation has ever made.
It seems like every job posting now has around 50-100 people who apply. To weed out this many people en masse they will make you do just about anything - tests that have little application to the job that you are applying for, bark like a dog, sing the interviewer's favorite Barbra Streisand song, paint a painting of a nice wilderness scene, tune the carburetor on the interviewer's old Triumph motorcycle... Many of the people are well-qualified and even over-qualified! To weed them out on that alone would go nowhere.
If I had to tell you how many times I've been asked something stupid and cliche like "Tell me about a time when you experienced change" or "Tell me about a time when you faced challenge" I might go postal. It's almost like HR people invent these questions to pad their interviews because they don't really understand what or who they are interviewing for. I long for the days when a hiring manager or, god forbid, the company owner/proprietor calls and asks you "So, tell me what you are about and tell me why you think I should hire you."
They can treat applicants like total bastards and get away with it. With this kind of market what is really to stop them?
(In the end I admittedly had absolutely no idea how how to solve the problems, and didn't even attempt to. I got the job anyway.)
When I interview people - I feel it is my job to "extract" the best out of the candidates, and to find out what "their best" actually is. If I come away from an interview and don't have a strong feeling for a candidates abilities - good and bad - I feel as though I didn't do my job as an interviewer. I've seen too many people "freeze up", or just be shy in interviews. These people maybe were VERY qualified - I feel it is always my job to understand that. My creedo is this: Get the people talking. Get them talking about what they do, and what they love. If you can do this - they'll go into the depths and bowels of their technical knowledge, working style, experience, etc.
Google invited me to interview for a Java programming job. They started the interview by informing me that I would be "the oldest person in the group" (I was 39 at the time). Then, I was invited to code a linked list in C on the white board while they watched. I can do this, I suppose, having done it 20 years ago while getting my computer science degree. And never done it since. I questioned the relevance of the problem pointing out that this was surely not required for programming in Java. It kinda went downhill from there...
Sure this technique has tons of false negatives, but I think it has fewer false positives than many other interview techniques. False positives are a much bigger problem then false negatives when hiring.
Looks like you just flunked the "sample size" test. Don't worry about it, brain teasers aren't for everyone.
Considering that these questions can be asked once, maximum 3 times, before someone posts it to the Internet, now that I've memorized all of the right answers there they go changing the rules. I'll never get that job now.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
partly - perhaps even mostly - due to this new strict 'screening' bullshit that everyone in the silicon valley seems to be doing, now. I've been out of work for more than a year now and the interviews I have been on have been markedly different than they were 20 years ago when I first came to the bay area. lately, the interviews are confrontational. they are assuming you are a liar, incompetant and many other bad things - and its up to you to disprove that. they do not seem to want you, they are there because their bosses told them to interview some new people.
this is quite different from the case where they really are looking to hire and are excited to have found a match (at least on paper) and just want to verify that you are who you say you are. even as little as 5 years ago, the interviews were not so negative. it was more of a verification that you met your paperwork. now, they don't bother to read your resume; or they assume its all lies and you have to start from ground-0 and prove an entire background to them. *over and over*, too; with each new guy that steps in for his 45min slot. show me linked lists; show me trees. show me O(n) stuff.
the problem for me is that I am starting to not care anymore about this level of detail. I'm 50 and have been doing C since I was mid 20's. I have done my time, to be sure; but I just don't really get into having to prove it over and over again, like my resume is all a pack of lies. it gets tiring and its making me question whether I really want to re-enter this field with these kinds of people 'running things'.
I do think that people like me are worth retaining in the software development field. true that I'm getting tired of the lower level details and things that are reference (like precise steps in deleting a linked list node) are of no interest to me when I can quickly get the correct sequence in a few minutes of search. there is too much to keep track of and older things do age out, its true. but older folks do have a lot of other things to offer. its a shame that we get passed over due to how the 'tests' are structured these days. I once was able to jump thru some hoops, but now, I'm just not so motivated to play their bullshit 'test me' games anymore. its a shame since I'm not alone in this and losing experienced guys like me is a real loss to the industry.
--
"It is now safe to switch off your computer."
I've interviewed lots of people using puzzles of this sort, and I find they work really really well for picking out the better programmers. You need to understand how to do it correctly, though. You're not looking for whether they "get the right answer". You're looking to see how they approach the problem and what sorts of solutions they try (even if they end up not working). When you interview a bunch of people this way, you find they split into a few groups, and the differences between groups are really obvious. For example, some people will just have no idea how to even approach the problem. Others will struggle to figure out an O(n^2) solution. And others will instantly take it for granted that of course there's a trivial O(n^2) solution, but you're obviously looking for something better than that. The differences aren't subtle.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
Because the entire point of those tests isn't to see if they get the right answer, it's to see if the candidate can work with the people in your office.
The trouble with brain-teaser puzzles and trick questions it that the entire point is usually to make the interviewer feel that they are smarter than the candidate.
If they make hiring decisions based on that kind of test, this is probably not the case. :-)
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Sounds like you're trying to weed the good artists. I have hired graphic artists as well. And it was a done deal pretty much before I even talked to him. There was one thing I looked at and that was portfolio, the interview was just a matter of finding out if they were an idiot or not. It doesn't matter what application a good artist uses to produce the artwork, as long as they produce good artwork. You can train on a piece of technology, as long as the person isn't stupid. But if you think you can actually train a good graphic artist, you're the fucking stupid one. Becoming a good graphic artist takes years of training and practice.
But I can see why you no longer do interviews, you actually fucking suck at them.
For instance if I say I know C# and you want me to bang out some code in Python, we're done. Because whether or not you realize it, I'm also interviewing you. And I have stopped the interview process in the middle because the interviewers didn't have their shit together. Seriously if you're going to fuck with me during the interview process you probably don't you have your shit together as much as you think you do. So things are really going to be fucked up once I get started, and I have start looking again. No thanks; find somebody else more gullible.
If someone is passing you on the right, you are an asshole for driving in the wrong lane.
Can he remain polite when frustrated? Can he admit that his first approach to a problem was wrong?
Why do you assume this will happen? Is it because you don't give them all the real requirements until after he takes a stab at the answer? That's a common trick that interviewers use to make themselves feel smarter than the candidate. What do you do when someone asks you to give them a real, written requirements list before they start, as they would get if this was a real work situation?
Can he work collaboratively at all, or does he have to solve problems entirely by himself.
If your "test" is simple enough to do on a whiteboard, many good programmers won't need any help after reading the written requirements.
Alex at The Daily WTF wrote about this problem back in 2007: http://thedailywtf.com/Articles/Riddle-Me-An-Interview.aspx
The interview is often a very good indicator of what the job is like. It's just as much of a way for the interviewee to evaluate their prospective employer as it is for the employer evaluating the employee.
Amen.
I once interviewed for a programming position at a very young and small company. Most of the interview was done by the owner, a business/marketing guy with some technical knowledge of the industry they develop applications for but with no real knowledge of software development (other than it takes longer than expected). For the programming test I was handed over to his lead developer. The test was crap but I probably did well on it. When I was back with the owner he asked how the test went. I decided to do my side of the evaluation. I told him I probably did well but that the test wasn't very good. I explained why. The test seemed like a sampling of multiple choice questions from quizzes and tests from a bunch of different CS classes. Given that a CS type degree was required the applicant presumably passed all these classes so nothing new was really learned. At least it was easy to grade, however like many things you get out what you put in. If a test is to be given it should test for something a degree does not necessarily demonstrate, the ability to actually develop code that solves the problem or task at hand. That is what you are going to pay people for. The preceding was not some kind of soliloquy, the owner seemed interested and asked questions of his own and the above gives an overview of what was discussed. The interview ran long due to this conversation. I was offered the position a few days later. The owners lack of software experience was a concern but I got a good vibe from our chat about the test and from his responses to some of my general questions. I took the job. It worked out well, he trusted us and took our opinions quite seriously when he needed to make decisions. Oh, my first task was to write a new programming test.
Today when I interview people and get to the part where I will answer any questions they have, and they say they don't have any, I point out that an interview works both ways. That this is their chance to find out if this company is a good fit for them. Usually there is surprise and a few seconds of confusion but most are able to come up with some questions at that point.
I find that the HR-built questions tend to drive me crazy at times, mainly because some of little to no bearing on the position being interviewed, and others may not *have* an answer.
Being asked how you resolved a conflict with a co-worker when you've never had one (can't say that now, but during an interview 5-10 years ago I could as I had only worked in smaller shops full of some pretty nice people) is frustrating as heck.
But back to the technical questions. I've found that making them is fun. Rather that trying to come up with obtuse technical questions, some basics mixed with real situations the company has faced works well (what would you check in situation X). Questions that try and get you to fill in entries that would be more easily available from a man-page are lame.
In my own experience, the best part of coming up with the questions is getting back answers that I'd never thought of. Sure they don't match the solution/issue given in my own experience, but finding cool new ideas that never crossed my mind is part of what makes tech fun.
While I don't really ask many brain teasers when interviewing people, one key benefit that brain teasers offer is to see how candidates do when facing an unexpected, off-the-wall problem. Do they freeze? Panic? Make stuff up? Give up? Or do they start thinking it through? This is really important when the job entails facing unexpected off-the-wall problems regularly, as it does in my shop (a top-ten computer science department where weird computing is not unusual). A similarly useful technique in interviews is to hand a candidate a stack of paper, each sheet of which has a snippet of code, the output of a command, or the contents of a standard system file (some of them should be obscure, some common), and ask them to simply identify the programming or scripting language, the command, or the system file, if they know what it is. It's a really quick way to see a candidate's breadth of knowledge and experience, and also (for obscure sheets) how they react to being faced with something they've never seen before. And yes, it does sometimes lead to surreal situations, such as candidates who claim to be e.g. Java programmers but can't recognize Java when they see it.
Like some of my PhD friends have told me, putting a technical quiz in front of well educated and experienced job candidate is a great way to insult them, and is deserving of a good punch in the face.
What you get from a quiz is a candidate who is intelligent enough to write a program that is plain to the interviewer. That is, it is neither a wise answer nor a clever one. It is simply an explainable one, and it is usually the explainable ones that show up in "For Dummies" books and have no practical value. You could be interviewing Einstein who would give you an answer that breaks ground in uncharted territory, but you wouldn't hire him because your mind couldn't comprehend his explanations. You could be interviewing Jesus, but his wise answers would be so over your head that you'd not hire him because you couldn't grasp how many risks were calculated in giving you those answers.