Slashdot Mirror
Tough Tests Flunk Good Programming Job Candidates
snydeq writes "Fatal Exception's Neil McAllister discusses the use of quizzes and brain-teasers in evaluating potential software development hires, a practice that seems to be on the rise. 'The company best known for this is Google. Past applicants tell tales of a head-spinning battery of coding problems, riddles, and brain teasers, many of which seem only tangential to the task of software development. Other large companies have similar practices — Facebook and Microsoft being two examples,' McAllister writes. 'You'll need to assess an applicant's skill in one way or another, but it's also possible to take the whole interview-testing concept too far. Here are a few thoughts to keep in mind when crafting your test questions, to avoid slamming the door on candidates unnecessarily.'"
Radical idea: have them write code for a few hours to solve a given problem - then see how their solution looks. This goes a long way towards judging their fit for the job. You can even give them a couple of data structures and algorithms references - on the job we use references all the time, and being able to implement something from a reference and apply it to a problem is a real skill.
Tests can be a good measurement of quality when the test is material that can be studied for. In school you have a test at the end of a class. For certifications, tests are meant to measure knowledge gained during training. In graduate school, qualifying exams are done to second year students who have time to prepare and hone their skills.
Testing somebody from a cold start, on subjects they have no practical way to prepare for seems like a good way to hire a trivia expert, but the productivity of an employee should be evaluated by his resume and portfolio.
...and I'm a darned good programmer. Sounds like they work to me.
"Please describe the scientific nature of the 'whammy'" - Agent Scully
Is not to play.
Whatever happened to tests like drinking the interview panel under the table?
Now that is a skill needed on the job.
I am anarch of all I survey.
Teams with diverse thinkers are often the most effective. The one who is not good at math puzzles may instead be good at understanding the customer's needs or the intuitiveness of user interface designs in the eyes of non-techies, and vice verse. They each can focus on their specialty, or at least help each other out in their weak spots.
Table-ized A.I.
If you get hundreds or thousands of applications for a single position, how else do you narrow them all down to one candidate?
Tell me wrong, but if google is full with EXCEPTIONAL developers, why there is not even a single product made entirely by them? Please, even one is enough (and don't say: Google) Anyway, my point is that these mehods are good enough for finding anything, but good developers, which is maybe their aim in the first place.
Jeff Atwood recommends the FizzBuzz test.
I agree.
While a programming test can be a valuable part of the interview process, it would be a mistake not to do some in person "tests" as well.
Why? Because the entire point of those tests isn't to see if they get the right answer, it's to see if the candidate can work with the people in your office.
There's no -1 for "I don't get it."
The interviewers, et al may be professional programmers, but they know less than jack about testing, evaluating skills, etc.
When confronted with this I ask "Who publishes and standardizes this test?"
The answer so far has always been "Well, it's just a bunch of questions we cooked up/"
Which, considering my views on hazing and one-upmanship, prompt me to rise and not so politely depart.
It sets a tone for the relationship like asking someone on a date and having them cover the check
The offshoring of software development over the past 15 years hasn't just trashed the quality of the software that many American businesses use, it has also trashed the ability of software developers to become managers.
The best software development managers were formerly software developers themselves. They know that experience is what counts. They know that bullshit HR tests don't work. But these kind of managers are now retiring or getting promoted to executive positions outside of software development. There's nothing but a huge void following them, since there have been very, very few software developers in America over the past 15 years.
The people we have following them often have no software development experience. Many of them are MBAs who don't even know of any programming languages beyond JavaScript, and they only know of JavaScript because they read about it once in some article that was hyping it. The worst are the "professional project managers" who don't even have any relevant college-level training in any useful field (yes, that's right, sociology majors don't know how to be software development managers).
We don't find good managers in the places where the software development was offshored to, either. Skilled management was never a factor there to begin with, and thus the void has always been present over there.
Offshoring software development has been one of the biggest economic mistakes that any civilized nation has ever made.
It seems like every job posting now has around 50-100 people who apply. To weed out this many people en masse they will make you do just about anything - tests that have little application to the job that you are applying for, bark like a dog, sing the interviewer's favorite Barbra Streisand song, paint a painting of a nice wilderness scene, tune the carburetor on the interviewer's old Triumph motorcycle... Many of the people are well-qualified and even over-qualified! To weed them out on that alone would go nowhere.
If I had to tell you how many times I've been asked something stupid and cliche like "Tell me about a time when you experienced change" or "Tell me about a time when you faced challenge" I might go postal. It's almost like HR people invent these questions to pad their interviews because they don't really understand what or who they are interviewing for. I long for the days when a hiring manager or, god forbid, the company owner/proprietor calls and asks you "So, tell me what you are about and tell me why you think I should hire you."
They can treat applicants like total bastards and get away with it. With this kind of market what is really to stop them?
(In the end I admittedly had absolutely no idea how how to solve the problems, and didn't even attempt to. I got the job anyway.)
When I interview people - I feel it is my job to "extract" the best out of the candidates, and to find out what "their best" actually is. If I come away from an interview and don't have a strong feeling for a candidates abilities - good and bad - I feel as though I didn't do my job as an interviewer. I've seen too many people "freeze up", or just be shy in interviews. These people maybe were VERY qualified - I feel it is always my job to understand that. My creedo is this: Get the people talking. Get them talking about what they do, and what they love. If you can do this - they'll go into the depths and bowels of their technical knowledge, working style, experience, etc.
Trades like / apprenticeships / schools are needed for the TECH field.
To many tests / certs are based on cramming for the test. Certs also have questions based stuff that you just about never see in the real work place or have software setups that seem to be a world where software is free.
The beast way to assess an applicant's skill is to have some kind of a apprenticeship or temp to hire.
Other good ways are hands on tests, go over past cases / stuff that happen in the office and asking what would you do?
Some bad ways are off base questions, basing on GPA, degrees and what school you went to.
GPA are bad due to filler class being part of them as well classes that are all about the test (good people at doing a job may be bad at tests and there are people who are good at taking tests but are bad at applying stuff that the covered aka the paper mcse)
I interviewed over the phone a couple times there.. They try to "ease" the difficulty by providing hints and asking you to "talk through" your process, etc. The problem is that these questions are often poorly defined (and I suspect made up on the fly), and one interviewer's "hints" were very distracting. In both cases I figured out the answers 5 minutes after hanging up, when I could finally be alone with my thoughts.
Google invited me to interview for a Java programming job. They started the interview by informing me that I would be "the oldest person in the group" (I was 39 at the time). Then, I was invited to code a linked list in C on the white board while they watched. I can do this, I suppose, having done it 20 years ago while getting my computer science degree. And never done it since. I questioned the relevance of the problem pointing out that this was surely not required for programming in Java. It kinda went downhill from there...
Sure this technique has tons of false negatives, but I think it has fewer false positives than many other interview techniques. False positives are a much bigger problem then false negatives when hiring.
Problem solving ability is what they are really trying to test. If you cannot figure out what to do, and lack basic problem solving ablilties you are one of these guys http://professionalsuperhero.com/ .
"Talk is cheap. Show me the code."
Bullshit. The vast majority of the questions are about using an appropriate data structure and applying it to a relatively trivial algorithm. If you don't want to learn these things, and for some reason are determined to be a developer, I suggest picking up Ruby on Rails. After a week or two you might even be able to make significant improvements to MRI.
Considering that these questions can be asked once, maximum 3 times, before someone posts it to the Internet, now that I've memorized all of the right answers there they go changing the rules. I'll never get that job now.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
The two biggest issues in programming are proper usage of programming language grammar and effective problem solving.
Great problem solvers with poor grammar skills write buggy software fairly quickly. Great "grammarists" with poor problem solving skills write good code but the code takes a long time to get done because the problem solving is difficult.
I remember working with a psychologist on an IQ interview (IQ tests are crap, proper IQ evaluation is done through interviewing with a trained clinical psychologist) and I pointed out that the solution given to a problem would not actually work and how my solution would. I had to work with her on beam deflection, but, she understood the issues. The point is that really creative problem solvers might come up with answers that don't fit within the test parameters which is exactly what you want with a programmer, innovative problem solving.
Has anyone considered pitting them against each other in mortal human combat? That would surely make the cream rise to the top!
There were two parts to it. One was a set of about 20 questions that I maybe had an inkling of the answer. It was a warm day, I was wearing a suit and stuffed into a room by myself to see how many I could answer. Are they testing my abilities or my tolerance to being in an uncomfortable environment? Most of the questions were ones that only someone with OCD would be able to answer off the top of their head. Information like what protocol goes with what port is stuff that I don't use on a daily basis, but I know perfectly well where to look for it when I do need to find out. That's what reference material is for.
the second part was "here's a problem, how would you code the solution". In this case the language could be anything, so I wrote in pseudo code and even went back over one of them because I though of a more efficient solution.
Things that can be looked up in a reference manual or on the internet should be left out of interviews (unless the question is along the lines of "How would you determine XYZ?"). Keep it to methods used to solve a problem and you'll get a good understanding of how a particular person works and how they will work for you.
These are really just a proxy IQ test with questions that only someone on the far right end of the bell curve can answer. The reason they do this is because requiring an actual IQ test to be taken is illegal, so they've found a loophole. If the company that was hiring you was not actually requiring that the questions be solved, then my guess is that they had a lower IQ bar than Google but didn't understand the reason behind what Google/Microsoft et al does enough to find some questions that were hard but still easier than Google's questions. Basically a cargo cult approach to hiring.
If I have seen further it is by stealing the Intellectual Property of giants.
I almost want to apply for a job just to play the puzzles.
No kidding. I was a software engineer for almost a decade and got some bizarre interview questions with no connection to any relevant problem I had to solve in nearly 10 years of writing apps. What was bizarre is, most of the time, the problems had no connection to anything the company did, either. Nothing about logistics, data import, connecting to a web service, or setting up batch jobs, none of the skills you need every day.
The worst offenders are the third party testing companies which seem to specialize in obscurity.
I switched to freelance writing. It pays better, it's age proof, there's no commuting, no dipshit third party testing companies to slog through, and I don't work for idiots.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
partly - perhaps even mostly - due to this new strict 'screening' bullshit that everyone in the silicon valley seems to be doing, now. I've been out of work for more than a year now and the interviews I have been on have been markedly different than they were 20 years ago when I first came to the bay area. lately, the interviews are confrontational. they are assuming you are a liar, incompetant and many other bad things - and its up to you to disprove that. they do not seem to want you, they are there because their bosses told them to interview some new people.
this is quite different from the case where they really are looking to hire and are excited to have found a match (at least on paper) and just want to verify that you are who you say you are. even as little as 5 years ago, the interviews were not so negative. it was more of a verification that you met your paperwork. now, they don't bother to read your resume; or they assume its all lies and you have to start from ground-0 and prove an entire background to them. *over and over*, too; with each new guy that steps in for his 45min slot. show me linked lists; show me trees. show me O(n) stuff.
the problem for me is that I am starting to not care anymore about this level of detail. I'm 50 and have been doing C since I was mid 20's. I have done my time, to be sure; but I just don't really get into having to prove it over and over again, like my resume is all a pack of lies. it gets tiring and its making me question whether I really want to re-enter this field with these kinds of people 'running things'.
I do think that people like me are worth retaining in the software development field. true that I'm getting tired of the lower level details and things that are reference (like precise steps in deleting a linked list node) are of no interest to me when I can quickly get the correct sequence in a few minutes of search. there is too much to keep track of and older things do age out, its true. but older folks do have a lot of other things to offer. its a shame that we get passed over due to how the 'tests' are structured these days. I once was able to jump thru some hoops, but now, I'm just not so motivated to play their bullshit 'test me' games anymore. its a shame since I'm not alone in this and losing experienced guys like me is a real loss to the industry.
--
"It is now safe to switch off your computer."
My last interview was for a "C" programming position (Point-of-Sale system for a large retailer) which was a very good fit for my experience and skills (30 years C programming, almost a decade on another large P.O.S. system).
Unfortunately, the only technical interviewer who talked to me was a Java programmer. He thought he was also a C/C++ programmer, but he was wrong. After 45 minutes of questions about details of Java class usage, it was clear that he couldn't tell a good C programmer from a bad one.
I didn't get the job, and I suspect that whichever Java programmer they ended up with is thoroughly miserable programming in C.
I recently got a new job and have definitely seen a lot of this. It ranges from the esoterically retarded ("Can you call a destructor in C++ directly?" (The answer is yes, but it's a terrible idea -- so is there any difference in practice between never doing something and thinking you can't? Why not quiz me on goto syntax -- that's at least useful sometimes)) to one company I really liked that sat me down for 3 hours with a Linux box and said "Solve these 3 problems as efficiently as you can" -- all of which were fair and managed to solve all of them quickly, correctly, and I thought the resulting code was beautiful in its simplicity, efficiency, and readability. I also had fun doing it (hey, some people write entire kernels for this reason;)). I've also failed one of these so-called online tests for reasons of the above. Their loss. But why would I want to work for anyone who thinks some incredibly stupid test means anything?
The job I ended up taking? I sat down with their lead dev and we talked shop for an hour. He wanted to know what I did and how I went about doing it and we talked about how they're using CUDA to develop higher level APIs to make it easier to use the GPU for computationally expensive operations. And then we ate really good Indian food with the rest of the team.
So if you're in charge of hiring, remember, that good programmers are a scarce resource, and we usually have more than one offer on the table. Don't waste our time and we won't waste yours.
-- Political fascism requires a Fuhrer.
Screening interview for a Morgan Stanley quant job consisted of being asked (somewhat tricksy) maths questions over the phone by someone from HR who declared at the beginning that she did not know much about maths; wasn't interested in me telling her how I was thinking about the problem and didn't expect me to look up any references.
The Morgan Stanley quant group is missing out on a lot of good mathematicians.
Good Programming Job Candidates Flunk Tough Tests
Is it too much to ask?
I've made it a point to ask ahead to confirm that there will be no "brain teasers" and if there may be, to kindly ask them to rescind the interview so that they may focus on more qualified candidates who share their core values (being a self-congratulating asshole who gets a boner doing brain teasers)
Frankly, the very fact they ask brain teasers tells me a lot more about them than my answer will ever tell them about me. The fact that I put it bluntly, but kindly to opt out in such an event should show my "free-thinking out of the boxness", though lower my score as a team-player (read: leader not follower). But the fact they put such a priority on stupid pissing-contests shows they lack creativity, have no clue how to actually assess someone based on their merits and skill and place little value on individual contributions as much as maintaining their "culture". All in all, they show they don't care as much about the individual's technological aptitude as much as "fitting in" - so if getting lost in a cube farm is your goal, study up on these brain teasers. But, if you're looking to develop hot software, go with a smaller organization where your contribution will be greatly felt and who don't have time to waste playing games with candidates.
A hiring manager asks his secretary for the resumes for an open position, and she hands him a stack of papers. He splits the stack in two and throws half of them away without reading them. She asks him why those people were rejected. "They're unlucky," he said. "We don't hire unlucky people"
A lot of companies are also just very bad at interviewing. A lot of times they'll just have a couple folks from the team you'll be working on talk to you for a while and you get in or not based pretty much on how much they like you more than any technical skills you may or may not have. You can skew this process more in your favor by working your social network, though for that to work you have to have not completely sucked at your last few positions.
If you implement it badly, you're not getting any higher or lower quality employees with the brain teasers than with any other method. And if you implement your interviewing method well, you're getting pretty much equally good employees no matter how you choose to do it. For most companies most of the time it's really just a matter of luck if they get good people.
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Sour grapes. Puzzles test your practical problem solving skills, and if you don't think that's a part of programming then you may be more suited to some executive position where you decide whether things should be blue or not.
Having built a software development team from scratch for a venture-funded startup, and having done tech vetting for years on consultants, I fully understand the difficulty in determining who's actually talented and who isn't. But I just don't buy that the 'puzzle' approach translates into 'great software engineer'. You may well get a bunch of bright and clever people who are good at puzzles, and it's good to have some folks like that around, but I suspect if everyone is that way, you'll end up with a bunch of engineers trying to out-clever each other -- and that doesn't translate into well-designed and readily-maintained software. IMHO. ..bruce..
(What do I think is necessary? See here and here.)
Bruce F. Webster (brucefwebster.com)
I was interviewing with a company that did a lot of mechanical engineering but lacked the software expertise. They basically offered me the job at the interview and I said, "Sure." Only one "formality" remained: a mechanical skills test. Now I was being hired strictly for software and would never, ever have to turn a wrench. The test consisted of a hand-cranked machine that moved a block of plastic around a table surface, flipped it over, around the corners, and returned whence it started. Lots of cams, gears, pulleys,etc. Four parts, 15 mins. per part. They "break" the machine and I have to find what they changed. The first part I got, a cam was out of phase. Second part I got half of it but couldn't find the 2cd part. Third part I was stumped and then the tester said, "We're done." I asked about part 4 and he said he had what he needed. A few weeks later, nothing. I call the hiring manager and ask what is happening and he tells me I failed the test and there was nothing he could do about it. I said I didn't take all four parts and he was angered about that but HR dug in their heels. He put a request that mechanical engineers take C++ test but no dice. I asked him if there was something else he couldn't talk about at work and asked him to call me after work and discuss this. He said, "It's the test, nothing more." For real. My friends in ME said that they would have had problems with such a test yet a software engineer is expected to pass. In the subsequent years, I saw positions I was very qualified to fill at the company, but I couldn't be bothered with them
A few months ago I was interviewed by a Google recruiter. One of the questions was "What is the number of bytes in an ethernet address?". My real answer to that question would be "Who cares", instead I gave the wrong answer. I deal with ethernet addresses daily, but I never bothered myself to count the bytes in them. And I don't think it is valid to claim that "if you are experienced in the Ethernet protocol, you should know the number of bytes in an ethernet address." The other questions were similar too. One cannot learn anything about my engineering experience or skill set by looking at my answers to those questions. (And Google couldn't.)
Briefly speaking, viable or not, I believe the only way to assess the work experience and skills of a software engineer would be to look at his/her projects, source code, documentation, and the processes used, and to question each. But I suspect there may be very few such recruiters in the world.
PS: I am sure there are many applicants to Google, hence there must be multiple stages in the screening process, but this does not warrant such stupid questions to reduce the number of candidates. Because they will certainly eliminate many suitable candidates in the process.
The essential elements of a good developer lay in your personality.
All of these things, good developers make and none of them are found in a math test. You might even say that these math quizzes do more to screen these people out than let them in. You don't need to screen programmers for math skills; if they were so bad at mathematics that simple algorithms confused them, they wouldn't even try picking up the trade, and advanced algorithms can't be divined on the spot anyway. There's a reason most of them have names.
Normal human response to uncertainty (noise behind a big rock): a spontaneous decision between fight or flight. If the interview is testing your human response to being faced with solving unknown problems, then the puzzles work. If the interview is testing your experience with any particular technology, then they obviously don't. Of course, if all they want is your expertise, then it's not a place that will let you grow in your skills. It's a place that's looking to squeeze you until your skills are obsolete and then replace you. Because technology changes fairly rapidly, you should be looking to solve unknown problems as new technology emerges. If you are not, then your shelf life is too short for a company to make a long term investment in you.
Any guest worker system is indistinguishable from indentured servitude.
I've interviewed lots of people using puzzles of this sort, and I find they work really really well for picking out the better programmers. You need to understand how to do it correctly, though. You're not looking for whether they "get the right answer". You're looking to see how they approach the problem and what sorts of solutions they try (even if they end up not working). When you interview a bunch of people this way, you find they split into a few groups, and the differences between groups are really obvious. For example, some people will just have no idea how to even approach the problem. Others will struggle to figure out an O(n^2) solution. And others will instantly take it for granted that of course there's a trivial O(n^2) solution, but you're obviously looking for something better than that. The differences aren't subtle.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
That's too much work. If it's a female applicant, you should just ask her to take her top off and if she does, hire her.
If it's good enough for Herman Cain, it's good enough for me.
You are welcome on my lawn.
I hate interviews, so I started up my own company. Half the time you work for someone else you're thinking you could do their job much better anyhow if only you were in charge. Great thing about having your own company, you get to be in charge, and you get to do whatever the eff you want to do, exactly how you believe it should be done.
It's not a common knowledge problem until you state that every woman knows that every other woman knows this. If the women have philosophy degrees, nothing much happens. If they don't, they will charge ahead on the assumption that "every other woman is just like me" as all logicians sans philosophy will do. In that case: on the third full moon, all the men depart on a very long hunting trip after several months of heightened debauchery. Assuming they know whether their women have philosophy degrees and what they assume about whether the women also know this. Or perhaps there's a hasty shotgun marriage before this all plays out with suitable supervision to constrain what A knows about what B doesn't know.
At my ripe age, I personally find it more entertaining to conduct a code review on the statement of problem than to actually solve it.
The pirate problem contains a misleading plural of unreliable plurality:
How does the lowest ranked pirate reliably defeat his immediate superior in a knife fight while folding his ballot slip? I think you need a set of complex interlocking manacles to implement the problem as stated. Interesting. At first it doesn't look like a crypto problem.
This whole hiring thing devolves into a common judgement problem. Good judgement is the way to hire good people. But how do you know the person doing the hiring has good judgement? Because you hired him or her in the first place, supposing you implemented this policy on day one. But no, you're saying you don't trust long inference chains after all, and you'd rather I just completed the problem set?
Perhaps the real point of the Blood Queen of Bath problem is to prove you know enough about what logic implies to never assume it holds sway in the real world.
One time I went on an out-of-town training class for a particular FEM pre/post processing program that's popular in our industry. Funny story actually, I show up at their regional office and a couple of the sales reps who I knew looked at me blankly and said "Oh. That session was cancelled. Nobody signed up." That was pretty funny. So anyway, seeing as how I was working for a big customer of theirs at the time they decided to let me self-learn with their class materials and a machine. So I did that and after 1 day I find out about their macro programming tool and after 3 days I've written some pcl modules hooked in to GUI thingies. I know, not heroic but remember I'm not a software guy I'm a gearhead.
So the one fellow in charge of the office invited me along to dinner on the last night of my self-course and at the beginning of the meal suggested that I should apply for a job opening they have coming up for some kind of application engineer. Great I thought, flattering. I might have to take them up on that. Then I proceeded to tear into the lobster we were having with bare-handed gusto. I mean, fragments of lobster were flying all over. I may have had bits of it in my hair.
Still waiting to hear back on that job thing...it's only been 18 years...and they got bought...ahhh what the heck, between you and me it's a crappy program anyway.
Equine Mammals Are Considerably Smaller
I've sifted through hundreds of CVs and interviewed over 50 people. It's hard finding good developers. I don't think puzzles or other forms of intelligence test by proxy are a good way of finding them.
I try to get candidates to talk about the work they've done and communicate what was interesting about it. That one filters out those who don't really care. I also ask about situations where a commonly held way of doing something should NOT be applied. That filters out the ones who've got all the "right" answers by book learning, but haven't actually thought about it themselves. I also ask some ambiguous questions to see how they deal with them.
I've offered jobs to people who got many, many details wrong, but clearly demonstrated they were into writing good software and were easy to work with.
I've dealt with several of these brain teaser interviews. Given the quality of software from the companies in question, they don't do a good job of choosing qualified engineers. There are two ways to flunk: one is not to figure out the answer, the other is to know it at once!
My current employer did something quite different. They showed me a C expression, and asked me what the result is. I took a look at it, and said at once "you can't do that. It's ambiguous and undefined in the C specification." They all looked at each other, and then let me in on the secret: a programmer that had quit had written that line, and it was the cause of a bug in their product that took the rest of them MONTHS to find. The decision to hire me was made then and there.
Testing on school problems tests how well the candidate solves school problems. My employer's test was something that actually happened for real. And since I started there, I've found and fixed innumerable bugs. That isn't even my job responsibility; I do design and development. But every so often, I go on a extracurricular bug hunt just to have fun. They haven't minded. :)
School problem tests discriminate in favor of clever youngsters, and against the knowledgeable experienced. It should be no surprise that companies that only hire clever youngsters end up with software full of blunders that a knowledgeable experienced developer would never make.
I have a reputation for being hard on candidates, and I do like to push them way out of their comfort zone, I like to use red ink on their resumes when I think a skill they claim is unjustified.
But, I'm more interested in how they attempt to do things rather than whether they really solve the problem. Some people crank out code instantly which has bugs, others just freeze up and I have to go elsewhere, but it's all part of judging the person.
I guarantee somebody just wrote a book on how to interview. It makes the rounds in MBA schools then gets into "HR Today" magazine and simply everybody's got to use it. Same crap goes around every few years only to be replaced by the crap the year before.
I have the luxury of mostly interviewing PhDs who have already passed a couple rounds of initial screening. Abstract puzzles have always seemed to me to be weighted random number generators. Problem solving is one of the most important skills of a researcher in my field, but solving a problem without any preparation in 5 minutes doesn't correlate well with solving a problem through research over a few months.
My colleagues and I ask about multi-month or multi-year problems solved by the candidate in the recent past. We evaluate the work, his or her ability to describe that work, and then start probing a lot of the technical and logical details to make the candidate prove that he or she isn't just reciting what collaborators or an advisor did.
When I'm interviewing non-PhDs who don't have much experience to explore, I will ask some puzzle questions, but they're always taken directly from the work in our labs and are the kind of thing that someone in the field should be able to at least intelligently troubleshoot in a few minutes - that is, they're the kind of problems we expect our employees to actually encounter on the job.
As another poster pointed out, a lot of the common abstract puzzle questions aren't even well defined. Half the time it's easy to come up with two or three reasonable answers that are equally valid. In my experience they measure experience with this particular brand of puzzles more than anything else. As someone who's always done very well on that section of interviews, I suggest all interviewers who use them should instead look for ways to evaluate performance in real problems from their field.
"I zero-index my hamsters" - Willtor (147206)
"A few months ago I was interviewed by a Google recruiter. One of the questions was "What is the number of bytes in an ethernet address?". My real answer to that question would be "Who cares", instead I gave the wrong answer. I deal with ethernet addresses daily, but I never bothered myself to count the bytes in them. "
I was in the same situation, about the same date, so maybe they re-build their "fast question" from time to time as if they were passwords.
I deal with ethernet addresses from time to time too, and as you can expect, I never bothered myself to count the bytes in them, just like yourself.
The difference is that I *do* deal with that stuff from time to time, is not that I just *say* it, so it was a matter to recall "hey! there are six groups of two HEX in a MAC, so one byte per group for a grand total of six bytes".
So in the end, it's a matter of a Dunningâ"Kruger effect: you are not as good as you think you are.
"Because they will certainly eliminate many suitable candidates in the process."
Again showing you are not as smart as you think you are.
What do you think that worries Google (or other well-known compaines) the most? rejecting 100 good candidates or hiring a bad one?
You've got their resume, you've got the word of their past employers, they've given you some sort of portfolio, isn't that real measure of whether or not they can get the job done? You've got years of work you could go over, but instead you're thinking that in thirty minutes you'll be able to judge a man's skill?
It seems like an interview should be more about getting to understand someone's personality. Are they a good fit for your department? Maybe you could also spend some time clarifying things in their portfolio, trying to understand the work they really did. Giving them a paper test sounds like a waste of time to me.
I just took a multiple choice c# test for a potential position -- they used java terms and keywords all over the place in the questions and answers. I made it through the screening and removed my name from consideration for the position.
The moral is, sometimes you can tell just as much about the company as they can about you.
I had this aptitude test one time where the interviewers put us into this room with these uncomfortable chairs and they provided no writing surfaces. While we were proctored the test I dragged the only table in the room over to my chair so that I would have something to write on. Turns out, that sort of ingenuity was what they were looking for - it was all a trick question.
I do systems software.
My last encounter with trees, threads, priority queues, condition variables, and mutexes was this afternoon and any one I hired could be doing the same thing on their first day of work.
I don't risk giving guys their first job, although I include free software and interesting project classes the equivalent.
I ask every candidate regardless of seniority (you can get surprisingly far in industry without much skill beyond high-level hand waving)
1) A simple programming question with a couple of edge conditions. It can be solved in one loop although I'm very happy with candidates who use an obviously correct solution with a pair of inner loops which totals 17 lines including braces where not technically needed added in the K&R style. I don't mind pointing out an edge condition that's been missed, just that they address it in the end (in the real world a test would find it).
2) A simple data structure question. I give an API with three functions, ask what data structures the candidate would use, and what the operational complexity on the three functions would be. I don't want an optimal answer, just something workable, and don't mind answers like "some sort of" that would lead to a book or web lookup. I'd be happy enough with an STL container answer; although pretty much everyone without the basics doesn't do well at that abstraction level either.
Candidates worth hiring in my opinion and the majority who've passed hours of less hands-on interviews take about five minutes for each (maybe ten minutes for the first problem if they've been withering away as a project manager which is fine) after which I ask a five minute thread question and spend the remainder of the time talking about what they've done and assessing whether they have attitudes about software test and process commensurate with time in industry.
Ones not worth hiring can spend an hour on the first problem and still fail to get it when I explain how it would work in English with diagrams.
Google's been doing this for a while... look what it's gotten them: http://www.one37.net/blog/2011/11/3/googles-missed-opportunity.html?utm_source=loopinsight.com&utm_medium=referral&utm_campaign=Feed%3A+loopinsight%2FKqJb+%28The+Loop%29
Google reminds me of the time I went to work in France as a young American self-taught programmer. I found that the French programmers were extremely well educated and they always wanted to do things the "right" way... unfortunately, that led to rarely getting anything done on time and when it was done, it was an over-engineered mess (sound like Google).
As someone who has been through these for Microsoft and some other big name companies, I completely agree with the post. Although I was successful (I was offered a position), it just doesn't make sense. When you program, you don't do it in a pressure filled environment. You take your time, you gather requirements, you design and implement within the time available and you do the best job.
My perception of these tests has always been some smart-ass programmer who wants to show that he's better than anyone else making young programmers try and solve stupid puzzles as fast as possible. It's pretty ridiculous.
The company I work for now has the best idea IMO - when you submit your resume, you must also submit a working program that solves a business case outlined. You can of course take as long as you want, do it at home, and use internet resources. But the program isn't judged on just code alone, it's judged on many other elements. This course of action has allowed us to hire very intelligent and well rounded developers, and weeded out all those who were just "carpet bombing" resumes (as they wouldn't even read that they had to submit an application as well, or were too lazy to do it). In addition, we can immediately see the quality of code, style (assuming they didn't copy), and just as important how they think the UX/UI should be, etc.
Why more companies, like Google, don't do it this way is beyond me. I feel like they just want to be assholes and try to intimidate new recruits, or "show off."
A few times now I've gone through "logic tests" during interviews. When they give me that 4 page sheet and say I have 45 minutes to complete it, I go "Oh boy oh boy!" and usually finish the whole thing with five minutes to spare. I did those sorts of problems as a hobby as a teenager. I love them! Then they give me the personality test, at which point I am screwed.
The last job interview I had involved no testing or quizzing at all, just proof I didn't lie on my resume. As a result, I was hired by the end of the interview and I've been with the company for a bit over a year now.
Occasionally living proof of the Ballmer peak.
Yep, I said it.
I have been to several University nerd parties where a Google employee coaches his friends and prospective candidates as to how to answer some of the questions.
What I have notice is some of the questions require giving a wrong answer, and any answer other then the "one they want" will be a mark against you, no matter how correct it is.
So basically you need to get the answers or at least enough clues online to be able to pass the interview. You must study for it like the SAT's. I have even seen Google Interview Study parties. I consider this cheating, which is in my humble option complete bullshit.
I have been programming C since 1982 and C++ since it's inception in 85 or something like that. I have plenty of code in the Open Source and worked on so much code I can't even add it up at this point.
I am always the one who can fix the hardest bugs, Kernel panics/oops, pour through core dumps, clean up drivers, JAG the hardware, and do the board bring ups.
I have worked on code in over 100 languages counting all the flavors of assembly language and scripting language.
I have developed some of the cleanest architectures and have code a number products that are on the market today and in the BSD and Linux Kernels.
I deliberately keep to a subset of C and C++ and avoid many things that make machine parsing and code analysis in sed/awk/grep/find difficult.
Occasionally I needed to look up things when I see someone do weird obfuscation crap in their code.
http://churchofbsd.blogspot.com/2011/11/weird-obfuscation-crap-in-their-code.html
Some one that passed one of those interview tests probably thought he was being clever.
In professional code we don't want clever. Clever is BAD because clever makes the next guys job hell.
At this point my fingers know the language, my eye's just know what looks correct. I think what I want and it just pours out of me, but don't ask me to explain because I am not sure, I'd have to stop and think.
Much the same way I was with phone numbers (Back when we had to dial them) where I needed to actually dial the number to see where my fingers go to be able to tell you what the number was.
I find I code best when I am not thinking, I literally don't look too closely at the screen. I just keep myself distracted and only stop to consciously think about the larger design and architecture.
So I can't code on a blackboard. Just can't do it. I never was able to, and I am not about to try now after 30 years of VI on CLI.
As a result I can't get through most of the interviews like that. Fortunately I already make more money then most of those places would pay anyhow.
I don't need such abuse, I am not one of those sad old guys that can't find work doing Cobol any more.
I am always doing that which I can not do, in order that I may learn how to do it. - Pablo Picasso
Sounds like you're trying to weed the good artists. I have hired graphic artists as well. And it was a done deal pretty much before I even talked to him. There was one thing I looked at and that was portfolio, the interview was just a matter of finding out if they were an idiot or not. It doesn't matter what application a good artist uses to produce the artwork, as long as they produce good artwork. You can train on a piece of technology, as long as the person isn't stupid. But if you think you can actually train a good graphic artist, you're the fucking stupid one. Becoming a good graphic artist takes years of training and practice.
But I can see why you no longer do interviews, you actually fucking suck at them.
For instance if I say I know C# and you want me to bang out some code in Python, we're done. Because whether or not you realize it, I'm also interviewing you. And I have stopped the interview process in the middle because the interviewers didn't have their shit together. Seriously if you're going to fuck with me during the interview process you probably don't you have your shit together as much as you think you do. So things are really going to be fucked up once I get started, and I have start looking again. No thanks; find somebody else more gullible.
If someone is passing you on the right, you are an asshole for driving in the wrong lane.
These tests are too easy for all of these companies to use in eliminating people which they consider undesirable: too old, too young, too long out of work, wrong gender, wrong skin, whatever. It is not just the big companies, a lot of places are using online tests to screen candidates. Some of the tests I have seen serve no purpose other than to make the candidate feel completely worthless - so they can drive the rate down. Language tests "drill down" beyond the level of language trivia, where the answers they demand are likely wrong anyway. As a consultant for many years I did lots of technical interviews. The absolute best method is putting someone who actually has knowledge of the technical area, in the room with the candidate and let them talk. If you want to know how well they know PL/1 give them a PL/1 listing and let them tell you about it. Don't send in the Guru your hired last time, because he is only out to prove he is a better programmer than the candidate. That will never end well. On the other hand too often a project manager is hired who knows little or nothing about the implementation technology. For example PL/1. The project manager will eventually find the Hughes Book, which he will peruse for highlights. The Hughes Book uses a collection of words to describe PL/1 storage types which those of us who have done PL/1 extensively simply do not use. So the PM asks the candidate over the phone to name those types: The candidate replies automatic, based, controlled, and static. The candidate is disqualified because the list does not match the terms used in the Hughes Book. Another candidate who had a PL/1 class in college and read the Hughes Book the night before will get the job because he is working from the same script as the PM. This is the main reason knowledge professionals (including programmers) need to organize.
.. simple practical tests can rid you of the unsuitable candidates very quickly.
I went through an interview process a few years back for a company writing software which required life-and-death accuracy, so their interview technique was carefully designed. First test was with a recruiter .. basic C++ programming skills. Simple questions that are obvious to people who know the language .. like "Create a pointer to an array of 12 characters and populate the letters A through L into the array" or "Create a class with a member structure to store a street address". I was told that this test weeded out 90% of candidates .. 90% of people applying for the job were lying on their CV.
Next, I met up with the team I'd be working with. We just yacked for about an hour, before they sat me down in front of a terminal with a simple practical task.
Finally there was a formal interview with the Manager and HR person from the company.
I didn't get the job, but I did get the point.. and at the time, I wasn't right for the job. The point was to find the right person for the role because bringing in the wrong person could be deadly. It was a direct, simple, effective and (most importantly) cost effective way to interview hundreds of people. The testing process needs to fit the role. When the testing process is appropriate, the end result is good. If the testing process is some abstract HR tests then you can expect the results to answer the question and end up hiring a workforce consisting entirely of people with a single skill set.
have you noticed -- a lot of software is free this node is entirely based on free open source software. freshmeat, source forge, OSDN...
The position was for crash dump debug analysis teams. Let me tell you all 1 thing:
The questions weren't "stupid brain teasers", they were strictly comp. sci. related, see here, as I did a post on here it a few years ago -> http://developers.slashdot.org/comments.pl?sid=155172&cid=13007974
(That post of my experience was "modded up" as +3 interesting iirc, & others had the SAME test too, was nice to see!)
However, the FUNNIEST & MOST INTERESTING PARTS were the SHEER AMOUNT of replies afterwards, attempting diff. ways to do the questions answers (some failed because their methods only worked on specific data types, but interesting nonetheless - very "geek speak", lol!).
Anyhow/anyways - Things may have changed, but, in any event? Well... from the point-of-view I have, actual experience (& making it to the 3rd round of tests YEARS ago? That was my experience with MS - a humbling one actually, but I suppose one I needed... I know FAR more now, but it took time!)
APK
P.S.=> Personally, I thought the questions & tests were perfectly "inline" for the position... I apparently though wasn't "good enough" to make "their grade" back in 2003, but now? Who knows, but I didn't get "worse" in the intervening 8++ yrs. now (only stronger & better, & that interview humbled me some, but made me learn that I HAD TO LEARN MORE IS ALL... when life gives you lemons, make lemonade!)
... apk
If Microsoft does it, there's a good reason to not do it.
If Facebook does it, there's yet another good reason to not do it.
If someone is passing you on the right, you are an asshole for driving in the wrong lane.
The last job interview I had gave me homework to do. They pointed me to a public dataset, and had me write a web-app that could read and display it. Language, tools, platform, etc. didn't matter. Even getting it all done didn't matter. But I had to explain why I chose what I did, and how I would have implemented the parts I didn't finish.
This is pretty much like the real world is - where it's an open book kind of world. Where you can Google, grab open source libs, etc. A job where you couldn't look things up would be a kind of odd one.
C/Unix development/support role. I have had multiple supposedly experienced people fail to answer these questions.
1. What does static do when applied to a variable within function scope?
2. What does static do when applied to a variable not within a function?
3. Aside from multiplication, what does "*" do?
4. How do you list the files in a directory?
5. Tell me a few kinds of inter-process communication.
The preferred solution is to not have a problem.
,,, I got my first break in the biz via the old IBM test (not with IBM). God only knows what it was intended to prove, but apparently it somehow was able to pick out potential programmers. Not so, in my experience, being good at programming was an endevour which required shitloads of learning how stuff actually worked, no matter how good you were at the test. So IMHO, just a case of jumping through a hoop to get elsewhere. That said, learn z/Arch assembly language. As far as I can tell, no-one can do that shit these days, and you can make real bucks at it. Especially because everyone else who tries is totally crap at it.
Alex at The Daily WTF wrote about this problem back in 2007: http://thedailywtf.com/Articles/Riddle-Me-An-Interview.aspx
50% of the people you meet, work with, marry, catch stuff from, support with your taxes, ect. are by definition below average. Think about it!
I was asked to solve a network traffic issue and knew the answer immediately based on my experience. After all, that's why they were interviewing me. They weren't so much interested in the answer (a specific worm on a specific executive laptop), they were interested to see what changes I would introduce into the network to handle such problems in the future. Why I would introduce them, what the cost of each would be (approximately), and what other advantages I could expect from the changes I proposed. That was Google.
I've interviewed candidates at about a dozen different companies for every position in the org chart. I tend to that sort of approach. I'm not so much interested in the "do you know this" as much as how, why, where, and when you would apply it.
As for "can you code this in a specific language", who cares? I want to know *why* you'd code it. Coding is not engineering. It's a craft. I hire engineers.
Past employers lie (I've watched several of my current employers "puff up" someone recently laid off just to get them away from our unemployment benefits...)
Resume? You've got to be kidding.
You can't judge how somebody will do on a 6 month problem in 60 minutes, but you can at least see if they can handle what should be a 15 minute problem. People who can't pull that off within an hour or so tend to be "needy" all the time, and generally much less productive.
Sure, there is a baseline for skills which is somewhere around anyone can do this with 1/2 an interest and some time... skills can be taught.. A good employee is just about always how they get along in the company and with co-workers. A genius slacker would pass the tests and manage to suck your company dry while finding ways to enable his slacking.. An honest focused, caring employee will be there after hours and make a deadline. If you are looking for the genius dedicated employee.. well.. thats a sort of an oxymoron.
Comment removed based on user account deletion
I've been running a mid sized outsourcing shop for decades now and I've come to skip tests and interviews, it's not helping anything.
What I want to see is:
1. The scope and success of your personal projects. This will tell me if you're capable and productive on your own.
2. So you're capable and productive on your own, now you're in for a test period. You're assigned team-work. If team is more productive, then you're not only capable on your own but also in a team. Congratulations, you're hired.
Simple and effective.
It's 4oclock in the morning and I can't sleep - so I'll post this. :-)
Software Development is still very young, very dynamic and very complex.
I've been programming since 1986 and been in professional web software development since 2000. I've done a solid share of projects of all kinds and right now, once again, on the lookout for jobs and projects. Twice a week I get called up by recruiters asking me if I 'know XML' or can drive down to Munich for free to make a job interview for a project that smells of out-of-time, over budget and clueless gouvernance ten miles against the wind. 10 years ago money was free an nobody cared about wether a project was going to succeed or not. Nowadays money is more scarce, but the people in the pipeline haven't learned a single bit.
It occured to me the other week that our profession is very simular to that of doctors. Everybody knows there are complete hacks in the field that can ruin your health for good and seasoned professionals with the patients best interest in mind and a solid experience and will to do good, but only the pros in the field themselves have the capability to judge wether a given MD is from those 30% that are total screwups or it he belongs to the 30% that can make a difference.
In the last few weeks I've been dismissing companies left right and center just by reading their outlandish fantasy job requirements or just asking a set of questions simular to these and noticing what a crappy shop I'm speaking to.
I get simular stories from my cousin who's passed through 3 jobs in the last 18 months - and he's a *real* engineer - y'know, building Airplanes and shit (Airbus 380 and such).
The truth is, after being in the field for so long, I know for a fact that only a 5th or all software teams out there barely fit the most basic standards of working conditions I can even be productive in. 2 out of 3 of those fall flat in some other basic requirement. I think it's a crying shame, because I'd love to do some good and meaningful work in this field, but it's just so darn difficult to find a proper pipeline and the surroundings to make use of it.
Are there any seasoned programmers here who think it should be possible to build an international company that has everything online (vhost servers for versioning, building and staging, online project and task management, etc. ...) and delivers along the lines of a company like this one? I personally do, speak fluent English and German, have solid experience in Organisation (as a scrum master), have technical account management skills alongside my programming stuff and do believe it should be possible to a) deliver good software b) on time c) without crunching d) having fun while doing so and finally e) making a good living at that.
Reply to this post and let us hear your thoughts. I'll leave a mailadress somewhere further down as an reply tomorrow .... errr ... later and we can get together online somewhere. I'd rather work with some slashdotter from the other side of the planet I have never seen than with some disponent from the other town who can't tell Java from JavaScript and earns 3x my rates by renting me out to others.
Let's see if we can improve the industry just a little bit and make a dent in the universe.
My 2 cents.
We suffer more in our imagination than in reality. - Seneca
I came looking for study results, but all I got were assertions and recommendations based on those assertions.
So disappointed.
I definitely agree with the assertions, but I wanted to find out if I was right.
-josh
"IQ tests aren't crap. The idea, that the whole range of cognitive abilities can be expressed in a number is crap. And the idea that this number could is a usefull metric for anything is right out stupid."
So sure you are, young padawan?
"The best definition of intelligence I ever heard is: Intelligence is what you can measure with intelligence tests.
So if you slap together a test that tries to measure intelligence by testing language. concentration, logic and 3D-thinking, you'll get an accurate test that measures exactly that."
Just because you're smart doesn't mean that you know everything.
Now I'm not a researcher in the field, but there is indeed something more to IQ than the commonly believed and superficial dismissal.
Suppose you do just this: "slap together a test that tries to measure intelligence by testing language. concentration, logic and 3D-thinking"
and you look at the results. Say normalize the subscores within each category with the usual z transform (x-mean(x))/std(x).
Do this for thousands of people and do a principal components analysis. What do you find? The distribution is not a spherically symmetric Gaussian---there is one clear and statistically robust principal axis which explains a large fraction of the variance. This means that people who are good at some categories in the IQ test, also tend to be good at others. This is the meaning of 'generalized intelligence' and it is a non-trivial, empirical fact of humans.
It didn't have to be this way: outside of clear disease or disfunction, the performance on various apparently different test categories could have been uncorrelated on average.
The very categories in the IQ test are of course the ones which exhibit this phenomenon the most clearly---if other tasks were something like "sing on tune" or "catch a baseball" were included they would have a much lower correlation. There are unusual psychological tests involving immediate 'stimulus-response' of pressing the right button when the right or wrong words/figures appear on the screen, and results on some of them (e.g. counting milliseconds) can be in fact quite well correlated with IQ measured using traditional cognitive problems.
In sum, people's typical snarky dismissal of IQ measurement is ignorant. This is a frequent occurrence, especially on slashdot (snarky, but wrong) when otherwise smart people think their 15 seconds of "Blink" or nice social prejudice or their talk show host knows more than people who have done professional research in the field for decades.
When I interview candidates, I try to take them through all the different practical skills that are needed for the kinds of products I work on. I start with a little theory, then I move to database basics because "that's where the data lives," and then I get to programming. I try to gauge if the developed programmer experience matches the stated experience; if there's potential to learn, and if the developer has a good / bad attitude.
I start with a general quiz on the basics: What's the difference between a class and an object? What's the difference between the stack and the heap? I'm not looking for correctness; but a general comfort level with general programming topics and vocabulary. In general, I want a candidate that answers these questions from experience, as opposed to wrote memorization.
Then I ask the candidate to design some simple tables and queries based on our business. Nothing too complicated; but if the candidate doesn't get SQL or a relational database; it comes right out. I've spent too much time cleaning up messes from people who don't get programming with a relational database that this kind of filter very quickly separates the men from the boys.
Depending on what kind of time I have, I may ask the candidate how to convert data from the database into objects. The goal is to ensure that the candidate isn't going to blindly rely on an ORM and write code that runs 1,000,000,000 times more slowly then it needs to do.
Finally, I ask the candidate a very simple programming question that requires wrapping a basic collection. I don't care if they come up with the most optimal way right away; what I'm trying to judge at this point is how much of a "programmer's instinct" the candidate has established.
No, I will not work for your startup
There's a difference between a "brain teaser" and an analytical problem.
I've certainly been in interviews filled with "gotcha" problems that only has one right answer, and yeah those are a waste of time.
What you want is a general, yet realistic problem with a variety of solutions. That way you can find candidates who can solve problems and discuss things.
There's no -1 for "I don't get it."
The interview is often a very good indicator of what the job is like. It's just as much of a way for the interviewee to evaluate their prospective employer as it is for the employer evaluating the employee.
Amen.
I once interviewed for a programming position at a very young and small company. Most of the interview was done by the owner, a business/marketing guy with some technical knowledge of the industry they develop applications for but with no real knowledge of software development (other than it takes longer than expected). For the programming test I was handed over to his lead developer. The test was crap but I probably did well on it. When I was back with the owner he asked how the test went. I decided to do my side of the evaluation. I told him I probably did well but that the test wasn't very good. I explained why. The test seemed like a sampling of multiple choice questions from quizzes and tests from a bunch of different CS classes. Given that a CS type degree was required the applicant presumably passed all these classes so nothing new was really learned. At least it was easy to grade, however like many things you get out what you put in. If a test is to be given it should test for something a degree does not necessarily demonstrate, the ability to actually develop code that solves the problem or task at hand. That is what you are going to pay people for. The preceding was not some kind of soliloquy, the owner seemed interested and asked questions of his own and the above gives an overview of what was discussed. The interview ran long due to this conversation. I was offered the position a few days later. The owners lack of software experience was a concern but I got a good vibe from our chat about the test and from his responses to some of my general questions. I took the job. It worked out well, he trusted us and took our opinions quite seriously when he needed to make decisions. Oh, my first task was to write a new programming test.
Today when I interview people and get to the part where I will answer any questions they have, and they say they don't have any, I point out that an interview works both ways. That this is their chance to find out if this company is a good fit for them. Usually there is surprise and a few seconds of confusion but most are able to come up with some questions at that point.
I asked to him to write a certain program on the laptop, using the language of his choice. He chose C++.
He looked up APIs on google and an in man pages.
I'm pretty happy with the program. It's correct, no slower than necessary, reasonably readable. A few minor improvements can be made, and he started to find these before we ran out of time.
Unfortunately he did badly with other interviewers in classic whiteboard situations, so I don't think he'll be hired.
This is the second time I've interviewed this way. I learned that a small task takes much longer to code than I think. Also, that this measures something quite different from whiteboard questions.
A lot of those puzzles fall into a small number of categories:
Once you master the solution to one category, all the variations in that category are trivial.
"Freezing up" is precisely the problem; you can solve a lot of these puzzles by following your chain of logic to its end instead of abandoning it when it looks difficult or sketchy.
Whether these puzzles are good for hiring is another matter.
Why would you count the number of groups in the ethernet address? You learn how to divide 12 by 2 hence obtain 6 at elementary school. I think I am smart enough to do that, I just don't bother my brain with such bs.
When I prepared for an interview at Google (London) I looked up all the information they gave me to prepare for the interview - and they mentioned explicitly that they're not going to give you any strange brain teasers or really twisted questions...
They do give you a lot of problems that you have to solve, that show the way you approach solving a problem.
And usually you do it "with" the interview in some kind of fashion.
They stressed a lot on the interviews being a conversation where you have to explain and talk to the interviewer to show how you're thinking and how you're going to solve a problem.
Analytical thinking is a good start. You know you will hire a lad that will cope with 1% of the work to be done.
Alas developing software requires a few more qualities. Simple things like being structural in proceedings, abide to coding standards and being a nice guy to work with. But sometimes also the capability to foresee the road map of an API and to avoid interfaces you will regret later on. But most of all probably, a developer should have the patience and ability to analyse and solve a problem structurally.
I wish Google no harm, but the way they hire people guarantees a bunch of bright lads that produce brilliant products but potentially cause unmaintainable software to be created. I wouldn't be surprised if systems are rewritten over and over again. I probably wouldn't be a happy camper over there as IMHO there is more to developing software than being on the edge all of the time in order to secure yourself a small but decent and others humongous ones.
Having said that, my view of development at Google is through hearsay and I hope to be proven very wrong in my assumptions.
I hadn't the slightest objection to his spending his time planning massacres for the bourgeoisie... (P.G. Wodehouse)
While you should make the programmers write code and analyse code snippets, it is also necessary to ask them a few tough questions - riddles and procedures - based on standard curriculum. These would test their understanding of concepts, preparation and presence of mind. Also, while you may marginally improve your performance using brain dumps posted on the net, you cannot significantly improve your standing in an interview by just rote learning. OK
In a few hours? So they write a single stand alone program? How about their bug finding skills? Fixing bugs is easy, finding the bug is the tricky thing. How do you test that? Given them a single sheet of code and telling them to find the bug in it is to easy, you told them the bug is in that part of the code.
And how many is a few hours? When I look for a job I sometimes got to squeeze two or three interviews into a day as the company I am employed at suddenly remembers just how valuable I am and tries to milk every last bit of knowledge out of me... if your interviewee has half a day to spare on an interview... wonder WHY he has so much spare time.
Also, if the candidate is doing say a dozen interviews, that means he is spending a LOT of time doing silly tests. And if the test is very silly indeed, you just advertised that you are a silly employer.
And what if the candidate is far far better then the interviewer and writes code the interviewer can't comprehend? What if the interviewer is lousy at setting requirements? I have looked at tests that were clearly wrong, known to be wrong since they were copied from some website and the website mentioned the old version of the test was wrong...
Finding the right candidate is very difficult. It requires first of all that the company asks itself, who do we REALLY need. As the article says, does the position really warrant a PhD? For changing the logo on a wordpress site? "We want the best" sounds nice but can you afford to hire them? And keep them?
Even if you are thinking of writing the next google from scratch, if that is going to happen maybe sometime in the future, you can hire the best and brightest right now but they will leave before you are ever ready and take your budget with them.
I work in the lowest of the low of development, web development, LAMP. *Que cries of ridicule* and that means most of my work just isn't all that cutting edge. I am not writing the next 3D engine or get involved with any math higher then primary school level. Cosine? What for? What website needs that? Even square root I never had a use for.
The most important skill for a webdeveloper? His google skills. There is ALWAYS someone brighter out there who has already solved your problem, wrote code for it, debugged it and tested it for years with full doco. Write your own? WHY!?! I swear you can put together a killed website with the most basic coding skills but awesome copy and paste skills. And you get PAID for it!
You know what I want to hear when I pose you a complex question? "Wait, let me google that". THAT is correct and efficient use of your time as a developer. Know how to phrase your problem in a question you can google and then see if someone has already done it.
But noooo. Companies test for web developers who can implement a bubble search... WHY? If you are in web dev and you did your own bubble search you wasted your time. Far far smarter people have already done it for you.
And if you are that smart, why are you a web monkey?
Leave the rocket science to the rocket scientists and hire people that have the skill YOU need in your company.
MMO Quests are like orgasms:
You may solo them, I prefer them in a group.
I was sent a technical test for a company I was applying to. At the bottom of the test it said, "this should take around 1 - 2 hours". In fact it took me around 5 hours to do, after thinking about it for two days. I've yet to receive the "blow-out" from the agency but I'm surely expecting it, as I was totally honest with them about how long I spent on it. The trouble here is I seem to be more of a sculptor of code and an engineer of it. I chipped away at my implementation until I was happy with it, constantly improving it as time went on and I came up with new ideas, rather than knowing beforehand everything I was going to do and only spending time typing it out.
I'm not against such tests in general, especially for candidates without experience, but I have 10 years experience and a portfolio of finished products. Even so, being confronted with the technical test kind-of reduced me to feeling like a moron (if others can do this in 1 - 2 hours, why has it taken me 5?).
I'll use an analogy to protect the guilty, but at a certain company, I was interviewing for, shall we say, a test pilot position and they asked me how would I go about having people board the plane.
This was so mind bogglingly unrelated that I checked out of the interview process right then and there.
Puzzles work both ways. A good relevant one can help filter an unsuitable candidate. An irrelevant puzzle helps filter unsuitable companies. The division I was supposed to join went on to be disbanded about a year after that.
TFA mentions this: "Broad, conceptual questions are fine, but if you get specific, make sure that what you're asking actually applies to the position on offer."
"For example, if you're mostly a Java shop [...] don't give candidates a problem in Lisp just to throw them a curveball. [...] If the questions focus on trivia rather than your actual work environment, your test will have as much real-world value as a pub quiz, and you'll end up excluding candidates essentially for no reason."
*sigh* Our exam starts with the question "write a class definition for a circle in the language of your choice". An astonishingly few of our candidates are able to do that.
We also ask such questions as "what does 'refactor' mean" and get more blank stares than answers.
I use various programming puzzle tests like that when I'm doing interviews.
They generally have two goals:
1) When its a particularly stupid and irrelavent puzzle, its useful to see at what point someone will call out that its a stupid and irrelavent puzzle. The real world is full of people making stupid requests, and its good to feel out how someone reacts to them.
2) Its good to get a sense of how someone tries to solve a problem.
Generally I don't particularly care if someone actually gets the answer. Its how they think about the problem that matters.
And some quizzes are good to just eliminate people who lie about their skills.
Hi John;
Long time, no see. Tell Jesus M. "Hi" if you still talk to him... I have to come clean and tell you I work at Google now; I was hired in there after quitting Apple.
This whole thread has been pretty bogus.
The first this that's wrong about it is that Google doesn't use brain teasers in interviews. You have to write down what you talked to the candidate about briefly so that follow-on interviewers can switch tack and not ask the same question, and then you go back to your desk and often spend as much time writing up your interview as you did actually interviewing. If you asked something stupid like "why are man hole covers round?" or some other trick question, or a riddle, you'd be scrubbed from the interview rotation until you had gone through training again.
The primary reason for a formal education is not that you have learned anything that you couldn't have learned on your own, it's so that you share a common language with your coworkers and can talk about complex topics using the same terminology, and you and they can understand each other.
The reason for giving coding problems or similar problem solving questions is to gauge how you think about solving a problem. In general, it's to tell whether or not you have critical thinking skills. The ability to engage in critical thinking is frequently a skill most people don't acquire without a logic class and/or some hard science classes in something like physics. Testing and evaluation generally require some ability in statistics, also something most people do not learn until they've had a college course in the subject.
The reason for things like linked list problems is that many people who do the minimal effort curve to graduate with a CS degree never learn anything about memory layout or how pointers work, and without understanding that, you have no hope of understanding what it is your compiler is doing to your source code, or what it's like to try to handle 200,000 transactions instead of 120,000 in the same interval, without throwing additional hardware at the problem. Yeah, you can look up how a linked list works, if you needed one, or just #include and use the macros, but if you can't write one from scratch, not only is it likely you don't know how things aren't laid out in memory, you've just spoiled the interviewers follow-on questions where you get asked to modify the algorithm to get different behaviours out of it.
The candidate qualification problem is something that was inevitable when the accreditation standards change from "Programming in C" to "Database concepts using C (please learn C on your own)", and the dearth of systems where you had to care about such things because you were working without a net. I have another younger friend who is of the calibre of our generation, Michael Steil, who is an absolute C64 fanatic. He learned what we learned, but it was because he grew up in Germany with old computers with limited memory and no memory protection to save him from himself. Just like we did.
It's an unfortunate fact in our industry that we went through a period of time often referred to as "the dot com bubble", when thousands of CS students were hired before they could really learn any of the above, just to get a warm body to fill a cubicle so that the VCs would part with the next round of funding between step 2 and ??? before "profit!" was going to happen. A lot of these people are regretting it now, or they are heading back to school to get the paper credentials that will act as their latter day union card.
So a good resume or even the right sheepskin is not as good as a demonstration of actual problem solving skills, or an ability to communicate with other people who have those skills, and with whom you are going to need to be able to collaborate effectively. There are plenty of people left over from "the dot bomb" with glowing resumes based on being cubicle warmers, and neither you nor I would hire them.
I would say it's incredibly hard to cram for an interview, any i
The false positive / false negative comment is spot on: you know that the people who pass are probably what you're looking for.
People, mostly programmers, all think they're "great", all think they're "the best" because they can crank some PHP website or some toyish Android app in Java.
But that is not how it works. If you think you're great I suggest you go competing in algorithmic competition like TopCoder and get a reality check. There you'll see you're really not as good as you think you are. I used to be a high-ranked yellow coder there: there's no way I could make it to "red". Priceless humbling experience.
The quality of a programmer is not measured by success / money. It's measured by its ability to solve problems, first and foremost.
And that's what such puzzlers do test.
Sadly we're in a business where a lot of very mediocre programmers are very vocal: all the ones posting at the dailywtf are particularly bad. They're the frustrated ones. So of course you'll find there people whining that this isn't "real coding".
It's about finding out how applicants approach problems, not about solving a real-world problem.
... but a firm grip on real-world processes is even more important. Tricky questions based in fantastic scenarios test the former, while leaving an assessment of the latter untouched.
I applied around 2005-2006 for an european location, just after getting my degree in Computer Science.
I was enthusiastic about google, I thought they really were the people I wanted to work with.
At the time I was involved with many FOSS projects, and thought of them as a natural next step.
The impact with the bureaucracy, and most importantly their hiring process was pretty bad:
I was passed from one hiring manager to the next, had multiple interviews on the phone, they also called exactly during the times of the day I asked them to avoid. I had an interview with a mountain view engineer, who posed multiple problems. I solved most of them, although I admit I was kinda slow, I was disturbed by having to keep a phone in one hand listening/talking to the guy, while trying to concentrate on the non-trivial questions, and keeping the emotions associated with interviews in check.
To note the fact that this guy never heard the term "external memory algorithm", so when I mentioned that I was given a "what the hell is that?" answer. This really surprised me and at the time I did not have the confidence to challenge my interviewer.
In the end, after this long and painful process, they gave me one of their default answers; they determined I was not good enough for them.
Now years have passed, I have worked for other companies in other domains, and earned the appreciation of colleagues and management alike for my problem-solving skills, and my rigorous software engineering work.
I recently switched companies, but have never given any thought about applying for google again.
I would now probably be able to "prepare" for their interview process better, and would be much less influenced by emotions, especially since I am much more confident. But why would I go through that kind of unpleasant interview-marathon? Why would I accept that kind of degrading experience?
The last company that hired me interviewed me over the phone a couple times first (ca 40 mins each IIRC), then invited me over for an "interview-day" (they paid the expenses), in which I was interviewed for the whole afternoon, but I was interviewed well. I could take breaks, drink coffee, and I felt I was being treated well in general. This is the kind of company I want to work for, and am working for.
LOL, as far as "star struck" ala the film "Sunset Boulevard" -> http://www.youtube.com/watch?v=dkDLI43iiTs & Gloria Swanson's character? Heh, that's not my "state-of-mind" here @ all... I just wanted to point out that MS' questions in that timeframe (early 2003) weren't anything like this article's statements (things MAY have changed though in the intervening period is all).
APK
P.S.=> Funniest part was I didn't go to they, they came to myself (was shocking) - Never had that happen before: Usually I "pound the pavement" just like anyone else (or in the case of geeks, email quite a bit)... apk
Over the course of my 17+ year career as a engineering professional in the technology I have interviewed by 150+ people and conducted 150+ interviews. In all of that interviewing (I was an independent for 7 years so that accounts for the large number of interviews) I had only a single interview where I was asked ahead of time to work on a problem and bring in a design and present that design. I found that very refreshing and engaging. Only one other interviewer asked me ahead of time to think about a particular subject and then use that subject in the interview process as a launching point into other topics. The other 148+ interviews were your basic run-of-mill question answer hand-off sorts of things with a battery of questions and then long winded speeches.
As an interviewee, only one person took the time to look through my open source projects to see actually how I code. As a coder I find this quite bizarre because coding styles and skills vary as widely as text writing skills. I always give links to my open source projects so I can present an example of my work. You should be able to see if I am off my rocker within about 5 minutes of clicking through my open source code on a website. I have published articles and presented then to interviewers but rarely are they viewed and much less read. It seems incredible to me that you would hire someone to do a job without evaluating how they actually communicate on paper or how they actually design systems and write code. Would a publisher hire a writer without reading any of the writers work and just do a quick question answer session? The only exception to this is if you show up with a book you have authored and that book was widely read. Then in that case you either are thrown into the category of "he can write a book but it sucks and he can't code" or you walk into interview with "God mode" enabled.
It is interesting to note that the single company that asked me to submit work prior to the interview was a small, creative graphics firm that does a lot of creative game and graphics development. For small companies a few bad hires can potentially damage the revenue stream beyond repair so they can't afford to be "lazy" or as robotic in the process as larger firms. But, this doesn't not always hold true in the real world.
I find that the HR-built questions tend to drive me crazy at times, mainly because some of little to no bearing on the position being interviewed, and others may not *have* an answer.
Being asked how you resolved a conflict with a co-worker when you've never had one (can't say that now, but during an interview 5-10 years ago I could as I had only worked in smaller shops full of some pretty nice people) is frustrating as heck.
But back to the technical questions. I've found that making them is fun. Rather that trying to come up with obtuse technical questions, some basics mixed with real situations the company has faced works well (what would you check in situation X). Questions that try and get you to fill in entries that would be more easily available from a man-page are lame.
In my own experience, the best part of coming up with the questions is getting back answers that I'd never thought of. Sure they don't match the solution/issue given in my own experience, but finding cool new ideas that never crossed my mind is part of what makes tech fun.
If you want to measure something, then don’t measure other shit.
..on sourceforge and one on gitorious and a proven track record of successfully finished projects. I don't do "brain teasers". If someone wants to hire me, I am a freelancer, he might check whether or not I fit into the team. But I don't jump through hoops to prove my technical skills.
We've had a terrible time finding "programmers" that could even pass simple coding tests. We'd be happy with someone who understood general coding principals that we could send to training for a few weeks, but finding even that has been a serious problem.
While I don't really ask many brain teasers when interviewing people, one key benefit that brain teasers offer is to see how candidates do when facing an unexpected, off-the-wall problem. Do they freeze? Panic? Make stuff up? Give up? Or do they start thinking it through? This is really important when the job entails facing unexpected off-the-wall problems regularly, as it does in my shop (a top-ten computer science department where weird computing is not unusual). A similarly useful technique in interviews is to hand a candidate a stack of paper, each sheet of which has a snippet of code, the output of a command, or the contents of a standard system file (some of them should be obscure, some common), and ask them to simply identify the programming or scripting language, the command, or the system file, if they know what it is. It's a really quick way to see a candidate's breadth of knowledge and experience, and also (for obscure sheets) how they react to being faced with something they've never seen before. And yes, it does sometimes lead to surreal situations, such as candidates who claim to be e.g. Java programmers but can't recognize Java when they see it.
... a while ago, we spent so much time discussing stupid programming brain teasers, that I began to wonder how the company earns money with everyone focused on such playground stuff. I decided there and then that it's not the place for me to work.
I happen to work at Google and also happen to know that "brain teasers" aren't used in the interview process for the engineers now.
Go on, apply and come. Don't be scared, interviews are quite pleasant and not confusing at all.
(Obviously, this is just me saying and not an official company statement, blah-blah-blah).
Like some of my PhD friends have told me, putting a technical quiz in front of well educated and experienced job candidate is a great way to insult them, and is deserving of a good punch in the face.
What you get from a quiz is a candidate who is intelligent enough to write a program that is plain to the interviewer. That is, it is neither a wise answer nor a clever one. It is simply an explainable one, and it is usually the explainable ones that show up in "For Dummies" books and have no practical value. You could be interviewing Einstein who would give you an answer that breaks ground in uncharted territory, but you wouldn't hire him because your mind couldn't comprehend his explanations. You could be interviewing Jesus, but his wise answers would be so over your head that you'd not hire him because you couldn't grasp how many risks were calculated in giving you those answers.
The correct geek solution is almost certainly the wrong real-world solution. Take the pirate problem, for example. There's a "correct" game theory solution, but pirates probably don't know game theory. All four will be unhappy with the "correct" solution. So, given that pirates don't know game theory and acknowledging that neither do I, there are at least three better solutions...
The greedy bastard will keep 32 and give 34 each to two of the others (because 100 can't be divided evenly, and you don't want one of the voters you hope to please getting more than the other, and you don't want to keep more than the voters). The two who don't get any don't matter since they can't vote down the plan.
The peace-making plan is to divide it evenly. 20 each.
The plan with heart, and the greatest chance of survival, even long after the voting, is to divide it evenly between the other four and keep nothing. Maybe you could even count on increased loyalty.
Of course know-it-all geeks who got this from some source without having to solve it themselves are going to be unhappy with my answer, but I'm okay with that. I don't want to work with them, anyway. Hopefully the interviewer is instead interested in workable solutions.
I have done a number of these "tests" as a practical portion of an interview. I have found in most cases it is retarded. "Here is X problem, solve it using Y language, you have 30min and a sheet of paper and a pencil, good luck".
I never code that way. It would be different if they asked me what my solution would be, or to write it in pseudocode, or how I would go about it. I just think asking someone to write perfect code, with no references, is stupid. It most cases I end up piecing together my code from various other projects I have worked on, from other sources, books, etc.... But it is all just syntax. The solution for the most part (unless for whatever reason isn't easily supported by the required language) doesn't change. Anyway I am not a "programmer", and haven't applied for a "programming" job, but most would require some ability.
Anyway most tests I have seen are silly and are really not a good test of how good a programmer you are, but more of how familiar you are with one particular language syntax, which most of you will probably agree isn't everything. I think part of the problem is this stuff is put together by HR and Management staff, who don't know any better other than one of the requirements of the job is Y.
In defense of the GP some people believe quite strongly in the Myers-Briggs test. I was at an HR training where they encouraged us to go to HR and take the test. Other people in the room indicated they had their personality type in their email signatures to help other people to interact with them better.
So like anything it's a tool that some people find useful in helping them deal better with other people.
If you're a serious "I am not a number" or "don't categorize me" person then you probably won't like it, but you may be missing out on insights about yourself or the way you deal with others.
And no, I haven't taken the test yet, I've been kind of busy and had mostly forgotten about it until now.
Yes we do brain-teasers, especially asking questions which are sometimes quite ambiguous, deliberately.
The point is not to get to the super optimal O(1) solution in a few seconds - in fact, I would question if anyone could do that withing breaking a sweat that he might have encountered the question before...but to watch how he responses, process the information, communication skill to get the requirement and question clear, explaining the train of thought on the way, all those little interactions. If writing code is needed, I would just tell them not to care too much how to make a API call because I would lookup Google^wMSDN for that too.
Getting the answer correctly is a bonus, of course you can't fail too much...but we are more focus to hire someone who can make sense, able to learn on the job, passionate about the job...hardcore skills are really not that important for entry level, I don't really care if you can or cannot prove P=NP, we are not in the research department.
As someone who has lead a software engineering division, I have a procedure - give me the guys that the others don't want. So I get another interview where I talk to the people and find out what they know, rather than what they are tested for. My division has the highest retention rate, as well as highest customer satisfaction. Why? Because I get to know the people who come in to interview and find out what they have done and marry that to what we need, and the customer. So I've become the strongest detractor from the "give them a test" crowd. The "test-them" group seems to be made made up of people that don't want to take the time to get to know the candidate.
That is my 2 cents worth. I think testing people is worthless since it causes you to miss the really good guys. But it's hard to have some non-skilled weenie in HR give a useful interview - that is whey I do the interviewing for my division. It takes a lot of time, but the results are worth it. The ONLY reason I suspect places like Google do it, is to cut down on people to actually interview and it doesn't lead to better candidates, just the ones that are more willing to jump through your hoops to work for you.
The company best known for this is Google. Past applicants tell tales of a head-spinning battery of coding problems, riddles, and brain teasers, many of which seem only tangential to the task of software development.
Been there, done that, flunked the quiz!
Sorry, Put me in front of a computer with vi (with no auto complete builtin though some configure it to generate such, but I've found it usually slows down my coding more than speeds it up given the length of perl words), with a set of perl man manpages I can invoke in a separate window,and I can spin up some random demo perl prog in no time at all...of course please be there when I ask you about all the assumptions you left unstated.
Put me in front of a whiteboard...and try to get anything legible out of me -- let alone with keywords correct.
I'd even go on to say (this would be harder to allow) to let me have my own dir of self-written sample scripts I use all the time time -- because I'll often forget the basics of how to do a relatively simple algorithm if I known it is well encapsulated in one of my scripts and I can copy it on command whenever I need it as a starting point.
I even have a template of header code that has my most commonly used 'good programming practices' debug features/settings. Start a new file, I can save myself 5 minutes up front just by copying the template and commenting out/deleting portions I don't need.
Alot of my script progs (sometimes written explicitly as libs), are similar....Algorithms and shortcuts I find useful in my development -- without which I might spend hours recreating a bug free alternative.
So -- sure strip someone of their 'tools, and their collected knowledge' (which any good CS person would store on a computer, and not try to memory -- as it constantly needs updating and After 500-600 manuals, the memory access time really drops -- not to mention, for most people, the fidelity.
Might as well ask a race car driver to demonstrate how well they drive on a chalkboard....
In case anyone is still wondering manhole covers are round because Reuleaux triangle-shaped manholes and covers are too expensive to fabricate.
Another interesting article.
http://www.fastcompany.com/1788419/what-it-takes-to-get-a-job-at-google
I am always doing that which I can not do, in order that I may learn how to do it. - Pablo Picasso