Slashdot Mirror
Programming Interview Questions Are Too Hard and Too Short (triplebyte.com)
Programming interview questions can feel unnecessarily difficult. Sometimes they actually are, a new study has found. And this isn't just because they make interviews excessively stressful. The study shows that harder programming questions actually do a worse job of predicting final outcomes than easier ones. From the study: Programming under time pressure is difficult. This is especially true during interviews. A coding exercise that would seem simple under normal circumstances somehow becomes a formidable challenge under the bright lights of an interview room. Stress hormones cloud your thinking during interviews (even though, sadly, neither fight nor flight is an effective response to a menacing programming problem). And it can almost feel like the questions are designed to be perversely difficult. I actually think this is more than just a feeling.
Interview questions are designed to be hard. Because the cost of hiring a bad engineer is so much higher than the cost of rejecting a good engineer, companies are incentivized to set a high bar. And for most companies that means asking hard questions. Intuitively this makes sense because harder questions seem like they should result in a more rigorous screening process. But intuition turns out to be a poor guide here. Our data shows that harder questions are actually less predictive than relatively easy ones. Further reading: Programmers Are Confessing Their Coding Sins To Protest a Broken Job Interview Process.
Interview questions are designed to be hard. Because the cost of hiring a bad engineer is so much higher than the cost of rejecting a good engineer, companies are incentivized to set a high bar. And for most companies that means asking hard questions. Intuitively this makes sense because harder questions seem like they should result in a more rigorous screening process. But intuition turns out to be a poor guide here. Our data shows that harder questions are actually less predictive than relatively easy ones. Further reading: Programmers Are Confessing Their Coding Sins To Protest a Broken Job Interview Process.
take this string, now reverse it, now convert it to pig latin, now reverse that.. How many internal threads are used to perform writes in a ConcurrentHashMap, now bark like a seal..
Answers are never that important to me as an interviewer unless I am verifying they possess a certain skill. I am much more interested in how conscientious a candidate is in looking for the right answer. It is predictive of how they will solve real problems when they are unsupervised later after being hired.
I shortlisted for an interview and got ambushed on my arrival. There were 19 other candidates, and we're all ushered into a small room with 20 desktop spots on a table that went around the entire room. We were handed one single sheet of paper with a coding problem we were to find a solution for. All 20 workstations were Mac Minis. None of us were told there was a programming exercise, none of us were told it was a Mac shop.
I walked out of the "interview" in disgust. Eleven others went with me. By the looks on the interviewers' face, this has happened before.
I don't mind being put to the test, but I don't like being ambushed.
-- Karma whore? You betcha. --
Small hint: sometimes the answer is not what they are looking for...stress handling, bullshit-o-meter, implementation ideas, probable likability, etc...
Only once I had an interview where I didn't get the job (as a software dev - back when I was a junior, was totally not prepared). The key is to be prepared and actually tell them you don't know and/or you don't want to bullshit them if you don't know the answer.
Answers are never that important to me as an interviewer unless I am verifying they possess a certain skill. I am much more interested in how conscientious a candidate is in looking for the right answer. It is predictive of how they will solve real problems when they are unsupervised later after being hired.
Quoted in full for emphasis.
It's not always important to have a problem answered fully or correctly in the allotted time. It is far more important to see *how* the candidate tackles the issue, what steps they take, what mental tools they bring to bear, and most important of all, knowing when to ask for more information and/or assistance (and then paying attention to what they're asking for, and the reasons they state as to why.)
I used to do this back when I dealt with just sysadmins. One of the fun bits was to chattr +i a config file, but then have them alter it as part of the testing (but not give them the permissions to fully do that).
The point wasn't to torture anyone, but to see how they handle non-ordinary problems. To watch how they tackle it, how they determine what's going on, then know whether or not to ask for help - and to see if that request for help is specific ("you'll have to change the attributes back before I can update it"), or if it was vague.
That told me more about the skills of the candidate than whether or not they successfully knocked out a problem.
Quo usque tandem abutere, Nimbus, patientia nostra?
It's pointless to ask anything other than a basic programming question during an interview. IF you insist on verifying that a candidate knows the language they claim, do that OUTSIDE the interview. You can present them with a "how do you fix this error" or give them a bit of code and ask them what it does, but any detailed questions like "Solve this problem using recursion" really doesn't tell you anything.
Personally, I ask what seems to be a programming question, but really isn't. I ask them how they'd go about writing some algorithm and then start throwing system integration tests that are failing at them, their coworkers not being available to fix something and a deadline they don't think they can meet. The point is to find out how open they are to ask for help, how readily they will report problems meeting schedule and how they choose to interact with uncooperative coworkers. To me THAT's the stuff that is more important than being able to implement a swap sort on a linked list using recursion.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
... "Talk is cheap. Show me the code."
This might not work for all developers in all situations, but one of the best ways to establish credibility as a developer is to find an Open Source project you like and contribute.
When it comes to the technical aspects of an interview, any technical interviewer worth their salt should be able to go to Github and verify that you are the person who committed the code you claim to own. Take along hard-copy, annotated if necessary, and be prepared to talk to it, to explain why it is something special, elegant, efficient, and flexible.
If you can get testimonials from project leads or other contributors, to demonstrate that you're a good team player, so much the better.
There are several advantages to this approach:-
1. The body of work that you build up stays yours and can be taken from job to job
2. You can share this well in advance of an interview, giving your interviewer time to review it and thus lead to a good discussion
Yes, someone is going to argue that this approach could result in cheating and could allow you to claim ownership of code that isn't yours. This is not a new problem for the interview process to factor - and a good technical interviewer should be able to ask piercing questions able to determine if you really did write what you claim is yours.
It's a completely [utterly] different example of credibility, but I am for some reason reminded of that scene in the original, Black-and-White movie version of the "Dam Busters" raid from the Second World War. Barnes Wallace, the inventor of the bouncing bomb, needed the use of an RAF aircraft from which to test drop his prototypes. It quickly became apparent that he needed an actual Avro Lancaster bomber, but of course, in the middle of a war, these were in short supply. His response?
"Do you suppose that if we explained to your man in the Ministry that I designed the Lancaster that he might be willing to lend me one?"
The whole reason we hold interviews in the recruitment process is to try and establish whether the candidate has the chops to actually do the job. Nothing says that better than hard, real-world evidence.
A similar situation is seen in final college exams, where students are asked to answer/solve a difficult programming question against the clock. A too frequent outcome is that few students fully answer the question, while many very poor answers have to be opaquely scaled to meet an external requirement. Not the best mechanism to separate the sheep from the goats.
I was a lot happier when I bid contracts for work and stopped interviewing for jobs.
Being an employee sucks. The tax system is set up to screw over the working schlob. If you're smart enough to program well, you're smart enough to hack the tax system too. Programming is the only profession that has anything like trivial tests in interviews. The flip side of this is without professional credentials, that's where things go.
Otherwise.. stand up and do tricks.
Unionize already. Doctors are unionized. Lawyers are unionized. They call it something else, but that's what it is..
..don't panic
I think a big problem with these questions is not if they are truly hard or difficult, but the meta thinking involved. Many of them really come down to 'which trick did the person who designed this test have in mind?' or even worse 'which niche within the language is the person who wrote this obsessed about?'.
A lecturer asks students to learn the phone book by heart.
The physics students ask: "Why?"
The medicine students ask: "Until when?"
There's a longer version, but you get the point. Apparently some of these "hard interviews" are just testing how fast you can cram irrelevant knowledge into your head every sane person would look up when they need it. That someone knows how to write down a quick sort algorithm in c with every semicolon in the right place doesn't mean they know when best to use it.
And now for something completely different:
https://www.youtube.com/watch?...
"By the way if anyone here is in advertising or marketing... kill yourself." -- Bill Hicks
I have found it useful to give them questions that they may not know.
I had a test that seemed to be rather good at judging ones ability. And it took people an average of 4 hours to complete.
Question 1. I have an HTML file with a Picture of two boxes overlapping each other, and a box with two empty Div's asking them to make the other box look like the picture. (Non Web people will need to google to learn how to use style sheets, and position properitys)
Question 2. A simple Application form, that asks for the address, validates that the format is correct, and just pops up a text box giving the errors or showing the address in a proper US mailing address format. (The zip code with leading 0's gets them every time. )
Question 3. A SQL Stored procedure, that returns a table where the rows become the column. The code works correctly, however there is a null in the data, preventing it form working correctly. (So they either will need to do something with the null, or exclude it, extra bonus points if they state how insecure that code was.)
This seemed to have allowed the company to get some good employees, but still it isn't fool proof, a few people slipped threw the cracks, mostly because they actually had experienced those particular issues before, so they knew how to handle them, but turned out they would struggle on new problems.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
I've conducted dozens of interviews over the past 10 years, and the best predictor I've found is simply to talk to candidates.
I typically ask questions like:
What's your favorite *nix OS / distribution, and why?
What was your worst day, and how did you recover?
What was your greatest success?
What's your methodology when you're asked to start a new project from scratch?
Key is to go into detail on each of the points, and listen for confidence, honesty, bullshit, and specific words. Test individual responses. Get down to syntax, command arguments, file locations, protocols... but try to keep focus on what they've done.
So far out of the 8 people I've hired/recommended, only one didn't work out.
A government is a body of people notably ungoverned - AC
That just doesn't sound like it's going to bring out the best in candidates.
The best approach that I've found to making programming questions part of the selection process was having qualified applicants do the test prior to the first interview. We selected a few standard programming algorithms (Quicksort, Dijkstra's Algorithm and Hash table create) that we either gave the candidate broken code or the prototype which needed the code. The candidates were given one week to submit the tests at which point they would be reviewed and we'd decide whether or not to grant the candidate an interview. The review was based on operational correctness (did the code produce what we expected from given inputs) and performance (generally speed).
HR (this was at RIM, about ten years ago when they would hire anybody with a CS degree and was apparently breathing), raised holy hell as it eliminated at least three quarters of the applicants without us even talking to them. Well over two thirds of the candidates never submitted the tests with most of them saying "I graduated from ... and where do you come off seeing if I can program?" and others saying, "I'm too busy in my current job to do this." with HR supporting those arguments. About 10% came back and asked if we had any other tests - these 10% always did the test we gave them successfully (and sometimes brilliantly) and these were generally hired and we never had performance issues with them.
The big problem we had was explaining to HR (sorry "Organizational Development", RIM speak) that just because you graduated with a CS degree, that did not mean you could program and that we wanted people who wanted to be challenged. When it came right down to it, they were getting hammered by their management because we were only interviewing 10-20% of their "qualified candidates" and felt that our approach made them look bad as other divisions within RIM wanted to follow our approach.
My response was "tough shit" and lead to some interesting meetings with senior VPs at RIM.
That just doesn't sound like it's going to bring out the best in candidates.
The best approach that I've found to making programming questions part of the selection process was having qualified applicants do the test prior to the first interview. We selected a few standard programming algorithms (Quicksort, Dijkstra's Algorithm and Hash table create) that we either gave the candidate broken code or the prototype which needed the code. The candidates were given one week to submit the tests at which point they would be reviewed and we'd decide whether or not to grant the candidate an interview. The review was based on operational correctness (did the code produce what we expected from given inputs) and performance (generally speed).
HR (this was at RIM, about ten years ago when they would hire anybody with a CS degree and was apparently breathing), raised holy hell as it eliminated at least three quarters of the applicants without us even talking to them. Well over two thirds of the candidates never submitted the tests with most of them saying "I graduated from ... and where do you come off seeing if I can program?" and others saying, "I'm too busy in my current job to do this." with HR supporting those arguments. About 10% came back and asked if we had any other tests - these 10% always did the test we gave them successfully (and sometimes brilliantly) and these were generally hired and we never had performance issues with them.
The big problem we had was explaining to HR (sorry "Organizational Development", RIM speak) that just because you graduated with a CS degree, that did not mean you could program and that we wanted people who wanted to be challenged. When it came right down to it, they were getting hammered by their management because we were only interviewing 10-20% of their "qualified candidates" and felt that our approach made them look bad as other divisions within RIM wanted to follow our approach.
My response was "tough shit" and lead to some interesting meetings with senior VPs at RIM.
I just rebooted and forgot to login to /. so this is repeated AC post.
Mimetics Inc. Twitter
Maybe you could talk to them, instead of trying to deceive them?
Exactly. I also prefer (longer) questions that out candidates at ease rather than put them on the spot. For example: “What problem in your career did you most enjoy solving?” Then just take it from there and go in-depth. Interviewees tend to open up when they get a chance to talk about stuff they are proud of or enjoyed doing.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
I have found it useful to give them questions that they may not know. I had a test that seemed to be rather good at judging ones ability. And it took people an average of 4 hours to complete.
Question 1. I have an HTML file with a Picture of two boxes overlapping each other, and a box with two empty Div's asking them to make the other box look like the picture. (Non Web people will need to google to learn how to use style sheets, and position properitys)
Question 2. A simple Application form, that asks for the address, validates that the format is correct, and just pops up a text box giving the errors or showing the address in a proper US mailing address format. (The zip code with leading 0's gets them every time. )
Question 3. A SQL Stored procedure, that returns a table where the rows become the column. The code works correctly, however there is a null in the data, preventing it form working correctly. (So they either will need to do something with the null, or exclude it, extra bonus points if they state how insecure that code was.)
This seemed to have allowed the company to get some good employees, but still it isn't fool proof, a few people slipped threw the cracks, mostly because they actually had experienced those particular issues before, so they knew how to handle them, but turned out they would struggle on new problems.
This is the correct outlook.
Unfortunately, there are enough "interviewers" out there looking for canned code monkeys, that it really makes an already stressful activity (job hunting) into an even worse hassle.
Problem solving is important but having a good work ethic is at least as important and most interviews do little to identify that.
During an interview I'd realised I didn't want the job, so when they sprang an unexpected programming test on me I asked them for the project plan and the business case behind it. They said it was to check how well I could program, I replied that I want to check how well they're organised and what their policies were. It went on like that for a few minutes until the interview ended.
Maybe you could talk to them, instead of trying to deceive them?
You don't have to deceive them entirely - just emulate a problem or two to see how they handle it.
Quo usque tandem abutere, Nimbus, patientia nostra?
Part of a sysadmin job is to fix the mess of others, get lied to and deal with incompetents.
Sometimes, you'll have to fix a problem, and no one will be there to help you. You are the expert, you are supposed to fix problems, not the other way around. When asked, people will invariably tell you they did nothing. If there is some problem you can't fix (ex: that "read only" attribute), you'll probably have to tell the person who can fix it exactly what to do. Either because that person doesn't know (remember, you are the expert), or because he is some kind of higher up who specifically hired you not to waste his time.
What we do is provide a list of about 6 (depends upon skill set we're hiring) problems that equate to first or second year CS studies. They are all straightforward and require no fancy manipulations. The candidate picks one. We set them up with an environment, they can access Google (keep search history intact, no Googling the actual problem).
They are given an hour to do the problem; anyone on the team can do it in fifteen minutes or less. If they're having trouble, we will give them another hour. There is a proctor present to help with non-programming issues (for example, for C++ folks we'll help them compile if they're rusty with command-line gcc).
These types of simpler exercises provide insight into some of the fundamentals: can you write code? What methods do you use to solve problems? Can someone else easily understand your code?
The number of C++ 'programmers' who can't compare two lists based upon criteria, or web programmers who (no joke) can't code a page that dynamically retrieves data using a (provided) web service, or database folks who can't join two tables is staggering. The exercise weeds out people who have spent years - decades - copy/paste/trial-error coding. Sorry kids, we have real work to do.
The company I work for has multiple divisions. We primarily build technology. In some divisions, the people who write code are, indeed, just laborers who build exactly what is provided them.
But in a couple - magical - divisions, the people writing the code understand the intent, the business, and the objectives. It's much harder to find, hire, and develop these people. It's especially bad when someone is good at one facet but not all of them. It's really difficult to scale. That's why so many places "grow" to a point where things like the business, design, architecture, etc. are divvied up. They also end up being (individually) "expensive", at least on paper.
However, if you can make it work, the employees tend to love it. There is some survivor bias - people who don't like it are amongst those who cycle out. "I'd rather just code than deal with foo." It's also hard to scale.
Honestly, there's merit in both approaches. As for personal satisfaction, it depends where you land in terms of what you truly enjoy.
WTF?! In the real world, when are you going to need to write a program to escape a burning building? If the place is such a dumpster fire that they are trying to push out critical, last minute fixes with the customer standing in the waiting room, then good God, get me away from there!!
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba