Slashdot Mirror
Programming Interview Questions Are Too Hard and Too Short (triplebyte.com)
Programming interview questions can feel unnecessarily difficult. Sometimes they actually are, a new study has found. And this isn't just because they make interviews excessively stressful. The study shows that harder programming questions actually do a worse job of predicting final outcomes than easier ones. From the study: Programming under time pressure is difficult. This is especially true during interviews. A coding exercise that would seem simple under normal circumstances somehow becomes a formidable challenge under the bright lights of an interview room. Stress hormones cloud your thinking during interviews (even though, sadly, neither fight nor flight is an effective response to a menacing programming problem). And it can almost feel like the questions are designed to be perversely difficult. I actually think this is more than just a feeling.
Interview questions are designed to be hard. Because the cost of hiring a bad engineer is so much higher than the cost of rejecting a good engineer, companies are incentivized to set a high bar. And for most companies that means asking hard questions. Intuitively this makes sense because harder questions seem like they should result in a more rigorous screening process. But intuition turns out to be a poor guide here. Our data shows that harder questions are actually less predictive than relatively easy ones. Further reading: Programmers Are Confessing Their Coding Sins To Protest a Broken Job Interview Process.
Interview questions are designed to be hard. Because the cost of hiring a bad engineer is so much higher than the cost of rejecting a good engineer, companies are incentivized to set a high bar. And for most companies that means asking hard questions. Intuitively this makes sense because harder questions seem like they should result in a more rigorous screening process. But intuition turns out to be a poor guide here. Our data shows that harder questions are actually less predictive than relatively easy ones. Further reading: Programmers Are Confessing Their Coding Sins To Protest a Broken Job Interview Process.
take this string, now reverse it, now convert it to pig latin, now reverse that.. How many internal threads are used to perform writes in a ConcurrentHashMap, now bark like a seal..
Answers are never that important to me as an interviewer unless I am verifying they possess a certain skill. I am much more interested in how conscientious a candidate is in looking for the right answer. It is predictive of how they will solve real problems when they are unsupervised later after being hired.
I shortlisted for an interview and got ambushed on my arrival. There were 19 other candidates, and we're all ushered into a small room with 20 desktop spots on a table that went around the entire room. We were handed one single sheet of paper with a coding problem we were to find a solution for. All 20 workstations were Mac Minis. None of us were told there was a programming exercise, none of us were told it was a Mac shop.
I walked out of the "interview" in disgust. Eleven others went with me. By the looks on the interviewers' face, this has happened before.
I don't mind being put to the test, but I don't like being ambushed.
-- Karma whore? You betcha. --
Small hint: sometimes the answer is not what they are looking for...stress handling, bullshit-o-meter, implementation ideas, probable likability, etc...
Only once I had an interview where I didn't get the job (as a software dev - back when I was a junior, was totally not prepared). The key is to be prepared and actually tell them you don't know and/or you don't want to bullshit them if you don't know the answer.
Answers are never that important to me as an interviewer unless I am verifying they possess a certain skill. I am much more interested in how conscientious a candidate is in looking for the right answer. It is predictive of how they will solve real problems when they are unsupervised later after being hired.
Quoted in full for emphasis.
It's not always important to have a problem answered fully or correctly in the allotted time. It is far more important to see *how* the candidate tackles the issue, what steps they take, what mental tools they bring to bear, and most important of all, knowing when to ask for more information and/or assistance (and then paying attention to what they're asking for, and the reasons they state as to why.)
I used to do this back when I dealt with just sysadmins. One of the fun bits was to chattr +i a config file, but then have them alter it as part of the testing (but not give them the permissions to fully do that).
The point wasn't to torture anyone, but to see how they handle non-ordinary problems. To watch how they tackle it, how they determine what's going on, then know whether or not to ask for help - and to see if that request for help is specific ("you'll have to change the attributes back before I can update it"), or if it was vague.
That told me more about the skills of the candidate than whether or not they successfully knocked out a problem.
Quo usque tandem abutere, Nimbus, patientia nostra?
It's pointless to ask anything other than a basic programming question during an interview. IF you insist on verifying that a candidate knows the language they claim, do that OUTSIDE the interview. You can present them with a "how do you fix this error" or give them a bit of code and ask them what it does, but any detailed questions like "Solve this problem using recursion" really doesn't tell you anything.
Personally, I ask what seems to be a programming question, but really isn't. I ask them how they'd go about writing some algorithm and then start throwing system integration tests that are failing at them, their coworkers not being available to fix something and a deadline they don't think they can meet. The point is to find out how open they are to ask for help, how readily they will report problems meeting schedule and how they choose to interact with uncooperative coworkers. To me THAT's the stuff that is more important than being able to implement a swap sort on a linked list using recursion.
"File to fit, pound to insert, paint to match" - Aircraft Maintenance 101
I do a lot of rapid fire, easier questions when I interview. A person who is full of BS will get tripped up because you have to have a working knowledge of the subject to keep talking through rapid fire answers,.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
Why do people go to college to study information systems; analysis, design, project management, requirements, QA, etc. only to be stuck in a corner to code bits and pieces of a system they really have no control over?
What that makes you is pretty much the dude on the auto assembly line screwing on brake drums. You may be very good at it. You can probably put brake drums on any car ever made. You can do it blind folded and you can tell by feel if the drum is unbalanced.
But you are still just screwing on brake drums.
But hey, we need brake drums on every car, eh?
... "Talk is cheap. Show me the code."
This might not work for all developers in all situations, but one of the best ways to establish credibility as a developer is to find an Open Source project you like and contribute.
When it comes to the technical aspects of an interview, any technical interviewer worth their salt should be able to go to Github and verify that you are the person who committed the code you claim to own. Take along hard-copy, annotated if necessary, and be prepared to talk to it, to explain why it is something special, elegant, efficient, and flexible.
If you can get testimonials from project leads or other contributors, to demonstrate that you're a good team player, so much the better.
There are several advantages to this approach:-
1. The body of work that you build up stays yours and can be taken from job to job
2. You can share this well in advance of an interview, giving your interviewer time to review it and thus lead to a good discussion
Yes, someone is going to argue that this approach could result in cheating and could allow you to claim ownership of code that isn't yours. This is not a new problem for the interview process to factor - and a good technical interviewer should be able to ask piercing questions able to determine if you really did write what you claim is yours.
It's a completely [utterly] different example of credibility, but I am for some reason reminded of that scene in the original, Black-and-White movie version of the "Dam Busters" raid from the Second World War. Barnes Wallace, the inventor of the bouncing bomb, needed the use of an RAF aircraft from which to test drop his prototypes. It quickly became apparent that he needed an actual Avro Lancaster bomber, but of course, in the middle of a war, these were in short supply. His response?
"Do you suppose that if we explained to your man in the Ministry that I designed the Lancaster that he might be willing to lend me one?"
The whole reason we hold interviews in the recruitment process is to try and establish whether the candidate has the chops to actually do the job. Nothing says that better than hard, real-world evidence.
A similar situation is seen in final college exams, where students are asked to answer/solve a difficult programming question against the clock. A too frequent outcome is that few students fully answer the question, while many very poor answers have to be opaquely scaled to meet an external requirement. Not the best mechanism to separate the sheep from the goats.
I was a lot happier when I bid contracts for work and stopped interviewing for jobs.
Being an employee sucks. The tax system is set up to screw over the working schlob. If you're smart enough to program well, you're smart enough to hack the tax system too. Programming is the only profession that has anything like trivial tests in interviews. The flip side of this is without professional credentials, that's where things go.
Otherwise.. stand up and do tricks.
Unionize already. Doctors are unionized. Lawyers are unionized. They call it something else, but that's what it is..
..don't panic
I think a big problem with these questions is not if they are truly hard or difficult, but the meta thinking involved. Many of them really come down to 'which trick did the person who designed this test have in mind?' or even worse 'which niche within the language is the person who wrote this obsessed about?'.
A lecturer asks students to learn the phone book by heart.
The physics students ask: "Why?"
The medicine students ask: "Until when?"
There's a longer version, but you get the point. Apparently some of these "hard interviews" are just testing how fast you can cram irrelevant knowledge into your head every sane person would look up when they need it. That someone knows how to write down a quick sort algorithm in c with every semicolon in the right place doesn't mean they know when best to use it.
And now for something completely different:
https://www.youtube.com/watch?...
"By the way if anyone here is in advertising or marketing... kill yourself." -- Bill Hicks
I have found it useful to give them questions that they may not know.
I had a test that seemed to be rather good at judging ones ability. And it took people an average of 4 hours to complete.
Question 1. I have an HTML file with a Picture of two boxes overlapping each other, and a box with two empty Div's asking them to make the other box look like the picture. (Non Web people will need to google to learn how to use style sheets, and position properitys)
Question 2. A simple Application form, that asks for the address, validates that the format is correct, and just pops up a text box giving the errors or showing the address in a proper US mailing address format. (The zip code with leading 0's gets them every time. )
Question 3. A SQL Stored procedure, that returns a table where the rows become the column. The code works correctly, however there is a null in the data, preventing it form working correctly. (So they either will need to do something with the null, or exclude it, extra bonus points if they state how insecure that code was.)
This seemed to have allowed the company to get some good employees, but still it isn't fool proof, a few people slipped threw the cracks, mostly because they actually had experienced those particular issues before, so they knew how to handle them, but turned out they would struggle on new problems.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
I would have thought that most employers might be interested in your ability to perform under stress as well or at least your ability to handle it? If you can't program at all when under stress, that's not going make you a very effective programmer in the real world.
Please prove P=NP.
I've conducted dozens of interviews over the past 10 years, and the best predictor I've found is simply to talk to candidates.
I typically ask questions like:
What's your favorite *nix OS / distribution, and why?
What was your worst day, and how did you recover?
What was your greatest success?
What's your methodology when you're asked to start a new project from scratch?
Key is to go into detail on each of the points, and listen for confidence, honesty, bullshit, and specific words. Test individual responses. Get down to syntax, command arguments, file locations, protocols... but try to keep focus on what they've done.
So far out of the 8 people I've hired/recommended, only one didn't work out.
A government is a body of people notably ungoverned - AC
That just doesn't sound like it's going to bring out the best in candidates.
The best approach that I've found to making programming questions part of the selection process was having qualified applicants do the test prior to the first interview. We selected a few standard programming algorithms (Quicksort, Dijkstra's Algorithm and Hash table create) that we either gave the candidate broken code or the prototype which needed the code. The candidates were given one week to submit the tests at which point they would be reviewed and we'd decide whether or not to grant the candidate an interview. The review was based on operational correctness (did the code produce what we expected from given inputs) and performance (generally speed).
HR (this was at RIM, about ten years ago when they would hire anybody with a CS degree and was apparently breathing), raised holy hell as it eliminated at least three quarters of the applicants without us even talking to them. Well over two thirds of the candidates never submitted the tests with most of them saying "I graduated from ... and where do you come off seeing if I can program?" and others saying, "I'm too busy in my current job to do this." with HR supporting those arguments. About 10% came back and asked if we had any other tests - these 10% always did the test we gave them successfully (and sometimes brilliantly) and these were generally hired and we never had performance issues with them.
The big problem we had was explaining to HR (sorry "Organizational Development", RIM speak) that just because you graduated with a CS degree, that did not mean you could program and that we wanted people who wanted to be challenged. When it came right down to it, they were getting hammered by their management because we were only interviewing 10-20% of their "qualified candidates" and felt that our approach made them look bad as other divisions within RIM wanted to follow our approach.
My response was "tough shit" and lead to some interesting meetings with senior VPs at RIM.
What came first? the egg or the chicken?
The egg.
The life cycle of a chicken begins with the egg, results in procreation to produce more eggs, and eventually the death of the organism. Ergo, a "chicken" to this argument is viewed from the point of the egg.
Applying evolution, one must define a strict categorization of what a chicken is. Once that is established, there is no "slippery slope" logical fallacy, and you can determine, in binary, what came first on the life cycle, as a chicken.
An "almost chicken" laid a mutated "almost chicken" egg, which is now classed as a chicken. Ergo, the mutated "almost chicken" egg is the world's first chicken egg, which becomes the first chicken.
q.e.d.
Science advances one funeral at a time- Max Planck
That just doesn't sound like it's going to bring out the best in candidates.
The best approach that I've found to making programming questions part of the selection process was having qualified applicants do the test prior to the first interview. We selected a few standard programming algorithms (Quicksort, Dijkstra's Algorithm and Hash table create) that we either gave the candidate broken code or the prototype which needed the code. The candidates were given one week to submit the tests at which point they would be reviewed and we'd decide whether or not to grant the candidate an interview. The review was based on operational correctness (did the code produce what we expected from given inputs) and performance (generally speed).
HR (this was at RIM, about ten years ago when they would hire anybody with a CS degree and was apparently breathing), raised holy hell as it eliminated at least three quarters of the applicants without us even talking to them. Well over two thirds of the candidates never submitted the tests with most of them saying "I graduated from ... and where do you come off seeing if I can program?" and others saying, "I'm too busy in my current job to do this." with HR supporting those arguments. About 10% came back and asked if we had any other tests - these 10% always did the test we gave them successfully (and sometimes brilliantly) and these were generally hired and we never had performance issues with them.
The big problem we had was explaining to HR (sorry "Organizational Development", RIM speak) that just because you graduated with a CS degree, that did not mean you could program and that we wanted people who wanted to be challenged. When it came right down to it, they were getting hammered by their management because we were only interviewing 10-20% of their "qualified candidates" and felt that our approach made them look bad as other divisions within RIM wanted to follow our approach.
My response was "tough shit" and lead to some interesting meetings with senior VPs at RIM.
I just rebooted and forgot to login to /. so this is repeated AC post.
Mimetics Inc. Twitter
I've talked with several programmers here in Silicon Valley, who seem to state that HR and management wants you to code a whole project for nothing.
https://www.youtube.com/c/BrendaEM
I went to an interview and was handed a programming test to implement a web based vending machine. The test also required implementing a service that mocked a credit card processor.
Pfft, no problem. Mine even handles multiple card processors.
What part of "shall not be infringed" is so hard to understand?
Maybe you could talk to them, instead of trying to deceive them?
the quiz is not the interview. the actual interview process is a more involved two-hour live call. source: i do remote interviewing for them as a side thing.
the actual interview does have live coding -- but note that it's coding using whatever tools the candidate is comfortable with, not a whiteboard. also more involved question/answer things, and more. it gets a lot of additional signal beyond what the initial quiz does.
My blog: http://www.seebs.net/log/ --- My iPhone/iPad app: http://www.seebs.net/seebsfrac/
I have found it useful to give them questions that they may not know. I had a test that seemed to be rather good at judging ones ability. And it took people an average of 4 hours to complete.
Question 1. I have an HTML file with a Picture of two boxes overlapping each other, and a box with two empty Div's asking them to make the other box look like the picture. (Non Web people will need to google to learn how to use style sheets, and position properitys)
Question 2. A simple Application form, that asks for the address, validates that the format is correct, and just pops up a text box giving the errors or showing the address in a proper US mailing address format. (The zip code with leading 0's gets them every time. )
Question 3. A SQL Stored procedure, that returns a table where the rows become the column. The code works correctly, however there is a null in the data, preventing it form working correctly. (So they either will need to do something with the null, or exclude it, extra bonus points if they state how insecure that code was.)
This seemed to have allowed the company to get some good employees, but still it isn't fool proof, a few people slipped threw the cracks, mostly because they actually had experienced those particular issues before, so they knew how to handle them, but turned out they would struggle on new problems.
This is the correct outlook.
Unfortunately, there are enough "interviewers" out there looking for canned code monkeys, that it really makes an already stressful activity (job hunting) into an even worse hassle.
In the interview process just determine if someone can talk to talk. Save the proof for the first 30-90 days so people can work under normal circumstances. Some people suck at interviews anyway. For myself, I always bomb the easy questions because they are usually nonsense the compiler will catch, minor syntax, minor implementation semantics.
Unless you are hiring entry level these things are all minor noise and what separates talent from drones is higher level thinking. You are going to end up with someone who fits the old stereotype of an Indian firm that can only implement what you tell them verbatim but not think independently otherwise. At that point you might as well be outsourcing.
Problem solving is important but having a good work ethic is at least as important and most interviews do little to identify that.
My first reaction is "Why?" to these types of questions. I work for a small company now: they have things that need to be done. Coding in large enterprises tends to be "coding in the small", with little creativity required and everything spelled out for you.
If you post it, they will read.
How about ask questions that don't have definite answers but need technical skill to answer. I find these very useful both for getting a feel of a candidates depth of knowledge as well as there ability to reason?
I'm thinking things like: So you are a C# programmer. What characteristics of C# attack you to the language AND which ones do you find most annoying? What type of problems do you feel C# is best at solving and which would you recommend at least considering another language.
( just a hint, if you can't answer those type of questions about any language you program in , or at least come up with a descent opinion that you support with valid technical reason, you can't really claim you understand anything about the language.)
it's a sign of how bad the job market is for programmers that they can pull this crap. Thing is, even if those 9 bail they can just go to Congress and get more H1-Bs ("But. But. But we can't find anyone qualified!").
Heck, the current administration just added another 20,000 PHD candidates to the H1-B cap (they sold it as a benefit to workers because they'd require more PHDs ignoring that those PHDs are in addition to the existing 65k cap. Got away with it too...).
I'm really disappointed. I didn't expect much from this current crop of politicians, but I thought at least they wouldn't increase the caps. Heck, the H2-B (low skill temp labor) caps are getting doubled.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
I check to see if they spend enough time to understand the problem, do they ask questions to make sure the specs are, I like it if they ask, "do you want a fast algo or minimum memory algo?". I ask them think out aloud, and guide them towards the solution. Then the fun starts, I change the spec, I ask them to reimplement it optimizing for speed instead of memory or vice versa. Change it from "filter out odd numbers" to "filter out numbers divisible by another set of numbers", etc etc.
Right at the beginning I tell them, the test is not on programming. The test is on "Am I able to communicate well with you, and are you able to communicate back with me?"
I hire only PhDs and Masters from US universities, coding is necessary but not sufficient for my positions.
So, yes, there are hiring managers who would happily go through the same hiring process themselves.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I have worked for decades, for companies tiny and huge. I have a github account. I write articles about cutting edge techs. You give me some abstract problem to solve that has no bearing on what I do every day. You don't need me, I don't need you.
If you post it, they will read.
During an interview I'd realised I didn't want the job, so when they sprang an unexpected programming test on me I asked them for the project plan and the business case behind it. They said it was to check how well I could program, I replied that I want to check how well they're organised and what their policies were. It went on like that for a few minutes until the interview ended.
An "almost chicken" laid a mutated "almost chicken" egg, which is now classed as a chicken. Ergo, the mutated "almost chicken" egg is the world's first chicken egg, which becomes the first chicken.
q.e.d.
Not really.
Speciation occurs within a population which is mixing its DNA, not within one individual. It's really impossible to be so genetically different from your parents that you couldn't mate with them, and successfully mating is how we categorize a species. Within the population of a species you'll find a wide variance of DNA, and as time goes on, because they're mixing that constantly, the whole species is changing. Over time, that species may change so much that it's not the species it used to be, but this isn't happening at the individual level.
Take a subset of that species and isolate it genetically (usually by geographically isolating it) and over time the genetics of the two populations can drift apart to the point where we'd call them different species.
Ring species are particularly interesting, and a good example of how blurred this line is.
Species A can mate with B and D.
Species C can mate with B and D.
But Species A can't mate with Species C.
Generally, if things can produce viable offspring, we call them the same species. So here, A and C are pretty much the same species as B and D, but not the same species as each other?
Which came first, the chicken or the egg?
Neither.
They co-developed over hundreds of thousands of years, and there are thousands of different genetic types of birds that blur the lines between "chicken" and "not chicken". The variety in chickens is rather incredible. Check out Araucanas, Malay, Silkie, Ayam Cemani, Polish, Manx Rumpy, Modern Game, or Transylvanian Naked Neck chickens. Those are all chickens. However a whole lot of those are starting to look very not-chicken, and give them some genetic isolation and a few tens of thousands of years, and they won't be chickens anymore.
Velociraptor = Distiraptor / Timeraptor
My preferred approach is to use a question that has many different variations, and to start with the easiest variations and then ramp up the difficulty/complexity based the candidate's performance. I only have the candidate write code for one variation, usually the second-hardest one that we talked about, and then ask them how they would go about testing what they implemented. This approach has many advantages.
First, it lets me see how mentally agile they are. Weaker candidates have a hard time adapting their approach as the problem changes, and have a tendency to get locked into early solutions. Better candidates can see how the later variations offer more/different options/tradeoffs and can adapt.
Second, it lets me see how they think through easier variations, to get that strong "process" signal TFA mentions. This gives me good insight into how they approach problem-solving tasks. In some cases it also gives me a clue as to their honesty/integrity, because it's pretty clear when someone has already seen a simple variation of the problem and doesn't admit to it (during my intro I point out that if the candidate has seen the question before, they should let me know).
Third, it gives all but the very weakest candidates some early success, which helps to lower their stress level. There's a limit to how much I can do in that regard, because interviews are inherently stressful situations, but I think it does help... and I think people perform better when they're less stressed.
Fourth, it lets me tune the difficulty of the interview to the candidate's ability level. This is for the candidate's benefit, not to give me any information. Since I'm just one in a slate of interviews, unless I'm the last one the allotted time is fixed and there's no practical way to just cut the interview short (unless they're so bad that I feel I need to end the day entirely to avoid wasting everyone's time; this is very rare). This means that if the candidate is weak and I give them a hard problem, they end up spending the whole hour struggling vainly. This makes for an unpleasant hour for them, an unpleasant hour for me, and it increases their stress level for the next interview. If the candidate finds the first variations of the problem challenging, I just stay at the easy level.
This difficulty-adaptation happens very naturally, because I don't move on to a harder variation until they've solved the previous, easier, one, with whatever hints/tips I need to provide. I watch the clock to decide when to have them change gears from discussing increasingly-harder variations to writing code. On occasion I underestimate the amount of time it will take them to write code and have to tell them to stop before they finish. This happens most often with PhDs, actually, who are often bright and good problem-solvers, but don't have much coding experience -- often even after years in industry. On those occasions, I apologize for not allowing enough time, though I know it's still pretty painful for them not to be able to finish. Whether or not they finish doesn't affect my evaluation much. I get most of what I want to know about their coding skills in the first few minutes of watching them.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Definitely agree about the easy questions. When candidates can't answer easy questions correctly, that provides a strong indication that the candidate is a bad hire.
When candidates can't answer hard questions, what does that tell you? They need extra time to solve difficult problems? Who doesn’t?
Maybe you could talk to them, instead of trying to deceive them?
You don't have to deceive them entirely - just emulate a problem or two to see how they handle it.
Quo usque tandem abutere, Nimbus, patientia nostra?
When I interview candidates, I usually give one problem that's designed to measure how much of their chosen language they know. While I don't test for knowing language trivia, I do look for signs if they prepared for the interview. I tell them upfront that if they can't remember the arguments to a function to please ask, but if a pattern of unpreparedness appears, that's a minus.
I've had candidates that have outright refused to code. They'll sometimes even write pseudo code that describes a solution, but when I try to get them to translate it into their programming language, they refuse.
Others try to show off their coding prowess and usually fuck it all up in the process by increasing the complexity. I tell the candidates upfront to KEEP IT SIMPLE.
I don't require a complete solution for a hire recommendation, but there are a couple of key milestones they need to make.
I want them to show they can program, and can put a little thought into different approaches. I try to give hints along the way if they get stuck, or head off into the weeds. Giving hints is important, because it doesn't do anyone any good if the candidate is just standing there. If the candidate can take a subtle hint and incorporate it, that's a plus, because I also want to see how they work in a group or team. I've actually had candidates who thought I was lying to them when I've tried to give hints.
Anyway, the point is, in today's world of global collaborators, you can't just test for the candidate's knowledge of programming trivia, you've got to know if they can work as a group and learn from others. Being able to write working code is important, but so is teamwork.
FWIW, I recommend Hire about 3% of the time.
What. The. Fuck. Seriously, a "programming test" should be to prove that you can program, right then and there, in a few minutes on a whiteboard, not to design a finished system solo, over multiple unpaid hours. That right there causes the company to fail the interview, though I would have balked before wasting four hours of my time. And the question the devs were scratching their heads about the week before should also not be used as a serious interview question.
#naabhaprzrag, #sverubfr-000, #agi-fcbafberq, negvpyr[pynff*=' negvpyr-ary-'] { qvfcynl: abar !vzcbegnag; }
I used to love the idea of programming-intensive interviews, as a way to objectively assess my most-relevant-to-the-job skills. After doing quite a few them, I realised about my mistake. A theoretically excellent proceeding becomes really bad when being terribly mismanaged, what is a surprisingly common scenario in the IT/programming world.
I never had to solve a really difficult problem, but lots of them under unreasonably short time constraints (i.e., the holy grail of arbitrary, cheating-prone, useless-parrot filters). Now, I prefer to avoid these situations, unless when knowledgeable people and reasonable conditions are involved.
Custom Solvers 2.0 = Alvaro Carballo Garcia = varocarbas.
Part of a sysadmin job is to fix the mess of others, get lied to and deal with incompetents.
Sometimes, you'll have to fix a problem, and no one will be there to help you. You are the expert, you are supposed to fix problems, not the other way around. When asked, people will invariably tell you they did nothing. If there is some problem you can't fix (ex: that "read only" attribute), you'll probably have to tell the person who can fix it exactly what to do. Either because that person doesn't know (remember, you are the expert), or because he is some kind of higher up who specifically hired you not to waste his time.
Its about quality and consistency, same as any other engineering profession. It is were seldom about solving a tricky problem. Its about producing high and even quality that is predictable and stable. The algorithms you get from a mathematician, they are the best at it anyway,
I've had my share of whiteboard interviews and in general I've found they fall into two categories.
1 - A basic scenario to show you know the broad outlines of what needs to be done.
2 - Some esoteric problem that requires knowledge of obscure functions and is a pet project of an interviewer.
The first kind is useful. It lets them know how you think and that you know what's going on. The second one is a dumpster fire you're never going to answer to their satisfaction so you might as well thank them for their consideration and walk out the door.
My personal favorite interview story about the second was actually a logic question. The problem itself was flawed in that they didn't give a critical piece of information necessary to solve it. There was an assumption that was quite literally wrong. When they smugly told me how I should have solved it I pointed out that error. Needless to say I didn't get the job. Also needless to say I told everyone I knew about their lack of proper requirements for problems and to avoid them as an employer.
A good talker can use up the interview time on subjects they know well. Then you don't know their weaknesses. Skilled interviewers can deal with that situation, but how many people were hired for their skills interviewing job applicants? It's a complete crap shoot whether anyone conducting an interview is a skilled interviewer. Different types of interviewers choose different approaches based on what works for them.
Wish I had some mod points for you today, AC.
Hey, Windows users, there is no such thing as "forward" slash, there is only slash and backslash.
... if they are hired on the basis of their ability to memorize stuff that can be looked up in two seconds.
A strong team can teach newcomers the right skills. It is more important to find the right personality/fit for the team.
The fact that you made the interview, likely means you have the skill.
I can teach you to code. I can't teach you to not be an *sshole.
What we do is provide a list of about 6 (depends upon skill set we're hiring) problems that equate to first or second year CS studies. They are all straightforward and require no fancy manipulations. The candidate picks one. We set them up with an environment, they can access Google (keep search history intact, no Googling the actual problem).
They are given an hour to do the problem; anyone on the team can do it in fifteen minutes or less. If they're having trouble, we will give them another hour. There is a proctor present to help with non-programming issues (for example, for C++ folks we'll help them compile if they're rusty with command-line gcc).
These types of simpler exercises provide insight into some of the fundamentals: can you write code? What methods do you use to solve problems? Can someone else easily understand your code?
The number of C++ 'programmers' who can't compare two lists based upon criteria, or web programmers who (no joke) can't code a page that dynamically retrieves data using a (provided) web service, or database folks who can't join two tables is staggering. The exercise weeds out people who have spent years - decades - copy/paste/trial-error coding. Sorry kids, we have real work to do.
take this string, now reverse it, now convert it to pig latin, now reverse that.. How many internal threads are used to perform writes in a ConcurrentHashMap, now bark like a seal..
When I interviewed with them in the early '90s, they gave a "programming" test that wasn't too far off from that. It was the most ridiculous thing I have ever dealt with. Converting from one pseudo language to another and then look up things on a chart and then convert back to a pseudo language.
WTF?!
I got fucking migraine about 2/3rds the way through and had to stop before I puked. It was just nonsense busy work. I got an interview afterwards by what would have been my direct report and he didn't look like he slept in a week.
Get this, they were a goddamn Microsoft Visual Basic shop. And they later got hacked so bad that everything they had was taken. So don't tell me that those tests get you the best.
The company I work for has multiple divisions. We primarily build technology. In some divisions, the people who write code are, indeed, just laborers who build exactly what is provided them.
But in a couple - magical - divisions, the people writing the code understand the intent, the business, and the objectives. It's much harder to find, hire, and develop these people. It's especially bad when someone is good at one facet but not all of them. It's really difficult to scale. That's why so many places "grow" to a point where things like the business, design, architecture, etc. are divvied up. They also end up being (individually) "expensive", at least on paper.
However, if you can make it work, the employees tend to love it. There is some survivor bias - people who don't like it are amongst those who cycle out. "I'd rather just code than deal with foo." It's also hard to scale.
Honestly, there's merit in both approaches. As for personal satisfaction, it depends where you land in terms of what you truly enjoy.
I took the TripleByte pre-quiz. They've been hounding me for my 'skills' in development being a great fit for a company that is hiring. The quiz said I was a 'top' candidate.
I've never been a developer so I don't know why their analysis suggests I'd be a good coder. I don't trust their recruiting methods.
Yep, I never spell check.
More incorrect spellings can be found he
I need to post this at top level, but your post inspired this thought: one of the inherent problems with the process (that you're caught up in) is that I'm looking for a full-time job, where I'll likely have weeks or months or on-going work on a project. The typical HR process (tests, interviews, etc.) is a crap-shoot because I might happen to have the knowledge, luck, and work well under the undue pressure of a job interview, but maybe I don't at that time, but maybe I would be the best for the job if someone could evaluate my potential.
Sometimes I can (and do) solve complex problems very quickly and people think I'm a genius, but it all depends on A) the amount of knowledge I have on the topic and can remember when needed for processing, and B) some kind of luck / magic.
Otherwise it involves research, which could take hours, days, weeks, which nobody has during an interview.
And I'm not just philosophizing / theorizing as so many do; this is from direct experience, and observation of very wrong people getting hired and becoming coworkers.
Very flawed process. Wrong people get hired. Square pegs in round holes.
BTW, yes, I've hired people and/or taken part in the decision process. I administered very simple tests. In one notable case I hired the 2 guys who would have failed every current HR checklist item. By HR rules, they were the least qualified for the jobs (one hid his almost complete Cornell BSEE). But they were the only two who got all the test questions right. Both were awesome productive. The other one went to night school for a BSEE and graduated Summa Cum Laude.
My point: HR are generally (I wrote "generally") egotistical control-freaks and should not be allowed to make hiring decisions for technical workers, unless you have actual technical people who also wear HR hats part-time.
Instead of a "programming" question, I often want to determine how they think. I can hire a code monkey anyday. Those positions are generally my college students and recent graduates. After I hear how they solved a particular issue and whether the candidate can relate to biological life forms, I ask a logic question. First to see how they handle the pressure and second to see what they come up with. I rarely get a "correct" answer but the process the candidate goes through is always telling. Here's one - "You just bought a hammer and a nail. Together, the hammer and nail cost $1.10. The hammer is one dollar more than the nail. How much was the nail?"
Another, "why are manhole covers round?"
The Kai's Semi-Updated Website Thingy
The only time I've had to hire someone, I told the local college and universities that I was hiring. Got a boatload of applicants, and after the first couple of interviews it was obvious that these guys hadn't much programming experience at all. So in order to weed out candidates I made a test, of some terrible code. Basically pointer arithmetic and weirdness. Most recruits gave up after a minute. One guy worked through it. He got the wrong answer but I could tell by the way he was working on the problem, he understood how things worked, and was methodical.
He got the job of course, and turned out to be a fantastic employee.
So I would submit that the questions aren't being asked to show what you know, but how you work through the problem.
It's easy to marry, difficult and expensive to divorce. Maybe we're solving the wrong problem :-)
Hiring good people is a common problem that good answers are difficult to come by. I was trained once to (basically) ask "5 whys" whenever somebody said they could do something --- in order to determine how much THEY did vs they were present when somebody else did the work. Each level down was intended to determine whether this person what making crap up or had really worked on the task.
I also knew a guy who gave logic/thought questions - he wanted to see if the person could think. He later recanted his line of attack as being useless as judging skill level. He'd ask "how many quarters stacked does it take to be as tall as the Empire State building?" To which he expected questions such as "on their side, end to end, how wide is a quarter, how tall is the building?" Did they seek requirements clarification and how did they solve the puzzle?! (its a problem in estimation, there is no exact answer--the goal was to understand how they arrived at it). But many people could solve it and still not write working code.
But I've learned recently that asking somebody to write a program in the language they claim to know is important. Why? Because there are a lot of script people who cut/paste and ferry modifications/customization and have a cursory understanding of Coding, Logic, and Problem solving. Just because you can copy somebody else's code and make it compile does not mean you can code. You know Java? great, so do I. Write a simple program that solves something domain specific (don't worry - I'm not here to see if know J2E or ask trick questions). C#? even better, that's what we use here.. Write a simple program....
If you have the skills - we'll train you and help you grow beyond where you're at today. No trick questions. You know Java and we need C#? Fine - want to learn C# --- we'll train you.
What they don't know is that I'm interviewing them, too.
Does the manager know what he is asking when he reads a question and then has a conversation about it, or is she looking for me to parrot whatever is written down? (Heh! I don't know their gender)
Are they trying me to get me to do the job as part of an interview? I have literally had the question, "Set up a framework to automate the testing of http://www.xyz.com/".
Are they looking to show me how much they know and how insignificant I am? Usually, this takes the form of, "Can you answer this obscure comp sci question that you'll never actually ever use?" or "Do you know this goofy trick that uses an obscure language feature in a non-standard way to solve an even more obscure problem?"
If I get a hint of any of that, I walk out and tell the interviewer that their hiring process is stupid. I've been around enough to know that working there will suck anyway. The interview should be a discussion about what is on my resume, to confirm that I did what I claimed. I wrote the damn resume so that they could decide if my experience matched up with what they need. If my experience doesn't fit the job, why did they call me in to test my "test taking" capabilities? Because, they're numb-nuts that don't know what they're doing, and I don't want to work with numb-nuts.
The best interview question is the one you ask at the very beginning. "Which part of my resume makes you think that I am a good candidate for this job?"
Aah, change is good. -- Rafiki
Yeah, but it ain't easy. -- Simba
You DO realize that dinosaurs laid eggs millions of years before chickens, right?
I've given one question that totally stumped the two people I gave it two before I stopped asking it. One was in a total panic and visibly sweating over it. All I wanted to see was how they approached the problem, but I think candidates really think that they must have the right answer (and I have often had interviewees ask me if they got the right answer or what the right answer was).
The thing was, at my previous job I was asking all sorts of hard questions and having them answered. Not super hard to be sure, but enough that you knew some thinking was going on. After that though my standards have declined because so many candidates were just awful. I suspect a problem with recruiting and HR screening and not being in as glamorous a field.
What surprises me that I will apologize for a question in advance saying "I know this is simple but I ask it of everyone" and then the candidates stumble and flail around. And the candidates are NOT newly graduated students, but people who claim to have a decade or more of embedded systems experience looking for a senior position and yet they can't answer the simplest questions. Ie, how to clear a bit in a word with C, it's something you do all the damn time in small embedded systems and even if you haven't done it you can figure it out quickly with basic binary logic. My standards at this point are that I won't have to be a remedial tutor for the candidate.
The thing is, you MUST ask the programming questions because most of the resumes are exaggerations or outright lies. People cannot program merely by going to google every time there's a new bug or task. I see candidates who claim big things on the resumes, citing what great project they worked on but in actuality they just fixed bugs in the logging mechanism.
HR understands this technical stuff is worthless, nerd talk.
In their lofty awareness, they would like you to transcend hiring laws and just ask this single, simple question:
Are you 40 or older?
Elegant, free from leadership reproach, and so helpful.
CS students please don't let this deter you from pursing this career path because their job is already so very hard.
Next, HR would like to tell you about how force.com can make your job a lot easier.
This is a part of the job. It makes sense to see if the candidate can handle the job. There are no jobs available where you can get by just programming Fizz-Buzz every day. If the candidate can't do the hard stuff then the candidate is not worth hiring.
In other words, find out if the candidate can pull their own weight. Sure, relax the standards for entry level hires, increase them for junior level, and be very picky for senior hires.
I've never been a developer so I don't know why their analysis suggests I'd be a good coder.
You don't necessarily have to be a software engineer to be a decent coder - it's not as hard as some make it out to be. The lead in my group working on embedded stuff is a mechanical engineer, but he's a damned fine coder too, even though that's outside his "official" expertise.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
The code monkey is generally the entry level job. You work your way up beyond that. But even at some senior levels you are still doing coding, especially if the team or company is not large. It's rare to just drop big boxes in an architecture diagram and then let junior people do the programming from there.
(possibly GUI is weird in this regard from listening to others in that many companies have a UX designer who never touches a computer and yet has strongly held views on how things must be done)
But if you don't like programming at all, then go into product or project management instead. Not wanting to code in a computing system job is like saying you don't like to do math when your job is a physicist.
So, there's an old joke about how the trustees of a university want to find out if the professors know their stuff. They come up with the question "What's 2 plus 2?" as the test. First, they go to the math department and ask the question. The professors say, "Oh, that's easy. It's four." Then they go to the physics department. Those professors say, "Oh, it's 4.0000000 with an uncertainty of another decimal place." Then they go to the college of engineering and ask those professors. They say, "Just a minute, let me get my handbook."
Point being, that trying to program from memory is a waste of time. Trying to write code quickly is also a waste of time. One should be thinking about the approach and the logic rather than regurgitating the details of some function call. It's entirely possible that there are multiple solutions to the same problem.
The fourth part of the joke is when they go over to the business school's accounting professors and ask, "What's 2 plus 2?" The professor looks around to make sure nobody can hear and whispers, "What do you want it to be?"
Often the very useful people get turned into management. Sometimes they can still program but it's such a time sync that they spend less and less time doing this. I was promoted to management and I honestly feel like I am so much less useful now, I'd much rather be programming as a team lead or mentor and let someone else do the bureaucratic stuff and attend the useless meetings.
No, it'll be outsourced overseas, H1Bs are too expensive.
Feel through an assessment task because I couldn't come up with a viable Fibonacci generator and some services built around it fast enough and the one I finally managed to tape together was too clunky - which I was perfectly aware of. Having programmed for 34 years I felt quite silly and discovered that these questions are just as stupid as they were 20 years ago.
The other interview was a sweepingly broad question about how I would build a Microservice architecture Webshop by some 28 year old nerdy IT lead douche. Of course my reply didn't satisfy him.
These questions get asked by three types of people:
1) frustrated nerds in lead positions trying to prove to their own camp how smart they are and how dumb all potential replacements are
2) people who are seeking a performant and obedient fall-guy for a project about to blow up
3) interviewers who have no idea what they are doing and probably have a general governance problem either already in place or slowly brewing
The truth is this: Anyone who thinks they can assess a senior or even medium position with a few questions or a two hour progging assessment is on crack. I might be able to do that after 4 weeks. And I know a thing or two about tech team management.
I'd walk out of an assessment should someone ask one of me again without warning.
We suffer more in our imagination than in reality. - Seneca
Where I work, (and I got the job at 40+, now in my 50's) we have an entirely different hiring policy and different techniques from almost every post above here. I read almost ALL of the above input, and I was not going to give my opinion in here, but after reading it all - I was wondering where all of that experience came from.
So this is what we do:
We don't care about your age, gender, preferences, race, religion, political beliefs, school background, grades etc. We just care about a few simple principles.
- Are you afraid of learning new things?
- Are you entirely honest? (Our hiring staff is VERY skilled at spotting someone who's not entirely honest).
- How do you fit in socially with others?
- Can you jump into something that seem impossible to solve, and admit you just can't do it, but will be humble enough to ask for help from your colleagues?
- Can you take responsibility and learn from your shortcomings and appreciate the talents of others? Admit your mistakes, and not be ashamed of it?
We don't care for the worlds fanciest resumé, we don't care about HR companies and hiring staff, we're the ones you're going to work next to every day of your life, we simply want to know WHOM we're sitting next to. You - as a person, not what you know or whom you knew.
We also use something called "mentoring technique", this means that someone in our team gets appointed to be your mentor during your hire here, our staff is you in the future, so we hope you adapt our mentality, not to point fingers at you and blame. We don't run the blame-culture here, we teach ourselves and learn from each other, very much self driven - and we're one of the biggest companies in the world.
This approach is immensely successful, because it doesn't matter to us what you knew when you came in, what matters to us is your willingness to adapt, drive yourself, your endless curiosity, your genuine interest in making life better for others (our end users), and see your colleagues as assets rather than the competition. Everyone work at their own pace, no one is whipped into reaching certain goals, they set their OWN goals.
This kind of freedom, makes people REALLY want to know what they're doing, because they take pride in having a team with this mentality, and no pressure from anyone encourages positivity and genuine interest in the craft they work with.
When I got the job, I was literally thinking, they must be joking - I did tell them the truth, I'm not that knowledgeable, I know SO many that I can't even compare myself to 1% of their skills, and I make mistakes ALL the time, but I OWN my work and my mistakes, and I'm never afraid to admit I was wrong, neither does my colleagues. We're open minded, totally ready for change - because our everyday consist basically of improvements and total change in some areas anyway, so we're used to that by now. We literally EAT re-organization for breakfast. But my colleagues have ONE thing in common, their passion for the company and their awesome colleagues.
I can't tell you where I work for obvious reasons, but right now - they're even expanding their staff of brand new coders, one of them is 55+ and have literally no coding experience. I came in too, knowing very little - years later, I've been told by my manager that I'm in the top 10 range, which I still find so hard to believe, but I take what I can get.
What I'm trying to tell you out there, with this wall of text above - is to just go for it if you dream of it, it's not impossible. Sure - it'll be DAMN hard, but challenges are fun, and that's probably why you would be interested in coding anyway, right? Age means nothing, attitude and passion is everything.
What this world is coming to - is for you and me to decide.
When I interview I start with a question that is relatively easy that anyone who can code could solve. Then follow that up with a question that is more difficult that builds on the previous answer.
For example, I would ask a candidate to write recursive Fibonacci function (I would explain what Fibonacci numbers are). If they are able to do that, then I would ask them what would happen if their function was called like this: "fib(10000);".
...richie - It is a good day to code.
Way back in the dark ages, pre Reagan, I took the Armed Forces Vocational Aptitude Battery tests. I had no intention of being in the military at all, but it was another of those periodic standardized tests that all of the students are expected to take in high school. Except it was a bit easier than most tests. Apparently I had a perfect or near perfect score on it. After that I got so many phone calls and mails from military recruiters who were all baffled that I didn't want to join, and the wandered why I bothered taking the test.
But the Mac is Unix, whereas Cygwin is emulated Unix that runs noticeably slower than you want it to. Just don't bother with xcode or other built in tools since they've morphed into an iOS development system.
Interesting. I work for the largest corporation in America and I'm building and designing new systems far more than when I was at a startup. They worship me. Six months in and I'm juggling three new designs with hands on work. It's a blast.
Yep. When people refuse or avoid code they begin to smell bad. I wonder what the fuck they think our industry is. It is pure automation. I swear everyone is confused.
Mechanical engineer, "damn fine coder," just not buying it. Perhaps our definition of fine code is different.
Try being nice to them while fixing thier shit you autistic, ADD clueless bitch. Amazing how everything falls into place as you begin to make a difference.
The fish.
You DO realize that chickens are dinosaurs right? Try raising a few. Just don't buy them clothes or school books. It's a waste.
I think the article is bullshit and their methodology sucks, but Triple byte is a convenient way to screen a lot of companies quickly if you're looking for a new job. (Along those lines, companies like LinkedIn are offering $250k total compensation to new hires these days, although not entry level).
"First they came for the slanderers and i said nothing."
I give each of my candidates a simple test, something like:
- Write a command line program to accept a list of items as arguments.
- Divide the arguments into types (dates, words, numbers, etc.)
- Sort each list in an order appropriate for its type.
- Eliminate duplicates from each list.
- Display the results on the console.
I give them 2 days to complete and turn in their answers.
I'm looking for:
- Are all requirements met?
- Does it actually run?
- What approach was used?
- How simple is the code? (I'm looking for simple solutions, few lines of code.)
- How readable is the code?
- How well-structured is the code?
- Are there good unit tests included?
This eliminates about 75% of candidates. Those who pass, I bring them in and discuss how they chose their approach, among other things.
I've been shocked how few candidates can complete the test!
I'll qualify it - he's written embedded code for a number of commercial and military platforms (some of which now live in low earth and geosynchronous orbit) competently enough to to survive the required formal verification and review at multiple levels before approval/implementation. He'll freely tell you that he's not as good as a lot of the dedicated SWEs at the company, but his record and performance reviews tend to argue against that statement.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
The key to finding qualified people is asking simple questions about tasks developers do every day such as conditional statements, code check-in, and HTML/CSS. Who writes a sort every day? This has worked perfectly for me all but one time. Ironically, that person had a double Masters degree. They couldn't get the simplest tasks done.
But attitude is far more important than coding prowess. Flawed personalities kill teams.
The guy hiding his Cornell degree probably exfiltrated all of your company's IP and sold it to China.
Smart and dishonest is a dangerous mix.
It was long ago in a galaxy far far away. And the guy hadn't finished the degree. His wife completed her PhD and got a high-paying job far from Cornell and they had a kid. I don't know the whole story, didn't and don't care. He wasn't really being dishonest. It was a somewhat low-paying tech job and the even partial BS would likely have disqualified him. He needed a job, passed the fairly simple quiz, and ended up being an awesome employee for at least 7 years.
And I'm holding back laughter thinking about any kind of IP at that company. But a Japanese company did what the Chins do now- copied one of the products, but I thought that was silly because it was an inefficient design, and the market was small, and they likely lost money on it.
I went to an interview and was handed a programming test to implement a web based vending machine. After 4 hours of programming I went to the interviewer and said I'm not going to finish this today
I went to an interview and was handed a programming test to implement a web based car sales site, including commands, history, payment, invoicing, stock, rental, repairs schedule mgt... After a year of programming I was told it's ok.
Was well paid, though.
Slashdot, fix the reply notifications... You won't get away with it...