Slashdot Mirror
Google Respins Its Hiring Process For World Class Employees
An anonymous reader writes "Maybe you've been intrigued about working at Google (video), but unfortunately you slept through some of those economics classes way back in college. And you wouldn't know how to begin figuring out how many fish there are in the Great Lakes. Relax; Google has decided that GPAs and test scores are pretty much useless for evaluating candidates, except (as a weak indicator) for fresh college graduates. And they've apparently retired brain teasers as an interview screening device (though that's up for debate). SVP Laszlo Beck admitted to the New York Times that an internal evaluation of the effectiveness of its interview process produced sobering results: 'We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It's a complete random mess.' This sounds similar to criticism of Google's hiring process occasionally levied by outsiders. Beck says Google also isn't convinced of the efficacy of big data in judging the merits of employees either for individual contributor or leadership roles, although they haven't given up on it either."
This has led TechCrunch to declare that the technical interview will soon be dead.
It was and has been a PR move for all along, with people praising all that HR innovation and crap, in the end? It's all bullshit and no one has the slightest idea of what they are doing, would like to rub this one on the face of some writers who can only spit google this, google that, look it's so much innovation science!
Google has decided that GPAs and test scores are pretty much useless for evaluating candidates...
Doesn't this lead one to believe in grade inflation at universities? If everyone scores from 3.7 to 3.98, how do you tease apart who really did well.
Almost none of the questions I've seen have provided enough information to get past the "it depends" stage. That they make candidates make wild-assed-guesses and then try to justify them is possibly a good way to test for poor managerial qualities, but the answers never have the level of explanation that the real life answers have. The days when a back-of-the envelope calculation is enough are long gone (and probably never existed int he real world anyway). So it's good to see a major employer rejecting them. Shame it didn't happen 20 years ago/
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
also get rid of testes the people who are good at test cramming can master.
I truly hope you did not mean what you wrote.
Faster! Faster! Faster would be better!
"Have you ever built something that worked, show me, explain it." IMHO that is key to successfully hiring developers.
Equally important, and admittedly a little strange to some, it to ask about their personal programming projects. Nothing work related, nothing school related, just things that they sat down and programmed motivated by their own personal needs or curiosity. If a person can not offer "something" a warning bell is going off. I don't care how small, trivial, silly, etc the personal project is. I mostly want to see that personal projects exist. To me they are an indicator that the interviewee is someone who has a genuine interest in programming, that they are not merely someone who got a degree because a parent or guidance counselor told them it was a good career path.
But not because Google went about it wrong and screwed up its hiring process.
I've been now through a few hiring processes, have sat on Interviews, decision committees. And while I like to think that my Interviews and candidate ratings were spot-on (I correctly predicted one failure and one early resignation), I'm pretty sure that's just skewed by the small sample size. What I do know is that I went through all kinds of approaches, both as an interviewer and an interviewee. I've done brainteasers, role-playing, decision explanations, code walkthroughs, resume deep-dives, online candidate research, just shooting the breeze, and more. And I haven't found a single thing that strongly correlates with acing the interview or hiring a good worker. Resumes can lie (sometimes subtly), and you'll never find out without hiring a private investigator. Role-playing can confuse people, especially if they're trying to figure out what you're looking for. Brain teasers can be memorized, shooting the breeze can lead to unreasonable judgments (positive or negative), interviewers and interviewees can have a bad day, the other person doesn't like your first name, and a million other things.
Especially when you start talking 10s of thousands of interviews, you're actually looking at so much data, so many influencing variables that I doubt you can find one common variable that stands out from the rest. What I'm concerned about (and that comes partially from being married to someone in HR) is that there is still a drive to find the one process that will automate the hiring process. As far as I can tell, it doesn't exist. Well, let me walk that back a tiny bit: there's one thing that will work better than anything else: have the interview done by the best people you have, have them take it seriously, and spend some time on it. But it takes time, is fuzzy, and is entirely reliant on managers knowing who their best people are.
I'm glad to see that Google doesn't think Big Data is the answer to everything. I just hope that this percolates through to the rest of the HR universe. There's much too much of a drive to automate hiring, like performance reviews and firing has been.
Those who can, do. Those who can't, sue.
I have walked out of job interviews where they asked nothing but puzzles. I solve technical challenges and write code. If you really have trouble determining how many toasters you can use to cook 50 pancakes, guess what. I am not the right person for you. If you are looking for someone who can code their ass off, I am the right person.
I have interviewed hundreds of candidates over the years. I have been the hiring manager a few times and never once pulled the puzzle bullshit. I have found the best indicator in the world is to just casually bullshit about technology. You can very quickly find someones strengths, weaknesses and if they are full of it. In a casual chat, people let their guard down and you get a look in.
until (succeed) try { again(); }
As in, you had to go through a day long gauntlet of interviews asking irrelevant questions to get the gig. Surprise, they didn't get the best candidates that way!
I like TechCrunch's suggestions, as they closely mirror what the Google HR guy is implying, except for one thing:
"Finally, if they’ve gotten this far, give them an audition project. Something relatively bite-sized, self-contained, and off-critical-path, but a real project, one that will actually ship if successful."
It isn't as if I couldn't be fired on the spot in the first 3 to 6 months at any permanent job- there is this thing called being a new hire. If I had someone tell me they were going to provisionally hire me and rate my progress based on a project, fine. If they told me I would be a temp until the work is completed, I would then inform them that they will need to pay me at my contract rate until I am perm- otherwise, they are just getting me at a lower rate for contract work, and that is sketchy behavior at best.
I've just gone through interviews at Google and Apple.
At Google, I was asked mainly theoretical questions - big-O, maths/stats, etc. And one "real" architecture/design question at the end. There were 5 interviewers and maybe 7 questions, sometimes 2 per interviewer but usually just 1 that lasted the whole hour. According to my recruiter before the decision, it was maybe 50/50 that I'd get an offer, and I did very well on the real-system design question (by inference, not so well on the others :). I didn't get the job.
At Apple, I had a seven-hour interview with seven interviewers. There were many many questions, far too many to easily remember categories, but they were all focussed on things I might end up doing, or problems that I might end up encountering. I got the job. I guess I do better with "real world" issues than the "consider two sets of numbers, one is ... the other is ...) type.
I have the self-confidence^W^W arrogance to believe I'm an asset to pretty much any company out there, but interview processes are nothing more than a gamble. Sure you can weed out the obvious under-qualified applicants, but frankly (unless the candidate is lying, and in the US that's a real no-no, in the UK padding your CV seems to be sort of expected...) that sort of candidate ought to have been pre-vetoed by the recruiter before getting to the interview.
I've yet to see the interview that guarantees a good candidate will do well. It's all about preparation: can you implement quicksort or mergesort right now, without looking it up ? The algorithm takes about 20 lines of code... Some interviews will require you to have knowledge like that; others are more concerned with how you collaborate with other candidates; still others are concerned with your code quality (I've seen a co-interviewer downmark a candidate for missing a ; at the end of a coding line. I wasn't impressed ... by the co-interviewer. But that's another story); still others are ... you get my point. Whether you do well or not can depend more on the cross-intersectional area of the interviewers style and your own credo than any knowledge you may or may not have.
So go in there expecting to be surprised, prepare what you can, be prepared to do wacky things to please "the man" interviewing you. For a good candidate, over a large number of interviews, you'll do well. The problem is that we often want a specific job, and we get depressed by the first dozen or so failed interviews. There's nothing more you can do than pick yourself up and try again. It's instructive to note that second-interviews at companies often go better than first-interviews, possibly because you're forewarned about the style a bit more, and therefore a bit better prepared...
Physicists get Hadrons!
If only they could figure out how to hire HR people who aren't so f---ing stupid, maybe they could come up with a decent process. The zero relationship thing doesn't surprise me at all. I thought the brain teasers as an interview sounded like one of the dumbest ideas ever.
Maybe they could test people on their problem solving abilities or even on skills related to their jobs?
It's nice to see a large company try to objectively evaluate its hiring process and express some self doubt. All to often the hiring process at a company is assumed to be good because the company is successful, which is an obvious fallacy since many factors contribute to a company's success. In fact I wouldn't hire anyone who didn't immediately question such an assumption :)
All too often the hiring process at a company, or the admissions process at a university, is treated as though it were created with some magical special sauce, when in fact it does little more than reinforce some (often unstated) prejudices. It's especially troubling coming from organizations that supposedly value rational and scientific analysis.
And my follow-up question would be: how would you go about finding out? Oh, here's my laptop. Knock yourself out.
Brainteasers for me were never about someone getting the answer right, it's how they work through a problem where they don't know the answer. Yours is a perfectly good answer, and leaves plenty of space to explore how you go about your research. To me, that's far more valuable than someone who has memorized the answer to a brain teaser.
Those who can, do. Those who can't, sue.
Do they have some objective job performance metrics that the rest of the world seems to have missed?
Supposedly brain teasers are used to figure out how you think about problems. Of course, when some candidates know the answers coming in -- or are familiar with that type of brain teaser, despite having no application to the job they do -- they tend to think about the problem better than people who don't.
I have colleagues and friends who've gone through Google hiring in the last 2 years. I've seen excellent personnel whom I've recommended not get the interview for 3 months, finally be interviewed because it turned out they were still looking to upgrade their position, and finally given job offers _over one month_ after the interview. Every single one of them found another role in the meantime, including promotions in their old company as new budgets were made to include a new position for them. The people who are still available after such a lengthy process are those who've effectively paid aa quite large Google hiring tax, of either weeks unemployed or of months at a lower salary.. While Google pays well, they don't pay well enough for people to pay such a task on the mere _hope_ of getting the Google role.
I've also seen some excellent personnel rejected because they applied for a specific role, which had requirements not in the job description and for which they were not made an offer. They were then unable to apply their existing interview results for roles which better suited their skills and which were not published as available when they applied to Google. They had to start over from the beginning. Coupled with the long hiring time for Google, and these personnel were long gone by the time they were made an offer or even interviewed for the second role.
The days when a back-of-the envelope calculation is enough are long gone (and probably never existed int he real world anyway).
Very much disagree on the lack of back-of-the-envelope calculations. von Braun and co. solved some of the hardest problems of Satern V development with paper napkins. I use quick calculations and engineering judgement all the time, and hire folks who are good at them too. In fact, we often spend far too much effort doing excessive studies when a few minutes of napkin math would give you the 80% answer. However, being able to figure out brain teasers and being able to quickly perform sound engineering judgements in a real work environment are two very different things.
Actually dismissing a question as stupid can work too. I was once asked a bunch of questions regarding the performance of a half dozen sorting algorithms, I recalled the details of only a couple. My answer: "Sorry, its been years since my data structures and algorithms exam. I bought the Knuth books so I can look up this stuff rather than have to memorize it."
I view interviews as two way. I'm evaluating the company. For example if the "senior engineer" giving me the above test doesn't know who Knuth is I probably don't want to work there. He did, but he pointed out my unconventional answer to the manager of the team. A person with a business background not a technical background. This manager asked what "Knuth" was and I explained. He then got a big smile, he loved my answer. A few days later I got a job offer. I worked there for four years, he was a suit, but he was a good one. He shielded us from as much BS as he could and he trusted and generally accepted our technical recommendation even when he personally had doubts.
Most corporations don't care about GPA, especially once you've got a few years of experience under your belt. Although I did send a CV for a research programmer position at a scientific research company on the east coast. They're first contact with me was to send me a form asking for everything going back to my high school GPA, SAT scores, activities, and college transcripts (undergrad and graduate). This happened about 4-5 years AFTER I received my PHD, with several years of post-graduate research experience. Of course, the initial job ad said they were looking for, "outstanding scientists with world class credentials", so I should've interpreted the use of that language to mean that they were a tad pretentious.
For Google interviews the answer to "run of the mill" brain teasers should be "Hang on while I Google it" ;).
And if they say no, ask them if it's better to use Bing instead.
I've never applied to Google because I'd heard enough about the interview process to realize it was mostly an unintentional way of asking if you're a recent graduate with a mind uncluttered by practical on-the-job knowledge, so you can focus on algorithms and brainteasers that have very few real world applications (and none in the job you're applying for.)
Try Netflix. According to a recent Slashdot post, they prefer hiring people who've spent a few years at Google learning their trade.
I find what Netflix does very interesting - effects of scale can be serious. They also take reliability very seriously, as some people deprived of a promised premiere can be dangerous :) As is so typical for the "ooh, ahh" evaluation of tech, little heed is paid to Netflix because they're selling movies, never mind that the tech is the magic behind the service. Yet Facebook get loads of "oohs" and "ahhs". The best tech is usually the tech that gets noticed least. Nobody thinks much of running the faucet, yet you're dealing with a tech that's been a key factor in civilizations since, uh, since there have been civilizations (and probably before).
I suppose it's news that the internal study found no correlation between interview scores and job performance, but everyone at Google recognizes that getting hired is a crapshoot. Not totally random, of course; there are plenty of candidates who simply aren't going to get hired, ever, because they don't have what it takes. But (I'm speaking of engineers here, dunno about other areas), everyone knows that candidates who are of the caliber Google seeks may or may not pass the interview process, and whether or not they do is pretty much a toss of the dice. I've heard rumors of a an internal study that took successful Google engineers and put them through the interview and hiring process, obscuring their employee status... and about half of them were "re-hired".
Also, as McDowell's blog post says, Google has always instructed interviewers not to use "brainteaser" questions. It probably does still happen once in a while -- indeed one of my interviewers asked me a "bonus" question, after I'd already demolished his design/coding problem, which arguably falls into that category (I failed to answer it) -- but they're doing it wrong and the hiring committee will let them know it.
Anyway, so if Google's process has such random results, why do they continue to use it? Simple: because nobody has found a better way. And the study results mentioned are a little misleading if you don't understand them in context: The study was of tens of thousands of interviews and their correlation with the performance of people who were hired. And nearly all of the people who are hired by Google go on to have successful careers at Google. What the study shows is that the degree of success is not correlated with the strength of the hiring recommendations.
On the other hand, as someone who came to Google with 20+ years of industry experience already behind him as a basis for comparison, I'll tell you one thing about the Google hiring process: It hires good people. It also fails to hire a lot of good people, but there are vanishingly few plodders or obstructionists around. In the 2.5 years I've worked for Google I have worked with well over 100 engineers (my work tends to touch lots of teams), and I've met one, maybe two, who weren't bright, highly competent and very effective, and even those one or two would be good-performers most places. That is very different from my prior experience, and I worked with a lot of high-profile companies.
As another data point, at every one of my prior employers I was something of a star, commonly called a "genius" and similar in performance reviews. At Google... I'm merely competent, perhaps a bit below average. Many of my colleagues are much smarter than me, and the superstars at Google are absolutely brilliant. One woman in particular who I've worked with quite a bit is always at least four steps ahead of me. She constantly says things that I think are stupid... until I have time to catch up with her thought process. She also talks faster than anyone I've ever met, in an attempt to try to keep up with her brain, I think. Talking to her is exhausting, but exhilarating. I've taken to structuring my conversations with her so they are always interrupted after no more than five minutes because that's about all I can take before I need to go process for a while. My consolation is that I notice many other people interact with her in the same way. Overall, my experience of Google employees that they're all smart, energetic and talented, with a strong leavening of the truly brilliant, and that perception extends even outside of engineering. Hell, our building facilities manager is really sharp.
What I experience of my colleagues is exactly what Google aims to achieve: since there's no known way to make accurate hiring decisions, the interview process aims primarily to filter out candidates who aren't fairly outstanding. In the process, it excludes a lot of really talented people, but it's very effective at excluding basically all of the poor to mediocre candidates.
I'm just glad the dice went my way when I interviewed.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
"outstanding scientists with world class credentials"
you are
Science advances one funeral at a time- Max Planck
They often waive their official GPA requirements if you worked in the field while earning your degree. 25-30 hours a week as a programmer while going to college full time and most corps won't care whether your GPA was 2.5 or 3.5 when applying for a development job.
Not a bad approach. Several years into my BS I switched from full-time student to full-time employment and part-time student. My grades went down, but I actually learned more in my classes because I saw the applications. It also cured me of the suspicion that classes only taught ivory tower nonsense.
...started with a phone interview a couple of years back (2006 maybe?). I was asked some run-of-the-mill questions, then the bombshell: An obscure question about an obscure RFC that had to do with big integer number representations. I told the interviewer that I really didn't know, and would she like me to wing an answer or get back to her on it? She told me to wing an answer. So I did. Later, I looked up the RFC and saw that I was more wrong than right.
Strangely, they offered to fly me to Mountain View for a second interview. Not so strangely, I declined. And I've never regretted the decision.
If GPAs are not an indicator but Google thought they where then their sample should show a negative correlation. i.e. people who were hired with low GPAs against the policy must have had something going for them?
Check your facts. Google is pretty much the only large company challenging those letters: https://www.google.com/search?q=google+challenges+national+security+letters
Maybe picking employees is like picking stocks. Sure that guy has been doing well the past 10 years. He's from a top school. Then his wife leaves him and he hits the crack pipe. You have no way of predicting that.
Conversely, the next candidate is from Podunk U and slid by with a C average. He's got a passion for coding though and was going through a lot of teenage shit in school. A few years out, the open source projects he worked on in his free time taught him a lot and he's just entering what will turn out to be 15 years of solid coding performance that vault him into the top 1% of programmers. You can't see that coming either; because his resume looks like shit.
Finally, between these two extremes you have a lot of average people. Even with all the right bullet-points, they still fit a bell curve and you can't predict where they fit. The coin isn't heads or tails until you... hire it and find Shroedinger's cat stinking up the cubicle or purring contently.
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
Having done a fair amount of interviewing and hiring, I knew the day that the big G called me that I had to say no.
What baffles me is that Google "could" have looked at the history of hiring and found this out many years ago. I took classes with the HR director at Southwest Airlines, who themselves had recorded and performed the same evaluation of hiring practices since the 60's. They too found that technical skill was only a minor indicator of success. Southwest found that personal intent, ethics and attitude were bigger drivers of success than technical expertise.
In fact, many companies have done these long-term studies before, and found similar results. There are volumes and volumes of studies... so why did the "big data" company ignore the data? It's just ridiculous!
I can just imagine that Google has a big problem now...
I said no... but I missed and it came out yes.
1)Will I and the coworkers get along with this person.
2)Will they work hard.
Soooooo many people in the tech industry fail at these 2 points. I would much rather have someone who has skills in the same ballpark that meet 1 and 2 then someone who is an expert in the area but is an ass.