Slashdot Mirror
Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women (reuters.com)
Jeffrey Dastin, reporting for Reuters: Amazon's machine-learning specialists uncovered a big problem: their new recruiting engine did not like women. The team had been building computer programs since 2014 to review job applicants' resumes with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters. Automation has been key to Amazon's e-commerce dominance, be it inside warehouses or driving pricing decisions. The company's experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars -- much like shoppers rate products on Amazon, some of the people said. "Everyone wanted this holy grail," one of the people said. "They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those." But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. That is because Amazon's computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.
[...] Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity.
[...] Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity.
As hard as you want to say, sometimes you still need an actual person doing the job. That person will be biased in some way other another too, so I guess it's not a perfect system any way you look at it.
Train algorithm with data in hand, algorithm's output mirrors data provided. They can't possibly be shocked by this, can they?
Which has more power: the hammer, or the anvil?
People defend search engines vigorously when there's perceived bias in algorithmic results.
Why would Amazon's system receive different criticism?
When government reviews your hiring they expect you to show that your diversity level is consistent with the normal spread of minority groups ( some consideration of candidate pool MAY be given.)
In other words, if your only criteria is hiring whomever best for the job, you will likely be operating illegally and subject to fines and lawsuits. This is the product of laws that are designed to create social engineering based restrictions based on someones religious idea that any measurable discrepancy in minority placement must be corrected.
âoeTolerance applies only to persons, but never to truth. Intolerance applies only to truth, but never to persons.
Amazon trained their AI using the dataset that reflected their business practices as they currently are (flaws and all) but what they wanted was a data set for the practices they wanted to become (i.e. the ideal).
Finding a training dataset that reflects the ideal is going to be extremely difficult, particularly in an area where that ideal is so poorly defined.
It's called the "Kavanatron"
-5 political trolling
That a hiring algorithm would pick more men out of the applicant pool than women. Ohh wait! That pool is mostly menn you say? Statistics and probability be damned, we must achieve that diversity quota at all costs!!
It lets them make immoral business decisions but not be personally held accountable for them.
Facebook shows real estate ads only to white professionals. Amazon only hires male chinese engineers. Google endlessly manipulates its search for political reasons.
But when questions get asked, it's always that pesky old AI that did it!
Get used to it.
It could be that the tool worked perfectly and all the top candidates were men.
So...the AI looked ignored mad skills, because all resumes had them, and instead looked at past resumes of successful hires, found "women's" wasn't in most of them, and drew the conclusion that it correlated with a terrible candidate.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Bias is a non-factual prejudice against someone. That is why it is considered unfair. If the facts are that 80% of the population of people who do the work you want are named "Dave", then it is not a sign of a moral failing if your AI exhibits a strong preference for another Dave.
"We receive as friendly that which agrees with, we resist with dislike that which opposes us" - Faraday
Garbage in, garbage out.
If the training data has bias, then the AI will learn to have that bias.
The trick is developing training data that doesn't reflect the biases of the humans that performed the task in the past.
One of our competitors trademarked the term "hypothesis". From now on, we will call them "boneheaded ideas".
just remove the gender part of the application form, or otherwise obscure it so the AI doesn't factor it in. Surely they've thought of this?
Sorry TRUMP TRAITORS, you're going to WISH you had women to push around as you rot in that cell, treasonous faggots.
We have good expert systems that can do amazing things with ultra controlled inputs.
Pretending computers have a bias vs anything is really dumb, articles and submissions like this are for controlling the narrative and keeping people poorly informed. The general populace is much smarter than they are given credit for, especially when they have the right information.
People that push this kind of nonsense ought to be ashamed. Slashdot used to be about the cool tech, anyone can go to vice(or pick your politically correct preference) and look at all the current crop of slashdot articles there, why be here?
If the results are biased, the data is biased and the process is biased, maybe the bias is normal?
I can imagine the conversation at Amazon...
"Goddamnit! The top CV's picked from this impartial unbiased machine learning algorithm are all men! It must be discriminating against women somehow, even though we aren't including gender in any of the data. Tweak it until more women pop out, or we're fucked once this gets out on Twitter..."
Maybe it *was* picking the best ones... and men just make better software engineers?
Yeah, yeah, sorry double plus ungood thoughtcrime. Warm up room 101...
Of course the machines would devise other ways of sorting candidates that will be discriminatory because machines don't give a fuck about gender equality, parity, inclusive treatment or positive(my ass) discrimination, but plain and hard facts to classify, sort AND discriminate the input.
That is what true equality and meritocracy is all about. You want the better ones or the not so better ones, but gender equated, paired, inclusive treated and "positively" discriminated?
You want the machine to sort the production by quality or want to be inclusive and non discriminatory and mix in defective parts to fill in the quota?
If it didn't scan the names on each resume, then it wasn't gender-biased.
When you read this article it doesn't say anything about this algorithm not 'liking woman'. Based on the parameters it was given it chose to rank candidates based upon the factors it was trained to look for. It's also somewhat telling how the writers of this tripe chose to specifically highlight how the algorithm chose to downgraded candidates from two all female colleges without saying why they were downgraded. As if the fact that it's an all female school is more important than the quality of the candidates that came out of the school.
At the end of the day this bullshit is more about how the media writes headlights to illicit emotional reactions instead of reporting the hows and the whys of a situation. And on that note I'd like to see someone actually start writing algorithms to to replace tech reporters so we can get ride of garbage tier activist journalism like this article.
When the facts don't fit your ideology, change the facts!
Purge any submission to the system of a gender identifier... women's or men's anything... remove names in case that is factored... literally provide nothing in the submission that would definitively define a gender.
Then see what it does.
My experience with these systems is that they don't actually factor gender but that the end result of is that there is a gender imbalance.
However, if there is an imbalance and the system was given no indication as to gender then there is no gender bias.
You can't cite persecution or preference if the system can't even know. And generally these fairly common and consistent imbalances are made without reference to gender itself.
Generally it is factoring on other criteria that give the same result but which are not gender. Work experience is a big one... breadth of skill set is another.
And if you took the total population and look at which portion of the population had that work experience and breadth of knowledge, you'd find it more closely matched the hiring patterns of these systems. Which means it isn't factoring on gender.
Now... this is assumption to some extent on my part. I've audited these systems in the past and what I am describing above is the pattern I've seen.
As to what the Amazon system was doing... I'd have to audit it.
What I'd probably try is a word replacement/purge of all terms that would signify gender or I'd just change a bunch of rejected female resumes to say they were male and see if they got accepted and vice versa.
If the system actually changed its decision based on gender then that's a smoking gun that it is doing things on the basis of gender.
But I'd find that very surprising.
Machine learning is unpredictable so I'm hardly going to claim to know what the damned thing was doing. For that reason I wouldn't actually use machine learning in this application. I'd use a very clear rules based system where everything it was doing was known to the programmers.
Those systems are completely fine for this sort of work and you can very easily audit the code for them.
The best way to deal with this is to first be gender blind. You literally do not factor for gender at all.
That will give you an imbalance probably... you can make as many diversity hires as you need to after that. But your core hiring pool should be merit based unless you want to go out of business.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
The summary edited out what actually happened:
In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.
This comes up frequently in high-tech companies: If only we could automate decision-making without involving people! Imagine!
This is literally the dumbest thing you could do, right up there with "B people hire C people." As an interviewer, I always looked at resumes to guide my interview approach, but in most cases it was impossible to make any decisions based on a resume. Even if you assume that the person didn't outright lie, you're looking at 4-line summary of 3-year work periods written by a writer who is very subjective, has little clue what is valuable about their work, and also is frankly a terrible writer. I often had candidates who were tough to call after we had spent an hour discussing multiple problems which I brought to the table - how could reducing your information content by many orders of magnitude possibly help?
And, let me be frank, resumes are full of lies and half-truths. I could believe machine-learning your way to a good evaluator given hundreds of pages of writing, especially if you have supporting evidence, but that's impossible with a resume. Hell, it's impossible to get supporting evidence in a resume unless someone is referring the candidate, and if it's a referral, you're usually better off just talking to the referrer rather than reading the resume at all!
Now, if you could feed the system a candidate's entire history of code reviews, email interactions with others, perf write-ups, things they say in meetings, etc, then I'll grant that you could plausibly machine-learn your way to identifying the top performers. I don't like how much of work it misses, though.
You would think a company called "Amazon" would show preference to women.
is that the job market for tech is so crappy that they're writing special software to sift through the hundreds of resumes they get. Back in my day the hiring manager just looked over a few and picked one.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Pre-process the training data so that exactly half of the input is from male applicants, and half from female. If there are more male entries in the complete dataset, then randomly remove them until it's exactly 50:50. Yes, this means tossing out potentially valuable information. However, if male applicants make it to the top 5, then they could be further compared against each other using the full male dataset. Surely if I thought of this in 5 sec they could too?
If the problem is that not enough women are in the training data, then it's not a problem with the method, but rather an indication that they don't have good data. Fixing this problem will take time, as it will require populating the dataset by hiring more women.
You can lead a horse to water, but you can't make it dissolve.
Well, the last thing any of our new diversity-obsessed saviors want is to (openly) specify hiring criteria, and generally computers require you to specify things. So it's not surprising that we run into these little snafus.
That said, I would have thought that "AI" would be a, er, godess-send for these folks ... just train it for awhile, and nobody will have any way to prove why it makes the decisions that it does. Sounds perfect for "diversity" hiring.
Simply reading the instructions the AI was apparently using, according to the article, tells me that whoever created this AI was either (1) a moron or (2) a bigot.
I guess the academics biased 2-1 in favor of female candidates will now resign in disgrace ?
And sexist courses like Wimmins Studies will close ?
And teachers will concentrate on educating boys, who are behind girls at every level of education ?
Of course not: the feminazis, who rule academia and the media, have won.
Comment removed based on user account deletion
The claim that "the industry is dominated by men and therefore we couldn't train this in a gender-neutral way" is totally bogus from a machine-learning perspective. All that is needed to eliminate a bias arising from dataset imbalance is to balance the dataset.
More likely they realised that when using dispassionate criteria for optimal hiring, it would become very likely they'd not get the desired "Women > Men" politically correct outcome for all sorts of statistically valid reasons, and figured such optimal hiring was not worth its salt against all the money lost from lawsuits and bad PR in a time of a politically tense climate favouring women.
I completely agree with their choice, and would do the same. No need to feed oil to the fire
Yes.
Both this ridiculous garbage reporting and the apoplectic shitshow from ideologues in the press over James Damore's memo are not just the usual bland claims of sexism.
There are long known and well researched gender differences in interest preference going all the way back to infants - long before any possible way to for the results to be explained by 'societal sexism' or other such nonsense.
Feminist dogma is 100 percent counterfactual this basic and well researched science.
Hence why the over the top attacks on anyone and anything that brings to light these fundamental differences in the abilities of men vs women in technological jobs.
The reason there is such a huge disparity in male hires in tech companies is a direct result of those well established gender differences. The candidates being selected are at the very, very top end of the bell curve in both intelligence(where men have a significant advantage) and a lifetime of interest in and drive compared to female applicants in general.
Of course the usual 'argument' and response anyone pointing these basic facts out is screetching that the claim is women aren't as capable as men.
Any individual woman can be just as capable as a man in tech.
However, that is not true at the population level where men will significantly outpace women in the number of highly qualified candidates.
So you feed an algorithm existing employee data for it to learn who the best employees are so you can match against new applications. Seems like a reasonable way to find more employees that are specifically good for your company. Now the computer (let me re-emphasize that, unbiased piece of machinery) chose the best employees to build it's template... And as it turns out, the majority of their best employees were men.... What a shocker considering this is for software development and it's not the most female trade to begin with. All you've done is shown this fact.... That more men are software developers than women and by default, more men will have the desired skills than women. It only "sexist" because some feminazi's probably didn't understand how the algorithm actually works because they weren't good enough software developers.
The A.I. doesn't care about being politically correct.
Maybe the A.I. has computed something we're not aware of.
Unfortunately, people will force political correctness into the A.I. and we'll never learn the truth. /sarcasm (or is it?)
#DeleteFacebook
Look, it's very simple.
Hire women. Stop finding excuses.
Here at the UW we have tons of STEM majors, in fact most of our AI people are women.
-- Tigger warning: This post may contain tiggers! --
Hang this nazi faggot from his punkass bitch neck until he understands how that might limit his ability to speak English. Find his mother and rape her conservatively as he supports.
So the algorithm picked the best candidates who just happened to be men. Is that so hard to understand. So now they need to game the system to get less qualified candidates just to check off being politically correct.
After repeated efforts to automate rational discrimination based on ability, they were unable to avoid results that were statistically preferential to one sex. And, we just can't figure out why that is so. What a deep mystery.
We're a miserable excuse for a species therefore our computer programs mirror what miserable pieces of garbage we are. Of course it discriminates, because humans discriminate. Remember the racist chat-bot? It wasn't built that way but it became that way because humans are assholes. You all worry about Skynet happening for real? If it does it'll be our own damned fault because it'll just be a more efficient killer than we are.
When reality is so sexist, that we need to blame AI for it, instead of admitting, some people's explanations of why there are huge gender gaps all over the place are rather wishful thinking.
Tech geniuses create AI.
Make 500 models, teach it to recognize some 50,000 terms.
AI does HR's job too well.
Executives kill project.
Ok, So imagine you wanted to have an automated way of incrementally increasing your workforce percentage of females. One way would be to segment the training data, that is resumes of past hires who have remained at the company for 5 years, say, into a separate female employees set (and their resumes) and male employees set.
Bounce the incoming resumes against both models, and find the good matches according to some threshold.
Now you are free to tweak the ratio of candidates coming from both automated selection streams, to add a percent or two bias toward the female stream. This implements your desired (or legally mandated) social goal over time, with only a tiny impact on "fairness" of selection.
Where are we going and why are we in a handbasket?
https://www.smbc-comics.com/co...
XML is like violence. If it doesn't solve the problem, use more.
Strat is a nazi faggot, hang this bitch.
This is what happens when the "everything is sexist" crowd is involved. When a resume is completely devoid of any gender indications, they'd then just blame the margins to say it was determined that the choices of margins by the woman made it clear she was a woman and that's why she was ignored.
Duh! Males don't get pregnant and drop out of the work force.
job application software sucks and is easy to be Bias on any group that you want.
also they want way to much info up front.
Like California's new law, there's going to be diversity requirements. Smh. Any algorithm that doesn't fulfill the preordained conclusion doesn't fulfill the desired, utopian-idealistic agenda.
Assuming algorithm tries to find dependencies between incoming data (resumes, etc) and resulting performance (performance reviews, promotions), maybe it is just the truth?
"Most resumes came from men" - so what? Why would it become a factor in resume evaluation? If there actually was bias against women in hiring in last 10 years, all the hired ones would be really remarkable off-the chart good employees and algorithm would be biased toward them. On the other hand, if the bar was lowered to fill quotas, bias would be different.
It is interesting that Amazon decided to abandon the ideal altogether and disbanded the team - they realized they can't just bend reality, at best they can ignore it and pretend nothing happened.
I've been working on such projects for some time now. We've tried everything but the bias always surfaces. Not just against women, but of course also against many minorities. My colleagues at google, linkedin and a few other large companies tried and are trying too - so far it is like stepping over a minefield with ACLU and other organisations just waiting to expose bias in whatever those companies deploy.
Thing is even if you strip the obvious attributes you can still easily infer the gender and race from past work experience, club affiliations, hobbies, postal code, and so on. You can attempt to remove bias by tweaking the whole model and normalizing against those attributes but then you very quickly start getting garbage as output. Either you get accuracy (which can be defined: providing recommendations which are in line with hiring managers' preferences) or you get bias-free useless garbage.
For now only one or two companies in the US managed to implement, at meaningful scale, a certifiably bias free job matching algo, with emphasis on 'certifiably' (e.g. hirevue). AI in HR is so far good for screening, bad for matching, because, for large part, employers are biased. Just walk into google offices, or most of the bay area companies for that matter. Who do you see? Asians, whites. Hiring managers are biased, and an argument can be made that the skills are not uniformly distributed across genders, ages and races -- due to bias, racism and other factors contributing to uneven chances and interests. But hey, once you start using AI you are supposed to cure the world and offer bias free recommendations.
What does work, is relying on anonymized merit-based hiring tools that get the candidates to solve algorithmic puzzles and present the results without disclosing the identities or any other attributes of the candidates. But that is only relevant for a few professional markets such as it or accounting and generally is met with resistance from employers as this forces them to actually get invested in the recruitment process. And while those methods allow for hiring decisions to be bias-free as the hiring managers dont know the gender, age or anything else until late in the process - the output is still biased. For the same reasons. Skill distribution is not uniform across demographies.
The AI detected the patterns in the data and the consumers of that output didn't like it.
If you're going to say "nah I don't like what we're finding", then why ask the question in the first place.
Unfortunately the data aren't politically correct.
Last time that I reviewed the corresponding law , it was that your hiring process needed to be transparently not discreminating against a minority. e.g. if you were careful to make sure the hiring process was double blinded and only criteria which are gender/religion/ethnicity independent were exterminated, you were fine. I have not seen any evidence of what you state in the relevant case law.
We need more data to evaluate the claim. If it's matching the resume against the resumes of people that work at Amazon, the dataset is inherently imbalanced, and while you could fix it for women, that demonstrates a persistent problem that can apply to other things.
"Women > Men" is not a real thing* nor politically correct, be serious.
* I'm sure you can scour the Internet and find groups of radicals that say that, in the same way you can scour the internet and find Jews that passionately believe in Nazism and think the genocide thing was just a big misunderstanding)
Haha haha. You asked a system to sort people. That act is inherently discriminatory.
Then when the system tells you the best options you tell it that it's wrong.
From an objective stand point the humans are sexist for wanting women over men where a system that cannot be sexist told them men would do better.
And they blame it on resume patterns? What exactly is inherently a sex difference between men and women's resumes?
Slightly tangential to that, I thought the people's view was that gender shouldn't be a factor? Why did anyone even question the robot in the first place? Oh right, not enough women.
Seems like the most obvious issue here is a sexist slant in women's favor, per usual
Yeah. So more men apply for the jov
What the hell are you going to do about it? Purposefully hire a disproportionate amount of women applying? Hire unqualified women?
Am I reading this wrong? They took in more male applications so they think they need to make the system WANT to hire more women?
It said the best for the job were disproportionately men, it's tech. That's their bag.
They trained it to look for patterns and it found patterns so they shut it down?
But rather even when they had made sure it didn't it still thought the men were better and they didn't liked that.
Also I'd believe all these fake-equality people when they talk about men being crushed at their job, men dying of prostate cancer, man suiciding the most, male grades in schools, males without a sex partner or life companion so on so on.
They are feminists - not for equality.
Injustice activists.
AI really has become smarter than the average human.
and men are simply better.
Funny, how looking at the outcomes is assumed to be unbiased, given that many companies have idiots like James Damore who drive down the effective performance of women
Here's how you fix it. For any desired number of candidates, retain the first half of the list produced by the algorithm, then replace the bottom half with as many candidates from a ranked list of the top female candidates until the overall list reaches parity.
Good luck with any other method that does not essentially just do this.
Maybe what we *want* is not what nature, biology and statistics give us. How do we know our models of reality are accurate if we throw them out when they don't conform to our expectations? Models of reality should have the upper hand, not expectations, because that's the very definition of bias. If reality really was lopsided towards some and in disadvantage of others, the model describing that is not biased - we'd be if we expected it to be different.
If that question alone sounds like a heresy to you, it's a clear indication where we're headed.
All that is needed to eliminate a bias arising from dataset imbalance is to balance the dataset.
You say that like it's a trivial task. They are evaluating people for a job, and most of the current evaluation data is either subjective or biased or both.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
seriously, you cant scan through 100 resumes to hire someone? if it takes you more than 1 minute each (to do a rough screen/filter), you probably dont know what you are looking for or at. that's 1h40m to filter/scan for someone to hire that you are going to pay $30/40/50/60k+ per year, hopefully for many years. holy hell people are lazy. i've churned through 30 resumes over a cup of coffee. even crazier, companies pay recruiters 15-30% of salaries to do the screening. what a mess. i take it back, people really are beyond lazy these days...
All systems nominal. Oh, you want a system that doesn't work and makes bad decisions?