Slashdot Mirror
Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women (reuters.com)
Jeffrey Dastin, reporting for Reuters: Amazon's machine-learning specialists uncovered a big problem: their new recruiting engine did not like women. The team had been building computer programs since 2014 to review job applicants' resumes with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters. Automation has been key to Amazon's e-commerce dominance, be it inside warehouses or driving pricing decisions. The company's experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars -- much like shoppers rate products on Amazon, some of the people said. "Everyone wanted this holy grail," one of the people said. "They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those." But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. That is because Amazon's computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.
[...] Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity.
[...] Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity.
As hard as you want to say, sometimes you still need an actual person doing the job. That person will be biased in some way other another too, so I guess it's not a perfect system any way you look at it.
Train algorithm with data in hand, algorithm's output mirrors data provided. They can't possibly be shocked by this, can they?
Which has more power: the hammer, or the anvil?
When government reviews your hiring they expect you to show that your diversity level is consistent with the normal spread of minority groups ( some consideration of candidate pool MAY be given.)
In other words, if your only criteria is hiring whomever best for the job, you will likely be operating illegally and subject to fines and lawsuits. This is the product of laws that are designed to create social engineering based restrictions based on someones religious idea that any measurable discrepancy in minority placement must be corrected.
âoeTolerance applies only to persons, but never to truth. Intolerance applies only to truth, but never to persons.
Amazon trained their AI using the dataset that reflected their business practices as they currently are (flaws and all) but what they wanted was a data set for the practices they wanted to become (i.e. the ideal).
Finding a training dataset that reflects the ideal is going to be extremely difficult, particularly in an area where that ideal is so poorly defined.
It lets them make immoral business decisions but not be personally held accountable for them.
Facebook shows real estate ads only to white professionals. Amazon only hires male chinese engineers. Google endlessly manipulates its search for political reasons.
But when questions get asked, it's always that pesky old AI that did it!
Get used to it.
Bias is a non-factual prejudice against someone. That is why it is considered unfair. If the facts are that 80% of the population of people who do the work you want are named "Dave", then it is not a sign of a moral failing if your AI exhibits a strong preference for another Dave.
"We receive as friendly that which agrees with, we resist with dislike that which opposes us" - Faraday
Garbage in, garbage out.
If the training data has bias, then the AI will learn to have that bias.
The trick is developing training data that doesn't reflect the biases of the humans that performed the task in the past.
One of our competitors trademarked the term "hypothesis". From now on, we will call them "boneheaded ideas".
just remove the gender part of the application form, or otherwise obscure it so the AI doesn't factor it in. Surely they've thought of this?
We have good expert systems that can do amazing things with ultra controlled inputs.
Pretending computers have a bias vs anything is really dumb, articles and submissions like this are for controlling the narrative and keeping people poorly informed. The general populace is much smarter than they are given credit for, especially when they have the right information.
People that push this kind of nonsense ought to be ashamed. Slashdot used to be about the cool tech, anyone can go to vice(or pick your politically correct preference) and look at all the current crop of slashdot articles there, why be here?
If the results are biased, the data is biased and the process is biased, maybe the bias is normal?
If it didn't scan the names on each resume, then it wasn't gender-biased.
When you read this article it doesn't say anything about this algorithm not 'liking woman'. Based on the parameters it was given it chose to rank candidates based upon the factors it was trained to look for. It's also somewhat telling how the writers of this tripe chose to specifically highlight how the algorithm chose to downgraded candidates from two all female colleges without saying why they were downgraded. As if the fact that it's an all female school is more important than the quality of the candidates that came out of the school.
At the end of the day this bullshit is more about how the media writes headlights to illicit emotional reactions instead of reporting the hows and the whys of a situation. And on that note I'd like to see someone actually start writing algorithms to to replace tech reporters so we can get ride of garbage tier activist journalism like this article.
Purge any submission to the system of a gender identifier... women's or men's anything... remove names in case that is factored... literally provide nothing in the submission that would definitively define a gender.
Then see what it does.
My experience with these systems is that they don't actually factor gender but that the end result of is that there is a gender imbalance.
However, if there is an imbalance and the system was given no indication as to gender then there is no gender bias.
You can't cite persecution or preference if the system can't even know. And generally these fairly common and consistent imbalances are made without reference to gender itself.
Generally it is factoring on other criteria that give the same result but which are not gender. Work experience is a big one... breadth of skill set is another.
And if you took the total population and look at which portion of the population had that work experience and breadth of knowledge, you'd find it more closely matched the hiring patterns of these systems. Which means it isn't factoring on gender.
Now... this is assumption to some extent on my part. I've audited these systems in the past and what I am describing above is the pattern I've seen.
As to what the Amazon system was doing... I'd have to audit it.
What I'd probably try is a word replacement/purge of all terms that would signify gender or I'd just change a bunch of rejected female resumes to say they were male and see if they got accepted and vice versa.
If the system actually changed its decision based on gender then that's a smoking gun that it is doing things on the basis of gender.
But I'd find that very surprising.
Machine learning is unpredictable so I'm hardly going to claim to know what the damned thing was doing. For that reason I wouldn't actually use machine learning in this application. I'd use a very clear rules based system where everything it was doing was known to the programmers.
Those systems are completely fine for this sort of work and you can very easily audit the code for them.
The best way to deal with this is to first be gender blind. You literally do not factor for gender at all.
That will give you an imbalance probably... you can make as many diversity hires as you need to after that. But your core hiring pool should be merit based unless you want to go out of business.
I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
This comes up frequently in high-tech companies: If only we could automate decision-making without involving people! Imagine!
This is literally the dumbest thing you could do, right up there with "B people hire C people." As an interviewer, I always looked at resumes to guide my interview approach, but in most cases it was impossible to make any decisions based on a resume. Even if you assume that the person didn't outright lie, you're looking at 4-line summary of 3-year work periods written by a writer who is very subjective, has little clue what is valuable about their work, and also is frankly a terrible writer. I often had candidates who were tough to call after we had spent an hour discussing multiple problems which I brought to the table - how could reducing your information content by many orders of magnitude possibly help?
And, let me be frank, resumes are full of lies and half-truths. I could believe machine-learning your way to a good evaluator given hundreds of pages of writing, especially if you have supporting evidence, but that's impossible with a resume. Hell, it's impossible to get supporting evidence in a resume unless someone is referring the candidate, and if it's a referral, you're usually better off just talking to the referrer rather than reading the resume at all!
Now, if you could feed the system a candidate's entire history of code reviews, email interactions with others, perf write-ups, things they say in meetings, etc, then I'll grant that you could plausibly machine-learn your way to identifying the top performers. I don't like how much of work it misses, though.
You would think a company called "Amazon" would show preference to women.
is that the job market for tech is so crappy that they're writing special software to sift through the hundreds of resumes they get. Back in my day the hiring manager just looked over a few and picked one.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
Pre-process the training data so that exactly half of the input is from male applicants, and half from female. If there are more male entries in the complete dataset, then randomly remove them until it's exactly 50:50. Yes, this means tossing out potentially valuable information. However, if male applicants make it to the top 5, then they could be further compared against each other using the full male dataset. Surely if I thought of this in 5 sec they could too?
If the problem is that not enough women are in the training data, then it's not a problem with the method, but rather an indication that they don't have good data. Fixing this problem will take time, as it will require populating the dataset by hiring more women.
You can lead a horse to water, but you can't make it dissolve.
No that is an incorrect statement.
The Algorithm finding that more men then women are being hired. Decides that gender is a factor to be considered. It is weighing factors that really do not matter.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
Well, the last thing any of our new diversity-obsessed saviors want is to (openly) specify hiring criteria, and generally computers require you to specify things. So it's not surprising that we run into these little snafus.
That said, I would have thought that "AI" would be a, er, godess-send for these folks ... just train it for awhile, and nobody will have any way to prove why it makes the decisions that it does. Sounds perfect for "diversity" hiring.
Simply reading the instructions the AI was apparently using, according to the article, tells me that whoever created this AI was either (1) a moron or (2) a bigot.
Comment removed based on user account deletion
The claim that "the industry is dominated by men and therefore we couldn't train this in a gender-neutral way" is totally bogus from a machine-learning perspective. All that is needed to eliminate a bias arising from dataset imbalance is to balance the dataset.
More likely they realised that when using dispassionate criteria for optimal hiring, it would become very likely they'd not get the desired "Women > Men" politically correct outcome for all sorts of statistically valid reasons, and figured such optimal hiring was not worth its salt against all the money lost from lawsuits and bad PR in a time of a politically tense climate favouring women.
I completely agree with their choice, and would do the same. No need to feed oil to the fire
Yes.
Both this ridiculous garbage reporting and the apoplectic shitshow from ideologues in the press over James Damore's memo are not just the usual bland claims of sexism.
There are long known and well researched gender differences in interest preference going all the way back to infants - long before any possible way to for the results to be explained by 'societal sexism' or other such nonsense.
Feminist dogma is 100 percent counterfactual this basic and well researched science.
Hence why the over the top attacks on anyone and anything that brings to light these fundamental differences in the abilities of men vs women in technological jobs.
The reason there is such a huge disparity in male hires in tech companies is a direct result of those well established gender differences. The candidates being selected are at the very, very top end of the bell curve in both intelligence(where men have a significant advantage) and a lifetime of interest in and drive compared to female applicants in general.
Of course the usual 'argument' and response anyone pointing these basic facts out is screetching that the claim is women aren't as capable as men.
Any individual woman can be just as capable as a man in tech.
However, that is not true at the population level where men will significantly outpace women in the number of highly qualified candidates.
The A.I. doesn't care about being politically correct.
Maybe the A.I. has computed something we're not aware of.
Unfortunately, people will force political correctness into the A.I. and we'll never learn the truth. /sarcasm (or is it?)
#DeleteFacebook
So the algorithm picked the best candidates who just happened to be men. Is that so hard to understand. So now they need to game the system to get less qualified candidates just to check off being politically correct.
Do you really think Amazon hired a statistically significant number of 'Vassar girls'?
The one they did, likely didn't make it out of probation. Hence the 'AI' gave that trait a negative score. It could be right.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
When reality is so sexist, that we need to blame AI for it, instead of admitting, some people's explanations of why there are huge gender gaps all over the place are rather wishful thinking.
Tech geniuses create AI.
Make 500 models, teach it to recognize some 50,000 terms.
AI does HR's job too well.
Executives kill project.
Ok, So imagine you wanted to have an automated way of incrementally increasing your workforce percentage of females. One way would be to segment the training data, that is resumes of past hires who have remained at the company for 5 years, say, into a separate female employees set (and their resumes) and male employees set.
Bounce the incoming resumes against both models, and find the good matches according to some threshold.
Now you are free to tweak the ratio of candidates coming from both automated selection streams, to add a percent or two bias toward the female stream. This implements your desired (or legally mandated) social goal over time, with only a tiny impact on "fairness" of selection.
Where are we going and why are we in a handbasket?
That interpretation is very silly. They're trying to figure out "as a percentage of candidates with resumes like X that we interview, how many do well at the job", and make all such statistic correlations that they can.
It's not about "they have fewer women on staff", obviously. It's about "how do female candidates fare". Amazon, like all of the Big 5, is trying to address gender balance, and so will interview women with less chance of getting hired than their male candidates. They want more women hired, so they take a bigger risk, an accept the elevated cost in interviewer time. That's great, as it lets you hire more women without lowering the bar for them (accusations about Google's process aside).
But of course the side effect of that is that fewer women who get interviewed get hired. The AI of course found that correlation, and probably dozens of other correlations associated with protected classes, and used it. Pretty obvious in hindsight, but then lots of things are obvious in hindsight.
Socialism: a lie told by totalitarians and believed by fools.
I can imagine the conversation at Amazon...
"Goddamnit! The top CV's picked from this impartial unbiased machine learning algorithm are all men! It must be discriminating against women somehow, even though we aren't including gender in any of the data. Tweak it until more women pop out, or we're fucked once this gets out on Twitter..."
Maybe it *was* picking the best ones... and men just make better software engineers?
Yeah, yeah, sorry double plus ungood thoughtcrime. Warm up room 101...
Substitute women with H1B.
Top CVs picked are H1Bs. Double ungood though-felony!!
https://www.smbc-comics.com/co...
XML is like violence. If it doesn't solve the problem, use more.
Why is this downmod? That's what happened and was the issue and what they had to address.
Who is fantasizing this is an attack on women? That's disasterbation.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
job application software sucks and is easy to be Bias on any group that you want.
also they want way to much info up front.
I've been working on such projects for some time now. We've tried everything but the bias always surfaces. Not just against women, but of course also against many minorities. My colleagues at google, linkedin and a few other large companies tried and are trying too - so far it is like stepping over a minefield with ACLU and other organisations just waiting to expose bias in whatever those companies deploy.
Thing is even if you strip the obvious attributes you can still easily infer the gender and race from past work experience, club affiliations, hobbies, postal code, and so on. You can attempt to remove bias by tweaking the whole model and normalizing against those attributes but then you very quickly start getting garbage as output. Either you get accuracy (which can be defined: providing recommendations which are in line with hiring managers' preferences) or you get bias-free useless garbage.
For now only one or two companies in the US managed to implement, at meaningful scale, a certifiably bias free job matching algo, with emphasis on 'certifiably' (e.g. hirevue). AI in HR is so far good for screening, bad for matching, because, for large part, employers are biased. Just walk into google offices, or most of the bay area companies for that matter. Who do you see? Asians, whites. Hiring managers are biased, and an argument can be made that the skills are not uniformly distributed across genders, ages and races -- due to bias, racism and other factors contributing to uneven chances and interests. But hey, once you start using AI you are supposed to cure the world and offer bias free recommendations.
What does work, is relying on anonymized merit-based hiring tools that get the candidates to solve algorithmic puzzles and present the results without disclosing the identities or any other attributes of the candidates. But that is only relevant for a few professional markets such as it or accounting and generally is met with resistance from employers as this forces them to actually get invested in the recruitment process. And while those methods allow for hiring decisions to be bias-free as the hiring managers dont know the gender, age or anything else until late in the process - the output is still biased. For the same reasons. Skill distribution is not uniform across demographies.
But rather even when they had made sure it didn't it still thought the men were better and they didn't liked that.
Also I'd believe all these fake-equality people when they talk about men being crushed at their job, men dying of prostate cancer, man suiciding the most, male grades in schools, males without a sex partner or life companion so on so on.
They are feminists - not for equality.
Injustice activists.
All that is needed to eliminate a bias arising from dataset imbalance is to balance the dataset.
You say that like it's a trivial task. They are evaluating people for a job, and most of the current evaluation data is either subjective or biased or both.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC