Slashdot Mirror

← Back to Stories (view on slashdot.org)

Microsoft Developing a Tool To Help Engineers Catch Bias in Algorithms (venturebeat.com)

Posted by msmash on from the a-fight-within dept.

Microsoft is developing a tool that can detect bias in artificial intelligence algorithms with the goal of helping businesses use AI without running the risk of discriminating against certain people. From a report: Rich Caruana, a senior researcher on the bias-detection tool at Microsoft, described it as a "dashboard" that engineers can apply to trained AI models. "Things like transparency, intelligibility, and explanation are new enough to the field that few of us have sufficient experience to know everything we should look for and all the ways that bias might lurk in our models," he told MIT Technology Review. Bias in algorithms is an issue increasingly coming to the fore. At the Re-Work Deep Learning Summit in Boston this week, Gabriele Fariello, a Harvard instructor in machine learning and chief information officer at the University of Rhode Island, said that there are "significant ... problems" in the AI field's treatment of ethics and bias today. "There are real decisions being made in health care, in the judicial system, and elsewhere that affect your life directly," he said.

23 of 239 comments (clear)

  1. Wrong Bias by Anonymous Coward · · Score: 3, Insightful

    Correctly read as: "Microsoft is developing a tool to help developers detect wrong bias in their algorithms."

  2. Couldn't a tool developed by Shemmie · · Score: 2

    to detect bias in algorithms, be used in an attempt to insert bias into algorithms, without detection?

    Just spit-balling here.

    1. Re: Couldn't a tool developed by phantomfive · · Score: 2

      Equally, it can be used to avoid liability. You can say, "Maybe it's biased, but we did due diligence, it's not our fault!" Maybe though, maybe Microsoft is trying to avoid another Tay.

      --
      "First they came for the slanderers and i said nothing."
    2. Re:Couldn't a tool developed by green1 · · Score: 2

      Except in reality it's probably more like an algorithm that rolls the dice 6 times, and complains that it's biased if it doesn't roll one of each of the 6 numbers. That's no bias, that's how random works.

      Thing is, the real world isn't random. And the people who make these things are likely to try to fit a random pattern on to non-random data. For instance, if you have 30000 males, and 10000 females in a particular data set, and you pick a random person from that data set 500 times, you'll likely pick approximately 75% male. The "bias detection algorithm" will then tell you that your algorithm is sexist because it should have picked females 50% of the time. Your algorithm wasn't sexist, it was completely unbiased, and didn't even know the gender it was picking until after the fact. But the authors would suggest you tweak your algorithm until it picks an "unbiased" 50% female from your data set which is itself not 50/50.

      These efforts are almost never true efforts to eliminate bias, but are in fact efforts to introduce a politically correct bias.

  3. The bias of reverse bias by Citizen+of+Earth · · Score: 5, Insightful

    The main problem with this endeavor is that the "bias" they are trying to suppress is actually the opposite of bias. They seek to treat people differently on the basis of identity politics instead of on their actual behavior. The AIs will naturally be confused by being disallowed to latch onto the strongest signals in the data.

    1. Re:The bias of reverse bias by frank_adrian314159 · · Score: 3, Interesting

      The AIs will naturally be confused by being disallowed to latch onto the strongest signals in the data.

      Uh not unless it's a really crappy AI. If you haven't noticed, chances are any human directive will be treated as that by the neural network - another signal that is larger/more salient because it is input by a human. Just the way that the system would be designed to do unless you want it completely independent of human control.

      In short, don't project your own human confusion about neural nets onto the technology just because you don't like the implications of human control of machines.

      --
      That is all.
  4. Re:Unbiased approach. by ArmoredDragon · · Score: 2

    Of course it is. From what I understand, in nearly all cases the algorithms that make decisions about routine stuff don't even have access to information about the person's race, nationality, gender, etc. If so, how is bias even possible? It sounds like the individuals it disfavors may have some kind of adverse event in their history that was fed into the algorithm. I.e. missing down payments, drove 50mph over the speed limit, did 2 years in Virginia for possession of fentanyl, etc.

    Except in the case of car insurance, where gender is given, and is very biased against males, and for good reason. Bias against any other identifiable category, no matter how good of a reason, and the court of public opinion will summarily issue a guilty verdict, and then Hank Johnson will introduce a new bill banning algorithms.

  5. Except no by bug_hunter · · Score: 2, Insightful

    From the article:

    Northpointe’s Compas software, which uses machine learning to predict whether a defendant will commit future crimes, was found to judge black defendants more harshly than white defendants.

    So that was an existing algorithm that judged somebody on how they were born rather than their individual behavior.

    --
    It's turtles all the way down.
    1. Re:Except no by bug_hunter · · Score: 2, Interesting

      Here's a more more interesting question:

      Do you want a justice system that says:
      For the crime of breaking an entering:
      White person : 2 years
      Black person : 4 years
      Asian person : 1 year
      etc

      Do you imagine the groups on the larger sentencing of that spectrum having faith in the justice system?

      --
      It's turtles all the way down.
    2. Re:Except no by russotto · · Score: 4, Informative

      The COMPAS algorithm, while opaque, does not have race as an input. It was found its accuracy could be matched by an algorithm with just two variables: age and prior convictions. Even this simple model shows the same "bias" that COMPAS is accused of. The bias isn't in the algorithm; it's in the real world.

    3. Re: Except no by phantomfive · · Score: 4, Interesting

      I've found that to be a problem in my attempts to make neural networks: too often a complex network can be simplified to just a few variables that, once found, can be hard coded. In some ways it's really depressing.

      --
      "First they came for the slanderers and i said nothing."
    4. Re:Except no by bitkid · · Score: 5, Informative

      Slight tangent: The article cites the ProPublica study on the Northpointe software in which journalists (not statisticians) reported the software as biased. What they left out is that an independent study found this study showing bias to be wrong.

      Source: Flores, Bechtel, Lowencamp; Federal Probation Journal, September 2016, "False Positives, False Negatives, and False Analyses: A Rejoinder to “Machine Bias: There’s Software Used Across the Country to Predict Future Criminals. And it’s Biased Against Blacks.”", URL http://www.uscourts.gov/statis...

      In fact the ProPublica analysis was so wrong that the authors wrote: "It is noteworthy that the ProPublica code of ethics advises investigative journalists that "when in doubt, ask" numerous times. We feel that Larson et al.'s (2016) omissions and mistakes could have been avoided had they just asked. Perhaps they might have even asked...a criminologist? We certainly respect the mission of ProPublica, which is to "practice and promote investigative journalism in the public interest." However, we also feel that the journalists at ProPublica strayed from their own code of ethics in that they did not present the facts accurately, their presentation of the existing literature was incomplete, and they failed to "ask." While we aren’t inferring that they had an agenda in writing their story, we believe that they are better equipped to report the research news, rather than attempt to make the research news."

      The authors of the ProPublica article are no longer with the organization, but this article shows up in any news article about AI bias. The fake story just doesn't want to die...

      With all that said, I have some hopes that algorithms will help make truly race-blind decisions in criminal justice. It's easier to test them for bias than humans, and decisions are made in a consistent, repeatable manner.

    5. Re: Except no by phantomfive · · Score: 2

      That would be a benefit. In most cases I've found that neural networks have been wholly inadequate for the task I've chosen, and another approach is better (for example, a standard natural language processor with a strong domain processor to rank resumes. It is true you will get a small improvement at recognizing verbs and nouns with a NN without actually understanding meaning, but the improvement potential of building a solid domain model will make the NN look like a rounding error. You might say that using a neural network to build up a domain model is a good idea, but then you will need to spend tremendously more time building up a data set). Admittedly, I am not an expert, and there are definitely some domains where NN are very useful.

      --
      "First they came for the slanderers and i said nothing."
  6. Re:Unbiased approach. by Anonymous Coward · · Score: 4, Insightful

    Eliminating Bias from AI means discarding facts and data that violate SJW principals.

  7. What exactly is an algorithm bias? by misnohmer · · Score: 4, Interesting

    I've been reading stories in removing bias from algorithms but still don't get it. What is an algorithm bias? If the results don't have perfectly flat distribution across sex, race, religion, and other protected groups?

    1. Re:What exactly is an algorithm bias? by Actually,+I+do+RTFA · · Score: 4, Informative

      What is an algorithm bias?

      An algorithm that uses historic data, which was distorted by human bias, to predict future events. These reinforce human bias from the past. For instance, did you know that in 1864, practically no black people in the South ever paid a debt back? If you use that fact (which was, you know, caused by slavery) to figure that black people were higher credit risks, which meant higher rates, which meant more defaults, which meant worse credit, etc, your algorithm is biased.

      --
      Your ad here. Ask me how!
  8. This is actually an important research topic by bangular · · Score: 4, Informative

    I think we have to be a little more formal with terminology. The summary and most articles these days use "algorithm" and "AI" interchangeably. You can use an algorithm to train a machine learning model, but the model isn't really an algorithm in the classical sense.

    The trained model can definitely have bias based on the training data. The classical example is, train a word2vec or glove model on the texts of wikipedia, then find the vector representations of doctor and nurse. You'll find that nurse is considered a female term while doctor is male.

    This may be acceptable for trivial things like advertising or movie suggestions, but machine learning is now being used for important things like job application screenings. Many times the model can be very opaque and this bias may not seem obvious. Even worse, it seems every company now wants to have AI in their product, and may have half-rate data scientists that graduated from a data science bootcamp.

    The research I've seen on this subject is serious work. In the case of the doctor/nurse vector representation, the goal would be to make the occupation gender neutral. The tricky part is that you'd still want the model to retain certain qualities, like mother being female and father being male.

  9. Re:Unbiased approach. by fafalone · · Score: 3, Informative

    SJWs simultaneously complain that black people are being arrested more, and that an algorithm that predicts higher recidivism for blacks is improperly biased and should return the same risk for whites.
    Explain this to one, and you'll get a blank stare followed by an accusation that you're a racist.

  10. Re:Obey by serviscope_minor · · Score: 2

    Remember, Citizen: Equality means including an equal

    No, citizen, equality means not giving you a harsher conviction simply because people who look like you have been convicted in the past. What I don't really get is why you'e against true equality.

    --
    SJW n. One who posts facts.
  11. Re: Bias in - Bias out. by bitkid · · Score: 2

    First example you cite has been shown to be based on flawed statistics, i.e., the algorithm was shown not to produce biased results on the data. Bad things happen when journalists try to do statistical analysis.

      Reference: Flores, Bechtel, Lowencamp; "False Positives, False Negatives, and False Analyses: A Rejoinder to âoeMachine Bias: Thereâ(TM)s Software Used Across the Country to Predict Future Criminals. And itâ(TM)s Biased Against Blacks.â", Federal Probation Journal, September 2016, You can find the article here: http://www.uscourts.gov/statis...

  12. Comment removed by account_deleted · · Score: 4, Informative

    Comment removed based on user account deletion

  13. Re:Unbiased approach. by russotto · · Score: 2

    It constantly overestimates the real recidivism rate for black people.

    It does not. Take a look at this Washington Post article

    Note the first graph. For each risk score, chance of recidivism is approximately the same between blacks and whites.

    What ProPublica showed is the reverse, that black defendants who do not reoffend are more likely to receive a high score than white defendants who do not reoffend. Given that black defendants as a whole are more likely to re-offend, this is unavoidable without making the predictor biased against whites instead. The Post article goes into this.

  14. Comment removed by account_deleted · · Score: 2

    Comment removed based on user account deletion