Slashdot Mirror

← Back to Stories (view on slashdot.org)

Amazon's Facial Recognition Wrongly Identifies 28 Lawmakers, ACLU Says (nytimes.com)

Posted by msmash on from the closer-look dept.

Representative John Lewis of Georgia and Representative Bobby L. Rush of Illinois are both Democrats, members of the Congressional Black Caucus and civil rights leaders. But facial recognition technology made by Amazon, which is being used by some police departments and other organizations, incorrectly matched the lawmakers with people who had been arrested for a crime, the American Civil Liberties Union reported on Thursday morning. From a report: The errors emerged as part of a larger test in which the civil liberties group used Amazon's facial software to compare the photos of all federal lawmakers against a database of 25,000 publicly available mug shots. In the test, the Amazon technology incorrectly matched 28 members of Congress with people who had been arrested, amounting to a 5 percent error rate among legislators. The test disproportionally misidentified African-American and Latino members of Congress as the people in mug shots.

"This test confirms that facial recognition is flawed, biased and dangerous," said Jacob Snow, a technology and civil liberties lawyer with the A.C.L.U. of Northern California. Nina Lindsey, an Amazon Web Services spokeswoman, said in a statement that the company's customers had used its facial recognition technology for various beneficial purposes, including preventing human trafficking and reuniting missing children with their families. She added that the A.C.L.U. had used the company's face-matching technology, called Amazon Rekognition, differently during its test than the company recommended for law enforcement customers.

17 of 145 comments (clear)

  1. Well, it was half right... by magusxxx · · Score: 5, Funny

    ...they just haven't been arrested yet. :D

    --
    Care killed the cat, but satisfaction brought it back.
  2. AI sometimes isn't perfect either by foxalopex · · Score: 5, Insightful

    I think the whole idea of using face recognition is to cut the amount of work required by a detective to search through thousands of pictures. I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

    1. Re:AI sometimes isn't perfect either by ranton · · Score: 4, Insightful

      I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

      This is exactly correct, and why these statements from the ACLU are ridiculous. Would they rather the police just be looking for any tall black guy with a sweatshirt in the area? This type of technology simply provides more information to the police, but it still takes actual policemen and prosecutors to decide who is a real suspect and who should be charged.

      --
      -- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
    2. Re:AI sometimes isn't perfect either by jeff4747 · · Score: 5, Insightful

      You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

      What about recent law enforcement activities have given you any reason for this hope?

      You "match", you get arrested. And held until you can pay the bail for your "crime", or decide to plead to a lesser charge for time served.

      Meanwhile, your life is completely destroyed while you're in jail, because you can't work, can't pay your bills, lose your house because you can't pay the mortgage/rent, can't care for your kids, and so on. So there's a ton of pressure to plead to something just to stop the destruction. And when you do, the cops have "solved" the crime and look good.

    3. Re:AI sometimes isn't perfect either by Anonymous Coward · · Score: 3, Insightful

      Except it won't be. They'll just arrest the person and "let the courts sort it out." Which never recognizes the damage that simply being arrested by itself can cause to someone.

    4. Re:AI sometimes isn't perfect either by lrichardson · · Score: 3, Informative

      "What about recent law enforcement activities have given you any reason for this hope?"

      None. None whatsoever. A female acquaintance at the courthouse for a traffic ticket was arrested, and put behind bars overnight. There was a warrant out for someone with the same first and last name. Of course, the detail she is a petite Caucasian female and the suspect was a large black male might have tipped the LE officer off that her protests had some validity ... but as there is zero repercussions to LE pulling cr4p like this, it is going to continue.

      Personally, I'm getting really sick of hearing lawyers state 'He reacted in the way he was trained.' As though the police department training is to blame for lack of common sense, lack of knowledge of the law, and general lack of humanity.

    5. Re:AI sometimes isn't perfect either by Wycliffe · · Score: 4, Insightful

      The mere fact that innocent citizens show up on the radar at ALL for police trying to solve a crime is very troublesome.

      You want to absolutely minimize false positives.

      I disagree. I think you should set up the AI to always produce false positives and probably hide the percentage of the match as well. Just like a lineup it should always return the top 10 results sorted randomly regardless of how closely they match. That way the cops don't start relying on it as something that it isn't.

    6. Re:AI sometimes isn't perfect either by bigpat · · Score: 4, Insightful

      I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

      This is exactly correct, and why these statements from the ACLU are ridiculous. Would they rather the police just be looking for any tall black guy with a sweatshirt in the area? This type of technology simply provides more information to the police, but it still takes actual policemen and prosecutors to decide who is a real suspect and who should be charged.

      Yes and no. So when I have run image recognition through a neural net I get a percentage match... so it depends what the threshold for a match is set at. Is 65% considered a match or 95% or 99.9%? The devils in the details and I could see the percentage being obscured from the end user to the point of police and the courts treating it as a binary value rather than with any relative degree of certainty because the police and the courts want to be right and are under time constraints to be right and move on.

      So depending on the percentage match I could see people in the same racial group being matched... but would a court issue a warrant based on someone saying they are in the same racial group... because I could see the police saying that "they were a match using facial recognition" and the court just rubber stamping that because it obscures the real underlying lack of certainty. There is a real danger of abuse depending on how facial recognition is used (like any tool), but neural net algorithms are especially prone to obfuscation.

      On the other side, people are often terrible witnesses and have their own underlying lack of certainty that can be obscured without the reproducible and adjustable nature of image recognition. People are often wrong in their recollection and many people have gone to jail because of wrong identification by witnesses, sometimes even multiple witnesses.

      In other words their is uncertainty no matter what... the good and the bad news with AI is that you can begin to quantify that uncertainty. So image recognition is good news for improving accuracy over human perception, but bad news if it is either misunderstood or willfully abused to create the misconception of 100% accuracy.

    7. Re:AI sometimes isn't perfect either by sdavid · · Score: 3, Insightful

      One of the concerns with a high false-positive rate and large databases yields a lot of unnecessary investigations and if the rate is high enough it can facilitate investigation of individuals who have already been targeted. "This guy looks sketchy, let's run his photo through the database." How carefully is the photo going to be reviewed in that situation before he's detained and searched?

    8. Re:AI sometimes isn't perfect either by Wycliffe · · Score: 4, Insightful

      When you consider the large crowds in the public spaces where this system is likely to be deployed, a 5% false positive rate would result in unmanageable numbers to verify. -E.g. Times Square sees 300,000 people a day movement, resulting in 15,000 false positives a day. Even a 1% false positive rate would be too high, especially considering the cost in civil liberties involved to those falsely flagged.

      They aren't going to arrest 15000 people a day so there is no "cost in civil liberties involved to those falsely flagged" nor are they going to arrest 1000 people but it could help them quickly look at those 1000 people from a distance versus having to do the impossible job of trying to look at all 300k people. A large false positive is actually probably a good thing. If the false positive is too small like say only 0.01% then the cops might be tempted to arrest all 30 of those people without doing due diligence.

    9. Re:AI sometimes isn't perfect either by Areyoukiddingme · · Score: 3, Informative

      In other words their is uncertainty no matter what... the good and the bad news with AI is that you can begin to quantify that uncertainty. So image recognition is good news for improving accuracy over human perception, but bad news if it is either misunderstood or willfully abused to create the misconception of 100% accuracy.

      Given how both fingerprints and DNA matching have been painted with the 100% accurate brush (unjustifiably), I expect facial recognition will be too.

      Hilarity ensues.

    10. Re:AI sometimes isn't perfect either by cayenne8 · · Score: 3, Insightful

      I disagree. I think you should set up the AI to always produce false positives and probably hide the percentage of the match as well. Just like a lineup it should always return the top 10 results sorted randomly regardless of how closely they match. That way the cops don't start relying on it as something that it isn't.

      Being in a line up is voluntary.

      Here's the thing about the cop. They are there under pressure to get a conviction, especially if the crime is public, and heinous.

      They hope it is the correct person, but that doesn't always happen, and innocent people go to jail and get executed.

      Ok, so scenario, my face gets pulled up false positive. I was never there, but I don't have a real valid alibi.

      Witnesses, who are often very unreliable, especially if it was a violent, dangerous fast acting crime...ID's me as the suspect.

      Public opinion goes against me...social media goes against me, and I get convicted on circumstantial evidence.

      That is not a far fetched thing to happen, granted, hopefully rare, but not far fetched at ALL.

      Now, if my name had never come up as a false positive, I would have never been on the police radar, and would have never even remotely been on the radar for a crime I didn't commit.

      Look, I'm gung ho for criminals to get caught and judged. But I'm willing to let a few go free, to ensure that as close to zero innocents get convicted and have their lives ruined.

      --
      Light travels faster than sound. This is why some people appear bright until you hear them speak.........
    11. Re:AI sometimes isn't perfect either by ranton · · Score: 3, Informative

      Except it won't be. They'll just arrest the person and "let the courts sort it out." Which never recognizes the damage that simply being arrested by itself can cause to someone.

      And that practice will be something for the ACLU to combat. But always assuming the worst possible use of new techniques and technologies is not helpful.

      --
      -- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
  3. Garbage In, Gospel Out! by Thud457 · · Score: 4, Funny

    I find it hard to believe that only 28 members of Congress are criminals. This AI needs to go back for reeducation.

    --

    the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff

  4. Confidence threshold by DredJohn · · Score: 3, Insightful

    Increasing the confidence threshold would probably have reduced the 5% error rate.... From the article: The A.C.L.U had used the system’s default setting for matches, called a “confidence threshold,” of 80%. That means the group counted any face matches the system proposed that had a similarity score of 80% or more. Amazon recommended that police departments use a much higher similarity score — 95% — to reduce the likelihood of erroneous matches.

  5. I note that the NY Times. . . by Salgak1 · · Score: 3, Interesting

    . . . . does not appear to show the mugshots they reportedly matched. That would be a critical point in the argument. I've seen multiple "near-clones" of people over the years: it's entirely likely that the Congresscritters have some as well. . .

    Additionally, ARRESTED does not make one a criminal. Conviction does. The wrong people get arrested all the time: cops are FAR from perfect. . .and like Slashdot, they like simple solutions to their problems....

    1. Re:I note that the NY Times. . . by Whorhay · · Score: 3, Insightful

      I agree with you that arrested does not equate to being a criminal, but I've also met plenty of people who don't agree with that. Getting wrongly arrested can seriously screw with a persons life both in the long and short term. Missing a single day of work, or even being significantly late can lose a person a job. On top of that there is the cost of paying bail or a bail bondsman, and possibly a lawyer. And god help you and your family if you end up wrongly convicted.

      You're absolutely right that cops aren't perfect and like simple solutions. Which should make it incredibly obvious that giving them broad access to facial recognition technology is a bad idea. Our crime rates are sufficiently low enough that something like this will easily cause more harm than good in our society.