Slashdot Mirror

← Back to Stories (view on slashdot.org)

Amazon's Facial Recognition Wrongly Identifies 28 Lawmakers, ACLU Says (nytimes.com)

Posted by msmash on from the closer-look dept.

Representative John Lewis of Georgia and Representative Bobby L. Rush of Illinois are both Democrats, members of the Congressional Black Caucus and civil rights leaders. But facial recognition technology made by Amazon, which is being used by some police departments and other organizations, incorrectly matched the lawmakers with people who had been arrested for a crime, the American Civil Liberties Union reported on Thursday morning. From a report: The errors emerged as part of a larger test in which the civil liberties group used Amazon's facial software to compare the photos of all federal lawmakers against a database of 25,000 publicly available mug shots. In the test, the Amazon technology incorrectly matched 28 members of Congress with people who had been arrested, amounting to a 5 percent error rate among legislators. The test disproportionally misidentified African-American and Latino members of Congress as the people in mug shots.

"This test confirms that facial recognition is flawed, biased and dangerous," said Jacob Snow, a technology and civil liberties lawyer with the A.C.L.U. of Northern California. Nina Lindsey, an Amazon Web Services spokeswoman, said in a statement that the company's customers had used its facial recognition technology for various beneficial purposes, including preventing human trafficking and reuniting missing children with their families. She added that the A.C.L.U. had used the company's face-matching technology, called Amazon Rekognition, differently during its test than the company recommended for law enforcement customers.

12 of 145 comments (clear)

  1. AI sometimes isn't perfect either by foxalopex · · Score: 5, Insightful

    I think the whole idea of using face recognition is to cut the amount of work required by a detective to search through thousands of pictures. I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

    1. Re:AI sometimes isn't perfect either by ranton · · Score: 4, Insightful

      I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

      This is exactly correct, and why these statements from the ACLU are ridiculous. Would they rather the police just be looking for any tall black guy with a sweatshirt in the area? This type of technology simply provides more information to the police, but it still takes actual policemen and prosecutors to decide who is a real suspect and who should be charged.

      --
      -- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke
    2. Re:AI sometimes isn't perfect either by jeff4747 · · Score: 5, Insightful

      You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

      What about recent law enforcement activities have given you any reason for this hope?

      You "match", you get arrested. And held until you can pay the bail for your "crime", or decide to plead to a lesser charge for time served.

      Meanwhile, your life is completely destroyed while you're in jail, because you can't work, can't pay your bills, lose your house because you can't pay the mortgage/rent, can't care for your kids, and so on. So there's a ton of pressure to plead to something just to stop the destruction. And when you do, the cops have "solved" the crime and look good.

    3. Re:AI sometimes isn't perfect either by Anonymous Coward · · Score: 3, Insightful

      Except it won't be. They'll just arrest the person and "let the courts sort it out." Which never recognizes the damage that simply being arrested by itself can cause to someone.

    4. Re:AI sometimes isn't perfect either by Wycliffe · · Score: 4, Insightful

      The mere fact that innocent citizens show up on the radar at ALL for police trying to solve a crime is very troublesome.

      You want to absolutely minimize false positives.

      I disagree. I think you should set up the AI to always produce false positives and probably hide the percentage of the match as well. Just like a lineup it should always return the top 10 results sorted randomly regardless of how closely they match. That way the cops don't start relying on it as something that it isn't.

    5. Re:AI sometimes isn't perfect either by bigpat · · Score: 4, Insightful

      I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

      This is exactly correct, and why these statements from the ACLU are ridiculous. Would they rather the police just be looking for any tall black guy with a sweatshirt in the area? This type of technology simply provides more information to the police, but it still takes actual policemen and prosecutors to decide who is a real suspect and who should be charged.

      Yes and no. So when I have run image recognition through a neural net I get a percentage match... so it depends what the threshold for a match is set at. Is 65% considered a match or 95% or 99.9%? The devils in the details and I could see the percentage being obscured from the end user to the point of police and the courts treating it as a binary value rather than with any relative degree of certainty because the police and the courts want to be right and are under time constraints to be right and move on.

      So depending on the percentage match I could see people in the same racial group being matched... but would a court issue a warrant based on someone saying they are in the same racial group... because I could see the police saying that "they were a match using facial recognition" and the court just rubber stamping that because it obscures the real underlying lack of certainty. There is a real danger of abuse depending on how facial recognition is used (like any tool), but neural net algorithms are especially prone to obfuscation.

      On the other side, people are often terrible witnesses and have their own underlying lack of certainty that can be obscured without the reproducible and adjustable nature of image recognition. People are often wrong in their recollection and many people have gone to jail because of wrong identification by witnesses, sometimes even multiple witnesses.

      In other words their is uncertainty no matter what... the good and the bad news with AI is that you can begin to quantify that uncertainty. So image recognition is good news for improving accuracy over human perception, but bad news if it is either misunderstood or willfully abused to create the misconception of 100% accuracy.

    6. Re:AI sometimes isn't perfect either by sdavid · · Score: 3, Insightful

      One of the concerns with a high false-positive rate and large databases yields a lot of unnecessary investigations and if the rate is high enough it can facilitate investigation of individuals who have already been targeted. "This guy looks sketchy, let's run his photo through the database." How carefully is the photo going to be reviewed in that situation before he's detained and searched?

    7. Re:AI sometimes isn't perfect either by Wycliffe · · Score: 4, Insightful

      When you consider the large crowds in the public spaces where this system is likely to be deployed, a 5% false positive rate would result in unmanageable numbers to verify. -E.g. Times Square sees 300,000 people a day movement, resulting in 15,000 false positives a day. Even a 1% false positive rate would be too high, especially considering the cost in civil liberties involved to those falsely flagged.

      They aren't going to arrest 15000 people a day so there is no "cost in civil liberties involved to those falsely flagged" nor are they going to arrest 1000 people but it could help them quickly look at those 1000 people from a distance versus having to do the impossible job of trying to look at all 300k people. A large false positive is actually probably a good thing. If the false positive is too small like say only 0.01% then the cops might be tempted to arrest all 30 of those people without doing due diligence.

    8. Re:AI sometimes isn't perfect either by cayenne8 · · Score: 3, Insightful

      I disagree. I think you should set up the AI to always produce false positives and probably hide the percentage of the match as well. Just like a lineup it should always return the top 10 results sorted randomly regardless of how closely they match. That way the cops don't start relying on it as something that it isn't.

      Being in a line up is voluntary.

      Here's the thing about the cop. They are there under pressure to get a conviction, especially if the crime is public, and heinous.

      They hope it is the correct person, but that doesn't always happen, and innocent people go to jail and get executed.

      Ok, so scenario, my face gets pulled up false positive. I was never there, but I don't have a real valid alibi.

      Witnesses, who are often very unreliable, especially if it was a violent, dangerous fast acting crime...ID's me as the suspect.

      Public opinion goes against me...social media goes against me, and I get convicted on circumstantial evidence.

      That is not a far fetched thing to happen, granted, hopefully rare, but not far fetched at ALL.

      Now, if my name had never come up as a false positive, I would have never been on the police radar, and would have never even remotely been on the radar for a crime I didn't commit.

      Look, I'm gung ho for criminals to get caught and judged. But I'm willing to let a few go free, to ensure that as close to zero innocents get convicted and have their lives ruined.

      --
      Light travels faster than sound. This is why some people appear bright until you hear them speak.........
  2. Confidence threshold by DredJohn · · Score: 3, Insightful

    Increasing the confidence threshold would probably have reduced the 5% error rate.... From the article: The A.C.L.U had used the system’s default setting for matches, called a “confidence threshold,” of 80%. That means the group counted any face matches the system proposed that had a similarity score of 80% or more. Amazon recommended that police departments use a much higher similarity score — 95% — to reduce the likelihood of erroneous matches.

  3. Poor definitions of "matched" by GoRK · · Score: 1, Insightful

    I'd find the claims more credible if they defined what "match" meant and showed comparisons of which photos actually "matched."

    If you are familiar with the birthday problem, you must certainly realize that if you take a couple of sizable populations, such as, say, the 535 members of congress and try to match them up with another large set of data, say, 25,000 mugshots then certainly you are likely to find some uncanny resemblances even given the overwhelmingly huge variety inherent in a person's appearance. On top of this, you are using an algorithm that is designed to give a confidence value to be interpreted by a person and is therefore inherently tuned to match optimistically.

    To be clear I am not a fan of any of this, but the ACLU spreading a bunch of FUD really doesn't do anything to properly advance the discussion; instead it solidifies the argument that the opponents of facial recognition technologies are complete idiots who don't understand how the technologies operate or are applied.

  4. Re:I note that the NY Times. . . by Whorhay · · Score: 3, Insightful

    I agree with you that arrested does not equate to being a criminal, but I've also met plenty of people who don't agree with that. Getting wrongly arrested can seriously screw with a persons life both in the long and short term. Missing a single day of work, or even being significantly late can lose a person a job. On top of that there is the cost of paying bail or a bail bondsman, and possibly a lawyer. And god help you and your family if you end up wrongly convicted.

    You're absolutely right that cops aren't perfect and like simple solutions. Which should make it incredibly obvious that giving them broad access to facial recognition technology is a bad idea. Our crime rates are sufficiently low enough that something like this will easily cause more harm than good in our society.