Amazon's Facial Recognition Wrongly Identifies 28 Lawmakers, ACLU Says (nytimes.com)

Representative John Lewis of Georgia and Representative Bobby L. Rush of Illinois are both Democrats, members of the Congressional Black Caucus and civil rights leaders. But facial recognition technology made by Amazon, which is being used by some police departments and other organizations, incorrectly matched the lawmakers with people who had been arrested for a crime, the American Civil Liberties Union reported on Thursday morning. From a report: The errors emerged as part of a larger test in which the civil liberties group used Amazon's facial software to compare the photos of all federal lawmakers against a database of 25,000 publicly available mug shots. In the test, the Amazon technology incorrectly matched 28 members of Congress with people who had been arrested, amounting to a 5 percent error rate among legislators. The test disproportionally misidentified African-American and Latino members of Congress as the people in mug shots.

"This test confirms that facial recognition is flawed, biased and dangerous," said Jacob Snow, a technology and civil liberties lawyer with the A.C.L.U. of Northern California. Nina Lindsey, an Amazon Web Services spokeswoman, said in a statement that the company's customers had used its facial recognition technology for various beneficial purposes, including preventing human trafficking and reuniting missing children with their families. She added that the A.C.L.U. had used the company's face-matching technology, called Amazon Rekognition, differently during its test than the company recommended for law enforcement customers.

Comments

Well, it was half right...

[magusxxx]

...they just haven't been arrested yet. :D

Re:Well, it was half right...

[cayenne8]

I guess what they say is true....

...all congressmen DO look alike....

;)

AI sometimes isn't perfect either

[foxalopex]

I think the whole idea of using face recognition is to cut the amount of work required by a detective to search through thousands of pictures. I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

Re:AI sometimes isn't perfect either

[ranton]

I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

This is exactly correct, and why these statements from the ACLU are ridiculous. Would they rather the police just be looking for any tall black guy with a sweatshirt in the area? This type of technology simply provides more information to the police, but it still takes actual policemen and prosecutors to decide who is a real suspect and who should be charged.

Re:AI sometimes isn't perfect either

[wired_parrot]

When you consider the large crowds in the public spaces where this system is likely to be deployed, a 5% false positive rate would result in unmanageable numbers to verify. -E.g. Times Square sees 300,000 people a day movement, resulting in 15,000 false positives a day. Even a 1% false positive rate would be too high, especially considering the cost in civil liberties involved to those falsely flagged.

Re:AI sometimes isn't perfect either

[jeff4747]

You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

What about recent law enforcement activities have given you any reason for this hope?

You "match", you get arrested. And held until you can pay the bail for your "crime", or decide to plead to a lesser charge for time served.

Meanwhile, your life is completely destroyed while you're in jail, because you can't work, can't pay your bills, lose your house because you can't pay the mortgage/rent, can't care for your kids, and so on. So there's a ton of pressure to plead to something just to stop the destruction. And when you do, the cops have "solved" the crime and look good.

Re:AI sometimes isn't perfect either

[Anonymous Coward]

Except it won't be. They'll just arrest the person and "let the courts sort it out." Which never recognizes the damage that simply being arrested by itself can cause to someone.

Re:AI sometimes isn't perfect either

[lrichardson]

"What about recent law enforcement activities have given you any reason for this hope?"

None. None whatsoever. A female acquaintance at the courthouse for a traffic ticket was arrested, and put behind bars overnight. There was a warrant out for someone with the same first and last name. Of course, the detail she is a petite Caucasian female and the suspect was a large black male might have tipped the LE officer off that her protests had some validity ... but as there is zero repercussions to LE pulling cr4p like this, it is going to continue.

Personally, I'm getting really sick of hearing lawyers state 'He reacted in the way he was trained.' As though the police department training is to blame for lack of common sense, lack of knowledge of the law, and general lack of humanity.

Re:AI sometimes isn't perfect either

[Wycliffe]

The mere fact that innocent citizens show up on the radar at ALL for police trying to solve a crime is very troublesome.

You want to absolutely minimize false positives.

I disagree. I think you should set up the AI to always produce false positives and probably hide the percentage of the match as well. Just like a lineup it should always return the top 10 results sorted randomly regardless of how closely they match. That way the cops don't start relying on it as something that it isn't.

Re:AI sometimes isn't perfect either

[bigpat]

I'm sure the final step would be for a real person to verify the matches to see if there's false positives. The AI in this case would likely be setup to tend to produce false positives rather than outright missing matches because not being able to find anything is worrysome compared to finding a few false positives. You would hope the cops arn't crazy enough to start arresting people based entirely on the matching system and at least look at the profiles to confirm.

This is exactly correct, and why these statements from the ACLU are ridiculous. Would they rather the police just be looking for any tall black guy with a sweatshirt in the area? This type of technology simply provides more information to the police, but it still takes actual policemen and prosecutors to decide who is a real suspect and who should be charged.

Yes and no. So when I have run image recognition through a neural net I get a percentage match... so it depends what the threshold for a match is set at. Is 65% considered a match or 95% or 99.9%? The devils in the details and I could see the percentage being obscured from the end user to the point of police and the courts treating it as a binary value rather than with any relative degree of certainty because the police and the courts want to be right and are under time constraints to be right and move on.

So depending on the percentage match I could see people in the same racial group being matched... but would a court issue a warrant based on someone saying they are in the same racial group... because I could see the police saying that "they were a match using facial recognition" and the court just rubber stamping that because it obscures the real underlying lack of certainty. There is a real danger of abuse depending on how facial recognition is used (like any tool), but neural net algorithms are especially prone to obfuscation.

On the other side, people are often terrible witnesses and have their own underlying lack of certainty that can be obscured without the reproducible and adjustable nature of image recognition. People are often wrong in their recollection and many people have gone to jail because of wrong identification by witnesses, sometimes even multiple witnesses.

In other words their is uncertainty no matter what... the good and the bad news with AI is that you can begin to quantify that uncertainty. So image recognition is good news for improving accuracy over human perception, but bad news if it is either misunderstood or willfully abused to create the misconception of 100% accuracy.

Re:AI sometimes isn't perfect either

[avandesande]

I don't think this will be used for picking people out of a crowd but more for things like processing passport application or employment background checks. If you get a positive it just means digging deeper.

Re:AI sometimes isn't perfect either

[sdavid]

One of the concerns with a high false-positive rate and large databases yields a lot of unnecessary investigations and if the rate is high enough it can facilitate investigation of individuals who have already been targeted. "This guy looks sketchy, let's run his photo through the database." How carefully is the photo going to be reviewed in that situation before he's detained and searched?

Re:AI sometimes isn't perfect either

[Wycliffe]

When you consider the large crowds in the public spaces where this system is likely to be deployed, a 5% false positive rate would result in unmanageable numbers to verify. -E.g. Times Square sees 300,000 people a day movement, resulting in 15,000 false positives a day. Even a 1% false positive rate would be too high, especially considering the cost in civil liberties involved to those falsely flagged.

They aren't going to arrest 15000 people a day so there is no "cost in civil liberties involved to those falsely flagged" nor are they going to arrest 1000 people but it could help them quickly look at those 1000 people from a distance versus having to do the impossible job of trying to look at all 300k people. A large false positive is actually probably a good thing. If the false positive is too small like say only 0.01% then the cops might be tempted to arrest all 30 of those people without doing due diligence.

Re:AI sometimes isn't perfect either

[Areyoukiddingme]

In other words their is uncertainty no matter what... the good and the bad news with AI is that you can begin to quantify that uncertainty. So image recognition is good news for improving accuracy over human perception, but bad news if it is either misunderstood or willfully abused to create the misconception of 100% accuracy.

Given how both fingerprints and DNA matching have been painted with the 100% accurate brush (unjustifiably), I expect facial recognition will be too.

Hilarity ensues.

Re:AI sometimes isn't perfect either

[Areyoukiddingme]

Personally, I'm getting really sick of hearing lawyers state 'He reacted in the way he was trained.' As though the police department training is to blame for lack of common sense, lack of knowledge of the law, and general lack of humanity.

Worse.

Police department training has been publicly acknowledged to teach police (I refuse to call them "officers") to protect themselves first and foremost. Their safety is always paramount, and that's flat out bogus. With great power comes great responsibility. You get the uniform, the badge, and the gun, so you can damn well put the public's safety first, or quit the fucking job.

They've been playing the "it's a dangerous job" card for decades, when it's not even in the top 10, and personally I'm sick of it. If it was ACTUALLY dangerous, that'd be one thing, but it's not and we know it.

Re:AI sometimes isn't perfect either

[cayenne8]

I disagree. I think you should set up the AI to always produce false positives and probably hide the percentage of the match as well. Just like a lineup it should always return the top 10 results sorted randomly regardless of how closely they match. That way the cops don't start relying on it as something that it isn't.

Being in a line up is voluntary.

Here's the thing about the cop. They are there under pressure to get a conviction, especially if the crime is public, and heinous.

They hope it is the correct person, but that doesn't always happen, and innocent people go to jail and get executed.

Ok, so scenario, my face gets pulled up false positive. I was never there, but I don't have a real valid alibi.

Witnesses, who are often very unreliable, especially if it was a violent, dangerous fast acting crime...ID's me as the suspect.

Public opinion goes against me...social media goes against me, and I get convicted on circumstantial evidence.

That is not a far fetched thing to happen, granted, hopefully rare, but not far fetched at ALL.

Now, if my name had never come up as a false positive, I would have never been on the police radar, and would have never even remotely been on the radar for a crime I didn't commit.

Look, I'm gung ho for criminals to get caught and judged. But I'm willing to let a few go free, to ensure that as close to zero innocents get convicted and have their lives ruined.

Re:AI sometimes isn't perfect either

[ranton]

Except it won't be. They'll just arrest the person and "let the courts sort it out." Which never recognizes the damage that simply being arrested by itself can cause to someone.

And that practice will be something for the ACLU to combat. But always assuming the worst possible use of new techniques and technologies is not helpful.

Re:AI sometimes isn't perfect either

[Solandri]

Yes and no. So when I have run image recognition through a neural net I get a percentage match... so it depends what the threshold for a match is set at. Is 65% considered a match or 95% or 99.9%

The posts you're replying to aren't talking about a percentage match rate. They're talking about the two possible failure modes. (A) Failing to identify a suspect's picture, and (B) misidentifying someone who is not a suspect as the suspect. If you're only using the software to weed out "obvious" not-a-match photos, then type B failures are perfectly acceptable.

That is, you're not using the software to try to find the suspect, you're just using it to reduce the number of photos that a human has to look through manually. Doesn't matter if the match rate is 65% or 95% or 99.9%. As long as the suspect isn't in the 65%, 95%, or 99.9% of photos which are rejected (type A failure), it's a success. If there's a racial bias (actually it's a skin tone bias, nothing to do with race, just that different races tend to have different skin tones), it just means a human has to look through a larger percentage of a pile of photos trying to find a black suspect, than for a white suspect. i.e. Black suspects will need to be identified manually by a human more often than by the facial recognition algorithm. Precisely the opposite of what TFA implies.

Re:AI sometimes isn't perfect either

[Uberbah]

And that practice will be something for the ACLU to combat. But always assuming the worst possible use of new techniques and technologies is not helpful.

No assumptions necessary - any facial recognition is going to produce false positives, which will land innocent people in jail and some in prison. Then there's the Orwellian nature of using this technology to track everyone, everywhere.

So what would be helpful, is to burn this to the proverbial ground before it takes off, before Brandon Mayfield's become commonplace, but via facial recognition.

https://www.theregister.co.uk/...

Muslim-convert Brandon Mayfield spent 17 days in detention after an FBI Lab wrongly linked him to prints recovered by Spanish police investigating the 11 March terrorist outrage. US authorities matched digital images of partial latent fingerprints obtained from plastic bags that contained detonator caps to Mayfield, leading to his arrest.

Re:AI sometimes isn't perfect either

[jeff4747]

But always assuming the worst possible use of new techniques and technologies is not helpful.

This is a tool for the police to arrest more people. It will be used in the worst possible way.

Take field drug test kits as an example. So incredibly unreliable that they are not admissible in court.....but you can use them to arrest someone. And with their incredibly high false positive rate, they arrest a lot of people. As an added bonus, the people arrested via this method don't have legal recourse for the false arrest. The cop had probable cause due to the drug test kit.

So no, it is a terrible idea to wait until the police start with the worst possible use.

Re:AI sometimes isn't perfect either

[strikethree]

But always assuming the worst possible use of new techniques and technologies is not helpful.

Not exploring the worst possible use of new techniques and technologies is not helpful. I would go so far as to call it willful ignorance. But, that is just me. I absolutely do agree that making the worst case the default assumption to be not helpful, but if you do not examine the worst case... just, wow.

Re:AI sometimes isn't perfect either

[Agripa]

They aren't going to arrest 15000 people a day so there is no "cost in civil liberties involved to those falsely flagged" nor are they going to arrest 1000 people but it could help them quickly look at those 1000 people from a distance versus having to do the impossible job of trying to look at all 300k people.

Since they automated facial recognition, maybe they can automate arresting suspects. Or suspend drivers licenses, passports, bank accounts, etc. until they arrest themselves or pay the fine.

Garbage In, Gospel Out!

[Thud457]

I find it hard to believe that only 28 members of Congress are criminals. This AI needs to go back for reeducation.

Confidence threshold

[DredJohn]

Increasing the confidence threshold would probably have reduced the 5% error rate.... From the article: The A.C.L.U had used the system’s default setting for matches, called a “confidence threshold,” of 80%. That means the group counted any face matches the system proposed that had a similarity score of 80% or more. Amazon recommended that police departments use a much higher similarity score — 95% — to reduce the likelihood of erroneous matches.

Re:I expect inevertant programmed racial bias.

[Anonymous Coward]

I'll bet the racial bias was in the the criteria they were compared to: mug shots. The mug shots were skewed so there was more matches that were skewed. AI / computer recognition is only as good as the dataset you train it with, here they trained with a biased sample because they used mug shots from a biased justice system.

I note that the NY Times. . .

[Salgak1]

. . . . does not appear to show the mugshots they reportedly matched. That would be a critical point in the argument. I've seen multiple "near-clones" of people over the years: it's entirely likely that the Congresscritters have some as well. . .

Additionally, ARRESTED does not make one a criminal. Conviction does. The wrong people get arrested all the time: cops are FAR from perfect. . .and like Slashdot, they like simple solutions to their problems....

Re:I note that the NY Times. . .

[Whorhay]

I agree with you that arrested does not equate to being a criminal, but I've also met plenty of people who don't agree with that. Getting wrongly arrested can seriously screw with a persons life both in the long and short term. Missing a single day of work, or even being significantly late can lose a person a job. On top of that there is the cost of paying bail or a bail bondsman, and possibly a lawyer. And god help you and your family if you end up wrongly convicted.

You're absolutely right that cops aren't perfect and like simple solutions. Which should make it incredibly obvious that giving them broad access to facial recognition technology is a bad idea. Our crime rates are sufficiently low enough that something like this will easily cause more harm than good in our society.

Re:I expect inevertant programmed racial bias.

[bluefoxlucid]

There are six primary face shapes. Some argue that there are seven or nine; and some folks say there are approximately twenty-six faces and everyone is easily-confused with everyone else in one of those groups.

Black people have less dynamic range. Black people are black because their skin contains more melanin. White people can darken areas of their skin by producing more melanin, such as by tanning and freckles.

Black people reflect less light, and detail is less-obvious. Humans are pretty good at compensating; machines have trouble finding borders due to lack of contrast. Remember a human can see a face as a face, but then notice a pore as a pore and identify every fine detail as a separate aspect if we so choose; machines have to be taught to do that, then have to learn how to make the distinctions correctly.

Result? Facial recognition sucks. People train facial recognition by looking at small sets of faces and trying to find a way to differentiate specifically to that set; it continues to suck in practice on larger sets. Facial recognition sucks really bad if you're black because it thinks there are only five black people.

I can't readily recognize faces. I actually won't recognize people I know; and then I'll fail to recognize them entirely if they change their hair. It takes a few tries, and I mostly go by things like gait, body composition, behavior, voice, and so forth. I'm really sensitive to voice, and can identify someone's voice even if they manipulate it to disguise themselves--and can point out that someone sounds exactly like another person I know while also differentiating their voice with perfect reliability.

Let that sink in: I can't recognize you in real life even if I have an endless array of video and photographs for a few weeks or months ahead of time; but I can recognize your voice immediately from one sample, no matter what you do to distort it, short of electronic manipulation of a recording. Every voice is distinct. Twins have distinct voices.

Computers are known to be better at facial recognition than humans.

Facial recognition is hard.