Slashdot Mirror
When Sentencing Criminals, Should Judges Use Closed-Source Algorithms? (technologyreview.com)
Some judges in America have recently started using a closed-source algorithm that predicts how likely convicts are to commit another crime. Mosquito Bites shared an article by law professor Frank Pasquale raising concerns about the algorithms:
They may seem scientific, an injection of computational rationality into a criminal justice system riddled with discrimination and inefficiency. However, they are troubling for several reasons: many are secretly computed; they deny due process and intelligible explanations to defendants; and they promote a crabbed and inhumane vision of the role of punishment in society...
When an algorithmic scoring process is kept secret, it is impossible to challenge key aspects of it. How is the algorithm weighting different data points, and why? Each of these inquiries is crucial to two core legal principles: due process, and the ability to meaningfully appeal an adverse decision... A secret risk assessment algorithm that offers a damning score is analogous to evidence offered by an anonymous expert, whom one cannot cross-examine... Humans are in charge of governments, and can demand explanations for decisions in natural language, not computer code. Failing to do so in the criminal context risks ceding inherently governmental and legal functions to an unaccountable computational elite.
This issue will grow more and more important, the law professor argues, since there's now proprietary analytics software that also predicts "the chances that any given person will be mentally ill, a bad employee, a failing student, a criminal, or a terrorist."
When an algorithmic scoring process is kept secret, it is impossible to challenge key aspects of it. How is the algorithm weighting different data points, and why? Each of these inquiries is crucial to two core legal principles: due process, and the ability to meaningfully appeal an adverse decision... A secret risk assessment algorithm that offers a damning score is analogous to evidence offered by an anonymous expert, whom one cannot cross-examine... Humans are in charge of governments, and can demand explanations for decisions in natural language, not computer code. Failing to do so in the criminal context risks ceding inherently governmental and legal functions to an unaccountable computational elite.
This issue will grow more and more important, the law professor argues, since there's now proprietary analytics software that also predicts "the chances that any given person will be mentally ill, a bad employee, a failing student, a criminal, or a terrorist."
No
Long answer Nooooooooooooooo!!!!!!
This is as crazy as having closed source software determine guilt (think breathalyser, radar gun etc) -- which means it is probably inevitable.
In the legal system, on every encounter with an object running code, (firmware or otherwise) an accused party should move for discovery and expert analysis of said code, and how it was developed.
Were they using best-practices? Running valgrind? (lets see the exception list) Address-sanitizer, theread-sanitizer, clang static analyser? etc..
Do they have unit tests? Do they track code coverage of those tests? (it had better be 100% across the board)
Even using all the latest tools and with plenty of unit tests with good coverage, there are still bugs.
Ian Ameline
Government, as being tax funded, should use entirely open source software and open formats. Anything otherwise is favoring certain corporations (Microsoft formats for example) or having potential to be abused (FBI backdoors in government software).
We the people elected them. We the people should be able to inspect them.
Imagine a judge just picks a number out of the air based on his own experience, opinions and yes, prejudices.
That's using a closed source algorithm, except it runs on a wet carbon platform rather than silicon.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Use compassion and moral guidance. Remember, a prison is supposed to be about rehab, not outright punishment. The Nordic countries do it right.
So rise up, all ye lost ones, as one, we'll claw the clouds.
Government needs to be able to explain its decisions about citizens to the public. Private organizations don't.
So, proprietary analytical software that cannot justify its decisions is not acceptable for courts of any kind, for policing, for awarding government contracts, for public schools, etc.
It is acceptable for businesses, medical providers, employee evaluations, private schools, etc.
Absolutely it should. Conincidentally, I legally changed my name to 1;drop table.
As a lawyer, I will admit most lawyers are not very mathematically savvy, except when it comes to calculating legal fees, then they turn into Srinivasa Ramanujam.
Frank Pasquale left out a couple of details in his opinion piece. First, these algorithms are only used in determining sentences, not to determine guilt. At that point in the trial guilt has been determined beyond a reasonable doubt. At the sentencing phase rules of evidence do not apply anymore and almost anything goes. That's why the prosecutor puts crying victims on the stand. There have been two supreme court cases (Malenchik vs Indiana and Loomis v. Wisconsin) that challenged the use of algorithms in sentencing and both upheld (in the later with some minor restrictions) that these algorithms can be used in sentencing. The conditions in general are that the algorithm has to be scientifically sound. That was the case in both cases that were challenged as there is existing peer-reviewed literature that examined the algorithms. Frank neglected to mention that. The secrecy of the algorithms is a consequence of patent and copyright law btw. The algorithms in these cases are a scoring function. Math is not (and should not) patentable or copyrightable. In this case the consequence is that the manufacturers only recourse is to keep it a trade secret. That could be solved better, but in my opinion people shouldn't get their hopes up that there's some exploitable loophole in the algorithm or something.
We can debate whether assessments (actuarial prediction instruments) should be used in sentencing or criminal justice. I’m very much in favor as it does reduce bias and leads to reproducible results. It’s much easier to control for biases in decision making with statistical methods than it is to control or fix bias in humans. Does anybody believe that human judgement is less biased? You can read up on the work of Paul Meehl who spend his lifetime showing that even simple assessment tests outperform the judgment of trained clinicians. Part of the sentencing is taking into consideration how likely the perpetrator is to commit a new offense. Humans suck at making predictions and estimating probabilities. This is no different in criminal justice.
Let me end this with pointing out some of the positive change that systems like this have brought: early release from incarceration. Low risk prisoners are more frequently released early (not just from overcrowded California prisons when ordered to do so by a federal judge), and then put on probation/parole. And work out well it did: http://time.com/4065359/califo... The expected crime wave from federally mandated early release didn't materialize. In my opinion thanks to these prediction models.
There are many things wrong with policing and criminal justice in the US, but the move to what’s generally referred to as “evidence based practices” (incl. actuarial prediction instruments) has been pretty positive. The great part is that both Dems and Reps are behind the idea of risk assessments so we might actually see some change for the better.
With a lot of caveats, of course, but basically an individual has the right to 'an explanation of the decision reached after [algorithmic] assessment'. Described in http://fusion.kinja.com/eu-cit...
davecb@spamcop.net