Slashdot Mirror
When Sentencing Criminals, Should Judges Use Closed-Source Algorithms? (technologyreview.com)
Some judges in America have recently started using a closed-source algorithm that predicts how likely convicts are to commit another crime. Mosquito Bites shared an article by law professor Frank Pasquale raising concerns about the algorithms:
They may seem scientific, an injection of computational rationality into a criminal justice system riddled with discrimination and inefficiency. However, they are troubling for several reasons: many are secretly computed; they deny due process and intelligible explanations to defendants; and they promote a crabbed and inhumane vision of the role of punishment in society...
When an algorithmic scoring process is kept secret, it is impossible to challenge key aspects of it. How is the algorithm weighting different data points, and why? Each of these inquiries is crucial to two core legal principles: due process, and the ability to meaningfully appeal an adverse decision... A secret risk assessment algorithm that offers a damning score is analogous to evidence offered by an anonymous expert, whom one cannot cross-examine... Humans are in charge of governments, and can demand explanations for decisions in natural language, not computer code. Failing to do so in the criminal context risks ceding inherently governmental and legal functions to an unaccountable computational elite.
This issue will grow more and more important, the law professor argues, since there's now proprietary analytics software that also predicts "the chances that any given person will be mentally ill, a bad employee, a failing student, a criminal, or a terrorist."
When an algorithmic scoring process is kept secret, it is impossible to challenge key aspects of it. How is the algorithm weighting different data points, and why? Each of these inquiries is crucial to two core legal principles: due process, and the ability to meaningfully appeal an adverse decision... A secret risk assessment algorithm that offers a damning score is analogous to evidence offered by an anonymous expert, whom one cannot cross-examine... Humans are in charge of governments, and can demand explanations for decisions in natural language, not computer code. Failing to do so in the criminal context risks ceding inherently governmental and legal functions to an unaccountable computational elite.
This issue will grow more and more important, the law professor argues, since there's now proprietary analytics software that also predicts "the chances that any given person will be mentally ill, a bad employee, a failing student, a criminal, or a terrorist."
No
Long answer Nooooooooooooooo!!!!!!
This is as crazy as having closed source software determine guilt (think breathalyser, radar gun etc) -- which means it is probably inevitable.
In the legal system, on every encounter with an object running code, (firmware or otherwise) an accused party should move for discovery and expert analysis of said code, and how it was developed.
Were they using best-practices? Running valgrind? (lets see the exception list) Address-sanitizer, theread-sanitizer, clang static analyser? etc..
Do they have unit tests? Do they track code coverage of those tests? (it had better be 100% across the board)
Even using all the latest tools and with plenty of unit tests with good coverage, there are still bugs.
Ian Ameline
Government, as being tax funded, should use entirely open source software and open formats. Anything otherwise is favoring certain corporations (Microsoft formats for example) or having potential to be abused (FBI backdoors in government software).
We the people elected them. We the people should be able to inspect them.
it means a "diminutio" of defendant rights... if it will be used extensively we (well... you in the US) will have unfair processes
This is tricky because most people don't get the issues related to open/closed source. Remember that the majority of people now have smart phones and tablets with their walled garden of app ecosystems. The idea of libre software simply does not occur to them. They likely didn't grow up learning computers in the era when source code commonly shipped with the computer and you could inspect/tweak it if something went wrong.
That said, I firmly believe that any software which is used in determining innocence/guilt, sentencing, criminal culpability, civil liability, etc., should be open source. Sentencing is an interesting one, though, because in that case I think it would be acceptable (assuming there are no other alternatives) to settle for a closed source product with a public specification and comprehensive set of test cases which satisfactorily demonstrate that the software performs in conformance to the written spec.
There also needs to be a contractual commitment on the part of the vendor to update the software (and spec and test suite) in a timely manner when unexpected results are encountered for previously unspecific input conditions or edge cases, etc.
That said, if you have a strong opinion regarding this, then you are welcome to start an open source project, form a lobbying organization, and get your open source solution adopted in as many jurisdictions as you can manage. Given the way that governments work, you can probably form a firm that also provides paid support for the product.
Not even on the submitted stories page?
"I'm sorry, Citizen - the reason for your incarceration is not available at your security clearance level. Please report to you closest walled garden for mandatory fun."
-The Computer.
Imagine a judge just picks a number out of the air based on his own experience, opinions and yes, prejudices.
That's using a closed source algorithm, except it runs on a wet carbon platform rather than silicon.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Perhaps the secret is that the algorithm is a black box method, and no-one actually knows exactly why it scores the way it does?
Neural networks are all the rage now, and they are one of the absolutely worst when it comes to transparency and intelligibility.
Had the company worked with rule-based classifiers or other transparent machine learning I'm sure they'd be happy to show just how smart they are, snippet by snippet. After all, we only want to avoid showing others when we're doing something simple or stupid (or unethical, which might also be the case here).
Does nobody else really see how sick this is?
Just think how much money the state will save on legal costs if convicts are unable to appeal their sentencing. Sure, there's the chance of giving people a few more years in prison than is really appropriate, but has there ever been a time when the public called for more rights for convicted criminals?
AFAIK software have no legal right.
It should be the responsibility of the lawyer using the software. Closed or open source doesn't matter, what is important are the results and the legal entity backing them.
If a lawyer uses voodoo magic to find culpability, why not, as long as the facts are right. If not, he better get a good explanation. "magic" won't cut it. Same for the software. The good thing with open source is that it is easier to explain results when challenged.
I'd like opensource everywhere but I don't think it is the main point here. There is opensource bullshit too.
and let that be a lesson to all, because we know who you are, what you did, and we're coming.
It is obvious that the answer should be no.
Sentencing decisions should include the reasoning why the decision would be motivated.
"Because The Software Says So" is no more valid than "Because I Say So".
I think that if software is being used, not only should the software be Open Source, the sentencing document should include a mathematical explanation of the algorithm that the program uses together with the input parameters about the judged and various statistics that the decisions leans on.
"We mustn't be caught by surprise by our own advancing technology" -- Aldous Huxley
It's clear as day whom has been writing these, what they'll be indicating, and why they'd skew them in this way.
Long jail sentences equals free slave workers and great big contracts from the state. Profitable. Very, very profitable.
When sentencing convicted criminals, judges should, constrained by the minimum and maxim sentences prescribed by law, and in consideration of all known mitigating circumstances, give sentences in conformance with their JUDGEMENT as to the social effects and the effects on improving the criminal's future biography.. That is why they are called JUDGES, after all. For instance, a longer sentence may be chosen to be given to a criminal who has no credible family or community support, and a shorter one who possesses such relationships, if the judge believes this will improve the possibility of rehabilitation, and/or reduce the possibility of recidivism. People who do this work we call "Judges". We choose them, and charge them to judge what is best to do.
and now excuse me while I rewatch Gattaca.
Others have already shown why the answer to this question is No.
I would like to point out why No is always the answer to questions like this.
If the answer is Yes, the editor rephrases the question as a statement. I.E. no one writes a headline "Did Trump get elected?" Instead we write "Trump wins."
We only use a question when the real headline would be boring, so we re-write it to sound more exciting and make it a question so we can't get sued. So if we know that Trump has not resigned, but we want people to click/read our article, we write "Has Trump RESIGNED?" The answer is no.
Same thing with this question. The answer is obviously no, we should never trust anyone that says "I refuse to tell you why, but I think the judge should do X." You want us to listen to your legal argument then you HAVE to back it up with your thinking.
excitingthingstodo.blogspot.com
Use compassion and moral guidance. Remember, a prison is supposed to be about rehab, not outright punishment. The Nordic countries do it right.
So rise up, all ye lost ones, as one, we'll claw the clouds.
People's irrational belief in the accuracy of technology.
"Garbage in; garbage out" has been forgotten by the current generations.
AND - you can just bet some very sharp person will figure out how to game the algorithm.
CONSULTANT: Let's see, several drug offenses, rape, homicide - order the defendant to go to Church and "find Jesus".
COMPUTER: "200 hours of public service and time served."
First, it completely violates the concept of being able to confront your accusers.
In this case accusers for what you might do in the future.
Second, I acknowledge that what goes on inside a judge's head at sentencing is proprietary to the judge.
But usually, there is a feedback mechanism here it not re-electing the judge.
I don't see such a feedback mechanism there.
In fact, it kind of eliminates the judge's feedback mechanism because he can hide behind the machine for the calls he makes.
Overall, using a published algorithm to pick sentences could be a good thing.
If it's based on science and can be show to statistically improve the criminal justice system.
But Algo is unlikely to be able to handle the wide variety of circumstances it is confronted with.
To partner with Algo, the judge needs to understand what Algo's limits are so he can make rational choices as to when to use and not use the results.
A proprietary mechanism does not permit this.
I got a number of machine learning courses at uni, one specifically on neural networks. The professor used specifically this as an example where neural networks and other "black box" machine learning solutions would be a terrible idea, because while they may fit the data quite well they don't offer much in terms of explanatory value. Seeing them creep into this type of application should be very troubling for anyone with a working understanding of AI.
This is one step further from being locked up for considering committing a crime. This locks you up for longer because some supposed Source Of Truth (and that is assumed on blind faith) marks you as someone who might think of committing a crime in the future so you get punished before the fact.
Pour encourager les autres?
"Wait. Something's happening. It's opening up! My God, it's full of apricots!"
Government needs to be able to explain its decisions about citizens to the public. Private organizations don't.
So, proprietary analytical software that cannot justify its decisions is not acceptable for courts of any kind, for policing, for awarding government contracts, for public schools, etc.
It is acceptable for businesses, medical providers, employee evaluations, private schools, etc.
Absolutely it should. Conincidentally, I legally changed my name to 1;drop table.
Pick ten universities in the US (since this is US sentencing) with backgrounds in criminology and sociology. Have nine of them create programs to predict likelihoods of recidivism, etc based on a set of common inputs. Have the tenth do work on the aggregation piece and overall project management. When you do sentencing you throw out the high and the low and take the average of the rest. Hopefully this will help mitigate bias, computational mistakes, etc. The code for all the algorithms is available for per review as well as the test sets. Give the nine universities four year grants and then rotate to another nine, give the project management one a six year grant. Set it up so it's staggered and new universities are rotated in. A university is eligible again after twelve years or so to minimize the number of people involved in the old project. I think the most important component would be structuring it so that one particular school of thought can't hijack it. You don't want all the schools in one state or even in the same geographic region. You should try to avoid having the majority of the projects lead by professors who all graduated from the same school (or worse), under the same professor(s), etc. It can and should evolve to represent our (hopefully) growing understanding of the field.
In my mind this is one of the ways our tax dollars should be used.
https://www.law.umich.edu/news... For those too busy to read the citation, the research by Professor Sonja Starr indicates men receive prison sentences that average 63% longer than women convicted of the same crime. Women are also twice as likely to avoid incarceration altogether. The paper itself: https://papers.ssrn.com/sol3/p...
So what we have here a bit of a minefield. If we are to continue this practice, the algorithm must take gender into account, and purposely hand out sentences that are much less for females than males. This would become glaringly obvious during testing. Input identical parameters except gender. Codified and simply proven gender discrimination, built right into the program will result in a pretty short gender discrimination trial.
But if gender is not taken into account, and suddenly females get identical sentencing, you can bet there will be a lot of legal agitation from a different group.
I see it as an entertaining thing to watch unravel. I would expect to consume much popcorn.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
While the open-sourcing (for critical purposes) of such an algorithm is all well and good in theory, the fact is it would then be gamed for advantage.
If, for example, there is a statistically-significant correlation between a defendant showing up at a trial with a facial tattoo re-committing later, any intelligent defense attorney is going to either cover it with make up or get it removed.
-Styopa
Any algorithm can be gamed. Open sourcing the algorithm lets those with good legal advice game the system. This is not a "good thing".
If a DRM is injected to a Personal Computer as a trojan horse then what does the DRM?
We need to know what algorithms do this DRM, right?
It seems to be that we expect that DRM commits unauthorized actions against the buyer of the Personal Computer.
1. Is the case, or has it ever been, widely publicized? Yes goto 2. No goto 3. .5; Is the convict female? Yes SF := SF*.65 := SF+ (skin color-5)/20
2. Is the convict viewed favorably by the public? Yes = Minimum. No = Maximum. Break.
3. Sentencing Factor =
4. On a scale of 1 to 10 with 1 being skin color whiter than white, and 10 being blacker than black, SF
5. On a scale of 1 to 10 with 1 being the lowest decile and 10 being the top decile of convict's wealth & income SF:= SF+(5-wealth)/20
6. 0.7 = didn't graduate high school; 0.6 high school graduate; 0.5 attended college; 0.4 graduated college with bachelors; 0.3 has a PhD in liberal arts or a Master's in STEM; 0.2 Has a PhD in STEM; 0.1 has a MD, LLD degree; 0.0 Has an active Medical or Law license. SF:= SF+(education/1.5)
7. If SF 1 Maximum, otherwise sentence = Minimum + Range*SF. Break
These are government courts, so no. The courts need to be private, with competition among them, then it would be a private decision. As it stands today - no.
MY OTHER COMMENTS
We are talking about Law, so pedantry and precision is the way to go[*] ......
By definition an algorithm cannot be "open source" or "closed source". It might be proprietary ... but that is a different thing.
Personally, I think that using some sort of Big Data / AI / Machine Learning thing to abdicate a Judge's responsibilities would seem to be the wrong way to go, particularily if you are using them to predict somone's future behaviour -- why have a Judge at all if you are going to do that?
Furthermore, prediction is really hard, particularily about the future.
It all sounds a bit Minority Report / Pre-crime to me, and so does Frank Pasquale it seems.
---
[*] What have I opened myself up to there?
Yeah, we wouldn't want a justice system that couldn't be manipulated by rich peoples lawyers.
Frank Pasquale left out a couple of details in his opinion piece. First, these algorithms are only used in determining sentences, not to determine guilt. At that point in the trial guilt has been determined beyond a reasonable doubt. At the sentencing phase rules of evidence do not apply anymore and almost anything goes. That's why the prosecutor puts crying victims on the stand. There have been two supreme court cases (Malenchik vs Indiana and Loomis v. Wisconsin) that challenged the use of algorithms in sentencing and both upheld (in the later with some minor restrictions) that these algorithms can be used in sentencing. The conditions in general are that the algorithm has to be scientifically sound. That was the case in both cases that were challenged as there is existing peer-reviewed literature that examined the algorithms. Frank neglected to mention that. The secrecy of the algorithms is a consequence of patent and copyright law btw. The algorithms in these cases are a scoring function. Math is not (and should not) patentable or copyrightable. In this case the consequence is that the manufacturers only recourse is to keep it a trade secret. That could be solved better, but in my opinion people shouldn't get their hopes up that there's some exploitable loophole in the algorithm or something.
We can debate whether assessments (actuarial prediction instruments) should be used in sentencing or criminal justice. I’m very much in favor as it does reduce bias and leads to reproducible results. It’s much easier to control for biases in decision making with statistical methods than it is to control or fix bias in humans. Does anybody believe that human judgement is less biased? You can read up on the work of Paul Meehl who spend his lifetime showing that even simple assessment tests outperform the judgment of trained clinicians. Part of the sentencing is taking into consideration how likely the perpetrator is to commit a new offense. Humans suck at making predictions and estimating probabilities. This is no different in criminal justice.
Let me end this with pointing out some of the positive change that systems like this have brought: early release from incarceration. Low risk prisoners are more frequently released early (not just from overcrowded California prisons when ordered to do so by a federal judge), and then put on probation/parole. And work out well it did: http://time.com/4065359/califo... The expected crime wave from federally mandated early release didn't materialize. In my opinion thanks to these prediction models.
There are many things wrong with policing and criminal justice in the US, but the move to what’s generally referred to as “evidence based practices” (incl. actuarial prediction instruments) has been pretty positive. The great part is that both Dems and Reps are behind the idea of risk assessments so we might actually see some change for the better.
it will predict bad politicians
What matters are future results, and that the effectiveness of whatever algorithm is used is continually measured. There's also question of whether sentencing alone can effectively reduce recidivism. The Economist has an interesting article on making America's prisons work better.
In Cobb County GA near the Braves MLB stadium, in my opinion, you are now guilty until proven innocent:
http://www.11alive.com/news/investigations/the-drug-whisperer/437061710
I no longer take people to baseball games
There's an algorithm that will break that into paragraphs for you.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Locking a person away because you think they will commit future crimes, probably.
A justice system is about objectivity. It is not about punishing someone because you dislike them. Justice is blind, not a popularity contest.
Troll is not a replacement for I disagree.
When a judge "decides" on a sentence, there can be many factors that influence the result. Not all of them are strictly relevant or objective and not all of them are easily explainable. Different judges would give different punishments for the same crime. Hell, the same judge can give different sentences for seemingly identical circumstances. When a program computes a sentence then the same data entered each time will lead to the same result.
Whether it is an open-source program that gives a result or a proprietary algorithm is irrelevant. Judges decision-making processes aren't exactly open-source, either. It is far better for sentencing to be consistent than "personal" or explainable. It removes accusations of bias (although there is still dependency on the data fed in) and makes the law and the trial process appear far more objective and fair.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
if they want it to be critequed or not. this sounds a lot like precrime but i suppose judges already do this when looking at releasing criminals i just say know why and how the alga give its results.
With a lot of caveats, of course, but basically an individual has the right to 'an explanation of the decision reached after [algorithmic] assessment'. Described in http://fusion.kinja.com/eu-cit...
davecb@spamcop.net
You cannot challenge the logic, only the result. "journalists" like to talk about the arguments, but as we can see repeatedly, only the judgment counts. The next level of appeals always substitutes their own opinions. It is NOT a formal logic proof that gets refined by examination, no matter how much they try to tell you it is.
No. No. No. No.
With all the "tough on crime" minimum sentecing laws passed over the last 20 years, Judges dont even get a say in sentencing for state legal systems...
It's all up to the prosecutor and that's how plea bargaining works in the county... Reduced the crime charged and you reduce the sentence...
99% of the time I see any type of reporting on court cases they are inaccurate and it seems that no one that hasn't been in the system really understands how the system works for most people..
Take Oregon as example.. I see local news reporters talking about Parole and people that violate going back to prison.. doesn't work that way.. Oregon hasn't had a Parole system in 30 years... it's Post Prison Supervision and you dont go back to Prison for violations, you go to jail..
There is a HUGE difference and to think otherwise is just a myth..
There seems to be a massive hang-up over courts, sentencing, and punishment. People see the sentence as a punishment when it more importantly should serve a different purpose. When we have a person proven, in court, to be guilty of various crimes perhaps we should concentrate on the other P-word, protection.
Sentencing should serve society as well as issue any required chastisement to the criminal deemed necessary. Punishment is issued in anger. This person is so bad that something nasty must be done to that person to expiate his sins. But anger is almost always a counter-productive emotion. What really needs to be done is protect society from a known flawed component. Since fellow humans are involved as miscreants, prosecution, and juries simply discarding the bad component is highly improper. Mistakes happen and we owe it to ourselves to allow an out.
However, simply turning the flawed person back into society is wrong as well. It places you and me into the danger that the flawed person will fail again. So we put the person away for a period of time designed to perhaps give the person time to heal and protect society from the flawed person. For this an algorithm for sentencing guidance is probably a good thing. AIs learn patterns nicely. What traits would a flawed person display that would indicate a high chance that defects would be healed in a short time or would take a long time to be neutralized? Let an AI figure this out from the huge volumes of existing data and tracking modern results.
I am human and experience the urge to punish the particularly odious criminals with unspeakable punishments. Then I sit back and ask, more seriously, "What level of danger is this person to society for what period of time in confinement sequestered away from society for society's own protection." That should define duration. Best case the confinement should be fairly benign. Worst case some form of spanking is required and the conditions of the confinement are adjusted as the chastisement determined to be appropriate,
Protect society first, then if you must and it matters, punish the criminal.
{^_^}
Presumably there is some sort of machine learning being done here to predict re-offending. It would be straight forward for the creators to publish cross validation rates. I.e. Train their algorithm on 4/5 of the data, keeping 1/5 separate, and then seeing how accurate the predictions are on the 1/5. Repeat for each of the different fifths, and record the average accuracy overall. If the accuracy in predicting reoffending is say, 75% (I'll be that's higher than the real rate) then it'll be wrong one time in four, and the person being judged, as well as the judge, should have a right to know that. You would also want to look closely at the false positive and false negative rates. It's easy to create a predictor, it's hard to create an accurate predictor.
The entire legal system can currently be gamed by how much you pay your lawyer. so there really would be no change except for savy people who dont have the money for a lawyer could then have a better defense for themselves.
Your counterpoint is invalid.
Here is the URL for it for free (as in beer): https://www.vanderbiltlawrevie...
First, I didn't say the Nordic countries were doing anything right, that was somebody else.
No, you're right, I misspoke. I smashed two things together in my head and didn't clearly articulate them. Sorry, my bad.
The problem is that once they decided that, BOP no longer had to pretend to try to rehab inmates. Programs that were not seen as immediately beneficial to BOP were gutted. Because of this quality of life deteriorated markedly (inmates suffer tremendous boredom and programming is one of the ways they fight it; less programming = more boredom = more bad shit like gambling, fights, and yard politics).
Finally, BOP Minimums (called 'camps') are NOTHING like nordic prisons. Inmates sleep in open bays, the food sucks, there is little recreation opportunity. The only things inmates like about camps is the contraband. It is easy to get cellphones, alcohol, pornography, drugs, and outside food. Also, if your woman lives nearby or can travel you can get laid. Alternatively, there are hookers that service some of the camps, but I hear that's rare. In real life, some guys intentionally get caught doing something to get sent back up to the Low. Camps are not equivalent to nordic prisons.