Slashdot Mirror

← Back to Stories (view on slashdot.org)

Scientists Create AI Program That Can Predict Human Rights Trials With 79 Percent Accuracy (theverge.com)

Posted by BeauHD on from the moral-right-and-wrong dept.

An anonymous reader quotes a report from The Verge: Computer scientists have created an AI program capable of predicting the outcome of human rights trials. The program was trained on data from nearly 600 cases brought before the European Court of Human Rights (ECHR), and was able to predict the court's final judgement with 79 percent accuracy. Its creators say it could be useful in identifying common patterns in court cases, but stress that they do not believe AI will be able to replace human judgement. As described in a study published in the journal PeerJ Computer Science, the AI program worked by analyzing descriptions of court cases submitted to the ECHR. These descriptions included summaries of legal arguments, a brief case history, and an outline of the relevant legislation. The cases were grouped into three main violations of human rights law, including the prohibition on torture and degrading treatment; the right to a fair trial; and the right to "respect for private and family life." (Used in a wide range of cases including illegal searches and surveillance.) The AI program then looked for patterns in this data, correlating the courts' final judgements with, for example, the type of evidence submitted, and the exact part of the European Convention on Human Rights the case was alleged to violate. Aletras says a number of patterns emerged. For example, cases concerning detention conditions (eg access to food, legal support, etc.) were more likely to end in a positive judgement that an individual's human rights had been violated; while cases involving sentencing issues (i.e., how long someone had been imprisoned) were more likely to end in acquittal. The researchers also found that the judgements of the court were more dependent on the facts of the case itself (that is to say, its history and its particulars) than the legal arguments (i.e., how exactly the Convention on Human Rights had or had not been violated).

41 of 83 comments (clear)

  1. Dang it! by Anonymous Coward · · Score: 1

    Here I am, only weeks away from completing my graduate degree in Predictive Human Rights Jurisprudence, and some jerkhead has built an artificial intelligence to take away my job!

    I knew I should have stayed with medicine... we could have a cure for diabetic mice by now, if I hadn't listened to my advisor.

  2. data analysis or "AI"? by sittingnut · · Score: 2

    why do people slap "AI" label on unnecessarily? publicity?

    "The AI program then looked for patterns in this data, correlating the courts' final judgements with, for example, the type of evidence submitted ...a number of patterns emerged ...For example, cases concerning detention conditions ... more likely ...cases involving sentencing ...more likely"

    this is mere data analysis. or is that what so called "AI" amount to?

    and this,
    "judgements of the court were more dependent on the facts of the case itself "
    duh?!
    how smart of so called "AI" to find that out?

    1. Re:data analysis or "AI"? by ShanghaiBill · · Score: 3, Insightful

      this is mere data analysis. or is that what so called "AI" amount to?

      "Mere" data analysis is when a human looks at the data and tries to find patterns. But it is "AI" when the algorithm is open ended, and finds it's own patterns and correlations. That is "machine learning" is certainly a branch of AI.

    2. Re:data analysis or "AI"? by ShanghaiBill · · Score: 1

      Even if it is predictive, it relies on the 600 case training set being representative of future cases.

      Human intelligence also relies on the future being similar to the past.

    3. Re:data analysis or "AI"? by Anonymous Coward · · Score: 1

      5 years ago we called that data mining.

      I skimmed the paper. They converted the cases documents into word n-grams, did some clustering, made some type of n-gram by n-gram matrix that I've never learned about, did some more clustering, then fed those clusters into a SVM.

      Data mining is a much better term, but I guess that's not fashionable anymore. In 5 more years is everything going to be about multidimensional awareness (MA)?

      *All MA rights belong to Slashdot's ACs. Fear my pending patient.

    4. Re:data analysis or "AI"? by ShanghaiBill · · Score: 1

      ... then fed those clusters into a SVM.

      SVMs are used in machine learning, which is a branch of AI.

      Data mining is a much better term

      "Data mining" and "machine learning" are two orthogonal concepts. You can do data mining with or without machine learning. Machine learning can be used for many things besides data mining.

    5. Re:data analysis or "AI"? by TuringTest · · Score: 1

      AI has always meant that in academy. Maybe you were using some other definition from, say, literature?

      --
      Singularity: a belief in the "God" idea with the "demiurge" relation inverted.
    6. Re:data analysis or "AI"? by AthanasiusKircher · · Score: 1

      "Mere" data analysis is when a human looks at the data and tries to find patterns. But it is "AI" when the algorithm is open ended, and finds it's own patterns and correlations.

      I certainly agree that this falls under the classification of "AI" as a field. I'm guessing that part of the concern is also whether what was done here qualifies as "intelligence" rather than just a slightly more advanced algorithm for processing data.

      Most research studies these days use fairly complex statistical computations -- often, lamentably, that the researchers themselves don't fully understand (or at least don't fully understand the limitations of). So, basically by the time many researchers are looking at the "data" to search for correlations, the raw data has already been processed in rather non-transparent ways. Relatively few people are staring at the raw data and saying, "Hmm... there's a lot of high X values here -- let's run a test to see whether X correlates with Y." Instead, they either decided they were already going to run that correlation before the test already began, or they do the p-hacking thing where they just run dozens of different statistical tests and see whether anything "shakes out."

      Either way, humans are rarely DIRECTLY finding their own "patterns and correlations" anymore. They throw a bunch of stuff into a statistical package and see what pops out. The "AI" algorithms used in the present paper are certainly putting an extra layer of processing on top of that, but ultimately they're just doing a few more steps of statistical analysis and spitting out the patterns that emerge. The algorithms just tend to emphasize and weight certain things in the dataset a little more to make patterns pop out more easily.

      So, yeah, it's automating pattern-finding a bit more. But I can also see the point that it's really an extension of data analysis... ultimately the patterns that come out of this system aren't really meaningful. For example, according to the top-rated clusters of topics, judgments for one of the rights are highly likely to depend on whether the word "July" is found -- good for your chances! -- vs. whether "June" or "June applicant" or "dated June" is found -- which apparently causes you to be more likely to lose the case!

      Obviously that's ridiculous, but it shows the similarity in this sort of analysis to what you might get with a simpler statistical package that just tries out dozens of correlations. In both cases, the computer is just weighting the patterns it finds -- using algorithms dictated by humans -- and then spitting out a lot of nonsense and some things that look more interesting. It's then up to the humans to determine which are the interesting bits.

      While I know the term AI is used for this stuff, personally I'll reserve the term "artificial intelligence" for a system that actually has some fairly sophisticated threshold for realizing when the output is nonsense vs. when it's likely to be more interesting, and that determination isn't just a hard-coded aspect of the algorithm in question. Then the system would actually be doing something akin to "judgment," which implies "intelligence," rather than just being a more sophisticated pattern-finding stats package.

  3. AI by 110010001000 · · Score: 2

    God, not another "AI Program". We used to just call them programs.

    1. Re:AI by ArylAkamov · · Score: 4, Funny

      But it's twenty-sixteen! We future now!

      RC aircraft are DRONES
      Predictive models are ARTIFICIAL INTELLIGENCE
      Computer programs are APPS (APPING APPERS)!
      A can of wd-40 and a lighter is an LOL EPIC FLAMETHROWER!!1 ...The future sucks.

    2. Re:AI by Plus1Entropy · · Score: 1

      ... What's the A stand for?

      --
      Only crack the nuts that crack. You don't put the ones that don't crack in the sack.
  4. Actually no. by goombah99 · · Score: 3, Informative

    You can't even tell if it's better than a coin toss. For this statistic to have any meaning at all you need to know the frequency the plaintiff wins. For example, let is suppose that the plaintiff wins in 79% of cases. Then an "AI" that merely always guess the plaintiff won would be correct in 79% of cases.

    In fact given that it's unlikely the outcome is 50:50, then one would expect that such a dumb algorithm would be correct more often than not just by always guessing one side. It would therefore take very little extra "intelligence" so boost it over the top. In particular such intelligence could be simply an artifact of the data set. As an example suppost the data set contained 10% of plaintiffs whose names started with R. If this group of people won more often than the avergage, then simply learing to guess "win" anytime there was a plaintiff with an "R" name would improve the test. This is true even if you split the data up into cross validation sets, as the bias for "R" will persist on any randomly Chosen subset as well.

    thus the results probably are meaningless. Certainly the article is.

    --
    Some drink at the fountain of knowledge. Others just gargle.
    1. Re:Actually no. by ShanghaiBill · · Score: 1

      You can't even tell if it's better than a coin toss. For this statistic to have any meaning at all you need to know the frequency the plaintiff wins.

      Indeed. For instance, 99% of Japanese criminal cases end in conviction. So you could predict the result with 99% accuracy using the following algorithm:

      1. Flip a coin
      2. If it is heads, the defendant is guilty.
      3. It it is tails, go to step 1.

    2. Re:Actually no. by Shane_Optima · · Score: 2

      That's an interesting statistic of course, but it's a little less shocking or worrying than one might initially assume. For instance, I read that one of the reasons is that Japanese prosecutors will very rarely take something all the way to trial unless they are nearly positive they'll win--it's therefore as much a symbol of Japanese perfectionism / aversion to failure as anything else. Also, there's a lot of soft power in Japanese culture, so presumably they have other ways to pressure suspected criminals to stay in line in non-slam dunk cases... or maybe they have a lot of plea bargains, not sure. I never looked that far into it.

      The statistic that over 99% of people in modern Russia are convicted is more bluntly worrying to me. One might've hoped that, given their history, a distrust of authority would have taken root at some point, but this does not seem to be the case (or if it is, it's exceeded by the distrust of the accused.) And there's no use appealing to an overall lower trial rate or crime rate to explain that one.

    3. Re:Actually no. by Shane_Optima · · Score: 1

      err, over 99% of defendants*

    4. Re:Actually no. by imidan · · Score: 1

      For example, let is suppose that the plaintiff wins in 79% of cases. Then an "AI" that merely always guess the plaintiff won would be correct in 79% of cases.

      What in the world...?

      It's trivially obvious that the way you would build such a model is to take a set of cases, subset them, identify the predictors of outcome (none of which is who won) in the subset, regress (in some way) the predictors with the outcomes in the subset, and then attempt to predict the outcome of the cases outside the subset using the function derived using the subset.

      Of course you don't just count up the number of times the plaintiff wins, divide it by the total number of cases, and then call the probability of any given plaintiff winning 79%. That's just stupid. The point is that after training on a subset of cases, the algorithm predicts the outcome of other cases correctly 79% of the time.

      The right question to ask is how representative the sample is and how widely the result applies.

    5. Re:Actually no. by tgv · · Score: 2

      Sorry, but you're wrong. The remark is about the performance. 79% means nothing without knowing the baseline of an uninformed method. I think you can agree that a coin toss will produce the proper result in 50% of all cases. So if the performance of a system on a binary choice is 50%, it's as good as a coin toss, no matter how it's implemented. Suppose you make a system that always prints "plaintiff wins". Then its performance will be the actual win rate for the plaintiff. If that happens to be 79%, the system's performance is 79% without any knowledge.

      How representative the sample set is, is another question all together.

      BTW, the actual numbers for 2015 are

      Refused: 2930
      Granted: 3433
      Denied: 588
      Total: 6951

      So granted is 85% of all cases. So a system just printing "granted" will perform better (if refused is left out of consideration).

    6. Re:Actually no. by SharpFang · · Score: 1

      And considering "Refusal" is usually due to formal considerations that are well defined, the system should be able to predict refusal with a very high accuracy (...actually, the only inaccuracy would be human (clerical) error, when a case is wrongly passed or refused despite meeting or failing to meet the formal requirements) - and as result, with a system that has, say, 99.5% accuracy of determining between "Refused/Deliberated" (say, 0.5% of cases are wrongly refused or wrongly put under deliberation) then that makes "Denied" a 7.5% of all cases, so the system would be accurate some 92% of time telling either "Refused" or "Granted" basing on formal parameters of the application and discounting any actual legal/moral content, and never once serving "Denied".

      --
      45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
    7. Re:Actually no. by wonkey_monkey · · Score: 1

      That's the point.

      --
      systemd is Roko's Basilisk.
    8. Re:Actually no. by goombah99 · · Score: 1

      Wrong. Read the article. It was a binary classifier. It doesn't matter if you Crossvalidate (subsetting) if the baseline frequencies are 79% for one of the classifications.

      --
      Some drink at the fountain of knowledge. Others just gargle.
    9. Re:Actually no. by Shane_Optima · · Score: 1

      Yeah, I guess a key detail I forget to examine here is the fact that they're predominantly judge trials in Russia. It all more or less makes sense now, but is no less disheartening.

    10. Re:Actually no. by syntotic · · Score: 1

      The AI may be 100% correct but judges are biased. They have an incentive to be biased and act crazily and contrary to fact and evidence because otherwise they are not needed, an AI can give verdict. So why the AI failed in 31% of cases is the matter to discuss. It should be understood to the last detail.

  5. What does AI stand for by goombah99 · · Score: 1

    Another Improbable program.

    --
    Some drink at the fountain of knowledge. Others just gargle.
  6. Trial? by rossdee · · Score: 1

    Most of them never come to trial

    Terrorist like IISIS and Boko Haram will die rather than be captured

    The leaders of large countries (eg Putin, GWB and the chinese) never submit to international tribunal

  7. And regarding the remaining .21% ... by hcs_$reboot · · Score: 1

    ...humans were wrong.

    --
    Slashdot, fix the reply notifications... You won't get away with it...
  8. the endgame by prof_robinson · · Score: 2

    Seriously, what is the endgame here? Having robots adjudicate human rights? How in the world does that seem like progress to anyone?

  9. Re:Slighlty better than coin tosx. by frovingslosh · · Score: 1

    Is it really better than a coin toss? What is the outcome of most "Human Rights" trials. If it is about a 79% conviction rate or even just a 21% conviction rate then the outcome isn't very impressive at all.

    --
    I'm an American. I love this country and the freedoms that we used to have.
  10. Al not A.I. by frovingslosh · · Score: 1

    You misunderstand. This is an "Al" program, as in Al Gore or Al Franklin, extremely liberal and so sure of its view on human rights that it is wrong 21% of the time. A.I. would never make that mistake.

    --
    I'm an American. I love this country and the freedoms that we used to have.
  11. Re: by addisoncurtis · · Score: 1

    I am very glad on Scientist's this invention, but I want to ask you something as I don't know, is this a recent invention, I think this is an old research. What you say?

  12. Ummm by Archfeld · · Score: 1

    Not to be pissy or anything but let's face it. By the time a human rights violation has case has 'come to trial' it is nearly a forgone conclusion. Were 10K people massacred ? Was an entire countryside gassed ? Did the members of an entire tribe suddenly disappear ? Was a mushroom cloud, and nerve gas involved ? It doesn't take much of an AI or computer to come to a conclusion in those kinds of cases.

    --
    errr....umm...*whooosh* *whoosh* Is this thing on ?
  13. 79% doesn't seem great by 91degrees · · Score: 1

    In fact we shouldn't expect these to be too predictable. This is why they go all the way to the ECtHR in the first place

    1. Re:79% doesn't seem great by AHuxley · · Score: 1

      In the US you can get that way up in the federal system
      Conviction rate https://en.wikipedia.org/wiki/...
      "For 2012, the US Department of Justice reported a 93% conviction rate."

      --
      Domestic spying is now "Benign Information Gathering"
    2. Re:79% doesn't seem great by 91degrees · · Score: 1

      So an AI that said "guilty" every time would get much better accuracy here :)

      That's not really the best comparison though. ECtHR is more comparable to the US Supreme Court. Mostly dealing with appeals based on incompatibilities of legislation with fundamental law.

    3. Re:79% doesn't seem great by AHuxley · · Score: 1

      Default to guilt and then look for any strange "not guilty" outlier in every case to really seem accurate.

      --
      Domestic spying is now "Benign Information Gathering"
  14. Re:my AI by FyRE666 · · Score: 1

    My guess is that any forum you frequent will live down to that prediction...

    --
    Code, Hardware, stuff like that.
  15. I hope one day it can predict... by TheCarp · · Score: 1

    The outcome of Bush, Obama, Cheney and Yoo's trials.

    So they like facts more than legal arguments? Thats great, we have lots of those. Drone strikes targeted by cell phone data killing innocent people. A whole system of assasination based on paid informants and lies. A torture program that was swept under the rug rather than exposed and prosecuted....

    Lots and lots of facts for them.

    --
    "I opened my eyes, and everything went dark again"
  16. Re:The Problem Is by Attila+Dimedici · · Score: 1

    What does the state of Israel have to do with predicting the outcomes of cases before the EUROPEAN Court of Human rights? Considering that Israel is not a signatory to the treaty creating said court.

    --
    The truth is that all men having power ought to be mistrusted. James Madison
  17. Shell script by T.E.D. · · Score: 1

    I can predict Egyptian and Russian human rights trials with 100% accuracy with a simple shell script.

    echo "Guilty!"

  18. so is that good? by fish_in_the_c · · Score: 1

    So I coin toss could predict it with 50% accuracy. How well can a human being perform the same task? Not sure that 20% better then a coin toss makes your AI very intelligent. Of coarse I'm sure it is just the beginning of a lot of cool work.

    --
    âoeTolerance applies only to persons, but never to truth. Intolerance applies only to truth, but never to persons.
  19. Assuming you have "standing" by Tokolosh · · Score: 1

    If you don't, you have no rights.

    BTW, there is no right to "standing" or for the government to reveal that you even potentially have "standing.

    --
    Prove anything by multiplying Huge Number times Tiny Number
  20. Re:This couldn't be too hard by OakDragon · · Score: 1

    Well I wrote a program to detect racism in Internet comments.

    The first test, it returned notifications on ALL comment sections scanned.

    There were no false positives.

    --
    Dark Reflection