Scientists Create AI Program That Can Predict Human Rights Trials With 79 Percent Accuracy (theverge.com)

An anonymous reader quotes a report from The Verge: Computer scientists have created an AI program capable of predicting the outcome of human rights trials. The program was trained on data from nearly 600 cases brought before the European Court of Human Rights (ECHR), and was able to predict the court's final judgement with 79 percent accuracy. Its creators say it could be useful in identifying common patterns in court cases, but stress that they do not believe AI will be able to replace human judgement. As described in a study published in the journal PeerJ Computer Science, the AI program worked by analyzing descriptions of court cases submitted to the ECHR. These descriptions included summaries of legal arguments, a brief case history, and an outline of the relevant legislation. The cases were grouped into three main violations of human rights law, including the prohibition on torture and degrading treatment; the right to a fair trial; and the right to "respect for private and family life." (Used in a wide range of cases including illegal searches and surveillance.) The AI program then looked for patterns in this data, correlating the courts' final judgements with, for example, the type of evidence submitted, and the exact part of the European Convention on Human Rights the case was alleged to violate. Aletras says a number of patterns emerged. For example, cases concerning detention conditions (eg access to food, legal support, etc.) were more likely to end in a positive judgement that an individual's human rights had been violated; while cases involving sentencing issues (i.e., how long someone had been imprisoned) were more likely to end in acquittal. The researchers also found that the judgements of the court were more dependent on the facts of the case itself (that is to say, its history and its particulars) than the legal arguments (i.e., how exactly the Convention on Human Rights had or had not been violated).

Comments

Actually no.

[goombah99]

You can't even tell if it's better than a coin toss. For this statistic to have any meaning at all you need to know the frequency the plaintiff wins. For example, let is suppose that the plaintiff wins in 79% of cases. Then an "AI" that merely always guess the plaintiff won would be correct in 79% of cases.

In fact given that it's unlikely the outcome is 50:50, then one would expect that such a dumb algorithm would be correct more often than not just by always guessing one side. It would therefore take very little extra "intelligence" so boost it over the top. In particular such intelligence could be simply an artifact of the data set. As an example suppost the data set contained 10% of plaintiffs whose names started with R. If this group of people won more often than the avergage, then simply learing to guess "win" anytime there was a plaintiff with an "R" name would improve the test. This is true even if you split the data up into cross validation sets, as the bias for "R" will persist on any randomly Chosen subset as well.

thus the results probably are meaningless. Certainly the article is.