Algorithm Predicts US Supreme Court Decisions 70% of Time

stephendavion writes A legal scholar says he and colleagues have developed an algorithm that can predict, with 70 percent accuracy, whether the US Supreme Court will uphold or reverse the lower-court decision before it. "Using only data available prior to the date of decision, our model correctly identifies 69.7 percent of the Court's overall affirm and reverse decisions and correctly forecasts 70.9% of the votes of individual justices across 7,700 cases and more than 68,000 justice votes," Josh Blackman, a South Texas College of Law scholar, wrote on his blog Tuesday.

Comments

biased algorith

[Dthief]

I (read: anyone) can make an algorithm that fits any previous data (even only using data that precedes the "prediction")......testing future predictability is the only way this means anything.

Re:biased algorith

[mwvdlee]

If only he could have made some predictions, travelled to the future to test the predictions, then travelled back and put the results in his blog post.
Sadly, testing future predictability can only be done after the future has passed.

Re:biased algorith

[Chatterton]

That why you train your algorithm on all the available cases but the last year ones. Then you can test it on that last year of cases. For the system the last year is the "future" on which you do your testing.

Re:biased algorith

[Thanshin]

But once the future has passed, it's no longer future. So one can only assert to have tested the predictability formerly called future; also known as the Prince test.

Re:biased algorith

[Anonymous Coward]

That's why you should always divide your data set into one subset for fitting/training of the algorithm, and another subset to verify its predictive ability.

The algo doesn't know or care whether the data is actually from the future. That is irrelevant as long as it wasn't fitted on it.

Re:biased algorith

[Anonymous Coward]

Yes, and then when the algorithm doesn't work you finetune it a bit and test again and suddenly you end up with an algorithm that has been trained on all data without actually training it against all data.

One should be very skeptical against future predicting algorithms. Until they have been released in the wild for a while without the developer tampering with it it is pretty safe to guess that it more or less is another version of the Turk, even if its inventor doesn't realize it.

The same principle can be applied to market research or climate studies. If the algorithm used is tampered with to produce more accurate results one can assume that it is useless.

Re:biased algorith

[Euler]

You could train it with 80% of the historical data and see if it predicts the next 20% of historical data.

Re:biased algorith

[nedlohs]

But of course you tweak and change over time rather than having the first try work just perfectly and so that subset for verification ends up influencing the algorithm anyway.

Re:biased algorith

[phantomfive]

That's a good idea, but it only works once. If you keep trying it over and over with different algorithms, then the effectiveness decreases.

Essentially what you are doing there is manually running an evolutionary algorithm, discarding ones that don't fit the last 20%, and improving on ones that do.

Re:biased algorith

[LongearedBat]

I dunno. There may be changing trends, so what you suggest might train it to be accurate for a bygone era.

Trivial

[Chrisq]

Just identify the wrong decision and they are bond to pick it ;-)

Useless

[Jiro]

According to http://www.scotusblog.com/stat... the Supreme Court recently affirmed 27% of lower court decisions and reversed 73%. This means that if you guess that the Supreme Court reverses the lower court every time, you'll be 73% accurate. 70% accuracy is ridiculously low if you can get 73% accuracy *without* taking into consideration the records of each justice or any other kind of details.

Re:Useless

[sound vision]

Not really - The "algorithm" the grandparent has come up with can be written out as "The vote will be to reverse the ruling". Sure, you will get approximately 73% accuracy, assuming the distribution of the decisions remains the same. But it has zero utility as a predictive algorithm. Presumably, the algorithm that has been developed in TFA can predict both rulings to uphold and rulings to reverse with 70% accuracy. That's infinitely more useful than an algorithm that predicts rulings to reverse with 100% accuracy and rulings to uphold with 0% accuracy, which is what the GP poster did.

It's similar to

Re:Useless

[AthanasiusKircher]

70% accuracy is ridiculously low if you can get 73% accuracy *without* taking into consideration the records of each justice or any other kind of details.

First, your link only deals with the past court term. TFA deals with predicting cases back to 1953. Is your 73% stat valid for the entire past half century?

And even if it were, the algorithm is much more granular than that, predicting the way individual justices will vote. From TFA:

69.7% of the Courtâ(TM)s overall affirm and reverse decisions and correctly forecasts 70.9% of the votes of individual justices across 7,700 cases and more than 68,000 justice votes. Also, before someone objects, please note that (contrary to popular belief) SCOTUS does not always vote 5-4 according to party lines. For instance, your own link notes that 2/3 of last year's opinions were UNANIMOUS. 5-4 decisions usually amount for only 25% of cases or so in recent years, and of those, usually a 1/3 or so don't divide up according to supposed "party line" votes.

So, I agree with you that simply predicting reverse/affirm at 70% accuracy may be easy, but predicting 68000 individual justice votes with similar accuracy might be a significantly greater challenge.

Re:Useless

[fph il quozientatore]

Yeah, that's why we have better error measures than "70% accuracy", and competent people should use them.

Re:Useless

[fph il quozientatore]

To be more fair, I am not bashing the original paper here -- that one reports the full confusion matrix (http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463244).

Simplified algorithm

[mwvdlee]

if defendant.bank_balance > plaintiff.bank_balance
      winner = defendant
else
      winner = plaintiff

I'd guess about 90% accurate.

Re:Simplified algorithm

[dywolf]

my algorithm is even better, and even more accurate. its simple: What is the worst possible outcome for the citizenry?

Re:Simplified algorithm

[swillden]

my algorithm is even better, and even more accurate. its simple: What is the worst possible outcome for the citizenry?

I don't know about the accuracy of your SCOTUS result-picking algorithm, but you and mwvdlee have a good algorithm to get modded up on slashdot: Just express deep cynicism about the system. Doesn't have to be true in the slightest.

FWIW, I watch SCOTUS pretty closely, and I'd say their bad decisions are fairly rare. I'm unhappy with the outcome in a larger minority of cases, but it's not very common that upon reading the opinions and dissents that I find myself ultimately in disagreement with their conclusions. And in most cases I think they not only make the right legal call, but the right call for the citizenry (though that isn't, and shouldn't be, their primary focus).

Of course, you and I may well disagree about some of the decisions.

Re:Missing info

[mwvdlee]

It would be useful to know how many of the court's decisions are affirm vs reverse.

http://www.americanbar.org/con...

I did some tallying on table 3 and found the following numbers on total decissions;
Reversed: 58.48%
Vacated: 12.58%
Affirmed: 28.94%

The article doesn't mention whether "vacated" is counted separately or as a reversal.
The graph shows only reversed and affirmed, so I'm assuming vacated counts as a reversal.
If this is the case, reversed and vacated together is 71.06%.
So if you'd guess "Reversed" all the time, you'd be slighly more accurate than the algorithm.

Sweet! Now we can start the Judge program.

[Lumpy]

Install software in the helmet, Set the judges loose on the city....

I AM THE LAW!

Re:is it better than random?

[mrvan]

That is correct, but not what the GP meant. If you can model the distribution (e.g. you 'know' that B is 90%) then you can weigh your random guessing such that it is correct in >50% of the cases, even without looking at the case itself (it is still 'random' in that sense)

Extreme case: I can predict whether someone has Ebola without even looking at them with >99.99% accuracy by just guessing "no" every time, since the prevalence of Ebola is >.001%.

Suppose the supreme court has 70% chance of overturning (e.g. because they choose to hear cases that have 'merit'), then an algorithm that guesses 'overturn' 100% will have a 70% accuracy. A random guess that follows the marginal of the target distribution (e.g. guess 70% overturn) also scores >50% (58% to be precise).

Algorithm based on bias

[ranton]

I wouldn't be surprised if the primary predictive trait used is simply to check the biases of each judge and then assume they will vote along those biases. Assuming conservative judges will vote conservative and liberal judges will vote liberal should give you a pretty good score right off the bat.

Re:Algorithm based on bias

[AthanasiusKircher]

I wouldn't be surprised if the primary predictive trait used is simply to check the biases of each judge and then assume they will vote along those biases. Assuming conservative judges will vote conservative and liberal judges will vote liberal should give you a pretty good score right off the bat.

Only in a small minority of cases. Contrary to popular belief, most SCOTUS cases aren't highly politicized cases with a clear conservative/liberal divide. Most cases deal with rather technical issues of law which are much less susceptible to this sort of political analysis.

The Roberts Court, for example, has averaged 40-50% unanimous rulings in recent years (last year about 2/3 of rulings were unanimous). So, your idea of "assume conservative vote conservative, liberal vote liberal" would tell you nothing about maybe half of the cases that have come before the court in recent years. (Historically, I believe about 1/3 or so of rulings tend to be unanimous.)

And even with the closely divided cases, you have a problem. Of the 5-4 rulings (which in recent years have been only about 20-30% of the total rulings), about 1/4 to 1/3 of them don't divide up according to supposed "party lines."

In sum, I don't know what factors this model ends up using, but "conservative vs. liberal" is way too simplistic to predict the vast majority of SCOTUS rulings. If you could factor in detailed perspectives on law (which often have little to do with the stereotyped political spectrum), you might have something... but that would require a lot more work, particularly over the 50 years of rulings TFA deals with.

Re:Algorithm based on bias

[JoeMerchant]

Bear in mind, the model only gets it right 70% of the time, and a red-black roulette spin would get it right nearly 50% of the time.

Re:Algorithm based on bias

[ShanghaiBill]

So 40-50% are unanimous, and those should be easy to predict. For the remainder, predict party line, and you will get an additional 30-40% right. So an algorithm that gets only 70% right doesn't seem very impressive. Even simplistic guessing should do at least that well.

Re:not really that hard, theoretically

[tomhath]

The original intent was to prevent the government from having too much power by ensuring that citizens could form militias. Having arms available to everyone (not just the government's army) was an essential part of being able to raise a militia.

Re:not really that hard, theoretically

[argStyopa]

Nonsense, an editorial screed by the New Yorker is meaningless. And if you want to bring context into it, you'll lose even harder.

Firstly, judicial review wasn't even a principle until Marbury v Madison in 1803. So we're talking about the 19th century only.

In cases in the 19th Century, the Supreme Court ruled pretty much only that the Second Amendment does not bar state regulation of firearms. (For example, in United States v. Cruikshank, 92 U.S. 542, 553 (1875), the Court stated that the Second Amendment âoehas no other effect than to restrict the powers of the national government,â and in Presser v. Illinois, 116 U.S. 252, 265 (1886), the Court reiterated that the Second Amendment âoeis a limitation only upon the power of Congress and the National government, and not upon that of the States.â )

Although most of the rights in the Bill of Rights have been selectively incorporated into the rights guaranteed by the Fourteenth Amendment and thus cannot be impaired by state governments, the Second Amendment has never been so incorporated.

It's only since 1939 United States v Miller, that federal court decisions considering the Second Amendment have largely interpreted it as preserving the authority of the states to maintain militias - not the '150 year history' stated in the deliberately-misleading text of the quoted article.

(much of the above is clipped verbatim from http://www.loc.gov/law/help/se...)

In fact, it's ONLY in the latter 20th Century that we've even HAD this debate, as all constitutional commentary and understanding previous to that was universal in its understanding of the 2nd Amendment as an individual right, *not* dependent on being in a militia: http://en.wikipedia.org/wiki/S...

Of course, you further disregard that according to the US code, all males from 17-44 *are* by default in the militia. (http://www.law.cornell.edu/uscode/text/10/311)

Re:Replace them

[Impy the Impiuos Imp]

Lawyers: We want people to carry their rights with them, even when operating as a group of people Congress defined as a "corporation" because Congress cannot force them to give up their First Amendment rights.

Scotus (in the voice of Nomad): Logic correct. Opposing lawyers are in error. Must sterilize.

Is that really hard?

[Opportunist]

I'd actually have expected more. I mean, let's be reasonable here:

A 50% accuracy can be achieved by the average unbiased coin. Now throw in the rulings that are easy to foresee because any other decision would be politically suicidal and you should easily arrive at more than 70%.