Slashdot Mirror
20 Top Lawyers Were Beaten By Legal AI (hackernoon.com)
An anonymous reader shares a report:In a landmark study, 20 top US corporate lawyers with decades of experience in corporate law and contract review were pitted against an AI. Their task was to spot issues in five Non-Disclosure Agreements (NDAs), which are a contractual basis for most business deals. The study, carried out with leading legal academics and experts, saw the LawGeex AI achieve an average 94% accuracy rate, higher than the lawyers who achieved an average rate of 85%. It took the lawyers an average of 92 minutes to complete the NDA issue spotting, compared to 26 seconds for the LawGeex AI. The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes.
A computer once beat me at chess, but it was no match for me at kickboxing.
So rise up, all ye lost ones, as one, we'll claw the clouds.
The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes.
Was time used as a measurement, i.e. were the lawyers told to work as fast as they could? Cause that would likely affect the diligence put into the review.
We already know that computers are faster at processing data. We also know that stressing the crap out of people tend to make them work faster but with less precision. Adding such an element to the human input would bias the result while providing no additional information of value.
20 top US corporate lawyers with decades of experience in corporate law and contract review were pitted against an AI.
What the article isnt saying is that these lawyers were pitted against a combat AI in a parking lot in the back of a derelict convenience store in the dark hours of the morning. The AI successfully dismembered virtually all of them, despite their decades of experience in corporate law.
Good people go to bed earlier.
Are these really AIs? Or are they just expert systems trained to do a job? Do they sport new issues? or are they just really good at spotting the issues they've been trained to spot? I argue that all this "artificial intelligence" craze we've got going on right now has nothing to do with intelligence. It's just training a computer program to do one thing and to do it very well. Does anyone consider a calculator an AI? Why not, calculators are far more accurate than humans at math and do not make mistakes.
First they came for the lawyers, but I did nothing, for I am not a lawyer ...
Nae king! Nae laird! Nae yurrupiean pressedent! We willna be fooled again!
by the hour?
Seems like most court cases involving contracts and interpretation of the law are mostly about unpacking the Bullshit written into the contract or law. It seems absurd that one of the functions of the court is to divine the intentions of the law's author.
A law can be complex in that there are many moving parts that can interact in different manners, but the parts themselves and the manner in which they can interact should be clearly defined and not subject to interpretation.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
re-write all the lawyer jokes...
It took the lawyers an average of 92 minutes to complete the NDA issue spotting, compared to 26 seconds for the LawGeex AI.
I imagine sexbots will achieve similar results.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
A clever lawyer might have added some 'special' text to the NDA https://xkcd.com/327/
Modest doubt is called the beacon of the wise - William Shakespeare
I hypothesize that the AI is only good at spotting current and historic types of loopholes.
Here's my proposed test. A theoretical bad-actor NDA creator gets services of both a lawyer and the AI to review their document. They craft different ways to build in issues, with several cycles of submitting to both for feedback and modifications. (Since both of these would be available to someone trying t make a bad one.)
Final document is reviewed and scored.
My guess is that the human lawyers will be more adept at finding innovative issues in the NDA. But who knows until we test it.
It might be that the best path is a first pass by an AI to catch issues, then a lawyer-pass that can be significantly quicker since it doesn't have to look for the same issues the AI would.
LITTLE GIRL: But which cookie will you eat FIRST? C. MONSTER: Me think you have misconception of cookie-eating process.
In the linked article, there is a button to get the full study, but downloading the study requires giving one's name, email, phone, company, and "contracts reviewed per month". Not exactly paywalled, but not exactly free.
The title of the Slashdot story says, "20 Top Lawyers Were Beaten By Legal AI", which is not true. The top AI and top human scores were identical at 94%. Furthermore, although the lowest human score was 67%, the average was 85%, indicating that the distribution is skewed toward higher human scores. There is a distribution of human scores, but just a single AI sample point, so it's not clear what the AI distribution would be.
There is also a comparison of AI vs. human time. Of course, that's a misleading comparison, similar to the misleading Jeopardy comparison from a few years back. Computers will always beat humans at text parsing and button pressing. That was true decades before AI.
The other aspect that I don't see in the article summary is whether the seeded risks were all in the AI training set or if any were deliberately left out of the training set. I'd expect the AI to do extremely well in detecting risks similar to the training set. However, I'd expect the humans to do better in risks that deviated from the training set.
Boring, difficult tasks are where they bill most of their hours. Why would anyone who is paid $300/hr want to finish in 26 seconds? The lawyer who took 156 minutes just made $900.
You mean it will replace already poorly paid beginner lawyers in lawyer offices and make that occupation vanish into thin air.
They should put this AI on a website called something like "EULA-Buddy", where you can paste those 10-foot-long EULAs that come with every modern device or service. Then it could concisely tell you how much the EULA sucks. Maybe that would help reign in uncontrolled expansion of these ridiculous "contracts".
More questions: How were the NDA's "flawed", and did the agent that produced the flawed NDAs know anything about the AI's capabilities?
Boring, difficult tasks are where they bill most of their hours. Why would anyone who is paid $300/hr want to finish in 26 seconds? The lawyer who took 156 minutes just made $900.
Proving once again that time spent is a lousy proxy for value.