Slashdot Mirror
20 Top Lawyers Were Beaten By Legal AI (hackernoon.com)
An anonymous reader shares a report:In a landmark study, 20 top US corporate lawyers with decades of experience in corporate law and contract review were pitted against an AI. Their task was to spot issues in five Non-Disclosure Agreements (NDAs), which are a contractual basis for most business deals. The study, carried out with leading legal academics and experts, saw the LawGeex AI achieve an average 94% accuracy rate, higher than the lawyers who achieved an average rate of 85%. It took the lawyers an average of 92 minutes to complete the NDA issue spotting, compared to 26 seconds for the LawGeex AI. The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes.
A computer once beat me at chess, but it was no match for me at kickboxing.
So rise up, all ye lost ones, as one, we'll claw the clouds.
When did AI's become to legal to beat the crap out of attorneys?
What is an "issue" with an NDA? What is the "right" NDA. You can't make objective measurements about this. The lawyers actually think and reason, the bot does what it is programmed to. Obviously the bot is better at comparing the NDA to a "correct" NDA. But the bot can't think about whether that NDA suits the context in which it is used.
As a programmer, I have this to say about AI: Bullshit, don't invest, it can not replace humans for non-repetitive tasks that require reasoned judement.
Success at MMA or any other contact sport largely depends on the robot built around the computer. Does the chess boxing match in this joke take place before or after Boston Dynamics and its line of doglike robots?
People will be happy that some lawyers got beaten. I am not. This is how it begins.
First they came for the lawyers, but I did nothing, for I am not a lawyer ...
I will NOT welcome the AI-lawyer-beating overlords.
(Or perhaps it is Al and not AI, with lowercase L. Then: more power to you, AL. Kick their asses.)
Don't fight for your country, if your country does not fight for you.
The document used for testing was not created by an AI, was it?
In Soviet Washington the swamp drains you.
What kinds of "issues" are we talking about here. This seems highly subjective since one side's "issue" could be the other side's deal breaker.
The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes.
Was time used as a measurement, i.e. were the lawyers told to work as fast as they could? Cause that would likely affect the diligence put into the review.
We already know that computers are faster at processing data. We also know that stressing the crap out of people tend to make them work faster but with less precision. Adding such an element to the human input would bias the result while providing no additional information of value.
But to issue fix. Get back to me when the AI can negotiate those issues out of existence with the other parties to the transaction. We don't pay lawyers to issue spot. We pay them to resolve issues that for the most part any 12 year reading at grade level can spot. Love to see the AI run a conference call with 6 parties and 20 lawyer on the line. When that happens all humans everywhere are out of a job.
Ya...AI has become Market Speak these days.
Anything that can do something faster than a person is, "AI".
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
What this means is that legal advice is going to become dirt cheap. Now only the rich and connected can afford legal advice. If anyone can get good at least SOME legal advice for $20, the world might become more fair.
This kind of unbalanced situation happens all the time, especially with corporations. What if you could just send that silly NDA you got to an AI and it'll tell you if it's enforceable? A lot of law is just knowing what papers to file. That's something AI can be easily trained.
20 top US corporate lawyers with decades of experience in corporate law and contract review were pitted against an AI.
What the article isnt saying is that these lawyers were pitted against a combat AI in a parking lot in the back of a derelict convenience store in the dark hours of the morning. The AI successfully dismembered virtually all of them, despite their decades of experience in corporate law.
Good people go to bed earlier.
Are these really AIs? Or are they just expert systems trained to do a job? Do they sport new issues? or are they just really good at spotting the issues they've been trained to spot? I argue that all this "artificial intelligence" craze we've got going on right now has nothing to do with intelligence. It's just training a computer program to do one thing and to do it very well. Does anyone consider a calculator an AI? Why not, calculators are far more accurate than humans at math and do not make mistakes.
Still not better than Harvey and Mike...
by the hour?
To spot the fact that representing a client in court as a non certified attorney is illegal?
Seems like most court cases involving contracts and interpretation of the law are mostly about unpacking the Bullshit written into the contract or law. It seems absurd that one of the functions of the court is to divine the intentions of the law's author.
A law can be complex in that there are many moving parts that can interact in different manners, but the parts themselves and the manner in which they can interact should be clearly defined and not subject to interpretation.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
Bundy
re-write all the lawyer jokes...
I thought these kind of tasks are done my junior or paralegal?
You can have as many medbots, drone armies and robocops as your heart desires, but you're not messing with lawyers' jobs.
Non-conscious, non-sentient A.I.
Expert systems have their place, but this wasn't one of those.
It took the lawyers an average of 92 minutes to complete the NDA issue spotting, compared to 26 seconds for the LawGeex AI.
I imagine sexbots will achieve similar results.
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
Why don't they pit AI against janitors, or cab drivers to see. Stop taking jobs from us!!!! Oy vey!!!
A clever lawyer might have added some 'special' text to the NDA https://xkcd.com/327/
Modest doubt is called the beacon of the wise - William Shakespeare
The human lawyers outlawed AI lawyers. It took the human lawyers and average of 2.3 seconds to get the law passed. The AI lawyer objected within 1.4 nanoseconds. It was overruled by the human judge and the AI lawyer was summarily executed/unplugged.
First it was the truck drivers, then it was the financial anaylist and now the lawyers. AI will be more disruptive than the industrial revolution.
That'w why there's a big power grab by the elites, economic degradation, fake bombs, all the stops.
Mankind is on the verge of getting unbribable, consistent law with no retaliation for judgements. We're also on the verge of money with no central banking authority.
The elites want this for themselves, not for you!
(Decentralized) Code Is Law
IOW we don't have to wait for long now before a couple of millions lawyers are sacked.
A good beginning.
I sincerely hope that the same AI will write the new laws in the future instead of the lobbyists.
Expect automated legal work (AI or whatever you want t call it) to be outlawed. The legal field will not allow its revenues to be stolen by nerds.
Doubt it? Who makes the laws?(legislature) Who upholds the laws?(judges) What do all these people have in common? (they're lawyers)
Do you really think that the people who make uphold and argue the law, the people who profit from these practices, will allow an AI to take their jerbs?
I am FUCKING azmazed that LegalZoom hasn't been banned, let alone this shit.
Nobody takes three hours to look at an NDA. Either that NDA was grotesquely off-market or there was some other reason the lawyers were incentivized to waste that much time looking at it during the study. A lawyer who actually spends time looking at NDAs as part of his job will do it in 10 minutes usually.
Most NDAs are pretty much the same. If the information isn't the crown jewels, you look it over, look for the same small number of issues that come up over and over again and look for insane shit like non-compete agreements.
never could beat Andre Panza Kickboxing.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
I hypothesize that the AI is only good at spotting current and historic types of loopholes.
Here's my proposed test. A theoretical bad-actor NDA creator gets services of both a lawyer and the AI to review their document. They craft different ways to build in issues, with several cycles of submitting to both for feedback and modifications. (Since both of these would be available to someone trying t make a bad one.)
Final document is reviewed and scored.
My guess is that the human lawyers will be more adept at finding innovative issues in the NDA. But who knows until we test it.
It might be that the best path is a first pass by an AI to catch issues, then a lawyer-pass that can be significantly quicker since it doesn't have to look for the same issues the AI would.
LITTLE GIRL: But which cookie will you eat FIRST? C. MONSTER: Me think you have misconception of cookie-eating process.
that the lawyers jobs are at stake. I mean that. These are people who make a living manipulating our legal system. If they are made obsolete by AI they are not going to go quietly into that good night. They're going to look for targets to sue for quick cash.
The mega corps figured that out years ago. They had Congress pass a law that made Arbitration agreements legally binding and had the Supreme Court stacked with pro corporatists who upheld it (even though it's a pretty obvious violation of due process and centuries of legal precedent that says you can't sign away constitutional rights.)
What that means is lawyers can't sue mega corps (or even a mid sized corp with decent arbitration agreements). Again, they're not going to shrug their shoulders, sell their BMW for a 4 year old Yaris and go work and 7-11. They're gonna come after you and me. We've already seen this with stuff like Penda law suing regular joes. You'll see more and more of that and by people who aren't as incompetent as to get caught.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
In the linked article, there is a button to get the full study, but downloading the study requires giving one's name, email, phone, company, and "contracts reviewed per month". Not exactly paywalled, but not exactly free.
The title of the Slashdot story says, "20 Top Lawyers Were Beaten By Legal AI", which is not true. The top AI and top human scores were identical at 94%. Furthermore, although the lowest human score was 67%, the average was 85%, indicating that the distribution is skewed toward higher human scores. There is a distribution of human scores, but just a single AI sample point, so it's not clear what the AI distribution would be.
There is also a comparison of AI vs. human time. Of course, that's a misleading comparison, similar to the misleading Jeopardy comparison from a few years back. Computers will always beat humans at text parsing and button pressing. That was true decades before AI.
The other aspect that I don't see in the article summary is whether the seeded risks were all in the AI training set or if any were deliberately left out of the training set. I'd expect the AI to do extremely well in detecting risks similar to the training set. However, I'd expect the humans to do better in risks that deviated from the training set.
They should put this AI on a website called something like "EULA-Buddy", where you can paste those 10-foot-long EULAs that come with every modern device or service. Then it could concisely tell you how much the EULA sucks. Maybe that would help reign in uncontrolled expansion of these ridiculous "contracts".
Have the AI run through thousands of scenarios finding loopholes for crimes.... $$$
[($)]
Decided to check on who these "Top" 20 lawyers were. First on list from article named "Zakir Mir". Went to linkedin, confirmed it was him via his work for Allegiance International. The man graduated law school 3 years ago (2015). I'm sorry but 3 years of experience is not a "Top" lawyer.
This makes sense. We can overlook things quite easily even when trained, while for an AI it would be much easier.
I would be actually surprised if the AI could take over in an actual trial.
Like Accountants with "new" spreadsheet software.
This will free up lawyers time to do more interesting "work" and concentrate on the non-standard sections.
But there are billions of different contracts. An NDA is pretty easy to handle human or not. The ones I've had to sign were mostly unenforceable ... or I signed someone else's name at the company. Nobody ever noticed. Filing mistake, no doubt.
So the quickest lawyer billed for 1 hour, and the slowest for 3. IANAL but I review all NDAs and other contracts for my company, and a typical NDA takes me 10-15 minutes, but then I don't get paid more for taking longer. Despite my lack of formal legal training, my company hasn't been f**ked over by a bad NDA in the 20+ years I've been doing it. Any lawyer billing 3 hours to review an NDA is ripping off their client.
I hope we soon have lawsuits filed just for stepping out of our homes in the morning....
In that first episode where the creepy automated hologram lawyer playing pre-recorded defense, and the creepy automated hologram judge playing a prerecorded sentence to have the "perpetrators" organs harvested?
...but what does the AI know about bird law?
Tools like this should make lawyers less expensive and available to more people as they won't have to spend their time examining documents when a computer can do that. (Actually the junior lawyers are for that and for researching now.) Let the lawyers spend time with clients, in talks with the other sides lawyers, or in court. Speeding up the discovery process should get cases to trial sooner too.
The lawyers that have to worry are the ones that are really good at researching. Those that are good with clients and in the courtroom will benefit from the oncoming computer tools.
I noticed you are writing an NDA agreement. Can I make a few suggestions?
Any guest worker system is indistinguishable from indentured servitude.
They will sound like this: A dramatic reading of a lawyer commercial written by a bot
Would have been a much better title.
It is better than just sneaking something small past an Asemi-I. Sneak something big past. Something that no human would ever let through. Like the surety being the first born child. Or possibly something more enforceable.
This happens to simplistic AIs that mark English essays. Well written gibberish can get high marks. The sad thing is that the AIs generally do a better job than human markers, which is probably more a statement about the humans doing the marking than the AI.