Slashdot Mirror

← Back to Stories (view on slashdot.org)

AI Beats Humans at Reading Comprehension (bloomberg.com)

Posted by ryuzaki0 on from the beat-them-if-you-can dept.

In what is being called a landmark moment for natural language processing, Alibaba and Microsoft have developed AIs that can outperform humans on a reading and comprehension test. From a report: Alibaba Group put its deep neural network model through its paces last week, asking the AI to provide exact answers to more than 100,000 questions comprising a quiz that's considered one of the world's most authoritative machine-reading gauges. The model developed by Alibaba's Institute of Data Science of Technologies scored 82.44, edging past the 82.304 that rival humans achieved. Alibaba said it's the first time a machine has out-done a real person in such a contest. Microsoft achieved a similar feat, scoring 82.650 on the same test, but those results were finalized a day after Alibaba's, the company said.

17 of 171 comments (clear)

  1. No Kidding by tsqr · · Score: 3, Insightful

    Based upon the knee-jerk quality of many comments posted on /. this should not be a surprise to anyone.

  2. Comprehension, M'FR, Do You Read It ?!? by cstacy · · Score: 2

    Sadly, this does not surprise me.

    Most people don't read and have shockingly poor comprehension when they do.
    This has gotten much worse (at least in the US) over the past 100 years.

    LOL I didn't bother to read TFA so perhaps totally don't comprehend what it said...

    1. Re:Comprehension, M'FR, Do You Read It ?!? by alvinrod · · Score: 3, Insightful

      It's okay in a lot of areas, it's just that no one really gives a shit about the inner city school districts and so they've gone to absolute shit. If you remove that from the equation, the U.S. as a whole is quite comparable to other western democracies. The U.S. has seems more content to let this problem fester and to deal with the consequences rather than tackle it head on so the problem just goes from bad to worse in a lot of ways.

      On a side note, if there weren't so many useless (not as in they suck at their jobs, just useless in that their jobs don't improve educational outcomes in any measurable way) administrators soaking up money, we could pay teachers a hell of a lot more. The U.S. spends more on education as a percentage of GDP than other countries that do as well or better than us, and over time our spending on education as a percentage of GDP has increased. Even though you hear about cutbacks all the time (who pays attention when funding is increased?) the trend has been moving upward over time. So it's not strictly a money problem.

      Here's a good report (PDF warning) that has looked into how public education has changed in the U.S. over time. The increase in administrative staff has done nothing to improve outcomes and removing the excess would allow for an additional ~$11,000 in yearly salary for every teacher.

    2. Re:Comprehension, M'FR, Do You Read It ?!? by Dread_ed · · Score: 2

      Many people care about the inner city schools. In my city, one group has consistently tried for many decades to get under performing teachers and administrators fired, reassigned, or removed from inner city schools. Their counterparts in city government and school administration have rebuffed them by calling them racists, and demanding that under performing and detrimental administrators and teachers keep their jobs because of the color of their skin.

      Then one group tried to give children choices besides enforced shit schools guaranteed to end in high dropout rates, low test scores, etc. Charter schools were proffered. That was called racist and part of a war on education. They got pushed through anyways and are now a viable option for some children to escape from the working-as-designed life destroying school system as envisioned by our local school controlling party. I say some, because they are severely limited by design and as a result only some children will be able to attend.

      So yes, people do give a shit about inner city schools. Apparently it's the racists who want provide quality education, options for children and parents if their government enforced school is not what they are looking for, and to ultimately produce productive members of society. And, the it's the non-racists who are bolstering teacher's unions that prevent the firing of shit teachers, who demand that horrible teaching is enforced on all students, and who don't want there to be any other choice for these kids than terrible schools with terrible teachers.

      So yeah, keep saying no one cares. The truth is that the people who are in direct control of the schools are actively destroying the lives of the children within because of unrestrained self interest and the ability to throw down the race card on anyone who dares try to improve these kid's lot in life.

      --
      When the only tool you have is a claw hammer every problem starts to look like the back of someone's skull.
    3. Re:Comprehension, M'FR, Do You Read It ?!? by alvinrod · · Score: 2

      My point is that society as a whole doesn't really care, not that there isn't some individual person out there that isn't trying to solve the problem. You should be able to figure out that in plain language saying "no one" doesn't imply a universal quantifier across the entire population. Individual teachers probably care (until they get burned out) but they can do fuck all and probably have their hands tied by the system as much as anything. The same goes for other groups and individuals as well, who lack the political or financial capital to do anything on a large scale.

      The point is that Republicans don't really care because they're largely not based in the inner cities so from their perspective it's someone else's problem and they're not all that keen about being made to pay for it either. The Democrats don't really care either because to the extent that people in the inner city vote, they're sure as fuck not going to vote Republican so there's no incentive to do anything for that small part of their base when lip service does fine.

      Any small groups or individuals trying out different systems (whether they work or not) don't have the political power to accomplish anything themselves, especially not against any intrenched interests and the larger political parties are mostly apathetic towards that overall cause for reasons stated above. Even if those new systems do get better results, they're probably not so much better to the point that it becomes blindingly obvious that it's a better way. I'll admit that I haven't read much of any research on the topic though so if there are a large number of good studies showing that charter schools do a better job, I'm more than open to reading through them.

  3. Doug Lenat's Test by sycodon · · Score: 4, Insightful

    “Mary saw a bicycle in the store window. She wanted it.”

    Does Mary want the bike, the store, or the window?

    --
    When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
    1. Re:Doug Lenat's Test by amalcolm · · Score: 5, Funny

      Obviously, she was gagging for sex. The window shopping was just a distraction

      --
      Time for bed, said Zebedee - boing
    2. Re:Doug Lenat's Test by 110010001000 · · Score: 2

      It isn't unambiguous to normal people. That is the point, and that is the difference between intelligence and just following rules. You just proved his point.

    3. Re:Doug Lenat's Test by sycodon · · Score: 2

      Because people speaking in normal conversation always use the proper rules of English?

      That's the entire point of his work is to enable the computer to understand things that you and I intuitively understand, but which is vague and indeterminate to a computer.

      On the other hand, something AI could benefit from is a properly defined AI interface syntax. Like it does for coding, a properly defined syntax for interacting verbally with computers could move things ahead quite a bit by eliminating the need for the computer to try and figure out what the fuck you are talking about.

      --
      When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
    4. Re:Doug Lenat's Test by sycodon · · Score: 2

      That's the point entirely. People speak in many different ways and you intuitively understand what they are saying despite the sometimes unclear way they say things.

      --
      When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
    5. Re:Doug Lenat's Test by nospam007 · · Score: 4, Funny

      Two guys were watching a dog licking his balls.
      First guy says to the second: I'd like to do that too.
      The second guy replies: You'd better pet him first or he might bite you.

  4. This is just misleading. by xxxLCxxx · · Score: 2

    This is plain stupid as it is just misleading. Granted, scholars can barely read these days. Nonetheless, what is being defined as 'AI' here might lead to frustration when people actually expect some sort of 'intelligence' from it.

  5. Comment removed by account_deleted · · Score: 2

    Comment removed based on user account deletion

  6. A real test by wafflemonger · · Score: 2

    A much better test would be seeing if it could understand some deconstructionist literary criticism.

  7. Re:AI is extremely BROAD by fluffernutter · · Score: 2

    So far there is only AI capability when there is a well-defined set of rules. Chess and Go have a small set of rules. Language has rules that are complex but it's not like grammar isn't something that is studied and well understood. Compare that to an activity like driving, where you may need to judge if a bent and half-obscured stop sign is a legal one, or interpret whether a front end loader operator wants you to wait for it or pass around it, or interpret what construction workers mean by analyzing poorly laid out road markers, where the rules are almost infinite AI will have trouble.

    --
    Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
  8. Re: This says little about AI by rrconan · · Score: 2

    The test is https://rajpurkar.github.io/SQ...

  9. Re:This says little about AI by djinn6 · · Score: 3, Informative
    The questions are nothing like that. Here's the reading material:

    Packet mode communication may be implemented with or without intermediate forwarding nodes (packet switches or routers). Packets are normally forwarded by intermediate network nodes asynchronously using first-in, first-out buffering, but may be forwarded according to some scheduling discipline for fair queuing, traffic shaping, or for differentiated or guaranteed quality of service, such as weighted fair queuing or leaky bucket. In case of a shared physical medium (such as radio or 10BASE5), the packets may be delivered according to a multiple access scheme.

    And here's the questions:

    How are packets normally forwarded?
    Answer: asynchronously using first-in, first-out buffering, but may be forwarded according to some scheduling discipline for fair queuing

    How is packet mode communication implemented?
    Answer: with or without intermediate forwarding nodes

    In cases of shared physical medium how are they delivered?
    Answer: according to a multiple access scheme

    So the test taker only needs to find a selection of the original text that answers the question.

    The way I see it, the real issue with the "reading comprehension" quiz is that you don't need to actually comprehend the text to answer it. A better question than "How are packets normally forwarded?" would be something like "What are some situations where packets are not forwarded in the fifo order?" The first question only requires you to find the words "packets", "normally" and "forwarded" in the paragraph and answer with the rest of the sentence. The second question requires you to understand that the text is presenting 2 options, one is "normal" and the other isn't.

    There's also some official answers that are just plain incorrect. The answer to "How is packet mode communication implemented?" is the entire rest of the paragraph, not just "with or without intermediate forwarding nodes".