IQ Test Pegs ConceptNet 4 AI About As Smart As a 4-Year-Old

An anonymous reader writes "Artificial and natural knowledge researchers at the University of Illinois at Chicago have IQ-tested one of the best available artificial intelligence systems to see how intelligent it really is. Turns out–it's about as smart as the average 4-year-old. The team put ConceptNet 4, an artificial intelligence system developed at M.I.T., through the verbal portions of the Wechsler Preschool and Primary Scale of Intelligence Test, a standard IQ assessment for young children. They found ConceptNet 4 has the average IQ of a young child. But unlike most children, the machine's scores were very uneven across different portions of the test." If you'd like to play with the AI system described here, take note of the ConceptNet API documentation, and this Ubuntu-centric installation guide.

Comments

More like autistic-savant 4 year old

[schneidafunk]

From the article: “If a child had scores that varied this much, it might be a symptom that something was wrong,” said Robert Sloan, professor and head of computer science at UIC, and lead author on the study.

Re:More like autistic-savant 4 year old

[Metabolife]

Similar to how a typical Slashdot user might score amazingly well on the math section and then never score in real life?

Re:More like autistic-savant 4 year old

[Vanderhoth]

Obligatory, I for one welcome our new four year old mentally unstable electronic overlords.

Re:More like autistic-savant 4 year old

[Anonymous Coward]

Of course you have to compare it to a child. How else will it ever ascend to becoming a contestant on "Are You Smarter Than A 5th Grader?"

Re:More like autistic-savant 4 year old

[Anonymous Coward]

It is indeed an indicator of something wrong. Can you imagine an AI with the mores and intelligence of a small child with access to vast amounts of information? Have you ever seen a small child in the store? I HATE you mommy!. That that emotional maturity and intelligence and that mommy would be dead. Machines like this will be dangerous.

Re:More like autistic-savant 4 year old

[Anonymous Coward]

Damn it, I'm not even good at slashdot reading - I score bad at math and don't get laid! :(

Re:More like autistic-savant 4 year old

[Vanderhoth]

so five then?

Re:More like autistic-savant 4 year old

[Anonymous Coward]

Being as intelligent as a 4 year old human on an IQ test is not even remotely related to having the learning abilities of a 4 year old human.

Re:More like autistic-savant 4 year old

[ByteSlicer]

Except for the small fact that current AIs have no emotions whatsoever.

Re:More like autistic-savant 4 year old

[sjames]

He said IQ, not emotional maturity. So 5 1/2.

Sample output when tested

[Anonymous Coward]

No!

But

[Sparticus789]

Does the AI use contractions?

Re:But

[leonardluen]

for a 4 year old i am pretty sure the value of pi is "more" and possibly "with ice cream"

Re:But

[Laxori666]

No you see, by pi he meant the mathematical number - 3.1415... - not pie as in "food". Even so, the value of pie wouldn't be "more" and "with ice cream" - that's not the value of something, those are desires and descriptors of something. Value would be more like how many dollars it's worth or how many good deeds/grades one has to get to receive the pie.

Re:But

[Laxori666]

That's how we lost ol' Timmy. Asked him to give us a few digits of that good ol' pi and he just durn went and hung himself right in the ol' barn. We stopped teachin' the young'uns maths after that lil' incident.

Re:But

[TWiTfan]

The whistling thing I could understand, since he presumably had no respiration.

Re:But

[flyingfsck]

I'm wondering whether your post deserves a whoosh, or a huh? or whether you are a 4 year old AI...

Re:But

[RobinH]

The guy who built him deliberately made him *less* lifelike than Lore because the colonists didn't like how eerily human-like Lore was. His inability to use contractions was one of these things.

Re:But

[Beorytis]

Alexander Hamilton never used the word "whilst" but James Madison never used the word "while." This does not imply that either founding father was an ill-programmed android.

Re:But

[snadrus]

To be accurate, it would need
the same input inaccuracies (pi and pie verbally being the same), combined with
the order of learned experiences which influences weighting (pie before pi), combined with
a 4-year-old's limited capacity for context, combined with
need (eat) & desire (sweet foods).

We have a ways to go.

Re:But

[Anonymous Coward]

Not that it matters, but your XO emoticon means "hug and kiss" in straight culture.

Misleading crap

[Anonymous Coward]

We are nowhere near getting an AI that can navigate the world at the level of a 4 year old. All the program can do is simple tasks in vocabulary and such with no real understanding of those words. Nothing to see here.

Re:Misleading crap

[DougOtto]

Sounds like my ex-wife.

Re:Misleading crap

[Vanderhoth]

I think the point is, like in a real child's development this is a stepping stone, it's something A.I. has to go through in order for it to mature.

I never understood why people think a A.I. should learn any faster than a real child could. It's like people think because it's a computer it automagically knows everything there is ever to know, but in reality A.I. still requires training and positive/negative reinforcement just like really children do.

Re:Misleading crap

[ebno-10db]

We are nowhere near getting an AI that can navigate the world at the level of a 4 year old. All the program can do is simple tasks in vocabulary and such with no real understanding of those words. Nothing to see here.

The headline is the usual attention grabbing junk, but the article itself does a decent job of explaining it:

Sloan said ConceptNet 4 did very well on a test of vocabulary and on a test of its ability to recognize similarities.

“But ConceptNet 4 did dramatically worse than average on comprehension—the ‘why’ questions,” he said.

One of the hardest problems in building an artificial intelligence, Sloan said, is devising a computer program that can make sound and prudent judgment based on a simple perception of the situation or facts–the dictionary definition of commonsense.

Commonsense has eluded AI engineers because it requires both a very large collection of facts and what Sloan calls implicit facts–things so obvious that we don’t know we know them. A computer may know the temperature at which water freezes, but we know that ice is cold.

“All of us know a huge number of things,” said Sloan. “As babies, we crawled around and yanked on things and learned that things fall. We yanked on other things and learned that dogs and cats don’t appreciate having their tails pulled. Life is a rich learning environment.”

IQ tests mean little enough for a human being, for AI they're little more than cute. Most 4 year old's know if someone is mad at them (expression, tone of voice, etc.) and, from past experience, often know why someone is mad at them. They're also clever enough to pretend they don't know why someone is mad at them. Most importantly (and practically), they know to start acting cute before somebody kills them. Let me know when an AI program can do that.

P.S. This is not to disparage the AI work, just to keep things in perspective.

Re:Misleading crap

[Vanderhoth]

With the proper sensors they can look around, recognize faces and react to an locate sounds. I'm not sure if giving a machine a mouth so it can shove random objects into it is really Nobel prize worthy...

Re:Misleading crap

[Laxori666]

Ultimately the source of all our information comes from sensory input. That's how we know ice is cold, that things fall, some things hurt, others are pleasurable, etc. On top of that sensory data we construct an intelligent (symbolic) representation of the world, in tandem with a language (or several languages) which we share with others and can thus use to exchange ideas.

What AI researchers seem to be doing is skipping the sensory input part, because it's hard, and just trying to codify the intelligent representations directly. In light of the above, it's clear why common sense eludes AIs built on this principle.

Perhaps the approach that will ultimately succeed - and I don't see a convincing reason why we won't ultimately be able to build a sentient self-reflective machine which ends up being more intelligent than a human - is to mimic the human developmental approach from the start. Hook up a shitload of sensors to a massively parallel brain-structure-type thing and have it "learn" from there. We won't so much program it as direct its growth and evolution. This probably requires a ton more computing power than we have now, but seeing as how it on average doubles every 2 years, it will eventually catch up. The first artificial humans will likely be pretty unintelligent, but then they'll quickly surpass humans because we'll have bested evolution - we'll have figured out exactly what makes intelligence and sentience, and then we'll be able to turn that knob up to 11 and beyond in a relatively short period of time.

At that point we can only hope one of them doesn't go rogue and kill us all.

Re:Misleading crap

[tmosley]

Yeah, I had a sudden jolt of existential terror when I read the headline. All was calm again after reading the article.

For those NOT utterly terrified at the concept of a strong AI, I suggest you read some of Yudkowsky's writings on the subject here.

I think they have it backwards

[shadowrat]

They didn't assess how intelligent this AI is. They assessed the IQ test and found it to be a poor indication of intelligence.

Re:I think they have it backwards

[bunratty]

It's the most accepted way to give a number to intelligence for a human being. In other words, the score on the IQ test is correlated highly with the intelligence of the person who took the test. There isn't necessarily such a correlation for computer programs. A relatively simple computer program might score highly on an IQ test designed for humans, but that certainly doesn't mean it's as smart as a human. I've taught this idea to AI classes -- it's a question near the beginning of the Russell & Norvig AI book.

Re:OOOh IQ thread.

[MugenEJ8]

Yup... an apostrophe.

Re:OOOh IQ thread.

[ArcadeMan]

You have an IQ of 0xFFFFFFFF80000000?

IQ tests only apply to humans

[Anonymous Coward]

These tests don't tell us much about the power of an AI and here is why. If you give a human test with a million questions, then giving one more question is not going to tell you much more. You could probably remove some of the questions too without removing much information about how smart the person is. It turns out some of the questions are much more valuable when it comes to figuring how smart someone is. If you put enough statistics work into that, you'll be able to condense those million questions into a quite short list of questions that can be administered in an hour or so, to a human, yet still tell you almost as much information as the million question test did. That's what an IQ test is.

The problem is, if you give that test to an AI, then the IQ number you get at the end won't tell you how well the AI would have done at a million/billion/trillion question test. You do get that information for a human because the test has been carefully constructed to be like that. For an AI, all you learn is how well the AI does at the questions in the test, which is much less interesting than the information you get from a human taking an IQ test.

Unfortunately

[puddingebola]

Unfortunately the AI also lied that it had completed its arithmetic assignment so that it could go out to recess early. It is also suspected of taking an extra snack at snack time, and caused a disturbance during nap time.

Faking It

[Capt.Albatross]

"ConceptNet 4 did dramatically worse than average on comprehension—the ‘why’ questions.” - Robert Sloan, lead author of the study.

This comment strengthens my feeling that current AI is making progress in faking many of the accidental attributes of intelligence, but has not discovered the essence.

The development of childrens' mental abilities seems to accelerate over time, as if there is positive feedback, but this does not seem to have emerged in AI yet, especially if we factor out Moore's law. On the contrary, any given exercise in developing AI through machine learning seems to hit a wall of diminishing returns at some point. Is anyone aware of a project that has not experienced this effect?
 

The actual study? How did they do it?

[paskie]

Anyone knows where to access technical information about the actual study, or how did they conducted the IQ test? ConceptNet is just a database + a library with some NLP parsing tools and database (the concept hypergraph) accessors, but I wonder how did they actually conducted the test as that doesn't seem to be a trivial extension of the available tools...