Software To Evaluate Facial Expressions Developed

The Technology Review site has up an article on new software that can do quantitative analysis of facial expressions, detecting whether a person is smiling. The software was developed by the Omron Corporation, and can also estimate a subject's age and gender, or verify a person's identity from a database. Though the company doesn't yet know whether it plans to release the software commercially, there are a number of obvious applications. "Omron envisions the smile software being used in marketing, perhaps to evaluate consumers' reactions to a new product or to an advertising campaign. A smile checker could also help train customer-service staff to meet Japan's legendarily high standards ... A smile in isolation is easy to detect, but the bigger challenge is to develop systems that can recognize the concerto of facial actions that make up complex expressions like confusion, fear, and disgust." Thanks to jamie for the link.

Comments

yea

[Mutagenic]

great now my sexbot will know how im feeling

Re:yea

[mh1997]

great now my sexbot will know how im feeling

Without seeing your face, I am going on a limb and guess "horny."

Re:yea

[Anonymous Coward]

I for one welcome our evaluating expressionless faciallords, poured over hot grits naked and petrified.

Re:yea

[Bogtha]

I'm pretty sure it already knows.

Re:yea

[davidsyes]

Put that in the REAL DOLL. Sniffing and smile detection...

Be sure to endow the Doll with voice synthesis (and other servos, pumps, actuators, springing action, etc...)...

Then, to borrow a line from a certain 60's show...

him: "You'll let me know if you fell pain?..."

her: "You'll be the FIRST to know..."

Re:yea

[ZeroFactorial]

Now anti-social, chip-on-the-shoulder against humanity, virus makers everywhere will create viruses that only target the happy people.

If ignorance is bliss, we might finally see the end of AC's on slashdot!

Re:yea

[popmaker]

Yes! And when you reach orgasm it will take a picture!

'Facial' Detection?

[EveryNickIsTaken]

Wow. I need to get my mind out of the gutter today.

Re:'Facial' Detection?

[UbuntuDupe]

Not milk?

How much of it is ....

[iknownuttin]

racial or ethnic? I don't mean in a racial way ... but, there are some cultures that will shake their heads side to side to say "yes". So, what's up?

No, I didn't read TFA. Can't connect.

Re:How much of it is ....

[ShiningSomething]

This will recognize the facial expression of "smiling". As long as a smile has a meaning within a culture, it will be of use (provided it works, of course). It operates based on things like (from TFA)

the hallmarks of a smile--such as narrowed eyes, an open mouth, creases around the mouth, and wrinkles turning downward around the eyes So, it may be that in some places it detects discomfort and embarrassment, whereas in others it detects happiness. But to the extent that we all open our mouths and get wrinkles around our eyes when we smile, it should be pretty universal. I think.

Testing methods

[LiquidCoooled]

Confusion I can understand, just get the human interacting with the system.

Disgust as well, the system just needs to show goatse or something.

Fear is the simplest, it just takes an automatic gun and 20 seconds.

Airports

[BlowHole666]

How long till this product is in an airport near you figuring out if you are happy. If you are not you get corn holed with extra screening because you must be a terrorist. Perhaps this software will be used at the exit to a bar. If you are too happy or too sad, you are drunk. What if this software is used allover the place and it is saved. An employer could search you and find out if you are typically a sad or happy person and you then do not get the job because you are a sad person. Or could this be used for discrimination? Show someone of an African American hanging and if the person smiles they are a bigot? (Yes I think racism is wrong, it was just an example. Insert African American and hanging for something else if you want.)

Re:Airports

[skeletor935]

You, good sir, are way too paranoid. Also, you have cameraphobia

Re:Airports

[BlowHole666]

Oh no. I am just trying to fit in on Slashdot. Normally I would have the opposite views. I don't give a crap who takes a picture of me picking my nose. I Just tend to get trolled so I figured I would try something new. :) So yeah I don't care if they know if I am smiling.

Re:Airports

[Arthur B.]

What if they even use PEOPLE to gather that information, omfg !

Re:Airports

[BlowHole666]

OMFG! But this is a camera on every street corner. Not a Cop. Man I thought Slashdot loved the "Your invading my privacy stuff". Guess I found the one person who does not care about their privacy.

Re:Airports

[sm62704]

How long till this product is in an airport near you figuring out if you are happy. If you are not you get corn holed with extra screening because you must be a terrorist.

Is that a happy grin or an evil grin? Is that a real poncho or a Sears poncho? I doubt this software can tell (no, I din't RTFA ass eye yam knot knew hear)

Perhaps this software will be used at the exit to a bar. If you are too happy or too sad, you are drunk.

Neither the bar owners nor the bartenders care if you're drunk, and the policeman waiting outside in his squad car needs no software. Actually, the bartenders/owners would want a device that wouldn't let you out until you were either a) drunk or b) broke. ("that's a joke, son", says Mr. Leghorn)

What if this software is used allover the place and it is saved. An employer could search you and find out if you are typically a sad or happy person and you then do not get the job because you are a sad person.

HR can already do this without any technology whatever.

Or could this be used for discrimination? Show someone of an African American hanging and if the person smiles they are a bigot? (Yes I think racism is wrong, it was just an example. Insert African American and hanging for something else if you want.)

Sorry, that sentence didn't parse. Will pass this comment along to my user for human evaluation. Please do not leave the room, the authorities will be there shortly.

Outlaw the American Secret Police

Re:Airports

[owlnation]

How long till this product is in an airport near you figuring out if you are happy. If you are not you get corn holed with extra screening because you must be a terrorist.

You must be new to airports. The only happy people in them are likely be terrorists. Those are ones you want to screen. Any normal person finds airports to be the singular most hellish places on Earth.

Re:Airports

[GigG]

"How long till this product is in an airport near you figuring out if you are happy. If you are not you get corn holed..." Damn, I really wanted that sentance to end, "...by a congressman in the bathroom."

Re:Airports

[ObsessiveMathsFreak]

If they could read your mind, they would. In lieu of that, they'll always go for the next best thing.

Metaverse

[FiloEleven]

In a much overlooked section of Snow Crash, Neal Stephenson postulated that the real key to making the Metaverse a place where things happen and where people want to interact with each other was the inclusion of facial expressions. I tend to agree with him - look at how emoticons are used on the web today; we've had articles about their usage bleeding into corporate culture to help people understand the intended subtext of a message. A bunch of avatars walking around with a fixed look on their faces makes for a boring virtual world and more miscommunication than communication. Facial expression replication (which does not necessarily include recognition, but I think it'll help immensely) will be needed before virtual worlds really take off for society at large.

Re:Metaverse

[esecasco]

I have to agree...this tech could find very good usage in video games. In multi-player it could definitely add an element of interaction for all gestures. For example, if it can detect a facial expression, why not a head movement, so instead of a radio button to say "yes" you can just nod, and have your character do the same. Shake, tilt your head, etc, etc. Also, the element of expression could add a new level of interaction, in conjuction with voice communication. From someone screaming angrily with a war cry, to someone cowering as they see their allies die. And lets not forget single player. An NPC would be able to see if you're angry, or happy. NPCs could be coded to devise cunning ways of tricking you depending on what you're expressing.

One word.

[Anonymous Coward]

Thoughtcrime.

Re:One word.

[fractoid]

Isn't it more like... face crime?

But will it detect sarcasm?

[starglider29a]

A sarcastic smile? A sardonic smile? A wistful smile? A retail smile? The masked lie? "Mmmm... this casserole taste great, hunny"

Or the smile that Ballmer uses when describing Vista's security or sales milestones? The smile that Verizon execs wear when they promise "unlimited..."???

Re:But will it detect sarcasm?

[Albert Sandberg]

A sarcastic smile? A sardonic smile? A wistful smile? A retail smile? The masked lie? "Mmmm... this casserole taste great, hunny"

She will not need to use a computer to detect that, fool ;-)

Re:But will it detect sarcasm?

[davidsyes]

They can try electronic scratch n' sniff, teaching it how to recognize smell.

As for the chairs, be sure to give them motility, so they can watch for a ballmer tirade. Then, they can run away from him.

Re:But will it detect sarcasm?

[Anonymous Coward]

and finally we get to find out about the mona lisa. I've always wondered about that smirk.

Re:But will it detect sarcasm?

[Maian]

When it comes to subtle expressions, they vary a lot between people. Sarcastic expressions, for example, can be hidden (where the person relies on his words to convey the sarcasm). Some people can master the "retail smile" so well, you'd think they were worshiping their employers (hello Apple).

Re:But will it detect sarcasm?

[Anonymous Coward]

Or the smile that Ballmer uses when describing Vista's security or sales milestones? The smile that Verizon execs wear when they promise "unlimited..."??? These smiles are in the "Joe Isuzu Smile" category and are easily recognized.

What are the non-enforcement uses?

[pembo13]

How can this system be used outside of law enforcement?

Re:What are the non-enforcement uses?

[Rob T Firefly]

How can this system be used outside of law enforcement? A terrificly modern answer to the old "mood ring" craze?

Re:What are the non-enforcement uses?

[Speare]

How can this system be used outside of law enforcement?

Sony has already developed a camera with a simpler form of "smile detection." If engaged, the camera will scan the scene for all subjects to smile and then allow the shutter sequence to fire. Seems really gimmicky (useless in practice) to me, but a new tickbox on the carton equates to sales.

Sony Smile-Detecting Camera

Re:What are the non-enforcement uses?

[hansamurai]

I'd rather have a camera that detected when all the eyes were focused on it. Getting everyone to smile might be a tremendous feat, but getting everyone to simply look at the camera isn't as bad, and probably more important.

Re:What are the non-enforcement uses?

[Hrodgare]

As mentioned a few comments below, affect/emotion detection can be useful for people with autism. Rana El Kaliouby has been working on a system that does exactly this in the Affective Computing group at the Media Lab. My understanding is that they have built it into an appliance which is robust to poor lighting situations, which is pretty impressive. There is also a great deal of interest in this in the human-robot interaction and human-computer interaction communities in general, so that robots and computer systems can respond more naturally to people in social situations.

Re:What are the non-enforcement uses?

[stokes]

It could provide social cues for the mildly autistic and victims of brain injury who are incapable of interpreting facial expression.

Re:What are the non-enforcement uses?

[milo78]

Well, as a usability analyst I think it could be useful if integrated into something like Morae. So, in additional to a webcam video stream of a participant interacting with a site, you could get data on how they felt about it...without having to ask them. (because that ain't very reliable.)

Re:What are the non-enforcement uses?

[Froboz23]

There is a particularly scary application for this type of software. Imagine surveillance cameras scanning a political rally. (Putin's new Russia comes to mind, but you can insert your own favorite government here as you please.) An advanced system could detect dissent by individuals' facial expressions during the rally, and deal with it appropriately. Sometimes facial expressions happen subconsciously. You have to focus to maintain a poker-face.
</Paranoia>

From a computer-science perspective, I find this stuff fascinating. But anything that can gauge a person's emotions, especially if they don't know they're being watched, has a lot of room for abuse.

Re:What are the non-enforcement uses?

[fractoid]

Hell, I'd take it if it could detect whether my eyes are open or closed... I have an incredible ability to blink just as the picture's taken, even when it's in broad daylight and there's no flash.

Re:What are the non-enforcement uses?

[Magada]

Rally schmally. You protester types always forget there's life outside political activism. If you really want to get people interested, why not posit a scenario such as "you might get labeled a ter'ist sympathizer for displaying negative emotions in the presence of american national symbols". Favorite team unlikely to win? Better keep a straight face while they're playing the anthem, buddy!

Analyze This!

[Sponge Bath]

:-)

Expression analyzed: Terrorist Alert!

Re:Analyze This!

[fractoid]

You're software is off. Please adjust it to recognize the following faces:

:-) civilian
@:-) terrorist

Facial profiling. Hurrah!

Am I missing something?

[JetScootr]

...evaluate consumers' reactions to a new product or to an advertising campaign
All this time, I thought marketing people didn't give a damn WHAT our reactions were to their campaigns.
Oh, wait, I get it - smiles mean 'FAIL'. They'll look for the concerto of facial actions that make up complex expressions like confusion, fear, and disgust.

I'D LIKE TO SHOW IT MY O FACE

[Anonymous Coward]

OH oh oh oh

Testing Methodology

[hyades1]

They could test the software by letting it count the smiles in an audience as they watch marketing managers being put to death in various off-beat and creative ways. A dunk tank full of piranha, rolls of toilet paper in a parachute pack...that sort of thing.

Does it have a botox mode?

[140Mandak262Jamuna]

Come on even humans have difficulty telling the emotional state of someone who got botoxed ... How can the software tell?

Re:Does it have a botox mode?

[BlowHole666]

No I do not think it has a John Kerry mode.

Immunity

[Rob T Firefly]

This software is useless against certain subjects..

Other application

[Chris whatever]

This is not new , they have already put software like this in use for kids who have light autism. Some of these kids look at the mouth when other people speak so they miss the facial expression which conveys more than the tone of voice.

the software, which is not perfect, tries to get what the other person is feeling by facial expression and they even have a camera pointing at their own face to let them know how to understand facial expressions by studying themselves.

Re:Other application

[Anonymous Coward]

The flip-side of course being that autistic people end up being the first "shoot to kill to protect" targets because they didn't smile "correctly" when buying their Tube ticket etc. Welcome to 1933 - 1945.

Estimate Gender?

[Anonymous Coward]

and can also estimate a subject's age and gender

How in the world do you estimate gender? That's like trying to whether a light switch is on or off. (I'm counting hermaphrodites and dimmer switches as statistical anomalies in this case)

Looking at the mouth, or eyes?

[jackpot777]

Plenty of people smile with their mouth, but their eyes tell how happy they aren't.

Probably why Japanese emoticons reflect their emotions with the eyes instead of the mouth.

(^_^)(^_^)

The mouth isn't smiling, but the eyes say PAAARTY!

Re:Looking at the mouth, or eyes?

[Red Flayer]

(^_^)(^_^)

That's a pretty strange pair... I'd be a little shocked to see them like that in real life.

Normally what I see is:

(.)(.)

or if I'm at the right website,

( o )( o )

Re:Looking at the mouth, or eyes?

[hansamurai]

Except in the real world, Japanese often do not make eye contact while in conversation. This is for various reasons and of course a generalization with no doubt many exceptions, but compared to Westerners, they do make less eye contact. So if you were trying to imply that Japanese emoticons are an extension of the people, I would disagree. But they definitely show off emotion better than Western emoticons. It'd be interesting to trace their origins, I'm guessing they're mostly from manga/anime where a few original artists have influenced nearly everyone else in art style.

Re:Looking at the mouth, or eyes?

[ozphx]

I never understood the ^_^ until I had a chinese gf. When she smiles her eyes almost exactly look like ^'s. Its uncanny, and it made me laugh, and I couldnt explain why I was laughing :P

Vista Smile Recognition

[EmbeddedJanitor]

If it is anything like Vista Voice Recogition, I think Id rather use:

Are you smiling? [Yes] [No]

Re:Vista Smile Recognition

[cayenne8]

I guess I need to run out quick and patent my anti-Omron Smile software device.

I think I'll call it a mask ©!!!

Evil Sounding Corp Name

[riffzifnab]

Seriously, "Omron Corporation" sounds like something out of a bad video game. Now to figure out what kind of bad video game and what they (really) do.

Re:Evil Sounding Corp Name

[popmaker]

Sounds worse, though, if you exchange the first two letters.

nuances

[icepick72]

I wonder if it can distinguish between middle of a sexual climax vs. being in pain.

Re:nuances

[Anonymous Coward]

Any differences or similarities between those depend on what subcultural circles or doctors' offices you hang around in.

Re:nuances

[Anonymous Coward]

The pain starts 9 months later, and continues for 18+ years after that.

You really expect the software to be THAT good?

To Find Terrorists:

[Anonymous Coward]

such as this thug.

Costly

[Tarlus]

For the money it is costing them to develop this software, they could just pay some guy to look at people's facial expressions and say what they are...

This is very old

[biraneto]

A lot more resources can be found at this usefull site, Since this specific software may be new but the concepts and software for face, facial expressions, gender or age recognition may be older than the first PC web cam.

This is for airports.

[DragonTHC]

there was an article about the TSA using this a few months ago.

how many innocent citizens will they harass that are simply pissed off that they have to wait in line?

how many nervous travelers are they going to waste time on?

This is really the job of a human.

the TSA is the biggest farce in the last 30 years.

Estimating gender

[Anonymous Coward]

I like how the summary says the program can "estimate a subject's age and gender":

This subject is approximately female.

Mona Lisa Analysis

[Doc Ruby]

Wake me when it can tell me why the Mona Lisa is smiling.

Smile detection is very simple

[EmbeddedJanitor]

grep :-)

Not so hard...

[jemenake]

and can also estimate a subject's age...

if(face.oneSideDroops())
___stroke=true;
if(stroke)
___minAge=60;
if( ! face.movementDetected())
___minAge=90;
minAge += 32 - teeth.count();

See? It ain't so tough.

Testing the software

[Wowsers]

Did they test the software on politicians? Whatever expression is pulled, they more than likely they get a score of 'Liar, Liar, pants on fire!'

This could lead to great improvements in Eliza!

[Shag]

This is just what we've needed since 1966 - a way for the non-directional therapist to see facial expressions.

complex expressions?

[ozphx]

"complex expressions like confusion, fear, and disgust"

Thats easy to train. Theres plenty of examples if you google for "first goatse" ;)

Some test cases for the software...

[madbawa]

1) http://www.dubyasworld.com/dubya-quit-thinking.jpg
2) http://www.geocities.com/calann@sbcglobal.net/dubya.jpg

Research not limited to Japan

[BananaBender]

This is not exactly new research. The German project SIPBILD is aiming for the complete recognition of facial expressions, body language and gestures http://www.forsip.de/index.php?show=projekte_sipbild2&page=1&lang=en.
The project has been running for several years now and the last publications date back from 2006.

More information on the recognition of facial expressions can be found at the TU Munich under http://vision.in.tum.de/projects/mimic_recognition/.
I am really fascinated by the research but why the hell did it take so long to figure this one out? The problem seems obvious and the solution cannot be that hard...