Let Your Pupils Do the Typing

New submitter s.mathot writes: Researchers from France and the Netherlands have developed a way to—literally—write text by thinking of letters. (Academic paper [open access], non-technical blog, YouTube video.) This technique relies on small changes in pupil size that occur when you covertly (from the corner of your eye; without moving your eyes or body) attend to bright or dark objects. By presenting a virtual keyboard on which the 'keys' alternate in brightness, and simultaneously measuring the size of the eye's pupil, the technique automatically determines which letter you want to write; as a result, you can write letters by merely attending to them, without moving any part of your body, including the eyes.

Comments

Is this really new?

[Z00L00K]

I think that eye movement tracking have been studied for a long time as an input method, mostly for handicapped people that lacks movement in a major part of their body.

What about the system Stephen Hawking uses?

Re:Is this really new?

[110010001000]

This one is probably slightly different: rather than trying to track where you are looking, you examine the pupil change to try and find out what letter they are looking at. Sounds terribly unreliable and expensive.

Re:Is this really new?

[Martin Blank]

Hawking still uses a system activated by a muscle in his cheek, one of the few over which he still has some level of control, which is then detected by an IR sensor in his glasses. Earlier versions used a small joystick while he still had some control over a few fingers (or maybe it was just one), but the system has been adapted as he's lost more and more control.

This system might allow him to continue working even if he loses the last vestiges of control over his facial muscles.

Re:Is this really new?

[Anonymous Coward]

I understand the confusion, but this technique does not rely on eye tracking in the sense of measuring eye position. It relies on measuring the size of the pupil, while the eyes don't move. So our technique is more comparable to so-called brain-computer interfaces than to conventional eye tracking. (Of course, eye tracking and similar devices (such as the cheek system used by Stephen Hawking) have been around for years, and if (eye) movement is possible, these are more efficient.)

Re:Is this really new?

[110010001000]

Nice hype, but you are measuring the pupil size, there is no brain-computer interface, and you aren't "thinking" of letters. The system would work without letters. Ridiculous.

Re:Is this really new?

[Anonymous Coward]

Pupillometry can actually be really cheap! There are commercial eye trackers that can do it for about $100, and you could even use a bogstandard $15 webcam. The current study has two things going for it: 1) It's really quite amazing that it can track where someone is attending to using nothing else than the pupil size, and 2) This principle can be used for people who cannot move their eyes and/or other muscles. Even with completely paralysed or locked-in patients, pupil reflexes are highly likely to still be intact.

In this light, the pupil-computer interface can be a rather cheap alternative to EEG or fMRI brain-computer interfaces. The results in the PLOS ONE study even indicate it might be a quicker and more accurate system. (Of course, it still needs to be tested in actual patients; but the results in healthy controls are very promising compared to healthy-control studies of the aforementioned brain-computer interfaces.)

Sauce: Am neuroscientist / experimental psychologist at Uni of Oxford.

Re:Is this really new?

[mcgrew]

Indeed, this would only be helpful to someone who could neither type nor speak. It seems that writing this way would be very time consuming.

Re:Is this really new?

[sconeu]

My late wife had an eyegaze computer made by Tobii.

My first thought is, "How does this differ from a traditional eyegaze computer?"

Re:Is this really new?

[s.mathot]

Well it's actually very reliable and cheap: The eye tracker used for the video is €100, and selection accuracy (even for naive participants) is over 90%. It's slow though.

Re:Is this really new?

[Applehu Akbar]

"and you could even use a bogstandard $15 webcam."

Have any scammers tried this technique at ATMs using a tiny, concealable camera, now that EMV card readers are preventing crooks from skimming the users?

Re:Is this really new?

[Anonymous Coward]

Nope, but if you are one, you could think about using something like this: https://www.youtube.com/watch?v=KM_ruTFY8pY

Note that most trackers do require calibration to an individual; and that the above technique requires luminance differences to exist. Gaze tracking might be possible with a generic calibration (not specific to a person), but I wonder if it would be accurate enough. Interesting suggestion, though! Although I suppose that tracking the finger movements when they type the pincode would be more fruitful.

Re:Is this really new?

[wonkey_monkey]

Indeed, this would only be helpful to someone who could neither type nor speak.

Nor move any part of their body including their eyes.

It seems that writing this way would be very time consuming.

If the alternative is "not communicating at all" I don't think it matters.

Re:Is this really new?

[Anonymous Coward]

The summary definitely makes it sound like you're eye tracking in addition to pupil-size tracking. If the keys alternate in brightness, that would only tell you which half of the keys the user is looking at, not which particular key... unless I'm missing some handwaving mojo someplace.

Re:Is this really new?

[wisnoskij]

It would seem that the main problem is that it probably works great in a darkroom on a monitor specially designed for it. But not in the real word, not on a monitor with a giant glowing power button, and not if you want to show anything else on the screen at the same time.

Re:Is this really new?

[safetyinnumbers]

> when you covertly (from the corner of your eye; without moving your eyes or body) attend to bright or dark objects.

If this is accurate, you don't have to look at the letter. It sounds like, as the summary says, you're literally just thinking about a letter in your peripheral vision without looking directly at it and your pupil responds to its brightness.

Which, even if unreliable, is an interesting discovery.

Re:Is this really new?

[retroworks]

Call it as I see it.

Re: Is this really new?

[hackwrench]

The system may not need to show letters to get the letters right but somewhere in the brain there needs to be letters and somewhere in the machine there also needs to be letters.

As an aside, is anyone else having to relog in a lot on the mobile side of Slashdot?

Re:Is this really new?

[tricorn]

If you can't move your eyes at all, then I don't see how you can "attend" to anything. Vision isn't possible without continuous saccades, your eyes are always moving.

Re:Is this really new?

[naughtynaughty]

All letters display, it figures out which half your letter is in and re-displays just that set, repeat the down select until it figures out which you were "attending"

Re:Is this really new?

[marciot]

If you can't move your eyes at all, then I don't see how you can "attend" to anything. Vision isn't possible without continuous saccades, your eyes are always moving.

You can actually focus your attention on objects in your periphery without moving you eyes to it. In vision therapy there is an exercise where you stare fixedly at a dot in the center of the page while finding letters that are scattered throughout the page. You can actually do this with some practice, although it's not particularly natural. Here is the exercise:

https://visionhelp.files.wordp...

An interesting aspect of the test is that letters away from the center are printed much larger since the resolution of your vision drops sharply the further away you get from the fixation point.

Re:Is this really new?

[marciot]

If the keys alternate in brightness, that would only tell you which half of the keys the user is looking at, not which particular key... unless I'm missing some handwaving mojo someplace.

There is no eye tracking. As you have observed, the system can only tell which half of the keys the user is looking at, so the system is only asking a binary question at each stage. The trick is that the letters change places within the groups depending on which groups the user attends to in earlier questions. This is an example of a binary search. Suppose the letter is H, it would work something like this:

Is the letter in the set ABCDEFGHIJKLM or NOPQRSTUVWXYZ? (user fixates on first set)
Is the letter in the set ABCDEF or GHIJKLM? (user fixates on second set)
Is the letter in the set GHIJ or KLM? (user fixates on first set)
Is the letter in the set GH or IJ? (user fixates on first set)
Is the letter in the set G or H? (user fixates on second set)

It only takes five choices for the system to determine precisely what letter the user is looking at.

Re:Is this really new?

[Anonymous Coward]

If you can't move your eyes at all, then I don't see how you can "attend" to anything. Vision isn't possible without continuous saccades, your eyes are always moving.

You can actually focus your attention on objects in your periphery without moving you eyes to it. In vision therapy there is an exercise where you stare fixedly at a dot in the center of the page while finding letters that are scattered throughout the page. You can actually do this with some practice, although it's not particularly natural. Here is the exercise:

https://visionhelp.files.wordp...

An interesting aspect of the test is that letters away from the center are printed much larger since the resolution of your vision drops sharply the further away you get from the fixation point.

mod parent up

Re:Is this really new?

[KGIII]

You made me curious so I went looking. The prices aren't that bad though I don't see any options for open source. I wonder (I looked and didn't find any) if there are any projects doing this with open source? It seems like you could patch something like that together for less money and the software would then be the only real hard part. (Unless I'm missing something.) An above poster, seemingly in a university and working with this sort of thing, indicates that you can do that with a standard web cam.

I've no need for such a device but I'd be interested in supporting such a project - depending on a few things. It seems a noble quest and I'm not purely altruistic - I want to, someday, be totally "jacked in" to the 'net and simply use my brain instead of an external computing device. I think that would be awesome. I'd even let 'em stuff an ethernet port in my neck and a wireless antenna on the top of my skull. I imagine the results look better in my imagination than they'd really be but I think it'd be awesome to have near instant access to Google. I'd not need software so much, I'd just need logic (as I'm envisioning it) though filtered at the external hardware level might keep the noise to a dull roar. No display, pure internal "vision." It'd be great...

So, the more we move in this direction of alternative inputs and control. The closer we get to my brain-as-a-computer-with-internet point. The closer we get to that point, the closer I get to being able to rule the world! Or, more likely than that, post stupid shit to the internet. But, it'd enable communication, education, and creation levels hitherto unknown. Well, it would if it turned out to be as awesome as it is in my imagination. And yes, yes I really would let 'em mount a wireless antenna on my head. If they had a decent plan, I'd even volunteer to be a test subject. I'd *pay* to be a test subject.

Re:Is this really new?

[Martin Blank]

It's not a direct read, but the idea is the same. You're thinking of a letter, and your attention goes to the letter on the screen even if your eyes don't move at all (they mention this for use in locked-in syndrome, where there's no voluntary movement at all). The iris responds to a lesser degree than it would if it were to center on the letter, but it still responds to the brightness, an involuntary movement based on a thought.

It's not a direct brain interface, but it makes for an indirect one through. A reading of what the subject is thinking, even at so rudimentary a level as a binary choice like this, without relying on a conscious physical action can be seen as a form of brain interface.

Re:Is this really new?

[tricorn]

Except your eyes are still moving. You can keep it focused on a fairly small area, but if they completely stop moving, you can't see anything. When I had laser surgery for a small retinal tear, they injected a paralytic in that eye. As it took effect, I basically went blind. I could sense light and dark, and I could tell when there was movement, but there was no way to actually see anything. The other eye was fine.

Re:Is this really new?

[mikael]

Some systems work on detecting the direction the eyes are looking by measuring the shape of the pupil. Advertising and human-computer interaction people use eye-gaze detection systems that are used to measure how long and where a person is looking

Re:Is this really new?

[s.mathot]

That's correct (although not directly related to the technique): Your eyes always move a little bit, so when we say that "you don't move your eyes" here we simply mean that you don't make large eye movements to look at the letters. But your eyes are always jittering (around a dot center of the display in our case), making tiny movements called 'drift' and 'microsaccades'. Without these small eye movements, vision for things that don't move themselves disappears. I know this only from the literature--it must be quite fascinating to experience this yourself.

Re:Is this really new?

[mcgrew]

Well, of course. It may indeed be of help to Doctor Hawking.

Perfect

[110010001000]

This is perfect. We now don't even need to move our arms to watch netflix. The peak of civilization has now been achieved.

Oh, by the way, this type of system has already been deployed for paraplegics for many years. And no, you aren't "literally thinking" about letters, you are "looking" at letters. Learn what "literally" means.

Re:Perfect

[Opportunist]

Actually the new and improved part of this invention is exactly that people do NOT look at the letters.

Re:Perfect

[Anonymous Coward]

Except that the point of this tech is that you aren't in fact looking at the letters, you're mentally focusing on them outside of the focus of your vision. This is what they mean by "attending" to them, which frankly is a poor wording that doesn't really convey what they're trying to convey.

Re:Perfect

[110010001000]

Not really. You are "looking", but you don't have to move the eye to look. In fact there doesn't need to be "letters" at all. You are just "looking at" light/dark patterns and measuring the pupil response. No thinking required, which makes it perfect for people like me who don't like to think.

Re:Perfect

[Opportunist]

Careful when your boss is near, you might spell "asshole" before you're realizing it. :)

Re:Perfect

[Anonymous Coward]

Not quite so! Eye tracking has been used for many years (which is looking at letters); but measuring pupil size when participants are looking at a single location. They attend to the letter they want to type rather than actually looking at it. The pupil responds to the bright/dark background, and this is what's picked up by the algorithm. So the study is rather novel, and is a promising system for those who cannot move their eyes.

You could've know this, if you actually read the announcement and/or the paper ;)

Re:Perfect

[OzPeter]

This is perfect. We now don't even need to move our arms to watch netflix. The peak of civilization has now been achieved.

Netflix. Yeah, right.

Teachers having their charges do the typing?

[Anonymous Coward]

Did anyone else think this was going to be advice to teachers to let their charges do the typing instead of the teachers ?

Phew!

[Applehu Akbar]

So this isn't about exploitation of students, then?

Let your pupils do the typing

[PopeRatzo]

Right after they clean the blackboard and erasers.

Old technology.

[140Mandak262Jamuna]

Till about the 1990s, all over Indian even in very small towns there were "Typewriting and Shorthand Institutes" . In those institutes pupils have been typing since 1900s. .

They started morphing into programmer mills churning out dBase III, COBOL, coders and now they teach everything from Java to Ansys Fluid Mechanics R17.1 (Register for two courses and AutoCAD is free!)

Richard Stallman Does This.

[tlambert]

Richard Stallman Does This.

He has his graduate students do his typing for him; too many EMACS related Vulcan Nerve Pinches have left him with severe carpal-tunnel syndrome.

Oh. Not *those* kind of "pupils". Never mind.

Re:Richard Stallman Does This.

[hawk]

EMACS is indeed the only software that ever caused me physical injury.

After a multi-day editing binge on a CKIE keyboard I went to the campus medical center. Muscle strain on my left pinky from rotate/stretch/curl of my large hands to hit control ...

Now, I would never tamper with University property but a couple of days later days later, there was a little piece of plastic next to my keyboard , and the shift-lock no longer toggled allowing me to remap the control key to where God meant it tone. . .

hawk

Re:Richard Stallman Does This.

[djinn6]

I've been using two $10 USB foot pedals for control and meta. It's completely gotten rid of the pinky and thumb strains for me, at least while I'm coding.

Eye Movment?

[wisnoskij]

It sounds to me like they are implying that when the eye focuses on the brighter lights its pupil contracts, and in such a way the device can know which letter you want. Is that not what is happening? Because that definitely requires eye movement; Are they implying that you will just somehow know the differing brightness of the keys without even looking at them?

Re: Eye Movment?

[orngog]

Not implying, stating.

8 minutes!

[Anonymous Coward]

8 minutes to write a 5-letter word Some improvement will be needed before it can go mainstream!

"Attend to"?

[Will_Malverson]

Do you mean "look at"?

Think vs look

[enriquevagu]

They have developed a way to—literally—write text by looking at letters. Great advance!!

Why a binary search?

[marciot]

It appears as if the letters are separated into two groups and that within that group the alternation of brightness is perfectly synchronized to the other letters in the group and opposite the color to the letters in the opposing group. Smaller sets of letters are presented over time until one letter is chosen. So this appears to be a binary search that reads one bit at a time, based strictly on the phase of the brightness signal.

What makes me wonder is why this is so constrained. Could the brightness of each letter be controlled independently to encode the letter directly? Perhaps the user could be presented with a full keyboard, with each "key's" brightness modulated to a different binary code. Presumably then the code of the character that the user was fixating on could be read from the pupil diameter variation directly?