Slashdot Mirror
Best Device For Gesture Based Input?
jotaeleemeese writes: "A few days ago there was a discussion about gesture navigation in the Opera browser, that prompted my to buy Black & White, download Opera and get the evaluation version of Sensiva. Being a trackball user, I found gesture navigation too cubersome, I found a mouse not much better either. Then I thought a pen based device or a touchpad could be ideal for this kind of input, but before investing my hard cash buying something, I would like opinions from /.ers that have already tried something with these or other programs using gesture recognition and what the results have been."
If you're looking for a fully gesture-based input system, you may be interested in the KeyBowl, a no-key no-wrist-movement "keyboard". Contrary to what the pictures might imply, the two domes don't rotate. Rather, letters are formed by sliding the domes while keeping the wrists straight.
I don't own one of these (I have a Kinesis Contoured keyboard, which I'm very pleased with), but I'd be half-tempted to buy a KeyBowl if they weren't almost $400 (%$#!).
Alex Bischoff
---
Alex Bischoff
HTML/CSS coder for hire
One thing I noticed about Black and White is that if your input device has very low resolution (say your comptuer is overtaxed and only servicing interrupts every 100ms or so), then the gesture based input can be a real pain, but when I'm on a fast enough machine (with a good precision mouse) the gestures are easy to preform. The problem with slow input is that when you go around a curve, the mouse may only register at two or three points along the curve, and your software will interpolate that into a straight line between those points. If what you are trying to draw is curved, then there is a good chance the recognition software will get it wrong.
Down that path lies madness. On the other hand, the road to hell is paved with melting snowballs.
I read the internet for the articles.
Right click the application while holding shift. Choose "Run as...", enter the user you want it to run as (administrator in this case), enter password. *Presto*.
I think something like the old Nintendo power glove would be great, hit a button, gesture, hit a button to confirm, all of it fingertip controlled. Either that, or a touchpad/screen. What would be more natural for gesture based control than touching the screen and making the gesture?
You say you want a revolution....
I have a Wacom tablet (12x9) for drawing purposes, I figured this would work nicely for doing gesture based motion in B&W... bah... it's worthless for that... I've found that Black & White either has something in it to stop perfect motion from counting (so you can't use pentablets) or I'm a worse pen-writer than I thought... I mean, a giant "R" should work no matter what, and it's not hard to make a spiral. Basically, I stick with the Trackman Marble. It works. (oh, and the funky Wacom "4D" mouse doesn't help either.... wonder if I could configure it's thumbwheel to do nifty easy zooming for me though... hmm)
-Laz
The idea is that the human brain isn't good at discerning differences between short distances, such as "Move the mouse pointer to the menu bar, click within a .5 inch box, scroll down 2.5 inches to the appropriate menu item and release", however it's quite good at producing and remembering changes in directions. So, for instance, File|Save would be "Up, Left".
With just two gestures, it's possible to represent over 48 different actions. Add a third gesture, and that number goes to 288. Their research showed that their average subject had no problem remembering four levels deep!
Gesture interfaces are especially useful as a user-interface for blind people, where it's just not possible to choose items from a menu visually.
The cool thing is that gesture-based menus have been part of the Alias|Wavefront products since 1996.
-James
I gesture at my computer constantly. It doesn't increase my productivity or improve my computing, but it does make me feel a whole lot better....
For surfing p()orn sites and playing Tomb Raider, I have found that a life-size inflatable doll makes the best gesture-based "input" device.
--
Sheesh, evil *and* a jerk. -- Jade
Essentially the mouse has 2.5 degrees of freedom (relative x,y + activate mouse). A stylus is absolute x,y + tap code (morse code?) There are other devices which goes up to 6 (3D + twist, turn, roll). However this basically mirrors the major axes of your forearm. Imagine extending your forearm away from chest clenching a fist and moving or rotating the fist. Some other devices (e.g. wand) have been invented to mirror broad gestures but I don't think the tech is quite there yet to do real-time sign language which relies on alignment/spacing of the fingers. The problem is as always software applications which can recognise the gestures. Anywhich which is 2.5 degrees can be adapted to map to a mouse based input but you are really limiting yourself once you go to higher degrees of freedom. It would be easier to work out the complexity of the system you want to control, then work out the dimensions you can partition it into, then work out the type of device which best suits your needs. Note that even a high-end stylus can add extra levels of complexity such as angle, pressure, acceleration ... think caligraphy. The other factor is human support in that it is tiring to continously hold up your pinkie all the time. Tricks are to mount any coordinate device on something like glasses or helmet.
LL
As a guy who plays Black & White with a Logitech iFeel mouse, I've gotta say your initial take on mice needs to be revisited. Having the mouse kick back when you do something right, wrong, powerful, whatever, that means a lot, and it helps you get used to doing things the right way.
The only drawback is that it's too tiring for day-to-day use. I usually leave the feedback turned off when surfing the web, for example, because it just beats your wrists to death as you glide over a zillion links. I've got carpal tunnel, and the buzz that it makes when jumping over hyperlinks makes my wrists feel like they've been typing for hours.
It's remarkably cheap, too - it was $45 on the shelf the last time I looked.
What's your damage, Heather?
I got the hang of the controls quickly, but eventually I found that the mouse works well for maneuvering around, but was still unreliable when casting miracles. Often the gestures for casting a miracle would have to be slow and deliberate, and even then would take 2-3 trys to work. When you're trying to cast a miracle in a hurry, this is unacceptable.
I dug out a touchpad I won as door prize at a conference (which I hadn't opened until then) and gave it a shot. Casting miracles with a touchpad was an incredible improvement, the movement was much more intuitive to do with a finger rather than your whole hand. However, maneuvering in Black & White with touch pad is a pain, and in some situations it was impossible to use (fighting and other quick point-and-click motions).
So I got a serial mouse and starting using the mouse to play the game and the touch pad to cast miracles. This, IMHO, is the way the game was meant to be played. After a while, you'll find things that work better with the touchpad, such as interacting with your creature, and things that work better with the mouse, i.e. fighting and catching followers to feed to your creature. Give em both I shot, it's worth the $50 or so for a touchpad.
--- Rectum?! Damn near killed em'! - Confucius
I am currently using a Dell CPxJ for browsing with Opera. I had the regular drivers installed for the PS2 mouse...they blew for this. I downloaded the Alps drivers. This allowed me to click and drag, right click, everything could be set so I just placed my finger and drug...all's done
I think the key to anything is find something you are comfortable with, and then just make it work. Don't spend a lot of money on something you aren't going to be happy with. And when you do get it, don't half ass it!
------ This has been provided as a public service! ------
Grafitti (handwriting recognition in PDAs) is very much the same as the gesture based input in these systems.
Think about the act of dragging your mouse as the act of writing a little scrible to represent a letter that is places serially along an input stream.
They are very similar.
Eh...
For drawing pictures freehand on a 'puter nothing beats it. Pressure sensitive and integrates with Adobe and Corel. Darker, fatter lines when you press hard, lighter thinner lines when you ease up. You can actually sketch with this thing. Has a similar feel to a soft pencil or the spongy tipped ink pens. Put a piece of soft plastic over the tablet to provide a better feel of resistance to pen strokes. Nothing rough though. Anything rough will actually give you the effect of gravestone rubbings. It transfers the grain of the paper you are using to provide resistance directly to the screen. Yes, it is that sensitive.
If voting were effective, it would be illegal by now.
I noticed that some people seem to assume that the gestures in B&W are in the plane of the screen. The manual reads that these gestures are 'drawn on the ground'. I've found that gestures are more successful in the game if they are distorted to account for the slope of the terrain. Hence, doing a 'perfect' gesture on a tablet or with a mouse won't work too well, it needs to be put into perspective...
i am endorsed for the carrying of dangerous goods, please be giving me your depleted uranium
I have used a pen based system at my school to do drafting and some drawing, and I found it quit a pain. I was able to use the pen for normal mouse type movements, but the thing got to really be a pain. It eventually started to hurt my wrist, because it wasn't quite the right size and i had to move my fingers around strangely to grip it. So...i don't really like the pens...
:)
I also use a glidepad occasionally on my father's laptop. I hated it at first, but then I got used to it and I can use that more effectively than a mouse (as long as i avoid finger drag). If you go the glidepad route..make sure you get one that you can click by tapping.
The anti-salmon
I've only got experience of a mouse with gesture recognition, so I can't speak for any other device.
What I have seen is how much the 'refresh rate' of the mouse's position (temporal frequency?) affects the usability of gestures.
I've bought Black and White, and it has serious issues on Windows 2000. As in it doesn't run at all. Fantastic.
I've got a triple-boot machine (Slackware/Win98/Win2k), so I'm forced to run B&W in Windows 98 where the update rate of the mouse is pretty appalling.
Getting B&W to recognise some of the more complex gestures is a pain because the time between updates of mouse position gives the gesture considerably more 'jaggy' edges, making it look less like what you actually did with the mouse.
Windows 2000 has the refresh rate pretty high, so I'd have thought it's far easier to use gestures successfully on there.
I've not used the mouse much under Linux; my dedicated Linux box doesn't have a monitor, let alone a mouse, I just use it over ssh or X-Win32, so I don't know if the PS/2 refresh rate has been increased (or is configurable); the last I saw was that it wasn't particularly fast.
Opera's gestures are fairly simple (so far), not nearly as complex as some of B&W's gestures, so the rate isn't as critical. But, add more complex ones and you will see the difference.
It's not a new technology by any stretch of the imaginatio (emacs strokes mode anyone?) but it's very useful; even something as simple as Opera's 'back' gesture is so convenient, I wonder 'why didn't they put this in earlier!'.
Nice one Mr. Molyneux; he was always the king of games back in the good old days of Atari STs, and now something from his latest game seems to have started a bit of trend elsewhere in the software business.
This was an ancient piece of software that took your average Joystick and turned it into a mouse. Buttons = Buttons. Navigation was by moving your Joystick around. Beautiful and elegant. I can only imagine the "enhancements" that could be made with all the new force feedback and related controls on the new joysticks.
Imagine feeling a "kick" in the joystick to warn you not to click on the "Goat sex" link.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
Nah, it's great.
A whole 5 levels,
Which take a long time to complete, at least for me.
and WAY too much wood required to do anything. If I wanted to do the same task over and over and over again for hours on end...
Yeah, I think the wood requirements are too high, but you can do this:
Step 1: Teach creature to cast wood miracle
Step 2: Attach creature to village store with the compassion leash
And how do you become evil?
I dunno, my problem is becoming good; my hand looks all orange and veiny, with sharp fingernails.
- Synchronize a "frame" from the point of view of every camera. You must already know their "absolute" positions, which is relative to some zero-point. (Determined by the original location of the calibrator).
- For each pixel that a given camera sees:
- Assume that you are seeing a pixel at the nearest point that the second camera in your stereo set could also see. To draw a human comparison, bring your finger closer and closer to your eye, until with your other eye it passes the line of your nose and you can't see it anymore. This is the "closest point".
- Calculate where this point would appear in the other camera, as well as the sorrounding blocks of pixels, and see whether it matches what the other camera in the stero pair actually sees.
- If it doesn't match, assume that it must be farther than you initially assume. Repeat process.
- Repeat until you "converge"...ie, get images where many pixels in the area "line up" as calculated by the assumption that they are at absolute point x,y,z. This process actually is very similar to what your eye does if you ever notice when it's scanning for how far away something is. At first it assumes it's close, then keeps looking farther and farther away until the two images are brought together. Your brain is the only thing bringing the two images together! Your eyes are still an inch point five apart, silly.
:) In the same way, for each pixel (or rather, group of pixels large enough to identify a small area on an object), our software's "brain" converges the image for various distances until it finds a match.
- If you cannot find a match, assume that the other camera in the pair is not seeing that particular pixel, either because something near you is blocking the nearest area that the other camera is seeing, or because something near the other camera is blocking the line of sight that goes to what you're seeing, or because it's outside the line of sight of another camera entirely. This last is easiest because you don't even need to scan the pixels you know only one camera sees.
- Repeat this process for each stereo pair.
- Assemble every picture you have an absolute coordinate from (that a stereo pair can see) into a three-space.
Note that I've left out such things as massaging the image from different cameras in various ways (color, brightness, etc) to get them near, using more or less fuzzy "matches" depending on how much you might expect an object to differ at different angles, and calculating lighting sources based on the calibrator. While these are serious issues, they're really basic math stuff that's well-explored in the field of optical recognition, and it's basically a cut-and-paste of components, and, like I said, a $5,000 server can do thirty frames per second without having any graphics hardware specifically enabled for this stuff. The number of three-space "pixels" it ends up getting varies with conditions, but you can always do well enough to read standard braille that's reasonably close in proximity (1.5 feet) to a stereo pair of cameras. Needless to say, there are more useful applications to these kinds of technology than reading braille on your computer screenDeveloping a gesture recognition system. I did not mean to outline everything I did above, but it really is not involved, and a lot more viable than some people think. Anyway, the interesting thing about the three-space that you develop from the process above is that it is very easily analyzable. Not only do you have a solid "block" of where pixels are, but it's easy to tell lines that separate, for instance, individual fingers that overlap. In fact, the human brain uses more picture analysis than stereoscopic analysis, and our system is actually more precise than the human brain at finding the exact location of a point two or three feet away relative to a point near it, compared with the human brain, if you are given no color clues! When looking at a hand, therefore, we can pretty take the basic shape of a hand and (here is where we get tricky) apply a very fuzzy algorithm for fitting it to the hand that we actually see. It is "fuzzy" almost to the extent of being neural-netty (although we control it very much), since it not only needs to choose between an infinite number of ways that two hands can contort themselves, but also learn the size of individual aspects of it (which changes slightly), and their shape, and for this purpose also takes into account where the hand "used" to be in the previous frame, how fast it was moving over the previous few frames, and how likely it is to move in a certain way, with respect to speed and with respect to what positions are unnatural. All this is necessary to get 30 frames per second, because we aren't just interested in the "position" of the hand, but its important aspects (the relative bend in each joint). To test, we have another application that is ONLY given the absolute position of hands and the relative joints we are measuring, and then reconstructs the hands visually. You can therefore have all three programs running, the stereoscopic analyzer feeding the hand-position recognizer data, and the hand-position recognizer feeding the renderer data, so that your screen shows how the renderer is getting the info about where your hands are. Mostly, however you move your hands will be reflected on the screen, but if you move it very quickly and unusually you can still confuse the hand-position analyzer and get an image that's out of sync with what your hand actually is doing. This is independent of the stereoscopic anaylzer, which comes up with the correct data, which if you feed directly to the renderer you see always matches what your hand is doing, at 30 fps.
So now I've outlined how we get the position of joints, which includes quite a bit of fuzziness. But by far the most fuzziness is not in this, but in the actual "recognition" of a GESTURE. We've already gotten the first-generation information about what a gesture is by spending several hours each in front of a test server set up for it, already equipped with a popular voice command system, and agreeing to surf the web and do various other tasks the voice command system is equipped for (we didn't make that, it's just purchased off the floor somewhere) while also doing the gesture we have set up for each command. So we end up with "sample" gestures to analyze, and have already manually looked at the major indicators and drawn them up and programmed them. The way we have done the first time is very crude, however, eyeing as we have each sample ourselves, but we are now in the process of collecting second-generation information, so that when a user successfully uses a gesture and doesn't complain that it wasn't what he wanted, that particular instance of gesturing gets put into the database of gesturing instances associated with a gesture, and we are developing fuzzy logic to link these gestures more closely and reliably. The gestures make sense for the most part, such as having your right thumb open to the left with your other fingers closed, in a quick leftward motion to go back, or up and with a quick rightward motion to be right. Stopping is pushing your palm forward toward the screen, closed a window is putting your finger and thumb together and drawing your hand back, as if you're flicking the window away, and refresh is a sweeping gesture with your palm toward you, from bottom left toward top-right (only a small part of the way). The software recognizes a "gesture" because you perform it particularly fast and deliberately, so if you playing with your hands slowly, it doesn't misrecognize any of these.
Anyway I'm getting really tired of typing all this, and even though there is much, much, more, I'm just kidding. Wouldn't all this be cool though?
~
...anything that attaches to the head. It seems counter-intuitive, but could you imagine using a helmet or visor to play Quake? I can see the whiplash lawsuits at Id's door now... "Our client, D34thFr0mAB0V3, suffered a fractured sternum while attempting to 'rail-snap' a third party..." -t
IBM has a new laptop that is awsome for gesture navigation. It is large and heavy, but it opens up with a notebook on one side, and the laptop/monitor on the other. It has both a normal laptop mouse and a pen mouse. The pen mouse can be used on the screen, or on the pad beside the laptop. It comes with a documentation program that allows you to write/draw into the software itsself =) Its _REALLY_ cool... the pen allows you to do gesture type actions just like you where writing them down!
I find the mouse is excellent for this sort of thing. However, I have a Logitech Mouseman which fits my hand perfectly and I have very high sensitivity set and acceleration turned on. A gesture for me means moving my mouse within an area no bigger than about 1/4" x 1/4". Most people have their mouse sensitivity set way too low.
The only better device would be a 3D glove since you could do 3D motions, which gives a much larger domain for your gestures to be in, probably making it both easier to remember them and less likely you'll mess them up. But don't sneeze or you may delete you root directory.
BTW, Black and White sucks. A whole 5 levels, and WAY too much wood required to do anything. If I wanted to do the same task over and over and over again for hours on end I'd get a job in a factory and get paid for it. And how do you become evil? I taught my creature to eat people, I destroy entire villages, I set people on fire, fling them into mountains, sacrifice 'em all over the place, starve them to death and I'm a GOOD God? They got some good weed down at Lionhead, uh-huh.
I use the VersaPad from Interlink Electronics (69.95). It works VERY well with all these programs, and I strongly suggest getting one. Even if you have a mouse and want to enhance gesture recognition, it is a vey good deal.