Replacements For Mouse And Keyboard?

Qrious writes "I read Tad Williams' "Otherland" a while ago and was facinated by the interface that is used for controlling computers with your hands. Now I am wondering how far technology is in this field. Aren't replacements for mouse and keyboards around ? Gloves and a virtual keyboard ? A flick of the wrist that changes back and forth betweeen keyboard and pointing device ? Are these things already made ? If not, why not ? I for one would like to work with something faster than a keyboard. Typing in the air or on a table with some kind of gloves instead of a keyboard would take some getting used to, but would be a lot faster when you are used to it. Anyone else have ideas to some other way of interacting with computers that would make it easier on the hands and faster to control ?"

There are certainly a lot of radical extensions of the keyboard as we know it -- chording keyboards like the BAT and the Twiddler, Deep Ergonomic ones like the Kinesis, and many variants on the one-handed keyboard.

IBM of course is now selling ViaVoice for Linux, as well. But are there interfaces that are even more out there available now?

Comments

Ideas

[ecloud]

The usual objection against virtual keyboards is lack of tactile feedback. It's not just to let you know that you pressed a key, but to reclaim some kinetic energy; it takes less work if your fingers bounce back from the keys instead of you having to pull them back. Touchscreens are intuitively compelling, and also bad with respect to this feedback if used exclusively - you're punching fingers down against an immovable object and it could probably be rather fatiguing. Your finger is absorbing most of the impact energy. But maybe the problem is not so bad with gloves, because you don't have to move your fingers quite as forcefully in the first place, and vibratory or audio feedback would be enough to let you know that you hit a key. I'd probably get tired of having to put on the gloves, myself; like how could you eat or drink and compute at the same time while wearing them? (I do this a lot, I'm eating dinner right now.)

I believe that a touchscreen is a good replacement for a mouse; and a trackball would also be an even better replacement under certain circumstances (such as smooth navigation, as opposed to point-and-click). Inspired by the non-existent technology of the Starfire project, I've been pondering the idea of using a combination of a large screen for viewing documents, with no controls on it; a small touchscreen for application-specific controls; a trackball for smooth motion; and a keyboard for typing text. I could operate GIMP entirely via the touchscreen, or a combination of touchscreen and trackball. The trackball could be the 3-axis type which would allow some interesting 3D navigation (but Linux is short on apps which would make use of it, so far). Eventually the keyboard (and some other input modalities) could be gradually replaced by voice recognition.

The metawidget idea (that link is getting old, I need to write about my more recent ideas) would be useful in such a system to separate "control" functionality from the document and view parts of the MVC pattern. GIMP for example would keep a socket connection open with the touchscreen display software, on which it would exchange messages about which controls to make available, and receive messages about which controls were selected. So the palettes, the toolbox, and some context-sensitive stuff would be the main things on the touchscreen, and there would be enough real estate to have many more of them available at once, so that most actions can be done with fewer clicks. Cascading menus should be replaced with something more appropriate for punching a touchscreen (when you "dive in" to the next level, the next level replaces the current level; or perhaps with a menu structure that resembles the NeXT file browser). Eventually I will get around to putting up a website at www.metawidgets.org to discuss these ideas.

In short, I think improving ergonomics ought to be done in a holistic way rather than just putting more bells and whistles on existing devices like mice and keyboards. And there is more than one path to experiment with. I like touchscreens but they are not practical in every situation.

Now, I'm going to go ramble on a bit here about my ergonomic workstation idea, in case you haven't already had enough...

How about a glove that understands sign language?

[Kris_J]

Building motion sensing gloves that are used by pretending you have a keyboard is silly. I think someone should develop some (relatively skeletal)gloves that understand sign language, then you could enter data using an existing gesture set that probably doesn't have any RSI problems surrounding it.

Good basic idea, but not ASL

[FascDot Killed My Pr]

You can't use sign language for the same reason you can't use spoken language--computers can't yet decode a simple sentence.

However, you could use some restricted set of sign symbols with a very restricted "grammar". This would be analogous to saying "open program", "close window", "reboot" etc to your computer instead of saying "I want to calculate my gas mileage and then check my email". The advantage in using gesture instead of speech would be that the physical motion *may* be easier to detect than the speech units.

Of course, since we are now talking about an arbitrary set of physical motions that are intended to convey instructions to the computer we are right back to "virtual keyboards" or something very like it...
--

ditch the mouse, keyboard, glove, etc.

[dutky]

Have a look at the MIT Media Lab Responsive Environments Group and their current projects, specifically responsive surfaces as well as the Tangible Media Group and their senstable and m etaDesk projects.

These projects all make use of various methods of sensing user movement without the need to add extra bits of stuff to the person. One of the projects that I have heard about (I think it is part of metaDesk) uses an array of theramin-like* devices to sense the users hand positions over a table top. The devices are able to sense both hands along with palm and finger orientations.

* A theramin is an electronic musical instrument where the musician plays the instrument by passing her hand between a radio transmitter and reciever, interrupting the electromagnetic field and producing various tones and sounds. This is similar to the effect that you can see on either TV or radio reception as people move around relative to the receiver antena (obviously this doesn't happen if you have cable TV).