RSI, WIMPs and Pipes; What Next?

Tetard asks: "Long live the pipe! Since the `|' was invented by Doug McIlroy in 1973, has there ever been a more effective way of reusing tools and connecting data ? The mouse is a device of the Beatles era; Rather than try and provoke nostalgia in the older ones among us, I'm asking myself, as are others: when we don't try to reinvent the wheel, or at least improve it, why must we try and copy it every time ? Xerox PARC exposed us to WIMPs and we haven't done better: some innovation, some plastic surgery -- but no "paradigm shift" -- where's the creative destruction that will take us further ? Graphical component programming is turning us into click-happy bonobos^H^H^Hchimpanzees, as we fail to find new ways to manage and connect richer data streams. My web designer friends are damaged for life because of mice, and yet we persist... Where do we go from here ? If we ever invent the graphical pipe, let if have keyboard shortcuts." Yes, you've probably seen a similar question to this run by Ask Slashdot before, but this time I'm wondering if maybe we need new input devices before the WIMP paradigm is replaced with something better. Might any of you have ideas on what form these input devices might take?

For those interested, here are the previous stories that have handled this type of question:

• GUI Research - Is it Still Being Done?
• Pipes In GUI's?
• Pipes in GUI's? (Redux)

So what it will take to break us out of the WIMP box (or prison, depending on your bias), maybe new input devices would do it, but quite frankly, I wouldn't be surprised if a 3D interface might be another route (it would possibly spark interest in designing a new input device that would work better with 3D interfaces, or maybe data-gloves could serve this purpose?). Going on a limb, maybe this guy might just be the ticket.

Comments

Face Recognition

[Sludge]

With a sub-$100 webcam watching you, look at the point of the screen where you would click, and blink.

Are there lots of problems to doing this? Yes. Should that stop me from throwing out the idea? No.

Re:Face Recognition

[DarkFyre]

This can be done, and is done for physically disabled people who cannot use a mouse. The problem is that a sub-$100 'webcam' doesn't cut it. You need high-quality video in reasonably real-time in order for this to work. The cameras are more like $400-$500 (CDN, so that's $250-$350 USD).

The main problem with the systems right now is that they cannot track only eye movement. You need to use your whole head for large-distance traversals. You think RSI sucks in your wrist, wait until you start getting neck cramps from your favorite RTS game.

Re:Face Recognition

[JanneM]

Except that the eye isn't all that accurate. You can generally get a resolution of about 1 degree, no more. It's because the fovea (the sharp area of the eye) is about 1.5 degrees, and the eye will skip around slightly all the time within that area (the visual system needs a bit of movement at all times).

You can probably do a decent interface using that kind of accuracy, but you won't be doing any kind of precision work.

/Janne

Re:Face Recognition

[megaduck]

Great idea, but it doesn't get us out of the WIMP paradigm. You're just replacing your mouse with a more efficient type of pointer.

Really, that's the problem in a nutshell. We are so used to the WIMP interface, that the best we can vizualize is an improvement of the WIMP system. Until we can come up with a totally different metaphor for interacting with our computers, we won't see WIMP go away. Personally, I think it will take a "mad genius" type to break out of the WIMP paradigm and move us forward.

Re:Face Recognition

[sporty]

Until you sneeze and reboot your computer :)

Re:Face Recognition

[slamb]

There's a time and a place for diplomacy, and this isn't it.

<div diplomacy="off"> That's a dumb idea. </div>

My attention wanders and consequently my eyes do as well. I blink subconsciously. You can't change these things, and it's stupid to try. Don't make an input system that fails when they happen.

I think Douglas Adams does a good job of describing the failings of this type of input system:

A loud clatter of gunk music flooded through the Heart of Gold cabin as Zaphod searched the sub-etha radio wavebands for news of himself. The machine was rather difficult to operate. For years radios had been operated by means of pressing buttons and turning dials; then as the technology became more sophisticated the controls were made touch-sensitive - you merely had to brush the panels with your fingers; now all you had to do was wave your hand in the general direction of the components and hope. It saved a lot of muscular expenditure of course, but meant that you had to sit infuriatingly still if you wanted to keep listening to the same programme.

Douglas Adams, the Hitchhiker's Guide to the Galaxy

That's not even taking into account the way the eye inherently jitters, according to the other replies. Even without that, this wouldn't work out well except for extremely disabled people who have little other choice and aren't likely to complain about something that makes it at all possible for them to use a computer.

WIMPs? Pipes?

[cperciva]

What do Weakly Interactive Massive Particles have to do with pipes?

NLP

[Zen Mastuh]

Natural Language Processing has my vote. Some of these folks are working on it already. Wouldn't it be nice to say "move this thing over here", or some other combination of speech and gesturing, rather than all these inane menus and clicks? Someone still needs to develop the pipe infrastructure, tho. Just *don't* make it so narrow as to become worthless.

Re:NLP

[JanneM]

Nope. It will have a role in niche applications, but I don't see it ever being the dominant method of interaction. A couple of reasons:

Imagine a large room full of office workers. Now, imagine the same room with every worker talking to his wordprocessor or spreadsheet, trying to make him or herself heard over all the others, getting irritated and fatigued because of the constant noise of everybody else talking to their computers.

Imagine trying to do some work in an airport or on an airplane. Now, imagine trying to do the work using your voice _without_ other people hearing the budget details for your company or hearing the steamy endearments you will be mailing to your spouse.

Imagine talking. Now, imagine constantly talking all day, every day. Some actors and singers get permanent damage to their vocal cords - and they've had professional coaching and access to medical services. It could become RSI for your throat.

/Janne

You know what else?

[aiken_d]

We're still using *wheels* on our vehicles! I've got a nearly brand new car, and it's using 5,000 year old technology to get around? From Detroit to Stuttgart, automakers are clearly devoid of imagination!

Get over it -- sometimes it makes sense to stick with what works. A love of change for the sake of change is the one thing that unites geeks and fresh-out-of-school MBA's.

-b

Re:You know what else?

[srvivn21]

We are still using wheels, but we don't use them exclusively, and they have been *massively* improved over those 5,000 years.

If I want to make a trans-continental journey, I use wings. To ride across the harbor, a hull, and hydrodynamics. The wheels on your nearly brand new car are a far cry from the round stone of yester-century. They are even more advanced than the wheels you would find on a car just 20 years ago.

Change for the sake of change is one thing, but change for the sake of better usability, safety and economy is completely a different matter.

I for one appreciate the ride (and style)I get from my air-filled, steel belted radials wrapping polished aluminum rims.

Re:You know what else?

[mangu]

they have been *massively* improved over those 5,000 years

Can you name a single improvement to the concept of the wheel in those years? AFAIK, they are still a round thing that revolves around an axis. Sure, the machining precision of that roundness and that axis are many levels of magnitude better than 5000 years ago, but the concept is still the same.

New discussion site for post-WIMP interfaces

[Nooface]

This is exactly the topic of the new Slash site that I just set up at http://nooface.net. The goal of the site is to promote out-of-the-box thinking about truly next-generation user interfaces that are designed for new types of users and computing devices, and go beyond the WIMP (Windows, Icons, Menus, Pointing Device) method that most current interfaces are based on.

Nooface
In Search of the Post-PC Interface

Emacs, naturally

[tmoertel]

Might any of you have ideas on what form these input devices might take?

They might just take the form of Emacs.

I'm serious.

No, really.

Nothing gives me hand pain as quickly as using mice, especially ones with that wheely thing. Keeping my hands in good form over the "home row" of my keyboard -- and away from the mouse -- has virtually eliminated pain from my computing life. I spend half of my waking hours in Emacs, and I have come to love (and depend on) its there's-a-key-for-everything nature.

Dump the mice. Keep Emacs.

Why replace the current GUI paradigm?

[kurisuto]

I'm wondering if maybe we need new input devices before the WIMP paradigm is replaced with something better

This seems a bit like asking what it would take to replace the current way of driving a car (steering wheel, gas and pedal brakes, etc.) with something better. But the interface between humans and automobiles is pretty much a solved problem, and nobody seems to spend much time speculating on what a paradigm change in automobile control would be like.

There's a curious assumption which I've seen repeatedly-- namely, that a paradigm shift in human/computer interaction would be a good thing. Why, exactly? I see no reason to pursue a paradigm change for its own sake; I view it as a problem which has basically been solved for now, much as the problem of steering cars is a solved problem.

Re:Why replace the current GUI paradigm?

[tim_maroney]

There's a curious assumption which I've seen repeatedly-- namely, that a paradigm shift in human/computer interaction would be a good thing. Why, exactly?

That's an excellent question. By Kuhn's model of paradigm shifts, the shift must be preceded by a number of anomalies in the current paradigm. In command-line interfaces, the anomalies were numerous -- the need for constant relearning of old habits, the need for memorization, the ease of making errors, the computer being in control of the human rather than the other way around, etc. Eventually social factors caused the anomalies to be recognized as such, so that when a new paradigm was created, its values were widely recognized. Perfect recipe for a Kuhnian shift.

What are the anomalies today which would force a change in the paradigm? Serious question, not rhetorical. For starters, I'd say Gelernter's new project is an attempt to rectify some anomalies which have not yet attained social recognition as anomalies.

Tim

Gross motion v. fine motion

[www.sorehands.com]

Part of the problem is with fine motion control v. gross motion control.

Add a touch screeen so that when it comes to window selection less fine manipulation, you can use the large muscles of the arm and shoulder, then use the mouse for the finer that you cannot get with a touch screen.

The more important aspect is the comfort and the breaks. No matter what mechanism is used, trauma can accumulate over time. You need time to allow the body to recover from that trauma.

You are missing the point

[SClitheroe]

Use the right tool for the right job. That's the UNIX/Linux philosophy. If you think that the WIMP interface is cramping your style with respect to whatever task you are trying to accomplish, then you are using the wrong tool.

With regards to the command line or WIMP interfaces being old, and not particularly forward looking, you are also missing a fundamental point: A graphical "pipe" isn't innovative either. You're simply shoehorning two paradigms together, and even worse, two totally incompatible paradigms at that. The pipe is a useful metaphor and operator for stream-oriented I/O. The WIMP is useful for (obviously) visually oriented information, and its designed for a completely different purpose than the pipe. The WIMP is designed to allow humans to manipulate data and abstract objects in a visual manner. The pipe is designed to allow users to allow the computer to do the same, without intervention.

If you want an innovative computing interface, worrying about streams, or visual representations of data is a waste of time. You're going to have to come up with something totally new. One good example is the use of sound to communicate the health and performance of networks or systems.

3D Environments will lead to change

[Bonker]

The real problem with computer software advancement is that its all very firmly based in 2D land. WIMP environments are about as eficient as you can get. Even assuming that you can wrap your work habits around something as 'next-gen' as 'The Brain', you're still stuck in 2D land.

The real advance that will open innovation (real innovation and not some corporation's twisted idea of it) is the beginning of a 3D workstation environment.

We already have the primatives for this kind of environment in games like Unreal, Q3A, and Black and White. Assuming that we implement a 'graphical pipe' that will work for a truly 3D application system, ie: Allow 3D applications to pass information back and forth between each other semi-effortlessly, this will ignite a new 'interface revolution' similiar to what we experienced as a result of Xerox's early WIMP system and the first versions of Apple's MacOS.

Once programs and applications can truly be represented as 'objects' in a 3D environment, we'll end up with something like the 'God' interface in Black and White, where processes are represented by animated people and files are represented as other objects. Tasks best handled in 2-D such as composition, coding, or painting will still be easy to handle, but tasks best performed in 3-D such as file management, database management, and even some advanced programming tasks like linking and compiling files, will take place in a representational environment. Imagine opening up your HDD and pouring objects into it, then sorting them into containers based on type, as you would sort files into directories.

Eventually, I see us moving into something like Stephenson's 'Street' metaphor for shared environments.

Along with these advances, will come new interfaces. I think that eye-tracking cameras have the biggest potential, but we keep coming back to the data-glove in one form or the other. I know CADesigners who still have an old Nintendo powerglove hacked for basic 3D manipution tasks. We're also probably going to see a renaissance of 'body tracker' devices that will track human motion via sonar or laser. Any one of these has the potential to vastly reduce RSI injuries.

The real trick in jumping from 2D to 3D is reverse compatibility. All the shells I've seen that attempt 3D interaction do it badly. Even then, they fail completely when faced with most of the tasks we do on a daily basis, like write or paint. I think we're going to have to use 'easels' or something similiar. In Cowboy Bebop, Radical Edward's computing environment is shown as a multitude of 2D windows hovering around her in 3D space. This wouldn't be that hard to do, really.

Navigation will be the true challenge for any 3D application designer. It will be that itch to scratch that will spawn new, inventive input and coding ideas.

Re:3D Environments will lead to change

[Junks Jerzey]

The trouble with most GUI desktops is that they are designed for manipulating items on a GUI desktop and customing the GUI desktop rather than making the interface transparent. As a counterexample consider the Palm, where the idea is to make the UI be as lightweight and unobtrusive as possible, because people want to just take notes and view their schedules. A WIMP desktop is overkill, so they went with something an order of magnitude simpler.

A 3D desktop is a step in the opposite direction, placing more emphasis on the desktop itself than what people want to do.

Re:3D Environments will lead to change

[Mr. Sketch]

This would be good, but it would require a 3d interface. I think the only way to truly do a 3d environment is for it to exist in the physical world. Any 3d interface on a 2d screen will become kludgy pretty quick. The best way to do a fully 3d interface is to put in in the real world. Imagine if your desk *WAS* the computer. The desktop *WAS* your actual desktop. You open your draw to see 'real' manilla folders with names on the tabs for your documents and thumbing through them to find the financial report you were working on, 'grabbing' it and pulling it out and it appears on your desktop for you to work with. You open up another draw and see pens, pencils, markers highlighters, etc that you then 'grab' to select what you want to start writing with. You could just slide your hand across the desktop to move documents out of the way and tab or 'grab' a document that was 'under' the one you were working on and it comes to the top and you can begin working on that one.

This would require a lot of holography and motion tracking, touch sensors, etc, but it would be the ultimate in 3d interfaces. It would avoid klunky HUDs and gloves, etc that just detract from the actual work. You could even bring up a keyboard on the desktop and use that instead of a virtual pen or pencil.

3d interfaces would be nice, but on a 2d display I think it's best to stick with a 2d interface.

Forget WIMPs

[sulli]

WIMPs are such old hat. The new metaphor is MACHOs.

(WIMP = Weakly interactive massive particle; MACHO = Massive compact halo object)

Graphical Pipes

[EnglishTim]

There are several programs out there that make use of graphical pipes - Maya, Eddie, Shake, grphedit (DirectShow graph editor) to name but a few.

The nice thing about graphical pipes is the ability to easily and transparently connect several forms of data to one node. With command line pipes, you've effectively only got one input and output.

What would be needed would be graphical terminal programs (built into the OS, or at least window manager) for connecting these things together, Oh, and a standardized way of defining input and output types - I dunno - would MIME work there?

I expect somebody can point us at a project that has already done this?

A combo of input methods and predictive interfaces

[jht]

To improve on the classic WIMP interfaces, I immediately would exclude anything that had to be physically connected to the user. That eliminates data gloves as a significant mainstream input device.

Gloves work well technically (at least from what I've seen of them), but are fairly inconvenient to use. I suspect that some basic speech recognition will be mainstream in the not-too-distant future, because processing power is cheap enough to handle simple speech without impacting performance. Maybe eye tracking will be used some as well, but eyes tend to wander.

So I think the biggest trend in interface design over the next few years is going to be a return to simplicity. Fewer clicks, fewer mouse movements, and greater use of predictive interfaces - where the interface guesses what you'll do next based on experience and learning, and has it ready for you just in case. The Mac-style version of the UI (but not necessarily the Mac itself) will probably be the dominant strain overall - Microsoft is converging in that direction now as well as evidenced by the Luna interface in XP. I think mainstream mice return to two buttons (from the currently popular multi-buttoned models), or maybe even one. And mouse gestures will be used more instead of clicking in some cases.

Mice themselves I can see working by gyro rather than by ball or light sensor, which would allow a mouse to be held and moved within 3 dimensions (even if it only tracks in two), rather than skated over a desk in two dimensions. It's potentially a more natural hand motion. Keyboards probably won't change much due to inertia. They haven't really changed in basic layout for over a hundred years.

Of course, I may just be blowing smoke, since my intuition is no better than anyone else's. But the Slashdot crowd is very different from the "mainstream" - we are typically more forgiving of a complex interface and would trade off in favor of power over simplicity. So we may not be the best people to forecast.

Of Mice And Music

[istartedi]

The mouse is a device of the Beatles era

And the steering wheel is a relic of the Jazz age. I don't plan on giving up either of them any time soon.

NLP != Voice Recognition

[Greyfox]

A lot of people think NLP implies voice recognition and to a large extent that's true, but it also means being able to communicate with your computer using plain english sentences. Difficult because English is so damn ambiguous (Note to self: Perhaps Klingon would be a better place to start...) Still, if I want to tell my computer, "Book the least expensive flight from Denver to Miami on the 23rd of this month." I should be able to say that to the computer or type it and have the computer understand what I'm telling it to do. I should also be able to modify my command to specify times or airline or both and my computer should be smart enough to say "There's a flight for $100 less 5 minutes earlier than the timeframe you specified. Would that be acceptable?"

That sort of thing will be the wave of the future, and it will mean that apps will have to be smarter and communicate a lot more than they do today. My personal agent should reside on my local machine, not the network, and should watch out for my personal privacy. It should divulge only what is necessary to others in order to perform the commands that I give it. It should be flexible and configurable, but I should never have to configure it; it should learn what I like by how I interact with it.

Several large companies have been working toward this holy grail for years, but thus far not even common voice recognition much less NLP has emerged from their research. Sure there are some voice recognition packages out there, but there's very little integration, and AFAIK nothing at all in the NLP arena. We could start working toward the level of integration that would be a necessary foundation for a lot of this stuff, but I don't know that you could get the necessary level of cooperation in ANY software development community.

We need Star Trek interfaces

[AKAImBatman]

No, I'm serious. Computers should be working for us, not against us. Today, we don't give a computer a task and ask it to complete it, we instead worry about interfacing with it. The interface should be natural and change depending on what you are doing. Let's go through this using Star Trek (TNG) as our baseline:

Computers should have voice recognition for general tasks such as environment management and collabertion. The crew uses this to dim the lights for example.

Computers should have keypad interfaces for general reading and writing. You'll notice the workstation on Picard's desk for exactly this purpose.

Computers should have remote/networked interfaces for passing sneaker-net info. This means datapads and remote voice uplinks. (Hold on captain, I'll email the info to your chair! Yeah, right.)

Computers should be able to present information in the most efficient way possible. This doesn't mean the prettiest, but instead it means dynamic graphical/audio interfaces. These interfaces must be easy to create and use. This means that the readouts might one moment be tracking the progress of Warp plasma then next moment display starcharts of the ship's course.

All of this makes the Star Trek interface very useful and is quite possible with current technology. However, there is one thing that really makes it easy to use:

Computers must be able to automatically analyze, process, and display any information. Filters should be able to be applied automatically. Artifical intelligence should be able to translate speech from any language to any language and add new patterns without reprogramming. Answers to theoriretcal questions should be able to be answered given a current set of information on a subject. Information should be able to be retrieved and stored no matter what the format and new formats should be able to be analyzed and added to the database.

This is much more difficult due to the fact that computers today have a very monolithic nature. Pipes are a good example of dynamic linkage of programs, yet at the same time are very primitive. They require a great deal of operator understanding of the information in order to use.

What if we could build layers of software on top of layers of software, which are then again used to build greater layers of software? This would require that we accept standard functionality at each level of the layering process and then allow people to write ever simpler code due to this great deal of layering. Why should anyone be required to rewrite a quicksort once it has been written?

The first and best key we have today is the Java class file format. Class files are allowed to layer on top of others by nature. The computer can pull apart the structure and investigate the usefullness of a class for a specific function. With enough layers, the computer can make extremely intelligent decisions automatically. Witness JINI.

So here is my challenge to every programmer/newbie/manager/hardware designer in existance. Stop trying to focus on how we can optimize or tweak each individual computer function so that you gain temporary marketshare, outperform your competitor, or push your political agenda. Instead, begin pushing for hard standards in the industry. Don't worry if they have the "highest performance" or if they have the "coolest widgets". Instead worry if they have the best design, if they are technically "right" for the task at hand. Stop worrying about making languages that allow you to produce specific functionality in fewer lines of code, and worry about producing the highest level of quality.

Many people will find this a hard pill to swallow, but the end result will most certainly be worth it.

Re:We need Star Trek interfaces

[funwithBSD]

Yeah!
And the intellegent chair can read my ass for authentication!
Should help with my diet/training program as well...
"OOF! Based on your cheek patterns Captian, I am reprogramming your workout for an extra 15 minutes, and I have revoked your Krispy Kreme access on the replicator. Have a nice day" ;)

Re:We need Star Trek interfaces

[slamb]

Artifical intelligence should be able to translate speech from any language to any language and add new patterns without reprogramming.

Oh, okay. I'll have that for you tomorrow.

I'm really tired of people making these dramatic statements that are all but impossible to realize. There are people who have spent their careers trying to do what you mentioned in that paragraph, and they've gotten nowhere, because the task is so enormously complex. Translating from one language involves taking one set of nonsensical rules and ambiguities and replacing them with another. It can not be done without a complete understand of what is being translated. That's...hard, to make the understatement of the year.

What if we could build layers of software on top of layers of software, which are then again used to build greater layers of software? This would require that we accept standard functionality at each level of the layering process and then allow people to write ever simpler code due to this great deal of layering. Why should anyone be required to rewrite a quicksort once it has been written?

There are already so many levels of complexity in any piece of software that it's completely insane. Your idea is nothing new. Here's a partial list of the layers involved in a Java Virtual Machine I'm sure I am leaving many things out, putting them in a bad order, etc.:

1. Gates. Operations on ones and zeros.
2. Hardware components. I'm sure there are layers I am leaving out here and much more descriptive ways to say this...but I have only a token knowledge of these layers.
3. The instruction set architecture of the machine. Already a huge step up, this is a layer at which I can (slowly and painfully) produce useful software.
4. Assemblers. They take somewhat human-readable code and generate machine code from it. (Ever had to do this by hand? Not fun.)
5. High-level programming languages. They take relatively abstract concepts like loops, functions, etc. and create assembler from them.
6. Operating systems. (At this point, the "layers" metaphor shows its imperfections...high-level programming languages and operating system functionality are orthogonal, meaning they abstract different things. My order is arbitrary.) They make it possible to access devices in abstract ways, plus a million other useful things for software.
7. The core library. (libc on C-based machines.) Makes system calls to the operating system easier. Implements stuff common to lots and lots of programs.
8. Other libraries. There's actually many layers hidden here. For example, on UNIX, libxpm depends on libXt which depends on libX11 which depends on libm which depends on libc.
9. Various other processes, like the X11 server. These interact with client-side libraries through communication primitives provided by the operating system layer and made more accessible by the libc layer.
10. The Brahm GC. (For Kaffe. I don't know what GC Sun's VM uses or where exactly it fits in.) This makes memory allocation easier.
11. C-side Java support stuff. Part of the String class implementation, for example.
12. The virtual machine. Implements a completely different instruction set.
13. Java-side support libraries. A lot of the java.lang stuff, for example.
14. More Java-side support libraries. GUI stuff. This depends on stuff below. Extend to arbitrary depth.
15. Your code.

My point? Layering software is nothing new. If you want to add a layer, say so, but don't pretend it's a new concept.

So here is my challenge to every programmer/newbie/manager/hardware designer in existance. Stop trying to focus on how we can optimize or tweak each individual computer function so that you gain temporary marketshare, outperform your competitor, or push your political agenda. Instead, begin pushing for hard standards in the industry.

You know, I really hate it when people push for other people to make dramatic changes I suspect they don't understand (evidence: above) and use the word "we". It's entirely inappropriate.

Stop worrying about making languages that allow you to produce specific functionality in fewer lines of code, and worry about producing the highest level of quality

The language is one of those layers you advocated. The language itself is a piece of software to be reused. You suggested people add layers to make things possible with simpler code, and this is one way to do it. Don't knock it.

Re:Graphical pipes

[tim_maroney]

Pipes seem necessarily linked to the command line.

It's not even so much the command line as it is the pervasiveness of ASCII text for information storage throughout the system. Almost every program currently available for MacOS or Windows would have to be changed to start storing their files as ASCII text rather than as custom binary formats.

As XML conversion continues, this may become more feasible. However, few programs will use XML as their native format for efficiency reasons. Programs will need to have options for XML input and output.

This leaves aside the fact that pipes are a programmer-only feature, which no one else wants or could possibly use.

Tim

Possibly speech but needs more

[Mr. Sketch]

A speech interface would be nice, but only if it was supplemented with a standard mouse and keyboard (and maybe a touch screen) and would accept natural language commands. As far as the user interface goes it should have a complete abstraction from applications and the file system leaving the user to only be concerned with documents.

The reason they should also have mouse and keyboards are for security so passwords etc wouldn't have to be spoken (see the recent user friendly strip series for a humerous take on that), and so things you're doing could be kept somewhat private. Imagine starting up a long build or whatever on your machine and figuring you'd take a short break while everything compiles and telling your computer 'open mozilla. go to hot asian chicks dot com. click hot and horney', you might get more than a few head turns from local cube dwellers unless you bookmarked it and renamed it to something like 'intranet' but the renaming process would also have to be vocalized.

It should also accept natural language commands for complicated to speak text. The main example for this is programming. If I wanted to do:
for (int i = 1; i = 10; i++)
cout << i << endl;
I would like to just say 'for loop. local integer i from zero to ten step one begin. print i and end line. end loop'. instead of having to articulate each puntuation symbol as 'for open parenthesis int i equals 1 semicolon i less than equal ten semicolon i plus plus close parethesis. enter. c out less than less than i less than less than end l', not to mention if I had to put spaces in there too.

The next thing we would need is an abstraction from the use of applications and the file system which would go in very well with a speech interface. The user would only be concerned with documents and data. The user would just ask the computer to start a new report on photosynthesis and the computer could ask the user what to call it and they could just respond with a natural name like 'biology 101 mid-term'. Later the user would just ask the computer to open the biology 101 mid-term without having to care if it was opened with word or starwriter or kword, etc, it would just be there and they could work on it.

The abstraction from the file system would be a natural extension of this because the user doesn't need to know where anything is because the computer takes care of it for them. The user just needs to remember documents/files as he would anything else 'I was writing that letter to Bob', 'I was working on the bio mid-term', etc. This also furthers the use of a computer as a tool, because it would actually help you get things done and be easy to use by anyone because speech is a natural interface for us, but keyboards and mice are not.

The best example I can think of having something like a touch screen is for web browsing or editing documents/preparing presentations, drawing (but maybe a graphic tablet would better for that), etc. so instead of telling the computer to open the 'Read more' link, I could point and it would open whatever I pointed to.

Microsoft is trying to do this with things like the My Documents folder and automatically naming documents with the first line of the document, but it's still somewhat cludgy because it relies on keyboard and mouse interaction. They are kind of on the right track in terms of abstraction from applications and the file system, but still needs a ways to go. This is why they have the Documents folder in the start menu and New Office Document and Open Office Document on the start menu instead of the programs menu. This is also why they have extension associations with applications so the user can just click on a document and it will spawn the right application (or maybe they just stole it from macs).

These ideas are nothing new, I've seem them all somewhere else before, but I just thought I'd post them here for discussion because I think they're good ideas. It should also be noted that this type of interface is for the 'average' user not the average slashdot reader since we all like our keyboards and CLIs.