GUI Research - Is it Still Being Done?
Davor Buvinic asks: "In my spare time I like to study about GUIs. Recently,I was amazed with the new design that I saw in the previews of the future MacOS X, until I discovered in theWeb that things like file dialogs attached to windows dated from the earliest prototypes from the Apple Lisa (July 1981). My question is: Is there any news in GUI design? The newest design I probed was Rob Pike's ACME user interface for programmers. Is anybody
(individual or research center) working in a new GUI design? I mean a GUI for the mainstream, no immersive virtual 3D environments, that probably need a powerful Silicon Graphics to run." Have we done as much with the GUI as we possibly can, or are there other reasons behind the lack of technical innovation in most desktops?
You see them all the time in little Internet toys, and media programs. Look at the KAI graphics stuff for example. Look at any of the "skinnable" applets. I've seen some MP3 players that look downright weird.
The problem is, I hate most of them.
I'm afraid that GUIs (as they exist in the mainstream now) have been hard-coded into our brains. New GUIs have a backwards compatibility problem like you wouldn't believe; they have to be backwards compatible with people.
Unfortunately, we've learned the current GUIs so well, that any major departure is just "wrong."
Check out this link to see what's going on at CMU's HCII. All sorts of wacky stuff...
Shine on, you crazy diamond.
Secondly, there's much refining being done in the area of the GUI. Just look at some of the enlightenment screenshots to see what I'm talking about. Different, but very powerful. (Those screenshots have sucked more than a few new users into Linux!)
Everything else has been a "refinement" process in the area of GUI research. So, here's my idea for a new GUI:
One of the best features of the newest refined GUI's is customizeability - the ability to choose what the OS looks like. Let's take that to the maximum - a generic plugin-based system that lets skin authors completely change the feel of the OS. The User Interface would load plugin modules (swappable at will) that perform the following functions:
- Task management - switching between windows on the screen.
- File management - browsing the files on the hard disk
- Program launching - starting up programs from some sort of menu
- Menu management - if one is loaded, the active program's menus are displayed in this widget, ala MacOS X or NeXT.
- Others I can't think of right now...
This would allow users to completely change the look and feel of their desktop interface. The UI could switch from a convincing Mac clone to a Windows clone to a BeOS clone to a Palm clone to something completely new and uncharted in a matter of seconds! Of course, it would still be based on the same ideas of dialog, widgets, etc. as current interfaces, but it would be a step towards complete user-control of man-machine interaction.I think with the limitation of a 2D display with less than 100 dpi, we have approached the limit of what can reasonable be done with a modern GUI. The original GUIs came about as the result of cheaper raster based video hardware becoming available and supplanting the previous character based hardware.
The attempts at 3D GUIs don't do anything for me, when the display really isn't 3D, and that icon in the distance is illegible because my screen's resolution sucks. We need better and radically different hardware before any major advances in user interface design can occur.
-josh
I read an interesting, online-only article at Linux Journal about a 18 months ago on a topic called "color reactance". Essentially it advocated (and partially demonstrated) how you could have programs set "traffic lights" (or window frame colors, or something) to indicate states. For instance, a program that needs attention could be set to flash yellow whereas one that is finished could flash green (or whatever).
When I first say the Aqua screenshots, I thought Apple had done this. They have a trio of traffic lights on the upper right of every window. But it turns out they are just eye-candied versions of the old close/minimize/maximize buttons.
--
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
CLI's rely on human memory... we need to learn to speak the computer's "language", and often need to remember what the computer is currently up to. GUI's rely on our visual pattern recognition abilities. We "see" the commands we want to execute, and have a "finder" or "taskbar" to remind us what is going on. In both cases, the interface is driven by our choices of how we want to communicate with the system, and once you make that decision, a lot of the rest of the design is mostly asthetics.
The change will come when an interface that is obviously better than typing and clicking comes along. Whatever it is, it will need to be enough of a step up to be worth learning. There have been hundreds of "better" keyboards, but they don't get adopted by people because they are not enough of an improvement on the crappy qwerty (or dvorak) that we already know how to use. The next step to succeed will most likely be something completely different than a keyboard, and it will introduce the need for a radiacally different UI.
Information wants to be anthropomorphized.
- zoomable UIs (Pad++)
- two-handed user interfaces (e.g. toolglasses and magic lenses)
- smarter desktops(e.g. Apple Data Detectors, LiveDoc, CyberDesk)
The ACM CHI and UIST conferences over the past 10 years are full of good stuff that hasn't made it to production yet. It'd be great to see some of these ideas incorporated into open source projects.Check out the User Interface group which is part of the numerous research groups at Microsoft.
There are many universities such as MIT, Georgia Tech, and CMU which do user interface research, but studies (conducted at CMU) have shown that it takes a long time for advances done in universties to reach actual products.
--weenie NT4 user: bite me!
--weenie NT4 user: bite me!
"Computers are nothing but a perfect illusion of order" -- Iggy Pop
So, no, GUI research ain't dead. ("It's pining for the fjords."
To the editors: your English is as bad as your Perl. Please go back to grade school.
A lot of posters here make references to CHI (Computer-Human Interface) research groups at various universities. This just skims the surface. (Do a google search for "computer human interface" or "human computer interface" and follow any of the many links you'll find).
Is GUI innovation dead? Well, one of the things CHI people are working on are ways to improve GUI design. However, as is sadly too common, there is a huge barrier between what academics find and what is adopted in industry.
Remember: although Apple did do a *lot* of original work with GUIs, the core ideas came from academia (Even the Xerox PARC team were former students of Doug Englebart, the Stanford researcher who laid the important groundwork).
But where are the bold, new, designs? Why do all the improvemnts still look like dialog boxes and buttons?
Well, there may be hugely innovative stuff yet to be done - but the field is old by computer science standards. Most of the major ideas of how to get humans with keyboards and mice to interact with computers have already been done.
So does this mean *all* UI innovation is done? Nope. The old hardware assumptions - the human had a keyboard and mouse, the computer had a video display (and maybe a sound system) - will be overturned.
You will be able to use your eyes and hands to let the computer know what you want. Or, if that isn't accurate enough, you can still use the mouse. You can speak when that's more efficient, or type if typing would be faster (For things like "(" or "{" or "["). If your finger and eyes aren't accurate enough to point, go ahead and use the mouse.
All of these new ways of interacting with computers will lead to new ways of presenting data, and new ways of allowing users to modify data. The innovation won't be in GUIs alone, but a combination of GUIs with newer input/output devices.
Don't ask about innovation in GUI design, ask about innovation in human-computer interfaces overall.
Completely new paradigms are also being worked on - Ken Perlin's Pad is one good example, as is David Gelertner's Lifestreams.
PDA intercases, at least the better ones, are also an area of active research. WinCE is mostly a scaled-down WIMP UI, but the Newton is not. The Newton makes pervasive use of gestures (and not just handwriting - even cut, copy, and paste), as well as sound, animation, and a lack of anything resembling a desktop, "saving" files, or even files at all at the user level.
General references to UI research include Ben Schneiderman's textbook (good for learning just how complex the field is) and Baecker et al's collection (which has some of the recent results) and the pages of SIGCHI, the ACM's Special Interest Group for Computer-Human Interaction.
-----
Klactovedestene!
Something clicked while reading the AntiMac page.
The Trashcan/Recycle Bin metaphor should be extended. When you empty your trash can, the contents should be placed in Dumpster on your LAN. If you realize that you've deleted a file that you needed, you can go dumpster diving. Of course the LAN will have a twice weekly pickup, so if the garbage truck has already come, you'll have to travel to the Landfill (a tape/CD-RW archive of deleted files) to retreive your file.
Somehow, it seems kind of fitting to have a Dumpster icon appear in a Windows NT/2000 server window under Network Neighborhood, and a Landfill icon when you click on Entire Network.
The most disturbing things about Liunx GUIs is that the architects--and I hesitate to call them that--are not paying attention to any research or good advice. There are a number of good books and online resources about GUI design, and many of them go off in very different directions than Windows. So, yes, there is research going on and there are alternatives, but no one is listening. "Gotta clone Windows!" is the battle cry.
Two good examples are the Genera environment from Symbolics and the system software of the Apple Newton. The latter of these is astounding. It does away with a filesystem, and is based on scraps of information that are indexed and compressed on the fly, invisible to the user. Lisp Machine fanatics can tell you about Genera.
The biggest flaw of KDE and GNOME is that they aren't designed to solve any particular problem. They're just nebulous environments with doodads and gadgets. KDE, for example, seems to have been developed solely to allow people to tinker with and customize KDE. And what a lot of effort and code has gone into a project without a real point.
It would be nice to have a GUI that was more fitting for the small and well-engineered Linux kernel. A 1970s terminal window misses the mark. So does a crufty, minimalist interface sitting on top of X Windows. Are there any real alternatives besides the jump to KDE and GNOME?
Unfortunately, there are a couple of stupid reasons why pie menus aren't widely used. One is technical and one is political.
The technical one has been the lack of plug-in component architectures that allows new widgets like pie menus to be integrated into new and pre-existing applications. The other is that companies like Alias/SGI are abusing the patent system to discourage their competitors from using useful techniques like pie menus.
Some of the technical problems have finally been solved for Linux and X11! Thanks a lot to everyone who contributed.
NeWS took a stab at solving of those problems a while ago. You could download piemenu.ps to the NeWS window server, and replace all the linear menus in the system with pie menus, and download pietab.ps and replace all the window frames with tabbed window frames, that let you drag the tab anywhere around the edge of the window, and pop up optimized window management pie menus.
"Bring to front" was up, "Push to back" was down, the "Stretch edge/corner" submenu had 4 corners and 4 edges in the appropriate direction, so you could mouse ahead into the pie menus very quickly once you learned them, etc. Pie menus are great for window management, since the tasks are so spatial and you use them so often, you soon learn to mouse ahead very efficiently, it saves you a lot of time, and is very reliable.
The litmus test for a pie menu window manager is that you should be able to reliably start up programs and manipulate windows, even while the window system is busy starting up, paging and thrashing virtual memory, and only slugishly responding to input events. Mouse ahead is that good! Pie menus must be very careful how they synchronize and handle input events, never dropping any mouse clicks on the floor!
All that PostScript pie menus source code I wrote is freely available, but only runs on NeWS, which would be more effort than it is worth to resurrect.
When NeWS died and I had to start use X11 on a regular basis, I hacked pie menus into one of the window managers (that I called "piewm"), so I could use them to control the windows and run programs without going crazy with frustration at linear menus. That source code is also freely available, and it probably still works ok. But the code is not very reusable or up to date, since the X11 window manager is monolithic and does not use any plug-in component framework. It would be better to start with the following code instead.
When I ported SimCity Classic to Unix in 1992, I used the TCL/Tk toolkit, and implemented a Tk pie menu widget for the game, to select between city editing tools (bulldozer, road, residential zone, etc). I distributed the source code for the TCL/Tk widget for free, but it was not widely used in other applications, because it required a programmer to integrate the C and TCL source code into another program, then recompiling and relinking. At the time, TCL/Tk did not have a dynamically loadable component framework.
Microsoft has developed OLE (aka ActiveX) to solve this problem. It allows components written in any language to be loaded dynamically at run time and integrated with any other language, and it allows programmers as well as more casual interface designers to plug components together and configure them with property sheets.
I implemented an ActiveX pie menu control, so that pie menus can be used on web pages and in other Windows applications. The source code as well as the binary is freely available. Now it is quite easy for other people to integrate ActiveX pie menus into their own applications and configure them to their liking.
I've used ActiveX pie menus as a vehicle to experiment with all kinds of different layout and interaction styles. They've got lots of property sheets to set all the various modes and attributes, and you can type in a nested submenu tree as an indented text outline.
I implemented graphical menu items, but I still want them to be animated. A while ago I started adding the ability to read and write nested pie menu specifications as xml. I wanted to add all kinds of other features, but there needed to be an easy concise way to read, write and configure them all. I finally realized that I had hit a brick wall with ActiveX, in the face of all the complexity and things I wanted to be able to do with pie menus, compared to what could be done on a web page with dynamic html.
I want each menu item to be any dynamic html object, like a movie, or a java applet, or an ActiveX control. And I want the graphics and interactive feedback to exploit the full capabilities of dynamic html, like making the point size of the label grow continuously larger as you move the cursor into the slice.
I realized that it was going to be impossible to play keep-up with the capabilities of a web browser by adding feature after feature to my little ActiveX control, and what I really needed was for pie menus to be specified in xml, and implemented inside the web browser using dynamic html on the web page itself, instead of using a shrink wrapped plug-in control that opens and draws its own windows, but can't interact with the rest of the web page.
So I have basically shelved the ActiveX pie menu, and decided to rewrite pie menus in JavaScript and dynamic html, if I ever get around to it, and if the browsers ever get around to supporting dynamic html.
In the mean time, I have been working on the political problems that have kept pie menus and other useful techniques from being widely used.
I was at the computer game developer's conference several years ago. Since I was using 3D Studio Max at work, I stopped by the Kinetix booth, and asked them for some advice integrating ActiveX pie menus into their 3D editing tool.
They told me that Alias had "marking menus" which were like pie menus, and that Kinetix's customers had been requesting that feature, but since Alias had patented marking menus, they were afraid to use pie menus or anything resembling them for fear of being sued for patent infringement.
I told them that sounded like bullshit since there was plenty of prior art, so Alias couldn't get a legitimate patent on "marking menus".
The guy from Kinetix told me that if I didn't believe him, I should walk across the aisle and ask the people at the Alias booth. So I did.
When I asked one of the Alias sales people if their "marking menus" were patented, he instantantly blurted out "of course they are!" So I showed him pie menus on my laptop, and told him that I needed to get in touch with their legal department because they had patented something that I had been working on for many years, and had used in several published products, and I didn't want them to sue me for patent infringement.
When I tried to pin him down about what exactly it was that they had patented, he started weasling and changed his story several times. He finally told me that Bill Buxton was the one who invented marking menus, that he was the one behind the patent, that he was the senior user interface researcher at SGI/Alias, and that I should talk to him.
So I called Bill Buxton at SGI/Alias, who stonewalled and claimed that there was no patent on marking menus. He said he felt insulted that I would think he would patent something that we both knew very well was covered by prior art. I told him that companies try to made illegitimate patents all the time, and that I did not mean to insult him by repeating to him the misinformation that his marketing people were spreading around the computer industry, in his name.
I tried to explain how Alias's FUD had adversely effected the user interface design of 3D Studio Max, in spite of user requests, but he did not care about 3D Studio Max, since Kinetix was his competition. I asked him whose side he was on, the users or the patent lawyers.
He claimed to be on the side of the users, since he is such a well known user interface researcher, but I believe he has totally sold out to the point of abusing the patent system for profit, and is in the thrall of SGI corporate lawyers. Users beware.
A year or so later, I ran across a marking menu patent issued to Alias, that is probably the one the Alias sales people were spreading rumors about. Now it all makes a lot more sense in perspective.
At the time I found out about it from Kinetix, Alias had just applied for the patent on marking menus. The Alias sales people had heard about it, but could not keep their mouths shut, even though there were damn well supposed to. So they repeatedly spread Fear, Uncertainty and Doubt by bragging about this PENDING patent that they really didn't know much about. The only reason I ever learned about it, was that their FUD was so successful if effected Kinetix's plans.
When it got back to Buxton that they had leaked news of the pending patent to Kinetix, which was supposed to be secret, he was furious, but certainly wouldn't tell me what was really up, so he took his anger out on me instead. He wanted to keep me in the dark, so I didn't go to the U. S. Patent Office and inform them of all the prior art that was conspicuously missing from his patent. But I'll bet he was sure proud that the leak about the patent successfully discouraged Kinetix's plans to put marking menus into 3D Studio Max. It's a textbook example of successful FUD!
Anyway, I did not let that discourage me from my long term plan of incorporating pie menus into a mainstream product (The Sims from Maxis). That is the only way that a lot of people will ever be able to see them and get used to the idea.
Now, when the users of a program like 3D Studio Max demand a feature like pie menus, companies like Kinetix will not be fooled by FUD spread by corporations like Alias/SGI. They will realize that their kids play a game that has pie menus, and they seem to work ok, so there must not be anything wrong with using them for a 3D graphics editing program.
-Don
Pie menu web page:
http://www.catalog.com/hopkins/piemenus
Notes from a talk about Pie Menus I gave to BayCHI at Xerox PARC:
http://catalog.com/hopkin s/piemenus/NaturalSelection.html
A description of ActiveX pie menu features:
http://catalog.com/hopk ins/piemenus/PieMenuDescription.html
Take a look and feel free: http://www.PieMenu.com
This includes:
- The whole "registry" thing.
- Similarly, when applications 'talk to one another,' whether via OLE, COM, CORBA, RPC, HTTP, or ICE, this has rather a lot of effect on system behaviour, even when the protocols hide "below the skin."
- The use of serialized data transfer protocols ( e.g. - the "Save File" dialog) as opposed to persistent database schemes similarly can make systems work way different even though the appearance of what gets shown on screen may have minimal difference.
These three "views" all have in common that they have nothing to do with which GUI library you're using to build your applications, or what icons are used.There is various information that needs to be persistent to one degree or another. On Windows, this tends to be saved in the Registry of "renoun and much denigration."
On Linux, such data typically sits in the hordes of files in /etc and in $HOME/.*rc
The semantics of how this all works has rather a lot of effect on how applications start up, even though it sits "under the covers."
It's a small additional step to get to "transactional" systems, where once updates are "committed," they are really permanent. Think Tuxedo/Encina...
The fact that they're not particularly "visible" does not make them any the less important in the overall scheme of things.
After all, if the gentle user can shut down (perhaps pressing the power switch!), and expect to power up again tomorrow and have everything go to where it was when they pressed the switch, that has lots of effect on user behaviour, whether they "click on save" continually, or not.
My thought here is that a lot of the "HCI" changes taking place don't always need to involve things that are manifestly graphical. A Massively Improved World may "simply" involve systems that are reliable and provide persistent data as opposed to "3D Rotating Splash Screens."
If you're not part of the solution, you're part of the precipitate.
I'm assuming you've got a Windows system. Those who run Linux, like me, can easily emulate this train wreck in X with GTK, KDE, Xt, Motif, Athena, and straight Xlib applications.
Barf. Barf. Barf! Death to skins everywhere. Give me a good-looking, powerful, *standard*, incredibly intuitive interface. Hopefully someone's researching this.
- A.P.
--
"One World, one Web, one Program" - Microsoft promotional ad
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
Desktop
You are standing in an open field to the west of a bar. There are some icons in the bar.
>examine icons
You can't see any icons here!
>e
Launcher Bar
This is a narrow room with passages leading west to the Desktop and north to an xterm window. In addition, a set of stairs leads down into darkness. There is a Netscape icon here. There is a StarOffice icon here. There is a gaim icon here.
Your pointer is glowing with a faint blue glow.
>click netscape
What do you want to click the Netscape icon with?
>click netscape with pointer
A violent rumbling comes from the ground. A previously unseen door opens to the southwest, revealing a brightly colored splash screen. After a moment, the rumbling stops, and the splash screen is replaced by an instance of Netscape Navigator 4.72, process number 5188.
Your pointer has begun to glow very brightly.