Here's a link to our paper at KDD 2013 looking at why people hate your app. We crawled user comments on Google Play for about 100k apps, and then did some clustering and linear regressions to probe what people say when they give you low star ratings.
It turns out that a lot of low ratings often come right after an update, when people find out that their app doesn't work anymore due to incompatibilities. We also found some odd anomalies, like people saying they love your app but gave you a 1 star rating. If you want the very short summary, jump to Table 6. We divided up the comments by app type. For example, for games, people tended to complain about (1) attractiveness, (2) stability, and (3) cost. For other categories, the complaints were less consistent.
The description you have is sort of backwards. That is, fulfillment is an example of an intrinsic motivation, and intrinsic motivations are one way of getting people to do certain activities. The people who do a lot of these MOOCs have a strong intrinsic motivation to want to learn, what to challenge themselves, and have fun doing so. These are also classic examples of intrinsic motivation.
I think you're referring to cognitive dissonance instead (do a search for "boring task" in the linked Wikipedia article).
Also, to anyone saying "people are stupid" or "they deserve to get malware", you really are part of the problem. It's our job to protect people, to reduce complexity, and to ensure the safety of our systems and networks. Arrogantly dismissing others as being inferior or stupid is one reason why computer security, user interfaces, and software in general is in the state it is.
Instead of using the 99.99% figure, use natural frequencies to describe it. This blog post on the NYTimes talks about how people have a much easier time understanding frequencies than prior probabilities. This isn't just a problem of education, cognitive scientists already know that some representations are much easier to reason with than others, even if they are equivalent.
For example, many people get this problem wrong:
The probability that one of these women has breast cancer is 0.8 percent. If a woman has breast cancer, the probability is 90 percent that she will have a positive mammogram. If a woman does not have breast cancer, the probability is 7 percent that she will still have a positive mammogram. Imagine a woman who has a positive mammogram. What is the probability that she actually has breast cancer?
But, a lot of people get this framing right:
Eight out of every 1,000 women have breast cancer. Of these 8 women with breast cancer, 7 will have a positive mammogram. Of the remaining 992 women who don’t have breast cancer, some 70 will still have a positive mammogram. Imagine a sample of women who have positive mammograms in screening. How many of these women actually have breast cancer?
If your doctor smoked one pack of cigarettes every day, and said that you should stop smoking, your doctor might be hypocritical, but they wouldn't be wrong.
As an aside, I know many slashdotters don't believe you can train people to protect themselves from phishing. That is the standard conventional wisdom in computer security. However, we've actually demonstrated that you can, if you make it fun, timely, and relevant. We're commercializing some
micro games for security training and
a service for simulated phishing attacks
based on research we did at Carnegie Mellon.
You have a lot of good points and anti-points (ie things to avoid) with respect to practical implementation of web sites. However, I think it doesn't quite capture the essence of user interface design patterns. What you have is more of guidelines, which are useful, but don't provide actual examples of what to do.
Another way patterns differ from guidelines (and something the GoF missed) is that they should be hierarchical. Take a look at Christopher Alexander's original book on design patterns, and you'll see that he starts with large regions, continues on to cities, to neighborhoods, individual buildings, and rooms. Such a structure helps people simultaneously see the micro view as well as understanding how it fits into the macro view of things.
Part of the reason we wrote our book The Design of Sites was in reaction to things like "web pages that suck." There's a lot of information out there about bad web sites, but not as many practical examples and details on how to create good ones.
I'd accept DRM technologies... for corporations, that is, to use our personally identifiable information in ways that owners deem fit. This paper suggests that DRM might work (though there are obvious difficulties wrt getting corporations and government to use it):
Korba, L., Kenny, S. "Towards Meeting the Privacy Challenge: Adapting DRM," 2002 ACM Workshop on Digital Rights Management, Held in Conjunction with the Ninth ACM Conference on Computer and Communications Security. Washington, District of Columbia, USA. November 18-22, 2002. NRC 44956.
First, the article on ACM Queue is about Natural Programming and NOT Natural Language Programming. Please at least read the article before making an inane post.
Second, the concepts in Natural Programming are quite sound. Rather than re-using programming concepts that people often have difficulty with or are error prone (such as looping structures or boolean statements), they suggest alternatives and provide user tests that indicate that these alternatives have good potential.
A lot of people have covered a lot of great areas in computer science. Here's a short annotated list I've put together for an often-overlooked area, human-computer interaction.
As We May Think, by Vannevar Bush. Bush was the
Director of the Office of Scientific Research and Development, basically the precursor to NSF and DARPA. In this magazine article, he observed the problem of disseminating information, and noted that electronics may be a better medium (keep in mind that this was written in 1945). He also outlines what he calls the Memex, the first description of a hypertext machine. Bush's theme is that we need to create devices that will make it easier for us to store and access information, and ultimately solve problems better.
Sketchpad, by Ivan Sutherland. Couldn't find a link to a video, but this truly is one of the seminal papers in computer science. This paper introduced the first graphical user interface (graphical as in graphics, not windows and mouse), the first object-oriented system, the first zooming interface, and the first constraint solver. Best quote:
"I once asked Ivan, 'How is it possible for you to have invented computer graphics, done the first object oriented software system and the first real time constraint solver all by yourself in one year?" And he said "I didn't know it was hard."
-- Alan Kay on Ivan Sutherland.
The embarassing part is that, although this was done in the early 1960's, Sketchpad still looks cool and useful today.
Doug Engelbart's 1968 Demo. The link points to a video collection, which is easier to read than his papers. Engelbart is not the most exciting speaker, but keep in mind that in 1968 that people were still stuck using terminals and punchcards. What does he show them? The first mouse. The first hypertext implementation. The first use of video-conferencing. The first online help system. The first interactive word processor. Obviously a mind-blowing experience if you were there. As many people have said, this is the mother of all demos, and we still have not achieved many of his visions today.
The Computer for the Twenty-First Century by Mark Weiser. Although this was written in 1991, I think that this might be the most important paper of the 1990s. Why? Keep in mind that in 1991, people were still using desktop PCs, that wireless had not achieved momentum, and that sensors were very few and far between.
So what is the basic idea? That computers should not be constrained to the physical desktop, but should become an everyday and seamless part of our lives. And in this paper, Weiser and his team at Xerox PARC introduced location-based computing; devices of all form factors, from small PDAs to tablet PCs to electronic whiteboards; sensors for integrating the physical and virtual worlds; wireless networking to make it all connected no matter where you were (in their office building anyway). Weiser's vision is so influential, that there are now (literally) thousands of researchers working on what he called ubiquitous computing, as well as several research conferences devoted to this theme, not to mention the direction that the commercial world has already taken with PDAs, WiFi, sensor networks, and so on.
There are two primary reasons for getting a PhD, and they are for teaching at a university, and for doing research at a university or an industrial research lab. If you don't want to do either, or are unsure, then a PhD is probably not for you. The worst decision I've seen by people is applying to a PhD program because they don't have anything else to do and don't know what they want.
One thing that people often don't know is that PhD programs are more about thoroughness, communication, creativity, and endurance than about ability to hack. I'm in the CS PhD program at Berkeley, and I spend more time creating slides, writing papers, reviewing papers, talking to colleagues, and thinking up new ideas than I do programming. I enjoy it a lot, but depending on your goals, this may or may not be a good thing.
The endurance part is something that few people mention. Are you willing to devote the next 4-5 years of your life to research? Is there something that will get you up every day in the morning so that you can finish?
Check out the web pages of schools you'd like to apply to, and find projects and professors there that seem interesting to you. Also, many graduate programs are open to visitors and prospective graduate students. If you have the opportunity, visit those schools and see what the people and the research is like.
This is going to be a lengthy but serious discussion of MMORPGs.
Usually, when I read these kinds of articles about game addicts, I always think, "if only we could use his powers for good!" If only we could make it so that people get more out of games than just fun. If only we could actually get something genuinely useful at the same time (so we don't end up with stories like this one from The Onion).
My canonical example is Crazy Taxi. In this game, you drive a taxi, taking people from place to place in a pseudo-San Francisco city. You get more points for driving recklessly, getting as close as you can to crashing things without actually crashing into them. What if...you could actually learn the streets of San Fran while playing this game? I hate driving there because I don't know what the streets are, because of all the one-way streets, because of all the cars and pedestrians. But what if you could actually learn the streets incidentally while playing the game? You would actually be learning something useful beyond the game console.
Now, analogously, what if we could get something useful out of MMORPGs, more than just entertainment and player-killing?
Here's a crazy idea: what if we could actually simulate real problems of society in MMORPGs and harness the power of players in solving those problems? For example, homelessness or pollution?
What if these MMORPGs were modelled such that they actually reflected real aspects of the world, creating an environment where we could actually experiment with different public policies, or even have the numerous players (who are clearly very intelligent people) try to figure out different solutions to these problems? Try out different ideas that may eventually influence what we actually do in the real world?
One example that's pushing in this direction is University of Washington's UrbanSim, where they try to predict what the impact of different public policy decisions will be on the environment. (They also run tests on old data to make sure their model matches the actual results).
I'm aware of how difficult this would be, all of the barriers in making convincing and realistic models, in making an appropriate reward system to incentivize people, in actually convincing academic scholars in sociology and public policy as well as policy makers that these ideas can be realistically and feasibly implemented with the expected results. (I'm in the Phd program in Computer Science at Berkeley, I have a pretty good idea of how difficult it would be).
But think about the potential here as well. A simulation with thousands of people interacting with one another, where we could try out radical new ideas in solving problems. Think of it as SimSociety. Think of it as a variation of Doug Engelbart's vision, where we need to get better at solving problems because the ones we're facing these days are far harder than anything we've ever seen before. Players could be doing more than just having fun. They could also be making a difference, for the better.
I thought I would provide some useful resources to help developers build better user interfaces.
Paper Prototyping. The basic idea here is to do a quick mockup of the user interface with paper, and then to stick real users in front of it and test it. Why? Because it's fast, cheap, and effective. Rather than spending weeks on features that users might not need or understand, you can do it in a single day.
Contextual Design, by Holtzblatt and Beyer. This book looks at simple techniques for observing how end-users do their work, and then using those observations in the development of high quality user interfaces
Design of Everyday Things, by Norman. The classic book on why design is important, and some guidelines for design.
The Design of Sites, by van Duyne, Landay, and Hong. Ok, a shameless plug, this is my book on principles, processes, and patterns for web site design. We cover things like rapid prototyping, field studies, human capabilities, as well as 90 different user interface design patterns.
And lastly, here is the website for a 3-day Human-Computer Interaction Course that I co-taught last summer. We have our syllabus and all of our slides online, for free.
If I had had to summarize it into two pieces of advice, I would say:
Take some time to understand the end-users, their tasks, their tools, and their social organization. Try to see things from their point of view.
Do several iterations of rapid prototyping and testing before building the real thing. It will help tremendously.
Dan Olsen's Developing User Interfaces. Dan Olsen is a professor of Human-Computer Interaction, formerly at Carnegie-Mellon, now at Brigham Young. his book describes the nuts and bolts of the internals of GUIs, including rendering models, event queues, event handling, graphics, and so on. Depending on your level of expertise, it may or may not be useful.
If you have access to ACM's digital library, check out the proceedings of CHI and UIST. CHI stands for Computer-Human Interaction, and is the main conference in that field, and looks more at design issues and evaluation. UIST stands for User Interface Software and Technology, and looks more at the implementation issues. There are several papers there describing the design and implementation of several complex systems.
(If you have access to a university library, you can find those proceedings there as well)
Following up, check out the work of Bill Buxton. He and his students at Toronto have done a great deal of work with 3D interfaces. In fact, Bill Buxton also works at Alias | Wavefront, who create lots of tools for 3D modeling and rendering.
I'd also suggest looking at the Virtual Reality work done by others. I suspect that they have many issues in common, although the interaction is clearly different. Check out Alice by Randy Pausch and others (at Carnegie Mellon). Alice is sort of like Logo for 3D worlds. Fred Brooks, of Mythical Man-Month fame, also does work in this field. (I don't study virtual reality, so I'm not as familiar here)
You bring up a lot of great points, but I don't think that the original premise, that all interfaces need to be simplified, is correct. I feel that the real question is how to design these interfaces to match the (ever-evolving) context of end-users.
Computer pioneer Doug Engelbart said it better: if ease of use was the only critera, we'd be riding tricycles instead of bicycles. However, this should not be an excuse to make difficult-to-use interfaces! We have to look at deeper issues, including usability, learnability, reliability, and expertise, as well as fundamental ones like flow, aesthetics, user experience, and quality-of-life, when designing systems.
After all, the point of building all these tools and gadgets is so that they work for us, and not the other way around.
Great points, but keyboards don't work as well for small mobile devices, as well as large wall-sized devices (like SmartBoards). Another problem with keyboards is that it's difficult to do collaborative work with them. There's an implicit one-user-at-a-time model built into the modern desktop computer.
There's also a question of leveraging natural human abilities, such as speech, sketching, gesturing, and so on. These are the ways people communicate with one another, why can't we do this with computers?
The key to human-centered computing is to start with real human needs, not with technology. It's about empathizing with end-users, understanding their current tools, their work practices, their tasks, and their organizational context, and then iteratively desiging software to match their needs.
As University of Maryland Ben Shneiderman writes, "The old computing is about what computers can do. The new computing will be about what people can do."
Ease of use and power is not necessarily an either/or proposition as you so bleakly portray it. Certainly, some interfaces are well worth the time it takes to learn. However, a large majority of interfaces are not, because they do not:
Leverage existing knowledge
Leverage existing tasks
Leverage existing tools
Too many user interfaces are ad hoc and arbitrary, and do not focus on the end user and the task they are trying to accomplish. Try understanding your target audience first before dismissing them as "groundlings" or "the masses".
Simple question: do you think that usability development processes and open source development processes can ever be unified? Or are they intrinsically at odds with each other? Or is there a happy medium in between?
Here's my thoughts: interface design and usability seem to be of the "Cathedral" process of development, where a small group designs, evaluates, and iterates on an interface idea.
However, the open source process, or the "Bazaar" process, relies on a large number of people, which makes it difficult to have consistency. Furthermore, my perception is that people in the open-source world get "points" for cool code hacks, not for running usability tests or making an interface aesthetically pleasing.
As a quick follow-up: Be aware of the constraints that the Apple interface developers were working with when they developed their guidelines. Think about what hardware and software was like back then, and consider whether they will scale up well to what we have today. For example, with 20 megs of hard drive, file management was as serious a problem as it is today. Also, there are more and more input and output devices today than back when the guidelines were developed. The innovations that came out of Xerox and Apple were developed by careful observation and rapid usability testing with EVERYDAY people, not just technophiles. If we want great user interfaces for Linux, then we can play catch-up with de facto standards, and at the same time try to leap-frog ahead with true innovations EXPERIMENTALLY PROVEN to work reliably with everyday people.
Slashdot Mirror
Here's a link to our paper at KDD 2013 looking at why people hate your app. We crawled user comments on Google Play for about 100k apps, and then did some clustering and linear regressions to probe what people say when they give you low star ratings.
It turns out that a lot of low ratings often come right after an update, when people find out that their app doesn't work anymore due to incompatibilities. We also found some odd anomalies, like people saying they love your app but gave you a 1 star rating. If you want the very short summary, jump to Table 6. We divided up the comments by app type. For example, for games, people tended to complain about (1) attractiveness, (2) stability, and (3) cost. For other categories, the complaints were less consistent.
How have recent issues like sequestration, reduced NSF and NIH funding, and the government shutdown impacted your research?
The description you have is sort of backwards. That is, fulfillment is an example of an intrinsic motivation, and intrinsic motivations are one way of getting people to do certain activities. The people who do a lot of these MOOCs have a strong intrinsic motivation to want to learn, what to challenge themselves, and have fun doing so. These are also classic examples of intrinsic motivation.
I think you're referring to cognitive dissonance instead (do a search for "boring task" in the linked Wikipedia article).
I wrote up an article in Communications of the ACM about a year ago summarizing the state of phishing attacks.
My colleagues and I have also studied phishing extensively and have the most comprehensive peer-reviewed body of work in this area. Our studies include understanding why people fall for phishing attacks (PDF), evaluating how well simulated phishing attacks work (PDF) (the short answer is quite well, based on a study of 500 people), designing and evaluating a micro game teaching people about URLs works (PDF) (empirically tested with several thousand people), and more.
We've also commercialized our work, in terms of a service for simulated phishing attacks, the micro game for anti-phishing, and more.
Also, to anyone saying "people are stupid" or "they deserve to get malware", you really are part of the problem. It's our job to protect people, to reduce complexity, and to ensure the safety of our systems and networks. Arrogantly dismissing others as being inferior or stupid is one reason why computer security, user interfaces, and software in general is in the state it is.
But, a lot of people get this framing right:
If your doctor smoked one pack of cigarettes every day, and said that you should stop smoking, your doctor might be hypocritical, but they wouldn't be wrong.
Jagatic and others saw this in 2007 in their work on social phishing at Indiana University.
We saw the same in our PhishGuru work at Carnegie Mellon, on training people not to fall for phishing scams in 2009.
As an aside, I know many slashdotters don't believe you can train people to protect themselves from phishing. That is the standard conventional wisdom in computer security. However, we've actually demonstrated that you can, if you make it fun, timely, and relevant. We're commercializing some micro games for security training and a service for simulated phishing attacks based on research we did at Carnegie Mellon.
You have a lot of good points and anti-points (ie things to avoid) with respect to practical implementation of web sites. However, I think it doesn't quite capture the essence of user interface design patterns. What you have is more of guidelines, which are useful, but don't provide actual examples of what to do.
Another way patterns differ from guidelines (and something the GoF missed) is that they should be hierarchical. Take a look at Christopher Alexander's original book on design patterns, and you'll see that he starts with large regions, continues on to cities, to neighborhoods, individual buildings, and rooms. Such a structure helps people simultaneously see the micro view as well as understanding how it fits into the macro view of things.
Part of the reason we wrote our book The Design of Sites was in reaction to things like "web pages that suck." There's a lot of information out there about bad web sites, but not as many practical examples and details on how to create good ones.
I'd accept DRM technologies... for corporations, that is, to use our personally identifiable information in ways that owners deem fit. This paper suggests that DRM might work (though there are obvious difficulties wrt getting corporations and government to use it):
Korba, L., Kenny, S. "Towards Meeting the Privacy Challenge: Adapting DRM," 2002 ACM Workshop on Digital Rights Management, Held in Conjunction with the Ninth ACM Conference on Computer and Communications Security. Washington, District of Columbia, USA. November 18-22, 2002. NRC 44956.
Thoughts?
First, the article on ACM Queue is about Natural Programming and NOT Natural Language Programming. Please at least read the article before making an inane post.
Second, the concepts in Natural Programming are quite sound. Rather than re-using programming concepts that people often have difficulty with or are error prone (such as looping structures or boolean statements), they suggest alternatives and provide user tests that indicate that these alternatives have good potential.
If you want to see an example video of one of the projects, please see:
http://web.cs.cmu.edu/~pane/HANDS/HANDS.MPG (Warning: 73Megs)
So what is the basic idea? That computers should not be constrained to the physical desktop, but should become an everyday and seamless part of our lives. And in this paper, Weiser and his team at Xerox PARC introduced location-based computing; devices of all form factors, from small PDAs to tablet PCs to electronic whiteboards; sensors for integrating the physical and virtual worlds; wireless networking to make it all connected no matter where you were (in their office building anyway). Weiser's vision is so influential, that there are now (literally) thousands of researchers working on what he called ubiquitous computing, as well as several research conferences devoted to this theme, not to mention the direction that the commercial world has already taken with PDAs, WiFi, sensor networks, and so on.
There are two primary reasons for getting a PhD, and they are for teaching at a university, and for doing research at a university or an industrial research lab. If you don't want to do either, or are unsure, then a PhD is probably not for you. The worst decision I've seen by people is applying to a PhD program because they don't have anything else to do and don't know what they want.
One thing that people often don't know is that PhD programs are more about thoroughness, communication, creativity, and endurance than about ability to hack. I'm in the CS PhD program at Berkeley, and I spend more time creating slides, writing papers, reviewing papers, talking to colleagues, and thinking up new ideas than I do programming. I enjoy it a lot, but depending on your goals, this may or may not be a good thing.
The endurance part is something that few people mention. Are you willing to devote the next 4-5 years of your life to research? Is there something that will get you up every day in the morning so that you can finish?
Check out the web pages of schools you'd like to apply to, and find projects and professors there that seem interesting to you. Also, many graduate programs are open to visitors and prospective graduate students. If you have the opportunity, visit those schools and see what the people and the research is like.
This is going to be a lengthy but serious discussion of MMORPGs.
Usually, when I read these kinds of articles about game addicts, I always think, "if only we could use his powers for good!" If only we could make it so that people get more out of games than just fun. If only we could actually get something genuinely useful at the same time (so we don't end up with stories like this one from The Onion).
My canonical example is Crazy Taxi. In this game, you drive a taxi, taking people from place to place in a pseudo-San Francisco city. You get more points for driving recklessly, getting as close as you can to crashing things without actually crashing into them. What if...you could actually learn the streets of San Fran while playing this game? I hate driving there because I don't know what the streets are, because of all the one-way streets, because of all the cars and pedestrians. But what if you could actually learn the streets incidentally while playing the game? You would actually be learning something useful beyond the game console.
Now, analogously, what if we could get something useful out of MMORPGs, more than just entertainment and player-killing?
Here's a crazy idea: what if we could actually simulate real problems of society in MMORPGs and harness the power of players in solving those problems? For example, homelessness or pollution?
What if these MMORPGs were modelled such that they actually reflected real aspects of the world, creating an environment where we could actually experiment with different public policies, or even have the numerous players (who are clearly very intelligent people) try to figure out different solutions to these problems? Try out different ideas that may eventually influence what we actually do in the real world?
One example that's pushing in this direction is University of Washington's UrbanSim, where they try to predict what the impact of different public policy decisions will be on the environment. (They also run tests on old data to make sure their model matches the actual results).
I'm aware of how difficult this would be, all of the barriers in making convincing and realistic models, in making an appropriate reward system to incentivize people, in actually convincing academic scholars in sociology and public policy as well as policy makers that these ideas can be realistically and feasibly implemented with the expected results. (I'm in the Phd program in Computer Science at Berkeley, I have a pretty good idea of how difficult it would be).
But think about the potential here as well. A simulation with thousands of people interacting with one another, where we could try out radical new ideas in solving problems. Think of it as SimSociety. Think of it as a variation of Doug Engelbart's vision, where we need to get better at solving problems because the ones we're facing these days are far harder than anything we've ever seen before. Players could be doing more than just having fun. They could also be making a difference, for the better.
If I had had to summarize it into two pieces of advice, I would say:
(If you have access to a university library, you can find those proceedings there as well)
You bring up a lot of great points, but I don't think that the original premise, that all interfaces need to be simplified, is correct. I feel that the real question is how to design these interfaces to match the (ever-evolving) context of end-users.
Computer pioneer Doug Engelbart said it better: if ease of use was the only critera, we'd be riding tricycles instead of bicycles. However, this should not be an excuse to make difficult-to-use interfaces! We have to look at deeper issues, including usability, learnability, reliability, and expertise, as well as fundamental ones like flow, aesthetics, user experience, and quality-of-life, when designing systems.
After all, the point of building all these tools and gadgets is so that they work for us, and not the other way around.
Great points, but keyboards don't work as well for small mobile devices, as well as large wall-sized devices (like SmartBoards). Another problem with keyboards is that it's difficult to do collaborative work with them. There's an implicit one-user-at-a-time model built into the modern desktop computer.
There's also a question of leveraging natural human abilities, such as speech, sketching, gesturing, and so on. These are the ways people communicate with one another, why can't we do this with computers?
As University of Maryland Ben Shneiderman writes, "The old computing is about what computers can do. The new computing will be about what people can do."
See the original idea for mapping network traffic to the physical world, done 4 years ago at Xerox PARC. It's called the Dangling String, and it's a wire that shakes around. The more network traffic, the more it moves.
http://guir.berkeley.edu/projects/ic2d/
We're at UC Berkeley grad school CS researching user interfaces. You can check out the other stuff we're doing too.
http://guir.berkeley.edu/
Too many user interfaces are ad hoc and arbitrary, and do not focus on the end user and the task they are trying to accomplish. Try understanding your target audience first before dismissing them as "groundlings" or "the masses".
Here's my thoughts: interface design and usability seem to be of the "Cathedral" process of development, where a small group designs, evaluates, and iterates on an interface idea.
However, the open source process, or the "Bazaar" process, relies on a large number of people, which makes it difficult to have consistency. Furthermore, my perception is that people in the open-source world get "points" for cool code hacks, not for running usability tests or making an interface aesthetically pleasing.
As a quick follow-up: Be aware of the constraints that the Apple interface developers were working with when they developed their guidelines. Think about what hardware and software was like back then, and consider whether they will scale up well to what we have today. For example, with 20 megs of hard drive, file management was as serious a problem as it is today. Also, there are more and more input and output devices today than back when the guidelines were developed. The innovations that came out of Xerox and Apple were developed by careful observation and rapid usability testing with EVERYDAY people, not just technophiles. If we want great user interfaces for Linux, then we can play catch-up with de facto standards, and at the same time try to leap-frog ahead with true innovations EXPERIMENTALLY PROVEN to work reliably with everyday people.