Slashdot Mirror
Conducting a Unix Desktop Usability Study?
cyclop asks: "I am a close friend of a Ph.D. student on human interface usability. She's now working to tailor a KDE-vs-Gnome usability study (a pretty hot topic these days), and I have been called to help, as a long time GNU/Linux desktop user. What kind of advice -- both technical and theoretical -- would you give us on conducting a deep and objective study on the Unix desktop, that can be useful for the developers and the OSS community?"
"She has installed GNU/Linux and used both KDE and Gnome to get to know them, while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects. However, I feel that it's not enough and that there are a lot of potential pitfalls and misconceptions that wait for us, me being a geek and she being a Linux newbie. Moreover, she found that most of the previous studies on the web were quite sloppy, in comparison with common usability research standards."
The out-of-the-box setup is a compromise at best; and shouldn't be used to judge the overall usability for people who use the system more than once.
Get a slashdot poll on the topic and read the insightful comments.
They who would give up an essential liberty for temporary security, deserve neither liberty nor security
Get people who are not experts, see how many problems they run into doing simple tasks that they're familiar with on Windows. See how many of these they can solve themselves. Start half of them on Gnome and move them to KDE, do the other half in the reverse order.
It is probably also worth noting that most people (apparently including Linus) consider KDE more powerful, so KDE is kinda at a disadvantage.
I for one would like to see a study involving not just how easy it is to learn an interface, but once learned how productive one can be in said interface. For instance, I am proficient in both KDE and Gnome (and a myriad of other WMs which aren't mentioned here), but I feel I can get the most work done faster in KDE. Of course I do tweak quite a few aspects of KDE, but I digress. I would really like to see a productivity evaluation between already proficient users, confident with their skills on their respective interfaces, performing a series of common tasks and comparing the results.
Working in a DevOps shop is like playing in a band made up entirely of keytarists.
My advice? Don't have someone who's been a long time GNU/Linux user assisting her. Chances are, you're fond of either KDE or Gnome. Before the study has even started, I'm alarmed by potential bias. Let her do the study on her own, gather the facts and come up with her own conclusion. Isn't that what Ph.D.'s do?
"[...] while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects."
To me, the study is already flawed. You've dropped a load of information onto her lap, while a complete "newbie" doesn't have that same luxury. How can a usability study be unbiased in this manner? Who's to say you didn't provide her with REALLY good links to KDE information, while giving half-assed links to Gnome?
I like big butts and I cannot lie.
It's not clear that one can easily do an objective study on usability, as it can mean very different things to different people. It should at least be done with segregated populations (e.g. power-users vs. novices).
Some examples:
* A novice might look for how obvious it is to do a certain task, whereas an expert user might instead prefer what can be done fastest (e.g. notepad vs. emacs).
* Related: How much time does this person use a computer/this application can be an important factor. If I rarely do 3d design, I want to be told how to do everything, and have obvious controls (i.e. > 3 parameters might boggle my mind). However, if I work for Pixar, the verbose messages, and dumbed down controls (i.e. 30 parameters might just not cut it for what needs to be done).
* Certain paradigms might make sense to people who are used to using certain types of systems. Files and folders make perfect sense to many people, but certainly not to everyone (e.g. my mother). We think these simplified analogies work better for novices, but that isn't always the case. People think differently, and different analogies will make more/less sense dependent on their world view.
But if you must, at least don't do KDE vs. Gnome. What's the best possible outcome of that? ("So in summary, Gnome tended to be less confusing for newbies, but power users preferred the configurability of KDE...")
Instead compare either or both against Windows or Macintosh for tasks that your _specific target userbase_ would do. [If you haven't defined one or more use cases you've already lost.] This would be much more valuable.
Better yet, switch your topic to focus exclusively on accessibility (a11y). Every DE out there needs some accessibility love.
The first thing to understand is that you will have 3 groups of users:
#1. The ignorant users: These have never used a desktop before. These aren't as easy to find anymore. I worked with one woman back in the mid 90's who could not even use a mouse. She had to hold it still with one hand while she clicked the button with her other hand. After a week of solitare, she had the necessary muscle coordination to start learning the system.
#2. The tainted users: These have experience with systems other than the one you're testing. If your system isn't 100% like the one they're used to, they'll waste time clicking around where the functions are on their systems.
#3. Friends: These have worked on the system that they're being evaluated on.
Now, a system that is easy to learn for the "Ignorant" class may be incredibly un-friendly for more advanced "Friends".
Determine what functionality you want to measure and what GROUP you want to measure it for.
The real "ease" on an interface comes down to 2 things:
a. Can you quickly guess where a function is based upon your existing experience with it?
b. Once you know where a function is (you guessed at it before, you asked someone, you went to training), how easy is it to remember that 24 hours later, 1 week, 1 month, 6 months later?