Slashdot Mirror
Conducting a Unix Desktop Usability Study?
cyclop asks: "I am a close friend of a Ph.D. student on human interface usability. She's now working to tailor a KDE-vs-Gnome usability study (a pretty hot topic these days), and I have been called to help, as a long time GNU/Linux desktop user. What kind of advice -- both technical and theoretical -- would you give us on conducting a deep and objective study on the Unix desktop, that can be useful for the developers and the OSS community?"
"She has installed GNU/Linux and used both KDE and Gnome to get to know them, while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects. However, I feel that it's not enough and that there are a lot of potential pitfalls and misconceptions that wait for us, me being a geek and she being a Linux newbie. Moreover, she found that most of the previous studies on the web were quite sloppy, in comparison with common usability research standards."
The out-of-the-box setup is a compromise at best; and shouldn't be used to judge the overall usability for people who use the system more than once.
Get a slashdot poll on the topic and read the insightful comments.
They who would give up an essential liberty for temporary security, deserve neither liberty nor security
It *does* sound like perfect academic paper fodder.
Get people who are not experts, see how many problems they run into doing simple tasks that they're familiar with on Windows. See how many of these they can solve themselves. Start half of them on Gnome and move them to KDE, do the other half in the reverse order.
It is probably also worth noting that most people (apparently including Linus) consider KDE more powerful, so KDE is kinda at a disadvantage.
I for one would like to see a study involving not just how easy it is to learn an interface, but once learned how productive one can be in said interface. For instance, I am proficient in both KDE and Gnome (and a myriad of other WMs which aren't mentioned here), but I feel I can get the most work done faster in KDE. Of course I do tweak quite a few aspects of KDE, but I digress. I would really like to see a productivity evaluation between already proficient users, confident with their skills on their respective interfaces, performing a series of common tasks and comparing the results.
Working in a DevOps shop is like playing in a band made up entirely of keytarists.
My advice? Don't have someone who's been a long time GNU/Linux user assisting her. Chances are, you're fond of either KDE or Gnome. Before the study has even started, I'm alarmed by potential bias. Let her do the study on her own, gather the facts and come up with her own conclusion. Isn't that what Ph.D.'s do?
"[...] while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects."
To me, the study is already flawed. You've dropped a load of information onto her lap, while a complete "newbie" doesn't have that same luxury. How can a usability study be unbiased in this manner? Who's to say you didn't provide her with REALLY good links to KDE information, while giving half-assed links to Gnome?
I like big butts and I cannot lie.
It's not clear that one can easily do an objective study on usability, as it can mean very different things to different people. It should at least be done with segregated populations (e.g. power-users vs. novices).
Some examples:
* A novice might look for how obvious it is to do a certain task, whereas an expert user might instead prefer what can be done fastest (e.g. notepad vs. emacs).
* Related: How much time does this person use a computer/this application can be an important factor. If I rarely do 3d design, I want to be told how to do everything, and have obvious controls (i.e. > 3 parameters might boggle my mind). However, if I work for Pixar, the verbose messages, and dumbed down controls (i.e. 30 parameters might just not cut it for what needs to be done).
* Certain paradigms might make sense to people who are used to using certain types of systems. Files and folders make perfect sense to many people, but certainly not to everyone (e.g. my mother). We think these simplified analogies work better for novices, but that isn't always the case. People think differently, and different analogies will make more/less sense dependent on their world view.
But if you must, at least don't do KDE vs. Gnome. What's the best possible outcome of that? ("So in summary, Gnome tended to be less confusing for newbies, but power users preferred the configurability of KDE...")
Instead compare either or both against Windows or Macintosh for tasks that your _specific target userbase_ would do. [If you haven't defined one or more use cases you've already lost.] This would be much more valuable.
Better yet, switch your topic to focus exclusively on accessibility (a11y). Every DE out there needs some accessibility love.
The first thing to understand is that you will have 3 groups of users:
#1. The ignorant users: These have never used a desktop before. These aren't as easy to find anymore. I worked with one woman back in the mid 90's who could not even use a mouse. She had to hold it still with one hand while she clicked the button with her other hand. After a week of solitare, she had the necessary muscle coordination to start learning the system.
#2. The tainted users: These have experience with systems other than the one you're testing. If your system isn't 100% like the one they're used to, they'll waste time clicking around where the functions are on their systems.
#3. Friends: These have worked on the system that they're being evaluated on.
Now, a system that is easy to learn for the "Ignorant" class may be incredibly un-friendly for more advanced "Friends".
Determine what functionality you want to measure and what GROUP you want to measure it for.
The real "ease" on an interface comes down to 2 things:
a. Can you quickly guess where a function is based upon your existing experience with it?
b. Once you know where a function is (you guessed at it before, you asked someone, you went to training), how easy is it to remember that 24 hours later, 1 week, 1 month, 6 months later?
'nuff said.
An occasionally used website (like an airline) or kiosk (photo stickers?) must be useable on first contact.
A work desktop must be quickly learnable and facilitate productivity of intermediate to advanced users.
IMHO - the "best" interface would be one that starts off at a child's level and allows the user to set the degree of functionality and complexity based upon his/her knowledge and needs.
How about, how many repetitions of instruction does it take for an average user to remember how to perform one action after a week of not using it?
And, once one function is explained, how quickly can the average user deduce/guess at related functions? This is how you select "bold" text. Then let them find "italics" and "underline".
KDE vs Gnome? hey dude, you missed Mac OS X, the perhaps the best operating system based on UNIX http://www.apple.com/macosx/features/unix/
Use a commercially successful UNIX desktop as a reference point.
While OS X doesn't occupy the majority of desktops it is
a commercial success.
Neither the participants nor the study coordinator should know what operating system the test subjects are using! You might laugh, but all you need are people who have only used Windows or Mac!
Also, make sure to use more than Gnome or KDE! Use XFCE, Fluxbox, and other XWindows managers.
And don't forget to make sure that the study has the appropriate "power"!
And make sure that everyone is using the same system configuration (motherboard, processor, underlying flavor of linux)
If this chick is doing a PhD on usability, then she probably knows tons about how to do an objective usability study. If she doesn't then people on Slashdot definitely won't :)
But there are lots of countries with people who still haven't used computers.
Not to mention that an interface with settings 1-10 (1=brand new user, 10=expert) would make a lot of non-experts more comfortable with their computers.
An ideal interface would evolve with the user's experience level. Not trap an experienced user with a pre-school interface nor confuse a new user with expert-level options.
Tailor the choices available to the level of the user and let the USER choose how complex the interface he uses is. Just like books. When you started reading, you didn't read the books you read today. Those books followed very careful patterns on what words were used and how often they were repeated.
But since none of the interfaces out there are doing that yet, it really doesn't matter for this discussion.
I had a nice rant all written up, because this is definitely not News for Nerds, or even Stuff that Matters. But its not worth it. Its already a lost cause when people ask slashdot to do assignments that they should be lerning from for them.
Going to the dead tree repository (you know, what we used to call a library) and doing some basic research on design in other fields completely unrelated to computers would be more useful, but lazy, unmotiviated students can't be bothered.
Nope the answer is "Teh Intar-net", some hand-waving, yadda yadda yaddda ... a powerpoint presentation or 10 (novocaine for the brain), a paper written in large fonts, wide margins, triple-spaced, to bulk up the page count, lots of screen shots that take up ink and paper but communicate nothing, a ton of links cited as references (just grab the first few pages off google and cut-and-paste the linkies, but dont bother reading the content - its not like anyone else will check it) and you've got your 100-page piece of drek. Don't forget to spiral-bind it for extra credit.
One of these days a professor is going to require that assignments be hand-written; then we'll see people actually learn to communicate with an economy of words. It'll kill off the cut-n-paste gang. Big deal. They're already a waste of space.
My advice is to not just look at KDE and Gnome! Many people, myself included find both of them rather annoying and cluttered. There's a lot to be said for the customizability and simplicity of not using them. It's not perfect either, but I do wish people wouldn't assume that it's no longer an option.
Exactly. On the face of it, KDE and GNOME can appear to be similar to some people. You may even dislike KDE's Qt interface for a while until you get used to it. However, KDE has amazingly powerful technology underneath. Spend a few months doing all your work on KDE, giving yourself time to discover (discovery learning is good) its hidden strengths, and you'll never look back.
I don't often agree with Linus, but what he said about GNOME and KDE was spot-on: when you take the time to try them both, you'll surely see that GNOME is clearly inferior, and KDE clearly shines as a bright technology -- perhaps the best technology -- available in Free Software.
I disagree. Group #2 is probably the most important group. Keep in mind that being tainted does not only mean being exposed to Windows+Office, but also to any other desktop environment, including Linux, OS X or others.
This includes for instance the Linux FVWM users who will be frustrated that the default GNOME window manager does not offer an option to bind the mouse buttons in the "right way", or the OS X users who will be frustrated that KDE tries to copy Windows in some places and uses the "wrong" button order. Every person who reads this on Slashdot is already in group #2. The tainted users will be biased towards one environment or another, often without realizing it. So they will think that some applications will be more usable than others because of their past experience, which will strongly influence what they think is the "right" way to do things.
But being tainted does not mean being a zealot. There are many Windows users who enjoy using KDE or GNOME when they are confronted with these new environments for the first time. Even if they cannot instantly forget all their past experience with computers, they are still a very useful test group for usabilty.
-Raphaël
For goodness sake, usability is not checked in websites, mailing lists or blogs. The poster would be excused about following this path of action because he is not a human interface specialist, but I hope the PhD student is wiser about this.
Human interface usability is properly tested one way and one way only. By watching real people interacting with them. You can use muckups (drawn or computerized), storyboards, etc, but nothing beats putting the fat asses of a few users in front of a computer and collecting their impressions.
If you possibly can you tailor your investigation to a particular group of people, ideally one that would make the study useful to you (if you are testing usability for software in kindergartens you don't want to do your usability test in a bank's trading floor).
The laughable suggestions to use children only, experienced users only or unexperienced users only as the correct or more accurate way of gauging usability is, as the British say, a load of pants. People suggesting this should jump of a clift like the gerbills they are.
One would do such a think only if there is no choice or if one has particular reasons for doing so, but never as the preferred criteria for a generalized useful study, what may be good for children may be crap for old timers and viceversa, experienced users may find some things annoying that new users find useful and viceversa.
The first thing that many people fail to understand is that usability is a wholy subjective thing. Linus loving KDE (yeah, that Linus) is only probe that for Linus (yeah, that Linus) KDE is more usable. All the KDE zealots implying that this is the God given truth regarding usability in Linux should be forced to use Gnome untile the know better.
Usability should be studied only on groups with similar patterns of usage for it to be any useful. The wider you make your target group study, the more difficult it will be to find meaningful results.
If you target all Linux users, then you are in for the most subjective, meaningless, most likely useless study.
If you target Linux users with less than one year experience using Linux then you are into something. If you target Linux users with more than 10 years experience degrading penguins you would also find more useful results.
Target your audience and you will find good results for that group of people.
Make your sample too wide and be welcome to the scrapyard of useless studies.
IANAL but write like a drunk one.