Slashdot Mirror
Conducting a Unix Desktop Usability Study?
cyclop asks: "I am a close friend of a Ph.D. student on human interface usability. She's now working to tailor a KDE-vs-Gnome usability study (a pretty hot topic these days), and I have been called to help, as a long time GNU/Linux desktop user. What kind of advice -- both technical and theoretical -- would you give us on conducting a deep and objective study on the Unix desktop, that can be useful for the developers and the OSS community?"
"She has installed GNU/Linux and used both KDE and Gnome to get to know them, while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects. However, I feel that it's not enough and that there are a lot of potential pitfalls and misconceptions that wait for us, me being a geek and she being a Linux newbie. Moreover, she found that most of the previous studies on the web were quite sloppy, in comparison with common usability research standards."
The out-of-the-box setup is a compromise at best; and shouldn't be used to judge the overall usability for people who use the system more than once.
Get a slashdot poll on the topic and read the insightful comments.
They who would give up an essential liberty for temporary security, deserve neither liberty nor security
3-6 months perhaps?
"Usability" imho, in the usual meaning of the term, is a load of wank.
Who cares if the first time someone uses the environment that it takes a little orientation to get used to? In the real world, if a couple of weeks of pain makes you much more productive after that, it's a net benefit imho - the remainder of your time using the environment outweighs the significance of the learning time.
I'm not saying that initial learning is not important, but I think that these studies need to show both sides of the equation...
smash.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Get people who are not experts, see how many problems they run into doing simple tasks that they're familiar with on Windows. See how many of these they can solve themselves. Start half of them on Gnome and move them to KDE, do the other half in the reverse order.
It is probably also worth noting that most people (apparently including Linus) consider KDE more powerful, so KDE is kinda at a disadvantage.
I for one would like to see a study involving not just how easy it is to learn an interface, but once learned how productive one can be in said interface. For instance, I am proficient in both KDE and Gnome (and a myriad of other WMs which aren't mentioned here), but I feel I can get the most work done faster in KDE. Of course I do tweak quite a few aspects of KDE, but I digress. I would really like to see a productivity evaluation between already proficient users, confident with their skills on their respective interfaces, performing a series of common tasks and comparing the results.
Working in a DevOps shop is like playing in a band made up entirely of keytarists.
My advice? Don't have someone who's been a long time GNU/Linux user assisting her. Chances are, you're fond of either KDE or Gnome. Before the study has even started, I'm alarmed by potential bias. Let her do the study on her own, gather the facts and come up with her own conclusion. Isn't that what Ph.D.'s do?
"[...] while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects."
To me, the study is already flawed. You've dropped a load of information onto her lap, while a complete "newbie" doesn't have that same luxury. How can a usability study be unbiased in this manner? Who's to say you didn't provide her with REALLY good links to KDE information, while giving half-assed links to Gnome?
I like big butts and I cannot lie.
maybe on a woman... but I don't consider my own very intuitive; I can't figure out what its purpose is.
The Admin and the Engineer
It's not clear that one can easily do an objective study on usability, as it can mean very different things to different people. It should at least be done with segregated populations (e.g. power-users vs. novices).
Some examples:
* A novice might look for how obvious it is to do a certain task, whereas an expert user might instead prefer what can be done fastest (e.g. notepad vs. emacs).
* Related: How much time does this person use a computer/this application can be an important factor. If I rarely do 3d design, I want to be told how to do everything, and have obvious controls (i.e. > 3 parameters might boggle my mind). However, if I work for Pixar, the verbose messages, and dumbed down controls (i.e. 30 parameters might just not cut it for what needs to be done).
* Certain paradigms might make sense to people who are used to using certain types of systems. Files and folders make perfect sense to many people, but certainly not to everyone (e.g. my mother). We think these simplified analogies work better for novices, but that isn't always the case. People think differently, and different analogies will make more/less sense dependent on their world view.
But if you must, at least don't do KDE vs. Gnome. What's the best possible outcome of that? ("So in summary, Gnome tended to be less confusing for newbies, but power users preferred the configurability of KDE...")
Instead compare either or both against Windows or Macintosh for tasks that your _specific target userbase_ would do. [If you haven't defined one or more use cases you've already lost.] This would be much more valuable.
Better yet, switch your topic to focus exclusively on accessibility (a11y). Every DE out there needs some accessibility love.
The first thing to understand is that you will have 3 groups of users:
#1. The ignorant users: These have never used a desktop before. These aren't as easy to find anymore. I worked with one woman back in the mid 90's who could not even use a mouse. She had to hold it still with one hand while she clicked the button with her other hand. After a week of solitare, she had the necessary muscle coordination to start learning the system.
#2. The tainted users: These have experience with systems other than the one you're testing. If your system isn't 100% like the one they're used to, they'll waste time clicking around where the functions are on their systems.
#3. Friends: These have worked on the system that they're being evaluated on.
Now, a system that is easy to learn for the "Ignorant" class may be incredibly un-friendly for more advanced "Friends".
Determine what functionality you want to measure and what GROUP you want to measure it for.
The real "ease" on an interface comes down to 2 things:
a. Can you quickly guess where a function is based upon your existing experience with it?
b. Once you know where a function is (you guessed at it before, you asked someone, you went to training), how easy is it to remember that 24 hours later, 1 week, 1 month, 6 months later?
IMHO - the "best" interface would be one that starts off at a child's level and allows the user to set the degree of functionality and complexity based upon his/her knowledge and needs.
How about, how many repetitions of instruction does it take for an average user to remember how to perform one action after a week of not using it?
And, once one function is explained, how quickly can the average user deduce/guess at related functions? This is how you select "bold" text. Then let them find "italics" and "underline".
Get some novices (people who have never used the interface before) in, and a list of tasks to complete. Let some of them muddle through it on their own, and give others pointers on use of the help system, google, and man pages. (One of the tasks for the first group could be -finding- help on some of the things they won't be able to complete on their own.) This will help represent the range of people coming into it-some will have absolutely no idea what they're doing, others will have at least some support from other knowledgeable users who can at least point them in the right direction.
You also might want a broad cross-section of users-some who rarely or never use a computer at all, some who use one relatively frequently, and some "power users" from other operating systems. This could lead to a very interesting picture-do those that already "know how" on a different interfacee have a harder time with something new, or are they able to translate most of their knowledge and pick up more quickly?
As to a comparison between the two, you may wish not only to time how long it takes the users to complete their task lists, but also include feedback from them-were the help pages actually helpful, or did they just confuse the issue more? Was the experience relatively smooth and welcoming, or aggravating and frustrating? Was there anything the user expected to be/work a certain way that did not do as expected? Did the user find it necessary to work in CLI at any point, and if so, was this disorienting or frustrating, or relatively smooth? Did they ever think they had done something right when they really had screwed it up, and were any clues/warnings given them to this effect if so?
All these are factors in usability, and I'm sure anyone can list plenty that I missed. In the end, usability is determined by-well, the user. Since it is somewhat subjective (I find working in a command line far easier and more convenient then use of a GUI most of the time, but there are many that would disagree!), focus on what the end-user, presented with the interface for the first time, thinks of it overall.
To fight the war on terror, stop being afraid.
Neither the participants nor the study coordinator should know what operating system the test subjects are using! You might laugh, but all you need are people who have only used Windows or Mac!
Also, make sure to use more than Gnome or KDE! Use XFCE, Fluxbox, and other XWindows managers.
And don't forget to make sure that the study has the appropriate "power"!
And make sure that everyone is using the same system configuration (motherboard, processor, underlying flavor of linux)
a benchmark.
Using the OOMA method, let's say it takes a user on one system 2 minutes to figure out how to send an email, and 1.95 min on anther system. What the hell does that mean?
If you use other items as some sort of bench mark, people might begin to get a feel for the numbers.
Lets say it take 45 seconds to figure out a new blender, 5 minutes to use a new remote, 20+years to set the clock on the VCR. Now people reading your study have a reference they can relate to.
It would also help companies trying to make applience computer to know where they stand in relation to appliances.
The Kruger Dunning explains most post on
But there are lots of countries with people who still haven't used computers.
Not to mention that an interface with settings 1-10 (1=brand new user, 10=expert) would make a lot of non-experts more comfortable with their computers.
An ideal interface would evolve with the user's experience level. Not trap an experienced user with a pre-school interface nor confuse a new user with expert-level options.
Tailor the choices available to the level of the user and let the USER choose how complex the interface he uses is. Just like books. When you started reading, you didn't read the books you read today. Those books followed very careful patterns on what words were used and how often they were repeated.
But since none of the interfaces out there are doing that yet, it really doesn't matter for this discussion.
Well, since Unix has *NEVER* had an objective study of it's desktop done, you will make history as a pioneer. Since it's survived so many smear campaigns, yours will, unfortunately, just add to the hot air. What, exactly, is the *point* of such a study, anyway? What does it change? I have yet to read a single such study that swayed my choices one iota.
Sadly, you're off on the wrong foot already. KDE-vs-Gnome. Hey, Dr Kinsey, there's just a few other test subjects you're failing to interview: http://xwinman.org/. So actually, you're flunking already. You are not doing a "Unix desktop study". You are doing a "KDE-vs-Gnome" study, and your results will no more be applicable to Unix in general than a study of Coke-vs-Pepsi would apply to all beverages.
It does not go without saying: Don't be paid Microsoft shills. Don't be paid by *anybody* for that matter.
Now, if I studied dogs, I wouldn't start with everything I know about cats and try to fit it all around that by comparing dogs with cats at every possible point. Similarly, Unix never gets taken as an operating system on it's own right. Everything is instead stated "It is not as good as or just like or better than Microsoft." How about judging something just once based on it's own merit, the way anybody studying anything else is expected to do in any other field? Consider your subject as if other operating systems did not exist. God knows, Microsoft is talked about in this manner.
Unfortunately, the focus will of course be on KDE and Gnome, the Heckyl and Jeckyl whose sole point of contention is "I'M a perfect clone of the Windows environment!" "No, I am!" "No, me!" "NO, ME!" So in fact, you're not the least bit interested in considering even KDE or Gnome on it's own right - this will be a Windows-impersonator contest. Never mind that counting from the invention of computers: http://ei.cs.vt.edu/~history/Babbage.html, computers have been around for one hundred and eighty-two years, and only the last 20 years http://members.fortunecity.com/pcmuseum/windows.ht m has seen the existence of a desktop system known as Windows. For a ratio of 0.10989011 of computer's history, you are going to compare the one system whose sole claim is that it made a lot of money in the United States to two other desktops expressly written to mimic it.
I'm really sure the world will be enlightened.
#4. I have no social life : These have seen so much different systems that they understand well the general concept and can adapt themself very quickly to whatever system you throw at them. As open to new solutions as #1, but very quickly get as agile to whatever system as #3 is.
(In fact that was my case when I started learning both KDE and Windows 95 in parallel)
The best system ever should be as easy as possible for #1 out of the box, but need to be very easily configurable to whatever complex system #3 and #4 need.
If the system is newbie-friendly but can't evolve you'll end with Clippy and this kind of stuff that gets in the user's way with pointless tips (tips that would have helped a total beginner, but sorry now I know exactly what I want). Attracts #1 users, but repels #3 and #4.
If the system is configurable to extreeme you end up with emacs or vi : the most powerfull tool around you can't ever dream of in your wildest dreams, but you can't do anything without unless you've spent the first year learning it the harsh way. #3 only are interested, #4 must ponder if they want to re-learn everything once again (albeit they do it faster), #1 will prefer to commit suicide.
#2 are pointless, they won't accept anything that isn't their original system, they're the one that will never switch to MacOSX or Linux because it's not Windows+Office, and they'll cry each time MicroSoft revamps the interface and everything is moved around (Windows 3.11 -> Windows 9x -> Windows 2k -> Windows XP -> Windows Vista and same for the Offices). Just wait until the next "GUI is completly changed one more time" period, and they'll be as good as #1 users (or #4 if it's not their first change around).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Problem: Unix GUI usability low for casual users.
Action: Perform UNIX GUI usability studies every few days, post repeatedly on Slashdot.
Result: UNIX GUI usability studies improve, UNIX GUI usability stays same.
but I don't consider my own very intuitive; I can't figure out what its purpose is.
You're a geek, so just use the process of elimination. All you have to do is remove your nipples, then wait and see what stops working.
It's not rocket science...
"I've got more toys than Teruhisa Kitahara."
I'm a level 137 Microsoft Win-zard, I can now wield the CLI of fortitude and I've unlocked the 5th ring of the registry.
That will surely make the level 1s in this world eager to learn.
Alright. I cast BSOD of slaying on your Win-zard.