Slashdot Mirror
Conducting a Unix Desktop Usability Study?
cyclop asks: "I am a close friend of a Ph.D. student on human interface usability. She's now working to tailor a KDE-vs-Gnome usability study (a pretty hot topic these days), and I have been called to help, as a long time GNU/Linux desktop user. What kind of advice -- both technical and theoretical -- would you give us on conducting a deep and objective study on the Unix desktop, that can be useful for the developers and the OSS community?"
"She has installed GNU/Linux and used both KDE and Gnome to get to know them, while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects. However, I feel that it's not enough and that there are a lot of potential pitfalls and misconceptions that wait for us, me being a geek and she being a Linux newbie. Moreover, she found that most of the previous studies on the web were quite sloppy, in comparison with common usability research standards."
The out-of-the-box setup is a compromise at best; and shouldn't be used to judge the overall usability for people who use the system more than once.
Get a slashdot poll on the topic and read the insightful comments.
They who would give up an essential liberty for temporary security, deserve neither liberty nor security
It *does* sound like perfect academic paper fodder.
3-6 months perhaps?
"Usability" imho, in the usual meaning of the term, is a load of wank.
Who cares if the first time someone uses the environment that it takes a little orientation to get used to? In the real world, if a couple of weeks of pain makes you much more productive after that, it's a net benefit imho - the remainder of your time using the environment outweighs the significance of the learning time.
I'm not saying that initial learning is not important, but I think that these studies need to show both sides of the equation...
smash.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Get people who are not experts, see how many problems they run into doing simple tasks that they're familiar with on Windows. See how many of these they can solve themselves. Start half of them on Gnome and move them to KDE, do the other half in the reverse order.
It is probably also worth noting that most people (apparently including Linus) consider KDE more powerful, so KDE is kinda at a disadvantage.
You've got a lady friend and you have been called to help on conducting a deep and objective human interface study on the desktop?
Go for it!
(If you're wondering about the subject of this comment, the nipple is one of the most intuitive human interfaces btw).
Banu
I for one would like to see a study involving not just how easy it is to learn an interface, but once learned how productive one can be in said interface. For instance, I am proficient in both KDE and Gnome (and a myriad of other WMs which aren't mentioned here), but I feel I can get the most work done faster in KDE. Of course I do tweak quite a few aspects of KDE, but I digress. I would really like to see a productivity evaluation between already proficient users, confident with their skills on their respective interfaces, performing a series of common tasks and comparing the results.
Working in a DevOps shop is like playing in a band made up entirely of keytarists.
My advice? Don't have someone who's been a long time GNU/Linux user assisting her. Chances are, you're fond of either KDE or Gnome. Before the study has even started, I'm alarmed by potential bias. Let her do the study on her own, gather the facts and come up with her own conclusion. Isn't that what Ph.D.'s do?
"[...] while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects."
To me, the study is already flawed. You've dropped a load of information onto her lap, while a complete "newbie" doesn't have that same luxury. How can a usability study be unbiased in this manner? Who's to say you didn't provide her with REALLY good links to KDE information, while giving half-assed links to Gnome?
I like big butts and I cannot lie.
It's not clear that one can easily do an objective study on usability, as it can mean very different things to different people. It should at least be done with segregated populations (e.g. power-users vs. novices).
Some examples:
* A novice might look for how obvious it is to do a certain task, whereas an expert user might instead prefer what can be done fastest (e.g. notepad vs. emacs).
* Related: How much time does this person use a computer/this application can be an important factor. If I rarely do 3d design, I want to be told how to do everything, and have obvious controls (i.e. > 3 parameters might boggle my mind). However, if I work for Pixar, the verbose messages, and dumbed down controls (i.e. 30 parameters might just not cut it for what needs to be done).
* Certain paradigms might make sense to people who are used to using certain types of systems. Files and folders make perfect sense to many people, but certainly not to everyone (e.g. my mother). We think these simplified analogies work better for novices, but that isn't always the case. People think differently, and different analogies will make more/less sense dependent on their world view.
*) Get total PC newbies and Windows users to try using them and observe what they do, what they try, how and why they fail to succeed in their attempts. Let them talk verbosely about what they are doing.
.exe are apps etc.
*) Count the times you had to look in a manpage, in google, and how often you had to fire up a console for doing simple things (like creating a shared folder, browsing the internet, installing some plugins like flash etc.) Keep in mind: SIMPLE things! Trivial tasks done by the casual user.
*) For each system you need to learn how to use it, thats a fact. Unix users have to learn the concept behind the filesystem (nothing too fancy, but basic knowledge about what mounting is for example). This is comparable to the knowledge about the drive letters in Windows, the usage of backslash for separation in paths, that
*) Review the help system and documentation. Among other things, look for technical mumbo-jumbo. This is a common error. Stuff like SSH, SSL, CORBA, FUSE, pthreads etc. should never occur in enduser documentation.
*) Have a look at the menus. Are they cluttered or usable? How long did you have to search something in the menu?
This sig does not contain any SCO code.
But if you must, at least don't do KDE vs. Gnome. What's the best possible outcome of that? ("So in summary, Gnome tended to be less confusing for newbies, but power users preferred the configurability of KDE...")
Instead compare either or both against Windows or Macintosh for tasks that your _specific target userbase_ would do. [If you haven't defined one or more use cases you've already lost.] This would be much more valuable.
Better yet, switch your topic to focus exclusively on accessibility (a11y). Every DE out there needs some accessibility love.
The first thing to understand is that you will have 3 groups of users:
#1. The ignorant users: These have never used a desktop before. These aren't as easy to find anymore. I worked with one woman back in the mid 90's who could not even use a mouse. She had to hold it still with one hand while she clicked the button with her other hand. After a week of solitare, she had the necessary muscle coordination to start learning the system.
#2. The tainted users: These have experience with systems other than the one you're testing. If your system isn't 100% like the one they're used to, they'll waste time clicking around where the functions are on their systems.
#3. Friends: These have worked on the system that they're being evaluated on.
Now, a system that is easy to learn for the "Ignorant" class may be incredibly un-friendly for more advanced "Friends".
Determine what functionality you want to measure and what GROUP you want to measure it for.
The real "ease" on an interface comes down to 2 things:
a. Can you quickly guess where a function is based upon your existing experience with it?
b. Once you know where a function is (you guessed at it before, you asked someone, you went to training), how easy is it to remember that 24 hours later, 1 week, 1 month, 6 months later?
Don't go into it with the expectation that more like Windows or Mac OS is better. Sometimes they may be and sometime they may not be. Study people ranging from no experience up through experts.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
An occasionally used website (like an airline) or kiosk (photo stickers?) must be useable on first contact.
A work desktop must be quickly learnable and facilitate productivity of intermediate to advanced users.
Well we all know Linus's opinion. It's rather insightful: GNOME is overly simple and for dumb users, KDE is for smart people, haha. But, seriously, it is all about configurability. One of the nice thigns about both KDE and GNOME is all of the configuration you can do to them. The question is, how "stripped down" of a configuration are you going to start a user out with? Are you going to set up some nice buttons or put some useful help-guides on the desktop? For instance, I'm starting a cute little Web site I'm going to put on the desktop on my parents computer at home. I'm probably going to start them off with KDE, but that is because I get to set it up for them and give them a cute little guide to using it, and some simple pieces of software they might want to use.
IMHO - the "best" interface would be one that starts off at a child's level and allows the user to set the degree of functionality and complexity based upon his/her knowledge and needs.
How about, how many repetitions of instruction does it take for an average user to remember how to perform one action after a week of not using it?
And, once one function is explained, how quickly can the average user deduce/guess at related functions? This is how you select "bold" text. Then let them find "italics" and "underline".
Get some novices (people who have never used the interface before) in, and a list of tasks to complete. Let some of them muddle through it on their own, and give others pointers on use of the help system, google, and man pages. (One of the tasks for the first group could be -finding- help on some of the things they won't be able to complete on their own.) This will help represent the range of people coming into it-some will have absolutely no idea what they're doing, others will have at least some support from other knowledgeable users who can at least point them in the right direction.
You also might want a broad cross-section of users-some who rarely or never use a computer at all, some who use one relatively frequently, and some "power users" from other operating systems. This could lead to a very interesting picture-do those that already "know how" on a different interfacee have a harder time with something new, or are they able to translate most of their knowledge and pick up more quickly?
As to a comparison between the two, you may wish not only to time how long it takes the users to complete their task lists, but also include feedback from them-were the help pages actually helpful, or did they just confuse the issue more? Was the experience relatively smooth and welcoming, or aggravating and frustrating? Was there anything the user expected to be/work a certain way that did not do as expected? Did the user find it necessary to work in CLI at any point, and if so, was this disorienting or frustrating, or relatively smooth? Did they ever think they had done something right when they really had screwed it up, and were any clues/warnings given them to this effect if so?
All these are factors in usability, and I'm sure anyone can list plenty that I missed. In the end, usability is determined by-well, the user. Since it is somewhat subjective (I find working in a command line far easier and more convenient then use of a GUI most of the time, but there are many that would disagree!), focus on what the end-user, presented with the interface for the first time, thinks of it overall.
To fight the war on terror, stop being afraid.
Improve apon betterdesktop. The site is a collection of usability data with a focus on Linux apps. The front page gives more detail
peterrenshaw ~ Another Scrappy Startup
No. All she is "testing" there is how closely the desktop they're being "tested" on resembles the one they're used to.
Novell did this already.
http://linux.slashdot.org/article.pl?sid=05/10/11
So, to send email
KDE vs Gnome? hey dude, you missed Mac OS X, the perhaps the best operating system based on UNIX http://www.apple.com/macosx/features/unix/
Use a commercially successful UNIX desktop as a reference point.
While OS X doesn't occupy the majority of desktops it is
a commercial success.
Neither the participants nor the study coordinator should know what operating system the test subjects are using! You might laugh, but all you need are people who have only used Windows or Mac!
Also, make sure to use more than Gnome or KDE! Use XFCE, Fluxbox, and other XWindows managers.
And don't forget to make sure that the study has the appropriate "power"!
And make sure that everyone is using the same system configuration (motherboard, processor, underlying flavor of linux)
a benchmark.
Using the OOMA method, let's say it takes a user on one system 2 minutes to figure out how to send an email, and 1.95 min on anther system. What the hell does that mean?
If you use other items as some sort of bench mark, people might begin to get a feel for the numbers.
Lets say it take 45 seconds to figure out a new blender, 5 minutes to use a new remote, 20+years to set the clock on the VCR. Now people reading your study have a reference they can relate to.
It would also help companies trying to make applience computer to know where they stand in relation to appliances.
The Kruger Dunning explains most post on
If this chick is doing a PhD on usability, then she probably knows tons about how to do an objective usability study. If she doesn't then people on Slashdot definitely won't :)
1) Coming here to ask your question is a bad idea. Not necessarily because of the quality of most answers, but rather because /. readers represent such a miniscule portion of the real population.
2) Which brings me to my next point. Hire HCI experts, or take some classes on HCI. Testing OSS interfaces isn't any different than testing those of commercial software. You can do either user evaluations or predictive evaluations (w/o users). In fact, doing the latter first AND then the former is [usually] the best option. A cognitive walkthrough or heuristic evaluation can eliminate ~75% of the problems if done by around 4-6 evaluators. Then design an evaluation plan to be executed w/ users. Decide on benchmark tasks. Since you're comparing KDE with Gnome, and I bet those with Windows and/or Mac, you'll want benchmark numbers for all. Look for # of errors, time to completion, etc. These are easily quantifiable and thus comparable metrics.
Just off the top of my head, make sure to study these basic principles (not exhaustive): Learnability, Retainability, Predictability, Familiarity, Consistency, Dialog Initiative, Customizability, Generalizability, Observability, Responsiveness, Efficiency, Error Prevention, Error Recovery, Feedback.
Jakob Nielson and Don Norman are 2 of the most popular experts on HCI. Read Norman's "Design of Everyday Things" and Nielson's Usability Engineering.
But there are lots of countries with people who still haven't used computers.
Not to mention that an interface with settings 1-10 (1=brand new user, 10=expert) would make a lot of non-experts more comfortable with their computers.
An ideal interface would evolve with the user's experience level. Not trap an experienced user with a pre-school interface nor confuse a new user with expert-level options.
Tailor the choices available to the level of the user and let the USER choose how complex the interface he uses is. Just like books. When you started reading, you didn't read the books you read today. Those books followed very careful patterns on what words were used and how often they were repeated.
But since none of the interfaces out there are doing that yet, it really doesn't matter for this discussion.
Well, since Unix has *NEVER* had an objective study of it's desktop done, you will make history as a pioneer. Since it's survived so many smear campaigns, yours will, unfortunately, just add to the hot air. What, exactly, is the *point* of such a study, anyway? What does it change? I have yet to read a single such study that swayed my choices one iota.
Sadly, you're off on the wrong foot already. KDE-vs-Gnome. Hey, Dr Kinsey, there's just a few other test subjects you're failing to interview: http://xwinman.org/. So actually, you're flunking already. You are not doing a "Unix desktop study". You are doing a "KDE-vs-Gnome" study, and your results will no more be applicable to Unix in general than a study of Coke-vs-Pepsi would apply to all beverages.
It does not go without saying: Don't be paid Microsoft shills. Don't be paid by *anybody* for that matter.
Now, if I studied dogs, I wouldn't start with everything I know about cats and try to fit it all around that by comparing dogs with cats at every possible point. Similarly, Unix never gets taken as an operating system on it's own right. Everything is instead stated "It is not as good as or just like or better than Microsoft." How about judging something just once based on it's own merit, the way anybody studying anything else is expected to do in any other field? Consider your subject as if other operating systems did not exist. God knows, Microsoft is talked about in this manner.
Unfortunately, the focus will of course be on KDE and Gnome, the Heckyl and Jeckyl whose sole point of contention is "I'M a perfect clone of the Windows environment!" "No, I am!" "No, me!" "NO, ME!" So in fact, you're not the least bit interested in considering even KDE or Gnome on it's own right - this will be a Windows-impersonator contest. Never mind that counting from the invention of computers: http://ei.cs.vt.edu/~history/Babbage.html, computers have been around for one hundred and eighty-two years, and only the last 20 years http://members.fortunecity.com/pcmuseum/windows.ht m has seen the existence of a desktop system known as Windows. For a ratio of 0.10989011 of computer's history, you are going to compare the one system whose sole claim is that it made a lot of money in the United States to two other desktops expressly written to mimic it.
I'm really sure the world will be enlightened.
> Personally I think Apple has done a quite decent job of building a GUI on top of a UNIX
> core (the Darwin flavor of freeBSD).
Well, this is no real option.
Because for Apple users it is irrelevant whether it is build on foosys or Unix. When you run a C64 emulator on Linux which is distributed as a game console to play games you cannot say "this is a real Linux" only because a techie can open a linux console. For the average use it is C64. And for the manufacturer it it was a rather technical choice what runtime plattform he chose. When you look at a diamond you can also think of carbon and sure it is.
But all these Apple freaks which want to belong to the Unix family do not understand that the essence of Macs is not Unix as it was not PPC. And when Apple switches tomorrow to Intel and from BSD to foosys, Apple users will find other silly arguments to rationalise their apple preference.
In my opinion KDE is more productive for me. But it depends. What is the real problem is hardware drivers and configuration stuff. Not difficult to get beaten by the most proprietary plattform. When you do not have to support the whole hardware cosmos things get easier.
#4. I have no social life : These have seen so much different systems that they understand well the general concept and can adapt themself very quickly to whatever system you throw at them. As open to new solutions as #1, but very quickly get as agile to whatever system as #3 is.
(In fact that was my case when I started learning both KDE and Windows 95 in parallel)
The best system ever should be as easy as possible for #1 out of the box, but need to be very easily configurable to whatever complex system #3 and #4 need.
If the system is newbie-friendly but can't evolve you'll end with Clippy and this kind of stuff that gets in the user's way with pointless tips (tips that would have helped a total beginner, but sorry now I know exactly what I want). Attracts #1 users, but repels #3 and #4.
If the system is configurable to extreeme you end up with emacs or vi : the most powerfull tool around you can't ever dream of in your wildest dreams, but you can't do anything without unless you've spent the first year learning it the harsh way. #3 only are interested, #4 must ponder if they want to re-learn everything once again (albeit they do it faster), #1 will prefer to commit suicide.
#2 are pointless, they won't accept anything that isn't their original system, they're the one that will never switch to MacOSX or Linux because it's not Windows+Office, and they'll cry each time MicroSoft revamps the interface and everything is moved around (Windows 3.11 -> Windows 9x -> Windows 2k -> Windows XP -> Windows Vista and same for the Offices). Just wait until the next "GUI is completly changed one more time" period, and they'll be as good as #1 users (or #4 if it's not their first change around).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
So what GNOME really needs is a means of ramping novice users up to power users. Unfortunately, this is sadly lacking. For instance, let's try navigating in nautilus as a power user:
Open Parent -> Alt+Up
Open Location -> Ctrl+L
Close Parent Folders -> Ctrl+Shift+W
Close All Folders -> Ctrl+Q
Close -> Ctrl+W
Home Folder -> Alt+Home
Notice that we're using Alt key combinations, Ctrl key combinations, and Ctrl+Shift key combinations. My biggest problem with using nautilus effectively is mixing up these combos to either no effect orworse the wrong effect. I usually end up opening a terminal window in frustration after hitting Ctrl+Shift+W rather than Ctrl+W.
I guess my point is that even power users are looking for some consistancy to their interface and GNOME, in focusing on novice users, overlooks this.
I can't comment on how KDE compares in this respect as I've never used it
Your courageous and selfless spelling corrections have made me a better person.
Problem: Unix GUI usability low for casual users.
Action: Perform UNIX GUI usability studies every few days, post repeatedly on Slashdot.
Result: UNIX GUI usability studies improve, UNIX GUI usability stays same.
My advice is to not just look at KDE and Gnome! Many people, myself included find both of them rather annoying and cluttered. There's a lot to be said for the customizability and simplicity of not using them. It's not perfect either, but I do wish people wouldn't assume that it's no longer an option.
I'm a level 137 Microsoft Win-zard, I can now wield the CLI of fortitude and I've unlocked the 5th ring of the registry.
That will surely make the level 1s in this world eager to learn.
Alright. I cast BSOD of slaying on your Win-zard.
- "Send an email"
- "Create a new email then send it"
- "Write an email"
The answer from my point of view is "I don't know", but I'd be interested to know. Of course you need a load more people and a load more tests. But this is science, it's always toughCarpe Daemon
Exactly. On the face of it, KDE and GNOME can appear to be similar to some people. You may even dislike KDE's Qt interface for a while until you get used to it. However, KDE has amazingly powerful technology underneath. Spend a few months doing all your work on KDE, giving yourself time to discover (discovery learning is good) its hidden strengths, and you'll never look back.
I don't often agree with Linus, but what he said about GNOME and KDE was spot-on: when you take the time to try them both, you'll surely see that GNOME is clearly inferior, and KDE clearly shines as a bright technology -- perhaps the best technology -- available in Free Software.
For goodness sake, usability is not checked in websites, mailing lists or blogs. The poster would be excused about following this path of action because he is not a human interface specialist, but I hope the PhD student is wiser about this.
Human interface usability is properly tested one way and one way only. By watching real people interacting with them. You can use muckups (drawn or computerized), storyboards, etc, but nothing beats putting the fat asses of a few users in front of a computer and collecting their impressions.
If you possibly can you tailor your investigation to a particular group of people, ideally one that would make the study useful to you (if you are testing usability for software in kindergartens you don't want to do your usability test in a bank's trading floor).
The laughable suggestions to use children only, experienced users only or unexperienced users only as the correct or more accurate way of gauging usability is, as the British say, a load of pants. People suggesting this should jump of a clift like the gerbills they are.
One would do such a think only if there is no choice or if one has particular reasons for doing so, but never as the preferred criteria for a generalized useful study, what may be good for children may be crap for old timers and viceversa, experienced users may find some things annoying that new users find useful and viceversa.
The first thing that many people fail to understand is that usability is a wholy subjective thing. Linus loving KDE (yeah, that Linus) is only probe that for Linus (yeah, that Linus) KDE is more usable. All the KDE zealots implying that this is the God given truth regarding usability in Linux should be forced to use Gnome untile the know better.
Usability should be studied only on groups with similar patterns of usage for it to be any useful. The wider you make your target group study, the more difficult it will be to find meaningful results.
If you target all Linux users, then you are in for the most subjective, meaningless, most likely useless study.
If you target Linux users with less than one year experience using Linux then you are into something. If you target Linux users with more than 10 years experience degrading penguins you would also find more useful results.
Target your audience and you will find good results for that group of people.
Make your sample too wide and be welcome to the scrapyard of useless studies.
IANAL but write like a drunk one.