Slashdot Mirror
Conducting a Unix Desktop Usability Study?
cyclop asks: "I am a close friend of a Ph.D. student on human interface usability. She's now working to tailor a KDE-vs-Gnome usability study (a pretty hot topic these days), and I have been called to help, as a long time GNU/Linux desktop user. What kind of advice -- both technical and theoretical -- would you give us on conducting a deep and objective study on the Unix desktop, that can be useful for the developers and the OSS community?"
"She has installed GNU/Linux and used both KDE and Gnome to get to know them, while I provided her a number of links on background information and previous usability studies on both DE, and advised her to subscribe to relevant mailing lists of both projects. However, I feel that it's not enough and that there are a lot of potential pitfalls and misconceptions that wait for us, me being a geek and she being a Linux newbie. Moreover, she found that most of the previous studies on the web were quite sloppy, in comparison with common usability research standards."
3-6 months perhaps?
"Usability" imho, in the usual meaning of the term, is a load of wank.
Who cares if the first time someone uses the environment that it takes a little orientation to get used to? In the real world, if a couple of weeks of pain makes you much more productive after that, it's a net benefit imho - the remainder of your time using the environment outweighs the significance of the learning time.
I'm not saying that initial learning is not important, but I think that these studies need to show both sides of the equation...
smash.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
*) Get total PC newbies and Windows users to try using them and observe what they do, what they try, how and why they fail to succeed in their attempts. Let them talk verbosely about what they are doing.
.exe are apps etc.
*) Count the times you had to look in a manpage, in google, and how often you had to fire up a console for doing simple things (like creating a shared folder, browsing the internet, installing some plugins like flash etc.) Keep in mind: SIMPLE things! Trivial tasks done by the casual user.
*) For each system you need to learn how to use it, thats a fact. Unix users have to learn the concept behind the filesystem (nothing too fancy, but basic knowledge about what mounting is for example). This is comparable to the knowledge about the drive letters in Windows, the usage of backslash for separation in paths, that
*) Review the help system and documentation. Among other things, look for technical mumbo-jumbo. This is a common error. Stuff like SSH, SSL, CORBA, FUSE, pthreads etc. should never occur in enduser documentation.
*) Have a look at the menus. Are they cluttered or usable? How long did you have to search something in the menu?
This sig does not contain any SCO code.
Personally I think Apple has done a quite decent job of building a GUI on top of a UNIX core (the Darwin flavor of freeBSD).
Currently they have it working on two different processor families (the IBM Power series, and Intel).
Yes, it is proprietary, but that does NOT mean that "Aqua" is not a GUI desktop running on a UNIX system.
Why not compare the other UNIX desktops with what may be the best UNIX desktop running?
Don't get excited, it's just an honest question. After all, just because it was done by a commercial software shop doesn't mean it doesn't work...
How DOES the usability of other attempts at a UNIX GUI stand up to Aqua?
--Tomas
Don't go into it with the expectation that more like Windows or Mac OS is better. Sometimes they may be and sometime they may not be. Study people ranging from no experience up through experts.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
It might be better to test it on childern.
most people have used windows at one time and expect things to have that type of layout.
children who havent gotten used to what windows is like might find it a bit easier/harder to move around in.
you could have 2 groups of children
set one group of children to use gnome for the first week/month/year and kde for the second week/month/year whatever
and set the other group to use kde for the first week/month/year and gnome for the second week/month/year
and compare there reviews of how easy it is to move around.
however it might be better to test it out on teenagers are they will be able to take more infomation in.
Well we all know Linus's opinion. It's rather insightful: GNOME is overly simple and for dumb users, KDE is for smart people, haha. But, seriously, it is all about configurability. One of the nice thigns about both KDE and GNOME is all of the configuration you can do to them. The question is, how "stripped down" of a configuration are you going to start a user out with? Are you going to set up some nice buttons or put some useful help-guides on the desktop? For instance, I'm starting a cute little Web site I'm going to put on the desktop on my parents computer at home. I'm probably going to start them off with KDE, but that is because I get to set it up for them and give them a cute little guide to using it, and some simple pieces of software they might want to use.
Get some novices (people who have never used the interface before) in, and a list of tasks to complete. Let some of them muddle through it on their own, and give others pointers on use of the help system, google, and man pages. (One of the tasks for the first group could be -finding- help on some of the things they won't be able to complete on their own.) This will help represent the range of people coming into it-some will have absolutely no idea what they're doing, others will have at least some support from other knowledgeable users who can at least point them in the right direction.
You also might want a broad cross-section of users-some who rarely or never use a computer at all, some who use one relatively frequently, and some "power users" from other operating systems. This could lead to a very interesting picture-do those that already "know how" on a different interfacee have a harder time with something new, or are they able to translate most of their knowledge and pick up more quickly?
As to a comparison between the two, you may wish not only to time how long it takes the users to complete their task lists, but also include feedback from them-were the help pages actually helpful, or did they just confuse the issue more? Was the experience relatively smooth and welcoming, or aggravating and frustrating? Was there anything the user expected to be/work a certain way that did not do as expected? Did the user find it necessary to work in CLI at any point, and if so, was this disorienting or frustrating, or relatively smooth? Did they ever think they had done something right when they really had screwed it up, and were any clues/warnings given them to this effect if so?
All these are factors in usability, and I'm sure anyone can list plenty that I missed. In the end, usability is determined by-well, the user. Since it is somewhat subjective (I find working in a command line far easier and more convenient then use of a GUI most of the time, but there are many that would disagree!), focus on what the end-user, presented with the interface for the first time, thinks of it overall.
To fight the war on terror, stop being afraid.
a benchmark.
Using the OOMA method, let's say it takes a user on one system 2 minutes to figure out how to send an email, and 1.95 min on anther system. What the hell does that mean?
If you use other items as some sort of bench mark, people might begin to get a feel for the numbers.
Lets say it take 45 seconds to figure out a new blender, 5 minutes to use a new remote, 20+years to set the clock on the VCR. Now people reading your study have a reference they can relate to.
It would also help companies trying to make applience computer to know where they stand in relation to appliances.
The Kruger Dunning explains most post on
1) Coming here to ask your question is a bad idea. Not necessarily because of the quality of most answers, but rather because /. readers represent such a miniscule portion of the real population.
2) Which brings me to my next point. Hire HCI experts, or take some classes on HCI. Testing OSS interfaces isn't any different than testing those of commercial software. You can do either user evaluations or predictive evaluations (w/o users). In fact, doing the latter first AND then the former is [usually] the best option. A cognitive walkthrough or heuristic evaluation can eliminate ~75% of the problems if done by around 4-6 evaluators. Then design an evaluation plan to be executed w/ users. Decide on benchmark tasks. Since you're comparing KDE with Gnome, and I bet those with Windows and/or Mac, you'll want benchmark numbers for all. Look for # of errors, time to completion, etc. These are easily quantifiable and thus comparable metrics.
Just off the top of my head, make sure to study these basic principles (not exhaustive): Learnability, Retainability, Predictability, Familiarity, Consistency, Dialog Initiative, Customizability, Generalizability, Observability, Responsiveness, Efficiency, Error Prevention, Error Recovery, Feedback.
Jakob Nielson and Don Norman are 2 of the most popular experts on HCI. Read Norman's "Design of Everyday Things" and Nielson's Usability Engineering.
>#1. The ignorant users: These have never used a desktop before.
I totally disagree with this. what percentage of your potential audience has "never used a desktop before". Very close to zero, and getting lower all the time. MSFT did do one thing, and that was teach everday people what a desktop was, and how to more-or-less get around a computer.
There is no reason to try to design anything for this class of user, it is such a small portion of the populace that it can be said to be zero.
... hi bingo
If long term efficiency (productivity) is the thing to be compared, then I'd assert more than just Gnome or KDE should be on the table. After all, they both represent implementations of very similar metaphors and workflows.
I'd like to see alternatives such as a ultra-lean-configuration fvwm/twm class desktop (representing thinner organizational and interactive metaphors upon a shell-centric workflow), an ion desktop (completely different desktop metaphor), Mac OS X's desktop, and maybe just a straight linux console (for comparison purposes as I do not think a single shell w/o access to graphical apps would be competitive).
Personally, I've refined over the years an fvwm configuration w/o borders, w/o icons, w/o title bars, lots of keybindings...efficient for high shell and vim counts...but difficult for one handed (like with the other hand holding a phone or food) usage. Basicly, ion made sense to me in a lot of ways, but is difficult with apps and application workflows that assume 'normal' windowing, so I went pretty far down the minimalist road with fvwm. (Note that FvwmProxy is indespensible for such a desktop, and I'd say it is superior to Apple's expose in usefulness)
Well, since Unix has *NEVER* had an objective study of it's desktop done, you will make history as a pioneer. Since it's survived so many smear campaigns, yours will, unfortunately, just add to the hot air. What, exactly, is the *point* of such a study, anyway? What does it change? I have yet to read a single such study that swayed my choices one iota.
Sadly, you're off on the wrong foot already. KDE-vs-Gnome. Hey, Dr Kinsey, there's just a few other test subjects you're failing to interview: http://xwinman.org/. So actually, you're flunking already. You are not doing a "Unix desktop study". You are doing a "KDE-vs-Gnome" study, and your results will no more be applicable to Unix in general than a study of Coke-vs-Pepsi would apply to all beverages.
It does not go without saying: Don't be paid Microsoft shills. Don't be paid by *anybody* for that matter.
Now, if I studied dogs, I wouldn't start with everything I know about cats and try to fit it all around that by comparing dogs with cats at every possible point. Similarly, Unix never gets taken as an operating system on it's own right. Everything is instead stated "It is not as good as or just like or better than Microsoft." How about judging something just once based on it's own merit, the way anybody studying anything else is expected to do in any other field? Consider your subject as if other operating systems did not exist. God knows, Microsoft is talked about in this manner.
Unfortunately, the focus will of course be on KDE and Gnome, the Heckyl and Jeckyl whose sole point of contention is "I'M a perfect clone of the Windows environment!" "No, I am!" "No, me!" "NO, ME!" So in fact, you're not the least bit interested in considering even KDE or Gnome on it's own right - this will be a Windows-impersonator contest. Never mind that counting from the invention of computers: http://ei.cs.vt.edu/~history/Babbage.html, computers have been around for one hundred and eighty-two years, and only the last 20 years http://members.fortunecity.com/pcmuseum/windows.ht m has seen the existence of a desktop system known as Windows. For a ratio of 0.10989011 of computer's history, you are going to compare the one system whose sole claim is that it made a lot of money in the United States to two other desktops expressly written to mimic it.
I'm really sure the world will be enlightened.
Its not just the users who are dummies.
I was TOLD to take out passwords and logins on a database connected to the internet because "its too much bother for the workers to log in."
So if you know the server and the directory, you can "administer" it. Add inventory. Reprice inventory. Remove inventory. Make a contract selling a $10,000 item for $1.99. Change descriptions. And since it also rebuilds the web site, change the product pictures for pr0n and add links to competitors.
#4. I have no social life : These have seen so much different systems that they understand well the general concept and can adapt themself very quickly to whatever system you throw at them. As open to new solutions as #1, but very quickly get as agile to whatever system as #3 is.
(In fact that was my case when I started learning both KDE and Windows 95 in parallel)
The best system ever should be as easy as possible for #1 out of the box, but need to be very easily configurable to whatever complex system #3 and #4 need.
If the system is newbie-friendly but can't evolve you'll end with Clippy and this kind of stuff that gets in the user's way with pointless tips (tips that would have helped a total beginner, but sorry now I know exactly what I want). Attracts #1 users, but repels #3 and #4.
If the system is configurable to extreeme you end up with emacs or vi : the most powerfull tool around you can't ever dream of in your wildest dreams, but you can't do anything without unless you've spent the first year learning it the harsh way. #3 only are interested, #4 must ponder if they want to re-learn everything once again (albeit they do it faster), #1 will prefer to commit suicide.
#2 are pointless, they won't accept anything that isn't their original system, they're the one that will never switch to MacOSX or Linux because it's not Windows+Office, and they'll cry each time MicroSoft revamps the interface and everything is moved around (Windows 3.11 -> Windows 9x -> Windows 2k -> Windows XP -> Windows Vista and same for the Offices). Just wait until the next "GUI is completly changed one more time" period, and they'll be as good as #1 users (or #4 if it's not their first change around).
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
Insightful? Funny is more like it. Sometimes all you'll get is "Usability is for weenies go check you're spellign and grammar and use windows you pathetic luser" propmtly followed by a flamewar in which half of the readers actually agree with him.
I am Spartacus
Well, I've long suspected girls are good for something.
- "Send an email"
- "Create a new email then send it"
- "Write an email"
The answer from my point of view is "I don't know", but I'd be interested to know. Of course you need a load more people and a load more tests. But this is science, it's always toughCarpe Daemon
Exactly.
My sister's owned a computer since 1998. She's a whiz when it comes to email, using Word, or manipulating her photos in Photoshop. Her first machine was a Gateway monstrosity that she paid way too much for. Now she's got a G5.
A couple of months back, she was absolutely flabbergasted when she saw me switching between open windows... all this time she's had no idea that one can minimize a window to the taskbar and continue on in another.
...Rob
The American Dream isn't an SUV and a house in the suburbs; it's Don't Tread On Me.