Novell's Releases Linux Usability Testing Videos

sp3298622 writes "Novell is releasing primary desktop research, including over 200 videos and analysis of usability tests, at betterdesktop.openSUSE.org. Vice president of collaboration and desktop engineering for Novell, Nat Friedman: As a programmer, it's sometimes difficult to know how ordinary people with no technical experience are reacting to your software. Linux people tend to know other Linux people. In these usability tests, we selected test subjects who were experienced with Windows, but who had never heard of Linux, and asked them to perform basic tasks using the Linux desktop."

Comments

Please proofread.

[Anonymous Coward]

That headline is just embarassing.

Re:Define basic tasks

[krgallagher]

"Define basic tasks "

OK Here is (I think) the complete list from the article:

• Task: Add an entry in the addressbook
• Task: Make an account on this computer for your friend
• Task: Add a birthday reminder to the calendar
• Task: Change the wallpaper
• Task: Change the fonts on the computer
• Task: Please correct your computer's date/time
• Task: Create a playlist
• Task: Crop the photo of the cat
• Task: Email "A Tale of Two Cities" to arthur@ximian.com
• Task: Upload your photos to your website
• Task: Figure out if the computer is online
• Task: Find all the music by Bach
• Task: Edit the note that you made for yourself
• Task: Find out if the computer is online
• Task: Find the document "A Tale of Two Cities" on the computer
• Task: See if you have enough free disk space for a new game
• Task: Import the photos from the camera to the computer
• Task: Please login to the computer
• Task: Make the photo of the car brighter
• Task: Make a shortcut on the desktop for the addressbook program
• Task: Make a new photo tag
• Task: Make a note for yourself
• Task: Play the selected song
• Task: Remove some songs from the playlist
• Task: Please rotate the photo of the cat
• Task: Assign the "People" tag to all photos with people in them
• Task: Turn the volume down
• Task: Turn off the login music

I found it interesting that eight out of twelve succesfully completed the "Find out if the computer is online" task. I also wonder if these users could complete all these tasks in Widows.

Re:So much for this

[Twylite]

The required population size depends on what you're trying to test and how carefully you select your population. If you're trying to test the failure rate of moderately experienced Windows users performing tasks on a Linux system, and you can accurately identify and select moderately experienced Windows users with no prior Linux experience, then you only need a tiny population.

Testing the failure rate is important: 100% of 11 users succeeding at a task can give you at most ~ 90% confidence that all similar users will succeed. 1 of 11 users failing is a far stronger result, telling you that you can expect at least 9% of all users to fail.

Various usability experts suggest that as few as 5 or 10 individuals are required for usability testing, and the remaining usability issues are discovered and resolved via the bug reporting and maintenance processes.

Re:So much for this

[99BottlesOfBeerInMyF]

89% sounds like a very good success ratio for the date and time test. However, RTFA and you'll see that only eleven people participated, most of them female.

Eleven people is a pretty good sized group for a usability test. This sort of testing is pretty expensive and time consuming, it's not like a survey or something. From a group that size, you can get a pretty good idea of how the average person will try to accomplish a task and some problems they may encounter. I've worked on projects where usability tests included only three people to test the interface to a product costing tens or hundreds of thousands of dollars. I'm going to have to disagree that these results are meaningless. The important thing is how did the user who failed try to do the task. What stopped them? What problems did other users have?

Re:So much for this

[Bogtha]

89% sounds like a very good success ratio for the date and time test. However, RTFA and you'll see that only eleven people participated, most of them female.

So only one of them had problems? Sounds good.

If you don't have a diverse testing population, you aren't going to produce meaningful results. The idea is fine and all, but the results are mostly useless.

You shouldn't let the small numbers put you off. Respected usability professionals say you only need five people for meaningful results.

Re:Hm.

[databyss]

I believe your answer lies here: http://www.betterdesktop.org/welcome/reports/repor t-date-time.html

# Issues encountered:

1. The date and time configuration tool is not easily discoverable from the menus, and is not listed in Personal Settings.
2. Users assumed the root password request meant they had to log in as root.
3. Users wanted the click behavior of the clock applet to be similar to Windows.

# Recommendations:

1. Fix time and date settings to not require root access.
2. Add time and date settings to the Personal Settings window.
3. Make the clock applet behavior be more similar to that of the Windows taskbar clock.


Too bad the tests have such small sample sizes.

It would be good to get some big numbers on these issues, although I acknowledge that it would be very expensive.

Re:Hm.

[zootm]

Putting a CD into a Windows machine pops up a box, one of the options of which "Copy the music to your computer". Your boss must be really dumb.

Claiming it is non-trivial on Windows is fallacious though, it's certainly no more difficult. Windows Media Player is included with Windows (except in some versions in the EU!) and is capable of ripping CDs.