Portable Usability Labs As User Research Tools

Pete Gordon writes "Do Portable Usability and User Research Labs make sense in the software development life-cycle? This interview (my bias--it's with me, and I have a tool in beta now) covers some of the issues and questions on KDE's news site. I don't have the right answers necessarily, just looking for others input and opinions. Also, here are other links about the subject over the past few months. Info World and Harry's comparison."

Comments

Cost?

[Dancin_Santa]

Open Source projects, more than other types of projects, have serious financial constraints. Is the cost/benefit ratio of performing these labs worth it? Seeing as how Open Source projects typically form the backbone of systems and rarely form the front (user-facing) end, is it worth it to spend time and money on projects that will only be used by developers and hackers?

Re:Cost?

[metlin]

Short term, yes. Long term, no.

In the long term, it would be worth it. Hate it as you will, the precise reason Windows does so well in the market is its user interfaces.

User interfaces play a very very vital role in user behaviour, and usage.

I do not understand the argument that developers should not have good UIs. Why not? Would you not use a Visual IDE for your development if it had more features that you would use? Or would you rather that we all stick to CLI?

In fact, I really *like* Microsoft's Visual Studio .NET's IDE -- it's really quite well done, and very well designed with the developer in mind. And guess what? It increases my productivity by a significant amount when I code.

I'll just say this -- if Linux has to make it big, user interfaces _are_ a big deal.

There is a HCI maxim that says that the best designs are those that you do not notice -- that is what we should be striving for, Opensource or not. You never know who would be using your Opensource application for what.

And a good UI design is only going to help it.

Re:Cost?

[metlin]

And oh, I forgot to add this.

Anecdotal evidence - I was suggesting implementing some Opensource solutions to a company, and the CIO quotes JWZ -- Linux is nice if you're not constrained on either time or money (don't remember the exact quote).

He felt that rather than train the existing users, use Linux and fix the problems, he would take up a reliable and commercially tested solution - not merely in terms of how it works, but also in terms of usability and support.

I really didn't have a good enough rebuttal to that. However, if we do begin such practices, it adds more credibility, and results in better software - software that the industry would trust _more_ readily than ever before.

But then, that's just my opinion.

Re:Cost?

[sgtrock]

I think I understand the point that your CIO was making, although I don't know who or what JWZ is. I've heard similar comments in the past from people. I have found that the best response is to turn it around. Ask how much time and money is spent dealing with problems with an existing solution. :)

Seriously, OSS is not a panacea. There are some OSS projects which are mind-bogglingly successful precisely because they provide a solution that is more cost efficient overall. This includes usability for the pieces that matter. The classic LAMP implementation is one such solution that springs to mind. OpenOffice.org is another that meets the needs for many (not all!) organizations. Lyx is also one that I'm personally fond of for document creation.

However, this is not necessarily true of all OSS projects. Even OOo won't fit the bill for some organizations. The best advice that I can give is to stay calm and objective. When another project comes up where OSS might fit the bill, ask yourself if your research shows a shallow learning curve in addition to equivalent or more functionality. If so, suggest that a bake-off be done between your proposed OSS solution and the closed source candidate(s). Make sure that the bake-off targets usability and functionality.

If the OSS project passes, great! You've got a foot in the door. If it fails, don't be disappointed. Don't beat a dead horse and whine about how OSS Teh Way. Instead, use the opportunity to find out what the end users and other techies thought was missing. Use that information as part of your search criteria when you go looking for the next possible candidate.

It may take you a couple of tries before you get an OSS project in, but as long as you stay calm and professional in your relationships with the other people involved, it won't cost you any skin. Keep your CIO in the loop as you do this exercise. He'll appreciate it.

Usability

[RomSteady]

Any effort to get usability information is worth it, whether it's a full usability lab, or just sitting behind someone who is trying to use your software with a pad of paper.

The only people who don't think that usability is worth measuring are the people you wouldn't want working on UI to begin with.

Re:Usability

[tonsofpcs]

The only people who don't think that usability is worth measuring are the people you wouldn't want working on UI to begin with.

You mean like certain e-voting-machine producers???
I agree.

Also, reading one of the links on the KDE News article, they suggest 'Why Apple', giving reason that "The Macintosh was the pioneer in providing a Usable Graphical User Interface." This is completely untrue. Xerox made the first GUI, and I believe the first usable GUI was either Intuition (Amiga's Workbench) or GEOS. The original MacOS was not very easily operated imho.
See Wikipedia: History of the graphical user interface

Ppl and PC's

[Ex-MislTech]

As Engineers and coders etc etc, we tend to take alot of things
as granted, and already understood ie. intuitive for our mindset .

But for the common man sometimes it does not jive .

If you think that alot of ppl still have flashing 12:00 on their VCR's
its gonna take some serious simplicity to get past their (fear?) of
the technical or just grasp of it .

I think monitoring computer usage amongst beginners and maybe even
intermediates could show were ppl are frustrating themselves, and
perhaps tools that could be provided to ease the road more
travelled , ie. the electronic office/school/home .

Ergonomics is for human physical comfort, this might provide
one for mental comfort of sorts .

Best example I can offer is tech manuals that leave out a step
that is obvious to the person that wrote it or coded the app,
but leaves the first time user sitting there thinking its obvious
something was left out, but not sure how to proceed .

Computer literacy is still pretty weak IMHO, and on the level of
Linux and nomenclature of OS subcomponents even more so .

The more we understand the users, the better we can adapt the
interface .

Ex-MislTech

A good thing, but not indispensable

[TuringTest]

Elizabeth Neal has recently written on this subject, and the title says it all:

Why You Don't Need a Usability Lab

Promoting the mindset for usability and user-centered design inside the KDE project is a very good thing, though.

Re:A good thing, but not indispensable

[platos_beard]

I don't think the most important factor determining usuability is addressed in either article. User interface design should be done by user interface designers, not programmers. They need a completely different skill set. While programmers need to understand the working of a computer and be able to extract essential information from documentation, UI designers need to understand the people and processes in the domain they're designing for and be able to extract essential information from people.

Put simply, programmers need computer skills while designers need people skills. Sometimes they overlap, but no more than random variation dictates (and possibly somewhat less). And even if they do, its a different mindset while doing one job vs. the other.

And both jobs are hard. A good UI designer has to get beyond the specific suggestions from users to see what the underlying need. UI designers have to find a way to get users to envision a system that doesn't exist yet and figure out how it could work best. Prototypes are essential. Skills to run meetings are essential.

The toughest part is dealing with criticisms of proposed designs. Sometimes the criticisms are because the new design isn't understood well enough, but other times the criticisms reveal a design flaw. Distinguishing between the two, and correcting misunderstandings of the proposed design without stifling further criticism (which you need) is a delicate art.

Test, test, and then test some more

[TrebleJunkie]

Usability testing is absolutely essential to producing good software -- I've seen too many applications that left the developers hands and went right to market and were utter crap, because the developers put together what *they* thought people wanted, rather than actually verifying at any point that they were on the right track. (And then, most developers have the nerve to get pissed at the user for suggesting they make changes. Go figger.)

Usability testing also mitigates most of the round-and-round arguments developers will always have between themselves over some feature or another. Instead of butting egos, ask the users.

Portable usability test environments are not all that hard to come by. Here, we use a couple of Windows Laptops with TechSmith's Camtasia to record users sessions. We can take the laptop to them, present them with whatever we're testing, record the sessions, bring them back, play back the sessions, make our notes and changes, and go about our business. It works rather well for us, and it's much more affordable than building a dedicated facility. Much more convenient for the users, too.

Short-term testing alone is harmful

[mattdm]

Creating truly usable software is a difficult task, and it makes sense that we'd want to apply the Power of Science! to the problem. So, we get Usability Testing.

Generally, the usability tests I've heard of work like this: you get a bunch of people and stick them in a lab (portable or not), and watch everything they do with the program for a while, as they complete a checklist of tasks. It seems to be prefered to get users who have no previous experience with the program, to prevent "bias".

Well, that's great, but it doesn't really address usability. It addresses short-term pickupability. Now, that's really important, and it _should_ be addressed -- but if it's the *only* thing you're concerned with, you'll miss other important issues relating to long term software use.

There's a unix-geek saying: "Unix *is* user-friendly -- it's just picky about its friends". Like all such jokes, there's a kernel of truth here. There's a very steep learning curve to the command line tools that are at the heart of the Unix environment, but once you've gotten up it, they *enable* you as a user to more easily do complicated tasks that would be very tedious in a GUI.

I don't mean that this is about the CLI vs. GUI thing, though -- that's just an example. I'd certainly be frustrated if my web browser were exclusively designed based on the reports of people who use it for a few hours to complete basic tasks. I'm concerned about the line of thinking that removes features which save huge amounts of time every day simply because they might confuse new users.

I won't name any names, but I will cough subtly in the direction of the GNOME project and at metacity.

Please, usablilty people, don't just think of first impressions. Think of the long-term relationship. Both have to be good.

It's called HCI

[Dabel]

It's Human-Computer Interaction (HCI). Every year there is a large ACM conference on this called CHI. There are also hundreds of HCI researchers all around the world at some of the top institutions working on problems like this.

Georgia Tech and Carnegie Mellon have two of the bigger masters programs available. Each program pumps out between one to two dozen people a year who should be well equipped to perform usability testing, among other things.

And you don't need a whole lab. You don't need to videotape often, and you don't need to buy some special software/hardware (you can, and they help, but you can get a lot of mileage from much less). Jakob Nielson and his cohort Don Norman have published a few good books that should be accessible to the uninitiated. Often times, some scribbles on paper are a better choice than prototyping the interface (scribbles usually give you higher levels of feedback, as opposed to "The font is ugly.").

There really are much better sources than articles like this one where people are just discovering HCI methods (not to rag on the article). Do a little google searching (you now have the right keywords: usability, hci), read some books (amazon is bound to have something up your alley), and maybe even ask some people in the field. There's a lot of really cheap, really quick things you can do to help yourself out (lookup Nielson's Discount usability, or you can hire an HCI person onto your team, we're very worth the cost).

BTW: There are many more excellent sources than Nielson, he's just the easiest to cite for applied HCI in a short period of time.

Why many user experience peeps don't do OSS

[count0]

It's a nice thought - open source user experience design (user research, interaction design, functionality, UI, not just visual design).

However, on most OSS projects, if you don't code, you're a second-class citizen. There are regular threads every year on user experience lists about "why the OSS community should listen to us" that are filled with anecdotes of rejection by dev teams when a designer or usability person has tried to get involved.

I don't have any particular answers either, other than that I'm sure there are good OSS developers who would like UX design talent on the team - but there's not a real venue for getting them to work together, and there's not a culture of involving noncoders in most OSS projects.

Open Usability is trying to bridge the gap, but still has a long long way to go. (from getting profile in the OSS community and UX community to getting rid of the focus on 'usability' professionals and a focus on testing / evaluation)

Re:Why many user experience peeps don't do OSS

[TuringTest]

I would fight this battle with the "Pixels Are Code"(tm) mantra. The idea after this slogan is to remark that HCI experts create a precise, exact output which is a high level description of the final application; something akin to the intermediate bytecode produced by a compiler, just with a very descriptive language. I.e. UI experts produce one step of the design process.

You should persuade OSS developers that the work of a usability analist requires technical skills and produces careful designed specifications, then they will recognize your hacking ability as a part of the programming process.

The problem I've seen with OSS developers is that they are not familiar with the knowledge and glossary of the HCI field, so they tend to consider advice from a UI expert as mere opinions. In order to break this trend we'll need to better marketing the usability techniques, in a language that they could understand and relate to the "code-sharing" experience of Open Source. Thus the usefulness of the "P.A.C." mantra.

Re:You can't test your way to a great solution

[Pete Gordon]

I totally agree with this. I really want to communicate the information gathered from field research back to everyone involved in the product--especially the developers. That is the real goal of this lab!

Screen capture or not, doesn't matter. You could just use the video (multiple cameras) and audio to capture the business process of an individual or a focus group setting.

I really want to see a User-Centered Design approach.

Barbara Nelson at Pragmatic Marketing (a marketing IT shop that is focused on the Product Marketing Manager) said it this way...

"Listen to customers everywhere - online, onsite, at user group meetings, customer advisory boards, technical support, usability labs, point of sale, focus groups (in person and online), and through email. Anywhere they might be. Their natural habitat is by far the most fertile, but take advantage of other places they might congregate."
http://www.productmarketing.com/magazine/1/4/09bn. htm

Thanks!
Pete Gordon