You fought hard for funding, the contractors have finished, and the paint is almost dry. Naturally you want to make sure that your new usability lab earns it keep. But don't get too enthusiastic about having a full schedule if it means that your development process will suffer as a consequence.
Here's the thing: while audio and video recording of test sessions, focus groups, and interviews in a controlled setting can be very useful, so can ethnographic research and candid feedback. Unfortunately, these last two activities (and several others) cannot be done in a usability lab. Ethnographic research in particular must be done in the field. Hauling participants into your test facility, no matter how charmingly decorated, will not produce the same results at all. We need to see how users really work, in their own environments, with their own equipment, materials and other resources. This is especially true of third-party interactions, where aside from the computer and its immediate user, there is a customer, passenger, patient, or other "ultimate" user. These complex interactions simply will not be the same in an artificial environment as they will in the real world.
Furthermore, because you have much easier access to your organization's own staff (than to, say, your customers) there is a very real concern that the Web-based system you are developing will target the wrong users. Let's take an example. Suppose that you are in the travel business, developing a new customer-facing system for use by your staff. While it is very important that this system be highly usable by staff, it would be to no avail if it did not at the same time meet the needs of customers. And you really don't have much choice here. Either you go out into the field and observe real customers with real needs or you become a playwright, drafting scripts to be enacted on this stage that is your lab environment.
Of course, the problem is not limited to three-party interactions. In ethnographic observation we are able to win the confidence of users by granting them anonymity and by trying to see the problem through their eyes. This means spending time and being a careful listener more than an interviewer. Back in the lab, though, we have little choice but to become an interrogator. Users are not going to be free to carry out their normal routine, partly through a lack of time, but also because they are in a totally artificial environment. Also, anonymity is rarely availablecertainly not if sessions are being video recorded. In fact, there is probably no better way to get recitals of official policy than to invite your staff to sit in front of a video camera and tell you how they do their job. But there are much quicker and less expensive ways of getting that information.
What about usability testing itself? Surely you can do as much of that as you'd like in your new lab? Well, not necessarily. Most usability testing is based on prearranged tasks. If these tasks are not realistic, assessing users' performance of them is an exercise in self-deception. We need to have done enough fieldwork ahead of time to know what the real tasks are and how important each is in the scheme of things. What this comes down to is a usability testing "sandwich":
- Early fieldwork
- Usability testing
- Follow-up fieldwork
The early fieldwork is to find out who the users are, their goals, tasks, working methods, resources, artifacts, and other contexts of use. Usability testing is done in the lab with paper prototypes1, working prototypes, and eventually early releases of the system being designed. Then, to be sure that we have done all of the earlier work correctly, we check that the system really does meet users' needs by observing it in practical use.
If this approach leaves embarrassing holes in your lab schedules, there are two things you can do. You can either live contentedly with the knowledge that you will be able to handle a larger number of projects than originally anticipated or, if you would like to keep your facilities as fully booked as possible, sell your design teams on the idea of paper prototyping1. In my view this is always an excellent investment of effort and a good use of a usability lab into the bargain.
©2004 ACM 1072-5220/04/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2004 ACM, Inc.