Literature that provides practical techniques for conducting usability testing of mobile applications remains limited. Had the following tips and tricks been available when I conducted my first usability test of a mobile application in 1999, it would have saved me much grief.
1. Account for prior mobile experience, but do not assume too much. Most study recruits will have experience with at least one mobile device. However, do not assume that people know the device make or model, industry lingo (e.g., "converged device"), or acronyms (e.g., "SMS").
Mobile-device interfaces vary greatly. Participants’ performance may suffer if a study requires a device that differs from the ones they own. Conversely, a participant may perform inordinately well if using a familiar device.
Design both the recruiting screener and the study accordingly.
2. Bring extra batteries, electric wall chargers, and a backup mobile device. Batteries tend to run out at the worst time, so keep devices charged between sessions and have plenty of spares. For long, intense studies, keep the device plugged in or swap a fully charged battery between every session.
Have two of each device, especially if the hardware is also being tested. If testing a device that requires a Subscriber Identity Module (SIM, or smart card), have at least two SIMs available in case the account gets canceled on the day of the test or one jams in the device. Yes, these things actually happen!
3. Allow for additional time to restore devices to a default state between participants. When tasks call for sending, receiving, or entering data, allow extra time between sessions to return the device to its default state. For example, if testing a mobile mail application and one task requires deleting a message, you will need to restore that message for the next participant to delete. Or, if multiple devices are available, swap out the used one with a fresh one, and return them all to the default state after completing the day’s sessions.
4. The video camera and its position impact the usefulness of your recordings. Use a camera attached directly to a mobile device, if possible, to approximate the experience of participants holding the device. At least one such camera on the market offers wireless video streaming. Another option is a document camera with a "gooseneck" allowing the camera to angle over the device, a compromise between recording the screen and obstructing the participant’s line of sight. A ceiling-mounted camera works well as long as it can be manipulated remotely to follow the device. A standard video camera may also be used to record over the participant’s shoulder, though it risks making participants more conscious of being recorded. Be prepared to readjust in real time.
5. If the device must be steady to record what is displayed, make it less mobile. A device may be temporarily held in place through a variety of techniques. For example, affix an angled platform such as a book holder to the desk and place the device on it. Place fastening strips on the back of the device and the platform to hold it in place, though they may leave behind adhesive residue (and annoy the person you borrowed the device from!).
For a more elaborate but effective solution, construct an arm and clamp using a mobile-phone car mount to hold the device in place from the sides. Attach it to a piece of wood and use a C-clamp to hold the assembly in place (see Figure 1). The device may be positioned such that the keypad protrudes below, allowing participants to use it more naturally than if the device were placed directly on a flat surface.
6. Plan and test your lighting so that the mobile screen is clearly visible. Many mobile devices have reflective screens, so test the lighting and recording beforehand. Halogen floor lamps or track lighting on opposite sides of the room with dimmer switches work well. If the room has a window, block it completely or be prepared to add recalibration to your list of things to do between every participant.
Mobile screens use backlighting that turns off when not in use. When possible, set the backlight timer to its maximum interval. Otherwise, keep the tasks short and intense so that participants press keys frequently, keeping the screen illuminated.
7. Use lower-fidelity paper prototypes early in the product lifecycle. When constructing a paper prototype for a mobile application, create a simple outline of the device on which the application content will be displayed (see Figure 2). If a photorealistic device image is used instead, the participant may spend time commenting on ergonomics and appearance rather than the user interface.
If the paper prototype approximates the dimensions of the target device, have participants use a pointer such as the back end of a pencil to tap the button they want to press. Doing so makes it easier to see which button was pressed when later watching it on video. A practice task requiring participants to press several buttons on the keypad will ensure that this indirect interaction method does not affect the study.
8. Add to the realism of the interaction by getting creative with focus. One way to emulate focusthe highlighting that informs users where they "are" on the screenfor a paper prototype is to cut a section of an overhead projector slide in the shape of the state of Oklahomaor Idaho if you prefer! (See Figure 3.) The study administrator or "Wizard of Oz" operator holds on to the large portion and slides the thin part over each item or line on the paper screen as the user presses the navigation buttons.
Testing on a Live Network
9. Test the signal strength available in your study location for the exact device and network that will be used. Scout the test location on more than one occasion to ensure adequate signal coverage. Network coverage, while improving, is still not ubiquitous. Be sure to test the signal available on the specific device and the specific network, ideally at the same time of day as each planned session.
10. Plan to lose network coverage despite your best efforts. For most practitioners, only the application software and perhaps the device hardware is under your (company’s) control. Always have a fallback plan in case you lose signal coverage for the test device. A plan B will allow you to avoid dismissing the participant without gathering something useful.
Have more tips and tricks? Please share them at www.mobileixd.com/forum for the benefit of the mobile usability community and the participants who may otherwise needlessly suffer through an "unexpected" study experience!
Acknowledgements: Special thanks to Michael Longé of Designer Software for the photo, editing, and support, and to Tamara Adlin of Adlin, Inc for the inspiration.
Mobile Interaction Designer
About the Author
David Schultz has spent almost ten years improving the mobile user experience, one product at a time. He has toiled as a usability engineer, interaction designer and most recently as a user experience researcher. Millions of people enjoy products based on his designs on a daily basis. When not wrestling with mobile industry titans, David enjoys spending time with his spouse and young daughter.
©2006 ACM 1072-5220/06/1100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.