Factors to no longer overlook

XV.5 September + October 2008
Page: 74
Digital Citation

FEATUREThe researcher-tool mismatch


Authors:
Rob Tannen

Pity the typical user researcher—hand cramped from writing pages of subsequently illegible observation notes, arm sore from trying to hold a camera steady for hours on end, head aching from trying to assimilate so much information. And if all goes well, that was the easy part. Returning to the office, the researcher now faces piles of notes, recordings, transcriptions, and other unstructured information to make sense of, but little time to do so. The data collection itself is the lesser challenge compared with the organization, analysis, and communication of the findings (i.e., the “secondary” research project).

Early on in my career I recognized an irony in design research: Professionals who are striving to understand user needs for new products were often doing so with relatively low-tech, generic tools—pen and paper, basic audio-video equipment, and Microsoft Office. The user research profession has accomplished a lot with a limited toolbox, and guidance on how technology could benefit user research practitioners is rare (e.g., Wilson and Conte [1]). Fortunately, this has been changing, due to the introduction of technologies that directly support the user research process. For example, Techsmith’s Morae software, launched in 2004 [2], specifically streamlined the time and equipment needed for conducting and recording software usability testing.

I began studying how technologies were being utilized in user research and what tools could improve data collection and analysis. A survey of user research professionals I conducted in 2006 [3], found more than half of the respondents used videocameras to document their observations, with more than a third using computer-based video. Presumably, the use of various technologies in user research has grown since then. But while technology use was widespread, there was also strong evidence of dissatisfaction. More than half of the survey respondents indicated that user research technology lags behind technology that is being researched, and more than one-third reported that they had created custom technology tools or solutions for user research. In other words, technology is important for conducting research, but off-the-shelf solutions frequently did not meet the needs of user researchers.

Characteristics of User Research Technology

In defining the needs of user researchers, I identified the following characteristics of current and potential user research technology:

  • Documentation—capability to record occurrences for subsequent review. Documentation tools can range from low-tech (pencil and paper) to high-tech (digital video) but share the common function of improving accuracy and reducing dependence on memory.
  • Measurement—taking documentation a step further, measurement references an attribute against a known, typically quantitative, comparator. Measurement tools too can be simple (a tape measure, rating scale, or stopwatch) or complex (body-measurement scanning system).
  • Efficiency—referring to tools that allow user research tasks to be completed more quickly or with less effort than otherwise. For example, an online survey allows access to a greater number of respondents versus an in-person survey.
  • Enhancement—tools that allow researchers to observe phenomena that are invisible or otherwise difficult to access. For example, eye-tracking systems document and measure eye movements, which are inherently too fast or complex for capture without such a technology.

These four characteristics provide a framework to evaluate and compare the relative benefits of particular technologies, but they also give perspective on trends in technology utilization for user research. Traditionally, user research technology has focused on documentation and measurement—the ability to record and quantify what is happening for subsequent review and comparison. More recently, there has been a shift in the application of technology toward efficiency and enhancement: efficiency primarily through Web-based tools to provide access to users regardless of time/distance, and enhancement via the availability of advanced sensor technologies.

In tracking the emerging technology space, I have been using several new or recently launched tools that, while not always designed specifically for user research, have direct application to the field. And while no technology is a substitute for skill and technique, new tools can improve the speed, validity, and benefits of user research.

The goal here is to increase awareness of the growing field of user research technologies, rather than fully endorse particular products. In fact, researchers should be hesitant about adapting new technologies too quickly—reliability and simplicity are the priority for research tools so they don’t interfere with or impede the observation process.

Video Technology

While high-definition (HD) resolution is extremely popular for professional and consumer video, I advise against using it for most practical research applications. Higher-resolution video means larger file sizes (roughly three times that of standard video) and consequently more processing time for video editing and file management. Moreover, the benefits of high-definition recordings are lost if high-definition screens are not used to display the recordings. For example, video compression for online presentation will typically diminish the native clarity of HD.

Rather than higher resolution, the relevant advancement in video technology to user research is higher speed. Controlling the speed of video data capture has been a valuable technique for studying the physical sciences for years, but it had limited application in the user research field. For example, time-lapse video has been utilized in medical ethnographic research [4]. In this context, video samples are taken periodically (e.g. every 10 seconds), rather than continuously, to reduce the volume of information to be analyzed, while still capturing the relevant procedures and measures. Time-lapse video is emphasizing the characteristic of efficiency by reducing the data-set size.

High-speed video, which is essentially the converse of time lapse, is emerging as a new technique for user research. Relative to the standard video-refresh rate of about 60 fields per second [5], high-speed video can capture hundreds or thousands of fields per second. More information is captured, enhancing the researcher’s ability to observe occurrences that are too rapid or brief to perceive unaided.

We’re currently applying highspeed video to:

  • Improve our understanding of the steps in complex manual maneuvers that occur too quickly to see clearly with standard video analysis. For example, the details of manual surgical maneuvers, to inform the design of medical instruments.
  • Analyze the micro-ergonomics of everyday product interactions—from cooking utensils to consumer electronics—to identify motor-coordination hesitations, inefficiencies in hand travel, and similar improvement opportunities.
  • Explore rapid user responses and reactions (e.g., quick gestures and facial expressions) as correlates to usability and aesthetic characteristics for prototype evaluation and differentiation.

Until recently, high-speed video has been prohibitively expensive and technical to use. Casio has changed that with the launch of the Exilim Pro EX-F1 (http://www.exilim.com/intl/ex_f1), the first prosumer-targeted digital camera with high-speed capabilities. The camera provides several unique features, including the ability to record at up to 1200 fps per second (although 300fps and 600fps modes are more pragmatic).

Note that high-speed video requires sufficient lighting, so it is most appropriate for daytime-outdoor or studio use. And it requires significant file sizes for storage—about twice that of HD. But unlike HD, which provides an incremental aesthetic improvement, high-speed video provides a new perspective on observing physical behavior that expands the potential for identifying innovative product design opportunities. Extending visual perception to micro-seconds can reveal informative sub-patterns of movement that are overlooked or invisible at a standard time-scale. And as with any technology, thoughtful use and analysis is required for meaningful interpretation.

Audio Technology

Multi-tasking is an essential skill in user research. Whether interviewing a consumer, observing a task in real time, or usability testing a website prototype, the researcher needs to see and hear what is occurring while simultaneously writing down key information and insights. Recordings can back up the researcher, but handwritten notes are the primary means of documenting and reviewing data. At the same time, notes tend to be succinct and coarse, conveying the concept, but not the verbatim of an event.

The Livescribe Pulse (www.livescribe.com) is a pen-based computer that offers the ability to narrow the gap between what you observe and what you write. The pen records the ambient sound (e.g., interviewee’s voice, contextual noise) and synchronizes the audio recording to what you are writing. Subsequently, when you place the pen-point on a particular note (or sketch), the pen will play back the audio that was recorded when you had originally written the note. For user interviews, the researcher can quickly reference notes directly back to the audio of interviewees’ words for clarity and idea expansion. For ethnographic observations, both conversations and environmental sounds can be unobtrusively recorded (in stereo) while taking notes. While the device does not provide the highest-resolution audio quality, it increases the working bandwidth and accuracy of the design researcher.

Data Analysis Tools

After research data is gathered, the team needs to make sense of it all. A common challenge for design research teams is the process of organizing qualitative data into meaningful patterns of information. Identifying and prioritizing observations, user input and other “raw” content depends on an effective process. But the right tools can streamline these efforts.

Often the biggest challenge is knowing just where to start. For example, consider trying to make sense of the transcriptions from several dozen user interviews—you might have a mix of positive and negative feedback, anecdotes, opinions, and narratives. You could painstakingly sift through all of the conversations, highlighting meaningful terms, or perhaps search for keywords. Or you could leverage software that provides structure to text-based data to help guide your analysis.

One such example is IBM’s Many Eyes (http://services.alphaworks.ibm.com/manyeyes). While this Web application set is known to many for its beautiful graphing capabilities, I find the text-visualization tools most valuable for analyzing qualitative data. Data visualization has been promoted as an effective means of presenting data, but its enormous value in analyzing data has been largely overlooked. In particular, the Word Tree that “lets you pick a word or phrase and shows you all the different contexts in which it appears. The contexts are arranged in a tree-like branching structure to reveal recurrent themes and phrases.”

When applied to qualitative data (e.g., interview transcripts, free-text survey comments), the Word Tree allows a researcher to quickly scan through text-based content by searching via keyword or phrase. For example to see what a group of users said about a particular product feature, the researcher can create a word tree around the feature (e.g. “registration” or “installation”) or around particular terms that are likely to indicate problems (e.g. “difficult to”, “but”). Visually structuring the data around critical terms provides a starting point for reviewing and understanding qualitative data in an efficient manner. It is not a substitute for thoughtful analysis, but a head start. One significant caveat in the case of Many Eyes is that all submitted data is publicly viewable, so it’s not always suitable for proprietary data analysis. But it is free.

Research Management Tools

On a broader scope, the greatest challenge in design research may be data management. From managing information within a project to sharing research trends across an organization, a user research database can grow into a valuable corporate asset. But collecting and organizing research across projects is rarely done efficiently, if at all.

There is a need for techniques and tools that support better research-data management and communication. New specialized software tools are providing a platform for accomplishing this. For example, QSR International’s recently launched NVivo 8 (http://www.qsrinternational.com/default.aspx) provides a framework for entering, tagging, and querying various forms of qualitative data (including audio and video) across multiple projects. HTML output can be produced to readily communicate and present research findings. These types of tools will enable more effective collaboration among both localized and geographically distributed researchers and can provide a centralized repository for observational data.

The value of well-conducted research is extremely limited if it is not organized for effective communication. Arguably, providing a system for documenting and sharing research data will likely have a greater organizational benefit than any other research-related technology.

Researcher-Centered Tools

Until recently, the availability of technology to support user research has not kept up with the overall pace of technology development. Now that there are a range of tools available, it is time to provide guidance on which technologies are appropriate for the varied contexts of research applications. The four attributes: documentation, measurement, efficiency, and enhancement can assist practitioners in focusing on the relevant characteristics to support their needs.

The technology examples discussed here touch multiple phases of the research process, from data collection through analysis and organization. When used appropriately, these tools contribute to improved efficiency, detail, or information sharing. Ultimately, high-definition research is achieved when the researcher can focus on the meaning and patterns of the findings, rather than the clarity and organization of data.

For more information about characteristics of user research technology, you can download the slides from an overview presentation on the topic that I gave to the New York City chapter of the Usability Professional’s Association (UPA) “Technologies for User Research (TURe)” on October 17, 2006; http://www.nycupa.org/pastevent_06_1017.html

References

1. Wilson, C. and L. Conte. “21st Century Technology for Usability and User Interface Design Activities,” Usability Interface 7, no. 1 (July 2000) <http://www.stcsig.org/usability/newsletter/0007-tools.html>

2. http://www.techsmithcom/company/about.asp

3. Tannen, Rob. “Survey Results: Technology for User Research.” Designing for Humans, 10 December 2006. <http://www.designingforhumans.com/idsa/2006/12/survey_results_.html>

4. Wiklund, M.E., and S. B. Wilcox. “Time-Lapse Video Offers More Information in Less time.” In Designing Usability into Medical Products. Boca Raton, Fla.:CRC Press, 2005.

5. http://en.wikipedia.org/wiki/Digital_video

Author

Rob Tannen is director of research at Bresslergroup (www.bresslergroup.com) an award-winning product design and development firm. Rob provides expertise and training in design research, human factors, and usability to support product/interface definition and refinement. He is creator and editor of DesigningforHumans (www.designingforhumans.com), the IDSA’s blog for design research and human factors Rob is a Certified Professional Ergonomist (CPE). He earned a B.A. in cognitive science from Vassar College and M.A. and Ph.D. degrees in human factors/experimental psychology from the University of Cincinnati.

Footnotes

DOI: http://doi.acm.org/10.1145/1390085.1390102

Figures

UF1Figure. High-speed photography of finger movements supported the evaluation of alternative key size, shape and placement arrangement on a prototype handheld device.

©2008 ACM  1072-5220/08/0900  $5.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2008 ACM, Inc.

 

Post Comment


No Comments Found