Features

XIX.6 November + December 2012
Page: 24
Digital Citation

Foci and blind spots in user experience research


Authors:
Javier Bargas-Avila, Kasper Hornbæk

In recent years, HCI has been influenced by a movement known as user experience. Take a look at current job postings and you can’t help but notice that companies often look for “user experience specialists” (rather than “usability experts”) or “user experience designers” (rather than “interaction designers”). At conferences, an increasing number of papers describe “user experience” (rather than “usability”).

The phrase user experience (UX) denotes new ways of understanding and studying the quality in use of interactive products, ways that represent a reaction against usability research and “traditional” HCI. UX researchers have argued that usability research is too focused on optimizing task efficiency, removing problems in the user interface, and studying interactive products used for work. Traditional HCI is seen as having a mostly instrumental, task-oriented view of interactive products.

In contrast, UX focuses also on hedonic qualities of use and much more broadly on experience. Hedonic qualities concern, for instance, aesthetics, fun, and identification that people experience during interaction. This focus requires new approaches for designing and evaluating interactive products because existing methods are unable—it is claimed—to capture experience. Thus, many researchers in the field of UX state that they methodologically break new ground or study new facets of interactive products’ use.

Here we discuss whether this is really the case. To what extent is UX research novel, and to what extent does it build on usability evaluation or traditional HCI research? To answer these questions, we reviewed 51 publications from 2005 to 2009, reporting a total of 66 empirical studies of UX. We looked at which products these publications studied, the dimensions of experience they assessed, and the methods they used. The goal is to understand the foci and blind spots in UX research. More details of the review may be found in our 2011 paper [1]; while the present article focuses on usability researchers and their claims in general, the 2011 paper provides references for these attributions and contains more examples of what is done and not done in current UX research.

What Does UX Research Study?

Some researchers posit that the focus of UX research should be interactive products used in everyday life, whereas other researchers are willing to apply UX methods in any domain. We took a look at the products researchers study and found that the most frequent topics of research are consumer products, like mobile phones or apps, and art (each 21 percent of our sample; see Figure 1). The context of use is typically leisure (64 percent), whereas work or a combination of work/leisure is studied less frequently (each 18 percent).

It seems that the UX movement not only shifted the focus from usability to experience, but also changed the products and use contexts that researchers study. The broadening of products studied is one motivation for UX research. Nevertheless, many studies in our sample speak generally about UX, yet fail to reflect on the products used to study UX. The UX movement criticized traditional HCI for focusing on work-related products. The narrow focus on consumer products and art seems comparably harmful because it may preclude conclusions about experience in general or other types of products.

Many arguments against usability research, also made by UX researchers, question employing researcher-set tasks to study UX. We therefore looked at which types of use researchers study and found three groups: controlled tasks, where participants receive precisely detailed assignments; open-use situations, where participants receive open instructions that explain the next steps, without providing detailed instructions or goals; and user-initiated use, where participants are free to choose if, when, and how they use the interactive products. The analysis shows that 61 percent of the studies use open-use situations, suggesting that the UX researchers do study more open use of interactive products.

Many researchers emphasize that UX is heavily influenced by the context of use. When, where, and with whom a product is used will influence the type and quality of experiences triggered. In almost half of the studies we sampled, researchers controlled the context by conducting their inquiry in a fixed setting (mostly lab studies). Another third of the sample used uncontrolled context (e.g., giving users a mobile phone to carry with them for a week). These studies provided no descriptions of the physical or social setting of use, and only 21 percent described parts of the use context. Despite the importance of context for UX, and at odds with the trend of studying open-use situations mentioned above, most UX research is silent about context of use.

* What Dimensions of Experience Are Studied?

The definition of UX and its key characteristics (dimensions of UX) is continually under debate. Some researchers write of UX as an “alternative to traditional HCI,” whereas others write that usability criteria can be used to assess aspects of UX. Therefore, we looked at which dimensions researchers assess in their work (see Figure 2).

We call the dimension that is most frequently investigated generic UX. This dimension summarizes researchers that do not further specify on which aspects of UX they focus. Common statements are “...it is possible to get a very rich view on the all-encompassing user experience” or “This paper reports on fieldwork, targeting user experiences of and with…” Generic UX stems mostly from qualitative studies, where focus groups, interviews, or probes are used. The second most frequent dimension is emotions and affect (24 percent), followed by enjoyment (17 percent), and aesthetics (15 percent). These are often mentioned as core dimensions of UX. With respect to emotions and affect, the most frequently used instrument is Lang’s SAM scale [2]. With this scale, users provide ratings of emotion by selecting versions of a manikin that symbolizes different emotional states. The scale is well validated and widely used, but is a relatively simplistic form of measuring emotions. We were surprised that not a single UX study in the sample seeks to obtain more detailed measures of emotions, while at the same time many researchers agree that emotion is one of the key components of UX.

We also found many authors who propose new dimensions for understanding UX. These dimensions include constructs such as enchantment, engagement, tangible magic, aesthetics of interaction, and relevance. Certainly, this variety is inspiring to the UX movement and supports new ways of thinking about experience. The main problem with these new proposals—what we call dimensionality explosion—is that the relation to established constructs is rarely made clear. Let’s exemplify this with the notion of “enchantment.” According to McCarthy et al., enchantment is a “useful concept to facilitate closer relationships between people and technology” [3]. Although the notion of enchantment is interesting, we wonder how it relates to other constructs: Is enchantment a crucial and distinct component of UX? What is the difference between enchantment and established concepts such as, for instance, the experience of flow? Is flow a condition or a consequence of enchantment? No studies in our sample report how enchantment can be measured and consequently help clarify its role within UX and the difference between it and other concepts. We do not think that enchantment is any more or any less problematic than the other dimensions being proposed. However, if new dimensions are needed and proposed, they should at some stage be accompanied by studies that clarify how they differ from established constructs. Otherwise we risk ending up with an endless number of words that describe similar phenomena within UX, something from which the characterization of user satisfaction in usability research has suffered.

Most researchers agree that UX is multidimensional. Therefore we looked at how many dimensions researchers study in their work. Surprisingly, almost half of the studies assessed only one dimension; 71 percent of the studies assessed two or fewer dimensions. Thus, UX—despite being thought of as a multifaceted and complex construct—is often approached in quite restricted ways.

* How Do We Study UX?

The methods applied in UX research are often debated, in particular those concerning the suitability of qualitative and quantitative approaches. A key issue is whether UX may be measured and modeled, and if so, how. Therefore we looked at the methods currently applied by UX researchers. Half of the studies are qualitative (50 percent), whereas 33 percent use quantitative methods and 17 percent combine the approaches. These figures differ considerably from traditional HCI, where one estimate suggests that 14 percent of the studies are qualitative, whereas about 65 percent are quantitative [4]. The shift toward UX seems to be accompanied by a change in methodology.

Another important question is how UX data is collected (see Figure 3). We found that questionnaires are the dominant UX-assessment method (53 percent). Others apply qualitative methods from HCI that involve talking directly with users, using semi-structured interviews (20 percent), focus groups (15 percent) and open interviews (12 percent), as well as user observation (17 percent), analysis of video recordings (17 percent), and diaries (11 percent). We also found an emerging group of constructive or projective methods, such as probes, collages/drawings, and photographs. Objective measurement of UX via psychophysiology is rarely used. Thus, data on UX is mostly collected with methods borrowed from traditional HCI and usability research.

Most researchers agree that UX occurs before, during, and after interaction with products. Therefore we analyzed in which of these phases researchers assess UX. Measures of UX before interaction is rare (20 percent), while after measurements are the most frequent (70 percent). Correspondingly, the most frequent pattern is the combination of during and after measurements—similar to traditional usability research, where users are observed when interacting, and satisfaction is measured afterward. Anticipated use assumes an important role in the field of UX and is a major difference from traditional HCI. We took a closer look at what researchers measure before interaction occurs and found that only five studies looked at users’ expectations about products. Thus, it seems that before measurements in UX are still widely disregarded. The analysis of temporal aspects shows that current UX research contains no truly longitudinal studies. Some papers study experience over several weeks, but projects that cover typical product life cycles of several months or even years are missing.

The choice of methods in UX research suggests a revival of the debate about qualitative and quantitative research methods. In particular we see that it leads to dichotomous research. On the one hand, some researchers study very particular use situations, emphasizing richness of description and using mainly qualitative research methods (we call those uniqueness studies). On the other hand, some studies model the dimensions of UX, emphasizing findings that generalize and using mostly quantitative research methods (we call those dimension studies). Some studies overemphasize their methodological stance within either of these methodologies to the extent of damaging research quality. Many uniqueness papers do not report interview questions or protocols, rarely describe data analysis methods, focus mostly on generic UX, and contribute to the dimensionality explosion mentioned earlier. Many dimension papers do not attempt to study complex, ongoing interactions (often using screenshots or studying very short interactions), and some say very little about experience and in-depth reports on UX. Unfortunately, few studies combine qualitative and quantitative methods (or uniqueness and dimension studies), and we find a sad lack of references between the corresponding groups of papers.

With respect to methods invented for studying UX, we found three major issues in our sample. First, new methods are rarely validated and are mostly used without comparison to other methods (or the comparisons repeat some of the mistakes that have haunted usability research). Second, what we have called constructive methods (e.g., sketches, collages, probes) are not only rarely validated, but also raise many issues about the process and validity of interpreting their results. Third, some authors argue the need to rely on first-person methods for understanding experience. Past work on user-centered design and usability research emphasizes that we need to look at behavior—what people do—rather than listen to what they say and what they say they do. The tension between these two approaches needs to be understood much better.

Our review has characterized current foci and blind spots in UX research. Future work must strengthen the interesting work that this movement has generated. At the same time, we have discussed issues that have tended to be overlooked in UX research and that are equally important to address in future work.

References

1. Bargas-Avila, J.A. and Hornbæk, K. Old wine in new bottles or novel challenges? A critical analysis of empirical studies of user experience. Proc. of the 2011 Annual Conference on Human Factors in Computing Systems. ACM, New York, 2011, 2689–2698.

2. Lang, P.J. Behavioral treatment and bio-behavioral assessment: Computer applications. In Technology in Mental Health Care Delivery Systems. J.B. Sidowski, J.H. Johnson, and T.A. Williams eds. Ablex Publishing, Norwood, NJ, 1980, 119–137.

3. McCarthy, J., Wright, P., Wallace, J., and Dearden, A. The experience of enchantment in human–computer interaction. Personal and Ubiquitous Computing 10, 6 (2006), 369–378.

4. Barkhuus, L. and Rode, J. From mice to men–24 years of evaluation in CHI. Proc. Alt CHI ‘07. ACM, New York, 2007.

Authors

Javier Bargas-Avila holds a Ph.D. in cognitive psychology. He has published over 20 peer-reviewed papers in HCI journals and conferences on topics such as user satisfaction, mental models in website perception, visual aesthetics, and webform usability. Since 2011 he has worked for Google, where he currently focuses on YouTube internationalization, monetization, and analytics.

Kasper Hornbæk is a professor with special duties in human-centered computing at the University of Copenhagen; his B.Sc., M.Sc., and Ph.D. were also done at the University of Copenhagen. His research focuses on usability/user experience and on advanced user interfaces (e.g., visualization, tangible/shape-changing user interfaces).

Figures

F1Figure 1. Products studied in UX research.

F2Figure 2. Dimensions collected in UX research.

F3Figure 3. Data-collection methods in UX research.

©2012 ACM  1072-5220/12/11  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.

Post Comment


No Comments Found