Niamh McNamara, Jurek Kirakowski
Evaluation is a mainstream activity in HCI. For many years we saw the emergence of a plethora of techniques to measure user-orientated quality assessment of technology: usability, satisfaction, efficiency, effectiveness, learnability, usefulness, and so on. In recent times, however, the discussion seems to have moved on. Issues surrounding the wider relationship between people and technology or the user experience are popular. And we've seen a number of new concepts emerging, such as engagement, pleasure, presence, and fun, to name a few. Their proponents suggest that these concepts represent important aspects of usage that are omitted by traditional approaches to evaluation. We've heard it all before, of course, but the words are new.
However, does the emperor actually have new clothes? In their efforts to explore concepts related to the user experience, researchers have been slow to articulate how their proposals should be measured or indeed if they can be measured at all. Clearly what is needed is an organized discussion on what aspects of use need to be investigated when the time comes for evaluation and how they could be assessed. We propose that when evaluating technology, there are three primary elements that need to be considered, namely, the product, the interaction between the user and the product, and the experience of using the product. Each of these three elements represents a unique but interdependent aspect of usage. These are Functionality (product), Usability (interaction), and Experience (user experience). Each area asks a different question about usage using a different language of discourse.
What should we be evaluating? Someone sending a text message with a numeric keyboard may well agree that the usability of the interface is awful, but their experience may be very immersive and compulsive, and, as we see, different mobile-phone operators pack in different kinds of functionalities with their product, hoping to attract different market sectors. This is why we've suggested that Functionality, Usability, and Experience represent three aspects of usage that need to be considered when designing and evaluating technology .
Functionality is a technical issue and refers solely to the product. In this instance, the goal of evaluation is to answer the question, "What does the product do?" The usefulness of device features, maintainability, and reliability are some of the issues that could be addressed in such an evaluation.
Usability is a characteristic of the interaction between the user and the product. It is a user issue; therefore the product needs to be tested with real users. The criteria of support and goal facilitation are important here in answering the question, "Can I make the product do what I want it to do?"
Finally, the user experience considers the wider relationship between the product and the user in order to investigate the individual's personal experience of using it. According to McCarthy and Wright [7, 8] these questions might include "how the person felt about the experience, what it meant to them, whether it was important to them, and whether it sat comfortably with their other values and goals."
These three elements are not strictly independent of each other. For example, although usability is not a product characteristic, physical features of the product such as the level of functionality and the way in which features are implemented can have an impact on usability. It has even been suggested recently that device aesthetics can influence perceived usability prior to actual use . The device's appearance can also affect the user experience, since people express themselves through the consumer products they own. Finally, usability also influences the user experience. Poor usability would no doubt contribute to a negative user experience, which in turn would possibly discourage further use of the product or the inclination to buy another from the same source.
We saw some preliminary support for this framework emerging during the development of a questionnaire to measure user satisfaction with electronic consumer productsthe Consumer Products Questionnaire (CPQ). (We're hoping to bring more information into the public domain about this questionnaire in the near future.) Respondents completing the CPQ were asked to describe in their own words helpful and unhelpful aspects of devices they were evaluating. We gathered nearly 1,000 comments in the course of time, which we pooled and analyzed using Krippendorff's Content Analysis method . We were surprised that only 24 percent of the comments related to usability; 47 percent of the comments addressed functionality, performance, hardware limitations, and durability; and 23 percent of comments were related to general experience with the device, including aesthetics, usefulness, concerns about health and safety, cost of using the device, and entertainment value. The Content Analysis method has its own standards and criteria for reliability and objectivity. Doing this analysis was a learning experience for us and helped us see things more clearly.
How Should We Approach Measurement? The question of how best to assess each area of concern is not an easy one to answer. In terms of evaluation, it is important to realize that each area of investigation incorporates its own unique set of methodologies and assumptions and the significance of choosing an appropriate methodology when carrying out an evaluation in any of the three areas cannot be overemphasized. It should be clear that one could not use established usability evaluation methods (UEMs) like time on task, or number of errors to assess the user experience. Likewise, one cannot use UEMs to assess a product's functionality. It is vital that experience is not reduced methodologically to usability in just the same way that it would be a "category mistake"  to attempt to measure functionality in UEM terms.
Assessing Functionality. This might include determining what features should be provided by the device, as well as evaluating device performance, reliability, and durability. The analysis of user comments revealed that, in general, respondents appreciated the functions provided by electronic devices, but they were intolerant of unnecessary functions. The comments also indicated that users felt some devices lacked certain functions that they would find useful. This suggests that designing for a market position is important, and designers should take the trouble to ask for a broad description of what users would like to be able to do with a particular product rather than adding additional features for their own sake or because it is inexpensive to do so [10, 1, 2]. Of course it should be noted that this is not a foolproof option. It would be difficult for users to decide beforehand exactly what features they do and do not want, and to some extent the development of electronic devices is governed by an attempt on the part of the designers to "try it and see if there's a use for it."
Assessing Usability. User comments suggest that issues such as transparency, learnability, and the support offered to users through guides, manuals, and clear and informative device feedback are areas that require particular consideration: all the "traditional" preoccupations of usability engineering. The measurement of usability should be based on the definition of usability as quality in use, as enshrined in ISO 9241-11, as "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction" . There are numerous well-established usability evaluation methods, but be warned: Evaluators will have to determine the success with which these can be applied to new technology before they can be used with any degree of confidence.
Assessing Experience. As this is a relatively new area of investigation within HCI, there aren't any well-developed assessment methods for evaluators. Some people advocate broadening the usability construct to include more-subjective dimensions, while others call for the creation of completely new constructs that are distinct from usability. Interestingly, most approaches tend to characterize experience as dependent on product features, a method insightfully dubbed design reductionism by Hassenzahl, Beu, and Burmester . A more suitable approach, McCarthy and Wright's Felt-Life framework, argues against design reductionism, suggesting that user experience cannot be deduced from product features. They propose that usability professionals should not be concerned with designing an experience but rather designing for experience.
Advances in technology change the relationship between people and technology, and we have to keep up. But let's not just leap in. Successful measurement in HCI depends on a clear understanding of what we are measuring and how we should measure it. Without this understanding evaluators can offer little guidance to designers to assist them in the creation of technology that truly supports the user.
9. N. McNamara & J. Kirakowski, "Defining usability: Quality of use or quality of experience?," in Proc. IEEE International Professional Communication Conference, Limerick, Ireland, 10-13 July, pp.200-204, 2005.
University College Cork, Ireland
University College Cork, Ireland
About the Author
Niamh McNamara graduated with a first-class honors degree in Applied Psychology from University College Cork. She has undertaken research into the usability of the WAP (Wireless Application Protocol) service and evaluating Web site usability including a Web site for adults with limited literacy skills. She is currently pursuing a PhD on the development of a tool to evaluate user satisfaction with electronic consumer products.
Jurek Kirakowski specializes in quantitative measurement for human-computer interaction. His major research goal has been to show how the quality of use of IT products can and should be quantitatively measured in an objective manner. Dr. Kirakowski and the Human Factors Research Group in University College Cork have contributed the SUMI (Software Usability Measurement Inventory), and the WAMMI (Web site Analysis and Measurement Inventory) questionnaires which are by now de-facto standards in their respective areas. His personal Web page is http://www.ucc.ie/hfrg/jk.
©2006 ACM 1072-5220/06/1100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.