A tidal wave of interest in emotions is brewing among user-interface professionals. Have you heard the rumblings of this oncoming tsunami?
At ACM SIGCHI’s CHI 2003 conference a few months ago, Prof. Jodi Forlizzi, of the Human-Computer Interaction Institute and School of Design at Carnegie-Mellon University, organized a well-attended panel discussion about emotions in user-interface design (see program details at www.acm.org/sigchi/chi2003). I am indebted to her for promoting the discussion of this challenging topic and the opportunity to co-present at the panel discussion, which prompted this essay.
The speakers, and the questions posed after the panel discussion, raised powerful issues. Of particular note was one of the speakers, Pieter Desmet, University of Delft, whose dissertation research in emotions and product design are summarized in his publication Designing Emotions . Particularly appealing to the audience were his experiments using computer-generated, animated, visual "puppets" or cartoon-like creatures to exhibit possible emotional attitudes of consumers to product designs (like automobiles). These images directly communicated the emotions and did not require translation into words of a particular language, which always introduces some ambiguities or biases.
This CHI event follows some other recent events:
- At a conference at the Institute of Design, Illinois Institute of Technology (sponsored by the International Institute of Information Design in Vienna), Prof. Neil J. MacKinnon, Department of Sociology and Anthropology at the University of Guelph, presented his research on affect control theory . Conducted over many decades, Prof. Mackinnon’s research contrasts with George Herbert Mead’s classical cognitive, rational perspective on symbolic interaction. Prof. MacKinnon demonstrated an affective-attributes database in several languages covering many different concepts. With his computer-based tool, it is possible to construct scenarios of events and explore the likely emotional impact on participants. The capabilities seemed just a few steps away from allowing product and service developers to create use-case scenarios and exploring the likely emotional "meaning" of the artifacts and the interactions among people using them.
- At the Usability Professionals Association 2002 conference, Patrick Jordan of the Contemporary Trends Institute in London presented his tutorial "Pleasure with Products: Beyond Usability." His tutorial expanded the themes he introduced in his Designing Pleasurable Products: An Introduction to the New Human Factors . In his book he identifies four levels of pleasure in products and services: physiological, social, psychological, and ideological. These levels are intended to provide a basis for moving beyond usability to usefulness and pleasure.
Given the widespread interest emerging in many conferences, companies, universities, and professions, it seems appropriate to take a moment to survey the emotional landscape of human-computer communication and interaction.
The study of emotions began thousands of years ago. The early Classical philosophers and dramatists, most notably Aristotle, considered what emotions exist and how artistic artifacts of words, images, or sounds, among other media, might affect viewers, listeners, or participants. Much more recently, French Encyclopedists in late-18th-century Europe carried out thorough taxonomies of the emotions and how they might be expressed through facial gestures, hand gestures, and other non-verbal signs. Similiar studies have been made in India and China.
More recent theories of emotion are abundant. Several sources, which include the James-Lange theory (1884), John Broadus Watson’s Behaviorism theory (1913), the Cannon-Bard theory (1915), and Sartre’s Phenomenological theory (1939), are detailed by Calhoun . Buck  presents the following typology of emotions in 2002:
- Biological emotions (such as arousal, reward-punishment, fear and anger, and love and bonding): based on specific neuro-chemical systems
- Social emotions (such as pride and guilt): based biologically on attachment
- Cognitive emotions (such as interest, boredom, and curiosity): based biologically on expectancy
- Moral emotions: based on a combination of social attachment and expectancy
Concerning more practical approaches to analyzing emotions, in recent times one can point to the work of Paul Ekman, Professor of Psychology, Department of Psychiatry, University of California Medical School, San Francisco, who has studied facial-gesture details for more than 30 years and what they betray of emotions, making it, in his view, easy to determine if one is telling the truth or lying.
In the world of computer-based communication and interaction, it has been an unstated human-factors-oriented assumption for most of CHI’s history that communication and interaction take place in a somewhat neutral emotional state. Occasionally research on interaction is conducted in control rooms under alarm conditions, medical emergency-response systems, or fighter-aircraft cockpits during combat. Most consumer and professional systems have assumed that developers, and users, exhibit the calm demeanor of the proverbial lab scientist in a white coat and pocket protector.
Nevertheless, researchers like Prof. Rosalind Picard of Media Lab’s Affective Computing Research Group at MIT, have, for years, undertaken studies of human emotions, sensing human affect signals, recognizing patterns of affective expression, understanding and modeling emotional experience, synthesizing emotions in machines, affective computing applications, user interfaces with affective computers, affective communication, and affective wearable computers . A special issue of Interacting with Computers  features three papers, all of which she co-wrote. One of the more stimulating titles is "Frustrating the User on Purpose: A Step Toward Building an Affective Computer." This approach seems to fly in the face of conventional usability theory, which typically seeks to avoid troubling the user.
Meanwhile, in another realm, as the ACM SIGGRAPH conferences ably demonstrate each year at their justly famous theater presentations, SIGGRAPH animators are coming closer and closer to completely synthesized actors and actresses who can move whatever virtual facial muscles necessary to convey innumerable and perhaps unnamable nuances of emotions. We are probably only a few years away from Arnold Schwarzenegger, whatever his political future, licensing his detailed body-base measurements and other attributes to provide a completely synthesized producer- and/or consumer-activated Terminator to be used for whatever purpose market forces permit.
Today, consumer product and service developersfrom mobile devices to music services to vehicles to wearablesall "just wanna have fun," or at least think that consumers who are having fun, or being amused, will be more persistent purchasers, more productive practitioners, and more satisfied, stickier, word-of-mouth testifiers, returning for more, for ever more.
In my lectures and tutorials of about 15 to 20 years ago, I predicted the stage of user-interface development (UID) that would see genres of UID emerge, just as they have in literature and cinema. I foresaw that we might have funny UIs, sad UIs, gentle UIs, matter-of-fact UIs, even in-your-face UIs, depending on user preferences, moods, and circumstances. In those days, I likened the development of UIs to movie production and cautioned developers to hire professionals, not to let application programmers, or even user-interface designers, write the jokes for extremely funny menu hierarchies.
Now it seems the times have caught up with us; the moment is at hand to open the door widely to emotion as part of the user experience. Before we get too giggly with our delight in all this commotion, let’s pause for a moment and consider some issues that may challenge our cognitive, aesthetic, and ethical abilities and skills in the coming years.
The moment is at hand to open the door to emotion as part of the user experience.
Among other questions already being considered and others waiting for investigation are the following:
- What is the relationship of culture to emotion and particularly to pleasure? How should our product and service UIs respond?
- Considering pleasure, to what extent is Maslow’s (see ) hierarchy of needs (physiological, safety, belonging and love, self-esteem, and self-actualization) actually culture-biased and perhaps not as universal as some might consider?
- Will manufacturers move toward a simple, lowest-common-denominator, short-term pleasure-principle, guaranteeing that our products and services will have all the edifying substance of the latest television reality shows? If so, how fast will this movement take place? What viable alternatives are there?
- What exactly do we mean by fun? Are there 7±2 dimensions of fun, as there are for persuasion, trust, and intelligence? What are these dimensions? How can we develop templates for these dimensions? Keep in mind that according to some French intellectuals, there are exactly 18 levels of humor, Paulos  has developed theories of the geometry of humor, and books like that of Prochnow  have, for decades, guided presentations of humor, in this case listing almost 2,000 jokes, jests, wisecracks, epigrams, and amusing definitions.
- How can one reliably assess, measure, and compare pleasure, as opposed to the usability-oriented term "satisfaction"?
- Will all products and services eventually become rated according to the "pleasure" they provide, as Amazon.com and others now guide book, CD, DVD, and other media purchases according to statistics of who "liked" the item, or who "benefited" from it, as opposed to what critics think, or who purchased the selection?
- How can emotional involvement of the user, as opposed to moods or sentiments, consciously focused by the designer, improve attention, memory, performance, and assessment? How is this approach similar to or different from the practices of advertising in marketing?
- Ekman  proposed 7±2 basic emotions expressed through facial cues: anger, disgust, fear, happiness, sadness, and surprise. How do these map to particular states, or attributes of user-interface components (metaphors, mental models, navigation, interaction, appearance)? Can one construct a mapping tool or a database of patterns or case studies? Note that other researchers [e.g., 18] refer to 24 emotions developed by other theorists.
- Are visual, musical, verbal, or physical cues to emotions more effective for certain groups of people under certain conditions? What are the variables and the conditions?
- Which emotions are appropriate or ethical for user-interface designers to manipulate? Should some kind of professional ethical review committee exist to determine what is appropriate?
- To what extent should computers track the emotional states (i.e., appropriate data) of users? To what extent is it appropriate or ethical for the manufacturer or computer to inform the user that these data are being tracked or that emotions are being manipulated? Consider the uproar decades ago over subliminal advertising that placed consciously unobservable messages (e.g., "buy more coffee") into communications.
- How accurate must emotion-recognition (like speech-recognition) be in order to be useful? Legally monitored or sanctioned?
- Will emotional communities arise that wish to interact according to particular palettes of emotions?
- What are the most appropriate metaphors of emotion? For example, anger is sometimes described in terms of its being a hot fluid in a container: "blood boils," "inflammatory remarks."
Which emotions are appropriate or ethical for user-interface designers to manipulate?
Many resources, such as theories of emotion, of humor, of fun, abound, to keep the interested researcher and practitioner busy. Theories must eventually translate into practice.
The struggle some analysts and designers will face seems likely to be to persuade business managers, marketers, and heads of product and service development to timely and efficiently address some of the more exotic, ethical issues. An argument oriented to return on investment (ROI) seems in order. Who will step up to the plate to prepare it?
With or without such an examination, it seems likely that the emotional temperature of the design community has been raised. Soon we’ll be sizzling. Are you ready for fun?
1. Brave, Scott, and Nass, Clifford. Emotion in human-computer interaction. In Jacko, Julie A., and Sears, Andrew, Handbook of Human-Computer Interaction, Lawrence Elrbaum, Mahwah, NJ, 2003. The chapter contains an excellent bibliography.
9. Gage, Debbie and McCormick, John. Prada: the Science of Desire (December 16, 2002). Available at www.baselinemag.com/ article2/0,3959,772305,00.asp. The article is a case study of Prada’s use of advanced technology in its first Epicenter store: transparent doors on fitting rooms that are supposed to become opaque when customers step in.
Aaron Marcus, President
Aaron Marcus and Associates, Inc. (AM+A)
I acknowledge the assistance of AM+A Design/Analysis Intern Carmen Doerr, of Germany, who conducted initial research during June-July 2003 on the study of emotions in design, product and service development, culture, and user-interface development. I am also indebted to Brave and Nass  for several of the issues they raise in their chapter.
©2003 ACM 1072-5220/03/1100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2003 ACM, Inc.