XVIII.5 September + October 2011
Page: 24
Digital Citation

Emotion and motion

Katherine Isbister

back to top 

At the end of a day of writing, emailing, and conducting more meetings than I'd like via video-conference, my back hurts, my eyes are glazed, and I feel drained and dissociated. If I calculate hours of face time, I must admit that my computer is my closest companion. Unfortunately, I feel I'm not my best self during the many hours we share; I'm hunched over, arms gathered closely in front of my body, staring silently ahead, movement limited to, for the most part, poking and prodding with my fingertips. I leave our time together feeling drained and dissatisfied; my intellect has been engaged, but my body has been neglected—maybe even mildly abused.

It seems I am not alone in this dysfunctional relationship. Everyone I know has similar problems with their computers. Recently, this has become a topic of health research, which shows that engaging with computers all day, in the way we do, is bad for us and cannot be erased by a daily hour on the treadmill [1]. Computers once required this dumbing down of our physical capacities as the price of engagement. Like the monks who suffered while hand-copying manuscripts hour after hour before the printing press was invented, we simply had to accept the inevitable aches and pains that came with the use of a mouse, keyboard, and monitor. But with today's powerful processors and commercial-grade sensors and input devices, this is no longer true. It is time for a fundamental rethinking of our physical relationship to everyday computing.

There are a number of converging efforts in the HCI community to explore and rethink our physical interaction with computers. (Eva Hornecker's recent piece in this magazine gives a nice overview of some of these strands [2].) Making a transition this radical will require research from many angles. Here, I'll introduce a stance and a line of research that is focused squarely on the quality of the experience we're having physically when we interact with computers—how we feel as we move to interact with the technology. I believe this approach is essential to help us navigate the vast possibility space of body-based interaction. Without it, we run the risk of recreating dissatisfying and unhealthy modes of engagement with computers, forcing people into awkward and unsustainable—and still worse, joyless and demoralizing—postures and motions all day long (like the Taylorized factory workers of the early assembly lines).

Despite some interesting experiments with using movement in interfaces to convey emotion (for example, Sundström et al. [3]), thus far there have been no sustained and systematic attempts in our community to develop types of gestural- and movement-based interaction that thoroughly integrate considerations of moment-to-moment feel.

With the recent commercial success of movement-based gaming consoles and the growing distribution of multi-touch tablets and tables, the era of movement-based interaction is here. This is a window of opportunity for influencing emerging standards to set a more sustainable and satisfying course for everyday movement-based interaction.

back to top  Learning from Games and Game Designers

Game developers have led the vanguard in commercial distribution of movement-based interaction platforms. There have been movement-based games in arcades and at home for many years now, and at present all three major home game consoles come equipped with movement sensors. The tremendous success of Nintendo's Wii platform convinced game companies that movement was something players valued and wanted more of, and that it was a way to gain an even broader audience for gaming. The industry has radically embraced movement-based interaction and is racing to evolve best practices for using movement.

Of course, best practices are defined somewhat differently for games than for productivity applications. The core definition of success for a game is that a person enjoys the game enough to keep playing it, rather than that it helps the person to achieve a task. This puts a great deal of emphasis on how the experience feels moment-to-moment. As one game designer puts it:

Focus on second-to-second play first. Nail it. Move on to minute-to-minute, then session-to-session, then day-to-day, then month-to-month (and so on). If your second-to-second play doesn't work, nothing else matters. Along these lines, if your day-to-day fails, no one will care about month-to-month, either [4].

Game designers use nested sets of goals and discoveries that keep players cognitively challenged as one powerful way to maintain engagement. But another important way is the crafting of compelling "game feel" [5]. Game feel is the moment-by-moment sensation of the system's response to the player—the kinesthetic qualities of the experience created by coupling with the input device and seeing what happens in the game as a result. Designers and players talk about how certain games have really great game feel, something valuable in its own right.

A colleague and I wrote a paper for CHI 2009 that introduced these notions, which we framed as "suppleness," to the HCI community [6]. In this paper, we argued that designers must turn their attention to the qualities of coupling with the system—of the moment-to-moment interaction—to create more supple experiences and, along the way, invent a more supple design and evaluation process that can reliably produce these kinds of end-use experiences.

Since then, I've come to realize that this is an excellent organizing principle for approaching movement-based interaction design. What if we place considerable design emphasis on how a movement makes a person feel in her body—as she performs it, and at the end of an extended interaction with the system? This led to my lab group's efforts to pin down essential qualities of the feel of movement-based interaction, toward generating useful and well-grounded design practices and principles.

back to top  Chasing the Connection Between Emotion and Movement in the Lab

We have been using two complementary approaches to glean a better understanding of how to design movements to evoke desired feelings for players/users. The first approach is to closely read the work of commercial game designers. We examined movement mechanics from a broad range of Wii games categorizing motion qualities toward generating useful design patterns [7] When working within the same target design space, some designers craft very pleasurable and engaging movements, and some do not. For example, one dancing game might produce exhilaration and joy, the other tension and frustration, through the qualities of the movements that players have to perform over and over. This leads us to believe that articulating design principles and guidelines for movement feel should benefit game developers themselves, by helping them to codify and share their own successful techniques with one another.

We worked from the bottom up, looking closely at the movements, but also from the top down, looking for connections to known emotional effects of movement from psychology and communication research (see Isbister [8, 9] for more detail on some of the research findings we've found useful). We also coded the movements we observed using a subset of a taxonomy that has been used in past HCI work (for example, Sundström [3]), Laban Movement Analysis, which is based on dance notation developed in the early 20th century.

In parallel, we have taken up the challenge of determining how to isolate and evaluate the impact of movement on how the player is feeling. Games are complex media, with many components (animation, sound, and reward system, among others) contributing to the overall effect. We have been making use of classic social science methodology to create experiments in which we can isolate and examine the influence of movement on players.

In our first study, we had participants play Wii games that we precharacterized as low-, medium-, or high-movement games [10]. We rigged a simple self-report system, in which the players used classroom clicker devices to quickly rate how they were feeling after each round, using a two-dimensional model of emotion used for quite some time in psychology (arousal: low versus high energy, and valence: positive versus negative). We also videotaped the sessions and had multiple coders score the amount of player movement to verify that our low/medium/high categorizations of the games were in fact accurate (Figure 1 is an example of the kind of video record we analyzed). We found that sheer volume of movement led to higher self-reported arousal level but did not change the valence of the players' emotional states (they reported positive feelings in all three conditions).

Concerned that this approach did not fully isolate movement as a factor (the games represented different genres and had many other differences), we replicated this work using our own game prototype [11]. Wriggle (see Figure 2) is a simple movement-based game that can be played either with hat-mounted Wiimotes or using key commands. This allowed us to completely isolate the impact of movement on how players reported their emotional states, as everything about the game other than the input modality remained the same during play sessions. Our results paralleled those from the first study: Vigorous movement led to reports of higher arousal but not of more positive emotional valence. We also found an interesting but not statistically significant result: Players reported a greater sense of social connectedness when playing in the movement condition.

At this point, we believe we have a promising approach to understanding qualities of movement that affect how players feel. This approach combines the close examination of successful and unsuccessful designed movement mechanics with strategic prototyping and experimentation to isolate moment-to-moment effects of particular movement characteristics that have potential value in many interaction contexts. We are using this combined methodology in several projects in the lab, across a range of movement platforms.

back to top  Demonstrating the Value of Feel Outside Entertainment Gaming

Recognizing that some colleagues are dubious of the value of caring about movement feel outside of pure entertainment, we have also begun to demonstrate the value of considering movement feel in productivity/non-gaming contexts. Our lab is a partner in the NYU Games for Learning Institute, which focuses on discovering and validating design patterns for building effective learning games. As part of this research, we have begun looking at whether and how the feel of movement mechanics can actually contribute to learning.

Recent research has demonstrated that striking "high power" poses for just two minutes leads to changes in peoples' biochemistry (higher testosterone levels, lower cortisone levels) as well as greater risk tolerance when completing a sample task directly afterward [12]. In essence, this research supports the folk maxim that one should pose as confident in order to feel confident. What if we could leverage this fascinating phenomenon to combat math anxiety?

We have built a research prototype math game called Scoop, which allows students to practice placing fractions on a number line that they extend and contract through broad arm movements (see Figures 3a and 3b). Working in consultation with the authors of the power-pose study, we designed two versions of the game that differ only in the quality of the movement they elicit from the player. One version requires the player to take expansive (high power) positions to catch fractions, and the other keeps the player in a contracted position with arms tucked into the body (low power). We have just completed a pilot study testing this game with middle school girls, to see if there is variation in their self-reported math confidence levels after playing the two different versions of the game. Should we see promising results, we plan to develop an extended version of the power pose game as part of an innovative math-curriculum pilot program in New York City.

This is one limited example of the potential of movement to enhance task performance through feel. Studies have shown that being in a mildly positive mood enhances creativity [13]; in a similar vein, we believe that the feel of movement can be used to produce a range of performance- and attitude-enhancing effects on task-based system interaction.

back to top  Moving Forward

When the iPhone first came out, a game-designer colleague made an intriguing observation. She noted that the gesture used to wake the phone from its sleep mode was a caress—like stroking a kitten, really—and wondered whether this contributed to the affection people seemed to instantly have for the device. (So much nicer than tap, tap, tapping.) Please understand, I don't want to turn computers into surrogate kittens; I simply want us to have the pleasure of engaging our bodies in activities like caresses and other subtle, expressive, and satisfying motions as we move through our days.

If I must spend hours every day dwelling in the digital realm, I'd like to bring my body into play. Instead of feeling glazed and dissociated, I'd like to feel the way I do after a day at the ocean: tired in a good way, and grounded in my physical self. Could email sorting feel more like gardening or doing tai chi? Could having a meeting online feel more like cooking or playing a sport than sitting and chatting through a small window at a prison's visiting center? I want to bring my body and its pleasures and capacities back into the mix of doing my daily work. Maybe this could even reverse some of the alarming health trends that inactivity has wrought.

As we move deeper into the era of cheap and ubiquitous sensors, it will take the concerted effort of many very clever people to wisely reshape the ways in which we physically engage with computers. We'll almost certainly all be using movements to interact with computers in some way, but there's no guarantee that these will feel good unless we set this as one of our aims. I hope this article flushes other like-minded and hopeful individuals out of the woodwork, so that we can make this happen together. (Note: If you want to see me speak enthusiastically about this topic, with supporting video examples, see [14]).

back to top  Acknowledgements

Thanks to the Wriggle and Scoop teams for their hard work on game production and research (see http://socialgamelab.bxmc.poly.edu). Thanks to Kia Höök for being an inspiration and a marvelous and generous collaborator. Thanks to Microsoft for support of the Scoop project through a generous grant to the NYU Games for Learning Institute.

back to top  References

1. Rabin, R.C. The hazards of the couch. New York Times. Jan. 12, 2011; http://well.blogs.nytimes.com/2011/01/12/the-hazards-of-the-couch/?scp=1&sq=inactivity%20unhealthy&st=cse

2. Hornecker, E. The role of physicality in tangible and embodied interactions. interactions 18, 2 (March + April 2011), 19-23.

3. Sundström, p., Ståhl, A., and Höök, K. eMoto—Affectively involving both body and mind. CHI '05 Extended Abstracts. ACM, New York, 2005.

4. Brathwaite, B. Design truth 1. Applied Game Design. 2011; http://bbrathwaite.wordpress.com/2011/01/13/design-truth-1/

5. Swink, S. Game Feel: A Game Designer's Guide to Virtual Sensation. Morgan Kaufmann, Burlington, MA, 2008.

6. Isbister, K. and Höök, K. On being supple: In search of rigor without rigidity in meeting new design and evaluation challenges for HCI practitioners. Proc. of CHI '09 (Boston, MA, April 4-9). ACM, New York, 2009.

7. Isbister, K. and DiMauro, C. Waggling the form baton: Analyzing body-movement-based design patterns in Nintendo Wii games, toward innovation of new possibilities for social and emotional experience. In Whole Body Interaction. D. England, ed. Springer, London, 2011.

8. Isbister, K. and Schaffer, N., eds. Game Usability: Advice from the Experts for Advancing the Player Experience. Morgan Kaufmann, San Francisco, 2008.

9. Isbister, K. Enabling social play: A framework for design and evaluation. In Evaluating User Experiences in Games: Concepts and Methods. R. Bernhaupt, ed. Springer, London, 2010.

10. Isbister, K., Rao, R., Schwekendiek, U., Hayward, E., and Lidasan, J. Is more movement better? A controlled comparison of movement-based games. poster presentation at Foundations of Digital Games (Bordeaux, France). 2011.

11. Isbister, K., Schwekendiek, U. and Frye, J. Wriggle: An exploration of emotional and social effects of movement. Work in progress presentation at CHI 2011 (Vancouver, Canada).

12. Carney, D.R., cuddy, A.J.C., and Yap, A.J. power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science 21, 10 (Oct. 2010), 1363-1368.

13. Isen, A.M., Daubman, K.A., and Nowicki, G.P. positive affect facilitates creative problem solving. Journal of Personality and Social Psychology 52, 6 (June 1987), 1122-1131.

14. Isbister, K. Emotion and Motion, Google Tech talk, March 4, 2011; http://bit.ly/iUJb9s

back to top  Author

Katherine Isbister is an associate professor jointly appointed in the NYU-Poly computer science department and the NYU Game center (http://gamecenter.nyu.edu/). She is research director of the Game Innovation Lab at NYU-Poly (http://gil.cite.poly.edu). Her research focuses on understanding and enhancing the social and emotional expressive range of games and other interactive systems.

back to top  Figures

F1Figure 1. Video record of movement-based games study

F2Figure 2. Wriggle, a simple movement-based game.

F3Figure 3. Scoop, a movement-based game for earning math.

back to top 

©2011 ACM  1072-5220/11/0900  $10.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2011 ACM, Inc.

Post Comment

No Comments Found