ACM Interactions

At NYU ITP in fall of 2012, I created and taught a class focused on embodied play. We explored this question: “How can interactive design create powerful experiences by tapping into our emotions?"

First off, it is worthwhile to put ITP into context. The NYU ITP site states, “ITP is a two-year graduate program located in the Tisch School of the Arts whose mission is to explore the imaginative use of communications technologies—how they might augment, improve, and bring delight and art into people's lives.” It’s a dynamic environment that allows for open experimentation in pedagogy letting me create this one of a kind course based on my personal research at the NYU Poly Game Innovation Lab.

To get students started, we defined embodiment for the sake of the class as a digital experience engaging the whole body. Technologies encouraged for use within projects were Kinect, iOS, Unity, Lilypad, and Ardino. Next, we identified a few key traits of successful interactions. Researching embodied games revealed two main types of user interaction: anachronistic body-mapped control schemes and embodied control systems.

Anachronistic body mapping takes traditional input mythologies and reapplies them to the body, along the way short circuiting potentially new forms of interaction. When it comes to the human body, this kind of control is particularly insidious because it often comes at the expense of the user comfort and possible safety.

For example, in June of 2011, Tetsuya Mizuguchi released Child of Eden with this kind of control scheme. This game is beautifully rendered and is stunning for many other reasons; however, the control system could be improved. Players must stand with their arms held parallel to the floor for nearly the entire duration of the game. There is positively no way a user could play Child of Eden without significant physical discomfort using the Kinect alone. This kind of user experience seems like a stepping-stone toward a more robust design.

Instructions on playing Child of Eden showing an arrow control pad
mapped to hand tracking.

In sharp contrast, embodied control unites player and interface through a holistic map of the player’s complete body into the game space while also considering the affordances of lived experience. These are the types of designs leading towards the most organic kind of engagement. An exceptional embodied control system exists in Deepak Chopra’s Leela. It accounts for a user’s whole form in the game space while challenging past affordances accumulated within the human body during play. With a focus on opening and expanding the body, the interaction design is similar to the style of movement experienced during a yoga class.

Screen capture from Deepak Chopra’s Leela showing the breath
control level.

With this understanding, students began crafting experiences of their own. Design was approached as an evolving process. We went through the following steps: conceptual development, paper interfaces, testing, iteration of the interaction models, an optional round of testing, creating digital prototypes, and a final round of testing to gauge the success of the evolution of the project. Since emotional states are not always easily observed, 3rd-wave testing strategies proved critical tools for scoring user engagement in these kinds of experiences. Students were encouraged to use open-ended questions, talking aloud, video capture, abstract shapes, and self-reporting during their evaluation processes.

Presented here are a few the most successful projects:

Amelia Hancock: Birdw@tcher

Birdw@tcher plays with tensions that occur between city and wilderness settings. This new media interface whimsically transforms an iPhone into binoculars. Opening a “window” onto a mountainous 3-D space, users traverse the virtual and real worlds simultaneously. Through exploring, it is possible to spot hidden songbirds. As players become immersed in the experience, it begins to evoke a sense of meditation and reverie.

User testing: While testing, Amelia found that many users desired a balance between urban and natural environments. Players suggested enjoying a sense of discovery and found bird watching a surprisingly interesting and fun activity in this context. Multiple rounds of testing helped her to resolve issues of bird placement and user expectations.

Future plans: Currently, the environment that the user explores is limited and fixed. Next, she would like to translate this concept into a GPS-based augmented reality app for children, where they are able to both create their own bird drawings, as well as search for others, in real time and space.

James Borda: The Dragon

The Dragon is a lighthearted experience that incorporates both the body and breath of a player into a game. Transforming into a living, breathing dragon, the user is tracked via the Microsoft Kinect while wearing a headset microphone as they soar around a fantasy world. The installation immediately evoked a sense of joy and childlike abandon in most players. Adults showed a hesitancy to play this game, while children between 8 and 12 immediately jumped into the experience. However, during the ITP end-of-year show, adults brave enough to play visibly enjoyed themselves.

User testing: There is a broad range of natural ability to map one's gestures to the actions of the avatar onscreen. Some people were able to control the dragon very smoothly and intuitively, and others found it very difficult. The most successful group was children ages 8 to 12.

Future goals: James plans to expand the experience into an actual game with multiple levels and goals.

Phan Visutyothapibal: Sound Cubes

Sound Cubes is a multiplayer musical instrument allowing players to work together to create a musical composition. The game begins with six cubes moving up and down on the screen. Each cube contains a sound that complements the other cubes harmonically. Players are represented by circles on screen and are motion tracked via a Microsoft Kinect. When they move into the spaces of a cube, it triggers a sound. By working together, players create cords and melodies. Sound Cubes plays with affect in a more subtle way than the other pieces previously mentioned in this post. This instrument creates a very emergent social dynamic that gives players a sense of mutual collaboration and group involvement. Often users had to squat or jump to catch the animated cubes, which heightened the physical and emotional engagement experienced during play.

User testing insights: Phan explored a range of different interfaces before he finally settled on the animated cubes. His first impulse was to lay the notes out similarly to how they are seen on a piano keyboard. This approach proved difficult to play and led to the second design iteration.

Future goals: Phan ultimately wants this to be an educational tool for teaching music to children.

Without a doubt, many of the other interfaces created in the class were also compelling but these three projects exemplified the tight coupling between the user interaction and the physical body that tends to create the most enjoyable embodied experiences. Teaching a class around embodiment benefited from exploring an iterative design process that allows students a space to rapidly prototype ideas before producing them. Also, 3rd-wave HCI techniques gave students a toolset to quantify their experiences in new ways that particularly resonated with the designers and artists found at ITP. Hopefully, the lessons learned with this class will benefit and enrich the students’ design practices henceforth.

Phoenix Perry