What are the challenges and rewards facing designers developing digital games using alternative input devices? How can these games engage participants and give them unique interactive experiences beyond the casualness of mini-games? In an attempt to answer these questions, I will describe the design process behind two physical gaming installations I am currently developing.
First, some background. In 2005 I developed an interactive installation called MOVE in which participants were able to play through six different game modules: AVOID, CHASE, COLLECT, HIDE, THROW, and JUMP (see Figure 1a), each allowing players to prompt basic actions that avatars perform in video games. The installation used a floor projection setup where the presence and movements of participants were tracked using a camera (see Figure 1b). The projection on the floor was updated based on participants' actions and displayed graphics in an abstract language of simple 2-D shapes; a restrained color palette (white, grey, and red); and simplified physics simulations (collision, acceleration, friction, among others). These simple game modules could be experienced by only one player at a time. Moreover, they rapidly increased in difficulty, leaving players just a few minutes to experience each game module and giving them no chance to win (see Figure 1c).
The installation turned out to be quite successful, especially with a younger audience. Children between the age of six and 12 would often spend hours at a time trying to master the games, forming lines around the projection and replaying them over and over again. Why was this simple gameplay project so addictive for children accustomed to more sophisticated game experiences at home?
One main difference between MOVE's setup and traditional game consoles is that instead of using a gamepad or joystick to interact with the games, participants use their whole body. The various games would react to different gestures, such as running (CHASE), jumping (JUMP), waving one's arm (COLLECT), flapping both arms (THROW), and so on.
Besides the unusual experience of seeing a projected circle on the floor following you as you run around (CHASE), it seems that a closer mapping of the expected user input with the output generated on screen leads to a more intuitive, satisfying sensation. Take the unmitigated success of games such as Nintendo's Wii Tennis, where players wave their arms holding the Wiimote mimicking real-life tennis gestures in order to control their avatars on the virtual court.
The type of interaction required from participants is also simpler than the often complex button combination and hand-eye coordination required from traditional video games. Despite the fact that MOVE's game modules are very punishing and players last only a few minutes, they often feel that they failed because of poor body coordinationthe ability to jump, move, or react fast enough to prevent their on-screen avatar (usually a circle) from colliding with the incoming threat (usually another circle). This failure to act fast enough was viewed as something that could easily be overcome by repeated play, and failure led to less discouragement because moving one's body is a more natural act than orchestrating a correct combination of button pushes with fingers on a gamepad.
Another important element that differentiates MOVE from games played on a console or in an arcade is the placing of the screen. Because the projection is on the floor, players are more engaged with the game environment. Since they don't rely on the intermediary of a projected realistic avatar or silhouette mirroring their action on a vertical projection, there's less perceived distance between the space for input and output. And as players are not looking at an avatar reacting to their movements, their sense of identification is altered, and their interactioninstead of going through a two-step process, first acting and then seeing the result of the screenleads to a more direct, seamless type of interaction.
Finally, there's a performative quality in playing through the various modules and executing the different full-body gestures in front of an audience. While most adults would be uneasy and self-conscious, the games confer a sense of competition, pride, and drive to excel to younger players.
Even though MOVE ended up providing a very enjoyable, entertaining experience to most people who interacted with it, my initial ambition when designing it was to raise a certain consciousness about the combination and complexity of the various actions that participants accomplish when controlling their avatars in video games. I wanted to deconstruct basic avatar behavior by separating actions in individual game modules, and offer participants a way to perform those actions through a series of mini-games. The installation was a means to deconstruct a medium (video games) using the medium itself, a process that media theorist Erkki Huhtamo has described as "metacommentary" .
After MOVE, I became interested in the challenge of designing multiplayer, physical gaming installations that would give users a gameplay experience more complex than what MOVE provided. Single-player mini-games could offer only limited interactive experiences, even though they worked quite well within a physical gaming context.
The games could be learned very easily thanks to simple rules. Because they required intense physical movement, their duration was brief so as not to be too exhausting; because they were short, they allowed quick permutation of players, important in a gallery or mediaevent setting, which sometimes means large crowds.
Nonetheless, I was curious to see if it would be possible to give players a more complex gameplay experience through a physical gaming installation. Instead of providing a simple goal (avoid circle), I wanted to give players a combination of direct, short-term goals and indirect, long-term goals: what Eric Zimmerman and Katie Salen call "micro and macro goals" . Would it be possible for players to develop strategies by balancing priorities between micro and macro goals in a hectic, physical environment? Such a game would require a higher learning curve, thereby risking alienating casual or inexperienced gamers, and void the advantage that the physical setting and intuitive body input provides.
Longer gameplay would lead to deeper immersion for the players within the representational environment and could allow for a more complex depiction of system simulations, giving players more convincing and complex roles. Nonetheless, the attempt to depict a representational world could lead to a loss in the immediacy that the abstract shapes of MOVE afforded, which gave players a higher consciousness of their own actions as opposed to a feeling of immersion in a virtual world.
Creating a multiplayer game environment would allow emphasizing interaction between players, and creating interesting scenarios and tensions between collaboration and competition in the game, but at the risk of creating an overly confusing gaming environment for the players.
In Virtual Ground and Flying Machinetwo physical gaming installationsI attempted to address these issues.
After MOVE, I decided to develop a multiplayer physical game with a similar installation setup, a floor-based projection with graphics updated based on participants' movement tracked by a camera located on the ceiling and pointing down vertically to the projection.
This new installation, Virtual Ground, is a loose reference to the technical definition in which a virtual ground is part of an electrical circuit that is maintained in a state of equilibrium, usually through a process of negative feedback.
The first reason for using this title was my interest in developing a game around the analogy of electrical flow, in which a virtual electrical current represented by a line drawn between players would charge a particle which, in turn, would be used to gradually light up a grid-based surface projected on the floor.
At the beginning of the game, participants see a floor projection with a grid of 4x3 squares. Inside the grid, a small circle (particle) is rebounding within the boundaries of the grid. As participants enter the projected area, circles appear on the floor, following them and adjusting their movement to be always in the center of the participant's silhouette. A line appears between participants and adjusts itself dynamically according to their position (see Figure 2a). When the particle hits the line between participants, it gets "charged," and if it hits one of the borders of the grid, that border (corresponding to the width of one of the squares of the grid) gets charged as well. The goal of the game is to charge opposing borders of the grid in order to light up the rows between borders (Figure 2b). If the particle hits one of the circles of a participant while they are standing above a charged part of the grid, the grid turns dark again. The goal of the game is to charge the whole grid. This game can potentially accommodate as many players as the projection surface would allow, each new player connected by a line to two other players (Figure 2c). Once the grid is completed, a new grid with a higher resolution (8x6 squares) appears, offering a more difficult challenge (Figure 2d).
I assumed that in order to create a game with more complex game mechanics, it could be beneficial to keep the interaction elements at their most simple and intuitive so as to encourage an easy entry and low learning curve barrier for players, who could then focus on discovering the more complex rules of the game. After my experience with MOVE, I realized that among all the different interactive gestures I had asked participants to perform while playing the six different game modules, the one that was the most intuitive and easy to understand was simply to move on the projected surface and have a circle position itself underneath them. When participants moved, the circle would follow them and reposition itself with a slight delay underneath their feet. This simple relationship between player and visual graphic were used in the modules CHASE, HIDE, and AVOID, which were the games in MOVE that were the easiest for participants to understand without explanations.
Creating a game in which the only possible input from participants would be directional movement across a 2-D plane has led to successful gameplay mechanics in video games. The most famous example is maybe Pac-Man, an arcade game in which the player controls an animated mouth through a maze, avoiding ghosts and eating dots. A more sophisticated example using a 3-D space is Katamari Damacy, in which the player controls a small character pushing a sticky ball that increases in size as objects stick to it. The challenge resides in exploring the play area in search of objects to collect. Only objects smaller than the ball will stick to it, increasing the size of the ball and allowing players to collect increasingly bigger objects.
The second important reference to the concept of virtual ground is the idea of equilibrium, or negative feedback. In MOVE, the common gameplay mechanic behind each game module was that the longer players interacted with the game, the more the game increased in difficulty, eventually leading the player to lose. Ultimately, games were impossible to win, and were based on the typical setup experienced in early arcade games, where success was measured in high scoresin this case, how long players could last without losing. On the contrary, in Virtual Ground I decided to explore the opposite setup, offering participants a game environment in which they could never lose, where the game would regulate itself by increasing in difficulty based on player interaction, similar to the negative-feedback effect of a virtual ground. As more fields get colored, getting hit by a particle becomes more likely, and if hit, then the fields are set back to black, letting players try again.
Such a state has often been described as the holy grail for game designers; it is the closest association with the elusive concept of fun. A game is fun if a player achieves a state of flow between boredom and anxiety.
In his famous theory of flow, Mihaly Csikszentmihalyi describes a state where users are engaged in an activity where they are in complete controla state in between boredom and anxiety . Such a state has often been described as the holy grail for game designers; it is the closest association with the elusive concept of fun. A game is fun if a player achieves a state of flow between boredom and anxiety. In MOVE, this balance was achieved for only a limited time; players were overwhelmingly subjected to too much difficulty before losing.
To achieve a state of flow for casual players, one approach has been implementing what is called dynamic difficulty adjustment (DDA)a system behavior controlling the difficulty of the game and adjusting it based on player performance. One successful example is Wii Tennis, where a single player can face increasingly stronger opponents as they progress, and their results are monitored with a score system. If players start to lose too frequently, the system decreases the difficulty level by offering them weaker opponents. Players are less discouraged to continue and can improve their gameplay without feeling defeated.
In Virtual Ground, players who attempt to fill the grid with color have two main challenges to overcome: First, to position the line at such an angle to be able to charge two opposite lines to create a filled row; second, to avoid having their circle hit by the particle while standing above a charged area. The only possible penalty is having a row of color turn back to black if they get hit. That would simply force them to start over again to try to fill that row with color. This mechanic is not adjusting the difficulty of the game but is providing a self-regulating difficulty where players try to achieve a goal without ever losing. They are attempting to reach a goal, filling all the rows with color, and if they make a mistake, they simply fall slightly behind.
This approach should allow for an easier point of entry for casual players, while still offering deeper gameplay than MOVE's modules and increased difficulty at higher level for dedicated players. Thus, the game can be enjoyed either for a short period of time or for longer periods if one seeks to master the game.
This game was designed to offer players an interesting experience of tension between collaborating to fill the grid with color while at the same time struggling to maintain enough individual space to move around and avoid getting hit by the particle.
Flying Machine is a game currently in development designed exclusively for the Hybrid Playground platform . Hybrid Playground is an interactive installation project initiated by Clara Boj and Diego Diaz of Valencia, Spain, under their collective name Lalalab, and developed in collaboration with media artist and programmer Martin Nadal of Madrid. The concept behind the project is to combine mobile gaming and physical gaming within a playground to encourage interaction between children.
The installation consists of various sensors attached to playground devices such as swings, slides, merry-go-rounds, and spring riders that wirelessly send information about the children's movements to a server that in turn communicates with a PDA on which a digital game is unfolding using the physical interfaces of the playground as input devices.
This setup transforms a playground into a physical-digital interactive system in which a group of children play a videogame using physical actions and movements. The playground doesn't need any significant alteration except for installing sensors enclosed in boxes in order to detect various movements such as tilt, motion, and rotation. These boxes are quite compact and do not affect the handling of the various playground devices.
Lalalab designed a series of mini-games for Hybrid Playground in which teams of four players can play simultaneously. Each team has one leader, who handles the PDA with the digital game, and three players wearing identification bracelets. Each team has a specific color and an avatar in the digital game. These roles are interchangeable at any moment during gameplay. The players wearing the bracelets are tracked in the playground and perform the physical actions of the game. The fourth player, or leader, oversees the unfolding of the game on the PDA screen and acts as a guide to the other team members, giving them precise orders to undertake the actions that each game requires. Each element of the playground has various mini-games with different degrees of difficulty that players have to overcome. The movements of the children engaged with the physical devices of the playground generate the input required to control the avatars in the digital game.
Because the game system of Hybrid Playground is adaptable and allows for a wide range of possible game scenarios, Lalalab is interested in having game designers create original games for their platform. They asked me if I would be interested in designing a game exclusively for Hybrid Playground.
I became very interested in the challenge of designing a gameplay experience that would go beyond the successful mini-games. The feedback and experience they had testing those games for Hybrid Playground with children had been overwhelmingly positive. I felt that since their platform was successful with mini-games, the challenge of creating a game with a higher learning curve and more demanding gameplay was similar to the challenges I faced designing Virtual Ground.
My main inspiration for designing what eventually became Flying Machine was the work of Swiss artist Jean Tinguely, whose elaborate kinetic sculptures resembling useless, overly complicated machines, have fascinated me ever since I saw a retrospective as an eight-year-old growing up in Switzerland. The idea of controlling a virtual machine using elements of a playground seemed to be a possible direction for a game played by children between the age of six and ninethe main age group targeted by Lalalab for Hybrid Playground.
I decided on the design of a machine grounded on the floor, which a "crew" of children would have to "lift." Each child, controlling a different part of the machine, takes it through a coordinated effort to reach an altitude of a thousand meters, avoiding increasingly menacing obstacles along their way.
The machine consists of two balloons that have to be inflated for upward movement, a propeller for horizontal movement, three air cannons aimed at dissipating clouds, and a main cannon shooting at the bombs hidden behind the clouds (as shown in Figure 3a).
The children control the various parts of the machine using the playground elements as input devices. The machine's state and movement are represented on the PDA screen held and closely monitored by the "captain," who shouts orders to his crew: "left balloon, inflate faster!"; "propeller, go right", "air cannon, shoot!", and so forth (see Figure 3b and Figure 3c).
Every time a balloon collides with a bomb, the machine spins downward and declines in altitude, signaling to the children that it's time for them to switch roles, each now controlling a new part of the machine by moving to different elements in the playground, with another child taking the role of captain (Figure 3d).
The main challenge of the game is the careful coordination of all the different parts of the machine through vocal communication between the captain and its crew. Children activating the different sensors cannot see the consequence of their actions and thus have to rely entirely on the captain's orders. In turn, the captain must anticipate the movements of the machine, balloons being inflated to the point where they explode, clouds hiding bombs that need to be dispelled, bombs that need to be shot, then communicate effectively with timing to his/her crew the necessary actions they need to accomplish given the current on-screen situation (see Figure 3e).
To prevent the game from being overly difficult and frustrating for the children, I decided to implement a dynamic difficulty adjustment system similar to the one I designed for Virtual Ground.
This time, I found inspiration in the game and thesis paper of a USC student, Jenova Chen, who had written a paper titled "Flow in Games," in which he analyzes a game he designed . In his game, Flow, the player controls a creature through a 2-D top-down view as it evolves in a submarine environment, eating smaller organisms and avoiding bigger ones. As it successfully eats other organisms, the creature sinks in deeper waters, signaled by a darkening of the blue background representing the water environment. Chen was very interested in Csikszentmihalyi's flow theory, and decided to use a DDA system to control his game. He implemented a very simple strategy where the player, rather than being penalized if his creature got eaten by a bigger organism, would simply float back up to higher waters, regressing in a level with an opportunity to try again.
I decided to employ a similar strategy with Flying Machine, where instead of using a "health bar" feature common in video games to penalize players when they hit an obstacle, I would simply have the machine spiral down and lose altitude, giving the team time to switch controls and resume their ascension again.
In further iterations of the game, I would like to implement the following features: design of custom versions of machines based on a library of parts as well as the physical elements available in a given playground; power up collection as the machine is gaining altitude, to overcome more challenging obstacles; and enable multiple machines with their respective team to interact within a shared environment seen across multiple PDAs.
What makes working on these installations so fascinating and yet so challenging is the unique convergence of three different creative disciplines with very different sets of constraints: interaction design, interactive art, and game design. To be successful, these installations require a careful balancing between opposites in each one of these disciplines.
From an interaction design standpoint, these physical installations are attempting to move away from traditional input devices such as the mouse or gamepad and use alternative gestural interfaces. How does one balance the need for a required level of user-friendliness to make the installation accessible while at the same time introduce novel or at least unusual methods of interactions? Part of the answer can be found in replacing learned conventions, such as button clicks, with more natural body gestures closely mapped with the actions depicted within the environment represented in the game.
As interactive art, these installations are faced with the tension between providing a deconstructive discourse about the computational medium itself while at the same time engaging the audience by offering an immersive experience. Should the interface be a mirror or a window? Here again, the solution probably comes from giving participants an interface in which using their bodies in a natural way can free their mind from memorizing button combinations and let them reflect on the close relationship between their bodily actions and the designed system.
Finally, to return to the main question this article tries to address, can these physical installations provide more than casual game experiences? It remains to be seen since both projects are still a work in progress, but certainly one can draw inspiration from nondigital physical games, where the use of the body constrained to a fixed set of rules has often been successful at providing these games with deeper gameplay.
It appears that body mapping might be the strategic design approach to take in order to create a meaningful experience through these installations. Despite the fact that work in this direction has significant history, there isn't yet a set of established standards for designers and artists to follow, making the design of these projects all the more exciting to pursue.
4. Parts of the Flying Machine section were translated and adapted from "Hybrid Playground: Integración De Herramientas Y Estrategias De Juego Audiovisual Interactivo En Los Parques Infantiles," by Clara Boj and Diego Diaz.
Andrew Hieronymi's recent work focuses on the boundaries between games and art in physical environments. He has talked and exhibited internationally in art venues and media festivals, such as Ars Electronica, SIGGRAPH, FILE, IxDA Interaction, Microwavefest, and Futureplay among others. Hieronymi has received an M.F.A. from the Design | Media Arts department at the University of California, Los Angeles ('05), and a diploma of fine Arts from Ecole Supérieure des Beaux-Arts, Geneva, Switzerland ('98). He is a professor of interactive design and game development at the Savannah College of Art and Design in Savannah, Georgia.
Figure 3a. Flying Machine is controlled by inflating its balloons for upward movement, a propeller for horizontal movement, air cannons aimed at dissipating clouds, and a main cannon shooting at the bombs hidden behind the clouds.
Figure 3b. The machine's state and movement are represented on the PDA screen held and closely monitored by the "captain," who shouts orders to his crew: "left balloon, inflate faster!", "propeller, go right!", "air cannon, shoot!", and so forth.
©2009 ACM 1072-5220/09/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.