Steve Benford, Gabriella Giannachi
There is a fascinating and potentially deeply productive relationship between interaction design, theater, and performance. On the one hand, interaction designers are increasingly involved with artistic experiences in which participants perform with computers to express themselves or to engage in a cultural experience. On the other hand, the spread of computers into public settings means that our everyday interactions become ever more performative in the sense that they are witnessed by others nearby , who often embrace roles in these interactions . In either case, it is illuminating to view interaction as performance, an argument that lies at the heart of our new book, Performing Mixed Reality, published by MIT Press . Our overall goal is to lay the foundations for a “dramaturgy of performance” by establishing a framework of conceptsa language, if you liketo help express the different ways in which computers can be embedded into performative experiences. We intend this framework to guide practitioners and researchers who are entering the field of artistic, performance, and cultural applications of computing. However, we also aim to stimulate wider thinking in HCI in general around the changing nature of the extended user experience and the new challenges this raises.
Our first task has been to document how a new generation of artists has been at the forefront of establishing an emerging theatrical genre that we called mixed-reality performance. This name is intended to convey two key ideas: the creation of experiences that mix real and virtual worlds in rich and complex ways; and the combination of live performance by actors or participants with interactive digital media. The net result is a complex, hybrid form of user experience in which participants employ diverse interfaces in multiple settings, but as part of a coherent overall whole. We chart a series of landmark mixed-reality performances by various artists, most notably the U.K. theater group Blast Theory, who have explored and defined this new genre.
Based on the first Gulf War, Desert Rain (1997) provided a compelling example of mixing real and virtual worlds to create a hybrid theatrical set. This involved the use of a series of “rain curtains,” individual projection screens made of a fine water spray through which participants and actors could pass. Ethnographic studies revealed how the artists carefully orchestrated the experience to create key climatic moments in the performance.
Can You See Me Now? (2001) extended hybridity to encompass city streets. Online players were chased through a virtual model of a city by actors who, armed with handheld computers with GPS, had to run through actual streets in order to catch them, and who also streamed live audio so that online players could tune in to their experience (Figure 1). In this case, our studies revealed how the experience needed to accommodate so-called seams  in the underlying technical infrastructure, such as partial coverage of GPS and wireless networking.
Uncle Roy All Around You (2003) also connected an online virtual world to city streets, but it focused on how a performance might exploit the ambiguity of relationships between street players, online players, and public bystanders. Street players trawled the streets in search of the mysterious Uncle Roy, guided by location-based clues from the game, but also by online players who could track their progress and who were in possession of useful information, such as the location of Uncle Roy’s office. As the experience unfolded, the street players were invited to engage with props, locations, and actors in the city, for example, by entering Uncle Roy’s physical office and getting into a waiting car. Ethnographic studies revealed the significant challenges of orchestrating a distributed mobile experience on city streets.
Rider Spoke (2007) further explored the nature of mobile engagement with the city, in this case by asking participants to draw on locations as inspiration for personal stories. In a deliberately isolating and contemplative experience, cyclists explored a city at night, recording a series of stories, leaving them at key locations, and then listening to the stories of others (Figure 2).
A text-messaging adventure game for mobile phones, Day of the Figurines (2006), took a quite different tack, exploring complex temporal relationships between real and virtual worlds. The experience was designed to be slow, with 24 hours of a narrative set in a fictional town unfolding over 24 days in the lives of the players, who sent just a few text messages each day. In this case, our ethnographic studies revealed the challenges of managing highly episodic engagement, especially how players had to account for interruptions and changing patterns of phone use with family, friends, and colleagues.
More than 10 years after Desert Rain, Ulrike and Eamon Compliant (2009) was a challenging ambulatory experience in which participants explored the hidden corners of a city while also receiving a series of telephone calls that described the lives of one of two terrorists, Ulrike Meinhoff and Eamon Collins. Instructions demanded increasing compliance from participants, who were eventually led to an interrogation room where they experienced a one-on-one interview with an actor. Our studies focused on the multilayered nature of the instructions, which, although down a “thin channel,” managed to address the four different aspects of location, sequence of actions, public comportment, and relationship to the content.
In addition to this series of works by Blast Theory, our book also documents works by other creative practitioners, such as explorations of biosensing and thrill by Brendan Walker in his Thrill Laboratory, the Anywhere mobile city walk by Willi Dorner, and the Savannah mobile-learning game by Futurelab, in association with the BBC and HP.
Studying Mixed-Reality Performance as HCI
Beyond being landmark examples of an emerging form of artistic practice, these various mixed-reality performances also provide vehicles for research. HCI researchers have been actively involved in their design and implementation, often touring with them through the first few performances as they visit different venues and helping refine them until they are stable enough to continue their touring life in their final form. Not only has this involved developing interfaces and supporting technologies for the participants, but it has also often involved creating authoring and orchestration tools for the artists. A second role for HCI researchers has been to study the participants’ experiencesand here we have turned to ethnography to produce “thick descriptions” of experience from participants’ points of viewas well as to focus on the artists’ rationale and the work required to manage live performance from behind the scenes.
Beyond feeding back into further iterations of the experiences and underlying technologies, these studies have informed theoretical work through a series of conceptual frameworks that generalize key aspects of performative interaction for the wider HCI audience. Notable examples include:
- informing discussions of the potential benefits of ambiguity in designing interfaces that provoke reflection and interpretation ;
- contributing to the notion of “seamful design,” in which evident seams in the underlying technology infrastructure, for example, gaps in the coverage and accuracy of wireless communications and sensing technologies such as GPS, become useful resources within an experience ;
- illustrating approaches to designing the spectator experiences; that is, configuring different aspects of a performer’s manipulations of an interface and their subsequent effects to become more or less visible to spectators depending on the situation and the desire to create interfaces that might be thought of as secretive, expressive, magical, or suspenseful ; and
- shaping discussions of how experiences are framed; that is, how various rituals and conventions convey to participants what belongs to an experience and what does not and what it might be appropriate to do. Mixed-reality performances in particular show how experiences that take place in public settings can involve an ambiguous framing that deliberately blurs the boundaries between the fictional and the real .
This relationship between creating performances, studying them as they tour, and then deriving broader principles of interaction operates as an iterative cycle, with these activities feeding into one another in various ways (Figure 3). In general terms, we characterize the resulting research process as being “artist led,” in that artists contribute original and creative ideas for new experiences that drive the process forward, and “in the wild,” in the sense that experiences are rapidly and iteratively deployed to public audiences and so can be studied under realistic use. Indeed, many of the projects lead to the somewhat unusual situation in which participants pay the researchers to take part by purchasing tickets.
Trajectories Through Mixed-Reality Performance
Our ultimate aim in the book is to articulate a broader conceptual framework for describing the core of mixed-reality performance. This is based on the idea that, in spite of their apparent diversity, the essence of the various performances that we analyze can be expressed using the common concept of trajectories. Our goal is to provide researchers with a sensitizing concept for grounding their own studies in this field, as well as to distill the craft knowledge that artists bring to such experiences into a form that is accessible to the wider HCI community. In other words, the goal is to create a kind of boundary object  that might sit at the intersection of HCI and theater and performance studies.
Our first observation is that as we saw above, mixed-reality performances tend to be inherently hybrid in their structure: combining multiple real and virtual worlds in various relationships; incorporating multilayered timescales; establishing various roles such as actors, orchestrators, participants, audiences, and bystanders; and finally, integrating diverse forms of interface into a single experience . As such, they defy easy classification in terms of current genres of experience or interaction paradigms. They are not simple theater, games, or installations, but rather combinations of these. Similarly, they are not just virtual reality, augmented reality, mixed reality, mobile computing, tangible computing, or ubiquitous computing, but again often a new combination thereof.
What they do tend to share in common, however, is the idea of taking the user on a distinctive journey that retains an overall coherence. We argue that such a journey can be described in terms of trajectories. Moreover, we argue that the structure of such experiences can be expressed in terms of three fundamental kinds of trajectory:
The canonical trajectory expresses the plan for the experience that is to be found in scripts, set designs, media and code design, stage management instructions, and so forth that the artists create beforehand. 6A given experience may involve multiple canonical trajectories, either for different kinds of participants (e.g., street players versus online players in Uncle Roy All Around You) or to represent a choice of journeys for a given participant (e.g., various pre-planned routes through the city for a street player). Thus, canonical trajectories may have branching (and rejoining) structures. Critically, canonical trajectories have to pass through key transitions, moments at which the coherence of the experience is at particular risk and so requires careful attention in design or orchestration, including beginnings, endings, handing over interfaces, role transitions, crossing seams, and managing access to physical resources.
The participant trajectory expresses the actual journey that a given individual makes through a particular instance of the performance. The very nature of interaction means that participant trajectories may diverge from pre-planned canonical trajectories as participants make individual choices; for example, choosing a route through the city. As our various studies revealed, processes of orchestration then become critical for steering the participant back to an appropriate canonical trajectory. The interweaving of multiple participant trajectoriesapproaching, crossing, and separatingexpresses the social aspects of mixed-reality performance, such as designing encounters, establishing moments of isolation, and managing pacing.
The historic trajectory expresses the ways in which an experience may be documented and subsequently retold. Our experience suggests that supporting reflection and retelling is a vital but often-neglected aspect of mixed-reality performance (and indeed of interactive experiences in general), and so we argue that the historic trajectory must be designed in from the beginning through opportunities to document the experience as it unfolds and then draw on this material in various ways afterward.
Having very briefly introduced these core concepts, we show in the final chapter of our book how they can be used to capture key facets of the mixed-reality performance such as hybridity, orchestration, seams, encounters, and transitions, providing an overarching framework to integrate many of the examples, studies, and concepts that have emerged from over a decade of research in the field.
UX as Trajectory
Up to now our focus has been on describing this emerging genre of mixed-reality performance. Our final question is whether our discussion, and especially our notion of trajectories, has wider relevance to other domains within HCI. We argue that it has. One obvious place to look is at more mainstream cultural experiences such as visits to museums, galleries, and theme parks, following locative “mediascapes,” or playing pervasive games. These share many of the characteristics of mixed-reality performance in terms of their blending of real and virtual, and live action and digital media. Indeed, one could argue that artists who work at the cutting edge of performance art are paving the way for future cultural and entertainment experiences. Some of our own recent work in theme parks backs up this argument. Theme parks are replete with canonical trajectories, from highly constrained journeys through individual rides, to the ways in which rides and other attractions are arranged within the broader settings of the park. In terms of participant trajectories, participants follow their own routes through these, and there is ample evidence of processes of orchestration by the park (e.g., digital signage showing queue times) and participants themselves (e.g., using mobile phones to keep in touch and coordinate meeting times). Finally, supporting the historic trajectory defines an important business for parks in the form of souvenirs and automated photo- and video-capture technologies. In one recent project, for example, we have explored how visitors can mix their own photos with those taken by the park and fellow group members to create personalized photo-story souvenirs from each ridea distinctive form of historic trajectory (Figure 4).
We suggest that trajectories are to be found elsewhere in HCI, too. Might it be instructive to consider future experiences of travel, healthcare, learning, or meetings in terms of various forms of trajectory? Can we consider how these experiences overlap with one another, moving from foreground to background, in terms of interleaved trajectories? Might we ultimately even consider lifelong trajectories of experience?
We conclude by noting that others have argued for an extended notion of UX that moves beyond conventional usability to encompass the emotional and aesthetic dimensions of experience . The ultimate argument in our book is that HCI needs to consider how a user experience extends across many interface spaces, timescales, and roles, and how thinking in terms of trajectories can help us capture the new design challenges that arise. Perhaps mixed-reality performances provide some early glimpses of our future user experiences.
This work was funded by the Engineering and Physical Sciences Research Council (EPSRC) through Horizon Digital Economy research (EP/G065802/1).
4. Chalmers, M. and Galani, A. Seamful interweaving: Heterogeneity in the theory and design of interactive systems. Proc. of the ACM Symposium on Designing interactive Systems (Cambridge, MA) ACM, New York, 2004, 243252.
6. Benford, S., Crabtree, A., Reeves, S., Flintham, M., Drozd, A., Sheridan, J., and Dix, A. The frame of the game: Blurring the boundary between fiction and reality in mobile experiences. Proc. of ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, 2006, 427436.
Steve Benford is Professor of Collaborative Computing in the School of Computer Science at the University of Nottingham.
Gabriella Giannachi is Professor of Performance and New Media and Director of the Centre for Intermedia in the Department of English at Exeter University.
©2012 ACM 1072-5220/12/0500 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.