Exhibit X

Dream: Mixed Reality Technologies for Live Performance

When the Covid-19 pandemic closed London's theaters, digital studio Marshmallow Laser Feast (MLF) was already working on Dream, a mixed-reality live performance intended to bring together live and online audiences. Co-developed with the Royal Shakespeare Company, the Manchester International Festival, and the Philharmonia Orchestra, Dream's focus on the integration of live performance and immersive technologies has taken on a new significance since the closure of so many performance venues around the world. It was funded by the U.K. government's Audience of the Future program, which has brought together creative, industrial, and academic institutions to find new ways to engage audiences using immersive technologies.

Rather than re-create A Midsummer's Night Dream, the team mined the original text for any mention of plants, flowers, trees, or creatures. These references were then used to create a lush, immersive forest world within Epic's Unreal game engine. As with many of MLF's other works, this play revalues the natural world by revealing unseen and unknown presences in the forest to (human) audiences.

Actors wear motion-capture suits, which control the movement of the characters in real time within the Unreal game engine.

Tree models were generated as particle systems in Houdini software and brought into the Unreal game engine to create Shakespeare's forest.

The character Peaseblossom was designed with a root-growing algorithm.

The character Moth is composed of a flock of swarming moths.

Actors wear motion-capture suits, which control the movement of the characters in real time within the Unreal game engine.

Camera tracking is used to animate the character Mustardseed's face.

Actors perform in a Vicon motion-capture studio setup.

Actors travel through the real-time forest environment, which includes weather systems.

The mischievous forest sprites in the play have been modeled from elements of the forest's ecosystems. Tapping into the magical animism in Shakespeare's original work, the lead character Puck emerges from a pile of rocks, Peaseblossom is a woodbine plant come to life, and a flock of moths assembles to form the character Moth.

The show's performers control the animated characters by wearing motion-capture suits that track and translate the movements of their bodies into the virtual forest. Part of the score is generated live from the performers' bodies by the software tool Gestrument.

The live performances take place within a motion-capture studio, where large LED screens display the forest world, helping performers orient themselves and move. Audience members watch the livestream generated by the game engine; at certain points, they are invited to interact with the forest environment. Using a Web browser, they can shoot illuminated fireflies into the scenes that are rendered and displayed in real time, allowing the performers to follow and interact with them.

This play revalues the natural world by revealing unseen and unknown presences in the forest.

The Covid-19 pandemic has radically reshaped the relationships between culture producers and audiences for the foreseeable future. This project demonstrates new configurations of live performance and immersive technology that stay faithful to the dynamic and improvisational spirit of the theatrical arts while providing opportunities for geographically distributed audiences to participate from home.

  Acknowledgments

The project is funded within the Audience of the Future program by UK Research and Innovation through the Industrial Strategy Challenge Fund. Dream is generously supported by Miranda Curtis CMG, the Sidney E. Frank Foundation, Audrey Mandela, and Sean Phelan, and is an Epic MegaGrants recipient.

  Authors

Lara Houston is a researcher at the University of Sussex, exploring how creative practices can help people to imagine more sustainable futures, as part of the EU-funded CreaTures project. [email protected]

Marshmallow Laser Feast (MLF) is an experiential art collective working in the liminal space between art, technology, and the natural world. The collective creates specific visual languages that expand perception and inform our lived experiences. Their approach has earned them a reputation for creating the seemingly impossible—producing experiences that push boundaries, redefine expectations, and excite audiences worldwide.

  Footnotes

https://dream.online/home/

Copyright held by authors

The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.