XX.5 September + October 2013
Page: 32
Digital Citation

The magical features of immersive audiovisual environments

Rodrigo Carvalho

back to top 

For centuries, artists, scientists, writers, designers, engineers, and others have explored audiovisual environments to create augmented realities and temporary autonomous spaces. The constant evolution of new media technologies provides the creators with better tools each day, allowing them to merge existing physical spaces with the virtual—the digital and imaginary ones. The magical features of these environments have been attracting humans since long before the advent of new media technologies.

The need to understand the universe is a basic human trait, as is the more specific need to understand its rules so we can break them and create new worlds. Randall Packer and Ken Jordan write, "The fantasy of being transported into another world, to be taken wholly into the imaginary real, is a primal desire" [1]. This fantasy has existed from ancestral times, spanning civilizations and technologies. It is embedded in human consciousness and exists in a range of fields, from literature to science, visual art, and cinema.

In the research on media archaeology, we find throughout history a huge range of immersive media techniques, tools, and mechanisms used to create the audiovisual illusions and immersive states that propelled many of the new media technologies.

The Renaissance yielded rich progress in audio and visual fields. Scholars such as Giovanni Battista Della Porta, Johannes Kepler, and Athanasius Kircher were brilliant researchers in both artistic and scientific fields. In this period, important advances were achieved with light, lenses, and other optics technologies that became building blocks for the media technologies we know today.

Giovanni Battista Della Porta (1538–1615) was one of the most important Renaissance authors, who like many scholars of the age dedicated himself to a swath of study fields—alchemy, astronomy, natural sciences, philosophy, mathematics, and cryptography. His most famous work was an encyclopedia called Magia Naturalis, in which he "undertakes a colossal and daring journey through all areas of life; from zoological observations and the (alchemistic) transmutation of metals and the synthetic production of precious stones to the investigation and composition of special combinations of herbs and rituals for inducing abortions and performing quasi-genetic engineering and secret ciphers" [2].

Book XVII of this encyclopedia is dedicated to "divers mirror and lenses," where he explores optics, projection, and reflection. In his studies, Della Porta focused his attention mainly on the "visualization of the imagination"—in the dilations, changes of dimension, deformations, and transmutations of reality that can be achieved with lenses and mirrors. He also gave a full description of the camera obscura, or as he called it, the obscurum cubiculum, including how "hunting scenes and battles and other kinds of hocus pocus can be made and performed in a room ... Guest performances, battle fields, games, or what you will, so clear, distinct, and pretty to see as though it were taking place before your very eyes" [2].

"With such technical artefacts and their specific arrangements, Kircher established a tradition of visual apparatus. [M]edia machines were designed and built in such a way that their functioning mechanisms remained a mystery to the audience: The projected world must not be recognizable as an artificial construct."—Media theorist Siegfried Zielinski [3]

Della Porta's interests—in the camera obscura, other optical illusions, and the potential to create virtual realities, imagined scenarios, and narratives—are amazing, and his words about these topics are surprisingly aligned with current media technology, features, and expectations.

It is also important to highlight the work of Athanasius Kircher. In 1646 he published the book Ars Magna Lucis et Umbra (The Great Art of Light and Shadow)—a compilation of the state of the art in light, lenses, mirrors, and astronomy. The book also presented detailed descriptions and illustrations about the camera obscura and lanterna magica (see Figure 1).

Kircher collected many of the artifacts of his studies in his own museum, Museo Kircherianum, established in the Collegium Romanum in Rome. "The museum was full of marvelous optical and acoustic devices" [2] that would create illusions and delight his visitors. These devices and artifacts were mostly automata figures with hydraulic or pneumatic power that performed a series of automatic movements. Many of these devices were copies or recreations of Heron of Alexandria's "Theatre of Illusions" [3].

The lanterna magica was an image projector with a candle lantern—the ancestor of today's slide projector. Although Kircher was not the creator, he is often associated with the lanterna magica due to his extensive work in the field. (There are references to similar artifacts from many years before from many authors.) One of his breakthroughs was the "Smicroscopin" that appears in the second edition of the Ars Magna Lucis et Umbra from 1671 [2]. It was a device used to narrate stories that included a circular surface holding pictures that illustrated moments in the plot. By rotating the wheel, you could follow the storyline. Kircher presented the story of Christ divided into eight different moments/slides/scenes. This device would become one of the clear predecessors of the modern motion picture projector.

By the end of the 17th century, the lanterna magica became a common instrument throughout Europe. Traveling showmen would often set up magic shows with it [4]. A great example is Phantasmagoria, an audiovisual performance from the 17th century created by Etienne-Gaspard Robertson (see Figure 2). Robertson created an immersive audio and visual space that focused on the themes of terror, demons, and spirits from the darkness. Using projections, light tricks, transparent screens, smoke, magic lanterns, mirrors, projections on glass, and moving lanterns, among other techniques, he played with the perceptions of the audience, creating illusions, space-time ambiguities, and warped reality.

These illusions created by the control of light and shadow allowed him to manipulate, create, and negate the physical space. The space became a transition between the tangible and the virtual, with the objective to "engage the audience's perception and cognition, and play with their fundamental categories of world formation and orientation, of belief and confusion, of certainty and play" (Tom Gunning in [5]). Magic lanterns were already well known and somewhat ubiquitous, but in Phantasmagoria the lantern (or phantascope, as Robertson called it) was hidden. The audience saw the projected images but neither their source nor the screen, which created a feeling that something miraculous was unfolding. The ghosts seemed simply to appear in the air. Robertson created an immersive media space, a temporal alternative space where abnormal events occurred. His mastery of illusion techniques allowed him to play with the audience's perceptions and emotions.

* Alternative Worlds

A good metaphor to illustrate the features of these environments is the "Zone" from Andrei Tarkovsky's 1979 film Stalker (see Figure 3). Like these reactive environments, the Zone has magical characteristics and does not obey the laws of physics as we know them in the real world. It is a post-apocalyptic area with special powers, dominated by a space-time disorder—a place where the conventions of the real world do not fit.

The Zone is a restricted area, closed and guarded by the government. It is known to be a dangerous and unpredictable place, where space and time are unstable. For instance, if you walk in a straight line, it is possible that you'll return to the same place where you started. Or if you try to go back to the same place where you were a few minutes ago, that place might not be there anymore. Somewhere in the Zone there is a special room where it is said that a person's most profound desire becomes reality. To go inside the Zone and come back alive you must hire a "Stalker," who will guide you through. The idea of comparing the Zone and audiovisual interactive environments came up in 2008 in the Interzone project/thesis, an audiovisual reactive installation for transition public places made by the author and Daphne Polyzos.

Similar to the Zone is the "Pocket Universe" from the TV series Fringe. The Pocket Universe is an interdimensional space that was created by Dr. Walter Bishop in episode six from the fifth season (2012). It is a small, secret parallel universe, similar to the current universe but interdimensional, where many of the usual laws of physics don't apply, with "hallways running upside down or in vertical directions." In the character's own words, "There are few places on earth where two plus two don't equal four, and this is one of these places." Access to this place is gained by passing through a membrane that separates both universes. This episode from Fringe is a clear reference to Lewis Carroll's work Through the Looking-Glass, and What Alice Found There. In fact, the episode itself is titled "Through the Looking Glass and What Walter Found There."

"I don't know what happens here when there isn't any human being inside. But when someone gets inside, everything starts to move. Places that were safe became impassable." —From sci-fi film Stalker

Alice's looking glass is a potent symbol for crossing into and creating new worlds. This image is commonly used in literature, film, and technology/design concepts. It represents the fantasy of launching and manipulating new universes with different rules—the curiosity for the unknown and the unexplained that makes people jump to the other side, or inside the Zone, even knowing that it can be dangerous.

Interactive audiovisual spaces are magical places with weird powers, where physics, matter, and logic do not work in the same way as they do in the real world; they are metaphoric places of transition between the real and the virtual. They do not intend to simulate the real world; they intend to augment it, to cross it with non-real worlds, and to give new experiences and perceptions of space-time.

"The Zone is a very complex system. It changes at any moment and without prior notice." —From sci-fi film Stalker

An important parallel between Tarkovsky's Zone and modern interactive audiovisual spaces is the fact that the Zone is an empty, quiet space; the events happen only when someone enters. Intruders and stalkers feed the Zone with their personalities, thoughts, and desires, which cause the Zone's mutations and abnormal events.

The same happens in interactive spaces. Users are the trigger to audiovisual events. Their presence feeds the system with data that is used to create and interact with the image and the audio. When the users go away, the space becomes a quiet zone again.

The integration of the audience as a key part of the artwork was an important milestone, influenced by Allan Kaprow, who in the 1960s coined the term happening. As Packer and Jordan write of Kaprow, "He was particularly interested in the blurring of the distinction between artwork and audience. ... The ultimate integrated art" [1].

It is this generated data that gives life to the space. The audience appears now as coauthor of the interactive new media art pieces [6]. They are an essential element in the system, as they provide a direct performance of the experience. They affect the results and the narrative of the piece.

The digital era allowed the birth of a new kind of real-time performance. Myron Krueger describes one such performance: "The audio or control data from the musicians' controllers were transferred to the image-generating system and used as actuators for visual impulses" [7] and vice versa. A new concept of artist was also born, an artist that acts at a meta level [7]. He creates the mechanism and the instrument, and he defines the parameters, the sequences, and the possibilities. Often he is surprised by unexpected results.

The emergence of the computer represented a breakthrough in this quest for alternative new worlds. The computer took the position of a universal machine that was able to replace a series of different media tools and deal at the same time with image, sound, and text [8]. It was a medium that "would appeal to all senses simultaneously" [1].

The computer led to endless possibilities in the creation and simulation of new worlds, manipulating images, light, sound, shapes, space, and autonomous identities. It enabled the creation of "real" virtual worlds that could be experienced in space and time, crossed, modified, or destroyed. In virtual and mixed realities, humans are able to cross the mirror and jump into a 3-D world built with data in space—the cyberspace, a "graphic representation of data abstracted from the banks of every computer in the human system. ... Lines of light ranged in the nonspace of the mind, clusters and constellations of data," as described by William Gibson in his 1981 novel Neuromancer. A current example of a cyberspace kind of environment is RAM Dance, a toolkit from YCAM Interlab used to create mixed reality environments for dancers, where performers manage and interact with motion data materialized in a three-dimensional digital matrix (see Figure 4).

"The space of the multiscreen film, like the space of the computer, compresses physically the space." –Architecture historian Beatriz Colomina in [5]

back to top  Conclusion

Immersive audiovisual environments represent then a "mathematical wonderland" that Alice jumped into (Ivan Sutherland [1]), the primal desire of being transported into another world. Common elements can be found in everything from ancestral rituals in deep, dark painted caves, where the audiences were immersed in other domains and triggered into altered perceptions and higher states of consciousness [1], to the optics and lights devices of Della Porta and Kircher, to Robertson's Phantasmagoria, to Krueger's responsive environments, to Gibson's cyberspace and virtual realities, to today's mixed-reality spaces with mapped projections and augmented objects.

Technology, science, art, and society evolve, but this desire to understand the world around us and create endless alternative ones with magical and surreal features remains constant over the time. It is embedded in human consciousness.

back to top  References

1. Packer, R. and Jordan, K. Multimedia: From Wagner to Virtual Reality. Norton & Company, New York, 2001.

2. Zielinski, S. Media archaeology. CTheory. 1996; http://www.ctheory.net/articles.aspx?id=42

3. Zielinski, S. Deep Time of the Media. MIT Press, Cambridge, MA, 2006.

4. Burns, P. The history of the discovery of cinematography. 1999; http://www.precinemahistory.net/

5. Douglas, S. and Eamon, C., eds. Art of Projection. Hatje Cantz Verlag, Ostfildern, 2009.

6. Manovich, L. The Language of New Media. MIT Press, Cambridge, MA, 2001.

7. Krueger, M. Responsive environments, 1977. In [1].

8. Daniels, D. Sound & vision in avantgarde & mainstream. Media Art Net, 2004; http://www.medienkunstnetz.de/themes/image-sound_relations/sound_vision/1/

back to top  Author

Rodrigo Carvalho is a Portuguese designer and new media artist. He is currently a Ph.D. student in digital media at the University of Porto under the UT Austin/Portugal Program. His research work, supported by Fundação da Ciência e Tecnologia, is focused on the relations and synergies between sound, movement, and image in audiovisual real-time systems and environments.

back to top  Figures

F1Figure 1. Kircher's projection device. Illustration from Ars Magna Lucis et Umbra.

F2Figure 2. Etienne-Gaspard Robertson performing Phantasmagoria.

F3Figure 3. Scene from the film Stalker.

F4Figure 4. Dancer interacting in a virtual stage with RAM Dance, a coding toolkit to create environments for dancers. (From YCAM Interlab; https://github.com/YCAMInterlab/RAMDanceToolkit/wiki/Overview)

back to top 

©2013 ACM  1072-5220/13/09  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2013 ACM, Inc.

Post Comment

No Comments Found