Kim Halskov, Peter Dalsgaard
In the dungeons of Kronborg, a nearly 400-year-old Renaissance castle known from Shakespeare's Hamlet, there is a statue of Holger the Dane. Holger the Dane is a mythical figure who, according to one myth, will awaken and defend the country when an enemy from beyond the borders threatens the kingdom of Denmark. Annually, more than 90,000 people visit the castle, and while most of them visit the dungeons, they usually visit the statue only briefly.
How does a Danish myth relate to HCI? Well thanks to 3-D technology, we have been able to bring Holger the Dane "to life." In order to create an engaging experience that communicates some of the many legends about Holger the Dane, most of which are unknown to the public, we looked to 3-D projection. Three-dimensional projection on physical objects is a particular kind of spatially augmented reality (AR), which augments a physical object by projecting digital content directly onto it, rather than by using a device such as a mobile phone or a head-mounted display. During the past two decades, several research teams have been investigating 3-D projection on physical objects from a technical perspective [1, 2, 3]; in contrast, our interest is in using the technology to support engaged communication.
Beyond such explorations of 3-D projection, which primarily focus on the technical aspects of the technology, the most prominent examples of 3-D projection are, arguably, the works of Pablo Valbuena. His series of installations, entitled Augmented Sculptures, has been displayed at Ars Electronica and has attracted many online visitors to Youtube . Valbuena's installations are typically composed of camera-tracked, angular, and clear-cut geometric shapes in conjunction with 3-D technology, which is used to create the illusion of light sources moving across elements of the faces of the installations. Also, there is a growing number of installations that employ 3-D projection to mimic the third dimension on a 2-D surface. 555 Kubik is one example of such an installation . Projected onto the facade of Hamburg's Kunsthalle, the installation employs visuals that create a sense of depth in various ways: The tiles appear to move in and out of the facade, the interior of the building is revealed in perspective, and so forth. AntiVJ's installation, Enghien , employs similar forms of expression but takes further steps to both underline and break down the illusion. Projected onto a building facade, the installation first mimics moving light sources to emphasize the 3-D effect; then, it starts copying and apparently moving the physical architectural features, such as windows and balconies; finally, it deconstructs and eventually explodes these features of the building.
Operating in the digital 3-D world also enables the designer to virtually project on specific elements of a physical model, and thus pick out particular areas to augment.
These installations are either developed to explore technological potential or for artistic reasons. The artistically oriented installations primarily explore new means of expression and may be defined as staged events for audiences. In relation to these, the Holger the Dane installation may be characterized as more functionally oriented in the sense that the installation has been developed to support communication. In addition to this installation, our research team has developed a series of other installations that employ 3-D projection on physical objects, including an augmented rune stone exhibit at a cultural heritage museum, a contribution to the Venice Architecture Biennale on urban planning, and a design tool that was used in the development of the Danish pavilion at the 2010 Expo in Shanghai. These installations, along with Holger the Dane, are explored in more detail elsewhere .
When visitors approach the statue, "embers" behind the feet of Holger the Dane glow more brightly, and a sequence of narrative segments recounts his life (a video of the installation can be viewed online ). According to legend, six fairies gave special power to Holger the Dane, one of which appears in front of the statue. It flies around, while its shadow is cast on the sculpture, and fairy dust illuminates parts of the sculpture, imbuing Holger with his powers. The many legends about Holger the Danehe was held captive by King Charlemagne, became an outlaw, and fought Burmand the giantare conveyed in the narrative segments through visually abstract references. For instance, the fight with Burmand is conveyed by a shadow slowly covering the statue, to illustrate Burmand's approach. Next, sounds illustrate the fight between the two; then another fairy arrives, giving Holger the Dane additional strength and enabling him to split the shadow. Blood appears to splash his shield and one of the statue's feet. Finally, in the last sequence, Holger the Dane appears to have fallen asleep, suggested by the upper part of his body slowly moving with the rhythm of his breathing. The statue is a very popular photographic subject, and when a photographic flash is detected, the visual style of the projection changes.
In our research, we studied the Holger the Dane installation, and the related installations mentioned in the introduction , in order to explore the design potential of 3-D projection on physical objects. As a result of this work, we wish to highlight two areas we find particularly promising for designers venturing into the domain of 3-D projection.
Fusing digital and physical objects. Generally speaking, the use of 3-D projection in spatial AR provides designers with the opportunity to employ a wide range of visual effects and illusions. In addition to new means of expression, designers may also make use of standard visual effects that are parts of existing 3-D software, for instance 3-D Studio Max.
In the case of Holger the Dane, the first step of the process was to enlist a scanning company to assist us in creating a digital model, which was subsequently used to create a 3-D print in a scale of 1:4. This enabled us to carry out a number of experiments and tests of the installation in real life, without traveling to Kronborg Castle and disturbing the exhibition before the launch of the installation. The creation of a 3-D model that matched the statue and its surroundings was the first step in fusing digital and physical objects, and the scale model allowed us to explore some of the 3-D effects that may be used to strengthen this fusion.
One of the standard effects in 3-D modeling is the use of shadows to support visual depth. In the installation, the fairies that grant Holger his powers are modeled as 3-D objects and cast shadows on the 3-D model of the statue. When projecting the fairies and their shadows onto the physical statue, we create the illusion that the fairies are flying in front of the statue. In other sequences, we use shadows cast on the statue by objects that are not visible. For instance, the approaching giant is visualized by a shadow cast on the statue, and when Holger the Dane is imprisoned, shadows of prison bars are cast on the statue.
Another common technique with which we have successfully worked is the use of various particle systems, which can simulate fire, for example. Such particle systems may be placed in front of a model; they create quite realistic effects, for instance, in terms of reflections on the complex geometry of the physical object itself. The fire behind the leg of Holger the Dane was made using this strategy. Moreover, we used a particle system to visualize the fairies' dust.
A third technique, which also uses 3-D software, is incorporating various filters normally confined to the digital 3-D world to create visual effects on the physical model. For instance, we used the twirl filter to create the illusion that Holger the Dane was sleeping, as indicated by having the upper part of his body appearing to slowly move to the rhythm of his breathing.
Transforming the perception of space and materials. In addition to using various means to fuse digital and physical objects, one of the aspects we found to have the greatest impact on audiences is the use of 3-D projection to transform the perception of space and materials. The Holger the Dane installation offers a good example of how visual effects that emphasize and transform physical properties may be combined with the emphasis or transformation of semantic properties. The public perceives the statue as "rock solid"both literally, being cut from stone, and in terms of representing a firm and unyielding mythological warrior. In the installation, we emphasized this general perception by initially projecting an image of the statue onto the statue. The result is an increased contrast that emphasizes the physical properties and makes the figure appear even more solid. However, at a later point in the unfolding narrative, we employ the aforementioned twirl filter on the projected image, which makes the statue appear to move and twist ever so slightly. This illusion plays into the viewer's perception of the physical structure, as well as the permanence of the mythological figure now coming to life.
Operating in the digital 3-D world also enables the designer to virtually project on specific elements of a physical model, and thus pick out particular areas to augment. In the case of Holger the Dane, the fire is confined to a glow behind his legs, and blood appears to splash the shield and one of the statue's feet. Such effects may serve to further play on the perceived materiality of the physical object.
The installation has been in operation at Kronborg Castle since October 2010, and an initial study, supplemented by visitor interviews, indicates that many visitors find the installation fascinating and novel. The positive effect of this is that many visitors now make a longer stop at the statue, and the visuals prompt them to learn more about the myths concerning Holger the Dane. However, on the basis of our observations and interviews, it is also clear that the novelty of the installation causes some issues that must be addressed in future versions of the installation or in similar projects.
First of all, the installation truly stands out in comparison with the rest of the site with regards to the high-tech appearance of the installation in contrast to the ancient dungeons in which it is placed. Second, many visitors join a guided tour, and while the tour guides are usually the center of attention and chief narrators for most of the tour, when the tour reaches the statue, the installation takes center stage. This leads to a clash in narrative styles, and the tour guides are not yet comfortable in handling this situation. Third, many visitors in fact know little about the myths of Holger the Dane, save for the one that tells that he will rise when the nation is under threat. For this reason, some of the visuals can be confusing. As stated by one interviewee: "The visuals are very impressive, but what is up with the fairies?" These three issues all highlight the need for the installation to be more closely integrated into the context and practices of the castle as a cultural heritage attraction.
Visitors' spontaneous exclamationsfor instance, "Now he's locked up!" when bars are projected onto the statueindicate that some of the visual effects make sense to them. In particular, the effect of splashing blood and the sleeping sequence evoke visitor reactions. However, only a portion of visitors view the entire sequence of narrative events; one reason for this is that the sequence runs in a loop. If you enter in the middle of the sequence, visitors will miss some of the initial narrative fragments that build up to subsequent events; furthermore, many visitors leave when the sequence is finished and do not stay to view the initial parts that they missed. Also, few notice the effect of flash photography.
Our interviews reveal that visitors are split when it comes to evaluating the overall concept and visual style: Some find the installation a bit too "Disney World" and point out the clash between the ancient dungeons and the modern, tech-heavy installation, whereas others appreciate the mood and atmosphere created by the installation and also point out that the installation could be a powerful way of creating interest in the younger generations about the legend of Holger the Dane. The manager of Kronborg Castle and the cultural heritage communications manager were initially in favor of a more direct retelling of the myths but were ultimately captivated by the atmosphere of the relatively abstract final installation.
In contrast to conventional AR, such as what is found in mobile devices, spatial AR using 3-D projection presents interaction designers with unique opportunities and challenges. The most prominent advantage of using 3-D projection on physical objects is, arguably, that it offers viewers an experience of immediacy and physical presence. This is different from what may be achieved with traditional AR, which uses a screen as the media layer. Three-dimensional projection removes the screen as media layer and presents the virtual layer directly on top of the physical object or environment. This enables an experience of presence that is qualitatively different from screen-mediated AR.
In some settings, such as museums and exhibitions, this is a particularly promising strategy. For instance, many museums place an emphasis on presenting visitors with authentic objects, such as the original Holger the Dane statue, but also wish to communicate further information related to the object. Traditionally, this has been accomplished by presenting audiences with a separate channel of communication, such as accompanying leaflets, descriptions on signs near the objects, audio guides, and so on. When properly employed, 3-D projection keeps the emphasis on the object itself, while adding layers of information. In another project, we worked with projectionthough not 3-D projectionon a Danish rune stone, as part of an exploration of fusing projection and object, in an engaging approach to communicating information at a cultural heritage museum .
However, 3-D projection on physical objects is a complex process that presents the designer with a number of challenges. First, the process of developing a precise 3-D model of the physical environment may be complicated, particularly when dealing with complex structures, such as the Holger the Dane statue. Second, content must be custom-developed for the specific environment; although standard effects from 3-D software may be employed, other types of content must be carefully aligned with the physical environment. Third, successful implementation of 3-D projection requires a high degree of control over the physical location in which the technology is employed. The projectors must be calibrated very precisely, the lighting conditions have to fall within specific parameters, and the physical environment must not be altered. This set of challenges is most likely the reason that related examples of spatial AR are set up in controlled environments, either in laboratory settings or designated exhibition areas. Fourth, spatial AR installations must be integrated into the context, not just with regards to the physical space, but certainly also with regards to the established practices of the site.
We would like to thank the many colleagues at CAVI who took part in the production of the Holger Dane Installation and our collaborators at The Castle of Kronborg. Our research has been funded by the Danish Council for Strategic Research (Digital Urban Living, grant 09-063245).
2. Raskar, R. Welch, G. and Fuchs, H. Spatially augmented reality. Proc. of the First IEEE Workshop on Augmented Reality (IWAR'98). (San Francisco, CA, Nov. 1). A.K. Peters Ltd., Natick, MA, 1998, 6372.
3. Raskar, R., Welch, G., Low, K-L. and Bandyopadhyay, D. Shader lamps: Animating real objects with image-based illumination. Proc. of the 12th Eurographics Rendering Workshop (EGRW 2001) (London, UK, June 25-27). Springer, 2001, 89102.
4. Valbuena, P. Augmented Sculptures, 2007; http://www.pablovalbuena.com/p05.htm/
5. Rossa, D. 555 Kubik, 2009; http://www.today-andtomorrow.net/2009/07/23/555-kubik-facade-projection/
6. AntiVJ. Enghien, 2009; http://www.antivj.com/enghien/index.htm/
8. CAVI. Holger The Dane, 2011; www.cavi.dk/projects/3d_holger.php/
Kim Halskov is a professor of interaction design at Aarhus University and director of CAVI and Center for Digital Urban Living, Aarhus University. His research focuses on the theory and practice of interaction design.
Peter Dalsgaard is an assistant professor of interaction design at Aarhus University. His research focuses on the theory and practice of designing engaging interactive environments and the ways in which pragmatist philosophy may inform and inspire interaction design.
This article is based on a CHI 2011 paper by P. Dalsgaard and K. Halskov entitled "3-D Projection on Physical Objects: Design Insights from Five Real-Life Cases."
©2011 ACM 1072-5220/11/0500 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2011 ACM, Inc.