Elena Not, Daniela Petrelli
Thirty years ago, content in museums and cultural heritage sites consisted mostly of labels, explanation panels, written leaflets, and books. The aim was to help visitors make sense of exhibits and places, understand the history behind their creation and usage, and appreciate their aesthetic values. Guidelines for effective label design followed from museum and audience studies, such as these from Beverly Serrell :
- Labels should begin with concrete, visual references to the objects they interpret to bring them to life.
- Labels should relate to the big idea of the exhibit, not ramble without focus or objectives, or contain sub-sub-subtopics.
- Labels should emphasize interpretation (offering provocation) over instruction (presenting information).
- Labels should know their audience and address visitors’ prior knowledge, interests, and/or misconceptions.
In essence, content should fit what is on display and help to focus attention; it should stimulate the senses, curiosity, and reflection, and amplify the emotion of being in a particular place. Such recommendations have stood the test of time and are still valid today, in the midst of the museum digital revolution. Indeed, digital technology on the exhibition floor offers opportunities to deliver content in a range of media. However, digital interventions have too often been designed as a repository of information rather than as a source of provocation, and as a parallel experience rather than as a complement to the tangible aspects of the visit.
In meSch, digital content complements and augments the tangible, physical experience. It is a means to layer more content onto the physical space, to give a deeper meaning to what is on display, to tell multiple and contrasting stories, to trigger emotion and wonder. In a series of studies, we have experimented with content tightly intertwined to the physical elements of tangible and embodied interaction to create experiences that engage visitors in an immersive and personal way . The examples in this issue’s Demo Hour section (Atlantic Wall, My Roma Pantheon, the Loupe, Exploratory Station) show meSch installations where digital content is delivered at the very same time as the visitor physically explores objects and places, and where the possibilities of interaction between the user and the system expand to include smart objects and networks of sensors.
Co-creating with cultural heritage professionals, we explored how content should be designed to complement the tangible and interactive aspects of the visit. Within cultural heritage settings, content is king: Irrespective of the technology used (i.e., which combination of sensors and actuators), the ability to deeply engage visitors at both cognitive and emotional levels depends on the stories and the values communicated, as well as the way they are presented. The purpose of content design is then to create an evocative experience by intertwining the narratives with the sense of being in place and the bodily interaction. We intended to use personalization algorithms to automatically compose content that fits the dynamic and specific physical context of the visit. At an early stage, we engaged with heritage professionals to better understand their views on personalization. For them, it was less a matter of finding the perfect match between personal knowledge and information, and more a matter of triggering emotions. Content should clearly connect with what visitors see (objects, places) or to what they can imagine (ruins, culture), should reinforce the sense of immersion and participation (by stimulating action and by providing food for thought), and should support sharing and discussion among companions. Museum experts particularly valued the way in which content can hook into the feelings of the visitor—evoking a sense of discovery and surprise, drawing connections to personal interests, and matching their mood and desired type of experience . The many exhibitions and case studies co-designed with cultural heritage professionals allowed us to investigate content design strategies that diverged from the vision of an intelligent system that makes decisions on behalf of the visitor. What we propose is a participative personalization, where multiple narrative threads are offered to visitors who are actively engaged in choosing and shaping their personal visiting path (as opposed to following what the system tells them to do). The system then builds on the visit to offer personalized content generated to create a heritage-visitor bond and to support post-visit engagement.
In meSch we experimented with different sources and genres (diaries and letters, historical newspapers and propaganda, more traditional curated text and object descriptions), multiple media (audio, video, images and slideshows, animation, text), and delivery effects (surrounding sounds, theatrical recitation, creative writing, visual effects, quizzes). The many experiments enabled us to study in depth the impact of content design in shaping immersive and personally involving experiences. Some of the lessons learned provide practical guidelines on how to design content.
Narrative to guide physical exploration. The strongest synergy between physical and digital interaction occurs when visitors are allowed to touch original objects on display. At the Italian Museum of War, a few originals from WWI are on display on an interactive plinth and visitors are welcome to touch, hold, and explore them closely (see Exploratory Station in Demo Hour). The content is designed to focus the visitor’s attention on the object, to invite and guide a tactile exploration of the artifact and generate a sense of discovery. Here a narrator invites visitors to feel the weight, to look at the details, to consider how each object was used and why. Such explicit invitations to look at and interact with objects sustain the visitors’ attention and encourage group members to take turns and share the experience . A similar result was achieved with the Loupe (see Demo Hour), a phone within a magnifying glass frame that used AR techniques to recognize the shape of an object on display to present text. Here, creative-writing techniques such as cliff-hangers were used to invite an alternate reading, with object observation resulting in much longer engagement, extended reading, and higher satisfaction with the exhibition.
By offering multiple, often contradictory points of view on the same object, place, or event, we intended to push visitors to reflect on alternative and possibly conflicting ideas.
The evocative power of the medium. Content does not have to be informative. Indeed, the preoccupation with delivering information detracts from the emotion that lies in the sensory interaction with the material . We used poems and songs, satirical images, evocative sounds, and captivating animations to generate an emotional response rather than to inform. The medium is key. In the outdoor settings of the trenches and fortified camp of WWI, we used evocative sounds played at hotspots to attract visitors and invite them to get closer before a story was told (Figure 1). Diaries and poems of soldiers recited inside dark caverns or on the observation point looking down into the valley generated strong emotional responses: “Some of the personal accounts were poignant. We stayed silent as a form of respect.” “It resounds all around you.” “... explaining that a cannon shot reached the mountain in front makes you look at the landscape in a different way.”
|Figure 1. A storyscape in the trenches of WWI is activated by the presence of visitors wearing an interactive belt (worn by the man second from left); the lamp conceals the bespoke device.|
A further example of evocative content is the projection of animated drawings inside a 3D-printed replica of a kylix, a Greek drinking cup. The animation creates the illusion of movement, as intended by the ancient pottery makers, when the drawing is seen through the liquid in the cup (see sequence in Figure 2). These are calls to action that amplify engagement with the content via the embodied experience: to move closer and listen to the stories or to handle the kylix to start the animation.
|Figure 2. The interaction with the Feint: Illusion in Ancient Greek Art interactive installation projects an animation within a 3D-printed replica of a kylix (Allard Pierson Museum, Amsterdam, the Netherlands).|
Connect via shared stories. Led by museum professionals, we embraced the diversity of perspectives that characterizes the contemporary museum interpretation of historical facts, particularly war-related history (see Atlantic Wall in Demo Hour). By offering multiple, often contradictory points of view on the same object, place, or event, we intended to push visitors to reflect on alternative and possibly conflicting ideas. We shied away from didactical information delivery and sought more engaging approaches, finding value in combining personal stories and factual reporting to aid visitors’ interpretation of history. The historical context is made more real through poignant memoirs that bring past events closer to the present. When the visiting experience occurs on site, explicit references to the place and to local events are particularly important to create a sense of immersion in the past. Referencing local history helps the community discuss, appreciate, revisit, or discover their shared past, which is critical in the long-term care of cultural heritage. Non-local visitors respond very positively too, since the personal stories foster empathy and an affective bond via the shared human experience.
Curators are well aware that different visitors are touched by different stories according to their own interests, sensibility, and life experience (Figure 3). How to best match individuals and stories has been a matter of research on personalization for cultural heritage over the past 25 years. The challenge of maintaining a dynamic model of the current context of an individual visit is complicated by the fact that the personal motivations and mood for today’s visit are unknown; the social setting affects the behavior, and personal interests and knowledge are difficult to model. Instead of giving the system more intelligence by guessing a visitor’s temporary disposition and then delivering a single story selected by the system, in meSch we exploited tangible interaction to offer visitors a choice of stories and themes, with the opportunity to swap or to add more at any time (Figure 4; see also Atlantic Wall in Demo Hour). This participative personalization requires designing the content for the intended experience in synergy with the tangible elements of the interaction, for example, multiple smart replicas or theme cards. In this way, visitors acquire agency and the system adapts the content delivery to visitors’ actions: When multiple options are on offer, a visitor expresses preferences by choosing objects and moving in space (customization). The system senses these choices and delivers content at the right time to accommodate the visitor’s behavior (context-awareness), generating personalized output based upon the dynamic visitor’s model built throughout the visit (adaptivity).
|Figure 3. The multiple threads of curators’ stories. First line: daily life of soldiers explained to adults vs. children. Second line: historical documentation of sanitary problems vs. the point of view of propaganda.|
|Figure 4. A set of cards represents the different themes. Tangible interaction facilitates negotiation of content choices.|
The way the experience is personalized via a blend of customization, context-awareness, and adaptivity based on physical interactions makes the content seamlessly fit the visit dynamics. Instead of the optimal algorithm to match visitor and content, the challenge in participative personalization is to create rich and multifaceted narratives that complement tangible interaction and bring together the physical and the digital for an engaging and evocative experience.
Crafting content across the physical-digital divide requires separating the narratives from the interaction strategies that deliver content in the specific context of the visit. This approach reduces the complexity of the creation phase by assigning cultural heritage professionals the task of curating the narratives, while designers work on the tangible interaction to control when and how the content is delivered. Thus, the preparation of the content requires the following:
The crafting of personalized souvenirs is an excellent example of combining the flexibility of the digital with the material engagement of the tangible.
- Curators and communicators design the content focusing on their intended communicative goal (e.g., to recount stories of daily life during WWI). The content is split into fragments, and multiple narratives are created to provide different points of view (e.g., the official stance of military officers taken from formal records: “Notes on the visit to the Nagiá Grom stronghold, sector 4a - peak Biaena, Rovereto command centre, done on the 18th September 1917 by myself, General Arthur Arz von Straussenburg. The stronghold is in good condition and properly positioned.” Or how civilians lived the war, with the more intimate language of personal diaries: “It was the 27th May 1915, as miserable pilgrims we all left our homes and our dear village where we lived many happy years. From Mori we went to Rovereto station—we were loaded like cattle.”).
- Content items are annotated with semantic tags that describe how fragments are linked to one another and how they fit different visitors’ characteristics and experiences. The tags may map anything: the visitor profile (e.g., preferred language, school trip); aspects of the discourse structure (e.g., whether parts of the story are optional or should be delivered in sequence); thematic choices offered to visitors (e.g., historical information, personal accounts, poetry). For example, the semantic annotations for educational content may say that a given content item is in English, is for primary school children (i.e., has a simple lexicon), and is a trail based on soldiers’ diaries, where each item of content must be listened to in a prescribed chronological order to preserve the sequence of narrated events.
- A second set of annotations describes the conditions that control the delivery of content when a specific context occurs: a specific value for a given feature of the interaction context (e.g., a position in the exhibition space), of the social context (e.g., co-presence of group members), or of the environment (e.g., noise level). For example, interaction annotations may say that a certain content item (e.g., the sound of a train whistle) is to be played to attract visitors when they are in the proximity of but not close enough for the story.
- Computational rules describe how the annotated, structured content created so far is actually traversed. At run time, the rules test whether the semantic and contextual features associated with a content item match the current visitor’s choices and the interaction context. For example, a rule may state that “IF a smart object is placed over an interactive station, THEN play the content item annotated with the theme and language specific for that object.” More complex adaptivity rules can be written to take into account the interaction history, such as “IF a visitor comes to the same place a second time, THEN deliver further details.”
This layered annotation separates the content from the context and fosters modularity and reusability. The same context (interaction rules) can be reused to deliver different stories, for example a traveling exhibition adjusted to the local collection. Similarly, the same content can be delivered in another context, for example on a different configuration of IoT devices.
The same IoT network of sensors and actuators that delivers digital content in the physical space collects the data of the interaction. This log can be exploited to create personalized souvenirs as reminders of the visiting experience and as a means to continue the interaction online: At the end of the visit, the data collected by the smart object that visitors used in the exhibition is read to create a personalized hard-copy postcard (Figure 5). The crafting of personalized souvenirs is an excellent example of combining the flexibility of the digital with the material engagement of the tangible. Techniques derived from text generation have been used to produce short summaries of the individual visit as recorded in the interaction logs; the level of verbosity and the graphical layout could be controlled to produce artifacts as diverse as a visit receipt, a postcard, or a multipage leaflet . The personalized text paired with an evocative graphical layout creates an object to help the visitor remember the visit and retell the experience to others, and invites the visitor to further engage with the museum (Figure 5): “The postcard is a link to what you did and saw. In the end, when you look at it, you think ‘I’ve got to go back!’” “The postcard is not just a souvenir, but a continuation, a bond between the visitor and the curator.” The souvenir materializes the visit and can be a bridge to connect the onsite exhibition with a subsequent online exploration: A unique code on the postcard provides access to a personalized website automatically created on the bases of the log associated with the code. It’s designed to sustain interest and invite a long-term relationship with the cultural institution .
|Figure 5. Content automatically generated for personalized souvenir postcards.|
In meSch we collaborated with more than 40 cultural heritage professionals at different stages of the project, and at different levels of involvement, to co-design and evaluate an accessible authoring environment for content creation . Interesting new directions were suggested, including the use of the platform to engage the audience in the creation of new exhibitions, in the ethos of the participatory museum, or in maker sessions for young visitors as a way to appropriate the exhibition. These scenarios go beyond current skills, concepts, and objectives to change how content is created (e.g., by visitors, for other visitors) and deployed (e.g., for pop-up exhibitions), possibly feeding in fresh ideas for experimenting with different media types, narrative threads and genres, and customization choices, to fill the digital-physical gap in cultural heritage institutions.
meSch (2013–2017) received funding from the European Community’s Seventh Framework Programme, “ICT for access to cultural resources” (ICT Call 9: FP7-ICT-2011-9) under the Grant Agreement 600851.
2. Not, E. and Petrelli, D. Blending customisation, context-awareness and adaptivity for personalised tangible interaction in cultural heritage. International Journal of Human-Computer Studies 114 (2018).
5. Petrelli, D., Marshall, M.T., O’Brien, S., McEntaggart, P., and Gwilt, I. Tangible data souvenirs as a bridge between a physical museum visit and online digital experience. Personal and Ubiquitous Computing (2016), 1–15.
6. Not, E. and Petrelli, D. Empowering cultural heritage professionals with tools for authoring and deploying personalised visitor experiences. User Modeling User Adapted Interaction 29, 1 (2019), 67–120.
Elena Not is a senior research scientist at Fondazione Bruno Kessler, working on the application of ICT technologies to cultural heritage and smart cities. She has a background in computer science and is interested in methods for adaptive information presentation with the integration of different interaction modalities. email@example.com
Daniela Petrelli is a professor in interaction design at Sheffield Hallam University. She started researching technology for heritage in 1996. Her other research interests include personal memories, multilingual and multimedia information access, and visual analytics. Throughout her career she has received 12 international awards. She directed the meSch project. firstname.lastname@example.org
Copyright held by authors. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2019 ACM, Inc.