FeaturesSpecial topic: the meSch project

XXVI.5 September-October 2019
Page: 34
Digital Citation

Tangible interaction meets material culture: Reflections on the meSch project

Daniela Petrelli

back to top 

Leo Caillard's Art Game series (Figure 1) captures the essence of the meSch project: to engage heritage audiences both physically and emotionally through unexpected interactions so their attention remains on the heritage, not the digital technology. meSch stands for Material Encounters with Digital Cultural Heritage; tangible and embodied interaction is at its heart. The vision of augmenting the physical experience via digital technology and media is underpinned by the Internet of Things: Microprocessors, sensors, and actuators can be concealed within reactive spaces and smart objects that seamlessly blend with their surroundings. By itself, digital technology is shapeless. As electricity, it needs a medium to become perceivable. This medium could be an off-the-shelf digital device, as in Caillard's Art Game, or a bespoke digitally augmented artifact, as we have done in meSch.

back to top  Insights


ins02.gif Figure 1. Image from Leo Caillard's Art Game series.

The pliability of digital technology, paired with the ever-expanding range of elementary computational elements, allows us to radically rethink interactive experiences in museums and heritage sites. Bespoke design can bring places and stories from the past into the present, creating immersive experiences where technology complements heritage (as opposed to competing with it for the visitors' attention). Pervasive computing then becomes an addition to the exhibition designer's toolbox, offering new ways to engage users with digital content through tangible interaction. There is much potential in making the technology disappear and making the visiting experience more immersive, sensorial, and affective. This calls for the heritage sector to change from being a repository of facts and information (often delivered via digital technology) to becoming a provider of meaningful and fulfilling experiences:

The museum's preoccupation with the information and the way it is juxtaposed to objects ... immediately takes the museum visitor one step beyond the material, physical thing they see displayed before them, away from the emotional and other possibilities that may lie in their sensory interaction with it.... Yet museums' preference for the information over the material, and for learning over personal experience more broadly and fundamentally conceived, may risk the production of displays which inhibit and even preclude such affective responses [1].

Today many heritage institutions acknowledge the value of a personal, physical relation with the objects. Indeed, many museums offer handling sessions where, under certain conditions, visitors can touch some of the original items in their collection—and visitors love it. The tangible and embodied experience of heritage takes visitors well beyond any educational mission. Far from being perceived as boring and tiring, some museums are seen as restorative environments that with their unique aesthetics capture imagination, invite reflection, and facilitate recovery from mental fatigue. Such aesthetic experiences are devoid of information acquisition or learning objectives and instead are centered on the sensorial experience of being there [2,3].

back to top  Redefining the Heritage Experience With Mesch

The premises that information is not the most important factor in visiting heritage and that the material culture (the essence of heritage) has a strong appeal, combined with the availability of a rich IoT toolbox, open up opportunities for interaction design to use technology to create immersive interactive visiting experiences. At this crossroads of disciplines is where meSch lies, and where our extended and thorough investigation took place. As discussed in Avram et al.'s article in this section, the meSch approach was intensively collaborative and challenging for everyone. Museum professionals, product designers, and HCI and technology experts were all participating on equal terms, which initially required some adjustment. The museum professionals were invited to sketch in hardware as a way to understand that technology in museums does not equal screens and that instead it can be used to create interactive spaces and objects. By becoming acquainted with what is technically possible, the museum professionals could grow their creativity in the direction of "What can I do with this in my museum?" This was a push to think of how content from archives could become part of an exciting and new experience, moving away from the delivery of prepackaged information in place. It was also a push to explore ways of avoiding being didactic or prescriptive (see Not and Petrelli's article). Digital content is designed: to question visitors and their beliefs, to surprise and make them wonder, and to make them feel and empathize.

While heritage professionals explored what message they could deliver with this new digital medium, technologists built an understanding of the needs, constraints, and desires of heritage institutions and visitors alike. Frustration surfaced when hopes for advanced technological research were dashed by the very practical needs for installations that were cheap, usable by everyone without any training (who remembers the "walk up and use" systems?), and robust enough for tens of thousands of visitors to use over many months, a scale that makes any researcher in the wild shudder. There is a tension at work as heritage institutions aim to offer unique experiences but are unwilling to take risks by using emerging technology. The technical expertise and knowledge of the project then focused on building an IoT infrastructure that, like Legos, enables the fast prototyping of interactive installations by recombining the same set of embedded electronic components without the need for technical expertise. The low-threshold, high-ceiling IoT toolkit (see Marshall et al.'s article in this section) became a means for design thinking on the exhibition floor as we collectively embraced creativity and experimented with many very different concepts.

By creating different objects to hold and interact with, product design makes these visiting experiences feel very different from one another, despite all being based on the same technological components.

The flexibility of the toolkit simplified the production and enabled extreme customization and the making of batches of one, where the same technology, for example simple (yet very robust) NFC mechanisms, can be used in many different installations, such as the Atlantic Wall, My Roman Pantheon, and the exploratory station featured in this issue's Demo Hour, but also the Fort Pozzacchio exhibition (presented below), the Feint installation (see Not and Petrelli's article), and the WWI trenches storyscape (in Dulake et al.'s article). By creating different objects to hold and interact with, product design makes these visiting experiences feel very different from one another, despite all being based on the same recombined technological components.

Stepping back, it is now easy to see how the process of design becomes more complex as we had to consider archives and collections (content), ubiquitous computing (platform), and the many materials and shapes of objects (product). In an organic way, we developed a path of design that was holistic as we simultaneously designed the content, the technology, and the product for a specific heritage (Figure 2). It is essential to consider all three at the same time, as one affects the other. A fourth element to consider is the environment. Erik Stolterman talks about design being concerned with the ultimate particular rather than with the universal or generic. This is even more true in designing for heritage, as the place where the interaction occurs is unique, and this, in turn, strongly affects the design.

ins03.gif Figure 2. Designing tangible interactions for a specific heritage: the components of a holistic design approach.

back to top  Voices From Fort Pozzacchio

There is one particular story that, for me, illustrates holistic design in quite some detail, especially in terms of how the three elements—content, technology, and product—are intertwined, and the role played by the heritage environment. It is the design of the installation Fort Pozzacchio: Voices from the Past at the Museo Storico Italiano della Guerra in Italy (the Italian Historical Museum of War). This was the last major installation planned for meSch, and by this time we had learned a few important lessons. Content must be interesting, engaging, and to the point: Audiovisual media has a strong attractive power, but videos of well over 10 minutes, as some were in the Atlantic Wall (see Demo Hour in this issue), were far too long and only very rarely listened to in full. The multiple and contrasting points of view, however, were very much appreciated by the visitors to the Atlantic Wall exhibition, who were pushed to question their own assumptions and reflect on what they saw. The multiple stories were materialized in smart replicas that represented the different voices, which were easy to use and nice to hold, fostering an engaging experience. The NFC technology was simple for the museum to manage (they could make their own replicas) and extremely robust. It enabled the logging of the interaction and the printing of a personalized postcard that extended the interaction online. In terms of fitting the environment, the Atlantic Wall did not pose any constraint, as the exhibition was created ex-novo.

In designing for Fort Pozzacchio, we had to negotiate with an imposing environment, but we turned this to our advantage. Fort Pozzacchio was built by the Austro-Hungarians at the break of WWI and contested by the Italians in the years following. It is a very peculiar place and a masterpiece of engineering: a three-story fortress of galleries and caverns dug into the mountain where soldiers lived, observed the movement of the enemy, and fought for their lives and the honor of their country.

The interactive exhibition about Fort Pozzacchio was to complement the collection of WWI artillery hosted in a WWII bomb shelter dug in the mountain under the museum, an environment that closely resembles the fort. The challenge was to create an immersive experience in a site with a strong physical and historical identity.

The curators were keen to tell the story of how the fort affected the life of many before, during, and after the war. They brought to the creative workshop samples of personal diaries of soldiers and civilians, as well as official army journals. Over two days we discussed several concepts, such as a context-aware walking staff and a time-travel radio. Each concept was critically assessed with respect to the brief: an extremely simple technology, cheap to produce, unbreakable, easy to maintain for the museum, immersive and evocative for the visitors. At the end of the first day, to the disheartenment of some, we had not agreed on a concept. However, the rich discussion was instrumental in generating the proposal put forward by the curators on the following morning: Instead of choosing a person and following their story, the curators proposed to organize the content by theme. The themes were the fort itself, its building and dismantling; life in the nearby village from before to after the war; and a single battle told by the two enemy armies. This clear organization of the content gave pace to the design, which quickly converged on the concept of a timeline, with all the stories set down for the visitor to choose from. At the entrance the visitors would receive a simple object to activate multimedia content at the thematic stations; when leaving they would receive a personalized postcard with a summary of their visit as a souvenir.

The technology behind this installation is the same as the one in the Atlantic Wall, but the design is different. Here the visitor does not join a point of view represented by a smart replica, but rather uses a piece of the fort—a pebble—to choose the story they want to listen to at every point in their visit. Each interactive station has a theme and the bench is a timeline; each little bowl is a different time and a personal story (Figure 3).

ins04.gif Figure 3. Each station has a theme. The theme of this station is the fort itself. The cups are a timeline: the design (1912), the building (1913, 1914), and the fighting (1916) for the Fort on the left; the dismantling, shown in the animation, many years later (1935) on the right.

Following the creative workshop, the work progressed in parallel. The museum in Italy searched their archives for content that fit this concept of personal stories of people whose lives were affected by the fort over a period of 50 years. The designers in the U.K. prototyped the installation and the pebble. The technologists in the U.K. and Italy built the infrastructure and coded the software. We all used the IoT toolkit to work independently while synchronizing our effort. The content uploaded by the curators (in Italy) was used to test the tangibles and to develop the personalized postcard (in the U.K.). We then met in place with a prototype to discuss what type of media would be more impactful within that environment to create an emotional connection between the visitors and the stories. A white-on-black, hand-drawn animation was the counterpoint to stories of the fort, the engineer who sank the fort within the mountain, the civilians who built the road and the fort itself, the soldiers' living in it, and its dismantling after the war was over. The drawings slowly emerge from the blackness of the cavern like memories resurfacing to the present (Figure 3).

Voices from Fort Pozzacchio is the outcome of a process where the technology, the product, and the content have been designed to work in synergy to augment each other.

The second station told the stories from the village nearby: the villagers happy for the new road, their desperation when they had to flee their homes or join the army, and the slow return to normal life after the war. Their stories were made more vivid by soundscapes.

The last station, positioned deep within the mountain, held the memories of soldiers from enemy armies and that of the chaplain, all recounting the same battle. Excerpts from the diaries were recited by actors in video portraits that talk directly to the visitor and are projected on a canvas hanging in the cavern to create a sense of intimacy, as in a confession. It is a theatrical setting, where the projection is done precisely where the videos were recorded (Figure 4).

ins05.gif Figure 4. The dramatic setting for the display of video portraits of enemy soldiers remembering the same battle.

The reaction of the visitors was well beyond what we expected. They stayed longer in the artillery gallery (30 minutes more) despite it being a cold and damp place. Visitors commented on the experience, giving meaning to the pieces on display: "To my eyes, this is not a cannon anymore.... I now see a means of destruction, not of the enemy, but of the lives of normal people." The pebble empowered them; it gave them choice and a reason to listen, to share and discuss as a group. The pebble was a way to connect to the people from the past while their stories provoked personal empathy and affective responses: "With the pebble you take those stories with you" and "I feel like I am entering the lives of others." The personalized postcard they received when leaving made the experience tangible and provided a keepsake.

back to top  Holistic Design

Going back to Leo Caillard's sarcasm and the critical question of how we make information technology more engaging in the context of an exhibition, we have to acknowledge that it is not just a matter of what content is provided and which device is used to deliver it. Voices from Fort Pozzacchio is a unique and engaging visiting experience customized to the place. It is the outcome of a process where the technology, the product, and the content have been designed to work in synergy to augment each other. The technology is seamlessly integrated in the installation: No wire or screen is visible. The large-scale projections make the experience immersive with the design of the pebble (discussed in Dulake et al.'s article) strengthening the feeling of being there, being present. The timelines on the benches ask the visitor to become active, to engage, and to choose which story to listen to; the pebble is smooth, pleasant to hold, and easily slips into the bench cups. The stories, in turn, are all personal accounts from different people at different times; they have been designed to make visitors think beyond the time of the war, about how life was before and how slowly it returned after. The pebble, the stories and the sounds, the projections of animations and videos, the environment and the settings—all work together to invite a physical engagement and provoke an emotional reaction.

Seen through the lens of holistic design, we can try to deconstruct how the three components work together. The content must be designed to challenge the visitors' opinions with multiple voices and conflicting stories, and move them with personal and intimate accounts.

The interaction with invisible technology forces a sense of presence, of being here, now, and enables the emergence of new meanings for the displayed collection: Listening to the stories of the villagers makes visitors see the artillery on display with different eyes.

Carrying an object during the visit is very powerful. Holding a pleasurable object extends the sense of self and asks the visitors to make choices, to become active.

And finally, the place itself is evocative and theatrical. Our design exploits it to make the experience even stronger. The final station, the battle for the control of the fort, is reached after about 40 minutes of walking within a long gallery; feeling the cold of the stone, you see all around you the oppression of a place with no natural light.

back to top  Conclusion

If we want the visitors to be drawn into the stories, the experience has to be orchestrated. Then it is not a matter of information to be delivered, but rather of how this is done and how the visitor is gently pushed to become active, to choose, to question, to discuss. The full integration of technology and heritage requires us to approach the creation of an interactive exhibition as a collaborative project that combines curatorial, technical, and design aspects. As designers of interactive installations for cultural heritage, a holistic design approach invites us to consider and use many sensorial aspects that are often overlooked, but that are fundamental to creating a memorable experience.

back to top  Acknowledgments

meSch was an EU project (2013–2017) that involved 12 partners across six countries in a large-scale exercise of co-design and co-creation. More than 50 researchers contributed their thoughts and work to the meSch project, but only a handful of them have co-authored this section. I gratefully acknowledge the effort and creativity: Without your enthusiasm, energy and participation, meSch wouldn't have been so meaningful and impactful. Seven awards are a testimony to the relevance of our collective work.

meSch (2013–2017) received funding from the European Community's Seventh Framework Programme "ICT for access to cultural resources" (ICT Call 9: FP7-ICT-2011-9) under the Grant Agreement 600851.

back to top  References

1. Dudley, S. Museum materialities: Objects, sense and feeling. In Museum Materialities: Objects, Engagements, Interpretations. S. Dudley, ed. Routledge, 2010.

2. Kaplan, S., Bardwell, L.V., and Slakter, D.B. The museum as a restorative environment. Environment and Behavior 25, 6 (1993), 725–742

3. Paker, J. and Bond, N. Museums as restorative environments. Curator - The Museum Journal 53, 4 (2010), 421–436.

back to top  Author

Daniela Petrelli is professor in interaction design at Sheffield Hallam University. She started researching technology for heritage in 1996. Her other research interests include personal memories, multilingual and multimedia information access, and visual analytics. Throughout her career so far she has received 12 international awards. She directed the meSch project. d.petrelli@shu.ac.uk

back to top  Footnotes

Overview: https://www.youtube.com/watch?v=DReu2J7eWx4

Visitor experiences: https://www.youtube.com/watch?v=E-g7jmy9UQM&t=22s

back to top 

©2019 ACM  1072-5520/19/09  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2019 ACM, Inc.

Post Comment

No Comments Found