FeaturesSpecial topic: the meSch project

XXVI.5 September-October 2019
Page: 40
Digital Citation

Experimenting around IoT for heritage

Mark Marshall, Albrecht Schmidt, Thomas Kubitza

back to top 

As museums and cultural heritage sites face constant pressure to offer an exciting visitor experience, they are consistently looking for new ways to quickly create engaging interactions for the exhibition floor. Yet most heritage professionals are not experts in the creation of such experiences; rather, they have expertise in particular topical domains and also understand the needs of visitors. Most of the time they are cut off from the process of creating truly interactive installations because of the complexity of the task. Even envisioning new experiences and communicating the ideas to others to implement is beyond their abilities if this implies an in-depth knowledge of technologies.

back to top  Insights


Our goal in the meSch project then was to lower the entry barrier to exploring, experimenting, envisioning, prototyping, and deploying tangible and embodied interactive experiences—to create ideas and quickly deploy them in place within the heritage site or museum. We also aimed to empower heritage professionals to become part of the creative team, to make them able to negotiate and collaborate with developers and designers in the creation and deployment of the interactive installations.

In several use cases (see Demo Hour in this issue), we have demonstrated that the meSch technology can deliver exactly this. It is designed to allow an easy prototyping and deployment process, with a low entry barrier and a high ceiling, allowing a clear path from ideation to deployment. The meSch technology architecture (Figure 1) consists of 1) a local server, which stores the code defining the functionality and interactions. This also facilitates the connection to the various local interactive devices. Then 2) the local devices themselves (whether inputs such as NFC readers or distance sensors, or outputs such as loudspeakers or projectors), 3) a cloud-based server that hosts the content, and 4) an online application to allow for the creation of interactive experiences and the deployment of them to the local server (see [1] for more technical detail).

ins02.gif Figure 1. The architecture of the meSch platform.


Together, these components allow heritage professionals to create, deploy, and manage interactive experiences, based on existing base recipes—previously developed experiences, tested in other heritage settings and described as a set of ingredients and instructions. These recipes can be quickly and easily cloned and modified to work with different input and output devices, or to display different content. With more specific requirements, the recipes can be further modified at the code level. This allows multidisciplinary teams comprising heritage professionals, designers, and programmers to transition from plain reuse (the heritage professionals upload new content for the hardware configuration defined in a recipe), through configuration (the designer changes the hardware defined in a recipe for the same content to map onto a new device), to parameterization and programming (the programmer specifies the interactive behavior for a new hardware configuration).

back to top  Composing Interactions

One of the biggest challenges for the use of IoT for interactive experiences in cultural heritage is in the creation and maintenance of these experiences. Heritage professionals are not software engineers, so requiring them to write code is a clear barrier to the adoption of such technology. While some projects have looked at the use of visual programming languages and end-user programming, we instead focused on the creation of an editor that would allow them to compose interactions—to get a basic experience up and running—without the need to code. While the capability is there to modify or write code, heritage professionals can instead simply reuse, reconfigure, and/or change parameters of existing experiences. Together with features such as programming by demonstration [2], we enable heritage professionals to create, deploy, and test engaging interactions quickly and easily, allowing them to focus on the experience itself rather than the code.

This editor presents a magazine of deployed recipes from which users can select the one that implements the interaction in which they are interested (Figure 2). Once a recipe has been chosen, it can then be customized based on three aspects: content, appliances, and behavior. The content editor (Figure 3) specifies all of the multimedia available for delivery in the interaction (be it video, audio, images, text, or any other form of more complex output such as quizzes or trails); the appliance editor (Figure 4) covers the physical devices (including sensors and actuators) that trigger or present content; and the behavior editor allows for a level of customization of how this triggering occurs. Each of these aspects are linked through a process of tagging.

ins04.gif Figure 2. The meSch magazine.
ins05.gif Figure 3. The meSch content editor.
ins06.gif Figure 4. The meSch appliance editor.

back to top  Concealing Complexity

Tagging allows users to quickly and easily connect content, appliances, and behaviors. Content items are audio, video, images, or text files to be delivered by the system. Appliances are input and output devices that trigger the delivery of content, such as NFC tags and readers, beacons, projectors, loudspeakers, and printers. Behavior provides simple, meaningful controls to modify parameters of the system, such as thresholds for input devices.

A particular content item is associated with one or more tags. Each appliance is also associated with one or more tags. When appliances are activated, their tags are checked and the corresponding content items are delivered. For example, should an NFC that has been tagged "English" be placed on an NFC reader tagged "Introduction," then the content item tagged with both "English" and "Introduction" will be played on any output appliances (e.g., a projector) also tagged with "English" and "Introduction."

Should the heritage professional wish to add another class of content, such as a translation to German, then they need only upload the content file, add a "German" tag to this content, and also to an NFC tag and a display device. This new NFC tag will then trigger the German content, without the need for writing code.

This method offers the advantage of simplicity. No programming is necessary, changes can be easily made, and the system can be extended. Heritage professionals can quickly and easily create interactive experiences and deploy them to their particular site. It also offers a level of extensibility, in that output devices may support multiple types of content. A device being used to display text and images may also be able to present audio and video, so the heritage professional can mix these types of content, or initially work with text and later simply add video content once it becomes available. New content is simply added to the editor, tagged correctly, and deployed. This approach also allows for the straightforward testing of an installation while creating it.

The strengths of this approach lie in the flexibility and simplicity that it offers. Users can reuse content in different contexts, delivering it using different hardware, or can use the same hardware configuration to deliver different content, or a different organization of the same content. Users can add new tags to a recipe, and this extends to any possible scenario. We have successfully used the tagging system to create interactive experiences that range from the presentation of fixed content, to content that changes based on the previous actions of the visitor, to quizzes and treasure hunts. This balance between flexibility and simplicity allows heritage professionals to create and explore a huge range of potential interactions and to customize them to their own specific goals, audience, and context. A tagging approach that allows multiple tags per item is very easy to understand and scales well to larger systems, as relationships are implicitly managed with the tags.

back to top  Instantaneous Deployment

When developing new interactive museum experiences, the ability to make and test changes without too much effort can be extremely important. Content or interactions may not work as expected in situ. This can be a particular issue on an outdoor site, or heritage sites that have unusual environments where environmental factors (light levels, background noise) can affect the content being displayed. It may also be necessary to tune parameters of input devices to meet the requirements of the space, such as setting thresholds on distance, presence, or activity sensors. So, for example, the meSchCases (Figure 5) made use of infrared distance sensors to detect visitors in front of the cases and thus estimate the interest level in the objects on display in each case. The threshold for detection of presence using these sensors had to be tunable to allow for the context of the space in which they were deployed—to ensure that only visitors standing directly in front of them were counted and not those en route to another display. By providing a straightforward method of modifying, deploying, and testing the interactive experience, we can enable heritage professionals to tune the experience to their own space and audience.

ins07.gif Figure 5. The meSchCases on display at Museon in the Hague, Netherlands.

In our case, this is enabled through three specific functions: appliance detection, behavior modification, and instantaneous deployment. Appliance detection lets users easily add new input and output devices to the system. When a new meSch-enabled device is turned on, it broadcasts its availability and its specification to the local server, which forwards it to the cloud-based server. The device then appears in the Appliances tab of the online editor, ready to use. As all devices describe themselves, it becomes straightforward to include them in the current experience. If multiple similar devices are present, we offer identification by interaction to pick a specific device to be selected (see [3,4]). For example, the developer interacts with the meSchCase she wants to change and consequently gets the direct reference to this object.

Behavior modification lets users quickly change a parameter of the system behavior. These parameters are defined in the recipe itself and displayed to the user in the form of input boxes and sliders. So, for example, if a presence-activated projection is triggering too soon, the user can slide the sensitivity slider to the left, deploy the recipe to the local server, and test with the new setting. This can be repeated until the correct setting is found. This feature was used in a number of the experiences developed during the meSch project, including the already discussed meSchCases and the Movement in Greek Art exhibition at the Allard Pierson Museum in Amsterdam.

This brings us to the instantaneous deployment function. This lets users perform a simple, one-click deployment of the recipe, including its behavior and content, to the local server. When the user clicks Deploy in the online editor, the cloud-based server sends the recipe and its associated files to the local server. In an effort to notify the user of the updated recipe being installed, the local server uses a sound icon—it beeps three times—to signal that the interactive experience is updated and can be tested.

Heritage sites are often outdoors, and others are in old, even ancient, structures built of materials ill suited to wireless networking.

The goal is to make the process of creating, editing, and deploying interactive experiences as smooth as possible. Heritage professionals should not need to deal with issues such as installing new appliances manually, editing code, or copying code or content to specific devices. All of this is managed by the meSch system so the heritage professional may concentrate on creating and optimizing the experience for their site and visitors. Many iterations can be made in a short time, similar to compiling while programming. Hence the creators can develop the experiences step-by-step by always having a working system, similar to agile development in software engineering.

back to top  The Challenges of IoT in Cultural Heritage

Over the course of this project, we have worked on a number of installations in heritage sites, ranging from single interactive installations [2] to complete interactive exhibitions [5]. We have subsequently found that heritage sites offer interesting challenges for the use of IoT.

First and foremost: the issue of infrastructure. Many of the heritage sites we have worked with have had restrictions on infrastructure, including little or no access to main power or minimal network connectivity. Where possible, we have attempted to mitigate these restrictions, whether through the use of battery-powered devices, the creation of our own network for meSch systems to communicate on, running networks over existing power lines, or the use of 3G/4G modems to allow for Internet connectivity during installation, configuration, and testing. As a further step toward reducing issues around network connectivity, the meSch system runs the deployed recipes and content from the local server: Once the system is up and running, an Internet connection is no longer required.

The physical and environmental characteristics of heritage sites can also pose a challenge. Heritage sites are often outdoors, and others are in old, even ancient, structures built of materials ill suited to wireless networking. In one particular case, we created an exhibition that ran in an underground bomb shelter hewn into the side of a hill (Figure 6). This space was cold, always quite damp, as well as dark and cramped. The thick stone walls did not allow wireless networking. We made use of powerline networking to connect individual components, as well as added a number of electrical safety features to the installed system.

ins08.gif Figure 6. A meSch installation in a bomb shelter in Rovereto, Italy.

Another challenge can revolve around issues of maintenance, startup, and shutdown. Many heritage sites lack dedicated IT support, so any potential system should be easy to start, stop, and maintain. For example, during an early experiment with the meSchCases we discovered that the museum in which they were being installed did not shut down or start up individual interactives each day; rather, power was removed or applied to the entire floor of the exhibition. This meant that any interactive device would need to automatically start upon the application of power and safely shut down upon its removal. As a result, most of the complex appliances supported by the meSch system now incorporate a small battery backup and shutdown system to manage safe shutdown and restart in such situations.

back to top  Conclusion

The use of IoT in cultural heritage offers both challenges and possibilities [6]. Our goal within the meSch project was to create a system that would allow heritage professionals to explore the possibilities on offer—one that would let them experiment with IoT technology and create a series of innovative interactive experiences. The system was designed so that users could craft these experiences without knowledge of the underlying technologies, and particularly without having to write code. Instead, heritage professionals can focus on choosing the type of interaction they wish to provide and creating and tagging the content that will be presented to their visitors, something that is significantly more important to them than the technology itself.

By abstracting the underlying code away and instead using a tagging system, we help heritage professionals to create these experiences. Through our experiments with heritage sites across Europe, we have been able to incorporate and test features to further lower the bar to the use of IoT in cultural heritage, resulting in interactive experiences that are easier to create, that can be quickly modified and tested, and, perhaps even more important, that will work more reliably within heritage sites.

back to top  Acknowledgments

meSch (2013–2017) received funding from the European Community's Seventh Framework Programme, "ICT for access to cultural resources" (ICT Call 9: FP7-ICT-2011-9) under the Grant Agreement 600851.

back to top  References

1. Kubitza, T. and Schmidt, A. meSchup: A platform for programming interconnected smart things. Computer 50, 11 (2017), 38–49.

2. Marshall, M.T., Dulake, N., Ciolfi, L., Duranti, D., Kockelkorn, H., and Petrelli, D. Using tangible smart replicas as controls for an interactive museum exhibition. Proc. of the 10th International Conference on Tangible, Embedded, and Embodied Interaction, ACM, New York, 2016, 159–167.

3. Kubitza, T. and Schmidt, A. Rapid interweaving of smart things with the meSchup IoT platform. Proc. of the 2016 A CM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, New York, 2016, 313–316.

4. Kubitza, T. Apps for environments: Demonstrating pluggable apps for multi-device IoT-setups. Proc. of the 6th International Conference on the Internet of Things. ACM, New York, 2016, 185–186.

5. Petrelli, D., Dulake, N., Marshall, M.T., Roberts, A., McIntosh, F., and Savage, J. Exploring the potential of the internet of things at a heritage site through co-design practice. Proc. of the 2018 Digital Heritage International Congress. IEEE, 2018.

6. Marshall, M.T. Interacting with heritage: On the use and potential of IoT within the cultural heritage sector. Proc. of 2018 5th International Conference on Internet of Things: Systems, Management and Security. IEEE, 2018, 15–22.

back to top  Authors

Mark Marshall is a lecturer in software engineering at Sheffield Hallam University in the U.K. His research focuses on the design, creation, and evaluation of novel user interfaces across domains including manufacturing, healthcare, education, heritage, and the arts. m.marshall@shu.ac.uk

Albrecht Schmidt is a professor of computer science at Ludwig-Maximilians-Universität München. He holds a chair in human-centered ubiquitous media and is working on novel interactive technologies and media. His research interest is the amplification of the human mind through digital technologies. He received a Ph.D. in computer science from Lancaster University. albrecht.schmidt@ifi.lmu.de

Thomas Kubitza is a Ph.D. candidate at the University of Stuttgart and cofounder and CEO of ThingOS GmbH. His research interests include the Internet of Things, cross-device interaction, and end-user development. He received an M.Sc. in computer science from the University of Duisburg-Essen. thomas.kubitza@ThingOS.io

back to top 

Copyright held by authors. Publication rights licensed to ACM.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2019 ACM, Inc.

Post Comment

No Comments Found