Making/breaking

Augmenting Media Experiences with Affective Haptics

Within our Distributed and Interactive Systems research group, we focus on affective haptics, where we design and develop systems that can enhance human emotional states through the sense of touch [1]. Such artificial haptic sensations can potentially augment and enhance our mind, body, and (virtual) social connections. In three works—voice communication, news consumption, and virtual embodiment—we explore the effects of enriching media experiences with thermal and vibrotactile affective haptics, and how such stimulation influences our affective perception.

  Thermalwear

In ThermalWear [2] (Figure 1), we explore how a Peltier element and heat sink embedded in a vest worn on the upper chest can thermally augment the valence of neutrally spoken voice messages. This requires fast thermal stimulation, given the brief duration of voice messages. We do this by controlling the Peltier behavior using a custom-built ESP microcontroller. Given wearability and thermal conductivity properties, we designed a neoprene vest with a silk cutout, avoiding direct Peltier-to-skin contact. We learned that warm stimuli increase the perceived valence of voice messages, while cool stimuli lower it.

Figure 1. (a) ThermalWear hardware components; (b) Peltier embedded in neoprene vest; [c] participant experiencing the ThermalWear prototype.

  Feelthenews

In FeelTheNews [3] (Figure 2), we built on our earlier prototype, with the addition of vibrotactile stimulation using a Lofelt L5 actuator. We explored augmenting one-minute news videos with haptic stimulation. Stepping away from wearability and fast thermal stimulation requirements, this prototype houses the hardware in a box that users can place their hand and forearm on. We designed three stimulation patterns: a) matching, where vibration mimics audio and temperature matches video valence; b) negative, with 70Hz vibration and 20°C temperature; and c) positive, with 200Hz vibration and 40°C temperature. In contrast to ThermalWear, we find no statistical differences in perceived valence ratings. Participants mentioned actively suppressing the induced sensation, or being distracted by the video. Some mentioned that inappropriate intensification of the news content and neutralization of the emotional impact of the video content also played a role.

Figure 2. (a,b) FeelTheNews hardware components; (c,d) pilot participants experiencing the FeelTheNews prototype.

  Hapticbiosignal-Proxemics

Whereas in FeelTheNews we had fixed haptic stimulation intensities, we wanted to explore how haptics using varying intensities influence users under virtual reality proxemics settings. In HapticBiosignalProxemics (Figures 3 and 4), we are exploring how equipping an embodied virtual agent in VR with artificial biosignals (vibrotactile heart rate and thermally actuated body temperature) influences users' interpersonal distance (IPD) and affective perceptions. Here, we embedded the Peltier element in a neoprene and silk armband to ensure suitability for VR. For vibrotactile stimulation, we used the Oculus Touch's internal actuator. The stimulations became more intense (i.e., higher amplitude vibration and warmer) as users approached an agent. One key finding is that while thermal stimulation decreased objective but not subjective IPD, vibrotactile heartbeats increased both.

Figure 3. Peltier element prototyping: (a, b) the initial design of the armband using tape to hold the element in place, which was later upgraded to (c) a 3D-printed casing.

Figure 4. (a) Hardware components; (b) participant's view of a trial with a male agent telling a positive story at a large interpersonal distance; and (c) participant's view of a trial with a female agent telling a negative story at a short interpersonal distance.

We learned that warm stimuli increase the perceived valence of voice messages, while cool stimuli lower it.

From these projects, we learned that augmenting media experiences using affective haptics is complex. Evaluations revealed several areas for design: high variance across human skin sensitivities; differing preferences for actuator body placement and social acceptability issues; undesirable effects and secondary interactions (e.g., moving in VR makes you sweat); misinterpretation of the actuation intent; and the complexity of experimenting with and creating custom hardware to meet design requirements. These observations, we believe, pave the way for the next phase of exploration and more human-centric experiential research on combined thermal and vibrotactile affective haptics.

  References

1. Tsetserukou, D., Neviarouskaya, A., Prendinger, H., Kawakami, N., and Tachi, S. Affective haptics in emotional communication. Proc. of 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops. IEEE, 2009, 1–6. DOI: 10.1109/ACII.2009.5349516.

2. El Ali, A. et al. ThermalWear: Exploring wearable on-chest thermal displays to augment voice messages with affect. Proc. of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2020, 1–14; https://doi.org/10.1145/3313831.3376682

3. Ooms, S., Lee, M., Cesar, P., and El Ali, A. FeelTheNews: Augmenting affective perceptions of news videos with thermal and vibrotactile stimulation. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2023, Article 137, 1–8; https://doi.org/10.1145/3544549.3585638

  Authors

Simone Ooms obtained her master's degree in industrial design at Eindhoven University of Technology while collaborating with Centrum Wiskunde & Informatica (CWI), the Dutch institute for computer science, on haptic and virtual reality research. As a Ph.D. student at Utrecht University, she continues to explore technological interventions for mental health applications for teenagers. [email protected]

Thomas Röggla is a software engineer in the Distributed and Interactive Systems group at CWI. He has an M.Sc. degree in computer science from VU Amsterdam and has worked on projects involving broadcasting, media production, physiological sensing, visualization, and cocreation. [email protected]

Pablo Cesar leads the Distributed and Interactive Systems group at CWI and is a professor of human-centered multimedia systems at TU Delft. His research focuses on measuring and evaluating the way users interact and communicate with one another using a wide range of decentralized digital systems. [email protected]

Abdallah El Ali is an HCI research scientist in the Distributed and Interactive Systems group at CWI. He leads the research area of affective interactive systems, where he focuses on ground-truth label-acquisition techniques, emotion understanding and recognition across the reality-virtuality continuum, and affective human augmentation using physiological signals. [email protected]

Copyright held by authors

The Digital Library is published by the Association for Computing Machinery. Copyright © 2024 ACM, Inc.