ACM Interactions

During the 2023 CHI conference, we ran a one-day workshop to consider the design space at the intersection of the body and materials [1]. The workshop gathered designers, makers, researchers, and artists to explore current theories, approaches, methods, and tools that emphasize the critical role of materiality in body-based interactions with technology. We were motivated by developments in HCI and interaction design over the past 15 years, namely the “material turn,” which explores the materiality of technology and computation and methods for working with materials, and “first-person” approaches emphasizing design for and from lived experience and the physical body. Recognizing the valuable contributions of approaches that foregrounded materiality [2] and the body [3] in HCI, we proposed to explore the intersection of these two turns. 

Material interactions is a field of wide-ranging interest to HCI researchers, starting with research on the design and integration of physical and digital materials to create interactive and embodied experiences [4], progressing to a growing interest in the contextual aspects of material interactions and their impact on personal and social experiences [2], and culminating in formalized method propositions such as “material-centered interaction design” [5]. The latter is a fresh approach, urging interaction designers to broaden their view beyond the capabilities of the computer and embrace a practice that imagines and designs interaction through material manifestations. Similarly, the role of the body in design has gained traction, through approaches such as soma design [3], embodied interaction, and movement-centric methods [6]. These have positioned the centrality of first-person accounts and physical engagement to research and design from the body, shifting the design space beyond security and efficiency, which are primary goals, for example, in the field of human factors, to deal with other dimensions such as fun/joy, play, entertainment, mental health interventions, and so on. In the workshop, the methods and prototypes brought in aimed to explore experience, for instance in terms of perceptions and feelings toward materials (Figure 1) or toward the person’s own body (Figure 2). In both cases, the interaction with the prototypes brings the subjective experience into focus. We share a keen interest with previous research in working across physical and digital materials, as well as in centering the body in design. Additionally, we have a particular interest in the intersection of these approaches in which we find a new design space: making the body central material in designing experiences, going beyond a view of the body as the medium with which to explore the material—its potentials, its affordances—instead including the body as part of the material being designed with and for. We are excited to develop this design space through a formal proposition, as we progress with our research.

Our proposed approach is both timely and critical, as it addresses the complex interplay between the body and emerging technologies. However, this approach is not free from challenges, as discussed in the workshop. These technologies not only have a tangible impact but also influence immaterial aspects, creating unpredictable outcomes that must be carefully considered during the design phase. As technologies increasingly become a ubiquitous part of our lives, they affect our bodies in both physiological and psychological ways. Furthermore, moving this research beyond the confines of a laboratory not only introduces ethical concerns in relation to health, well-being, safety, and social impact, but also opens new opportunities. For example, in terms of social impact, the technologies can directly affect not just individual bodies, but also the interpersonal relationships those individuals maintain. Therefore, any interaction with embodied technologies carries real-world implications that must be approached with caution, but also with curiosity about the opportunities they may open. 

The Workshop
Throughout our day-long event, attendees took part in various activities and were organized into four thematic groups: material enabling expression; material as a catalyst for human action; material enabling reflection, awareness, and understanding; and material supporting the design process for (re)creating the existing and the yet-to-exist.

In the first activity, participants shared their prototypes or methods within their distinct groups. Prototypes were examined in relation to each group’s theme and participants individually noted similarities, differences, and opportunities among them. The second activity mixed groups for participants to discuss key points across themes, to uncover overlaps and opportunities. 

<
Figure 1. One of the methods (Materials Gym) presented at the workshop. EMG is used alongside a smartphone application to continuously collect data on people’s interaction with materials: Speed and aspects of movement reveal information about the experience in relation to people’s perceptions of and preferences for materials qualities (see https://discovery.ucl.ac.uk/id/eprint/10172204/).

Figure 2. A physical prototype (Soniband) that one participant brought to the workshop. This is a wearable band that provides real-time sonification of movement through a variety of sounds, some of which build on material metaphors, such as water, wind, or mechanical gears, and which can impact on the wearer’s body perceptions and feelings, movement, and emotional state (see https://doi.org/10.1145/3411764.3445558; https://doi.org/10.1145/2858036.2858486)

To increase enthusiasm into our exploration, the workshop featured a panel with four guest speakers who presented their research through a provocation: Kristii Kuusk (Estonian Academy of Arts) talked about the impact of material on people’s bodies combining affordances of technology and traditional materials; Pedro Lopes (University of Chicago) talked about making the body the material from which we build the actuator of the designed technologies; Hasti Seifi (Arizona State University) talked about the need for integrating basic research on haptic sensory perception and language into software tools for designing body-based haptic experiences; Paul Strohmeier (Max Planck Institute for Informatics) talked about how material and body experiences are shaped by agency, control, and reflective processes. The panelists conversed about material experiences and engaged in a Q&A session with participants. To conclude, participants mapped the design space of “body x materials” with their original groups, considering challenges, opportunities, state-of-the-art, theories, and methodologies.

Figure 3. Participants engaging in workshop activities.

The diversity of backgrounds of workshop participants (position papers available at www.rca.ac.uk/body-materials) resulted in multidisciplinary discussion groups where different disciplines, theories, methodologies and application domains were considered (including HCI, interaction design, neuroscience, arts and crafts, and material design). Through discussions, this diverse group of participants arrived at shared conclusions. Participants highlighted the importance and timeliness of the topic. Many emphasized the need to establish a clear vision of the relationship between the body and materials for HCI. They foregrounded the challenges posed by the complexities of reconciling detailed fine-grain descriptions with a holistic perspective of the overall experience, which arise both from the dynamic nature of the experience of materials, conveyed through sensorimotor feedback, and the continuous interpretation of the material qualities involved in the act of touching. Understanding the underlying processes leading to these interpretations and how awareness levels affect interaction was perceived by participants as critical. Several participants stressed the importance of incorporating first-person perspectives into design.

The discussion spotlighted additional challenges stemming from language limitations and the necessity of cultivating a language specific to touch. Two key points emerged from this discussion. First, participants underscored the importance of establishing a shared vocabulary that enables effective communication among researchers who may have diverse backgrounds. Second, it was considered that in the translation of touch experiences into verbal descriptions significant information tends to be lost. Participants also explored the optimal data format for representing body-based research, acknowledging the existence of ongoing attempts. 

Finally, the topic of shifting agency between the body and materials was explored. Two approaches were highlighted: empowering individuals to create their own unique material experiences as part of the design process, and the concept of “controlling the body to experience something: where the body acts as an actuator. The tension between enabling individuals to engage with the experience and exerting control over the body for a specific experience was also acknowledged during the discussion.

For those interested in working in this novel design space at the intersection of body x materials, we synthesize three key takeaways from our workshop. These takeaways emphasize: 1) the significance of comprehending the fundamental processes underlying experiences within this intersection, 2) the need to employ appropriate methods to achieve this understanding, including the integration of first-person perspectives into the design process, and 3) the importance of establishing a shared vocabulary. The latter includes facilitating the translation and representation of experiences and research outcomes related to the body x material interactions.

Conclusions
Everyone agrees that mapping the state-of-the art of the body x materials space is necessary for creating a more significant impact beyond lab experimentation. Nevertheless, a cautionary note was also struck—participants noted that it is essential to slow down and account for the ethical implications of how this research affects people psychologically and otherwise. To move the field forward and generate impact, participants agreed that interdisciplinary conversations must be fostered and the real-life implications of materiality x body understood, both in terms of possibilities and positive and negative effects of our work. The atmosphere at the workshop was open and interdisciplinary, and it was clear among participants that more such interdisciplinary conversations around the real-life implications of body x materials need to be nurtured to bring a shared understanding to the field. With this aim, the workshop participants planned to consolidate existing materials and findings at future workshops, including one that was run at the IEEE World Haptics 2023 conference and attempt to have a repeat workshop at CHI 2024 to further explore the directions that emerged. The long-term goal is to build an interdisciplinary community and open the design space for material-enabled, body-based multisensory experiences by integrating research from various perspectives. 

Acknowledgments
We would like to thank all workshop organizers (the authors of this piece were joined by: Hasti Seifi, Judith Ley-Flores, Nadia Bianchi-Berthouze, Marianna Obrist and Sharon Baurley), guest speakers, and participants for their fantastic contributions. We acknowledge funding by: the Spanish Agencia Estatal de Investigación (PID2019-105579RB-I00/AEI/10.13039/501100011033) and the European Research (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 101002711). The work of Bruna Petreca and Ricardo O’Nascimento was funded by UKRI grant EP/V011766/1. For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) license to any Author Accepted Manuscript version arising. 

Endnotes
1. Petreca, B.B. et al. Body x materials: A workshop exploring the role of material-enabled body-based multisensory experiences. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2023; https://doi.org/10.1145/354454...
2. Giaccardi, E. and Karana, E. Foundations of materials experience: An approach for HCI. Proc. of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, 2015, 2447–2456; https://doi.org/10.1145/270212...
3. Höök, K. Designing with the Body: Somaesthetic Interaction Design. MIT Press, Cambridge, MA, 2018.
4. Ishii, H. and Ullmer, B. March. Tangible bits: Towards seamless interfaces between people, bits and atoms. Proc. of the ACM SIGCHI Conference on Human factors in Computing Systems. ACM, New York, 1997, 234–241.
5. Wiberg, M. The Materiality of Interaction: Notes on the Materials of Interaction Design. MIT Press, Cambridge, MA, 2018.
6. Wilde, D., Vallgarda, A., and Tomico, O. Embodied design ideation methods: Analysing the power of estrangement. Proc. the 2017 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2017, 5158–5170; https://doi.org/10.1145/302545...

Ana Tajadura-Jiménez

Bruna Petreca

Laia Turmo Vidal

Ricardo O'Nascimento

Aneesha Singh