Laewoo Kang, Steven Jackson
Can art practice be a method for HCI research and inquiry? Over the past several years, work by our lab [1,2] and others [3,4] has highlighted the value and possibility of experimental collaborations with artists as a method of HCI research and design. By offering opportunities to integrate aesthetic and theoretical inquiry into more multidisciplinary platforms, such collaborative artistic encounters have suggested alternative approaches to problem discovery and theory building, and revealed sometimes surprising sites of creativity and knowledge in HCI research and design. In our lab, collaboration with artists and musicians has helped shape methodologies and heuristics that may elude more behavioral, engineering, or traditionally ethnographic approaches.
In this article, we introduce three such projects and discuss the potential of participatory and collaborative art interventions as a method of HCI research and inquiry. Since 2013, our research team has worked with multimedia artists and experimental musicians to produce hybrid objects that are at once collaborative forms of art production and exploratory sites in which questions core to the field are addressed in playful, suggestive, or speculative ways. Members of our lab at Cornell have engaged in these collaborations in multiple and mixed roles, from multimedia artist and engineer to ethnographic researcher and project manager. Through ethnographic interviews, participant observation, reflective building, and performative practices, we have simultaneously built and studied the process of these collaborations. Here we report on central themes and findings emerging from this work and explain how collaborative and reflective art-making practices can extend HCI notions of creativity, collaboration, and design.
Scale  is a collaborative art project derived from a longer series of repair studies [5,6] that explores the creative potential embedded in the fixing and repurposing of broken and obsolete technologies (Figures 1 and 2). It began in fall 2013 with an interview study of 17 artists (including circuit-bending musicians, media artists, and hackers) who engage in processes of technological breakdown and repurposing as central methods or topics of their work. Throughout the course of this project, our research team built a relationship with Brooklyn-based visual artist Taezoo Park (http://www.taezoo.com/), whose Digital Being series shared a particular affinity with our prior theoretical work and early ethnographic findings.
|Figure 1. Scale, a collaborative artwork with Taezoo Park.|
|Figure 2. In Scale, the audience’s interaction turns a pile of e-waste into an aesthetic installation.|
In collaboration with Park, in summer 2014 we developed an interactive installation created from more than 100 examples of broken and discarded technologies, sourced from local recycling centers, reuse-it shops, and street corners. These were then fixed, hacked, or repurposed, and, through the intervention of Arduino and other DIY tools, reconfigured into the Scale installation. When an audience member steps on a repurposed scale at the front of the installation, this pile of broken and lifeless objects is set in motion, triggering odd functionalities, surprising connections, and eerie or troubling forms of beauty. Through such fleeting and transitional experiences, the installation suggests a space of play, reflection, and speculation in which broken and wasted items take on “lives” and futures beyond their original functions and contexts.
Over the course of a yearlong collaboration, we participated in three separate installations: at the 2013 World Maker Faire in New York; at the Olive Tjaden Gallery in Ithaca, New York; and at ACM CHI 2014 in Toronto (images and footage from these various installations can be found at the project website: http://cornellhci.org/scale/). It also led to countless interactions with audience members ranging from fixers, hackers, and DIY hobbyists to HCI and other researchers, all of whom shared with us their stories, problems, and critical reflections on technology breakdown, obsolescence, and reuse. This integrated experience of making, reflecting, and exhibiting drove conversations and understandings around repair in ways unavailable to us through design and ethnographic work.
With this experience fresh in hand, in February 2014 we launched a follow-up collaboration with the intent of modeling new and more symmetrical forms of engagement between media artists and social science theorists. To that end, we recruited three New York City–based media artists (Taezoo Park, Hyuns Hong, and Sukmo Koo) and asked them to work with ideas and concepts generated by three scholars of infrastructure studies (Geoffrey C. Bowker, Paul Edwards, and David Ribes). Our goal was to chart new passages between the theoretical and the aesthetic-collaborative, and to support new mixed-media conversations in which words would be answered by things, and vice versa.
Following a schematic designed (we thought) to reflect these principles, our collaboration adopted the following steps. Artists were invited to read selected papers offered by our infrastructure participants and then work collaboratively to produce prototypes reflecting, engaging, or disputing ideas encountered in the readings. Artists and theorists would then meet via teleconference to discuss the relationship between readings and prototypes, along with any other thoughts or ideas provoked in the encounter. Artists would then produce a final collaborative version of the prototype, now inflected by the additional round of feedback. Members of our research team were once again engaged on both sides of this collaboration, participating in the making of the artwork, interfacing with the “theory” participants, and conducting ethnographic study throughout in the form of participant observations, interviews, and reflective diary studies.
However, unlike the Scale project, this collaboration led to neither significant installation success nor discernible theoretical discovery or insight. Part of the difficulty seemed to stem from the radically different senses of collaboration and the relationship between ideas and practical work (whether material or textual) in our two participant groups. Whereas the theorists tended to look at the prototypes as implementations or reflections of ideas that lived primarily in text, the art group appeared to proceed through more intuitive and improvisational practices whose commitment to theoretical ideas seemed uncertain and subject to change in the face of emergent material and aesthetic opportunities. While the work did eventually lead to a collaborative sculpture (This Is Not a Television; Figures 3 and 4), its relationship to theoretical ideas remained unclear and it did not have the kind of installation success achieved by the earlier Scale project, with public display limited to a one-week exhibition at the FIG Fashion Show in New York in February 2014. Post-hoc interviews and reflections suggested that this outcome was in part due to a failure of common ground, and the time required to build mutual understanding, reciprocity, and commitment across the disparate worlds of the artists and researchers—a fact exacerbated by the tightness of the pre-designed structure and schedule.
|Figure 3. The collaborative process of This Is Not a Television.|
|Figure 4. The final version of This Is Not a Television.|
From the above two projects, we learned much about the complex and improvisational nature of collaboration, and how collaborative ideas and objects may be both constructed and transformed in the face of ongoing situational factors like materials, tools, and sites, as mediated through networks of trust, shared vision, and mutual commitment. This led to further curiosity around how artists use improvisation as a creative and collaborative method. What features or activities of improvisation enabled successful results in some collaborative art settings but not others? How could we incorporate this insight into the theoretical and methodological puzzles of HCI?
In 2015, we launched what would become the Intermodulation project with a series of interviews with 10 musicians actively employing improvisation as a method of musical discovery and performance. Through relationships established in the interviews, we then had the opportunity to collaborate with two groups of experimental musicians—the Electric Golem (http://levelgreen.com/electricgolem/) and Powerdove (http://annielewandowski.com/)—with the goal of producing public audio-visual performances in which the musicians would perform in concert with the interactive art installations created by members of the research team.
What features or activities of improvisation enabled successful results in some collaborative art settings but not others?
In both cases, plans and expectations of final performance were not pre-arranged in detail but rather emerged and evolved over the wider course of the interaction. As the installation remained under construction through early stages of collaboration (and was tweaked and revised all the way to the end), neither musicians nor artists were able to think toward the final results in a fully determinate way. Other important factors such as performance spaces, stage settings, and playlists also remained undecided in the early stages of collaboration, taking form and shape only late in the process. Ultimately, members of our lab developed a series of three interactive artworks—Breaking AndyWall (audio version, Figure 6), echo I (Figure 6), and Intermodulator (Figure 5)—that were performed live with the musicians over the course of a series of performances in Ithaca and Brooklyn (both in New York) from November 2015 to September 2017. More videos and images can be found on the project website: http://cornellhci.org/intermodulation.
|Figure 5. Members of Powerdove experiment with Intermodulator.|
|Figure 6. echo I (left) and the interactive audio version of Breaking AndyWall (right).|
Through this lengthy process, we learned to reflect on the interdependent nature of collaborative improvisation, along with the arts of active listening and situational adjustment by which participants modulated, played with, and departed from standard or intended practices. We also learned how practices of collaborative improvisation depended not only on stable and harmonious relations with trusted partners, but also forms of risk and tension produced from the encounter with discrepant and discordant voices and visions. This lesson strongly inspired the final design of the Intermodulator, built around the moiré effect, produced by two fans moving in opposite directions. It also led to wider theoretical reflection on features of collaborative improvisation that might be brought more centrally into HCI research and design.
How can we employ artistic and experimental collaboration as a method for HCI research and inquiry? And what are the advantages and limitation of this style of research? The program of work above suggests three general answers to these questions.
First, our experiments have shown how art collaboration can be paired with ethnographic approaches to produce more resolutely socio-material forms of research practice. Vis-à-vis more traditionally ethnographic forms of HCI work, this approach goes beyond simple interview or observation to include rich material engagements, aesthetic explorations, and complex social interactions. Such integrated and holistic approaches offer HCI researchers deeper and more multidimensional access to the collaborative and creative processes under consideration, and provide richer insight into how human actors, materials, technologies, and situations are related and entangled.
Second, our work has shown how artworks produced from collaborative and critical socio-material practices can themselves support sites and objects of HCI inquiry through which people can engage topics of research in simultaneously aesthetic and critical ways. Beyond findings embedded in careful organization and critical reflection on practice, the artworks themselves can provide audiences heuristic spaces in which they can engage and discuss questions of HCI concern.
Finally, in line with other findings from studies of creativity and collaboration, our studies reveal the complex and improvisational nature of much collaborative work, wherein findings and products emerge from engagement with specific, material, and sometimes accidental encounters. Research approaches built around collaborative art practice may share these constructive and open-ended tendencies, embedding forms of flexibility that may exceed and challenge traditional linear models.
But such forms of improvisation and open-endedness are not without limit. Since the collaborations remain a part of research, artistic practices and outcomes may be subject to forms of structure and accountability less prevalent in a purely aesthetic environment. Interactions between the intuitive freedoms of art practice and the discipline of research structure may produce forms of emotional and intellectual tension, both welcome and unwelcome, that in turn invoke risk but also the possibility of learning and discovery for both artists and researchers. Our work has attempted to provide useful models of an open-ended and multidisciplinary research practice wherein artists and researchers can experiment, play, and learn together, arriving at findings, outcomes, and forms of expression unavailable to each individually. We believe such efforts represent modest but useful steps in the ongoing effort to deepen and extend the methodological and imaginative toolkit of HCI.
1. Kang, L., Jackson, S., and Sengers, P. Intermodulation: Improvisation and collaborative art practice for HCI inquiry. Proc. of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2018.
2. Kang, L., Park, T., and Jackson, S. Proc. of the Extended Abstracts of the 32nd Annual ACM Conference on Human Factors in Computing Systems. 2014, 399–402; http://doi.org/10.1145/2559206.2574831
3. Benford, S., Greenhalgh, C., Crabtree, A. et al. Performance-led research in the wild. ACM Transactions on Computer-Human Interaction 20, 3 (2013), 14:1–14:22; http://doi.org/10.1145/2491500.2491502
4. Brynjarsdyttir Holmer, H., Disalvo, C., Sengers, P., and Lodato, T. Constructing and constraining participation in participatory arts and HCI. International Journal of Human Computer Studies 74 (2015), 107–123; http://doi.org/10.1016/j.ijhcs.2014.10.003
6. Jackson, S.J. and Kang, L. Breakdown, obsolescence and reuse. Proc. of the 32nd Annual ACM Conference on Human Factors in Computing Systems. 2014, 449–458; http://doi.org/10.1145/2556288.2557332
Laewoo (Leo) Kang is a Ph.D. candidate in the information science department at Cornell University as well as a multimedia artist (http://www.laewoo.com). His work explores art-research integration in the fields of human-computer-interaction (HCI), design, and media art. firstname.lastname@example.org
Steven J. Jackson is an associate professor of information science and science and technology studies at Cornell University (https://sjackson.infosci.cornell.edu/). His work centers on questions of value, innovation, and governance in contemporary computing practice, with special emphasis on problems of infrastructure, collaboration, sustainability, and repair. email@example.com
©2018 ACM 1072-5520/18/03 $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.