Miriam Sturdee, Makayla Lewis, Nicolai Marquardt
A roomful of 28 researchers in various stages of their careers sits nervously in a brightly lit room, the tables in front of them strewn with assorted pens, paper, and Post-it notes. They are all here to see how they might improve their drawing skills and subsequently apply these skills to their own research.
"I can't actually draw," says one nervously. "Is that OK?"
"Of course it is," we reply, in chorus. "But actually, everyone can draw."
The myth that you either can or can't is what prevents most of us from ever picking up a pencil after our school days, making little effort other than the odd doodle on a notebook during meetings or phone calls. That friend you sat next to in art class who could sketch people who really looked like people, that teacher who tutted and sighed when you presented your still life... forget them. It isn't important anymore—this is about you, your ideas, and how you gain confidence in sketching.
Research in developmental psychology suggests that sketching develops along the same pathways in the brain as language , so we all have that capability as we learn and grow. But many of us simply stop when it is no longer a pleasurable, childlike activity, or when it becomes an optional subject at school. Reengaging with sketching at a later age may require more effort—or a leap of faith—but it is no more out of reach to the average HCI researcher than learning a few words in a foreign language. N'est-ce pas?
Breaking down the perceived barriers to starting again, and simply gaining confidence in sketching, are the biggest challenges. We focused on these as part of the "Applied Sketching in HCI" course launched this year at CHI , a hybrid tutorial based on previous successful events by the authors, offering a hands-on overview of how sketching can be (re)learned, utilized, and analyzed as part of the HCI research process.
The non-exhaustive list of uses for sketching in HCI spans the recording of information (visual note-taking, such as sketchnoting or visual facilitation for groups; for more on sketchnoting, see Blog@IX on page 6); journeys of self-discovery (subjective sketching practice as a method of investigation); co-creation of sketches; telling user stories; sketching as data collection; and sketching analysis or coding. In different combinations, sketching can thus support all stages of the research process. And it doesn't have to be perfect, neat, or photorealistic—"Ideas not art" is the mantra of the sketch-informed. In fact, it is the ephemeral, often lightweight nature of sketches that makes them ideal for suggestion, alternative interpretation, and evolution.
In different combinations, sketching can support all stages of the research process. And it doesn't have to be perfect, neat, or photorealistic—"Ideas not art" is the mantra of the sketch-informed.
We start our session with a basic warm-up task called mark making (lines, circles, and swirls), using a stark black pen. There are no mistakes here, so pencils and erasers are not permitted. We look at people—our users—and how they move and express themselves, and try to capture something of their humanity. We tell stories, give examples, and sketch along together (Figure 1); the atmosphere is inclusive, encouraging, and most of all, creative.
|Figure 1. A sketching course attendee's sketch of people and actions.|
We bring together a visual library of iconography relating to different disciplines within HCI, the Post-it notes spanning meters and meters of the session-room wall, suggesting different perspectives and styles (Figure 2).
|Figure 2. Everyone in the CHI sketching course joins in when creating the icon library.|
We break, return, and continue with a master class in photo tracing and scene composition. In one corner, the daughter of two participants joins in, drawing spectacular dragons with sparkling ferocity, while her parents apply their newfound skills to storyboarding (Figure 3).
|Figure 3. An attendee shares a storyboard.|
We culminate our activities with a lively session of HCI Improv (Figure 4), an audience-participation activity where teams compete to create products, services, and scenarios for randomly generated users, problems, and contexts, before turning our attention to the task of sketch analysis: how we code, analyze, and generate requirements from sketched imagery.
|Figure 4. Live sketching by Makayla Lewis during the HCI Improv.|
By the end of the session, all attendees are sketching user experiences, scenarios, and hoping to take their skills back to centers of research and learning. We pick up the papers left behind, cheered by how little remains of the created works, thus confirming their value to the participants. However, our work does not begin, or end, here.
Sketching is an unsung hero of many parts of the academic-research process but could be seen as the remit of the few, cherished practitioners with the confidence to wield pencil and paper in the public domain. However, sketching in HCI goes beyond traditional mark making; it can be an integral part of focused research projects, aimed at elaborating upon this human activity in the age of the computer.
Some years ago, we the authors came together by chance at CHI, having been introduced over social media by mutual friends who saw our shared interest and activity around sketching in research and urged us to meet. While presenting work on comics, creating sketch-noted documentation of workshops, and attending sessions, we put together the bones of a plan to bring together those already engaged in sketching research to further elucidate this area as an emergent field. Our first workshop at Designing Interactive Systems (DIS) 2017 led to the creation of the Sketching in HCI network (https://sketchingdis.wordpress.com/), bringing together researchers from areas as diverse as gamification, jewelry design, shape-changing interfaces, and InfoVis, while an entry-level tutorial we ran at NordiCHI 2016 (https://sketchinghci.wordpress.com/) proved there is a burgeoning interest in learning how to use sketching in research by a wider body of novices.
For the past few years, CHI has hosted courses in sketching, sometimes more than one. In the meantime, we've seen the rise of visual facilitation in the workshop setting, and the graphic recording of conferences has become a popular method for note-taking and reflection, engagement, and the dissemination of proceedings (Figure 5). Concern for sketching practice in HCI is also not limited to special interest groups and courses. Researchers use sketching either formally or informally in various avenues of research, evident in a number of demos and presentations at the CHI conference in Montreal this year. This interest is also not limited to process and presentation; for example, trying to breach the divide between digital and paper-based drawing has long been a focus of those who seek to unlock the mystery of why we appear to prefer visual thinking offline, or how we might merge AI and the drawn image. Others seek to harness the power of the stylus and the human mind in novel tools and systems designed to enhance user experiences.
|Figure 5. Ultra Haptics workshop sketchnote by Makayla Lewis at CHI 2018.|
Sketching can be a valuable part of all stages of HCI research, but we are perhaps most familiar with it as part of the ideation process, forming links in the mind and creating novel forms from the ether . These insights do not have to be at the hands of the researcher, either; the same encouragement can be used to elicit input from study participants for subsequent analysis or development. Sketching especially comes into its own when we summarize research  or examine things that do not yet exist, as with the interfaces of the future . All made objects were represented by thoughtful imagery at some point; by putting the power of visualization into the hands of as many researchers and participants as possible, we expand the potential of our field exponentially.
Curious, and wanting to understand and promote sketching in HCI, we documented instances and uses of sketching in all its forms at CHI, including our own contributions. We collected what we found using the #CHI2018 and #SketCHI hashtags together on Twitter (https://bit.ly/2Q5XNS2), as well as encouraging others to do so. This record remains in the public sphere and provides a visual (and sketched) narrative of the conference relating to the human-drawn image.
As part of the drive to encourage interest in sketching, we chose to bring sketching to our Alt.CHI paper session, which was already hosting group chanting, a live painting demonstration, and 15 minutes of silence. Alt.CHI is a daring collection of challenges, provocations, and explorations that live at the fringes of HCI, therefore presenting a unique opportunity to explicitly disrupt the status quo while presenting highly original works.
Under the umbrella of the newly realized #SketCHI hashtag, we sought to actively engage the audience to take part in a sketching exercise by creating a cyclical presentation-exhibition of discourse and imagery. Sketch and the Lizard King: Supporting Image Inclusion in HCI Publishing  is a comic-based analysis of image use in HCI that suggests leniency in page length or format when publishing work that would benefit from increased visuals. It culminates in a call to arms for CHI to consider changes to how it presents research.
By enabling the audience (supplying pens, pencils, paper, and Post-it notes) and presenting the talk entirely using a live Twitter feed, we solicited responsive sketches, examining and celebrating the drawn images as they arrived amid the predetermined material. Those who did not use the platform were able to submit their images to the authors after the session, and these contributions were then added to the feed. This unusual methodology not only exemplified audience participation but also ensured that the forum could continue beyond the allocated room and time. By removing the perceived barriers to sketching (lack of materials, encouragement, or confidence), we found ourselves in a room full of nascent artists and practitioners, beginning discussions that were to span the conference and beyond.
Later, during the scheduled Town Hall meeting, we were driven to ask the question: Will CHI consider a pictorial or highly visual format in future conferences? The positive murmurings of the crowd suggested there is the potential for uptake in this area, further expanding the archival formats available to researchers at CHI.
To engage those who already had an interest in sketching prior to the conference, we hosted a hands-on special interest group (SIG) at CHI , where our discussions took an active turn as we roamed the venue, getting to know the scenery and inhabitants before committing them to paper (Figure 6). A diverse mix of attendees ensured that the scope of the session was broad and the imagery illuminating. Sketching on location became a conversation opener, a master class, and a social event, lasting well into the break before the lure of hot Canadian bread pudding, custard, and cream split up our close-knit group.
|Figure 6. A sketch by Miriam Sturdee made in the Palais des Congrès during the CHI 2018 SIG.|
We began in the main hall and on the front steps of our convention center, Montreal's Palais des Congrès. Striking architecture and curious pigeons found a home on the attendees' pads as we discussed the significance, potential benefits, and pitfalls of sketching in HCI. We moved through the venue, asking how we could increase the visibility and uptake of sketching within the HCI curricula, and how to engage with students who view even physical computing as a technical, coded skill set without room for creative expression. In the public spaces surrounding the Palais, we brought shop fronts, cafes, and pedestrians into the embrace of the sketched image, while considering the future of sketching in HCI.
Engaging with our students was perhaps the most challenging aspect of sketching as a tool for HCI—and the most pertinent. Enabling and encouraging our younger community members will ensure a diverse skill set and an interest that can be sustained throughout lengthy academic careers. Best practices in teaching and workshop facilitation also proved to be a popular topic, bringing the group together regardless of their area of interest within HCI as a whole.
The success of the SIG, and how much was realized in the short 90 minutes, propels us to host similar events at the forthcoming CHI in Glasgow and other sponsored ACM conferences. By allowing practitioners and researchers to come together in this manner, collaborations and contacts are made and the prospect of sketching in HCI could be strengthened.
Sketching can be seen as a particularly human activity. In our earliest years as Homo sapiens, we made marks on cave walls to depict the world around us. That world is forever changing, but this simple method of recording our surroundings has lasting appeal. The tools available to us may also be different, such as the advent of the tablet computer and digital stylus, but we currently remain the sole perpetrators of this form of output. Could this be about to change?
Paul the Robot has been built to draw portraits of people as a naive artist and observer . Advances in neural networks are training computers to recognize the styles of famous painters and reuse them in creating original artworks in the same vein. The next steps for sketching in HCI could see machines as the foci of creative and expressive practice. We might compare the sketches of a machine to those of a child learning to draw and attribute some form of humanity to the resulting images. Interactive systems can also teach us how to better wield a pen, gently shifting our lines to conform to aesthetic ideals suggested by the computational analysis of thousands of similar images or training our muscles to recognize their relationship to the mark on the page using playful interfaces. Perhaps the sketch of the future embraces other formats of the same descriptor—pieces of code, movement, algorithms.
The evolution of sketching research does not negate its roots but rather offers further avenues of inquiry, widening its reach. Sketching is one technique that transcends disciplines, subdisciplines, and social groups, and this potential to bring together groups within HCI should not be overlooked. By creating working groups aimed at promoting sketching as discourse, analysis, and research practice, we can collaborate on developing sketching in practice and reap the rewards of an engaged, expressive, and interactive researcher base.
Not only do we wish to elevate sketching practice and methodologies as a focus for research, but we also call on CHI to embrace and promote all visual forms of documentation, such as the pictorial, a recent development in archival formats used by the DIS and Creativity and Cognition conferences. To actively engage with imagery in relation to technology—and not limit it to the simple outputs of machinery and computers—furthers the interdisciplinary embrace of CHI, and therefore that of ACM. Blending diverse practices and subject matter breaks us out of our comfort zones and can lead to discovery and purposeful reflection. And it could all start with a sketch. Sketching as we know it may evolve during the age of the computer, but it is here to stay.
Join the Sketching in HCI Slack Group: https://docs.google.com/forms/d/e/1FAIpQLSeH8UQCQX4rN0cEcJGPfi63w6tqfNmKNR7t0wI_eRDfS_DiRQ/viewform
2. Lewis, M., Sturdee, M., and Marquardt, N. Applied Sketching in HCI: Hands-on Course of Sketching Techniques. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2018, Paper C08.
4. Lewis, M., Perry, M., and Louvieris, P. Designing smart money on a digital ledger: Central bank issued digital currencies. HCI for Blockchain: A CHI 2018 Workshop on Studying, Critiquing, Designing and Envisioning Distributed Ledger Technologies. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2018, Paper W28.
5. Sturdee, M., Hardy, J., Dunn, N., and Alexander, J. A public ideation of shape-changing applications. Proc. of the 2015 International Conference on Interactive Tabletops & Surfaces. ACM, New York, 2015, 219–228.
6. Sturdee, M., Alexander, J., Coulton, P., and Carpendale, S. Sketch & The Lizard King: Supporting image inclusion in HCI publishing. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2018, Paper alt15.
7. Lewis, M., Sturdee, M., Marquardt, N., and Hoang, T. SketCHI: Hands-On Special Interest Group on Sketching in HCI. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2018, Paper SIG09.
Miriam Sturdee (@AsMirry) is a postdoctoral fellow at the University of Calgary researching sketching in visualization. She obtained her Ph.D. and subsequent postdoc with Lancaster University, looking at sketching for the design and development of shape-changing interfaces. Her interests in sketching in HCI combine her M.F.A. in Illustration with her initial studies in psychology. firstname.lastname@example.org
Makayla Lewis (@maccymacx) is a research fellow at Brunel University London looking at smart money and user experience. An accomplished visual thinker and sketcher, she organizes the monthly SketchnoteHangout and SketchnoteLDN, among other sketching events and courses, including ones at NordiCHI 2016 and DIS 2017. She has also provided visuals and sketchnotes for conferences such as CHI and ISS. www.makaylalewis.co.uk
Nicolai Marquardt (@nicmarquardt) is an associate professor in physical computing at University College London. At the UCL Interaction Centre, he works on projects in the areas of ubiquitous computing, interactive surfaces, sensor-based systems, prototyping toolkits, and physical user interfaces. Together with Saul Greenberg, Sheelagh Carpendale, and Bill Buxton, he is co-author of the Sketching User Experiences Workbook (Morgan Kaufmann 2011). email@example.com, www.nicolaimarquardt.com
Copyright held by authors. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.