Robert Phillips, Dan Lockton, Sharon Baurley, Sarah Silve
Investigating how people understand the systems around themfrom technology to democracy to our own bodiesis a common research goal across many disciplines. One of the practical aims is uncovering differences between how people think systems work and how they actually work (particularly where differences can cause problems) and then addressing them, either by trying to change people's understanding or by changing the way people interact with systems so that this better matches people's understanding .
Being able to say that you understand a system is essentially saying that you have a model of the system . In HCI and other people-centered design fields, attempting to characterize people's mental models of technology in which their behavior plays a role can be a significant part of user research. Users' mental models will perhaps only rarely accord exactly with designers' conceptual models of a system , but this is not necessarily a problem in itself: "[A]ll models are wrong, but some are useful" . Mental models should not be assumed to be static constructs covering the whole of a system; multiple models working at different levels of abstraction can be relevant in different circumstances, from complex work domains to simple everyday interactions .
Having understandinghaving a modelgenerally implies an ability to communicate that model , so many research methods involve participants explaining their understanding of a system to the researcher, for example, through think-aloud protocols, structured interviews, and exercises (e.g., with "black box" products). Other research methods are based on participants "revealing" their understanding through actions, such as card sorting, cultural probes, shadowing, or reenactment. Each has its advantages and limitations: There will inevitably be interpretation and post-rationalization by both researcher and participant.
Communicating models to others can also be a way of exploring one's own understanding. Techniques such as rubber-duck debugging and the cardboard analyst  involve explaining something to inanimate "others," in the process forcing the explainer to clarify and structure the explanation. The idea can be taken further: Reddit's "Explain Like I'm Five" and "Explain Like I Am a..." communities invoke the challenge of explaining concepts (scientific, cultural, political) as if the "other" were a particular category of person (e.g., a five-year-old) with assumed knowledge, cultural references, and models of the world. There are obvious parallels with "learning by teaching" methods wherein pupils teach their peers, in the process potentially debugging their own understanding, and Joseph D. Novak and D. Bob Gowin's work with concept mapping in education .
Instructions are a common example of communicating models of technological systems and can act as a boundary object between designers' conceptual models and models developed by users. In particular, "lay" instructionsinstructions created by one user for anothercould be used as a research method for investigating users' mental models. There are also opportunities within HCI for novel instruction formats to help establish new mental models for users unfamiliar with systems, such as Clara Gaggero and Adrian Westaway's Out of the Box for Samsung  or the use of video or narrated slideshows (e.g., Emma Rose Metcalfe's HowDo app ).
Understanding the models people have of how systems work can be important in public engagement with science, health, democracy, and environmental issues. One engagement domain less explored so far from an HCI perspective is the growing "open design" or "Maker" movement. It presents both challenges and examples of empowerment around people creating artifacts that respond to their own contexts of use. The Maker movement focuses on helping people understand not only how technologies work, but also how people can take part themselves. It is driven by openness. In a recent study, more than "90 percent of respondents ... wanted to share their respective projects for review, feedback, and learning new skills" . Sharing happens via Maker-generated tutorials, kits, events, and online communities, and often through instructions, schematics, and examples that can be adapted and remixed.
The "Maker's Bill of Rights" , which defines an open approach to making, specifically states that "schematics shall be included," illustrating the need for instructions that can help makers develop accurate understanding. Nevertheless, poor instructionsfrom the perspective of novice makers or those learning a new fieldcan lead to giving up or being put off starting or completing projects without understanding how the system works. In general, if instructions are confusing or poorly designed, they can present stumbling blocksor even something like Robert Pirsig's "gumption traps"points where setbacks not covered by the instructions, or clashes between users' mental models and how the system really works, lead to friction or failure. It is not uncommon to hear that particular instructions were "difficult to follow" or that "I gave up at this point," and sometimes this, when linked to poor interface design, can lead to actions both inefficient and undesirable for users (e.g., ).
We decided to investigateinformallylinks between instructions and mental models in the Maker movement context, via a stall at the 2012 Brighton Mini Maker Faire in the U.K. The event attracts a mixed audience, from professionals selling hardware to complete beginners of all ages, with an average attendance of 7,000 visitors. Our stall comprised a popup wall of a variety of "inspirational" instructions and a table of artifacts (Figure 1) relating to two practical activities we asked visitors to consider: tea making and fire making. Making tea is familiar to the British public, representing a shared reference point. Making fire is a skill previously used daily by humans, not just for survival but also as a "communicative event" for people to gather 'round with a common cause [13,14]similar to tea making, in fact. The artifacts were both for inspiration and to enable reenactment of the activities; an Arduino-based LED "fire" was also created to attract people.
We asked visitors to complete short questionnaire postcards about their experiences with instructions, then to choose one activity and create instructions for how to carry it out for someone else (Figure 2). The choice of formatgraphical, textual, flowchart, and so onwas open.
Here, our intention is to explore ways in which the exercise could be useful to designers, rather than presenting a full experimental write-up. So we'll consider the results informally, in terms of insights for developing the method. Over the day, 98 visitors took part, ranging in age from five to over 60 and including a director of a major international design firm. Figures 3 to 8 show a selection of instructions visitors created.
The diversity in styles reflects distinctions such as that between structural diagrams (overall assembly) and process diagrams. One participant made and posted to YouTube a video on tea making. Differences in complexity and detail are noticeable, not just in cases such as Figure 5 (created by a six-year-old), but also in the inclusion of alternative choices, for example, whether milk or sugar were wanted in the tea. While time was not explicitly limited, the ad hoc nature of participation meant few visitors gave attention to "optimizing" their instructions.
One emergent theme was how participants made use ofor assumedprior knowledge on the part of the unknown "others" for whom they were creating the instructions. Some asked explicitly about gauging the target audience for the instructions, as it changed what information was required. Many used no annotation or written instructions, relying on drawings; some even created their own pictograms. One of the most detailed responses described the soil types of where not to make a fire (Figure 4). Where language was used, it was generally accessible and understandable. There were differences between the two activities in how instruction making was undertaken: Some participants who developed instructions for fire making stated that they had never made a fire and sought direction from us, while the common understanding of the tea-making task potentially meant participants overlooked problems that non-regular users, or those with special needs, might have to overcome.
In itself, the range of instruction formats has some implications for instructional design in the Maker context, but here we are concerned with extracting insights around modeling from the instructions that people created. We analyzed a sample in more detail, assessing each according to criteria intended to allow comparison between people's models:
- the entities present (objects, resources such as air, water and electricity, people), including which entities are used with each other, the order in which entities are used, and how entities are changed through the processes described
- optional elements such as if-then loops
- caveats or suggestions to take special care
- how other people are included in the instructions, if at all
- "gray" areas where the model or process was unclear.
These criteria draw on concept mapping  but were chosen here based on features present across the sample. The aim was to enable a representation of each model to be derived from the instructions: Process diagrams (simple flow charts) were considered the most appropriate given the step-by-step nature of instructions, although other formats would be possible. Extracting the entities present, the relationships between them, the order of actions, and any loops enabled the creation of a consistent form of diagram even where instructions were in a different format, making it possible to see more clearly the similarities and differences between models. (The aim is not to determine the exact models people have, since, as noted earlier, these models may not be stable and participants did not necessarily include their entire model in their instructions.)
For example, Figures 9 to 12 show two sets of instructions for making fire and the process diagrams extracted. The aims of each are the same, but the details of the models differ. Each emphasizes particular aspects. Figures 11 and 12, for example, indicate the importance of dry materials, airflow, and a nonflammable surfaceas potential setbacks if not attended towhile a much simpler model is given in Figures 9 and 10. Some models, like Figure 4, emphasized safety, while others concentrated on the step-by-step nature of building a fire and keeping it going through progressively adding more material. In some cases, explicit recognition of physical or chemical requirements such as oxygen (Figure 3) is given.
Similarly, with the tea-making instructions, some emphasized general principles, such as applying heat to water (whether by an electric kettle, on a hob, or otherwiseFigure 7 mentions "appropriate energy"), while others were based on particular types of kettle or included more options about the tea itself (types of tea, or choice of milk and sugar) rather than physical aspects of making the drink. Some mentioned measuring the quantity of water that should be heated, while others suggested filling the kettle regardless of how many cups were required.
We are aware of the limitations of the Maker Faire context, and our choice of activities was not directly HCI-focused. There are changes we will make in applying the method in further research (including work on interaction with domestic energy systems), such as introducing the exercise via a scenario. This would clarify the "other" for whom the instructions are being createdfor example, "Imagine your friend is house-sitting for a week while you're away on holiday, and you're writing a note to explain how to use the heating/alarm," or even "Please explain to the next participant how to use the system you've just used."
Despite the limitations, the method has potential for development within design research, both to explore users' mental models and to highlight particular issues, for example, where models differ, where they emphasize some elements over others, or where points of friction arise. These could all be significant to the adoption of systems or to influencing behavior change, and to public engagement around issues where understanding is important. Indeed, these insights could ground further stages of user research by surfacing issues, enabling the tailoring of interviews or observation (e.g., if someone's instructions suggest particular frustration with certain aspects of a system, these could be investigated in more detail).
Jeff Johnson and Austin Henderson  suggest that in developing conceptual models for new technology, designers should "support how people want to think about doing their tasks," and user-generated instructions for existing tasks could provide a starting point here. Equally, with an exercise based around a system actually being redesigned, it would be possible to analyze instructions created by users by clustering similar models to enable the segmentation of design options. Possibilities here include something like mental model personasbut that is for another time.
Thank you to all the participants, to the Brighton Mini Maker Faire organizers for the welcome, and to Hugh Dubberly for his suggestions with this article.
1. Nielsen, J. Alertbox: Mental models. Nielsen Norman Group. Oct. 18, 2010; http://www.nngroup.com/articles/mental-models
8. Gaggero, C. and Westaway, A. Out of the Box, 2011; http://vitaminsdesign.com/projects/out-of-the-box-for-samsung
9. HowDo blog; http://blog.how.do
11. Jalopy, M., Torrone, P. and Hill, S. The Maker's Bill of Rights. 2006; http://blog.makezine.com/2006/12/01/the-makers-bill-of-rights
Robert Phillips is a product designer and a Ph.D. candidate at Brunel University investigating the partnership between open design and citizen science. He is interested in the new opportunities that digital fabrication can bring to the design process and is fascinated by people-orientated solutions. His industry background has included: human-computer interaction, ethnography, material development, design for manufacture, technology applications, concept generation, and territory-exploring workshops.
Dan Lockton specializes in people-centered design for behavior change and the public understanding of everyday systems. As a senior associate at the Helen Hamlyn Centre for Design, Royal College of Art, London, he works on influencing more sustainable home-energy use as part of the pan-European SusLabNWE project. He is founder of Requisite Variety, consulting and running workshops on design, people, and systems.
Sharon Baurley is head of design at the School of Engineering & Design, Brunel University. She joined Brunel in May 2010, and was formerly at Central Saint Martins College of Art and Design (UAL). Her research focuses on open design digital tools and social technologies so that consumers can have design conversations with brands, funded by the RCUK Digital Economy program. She is currently working with Horizon, a digital economy research hub.
Sarah Silve is a lecturer in product design at Brunel University. Her background is in silversmithing, jewelry, and laser forming for crafts applications. She is interested in step changes of technologies and processes for creative outputs that evolve practice and design; current projects include CNC paste deposition modeling. Her research encompasses new materials, laser material processing, rapid prototyping, user interfaces, and open design.
©2013 ACM 1072-5220/13/09 $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2013 ACM, Inc.