ACM Interactions

Authors:
Carman Neustaedter, Phoebe Sengers

---

 

Designers, developers, and researchers frequently use their own systems during design in order to test concepts, learn through actual usage, or find and fix software bugs. In fact, many would argue this is an important step before putting a design before other end users; such "eat your own dog food" methods have been applied in industry for years. Yet self-usage as a means for better understanding system design has a dubious reputation within HCI research, where it appears to clash with user-centered design principles and visions of researchers as disinterested observers. In this article, we explore the value of what we term autobiographical design as a way of developing systematic understanding of a system's potential. Autobiographical design occurs when people build a system, use it themselves, use this experience to learn about the design space, and evaluate and iterate the design based on their own experiences. Thus, we are talking about much more than a "dogfooding" approach; we are talking about long-term, genuine use, during which new knowledge is obtained.

Our interest in this topic began with our own design research work; we had each studied systems we had developed and used extensively ourselves [1,2,3]. To build on this, we also interviewed 11 senior researchers from a diverse set of backgrounds (e.g., academic versus industry, a variety of specific disciplines) whom we felt had used a variant of autobiographical design in their work. Table 1 lists these researchers along with their affiliations and the systems they discussed with us so that we may give them credit for their intellectual contributions. Here, we explain what you can and can't know through autobiographical design and lay out best practices for developing research understanding from this kind of work.

  How Is Autobiographical Design Done?

By definition, autobiographical design refers to intensive self-usage as part of design research, and all of our experts but one had engaged in such design. Sometimes the researcher was part of a design team that had all used the system; sometimes only some team members had used the system; and sometimes a researcher worked alone. In one case (Tom Erickson's), the researcher was the sole user of the system, but in all others, researchers used the system with their work group, alongside other users, or with their families. Some drew on self-usage throughout the design process, while others did so in the initial stages of design before moving on to test with other users once their system was stable.

Compared with a typical user-centered design model, in which user requirement extraction leads to design, which leads to implementation, our respondents reported a much earlier, faster, and tighter coupling between user input and implementation. Autobiographical design allowed researchers to rapidly start using and learning about their designs because they could sideline some usability or finishing issues that might otherwise be time-consuming. Most researchers continually tinkered with their systems, making alterations and experiencing and evaluating the results immediately. Reid Priedhorsky, for example, told us, "Any design process benefits from having an end user on the team...and the reason for that is it shortens the feedback cycle... If you have somebody in the community on your team, they are right there—you can just ask them. If you don't have someone on the team, you have to go to your panel or interview someone...it's just longer."

Conversely, autobiographical design requires building a system from the start that really works—and not just in test cases, because the design is actually being used. As Whittaker said, "This is a generic problem with building systems for ourselves. You do need to have an end-to-end system, in particular if you are trying to substitute something that is really important to people." Sometimes it is possible to quickly create an initial simple system and then iterate its feature set; but if a system requires substantial complexity before it can be realistically used, then autobiographical design may not be feasible.

Our experts repeatedly insisted that autobiographical design should rest on designers' genuine need for the system, which leads to a real engagement with the system that otherwise would be unlikely to happen. Erickson, for example, felt genuine usage was the main distinction between autobiographical design and "dogfooding" as a debugging practice: "[G]enuine usage is [when] you have this real goal or often a set of interweaved goals that you are really using the system to pursue. I think that genuine usage is...really valuable because it causes you to use the system and try to integrate it with other parts of your life in practice in ways that you wouldn't tend to if you were just trying to test it."

This can be a particular problem when some members of the design team who are not users have difficulty empathizing with those who are.

  What Can You Learn From Autobiographical Design?

In HCI research practice, evaluation with other users is the gold standard for establishing knowledge about a system. Yet our respondents felt that they had learned a great deal from trying the system themselves. Compared with standard evaluations of systems, which may last from one to three weeks, the evaluation of autobiographical design is very long term. Almost all of our respondents used their systems for more than a year, allowing them to deeply understand the effects of the system on real practice, as opposed to novelty effects. Evaluation timeframes are bolded in Table 1 to highlight the duration.

All respondents emphasized strongly that autobiographical design cannot produce results known to be generalizable to a broader community of users, yet beyond this they felt that autobiographical design provides other worthy lessons.

For example, respondents found that autobiographical design allowed them to see "big effects"—major things that could make or break a system, and genuine, as opposed to discretionary, needs—in a way that might not be as likely without experiencing the system oneself. Saul Greenberg, for example, said, "You get to see what matters... being the key user anchors you to what matters and what doesn't matter... The things you see when you use it yourself are the big things, you don't need a deployment, you will see this stuff."

Sometimes big effects were surprises in usage. Intuitively, one would expect it would be difficult to be surprised by the usage of one's own system. Yet respondents reported that their practices sometimes violated their design expectations, leading to better understanding.

Several respondents felt that autobiographical design allowed them to uncover detailed, subtle understandings that they likely would not have found with other user-centered design techniques because they might have seemed unremarkable. This was enabled both because of the large amount and quality of time they spent with the system (perhaps equivalent to an ethnographer living at a user's home or shadowing them throughout their workday) and also because of their first-person perspective on it (see [4]). Gregory Abowd, for example, said that his motivation for designing a system for his autistic son was to understand that space deeply before he designed for others. He felt his personal experiences gave him a true, deep understanding of the situation of use that would be difficult to achieve otherwise. Such personal, experiential understanding then becomes a resource for design.

Some respondents described how autobiographical design made them more responsible designers because they felt the personal impact of their systems. By putting themselves in what Abowd termed "harm's way," designers have a personal stake in the outcome and feel its potential impact more strongly. As Carl Gutwin put it, "You are confronted with your design decisions in a very intimate way when you use an API you designed...you really, really care about getting it to work right."

As a research method, autobiographical design seems best suited for exploratory systems that fill a new design niche, where there is no existing system or established culture of use. If a system already exists to fulfill a need, it seems difficult to achieve genuine usage. Thus, respondents saw autobiographical design as more appropriate for learning at the early stages of innovation, when proof of generalizability is less important. As Steve Whittaker said, "I think in cases where it is not a well-understood problem...things don't occur to you in initial design because the space isn't well understood, and it's only when you actually have the system in place that you discover what the fundamental flaws are and what the possibilities are."

As a counterbalance to the aspects of autobiographical design focused on oneself, some respondents said it can be helpful to have additional input from non-users (e.g., colleagues, supervisors, professors) or secondary users (e.g., colleagues, family members), for example, by having them critique design ideas as they are put into the design. Additional lessons can also come after moving from autobiographical design to broader user testing. Even so, it is not always the case that testing with additional users will produce new understanding. For example, Bill Gaver told us that when his team deployed the Video Window more broadly, the experiential results had all been anticipated by his own use. The deployment did, however, cause the team to think about new issues, such as safety and reliability.

Most of our interviewees kept no formal records of their experiences to learn from. In contrast, in our own work we used extensive data collection, including tracking system usage, documenting changes to the code, and contributing to communal blogs shared among project participants to keep track of our developing experiences with the system. As a practical matter, we would encourage researchers using autobiographical design to formally collect information during the course of use in order to support more accurate analysis of the design process and its consequences.

  Broader Implications

It is common in the design community for designers to rely on their own intuition when creating new products, and many will use their own designs during the early stages. Some might call this dogfooding, whereas others might simply call it design. Here designers utilize a repertoire of (often tacit) knowledge learned through years of schooling and/or professional experience to guide their decisions [5]. This is considered allowable practice, and why not? Designers have a wealth of knowledge, and this should be utilized as part of design processes.

Yet in the world of research, there is a strong bias against methods that appear to compromise objectivity. Are researchers "cheating" when they test on themselves? Is autobiographical design inherently biased through personal motivation? Though our respondents generally felt the answer to these questions—at least under the correct circumstances—was no, they still struggled with them. Gaver, for example, experienced an uncomfortable tension between designing for himself and the thought that he might benefit professionally with a publication: "I would repeat the way it worked with the Video Window...I would design something for my home because I wanted it, and then I might be willing to consider publishing it, but I would want the motivation to be one of doing something for my home, not one of generating a new publication."

Wendy Kellogg objected to the term autobiographical design and referred to a similar perception of selfishness: "I just think of it not as designing for myself and then foisting it off on others—usually we're designing with others in mind, and if we can use it ourselves for something real, that gives us access to this huge, rich... set of feedbacks and insights that are valuable in design."

In reaction to perceptions that the research community would react negatively, we found that authors generally limited reporting self-design in their papers. As Dan Cosley said, "You know, from a sort of very tactical point of view, there are people who just will not accept the idea of experimenting on and designing for yourself, and...if your goal is to reach those people, then self-design is kind of self-destructive."

Researchers used a variety of tactics to circumvent expected negative reactions. For example, Kellogg's team interleaved stories of their own use with those of others to make it clear they had not relied only on their own experiences. Several respondents suggested following up with broader user testing, even when they thought this was unnecessary. A few respondents, however, reported that full disclosure, at least for motivating system design, resulted in little negative reaction.

The widespread practice of underreporting autobiographical design seems a lost opportunity for HCI to connect the worlds of research and practice. The experiences of our experts suggest best practices, not only for engaging in autobiographical design, but also for developing reliable knowledge based on it.

In many situations, autobiographical design can provide detailed, nuanced, and experiential understanding of a design space. This can be done early in the design process to tinker with an idea, over extended periods of time (from months to years) to learn about long-term adoption and real usage, and when it might otherwise be difficult to deploy a design because of technical complexities. Genuine usage is hard to come by in the early design stages through typical user-centered design practices, but can be accessed through autobiographical design. Such genuine usage supports reflection-in-action [6] and, as argued by our respondents, allows researchers to draw on intuitions and nuances of experience that escape formal analysis.

Yet, as our results show, there are clearly also situations when autobiographical design is not appropriate. It is not easy to apply when a system already exists to meet users' needs, or if the designer is not directly involved with technology-building. It does not prove that a system can or will be widely adopted. Neither does it establish generalizability. To see if a design works for others, it can be helpful to conduct broader studies using other methods.

As with other methods, doing autobiographical design well requires a degree of rigor. By rigor, we do not refer to scientific rigor or to scientific method, but rather to careful, critical reflection on one's work processes and respect for the unique hallmarks of quality of autobiographical design research. Such hallmarks that emerged from our interviews include an extensive period of genuine, intensive use, measured in months or years (though shorter time periods of intensity could still be valuable); surprises in usage that lead researchers to rethink or further develop initial design conceptions; improvements to design driven by specific, documented incidents of use; and careful articulation of the impact of design decisions on experiential qualities of the system in use.

  Acknowledgements

We thank our interview participants for their contributions to this article. This work was supported in part by NSF grant IIS-0847293 and by Intel Research through the Intel Science and Technology Center for Social Computing. This article is based on "Autobiographical Design in HCI Research: Designing and Learning through Use-It-Yourself," by Carman Neustaedter and Phoebe Sengers, in Proceedings of DIS 2012, (June 2012), 514–523.

  References

1. Boehner, K., Sengers, P., and Warner, S. Interfaces with the ineffable: Meeting aesthetic experience on its own terms. ToCHI 15, 3 (2008).

2. Judge, T.K., Neustaedter, C., and Kurtz, A.F. The Family Window: The design and evaluation of a domestic media space. Proc. CHI 2010.

3. Neustaedter, C. My Life with always-on video. Electronic Journal of Communication. (2012)

4. Höök, K. Transferring qualities from horseback riding to design. Proc. NordiCHI 2010.

5. Snodgrass, A. and Coyne, R. Interpretation in Architecture: Design as a Way of Thinking. Routledge, London, 2006.

6. Schön, D. The Reflective Practitioner. Basic Books, New York, 1984.

  Authors

Carman Neustaedter is an assistant professor in the School of Interactive Arts and Technology at Simon Fraser University and director of the Connections Lab (http://clab.iat.sfu.ca), an interdisciplinary research group focused on the design of ubiquitous and mobile technologies for connecting people over distance and time.

Phoebe Sengers is an associate professor in information science and science and technology studies at Cornell University, where she runs the Culturally Embedded Computing Group. Her research interests include critical approaches to sustainable HCI and humanities-and-arts-based HCI methodology.

  Figures

Figure. An early version of the Family Window, designed autobiographically by Neustaedter and his collaborators to connect his home to his parents' home across the country with a media space video link. Neustaedter and his family used the system for a year.

Figure. Images captured by Affector, a system for affective communication by office friends. Affector was used by Sengers over a period of three years.

  Tables

Table 1. Interview respondents and described systems.

 

©2012 ACM  1072-5220/12/11  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.