ACM Interactions

If I am right that HCI and neighboring fields will increasingly rely on the essay as a means of scholarly contribution and debate in the future, then it follows that the construction, articulation, and criticism of intellectual positions will become increasingly important.

In Humanistic HCI, we talk about the essay, the epistemic roles of positions, and how they should be peer reviewed. We defined a position thus:

[A] position is not merely a proposition; it instead holistically comprises an expert-subjective voice; a theoretical-methodological stance; its own situatedness within a domain; and a pragmatic purpose. (73)

But as I read design research papers in the fragmented and emerging subdomain of research through design (and similar practices, including constructive design, critical design, and so forth), I have been frustrated with how researchers characterize others' positions, especially ones they disagree with. 

The purpose of this post is not to discourage disagreement.

It is, rather, to seek ways to support disagreement in a scholarly manner.

I approach this topic by way of analogy. When learning social scientific methods, such as interviews, we are encouraged to write down as closely as possible, even verbatim, what participants say. We are discouraged from writing down our reactions. This is partly to minimize the risk that the reactions come to stand in for what research subjects actually said.

I think that risk is sometimes realized in research through design work, that is, that sometimes in our writings our reactions stand in for the actual positions of other researchers, and that this is hindering intellectual progress.

Recommendation: When critiquing or positioning oneself against prior work, one should summarize the position (its claim structure, voice/stance, theoretical and methodological underpinnings, and pragmatic goals/consequences), before and as a condition of, expressing one's reaction to or critique against it.

An example

Let me exemplify what I mean.

I begin by summarizing the argument of a well-known paper theorizing research through design (RtD): Zimmerman, Stolterman, and Forlizzi's 2010 paper, "An Analysis and Critique of Research through Design: Towards a Formalization of a Research Approach."

My summary of the authors' position:

Zimmerman et al. begin with the claim that research through design is an increasingly important practice, but one that it is not well theorized; as a result, it faces many practical challenges. The intended contribution of the paper is that the authors "take a step towards formalizing RtD as a legitimate method of inquiry within the HCI research community by detailing how RtD can lead to design theory" (310). To do so, they provide a critical literature review, summarize interviews with 12 RtD scholars, and analyze several "canonical" RtD projects (as identified by the interviewees). In their findings, they present several specific ways that RtD practitioners produce theory and frameworks as well as artifacts that "codify the designers’ understanding of the current state, including the relationships between the various phenomena at play therein, and the description of the preferred state as an outcome of the artifact’s construction" (314). Other findings they present is that their interviewees expressed a concern that a romantic conception of the designer-as-genius was inhibiting their ability to present RtD as research; that tacit or implicit knowledge was a key outcome of RtD and that, by definition, such knowledge is difficult to articulate; and that standards of RtD documentation were lacking. Near the end of the paper, Zimmerman et al. offer some recommendations: "there is a need for serious development of RtD into a proper research methodology that can produce relevant and rigorous theory" (316); "A need exists for more examples where the intentional choice and use of the RtD approach as a methodology and process is both described and critically examined" (317); and "Researchers who engage in RtD need to pay more attention to the work of other design researchers [...] It is of the utmost importance that RtD is analyzed and critiqued in a serious and ambitious way" (317). The paper concludes by observing that RtD is "alive and well" and "recognized by the design and HCI communities," but that "there is still a lot to be done when it comes to establishing RtD as a recognized and well-developed research approach" (318).

I believe that this summary represents their position, as I defined it earlier:

[A] position is not merely a proposition; it instead holistically comprises an expert-subjective voice; a theoretical-methodological stance; its own situatedness within a domain; and a pragmatic purpose. (73)

I also believe that were I to show the above summary to Zimmerman et al., that they would agree that that was their position. I think most readers of that paper in the community would agree that the above summary was overall faithful to the paper. It also included their own words, set in an appropriate context.

And then at that point I could launch my critique of it, because only then are we all on the same page.

But too often, this is not what happens in design research.

Consider the following quotes from the abstract and introduction of Markussen, Krogh, and Bang's 2015 otherwise excellent paper, "On what grounds? An intra-disciplinary account of evaluation in research through design." The bold is my own; it reflects statements about the Zimmerman et al. paper I just summarized.

In the research literature that is initially reviewed in this paper two positions are located as the most dominant representing opposite opinions concerning the nature of such a methodology. One position proposes a cross-disciplinary perspective where research through design is based on models and standards borrowed from natural science, social sciences, humanities and art, while the other position claims a unique epistemology for research through design insisting on its particularities and warning against importing standards from these other disciplines. [...] This “state of the art” has led some researchers to call for a policing of the research through design label, working out a formalized approach with an agreed upon method to document knowledge (Zimmerman, Stolterman, & Forlizzi, 2010). Other researchers, however, argue for appreciating the controversies and proliferation of research programs currently characterizing the field (Gaver, 2012). In caricature it can be noted that representatives of the first group works to associate design with changing existing research traditions (natural, technical, social sciences and humanities) dependent on the deployed methodology and measures for evaluation whereas the latter works to position design outside classical research and science. (pp. 1-2)

Instead of summarizing Zimmerman et al.'s argument, this paper asserts what appears to me to be a reactive interpretation of that argument: that Zimmerman et al. advocate "models and standards borrowed from natural science, social sciences, humanities and art," that their paper "calls for a policing of the research through design label," and that it seeks to work out "a formalized approach with an agreed upon method."

The whole Markussen et al. paper is then positioned in the introduction as a response to these two prior "positions" in research through design: one exemplified by Zimmerman et al. (the police model) and the other by Gaver (the sui generis model).

But do these two positions actually exist as such?

The problem is that Zimmerman et al. do not call for "policing" that I could see. Nor do they concern themselves with "labels." And while they do use the unfortunate term "formalizing" in several key locations, including the title (!), a reading of their position—what they claim, the expert stance behind those claims, the theoretical and methodological underpinnings by which those claims were made possible, and the pragmatic goals of such claims—suggests a much less controversial project.

Zimmerman et al. spoke to RtD practitioners about their accomplishments and challenges; they sought to gather and organize the accomplishments into a theoretical perspective that others could leverage; they sought to acknowledge practitioner challenges and envision how design research might help them overcome them, in a way that reflects their voices (i.e., the interviews) and their projects (i.e., the exemplars that they co-identified). This doesn't sound like "policing" to me; neither does it sound like natural science. It also doesn't sound like "formalizing" as I understand the word—I personally wish Zimmerman et al. hadn't used that term, or at least had clarified how they were using it, because I think it opened the door to that reactive interpretation that they are "policing."

The upshot of all of this is that once they get down to the business of their own contribution, Markussen et al.'s project looks to me like a welcome extension/expansion of Zimmerman et al.'s. Markussen et al. for instance argue that research on RtD needs to do a better job of attending to "how evaluation is actually being practiced within design research itself" (p. 2). And while I would say that Zimmerman et al. in fact did pay attention to that, nonetheless Markussen et al. paid more attention to it, and offered a more substantive account of that evaluation (they identified five different methods of RtD evaluation). Theirs is an original contribution, one that I intend to try out in my practice (which is probably the highest compliment I can give). And it is based in a reasonable critique of where Zimmerman et al. left off.

What I am pointing to in this instance—but I have seen it elsewhere in design research—is a tendency to offer straw man accounts of prior work that I believe are derived from reactive interpretations rather than a sober and scholarly account of what those positions actually were.

I hope that this blog post provides some practical and actionable guidelines to help design researchers avoid this problem.

If we cannot avoid the problem, we face the consequence of obfuscating where the agreements and disagreements actually are. We are encouraged to fight hobgoblins that aren't even there. At its worst (perhaps—hopefully—not in this example), it sows controversy and division. This can undercut the research community's shared desire to find common ground and learn together.

Summary

Part of the discipline of humanistic argumentation is taking others' positions seriously, even when one wants to criticize them.

To do so, one must first adequately characterize the position as such: its claims structure, its speaking voice and stance, its theoretical and methodological underpinnings, and its pragmatic purposes (and consequences).

An important goal of doing so is to ensure that everyone—including the original authors—is on the same page, that is, that this really is what that earlier position entailed, before the critique begins.

We want a community of learning, one that can accommodate informed disagreements, but we do not want a circular firing squad.

NOTE: This post was lightly edited and reblogged from interactionculture.wordpress.com

Jeffrey Bardzell