Blogpost

XVIII.6 November + December 2011
Page: 10
Digital Citation

Minority retort


Authors:
Joseph Kaye

The enormous intellectual diversity of the HCI community is at the heart of what makes this field an exciting place to work [1]. At the same time, the breadth of this diversity is not without problems. In a blog post, James Landay complained about the raw deal he felt his work received from reviewers at CHI: “The reviewers simply do not value the difficulty of building real systems and how hard controlled studies are to run on real systems for real tasks” [2].

As I wrote at the time, this is a familiar experience. But I believe this sense of frustration is far from unique to those who build “real systems.” My experience is that it’s not the difficulty of building real systems that reviewers don’t understand, but rather—wearing my ethnographer hat—the difficulty of studying real systems in use. At CHI and related conferences there is often the sense that whichever intellectual discipline one identifies with, one feels like an intellectual minority, unappreciated and drowned out by louder, more numerous voices. And, without a doubt, that is true. However—and this is the important bit—it’s true for everybody. All of the intellectual disciplines at CHI are minorities. Some are more in the minority than others, but there is no single dominant intellectual discipline.

Let’s say that—and I’m going to make up numbers here for the sake of argument—22 percent of the CHI community consider themselves people who build “real systems” of the kind Landay describes, so they feel as if they’re in the minority and that 78 percent of CHI just does not understand how difficult it is to do the work they do. Similarly, there are 22 percent who consider themselves to be ethnographers, who feel spurned, misunderstood, and ignored by 78 percent of CHI. And furthermore, there are only 22 percent who consider themselves to be doing engineering, 22 percent doing design, doing ICT4D, doing management, user experience, health, sustainability, entertainment, HCI for kids, HCI for pets, and so on.

Of course, not all those 22 percents are the same; some disciplines are more dominant in HCI. And indeed, we could develop some system to divide up work at CHI into categories and then come up with numbers that are more accurate than 22 percent—but in so doing there is a danger in emphasizing those numbers at the expense of the larger point I want to make in this article: All disciplines are in the intellectual minority at CHI. This is a result of the diversity in the field, and all the evidence indicates this trend toward increasing intellectual diversity will only continue. Indeed, I believe the field can only remain relevant if such intellectual diversity continues and further broadens. But in the last few years I have sat in member meetings at CHI and in program committee meetings at CHI and CSCW and listened to friends and colleagues in the community—arguably part of the CHI establishment, if there is such a thing, by their very presence at these meetings—bemoan the lack of respect and appreciation their work receives from CHI. So what will it take us as a community to move forward and embrace ongoing change so we can continue to be a growing and changing field that is not left behind?

To answer this, I propose we must actively recognize in HCI the notion of epistemic cultures: distinct intellectual communities that recognize appropriate topics of research and ways of creating and validating knowledge. Karin Knorr Cetina uses the term in her book of the same name to discuss the different ways two different “hard” sciences—high energy physics and molecular biology—create and validate knowledge [3]. Both are scientific disciplines that create new knowledge, but, as Knorr Centina points out, they have extremely different understandings of what knowledge is and how it should be created. These differences are evident not just in factors external to research, such as methods of peer review, but in the fundamental ways in which these disciplines conduct research.

Let’s take a very brief example to illustrate the difficulties of how different epistemic cultures coexist in HCI and how misunderstandings of those different approaches could arise. Psychological research has developed a set of marks of quality around doing experiments. A standard experimental psychology study might need to have 30 subjects to show the effect being studied to a reasonable level of doubt. (There’s some math about why that is the case, but that’s not important right now.) The number of subjects in a study is referred to as n, so in our example n=30. n has come to be an important mark of quality in psychological experiments. The results of a study in experimental psychology with n=12 will be seen as much less reliable or valid—and thus less publishable —than a study with n=30. A suitable n is recognized as one of the marks of quality in experimental psychology.


At CHI and related conferences there is often the sense that whichever intellectual discipline one identifies with, one feels like an intellectual minority, unappreciated and drowned out by louder, more numerous voices.


Let us compare that notion of validity in an experimental psychology experiment to an ethnographic investigation of a single extended family. In a paper for CSCW, my co-author Elisa Oreglia and I studied how members of three families in northern China gave each other cell phones as gifts [4]. I would argue—and our reviews tend to agree, I’m happy to say—that our paper is a “good ethnography”: By the epistemological standards of ethnographic work in CSCW, it is seen to be a valid creation of knowledge. Importantly, it is not a “bad experiment” because n=12 rather than n=30. That mark of quality is specific to a different epistemic culture.

And it’s precisely that distinction—between a good ethnography and a bad experiment—that makes it important to be explicit about epistemic cultures in HCI. In other, more established and less inclusive fields, it is possible to assume that everyone shares the same epistemic culture as you: the reviewers, the other readers, the editor of the journal. For better or for worse—and I stand solidly on the side of better—HCI is a wonderfully mixed bag of epistemic cultures. We must be rigorous about keeping research to epistemologically appropriate marks of quality—watching six different people for a few hours each is not an ethnography—but we need to recognize there are many different ways to do good work in HCI. There is an unfortunate tendency to see one’s own epistemic culture as the default and right way to make knowledge, and my personal experience suggests this is particularly apparent among those who have been trained solely or primarily in positivist science traditions. We must recognize that such assumptions are not acceptable within the CHI community. As we see over and over again at CHI, at CSCW, at DIS, and at our other conferences and publication venues, HCI has a huge variety of ways to create knowledge—design, ethnographic investigation, deployment of working systems in the real world, and many more—and each are themselves valid ways of making new knowledge, with standards of quality and of validity.

One example of taking these wide varieties of epistemic cultures seriously is in the approach the CSCW conference took this year to assigning papers to associate chairs, in which everyone had the chance to read titles and abstracts of all submitted papers and could nominate themselves for particular papers, providing an opportunity for the matching of both expertise and epistemic culture. CHI attempts to solve the problem by having authors submit to a subcommittee focused on a particular topic, which, while an improvement on assigning papers based only on author-selected keywords, suffers from the fact that it can be difficult to recognize epistemic cultures from committee descriptions, and particularly hard if one is part of the more than 50 percent new authors at the conference every year. Neither of these solutions is perfect, but both are steps toward helping the HCI community grow and incorporate new ways of knowing and new ways of making knowledge.

References

1. There is an open question about the degree to which CHI incorporates other forms of diversity, but that’s an issue for another article.

2. Landay, J.A. I give up on CHI/UIST. Dub for the Future blog (Nov. 7, 2009); http://dubfuture.blogspot.com/2009/11/i-give-up-on-chiuist.html

3. Knorr Cetina, K. Epistemic Cultures. Harvard University Press, Cambridge, MA, 2012.

4. Oreglia, E. and Kaye, J.J. A gift from the city: mobile phones in rural China. Proc. CSCW’12 (forthcoming 2012).

Author

Joseph “Jofish” Kaye is senior research scientist at Nokia Research Center North America. His work explores the social and cultural effects of technology on people and vice versa.

©2011 ACM  1072-5220/11/11  $10.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2011 ACM, Inc.

 

Post Comment


No Comments Found