XXII.2 March + April 2015
Page: 48
Digital Citation

The big hole in HCI research

Vassilis Kostakos

back to top 

At the 2014 CHI conference, my group published a paper that presented a bibliometric analysis of the conference itself over the past 20 years [1]. The extent to which the conference reflects the entire field of HCI is debatable, but it is acknowledged that this is the flagship conference of the field. Our analysis did not look at citations, or even authors, as this work has been previously published [2]. Rather, we performed a type of bibliometric analysis known as co-word analysis [3,4], which considers the keywords of papers, how keywords appear together on papers, and how these relationships change over time. Co-word analysis can map the "knowledge" of a scientific field by considering how concepts are linked. This analysis has been conducted for various other disciplines including psychology [5], software engineering [6], and stem cell research [7]. Therefore, in addition to using an established methodology, we are also able to compare the findings of our analysis with previous findings, and in this way identify how our own community's research compares with that of other disciplines.


By far the most important finding in our paper was that CHI has systematically lacked mainstream or motor themes. To understand the significance of this finding, it is important to explain motor themes. Co-word analysis identifies clusters of keywords that often appear together on papers; these clusters are called themes. For each theme we can calculate a number of metrics, including density (the internal cohesion of the theme) and centrality (how "central" a theme is to the whole field). Figure 1 shows a strategic diagram that uses these two dimensions (density and centrality) to identify four distinct states a theme can hold. A theme begins its life with low centrality and density in the Chaos quadrant. As the theme becomes more central to the community, it moves to the Bandwagon quadrant. The theme eventually matures its internal cohesion and moves to the Mainstream quadrant, where the motor themes of a community lie. Finally, a theme will lose its centrality in relation to the field and move to the Ivory Tower quadrant, subsequently dying away by returning to the Chaos quadrant.

back to top  Insights


Motor themes are the heart and soul of a discipline, its main topics or schools of thought. Surprisingly, we found that, unlike other disciplines, CHI has consistently lacked motor themes. Our analysis suggests that researchers who publish at the CHI conference do not systematically get behind a small number of topics to advance them sufficiently into the mainstream. Rather, our analysis showed that most research themes at CHI remain at the Bandwagon or Chaos quadrants. We simply roll from topic to topic, year after year, without developing any of them substantially.

The discussion of our paper remained moderate and stuck to presenting the facts and figures of our analysis. Here, I wish to provide a critical view of our findings and argue that for experienced HCI researchers, these findings are not surprising at all: In fact, they clearly demonstrate some of the skewed ways in which our community values research.

Our analysis suggests that researchers who publish at the CHI conference do not systematically get behind a small number of topics to advance them sufficiently into the mainstream.

The first point I wish to make is that a lack of motor themes should be a very worrying prospect for a scientific community. Therefore, HCI researchers should ask themselves whether CHI is or wants to be a scientific conference. If not, then the reasonable question is: Where is the scientific conference on human-computer interaction? If, on the other hand, we want to claim that CHI is a scientific conference, we ought to consider why it lacks mainstream themes.

My argument here is that the reason our discipline lacks mainstream themes, overarching or competing theories, and accumulated knowledge is the culprit known as implications for design. This is the mantra of our discipline that requires our papers to justify what implications for design our results have. While previous researchers have argued the perils of this approach, our work is the first to provide systematic and justified grounds against it. I argue that our eternal focus on implications for design is behind our discipline's lack of motor themes. Here is why:

  • Implications for design put practitioners' needs above those of researchers. Somehow our discipline believes that our research output should be more relevant to practitioners rather than to fellow researchers. Many will argue that CHI and human-computer interaction make up a multidisciplinary field with practitioners, and therefore implications for design are a way to tie the field together. However, one only has to look at the medical field for a strong counterargument. The medical field brings together scientists and practitioners, and in fact began as a "practice," but it maintains strong scientific approaches involving repeatability and reusability of findings that substantially develop research themes.
  • Implications for design give preference to contextual knowledge. The reason is that design is, indeed, practical and contextual, and therefore requires contextual advice. However, this means that more and more of our research provides only contextual knowledge, which is simply useless under slightly different situations. As a result, our field completely lacks accumulated knowledge that could potentially evolve into motor themes.
  • Implications for design are usually just a well-crafted argument. If we scrutinize most HCI papers that offer "Implications for Design" sections (including my own!), we will find that these are extremely polished pieces of text that do a wonderful job of not providing any reusable data, theory, or tools, yet manage to convince us that the implications are important. A further downside is that the use of English language is getting out of hand in our discipline. Unlike other disciplines that objectively and clearly describe their findings in scientific terms, our discipline has simply gone overboard and turned research into a prose competition. Doctoral candidates wonder: If I design the experiment, obtain the results, and report the results, why is that not enough? Why should I dress up my paper with eloquent language?
  • Implications for design demotivate incremental research. Because implications for design are typically an elaborate piece of English prose, it is in fact very difficult to reliably demonstrate that my research has improved on previous implications for design. If previous research gave me a tool or data, it would be much clearer how to improve those, but how do I slightly improve a long argument?
  • Implications for design demotivate repeating studies. If my research helps you design (because I offer you implications for design but no tool, theory, or data), then my research does not help you with your research. Given the billions spent on HCI research globally, how much of that research can be reused today? Frankly, very little, but there are a ton of implications for design.

As a way forward, I wish to urge our community to consider ways to establish motor themes. We simply need to get behind a small number of themes and develop them sufficiently. We need to stop conducting completely unrelated studies year after year and figure out ways to accumulate and develop knowledge. One way to do this is to change our mantra to "implications for research." For instance, we could expect papers to explicitly argue whether their findings are repeatable, how their findings can be reused by other researchers in an experiment or study, and how their findings can be objectively improved. In addition, we could expect that papers explicitly provide tools (software), datasets, or testable theories.

Finally, I wish to point out that I am very optimistic about the future of our discipline. Our work has been very well received by the community (getting an honorable mention for the paper and one of the best-talk awards for its presentation), a number of researchers have provided us with positive and encouraging feedback, and a number of new initiatives have sprung up our field to make it more scientific in the sense of repeating studies, incremental research, and reusable findings.

back to top  References

1. Liu, Y., Goncalves, J., Ferreira, D., Xiao, B., et al. CHI 1994–2013: Mapping two decades of intellectual progress through co-word analysis. Proc. CHI 2014. ACM, New York, 2014, 3553–3562.

2. Bartneck, C. and Hu, J. Scientometric analysis of the chi proceedings. Proc. CHI 2009. ACM, New York, 2009, 699–708.

3. Callon, M., Courtial, J.-P., Turner, W.A., and Bauin, S. From translations to problematic networks: An introduction to co-word analysis. Social Science Information 22, 2 (1983), 191–235.

4. Callon, M., Courtial, J.-P., and Laville, F. Co-word analysis as a tool for describing the network of interactions between basic and technological research: The case of polymer chemistry. Scientometrics 22, 1 (1991), 155–205.

5. Le Marc, M., Courtial, J.-P., Senkovska, E.D., Petard, J.-P., and Py, Y. The dynamics of research in the psychology of work from 1973 to 1987: From the study of companies to the study of professions. Scientometrics 21, 1 (1991), 69–86.

6. Coulter, N., Monarch, I., and Konda, S. Software engineering as seen through its research literature: A study in co-word analysis. Journal of the American Society for Information Science 49, 13 (1998), 1206–1223.

7. An, X.Y. and Wu, Q.Q. Co-word analysis of the trends in stem cells field based on subject heading weighting. Scientometrics 88, 1 (2011), 133–144.

back to top  Author

Vassilis Kostakos is a professor of computer engineering in ubiquitous computing at the University of Oulu, where he directs the Community Imaging Group. vassilis@ee.oulu.fi

back to top  Figures

F1Figure 1. Strategic diagram showing the various stages of a research theme.

back to top 

Copyright held by author. Publication rights licensed to ACM.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2015 ACM, Inc.

Post Comment

No Comments Found