Features

XXIII.3 May + June 2016
Page: 38
Digital Citation

Research contributions in human-computer interaction


Authors:
Jacob Wobbrock, Julie Kientz

All scholarly fields strive to contribute new knowledge. In the field of human-computer interaction (HCI), this new knowledge increasingly comes in rich forms like videos and demos, but the archival research paper remains the most widely used and accepted capture and delivery mechanism for research knowledge. The knowledge contribution made by a research paper—or more precisely, made by the work a research paper describes—is any research paper’s central feature. For example, a theoretical physics paper may contribute a new mathematical model for the behavior of light near black holes. A civil engineering paper may contribute a new method for stress-testing bridges. A social anthropology paper may contribute an account of people’s reactions to teen pregnancies in rural religious communities. Whatever the field of inquiry, whatever the phenomenon of interest, every research paper strives to make a research contribution by offering new knowledge. In an effort to distinguish this kind of knowledge from everyday know-how, some scholars even capitalize the term: Knowledge.

ins01.gif

In the whole of human inquiry, there are, of course, countless specific research contributions to be made. But the types of these contributions—the general forms this new knowledge takes—are relatively few. The three examples in the opening paragraph each make a different type of contribution: The first is theoretical, the second is methodological, and the third is empirical. These are three different research contribution types—the knowledge they contribute comes to us from three different “ways of knowing”[1].

Insights

ins02.gif

As is often observed, HCI is highly inter- and multidisciplinary. It is also young. It has taken a few decades for the types of knowledge in HCI to emerge, converge, and stabilize, and new ways of knowing still swirl about [2]. But over time, an identifiable pattern of research contribution types has evolved. These types were recently refined and put to use in the submission process for ACM CHI 2016, the flagship conference in HCI.

In this article, we offer an encapsulation of the research contribution types in HCI, endeavoring to provide definition to our field and the types of knowledge it produces. We also describe how different contribution types have been codified in the CHI submission process. For the CHI 2016 conference, we show how the submitted and accepted papers were distributed across contribution types, providing insight into the kinds of knowledge the HCI field develops and disseminates.

Seven Research Contribution Types in HCI

In 2012, the first author posted an unpublished whitepaper to his academic website describing seven research contribution types in HCI [3]. This whitepaper was used as a source in creating the contribution types for the new CHI keyword structure. The whitepaper presented seven research contribution types, how they are judged, and specific examples of each. Here we summarize those contribution types, with examples.


It has taken a few decades for the types of knowledge in HCI to emerge, converge, and stabilize, and new ways of knowing still swirl about.


Empirical Research Contributions

Empirical research contributions are the backbone of science. They provide new knowledge through findings based on observation and datagathering. Data may be qualitative or quantitative, aspiringly objective or unapologetically subjective, from the laboratory or from the field. In HCI, empirical contributions arise from a variety of sources, including experiments, user tests, field observations, interviews, surveys, focus groups, diaries, ethnographies, sensors, log files, and many others.

Empirical research contributions are evaluated mainly on the importance of their findings and on the soundness of their methods. If empirical findings are uninteresting or unimportant, or if the methods by which those findings arise are sloppy, imprecise, or confounded, then empirical contributions are judged unfavorably.

* Examples

  • Interview study. Burke, M., Kraut, R., and Williams, D. Social use of computer-mediated communication by adults on the autism spectrum. Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW ‘10). ACM Press, New York, 2010, 425–434; http://dx.doi.org/10.1145/1718918.1718991
  • Diary study. Czerwinski, M., Horvitz, E., and Wilhite, S. A diary study of task switching and interruptions. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘04). ACM Press, New York, 2004, 175–182; http://dx.doi.org/10.1145/985692.985715
  • Quantitative lab experiment. Lee, S.C. and Zhai, S. The performance of touch screen soft buttons. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘09). ACM Press, New York, 2009, 309–318; http://dx.doi.org/10.1145/1518701.1518750
  • Crowdsourced study. Pandey, A.V., Rall, K., Satterthwaite, M.L., Nov, O., and Bertini, E. How deceptive are deceptive visualizations?: An empirical analysis of common distortion techniques. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘15). ACM Press, New York, 2015, 1469–1478; http://dx.doi.org/10.1145/2702123.2702608
  • Qualitative field study. Poltrock, S.E. and Grudin, J. Organizational obstacles to interface design and development: Two participant-observer studies. ACM Transactions on Computer-Human Interaction 1, 1 (1994), 52–80; http://dx.doi.org/10.1145/174630.174633

Artifact Contributions

HCI is driven by the creation and realization of interactive artifacts. Whereas empirical contributions arise from descriptive discovery-driven activities (science), artifact contributions arise from generative design-driven activities (invention). Artifacts, often prototypes, include new systems, architectures, tools, toolkits, techniques, sketches, mockups, and envisionments that reveal new possibilities, enable new explorations, facilitate new insights, or compel us to consider new possible futures. New knowledge is embedded in and manifested by artifacts and the supporting materials that describe them.

Artifact research contributions are evaluated according to the type of artifact that gave rise to them. They are often accompanied by empirical studies but do not have to be, and sometimes should not be [4]. New systems, architectures, tools, and toolkits are evaluated in a holistic fashion according to what they make possible and how they do so [5]. New input and interaction techniques, by contrast, are evaluated precisely and quantitatively so as to isolate their human performance benefits. New design expressions, including sketches, mockups, and envisionments, are evaluated by how insightful, compelling, and innovative is their portrayal. Of particular importance is how well designs negotiate trade-offs and hold competing priorities in balance.

* Examples

  • Input device. Baudisch, P., Sinclair, M., and Wilson, A. Soap: A pointing device that works in mid-air. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST ‘06). ACM Press, New York, 2006, 43–46; http://dx.doi.org/10.1145/1166253.1166261
  • System. Dixon, M. and Fogarty, J.A. Prefab: Implementing advanced behaviors using pixel-based reverse engineering of interface structure. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘10). ACM Press, New York, 2010, 1525–1534; http://dx.doi.org/10.1145/1753326.1753554
  • Hardware toolkit. Greenberg, S. and Fitchett, C. Phidgets: Easy development of physical interfaces through physical widgets. Proceedings of the ACM Symposium on User Interface Software and Technology (UIST ‘01). ACM Press, New York, 2001, 209–218; http://dx.doi.org/10.1145/502348.502388
  • Input technique. Grossman, T. and Balakrishnan, R. The Bubble Cursor: Enhancing target acquisition by dynamic resizing of the cursor’s activation area. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘05). ACM Press, New York, 2005, 281–290; http://dx.doi.org/10.1145/1054972.1055012
  • Envisionment. Ishii, H. and Ullmer, B. Tangible bits: Towards seamless interfaces between people, bits and atoms. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘97). ACM Press, New York, 1997, 234–241; http://dx.doi.org/10.1145/258549.258715

Methodological Contributions

Methodological research contributions create new knowledge that informs how we carry out our work. Such contributions may improve research or practice. They may influence how we do science or how we do design. They may improve how we discover things, measure things, analyze things, create things, or build things.

Methodological research contributions are evaluated on the utility, reproducibility, reliability, and validity of the new method or method enhancement. Useful methods that can be reproduced and provide reliable, valid results are judged favorably. Knowing that a method has these properties requires repeated validation.

* Examples

  • Method application. Consolvo, S. and Walker, M. Using the experience sampling method to evaluate ubicomp applications. IEEE Pervasive Computing 2, 2 (2003), 24–31; http://dx.doi.org/10.1109/MPRV.2003.1203750
  • Method innovation. Druin, A. Cooperative inquiry: Developing new technologies for children with children. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘99). ACM Press, New York, 1999, 592–599; http://dx.doi.org/10.1145/302979.303166
  • Method adaptation. Millen, D.R. Rapid ethnography: Time deepening strategies for HCI field research. Proceedings of the ACM Conference on Designing Interactive Systems (DIS ‘00). ACM Press, New York, 2000, 280–286; http://dx.doi.org/10.1145/347642.347763
  • New measures. Soukoreff, R.W. and MacKenzie, I.S. Metrics for text entry research: An evaluation of MSD and KSPC, and a new unified error metric. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘03). ACM Press, New York, 2003, 113–120; http://dx.doi.org/10.1145/642611.642632
  • New instrument. Suh, H., Shahriaree, N., Hekler, E.B., and Kientz, J.A. Developing and validating the User Burden Scale: A tool for assessing user burden in computing systems. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘16). ACM Press, New York. To appear; http://dx.doi.org/10.1145/2858036.2858448

Theoretical Contributions

Theoretical research contributions consist of new or improved concepts, definitions, models, principles, or frameworks. They are vehicles for thought. Whereas methodological contributions inform how we do things, theoretical contributions inform what we do, why we do it, and what we expect from it. Theories may be qualitative or quantitative. They may have both descriptive and predictive power—by distilling the essential features of a phenomenon, they are also able to suggest how that phenomenon will behave. Fully developed theories offer explanatory accounts, not simply observing that but explaining why. Theories should be testable and falsifiable; if they are not, they do not qualify as scientific theories [6].

Theoretical research contributions are evaluated based on their novelty, soundness, and power to describe, predict, and explain. A theory that accounts well for observed data from a specific situation but has no ability to generalize to new situations is of limited use. Conversely, a theory that is so broad it can account for just about anything probably does not contain any true descriptive power. Theory validation is almost always achieved through empirical work.

* Examples

  • Thought framework. Bellotti, V., Back, M., Edwards, W.K., Grinter, R.E., Henderson, A., and Lopes, C. Making sense of sensing systems: Five questions for designers and researchers. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘02). ACM Press, New York, 2002, 415–422;. http://dx.doi.org/10.1145/503376.503450
  • Design space. Card, S.K., Mackinlay, J.D., and Robertson, G.G. The design space of input devices. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘90). ACM Press, New York, 1990, 117–124; http://dx.doi.org/10.1145/97243.97263
  • Conceptual model. Lee, C.P. and Paine, D. From The Matrix to a Model of Coordinated Action (MoCA): A conceptual framework of and for CSCW. Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ‘15). ACM Press, New York, 2015, 179–194; http://dx.doi.org/10.1145/2675133.2675161
  • Design criteria. Nathan, L.P., Friedman, B., Klasnja, P., Kane, S.K., and Miller, J.K. Envisioning systemic effects on persons and society throughout interactive system design. Proceedings of the ACM Conference on Designing Interactive Systems (DIS ‘08). ACM Press, New York, 2008, 1–10; http://dx.doi.org/10.1145/1394445.1394446
  • Quantitative model. Wobbrock, J.O., Cutrell, E., Harada, S., and MacKenzie, I.S. An error model for pointing based on Fitts’ law. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘08). ACM Press, New York, 2008, 1613–1622; http://dx.doi.org/10.1145/1357054.1357306

Dataset Contributions

A dataset contribution provides a new and useful corpus, often accompanied by an analysis of its characteristics, for the benefit of the research community. Benchmark tests may accompany datasets to standardize comparisons. Datasets enable evaluations of shared repositories by new algorithms, systems, or methods.

Dataset research contributions are judged favorably by the extent to which they supply the research community with a useful and representative corpus against which to test and measure. Often, datasets are published with new tools that enable researchers to work with the new corpus. Dataset contributions often accompany methodological contributions where a new method is meant to exercise the dataset.

* Examples

  • Test corpus. MacKenzie, I.S. and Soukoreff, R.W. Phrase sets for evaluating text entry techniques. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI ‘03). ACM Press, New York, 2003, 754–755; http://dx.doi.org/10.1145/765891.765971
  • Benchmark tasks. Myers, B. et al. Using benchmarks to teach and evaluate user interface tools. Unpublished manuscript (1997); http://www.cs.cmu.edu/~amulet/papers/benchmarks.pdf
  • Corpus creation. Paek, T. and Hsu, B.-J.P. Sampling representative phrase sets for text entry experiments: A procedure and public resource. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘11). ACM Press, New York, 2011, 2477–2480; http://dx.doi.org/10.1145/1978942.1979304
  • Repository. Plaisant, C., Fekete, J.-D., and Grinstein, G. Promoting insight-based evaluation of visualizations: From contest to benchmark repository. IEEE Transactions on Visualization and Computer Graphics 14, 1 (2008), 120–134; http://dx.doi.org/10.1109/TVCG.2007.70412
  • Global dataset. Reinecke, K. and Gajos, K.Z. Quantifying visual preferences around the world. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘14). ACM Press, New York, 2014, 11–20; http://dx.doi.org/10.1145/2556288.2557052

Survey Contributions

Survey research contributions and other meta-analyses review and synthesize work done on a research topic with the goal of exposing trends and gaps. Survey contributions are appropriate after a topic has reached a certain level of maturity. It is not uncommon for surveys to have references numbering in the hundreds. The journal ACM Computing Surveys is solely devoted to publishing survey contributions. In HCI, the journal Foundations and Trends in HCI regularly publishes survey contributions.

Survey research contributions, and meta-analyses in general, are evaluated based on how well they organize what is currently known about a topic and reveal opportunities for further research. To be effective, survey contributions must not be mere laundry lists of prior work. Good surveys exhibit completeness, depth, maturity, and organization.

* Examples

  • Techniques. Balakrishnan, R. “Beating” Fitts’ law: Virtual enhancements for pointing facilitation. International Journal of Human-Computer Studies 61, 6 (2004), 857–874; http://dx.doi.org/10.1016/j.ijhcs.2004.09.002
  • Emerging topic. Coelho, J. and Duarte, C. A literature survey on older adults’ use of social network services and social applications. Computers in Human Behavior 58 (2016), 187–205; http://dx.doi.org/10.1016/j.chb.2015.12.053
  • Tools. Johnson, G., Gross, M.D., Hong, J., and Do, E.Y.-L. Computational support for sketching in design: A review. Foundations and Trends in Human-Computer Interaction 2, 1 (2009), 1–93; http://dx.doi.org/10.1561/1100000013
  • Domain. MacKenzie, I.S. and Soukoreff, R.W. Text entry for mobile computing: Models and methods, theory and practice. Human-Computer Interaction 17, 2–3 (2002), 147–198; http://dx.doi.org/10.1080/07370024.2002.9667313
  • Technology. Shaer, O. and Hornecker, E. Tangible user interfaces: Past, present and future directions. Foundations and Trends in Human-Computer Interaction 3, 1–2 (2009), 1–137; http://dx.doi.org/10.1561/1100000026

Opinion Contributions

Opinion research contributions, also called essays or arguments, seek to change the minds of readers through persuasion. Although the term opinion might suggest a less-than-scientific effort, in fact, opinion contributions draw upon many of the above contribution types to make their case. Opinion contributions are considered a separate research contribution type not because they lack a research basis, but because their goal is to persuade, not just inform. Along with persuasion, the goal of opinion contributions is to compel reflection, discussion, and debate.


Opinion contributions are considered a separate research contribution type not because they lack a research basis, but because their goal is to persuade, not just inform.


Opinion research contributions are evaluated on the strength of their argument. Strong arguments credibly use supporting evidence and fairly consider opposing perspectives. They focus on topics of wide interest and should be broadly accessible.

* Examples

  • Evaluation. Bernstein, M.S., Ackerman, M.S., Chi, E.H., and Miller, R.C. The trouble with social computing systems research. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI ‘11). ACM Press, New York, 2011, 389–398; http://dx.doi.org/10.1145/1979742.1979618
  • Prioritization. Dourish, P. Implications for design. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘06). ACM Press, New York, 2006, 541–550; http://dx.doi.org/10.1145/1124772.1124855
  • Application. Greenberg, S. and Buxton, B. Usability evaluation considered harmful (some of the time). Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘08). ACM Press, New York, 2008, 111–120; http://dx.doi.org/10.1145/1357054.1357074
  • Vision. Newell, A. and Card, S.K. The prospects for psychological science in human-computer interaction. Human-Computer Interaction 1, 3 (1985), 209–242; http://dx.doi.org/10.1207/s15327051hci0103_1
  • Definition. Oulasvirta, A. and Hornbæk, K. HCI research as problem-solving. Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI ‘16). ACM Press, New York. To appear; http://dx.doi.org/10.1145/2858036.2858283

Contribution Types at the ACM CHI Conference

The ACM CHI conference has increasingly come to consider research contribution types as part of its submission process. From 1996 to 2001, authors were asked to indicate a paper type, although it was never referred to as a “contribution.” Examples from 1996 included “experience papers” and “systems papers.” The year 1998 was the first that the call-for-papers (CfP) included the word contribution. By 2000, lengthy contribution descriptors appeared, such as “a thought-provoking, well-substantiated analysis of an HCI-related issue,” or “experience gained in adapting designs and applying other HCI contributions to real-world conditions, presented in the form of a design briefing or case history.” From 2002 to 2008, authors were asked to submit a 30-word freeform “contributions and benefits statement,” only for use in the program. From 2009 to 2013, authors were asked to choose from a list of contribution types, but from 2014 to 2015, contribution types were listed in the CfP but not selectable when submitting. In 2016, contribution types appeared as optional submission keywords devised by Frank Bentley and Jofish Kaye and influenced by Wobbrock’s whitepaper [3]. An important difference from Wobbrock’s original list was that the empirical contribution type was split in two, one for studies of systems and one for studies of people. Tables 1, 2, and 3 show how research contribution types were codified over the past decade of CHI.

The earlier CHI contribution types embodied a mix of concerns—not only types of knowledge, but also types of work, substantive concerns, and methodological approaches. Eventually, lengthy descriptors gave way to pithy terms focused on the types of knowledge created.

The CHI 2016 submission process made it optional for authors to indicate their papers’ research contribution type. Of 2,316 submissions, only 130 opted not to indicate any contribution (5.6 percent); 1,481 indicated exactly one contribution (63.9 percent); 705 indicated more than one (30.4 percent). On average, a submission had 1.29 contribution types indicated (SD=0.66). Figure 1 shows how contribution types were presented to authors during the CHI 2016 submission process.

So how has the HCI research community engaged with these eight types of knowledge? The distribution of submitted and accepted papers across all eight of CHI 2016’s research contribution types is shown in Figure 2.

Of the 2,316 submissions to CHI 2016, nearly half (44.0 percent) were empirical studies of system use, and almost the same percentage of such papers made up the final program, with an acceptance rate (23.5 percent) almost identical to that of the overall conference (23.6 percent). Empirical studies of people also dominated submissions (28.4 percent) and had the best acceptance rate of any contribution type (26.3 percent). Artifact or system papers were also prevalent in submissions (24.2 percent) and had an acceptance rate (23.9 percent) similar to that of the overall conference.

In contrast, acceptance rates for the less prevalent contribution types— Method, Theory, Essay/Argument, Meta-Analysis/Literature Survey, and Dataset—were lower than the overall conference acceptance rate of 23.6 percent. Most were around 17 percent, with Dataset research contributions having the lowest acceptance rate (16.7 percent) as well as being the least submitted (1.3 percent). Essay/Argument contributions enjoyed the highest acceptance rate of this bunch (21.6 percent), yielding 19 accepted papers in the final program (3.5 percent).

It is clear from Figure 2 that empirical studies and technology artifacts dominated the research activities of the CHI 2016 community, and that other research contribution types were not only less common, but also less likely to be accepted. We stress that the level of research activity should not be confused with the value placed on particular contribution types once they are published. For example, Method and Theory contributions may be hard-won publications, but they may be highly cited and relied upon for years to come.

Conclusion

As the HCI field matures, it is important to reflect on the knowledge it produces and the forms that knowledge takes. Such reflection can be challenging in a broad and diverse field, and while we should be wary of constraining our imaginations, we should embrace giving definition to the knowledge we produce. Doing so provides a valuable map for navigating the field of HCI and helps newcomers take their first steps. It is, after all, those newcomers who will discover new possibilities and push the frontiers of our knowledge even further.

Acknowledgments

We thank Frank Bentley, Jofish Kaye, and James Fogarty for their feedback on early drafts.

References

1. Olson, J.S. and Kellogg, W.A. Ways of Knowing in HCI. Springer, New York, 2014.

2. Höök, K., Bardzell, J., Bowen, S., Dalsgaard, P., Reeves, S. and Waern, A. Framing IxD knowledge. Interactions 22, 6 (2015), 32–36.

3. Wobbrock, J.O. Seven research contributions in HCI. Unpublished. Available at http://faculty.washington.edu/wobbrock/pubs/Wobbrock-2012.pdf

4. Greenberg, S. and Buxton, B. Usability evaluation considered harmful (some of the time). Proc. of CHI 2008. ACM Press, New York, 2008, 111–120.

5. Olsen, D. Evaluating user interface systems research. Proc. of UIST 2007. ACM Press, New York, 2007, 251–258.

6. Popper, K. The Logic of Scientific Discovery (2ed). Routledge Classics, London, 2002.

Authors

Jacob O. Wobbrock is an associate professor in the Information School and, by courtesy, in Computer Science & Engineering at the University of Washington. He chairs the Master of Human-Computer Interaction & Design program and is a founding member of the DUB Group. wobbrock@uw.edu

Julie A. Kientz is an associate professor in Human Centered Design & Engineering and, by courtesy, in the Information School and in Computer Science & Engineering at the University of Washington. She directs the Computing for Healthy Living & Learning Lab and is a founding member of the DUB Group. jkientz@uw.edu

Figures

F1Figure 1. Optional checkboxes for the eight CHI 2016 contribution types. Authors could select none, one, or more than one.

F2Figure 2. CHI 2016 submissions and acceptances by contribution type, sorted by descending number of submissions.

Tables

T1Table 1. Contribution types for CHI 2006–2008. Authors also submitted a 30-word contributions and benefits statement.

T2Table 2. Contribution types for CHI 2009–2015. Through 2013, authors were asked to choose a type when submitting.

T3Table 3. Contribution types for CHI 2016.

©2016 ACM  1072-5220/16/05  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2016 ACM, Inc.

Post Comment


@Andreas (2016 05 10)

Great analysis and insight. Here is a point to ponder for the CHI committee members (and the rest of the community):

The authors state that “If empirical findings are uninteresting or unimportant [..] then empirical contributions are judged unfavorably”. I am sure that many researchers will have found their methodologically sound work rejected at many HCI-related conferences to be rejected in reviews, for these precise reasons.

Is it acceptable to a scientific community that work would be rejected on such partial, subjective basis? Uninteresting - to whom? For which reasons? Unimportant - in what context? To whose work?

I will write my own answer to that question below, and welcome others to provide their views as well.

The entire field of science is rife with examples of work whose value did not become apparent until years, or decades later, when other technological and theoretical conditions matured enough to make a contribution relevant. As a simplistic example, take Einstein’s theory of relativity, without which it would have been simply impossible to implement the global positioning system (GPS).

Some would argue that leading conferences have a responsibility to promote not just a diversity in submissions, but also to ensure that the criteria for reviews are clearly set and appropriate to a scientific field. We must also not forget that many of the reviews are undertaken by junior researchers, for whom reviewing papers is part of their formative process and whom experienced researchers such as PC members, have a responsibility to help develop. As such, criteria for quality reviews must not only be set, but also enforced.

CHI’s rebuttal mechanism is a step in the right direction, but it places the burden of proof to the authors, not the PC members who are solely responsible for picking appropriate reviewers. But it is not enough - serially bad reviewers are not excluded (or given time off) from the process. Submitters who are also reviewers often have “skin in the game” which may bias reviews. Papers which critique work in previous literature are given to the authors of the critised work to review (since they are regarded as experienced in that field), who of course have many reasons for putting down that work.

Of course, this is all human nature and in a world where conferences are always short of people to do reviews, these phenomena will never go away. But without a proper, and commonly agreed mechanism for feedback on the quality of reviews, as well as for enforcing the criteria appropriate to reviewing, we are doing no better than hoping “it will all work out OK in the end”.