Danielle Wilde, Kristina Höök, Danielle Wilde
The Blogpost in the March + April issue of interactions by Kristina Höök, technical program chair for CHI 2012, struck a nerve. While the column starts out well, the direction it heads down isn't that fruitful. Here's my view: People don't bring tech to CHI because CHI doesn't respect tech (in pretty much any form: novel tech, systems that test new tech, among others).
As a tech-oriented researcher, I cannot get papers published easily at CHI. I've had a few, but those were outliers that only "squeaked" in. The metrics CHI reviewers use for tech papers are not appropriate for technology-focused research. (Where's the study? Where's the killer app for this tech, validated by ethno-whatever studies of some important domain?) Indeed, "real" tech papers have such a low chance of acceptance at CHI that nobody in their right mind would submit them to the conference. I think the CHI community knows and largely accepts this, which is why conferences like UIST were created. Fine.
But the idea that the same folks should bring their demos to CHI and show them off is absurd. A CHI demo is not refereed; therefore, our peers do not respect it. Doing a demo at a conference, when you have a paper published about the work, is great. You have a respected contribution, and you are letting folks experience it.
But just doing a demo at CHI is like paying a ton of money to be a "dancing bear" in a circus. If the CHI research community does not want the work in the papers track, why would technology researchers go to the extreme trouble and significant expense of doing a demo at CHI? It's much more economical and effective to do demos at conferences that will publish the work.
So there you have it. If you want technology at CHI, start accepting the work into the papers track, and stop trying to create new venues for dancing bears.
School of Interactive Computing
Georgia Institute of Technology
I totally agree! This is why CHI changed this year, electing to have two kinds of demos: "only" demo and paper + demo. I think the Interactivity chairs worked really hard to raise the level of demos as well as raise the respectability of them.
I do wish we could make the demos archival, and not just refer to papers as archival and demos as "add-ons." But how can we make them archival? And how can we raise their status? This is a really important discussion. Right now, we are having it behind the scenes of CHI. I wish we could bring it out into the open. I believe that CHI will become less relevant unless and until we address this situation.
Small Steps First
I'm curious as to why demos are perceived as not peer reviewed?
I am one of the co-chairs of the Interactivity Explorations track at CHI this year. We went to great deal of effort to assemble a cast of high-quality reviewers for this track. Submissions went through a rigorous reviewing process (with a minimum of three reviews per), except in cases where we invited works and the authors were happy to not go through the review process. In these cases the works will be marked "invited" when demoed at CHI to distinguish them from works that have been peer reviewed.
In Australia, where I am currently based, exhibitions are accepted as publications. There are different criteria to assess the weight of the publication (just as there are for book chapters, journal articles, full conference papers, extended abstracts, and so on). Personally, I push demoing through where possible (and appropriately peer-reviewed) under the exhibition criteria, and am mindful to speak to the value of live demos whenever the opportunity arises.
I do make a distinction between a proof-of-concept research demo and a full-fledged research outcome, where knowledge is embodied in the completed artifact. In my opinion, such works warrant being championed as "publishable" and "published" when exhibited. They should be considered of equal (and, in some cases, of more) value than a written article, as they give access to knowledge that is not otherwise readily available.
The publication of artworks process I refer to is particular to fine art, and is being used more often in design research. While adapting it to HCI is still considered a bit radical, I believe it is inevitable, as value is increasingly understood and given to physical embodiments of work that afford embodied understanding in complement to what is provided in a written publication. Again, I speak of instantiations where the knowledge is embodied in the artifact, and best accessed through embodied engagement.
Bringing such changes to culture and to thinking requires great effort, though I sincerely believe the effort is warranted, and longed for by many.
For those who do not yet fully understand the value of demos, or the importance of recognizing their value throughout the entire system (and I include the antiquated administrative processes here), I believe it's only a matter of time and exposure. Yes, there are places where that value is already understood and supported, but institutions like CHI lag in this regard. This lag is exactly why I believe researchers should fight for culture change within CHI, and is why the Interactivity Explorations track was created.
It's easy to do what works, to take the path of least resistance, but doing so increasingly diminishes relevance (research is, after all, supposed to be the generation of new knowledge). If we can remind people of, or present the value of embodied experience and its role in knowledge generation (especially in institutions such as CHI), we can effect the necessary culture change. As the people in this conversation already know only too well, such knowledge is not lesser than that supported through qualitative and quantitative research methods, it is merely different.
Do we drop CHI because of this? No. We do what we can to change it from the inside. Hopefully, the small steps we are managing to take this year will help to set the stage for more positive and dramatic actions for the future. The important thing is not simply to complain, but to determine how best to take action. Even tiny actions can help to effectuate big changes.
Visiting Research Fellow
©2012 ACM 1072-5220/12/0500 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.