ACM Interactions

ACM conference program committees may not violate the Geneva Conventions or international law. But they are unquestionably un-American.

Presumption of innocence?—forget about it. “Until the rejection quota is reached, we must pass death sentences.” Double jeopardy?—bring it on! If the initial set of reviewers can’t agree to hang an author, have another program committee member read the submission after dinner and a couple drinks. Still no resolution? Yet another PC member the next day, as the clock ticks. We head to the airport only when 80 percent of those colleagues who entrusted us with their hopes and dreams are swinging from the gallows. And we will tweet friends and family: “Mission accomplished, another job well done!”

Confine smart, well-intentioned, hard-working, volunteer program committee members in a room often enough, put them under these pressures, and what happens?

Orcs, said Tolkien, were Elves who “by slow arts of cruelty were corrupted and enslaved.” I tend toward the Warcraft view that Orcs have nobility along with their savage destructiveness. Unquestionably they are remorseless killers, but they’re industrious and clever enough, albeit not flexible thinkers.

The ‘fresh eyes’ fallacy

Program committees feel that a fair approach to handling a disputed or borderline submission is to assign another reviewer. But bringing in a fresh Orc is clearly not in the interest of the authors. Trained reviewers are negative reviewers. If reviewers are not overwhelmingly negative, how will our major conferences manage to reject 75 percent to 90 percent of submissions?

The unfairness of bringing in fresh eyes is well known to journal editors. Journal reviewers have a gentler charge. Consider them Elves. Fewer initial submissions are rejected. Constructive criticism and suggestions for improvement are emphasized. Even reviewers who recommend rejection realize that if other reviewers do not, the decision could be “revise and resubmit”; therefore, they typically offer guidance for improvement. That said, a journal reviewer is tasked to find areas for improvement. Almost no journal reviewers give the first version they encounter an “Accept” recommendation. They find flaws and might tend to amplify their significance, demonstrating diligence by stressing the necessity of revising.

Journal editors (myself included) rarely bring in “fresh eyes” to examine a revision unless a reviewer has dropped out or specific important expertise was missing in the initial set of reviewers. Scientific idealism might suggest that additional perspectives could only be good, but adding new reviewers is widely considered unfair to authors, a form of double jeopardy. There is no perfect paper. New reviewers, if diligent, will perceive and amplify new flaws and insist on another full revision. If an editor were to bring in a new set of reviewers for each revision, no paper would ever be accepted. And that is with Elves!

Facing conference program committee Orcs swinging swords and maces, you will be hit. I co-chaired a recent program committee for which not one of 415 submissions received all 5s from the first three reviewers; 4s as well as 5s were in short supply. Just hope that the lethal blows fall elsewhere, and hope that they don’t send fresh Orcs your way. My CHI 2013 submission was liked a lot by two of the three initial reviewers, but after four (4!) program committee Orcs came at it, it wandered off and died.

Long ago, as an Orc tiring of blood, I tried an experiment. At that time, we submitted free-form reviews in email. For each paper submission, I wrote two reviews. One assumed the usual acceptance rate of about 20 to 25 percent. This review focused on finding flaws and identifying grounds for rejection. For the second review I assumed an acceptance rate of around 50 percent, familiar from IFIPS and HICSS conferences. This review focused on what was interesting in the paper, while also mentioning the things that would be good to fix. To my surprise, it was easy to write two markedly different reviews for the same paper. Both flowed out. Most of us, Gollum-style, have an Elf and an Orc in us. The Elf emerges to review for workshops and 50 percent conferences, and to help students. We summon our Orcs for program committee meetings, NSF panel reviews, and other less cheerful occasions. In that experiment, I submitted both sets of reviews and ratings to the Orc-in-Chief, who tossed out the Elvish scribbling and got on with the task at hand: rejecting papers.

Looking back in time, we might ask, “Why were Orcs created?” Tolkien said it was a consequence of an obsessive quest to demonstrate originality and creativity, opposed by the Elves, among others. We desired to demonstrate the creative originality and rigor of our conferences to Elvish journal disciplines. And we too began as Elves. In the early years, CHI program committees strongly proclaimed that we did not have quotas, that we would accept all work of merit. After a while this became difficult to maintain. Then there was a meeting where this proclamation was met by a hesitant, “But we really do have quotas, don’t we?” “No,” some insisted for the last time, but embarrassment over quotas was finally wrung out of us. Did someone hear Tolkien chuckling in the distance?

Jonathan Grudin