Cosmin Munteanu, Heather Molyneaux, Susan O'Donnell
For many HCI researchers, validating their experiment or study protocols from an ethics point of view is a routine operation, an essential step when planning their research. While this formal process varies by country, the underlying principles are universal. For a large amount of HCI researchcontrolled lab-based usability evaluations of a user interfacethe ethics review process is a simple formality: receiving approval from an administrative body responsible for ensuring all research follows sound ethical principles. In Canada, research ethics boards (REBs) exist in all universities and are guided by a formal policy created by the three federal research-funding agencies . In other countries the research ethics guidelines are less centralized (for example, specific to each university); nonetheless, the principles and implementation share many common aspects across countries and disciplines and are the subject of increased scholarly attention (as illustrated by workshops such as .)
We are now witnessing an increase in field studies within HCI, attributable to a significant increase in efforts dedicated to mobile interactions and the use of information technology (mainly mobile) in developing countries or to provide various types of support (such as language, medical, and educational support) to at-risk or marginalized user groups. Such research is bound to run into the same ethical challenges as those faced by researchers in the social sciencesdilemmas (or "moral panics," as termed by Will van den Hoonaard ) encountered while conducting fieldwork that could not have been anticipated during the planning and formal approval process. Balancing practical research concerns with formal ethics requirements is common in the social sciences. However, these challenges have been mostly foreign to HCI researchers (many based in computer science departments), who often lack exposure to the messy realities of fieldwork and are unfamiliar with engaging their institutional ethics board in research planning. Attempts to formalize universally accepted principlessuch as ensuring that participants' privacy and confidentiality are respected and that participants are not harmed or exposed to risks has led to the ethics approval process becoming increasingly bureaucratic, with ethics boards and templates "specialized" to a certain discipline.
In our own research, we have encountered ethical challenges in developing and evaluating a mobile support application to assist low-literacy adults. This experience has been eye-opening for us. We are sharing it here to prompt other HCI researchers to reflect on their own approach and conduct for ethically difficult empirical research and to encourage them to become more engaged in the ethics-review and policy-making processes. Fieldwork with vulnerable populations is common within the social sciences and is now starting to become more pervasive in HCI. It is time for our engagement in the ethical aspects of such research to catch up as well.
The literacy problem and a solution. In countries like Canada, low-literacy adults make up a sizeable proportion of the adult population. Unfortunately, for economic and socio-demographic reasons, current programs designed to provide learning support and resources to low-literacy adults largely fail to reach and retain those who would benefit most. To help such adults become increasingly literate and independent, we developed a mobile language assistant (ALEX, described in ), to be used both in the classroom and in daily life. We conducted a six-month exploratory study with 11 participants enrolled in two adult literacy classes. Such classes are informal, resembling one-on-one tutoring, with several adult learners working independently on their assigned subjects but also interacting with each other and the teacher. Each participant received one tablet running ALEX, and we collected data by frequently observing participants in the classroom and listening to participants' verbal accounts of usage elsewhere.
We encountered numerous challenges in preparing and conducting our research, mainly stemming from the study environment and the nature of our user group. We discuss here the ethical principles and guidelines we found difficult to navigate during the study and refer the reader elsewhere  for a description of the methodological challenges.
Informed consent. The literacy program is for adults who completed only a few years of formal schooling and can carry out some non-complex reading and writing tasks, such as some newspaper reading and writing a very simple letter. During enrollment in our study, we explained the research objectives, the consent forms, and how to use ALEX, and we encouraged participants to review the consent forms with the teacher, a family member, or a friend. After a review period of up to two weeks, participants decided if they wanted to continue with the study.
The adult learners had difficulties understanding the forms; most signed without reading. Only one participant read the consent form, although researchers suspected the participant treated it as a reading challenge in itself. Another participant jokingly drew the comparison with signing a new cell phone contract. Despite our efforts to phrase the consent form in accessible, plain language, our final version was still worded in a formal and fairly rigid way to satisfy the requirements of the ethics review process. We found ourselves trying to explain the details of this form to participants, only to be cut short in our explanations by the participants, who then signed it to get the formalities over with quickly.
A similar departure from established ethical requirements occurred with the device review process. Under normal circumstances, participants have a chance to review and familiarize themselves with the application, usually for a few minutes. To avoid overwhelming participants by demanding they agree to participate right away, this review period was extended to a week. During this time, we visited the classes daily with the stated intention of assisting participants strictly with technical troubleshooting. However, although not officially enrolled in the study yet, most participants started to offer their feedback, suggestions for improvement, and information on how they were using the application. We faced a procedural ethical dilemma of whether we should collect and use such data before having consent forms signed. It was clear that the information forthcoming from participants was valuable and would not likely be available after that initial familiarization week was over.
Privacy and confidentiality. A routine ethical expectation is that participant privacy and confidentiality are guaranteed, particularly for research involving vulnerable user groups. However, it was difficult to conduct our study while fully ensuring confidentiality. In the classroom, participants openly discussed the study details and their use of the application both when researchers were present and at other times. We conducted interviews as privately as possible and no audio was recorded, but the daily observation sessions were often a mix of technical support for application troubleshooting, feedback from participants, and personal stories on using the application, all shared publicly in the classroom.
Ethical guidelines exist for consent forms for studies involving group interactions in which participants are explicitly informed about decreased privacy, but our protocol was not designed for this situation. We chose not to revisit the already signed forms after noticing these group interactions, since signing the consent form was already a tedious task for participants, and since many non-participants were also included in these group interactions. Despite the departure from the privacy protocol outlined in our original application, we decided to continue collecting data in this manner since privacy risks were minimalthe adult learners were already sharing many other personal details in the classroom.
Contemporary HCI research involving vulnerable participants has its own unique ethical challenges. These are becoming more prominent and do not fit the ethical templates to which we are accustomed.
Voluntary participation. Human-subject experiments require participants to enroll voluntarily and without coercion. Not all the learners in the classroom enrolled in the study. Some did not want the burden of having a device that was not theirs; others were in the literacy program only for short periods of time. Yet almost all these non-participants used our application, as most participants willingly lent them their devices in the classroom (as well as to others outside the classroom who were also not part of the study). While nobody was pressured to enroll in our study, the non-participants who used the borrowed devices became, in a way, involuntary subjects. Their interactions with classmates were described to researchers by the study participants, and, on occasion, directly to researchers by these non-participants. This raises the ethical (and moral) question of whether data collected from non-participants should be included in researchers' analysis.
Exposure to risks and harms. Informing participants of any risks to which they are exposed is an essential ethical component of any study. Typically in HCI research, such risks are not greater than those encountered by using computing devices in everyday life; such statements still must be disclosed to participants before they enroll in the study. However, formal disclosure of risks is constrained when the application or device to be evaluated is used by non-participants, a very common occurrence in our field study.
Data collection. During the data collection we also found ourselves in situations where we needed to improvise with respect to the approved protocol. The initial plan was to administer questionnaires at the end of the study as part of an interview. However, the participants' literacy levels made this difficult. Even questions phrased, with the help of teachers, at appropriate literacy levels did not elicit meaningful answers; participants did not answer them at all or they provided answers from which no useful information could be extracted. Instead, researchers had to adapt, on the fly, the questions to the literacy level of individual participantsoften rephrasing questions, or setting them in the context of a personal story in order to elicit an answer.
Another departure from protocol was the "unplanned" data collection that occurred in places outside the classroom. Being in a small city, we encountered participants in various public places. Participants took advantage of such encounters not only to ask various technical support questions but also to relate their experiences using the application. Again we faced the dilemma of whether to ignore this data; in fact, we suggest it would be impossible and perhaps even unethical for researchers to ignore valuable data when conducting their analysis.
Participant-researcher rapport. One of the most significant challenges, both from an ethical and a moral perspective, was the familiarity between researchers and participants that developed naturally during six months of daily visits. A positive consequence was that participants were unreservedly providing feedback to researchers; however, participants also expected researchers to become more intimately involved in the class proceedings (for example, by answering questions, sharing personal details, and attending the holiday party). Participants regarded researchers as members of the community formed around the classrooms, exemplified by a request made by a participant for a ride to a medical appointment after class. While all efforts were made to ensure unbiased data collection (as discussed in ), it was difficult to maintain a social distance between researchers and participantsfor example, the loan of the mobile device to one participant was extended beyond the data-collection period as her school-age child started using the device (and our application) for homework.
Looking forward. Contemporary HCI research involving vulnerable participants has its own unique ethical challenges. These are becoming more prominent and do not fit the ethical templates to which we are accustomed. Our own challenges during our field study with low-literacy adults that we have reflected on here illustrate the difficulty of balancing formal ethics requirements with the practicalities of HCI field research. We believe it is upon us, as a community, to take an active role in managing such challenges by becoming more aware of the ethical implications of conducting field research, especially with vulnerable populations. We invite colleagues to join the interdisciplinary efforts of creating ethical guidelines and formal review processes that are flexible and reflective of the diversity of research methods and real-life cases we encounter.
1. Tri- Council Policy Statement: Ethical Conduct of Research Involving Humans, 2nd edition. 2010; http://pre.ethics.gc.ca/
Cosmin Munteanu is a researcher with the National Research Council Canada. He works on improving humans' interaction with information-rich media and interactive systems through natural language, for mobile devices, assistive technologies, and mixed reality. He also serves as an internal reviewer for applications to the NRC Research Ethics Board. firstname.lastname@example.org
Heather Molyneaux is a research council officer with the National Research Council Canada. Her research areas are new media, representation, gender, community, and technology usability. email@example.com
Susan O'Donnell is a researcher with the National Research Council Canada and the vice-chair of the NRC Research Ethics Board. A social scientist, she specializes in the social aspects of ICT (information and communication technologies). She has conducted research with marginalized communities in Canada and several European countries. firstname.lastname@example.org
COPYRIGHT HELD BY NRC.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2014 ACM, Inc.