ACM Interactions

Authors:
Niels van Berkel, Kasper Hornbæk

---

 

The field of human-computer interaction covers a broad set of methods, viewpoints, and application areas. While the real-world impact of our research is typically impossible to predict, HCI researchers generally seek for their work to have implications that go beyond an individual paper. For example, a study on the use of datalogging tools by hospital staff can provide design implications for patient information systems. A paper on privacy on the Web might provide policy implications on the regulation of tracking technologies. Finally, an analysis of methodological flaws in published papers can provide new analysis guidelines with implications for applying research methods. As these examples highlight, implications from a single study may span a variety of levels and types of potential impact. Here, we present seven types of implications commonly encountered in HCI research and outline the specific aspects that reinforce their impact.

  Insights

→ Implications from HCI research can be highly diverse, extending beyond the archetypical "implications for design."
→ We describe seven distinct types of implications and how they affect researchers from HCI or adjacent domains, practitioners, and society more broadly.
→ Being explicit about an article's implications supports readers in identifying concrete takeaways and forces authors to reflect beyond the technology or context being studied.

The purpose of authors in raising an implication is to highlight and summarize to the reader what to do with the insights presented in a paper or report. By being explicit in the intended implications of a scientific article, we pursue three distinct goals. First, to support the readers of HCI articles or UX reports to clearly identify different takeaways. Additionally, this assists in the uptake of findings by researchers outside HCI. Second, to allow readers and reviewers to more clearly assess whether a paper's method and results match the authors' intended implications. Third, to engage authors in constructing their arguments and in considering the broader implications of a given contribution. In doing so, authors may be encouraged to derive implications that extend beyond the specific technology or context of use being studied. While in this article we focus on scientific papers, we believe our framework can extend to any format of research report.

We draw inspiration from Jacob Wobbrock and Julie Kientz, who provided a valuable overview of seven types of research contributions in HCI [1], such as "empirical research contributions" and "artifact contributions." Rather than being preconceived like research contributions, implications emerge from the evidence presented in a paper. Implications therefore require careful reflection based on the knowledge obtained and presented in the paper. The widely shared expectation for HCI research to derive implications for design has raised criticism, as exemplified by Paul Dourish [2]. By explicitly describing other types of implications in HCI research, we seek to draw out more diverse and thereby more substantial HCI research implications. Specifically, we describe seven distinct types of implications and discuss how authors can design their studies and research reports to support the intended implications.

  Seven Types of Implications of HCI Research

The seven implications of HCI research that we describe next affect three distinct groups of stakeholders. Implications for methodology, implications for theory, and implications for HCI directly affect HCI as a field. As such, the primary target audience for these implications is researchers within HCI or adjacent domains. Implications for design and implications for practice primarily seek to inform practitioners, be they designers, developers, or other domain experts. Finally, implications for policy and implications for society aim to guide politicians, companies, or citizens in regulating or using computing in general.

Here we summarize each of these seven implications and describe the necessary considerations and common pitfalls in pursuing them in HCI research. For each implication, we provide examples found in the contemporary HCI literature.

Implications for methodology. Methodology implications aim to inform the way we design and analyze studies within HCI. These implications focus on aspects such as the selection and recruitment of participants or the analysis of data or reporting thereof. Methodology implications can also follow from the review or development of data-gathering instruments and tools, informing how research data is collected. As such, methodology implications seek directly to inform the ways in which we construct new knowledge. By improving our methodologies, we can study phenomena of interest in ways that are more reliable, insightful, or reproducible. Consequently, this type of implication is often highly specific and exclusively targets HCI researchers.

Contributions that involve implications for methodology often use comparison against a baseline, typically the established way of doing things. In doing so, researchers can provide a complete account of both the advantages and drawbacks of their methodological implications. For example, a new complex and computationally intensive analysis may increase the confidence in a specific statistical analysis, but doing so increases the required computing power and decreases the comprehensibility of the results. Researchers seeking to improve our methodological practice often follow a narrow empirical study design to isolate the impact of specific methods. Although the above description outlines a highly structured approach, implications for methodology can also be identified coincidentally throughout the process of conducting research.

Examples

• Nicola Dell et al. studied how demand characteristics in an evaluation affect its outcome. Their insights provide an understanding of the role of participant bias in HCI and suggestions for minimizing this bias. Nicola Dell et al. "'Yours Is Better!': Participant Response Bias in HCI." CHI'12, 1321–1330.
• Erin Treacy Solovey et al. provided guidelines for the correct application of functional near-infrared spectroscopy brain sensing in HCI studies. These guidelines follow from an empirical evaluation of signal interference caused by user behavior, allowing for higher data quality. Erin Treacy Solovey et al. "Using fNIRS Brain Sensing in Realistic HCI Settings: Experiments and Guidelines." UIST '09, 157–166.
• Nora McDonald et al. analyzed how the concept of reliability is discussed in CHI and CSCW papers. They also highlighted best practices for the community in identifying and motivating appropriate analysis techniques for qualitative data. Nora McDonald, Sarita Schoenebeck, and Andrea Forte. "Reliability and Inter-Rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice." Proc. ACM Hum.-Comput. Interact. 3.CSCW (2019).

Implications for theory. Theoretical implications concern the basic constructs of HCI and our understanding of how they affect each other. Such implications could be modifications to an existing model of acceptance of technology or a framework for understanding the impact of presence on decisions in VR. They may concern general propositions of what is valuable in HCI or the specifics of the processes involved when we press a button. The purpose of creating implications for theory is to improve our ability to understand and predict phenomena in interactive computing.

---

When done well, implications for theory help HCI researchers see new mechanisms or processes that drive interactions with computers.

---

When done well, implications for theory help HCI researchers see new mechanisms or processes that drive interactions with computers. To do so effectively, they need to explain why existing theories are unable to account fully for some phenomena. Effective implications for theory often arise from studies that are designed to test theoretical ideas, rather than as an afterthought in the discussion of results. Implications for theory may also be suggestions on how to apply theory to cover more cases, explain things better, or combine previous distinct models or frameworks. Simply stating that we need more work on a particular topic or that something is complicated are weak implications for theory.

Examples

• Jelmer P. Borst et al. compared the impact of the duration, complexity, and moment of smartphone interruptions. Based on this, they suggested that theories on interruptions account for both the interrupted and the interrupting task. Jelmer P. Borst, Niels A. Taatgen, and Hedderik van Rijn. "What Makes Interruptions Disruptive? A Process-Model Account of the Effects of the Problem State Bottleneck on Task Interruption and Resumption." CHI '15, 2971–2980.
• April Tyack and Elisa D. Mekler reviewed the current uses of self-determination theory in games research. Based on the review, they gave a range of implications for the use of this theory, including how to use its mini-theories more productively. April Tyack and Elisa D. Mekler. "Self-Determination Theory in HCI Games Research: Current Uses and Open Questions." CHI '20, 1–22.
• Ali Alkhatib and Michael Bernstein drew on the existing theory of street-level bureaucracies to present a theory of street-level algorithms. This idea has implications for how HCI and AI researchers should think about algorithmic decision-support systems. Ali Alkhatib and Michael Bernstein. "Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions." CHI'19, 1–13.

Implications for the HCI community. Implications for the HCI community may follow from studies or reflections on how we operate as an academic community, for example, through bibliographical analysis or a critique of ethical shortcomings. Frequently, these implications are based on an analysis of current practice and point inward to the HCI research community. As such, these implications can extend to widely different elements of HCI research into and education about, for example, the ways in which we organize conferences or handle academic misconduct.

In addition to empirical approaches, case studies can provide in-depth reflections on existing practices within the community. Contrasting HCI practice to other scientific fields is another valuable source for raising implications. For example, following the replication crisis in empirical research fields such as social psychology, various HCI conferences and journals have taken steps to allow researchers the ability to increase the transparency and reproducibility of their research. Given the diversity of HCI research and researchers, those proposing implications for HCI research must carefully consider the varying research practices and "rules" that exist across various related and intertwined disciplines. In the aforementioned example of research transparency, different viewpoints have emerged regarding participant privacy and the sharing of observational or interview data.

Examples

• Jessica Pater et al. presented a systematic review of participant compensation reporting in HCI. The authors provided a suggestion for standardized reporting of participant compensation. Jessica Pater et al. "Standardizing Reporting of Participant Compensation in HCI: A Systematic Literature Review and Recommendations for the Field." CHI '21.
• Anna Offenwanger et al. assessed the state of gender representation in HCI research through interviews and a meta-analysis. Based on these results, they highlighted suggestions on how to promote more-equitable research in HCI. Anna Offenwanger et al. "Diagnosing Bias in the Gender Representation of HCI Research Participants: How It Happens and Where We Are." CHI '21.
• Vassilis Kostakos reflected on an analysis of papers in HCI, which indicated a lack of motor themes in the HCI research community. Based on this data, Kostakos called on the community to explore new ways to more systematically accumulate and develop reusable knowledge. Vassilis Kostakos. "The Big Hole in HCI Research." Interactions 22, 2 (2015), 48–51.

Implications for design. The prototypical implications of HCI work are implications for design. These implications seek to inform the design of technology, bridging the gap between research findings and real-world design challenges. They can follow from both observational and interventional studies and can include a variety of outcomes, including design procedures, design guidelines, and various design representations. The target audience for this type of implication is often industry practitioners, such as user experience designers.

Regardless of the specific type of design implication, researchers must ensure its real-world applicability. This often requires working with the target end users to understand both the problem domain and its respective challenges. One way to validate the usefulness and validity of implications for design is to present and evaluate them with designers or other relevant stakeholders. A common pitfall in presenting implications for design is their limited generalizability, which reduces their value to others.

Although the ubiquitous nature of implications for design sections in HCI papers has been repeatedly critiqued, their occurrence remains commonplace. Paul Dourish commented on the fact that ethnographic studies are primarily evaluated on the implications that a study can provide for design [2]. This idea, he argued, further gives rise to the incorrect belief that design is the "natural end-point of research inquiry" [2].

Examples

• Jin Huang et al. evaluated target selection accuracy by manipulating a range of factors. Based on these results, they put forward recommendations for designing interfaces for optimal target selection accuracy. Jin Huang et al. "Modeling the Endpoint Uncertainty in Crossing-Based Moving Target Selection." CHI '20, 1–12.
• Johanna Brewer et al. reported the results of an online community-driven moderation intervention. Based on their experience, they presented implications for design activism in online platforms. Johanna Brewer, Morgan Romine, and T.L. Taylor. "Inclusion at Scale: Deploying a Community-Driven Moderation Intervention on Twitch." DIS '20, 757–769.
• Md Momen Bhuiyan et al. collected news credibility assessments from both professional and nonprofessional stakeholders. Based on these results, the authors draw conclusions on how designers can best integrate and filter news credibility assessments. Md Momen Bhuiyan et al. "Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria." Proc. ACM Hum.-Comput. Interact. 4.CSCW2 (2020).

Implications for practice. While the term practitioner in HCI research often refers to those in design-related roles (e.g., a UX designer), the design and evaluation of sociotechnical systems also lead to implications for other domains. The target audience for implications for practice can be specific professionals, such as teachers or healthcare staff, or those in leadership positions (such implications are often seen in information systems under the name "managerial implications"). Implications for practice suggest actions for these professionals, detailing how the results of a study inform their practice.

Similar to implications for design, implications for practice benefit from real-world validation to ensure their relevance to practice. Therefore, working with the target population of professionals is almost a necessity to understand their challenges and ensure that the provided implications are relevant in practice. Implications for practice furthermore need to clearly indicate how a system provides advantages or disadvantages to the practice at which it is aimed. This requires a more holistic approach to system evaluation than is typical for design implications.

Examples

• Gloria Mark et al. studied the impact of email on employee productivity and stress by cutting off email access for study participants. Based on these results, they discussed the positive impact that reducing email can have on employees. Gloria Mark, Stephen Voida, and Armand Cardello. "'A Pace Not Dictated by Electrons': An Empirical Study of Work without Email." CHI '12, 555–564.
• Krishna Subramanian et al. investigated the use of an interactive tool for teaching statistics. Based on their evaluation, the authors highlighted the potential of alternative pedagogical tools for educators. Krishna Subramanian et al. "Stat Playground: A Sandbox for Learning Practical Statistics." Human-Computer Interaction—INTERACT 2019. Springer International Publishing, 156–165.
• Andrea Grimes and A.J. Brush investigated how working parents manage their schedules. Following this, they outlined how calendars can better support busy professionals in their information needs. Andrea Grimes and A.J. Brush. "Life Scheduling to Support Multiple Social Roles." CHI '08, 821–824.

Implications for policy. Policy implications seek to inform or persuade regulators, politicians, and others in governing positions. Given the crucial role of digital technology in society, the findings of HCI researchers may be valuable to those responsible for regulating or implementing new technologies. This includes both taking control of the problematic sides of technology (e.g., privacy, addiction) and the adoption of technology in new domains. Lazar et al. highlighted a lack of engagement from HCI researchers toward public policy, arguing that "members of the HCI community need to engage, on a regular basis, with regulatory processes, at the regional, national, and multinational levels" [emphasis added; 3].

---

Implications must provide a significant and clear benefit to others, whether they are HCI researchers, practitioners, or other stakeholders.

---

Accessibility has been referred to as one of the few areas in which HCI has substantially affected policy development [3]. In this domain, the HCI community has a rich history of understanding, working with, and involving relevant stakeholders. Building on this core strength of HCI research is likely to increase the strength of implications for policy. Additionally, close collaboration with existing special interest or advocacy groups can help ensure that research findings are relevant to the case at hand. Finally, the use of meta-analyses is a relatively uncommon approach in HCI research that can help distill evidence on a debated topic. Such aggregated evidence can provide convincing data on which to base implications for policy.

Examples

• Kaiwen Sun et al. interviewed children to understand how their privacy risk perceptions form. Based on this stakeholder understanding, the authors provided recommendations for a more evidence-based approach to delivering privacy information for children. Kaiwen Sun et al. "'They See You're a Girl If You Pick a Pink Robot with a Skirt': A Qualitative Study of How Children Conceptualize Data Processing and Digital Privacy Risks." CHI '21.
• Matthew J. Bietz et al. studied the development of middleware software across two supercomputing centers. Their results highlighted challenges in balancing stakeholder needs and the role of national funding policy in overcoming these hurdles. Matthew J. Bietz, Drew Paine, and Charlotte P. Lee. "The Work of Developing Cyberinfrastructure Middleware Projects." CSCW'13, 1527–1538.
• Michael S. Bernstein et al. reflected on an ethics review process as applied to funding applications at their institution based on a mixed-method evaluation of former applicants. Based on their analysis, they identified a set of recommendations for future iterations and the potential uptake at other institutions. Michael S. Bernstein et al. "Ethics and Society Review: Ethics Reflection as a Precondition to Research Funding." Proceedings of the National Academy of Sciences 118.52 (2021), e2117261118.

Implications for society. Given the frequent and widespread interaction with technology across all aspects of life, HCI research increasingly points to wider implications for society. Going beyond the specific design or methods of studying specific technologies, implications for society focus on a technology's societal impact. The purpose of these implications is to raise awareness, stimulate reflection, and prompt action in relation to the impact of emerging technologies on our lives. While the target audience includes HCI researchers, it is generally broader and can include technologists, stakeholders, and interest groups.

Well-argued implications for society give rise to reflection on the impact of technology on society. This requires connecting HCI research to broader concerns or developments in society, as well as a thorough understanding of the concerns of relevant stakeholders. Often critiquing existing approaches, these implications can provoke existing norms or practices. Providing real-world examples of the consequences of existing practices can support in providing tangible evidence of what otherwise might appear as hypothetical concerns.

Examples

• Ine Beyens et al. studied the mediating role of adolescents' fear of missing out in their usage of social media. Their results showed that this fear results in increased stress levels, and suggested a link to other common issues among adolescents. Ine Beyens, Eline Frison, and Steven Eggermont. "'I Don't Want to Miss a Thing': Adolescents' Fear of Missing Out and Its Relationship to Adolescents' Social Needs, Facebook Use, and Facebook Related Stress." Computers in Human Behavior 64 (2016), 1–8.
• Dilrukshi Gamage et al. studied online conversations around deepfakes. Based on these insights, they provided suggestions on how to mitigate the harm around deepfakes. Dilrukshi Gamage et al. "Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implications." CHI '22.
• Josephine Lau et al. investigated privacy perceptions and behavior in the context of smart speakers. Their results highlighted tensions among primary, secondary, and incidental users of smart speakers, and a mismatch between these users' needs and existing privacy controls. Josephine Lau, Benjamin Zimmerman, and Florian Schaub. "Alexa, Are You Listening? Privacy Perceptions, Concerns and Privacy-Seeking Behaviors with Smart Speakers." Proc. ACM Hum.-Comput. Interact. 2. CSCW (2018).

  Raising Research Implications

In this article, we have outlined seven distinct implications for HCI. Implications take the shape of a logical argument in which an implication follows from the evidence outlined in the preceding sections of a paper or report. The arguments that support an implication can be deductive or inductive—with weak evidence leading to unconvincing implications. Implications must provide a significant and clear benefit to others, whether they are HCI researchers, practitioners, or other stakeholders.

An individual paper can conceivably result in implications across more than one of the outlined implication types. For example, Murphy et al.'s work on privacy practices at an emergency ward identified a gap between policy and real-world practice. This resulted in both implications for design, focused on privacy-enhancing technology that does not interrupt clinical work, and implications for policy, arguing for departmental-level privacy policies rather than a one-size-fits-all policy across the entire hospital [4].

Specificity. Identifying the implications of an individual paper requires a careful balance. On the one hand, it is critical not to overstate the paper's impact, as factors such as the study context, the investigated population, and the methodological choices may limit the generalizability. On the other hand, restricting a paper's implication merely to the particular context and constraints in which a study was conducted limits the significance and the advancement of the field. To strike this balance, implications must be specific and clearly articulated. This can be achieved through a well-supported argument that outlines how the implication can be applied and the circumstances under which it is applicable.

Alignment with evidence. Implications require a carefully written presentation to support the reader in forming a common thread that connects any presented evidence with the eventual implications of the work. When such a connection does not exist, an implication cannot be considered valid and should not be made. This aligns with Wayne Gray and Marilyn Salzman's concerns described as "conclusion validity" [5], in which they outline claims regarding elements not investigated in the paper or that are contradicted by their results.

Applicability. For implications to have real-world impact, they must be implementable by the intended stakeholder, be it a researcher, a practitioner, or a policymaker. Ensuring the applicability of implications can make these stakeholders more likely to utilize your implications. For example, a prototype application may provide evidence of the feasibility of a proposed technology, while a user test may indicate preliminary evidence of end-user acceptance. Providing practical support to these stakeholders, such as the contribution of reusable code, can further strengthen the applicability of implications.

  Conclusion

This article highlights seven distinct types of implications encountered in HCI research and practice. This is neither a definitive nor an exhaustive list, and while a nearly countless list of implications can be imagined, we sought to establish implications that could reasonably be expected to be the result of research within the area of HCI. We hope that a deeper reflection on an article's implication(s) can drive a more diverse set of implications beyond the familiar implications for design, ones that are more specific, closely align with the presented evidence, and can lead to real-world impact.

  References

1. Wobbrock, J.O. and Kientz, J.A. Research contributions in human-computer interaction. Interactions 23, 3 (2016), 38–44. DOI: 10.1145/2907069

2. Dourish, P. Implications for design. Proc. of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 2006, 541–550. DOI: 10.1145/1124772.1124855

3. Lazar, J. et al. Human–computer interaction and international public policymaking: a framework for understanding and taking future actions. Found. Trends Hum.-Comput. Interact. 9, 2 (May 2016), 69–149. DOI: 10.1561/1100000062

4. Murphy, A.R., Reddy, M.C., and Xu, H. Privacy practices in collaborative environments: A study of emergency department staff. Proc. of the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, New York, 2014, 269–282. DOI: 10.1145/2531602.2531643

5. Gray, W.D. and Salzman, M.C. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction 13, 3 (1998), 203–261. DOI: 10.1207/s15327051hci1303_2

  Authors

Niels van Berkel is an associate professor at Aalborg University's Human-Centered Computing group. His research focuses on collaboration and decision support, self-reports, and digital health. He is also interested in methodological concerns surrounding the real-world evaluation of intelligent systems. [email protected]

Kasper Hornbæk is a professor of computer science at the University of Copenhagen. He works on user experience, shape-changing interfaces, large displays, and body-based interaction. He is also interested in the methodology of HCI, including the role of replications, measures of usability, and solid experimental work. [email protected]

 

Copyright held by authors. Publication rights licensed to ACM.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.