ACM Interactions

Authors: Viktoria Pammer-Schindler, Erik Harpstead, Benjamin Xie, Betsy DiSalvo, Ahmed Kharrufa, Petr Slovak, Amy Ogan, Joseph Jay Williams, Michael Lee
Posted: Tue, August 04, 2020 - 11:09:15

---

The field of human-computer interaction (HCI) has always been interested in aspects of learning. HCI researchers have spent decades investigating how people learn to use interfaces, with designing for learnability being a key HCI design principle. And with technology constantly changing, everyone from developers to end users must constantly (re)learn how to use, adapt, and improve the tools around them.

More recently, there has been an even greater interest in learning within HCI communities, perhaps best evidenced by the increase in learning-relevant papers submitted to CHI. This, in turn, led to the creation of the newly established Learning, Education, and Families subcommittee, with 191 papers submitted in its first year at CHI 2019, and 179 papers submitted for CHI 2020. Many factors contributed to this boom: more learning scientists engaging with the HCI community, a renewed university-level emphasis on online learning technologies (e.g., MOOCs, learning-management systems), and an increased interest and awareness of the necessity of lifelong learning and the possibilities of using computer technologies to support this need. This interest, in turn, has surfaced the realization that a) user-centered design is critical for educational technologies, which may be particularly research-intensive when involving more novel technologies such as mixed reality or AI-based systems, and b) designing for learning has specific design and interaction issues that go beyond more generally applicable design knowledge.

The SIG at CHI 2019

At CHI 2019 we organized a Special Interest Group (SIG) on Learning, Education, and HCI to bring together researchers across the CHI community and discuss the position of learning within the field [1]. Our goal was to gather a broad set of perspectives from across the field and better understand what community members mean when they talk about learning. Here are some of the key insights from that SIG:

• HCI researchers interested in learning and education design for a vastly heterogeneous set of learning and educational settings. Subsequently, in order to enable communication and cross-fertilization in this HCI subcommunity, a clear foundation in learning sciences should be presented with each research contribution. This addresses the interdisciplinary challenge of designing for learning, which means integrating knowledge from different disciplines, while at the same time allowing the HCI community as a whole to increase shared knowledge about learning.


• HCI researchers are aware of the tensions between joy and usefulness in learning, and of the complex issue of motivating and nudging learners in order to foster learning that is as engaging and effective as possible.


• HCI contributions on learning and education may need to be exploratory, qualitative, and longitudinal rather than quantitative and experimentally comparative, especially if the contributions involve novel interventions to promote learning. This exploratory nature is common in HCI research; in this inherently multidisciplinary domain, such an approach needs to be further integrated and aligned with additional appropriate methodologies.


• HCI can bring to learning and instructional sciences significant knowledge about the design of interventions, such as methodological knowledge about participatory design and similar design and research methods. HCI can also bring design knowledge, such as affordances of different types of technologies, as well as usability and user experience best practices.

The SIG was organized as three roundtable discussions, such that at every point in time each of the following three questions was being discussed at two or three tables:

• What does learning mean to us?
• What are valid ways of evaluating a learning contribution?
• How can we foster the relationship between learning and HCI?

Results of Discussions

Below we summarize the discussions had on these questions by over 50 discussion participants at CHI 2019 in Glasgow and highlight overarching issues for research in human-computer interaction on learning and education.

What does learning mean to us? This question turned out to be successful at highlighting the wide variety of different types of learning that HCI researchers are interested in, such as: learning for different age groups; formal and informal learning; understanding learning from the perspective of different learning theories, such as constructionism, behaviorism, or sociocultural learning theories; and designing for learning based on different paradigms for structuring learning, such as experiential or collaborative learning.

The major insight, at a meta level, that we draw from this face-to-face discussion is that interdisciplinary discussion in HCI on learning needs to be open to a wide variety of learning and instructional theoretical backgrounds; at the same time, we need to ensure that discussions of learning remain accessible to the wide variety of HCI researchers. The discussion indicated that to meet both goals—inclusivity and accessibility—every learning contribution in HCI would need to present a solid explanation of its learning and instructional sciences background. While this may feel redundant to us, some scaffolding of learning theory is needed to communicate the assumptions underlying the work for other HCI researchers interested in learning and education.

In parallel, in an even more heterogeneous fashion, this question led SIG participants to reflect on issues that are not only fact-based but that also relate to values with respect to learning, such as whether HCI designers and researchers should prioritize engagement over effectiveness, or how much guidance versus how much freedom and associated challenge is suitable for learners, acknowledging that every dichotomy can be resolved by answering “both.” Even acknowledging foundational insights from learning and instructional sciences, for example on the effectiveness of guidance, these questions and others remain issues of concrete complexity in every single instance of technology design. HCI researchers therefore emphasize that one way to move forward is to look at persuasive and motivational design, which aims to incite motivation in learners to achieve as much joy and usefulness as possible at the same time.

Finally, we identified a fundamental tension: learning versus education. Participants tended to share the view that the two are not the same; that while the former emphasizes the change within the learning entities (and despite acknowledgement that learning can happen at an informal team or formal organizational level, the focus of most participants was on human individuals as learning entities), the latter emphasizes the design of formal environments and structures that lead to learning. It would seem then, in principle, that HCI researchers who explore how to set up computational environments that are conducive to learning should feel closer to education; however, this was not the prevalent feeling. This is an interesting development that we can only ask the community of researchers interested in the interdisciplinary field to take up in argument and discussion.

What are valid ways of evaluating a learning contribution? This question was discussed in order to understand the extent to which there was a shared agreement on what kinds of contributions the subcommunity of HCI research on learning and education is uniquely positioned to make. We highlight here that this question was not intended to provoke general discussion on what a CHI contribution is, or to question other research communities’ understanding of the matter.

From a learning viewpoint, what constitutes a suitable evaluation hinges on the goal of the intervention, the perspective on learning, and the learning context. For instance, from a cognitive perspective, a learning intervention can be evaluated in terms of performance of the learner on pre- and posttests of knowledge. On the other hand, from a sociocultural perspective, an appropriate study would consider a learner’s broader context, including social norms, learning culture, and available artifacts in the learner’s environment. Inherent to theoretical framings are assumptions as to what factors are important to learning and how to evaluate them. However, beyond underlying theories, a point of discussion was that while HCI researchers focus on designing technologies for or with learners, learners seem to be excluded from the earlier, yet crucial, stage of defining the goals of an intervention. This has important consequences for evaluation, in that researchers should evaluate not only what they think is important, but also whether the learners themselves have met their own goals. What researchers consider measures of success or failure of an intervention may be very different from how learners value an intervention. Consequently, the validity of learners’ assessment of an intervention as part of a scientific evaluation of the intervention remained a contentious issue among the workshop participants. This is an example of how exploratory and participatory design methods need to be integrated and aligned with other research methodologies, such that interventions can be suitably evaluated from multiple perspectives.

Beyond this, the question remains of what particular research approaches are specific and inherent to research at the intersection of learning sciences, instructional sciences, and HCI. A major realization was that HCI research is ultimately interventionist, as it is design oriented. So, while deep understanding of given contexts is relevant to design, the ultimate goal of HCI is to construct design-relevant knowledge. Thus, it is critical to focus on the learning process in addition to outcomes. This was highlighted as especially important when the goal is developing skills rather than knowledge. Examining the process helps identify what was termed “struggle points” throughout the learning process and how these relate to, or can be mitigated by, the introduction of technology.

When talking about formal education in schools, the schools’ culture and disposition to learning, plus the level of emphasis on summative tests and ways of evidencing learning (e.g., high school versus primary) are found to have significant consequences on the design and evaluation of educational technology interventions.

How can we foster the relationship between learning and HCI? While learning has always been a subcurrent within HCI, the modern incarnation of the field of educational technology has largely grown around several subcommunities adjacent to but separate from HCI. In thinking about how we might build more connections between these communities and HCI more broadly, we asked what learning work might be able to contribute to HCI and what HCI work can contribute to learning sciences.

In some discussions, participants expressed how learning theories and practices might inform HCI and vice versa. For example, in the HCI traditions of participatory design, learning for the participants is one of the key elements, but few designers look to literature on learning to help structure their participatory design activities. Learning scientists see potential for their theories and knowledge to help shape this work [2]. In addition, participatory design practices have much to teach the learning sciences in terms of ways to invite teachers, administrators, parents, students, and stakeholders to be more involved in design or educational technology and information [3].

Another example is in learning analytics research, which has only recently started to substantially take up HCI methods to design and evaluate analytics and visualizations, such that in learning analytics literature, using HCI methods for design and analysis is considered a novel contribution (e.g., [4,5]). This illustrates that even between two deeply specialized fields such as analytics and HCI, which both could be considered to be highly related to computer science, a constant interdisciplinary communication and transfer of knowledge is necessary and challenging.

Another perspective gained from this question was the possibility of leveraging new insights from learning and education research to address a need for updated curricula around how to teach HCI concepts themselves. The landscape of HCI pedagogy has become broad, multi-institutional, and international. We see some initial work that highlights the potential for work on learning to inform how to judge which forms of HCI education are more or less effective for which uses, as well as inform best practices in the sharing of instructional materials [6]. As HCI education develops, the community needs to ask: What are core learning goals [7]? What is the balance between practice and research [8]? How do we identify the most successful pedagogy when teaching traditions in computer science, psychology, and design can be so divergent [9]?

An overarching issue in discussions was the relationship between research and (educational) practice that seems to be particularly relevant for this subcommunity. To understand learning and education, and to observe the expected impact, significant interaction with stakeholders outside of labs is required. While we recognize that laboratory studies have their place in HCI and other communities, we also recognize that the context of educational technology is in a broader practice of education and adoption and use. Though we can build technologies, test them in the lab, and prove that learning happens if they are used, we also need HCI research to design for teacher, administrator, and parent use so they are able and motivated to provide access to students from all walks of life.

We conclude with reiterating the goals of this SIG and the larger initiative to recognize learning and education within HCI research. We want to 1) discuss inclusive cross-disciplinary perspectives on learning, 2) define future directions and qualities for learning and education contributions in CHI, and 3) build a community across research/practice boundaries. By doing so, our hope is to not be stringent or imposing on what learning is or is not, while still basing HCI research on what is established knowledge in the learning sciences.

Moving Forward

The roundtable discussions were a highly interactive and rewarding format for discussions, such that at any time there were at least eight different discussions ongoing in the room. This was despite the fact that there were only three core questions asked, such that multiple tables, and multiple rounds of discussions, engaged with the same question. This highlights the high interest in learning within the HCI community, and the timeliness of the special interest group and the new subcommittee for learning, education, and families established at CHI 2019, and continued at CHI 2020.

Discussions at any time were also of a very heterogeneous level, showing that there is, as yet, no shared systematic way of answering the foundational questions of what learning means in an HCI community, how to foster learning through technology design, and specific characteristics of a high-quality contribution to the intersection between learning and HCI. We see this as a push, for ourselves as well as others researching and practicing at the intersection of learning and instructional sciences and HCI, to continue interdisciplinary discussion, and at the same time try to identify particular knowledge and perspectives that come out of the intersection of HCI and learning research.

Endnotes

1. Xie, B., Harpstead, E., DiSalvo, B., Slovak, P., Kharrufa, A., Lee, M.J., Pammer-Schindler, V., Ogan, A., and Williams, J.J. Learning, education, and HCI. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Paper SIG09. ACM, New York, 2019.

2. DiSalvo, B. Participatory design through a learning science lens. Proc. of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2016.

3. DiSalvo, B., Yip, J., Bonsignore, E., and DiSalvo, C. Participatory design for learning. In Participatory Design for Learning. Routledge, 2017, 3–6.

4. Ahn, J., Campos, F., Hays, M., and DiGiacomo, D. Designing in context: Reaching beyond usability in learning analytics dashboard design. Journal of Learning Analytics6, 2 (2019), 70–85.

5. Buckingham Shum, S., Ferguson, R., and Martinez-Maldonado, R. Human-centred learning analytics. Journal of Learning Analytics6, 2 (2019), 1–9.

6. Vorvoreanu, M., Gray, C.M., Parsons, P., and Rasche, N. Advancing UX education: A model for integrated studio pedagogy. Proc. of the 2017 ACM CHI Conference on Human Factors in Computing Systems. ACM, New York, 2017.

7. Churchill, E., Preece, J., and Bowser, A. Developing a living HCI curriculum to support a global community. Extended Abstracts on Human Factors in Computing Systems. ACM, 2014.

8. Gray, C.M., Stolterman, E., and Siegel, M.A. Reprioritizing the relationship between HCI research and practice: Bubble-up and trickle-down effects. Proc. of the 2014 Conference on Designing Interactive Systems. ACM, 2014.

9. Wilcox, L., DiSalvo, B., Hennemann, D., and Wang, Q. Design in the HCI classroom: Setting a research agenda. Proc. of the 2019 on Designing Interactive Systems Conference. ACM, 2019.

---

Posted in: on Tue, August 04, 2020 - 11:09:15

Viktoria Pammer-Schindler

Erik Harpstead

Benjamin Xie

Betsy DiSalvo

Ahmed Kharrufa

Petr Slovak

Amy Ogan

Joseph Jay Williams

Michael Lee