XXIV.1 January - February 2017
Page: 54
Digital Citation

Educational playgrounds: How context-aware systems enable playful coached learning

Oliver Korn, Alan Dix

back to top 

With edutainment and serious games, education has often been among the first domains to adopt new interaction paradigms. However, on the technology side, this domain remains conservative: Education is driven not by technology but rather by people. Thus, apart from examples like Moodle, MOOCs, and smartboards, much of HCI’s potential does not find its way into mainstream education. While we work on visions of smart homes, smart factories, and even smart cities, the idea of smart education is typically associated with top-level educators themselves rather than smart devices and augmentations. Here, we present the vision of a context-aware system that supports educators and offers students what we call playful coached learning (PCL).

back to top  Insights


Before going too far on the scope of this vision, however, we have to limit its field of application. PCL is not going to help students learn languages or teachers grade essays; it focuses on practical, hands-on learning scenarios by augmenting physical work areas. This emphasis on physicality really changes the audience. Despite the rhetoric of participation, the majority of MOOC participants tend to be highly educated and from higher socioeconomic levels. But by looking at informal learning, it is clear there is a strong practical-skills element in, for example, YouTube videos showing how to fold a photographic screen. PCL potentially opens up smart education to a wider group of people and tasks.


At the same time, PCL can improve the quality of academic and job-related education. Imagine a student in a STEM field who has to learn how to a fit a circuit board or how to assemble a mechanical arm. In such exercise or lab scenarios, there will typically be a student-teacher ratio of at least 10 to 1, often worse. This makes it hard for teachers to distribute their support adequately, as typically the students with greater problems will demand the most attention, while the ones doing OK or fine will receive little or no feedback. The sidebar “Playful Coached Learning: A User Story” illustrates how a PCL system would change this.

back to top  Technological Components—An Integration Task

On a technological level (see Figure 1 and technical setup sidebar), PCL builds on a context-aware system. It creates a 3D representation of the working area and the users, especially their hands, arms, and face. PCL recognizes the user’s actions and triggers the multimodal and gamified presentation of hints and instructions, typically by in-situ projection, tangibles, and audio. Based on the (ideally automatic) identification of the user, the system assesses a history of learning and skill development to detect changes and adapt to the individual difficulty or guidance level.

Several of these components, such as in-situ projection [1], have been in use for some time already. Even the combination of the core technological components like depth sensors, cameras, and projectors has been realized prototypically in other domains, including assembly processes in production [2], surgery in the health domain [3], and rehabilitation [4]. Finally, the concept of using this technology to apply gameful design to STEM activities has been envisioned before [5]. Thus, PCL is not a disruptive but rather a combinatory innovation.

back to top  Pedagogic Effects: Self-Correcting Materials and Early Feedback

One of the aims of Montessori education is to have self-correcting materials, for example stacking blocks so that the child knows when things have fitted properly, without a teacher saying so. A similar strategy called Poka-yoke (Japanese for “mistake-proofing”) is applied in engineering to make physical work processes more intuitive. Among other things, this approach generates two critical pedagogic effects:

Autonomy and reduced external judgment. A teacher identifying errors, however helpful, can be experienced as stressful; the feedback shifts the focus to external motivation (satisfying the teacher) rather than internal (getting it right). In some ways, PCL is an automatic coach. As interventions will be designed with a transparent and simple feedback mechanism, they are much closer to the physical activity and generate less distraction.

Early error detection. For many tasks, one is aware that something went wrong only well after the event causing the problem: for an electronic circuit when you turn it on, for route finding when you are lost, and so on. This late error detection causes frustration (extra effort) and requires complex diagnosis (where did I go wrong?). However, the most damaging is “practiced” erroneous behavior, which happens over and over. In contrast, PCL will feature a stealth mode that intervenes when errors are about to be made and, depending on the user’s level of guidance, offers potential solutions. For physical work, this is essential: Physical actions are learned tacitly, so erroneous physical training is hard to unlearn. In addition, the stealth mode reduces stress (a barrier to learning) as well as the perceived risk, the “what if I mess it all up?” feeling that blocks creativity and self-learning.

back to top  From Playful Education to Emotion-Aware Coaching

Playful approaches in learning are not new at all: They lie at the core of pedagogy. Accordingly, learning and play have always been interwoven, often struggling for dominance. While playful design or gamification are methods to integrate them, the resulting solutions often do not incorporate the user’s freedom of will, a quality philosophers like Bernard Suits deem essential for play [6]. Without the feeling of having a choice, playful education can create aversions: Though it is acceptable to be obliged to learn or to work, nobody wants to be forced to play. If a system becomes aware of the user’s real-world interactions, this does not solve the problem of free will. But it strongly contributes to the user’s sense of interaction and exchange, which in turn make his or her actions meaningful and raises the motivation to engage in potentially tedious processes like studying.

However, there is more to learning than the user’s interactions with artifacts. PCL envisions reaching the competence of a dedicated teacher (with enough time for the students). Thus, it is not enough to add gamification elements like badges, levels, and achievements. Neither does it suffice to know the student’s learning history and be aware of his or her current actions. A good coach must also consider a student’s emotions. While gamification and playful design help to raise overall mood and motivation, this remains a one-way street unless a system can interpret emotional cues.

From recent experience, it appears that in educational settings, only non-invasive techniques such as facial expression will be accepted for obtaining these cues. Even for this feature, the emotion analysis will need to be black-boxed; that is, emotion records will be neither externally accessible during a learning session nor saved afterward. However, during the research and design phase of the PCL system, we plan to use biosignals like galvanic skin response (GSR) or encephalography (EEG) as additional data sources [7]. While the aim is that the emotional cues from these invasive data sources and the non-invasive facial expression analysis will converge, we are well aware that reliable emotion recognition is highly dependent on advances in the field of affective computing. It is in this area that PCL will require the most development effort in order to create a truly satisfactory user experience.

back to top  Summary

PCL is a good example of a combinatory innovation: Most components are already in place, but they have not yet been combined and tailored to fit the field of education. We think a system that directly assists users in practical learning tasks will help increase the overall quality of education. Additionally, it will reduce the stress for trainers and educators who must teach large groups with limited time resources. A motivating learning experience that incorporates the emotional cues of the student will help to raise motivation for self-learning and contribute to practice and skill acquisition.

On the path toward PCL, there already is a first large-scale research project on the way: KoBeLU (context-aware learning environment). We are happy to collaborate with researchers from the areas of education, affective computing, pattern recognition, and machine learning. Therefore, if playful coached learning is something that might interest you or your students, do not hesitate to contact us.

back to top  References

1. Pinhanez, C.S. The Everywhere Displays projector: A device to create ubiquitous graphical interfaces. Proc. of the 3rd International Conference on Ubiquitous Computing. Springer-Verlag, 2001, 315–331.

2. Korn, O., Schmidt, A., and Hörz, T. Assistive systems in production environments: exploring motion recognition and gamification. Proc. of the 5th International Conference on Pervasive Technologies Related to Assistive Environments. ACM, 2012, 9:1–9:5; http://doi.org/10.1145/2413097.2413109

3. Rüther, S., Hermann, T., Mracek, M., Kopp, S., and Steil, J. An assistance system for guiding workers in central sterilization supply departments. Proc. of the 6th International Conference on Pervasive Technologies Related to Assistive Environments. ACM, 2013, 3:1–3:8; http://doi.org/10.1145/2504335.2504338

4. Korn, O., Brach, M., Hauer, K., and Unkauf, S. Exergames for elderly persons: Physical exercise software based on motion tracking within the framework of ambient assisted living. In Serious Games and Virtual Worlds in Education, Professional Development, and Healthcare. Information Science Reference/IGI Global, Hershey, PA, USA, 2013, 258–268.

5. Korn, O. Industrial playgrounds: how gamification helps to enrich work for elderly or impaired persons in production. Proc. of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, ACM, 2012, 313–316; https://doi.org/10.1145/2305484.2305539

6. Suits, B. The Grasshopper: Games, Life and Utopia. Broadview Press, Peterborough, ON, Canada, 2005.

7. Schmidt, A. Biosignals in human-computer interaction. Interactions 23, 1 (2015), 76–79; http://doi.org/10.1145/2851072

back to top  Authors

Oliver Korn is a full professor of human-computer interaction at Offenburg University in Germany. He started his career in industry: In 2003 he co-founded the Fraunhofer spin-off KORION, developing simulations and games. He now works on the convergence of digital technology and real life, focusing on affective computing, gamification, and augmented work and learning. oliver.korn@acm.org

Alan Dix is a professor in the Human-Computer Interaction Centre at the University of Birmingham and senior researcher at Talis. He has worked in HCI for over 30 years and is the author of one of the field’s major textbooks. His recent work is focused on learning analytics, pedagogic issues related to novel technology, and the future of the textbook. alan@hcibook.com

back to top  Figures

F1Figure 1. Vision of a user/student working with the PCL system.

back to top  Sidebar: Technical Setup of a PCL System

User / Student. Assessment of facial expressions (video camera) and potentially eye gaze (eye tracker). In the design phase, we will additionally track brain activity and skin conductance. The aim is to determine if the user is fatigued, stressed, or distracted, and eventually how motivated and happy he or she is.

Tools and Actions. Assessment of body and especially of hand movements (depth camera) to predict what the user will be doing. This is required for early error detection (and the “stealth mode”). Before the soldering iron makes an error permanent, cautious feedback prevents irreversible “expensive” mistakes. Analyzing tremor and manner of movement could also supplement the facial analysis. In addition, by measuring movement paths and task completion times, PCL might be able to assess the user’s skillfulness: tentative or clumsy versus confident and fluid movements.

Artifacts / Work pieces / Tangibles. Assessment of task progress and performance: How much has the student done, and how well have they done it? We will assess object profiles (using depth cameras, potentially stereoscopic cameras, and object recognition) to detect if physical tasks are performed correctly. In addition, tangible objects will serve as projection areas, containing, for example, guidelines or help videos.

back to top  Sidebar: Playful Coached Learning: A User Story

Peter is a mechatronics trainee at a car manufacturer. He really likes his training but feels a bit lost in the practical exercises. His instructor is often stretched thin, firefighting the major problems in a group of 15 students. This situation changes when prototypes of the new PCL table are set up. At the beginning, Peter is skeptical that the odd apparatus with sensors and a projector will really help. Indeed, during the first session, the system asks a few technical questions, projected on a whiteboard that the system asked him to place on the assembly grid. It is just a fancy multiple-choice app, he thinks.

However, this changes as the system asks him if he is up for assembling a mechanical arm. Peter had tried this task once, but somehow the arm did not move correctly—it is a tough challenge. The projector then highlights the trays to pick parts from and the correct assembly locations. It even asks him to turn the arm so that it can “see” what Peter is doing. Sometimes, when Peter is uneasy or hesitates longer, it offers to show a video of the current process. In addition, after each action it makes comments, sometimes critical but mostly positive.

Once Peter is just starting to solder a cable when the system beeps loudly and projects a red cross directly onto his hand—he is in the process of using the wrong type of solder. After installing the third of four joints, the system comments: “You can do better: You completed the first two joints in under three minutes each, and now you are taking more than five minutes.” Of course, he then installs the fourth joint in under three minutes. The system reacts by displaying “Achievement: Joint Venture.” While this reminds him of the games in his Steam library, he actually likes the continuous feedback, the scores, and the measurements. The PCL system seems to know what he can do, trying to challenge and motivate him. When the instructor comes by to watch, he agrees: a job well done. Next time Peter will try to assemble the arm in stealth mode, asking the system to offer no instructions and guidance and intervene only when errors are about to be made.

back to top 

©2017 ACM  1072-5520/17/01  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2017 ACM, Inc.

Post Comment

No Comments Found