Jessica Cauchard, Mohamed Khamis, Jérémie Garcia, Matjaž Kljun, Anke Brock
The field of human-drone interaction (HDI) has been steadily growing over the past few years, especially as drone technology has become increasingly present in human spaces. The Covid-19 crisis has shown that "drones were ready"  and that, beyond taking photos and videos, drones could be used for a range of novel applications, from food and medication deliveries to population monitoring and law enforcement . This crisis has highlighted the presence of drones all over the world, showing us that drones are here to stay, revolutionizing our technological habits much like cars and phones did at different times in history, and achieving a strong presence—even stronger than that of ground robots.
The potential of drones to support us in everyday life seems to have been a sufficient incentive for people to accept their presence, despite uncertainties around topics such as security and privacy. Research efforts in HDI have brought forth a plethora of challenges and opportunities that are unique to this platform and that need to be addressed to ensure drones' sustainability. Their acceptance and rapid expansion showed different patterns compared with ground robots, and, so far, prior research work on interacting with ground robots does not appear to directly apply to flying ones. The complexity of flight adds new dimensions to be considered, so existing human-robot and human-computer interaction (HRI/HCI) methodologies need to be adapted for drone research. We therefore propose that these communities could greatly contribute to the work on HDI in supporting the future design, development, and integration of these new devices.
In this article, we present a roadmap of the grand challenges faced by the HDI community. This roadmap corresponds to the outcomes of a workshop held at CHI 2019 in Glasgow, U.K. , which brought together researchers and practitioners active in the field. We highlight opportunities and challenges that will drive future research, arranged in three themes: 1) dealing with increasingly autonomous drones, 2) drones in public spaces, and 3) interaction techniques beyond remote control. The article concludes with a discussion around the need to establish design and evaluation guidelines, as well as theoretical grounding for HDI research.
In addition to the traditional manual flight mode, drones are increasingly automated, that is, they can perform tasks automatically. However, they are not autonomous yet, since most regulations worldwide prevent flying machines from making decisions without any human intervention. Current applications of automated flights include, but are not limited to: fast delivery of supplies to isolated areas, search and rescue support, as well as inspections of vast areas.
The complexity of flight adds new dimensions to be considered, so existing human-robot and human-computer interaction methodologies need to be adapted for drone research.
The operation and integration of automated drones in existing airspace or in social spaces poses specific challenges that will be accentuated with autonomous behaviors. Technical challenges such as transitioning between indoor and outdoor positioning systems, landing and charging autonomously, and long-range communication are active research topics in the drone (UAV) community, including among manufacturers. Interaction challenges, as discussed below, are more idiosyncratic in nature and require significant future research. Here we describe some existing directions and possible future ones.
Beyond data capture. We envision that drones will be able to act in the physical world in much more complex ways. For instance, drones could support workers in completing various tasks in hard-to-reach places, such as at high altitude (e.g., cleaning windows and gutters), in dangerous terrains and situations (e.g., disarming landmines), or over obstacles (e.g., soldering an unstable bridge over a gorge). Technical advances are necessary for the drone to be sufficiently precise and stable, as well as dedicated visualizations and interaction strategies to automate and operate the drone to support workers in their tasks.
Human-drone partnership. Drones with increased capabilities also suggest the need for tools and standardized means of interaction, from full user control to full automation, as well as the ability for people to author automated and autonomous behaviors. Our community needs to investigate how to support on-the-fly reconfiguration and how to switch to manual control when it might be necessary to update planned activities for safety, efficiency, or legal reasons. Reciprocally, drones should be able to react to new situations and offer guidance to users in how they can best support them in their tasks. Existing approaches in air-traffic control or robot programming might prove successful but need to be thoroughly explored.
Social drones. The field of robotics has opened the path to social robots accompanying and supporting people in their daily life. Autonomous drones will be able to adapt their behaviors to users' habits or to learn new ones. They could live in homes, help people get dressed, bring them groceries, and even keep them company. They could further follow people outside, encourage them to exercise, or even navigate them to places. We envision social drones to build upon HRI research while presenting new challenges due to the drones' flying ability. We suggest that all drones should feature social abilities to communicate, not only with users but also with affected bystanders, supporting their acceptability in public spaces.
Drones are usually designed with a single communication channel between them and their pilots. Yet, in human environments, whether in private or public spaces, other people will be affected by the drones' presence; we refer to these people as affected bystanders . In current designs, pilots cannot communicate or convey information to other people via the drone, and affected bystanders have extremely limited options with which to interact with both drones and pilots. These needs for additional communication channels have materialized during the Covid-19 pandemic, where drones have been used for novel use cases, such as in remote medicine and to monitor social distancing, applications for which added communication channels would have improved people's experiences.
Affected bystanders have shown curiosity but also distrust, and sometimes fear, due to the lack of understanding of a drone's goals and intentions. We therefore need solutions to inform them of the drone's purpose, what its intentions are, who is controlling it, and so forth. This brings forward several questions, including: 1) What is the best way to communicate information via the drone? 2) What kind of information can be conveyed? 3) What form factors (e.g., size, shape) and behaviors (e.g., movement, approach) best match different goals and intentions? and 4) How does the drone's ownership impact the perceived privacy of bystanders? Potential solutions include the use of: onboard documentation; fixed visual design, such as branding, to represent the owner's or the drone's purpose; adaptive visual feedback, such as lights (as used for maritime signals) or the projection of information on surrounding surfaces; audio feedback; and the use of a swarm of drones conveying information through various flight formations. This information can also be conveyed externally, for example, using dynamic public signage to state drones' operations in a dedicated area, or more specific information such as altitude, speed, or the data being collected.
|Drone users are shifting from experts to lay users.|
|Person interacting with a drone using body and mid-air gestures .|
We also suggest that new interaction strategies are needed to query a drone's plans or to request taking control of the drone (to various extents) by inputting commands without having access to the remote control. For this, various sensors can be mounted on drones to enable human input. Researchers, designers, and manufacturers have explored various possibilities, such as texting the drone and interacting with projected information, as well as using gestures, QR codes, maps, gaze, voice, brain-computer interaction such as electro- and magnetoencephalography (EEG and MEG), electromyography (EMG), and combinations of these, thereby creating multimodal interfaces [2,5,6]. While drones are envisioned to be integrated into communities and become more prominent in our daily lives, there is a lack of understanding of how to best deal with multiple users or crowds, and of how to design multi-user interaction or shared control with drones in public spaces. Another under-investigated area is turn taking. While this topic has been researched in ground robotics, it is not clear how and when humans and drones should take turns in social interactions. This underlines the need for novel interaction techniques beyond remote interaction.
There are myriad promising applications for HDI, and we posit that the type of interaction will likely depend on the application at hand. In particular, as presented in the previous section, we suggest that the use of a remote control becomes obsolete in collocated interaction with autonomous drones, where affected bystanders and drones can directly communicate with each other. Moreover, drones can be thought of as flying interfaces that can additionally provide haptic feedback and tactile interaction. While drones and tactile may not appear compatible due to the drone's external propellers, studies have shown that people get close to drones and are often interested in touching them . During the workshop, we discussed ideas for HDI applications that leverage these types of feedback to deliver great potential, including using drones for haptic actuation. For example, drones could be used to support people exercising by pulling them back for resistance training, or could deliver tactile feedback, as already demonstrated in current prototypes, to increase realism in gaming and virtual reality environments, or to support visually impaired users.
Thanks to new form factors, we are witnessing the introduction of safe-to-touch drones. This novel affordance is bringing new opportunities to the field so that people can directly interact with a drone by touching it, moving it, and even displacing it . Touch is then possible in collocated environments when the drone is within arm's reach. How touch is performed is not trivial. For example, when applying pressure on a drone, the flight path is modified, affecting the drone's stability. A technological leap is needed to ensure that drones will be able to fly within the constraints of safe touch and beyond. In addition, the tactile feedback received when touching the drone needs to be taken into account.
Haptic feedback can be designed so that the user feels specific sensations when touching the drone. Several techniques have recently emerged using the drone as a proxy to a virtual world. Haptic sensations are additionally provided by the propellers' movements and the wind generated. Haptic channels need to be further investigated to fully understand how the final interaction would be perceived. It is important to note that any changes in the drone's movement additionally affect the sound it produces, so that haptic input actually receives multimodal feedback (audio, visual, and haptic).
As drones become increasingly autonomous, this tactile modality can be considered as an option for "safe collision" without the drone backtracking from the user when the person is reaching out. We imagine that, depending on the task at hand and the level of autonomy, the tactile modality will have to be coupled to other modalities of use, such as voice and gestures. We further wonder how tactile interaction will evolve with the use of multiple drones and swarms, so that swarm behaviors could be affected by the physical displacement of one drone only, which influences the whole swarm (as in ). Working with swarms presents specific research challenges: scalability being a major aspect, in addition to autonomy and shared control mechanisms.
We propose that the interaction challenges are technical, social, and regulatory. This research space needs to be further defined and investigated so that we can build a future where people could safely interact with drones.
When specific types of interfaces become mainstream, guidelines for design and evaluation emerge—as has been the case for, for instance, graphical user interfaces. Design and evaluation guidelines are recommendations that intend to provide clear instructions and advice to designers and developers on how to implement and evaluate specific principles, such as efficiency, effectiveness, learnability, and consistency (https://www.interaction-design.org/literature/topics/design-guidelines).
With the use of drones becoming more common, it is essential that our community establishes theoretical foundations and HDI-specific design guidelines. We suggest that future HDI researchers must address the following challenges: 1) Who is communicating, and where is the locus of control? For example, we can consider one person to one drone, one person to several drones, several people to one drone, several people to several drones. 2) How can we teach high-level behavior to the drone so that it is not necessary to constantly control it? For example, how will the drone communicate that it is here to help someone? 3) How do we provide baselines that can create comparable results between interaction designs? For example, how can we generate predefined scenarios for HDI and tools and metrics for evaluation. 4) How do we consider cultural nuances in HDI? For example, how can we adapt drone behaviors to social norms?
With the use of drones becoming more common, it is essential that our community establishes theoretical foundations and HDI-specific design guidelines.
The development of these design guidelines will require knowledge exchange between researchers and designers. Forums exist but are limited to specific topics and fields. We suggest that a repository of use cases and results could help in this regard. Furthermore, the development of the field needs to encompass exchanges of both a software and hardware nature, such as by providing open source software or designs for 3D printing bespoke drones. Another common problem in technology development is the standardization of data protocols, which the community should aim for. Currently, different technologies exist for controlling and exchanging data with and between drones, and each drone is typically delivered with its own application programming interface. Some researchers and designers adapt those interfaces to their own needs , but generic environments also exist (https://wiki.paparazziuav.org/wiki/Main_Page).
Finally, developing design and evaluation guidelines is a lengthy endeavor, and this emerging community needs time and experience to filter out best practices. We believe that it can learn from other young communities that have emerged, faced similar challenges, and blossomed over the past years, such as AutomotiveUI and SIGACCESS.
There is a clear growing interest from the research community, industry, and among designers to leverage advances in drone technology and realize the many promising applications of HDI. This article discusses how the current transition to fully autonomous drones paves the way to novel applications and future opportunities. HDI is particularly promising in human environments, allowing for a full transition from drones being used as tools to drones becoming social beings, able to help users and affected bystanders alike. These developments underline the need to further research interaction strategies for bidirectional human-drone communication, beyond traditional remote control.
In this article, we compile a series of promising directions and research questions that can be used as a roadmap by the HDI community. In particular, we highlight a number of technical, methodological, and conceptual challenges that are encountered when designing interactive drone applications, and that may impede the acceptability of specific applications—even the adoption of drones altogether. We emphasize that one of the most important steps will be to compile guidelines and theoretical models for the design and evaluation of future HDI technologies and applications. Creating a strong interdisciplinary community will facilitate this—one that includes researchers and practitioners from HCI, HRI, design, ubiquitous computing, and privacy. Finally, while rich in ideas, this emerging community also needs to collaborate with researchers from other fields such as materials science, physics, and mechanical and electrical engineering, as well as with legislators, to provide solutions that would address all the needs of human-drone interaction research.
We thank Markus Funk for co-organizing the workshop and all participants who contributed to the discussions and activities.
1. Williams, A. The drones were ready for this moment. The New York Times. May 2020; https://www.nytimes.com/2020/05/23/style/drones-coronavirus.html
3. Brock, A.M., Cauchard, J., Funk, M., Garcia, J., Khamis, M., and Kljun, M. Ihdi: International workshop on human-drone interaction. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2019; https://doi.org/10.1145/3290607.3299001
4. Baytas, M.A., Çay, D., Zhang, Y., Obaid, M., Yantaç, A.E., and Fjeld, M. The design of social drones: A review of studies on autonomous flyers in inhabited environments. Proc. of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2019, 1–13; https://doi.org/10.1145/3290605.3300480
5. Khamis, M., Kienle, A., Alt, F., and Bulling, A. Gazedrone: Mobile eye-based interaction in public space without augmenting the user. Proc. of the 4th ACM Workshop on Micro Aerial Vehicle Networks, Systems, and Applications. ACM, New York, 2018, 66–71; http://doi.acm.org/10.1145/3213526.3213539
6. Cauchard, J.R., E, J.L., Zhai, K.Y., and Landay, J.A. Drone & Me: An exploration into natural human-drone interaction. Proc. of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, New York, 2015, 361–365; http://doi.acm.org/10.1145/2750858.2805823
7. Braley, S., Rubens, C., Merritt, T., and Vertegaal, R. Griddrones: A self-levitating physical voxel lattice for interactive 3D surface deformations. Proc. of the 31st Annual ACM Symposium on User Interface Software and Technology. ACM, New York, 2018, 87–98; https://doi.org/10.1145/3242587.3242658
Jessica Cauchard is an assistant professor in the Department of Industrial Engineering and Management at Ben Gurion University of the Negev in Israel. She is the director of the Magic Lab, a research group focusing on the design, development, and evaluation of interaction techniques for novel devices such as drones. email@example.com
Mohamed Khamis is a lecturer/assistant professor at the University of Glasgow. He received his Ph.D. from LMU Munich. His research lies at the intersection of human-computer interaction and security. Within human-drone interaction, he is interested in the privacy implications of drones and how users interact with social drones. firstname.lastname@example.org
Jérémie Garcia is an assistant professor in the interactive informatics team at ENAC, Université de Toulouse, France. He is studying new ways to interact with complex or automated systems, such as drones or air traffic control, that allow users to express, develop, and evaluate ideas in effective and creative ways. email@example.com
Matjaž Kljun is an associate professor in the Department of Information Sciences and Technologies, University of Primorska, Slovenia. He also co-directs the HICUP lab and is a co-founder and chair of the ACM SIGCHI Chapter Bled, aiming to promote HCI and UX in the country. His research interests span numerous fields related to human-computer interaction. firstname.lastname@example.org
Anke Brock is an assistant professor at ENAC, Université de Toulouse, France. Her research interests are at the intersection of human-computer interaction, aeronautics, and accessibility. email@example.com
Copyright held by authors. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2021 ACM, Inc.