Yomna Abdelrahman, Albrecht Schmidt
Human visual perception is the ability to see, process, and understand stimuli in an environment. Despite its capabilities, this system is limited by the visible luminance range that it can detect. In fact, the perceivable spectrum of the human eye comprises less than 1 percent of the electromagnetic spectrum. Inspired by other creatures' advantages in visual perception and our own desire to extend our perceptual capabilities, we have built extensive tools to enhance our vision, such as lenses and glasses. But while these tools enhance the quality of the perceived spectrum (e.g., sharpening images for shortsighted users), the research focus has shifted to building tools that let us see a wider spectral range, to uncover what is naturally invisible. For example, thermal imaging lets us view the thermal map of a scene, adding an extra layer of information about the objects in our vicinity, enabling novel interactive experiences with our environment.
Thermal imaging application history. It's apparent that thermal imaging is now expanding beyond military, medical, and other specific context usages to more diverse applications. Looking into the published research over the past 20 years, it's clear that thermal imaging has gotten the attention of HCI researchers, enabling novel interaction techniques and modalities (Figure 1).
|Figure 1. Application domains of thermal imaging over the past 20 years.|
Thermal cameras capture the thermal map of a scene. They operate in the far-infrared spectrum with wavelengths between 7.5 and 13 μm. There are multiple differences between properties of thermal imaging and those of visible light. For instance, thermal imaging is light invariant—thermal cameras operate regardless of the lighting conditions (Figure 2).
|Figure 2. (left) RGB view; (right) equivalent thermal camera view.|
Seeing through the past. Thermal cameras can capture the heat traces left behind due to the heat transfer from recent interactions. These traces do not decay instantly, enabling the sensing of past interactions. Such a feature could be deployed in various use cases, from interactive surfaces [1,2] to enhanced surveillance systems. Consider Figures 3 and 4, which show the traces of someone passing by even after they have left.
|Figure 3. Step traces after the user is gone.|
|Figure 4. Traces of object interaction.|
Mirrored sensing. Another unique feature of thermal imaging is its different reflective behavior compared with RGB. Consider Figure 5, where a metal surface exhibits a mirror-like reflection in the thermal spectrum, yet is not reflective in the field of view (FOV) for the light spectrum.
|Figure 5. (left) RGB reflection behavior; (right) thermal mirror-like reflection for the same surface.|
In our early research, we investigated how to exploit this feature to extend the interaction space (Figure 6) and introduce a novel interaction modality . Clearly this feature is material-dependent, so we analyzed the relevant properties and depicted a design space supporting thermal reflection interaction . We envision that novel modalities or dimensions of interaction can pave the road for novel interactive systems, for instance multi-user interaction or even gaming, where the user in front of the camera is the active user. Additionally, it could add a "smart" aspect to existing devices, such as making any smart TV gesture-enabled (Figure 7).
|Figure 6. (left) The field of view (FOV) of the camera in gray. (right) The extended interaction space due to thermal reflection, highlighted in colors.|
|Figure 7. Potential use cases for thermal reflection interaction.|
Thermal biometrics. A body of recent work recognizes the need for a more transparent usable authentication mechanism. To be truly usable, authentication should be seamless and require no explicit actions from users. Biometric authentication is a promising approach because biometric characteristics can be used to simultaneously identify and authenticate a user. The ability of thermal cameras to detect subtle temperature changes enables the unique vein patterns on the back of our hands to be captured. We investigated using veins on the back of the hand for contactless and seamless authentication (Figure 8). It is invariant to changes in how the hand is posed as well as changes to the environment. Being accurate and invariant, vein-based authentication has the potential to seamlessly authenticate users of desktop computers and tabletops.
|Figure 8. Veins from the back of the hand captured by a thermal camera.|
The dark side of thermal imaging. As the writer David Wong once said, "New technology is not good or evil in and of itself. It's all about how people choose to use it." Although thermal imaging enables seamless authentication and novel interactive techniques based on the heat traces, it raises a threat in the form of thermal attacks. As shown in Figure 9, the heat traces could be extracted from smartphone screens and used to infer the PIN or pattern used for authentication. In our exploration , we found that the most-used smartphone authentication techniques, namely PINs and patterns, could be uncovered with up to a 100 percent success rate, even 30 seconds after authentication.
|Figure 9. Heat traces for sample PIN and pattern.|
A window into our mind. A technology with the capability of sensing and inferring physiological signals has the ability to provide a window into our mind that can be used to adapt system behavior accordingly. Internal-state information is largely invisible from the outside of a user's brain, and introspection often fails to reason about it in an unbiased and objective way. However, our emotional state influences our body temperature, in particular our facial temperature (Figure 10), as the face contains critical blood vessels connected to our autonomic nervous system. Using thermal imaging, these cues give glimpses into our internal states, such as our cognitive load. Thermal cameras overcome the limitations of the contact sensors used in related research. It is also more robust than other contactless approaches, since the temperature signature is more resistant to conscious manipulation [4,5].
|Figure 10. Facial temperature captured by a thermal camera.|
Thermal imaging technologies. Thermal imaging has entered the commercial market and is available in different forms, such as integrated into smartphones and as attachments to phone or USB cameras (Figure 11). Reductions in price and size come at the cost of resolution and thermal sensitivity. Noncommercial thermal cameras can detect subtle changes in temperature (up to 0.005°C), but while this ability is reduced in most commercial cameras (up to 0.04°C), it does not affect their potential utility. Interestingly, although thermal imaging operates in a different spectrum from RGB, known computer-vision techniques that use existing open source libraries (e.g., OpenCV; https://opencv.org/) work with the images generated from thermal cameras. Additionally, having the temperature information for each pixel enhances existing techniques such as person-detection, as our body temperature can be easily extracted from the background. Applying the typical image-processing techniques shown in Figure 12, different features could be extracted, such as touch points, facial temperature, and vein patterns.
|Figure 11. Evolution of thermal cameras in terms of size and cost. Source: https://www.flir.de|
|Figure 12. Thermal-image processing for diverse features extraction .|
Thermal imaging over the next 10 years. Thermal imaging has witnessed a vast increase in application as well as specification, where it has been involved in HCI research, for instance, in building novel interactive techniques. We envision that with the DIY movement, where sensors are commercially available and building your own hardware becomes more common, thermal imaging will be further deployed in a variety of use cases, from building interactive systems based on thermal imaging to replace current technologies, to deploying it in educational settings to monitor cognitive load, to replacing current authentication techniques. Furthermore, thermal cameras could be included in all commercial smartphones, as a feature as indispensable as current cameras. This will open up a new field of research in HCI, namely the implications and privacy concerns raised by using cameras that operate in the non-visible spectrum.
1. Abdelrahman, Y., Sahami Shirazi, A., Henze, N., and Schmidt, A. Investigation of material properties for thermal imaging-based interaction. Proc. of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, 2015, 15–18; https://doi.org/10.1145/2702123.2702290
2. Sahami Shirazi, A., Abdelrahman, Y., Henze, N., Schneegass, S., Khalilbeigi, M., and Schmidt, A. Exploiting thermal reflection for interactive systems. Proc. of the 32nd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, 2014, 3483–3492; https://doi.org/10.1145/2556288.2557208
3. Abdelrahman, Y., Khamis, M., Schneegass, S., and Alt, F. Stay cool! Understanding thermal attacks on mobile-based user authentication. Proc. of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2017, 3751–3763; https://doi.org/10.1145/3025453.3025461
4. Abdelrahman, Y., Velloso, E., Schmidt, A., Abdelrahman, Y., and Schmidt, A. Cognitive heat: Exploring the usage of thermal imaging to unobtrusively estimate cognitive load. Proc. of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 13, 3 (2017), Article 33; https://doi.org/10.1145/3130898
5. Ioannou, S., Gallese, V., and Merla, A. Thermal infrared imaging in psychophysiology: Potentialities and limits. Psychophysiology 51, 10 (2014), 951–963; https://doi.org/10.1111/psyp.12243
Yomna Abdelrahman graduated from the German University in Cairo in 2010. She then earned her master's degree in international computer hardware and software (INFOTECH) at Stuttgart University, Germany. She completed her Ph.D. on thermal imaging and novel interactive systems in the hciLab at the University of Stuttgart. Recently she joined University of Bundeswehr Munich as a post doc. firstname.lastname@example.org
Albrecht Schmidt is a computer science professor at the Ludwig-Maximilians-Universität in Munich. He works at the intersection of ubiquitous computing, digital media, and human-computer interaction. His research interests are in creating digital technologies to augment perception and the human mind. He has a Ph.D. from Lancaster University. email@example.com
©2019 ACM 1072-5520/19/01 $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2019 ACM, Inc.