Buzzwords like ubiquitous computing, the Internet of Things, and cyber-physical systems reflect how computing and communication have become the technological core of many things we use in everyday life. This fundamentally changes the impact of human-computer interaction. Many interactions in the real world, such as listening to music, unlocking a car, paying at the self-service check-out, and even pouring a beverage (see Figure 1) have already become user experiences strongly influenced by interaction technologies and software.
To envision, design, and implement engaging and enjoyable interactive experiences in this changing world, we need a solid and comprehensive understanding of the interaction technologies available. The opportunities and limitations for interaction between humans and computers (and everything based on computing technology) are today effectively determined by available input and output technologies. In this new forum we will scout interaction technology trends and relate them to the body of work in human-computer interaction. We will also discuss new basic technologies that have the potential to affect how we create user interfaces and design interactions in a world of ubiquitous computing.
While looking through the proceedings of CHI 2014, I wondered if the HCI research community is really setting trends in human-computer interaction or if the field is just following technology trends set somewhere else. Looking more closely, the answer is not simple. In short: Yes, the HCI community is greatly influenced by the technologies available, but at the same time it is not just following trends—it explores and creates new technologies as well. Looking back at the history of the field, there seems to have been a successful symbiosis between creating technologies and studying available technologies for several decades. Examining this more closely, it is apparent that there is great value in using and studying available technologies, as well as in creating new technologies to form the basis of future interactions.
Searching for the term multitouch in the ACM Digital Library reveals that the research community looked at this now omnipresent topic for the first time in 2002. Once multitouch phones became widely available in 2007, there was an exponential growth for several years, flattening in 2013. Figure 2 shows the number of search results for the term multitouch in the ACM Digital Library and in CHI proceedings (the data is not normalized to an increasing number of publications every year).
Looking at only these numbers is misleading. The HCI research community had been investigating the underlying concepts long before the term became mainstream. Bimanual interaction was already being researched in the 1980s and 1990s. Pierre Wellner's digital desk showed many of the interaction concepts, including several now common multitouch gestures, and work at Xerox PARC on Pads, Tabs, and Boards created the conceptual and to some extent technological basis for many current touch devices.
The number of publications indicates that the research community follows technology trends very closely and that a large number of researchers work with recently introduced technologies. This includes analytic work (e.g., uncovering the shortcomings of new technologies, assessing their impact on society, or measuring the performance of technologies) as well as creative and constructive work (e.g., using technologies to extend computing into new domains, appropriating technologies to build new interaction devices, or creating new interaction concepts based on technologies).
Entirely novel concepts and completely new technologies are less prominent in publications but are an integral part of HCI research. In many of our community's conferences, such as CHI, UIST, and TEI, new interaction technologies are highly valued. It is hard to name a single interaction technology that became successful in the mass market where similar concepts and technologies had not been published and discussed in the research community long before. Prominent examples range from pointing devices to gestural interaction and interactive watches. It should, however, be kept in mind that there are many interaction technologies published in the research community that will never make an impact in products. I still argue that many of these technological advances have value as intermediate steps, inspiring people to create new technologies and devices.
Research inventing new interaction technologies and interactive systems is less common, for many different reasons. The effort required to develop, implement, and study new interaction technologies or systems is typically huge compared with using or studying commercially available devices. The quality, and in particular the reproducibility, of the evaluation is typically lower with new interaction technologies, as they may work less reliably, and reproducing the results may require reimplementing hardware and software. A further point is that new interaction technologies prototyped in the lab, even if successful and promising, may lack immediate commercial potential. Unfortunately, current metrics in academia (e.g., the number of high-quality publications and technology transfer) set incentives for following trends rather than for creating them.
The phenomenon of research picking up technology trends is not new. Looking back at the field, technologies, from interactive command lines to graphical workstations to the Internet and social networks, have always shaped HCI research. In my view the reasons for following technology trends are pragmatic in several ways:
- The potential for new opportunities in human-computer interaction enabled through new technologies is massive and has not been explored fully when technologies hit the market. There are many low-hanging fruits to be picked that may even have a chance to impact peoples' lives.
- Commercially available technologies enable us to study interaction concepts that have been proposed before but could not be validated in realistic studies. With commercially available technologies, it is much easier to run clean and reproducible studies in realistic settings.
- Implementing new interaction concepts for widely available technologies has the benefit of being relevant to many people and includes a potential for commercial success.
- Creating new technologies is out of reach for many HCI researchers, in particular when it requires building sensing, actuation, or hardware systems.
Interaction technologies are typically designed for a specific market or with specific use cases in mind (e.g., Microsoft Kinect as game input). In research, such technology is appropriated and often put into a new context (e.g., gesture control in an operating theater) to experimentally validate new ideas for interaction techniques and applications.
The effort required to develop, implement, and study new interaction technologies or systems is typically huge compared with using or studying commercially available devices.
The downside of using off-the-shelf technologies and "just" conducting studies is that the capabilities of the device available will strongly impact the research. Looking at interaction concepts for gestural control, it becomes apparent that the limitations of the technology (e.g., temporal and spatial recognition of fingers with the Kinect) influence the interactions designed.
Available technologies have a clear impact on research that is carried out. For most research groups, the cost of technology plays a major role too. Once technologies become available to the mass market, they can and will be widely used in research. The following three examples highlight this relationship, in which the HCI community has been massively affected by the available mass-market technologies:
- Interaction in mobile settings. Mobile phones have become the dominant computing platform. Mobile devices have become cheap, ubiquitous, and easily programmable, so a lot of HCI research uses these mobile technologies. In many cases, research previously done in the stationary domain is redone in the mobile domain; the novelty comes through extending it to use cases with new requirements (e.g., Keystroke-Level Model for mobile users). Mobile technologies have fundamentally changed how and where people interact with computers; hence, understanding mobile interaction is commercially relevant.
- Application stores. Application research in the large is enabled by effective means for distributing applications and monitoring usage. The Apple App Store and Google Play are marketplaces where researchers can distribute applications to run experiments on a global scale. The innovation here can be seen in the method, which is to piggy-back research questions onto applications. This study method is not new, as companies with large user bases have followed similar approaches, but having it available to many researchers with little effort allows new research questions to be explored.
- Gestural interaction in 3D space. Tracking hands and fingers has been difficult with traditional cameras. With technologies such as Microsoft Kinect and LeapMotion, it became very easy to acquire the tracking information. With few programming skills, a researcher can now get the 3D position of an abstracted skeleton or a vector for each finger. This has enabled many researchers to create functional prototypes of interactive systems that use gestures as the input modality. The wide availability of these new input devices has provided the basis for making gestural interaction mainstream; research helps provide the usage scenarios.
There are many more examples of technologies that have strongly steered the direction of research in HCI. Further examples of technologies that act as enablers and have had a clear impact include 3D printing, high-resolution screens and 3D displays, mobile projectors, as well as 3G and LTE wireless technologies.
Interaction technologies are not exclusive to human-computer interaction. In particular, basic technologies often have their origins in other fields. The following example highlights this.
Measuring and applying electrical signals to the body. Over the past few years, the HCI community has been exploring technologies to measure EMG signals to recognize motion and actuate the human body by applying electrical signals. In medicine and biomechanics this is old news, at least from a technology perspective. Nevertheless, there is a clear scientific value in transferring knowledge between fields because the research questions and goals differ greatly.
Physical interaction, 3D printing, laser cutting, rapid fabrication, printing of circuits, displays and sensors, biomechanical modeling, and implanting electronics are only a few apparently new technologies. They are new to HCI but have been around in other disciplines for a while. In my experience, researchers from these fields sometimes think HCI is not doing original research, but rather only applying their technologies to create (at best, interesting) applications. I believe there is clearly a value in HCI researchers looking at these technologies and bringing an interaction perspective to these fields. But at the same time, we should not be ignorant of their achievements. There is a great potential for interdisciplinary research within areas of basic technologies (e.g., printing) and within the biomedical field.
When following trends and creating new types of interaction, it is important to step back from the actual capabilities of a given device or technology. It is fine to be inspired by technologies, but the limitations of the current technology available should not constrain the design. I strongly argue for a separation of new concepts and their implementation using available technologies. Following this approach, research can be more fundamental and forward-looking, and the actual realization based on current technologies will enable experimental research.
It is often argued that many of today's technologies are in search of a problem to solve. Technologies become available and they drive innovation, but are they useful to people? I think this strict separation of technology push and human need is unrealistic—it is a much more cyclical process. Humans have needs; technologies help to fulfill the needs; new human needs arise. This interaction between technologies and human needs can be well observed in human communication, from landline phones to social networking software. Therefore, it is clear that knowing about technologies is relevant to creating solutions that better serve human needs.
Over the coming issues, we plan to highlight different technologies and trends. This will include novel and exciting contributions to technological research in human-computer interaction, as well as exciting and inspiring technologies from other fields. By discussing new technologies, we acknowledge the important impact of technologies on our field and hope to inspire new research in HCI.
There are many technologies bound to become mainstream over the next few years. It is worthwhile to look back at some "old" papers on these "new" technologies to anticipate some ideas that will be reinvented (and hopefully improved) in the years ahead. Here are some of my guesses and suggestions for further reading:
- Television and game controllers start to integrate gesture and voice recognition. This works now much better than 35 years ago, but it is worthwhile to read up on .
- Eye-gaze interaction technologies are becoming cheap and may be in widespread use in the near future. Before inventing new concepts, it may be useful to look back at .
- Smart watches are getting popular, and interaction with small wrist-worn devices is not simple. Here are two pointers that may be useful to know [3,4].
1. Bolt, R.A. "Put-that-there": Voice and gesture at the graphics interface. Proc. of the 7th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, 1980, 262–270. DOI=10.1145/800250.807503; http://doi.acm.org/10.1145/800250.807503
3. Blasko, G. and Feiner, S. An interaction system for watch computers using tactile guidance and bidirectional segmented strokes. Proc. of the Eighth International Symposium on Wearable Computers. IEEE, 2004, 120–123.
4. Kim, J-S., He, J-S., Lyons, K., and Starner, T. The gesture watch: A wireless contact-free gesture based wrist interface. Proc. of the 11th IEEE International Symposium on Wearable Computers. IEEE, 2007, 15–22.
Albrecht Schmidt is a professor of human-computer interaction and cognitive systems at the University of Stuttgart. His research interests are at the intersection of ubiquitous computing and human-computer interaction, including large-display systems, mobile and embedded interaction, and tools to augment the human mind. He received a Ph.D. from Lancaster University.
Figure 1. A drink dispenser with a tablet computer as a user interface in a hotel in Gothenburg, Sweden. The graphics and animations are great, but the tangible experience is poor. In terms of overall user experience, the new computer-based design is lagging behind the traditional design.
Figure 2. Research that looks at multitouch became very popular after multitouch devices became commercially available. This figure shows the number of papers found in the ACM Digital Library and CHI conference proceedings that include the term multitouch.
Copyright held by author. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2015 ACM, Inc.