Vega LTE Launching
One day I was watching a TV commercial for Apple’s iPad2. The script definitely had an Apple ring to it. The ad began with, “Now we can watch a newspaper,” and ended with, “and touch the stars!” This upset me at first, because it seemed to overstate what the product could do and sounded a little too clever. Later, however, I found I could not deny any of its claims. Hmm, I thought, Apple has done it again.
The ad struck me because I realized its message should in fact be one of the core principles of interaction design! More specifically, I believe the ad revealed one of the most important secrets of “digital material,” in terms of what users actually manipulate and interact with when using their interactive devices. The secret is this: The conventional physical constraints on our interactions with digital material are being removed. (The term digital material is adapted from Jonas Löwgren and Erik Stolterman’s book Thoughtful Interaction Design , and I redefine it in the following manner: Digital material is any manifested and experiential phenomenon experienced through digital means.)
Apple’s ad also highlighted what users really care aboutthat is, what they like to interact with. What’s important to users is the content itself, not the device and the interface, which only provide the channels for the user to reach the core (i.e., the content).
This brief but significant experience led to new insight regarding what interaction design actually isor should beand what will come in the future. We need to loosen or even break the boundary of a specific interface type and shift our focus to the core value of the items with which we actually desire to interact. At the very least, the starting point of our designs should be the content, instead of the peripheral means to support our interactions with it.
Fingertip Touching: The Driving Source of True Naturality in Interaction Design
Digital materials were alien when they first came into the world. Laypeople could not manipulate them without an interface. We needed expertise from areas such as psychology and computer science to make our computing devices usable. With much help from these fields, interfaces to access and manipulate “computing materials” have evolved to make this easier and more effective. Consider touch-pad devices. Returning to the iPad commercial, now it does not seem awkward to flip through the pages of a digital book and explore 3-D space on a touchscreen. The analog look and feel of digital space has become more natural, and we no longer see it as a cheesy imitation of things in the physical world. With the touchscreen, digital items are directly tangible without any mediator to manipulate them. This direct fingertip touching of digital material is what brings, I think, the naturality of interaction.
It is impossible to enumerate how many things in my life I have touched with my fingertips. Since birth, touch has been one of the most natural ways of learning new things and understanding the world around me. The countless number of nerve endings on my fingertips combined with my visual sensorsmy eyeshave allowed me to accumulate all of my experiences of the world around me. As we now start to “touch” digital material directly with our fingertips, a new learning channel to the digital world opens up. Apple may have understood the significance of fingertip touching quite deeply. It has even changed the rules of the touch trackpads used on laptops. Unlike conventional trackpads, scrolling on new Apple laptops corresponds directly to the movement of a user’s fingertips (i.e., moving one’s fingertips down on the trackpad means the content will move down, and vice versa). We have now entered the era of real touch of virtual digital materials with interactive digital artifacts, handling them as we do physical materials. Our accumulated memories of touching things with our fingertips also allow us to project sensory experiences onto what we touch on a flat screen. If this is true, HCI researchers and designers should now rethink what we actually need to design for in interactive artifacts.
However, the image conjured by this advancement may not always be bright. If we consider very young children who face these touch devices not long after they are born while building memories of touch experiences with physical things in parallel, it is not clear how their perception of the physical and digital worlds will be shaped or how the symbiosis of these two worlds will affect their ways of thinking and living in the future. This could be an interesting area to research.
Is the Focus on Gesture a Good Thing?
When the touch interface was introduced, the concept of gesture interface design became significant. All fingertip movements came to be viewed as gestures. But the process of directly adapting the gestures we normally use for human communications in different cultures, such as handshakes, greetings, and signal gestures, into the context of interface design is ineffective. This effort tends to confuse our understanding of gestures in HCI.
The term gesture interface seems misplaced. Gestures on a touch interface are mostly naturally emerging phenomena, which arose due to the pursuit of naturality in interaction. We do not think of gestures in our interaction as messages for communication, unlike the commonly used gestures in our daily lives. Interaction gestures basically serve to manipulate the digital materials we “touch.” Even remote gestures, especially when they are analogous to the naturally emerging gestures we use to manipulate digital materials, can be considered natural. For example, the recent smartphone model Sky Vega introduced a gesture-recognition feature through which a user can take an incoming phone call even when the touching action is not possible (Figure 1). The Microsoft XBox 360 with Kinect technology is also a good example of an appropriate application of gesture interface. The use of gestures here is not primarily for communicating with the machine, but rather more for directly interacting with the content in the virtual world through real, natural, physical actions. People do not have to remember any symbolic-language-like gestures.
|Figure 1. Screenshots from the Sky Vega LTE advertisement for the gesture-recognition feature: A user taking a call while she is making dough.|
In this regard, whether or not we design gesture interactions is not an important issue. The key question is why gestures, and what do we care about when designing gestures in our interactions. These issues matter because if we do not care about them, no gesture will be natural. It will most likely be executed merely for the sake of making a gesture. Donald Norman also commented on current approaches in gesture interfaces: “Most gestures are neither natural nor easy to learn or remember” . What we must design for gesture interactions should not be a set of prescribed rules, as we see with GUI examples. As Norman further states, “Gestures are ephemeral” . People may not easily (or ever) remember what input units they made when they are gesture-based. People do not think about their input behaviors. They care about what they do with the target content and what they accomplish through their interaction with this content.
Disappearing Interfaces: Interactivity Instead of Interface
At present, the seamless symbiosis of the physical and digital worlds, which we have longed for since we were first introduced to computers, is beginning to be realized. And this merging will come to life when the interface starts to disappear. Many others have noticed this phenomenon as well, including well-known visionaries such as Norman, who wrote in The Design of Everyday Things, “The real problem with the interface is that it is an interface. Interfaces get in the way. I don’t want to focus my energies on an interface. I want to focus on the job” . Nicholas Negroponte adds, “Therein lies the secret to interface design: Make it go away” .
Their visions are finally starting to come true. From this perspective, the debate on which types of interfacesGUI, TUI, touch, gesture, speechare better than others may not be fruitful. Whichever interface type is used, the important thing becomes what people can do instead of which interface people can do it with.
In the era of interface design, the visible and cognitive qualities of interfaces normally significant to people’s eyes are what determine the quality of the design of interactive artifacts. However, now with touch-interface devices, such as the iPhone and Android smartphones, we face similar form factors and graphical qualities. The differentiation between these designs is focused more on the qualities of interactivity than the interface. The direct fingertip-touching action produces a more significant user experience, as it’s based on the quality of touching as opposed to merely looking.
Thus, what should be the target of our research activities? Shouldn’t we extend our focus from new interface materials, platforms, and styles to creating the best experience when interacting with the target content itself? Regardless of the interface type, users will fall in love with what they interact with if they are fully engaged with what they care about. The unnecessary decorative animations and “wow effects” of inputs will eventually die out if they are ends rather than means.
Some Thoughts on the Future of Interaction Design and Research
One of my research interests promotes this new direction for interaction design. In some of those studies, we defined ways of conceptualizing and concretizing the quality of interactivity so that designers can consciously design the various qualities of interactivity of an interactive artifact as they intended [5,6]. With this approach, we intentionally separated the concept of interactivity from any visible interface. It is all about what users feel at the moment they interact with digital materials.
How will this phenomenon of the disappearing interface affect the fields of HCI and interaction design? Here are some possible future developments:
- The HCI research space will shift from interface invention to quality exploration. As I mentioned in my recent research examples, this movement toward a disappearing interface will create new ways of thinking in interaction design. We will need to create new methods, tools, and courses to teach and design interactive artifacts. In line with this movement, we will also see more thriving cases of smart material inventions, which will enhance the intuitiveness and naturality of interactions.
- The exploration of the effects of interactive quality design will be an important research subject. Interactive quality design for most natural physical feelings is one possible research area. Based on how we manipulate interactive qualities, people can experience feelings of certain things that do not actually exist in reality. Other than this, research on how the operational feelings of digital materials relate to their interactive qualities will be another interesting topic. In addition, as stated earlier, what these “pseudo feelings” mean to people when they start to experience them at a young age will be another important issue to investigate.
- Treating interaction design targets as if they are alive will open up a new design space. In this regard, the field of human-robot interaction (HRI) will further evolve and thrive even more in interaction design. We already have the example of Siri, which although not a robot in the strictest sense, has become quite successful. When we think about robots, we do not particularly consider creating artificial interfaces to interact with a robot. The channels that allow us to interact with a robot are the direct parts of its body, as if its hands and feet and eyes and ears were not artificially attached to it and could access or be accessed by outer elements. If I extend my view of what we interact with from an artifact to a new material, the recently proposed concept by Ishii et al., radical atoms , also resonates with the direction I foresee here.
- The research effort on defining standardized sets of gestures or input languages will decrease. Commands that are extremely useful because they are digital, such as undo and redo, may still require some standardized symbolic languages for a while. However, as time passes I think such commands will be replaced by more natural means, such as actual verbal language. Nevertheless, the advancement of sensing technologies will continue. Precise and prompt detection of people’s movements and gestures will contribute to more natural interactions.
- In the same vein, standardized platforms of interfaces such as the GUI will also decrease. This will be one of the core phenomena we will face as the concept of the disappearing interface enters our reality. The ways of focusing on the target content that users actually care about and interact with will be the most important issue in interaction design. We should research and develop more guides and tools to support this in the near future. I believe this has already begun.
Digital materials are what overcome or ignore physical constraints. This is the secret of digital material. Thus far, we have enabled the overcoming of physical constraints with the use of formalized interfaces. However, we now touch and manipulate digital materials directly. The secret of digital materials is realized when this fingertip touching of the digital material becomes possible. To maximize the benefits of this secret, I believe we should address this immediacy of access and the use and manipulation of digital materials in interactive design, instead of continually focusing on interface types and styles (although such inventions will continue and are necessary). Focusing on the core of what we interact with rather than the interfaces that mediate the interaction is what matters. That is what the concept of the disappearing interface is about, and that would be the true nature of a natural user interface.
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science, and Technology (2010-0023206).
5. Lim, Y-k., Stolterman, E., Jung, H., and Donaldson, J. Interaction gestalt and the design of aesthetic interactions. Proc. of the 2007 Conference on Designing Pleasurable Products and Interfaces (Helsinki, Finland). 2007, 239254.
Youn-kyung Lim is an associate professor in the Department of Industrial Design at KAIST in South Korea. Her current research focuses on experience-centered design and the aesthetics of interaction, as well as prototyping in interaction design, especially for discovery-driven creativity.
©2012 ACM 1072-5220/12/0900 $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.