Tangible and embodied interaction is a diverse research field, in terms of both the range of interaction techniques that are explored (such as tangibles, wearables, and body gestures) and the disciplines that are involved (HCI, design, psychology, physical computing, and interactive art). What is common to the majority of practitioners is a desire to design ways of physically interacting with computers that fit with our innate abilities. Philosopher Andy Clark proposes that one distinction between well-fitted and poorly fitted technologies is the extent to which they require a person's attention during use: "A transparent technology is a technology that is so well fitted to, and integrated with, our own lives, biological capacities, and projects that they are (as Mark Weiser and Donald Norman have both stressed) almost invisible in use. An opaque technology, by contrast, is one that keeps tripping the user up, requires skills and capacities that do not come naturally to the biological organism, and thus remains the focus of attention even during routine problem-solving activity" . Clark points out that technologies typically require a period of training before they become transparent in use, and the shorter the training period, the more fitted they are to our innate abilities. Similarly, in psychology, dual-processing theory makes a broad distinction between automatic and controlled processing in human performance. An automatic process, like transparent tool use, occurs without requiring active control or attention and typically develops through training. In contrast, a controlled process does require attention but can be used in novel situations when a person cannot use learned automatic behaviors. Extensive laboratory testing has shown that these processing modes are complementary and that people transition between them depending on their situation and goals.
The focus of cognitive psychology research has been on quantifying subjects' performance, rather than on investigating changes in their phenomenal experience. However, there is a notable difference between the phenomenal experience of using a transparent technology, in which a person's attention is on the task being performed and they have little or no awareness of the technology itself, and using an opaque technology, which remains part of a person's phenomenal experience and is actively attended to. Distinguishing different modes of tool use, and, more generally, ways of engaging with the world, on the basis of phenomenology was first proposed by Heidegger, whose work has been particularly influential in the field of tangible and embodied interaction. He argued that in one mode, ready-to-hand, a person uses a tool as though it were an extension of themselves; their focus is on the task they are trying to accomplish, and they are unaware of the toolit is phenomenally transparent. In another mode, present-at-hand, a tool is treated as a distinct and separate entity and a person is aware of its properties, such as size, shape, color, and massit is phenomenally present. Heidegger also described a third mode of tool use that lies between the two extremes of ready-to-hand and present-at-hand, which he called un-ready-to-hand. This mode can result from the disturbance of previously ready-to-hand tool use. For example, a phenomenal experience with which many people are familiar is becoming suddenly aware of a previously phenomenally transparent computer mouse when the wheel becomes clogged with fluff and the screen cursor no longer moves as one expects. However, philosopher Mike Wheeler, who gives a particularly clear and insightful treatment of Heidegger's notoriously difficult philosophy, makes clear that "[t]he key point about the domain of the un-ready-to-hand seems to be that the environment presents the agent with a practical difficulty to be overcome, one that (however mild) demands an active problem-solving intervention" . He argues that in Heidegger's framework, the un-ready-to-hand mode covers a spectrum of cases, some involving automatic, real-time adaptive behavior that is therefore closer to the ready-to-hand mode of engagement, others more controlled and reflective and similar to present-at-hand engagement.
Many discussions of Heidegger's distinction of the three modes of tool use focus on hammers, as this was the example tool frequently used by Heidegger himself. However, neuroscientist Paul Bach-y-Rita has described a particularly vivid example of a change in the phenomenal experience of using a technology quite different from a hammer. In this example, the interface not only became phenomenally transparent in use (ready-to-hand) after users received extensive training, but users also had a new phenomenal experience when they started to use the technology transparentlya phenomenological aspect of tool use not discussed by Heidegger. In 1969, Bach-y-Rita and his colleagues first reported a remarkable device that they claimed enabled blind people to "see" . The subjects, all of whom had been blind from birth, sat on a dentist's chair with 400 vibrating pins built into its back, arranged in a 20-by-20 array. When the subjects were seated on the chair, these vibrators pressed against their backs and they felt vibrotactile stimulation through their clothes. Next to the chair was a video camera, mounted on a wheeled stand so that the subjects could move it around. The square, 400-pixel black-and-white camera image was mapped onto the array of vibrators in the dentist's chair. If a pixel in the image was black, the corresponding chair vibrator was switched on and a participant would feel vibrotactile stimulation on his or her back; if a pixel was white, the corresponding vibrator was switched off. Subjects moved the camera to scan the area in front of them, where experimenters placed geometrical shapes and everyday objects, such as cups, chairs, and telephones, and they felt a change in the vibrotactile stimulation on their backs as the camera image changed. Bach-y-Rita called this device a tactile vision sensory substitution system (TVSS), as tactile information substitutes, or stands in for, the visual information from the camera.
Operationalizing different modes of tool use is a "phenomenal" challenge for embodied interaction, both because it concerns subjective experience and because it is remarkably difficult.
Initially, participants were trained to use the vibrotactile stimulation on their backs to discriminate between simple shapes placed in front of them, such as circles, squares, and triangles; they then progressed to recognizing common objects. Bach-y-Rita measured subjects' performance with the TVSS in two ways: First, he recorded the amount of time it took participants to recognize an object or shape; second, he recorded the subjects' verbal reports of their experience of using the device. After 10 hours of training, they were able to identify the common objects placed in front of them in under 20 seconds, whereas initially it took subjects up to eight minutes. Furthermore, after training, they spontaneously reported a change in their phenomenal experience of using the TVSS. One of the participants, who went on to do a Ph.D. with Bach-y-Rita and provides the most detailed descriptions of the subjective experience of using the TVSS, describes this change as follows: "Very soon after I had learned how to scan, the sensations no longer felt as if they were located on my back, and I became less and less aware that vibrating pins were making contact with my skin" . Not only did the TVSS become less phenomenally present, but participants also started to phenomenally experience objects in front of them, rather than vibrotactile stimulation on their backs. Behavioral evidence for this new phenomenal experience came from the response of an experienced TVSS user to an experimenter accidentally increasing the zoom on the camera and thereby magnifying the image: "The startled subject raised his arms and threw his head backward to avoid the "approaching" object. It is note-worthy that although the stimulus array was, at the time, on the subject's back, he moved backward and raised his arms in front to avoid the object, which was subjectively located in the three-dimensional space before him" . This behavioral response suggests the TVSS had become well fitted to the subject's innate abilities, at least to the extent it could facilitate an automatic looming reflex.
Operationalizing the distinction between different modes of tool use would provide a way of determining how well a technology fits people's innate abilities, and it would also be an important first step in developing a common language for embodied interaction theorists and designers. Two benefits follow from this: First, comparisons could be made between different technologies in terms of their ability to support phenomenally transparent interaction; second, design experience could help inform theoryfor example, which interaction techniques support transparent use and which are more appropriate for more reflective interaction?
So how might we systematically identify different modes of tool use? One method is to use first-person reports of phenomenal states. However, two concerns with this approach are the accuracy of self-reports and that interrogating participants while they are using a technology might shift their mode of tool use. Another method is to measure people's performance of a task. A worry with this approach is that the measure might be so specific to a particular task and interaction technique that it would be difficult to compare different systems. Finally, differences in a person's focus of attention are central to most accounts of the mode of tool use and seem a fruitful avenue to explore. The challenge is to find reliable measures of attention shifts.
We are testing these approaches in an ongoing experiment in which subjects use a mouse to play a simple computer game in which the task is to herd one or more evasive sheep so they remain in a specified region of the screen. Our study is an extension of an experiment recently reported by Dotov, Nie, and Chemero , who used a video motion-tracking system to measure the hand movements of participants playing a similar herding game, both when the mapping between the mouse and the cursor on the screen was normal and when it was disrupted by randomly shifting the position of the cursor on the screen, causing it to "jitter." While playing the game, their subjects also counted backward in threes from 400 (400, 397, 394, ...), and their counting rates were recorded. Dotov and colleagues found that when the mouse and cursor had a normal mapping, the participants' hand movements showed a power law scaling across time scales ranging from around 100ms to 1.5 seconds. It has been proposed that the 1/fβ scaling found in the analysis of some human behavior (for example, eye movements, mental rotation, and postural sway) indicates a particular type of dynamicsinteraction-dominant dynamicsthat result from the complex interaction of a number of physiological processes that extend to the periphery of the body and perhaps to tools. Dotov and colleagues argue that this power law scaling relationship in their subjects' motor behavior is not only a signature of an integrated tool-body system but also of skilled, ready-to-hand tool use, that could be used to compare different interaction techniques for different tasks. They found that when the mapping between the mouse and the cursor was disrupted, the 1/fβ scaling in the participants' hand movements was significantly reduced and their counting rates decreased. They argue that the first finding is a result of subjects using the mouse in an un-ready-to-hand mode and that the latter finding can be explained by a shift in the participants' attention to the herding task.
We have begun to use a range of techniques to further investigate the behavioral and phenomenological changes that occur when the mapping between the mouse and the cursor is disrupted during the herding task used by Dotov and colleagues. In particular, we want to gain greater insight into subjects' phenomenal experience of using the mouse/cursor tool in the different conditions and whether these are correlated with shifts in the focus of their attention. First, we use a Tobii T60 eye tracker to record participants' eye movements. Second, we use a color-change paradigm to measure whether participants' ability to detect changes in the hue of the cursor they are controlling is different in the normal and disrupted conditions. Third, we use a "think aloud" protocol to record participants' descriptions of what they are doing during the experiment. Fourth, we attach an accelerometer to the mouse to record hand movements. We have tested the effects of four different mouse perturbations: reverse left/right, reverse up/down, mirror reversal, and lag.
In contrast to Dotov and colleagues, the results of our initial experiment show the following: First, cursor perturbations do not lead to a reduction in 1/fβ scaling; second, there is as much variation between trials within a condition as between conditions; third, subjects do not notice color changes in the cursor; fourth, participants' visual attention stays primarily on the evasive sheep, rather than on the cursor, even during perturbations; and fifth, during perturbations the mouse is reported to become phenomenally present by some participants (we don't know about the others).
Operationalizing different modes of tool use is a "phenomenal" challenge for embodied interaction, both because it concerns subjective experience and because it is remarkably difficult. However, developing a means of identifying transparent tool use would provide one way of determining whether an interaction technique fits with our innate abilities, enable different techniques to be compared, and potentially allow design to inform theory. Finally, it is important to note that phenomenal transparency captures one sense of fit, but there are others. For example, Bach-y-Rita reported that the TVSS, even when used by well-trained subjects, did not generate emotional content: A congenitally blind user shown a picture of his wife could describe the details in the image but did not experience any emotions. One hypothesis is that there can be an emotional fit between a person and a technology only if they learn to use them in a community of users, rather than in isolation in a laboratory another research challenge that the tangible and embodied interaction community could explore.
Jon Bird is a senior research associate at the UCL Interaction Centre, UCL, London. His research focuses on developing novel sensory augmentation devices and technologies to facilitate behavioral change. He uses a rapid prototyping methodology and tests designs in the lab and in real-world contexts ranging from homes to supermarkets.
©2011 ACM 1072-5220/11/11 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2011 ACM, Inc.