Over the past few years, I have been working on LinguaBytes , a project aimed at developing a play-and-learning system that would stimulate the language development of non- or hardly speaking children between one and four years old. The project resulted in a modular tangible interface that allows children and their caregivers to read interactive stories and do games and exercises aimed at early language development.
In many cases, the delayed language development of non- or hardly speaking children is caused by retardations in their cognitive and/or motor skills, often due to brain damage. Other such children may have an autism spectrum disorder or be culturally deprived. In most cases, however, the children I worked with had a form of Cerebral Palsy (CP), a non-progressive brain injury. Typically, children with CP have trouble controlling their muscles, often making them not able to walk, talk, eat, or play in the way most children do; their facial, gestural, and verbal expressions can be hard to interpret as well, making their reactions to their surroundings less communicative. Additionally, since much time is necessary for physical care, less time and attention than normal are left for parents or caretakers to spend on play and communication, which further diminishes the child’s learning opportunities. It is evident that neither of these factors contributes to an optimal learning situation, which is why the need for an interactive system such as LinguaBytes surfaced.
From an interaction design perspective, however, these children are not your average target group. For one, children between one and four years old are pre-literate, eliminating many of the interaction design tools we have at hand, such as the extensive use of icons and menu structures. These familiar interface elements pose difficulties for such children, as they require an understanding of abstract representations of which these children are incapable. Also, since children with CP can have severely diminished muscle control, many of them lack the motor skills to use a traditional computer interface; they typically use two or three large programmable buttons instead. Consequently, these children have fewer opportunities for taking initiative, as they have to wait for the moments when they are allowed to push the button. As far as I am concerned, this type of interaction can hardly be called interaction at all, as it is very one-directional. Moreover, in many situations it is an adult that sets up or even operates the interactive tool for the child, which is inadequate because the form of current interfaces and interactions capitalizes on skill sets other than those of very young children with CP.
The Apparent Benefits of Tangibility
All these factors made it clear that an alternative was needed, but they also made for a very unusual interaction design context. Consequently, not much related work was available at the time I started my doctoral research. This meant that I had the luxury to more or less choose my own path. As a starting point, I decided to build on tangible interaction, for various reasons.
First, you need only look at typical developing children in this age rangeI myself have a two-year-old sonto understand that the way they naturally interact with the world is not through buttons or menu structures. Young children explore and try out stuff. In other words, their natural interaction style is tangible. Second, considering that children with CP can have a very limited action radius, they typically also receive fewer tactual stimuli. To counter this I thought, If the physical world is too far away, why not bring it closer? I saw tangible interaction as a way to help children literally get more grip on the world around them. Third, I thought that if I could make as much of the interaction as tangible as possible, I could create more opportunities for play and communication: I could make anything tangible, from the menu structure itself to any action we would normally design through clicks, drags, or gestures. Fourth, but certainly not finally, I assumed that by making the interaction tangible I could create a shared space for interaction: a table filled with physical initiatives, mediated by a central point of attention (i.e., digital output). Knowing that social partners play an essential role in a child’s development, I specifically did not want LinguaBytes to replace social interaction but rather to facilitate it by creating a shared space for interaction and communication; LinguaBytes could provide children with a platform for participation through tangibility.
To reach all of this, I approached the development of LinguaBytes from my strengths: I am a hands-on designer-researcher who needs a first-person experience in situ to get a grip on the solution domain. I like seeing my assumptions in action. In my opinion, a design is driven by so many considerationssome coming from the research domain, some from the design domainthat a prototype cannot be seen as merely the physicalization of a design; it is the physicalization of the rationale behind the design. A prototype embodies the choices a designer-researcher makes based on the knowledge he or she has at a specific moment. This means that when a test subject interacts with a prototype, he or she interacts with the designer-researcher’s line of thought.
I went through five total stages of “seeing my assumptions in action” or, more seriously speaking, five research-through-design iterations. In these iterations I designed and experimentally evaluated experiential prototypes of incremental realism that allowed me to refine my insights. Where the first two iterations were more focused on explorationestablishing the boundaries of my solution domainthe last three had more a character of refinement, aimed at determining and designing the details of LinguaBytes.
The final result was a modular tangible interface consisting of five modules for input, output, and control, together with a wide range of tangible input materials16 story booklets, eight thematic backgrounds, 236 input figures, 31 word cards, and several programmable RFID-labelswith which 16 stories and 220 games and exercises could be done in a physical and playful form. Using a large collection of playful materials, children could “read” interactive stories and do linguistic exercises together with a parent, caregiver, therapist, or teacher.
Practically, this meant that a child and adult would sit together with a collection of input materials in a shared space (see Figure 1) in front of their own zone of influence (their own interface). The child could typically use the RFID-tagged materials to trigger animations or stories on a screen, either by placing them on their interface (in the case of exercises; Figure 2, middle right) or sliding them through their interface (in the case of stories; Figure 2, top right). Based on the animations or evolving storyline, the child and adult would engage in a “conversation,” as such stimulating initiative-taking and turn-taking as well as practicing language.
I made three identical prototypes of the final design. Two were tested during 10 months at different rehabilitation centers in the Netherlands. The third prototype was kept as a backup, in case one of the two others stopped working. In order to benefit from the wide in-house expertise at the rehabilitation centers, as well as from the variety of available children, I allowed any therapist, teacher, or caregiver to use LinguaBytes in their profession. At the beginning of the evaluation period, I gave a short demonstration to get staff acquainted with LinguaBytes’ possibilities and instructed them to (ab)use the prototype freely, and to move it or modify it in any way they deemed appropriate in the context of their work.
Overall, the evaluation showed promising results with regard to the learning effectalthough this is extremely difficult to assess thoroughly and was not the focus of my researchbut it also revealed interesting insights for the field of tangible interaction. I discuss a few here, by relating to the five strengths of tangible interfaces as identified by Orit Shaer and Eva Hornecker : collaboration; situatedness; tangible thinking; space-multiplexing and directness of interaction; and strong-specificness, which enables iconicity and affordances.
Collaboration is the quality of tangible interfaces to facilitate collaboration because they: draw on our familiarity with real-world affordances, lowering the threshold for engaging with a system and thus increasing the likelihood to actively contribute; use multiple access points, allowing for simultaneous interaction and easy participation; draw on the fact that manual interaction with objects is observable and has enhanced legibility due to the visibility of the physical objects; and draw on the fact that tangible objects can be handed over and shared more easily than graphics, thus fostering shared discussion.
I can fully support this apparent strength of tangibility. With an interface that connected to the world of childrena generally playful worldthey could join in the interaction. The last aforementioned factor was pivotal in this: The fact that tangible materials could be handed over and shared more easily than, for example, on-screen graphics meant that LinguaBytes could be used in communication, and thus as a means of participation. In earlier explorations I have tried out different interaction styles, from touch interfaces to tangible ones, and where the former led to frustration, the latter led to initiative: Children could grab, point at, and hand over materials to their caregiver almost as an alternative form of communication.
Situatedness implies that the meaning of tangible interaction devices can change depending on the context in which they are placed, and, conversely, that they can alter the meaning of the location. According to Farnaeus, Tholander, and Jonsson, the consequence of this is that interaction designers should also be “thinking about the interactions around the system, and how people interact with each other even when this activity is not directly directed at the interface. Physical interaction will often result in many manipulations of interface elements being performed ‘offline,’ directed at the social and physical setting” . I fully agree with this. Moreover, LinguaBytes was fully aimed at “offline” interactions, those between a child and her caregivers or peers. LinguaBytes created opportunities for social behavior, either directed through its digital component or outside it. While evaluating LinguaBytes, most of my attention went to the social interaction between child and caregiver, merely considering the mediating role LinguaBytes could play.
Tangible interfaces leverage the connection of body and cognition by facilitating tangible thinkingthinking through bodily actions, physical manipulation, and tangible representations. I also agree with this apparent strength. In LinguaBytes tangible thinking manifested itself most notably in syntactic exercises where children could physically construct sentences and identify in a hands-on way the impact of grammatical elements on the meaning of their combination.
Another strength of tangible interfaces is space-multiplexing and directness of interaction. I focus on the directness of interaction here. Looking at LinguaBytes, I identify the following strengths of tangibility:
- Objects are not bound to the 2-D virtual world. This means that they can be used for other purposes outside the interface.
- Objects allow for eyes-free interaction.
- Objects can be used as an appealing alternative for dedicated menu functions at any time during the interaction. In LinguaBytes I designed thematic backgrounds as an alternative to a menu. These formed an integral part of the social interaction between caregiver and child, thus turning an otherwise flow-disturbing functionality into an opportunity for learning.
These factors make it so the interaction through tangible interaction objects is not only more direct, but also better balanced between the strengths of the virtual world and those of the physical world.
Finally, with regard to strong-specificness, space-multiplexing means that interaction objects do not need to be abstract and generic but can be strong-specific, dedicated in form and appearance to a particular function or digital data. I have illustrated this here through the example of the thematic backgrounds.
As I write this, the two prototypes are still being used in the real world. We are currently planning a follow-up study with a total set of 10 prototypes. I find this very promising, not merely because it makes me feel good about my work, but more so because I see it as a sign that society and industry are getting more and more ready for a tangible interaction paradigm.
One subject I still want to address, though, is doing longitudinal studies with tangible interfaces, as it presents the consequences of tangibility in a more critical light (both for industry as well as for academic research). Looking at my five iterations, the major difference between the last one and the first lies in the duration of the testing period and the consequential quality of the prototype. To put it bluntly, the earlier prototypes were tested for a few weeks at the most, which determined the maximum period they had to maintain their quality. This is pure pragmatism: There is a trade-off between the necessary quality of a prototype and the consequential time investment. The final prototypes, on the other hand, were to endure for a year of intensive day-to-day testing. As a result, I had to solve quite a few prototyping issues to guarantee durability and safety. For example, whereas the earlier prototypes often needed instant fixes between test sessions, I could not afford driving between test locations scattered over the country every time some animal lost its headliterally. This forced me to completely rethink my prototype production method. An impactful issue here was the prototype’s scale. The final prototype saw an explosion in its vastness compared to the earlier ones. Moreover, it had to be manufactured in threefold by myself, which meant lots and lots of work. And although I am not afraid to get my hands dirty, it does illustrate that there is a consequence to doing longitudinal research with tangible interfaces. Ultimately, I spent eight months designing, animating, voice-overing, programming, and prototyping to get to a sufficient level of quality for a 10-month testing period.
The important question is, of course: Was the amount of work worth it? I am convinced it was. First of all, for my research it was essential that LinguaBytes could be adjusted to the developmental level of individual children, while taking their personal needs, skills, and preferences into account. This required a broad range of learning materials. Second, as the only way to draw conclusions about LinguaBytes’ language stimulation properties was to longitudinally test it, large quantities (and very durable) hardware and software were needed. Third, to properly determine LinguaBytes’ required adaptability and adaptivity  it was necessary to get insight into the complete system. To achieve this, I felt it necessary as a researcher to leave as little as possible to the imagination; doing research in complex real-world contexts simply requires detailed design prototypes that allow for diversity, subtlety, and richness during their confrontation with the world. Only when we take the whole human seriously in interaction design can we make it truly meaningful.
3. Fernaeus, Y., Tholander, J., and Jonsson, M. Beyond representations: Towards an action-centric perspective on tangible interaction. International Journal of Arts and Technology 1, 3/4 (2008), 249267.
Bart Hengeveld is an associate professor in the Designing Quality in Interaction group at TU Eindhoven. He has a master’s degree in industrial design engineering from TU Delft. In 2006 he started his doctoral research, LinguaBytes, at TU Delft, transferring to TU Eindhoven in 2007, where he defended his research in 2011.
Figure 1. LinguaBytes was designed to facilitate communication between children and their caregivers. It does this by generating spaces where they can meet and communicate and by offering materials that enable communication. Four spaces for (inter)action can be identified: two spaces where either the caregivers or the child is in control, and two other spaces where they share control. The child is primarily in control of input, the caregiver of the choice and timing of stories and exercises, and as such of giving structure to the child’s linguistic development.
Figure 2. LinguaBytes’ interface consists of output and input modules (top left). The input modules are especially designed for doing language exercises (top right) or interactive story reading (middle right) through the use of RFID-tagged input figures and story booklets.
©2013 ACM 1072-5220/13/11 $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2013 ACM, Inc.