Orit Shaer, Amon Millner, Caroline Hummels
The technological advances of the past two decades have given rise to an increasing number of creative practices and research areas that seek to overcome the longstanding separation between the physical and digital worlds. The first conference on Tangible and Embedded Interaction (TEI) took place in 2007 in Louisiana, motivated by the field's growth over the previous decade. The conference called for attendees to explore novel experiences that bridge bits and atoms through research in human-computer interaction, design, interactive arts, tools, and technologies. Since then, the annual TEI conference brings together researchers, designers, engineers, and artists who provide an innovative and cross-disciplinary perspective on physical/digital interaction design and technological innovation.
In February 2012, the sixth TEI conference, sponsored by the ACM, convened in Kingston, Canada. The attendees enjoyed working in a time when decreasing hardware prices and increasing computing and sensing capabilities, as well as emerging interaction paradigms, enabled them to explore the TEI design space in rich ways while addressing known challenges and setting new directions.
The 2012 conference also hosted a Graduate Student Consortium (GSC), a pre-conference event in which young scholars participated in a daylong critical discussion and review of their work with faculty mentors. Mentoring the 12 participants of this GSC made us appreciate the vibrancy and diversity of areas that shape ideas among the future leaders of the TEI community. Indeed, working closely with graduate students maturing in TEI-related fields underscored how exciting the various, constantly evolving TEI communities of practice can be. Through the perspective of the GSC, we had an opportunity to consider the trajectories of ideas explored by emerging innovators in this field. We are eager to see how these innovators will affect the future of TEI.
Here, we reflect on the evolution of ideas, innovators, and interactions contemplated and developed by this community since the first TEI conference in 2007. We will revisit where we have been as a conference community, take a look at where we are now, and offer ideas of where we are headed. We begin by reviewing the research agenda set for the field by the panel held at TEI 2007. We then examine the themes explored in TEI conferences since, and review the agenda-setting panel held in TEI 2012, discussing the expanding scope of this community. We continue by reflecting on the evolution of innovators, considering the role of TEI Graduate Student Consortia and student innovation contests in shaping future leaders in the field. We conclude by profiling promising innovators, highlighting career choices of emerging TEI leaders.
As a starting point, we revisit the agenda set for this field by the closing panel of the first TEI conference, which pulled together audience-selected scholars from a wide range of disciplines from both academia and industry: Mike Kruzeniski, Trevor Pering, Bruce Thomas, Paul Marshall, Astrid T. Larssen, and Thecla Schiphorst (with Eva Hornecker, Caroline Hummels, and Robert Jacob facilitating). The panel participants collectively discussed an agenda that shaped the direction of conferences that followed. The agenda included questions such as what would and would not be considered TEI work. For example, panelists discussed the merits of interactions mediated by body movement, where nothing is being manipulated tangibly, compared with interactions with physical controls in a car. The mindsets that the field would receive and promote came under scrutiny; tangible-solution-oriented design came into question. Panelists debated what it meant to conduct research as explorations. How would TEI prove its value to outside communities?
The panelists concluded by posing the following questions and challenges for the TEI community to address:
- understanding and describing the diversity of qualities in tangible interaction;
- developing actuated interfaces;
- understanding how to embed tangible interfaces into our lives;
- investigating the theoretical foundations of physicality from diverse perspectives;
- developing hybrid methodologies that embrace both engineering questions and poetic approaches; and
- defining metrics to compare and evaluate tangibles, as well as conducting more evaluations.
Rising to the challenges set in the inaugural TEI conference, a major trend in TEI 2008 and 2009 was mechanical actuation. This body of work included robotic devices, actuated fabrics, and power-generating garments. Also, many new systems experimented with embedding tangible interfaces into our homes by augmenting objects such as alarm clocks, door locks, wallets, and faucets. New toolkits and enabling technologies were also presented, focusing on the empowerment of designers through rapid prototyping.
In 2010, the conference expanded its focus to encourage a broader interpretation of the original vision, which focused on the tangibility of bits. To reflect the broadening scope of the conference, the word embodied became a part of the conference title, making TEI mean Tangible, Embedded and Embodied Interaction. The work presented in TEI 2010 and 2011 was diverse and bridged science, technology, design, and art. However, several focus areas emerged, including the development of novel touch sensors, embodied learning, and whole-body interactions. More presentations highlighted work that was evaluated in the lab or in-situ, shedding some light on the strengths and limitations of tangible interactions. TEI 2011 also held the first student design challenge, which culminated in a Superhero Fashion Show. This challenge encouraged students to experiment with various materials and techniques and to envision how TEI can enhance human capabilities.
The theme for TEI 2012 was Fold/Unfold, reflecting a major emerging TEI trendinteraction with flexible displays. New toolkits and enabling technologies seemed to be moving away from mechatronics to leverage depth-camera sensing and enable interaction across devices. Many works in this conference demonstrated the growing interest of the community in handcrafted and fabricated artifacts and interfaces.
In the six years that have passed since the first TEI conference, the works presented have involved a wide range of application domains, including sound and music, creativity and artistic expression, physics, chemistry, geoscience, visualization, social networking, and learning. The TEI community as a whole has tackled and advanced many of the challenges posed by the inaugural 2007 panel, and seems to be reaching new levels of visibility. Some novel work presented in TEI conferences matured to become commercial products (e.g., Sifteo, Cubelets, Reactable, LittleBits), whereas many other existing products are adopting TEI technologies. In addition, we have seen the community embrace new hybrid methodologies, encouraging the integration of perspectives that encompass engineering principles, scientific paradigms, design thinking, and artistic sensibilities.
Examining the themes that emerged in the most recent TEI conference made it clear that the TEI community has expanded its reach into realms well beyond what the TEI 2007 conference attendees would have conceived of.
The TEI 2012 closing panel examined the expanding scope of TEI through the lens of three frameworks: organic user interfaces, natural interaction, and reality-based interaction. Amanda Parkes moderated the panel that featured four leading researchers in TEI: Roel Vertegaal from Queens University, Rob Jacob from Tufts University, Mili John Tharakan from the Swedish School of Textiles, and Bill Buxton from Microsoft Research. The panel focused on two overarching themes: reality as a relative, ever-shifting concept and materiality.
As a starting point, each panelist presented his or her position: Roel Vertegaal introduced the concept of organic user interfaces, which strive to create computational everyday thingscomputer interfaces or displays that basically can have any shape or form. He emphasized the increasing importance of industrial design and design thinking as the focus shifts from devices and technology to products. Rob Jacob presented reality-based interaction as a framework that unifies emerging interaction styles through the observation that novel interfaces tend to leverage users' knowledge and skills of the real, non-digital world to a greater extent than traditional user interfaces. The framework provides designers with a lens for considering new and existing interfaces to understand which parts of an interface do and do not follow things that users know from the real world and what trade-offs designers make to gain desired qualities in return for giving up reality-based interaction. Mili John Tharakan discussed the importance of studying materials beyond their physical properties to understand how they reflect multiple personal, social, and cultural narratives that are valuable in the design of interactions and new kinds of interfaces. She also highlighted the use of craft techniques that can satisfy both the soul and the senses. Finally, Bill Buxton stressed the importance of considering interfaces in context while also considering the skills required for using them. He introduced the term where-able computing, spelled to reflect the notion that "where in space, where in context" is the essence of what ubiquitous computing is about.
The panel moved on, with participation from the audience, to discuss reusability and the use of materials from the real world, such as fabrics, textiles, and paper that can create strong affordances as well as social meaning. The panel also discussed the role of natural elements such as water, earth, and air in natural interaction, and examined learnability through the discussion of special-purpose interfaces (such as cars), expert user interfaces, the power of hyper-contextualization, and virtuosity and the desire of users to acquire new skills.
The panelists concluded by encouraging the TEI community to transition from a techno-centric perspective to human-centric perspective, which requires interdisciplinary thinking and heterogeneous expertise best pursued by "renaissance teams."
Having considered the evolution of ideas in TEI, here we reflect on the trajectories of TEI innovators. We begin by examining the role of the TEI Graduate Student Consortia and student design contests in preparing future leaders. Then we look at trajectories taken by young scholars into academia and industry in a range of areas.
Since 2010, the GSC has been serving as a venue for the TEI community to prepare budding scholars to carry on and append to the TEI agenda. The intimate gathering promotes the national and international exchange of research, methods, and ideas at the intersection of the diverse TEI subfields. The consortium gives participants opportunities to develop the research and design skills necessary to become the next generation of scientists, engineers, designers, and artists who will shape the technological and socio-cultural landscape of the future of computing and our daily interaction with the world.
Whereas GSCs allow students to sharpen their research skills, Design Challenges give students opportunities to hone their creative and practical skills and take a step toward applying their skills to reach a large audience. The response to challenges such as "design a superhero costume using TEI techniques" or "make something that TEI attendees will deem cool enough to buy" has drawn competitors from far and wide. In 2011, 12 teams made computationally enhanced superhero costumes, competing for awards such as "most creative" and "best presentation and style." In 2012, seven teams jockeyed for prize money to realize a product that TEI attendees voted on with their wallets.
GSC mentors who have experience in the academic, industrial, and artistic communities relevant to the TEI conference run the GSCs. They receive applications from dozens of graduate students (master's and Ph.D.) and select a dozen students to engage in a daylong session with their peers and senior TEI researchers at the conference. The National Science Foundation has been generously providing support for the GSC events, fostering the mentoring and interaction of graduate students in the field. Mentoring a next generation of TEI researchers is crucial if the field is to retain its initial vigor and openness as it gains a foothold in the academic establishment of human-computer interaction research.
GSC participants arrive the day before the main TEI conference program begins. They have opportunities to get to know each other through social functions, and then they get down to work. Students present their work to GSC mentors and peers to solicit feedback from diverse perspectives; the mentors critique each presentation and provide insights about how the work of different participants resonates with the dovetailing agendas in the TEI field. After the GSC, participants share posters at the main conference, and their short papers are published in the proceedings.
We surveyed the alumni of all three TEI GSCs to date to hear how their participation affected their work and the career choices they made following graduation. The students surveyed represented a range of fields, including industrial design, digital-media arts, human-centered computing, and computer science. Naturally, the GSC influenced them in different ways, but their responses also reveal common themes.
Participants hoped to get different things out of the TEI GSC. Responses ranged from meeting new people to getting inspiration; from augmenting research advice received from advisors to leveraging peers to work through problems; and from steering clear of late Ph.D. roadblocks to getting clarity on the big picture. All participants applied to the GSC to seek expert advice from people to whom they wouldn't normally have access. Participants credited the GSC for giving them new insights about different aspects of their work, including setting future directions, identifying and engaging complementary communities, and putting theory into practice. All participants mentioned the GSC as being helpful in refining their research questions.
Examining the professional routes GSC alumni have taken following graduation highlights how academic and industrial arcs can lure promising students. Next, we profile a subset of professionals who engaged in TEI activities as graduate students in ways they believe influenced their post-graduate decisions.
We profiled a subset of professionals who as graduate students engaged in TEI activities that helped shape their career paths. Jill Fantauzza-Coffin, Jamie Zigelbaum, and Marcelo Coelho have startups focused on the intersection of art, design, and technology. Jill's focuses on products. Jamie and Marcelo's centers around designing interactive installations. All three TEI community members participated in multiple conferences. Jamie and Marcelo had papers at the 2007 inaugural conference and co-chaired TEI 2010, and Jill attended several TEI conferences before participating in the 2011 GSC and presenting a full paper at the 2012 gathering. In their own words, they describe the ways in which TEI influenced where they are today.
Jill Fantauzza-Coffin. I participated in the 2011 TEI GSC because, like the conference itself, this was a forum that takes into account the changing nature of building and invention. Boundaries between the building disciplines are shifting. New digital tools, communication platforms, entrepreneurial business models, manufacturing equipment, and materials are changing the vision of novel artifact creation. The TEI conferences seem to prioritize staying open to future possibilities and interdisciplinary ideas, even if the methodologies are mixed, interdisciplinary, or otherwise non-traditional.
I myself am an artist-inventor. In my practice, I build technology from a visual arts foundation. Typically, I create art installations that in process necessitate the creation of new technologies. For example, I am currently working on an art installation based on the tactile sensation of heartbeats. The work I have had to do to actualize this installation has resulted in inventions and an international PCT patent application in the haptics realm. For me, the acts of creating artwork and creating technology are very close, though the artifacts I release into the world have different trajectories. At TEI, I am able to share this work, as well as my more scholarly studies on the creative practices of artists and engineers.
Whenever I attend a TEI conference, I feel a palpable sense of creative camaraderie and support, as if we are creating visions of future technologies together.
Jamie Zigelbaum and Marcelo Coelho. We co-chaired TEI at the MIT Media Lab in 2010 as first-year Ph.D. students. We had both been active in the TEI community since it started in 2007 as authors and reviewers, but this was our first committee assignment. As co-chairs, we were exposed to every process in the ecology of the conference and had a hand in both revising and expanding TEI's offering, including launching Explorations, Studios, and the Graduate Student Consortium while securing ACM sponsorship for the first time. Working together to create TEI 2010, we found that we made a good team and had an easy time collaborating. It was this insight that served as the catalyst for our studio, Zigelbaum + Coelho, a vehicle we've developed since winning the 2010 W Hotels Designer of the Future Award from Design Miami/ Basel. Together we operate across design, technology, science, and art. Our work utilizes physical, computational, and cultural materials in the service of creating new, but fundamentally human, experiences. We have exhibited internationally in venues such as Ars Electronica, the Creators Project, Biennale Internationale de Saint-Etienne, as well as contemporary art and design galleries. Our work has won awards, including Best Music Video and Video of the Year from the 2011 British Video Music Awards; Honorary Mention, 2011 Prix Ars Electronica: Interactive Arts; Honorable Mention, I.D. Magazine 2011 Annual Design Review: Interactive; and the U.S. National Congress on Computational Mechanics Award.
Danielle Wilde and Peter Bennett are GSC alumni who also credit the GSC for sparking their research careers.
Danielle Wilde. I am an independent artist and design researcher, working in collaboration with organizations such as the CSIRO, Australia's national scientific research organization, Neuroscience Australia, Australian Wool Innovation, and others. My Ph.D. investigated the poetics of embodied engagement afforded by pairing technology with the body to facilitate real and/or virtual extension. It was the first fine-arts, practice-led Ph.D. undertaken at CSIRO, and was awarded the vice chancellor's medal for excellence. I also received the inaugural prime minister's Australia Asia Award to undertake doctoral research at the University of Tokyo, and completed fellowships at a number of leading U.K. HCI, robotics, and smart materials research labs, and the Studio for Electro-Instrumental Music (STEIM), in Amsterdam. I hold an M.A. in interaction design from the Royal College of Art in London. I do not hold an undergraduate degree.
I first attended TEI in 2009, when I was a student volunteer. The following year, I was a participant in the first GSC. Mentoring and engagement with the TEI community was an important factor in my growth. Being a student volunteer provided access into the heart of the TEI conference very early in my doctoral studies. The GSC afforded, most important, critical examination of my work from a range of perspectives, deeper understanding of how to frame my contribution, and valuable advice on publishing and engaging with the broader community. The GSC facilitators generously shared vast expertise and knowledge. The structure and collegial atmosphere of the program gave me confidence to engage, and to form lasting relationships with an inspiring and inspired group of newly discovered peers, as well as members of the broader TEI community.
Peter Bennett. I am a post-doctoral research assistant in the University of Bristol's Interaction and Graphics group. My Ph.D., from Queen's University Belfast, focused on time, tangible interfaces, and the design of new musical instruments. My Ph.D. research resulted in the design and development of the BeatBearing, a tangible music sequencer. Previously, I studied for an M.A. in design, and an M.Eng. in cybernetics.
I first attended TEI in 2009, which gave me a great introduction to the TEI community. The following year I was accepted into the Graduate Student Consortium. At that point I was near the end of my Ph.D., and it was perfect timing for the feedback and constructive critique to really hone my thesis and subsequently my Ph.D. defense. I enjoyed the experience of taking part in the critique of everyone else's projects, especially as the work in progress could generate a lot of interesting discussion that may not have taken place around the more finished projects presented in the main conference. It was useful having the consortium before the conference, both as a way to meet a great group of people to hang around with at the conference and also because discussions started in the GSC could continue and develop over the following three days. I returned again this year to TEI 2012 to present my current project, the ChronoTape, as a paper and demo.
These are just a few of the amazing TEI alumni. Many more are emerging as TEI leaders.
Reflecting on the TEI GSCs is only one way to explore trajectories in TEI. These venues have helped to shape the ways in which the TEI community evolves, preserving the notion that ideas are powerful. The impact of preparing innovators in an emerging field is especially important as the community continues to expand in both thought and number. We are looking forward to seeing where emerging TEI innovators will take usoffering rich explorations of existing agendas, making an impact through commercialization, and establishing new agendas to pursue. We have seen this happen from 2007 to 2012 and have reason to be excited about TEI conferences to come.
We thank all conference organizers, the many members of the TEI community, and the GSC participants for gatherings full of creative energy. We also thank the panel participants, whom we have quoted in this article. This work was partially funded by NSF award IIS-1143513.
Orit Shaer is the Clare Boothe Luce Assistant Professor of Computer Science and Media Arts and Sciences at Wellesley College. Her research focuses on how novel human-computer interfaces can leverage users' cognitive, motor, and social skills to foster collaborative problem solving. She co-chairs the ACM 2012 Conference on Interactive Tabletops and Surfaces.
Amon Millner is a Visiting Assistant Professor of Computing Innovation at the Franklin W. Olin College of Engineering, cofounder of Modkit, and a visiting scientist at the MIT Media Lab. Millner directs the EASE Lab, which promotes empowerment through art, science, and engineering.
Caroline Hummels is a professor in the Department of Industrial Design at Eindhoven University of Technology, where she is heading the Designing Quality in Interaction Group. Her activities concentrate on designing for transformation through aesthetic interaction with open, disruptive, innovative systems. She is one of the founders of TEI.
Figure. Photos from GSC participant presentations. (left) Jing Hua: Interacting with Virtual Flowers in a Physical Garden. Jifei Ou, Offenbach Academy of Art and Design. (right) SymbiosisO, a textile surface that senses human touch and grows a cell pattern under the person's hand Eszter Ozsvald, New York University, Tisch School of the Arts ITP.
©2012 ACM 1072-5220/12/11 $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.