Two decades ago, three paradigms were born that have since grown mightily. The Web, for which Tim Berners-Lee released the first browser and server in 1991, needs no introduction. Mark Weiser’s 1991 ubiquitous computing (or ubicomp) vision has similarly blossomed, with many of his postulations now regarded as lived reality . Though its birth was quieter (first as a two-week student project), Durrell Bishop’s 1992 “marble answering machine” will also be familiar to many interactions readers . With its physical marbles acting as embodiments of voice messages, and their manipulation offering fluid engagement with digital “contents” and metadata, Bishop’s conception has greatly influenced both tangibles researchers (e.g., as discussed in ) and, more broadly, the interaction community.
The Web, ubicomp, and tangibles were first articulated as entirely independent phenomena, with few hints of mutual awareness. Though separated at birth, their present and future are entwined with increasing intimacy. To borrow a quantum metaphor, we see the Web, ubicomp, and tangibles as entangledboth with each other, and with crosscutting notions of space, form, light, and time. More than a philosophical diversion, this entanglement is actionable, with real implications for substantial impact. Here, we illustrate this with and consider future prospects for computational STEAM (science, technology, engineering, the arts, and mathematics) and cultural artifacts, which we argue may also be seen as productively entangled.
Consider space. People presently use cyberspace, the Web, the Net, and the cloud somewhat interchangeably as descriptors for online realms of diverse virtual artifacts. In parallel, physical spacearchitectural space, urban space, and beyondis a central, if too often neglected, canvas and foundation for both ubicomp and tangibles. Indeed, Bishop has long expressed frustration that in the excitement over his marbles and machine, many forget that his conception centered upon a spatially situated ecology of interacting products.
We find deep intersection and resonance between virtual and physical spaces. Introducing “tangible bits,” Hiroshi Ishii and I spoke of “dual citizenship”for both humans and tangiblesbetween physical space and cyberspace . Citizenship should not be taken lightly. The tangibles of Bishop, ourselves, and others are first-class members of both physical and cyberspace, of corridors and clouds, with all the rights, privileges, and responsibilities accorded to each. Conversely, it is fruitful to consider pervasive physical-world entanglements of cloud content, especially as they relate to form, light, and time.
Like space, form evokes many productive entanglements between cyberspace and the physical world. Webster introduces form in terms of shape, structure, and nature. Again, consider the Web. The Web manifests itself in many different forms. Some pages are plain text; others are richly and dynamically structured, heavily interwoven with hyperlinked content. These reflect the diverse (and in the long view, still embryonic) nature of Web contentfrom simple documents to perpetually evolving Facebook pages and online forums, and far beyond.
In a different sense, one can view the <form> tag and its manifestations as one early foundation upon which the Internet’s commercialization is grounded. Further afield, in William Gibson’s Mona Lisa Overdrive, the character Gentry “was convinced that cyberspace had a Shape, an overall total form; [and] this obsessive conviction that the Shape mattered totally.” Recent echoes can be seen in the March 2012 “Operation Global Blackout” threats against the Internet’s domain name system (DNS).
In the original ubicomp vision, the diverse, complementary physical forms of Weiser’s grounding artifactstabs, pads, and boardstogether with their spatial situatedness, are foundational to the underlying argument. As for tangibles, form-giving and shape languages are for many practitioners (especially product designers) among the most central intellectual activities and concerns.
Why is this discussion of form relevant? Whereas the tangibles community’s associations of physical objects and Web content date at least to the hypercards discussed in Ishii and my 1997 paper , common practice stands at a “pre-HTML” stageone object, one linkrarely, if ever, supporting entanglements between diverse systems of artifacts by disassociated creators. Uses of QR codes and augmented-reality tags from magazines and posters offer a clear, if highly limited, example of entangling physical and virtual forms. But again, such present uses stand at less than a Web 1.0 stage of maturity (at least in the West), one of minimal human legibility and limited malleability for many visual and physical design goals.
In short, our current virtual and physical forms are at best loosely entangled, bringing varied costs and lost opportunities. In a time when attention is among the scarcest of human resources, modal isolationism between online, screen, and tangible realms can be seen as fostering psychosis-like, “loss-of-contact” states. For example, the decades-long waxing and waning of online virtual worlds might be seen as a partial consequence of their insufficient entanglement with our physical world, which otherwise would allow increased human awareness and presence in both our on- and off-screen orbital phases.
From the vantage of human habitats, light itself can be seen as undergoing not only technical but also conceptual transformations. The progression toward pervasive use of solid-state (e.g., LED) light sources deepens prospects for computational entanglements . To be sure, an LED light can be controlled by a 19th-century switch as easily as like-vintage incandescent bulbs. Simultaneously, with our habitats now bathed in Wi-Fi and other Net-linked carriers, it is progressively easier to envision a time when the majority of artificial illumination is cloud-entangled.
With Web-entangled light, shadows of cloudspacemediations of people and processes near and far, in countless variationstake on new liberties for play beyond the bounds of our rapidly multiplying screens . Where graphic design has long concerned positive and negative space, the kinds of and ratios between emissive and non-emissive surfaces and fine- and coarse-grain light (both spatially and temporally) could give rise to analogous themes, with substantial implications for attention and interaction. Marshall McLuhan presaged these themes with his assertion that “the medium is the message” and discussions of “hot” and “cool” media (considering varying implications for audience engagement across different mediums). High-intensity emissive and projective displays can tend to seize visual attention. Alternatives such as diffusely illuminated passive artifacts or bistable (e.g., e-ink) displays, entangled with cloud-sourced stimuli, prospectively offer legible, actionable, aspirational, and inspirational ambient immersion in mixed physical/digital space [3,4]. Several of our recent efforts in this arena are visible in Figure 1.
For decades, HCI has fixated its concerns at and around the 100ms threshold of interactivity. Inspired by physicist Freeman Dyson’s powers-of-ten consideration of past and future time , we briefly consider powers of approximately 10,000: milliseconds, minutes, months, and millennia.
Milliseconds and minutes are well understood to HCI, as today’s native habitat for all things digital. As a tool for thought regarding longer timescales, the shrines and temples of Japan offer both a compelling example and our inspiration for Figure 2, which shows signage entangled with ancient Japanese practices accompanying the maintenance of temples and shrines . These buildings in many cases date back a thousand years or more, with wood a dominant structural material (and earthquakes a too-frequent companion). Given this, maintenanceand the funding necessary to support ithas both pragmatic and spiritual significance.
At one extreme, the temple of Ise has been rebuilt every 20 years since approximately 680 A.D., with the 62nd reconstruction scheduled for 2013 . The exact rationale for this long tradition is unclear; among various asserted spiritual connotations, one suggestion involves training each new generation of architects in the traditional techniques of their ancestors. Figure 2 depicts a wall recounting temple benefactors across a long period of time (here, for Nan’end at Kfuku-ji in Nara). Of course, not all Japanese architectural elements are bound to the eternal. For example, rice paper windows and walls were often replaced every one to two years.
These examples encourage us to contemplate prospects for interface artifacts with aspirations toward heirloom and UNESCO World Heritage status. Of equal interest are compostables, with aspirations toward net-positive total cost of ownership/whole life costs .
Thus far, we have considered philosophical entanglements between space, form, light, and time. Having done so, it is worthwhile to consider examples illustrating how these ideas can be operationalized. We consider two diverse examplescomputational STEAM and cultural artifactspartly with an eye toward generalization, and partly to explore their own mutual entanglement.
Computational STEAM is an extension of computational science, which has been heralded as a third pillar of scientific inquiry, joining theory and physical experimentation . It concerns scientific inquiries that can be conducted only through the use of high-performance computing. For example, in our work with computational genomics, we routinely task hundreds of computer cores for days or weeks on end to analyze hundreds of millions of DNA fragments, as part of efforts uniting dozens of collaborating science teams spanning the globe (e.g., ). Computational science is also deeply interwoven with the Web’s origin. Berners-Lee conceived the Web initially as an enabler for CERN particle physics efforts, where single papers sometimes include more than a thousand co-authors. Similarly, Marc Andreessen and team developed the Mosaic Web browser (evolving into Netscape and Firefox) at NCSA, a supercomputing center.
STEAM is a variation of the familiar acronym STEM (science, technology, engineering, mathematics), with the added “A” signifying the arts (including design, fine and performing arts, craft, and architecture). STEAM has been the subject of growing activity, for instance, the recent NSF-funded workshop “Bridging STEM to STEAM” at the Rhode Island School of Design.
One perspective on STEAM-entangled arts draws analogies to the doping of semiconductors (a process fundamental to microprocessors and most solid-state electronics). There, a relatively few “dopant” atoms can radically transform material properties. Similarly for STEAM, the arts can play transformative roles in representing and communicating complex STEM activities, provoking contemplation of their implications, anchoring these in diverse cultural contexts, and inspiring broader impacts.
Our team has developed tangible interfaces servicing computational STEAM for roughly a decade. Figure 3 provides one example. Here, a Microsoft Surface, Apple iPad, and custom electronics devices are being used to collaboratively engage with a visualization of the Deepwater Horizon oil spill. As with most digital interfaces, the depicted interaction is primarily conducted at the milliseconds and minutes timescales.
While sharing these short-timescale dynamics, the larger temporal and human scales of our computational genomics effortswith hundreds of collaborators, each perhaps averaging an approximately 1,000-hour investment over years on a single projectstretch the capacities of traditional HCI, and have compelled us to embrace entanglement. The illustrations of Figure 1 depict an architectural-scale project exploring entanglement, here with corridor content engaging both computational STEAM and cultural events. Cartouches and casierspart Web-born cloudstuff, part physical avatar, loosely serving as the words and sentences of simple tangible grammars represent elements such as people, SVN data repositories, papers, funded research projects (legacies of Kfuku-ji and Ise), supercomputers, and more.
As pictured, we’ve restructured light so that all illumination is sourced below the knees (allowing the floor to be safely traversed), through screens and Surfaces, or through back- and side-illuminated cartouches by way of hundreds of Web-entangled RGB LEDs. This ecology of interactors allows the state of diverse entanglements to be monitoredthe physical and virtual state of people, data, and other thingsand acted upon. Arrays of embedded Microsoft Kinect sensors are allowing the act of pointing to these cartouches to retrieve their associations to the integrated screens, and simultaneously to change the ensemble LED states to show related entanglements. Small, illuminated, capacitively sensed cartouches ringing the screens’ bezels and other locales are providing complementary direct-touch access to the same or complementary information.
Detailed elaboration upon this entangled wall is a topic for other venues. Here, we feel it primarily important to note the temporal dimensions of this wall content. Many organizations’ data repositories for papers, grants, talks, and their ilk span decades (if not longer). Similarly, tenured faculty often reside for decades; but these durations are also marked by daily showers of activity entangled with numerous students and colleagues near and far.
Closely synergistic with the Japanese temples, we intend for the cartouches representing funded projectsand through them, images, videos, papers, talks, datasets, participants, and other entanglementsto live over long durations (months to millennia) on these walls, entangled through, for example, Google Scholar citation tracking of the research funding’s impacts long after the students have gone.
Even the supercomputers represented by the wall’s cartouches have surprising entanglements with Ise and its kin. A supercomputer, whether costing $200 thousand or $200 million, has an expected lifetime of roughly five years. Within this narrow window, such machines are enswirled by a mad race to make the most of their brief lives.
Yet many supercomputing results transcend time. For instance, the first supercomputer represented within Figure 1, Supermike, was used to calculate Hurricane Katrina storm-surge simulations, calculations partially motivating the first mandatory evacuation of New Orleans. While the storm claimed thousands of lives, in the absence of Supermike, these losses might have been far greater. While Supermike has long since been decommissioned, this is a computational result worth remembering a thousand years (and more) from now. Our entangled wall provides a step toward a plausible path for both bearing witness and providing access to this impactful heritage.
For the remainder of this article, we consider another past- and future-oriented trajectory: the entanglement of cultural artifacts. To begin, consider the term plurality. One interpretation is a core thesis of ubiquitous computing: trends toward more numerous and functionally diverse computational devices.
Plurality is also used as a synonym for pluralism and cultural pluralism defined in the 21st Century Lexicon as “a condition in which many cultures coexist within a society and maintain their cultural differences.” In his memoirs, Dyson writes, “in biology, a clone is the opposite of a clade… This…has its analog in the domain of linguistics. A linguistic clone is a monoglot culture… Linguistic rejuvenation requires the analog of sexual reproduction, the mixture of languages and cross-fertilization of vocabularies… In human culture as in biology, a clone is a dead end, a clade is a promise of immortality. Are we to be a clade or a clone? This is perhaps the central problem in humanity’s future.”
Dyson’s proposition has deep implications for HCIand far beyond. On the one hand, the systems we have illustrated in Figures 1 and 3 heavily leverage mainstream smartphones and tablet computers. From the perspective of early 2012, paralleling the past few decades of desktop computing, the convergence of mobile devices toward progressively fewer operating systems again raises the prospect of HCI “clone culture.”
As noted by the late Kees Overbeeke , this trajectory remains populated by computational devices intentionally designed with universal “Ulm School” product language, abstractly sculpted to find equal comfort across diverse cultures. As an illustrative counter-example, Overbeeke suggested the Dutch bonensnijmolena particular kind of bean cutter pervasive in traditional Dutch culture, and unknown to othersas one of myriad examples of culturally specific design, with strong implications for computational artifacts. Sister examples include Japanese genkan, Chinese red envelopes, and Swiss Appenzeller festival hats.
We close by complementing our Ise temple example with two additional culturally specific artifacts, birthed centuries and worlds apart, but each offering incisive lenses and prisms toward entangled futures. The first is a page of “Canon Tables” within an illuminated manuscript now roughly 1200 years old (depicted and described in Figure 4). Its painted “illuminations” of gold and silver cry out for awakening as perhaps still-functional capacitive touch sensors; the precious gems encrusting its covers, as lenses for entangled digital light .
Today much conventional wisdom casts physical books as all but obsolete. This ancient monastic handwork, however, suggests a profoundly different alternative. The entangled “function” of the excerpted pageas a comparative cross-index to the Christian Gospels, reinterpreted as illuminated hyperlinks to otherwise scroll-like digital textsis at least as relevant today and appropriate to its physical medium as when it was first inscribed. Far from limited to religious studies, such an approach could be equally cogent for secular messages as diverse as competing budgets from divergent political factions and scientific domains such as comparative genomics.
A world away in medium and message, but a latent sister in entangled prospects, the memory boards of the Luba Lukasa use bits of shell, wood, and stone inset over generations to record important functional and spiritual features of an embodied memory landscape (see Figure 5; ). Originally passive placeholders for an oral culture, each pinned piece can be seen as a prospective cloud-entangled interactorfor recording or accessing stories or music (whether ancient or live), communicating with remote people, monitoring natural phenomena, and countless other prospective entanglements.
Be they of Lukasan, Lakotan, or Laotian handsby monks, artisans, chancellors, or childrentangibles and broader entanglements can be viewed as mediums through which cloudstuff can be interwoven with diverse spaces, forms, lights, and times, yielding artifacts ranging from enduring heirlooms to compostable transient materials.
Although we have prototyped contemporary entangled cultural artifacts, including Web-entangled Nepali prayer boxes and mne-mosynetic music boxes, we have been counseled toward caution in sharing them prematurely. The rationale concerns the importance of cultural provenanceof such artifacts arising from the hands, minds, and souls of native voices, rather than being imposed by interested outsiders. We hope these historical instances offer exemplars and stoke enthusiasm for near-at-hand, high-impact prospects. We believe the most promising of these will likely not be purely imitative of past artifacts, but rather will be hybrid entitiesas it is for all children, bearing both ancestral along with novel and distinctive traits.
We hope the lens of entanglement may provide a fruitful view to the rich interplay and complementarity of the Web, ubicomp, and tangibles. We have also endeavored to illustrate old and new dimensions to synergisms and fusions between the STEM disciplines with arts and culture. Ours is a fertile and demanding time for new mediums and messagesand one crying out for paths that engage our world’s many peoples in their responsible production, consumption, and use.
The research described and pictured was the product of many hands and minds. Figure 1 efforts were co-led by Narendra Setty, Landon Rogge, Chris Branton, Jeremy Grassman, Rod Parker, and Robert Kooima. Wall2 and Wall3 efforts were co-led by Debra Waters, Colby Jordan, Courtney Barr, Phil Winfield, Kevin Nguyen, Stephen Tanguis, and Jesse Allison. Co-implementation of these projects was by Benjamin Birk, Christian Dell, James Hamilton, Phillip LeBlanc, Alex Reeser, Rajesh Sankaran, Suroj Shrestha, Cornelius Toole, Christian Washington, André Wiggins, Tim Wright, and many others at LSU. Thanks to John Underkoffler of Oblong, and Bernt Meerbeeck, Dzmitry Aliakseyeu, and others of Philips Research, for inspiring discussions on the future of light(ing). Special thanks to Miriam Konkel, Joanna Berzowska, Kann Conynx, Nadine Couture, Hali Dardar, Elise van den Houen, Hiroshi Ishii, Robert Kahn, Kees Overbeeke, Rita Rodriguez-Anger, Sho Shibata, Joel Tohline, Jean Vanderdonckt, and Ron Wakkary for many invaluable supporting conversations. This research was funded by NSF (CNS-0521559, CNS-1126739, IIS-0856065, and EPSCoR), NIST CDI, LSU CCT, and the Louisiana Board of Regents.
2. Bishop, D. Marble answering machine. 1992; http://tangint.org/v/1992/bishop-rca-mam/
4. Aliakseyeu, D., Mason, J., Meerbeek, B., Van Essen, H., and Offermans, S. The role of ambient intelligence in future lighting systems. Ambient Intelligence, vol. 7040 of Lecture Notes in Computer Science. Springer, 2011, 362363.
13. Related prospects have been compellingly illustrated in the books, walls, and clothing of Buechley and collaborators .
Brygg Ullmer is an associate professor at LSU, jointly in computer science and the Center for Computation and Technology (CCT). He completed his Ph.D. in the MIT Media Lab’s Tangible Media Group. He has held positions at Interval Research, Sony CSL (Tokyo), Hong Kong Polytechnic Institute, and Zuse Institute Berlin. His research interests include tangible interfaces, visualization, computational genomics, and computationally mediated craft.
Figure 1. Hallway space with computationally + physically entangled content. Back- and side-illuminated cartouches display activity in document space (SVN-based code, papers, designs, etc.) and people space (both via social media and physical presence). Interactivity with entangled content under way through wall display, Microsoft surface 2 (SUR40, under retrofitting for convertible vertical + horizontal use), smartphones, addressable RGB LED strips, capacitive sensors, and arrays of Microsoft Kinects (detecting both pointing and presence). All interoperate with complementary technologies both in this and other spaces.
Figure 3. Microsoft Surface and cartouches + casiers , used to engage with a visualization of the Deepwater Horizon oil spill. Here, one casier is back-illuminated and sensed by an Apple iPad tablet; the other, physically identical, is bidirectionally mediated by the Surface itself.
©2012 ACM 1072-5220/12/0700 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.