Steve Harrison, Scott Minneman, Maribeth Back, Anne Balsamo, Mark Chow, Rich Gold, Matt Gorbet, Dale Mac Donald, K. Ehrlich, A. Henderson
Crossing the atrium of the Tech Museum of Innovation in San Jose, visitors see objects in a darkened gallery space not unlike the rest of the museum. They are big, furniture-sized things. They are inviting and playful. Presumably, they explain some aspect of technology, sitting grouped together. But they really do not have a quickly recognized form or a theatrical surround that says "This is about earthquakes." Or "lunar exploration" or "chip fabrication."
Step up to the joystick on the exhibit to the right. A cartoon image of a young boy is projected on the screen. Kids crowd in around you as you read about Henry and his world. (See Figure 1.) There is a world of cartoon images. Lines trail off to small drawings falling away as though seen through a fisheye. As you push the joystick, another image of Henry rolls into view along one of the lines, and a comic-book dialog bubble appears. The story Henry tells is of the things in his imagination and his everyday world. One image leads to the next and then to the next.
Over to the left of Henry is a sort of work bench with a touch-screen workstation sitting on it. No pictures on the screen this time, just a title ("Harry the Ape") and a long paragraph of text. The story is about the creatures that live in Harry's fur. Sprinkled around the paragraph are some pastel triangles. The instructions say to touch a triangle; the sentence gets longer! Reading the story, poking all the triangles, getting all the details, you get a pretty good idea that Harry has quite a menagerie in his fur.
Kids come up and read along. They want to find out just how many amoebae live on Harry and what all their names are. Probably a good time to move onto another exhibit.
Passing up some other exhibits because they're too crowded, head to the back of the area where another work bench has a sort of giant-sized magnifying glass. Some cards with pictures on them are lying about. One, under the magnifying glass, is of a woman in a big hat carrying a large bag. Peering through the magnifying glass, you see a skeleton superimposed on the woman. A slight push to one side and the "x-ray" shows a fish hidden in the large bag. Other cards are in a bin on the work bench. Each has some sort of hidden image that makes more of the story on the card. On the back of the card is a technical explanation of "data glyphs." Data glyphs are a form of steganography that encodes information in slashes small enough to be used as half-tone dots. So the "magnifying lens" contains a camera that scans the dots. After the pattern is decoded, the image is rotated and identified, an overlay image is selected, and the position for the overlay is determined. A half-silvered mirror reflects the image displayed.
What do all these things have to do with one another? Before we look at more XFR, let's try to answer that question.
This is a show about reading. These objects were created to remind visitors of the centrality of reading in our technological culture and ask them to reflect on how that may change.
Reading is part of how we understand the world, how we play, how we share knowledge, and even how we think. Although reading text is the dominant form of reading, it is important to realize that most of the text we readletters, numbers, words, and sentencesis surrounded by illustrations, photographs, and other kinds of symbols. We read it all.
One common image of reading is of a person sitting quietly alone with a small, paperback book in hand. Reading, however, has changed over time and place. Just 500 years ago, monks read aloud from giant, illustrated manuscripts while standing up in crowded rooms. Reading silently to one's self or even owning a personal copy of a book was unheard of. In some cultures, reading is done from walls ("epigraphic" reading), and in other cultures most reading is done from paper ("bibliographic" reading). In this culture, we are surrounded by opportunities to readbooks, magazines, trading cards, billboards, T-shirts, appliances, signs, newspapers, toys, memos, bills, and birthday cards. In movies and on television words scroll, stretch, and fly.
Technology exerts a powerful influence on these changes in reading. As technology changes, so does readingclay tablets, papyrus, the printing press, the copy machine, and the laser printer have each changed reading practices. The Internet might really be called the "World Wide Reading Machine." As technology continues to change we can expect new forms of reading to emerge.
Computers, in particular, have dramatically affected reading in many ways: they make text dynamic and they enable readers to interact with text in new ways. E-mail revives the reading and writing habits of letter writing. Desktop publishing and the Web browsing make communication with a community of people easier. Today, almost everything we read, including paper-based printed matter, was made using a computer.
The Research in Experimental Documents (RED) group at Xerox's Palo Alto Research Center (PARC) took these observations and created XFR: eXperiments in the Future of Reading. It presents exhibits that explore how digital technologies influence reading and also how that changes the process of authorship. Authoring now includes not only writing the words (what was traditionally called "the content") but also selecting the font, the colors, the mode of interaction, and even modifying the medium of communication itself. (For more about us, see the accompanying sidebar, "The Who of XFR.")
In developing this exhibit, we have come to call the process by which the content and the medium are simultaneously authored "total writing. Some of these exhibits, such as "Harry the Ape" (also known by its media as "Fluid U/I" or "Fluid Fiction"), present new document forms; others, such as "Henry's Hyperbolic World," offer speculations about new media genres. Exhibits such as The Reading Wall and Walk-in Comix reinvent traditional book forms, and Listen Reader imagines a new kind of paper-based book. What all have in common is that the form, content, and reading situation are designed together.
Some people believe that text and reading will "converge" onto a single device. We believe otherwise: we speculate that there will be an explosion of reading forms and media, each unique and meaningful in its own right. The practices and uses of reading are always changing; this explosion will accelerate that pace. These wonderful new media and new genres will not supplant old forms; print-based books will not disappear, but digital technologies may change the process of book production and perhaps their form.
Some of the pieces are compelling, and visitors often ask if they are available as products somewhere. Although this attention is flattering, the experiments are not necessarily what we predict reading will be like in the future (though it certainly would be a fun future!). Rather, these museum demonstrations help us think, research, and talk about the future of reading by giving us concrete examples of new reading devices and experiences.
It is more accurate to think of this as a vision of a future world, not of future products. Having all these items adjacent to one another conveys the idea that these wildly varied new forms stand for the wildly varied forms that really will be present in a reading-based culture in the near future. This is not to say that these might be the basis for products, but these are polemics in physical form, arguing our case about the robustness of reading.
Why is that important? Why should designers pay attention to objects that are acknowledged to be stand-ins for other products?
One reason is embodied in a PARC axiom: "The best way to predict the future is to invent it." This is a method that designers have in their control to direct the products and to influence the direction of technology and society. Along the lines of "concept cars" that automobile manufacturers display at car shows, these exhibits speak both to a buying public and to the companies that produce them. Perhaps the future that is invented will be the infrastructure that enables a range of technologies to be deployed or perhaps it will alter the consuming public's understanding so that markets are created for new products. If that were the only reason then it would be a fairly sad story, locating the speculations as primarily means to a consumer end.
The more important reason to pay attention is to see that there are many "obvious" things in the world that need to be challenged. We take on "reading" because it so pervasive and it is so easy to posit its downfall in the light of an electronically delivered image culture. Just as easily, we could use speculative processes to suggest that
- Not everything can or should be designed with a user in mind.
- Affordances are only starting points for understanding our relationship to the things we make.
- Making ideas concrete is hard.
- There are no solutions, just trade-offs.
Each exhibit has a number of design ideas within it. Lets take a look at a few more of them. (For a complete list of all exhibits, see the sidebar "The Exhibits.")
Besides turning pages, how can the body be involved with reading? We answered this question with the speculation that coffee-table books and coffee tables might one day become one and the same. Projected on the white surface of the Tilty Table is a high-resolution image that makes it appear as if the table is a glowing screen. When visitors tilt the table, the images on the surface change in direct response. Each of the three Tilty Tables provides a different reader experience.
The Reading Table (Figure 2) allows you to move through a large document that, if printed, would be more than 30 feet on one side. Readers feel as if they are surfing or gliding across the images of hundreds of captioned cartoons exploring ideas about reading, writing, words, and telling.
Gamelike in its use of the interface, the Peace Table (Figure 3) has a virtual ball that, when rolled over hotspots on the table, triggers the display of the word peace in different languages.
The Folk Tales from All Directions Table presents eight folk tales from around the world. The stories come sliding down across the table from the direction in which the table is tilted, with each side and corner being a different "corner" of the world. Tilting slightly off to one side will mix two stories together.
The Tilty Tables experiment explores many areas: how to read large documents in new ways, how words and images work together to form meaning, how we use our bodies to read, what it feels like to explore an infinite plane of stories, how a collection of words is transformed into a story, and how we respond to computer-authored text.
What if the audio illustrations could be just as deep and complex as pictorial illustrations? What if books could have acoustic environments like cinema? What subtle augmentations can digital technology add to books and other existing static media?
Three Listen Readers are placed in a quiet corner near the rear of the exhibition. Each one consists of a big, fluffy chair and a small table that can be pulled up to the chair. Mounted on each table is an illustrated children's book. As the visitor moves her hands over these books wondrous sounds emerge, placing the reader in a sonic atmosphere that seamlessly changes as the story unfolds from page to page.
Working with Chronicle Books, we identified three books. Dedicated to John Coltrane (and using fragments of his music, with the generous permission of Atlantic Records), Hip Cat is the story of a saxophone-playing cat told in a jazz-poetic style. It really gets readers into the book. The other stories are Frank the Monster Who Wanted to Dance and Armadillo Ray (in Spanish).
Each of the three children's storiesand the acoustic world created while reading themis different. Each reader gets a deeper, immersive experience of each book. But each chair is the same: speakers are built directly into the chairs and sensors are hidden in the lectern. The Listen Reader chairs use radio-frequency identification (RFID) tags embedded in each page of each book to tell which page is opened in the book. Electric field sensors located in the book's binding sense the proximity of the visitor's hand. The movement of hands to turn pages determines what sound and music samples to play, at what volumes, at what pitches, and at what apparent location. Sophisticated readers can "perform" the book by passing their hands over the pages swirling the sounds as they go.
The book must be read by the readerthe chair does not do it for the visitor. This is a different experience for the reader than the commercial children's books that offer a simple button-based sound experience (where, for example, if you touch the picture of a duck, the duck quacks). We see parents reading aloud to children, children reading aloud to each other, and people reading silently. Often, the music and sound effects subtly pace the reader.
The Reading Eye Dog
Emblematic of XFR is the metal talking dog. The Reading Eye Dog (a.k.a. RED2) consists of an assemblage of industrial video cameras, custom image-processing software, a commercial optical-character-recognition program, and a commercial speech synthesizer. Functionally, it is nearly identical to products developed to aid visually impaired people.
The form of delivery is important. RED is a dog, not some unapproachable box. Everyone loves the dog. Even though it sounds like the synthesized voice that Stephen Hawking uses, visitors find the experience compelling.
Tracing the evolution of the form of RED is another way to understand how significant this is. What became RED began as a desk with a camera mounted on it. The exhibit was intended to explain the difference between machine "reading" and human reading and to expose some issues dealing with literacy. As we began to play with the idea of reading aids, we considered applying it to the enormous amount of text in the environment. Perhaps there could be a machine to read all the text in the environment. Maybe it should sit on your shoulder like a pet parrot. From this animalized form, we jumped to the idea of seeing-eye dogs that aid the visually impaired.
But what personality should a reading eye dog have? Where should it be so that it can explain itself to visitors? Where would it go in an imagined future world? How would that reading situation be communicated to a visitor? As you can see from this small sample of design sketches, we worked through many ideas to get to RED.
One large design challenge was making clear where the dog was looking and what it could see. Devices that read aloud for the reading-impaired usually have fixed platens with covers (like scanners) and not cameras set as eyes. The physical constraints of the system guided some aspects of the form. Having even, bright lighting is essential to be able to read fine text. That meant that people should not cast shadows. Visitors would be walking up, so the reading material should be around eye level for nine-to 14-year-old kids. The dog might be big enough to look over a visitor's shoulder, but that would have created a much less approachable character. It might also have conveyed the idea that this is a big, ponderous technology.
The critter needed to look "alert." The dog's apparent attention was needed to direct the visitor's attention to the reading material.
Once we had some idea about the dog's position and the position of the reading material, we needed to show what the dog could see. Displays ended up on both the chest and back, so visitors need to be in one specific location to observe the action. The stepwise nature of the process was made visibleand slowed down at various stages.
A careful review of the design sketches (Figure 4) will show that the imagined ways of interacting with the dog were not always practical or comprehensible to visitors. What would trigger the dog? Changing the image? Could reading be interrupted? What should happen if the source document is covered up but the image has already been captured? Often these user experience issues were answered by selecting the most pedagogically effective mode; other times, design decisions were based on usability. There were no hard and fast rules.
These exhibits were the tangible result of a large brainstorming process. See the accompanying sidebar, "A Few Things We Didn't Make," for a sampling of the many ideas we did not turn into exhibits.
Although we can tell you here our arguments about the continued centrality of reading to our culture and how digital technology will create divergent, not convergent forms, we know that the experience of reading is not the same as the experience of being there. We are really pleased with and proud of XFR. It was a great design project with a crowd-pleasing result. There were design challenges along the way and some great ideas that might be revisited. We trust that the reader has benefited from this story about our exhibition and will continue to think about the richness of reading.
Other exhibits exist that we do not describe here; all the exhibits are listed in the sidebar. To find out about them and get more details about the ones we quickly describe, go to www.parc.xerox.com/xfr. As of this writing, we do not know where XFR will next be seen, but the Web site should list the schedule as it develops.
We would like to thank the Tech Museum of Innovation in San Jose for inviting us to create an exhibit for them. We also want to thank the many executives at Xerox who said "yes" when all that existed of XFR was a set of PowerPoint slides. We would like to particularly thank those in Integrated Marketing Operations who funded XFR. XFR also was the work of our many consultants whose special talents made XFR look so good and work so well, particularly, Shilajeet "Banny" Banerjee, Jonathon Cohen, Mark Meadows, Mary Kate Meyerhoff, Terry Murphy, and Maureen Stone.
Steve Harrison, Scott Minneman, Maribeth Back, Anne Balsamo, Mark Chow, Rich Gold, Matt Gorbet, and Dale MacDonald
Xerox Palo Alto Research Center
3333 Coyote Hill Road
Palo Alto, CA 94304
Design Column Editors
89 South St, 2nd Floor
Boston MA 02111
Rivendel Consulting & Design, Inc.
P.O. Box 334
8115 La Honda Rd. (for courier services)
La Honda, CA 94020 USA
Figure 1. Henry's Hyperbolic World uses the hyperbolic browser developed at PARC and marketed by Inxight. Intended to display complex hierarchies such as large Web sites, in this use the hyberbolic interface presents a multipath cartoon of a child's world.
Figure 4. Design sketches tracing the development of RED, The Reading Eye Dog. Intended as both a friendly reading companion and a pedagogical museum experience, the robotic form clearly demonstrates how users' expectations are shaped by the form of an object.
The Research in Experimental Documents group (Maribeth Back, Anne Balsamo, Mark Chow, Rich Gold, Matt Gorbet, Steve Harrison, Dale MacDonald, and Scott Minneman) predates the XFR project. Its members have worked on a number of small-scale group projects as well as numerous individual and small-team projects. They range from long term studies of design process to basic work in ubiquitous computing to video production.
The interdisciplinary breadth of the RED group is represented by members' backgrounds in architecture, engineering, and product design, as well as in education, cultural theory, and sound design. In addition to their professional training in various design fields, each of the eight members of the group also bring experiences in the realms of art and technology. One had worked on professional film, two others had collaborated in the creation of several public interactive art installations, two had worked on the Media Lab's Brain Opera, one was a former performance artist, and another had designed a cross-cultural multimedia documentary.
The group was assembled to pursue a very particular research charter at Xerox PARC: to investigate and design new genres, including new document forms, using new research methods. This charge provides the Corporation with early insights about the potential for new media and new opportunities for existing media.
What Pulls Us Together
Our value systems probably overlapped more than our interests. Central to our values is the idea that art and design have as much to contribute to insightful enquiry as does engineering and science. Everyone came from a background of collaborating with others, so another value is that of collective action. 0In that, we value having an opinionparticularly, aesthetic and intellectual opinions. Not that we share a particular aesthetic or philosophical stance, just that having one that is brought into play in discussions and guides design action is important.
And, of course, we share a common goal: the development of new genre. We value this, not for its own sake, but that as a means to make a statement about the state of the world and the direction of technology.
The Where of RED
These values translate directly into action. Much of this can be summed up as valuing a studio culture. As we valued a studio culture, we created some aspects of a studio. Physically, there were a couple of studio spaces that emerged over the course of the project: our regular meeting room, a multimedia work room, and a large lab space on a lower floor of the building. The spaces were first used to create prototypes of the exhibits. Later, the studios became our multimedia production and exhibit fabrication spaces for the most part. There were tables of workstations where the content of much of the exhibits was created. There were piles of graphic material on the tables and shelves, piles of aluminum extrusions and hardware on the floor, and everywhere there were boxes of computer parts, video arcade controllers, and video projectors.
Often inhabited simultaneously by a couple of RED members, work in the studios would be carried out publicly. This meant that we were free to look over each other's shoulders and kibitz. It kept us abreast of the state of the content and exhibit development. It meant that responsibilities were constantly negotiated and re-negotiated.
And the work was often done in meetings. During the first 10 months or so, we met three times a week for two to three hour session, mostly to brainstorm and to review progress. Later in the project, scheduled meetings went to project status meetings once a week, but we often found ourselves working in the same room unofficially about as much as before. And this was in addition to the development and production work we carried out together in our studio spaces.
There are about a dozen exhibits that make up XFR. Some exhibits are one-of-a-kind, some have multiple instances. * Indicates exhibit is described in the accompanying article. To find out more, visit: www.parc.xerox.com/xfr
The Book Artists' Studio
At the back of the exhibition is a studio where artists and graphic designers who specialize in limited edition and one-of-a-kind books create books. They have state-of-the-art software and computer hardware, high-speed digital color printer/copier, binding and trimming gear, and laser cutter.
Fluid Fiction: Harry the Ape
Demonstrating another use for Fluid User Interfaces created at PARC, Fluid Fiction presents a story where each sentence has alternate, longer, and more detailed endings. Accessed using a touch screen, the visitor can expand each sentence to discover more things are living on Harry. *
Glyph-o-Scope: The Hidden Worlds Revealed
This exhibit uses digital data encoded in half-tone dots. Visitors look through a half-silvered mirror to see an image superimposed on an image on a printed card. *
Hyperbolic Comics: Henry's World
Henry's world is a tree-structured narrative about the things and activities of a seven-year-old presented using the hyperbolic browser from Inxight Software. Each node in the tree is a cartoon illustration. *
ListenReaders: Hip Cat, Armadillo Ray, and Frank, the Monster Who Wanted to Dance
Each ListenReader chair has a book on a stand. Each chair creates an audio environment to accompany each page in the book. *
The ReadingWall: Episodes in the History of Reading
Each of the three panels of the ReadingWall describes the changing practices of reading in a different epoch. Details are shown on a moving flat panel display.
RED: The Reading Eye Dog
Shaped like a robot dog, RED reads printed material put in front of it. *
SpeederReader: PodKayne's Dispatches from the Inhabited Planets
SpeederReader uses an arcade-like driving interface to present stories one word at a time. A speedometer shows how fast the visitor is reading.
Tilty Tables: The Reading Table, The Peace Table, and Folk Tales from All Directions Table
Each Tilty Table is a 2-D reading surface that is controlled by tilting it. *
Walk-In Comix: Slip to Text
The graphic novel, Slip to Text, is read by walking trough a maze.
WHYRL?: What Haven't You Read Lately?
Each of the two WHYRL?s presents a slide show of images of text in the environment. Each digital image is selected probabilistically based upon the characteristics of the last slides shown. Sometimes, questions to make the visitors think about the presentation are superimposed over the images.
Constraints emerged over the course of the developing XFR; it would have been impossible to enumerate them in advance of designing them. (And even now, in retrospect, it is hard to capture all of them!) To suggest how much the design process was a matter of editing and combining, here are just a few of the ideas we did not put in XFR and some of the reasons why we dropped them:
The race would have been a game where visitors would see if they could read faster than a machine. It might have been fun for the visitors to see if they could beat scanning software, but we decided against developing it. It would have taken too many steps to explain in a museum setting, it did not create a new genre of reading or interactive exhibit, and it did not demonstrate the centrality of reading.
Pager Short Story
Visitors would have seen a story unfold across a forest of pagers. While interesting and demonstrating how existing media gets re-purposed into new document types, it was stopped because it was hard to deploy in a museum setting and required a complex network infrastructure that made it vulnerable.
The Adventures of the Red Dot
Intended to showcase a paper-moving technology under development at PARC, a paper dot would dance around illustrating the behavior of characters in a story. The technology was not readyor more accurately, the technology developers were not ready.
The Walk-Up Comic Book
Using projectors and proximity sensors, visitors would have immersed themselves in an interactive comic book adventure. It metamorphosed into Walk-In Comix. Where Walk-Up would have emphasized the potential for interactivity, Walk-In extrapolates a new medium from low cost wide-format printers. The idea that a new kind of novel could be published if people could inexpensively print wall-sized images was central to our message. It demonstrated how changes in media tools create opportunities for new document types.
We planned on having looping videos of people reading play on monitors scattered around the edges of the exhibition to set the context for the XFR show. The other exhibits turned out to be so obviously about reading, it was deemed unnecessary.
The Readers Theater
This would have been a place where stories and essays would be read aloud with multimedia accompaniment. Performative reading was an area that intrigued many members of RED and came up often in our thoughts about future reading activities. This exhibit had the logistical problem of either getting visitors to perform in front of other visitors or arranging for professionals to perform. And then there was the problem of what to do with the exhibit and the technology when no public readers were present.
Shakespeare Minus One
Using simple recognition tricks, technology would have been used to fill-in the other actors while visitors would perform a reading of a scene from Shakespeare. The other characters would have been played by modified programmable Barbie and Barney dolls. This exhibit also would have explored the performative possibilities of reading. Deemed too hard to do reliably in the noisy Tech Museum environment and dropped fairly early on, the idea came up repeatedly long after it was officially dropped.
Demonstrating that new document types are often combinations of unlikely pre-existing forms, PPtO would have used recognizable templates to augment a pre-recorded snippet of an opera. Kids (and most adults) do not relate to opera, so locating material that visitors would understand and find engaging proved difficult. Also, PowerPoint opera is an ironic idea. Irony is the fodder of modern art museums, but rare in science and technology museums; while being out of place is a method of creating new document types, irony could easily have been misunderstood by most visitors.1
The International Children's Book Project
With stories solicited from children from around the world, this would have been both an exhibit and an event. A collection would have been printed each day for visitorseach one unique, printed and bound right before the visitors' eyes. There were lots of problems with implementing the ideacopyright ownership, privacy, production, staffing, cost, coordination, access, censorship, litter, etc.
The ReadingRoom would have demonstrated that text is everywhere in the human environment, from little labels on fruit to giant billboards along the Interstate. With ever-changing displays on the face of everyday printed objects, the exhibit would have consisted of a tableau of an apartment kitchen with a view to a street beyond. The newspaper on the table and the boxes of cereal might show web pages, the refrigerator notes might be the day's news, and the stop sign outside might be a poem by e.e. cummings. While fairly esoteric, a number of prototypes were created. Most people who tested the prototypes were confused. It was dropped in part because it did not present the visitor with enough "distance." That is, people thought that these were new document types (which they were), but did not also see that it was commentary on their own environment, too. Try as we could, we did not find a way of breaking through this set of expectations established by the interactive museum exhibit genre.2
1 In our companion "Tools and Methods" article, we discuss some exhibition-wide attempts to break the conventions of the technology museum exhibit. Arguably, using irony would have been one clear way to have done that. But having just one ironic piece was generally thought to be more confusing than ironic.
2 The issue of exhibit genres and museum genres is explored in the accompanying article in the "Tools and Methods" column.
©2001 ACM 1072-5220/01/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2001 ACM, Inc.