X.2 March + April 2003
Page: 15
Digital Citation

Personal digital historian

Chia Shen, Neal Lesh, Frédéric Vernier

back to top 

One of the most enjoyable parts of life is sharing our experiences with others, whether with family over dinner, friends we see occasionally, relatives who live far away, people within our community, or colleagues we work with. People often use records of their past, such as photographs, videos, and various types of documents, as tangible conversation starters to share their stories and to build a shared culture and history. In the past 10 years, we have informally observed where and how people share photos and other supporting documents. Figure 1 is a table at which people have laid out their photos randomly for others to view.

Recent advances in technology have made it possible to easily amass large collections of digital recordings of our daily lives. These media offer opportunities for new user experiences beyond conventional digital photo albums [1, 2]. In this article, we describe the design of the Personal Digital Historian (PDH), an interactive system that facilitates face-to-face conversation and story sharing. PDH uses a specially configured digital tabletop, on which images related to the story can be easily displayed and viewed by everyone. In this system, users select stored digital archives such as photographs, video, and text documents and display the images grouped by people, time, place, or events, while the story is being told. By using the term "story sharing" instead of storytelling, we emphasize in our design the supporting of interaction and conversation among the participants, rather than the performance of a storyteller. Hence, our design focuses on providing the right tools and visualizations for the listeners of the story as well as the story sharers. (See the sidebar, "PDH Usage Scenario," for a sample interaction scenario.) Our goal is to provide a new digital content user interface and management system allowing face-to-face casual exploration and visualization of digital contents. Figure 2a is an artistic rendering that embodied our vision of the PDH table at the beginning of our design phase in the fall of 2000.

The current PDH table is implemented using our DiamondSpin ( circular tabletop Java toolkit. Unlike conventional, single-user, desktop, user interfaces, DiamondSpin is intended for multiuser collaborative applications. For the physical PDH table, we used a standard tabletop with a top projection (either ceiling mounted or tripod mounted) that displays on a standard whiteboard as shown in the right image of Figure 2b. We used two Mimio [4] styluses as the input devices for our first set of user experiments. Currently, PDH is also running on the DiamondTouch, multiuser, touch-tabletop surface [2].

back to top  Conceptual Model

Our first goal was to focus on developing content organization and retrieval metaphors that are easily understandable by users and can be used without distracting from the conversation. The folder structure used in conventional desktop systems are too restrictive for the PDH usage scenarios. We learned from our interview studies reported by Shen, Lesh, et al. [5] that a story-sharing system must support flexible narrative, rather than rigid, preauthored stories. The contents must be structured so that new members of the group can understand and retell the stories themselves, but not so much structure that people are locked into one way of describing the relevant events.

Given the intended application domain of PDH, we adopted a model of organizing the material using the four questions essential to storytelling: who, when, where, and what (the four Ws). Adopting this model allows users to think of their documents in terms of how they would like to record them as part of their history collection, not necessarily in a specific hierarchical structure. The four Ws are encoded in the form of metadata that can also be used to group material. This grouping is not an exclusive hierarchy; each meta-data keyword can belong to multiple groups. Therefore, documents are not defined by their physical location in a file system, but rather by their intended usage scenario. The user then can make selections among the four Ws and PDH will automatically combine them to form rich Boolean queries implicitly for the user.

back to top  Interface Design

Desktop interfaces in general have not been designed for multiuser, face-to-face conversation in a social setting. The vertical desktop display that most people use is fine for a single user but not conducive to group conversations. What we often see is that the most natural way for people to converse is to gather face to face around a horizontal table. The design of PDH has been guided by the following principles:

1. Allow natural, face-to-face conversation. Make it easy for people to see the display in their own way and their own orientation and eliminate typing, which can detract from conversations. See "Continuous Orientation and Resizing" and "Flexible Control Panel."

2. Afford easy-to-learn interactions. Use touch and fluid movement as the primary interaction mechanisms. See "Image-Based Visual Bookmarking" and "Content Layout and Visible Commands."

3. Flexible organization of content. Provide the tools for users to tell the same story in different ways with different viewpoints. See "Views and Context."

4. Simple content browsing and retrieval. Avoid complex Boolean query formulation. See "Implicit Formulation of Visual Queries."

bullet.gif Continuous Orientation and Resizing

When multiple people gather around a table, no single directional viewing angle or orientation is ideal for everyone present. The PDH table uses a circular display metaphor [6] that can provide a continuous orientation for multiple people. Users can then share documents simply by rotating individual items or the entire tabletop display. All documents can be freely resized. This mimics the experience of sharing physical documents on a tabletop while offering the advantages of working with digital documents.

bullet.gif Flexible Control Panels

Our user interface has curved control panels on the perimeter of the tabletop. The Who, When, Where and What menu bars in the control panel let the user display contents based on these four dimensions, as described in more detail in the next subsection. The leftmost menu item is an ink bottle used as a digital ink pad for freeform strokes; the Show button is for displaying contents after selecting the filtering values in any or all of the four Ws; the horseshoe symbol in the middle of the control panel is for rectangular linear layout (described in Figure 8); the Tools menu item provides a popup menu of many other functions, including a function to invite another user to the table by instantiating an additional control panel. Control panels can be repositioned anywhere along the perimeter of the table. The number of control panels is not fixed but varies by the number of people around the table.

bullet.gif Views and Context

To facilitate flexible content browsing during story sharing—that is, to allow users to bring out the same contents within different contexts—we use the concept of "views" in PDH. Contexts are the textures and background a user chooses to share a part of the story with. Each view represents a virtual tabletop with a different context background. In particular, each of the four Ws mentioned earlier is a view of the data organized by that context. Figures 4, 5, and 6 illustrate the Who, When, and Where views of the same collection of data. The What view is similar to the Who view except the thumbnails in the view represent events and objects rather than people. The user can easily switch between the different views during a conversation by simply selecting the menu items on the control panel. Users can select one or more persons from the Who view and one or more events from the What view by touching the thumbnails displayed in these views; selecting the items will change the thumbnails from black and white to color. At the same time, the name or event name of the selected item will also be displayed above the menu bar on the control panel. This reminds users of what the current selections are and allows them to easily retract the selections by touching the names on the button. Users can select a time range in the When view and a location from the Where view by sketching a freeform stroke with digital ink to indicate the region of interest.

bullet.gif Implicit Formulation of Visual Queries

When selections are made in any of the four W views, that subset of the images will be highlighted in color and prominently displayed in other views. The semantics of selecting people, for example, is to tell PDH to highlight only images containing at least one of the selected people. For example, if a user selects two friends in the Who view and then switches to the Where view, she would see where she has traveled with either of these friends by observing where the highlighted pictures appear on the map. If the users select several people in the Who view and several objects in the What view, PDH will highlight only pictures that contain at least one of the selected people and at least one of the objects. We found this semantic use of AND/OR to be the most natural way for users to quickly form queries. Similarly, users can select regions of interest in time or space with the When or Where views by drawing a freeform stroke using digital ink.

bullet.gif Image-Based Visual Bookmarking

During a conversation, people often branch out to different topics and threads and then return to some previous discussion point. We facilitate this style of discourse in PDH by letting the user select an item on the desktop as a landmark by dragging it to the bookmark area. Landmarks in the PDH system perform a similar function to bookmarks in browsers by providing a fixed point to which users can easily return. We have found that people use this feature at times to sequence their story vignettes and at other times to repeatedly return to certain points in the conversation.

bullet.gif Association Generation and Presentation

We have designed certain functions of PDH to support the serendipitous discovery of information by associating the currently selected item with other related items that could come from the user's personal information or from public sources.

Users may explicitly request an association to be made; we call these active associations. Users can actively request documents associated with a picture in the display. When the user performs a long touch (by holding down the touch for 0.5 second) on an image, a contextual menu with eight items pops up that allows users to easily request more pictures that are similar along any of the who, what, when, or where dimensions. This contextual popup menu, like all documents on the tabletop, can be dragged or rotated.

Alternatively, the system can suggest associations spontaneously. Passive associations are displayed around the perimeter of the circular display as shown in Figure 7 (although users can turn this feature off). In our current system, passive associations correspond to the result of a disjunctive variation of the query that was used to retrieve the images on the table (that is, carrying out an OR Boolean operation in place of an AND Boolean operation).

bullet.gif Content Layout and Visible Commands

The tabletop quickly becomes crowded with images. To overcome this problem, we designed and implemented various layout strategies [6] to help the user. As shown in Figure 7, one layout strategy is to use a particular scale factor for each image on the tabletop so that the images near the perimeter of the table are larger. The middle horseshoe icon in the control panel shown in Figure 3 is a magnet symbol. When activated, all the images on the tabletop will be rotated to face the direction of this control panel (as shown in Figure 8). This provides users the flexibility to lay out story vignettes. Because the entire PDH tabletop can be rotated, each user sitting around the table can bring this layout to his or her own viewing angle.

Figure 8 also shows how the PDH table supports visible commands for easy learning of user interface (UI) functions. The four corners of each image document are used for visible commands. The two red corners on the upper left and lower right are used for easy resizing of the document. The green corner with a question mark on the upper right is user for instant enlargement and display of the four Ws information for that particular image. The purple corner on the lower left is the handle for free rotation of the document.

back to top  Input Modes

The primary input modes in PDH are touch/click or draw with digital ink. No keyboard is required. A virtual keyboard appears when it is necessary to enter text. For example, when the user creates a group to represent a set of items, such as grouping people into Family and Colleagues, she must name that group so that it can be stored and then retrieved later. To maintain consistency among UI affordances, UI functions are grouped into categories:

  • Touch for selection
  • Touch-and-dwell for popup, context-sensitive menus
  • Digital ink for selecting regions of interest
  • Drag-and-drop for moving contents or rotating the tabletop

bullet.gif Anywhere, Anytime, Multilevel Annotation

The PDH system provides an annotation tool that supports collaborative and continuous history building using photos and documents. The historical data of personal collections are stored in a database. The annotation tool operates on a desktop computer to make it easier to type in the annotations. For example, the annotation user interface for annotating Where is supported with a map view. By clicking on a map location, the PDH annotation tool provides the GPS longitude and latitude values for the user. Similarly, the PDH annotation tool provides a calendar for annotating When and a dictionary of names for annotating Who. The What interface is similar to that of Who except that we provide a dictionary of keywords.

back to top  Looking into the Future

We have conducted several qualitative user studies in the past year. In one such study family members shared photo collections during the holidays. In another study, our research lab shared their collective history as represented by photos of the people in the lab, events, and technical reports (reported in [5]). In general, people really enjoy the informality and pleasantness of a tabletop display for sharing stories and the images associated with those stories. One user commented, "I am totally forgetting that I'm interacting with a computer!" User reactions, comments, and answers to questionnaires have provided us with valuable feedback that we can use to improve and continue to develop our system. Meanwhile, we are also partnering with academic research groups to start bringing the PDH table to other locations, including tables at local cafés where the table is being used to support peripheral awareness by continuously and automatically displaying contents that are related to the community and the locale of the café and museums where the table is used by children to help them tell stories.

back to top  Acknowledgments

MERL colleagues Paul Bearsley and Baback Moghaddam helped with the early conceptual formulation of the PDH project. PDH would not have its aesthetic appeal without the design work of Ryan Bardsley of Tixel HCI. Ryan's artistic talent and industrial design skills brought our early vision into the first set of visual designs. Clifton Forlines designed and carried out the user studies that provided us with new insights into our design. Jeana Frost of the MIT Media Lab conducted interviews with senior lab members and collected and annotated the MERL lab history collection. Gisele Azevedo designed and implemented the first version of the PDH annotation system and image database. The DiamondTouch team at MERL Cambridge Systems Lab broadened our project's horizon and opened up possibilities with their multiuser touch technology. Kent Wittenburg at MERL has provided ongoing valuable discussions and suggestions. We have also benefited greatly from discussions with our colleagues Chuck Rich and Candy Sidner.

back to top  References

1. Balabanovic, M.; Chu, L.; and Wolff, G. Storytelling with digital photographs. In Proceedings of CHI2000 (The Hague, The Netherlands, April 2000).

2. Dietz, P. and Leigh, D. DiamondTouch: A multi-user touch technology. In Proceedings of ACM UIST'01, (Orlando, FL, Nov. 11-14, 2001).

3. Kang, H. and Shneiderman, B. Visualization methods for personal photo collections: Browsing and searching in the PhotoFinder. In Proceedings of IEEE International Conference on Multimedia and Expo (ICME; New York, NY, July 30-Aug. 2, 2000).

4. Mimio,

5. Shen, C.; Lesh, N.B.; Vernier, F.; Forlines, C.; and Frost, J. Sharing and building digital group histories. Presented at ACM Conference on Computer Supported Cooperative Work (CSCW; New Orleans, LA, Nov. 2002).

6. Vernier, F.; Lesh, N.B.; and Shen, C. Visualization techniques for circular tabletop interfaces. In Proceedings of Advanced Visual Interfaces (AVI; Trento, Italy, May 22-24, 2002).

back to top  Authors

Chia Shen
Mitsubishi Electric Research Labs (MERL)
Cambridge Research Lab (CRL)
201 Broadway, Cambridge, MA 02139

Nesh Lesh
Mitsubishi Electric Research Labs (MERL)
Cambridge Research Lab (CRL)
201 Broadway, Cambridge, MA 02139

Frédéric Vernier
Mitsubishi Electric Research Labs (MERL)
Cambridge Research Lab (CRL)
201 Broadway, Cambridge, MA 02139

Kate Ehrlich, Director,
Collaboration divine,
One Wayside Road, Burlington, MA 01803

Austin Henderson, Director,
Systems Laboratory Advanced Concepts & Design Pitney Bowes
35 Waterview Drive MS 26-21, Shelton, CT 06484

back to top  Figures

F1Figure 1.

F2Figure 2. Artistic rendering of the PDH table by Ryan Bardsley, Tixel, HCI (a) and the current PDH table in action (b).

F3Figure 3. The PDH control panel showing the main menu bars and selections.

F4Figure 4. The Who view showing thumbnails labeled with metadata of the people in a user's data collections.

F5Figure 5. The When view showing the images laid out along a date line. There are two control panels, one on the bottom and one on the left side. The green freeform stroke indicates the user is selecting years 1991 to 1996.

F6Figure 6. The Where view, with three control panels each placed about 90 degrees apart on the perimeter showing thumbnail images superimposed on their associated places.

F7Figure 7. Fisheye layout used to manage the space on the table and Passive associations in black and white around the perimeter of the circular display.

F8Figure 8. Rectangular linear layout for personal directional viewing and visible commands.

back to top  Sidebar: PDH Usage Scenario: Meeting with a Friend at a Café

Two friends, John and Mary, meet at a café. They sit around a PDH interactive coffee table, catching up on what's been happening in each other's lives. John asks, "How are your kids?" Mary replies, "They're doing great" and selects images of her children, four-year old Max and seven-year old Nadine, which appear on a view of her digital collection organized by the people who appear in them. She switches to a calendar view of the same collection, which now displays her collection chronologically. The pictures of her children are highlighted and displayed prominently. John rotates the circular display toward himself to admire the pictures taken of the children over the years. Mary asks, "When was the last time you saw them?" John says, "Must have been about two years ago" and circles the last two years on the calendar view with a freeform stroke, zooming in on the more recent pictures. After a moment, Mary presses the summarize button, and about 25 pictures of her kids taken in the last two years appear on the screen. John expands one of the pictures of Nadine, saying "Hey, isn't that the book I gave her? Does she like it?" Mary says, "It's one of her favorites! Listen to this" and plays an audio recording of Nadine reading the book, by pressing an icon near the picture.

John then says, "It must be hard to travel with two small kids." Mary responds by switching to the world map with photo thumbnails of her kids from the last two years shown at the location they were taken, and says, "Not really, I actually take them on many of my business trips." She pans to Europe and zooms into Copenhagen. Mary then enables the "association generator" by touching a button on the control panel, which generates a stream of photos sliding along the perimeter of the display at a pleasant rate. New ones enter from one side and disappear at the other. They are related to the images she has requested in a variety of ways, but do not meet the exact people, time, and location constraints she has specified. When a photo of a previous visit to nearby Legoland catches Mary's eye, she moves it to the center of the table and continues the conversation...

back to top 

©2003 ACM  1072-5220/03/0300  $5.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2003 ACM, Inc.

Post Comment

No Comments Found