Sus Lundgren, Theo Hultberg
Theo had just discovered the musician Robert Fripp, who is famous for recording himself while playing and then playing it backin effect, playing along with himself .
Theo: "What if we do that in interaction design?"
Theo:"What if we use time as a kind of design material?"
The question kick-started an intense brainstorming session and a project that was to last for a couple of months. Theo has a point. Most of the time we do not start out by considering time in our designs; we let other parameters steer design and let the use of time become a consequence. "The time dimension, if we may call it that, is left to take care of itself," wrote design method guru John Chris Jones . For instance, when the user goes for lunch and leaves the interactions article she is writing behind in Word, was it actively decided that the time "in" Word should stop? Or, that if she later needs to open an older version of the document, time must back up and restart? What if, we asked, time "in" a program really does pass? How would this affect use? And design?
Our approach was not as original as we first thought. Looking around, we found creative examples of time use all around us. "Eternal Sunshine of the Spotless Mind," for instance, a film in which two lovers erase but nonetheless rediscover their love story, features time in fragmented moments along a distorted timeline. In another movie, "Time Code", separate but converging storylines play out simultaneously across a four-way split screen. The TV series "24," about recklessand sleepless!federal agent Jack Bauer, takes a totally different stance. Here, a minute is a minute both for Jack and for the people viewing him. Each season consists of 24 one-hour episodes (including commercial breaks), ultimately composing one full day. The same take is used in the online game World of Warcraft, where game-world time passes regardless of whether or not you are logged in, and where the annual Feast of Winter Veil is celebrated in December. Here, time is live, unlike tycoon or simulation games, where the player can stop time when building or play it forward at any pace she or he wishes. Or why not look at the new and original game Braid, in which the user has to manipulate time in different ways. In one version the player cooperates with an earlier version of himself, in order to solve some quests.
As for creative use of time within interaction design, Lars Hallnäs and Johan Redström have experimented with time in their slow technology program. In the "Sound Mirror," a microphone connects to a speaker that plays back what is recorded . The recordings are played at a later time, ostensibly at random. However, in general time tends to be discussed rather than used in the interaction design community. Youn-kyung Lim and her co-researchers touch upon time when presenting attributes of interaction gestalt . One of the attributes is related to time, namely "time depth"; two others are speed and pace, which are related to movement over time. In accordance, Tom Djajadiningrat with colleagues consider "timing rhythm and flow" to strongly influence the aesthetics of interaction . In HCI, time is of course measured, added, quantified, applied, and visualized .
Looking at these examples, we can discern different themes of time use. Similarly to how Lim and company discuss interaction gestalt using a language of attributes such as pace, directness, and orderliness, we think one can discuss temporalitythe relation to timein terms of these themes.
- Live time is the time we live by. It passes regardless of what the user does, like in World of Warcraft, or the live broadcasting of the Super Bowl (a rerun would not be live time though!).
- Real time is the theme used in "24"a minute is a minute, but it has no connection to time in the "real world."
- Unbroken time is when time passes in an unbroken sequence, but the speed of time may be altered or even stopped, as in Zoo Tycoon or Sim City. Many software programs use this theme. If the user does not do anything, the program (or the time in the program, seen in terms of changes) "stops."
- Sequential time is when events are still in chronological order but some events are skipped, i.e., scenes that jump in time by a couple of days or weeks, something that often occurs in movies and TV series.
- Fragmented time is when events are shuffled in time. This too is frequently done in movies, books, and TV, e.g., when a person remembers something from their childhood. It's the equivalent to opening an earlier version of a document; in a sense you are now jumping back in time.
- Juxtaposed time is when events that originally happened after each other are shown simultaneously, like when a time traveler meets himself.
This list is not definitive. One could identify many more steps in between, and one could also make the case that some of the types of time we list here are similar enough to fall into a single category. However, these six themes may well serve as a basic set of concepts that we can use when discussing how to apply time.
First, note how they distribute themselves along two parallel scales: one describing how close to reality the use of time is, the other denoting the level of control that the author, designer, or perhaps even user has. We all know that we as users often prefer a certain amount of control, but do we need as much control as in juxtaposed time? There is something to gain in staying close to reality too; World of Warcraft is mesmerizing in part because battles are fought whether or not you are there to participate.
To answer Theo's original question, and to explore the themes, we imagined a small, simple drawing application that would allow the user to do only one thing: draw lines. Then we asked, what if? What if one applies fragmented time, real time, or any other one of the themes to this very basic application?
The result was four drawing applications that apply time in different ways. We tested them ourselves but also let others try them; four users have tested each one, plus a class of 10 interaction designers. If you want to try visit http://demo.iconara.net/temporal-paint/ and let us know how it goes!
In the first application, Echo, the lines you draw gradually fade away by becoming thinner and thinner (unbroken time) until they disappear completely. After a more or less random interval, they reappear as "echoes" (fragmented time), only to again fade away, and come back, and so on. This was quite frustrating to the users, first because it took a while before they understood how the application worked, and second, because the user had no control whatsoever over when a certain line would reappear. A common strategy was to draw the same lines over and over again to make sure they would "stay," or rather that some instance of them was echoed at every moment in time. Only a few test subjects utilized the echo-effect by creating something that could be described as a mix between an animation and an association game; once some lines had disappeared, new lines were added, changing the motif from, say, a house to a flag. When we later added the possibility of drawing static lines by pressing Ctrl, the application changed drastically. We instantly developed a technique to draw animated scenes, such as drawing a static cloud and echoing lines as "rain" under it.
The second application, Perspective, was inspired by perspective drawings. Thus, the lines in Perspective slowly move toward the center of the canvas (real time), shrinking in the process, so that it appears if the user is zooming out. This zooming effect is almost immediately visible, and most test subjects quickly understood what happened. Also, the pace is slow enough for the users to plan what they wanted to do. Since the effect was predictable, most users felt at ease in a way that we did not see with Echo. All but one user worked with the effect, drawing typical perspective drawings, such as a long street stretching toward the center of the canvas. Again we added a feature, by allowing users to change the direction and/or speed of how lines moved (changing the theme to unbroken time), but in this case the change did not encourage any entirely new drawing approaches.
The last two applications toyed with changes in color. In ColorWheel the color changes continuously, whether or not you are drawing (real time), but in ColorPen the color changes only when you draw (unbroken time)all the red color has to be used up before you can use the orange color. Apart from the color of the line, none of the applications gives any other indication of the current color; the only way to discover it is to draw. As it turned out, users had a difficult time understanding how or why the colors changed, which made most of them ignore the color when drawing, and ultimately ignoring the temporal effect entirely. Others drew in different parts depending on the color they got (when blue, sky; when green, grass).
There seems to be a connection between how visible the temporal effects are and how easy it is to work with an application. In the case of Perspective, perhaps the most liked of the four, the effect is immediately apparent, but in ColorPen and ColorWheel, the underlying logic is hiddenit's quite hard to figure out what's going on. Echo falls somewhere in between. One of the effects, the fade, is obvious, but the other, the echo, has a randomness that makes it impossible to predict. Of course this has to do with control issues; if users understand how the effect works, they can perhaps create strategies to utilize it, as in Perspective. This, the possibility to utilize and work with the effect in some creative way, seems to be another key factor when applying temporal effects to a design. Again, the small change in Echo, where one can draw static lines, increased control and resulted in a somehow better product. Note also that this version of Echo was the only application where we allowed users to control the use of time rather than just analyzing and utilizing the effects of it. A possible next step would be to study how users react to different ways of controlling time.
Our work was experimental, so one may well ask, is it even possible to make "usable" products this way? Well, let's imagine adding a database and "fragmented time" to a search engine; now we can make searches for different points in time to see how the hit list changes and how some hits appear, disappear, and/or climb. Or let's add "real time" to Word and decide that newly written text is blue, however slowly fading to black after a few hours of active (unbroken) time. Or combine this effect with live time, letting the color of the text depend on the time of day, so that we easily can find that paragraph we were working on before lunch, and so on. These effects could be very useful for work in long documents, such as, a dissertation, in which the user frequently jumps back and forth. Note that even if we started with a very simple application as a basis for idea generation and development, there is no reason for the applications to be simple and easy to use. But from our limited testing, it appears that at least too much uncontrollable or unforeseeable behavior makes the experience less enjoyable.
Notably, most of our inspiration came from the entertainment sector, and it may be the case that unusual uses of time work best in noneffective applications. On the other hand, this may just be an old habit that's hard to break. We seem to have no problems at all dealing with fragmented time or even juxtaposed time in these contexts, so it may be a matter of letting creative use of temporality expand into new disciplines, such as interaction design.
Why should we leave such an intriguing way to enhance, steer, and/or inspire interaction to the game industry alone?
Sus Lundgren is a Ph.D. student and teacher at Chalmers University of Technology in Gothenburg, Sweden. She has spent a ridiculous amount of live time writing her Ph.D. thesis on teaching aesthetics in interaction design, and hopes to successfully finish by the end of 2009. With a background in GUI design and gameplay design, Lundgren is now shifting her focus toward interaction design didactics and design methods.
Theo Hultberg is a rich Internet application developer based in Gothenburg, Sweden. The aspect of interaction design he finds most fascinating is time, yet he never seems to find any to spend completing his master's thesis.
©2009 ACM 1072-5220/09/0700 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.