Thoughtful theory of humanity

XVI.6 November + December 2009
Page: 28
Digital Citation

FEATUREReflections on representation as response

Kirsten Boehner

back to top 

I'm terrified of flying over the ocean. This is an unfortunate neurosis, considering my constant trans-Atlantic travel. My husband tries to assuage my fear of the dark ocean and its creatures engulfing me: He assures me I'd be dead on impact. Somehow this fails to comfort me.

While writing this article, a search for the remains of Air France Flight 447 is under way. I want the black boxes (or the orange cylinders) to tell us what happened. I want the people touched by this tragedy to have a small degree of peace. For myself, I want to be able to ask the pilot on my next flight, "Excuse me, have the Pitot sensors been replaced?"

This incident has prompted reflection for me beyond mortality, chance, and my own fears. I am coauthoring a book that promotes a "representation as response" orientation for computing design, yet I seem to be longing for the more dominant "representation as reality" approach. Why?

back to top  The Ideal of Reality

A representation as reality approach forges a one-to-one correspondence between something in the world and its abstract computer representation. This approach is at play, for example, in Intel's vision for a "proactive home" that could detect and deter intruders, adapt light levels and music for an impromptu party, or alert distant relatives if grandma were to fall in the shower. Implementing this system depends on matching events in the world, such as "upright grandma in shower," with modeled states and programmed responses of the computer system, such as "all is well; do nothing." This approach underlies the majority of advances in computing. It may also inform what happened to Flight 447.

An Alternative: Representation as Response. I am interested, however, in an alternate approach to the proactive home in particular and digital systems in general—representation as response. In this approach the system design and evaluation do not strive to mirror reality, but rather to provide a lens through which to apprehend or interpret reality [1]. Consider, for example, the Home Health Horoscope (HHH) [2] and the Tableau Machine (TM) [3]. The HHH (partially funded by Intel's People and Practices Group) generates a horoscope projecting the needs or dispositions of the home based on past activities. The TM animates real-time visual displays of the energy, density, and flow of home activity. Both systems use a complex array of sensors to model home activity, yet neither proposes to understand exactly what is happening. Instead they present contingent representations that require active participation and interpretation. But contingency is not what I want in relation to Flight 447.

Is there a limit, then, to the representation as response approach for HCI? Could an interpretive approach work for safety-critical systems? One affirmative response points out that playful/interpretive experiences are not limited to play. Another produces examples of safety critical systems that require a degree of openness to interpretation [e.g., 4, 5]. Notwithstanding these arguments, my reaction to Flight 447 has caused me to examine whether there is anything necessarily wrong with the representation as reality approach. Why not use technology to protect grandma at home or prevent future plane accidents?

Trouble in Paradise. The utility of systems that strive to model reality is obvious; systems designed in this manner, however, may lead to unintended, and undesirable, consequences.

Incomplete, Flat, and the Old Switcheroo. All designs have unintended consequences. Yet let's examine consequences that occur because of, not in spite of, the representation as reality approach. The beleaguered history of symbolic AI demonstrates that world models of reality are always incomplete and require constant refining [6]. This inherent challenge of attempting to model reality [7] demonstrates a further consequence of this approach: the loss of nuance. Complexity is flattened into models that a computer can track and understand. Lastly, these incomplete and simplified models, even if offered originally as approximation, take on a life of their own and, in a sense, become reality.

The Switcheroo in Action. Every day we encounter the implications of representation as reality. Anyone who has been denied insurance unjustly, been detained unnecessarily at customs/immigration, and received unwanted direct mail has experienced a mismatch between a model and his own life. Abstract models serve a purpose. They help us learn from singular cases and detect predictable patterns. They focus our attention on what is considered the most important information. But they also hold the seeds of reification. Once codified, whether in public policy or computational systems, models are inherently difficult to detect and modify.

Imagine, for example, a natural language processing application that parses text and measures it against a model of veracity. Managers could gauge their employees' loyalty; an online dater could rest easier about her potential partner; and governments could add it to their intelligence arsenal. Sounds reasonable, depending on the model's accuracy, who it is used on with what permission, and what is done with the information. And this is exactly the point. Abstract models seem benign in the abstract, yet can be potentially troublesome in the living.

Values in Action. Let's return to grandma. The proactive home can provide peace of mind and signal events of concern. There is something comforting about handing data over to smart machines that augment the reach of our senses and process enormous amounts of data in a manner seemingly free from human error. To be sure, the system will have false alarms and missed emergencies, for example if grandma falls in an unmonitored part of the house, thus requiring more and more sensors. The system will also introduce its own error rate due to technical limitations. But the success/failure ratio of the proactive home is only one way of looking at it.

The system will also change the way grandma and her relatives live and relate to each other. It will affect perceptions of health and safety. It will challenge the balance between protection and privacy. The goal to keep grandma safe is a valid one, but along with the readily apparent benefits, there are consequences to account for and consider.

Trade-offs are the heart of the matter. The critical point is not that representation as reality is better or worse than representation as response as a design philosophy, it is that each approach frames the same problem differently. It's a matter of recognizing that each approach will have intended and unintended consequences, some that cannot be anticipated but many that can. It's a matter of choosing what we value. It's a matter of epistemology.

back to top  Epistemology Matters

Given what I've described regarding the limitations of the representation as reality approach, why do I long for an indisputable account of Flight 447? Why do I want black boxes that were designed with the representation as reality approach?

Separating Mindsets and Methods. In asking these questions, I have confounded methods and mindsets. An approach to research or practice typically consists of an epistemology—or mindset—and techniques for putting this epistemology into practice. At its simplest, an epistemology is a bumper sticker: "Life Is a Dance. Get Moving" would appeal to those who see life as fundamentally about rhythm and engagement, whereas "Life Sucks and Then You Die" describes a darker, more cynical view. As researchers and designers, the epistemologies guiding our work are a bit more complex (maybe), but the idea is the same [8].

Mindsets and methods complement but do not replace each other. A positivist scientist might tend to value experimental methods but could also use experiential ones, in an effort to deduce the truth. On the other hand, a constructivist scientist might favor experiential methods but could also use experimental ones, in an effort to illuminate how truth unfolds. The mindset determines how methods are employed and to what ends.

Reality as Mindset or Method. The representation as reality approach describes both a mindset and a method. As a mindset, it construes reality as predeter-mined rather than an activity of interpretation. Information or representations about the world are tantamount to meaning in the world. As method, representation as reality consists of a one-to-one correspondence between objects and symbols. For example, sensor readings of electricity usage in the proactive home track as faithfully as possible the actual electricity usage.

Mixing It Up. Representation as reality methods are often motivated by a representation as reality mindset, but this does not have to be the case. The proactive home employs one-to-one correspondence techniques but could also be implemented, interpreted, and evaluated from a representation as response mindset. For instance, its success could be evaluated in terms of redefining family relationships or conceptions of the home.

The HHH and the TM systems, designed from a representation as response mindset, use some representation as reality techniques. The system designs employ complex models to approximate what is "actually" happening in the home—e.g., daydreaming versus partying—in order to represent this in a meaningful way. The shape of the design and its assessment do not hinge on an isomorphic mapping between system interpretation and actual events. The system could be wrong and still work as long as it is wrong in an interesting or useful way [2].

The black boxes of Flight 447 are also designed faithfully to track as much information as possible employing representation as reality methods to recreate events of the flight. The technology need not embody a representation as reality mindset, however. Data could be interpreted in a myriad of ways: conjecture, best guess, or even a declaration of what happened beyond a shadow of a doubt. The latter is what I'm hoping for, although the desire for certainty does not require a representation as reality mindset.

Skirting the Po'Mo Wonderland. A representation as response mindset accepts that there is reality. It is not an invitation to escape into a postmodern wonderland. There are causes and effects (sometimes less linear than we'd like) and seemingly objective accounts. Medical science, for example, can find the cause of a disease. Yet what is known is always a foil for the unknown. Diseases, for example, can evolve. Black boxes may faithfully track some information, yet unaccounted-for information might tell a completely different story. Saying that the story is incomplete, however, is not saying that one should abandon the search.

Marine biologists can read ringed sediments of a piece of coral to determine saline content and chemical composition of the ocean across centuries. The coral then is a kind of natural "black box" of information. Is this an example of representation as reality or response? Depends on what we do with the information. For our designed systems, our approach determines what and how we track, how we process it, what we do with the information, and how we determine (and define) whether the system is working or not.

A representation as response mindset takes as a given the limitations of representation as reality. But rather than correcting these, the gap between reality and the represented is something important to acknowledge. Due to this gap, the representation as response approach will shy away from systems that wrest authority from situated actors in place of an abstract computational model. This does not mean all systems should directly report only raw information collected. Instead, it is a call for making models apparent as well as open to change. A representation as response mindset is about designing systems for engaging with and probing reality.

Strawmen Are Made of Straw. Finally, in describing epistemologies, I've slipped into reifying abstract representations. Using terms such as "representation as reality" and "representation as response" divides them into neatly separated categories, whereas in practice, epistemologies are fluid and motley.

The table here highlights useful comparisons, not to rigidly classify one school from another, but instead to prompt reflection and discussion about how under-lying values and beliefs influence design and evaluation. The table then is offered as a representation as response, not reality.

By comparing the approaches in this way, their similarities become evident. Yet their differences are profound. Instead of a spectrum, they are like adjacent positions on a circle—in one direction close, but in another, a vast distance separates them. Either approach could be employed in the Flight 447 case and like scenarios, but the designs, assessments, and consequences would differ dramatically.

back to top  Making It Real

Through this reflective exercise, I compared methods with mindsets and revisited the imperative that epistemology matters. My desire for certainty in the face of inherent ambiguity in the world does not require a deterministic approach to design. I am not (necessarily) a closet positivist because I look for answers. Certainty and ambiguity have been presented as opposites but perhaps are better understood as forces in balance with each other.

What does this mean for practice? If epistemology matters, should designers become philosophers? Some would argue that they already are [9], whether they acknowledge it or not. And herein lies the essence of the idea: Acknowledging the assumptions and values at play provides a powerful lens on design at both a broad conceptual level and at the personal practice level.

Seek Out Shifting Ground. One fruitful way to draw out under-lying values and assumptions is to experience the clash of epistemologies. The field of HCI, with its range of disciplines, is quite used to these intersections. Important dialogue has emerged when collaborators reach across disciplines to find a wider than expected gap. From a representation as response perspective, this gap is a fertile ground for insight.

The recent interest in inter-secting the sciences with the arts and humanities in computing design is epistemological reflection in action, examining conflicting values and divergent practices. A bricolage approach might mine different techniques, such as using situationist art practices as a tactic for participatory design. A dialectic approach might look to create a third space between art and science. An antagonistic approach might try to revolutionize science through provocative art and art through provocative science. All of these possible conversations would be interesting for HCI and the design of interactive systems.

Raise Foundational Questions. Observing or participating in an exchange across epistemologies raises foundational questions, particularly when it's at the proverbial paradigm-shift level. Yet foundational questions can always be usefully revisited even on a small scale. Why is one mindset or technique being used in place of another? What is gained, what is lost? What would the effects be if the driving mindset and/or the pursuant techniques were tweaked?

Foundational questions serve as, well, foundations. We ask them so we can move on. But they can also serve as blinders, boxing us into static patterns of practice and ways of thinking. Revisiting such questions illuminates overlooked areas for design and provokes new ideas. Questions of epistemology force us to examine what we focus on, ignore, assume, and value.

Draw Out the Personal. Perhaps the most obvious antidote to abstraction is personal experience, such as the passport-control delay, where models are fine until they completely misrepresent your own personal situation. One strategy then is to draw out and draw on one's own personal lived experiences.

We tend to erase the designer or researcher's fingerprints in representing our designs due to the requirements of blind review, principles of scientific objectivity, and the issue of relevance. Does anyone really care to hear about my fear of flying? Certainly in the age of twitterdom, we might have sparse appetites for more personal details woven into our publications. But personal experiences inform choices and reform epistemologies. My take on representation as response is slightly different from that of others. Finding ways of drawing out and sharing these subtle fissures is as important as distinguishing tectonic plates in terms of provoking and advancing the fields of interactive design and HCI.

Make It Messy. Published work in HCI tends to recount linear narratives using a normative template. We rarely acknowledge failed systems; nor is there room for discussing the difficult choices and trade-offs we encounter along the way. Choices are made and forgotten. Our stories of design are neat and concise.

As designers are required to focus on more wicked and messier design problems, perhaps we need channels to present the messiness of the design process. Not simply about design in the abstract, but a forum for examining how design choices emerge on a smaller individual scale. Doing so will ideally draw out epistemologies at play.

The suggestions to seek out shifting ground, ask foundational questions, draw out the personal, and make it messy are an encouragement to collide, examine, and ultimately revisit our design epistemologies. But by challenging fixed epistemologies, and encouraging their dynamic interplay, we lose the sureness of our footing. In the May + June 2009 issue of interactions, Bruce Sterling suggested that design has more to learn from literature [10], although if we take on the idea of including more complex narratives and even personal accounts, perhaps the opposite is also true. What stories do we enact through design? What stories about design do we project and promote? What new stories could we tell?

back to top  Acknowledgments

Many thanks to Phoebe Sengers, Bill Gaver, Katherine Isbister, Michael Mateas, Kia Höök, Lucian Leahu, and Alex Taylor.

back to top  References

1. Bolter, J. and Gromala, D. Windows and Mirrors: Interaction Design, Digital Art, and the Myth of Transparency. Cambridge, MA: MIT Press, 2003.

2. Gaver, B., Sengers, P., Kerridge, T., Kaye, J. and Bowers, J. "Enhancing Ubiquitous Computing with User Interpretation: Field Testing the Home Health Horoscope." Proceedings of CHI 2007, ACM Press: 537–546.

3. Pousman, Z., Romero, M., Smith, A., and Mateas, M. "Living with Tableau Machine: A Longitudinal Evaluation of a Curious Domestic Appliance." Proceedings of UbiComp 2008, ACM Press: 370–379.

4. Denef, S., Ramirez, L., Dyrks, T. "Letting Tools Talk: Interactive Technology for Fire Systems." Proceedings of CHI 2009, ACM Press: 4447–4452.

5. Sengers, P. and Gaver, W. "Staying Open to Interpretation." Proceedings of DIS 2006, ACM Press: 99–108.

6. Leahu, L., Sengers, P., & Mateas, M. "Interactionist AI and the Promise of Ulbicomp, or, How to Put Your Box in the World Without Putting the World in Your Box." Proceedings of Ubicomp 2008, ACM Press: 134–143.

7. Rod Brooks has said, "The world is its own best representation." See Brooks, R. A. "Intelligence without Representation." Artificial Intelligence 47, 1–3 (1991): 139–159.

8. Bannon, L. J. "The Politics of Design: Representing Work." Communications of the ACM 38, 9 (1995): 66–68. See Winogrand and Flores [9] for early work on epistemologies in system design.

9. Winograd, T. and Flores, F. Understanding Computers and Cognition: A New Foundation for Design. Norwood, NJ: Ablex Publishing Corporation, 1986.

10. Sterling, B. "Design Fiction." interactions 16, 3, (2009): 20–24.

back to top  Author

Kirsten Boehner is writing a book on interpretive approaches to human-computer interaction. Her current research revolves around moments of transformative participation, in particular dialogic arts practices and the intersection with information technology design. She recently completed her Ph.D. in communication and postdoctoral research in information science at Cornell University.

back to top  Footnotes


back to top  Figures

UF1Figure. The Home Health Horoscope printer and a sample daily message.

back to top  Tables


back to top 

©2009 ACM  1072-5220/09/1100  $10.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.

Post Comment

No Comments Found