XXII.5 September-October 2015
Page: 48
Digital Citation

After interaction

Alex Taylor

back to top 


It is, arguably, where it all began for interaction. Doug Engelbart's retrospectively titled "The Mother of All Demos" spectacularly set the scene for what we do in HCI and especially for what we imagine interaction to be. Engelbart and his team showcased a remarkable collection of technologies for seeing and manipulating data. The demo is best known for the introduction of the mouse as an input device, but also presented was an integrated teleconferencing system and the simultaneous collaborative editing of a text document. Think computing in the late 1960s, but with the mouse and something akin to Skype and Google Docs. It's hard to imagine how extraordinary this must have seemed at a time when telephone adoption was only just reaching a plateau in the U.S. and, for all but the technological elite, the idea of networked computers was the stuff of science fiction or, more likely, just unthought of.

back to top  Insights


Taking nothing away from the remarkable technological achievements, Engelbart's film stands, of course, as a piece of theater. Through a series of clicks and drags, complex computational tasks are achieved that the then audience were well aware would ordinarily have taken numerous keystrokes, if not lines of code to execute. The magic of the demo is precisely that it seems to be the interaction that is, well, magical. The remote collaborations, especially, but also the use of the mouse for controlling various inputs, locates the action between human and machine. The careful choreography prefigures an interface, foregrounding a very particular set of relations between user and computer. Engelbart's demo literally performs the interface and the interaction.


This interaction has become a cornerstone for HCI and interaction design (IxD). It is simply taken for granted in what we do. Indeed, as even not so attentive readers will have noticed, in its plural it is the name given to this publication. There is much more to this, though, than names. Interaction does some conceptual work. It suggests that the applied and research work we do is concentrated at the human-machine interface. Yes, we can and have stretched this to include the possibilities of many humans and/or many machines, but nevertheless our business has been all about human-machine interactions, and in some cases manifold interactions.

Interaction as a concept is pernicious in that it diminishes our responsibilities and misses the proverbial trick when it comes to making better things and, ultimately, making things and worlds better.

Here, I want to argue that as a concept, interaction hinges on an outmoded notion of technology in use. I'll argue that technology use is, in fact, already and always has been about a lot more than human-machine interactions (at least in how interaction is regularly imagined in HCI and IxD). I want to suggest that what we have been doing by both investigating and designing technology is participating in and to some extent configuring dense, interconnected relationships of humans and non-humans. That is, we have been assembling and reassembling human-machine hybrids, often in great numbers. And rather than working at a neatly defined interface, we have knitted together and entangled ourselves in these interwoven networks of relations, and go on doing so.

There may be no controversy here, but what I want to add is that, under the rubric of HCI and IxD, we have been giving form to networks that mobilize and entangle not just people and machines, but also produce what we might think of as worlds—social, technical, scientific, intellectual, organizational, political, ethical worlds (to name just the obvious). Each and every time we have observed or set up an interaction between humans and machines, and brought into being particular relational networks, we have also given shape to distinctive regimes of knowing and being in the world. We have been invested in—to borrow a phrase from the philosopher Nelson Goodman—"world making" [1].

To make this more concrete, let us return to Engelbart's demo. As I've said, the demo creates the illusion that it's the communication from humans to machines (and back again)—the human-computer interaction—where the action is. However, much is also being done beyond this narrow stage, behind (and to the sides of) the scenes. Take the most obvious. The machines here have been configured just so to perform this demo. What we don't see are the connecting cables, the binding code, the collective organization, the infrastructures necessary to make things work [2]. Hidden is an entangled network of things and people that precipitates particular kinds of relationships and prioritizes particular ways of understanding the world.

The relations go much deeper than any discrete interactions between user and machine. It's startling, for example, to see how Englebart behaves very much like a machine. Admittedly, this may be more an indication of the era in which the film was made, but still we see that a very particular rhythm has been composed for this performance that feels peculiarly inhuman. The interface and the curated interactions necessitate a "body work" that entwines user and machine, configuring how one is in the world and thus how one reasons about it [3]. To imagine the human and machine as separate, mediated only by discrete interactions, belies how tightly interwoven things are.

So it is not merely the interface that has been designed; the assemblies of humans, machines, and interactions are all being (con)figured in some way. An "imaginative landscape" is conjured up with its own "extended network of human labors and affiliated technologies" and its particular elisions or "modes of erasure" [2]. This, then, is world making. The technologies presented in Englebart's demo aren't incidental to a world (or worlds) in the making—they are integral to it (them). A network of humans and nonhumans are co-constituting ways of seeing, knowing, and doing in very special ways; they bind and entangle to perform worlds.

I want to argue, then, that interaction as a concept presents a troubling framing of what we do. It is not just its narrowness. I believe it to be pernicious in that it diminishes our responsibilities and frankly misses the proverbial trick when it comes to making better things and, ultimately, making things and worlds better. My hope here is to propose a counter framing, one that tentatively sets the stage for a different encounter with the relations between humans and computers. This will be a framing that embraces the relations that are brought into being when we design things, and strives—albeit with hesitance and care—toward a thriving and flourishing of these relations in the worlds we bring into being.

To begin sketching out this line of argument, an important point to make is that the decision to concentrate our attentions on the interface (at the cost of seeing the wider relations) has its roots in a very particular metaphysics. Here, I am talking about a materialism where nonhuman matter is passive, non-vital—it is measurable, rule-governed, and subject to universals, and crucially (because of its passivity), it is understood to be in the service of humans [4]. In this light, it's understandable that we might conjure up a notion of interaction where on the one side, humans, and on the other, nonhumans (or computers), are neatly separated. Interaction design from this point of view is unproblematically the job of improving efficiency, productivity, or the experience by (re) designing the interface. Also, it is, as we say, human-centered, meaning we take the "exceptional" qualities of being human—thought, cognition, physiology, and so on—to be what guides the design of interactions with things. And yet, even with just a modicum of care, it's difficult not to see holes in this thinking.

First, there is the nagging question of how exactly we decide where the line is drawn between humans and nonhumans/computers? Already HCI has a broad body of work that problematizes the interface and interactions seemingly located between human and computer. Norman blurred the boundary, shifting some of what was thought to be human potential into things [5]. He wrote of things affording particular actions, famously detailing the "instructions for use" embedded in such mundane objects as door and mug handles. Work coming from the same scholarly tradition has demonstrated that cognitive capacities can extend beyond the individual's mind and into the interactions between people. Thus, models of interaction might be something shared, not just in the head, but also in the world. Susanne Bødker [6] and Lucy Suchman [2] have recognized the fluidity and instability of the interface. Bødker showed interfaces to be multiple as we tackle practical problems using computers; and, by calling attention to the emergent qualities of plans and actions, Suchman showed how our interactions with machines are highly contingent.

To a greater or lesser extent (and with, it must be said, quite different epistemic grounds) what these perspectives also invite are questions around agency. They begin to question not only the idea that agency is solely in the hands of human users, but also how it might be distributed, albeit unevenly, across relational assemblages of humans and nonhumans. In other words, they invite questions about where, exactly, we understand the interface and interactions to be, how discrete they really are, and whether there really is an intrinsic separation to be bridged between human and machine. So, building on the thinking from scholars like those noted here and many further afield, it seems fair to ask whether we have reached a stage where interaction raises more questions than it answers—whether, when carefully scrutinized, it starts to unravel and lose purchase as a useful concept.

I want to propose, then, a move beyond interaction. My first interjection is to contest the kind of materialism that treats human and nonhuman things as fundamentally separate—that imagines there is in some way an intrinsic or essential difference between the two. With a now well-established mode of thinking in the social sciences and humanities underlying my argument, my proposal here is simply to invite hesitation when it comes to working at the intersections between humans and nonhumans. Might we ask why the divisions must be cut in this way? Is agency so neatly divided? What might happen if we imagined other kinds of separation? And what might happen if we began to see these separations as enacted and not essential, but performed in very much the same way as Englebart's "The Mother of All Demos"?

Taking this invitation to hesitation seriously, we might begin to imagine a new materialism or sociomaterialism [4], where things as well as humans are enlivened and where it isn't any sort of "natural" division between humans and nonhumans that dominates our thinking, but rather a much richer notion of relations that are always under way and always performing some kinds of worlds. The design challenge here shifts away from working within a stable and tightly demarcated interface to making sense of a wider set of fluid relations, ones that we ourselves are inexorably entangled in and thus must be accountable for. The emphasis broadens to be about what worlds we are making, and indeed, the kinds of worlds we want to make.

Take something close to my own research interests, London's public rental-bike scheme. In terms of interaction, the challenge would seem to be one of providing the public with quick and easy ways to rent bicycles. With a network of 700 docking stations across the city and 10,000 bikes moving between them, complexity becomes an especially important factor to monitor and handle in designing efficient technologies and services. Interaction concentrates the action between human and non-human actors, and design is then a remedial business of improving, serving, or sometimes enriching the user's interactions with the scheme's systems. But this fails to account for relations with the wider organization of the city and civic/social life. Undeniably apparent is that the technologies are implicated in a panoply of infrastructural, economic, and political concerns, and yet, seen through a lens of interaction, the relational threads binding these strewn-together worlds are somehow lost. As if by magic, interaction collapses into a narrow problem to be "solved."

We might imagine how the bike scheme could be designed to enliven the connections throughout the city by allowing relations to intensify and flourish.

In contrast, a relational framing gives us a sense that something rather different is at stake in the technological and infrastructural capacities we might design for. As with other sprawling and densely populated cities in the global North, contemporary London is undergoing a project of regeneration and gentrification. Programs supported often through public-private partnerships have been undertaken to redevelop neighborhoods seen as economically deprived in central and greater London. Yet, while there have been visible improvements to these areas, the programs have not been without controversy. Commentators write of a "cleansing" of neighborhoods as the regeneration has driven up prices and driven out financially less-stable cross sections of the populace. In other words, a diversity that has arguably been the hallmark of London, and been the route of much of its success as a cosmopolitan city, is eroding. While great towers are going up, peculiarly singular ideas of regeneration are exerting a flattening force on the city. London is being smoothed over, subjected to a force where qualities of difference are disappearing in favor of an aesthetic of sameness.

The bike rental scheme—its own instance of a public-private partnership—cannot be properly understood outside this worldly regime. Its technologies of interaction are inexorably bound up with the flows and trajectories of urban and civic life in London. As an example, through the scheme's mechanisms for payment and bike logging, journey times on the rental bikes turn out to be highly constrained, with over 95 percent of all journeys falling under a free 30-minute threshold. Set alongside the divisive decisions on where to locate the bike's docking stations, the scheme's own data paints a picture of a patchworked city with hubs in the financial districts and dense spokes funnelled to the residential neighborhoods that serve them (Figure 1). Large areas to the east and southeast are rendered invisible in these cycle-data routes. The physical and computational infrastructures are then knitting new layers of physical movement and computation into an historic infrastructure that shapes and undergirds London; they are etching different geographies of human-machine circulations into and across the urban topography, and giving shape to distinct rhythms coursing through the city. In this instance, they also bring into being a network of nodes and connections across London that perhaps unsurprisingly correspond to where wealth and prosperity are accumulating.


With such an intermingling of people, things, agencies, and infrastructures, it feels peculiar—misleading even—to cement and narrow in on human-machine interactions and to center agency around the human. These networks of transport (and the technologies and infrastructures that are part of them) present so much more than neatly demarcated interactional problems between human and computer. They invite a deeper understanding of the unfolding relations between humans and things, and their capacities for enabling particular forms of social and civic life. With a great deal of hesitance and care, we might wonder how the machines and infrastructures could play into a different kind of atmosphere in the city, one that affords a multiplicity of worlds—of logics, beliefs, values, and so on. We might imagine, for example, how the bike scheme could be designed to enliven the connections throughout the city, not by funneling or flattening them, but rather by allowing the relations between people, geography, infrastructures, and technologies to intensify and flourish. What this means for the design of specific technologies I leave open. My point here is that we see the possibility of framing our relations with technology very differently, and, in doing so, we might begin to see the scope of our work open up to a much richer set of possibilities.

In closing, then, I want to ask in more general terms what it could be to work with these much wider and denser networks of relations. Interaction presents our work as deceptively up an idea of a neatly demarcated intersection that we might work on, an interaction that we can stand outside of and unproblematically design and optimize. Paying closer attention, though, we see very quickly that we have always been entangled in worlds in the making, worlds where we differentiate, prioritize, and ultimately exert particular kinds of value.

And so, after interaction, with hesitance and care, might we ask how we want to understand and design technologies in these worlds? Resisting the forces that narrow or flatten worldly capacities, and seeing agency as emerging in the manifold relations, can we think of an orientation to design that seeks to thicken the relations, that is about how the multiple worlds—in their combination and through a dependence on each other—thrive? This I take to be a more responsive and responsible framing of design, and, I would argue, one that is far more open to the possibilities of something different and, perhaps, better.

back to top  References

1. Goodman, N. Ways of Worldmaking (Vol. 51). Hackett Publishing, 1978.

2. Suchman, L. Human–Machine Reconfigurations: Plans and Situated Actions, 2nd Edition. Cambridge Univ. Press, 2006.

3. Myers, N. Molecular embodiments and the body-work of modeling in protein crystallography. Social Studies of Science 38, 2 (2008), 163–199.

4. Orlikowski, W.J. Sociomaterial practices: Exploring technology at work. Organization Studies 28, 9 (2007), 1435–1448.

5. Norman, D.A. The Psychology of Everyday Things. Basic Books, 1988.

6. Bødker, S. Through the Interface: A Human Activity Approach to User Interface Design. Erlbaum, Hillsdale, NJ, 1991.

7. Siddle, J. London maps and bike rental communities, according to Boris Bike journey data. The Variable Tree (blog); http://vartree.blogspot.co.uk/2014/03/london-maps-and-bike-rental-communities.html

back to top  Author

Alex Taylor is a sociologist at Microsoft Research. He has investigated various aspects of everyday life, and has recently begun obsessing over computation and wondering what the compulsion for "seeing data everywhere" might mean for humans and machines. ast@microsoft.com

back to top  Figures

F1Figure 1. "Connected clusters" of bike docking stations by James Siddle [7].

back to top 

Copyright held by author. Publication rights licensed to ACM.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2015 ACM, Inc.

Post Comment

No Comments Found