ACM Interactions

Authors:
Davide Zhang

---

 

The meaning of spatial design is changing, and its implications are far-reaching. Traditionally rooted in architecture, landscape, and urban planning, spatial design has been brought into the limelight with the recent popularity of mixed-reality headsets, augmented-reality platforms, and real-time photorealistic rendering. Apple, for example, has dedicated sessions on the principles of spatial design. If we assume that the way we use language helps shape our culture, then this recent shift in the definition of spatial design will have a profound impact on how we understand space, how we design spaces, and what makes a good space, which touches the foundation of disciplines like architecture. For one, our homes as interfaces have very different dynamics from current headset user interfaces.

  Insights

→ "Spatial design" is shifting from traditional architectural disciplines toward digital interfaces.
→ Current interfaces on spatial computing hardware underutilize the spatial medium.
→ There is an opportunity to integrate HCI research with architectural paradigms to create actionable concepts around embodiment, intimacy gradients, and ambient transitions.

But the concept of "spatial" in the context of digital design is not new. It evolved from explorations of text-based virtual places like multiuser dungeons, to the initial hype surrounding VR in the late 1980s, to the emergence of 2D graphical virtual environments, to immersive CAVEs, to collaborative virtual environments (CVEs) and "inhabited information spaces" in the 1990s and early 2000s. Some efforts focused on practical applications for communication and collaboration in virtual spaces and most attempted to re-create the physical world in digital space. Others explored how data might be visualized and digitally rendered to afford spatial investigation of abstract data spaces, building on ideas from architectural environments [1,2]. CVEs and graphical virtual landscapes like Second Life (launched in 2003), World of Warcraft (launched in 2004), and others laid the groundwork for understanding "spatial" not just as a visual representation involving landscapes and avatars but also as a multifaceted concept encompassing a sense of "place." At the same time, designers of flat screen graphical user interfaces have drawn on physical metaphors—skeuomorphism in the early days of app design and more recently Google's Material Design system, which borrowed motifs like shadows, layers, and "physical" depth to guide interface layouts. The expanding use of this term in the context of spatial computing falls into this lineage.

"Spatial" in the context of technology is not a recent pursuit either. A vast portion of the global population already benefits from spatial computing capabilities: positioning, remote sensing, GIS, and spatial data science. They power ride-sharing, GPS, location-based social media, and even social gaming experiences like Pokémon Go [3]. Computing has thus embraced the spatial dimension, yet physical spaces and digital interfaces each come with decades (or centuries) of accumulated design heritage that must be reconciled.

Recent discussions on architecture in the context of HCI highlight the potential common ground these two fields share. As embodied interaction becomes more central to how we live and work, both architecture and HCI stand to benefit from an ethnographic lens that details how people move within, perceive, and interact with spatial context [4]. This shifts the two disciplines away from purely designer-centric intentions and toward user-driven behaviors unfolding in complex digital-physical hybrids.

Yet, as consumer head-worn spatial computing platforms emerge, we see a disconnect between the interface and the 3D environments offered by this medium. The interfaces follow the classical deterministic HCI pattern, while the 3D environments approach "spatial" literally as stylized 3D assets and skyboxes. Therefore, there is an opportunity to look carefully at decades of work on virtual environments, immersive interfaces, embodied interaction, human-building interaction [5], and ubiquitous computing to synthesize them into actionable frameworks.

  A Tale of Two Interfaces

In a playful nod to a certain 19th-century novel, let us begin by juxtaposing the design ethos of physical architectural spaces with digital HCI [6]. While this comparison necessarily simplifies a vast body of work, contrasting the two can expose different approaches to "space" and interactivity. It is critical to acknowledge that HCI spans a broad array of approaches—tangible computing, ubiquitous or pervasive computing, gestural interfaces, body-tracking systems, and more—that often break free from the notion of a 2D digital screen. Similarly, not all architecture is about quiet, ambient spaces; environments like Vegas casinos are meticulously designed to capture attention and even employ overstimulation techniques like infinite scrolling to form addiction [7]. Consequently, the comparisons that follow refer to typical or mainstream tendencies in each field, rather than universal truths.

Conflict of objectives: Task vs. activity. Many digital interfaces—even consumer software in mixed-reality headsets today—are task oriented, guiding users through discrete actions. Users input commands, the system computes results, and outputs appear on a screen. These looped interactions can feel deterministic: The user is "outside" the system, telling the computer what to do [8].

Architectural interfaces, on the other hand, are often activity oriented, concerned with supporting human behavior at many scales, from daily routines in a living room to the flux of travelers through an airport terminal. The interfaces change based on the number of humans, proxemics, social relationships, surfaces, materials, lighting, and objects in space. Notably, humans are part of the interface. People coinhabit physical spaces, and their behaviors dynamically reshape these spaces (Figure 1). Christopher Alexander and colleagues capture this fluidity by describing patterns of intimacy gradients, showing how we partition a house into more private or more public zones that promote or constrain different activities [9]. This resonates with parts of the HCI literature on activity theory, which likewise emphasizes broader, evolving activities rather than isolated tasks. Tangible and embodied interaction, as well as ubiquitous computing in HCI, do also consider how environments and physicality shape the usage of space.

Figure 1. Abstracted comparison between the architectural interface (left) and classical HCI interface (right).

Conflict of immersion: Objectivity vs. subjectivity. We often approach conventional digital interfaces as external objects—screens and devices that we can manipulate at a distance, forming clear mental models of "the phone," "the laptop," or "the headset." Even gestural- or voice-based systems remain, in most cases, physically separate from our bodies.

Architecture, however, tends to envelop us. Spaces are perceived through ever-changing, subject-centered viewpoints, and we can rarely perceive the entire building simultaneously. We are, in effect, "inside" the interface. Yet immersive environments have existed in HCI for decades as VR labs, CAVEs, and collaborative virtual environments [1]. So "being inside the interface" is not new. The point is that consumer spatial computing devices are making these immersive experiences more commonplace and integrated into everyday life, but they are often literal interpretations of spaces as 3D assets, rather than the expanded views developed over the decades. As Muriel Cooper's pioneering "Information Landscapes" first demonstrated, data itself as a 3D navigable environment opens new ways of perceiving and interacting with information [10].

Conflict of mediation: Layers of input. Human-computer interaction is mediated by input devices such as a mouse, joystick, keyboard, touchscreen, and sensors enabling gesture and body tracking. Our intent is translated to the usage of an input device. The input device then sends digital commands that are processed by a computer before an output is returned. Sometimes, the feedback loop is so quick that it creates the illusion of unmediated interaction. For example, real-time camera feeds and natural language interfaces remove the need to operate an input device, but the input device is still present—only the feedback chain is compressed.

In physical environments, the body itself can be an interface—simply walking through a room changes the relationship of occupant to environment. In spatial computing, we see attempts to reduce friction in similar ways: full-body tracking, eye tracking, and environment-aware sensors strive to create experiences that feel "unmediated."

Conflict of attention: Background vs. foreground. Digital interfaces in mainstream consumer markets are often intentionally attention-seeking, shaped by business models that rely on capturing as many user interactions (and as much screen time) as possible [11]. This maximalist approach is in contrast with architectural spaces, which can either be quietly ambient (a calming retreat) or intensely engaging (a concert hall). While it is true that some physical spaces, such as Times Square or a flashy retail showroom, are designed explicitly to catch the eye, many others aim for a more subtle, atmospheric role. Human-computer interfaces capture the entirety of one's foreground, whereas the architectural interface recedes to the peripheral vision or background.

If we let the attention-seeking paradigm take over spatial computing, we might arrive at a future where spatially anchored virtual ads and content take over our physical space to a point of overstimulation and desensitization. We have an opportunity to design more ambient, less distracting interfaces that align with how architectural spaces quietly support activities in the background.

Conflict of affordance: Physical vs. minimal. Modern digital interfaces favor minimalist designs with flat icons and minimal shadows, emphasizing simplicity and abstraction. Before the current minimal and flat trend, interfaces did go through a stage of skeuomorphism, where they mimicked physical referents stylistically to create affordance. However, as James Gibson points out in his theory of affordances, affordances are embedded not only in the physicality of the object itself but also in the environment in which it is situated—it is an invariant property of an ecological object [12]. Digital icons only suggested physical referents, but they lacked the deeper contextual cues that come from an object existing in a place, with surrounding materials, lighting, and constraints.

However, HCI has also explored embodied, tangible, and ubiquitous computing for many years—where physical objects become part of the interface. The question is: How do we bring those real-world affordances back into "spatial interfaces" at scale?

  Bringing 'Space' Back to 'Spatial Design'

Here are some ways we can leverage paradigms from physical spaces to design for spatial computing interfaces.

Concept 1: Situated embodiment. The key idea is to integrate the sense of colocated, embodied interaction with people, objects, and environmental conditions.

In a physical space, a group that is interacting is always colocated within the same space. Through interaction with other humans, we dynamically shape the architectural interface, which in turn shapes the way we engage with it. How can we bring this situated, embodied experience from architecture to human-computer interfaces?

The first aspect is embodiment: interacting with other humans. In the real world, we interact with humans as they are. In the digital realm, collaborative virtual environments explored this decades ago. But we now have consumer headsets that let us place embodied representations of remote humans, or avatars, in the same space. This was the original vision of Microsoft Mesh, where real-time 3D avatars collaborate in the same physical space. Today, the closest implementation to this vision is Apple's Spatial Personas with SharePlay on Vision Pro, where realistic 3D avatars are colocated in the same physical space to engage in the same activity.

But there is a limit to how many embodied digital avatars we can fit in a particular space. This is due partly to the physical dimensions but also to the intimacy gradient of spaces, which refers to the relationship between private and public space in a house [5]. A bedroom is usually perceived to be a more private and intimate space than the living room, and we are comfortable with fewer people present. Digital space has an intimacy gradient as well. From one-to-one video calls to mass multiplayer games, we constantly traverse the private-public spectrum in the digital realm. How can we experience colocated embodied experiences across the physical and the virtual private-public spectrums?

The diagram in Figure 2 explores embodiment possibilities across the physical and virtual intimacy gradients. For example, when we are in a digitally private and physically public scenario, such as a work conversation in an airport, spatial computing can create a private virtual boundary for human-scale 3D avatars. In a digitally public and physically private scenario, embodiment can be abstracted to 3D geometries or heatmaps to leverage the qualities of physical space but still be represented in a way that is comfortable to the privateness of space.

Figure 2. Embodiment diagram based on physical and digital intimacy gradients.

The second part of this concept is situatedness. How can spatial interfaces colocate us and other people, objects, and spatial conditions? From the perspective of the user, we can create a situatedness gradient showing how much of another space is situated within our space (Figure 3). For example, a teleported avatar in our space is different from a teleported avatar with half of their space merging with ours. As we move through different levels of digital integration, from simple avatars to fully immersive virtual spaces, we begin to see how these digital experiences can reshape our interaction with both physical and virtual environments. Just as buildings can be seen as a changing medium of data rather than a static object, situatedness is experienced in motion across time. This notion of a large amount of data extending over longtime frames falls into the line of inquiry of HCI [4].

Figure 3. Situatedness diagram.

Due to recent leaps in computer graphics, we can prototype the spectrum of situatedness with space-time Gaussian splats today [13]. Using advanced real-time engines and application programming interfaces, we can now situate anything from small objects to partial spaces and even full environments across time, creating a more integrated and immersive spatial experience.

Concept 2: Dynamic ensemble. The key idea is to give interfaces an elastic quality, such that adding or removing elements changes functionality in flexible, context-aware ways.

Physical rooms can accommodate a surprising amount of rearrangement (e.g., removing a coffee table, relocating chairs) before they lose their essential identity. If you swap out a couch for a large dining table, the function of the room might pivot from "living room" to "dining area." In other words, architectural interfaces can change their functionality dynamically. Can human-computer interfaces leverage the built-in elasticity in architectural interfaces?

Responsive UI today partially achieves this by shifting and rearranging UI components based on the dimensions of the canvas. By following the hierarchical zones dictated by the UI's information architecture, the interface can maintain its functionality across different canvas sizes, such as laptop screen, tablet, and smartphone, but they don't have the ability to change functionality on the fly.

Three-dimensional interfaces can not only extend this elasticity but also embrace dynamic functionalities. As UI elements are positioned near certain objects, new functions might become available, such as a note-taking widget appearing near a desk, or a cooking assistant appearing near a stove (Figure 4). Beyond a certain point—like switching from a "living room layout" to a "kitchen layout"—the interface might reorganize or highlight different features, similar to how a room's identity shifts with certain furniture.

Figure 4. Diagram for expanding UI from 2D to 3D while preserving hierarchical relationships and introducing dynamic functionalities.

This aligns with the notion in spatial computing to move beyond static overlays or discrete apps, and instead use flexible, location-aware systems that adapt to user context. Like the user-facing front, the backend data layer can fluidly join or leave an integrated 3D interface [3].

Concept 3: Ambient continuity. The key idea is to create a flexible interface that morphs from peripersonal to extrapersonal, foreground to background, attention-seeking to ambient, object-centered to subject-centered, flat to immersive, point-of-view to god's eye.

Architectural spaces are immersive, subjective, and ambient. Imagine an interface that is unbounded by the attention-seeking, 2D rectangle we always hold in our hands. It seamlessly bridges from being within arm's reach to being in the environment, being perceived as an object to being perceived from a POV, and being outside of an interface to being inside of it. It foregrounds itself when needed and blends into the background when not.

How can spatial interfaces bridge this duality?

One way is to literally break apart the 2D interface we are familiar with and make it architectural (Figure 5). For example, we could expand the interface along a third axis according to its information hierarchy. Actionable UIs, such as buttons and calls to action, are closer (less z-depth). They fade away as the UI becomes more ambient. And content, like text and images, is expanded farther (more z-depth).

Figure 5. Continuity diagram: mapping UI to architectural conditions.

The content that is far away blends with the surfaces and architectural conditions, such as walls, windows, ceiling, edge, corner, and floor (Figure 6). The wall is naturally a display canvas for content, as is the window, which favors the image more. The ceiling and floor are often overlooked and are great for subtle background activities or notifications. The edge and the corner are often leveraged by architects but underutilized by the rest of us. The meeting of multiple planes creates a visual tension that lends itself to decor or widgets on display. Architectural conditions are sometimes associated with latent meaning that affords the interaction modality. A door invites and activates upon proximity; the floor traces our movement and reflects it back to us; and walls have been drawn on and carved into. We don't need to look far to envision ways of interacting with an interface that adheres to ambient continuity.

Figure 6. Options for mapping UI to architectural conditions.

Central to this concept of ambient continuity is three-dimensionalizing a 2D interface. It's as simple as adding z-depth to UI elements, but how do we decide the amount of z-depth for each component? In a two-axis world of 2D UIs up until now, design systems have all followed an implied order from up to down and left to right. This order itself is a function of how we perceive time and structure our language, which is a fascinating topic beyond the scope of this article. For example, in both Apple's Human Interface Guidelines and Google's Material Design system, headers are at the top of the page, and going up one level is implied at the left of the interface.

The introduction of a third axis brings an immense new opportunity for a new hierarchical order, one that is based on space, or more specifically, the concept of ambient continuity (Figure 7). As interfaces become closer to us, the information density increases, resulting in the attention-seeking layouts we are familiar with today. As they go farther away, the information density decreases and they morph into condensed widgets, ultimately blending into the environment as architectural elements, 3D objects, or animated phenomena.

Figure 7. Information density diagram—variation in info density with distance.

Another way of achieving ambient continuity is to make the interface form-fitting and map UI to architectural objects (Figure 8). This strategy suits not only the interfaces of smart appliances but also real-world counterparts of digital tools, like a clock, a calendar, and other gadgets that become features in our smartphone.

Figure 8. Continuity diagram—mapping UI to architectural objects.

Figure 9 shows several ways, depending on the nature of the interface, to map it to objects. Form-fitting is self-explanatory and establishes a strong connection between object and information. A digital twin allows us to track and customize physical objects digitally. A 3D graph treats the object as a node and connects spatially to other objects or UI components. In a way, this surfaces the latent connections between objects in our space and equips them with a digital layer. Lastly, the interface can also be partially occluding or embedded into the object to highlight parts of the object itself.

Figure 9. Options for mapping UI to objects.

The third way of achieving ambient continuity is for interfaces to create architectural conditions on their own, continuing the lineage of Muriel Cooper's information as landscapes (Figure 10). Just as data intrinsically exhibit qualities of a field, user interface patterns possess similar qualities to those of physical space.

Figure 10. Continuity diagram: UIs becoming architectural conditions themselves.

Concepts like partition, column, hierarchy, compression, shadow, enclosure, and object can be applied to both 2D UI and interior spaces (Figure 11). Instead of adding z-depth to UI components, we can apply 3D transformations to create spatial conditions. Vertical navigation bars can wrap around to create a column; elements on top of one another can project real shadows; callouts can be curved to compress and thereby emphasize the space; subsections can be enclosed three-dimensionally; illustrations and animations can morph into physical objects that embody the same quality, like a rippling curtain.

Figure 11. Options for UIs becoming architectural conditions.

  Conclusion: Toward a 'Spatial Interface Design'

"Spatial design" has historically referred to the expertise of architects, urban planners, and interior designers, but it is increasingly being adopted by the spatial computing industry for interface design. Behind this shift in language is a rich lineage of spatial approaches in digital design and computing, spanning everything from inhabited information spaces to location-based social networks. Recent explorations hint at a potential convergence of architecture and HCI, viewed through an ethnographic lens. What if, instead, we articulate a more specific notion of spatial interface design—one that explicitly merges design research on immersive HCI, collaborative virtual environments, human-building interaction, and architecture's long heritage of shaping human activities and experiences?

By drawing insights from decades of research, recognizing that architecture and HCI are not monoliths, and leveraging architectural paradigms, we shed light on actionable concepts around presence, embodiment, intimacy gradients, elasticity of function, ambient transitions, and activity-centered design. They invite us to see the entire environment as an interface and embrace the human capacity to inhabit spaces physically and digitally.

  References

1. Snowdon, D.N., Churchill, E.F., and Munro, A.J. Collaborative virtual environments: Digital spaces and places for CSCW: An introduction. In Collaborative Virtual Environments: Digital Places and Spaces for Interaction, E.F. Churchill, D.N. Snowdon, and A.J. Munro, eds. Springer, 2001, 3–17; https://doi.org/10.1007/978-1-4471-0685-2_1

2. Snowdon, D.N., Churchill, E.F., and Frécon, E. eds. Inhabited Information Spaces: Living with Your Data. Springer, 2004; https://doi.org/10.1007/b97666

3. Shekhar, S. and Vold, P. Spatial Computing. MIT Press, 2020.

4. Sauda, E., Karduni, A., and Lanclos, D. Architecture in the Age of Human- Computer Interaction. Routledge, 2024.

5. Alavi, H.S., Lalanne, D., Nembrini, J., Churchill, E.F., Kirk, D., and Moncur, W. Future of human-building interaction. Proc. of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2016, 3408–3414; https://doi.org/10.1145/2851581.2856502

6. Dickens, C. A Tale of Two Cities. James Nisbet & Co. Limited, 1902.

7. Schüll, N.D. Addiction by Design: Machine Gambling in Las Vegas. Princeton University Press, 2012.

8. Kirsh, D. Do architects and designers think about interactivity differently? ACM Transactions on Computer-Human Interaction 26, 2 (2019), 1–43; https://doi.org/10.1145/3301425

9. Alexander, C.., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., and Angel, S. A Pattern Language: Towns, Buildings, Construction. Oxford University Press, 1977.

10. Cooper, M. Information Landscapes. Back Office; http://www.revue-backoffice.com/en/issues/01-making-do-making-with/nolwenn-maudet-muriel-cooper-information-landscapes

11. O'Brien, H.L., Roll, I., Kampen, A., and Davoudi, N. Rethinking (dis)engagement in human-computer interaction. Computers in Human Behavior 128 (2022), 107109; https://doi.org/10.1016/j.chb.2021.107109

12. Gibson, J. The theory of affordances. In Perceiving, Acting, and Knowing: Toward an Ecological Psychology, R. Shaw and J. Bransford, eds. Lawrence Erlbaum Associate, 1977, 67–82.

13. Li, Z., Chen, Z., Li, Z., and Xu, Y. Spacetime Gaussian feature splatting for real-time dynamic view synthesis. arXiv: 2312.16812v2, Apr. 4, 2024; https://doi.org/10.48550/arXiv.2312.16812

  Author

Davide Zhang is a technical product designer at Microsoft, currently working on Windows Volumetric Apps. Previously, he prototyped haptic wearables at MIT Media Lab and studied computer science, architecture, and design technology at Harvard University and Columbia University. He is passionate about mixed-reality experiences and interactions. [email protected]

 

Copyright 2025 held by owner/author

The Digital Library is published by the Association for Computing Machinery. Copyright © 2025 ACM, Inc.