Authors:
Nour Halabi, Evan Jones, Pejman Mirza-Babaei
Games, as a platform, have the power to transport us to different worlds, providing unique and captivating experiences. In the realm of virtual reality, this potential is amplified, offering a level of immersion that was previously unimaginable. What makes VR even more exciting is the possibility of gesture-based interactions, where the players' bodies become the controllers, bridging the gap between the virtual and physical worlds. In this article, we delve into the realm of gesture-based VR interaction design, exploring the experiences they unlock and how they have changed not only the gaming landscape but also the interaction design space. Our exploration is inspired by a collaboration between Stitch Media, a Canadian game development company, and UXR Lab, an academic research lab at Ontario Tech University, on the development of Broken Spectre, a commercial VR game that utilizes gesture-only interactions to enhance player immersion.
Insights
→ Gesture-based VR interaction offers unparalleled immersion and engagement, blurring the boundaries between the real and virtual worlds.
→ Design considerations such as intuitiveness, tutorialization, and accessibility are crucial for creating user-friendly and fulfilling gesture-based interactions.
→ The collaboration between game development studios and academic HCI research labs may provide valuable insights and guidance for designing effective gesture-based interactions.
VR platforms such as Oculus Quest have revolutionized the gaming landscape through groundbreaking innovations such as full-body movement and gesture-based interactions. These platforms leverage advanced technologies such as microelectronic sensors, including gyroscopes, magnetometers, and accelerometers, to determine the orientation of a user's body, providing inside-out tracking and hand-tracking capabilities. Combined, they represent a fundamental change in the way we think about designing game interactions. For example, by freeing players from the constraints of traditional controllers, VR platforms are enabling players to use their own body parts (e.g., head, hands, legs) as intuitive input mechanisms. Unlike in the real world, however, the absence of proper force feedback limits the ability to make interactions feel truly natural. Moreover, VR experiences may create a mismatch between the vestibular system and other sensory stimuli; for example, when your vision indicates an acceleration of some sort but that acceleration doesn't match with your movement in the real world, it could lead to the unpleasant sensation of motion sickness (also referred to as simulation sickness in VR design). Balancing the enticing possibilities of this platform with the complications it poses is the challenge of designing for virtual reality.
Gesture-based VR interaction has the potential to revolutionize gaming experiences, offering players unparalleled levels of agency, presence, and immersion. Agency involves the control and influence that players feel over their virtual surroundings. By using their bodies for in-game actions, players experience a heightened sense of agency, as their physical movements seamlessly translate into virtual actions. This empowerment allows them to shape their experiences and make meaningful choices, resulting in a deeper sense of engagement and presence. Presence is the feeling of truly "being there" in the virtual environment, both physically and mentally. Gesture-based interactions play a significant role in enhancing presence by establishing a direct and immediate connection between the player's physical body and the virtual world. When players can perform gestures that mirror real-world actions and interact with objects directly, they become immersed in the game world. Immersion is crucial, as it integrates the player's senses, emotions, and actions with the digital experience, creating more captivating, memorable games. The seamless and responsive nature of gesture-based interactions elevates the level of immersion that players can achieve, allowing them to fully embrace the narrative and challenges presented by the game. This level of immersion has the potential to enhance storytelling, deepen emotional connections, and empower players to fully embody their virtual counterparts and shape their individual destinies within the game.
Designing for Immersive Gesture-Based VR Interaction
Designing controls that feel natural and intuitive is an exciting challenge. Game designers often have their own expectations and ideas about how a game should play, but it is crucial to align these with the player's actual experience. We strive to understand the player's needs and requirements, taking inspiration from their mental model—a framework that helps us grasp their expectations and how they interact with the system. By designing games that take the player's mental model into account, designers can create intuitive and user-friendly systems based on familiar interaction patterns.
Through our collaboration, an understanding of the key design considerations emerged, which we dive into below. From the designer's perspective, considerations such as intuitiveness, environment, tutorialization, and accessibility are crucial (Figure 1). First, designers should align gestures with the player's mental model, providing intuitive interactions that are distinguishable from one another. Second, the environment, both physical and virtual, plays a significant role. Designers must consider the affordances and technical constraints of the system, such as the hand placement and orientation being accurately captured by the camera or the visibility within the digital environment. Third, tutorialization is important for introducing novel interactions to players and avoiding misinformation when adapting and learning new ways of interaction. The lack of tangible feedback generated by controller-free interaction in the virtual world may require both additional time for players to get accustomed to and adequate attention to other feedback sources (audio and visual). Lastly, accessibility should be addressed, taking into account factors such as play space, handedness, and the ability to play seated or standing. Providing options that cater to different user needs expands the accessibility of the game. For example, in Broken Spectre, players are given the option to sit while playing, which makes it accessible for a broader range of users. Another example is the rock climbing interaction, which has a significant role in Broken Spectre. Through iterative development to the climbing gesture, two modifications were implemented that take into account the discussed considerations. One is where the players' virtual hands can securely latch onto rocks after performing a grasping motion. The other is a visual indicator that simultaneously informs the player which rock is eligible for snapping when gazed upon and when their hand is fully snapped to a rock.
 | Figure 1. A reflection on designer considerations and player expectations. |
Players, however, have their own expectations when it comes to gesture-based interactions in VR. They expect comfort to be taken into account by avoiding tedious or repetitive motions. Accounting for the ergonomics of the interaction and conducting user testing will help designers decide which gestures are unsuited for continued repetition over a long time period. Another key expectation is interaction fidelity, which we discuss in greater detail below, where players' level of immersion is influenced by their expectation of having varying levels of interaction fidelity, such as the ability to mimic real-world actions in certain situations. Flexibility in interactions empowers players to tailor their experience, while error prevention is necessary to avoid frustration when the resulting behavior does not match what the player expected from the gesture or the player is uncertain why an error occurred. To ensure a seamless immersive experience, minimizing errors and providing reliable feedback is necessary.
The introduction of gesture-based VR platforms has unlocked a multitude of experiences that were previously unimaginable.
These considerations serve as a valuable guide for developers, helping them navigate the challenges and complexities of designing gesture-based interactions in VR, ultimately leading to the creation of captivating and immersive gaming experiences.
Engaging with Objects Through Gestures: Exploring the Spectrum of Interaction Fidelity
Engaging with objects in the virtual world through gesture-based interactions is another key element for immersion based on varying levels of interaction fidelity. Interaction fidelity is a spectrum that refers to the realism and responsiveness of the virtual world's response to the player's physical actions.
The player's "mental model," their intuitive understanding of how things function, greatly influences their gaming experience. By aligning the designer's perspective with the player's understanding, we can bridge the gap and create seamless experiences.
At the lower end of the spectrum, we find basic interaction fidelity, where virtual objects respond in a simplified and generalized manner. At this level, the focus is on conveying a basic sense of object presence and interaction. While the level of realism and responsiveness may be limited, it still allows users to engage with and manipulate virtual objects to some extent. Basic interaction fidelity is suitable for introductory experiences or situations where a more casual and accessible approach is desired. Moving up the spectrum, we encounter moderate interaction fidelity, which enhances the believability of object interactions. Here, virtual objects exhibit more-realistic behaviors and responses. Physics simulations, material properties, and object dynamics are taken into account to create a heightened sense of authenticity. For example, when we pick up a virtual cup, it may have weight and react to our movements as we tilt it or pour liquid from it. At the highest end of the spectrum, we find advanced interaction fidelity. This level aims to replicate the real-world interactions with the utmost precision and realism. Virtual objects respond in a highly realistic manner, closely mirroring the physics and behaviors of the real world. Every gesture, subtle movement, and touch is accurately captured and translated into the virtual environment, creating a profound sense of presence and agency.
Broken Spectre creatively leverages these interactions, enabling players to seamlessly pick up, manipulate, and interact with various objects in the game world. For instance, players can extend their hand to grab a virtual object and perform gestures to manipulate it (Figures 2 and 3).
 | Figure 2. A screenshot from the game Broken Spectre showcasing the use of the compass movement gesture and the representation of a ghostly figure. Upon gaze detection, the figure appears ahead to represent preordained locations where the player can teleport to once they confirm the movement by closing their hand. |
 | Figure 3. A screenshot from Broken Spectre showcasing the use of swing gestures (interaction) to cut through thick grass with a sickle that mirror real-world actions. |
Conclusion
Gesture-based VR interaction has opened up new avenues for immersive experiences in gaming. The insights shared in this article highlight the significance of agency, presence, and immersion in designing effective gesture-based interactions. By empowering players with control and influence over their virtual surroundings, creating a sense of "being there" in the virtual environment, and enhancing overall engagement and immersion, developers can fully harness the potential of gesture-based VR interaction. Through the collaborative efforts of Stitch Media and UXR Lab, key design considerations have emerged, encompassing intuitiveness, environment, tutorialization, and accessibility from the designer's perspective, as well as comfort, interaction fidelity, flexibility, and error prevention from the player's perspective. By integrating these considerations, developers can navigate the complexities of designing gesture-based interactions, resulting in user-friendly and captivating gaming experiences. As we venture further into the realm of gesture-based interactions, there are still many unexplored research avenues and exciting possibilities to be discovered.
Authors
Nour Halabi is a designer and researcher specializing in the fields of games user research and human-computer interaction at Ontario Tech University. With a focus on unlocking effective communication channels for player data, she utilizes the power of data visualization techniques, player issue severity assessment, and immersive interaction design. Her passion for collaborative research extends to both academic and industry projects, striving to bridge the gap between theory and practice. [email protected]
Evan Jones is a two-time Emmy Award winner. His work combines Web, mobile, and games with TV, film, radio, and the real world. He founded Stitch Media in 2007 with a vision that interactive stories combine distinct elements into something greater. He has collaborated with dozens of creative and technical teams to achieve award-winning projects across many platforms and genres. [email protected]
Pejman Mirza-Babaei is an interaction design consultant, author, and research professor at Ontario Tech University. He coauthored The Game Designer's Playbook: An Introduction to Game Interaction Design (2022) and coedited Games User Research (2018). He has worked on pre- and postrelease evaluations of more than 30 commercial games and other interactive products. [email protected]
Sidebar: Suggested Further Reading
Jicol, C. et al. Effects of Emotion and agency on presence in virtual reality. Proc. of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2021; https://doi.org/10.1145/3411764.3445588
McMahan, R.P., Lai, C., and Pal, S.K. Interaction fidelity: The uncanny valley of virtual reality interactions in virtual, augmented and mixed reality. Visual, Augmented and Mixed Reality: VAMR 2016. Lecture Notes in Computer Science 9740. Springer, Cham, 2016; https://doi.org/10.1007/978-3-319-39907-2_6
Rogers, K., Funke, J., Frommel, J., Stamm, S., and Weber, M. Exploring interaction fidelity in virtual reality: Object manipulation and whole-body movements. Proc. of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2019; https://doi.org/10.1145/3290605.3300644
Stahlke, S. and Mirza-Babaei, P. Chapter 7: Rejecting your reality. In The Game Designer's Playbook: An Introduction to Game Interaction Design. Oxford Univ. Press, 2022.
Warren, L.E. The Effect of Interaction Fidelity on User Experience in Virtual Reality Locomotion. Master's thesis, Virginia Tech, 2018.
Copyright held by authors. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.
Post Comment
No Comments Found