Authors:
Jie Li
I recently had a fascinating museum visit with my 6-year-old daughter, traveling through a giant "human body" at the Corpus Museum in Leiden, the Netherlands. The one-hour trip started at the knee and went all the way up. We witnessed how platelets helped heal a bleeding wound and how it felt to be inside a womb. We stood among the stomach, liver, and pancreas to watch them collaborate to make digestion work. We walked past the heart and the lungs to see red blood cells queuing up, catching oxygen by passing through the lung capillaries and releasing the oxygen to the rest of the body with the help of the heart. The highlight was inside the mouth, where my daughter got excited counting how many teeth the giant had and jumping on its soft tongue, which lit up at different zones to explain the flavor it can taste (Figure 1).
 | Figure 1. The mouth of the giant human body at the Corpus Museum (image painted by the author based on her impressions of the visit). |
Unlike professionally illustrated textbooks or encyclopedias, having such an immersive trip inside a human body is enchanting. It requires a lot of resources and work, however, to build and maintain the 5-floor-tall body. The experience is also difficult to diversify. Unlike other museums that frequently curate different exhibitions to attract return visitors, this human body trip has an absolute "wow" effect for first-time visitors, but it's not the type of museum people want to visit a few times a year.
Insights
→ Today people can experience a range of immersive experiences, from immersive museums to high-fidelity virtual worlds.
→ The boundaries between immersive and real-world experiences are increasingly blurred.
→ We need to find ways to engage a broader range of users in an inclusive and engaging metaverse by carefully considering content, interaction design, and accessibility.
CAVE: Customizable Immersive Experiences
Without requiring physically building such a giant human body but still providing an immersive experience, a cave automatic virtual environment (CAVE) is another option. CAVE is typically a cube-like room with 3D images or videos projected onto its walls to create a walk-in immersive environment. The automotive industry has been adopting CAVE to aid its designs and to evaluate interior concepts such as instrument reachability, displays, and reflections in the windshield (Figure 2) [1]. The CAVE concept has been brought to the public in the format of "immersive art," such as the 2019 immersive Van Gogh show in Paris, where visitors were immersed in Van Gogh's masterpieces, walking among sunflowers or lying beneath the stars (Figure 3). In CAVE, experiences can be easily diversified by changing the projected 3D images or videos. You can bring family and friends and share this immersive experience with them. The CAVE cube sets a boundary between the real word and the projected one.
 | Figure 2. CAVE assists car designers in testing designs before production (image painted by the author). |
 | Figure 3. The 2019 immersive Van Gogh show in Paris (image painted by the author based on her impression of the show). |
The Boundary between the Virtual and the Real Worlds
In the previous two examples of immersive experiences, the boundary is a physical one: People can exit the giant human body through hidden side doors or leave the CAVE by stepping out of the cube. Today, be it a head-mounted display (HMD) or a smartphone screen, we can remove ourselves from the virtual world simply by putting away these devices. In the past two decades, the interface that we use daily to enter the virtual world has not fundamentally changed. Although the computational power has increased enormously and a tiny smartphone can do much more advanced things than a desktop computer could 20 years ago, the interface is still screen based, from big, low-resolution noninteractive monitor screens to high-pixel-density OLED touchscreens. Even though people complain about excessive screen time and gradually losing touch with the people around them, it is still simple to cut ourselves off from the infinite scrolling on social media platforms: Just turn off the screen.
The main issue of wearing an HMD is not just the ergonomic discomfort but also the isolation. It keeps you isolated from family members who are supposed to be enjoying the fun with you.
With the rapid development of extended reality (XR) technology and commercially available HMDs, the rise of high-fidelity virtual worlds, and the various ways of crafting virtual avatars and interaction techniques, we are witnessing a big paradigm shift in how people engage in the virtual world. Popular applications, such as the award-winning virtual reality (VR) games Beat Saber (https://beatsaber.com) and The Under Presents (https://tenderclaws.com/theunderpresents), have motivated many customers to buy an HMD. In Beat Saber, by holding two lightsabers, players compete in slashing the visualized music beats flying toward them in a neon virtual world (Figure 4). The Under Presents immerses users in virtual live theaters where they can explore novel, interface-free interactions, such as reaching their virtual arms out and pulling the destination toward them rather than teleporting to it.
 | Figure 4. A Beat Saber user playing in the neon virtual world (image painted by the author). |
Although millions of VR HMDs have been sold in the past few years thanks to these popular applications, mainstream adoption is still a long ways away. I often hear comments from HMD owners that the main issue of wearing an HMD is not just the ergonomic discomfort but also the isolation. It keeps you isolated from family members who are supposed to be enjoying the fun with you (unless, for example, you buy an HMD for each person). You could cast what you are seeing on the HMD to a TV screen, but watching on a 2D screen is far from the immersive experience you are having on the HMD. In this scenario, the boundary between the virtual world and home is an obstacle for families to adopt this new form of entertainment.
Definitions of the Metaverse
The Corpus Museum physically built a giant human body to offer visitors an immersive trip, but it is not the metaverse. Second Life (https://secondlife.com), a Web-based 3D virtual world, is a metaverse that has existed for nearly two decades and has millions of users. People "live" in Second Life in parallel to their real lives. Users can build social relationships and engage in entrepreneurial activities such as creating and selling virtual real estate and digital goods. It has an internal economy with a closed-loop virtual token called the Linden dollar. Users have embodied presence by creating their avatars. Second Life has many characteristics of a metaverse, but it is not the metaverse that we talk about today.
What is the metaverse? In Ancient Greek, meta is used as a prefix to mean "after" or "beyond," which continues in the English language today. The term metaverse was coined in Neal Stephenson's 1992 science fiction novel Snow Crash, as a combination of the words meta and universe [2]. So, metaverse literally means "beyond universe," a reality that coexists with our reality. As described by Charlie Fink [3], in the metaverse "you will have a secure personal identity, and you will move seamlessly from place to place, just as you move from website to website. It will be spatial, 3D, and any device with a browser…will be able to access it. A headset won't be a requirement. It has to be social. It has to have an economy, which means ways to make and sell stuff." In other words, the metaverse will remove the barrier of geographical distance and physical limitations, enabling everyone to do things that they do in the real world and even things that they cannot. Users can smoothly travel between an infinite number of parallel virtual spaces and the physical world. In an interview with Mark Zuckerberg in the Verge [4], the author first summarizes venture capitalist Matthew Ball's view that a metaverse "has to span the physical and virtual worlds; contain a fully fledged economy; and offer 'unprecedented interoperability.'" She then paraphrases Zuckerberg, writing that the metaverse "will be an 'embodied Internet,' operated by many different players in a decentralized way." In short, the metaverse is a network of digital reality that is cocreated by different players. It combines aspects of social media, online gaming, VR, augmented reality (AR), and digital currencies to seamlessly connect interactions in the physical world with interactions in these virtual ones.
Engaging Users with Lifelike Experiences: Laboratory Prototypes
The reason why Second Life has been successful for two decades is that many people are still actively using it, creating experiences and sharing them. A promising future for the metaverse means that players are going to build an enormous number of virtual worlds. Inevitably, there probably will be many abandoned ghost virtual worlds. How do we keep virtual worlds prosperous? In the era of the experience economy, people are not easily satisfied with physical goods, which are fungible with many other products. People are time-starved and do not want to waste their energy on ordinary things; they're craving diverse, unique experiences that are intangible but still memorable [5]. Engaging more users in the metaverse is not just about selling equipment or creating a fixed experience such that they only wow once (e.g., the giant human body trip). It is more about delivering diverse and lifelike interactive experiences.
Many industrial and academic laboratories have explored ways to provide these lifelike interactive experiences. Microsoft Research has presented an end-to-end system for AR and VR telepresence called Holoportation (https://youtu.be/o00mn1XbClg) [6]. The system demonstrates high-quality, real-time 3D reconstructions of an entire space, including people, furniture, and objects, using a set of depth cameras. These 3D models are transmitted in real time to remote users wearing VR or AR headsets to see, hear, and interact with other participants in 3D, almost as if they were all in the same physical space. Meta's Reality Lab is pushing the limits of capturing hardware and trying to build lifelike codec avatars for diverse humans [7]. Codec avatars are built based on the captures by hundreds of high-resolution cameras, which are used to train AI systems. Google launched Project Starline, a 3D video chat system that enables users to look through a magic window and both see their loved ones, life size and in 3D, sitting across from them, and talk with them. Project Starline offers a feeling of presence similar to AR or VR, without the need to wear bulky headsets or trackers [8]. These systems require a lot of high-end hardware to run, however, so they are not easy to replicate outside of research laboratories.
People are time-starved and do not want to waste their energy on ordinary things; they're craving diverse, unique experiences that are intangible but still memorable.
In academic laboratories, researchers are trying to offer lifelike immersive experiences using lighter hardware setups. Our team [9] introduced a walk-into-the-movie experience (Figure 5). Each user was captured by three depth cameras and their 3D avatar was reconstructed and transmitted to a virtual apartment. When users met one another's photorealistic 3D avatar inside the virtual apartment, they had a sense of social presence and a feeling of walking into the movie. The idea behind this prototype is to demonstrate the future possibilities of watching a TV show or movie. Soon, when you are talking about a movie you watched last night, you might say, "I saw my friend's hologram standing behind the detective. She read the forensic report the detective was holding."
 | Figure 5. The walk-into-the-movie experience: The users' photo-realistic avatars copresent with the virtual movie characters in the virtual movie. |
Compared to the capturing systems of Microsoft Research, Meta, and Google, the virtual movie setup is much lighter. Even so, we cannot imagine that users would set up and calibrate the cameras themselves at home. To enable more users to engage with the new experiences created in laboratories requires extended collaboration between industrial and academic players to further develop the technology—for instance, making the depth cameras smaller and easily installable at home and developing software to help users calibrate them.
Engaging a Wide Range of Users with Accessibility Design
Whether using an HMD or walking into a CAVE, an immersive experience is still rare for most people because of the hardware investment required. To minimize the accessibility obstacles to virtual worlds, many Web-based applications (e.g., Mozilla Hubs, Virbela) have sprung up. Without a bulky HMD, users can enter a virtual world using a Web browser on a laptop or mobile phone. Building a Web-based accessibility solution is only a start to engaging more users. People with motor disabilities, impaired vision, or chronic pain may find virtual experiences less enjoyable if they require a lot of physical movement to get engaged, or if they contain primarily visual stimulations.
Furthermore, most of the available HMDs are not designed for children. I often need to hold the HMD on my daughter's head to enable her to experience what an immersive three-dimensional virtual world is. The primary concerns regarding the digital-native generation are about addictions and problematic behaviors that may be caused by the combination of unique components in the virtual world and the gamification elements [10]. When designing for accessibility for children, finding a way to smartly implement intervening techniques is essential to ensuring supervised use of the metaverse.
Concluding Remarks
Both academic researchers and technology companies will play an essential role in advancing the development of the metaverse and turning it into high-quality, accessible services for end users. New accessibility design standards and usability testing methods for metaverse services must be developed to widen the user group and, most importantly, ensure user safety and comfort. Next, the development of new technology and software solutions, such as more-user-friendly HMDs and more-portable depth cameras that are also easier to calibrate, is necessary to turn laboratory prototypes into market-available products. Establishing the metaverse economy is also important, which requires defining business strategies and enabling business owners to seamlessly monetize their products and services in both the physical world and the metaverse.
The metaverse is within reach. Before the smartphone era, many people read newspapers and paper books on public transportation. Now they are often busy with their phones. They may read on phones, but they also do many other things: working, grocery shopping, playing games, and just being absorbed in the little screen. With the metaverse, we will witness a new revolution in digital interactions. Hopefully, the future metaverse will not pull people into the little screen in front of them but instead offer an enormous network of virtual worlds in which to engage a wide range of user groups, where people work, live, and connect. People are not limited to a screen but rather are open to incredible interaction possibilities, with the virtual environment and with other people. To ensure the metaverses are attracting users and everyone has an equal opportunity to get engaged in them, we need to consider the design of the content and interactions, the hardware and software, and the accessibility for the elderly, children, and people with disabilities.
Acknowledgments
I would like to acknowledge and give my warmest thanks to Pablo Cesar, Jonathan Lupo, Alexandra Diening, Abdallah El Ali, and Haozheng Zhu, who helped make this article possible.
References
1. Zimmermann, P. Virtual reality aided design. A survey of the use of VR in automotive industry. In Product Engineering. Springer, 2008, 277–296.
2. Stephenson, N. Snow Crash (First Edition). Bantam Books, New York, 1992.
3. Fink, C. This Week in XR: Is there a cure for metaverse mania? Forbes. Aug. 12, 2021; https://www.forbes.com/sites/charliefink/2021/08/12/this-week-in-xr-is-there-a-cure-for-metaverse-mania/?sh=ac14b6c16228
4. Newton, C. Mark in the metaverse: Facebook's CEO on why the social network is becoming 'a metaverse company. The Verge. Jul. 22, 2021; https://www.theverge.com/22588022/mark-zuckerberg-facebook-ceo-metaverse-interview
5. Pine, B.J. and Gilmore, J.H. Welcome to the experience economy. Harvard Business Review (Jul.–Aug. 1998).
6. Orts-Escolano, S. et al. Holoportation: Virtual 3D teleportation in real-time. Proc. of the 29th Annual Symposium on User Interface Software and Technology. ACM, New York, 2016, 741–754.
7. Meta. Facebook is building the future of connection with lifelike avatars. Mar. 12, 2019; https://tech.fb.com/ar-vr/2019/03/codec-avatars-facebook-reality-labs/
8. Lawrence, J. et al. Project starline: A high-fidelity telepresence system. ACM Trans. on Graphics 40, 6 (2021), 1–16. DOI:10.1145/3478513.3480490
9. Li, J. et al. Evaluating the user experience of a photorealistic social VR movie. Proc. of 2021 IEEE International Symposium on Mixed and Augmented Reality. IEEE, 2021, 284—293. DOI: 10.1109/ISMAR52148.2021.00044
10. Kaimara, P., Oikonomou, A., and Deliyannis, I. Could virtual reality applications pose real risks to children and adolescents? A systematic review of ethical issues and concerns. Virtual Reality 26, 2 (2022), 697–735. DOI: 10.1007/S10055-021-00563-W
Author
Jie Li is head of research and insights at EPAM Netherlands. She holds a Ph.D. in human-computer interaction from Delft University of Technology. Her work focuses on designing and evaluating user experiences for emerging technologies such as social virtual reality. She is also a cake artist and owns a boutique café called Cake Researcher. [email protected]
Copyright held by author. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.
Post Comment
No Comments Found