Authors:
Victoria McArthur
On June 23, 2018, 25-year-old assistant soccer coach Ekkapol Chantawong led 12 team members, ages 11 to 16, on an exploration of the Tham Luang Cave in Thailand. Soon after the team entered the cave, heavy rainfall flooded a significant portion of the 6.2-mile system, trapping them inside. Nine days later, on July 2, the group was found deep within the cave on an elevated rock, a couple of miles from the cave's main entrance. Rescue efforts were impeded by rising waters, limited visibility, strong currents, and a complex tunnel system in the cave with extremely narrow passages. To get the team out before the next monsoon rain, an international team of more than 10,000 people, including nearly 100 expert divers, raced to retrieve the boys. Divers from several countries worked with Thai Navy SEALs to extract the team and their coach between July 8 and July 11. The divers carefully moved the boys, who had been given anesthetic to render them unconscious for the three-hour journey, through the tunnel system. The rescue was covered by social media and news outlets worldwide.
One of the most striking examples of media coverage of the rescue was an augmented reality (AR) article released by the New York Times on its mobile app on July 21, 2018. AR is an interactive experience in a real-world environment that involves "augmenting" physical spaces or objects with computer-generated information or virtual content. The AR experience utilizes a reader's mobile phone to present text and images describing the rescue (Figure 1). Using the phone's camera and built-in motion tracking technology, the New York Times app not only presented a chronological account of the rescue but also rendered narrow passages as AR "windows" in the reader's physical environment. By juxtaposing the passages in this way, the Thai cave AR experience connected the reader to the story in ways that other forms of digital journalism could not, providing the necessary context to appreciate the impact that the narrow passages had on the rescue.
 | Figure 1. Screenshots of "Step Inside the Thai Cave in Augmented Reality," by the New York Times, shown in the author's dining room. |
Insights
→ Emerging technologies such as augmented reality (AR) have the potential to connect audiences to stories in new and exciting ways.
→ An archive of AR journalism can help inform future practice.
→ Codesign approaches between HCI researchers and journalists can support the advancement of AR journalism.
With the pervasiveness of mobile phones, journalism has increasingly sought to mobilize digital media to share stories with the world. AR journalism has the potential to create deep empathy for viewers by immersing them in these stories [1]. The Thai cave AR experience is just one example of how emerging technologies can connect audiences to journalism in new and exciting ways. Improvements to mobile phone technologies such as cameras, internal motion sensors (accelerometers and gyroscopes), and overall processing power make it easier for more users to experience AR content on their own devices. As these technologies continue to advance, there is greater potential for AR journalism to push the boundaries of immersive storytelling and reach a broader audience base.
Without an archival platform for AR journalism, journalists and researchers cannot learn from past examples and we cannot innovate based on past techniques.
Journalists who wish to explore the use of AR in their field could learn a lot from past examples of AR journalism. The ability to access similar stories helps inform future practice. Private news organizations, however, have not prioritized archiving these stories for the long term. For example, the New York Times has not maintained its extended reality content, rendering the AR portions of the Thai cave story inactive. This leaves the future of AR journalism dependent on fragments of user-generated material available via YouTube videos or screenshots. Without an archival platform for AR journalism, journalists and researchers cannot learn from past examples and we cannot innovate based on past techniques.
Why Should Journalists Care About AR?
As storytellers, journalists have a responsibility to share the complex stories of our world. Journalists have long faced difficulties in explaining complex issues to the public. Today, this difficulty is compounded by anti-science attitudes and the emergence of "fake news" culture.
In the past 10 years, journalism has seen a shift in nonfiction storytelling, moving beyond traditional media (e.g., text, images, video, and websites) to explore new and emerging technologies such as augmented and virtual reality. For example, journalists such as Nonny de la Peña, who is credited with helping create the genre of immersive journalism, have employed VR to put viewers in the story. These emerging technologies offer compelling ways to present stories that improve user engagement. Technological barriers, however, continue to limit participation and innovation in this field. The skills needed to build AR and VR stories (3D animation, modeling skills and software development, to name a few) are not typically taught in journalism programs, and such content usually takes much longer to produce than traditional forms of journalism. This skills gap is further challenged by the fact that the narrative affordances of AR are radically different from traditional narratives as a cognitive and communicative structure. In other words, journalists would not only need to acquire additional technological skills to develop stories in AR but also to build new mental models to understand how to leverage those advantages effectively in their stories.
Understanding the unique capabilities of each medium or platform helps us choose the best fit for the kind of story we are trying to tell. For example, when deciding whether to tell a story using AR, it is important to ask, "What is it about AR that will help me tell the story in ways that I cannot using other media?" In other words, we want to avoid choosing AR for AR's sake without properly leveraging its storytelling abilities. The importance of understanding storytelling affordances is explored in Ulrike Spierling's work on interactive narrative design [6]. Spierling describes the need to conceive the story's interaction model and narrative as two separate yet connected components. Here, the interaction model defines how inputs and outputs will be handled in interactive digital media. In the context of AR journalism, the interaction model determines how users will trigger AR content, and how they will interact with the story to experience some or all of its components. Spierling writes, "In this model design process, authors need to anticipate potential story states to occur during interaction" [6]. Interactive journalism takes a bold departure from linear journalistic storytelling by giving audiences agency in how they experience aspects of the story. Moving beyond interaction, AR gives storytellers the ability to annotate spaces and situate a story within their everyday lives. To effectively utilize the medium, journalists will need to design stories specifically for AR. The question then becomes how they can use AR to connect stories to real people, places, or objects in ways that are meaningful to their audiences.
De la Peña notes that "[a] possible objection to immersive journalism may be that it may strain the credibility of journalistic integrity, undermining the ability to bring the true facts to the public" [1]. Audiences are already critical of what they see in the media, and other forms of journalism (e.g., video, text, photography) are understood as mediated representations. The audience's relationship with AR storytelling, however, is still in its infancy. De la Peña argues that the value in immersive journalism is helping audiences experience the facts, rather than creating a more complete version of the story [1].
Journalists already operate under a core set of five ethical principles or tenets: truth and accuracy, independence, fairness and impartiality, humanity, and accountability [3]. These principles ensure that they represent the truth as accurately as possible and without conflict of interest. The design of AR stories—the typical workflows and asset-creation practices—changes in the context of these tenets [4]. Max Hao Lu et al. note that journalists are bound by ethical guidelines that govern how little they are allowed to edit visual content to be presented in their stories [4]. These guidelines were developed with photojournalism and video in mind. AR content, however, typically consists of 2D or 3D assets. It is considered standard practice for artists creating AR content to have stylistic control over the size, color, and texture of 3D objects. Furthermore, to ensure AR experiences work on as many devices as possible, 3D artists will often optimize the geometry and textures of objects to reduce their size in order to improve the performance of the AR application. While these types of artistic liberties are considered common practice in AR content design and are done purely for technological and aesthetic purposes, they could potentially clash with the guidelines governing visual representation of news content in AR and VR [4].
In sum, the problems noted above include the need for a robust archive of AR journalism projects, the need to democratize or reduce technological barriers to the design of AR journalism projects, and the need to create workflows and best practices that support the design of immersive journalism—ones that are informed by the tenets of journalism. I argue that such problems are the kinds of issues the HCI community is well equipped to address. In the following section, I propose how we can mobilize to support journalists in these endeavors.
Why Should HCI Care About Journalists?
Human-computer interaction is a multidisciplinary field that is largely concerned with the study and design of interfaces and, more importantly, with studying the interactions that occur between humans and computers. HCI represents the intersection of several fields, including but not limited to psychology, sociology, information systems, communications, and computer science. Prior to the 1970s, computers were largely used by computer engineers and scientists. Interaction design was not prioritized as a research paradigm, since those using the computers were likely the same people who would have written the user manuals. Thus, the field of HCI is said to have emerged to better understand how interface design could be studied to accommodate a more diverse user population.
HCI researchers are currently supporting journalism in a variety of ways [2,5]. For example, Saniat Javid Sohrawardi et al. conducted a design role-playing exercise with journalists to better understand how they rely on digital tools to detect deepfakes [2]. While large media organizations have protocols in place to verify the authenticity of online content, the emergence of deepfakes has resulted in journalists heavily relying on detection tools, the authors note. They argue for a more cautious release of such tools and proper training to reduce the risks inherent in the current framework, particularly with regard to false positives and false negatives [2]. Sachita Nishal et al. conducted a user-centered study with science journalists to better understand their use of information subsidies [5]. The task of news discovery is a crucial and complicated process, requiring careful monitoring of scientific journals, preprint servers, conferences, and corporate blogs. Researchers explored the design of algorithmically generated subsidies to identify the design needs of this unique user group. Results of the study highlighted the need for user agency, flexible interfaces, and the role of generative models in such tools [5].
Due to the aforementioned technological barriers, the implementation of AR journalism is still firmly in the domain of technologists. The HCI community, however, boasts several decades of advancements in technology development, research methodologies, and increasingly inclusive collaborative methodologies, such as codesign. It is a collaborative methodology that engages stakeholders at all stages of design and employs qualitative approaches, such as participant observation and cognitive walk-throughs to evaluate how effectively each iteration of the prototypes, tools, and frameworks meet the stakeholders' unique needs. This paradigm exists in opposition to the more traditional top-down cycle of interface design, which typically excludes stakeholders until the final stage of usability evaluation.
Codesign approaches with journalists as stakeholders can support the advancement of journalism in AR. A collaborative approach to the development of an archive of past works would provide a meaningful reference to document the creative process and help practitioners understand how each story leveraged the assets (e.g., 3D objects, 2D objects, audio files, etc.), environment, user interactions, narrative structures, and hardware (e.g., was the story designed for mobile devices, VR headsets, etc.). The archive could host files or viewers, allowing users to experience the story for themselves. For older projects that can no longer be supported, a collection of audiovisual and textual documentation could be accessed for historical purposes. This repository would be structured in a way that allows for the analytical study of different projects and serves as a collective memory to provide a reference point for future innovation.
Journalists have much to contribute to the design and development of the digital tools they use in their professional practice. Technological barriers introduced by the complexity of tools currently used to develop and publish AR, such as game development platforms like Unity, make it almost impossible for journalists to create such stories on their own. The alternative—outsourcing the development of AR journalism to technologists and excluding journalists from the design process—represents a missed opportunity for journalists to leverage the abilities of immersive technologies in ways that reflect the narrative techniques specific to their trade. This reduces authenticity and potentially negatively affects the power of the narrative and consequently consumption. Here, codesign could also be leveraged to develop new tools and plug-ins to bridge this technological gap and empower journalists to design AR stories themselves.
Conclusion
HCI offers unique, interdisciplinary perspectives to practitioners engaging with technology and these perspectives can be very useful in empowering journalists to define AR journalism in their own terms.
There is currently no all-encompassing archive of AR journalism, and the stories that have been catalogued by their publishers are inconsistently documented. Past examples of AR journalism have a great deal to offer to the development of ethical guidelines and best practices that will inform the development of AR stories in the future. In the absence of an archive, innovation will come slowly, and missteps will be repeated frequently. AR may not be suitable for all stories, but its unique ability to add layers of context to people, places, and objects makes it an exciting platform to leverage in the field of journalism. I see great potential for AR to revolutionize journalism, and for journalism to identify new and exciting ways to push the boundaries of storytelling in immersive media.
Before this can happen, however, significant technological barriers must be overcome. These barriers are not limited to ease of use—usability must be addressed in the context of the tenets of journalism and ethical principles that govern visual representation in journalistic stories. I argue that, through collaborative approaches, HCI researchers are well suited to support the design of a meaningful archive of past works of AR journalism and to support the design and development of new tools that help journalists design and develop AR stories. The future of journalism lies at the intersection of storytelling and technology, and it is imperative that we bridge this gap to democratize immersive storytelling and foster deeper connections with audiences worldwide.
References
1. De la Peña, N. et al. Immersive journalism: Immersive virtual reality for the first-person experience of news. Presence: Teleoperators and Virtual Environments 19, 4 (2010), 291–301.
2. Sohrawardi, S.J., Wu, Y.K., Hickerson, A., and Wright, M. Dungeons & deepfakes: Using scenario-based role-play to study journalists' behavior towards using AI-based verification tools for video content. Proc. of the CHI Conference on Human Factors in Computing Systems. ACM, New York, 2024, Article 776, 1–17.
3. Ljajić, S. Media, ethical norms and media literacy education. Facta Universitatis. Teaching, Learning and Teacher Education 4, 2 (2020), 185–94.
4. Lu, M.H., Calandra, D., and Thawonmas, R. Artificial reality: Immersive but factually dishonest AR experience. IEEE Consumer Electronics Magazine 12, 1 (2023), 8–11.
5. Nishal, S., Sinchai, J., and Diakopoulos, N. Understanding practices around computational news discovery tools in the domain of science journalism. Proc. of the ACM on Human-Computer Interaction 8, CSCW1 (2024), Article 142, 1–36.
6. Spierling, U. Interaction design principles as narrative techniques for interactive digital storytelling. In Interactive Digital Narrative: History, Theory and Practice, H. Koenitz, G. Ferri, M. Haahr, D. Sezen, and T.I. Sezen, eds. Routledge, New York, 2015, 159–73.
Author
Victoria McArthur is a digital artist (AR/VR) and HCI researcher in the School of Journalism and Communication at Carleton University in Ottawa, Canada. She is cofounder of MARVEL, a research lab supporting the design and development of new AR and VR technologies. victoria. [email protected]
Copyright is held by the owner/author. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2024 ACM, Inc.
Post Comment
No Comments Found