XXV.4 July - August 2018
Page: 66
Digital Citation

Evaluating an AR-based museum experience

Andrew Hunsucker, Emily Baumgartner, Kelly McClinton

back to top 

Augmented reality (AR) is an emerging technology that could soon become an established paradigm in computing. Although it incorporates hand gestures and voice commands into the user’s experience, it bypasses traditional user command tools such as a touchscreen, mouse, or keyboard.

back to top  Insights


This paradigm creates challenges not only for interaction designers, who must invent new ways for users to interact intuitively with these devices, but also for researchers who have to evaluate the experiences. Because AR occurs within the real world and its experiences are privileged to a single user, researchers will have a difficult time correlating their external observations to what the user is seeing. In this article, we explore the lessons learned throughout the eight weeks of usability testing of a HoloLens in a museum context, both what we learned about the challenges of conducting this research in the face of the environmental constraints in the museum, and how we made sense of the user’s reactions in order to improve our design. Interestingly, the problems we faced as researchers provided insights into problems designers will have to solve to make AR viable in the real world.

back to top  The Project

Our exhibit of the Resting Satyr statue was featured in the center of the Eskenazi Art Museum’s gallery room, surrounded by an assortment of other artifacts displayed in glass showcases and on the walls. The original Resting Satyr, an ancient Greek work likely by sculptor Praxiteles, has a large number of replicas compared with other statues from its period. The exhibit’s objective was to enable users to compare the physical Resting Satyr statue alongside some of its copies, which are spread all over the world. To do this, we leveraged AR to display virtual copies, which were viewable through the HoloLens in the form of 3D models, allowing users to compare the physical artifact to its virtual copies. We studied this interaction in a museum context. This method of comparing a physical object to digital representations of similar ones is an interaction with applications in many other contexts. Examples include zoos, where patrons could view animals that they normally would not be able to get close to, and the medical field, where a doctor could compare a patient’s visible symptoms with the symptoms of a virtual patient.

Our approach to harmonizing the museum’s artistic and historic presentation with emerging technology took great care in addressing the variety of novel predicaments woven throughout our design process.

back to top  Challenges

Although our team was aligned on the strategy and goals, the usability testing proved to be the most difficult part of the project. The luxury of conducting our tests in a lab environment was not a realistic option. There were several reasons we needed to test in the actual gallery with the physical piece of art. First, we needed to allow the user to compare the real and virtual statues; a stand-in object, like a box, would not suffice. Second, asking museum staff to relocate a priceless artifact to a separate location for testing would have been unreasonable. As usual, there were also benefits to doing our testing in the context of use. For example, we did not receive immunity from the museum’s protocol for its patrons; the guards kept a close eye, ready to interrupt our tests in the event that our usability participants’ gestures threatened the artifacts. Of course, this reflects a user experience issue in its own right.

As with most emerging technologies like the HoloLens, we faced a number of technical limitations that created experiential complications and distractions. For example, while wearing the headset, one’s field of view is noticeably diminished, with the available view constrained to the center and omitting the peripheral view. This particular technical limitation makes it easy to overlook parts of the experience that exist outside the central view. Because peripheral vision is nonexistent in virtual space, we discovered users often missing important details during the experience—such as the moment when virtual statues would be replaced with new versions and which buttons were being activated—simply because they had not pivoted their head. This limitation was noticed on both a design level and a user level; complaints about the lack of peripheral vision became our most consistent piece of feedback. We attempted to apply traditional screen-based methods like clearly labeled buttons, animations, and action-based colors, but these were commonly missed by our users. What we learned (and a problem we didn’t solve during our testing) was that designers will have to develop new ways of drawing attention in AR.

ins02.gif A user’s view of our AR exhibit through a HoloLens. Pictured is a physical Resting Satyr statue positioned next to two digital representations, as well as supporting interactive tools (i.e., audio, podiums).

Regarding research methodology and tactics, one unexpected challenge was that initially, we could not see what the user was seeing through the headset. This was a major departure from traditional usability testing, which has myriad screen-sharing and recording software available. However, we did eventually learn of a Microsoft app that allowed us to record the user view during the test. Unfortunately, this recording degraded the quality of the experience, and as the app got more complex and more features were added, we simply stopped using the recording feature. By the time we stopped using it, however, everyone on the team knew exactly what the experience looked like and could predict what the user was seeing based on where they were positioned. This suggested to us that, more than ever, researchers will need to carefully observe the actions of their users, and have a strong grasp on what the user is seeing as they move to different parts of the experience. Conducting the research with this awareness will provide a helpful guide for thoughtful follow-up questions and mitigate any confusion during the course of the study.

Because peripheral vision is nonexistent in virtual space, users often missed important details during the experience.

Our process was simple. Each week we would perform testing, usually with some specific goal in mind related to questions to which we wanted answers. We would then create a report of user reactions, bug fixes, and feature requests that we would give to our art history expert and developer in preparation for the next week. Each round of testing offered several new insights; its influence and ability to shape the design was vital to our process.

back to top  Iterating the Design Through Usability Testing

Although augmented reality is situated in the real world, its technology can still create an overlay that distracts from the real world. Much like smartphones, this technology could cause accidents or cause the user to unintentionally ignore actual hazards. Our testing took place in the center of the gallery, with no ability to make changes to the existing layout. We didn’t know initially whether users would cautiously walk around the holograms as if they were real objects or carelessly walk through them. Because of the safety issues, we enforced various protocols designed to prevent harmful participant behavior, such as warning users of the dangers in advance and disrupting the AR experience with warnings if participants wandered too close to a showcase. However, we quickly learned that the majority of participants responded to the artificial statues with authentic caution. This inhibition had a major impact on their experience. We noted in our first tests that participants weren’t willing to view the exhibit from the rear because they felt the virtual statues created a barrier. While this suggests how compelling the AR experience was, designers will have to consider what behavior to foster and what behavior to discourage based upon research insights and observations, and how to do so without degrading the experience.

To showcase more statues, we decided to place two statues into a single position, adding a button to toggle between them, thus adding walking space around the exhibit. In order to achieve this, we had to decide where to place the button in the augmented space. We approached the problem by following accepted usability guidelines for “on-screen” experience [1], since AR practices are still developing. We grouped the button with the object it controlled, initially placing it on the far left of the exhibit. However, because of the HoloLens’s narrow field of view, participants were unlikely to see the button at all. We then attempted to place the button in the center of the exhibit for better visibility, but in that case, activating the button did not guarantee that participants would notice the statue change. We also had issues with the look and feel of the button. No matter what call to action was placed on the button, users were not sure what type of interaction was required.

This revealed a general lack of knowledge about AR from our participants; they simply did not have the comfort level to experiment in AR. With a smartphone, users by now know the repertoire of possible actions (i.e., tap, swipe, drag, hold) and will attempt them when uncertain. In contrast, in the AR experience, users don’t have the interactive memory (i.e., they aren’t familiar with the types of interactions that are possible in AR) to support such experimentation. In addition, the fact that the AR experience happens in the real world increases the degrees of freedom designers have about where to place interactive controls. This lack of natural constraints is challenging for both designer and user. Designers will have to create new onboarding tools to help users acclimate to this new interactive medium, complemented by new interactive norms in the AR space.

In studying how users compared virtual and physical objects, we learned a few lessons as well. For instance, the virtual statues often overpowered the physical statues in the darkly lit gallery. This privileged the virtual statues, leading to greater interest from users. In addition, while the art historians we tested invariably recognized the significance of the exhibit, those with less knowledge were unable to relate the physical statue to its virtual replicas. Helping patrons understand the artistic significance of an exhibit continues to be a challenge for museums, and AR as a technology cannot solve this problem alone. It can, however, give museum curators a new tool to develop thought-provoking exhibits.

back to top  Discussion

Augmented reality is an emerging technology and possibly a new paradigm in computing. Understanding user expectations and the perceived value of AR technology will be essential for the future. Throughout this project, we have experienced the challenges of usability testing in AR. For example, while many usability studies are performed in a controlled lab setting, we feel it is ideal to test AR experiences in the real setting. Because AR experiences are embodied, it is critical to understand how a user will interact with the space around them. Is there enough space? Will the 3D models you’re working with contrast with the background? If you’re in a public space, will the interactions you’ve designed interfere with other people’s use of the space?

For AR designers and researchers, the biggest challenge to overcome is the user’s discomfort with the technology. When smartphones appeared, they had a history of similar devices that helped users acclimate. Before smartphones, there were flip phones, and touch screens, and music players. The smartphone worked because it combined many different technologies into something familiar yet revolutionary. It is possible AR will follow the same incremental path. Our work simply scratched the surface of user experience in AR and how designers will be able to engage their users in this new space. But it is clear that people immersed in screen-based technology will need to be carefully guided into this form of computing.

We see usability testing as an ideal way to understand user reactions to this medium. Until AR becomes a ubiquitous technology, it will continue to be new and strange to many users. Through usability testing, designers can gain an understanding of how interaction methods and experiences will work for different user groups. Engineers and developers will continue to make the technology smaller, more efficient, and more powerful. But it is up to the designers and researchers to make it user-centered.

back to top  References

1. Krug, S. Don’t Make Me Think, Revisited: A Common Sense Approach to Web Usability. New Riders, 2014.

back to top  Authors

Andrew J. Hunsucker is a Ph.D. candidate at Indiana University studying design communication and augmented reality. He has previously published at CHI and has written for XRDS magazine. He expects to graduate in May 2019.

Emily Baumgartner gathers and translates user needs by designing and executing meaningful research methods at innovatemap, a digital product agency. She studied communication and returned to school to earn her M.S. in HCI from Indiana University which has helped her truly understand the user experience and champion the design of strategic, useful products.

Kelly Elizabeth McClinton focuses on the application of computational methods to material culture studies. This research has extended during her time at Indiana University to the ancient collection in the Eskenaz Museum of Art, where she has developed digital exhibitions with the acting ancient curator. Her current research focuses on Roman domestic spaces and the analysis of visual programs within the houses of Pompeii.

back to top 

©2018 ACM  1072-5520/18/07  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.

Post Comment

@Mark Albin (2012 06 30)

This is a very interesting article about social bots, thanks for sharing.

@Aman Anderson (2012 07 18)

This is great
“So what’s the center of a design? In one sense, it is the designer’s nuanced understanding of the problem or opportunity at hand. The focus of design is problem solving, not self-expression.” - Uday Gajendar, Interaction Designer

@Bill Killam (2012 07 31)

This is a long overdue article.  And I couldn’t agree with it more.  I’m current working on yet another Federal RFP that is asking for us to do work using short cut methods that are likely make it harder to get them quality results, and we can probably propose a cheaper and more data rich approach if they didn’t specify how we had to do the job.  Sad.

@Demosthenes Leonard Zelig (2012 08 12)

Great Article, it is funny to notice that such huge corporations do not even bother to do a market research before releasing products on a new market. However, I guess we are still learning from our mistakes. (2012 10 24)

Hi everyone, In the Technological University of Panama there is also a movement. There is a 2 years MS in IT with a specializtation in HCI. We are also trying to include HCI as part of our main curricula. This year we started a research with a company interested on incorporating usability in their development. We expect to receive a Fulbright Scholar next year in this area…


Karla Arosemena

@John Michael Sheehan (2012 11 06)

There are thousands of blogs that requires comments on them. What is the intention of blog comments? Sent From Blackberry.

@Junia Anacleto (2012 11 07)

A very shallow and naive view of a much more rich and complex context.
I am still waiting for a fair position paper to be presented.

@Rick Norton (2012 11 17)

Excellent article raising significant issues that are largely overlooked.  The prospect that the collapse of sustainability for a growth/consumption related societal model is inevitable, is a topic I have often wondered about, given the nature of capitalism as we know it today.  Even the “Great Recession” of current times gives me pause to wonder just how long we can keep this economic engine going before we have to face the reality that we are all going to have to learn to “live with less”.  (A quantitative assessment, not necessarily qualitative.)

Keep up the good work.  Hopefully, you will raise awareness of these topics.

@Noah McNeely (2012 11 27)

Very nice article, that raises meaningful questions.  I actually think that the idea of sustainable products and sustainable product development is a bit of a myth.  All products consume energy and other resources in one form or another during their production, use, or re-use.  The key, ultimately is to balance resource consumption with resource production, but we will always need to be producing new resources.  See my blog post on the subject at ( ) (2012 11 30)

The quote in the article mis-contextualize James Landay ‘s essay. James actually is actively working to break down those stereotypes, but you can’t do that without understanding what the deep problems are.

James’ blog post on this is at

@Lee Crane (2012 12 03)

This is a topic that is thought provoking and important.  The message explores how humans can escape and survive the world they have jumbled.  So many of the theories and ideas are basic.  Our future may look a lot like the distant past.  And indeed we may be happier for it.

@ 4996484 (2012 12 19)

this is a great article David and Silvia!  I’‘m so excited that you guys wrote this up and are showing everyone the complexities in this space. I hope Interactions features more of this kind of research on China.  Although I agree w/ @landay’s assessment of China’s creativity problem - but he’s working with a very different population than you guys. I think you research is absolutely on point - creative folks are going to hacker spaces like Xinchejian, they aren’t ending up in institutions like Tsinghua!  I explain more here:

@Joe (2013 01 04)

I think that if you study the Elliot Wave Theory it can answer your questions.

@Rafeeque (2013 01 06)

good one

@zhai (2013 01 17)

Enjoyed reading this article. I finally got why Harold wants to call it “the Fitts law”. If enough people write it that way I would never have to correct another submission making the embarrassing mistake of ‘Fitt’s law”.

I did not completely get the following remark though:

        “The Accot and Zhai paper about the Fitts Law [3] has a clever title that illustrates
        the rules on letters, “More than dotting the i’s…”—a bad pun on eyes.”

I came up with the title, but the word “eyes” never came to my mind. We meant that the point-and-click style of UI is like dotting the i’s everywhere—- placing a click on constrained targets as the fundamental action in interaction. Why not using ” Crossing the t’s ”  as an alternative action?  Indeed, we presented models of a new style of UI, which systematically reveals when crossing is superior to clicking,  hence the subtitle of the paper “Foundations for crossing-based interfaces.”

Shumin Zhai

@Mohamadou M. Amar (2013 03 22)

I am a Doctoral student in I/O Psychology with Touro UW and need to access your articles.

@Mohamadou Amar (2013 03 22)

Need access for Doctoral Research

@William Hudson (2013 04 09)

Gilbert overlooks the important issue that the ‘big boys’ largely do not appreciate the need for design all and the problems that real people have with technology. I admit that we’ve had a hard time selling UCD but I am not persuaded by the arguments here to abandon it. Perhaps have a look at my article on a similar subject - User Requirements for the 21st Century - where I take a more pragmatic view of trying to address real users’ needs in the development process.

@ 0343665 (2013 04 29)

Fantastic text. I came here by searching for people that quote the Standford study on multitasking. The introduction is fantastic as it builds up an argument that attention has some features that do not change over time.

@Simon Taylor (2013 04 30)

not wanting to do anything so grandiose as building a (technology for) a world parliament, I have in essence been working on the same problems and facing the seven challenges with a project called ‘company.’ []

In 2011, working with senior software developers - gratis - although neither the ethical undertaking nor the promise of sweat equity were enough to keep them involved - I established the technical feasibility of ‘company.’
In 2012, turning from the ‘voluntary’ ‘principled’ participation model - because the attractions of real paying jobs had lost me my team - I received financial support from the New Zealand government. This part-funded an Intellectual Property Position Review - which government considered a pre-requisite - as commercial due diligence - to investing in an initial build, or beta. The IPPR recommended I do proceed… However, government offers only part-funding and without a team - either technical or commercial - there has been little to no investor interest.

As things stand at present, I have the tools and schematics for a beta build of something which would fit the sort of use imagined here. If you have any interest in helping, please contact me.

Simon Taylor