ACM Interactions

Authors: Garreth Tigwell, Zhanna Sarsenbayeva, Benjamin Gorman, David Flatla, Jorge Goncalves, Yeliz Yesilada, Jacob Wobbrock
Posted: Tue, October 06, 2020 - 1:29:59

---

Mobile devices are our constant companions. We use them in varied contexts and situations, such as outside on a cold street, lying down in a dark bedroom, and commuting to work on a train. These situations challenge our ability to use mobile devices and can negatively influence our interactions with them. For example, we slow down or make more errors when typing, or select a wrong button. These adverse contextual factors have been referred to as situationally induced impairments and disabilities (SIIDs), or sometimes just situational impairments for short.

The experience of SIIDs during mobile interaction applies to users of all abilities, as SIIDs adversely affect both non-disabled and disabled user groups [1]. But SIIDs have been shown to exacerbate negative user experiences in mobile interaction for people with disabilities [2]. Thus, by accommodating SIIDs during mobile interaction, we can provide solutions to improve user experience for people with and without disabilities [1]. 

Research in the field of SIIDs is conducted in four main areas: Understanding (e.g., [3]), sensing (e.g., [4]), modeling (e.g., [5]), and adapting (e.g., [5]). Understanding provides knowledge about the effects of SIIDs on mobile interaction; sensing allows building mechanisms to detect the presence and extent of SIIDs; while modeling and adapting enable the creation of models and interfaces to accommodate SIIDs. The main criticism about research conducted on SIIDs is the lack of systematic knowledge of the effects of underexplored SIIDs and the combination of different SIIDs on mobile interaction. Furthermore, there is an absence of built-in sensing, modeling, and adapting mechanisms on conventional smartphones to detect and accommodate these SIIDs. Therefore, the research community should strive to push the SIIDs research agenda further in order to build a roadmap for future work in the field.

Figure 1. Workshop organizers and attendees enjoying a meal after an engaging full-day workshop.

Workshop purpose, structure, and goals

We organized a one-day workshop for CHI 2019 in Glasgow, Scotland, called Addressing the Challenges of Situationally-Induced Impairments and Disabilities in Mobile Interaction. The purpose of the workshop was to assemble researchers whose work is related to SIIDs so that we could identify current research gaps and define new directions for future research. The workshop included five main events throughout the day: “lightning” presentations, focus groups, a panel discussion, exploring scenarios, and a town hall meeting, followed by an evening meal (Figure 1). A detailed structure is found in our workshop proposal paper [2].

The four goals of our workshop were to: 1) provide a space for organizers and participants to share their expertise and insights on SIIDs, 2) engage attendees to discuss and identify gaps in the SIIDs research space, 3) ideate new solutions that could mitigate the effects of SIIDs, and 4) create and strengthen an international collaborative SIIDs research network.

The workshop included 17 participants (including organizers), collectively representing universities in eight countries across four continents. Our workshop call for submission was purposely made to be broad to allow participants to submit whatever they felt was relevant. For example, submissions could be in the form of position papers, case studies, empirical studies, or new interaction methods. Submissions were limited to eight pages (including references) and we used arXiv to keep a record of the accepted papers (the proceedings can be found at https://arxiv.org/html/1904.05382). Our participants were each given four minutes to discuss their work during the lightning presentations event (Figure 2) so that we could dedicate as much time as possible to identifying necessary future work.

Figure 2. A workshop attendee giving a short presentation about their work.

Activity 1: Focus group activity

Our attendees were divided into groups and given 1.5 hours for the first activity:

• Task: The groups reflected on the lightning presentations and deliberated on challenges they have faced when conducting SIIDs research.
• Outcome: The groups highlighted key parts of their discussion to share with the other workshop groups.

A list of example questions were provided to drive the discussion related to research methods and equipment, recruitment, ethical challenges for SIIDs studies, challenges utilizing sensor data and modeling SIIDs, and the limitations of adaptation. Figure 3 provides an example of one group’s record of challenges that they identified.

Figure 3. An example of challenges discussed by one of the groups.

Recruitment can be a challenge for SIIDs research, particularly when seeking to run longitudinal and ecologically valid studies—the costs incurred can be higher. Lab studies are beneficial for isolating factors to study, but more work needs to be conducted outside to increase the ecological validity of results. There should be particular consideration given to active observation approaches rather than only passive observation. Guided tours might be one way to mitigate some of the challenges of in-the-wild research and to address IRB concerns for participant safety. People with permanent disabilities should also be recruited, since they can also experience situational impairments and need very specific solutions to address SIIDs.

Modeling and adaptation are promising solutions to addressing SIIDs. Artificial intelligence and machine learning research could support the efforts of HCI researchers, but there is a lack of ground-truth data. It is important to parameterize the environment to help understand the relevant context; it may be possible to do this in an unobtrusive way such as by using smartwatches for sensing. However, although logging data can help address SIIDs, it may cause people to become uncomfortable, as devices become more aware of context, which highlights various legal, security, and privacy challenges. There was also concern that through addressing SIIDs, people’s own skill development might diminish, as devices work more independently and people may be encouraged to interact at times when they should be focusing on their environment (e.g., while driving).

Activity 2: Exploring scenarios

Our attendees were divided into groups and given 1.25 hours for the second activity:

• Task: The groups explored random scenarios and focused on the gaps in understanding, sensing, modeling, and adapting, in regards to the context, technology, and task.
• Outcome: The groups recorded the unique issues identified from the card game and shared the findings during the town hall meeting.

Figure 4. Example scenarios created by taking one card from three piles (a situation, a piece of technology, a task).

We gave each group a set of prompt cards to facilitate this activity. The cards covered three categories (a situation, a piece of technology, a task). We had at least 10 ideas for each category. The cards were laid out in the three categories, stacked in a random order, and the groups drew a card from the top of each pile so that the three cards made up a scenario (see Figure 4). The group could draw three cards that propose something common and relatable, such as “jogging,” “using phone,” and “reply to a message,” or something less familiar, such as “outside in the rain,” “wearing an AR headset,” and “call an Uber.” Sometimes a combination of cards was drawn that suggested an unlikely scenario, but the purpose was to quickly generate unique scenarios that the group could use to identify challenges and possibly where new research efforts need to be focused. We added wildcards to each pile so the group could determine their own entry (e.g., a technology wildcard would allow the group to invent some future mobile device to add to the scenario). The benefit of this approach was that the groups were less constrained to the ideas written on our cards.

Here are three examples generated by the participants and the issues that were considered:

• On a beach under an umbrella using a tablet and needing to unlock the device. The group considers that a person may have sand on their hands, and depending on how they unlock their device, this might be a challenge. For example, if using a fingerprint to unlock and it does not work due to sand, what other unlock methods are there? This highlights the need for various fallback methods to address SIIDs. But also the user may want control of what those fallback mechanisms look like—maybe the user does not want to compromise on biometric unlocking? Perhaps a user’s voice signature could be used for unlocking?


• Running in a park and receiving an email response on a smartwatch. Focusing on the email will distract the user from running and their environment. It could be difficult to read a long message and to respond to the email, which means more focus has to be given to the task. A user may have privacy concerns for reading in public depending on the content of the email. There may also be network inconsistencies while outside. The technology should aim to sense the level of danger and possibly delay notification if it is not an urgent task. Perhaps the smartwatch could detect that the user is running and defer its interruption?


• At the airport sending a message using a laptop. A person traveling can be stressed, tired, and likely distracted as they are listening for important flight information. The user is likely concerned about conserving their battery and dealing with limited network connectivity. Here, it is not only environmental factors that lead to SIIDs, but also internal factors. For example, the concept of emotions is purely situational; it is unclear if emotions directly influence the way we interact with mobile devices. In this scenario, the device could detect that the user is in an airport and make the changes necessary to reflect the current mood of the user. Perhaps future laptops could detect users’ stress levels and avoid contributing to information overload?

It is clear we need to be accurate and flexible with SIIDs solutions. There is little room for error when sensing the environment for potential dangers and determining the best method for interaction. Furthermore, an individual user will have their own needs that must be met and these are not likely to be static over time or for particular tasks. For example, some emails are urgent and others are not. There needs to be an easy way for the user to make these aspects known to the device to enable alternative modes of interaction.

Should we always adapt/accommodate?

Currently, most research in the area of SIIDs has been conducted within the laboratory environment. A laboratory environment limits our understanding of SIIDs, as it strictly controls and excludes the effects of accompanying factors that might be present in a real-world scenario. For this reason, we suggest that more research should be conducted in the wild. It is also necessary to study the effects of combined SIIDs, as it is common for a user to experience the effects of multiple SIIDs at once, for example being outside in a cold and noisy environment late at night. We argue that these future directions would increase our understanding of SIIDs and create new insights, potentially revealing new behaviors of people observed under realistic conditions with multiple SIIDs present.

We also suggest further understanding and investigation of the effects of SIIDs on mobile interaction according to the 2D space presented by Wobbrock et al. [1]: from-within (emotions, mood, mental well-being), from-without (difficult terrain, lack of connectivity/power), and mixed (combination of external and internal) factors. It is pivotal for the research agenda to understand the effects of SIIDs from this perspective in order to progress further by building sensing, modeling, and adapting mechanisms for these SIIDs. Furthermore, if the similarity between the effects of underexplored SIIDs and permanent impairments is established, it can further enable the creation of design solutions to accommodate users of all abilities (i.e., permanently, periodically, or situationally impaired). 

Moreover, research has shown that the personal and individual characteristics of users are very important when building sensing, modeling, and adapting mechanisms to address SIIDs. The challenges of building individual models and adaptive interfaces [1] can be overcome by optimization algorithms and formulating cost functions [2]. These should be sensitive to any of the user’s privacy and security concerns. In addition, it is also important for these mechanisms to provide adequate judgements of SIIDs to decide if the adaptation should take place at all, especially in high-risk, high-cost situations, when visual and attentional needs should be focused on tasks of higher priority (e.g., crossing a busy road).

Finally, we suggest that the research should expand to include a wider range of devices [6], for example smart watches, fitness and activity trackers, and other less common wearable technology such as AR glasses, which we foresee becoming the new norm. Considering potential SIIDs when designing new technology is important in order to build in necessary solutions from the outset, rather than after the technology is adopted by the user. Our CHI 2019 workshop highlighted these (and many other) issues that the attendees and, we hope, many other researchers will undertake, creating more aware, responsive, accessible, and safe mobile technologies that are usable by everyone.

Endnotes

1. Wobbrock, J.O., Gajos, K.Z., Kane, S.K., and Vanderheiden, G.C. Ability-based design. Communications of the ACM 61, 6 (2018), 62–71.

2. Tigwell, G.W., Sarsenbayeva, Z., Gorman, B.M., Flatla, D.R., Goncalves, J., Yesilada, Y., and Wobbrock, J.O. Addressing the challenges of situationally-induced impairments and disabilities in mobile interaction. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, New York, 2019, 1–8.

3. Sarsenbayeva, Z., van Berkel, N., Hettiachchi, D., Jiang, W., Dingler, T., Velloso, E., Kostakos, V., and Goncalves, J. Measuring the effects of stress on mobile interaction. Proc. of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 1 (2019), 1–18.

4. Goel, M., Findlater, L., and Wobbrock, J. WalkType: Using accelerometer data to accommodate situational impairments in mobile touch screen text entry. Proc. of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, 2012, 2687–2696.

5. Gajos, K.Z., Weld, D.S., and Wobbrock, J.O., Automatically generating personalized user interfaces with Supple. Artificial Intelligence 174, 12-13 (2010), 910–950.

6. Akpinar, E., Yeşİlada, Y., and Temİzer, S. The effect of context on small screen and wearable device users’ performance - A systematic teview. ACM Comput. Surv. 53, 3 (Jun. 2020), Article 52; https://doi.org/10.1145/3386370

---

Posted in: on Tue, October 06, 2020 - 1:29:59

Garreth Tigwell

Zhanna Sarsenbayeva

Benjamin Gorman

David Flatla

Jorge Goncalves

Yeliz Yesilada

Jacob Wobbrock