Alex Mihailidis, Jennifer Boger, Marcelle Canido, Jesse Hoey
Dementia impairs people’s ability to perform routine activities: They cannot remember the proper sequence of steps or how to use the necessary tools. Strategies commonly used by caregivers involve continually providing reminders or cues. Family caregivers find assisting their loved ones to be particularly upsetting and embarrassing, as it necessitates invasion of privacy and role reversal. This difficult situation often results in the family caregiver not being able to cope, and the affected person being placed in a care facility.
In response to the unique needs of older adults with dementia, we have been developing a new prompting device that uses artificial intelligence (AI) to automatically monitor an older adult during a common self-care activity (i.e., hand washing) and provide prompts as needed.
This article covers the most recently tested prototype of a new prompting device, named the COACH (Cognitive Orthosis for Assisting aCtivities in the Home). COACH was developed to guide a user while washing their hands. The device had four integrated systems: the tracking, state monitoring, Markov decision process (MDP) policy, and prompting modules, as represented in Figure 1.
The tracking system identified the position of the user’s hands and towel by applying background subtraction and color matching on images acquired through a video camera mounted over the sink. These techniques were used to isolate skin-toned objects (i.e., the person’s hands) and to determine the two-dimensional coordinates of the user’s hands. Figure 2 illustrates both the original and tracked images. These coordinates were then passed from the tracking system  to the state monitor, which determined which hand-washing step the person was completing.
An MDP model  is one type of an AI technique that can be used to develop a planning/decision-making system. It makes continual observations and decisions based on the state of the world, in this application from observations collected by the tracking module. Using this information, the model can be solved to determine specific actions the system should take for each state it could find itself in.
Six hand-washing steps were identified (Figure 3), with several different step paths that were considered for successful hand washing. Using these different paths and various variables (e.g., user’s hand position, water-flow status, last successfully completed step, and the last prompt given to the user), the possible MDP actions that resulted were either a prompt being given if (and only if) the user became "stuck" at a step, and/or the caregiver being called if the user did not respond to the most specific prompt or regressed in the task too many times. A detailed description of the specific MDP developed can be found in issue 10 of IEEE Transactions on Information Technology in Biomedicine .
The prompting module had 20 available actions: 18 were different prompts to the user to attempt one of the six steps (in three levels of specificity); the other modules were to either observe or call the caregiver to intervene. Levels of prompting specificity were minimal (e.g., "Use the towel"), moderate (e.g., "Use the blue towel"), and specific (e.g., "John, use the blue towel on your right"). The system was designed to give the lowest level of detail possible while still having the user complete the activity, thus encouraging the user to be as cognitively involved in the activity and as independent from a caregiver as possible.
Clinical (Efficacy) Study
An efficacy study was completed in a retrofitted test washroom to collect data on participant and system performance. Participants were recruited from a Toronto-based long-term-care facility. The primary inclusion criterion was clinical diagnosis of moderate to severe dementia (from our experiences this group requires and benefits the most from cueing during routine activity completion).
A modified withdrawal-type single-subject research design was used to test the device. This research design consisted of a baseline phase (no computer guidance), A, and an intervention phase (with computer guidance), B, tested in the order A1-B1-A2-B2-A3.
Case Study"Mrs. Smith". Mrs. Smith was an 84-year-old female resident in the long-term-care facility. She had a diagnosis of severe Alzheimer’s diseasehaving a Mini-Mental State Examination (MMSE) score of 1 out of 30 (a score of 12 and below is classified as severely impaired). She had good hearing in both ears and was fluent in English.
Mrs. Smith required constant reminders and prompting from the caregiver and/or device throughout all trials. She was highly dependent on the use of detailed prompts, including the caregiver physically completing the task for her, with the most assistance being required during the "use the soap" step. In fact, 35 percent of all prompts provided by the system were for this particular step. Furthermore, it was observed that she was easily distracted from the hand-washing task, often completing erroneous tasks such as wiping the sink with the towel.
For each test phase, two primary-target behaviors were observed: 1) the number of hand-washing steps that the participant completed without any interaction with a human caregiver; and 2) the number of interactions between the human caregiver and the participant for successful completion of hand washing.
Table 1 presents a summary of the overall data collected for each of Mrs. Smith’s target behaviors (Figure 4), representing the overall means per test phase, and are the mean values collected between two independent raters. It was observed that this participant’s level of dependence on the caregiver decreased when the device was introduced, and then increased when the device was removed. The observed changes in the total number of interactions between the participant and caregiver indicated a potential decrease in caregiver workload, which is an important finding to further explore.
The COACH assisted Mrs. Smith with approximately one-third of the hand-washing steps she was able to complete without assistance from a caregiver during the two intervention phases. The overall efficacy of the system was determined by using a total count of the number of hits (correctly identifying an error and providing a prompt); misses (not correctly identifying that an error was made, thus no prompt); false alarms (providing an unnecessary prompt); and correct rejects (recognizing that no error had been made, thus no prompt). These data were used to calculate two error rates: 1) Ew, the device not properly detecting that the participant made an error, and therefore not playing a corrective prompt; and 2) Ec, the device not properly detecting that the participant had performed a correct action, therefore erroneously playing a prompt. Table 2 summarizes the device-performance data.
It was observed that the majority of the system’s misses occurred during the "wet hands" step (58 percent of all misses), while the majority of false alarms occurred during the "turn off the water" step (48 percent of all false alarms). The system missed Mrs. Smith’s wetting her hands because it was observed that she would often not fully put her hands underneath the running water, but instead would lightly touch the water with her fingertips. This ambiguous user action caused the tracking and planning modules to be uncertain if this step had actually occurred. With respect to turning off the water, the system suffered from the limitation that if a particular step was done quickly, it was not observed by the tracking module, and thus not recorded as being completed.
The AI algorithms used to adjust the various parameters seemed to have worked efficiently and properly, including the selection of appropriate prompt detail levels based on Mrs. Smith’s performance. However, the participant ignored a large percentage of the prompts from the device, and often it was observed that she was not able to fully understand the directions being given. In fact, of the 325 total prompts that were provided by the system (including all false alarms), she responded positively to only 6 percent of them. For some of the hand-washing steps, five or more prompts from the system were often required before she responded properly or the caregiver was required to enter and assist her, especially during the "use the soap" step. This particular observation has resulted in more in-depth studies to be conducted by the authors on the types of interactions and prompts that are more effective with people with dementia , as well as the incorporation of video-based prompting strategies.
A cognitive device (the COACH) was developed using AI techniques to assist participants with moderate to severe dementia during hand-washing. This article presented an overview of this system, and its performance for one particular user who had severe Alzheimer’s disease. These data showed how such a technology may still be effective even for an "extreme" type of user, who was highly unresponsive to assistance provided by not only the technology, but also by a caregiver. The device was successful in the fact that it helped this particular participant perform more hand-washing steps without the caregiver, however, also identified several limitations in the technology, which are currently being resolved by the authors.
1. Boger, J., J. Hoey, P. Poupart, C. Boutilier, G. Fernie, and A. Mihailidis, "A Planning System Based on Markov Decision Processes to Guide People with Dementia Through Activities of Daily Living," IEEE Transactions on Information Technology in Biomedicine 10 (2006): 323 - 333.
2. Mihailidis, A., B. Carmichael, and J. Boger, "The use of computer vision in an intelligent environment to support aging-in-place, safety, and independence in the home.," IEEE Transaction on Information Technology in Biomedicine (Special Issue on Pervasive Healthcare) 8 (2004): 238-247.
University of Toronto
University of Toronto
University of Toronto
University of Dundee
About the Authors
In 2002, Alex Mihailidis received his Ph.D. in biomedical engineering from the University of Strathclyde in Glasgow, Scotland after attending the University of Toronto, where he received a M.A.Sc. in biomedical engineering and B.A.Sc. in mechanical engineering. He is presently an assistant professor in the department of occupational science and occupational therapy and the institute of biomaterials and biomedical engineering at the University of Toronto. His research focuses on the development of intelligent assistive technology and environments for older adults with cognitive and mobility impairments.
Jennifer N. Boger is a senior researcher in the department of occupational science and occupational therapy at the University of Toronto and a scientist at the Toronto Rehabilitation Institute. She received a M.A.Sc. in biomedical engineering from the University of Toronto and her B.A.Sc. in biological engineering from the University of Guelph in Guelph, Canada. Her research focuses on the development of artificially intelligent systems that enable older adults to age-in-place with greater safety and independence.
Marcelle Canido received her B.A. from the University of Toronto and in 2006 a certificate in gerontology from Ryerson University in Toronto, Canada. She is currently working for The Health Technology Exchange (HTX), a non-profit organization that facilitates connections in the medical and assistive technologies sector in Canada with a focus on the Ontario region.
Jesse Hoey received a Ph.D in computer science from the University of British Columbia in Vancouver, Canada, and his B.Sc. and M.Sc. in physics from McGill University in Montreal, Canada. His postdoctoral research was carried out at the University of Toronto, jointly in the department of computer science and the department of occupational science and occupational therapy. Currently, he is a lecturer in the school of computing at the University of Dundee, Scotland and an adjunct scientist at the Toronto Rehabilitation Institute. His research focuses on planning and acting in large-scale, real-world uncertain domains using video observations. In particular, he works on applying decision theoretic planning, computer vision, and machine learning techniques to adaptive assistive technologies in healthcare.
Figure 2. Tracking of hand, towel, and soap positions
for the activity of hand-washing for (a) the original image and
(b) the background-subtracted, color-matched image. The red spots
on the center of the user’s hands denote the (x,y) coordinates of
the hands that are passed to the planning system.
Figure 3. The acceptable pathways for the hand-washing
activity as defined by the system’s planning module. Any
combination of steps that takes the user from start to finish
through this diagram was considered successful hand-washing by
the planning system .
©2007 ACM 1072-5220/07/0700 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2007 ACM, Inc.