In Art Swanson's and Scott Lind's article, "Usability Testing EHRs," from our November + December 2011 issue, the authors refer to a proposed NIST protocol as the "Usability Evaluation Protocol (UEP)." The correct name of the protocol is the EHR Usability Protocol (EUP). A draft document describing the rationale for the protocol can be found at http://www.nist.gov/healthcare/usability/upload/Draft_EUP_09_28_11.pdf
The article provoked many comments from readers, some of which are presented here in edited form.
In the November + December 2011 issue of interactions, Swanson and Lind describe industry objections to a NIST plan for summative usability testing of electronic health record systems (EHRs). They begin by citing reliability issues of formative tests, but then imply those problems apply equally to summative tests. This is a little like ignoring your house's water meter because it won't tell you which faucets are leaky.
The authors assert that EHRs and their usage are too complex to allow comparison, describing seven challenges to justify their objection. In terms of measurement theory and statistics, their challenges fall into the well-established factors of validity and reliability, and the generality of samples. These same factors helped drive the ISO standard for summative usability testing that is the basis of NIST's plan, which is based on an adaptation of the ISO standard for reporting summative usability tests called the Common Industry Format (CIF). The objectives of the CIF are to allow usability to be included in customers' purchasing decisions, to replicate a test if needed, and to judge how well software products match the customer's own users, tasks, workflows, and context.
Swanson and Lind claim the seven challenges bring so much complexity that it's impractical and expensive to conduct all the needed testing. Understandably, they write from the perspective of EHR vendors, not the perspective of customers who need to buy the best EHR to meet their needs. But customers don't want to read every combination of every factor that could possibly be tested: They only want test reports on the EHRs that match their needs. There is currently no alternate means for customers to get this crucial information in a standard format.
To speed learning there are two lessons worth drawing from best practices in related areas of computing. The first is software testing. Despite conscientious testing users commonly find and report bugs in shipped software products. Yet, nobody has recommended we discount software testing. Instead, we try to improve it with both research and practice. Leading software companies track bugs closely, and strive to reduce rates release-over-release, despite differences in features and functions. The second lesson comes from commercial aviation, where a culture of quality and safety has guided remarkably successful innovations in interactive computing in flight decks. It may be the most important usability issue for the EHR industry is that a similar culture of quality and safety is still evolving for information systems that support clinical health-care and safety-critical decisions.
University of Washington
National Center for Cognitive Informatics and Decision Making
The article highlights the significance of usability testing EHRs and the challenges in doing so. It is a critical issue that should get a prominent place on the agenda of HCI and usability professionals, and this article does a great job of promoting the issue, particularly on a tactical level by sharing tips and lessons learned for usability testing in the medical context.
It is important to consider that the challenges of making usable EHRs go beyond the typical challenges of usability testing of any complex system. EHRs can be used in general practice as well as critical care. At stake are the quality of care and patient safety. We need to consider new methods and new usability metrics that can better reflect threats to quality of care and patient safety associated with the use of poorly designed EHRs. Most importantly, the very dynamic and unpredictable context of care could be a crucial aspect that must be considered in both the design and the testing of EHRs. The context must be taken into account by involving and engaging all healthcare disciplines and professions since any context in healthcare is cross-professional and must be incorporated in any usability testing.
The design and testing of usable EHRs should become a critical item not only on the agenda of HCI and usability professionals, but for all healthcare policy makers. Publishing articles like the ones featured in interactions is a positive step by sharing useful tactics. As a professional community, we still need to develop a broader strategy to address the unique challenges of the healthcare discipline.
Department of Psychology
Swanson and Lind elucidate essential issues in making EHR systems usable, useful, and appealing. We know things can be done. We know things can improve. A crucial catalyst is convincing top executives with budgets and decision-making power to get behind such initiatives. In this regard, our experience has shown it is important to prepare and present executive marketing presentations that show clearly what is proposed and the risks of doing nothing. These presentations are often crucial in paving the way for the more "mainstream" work of design and evaluation.
It does occur to me that the issues they discuss could be enhanced by consideration of topics not emphasized as much as our own experience shows them to be relevant:
- How do cross-cultural differences affect how information is collected, managed, shown, and used?
- How do cross-cultural differences affect team operations, workflow, concepts of leadership, and patient/family relations and communication?
- How do mobile devices affect any and all gathering, dissemination, and use of EHRs?
- How do social networks affect EHR systems? How might these be used to provide incentives to best practices and to reduce impediments to collection, maintenance, and reporting?
The article provides a very practical perspective on what needs to be done. I hope medical informatics policy makers will take noteand take actionto remedy a situation that is, in some cases, in crisis mode, and in others an embarrassment in consideration of the power of technology to improve our lives.
President, Aaron Marcus and Associates, Inc.
We think it's important to note that while our article was spurred by the NIST EUP, it is not about the EUP. It is about the challenges in usability testing EHRs. Some argue that the EUP does not aim to use summative testing to compare EHRs. While that may not be a goal of the protocol itself, there have been a number of stakeholders throughout the process who have indicated the protocol was likely a precursor to comparative testing, and the draft protocol references a common CIF reporting structure for results to be submitted. This is enforced by Keith Butler's comments about the NIST protocol and CIF describing the need for EHR purchasers to have concrete data about system usability when making purchasing decisions. This is the fundamental question: Is a protocol such as the one proposed by NIST going to generate repeatable, generalizable, and valid usability testing data that will benefit EHR customers? Butler argues from an academic perspective, and we argue from practical experience. We need to work toward common ground.
Butler is incorrect that our article "cit[es] reliability issues of formative tests but then impl[ies] those problems apply equally to summative tests"the article addresses both. He is correct in stating that we write from the perspective of usability professionals working for EHR vendors. And we would like to stress that our companies compete on usability and other factors every day. It's absolutely in vendors' interest to build systems that customers want to buy and that help users be more effective in their difficult jobs.
We strongly agree with the points Avi Parush makes about context and safety. Usability is one of many factors that contribute to safety of EHR systems. And we agree with Aaron Marcus's comments on the importance of executive support of usability processes as well as additional cultural and technical factors impacting EHRs.
This is an important topic and there are a number of alternative viewpoints on these issues. But we can't deny the challengeswe need to acknowledge and address them in order to move forward. We all have the same goalit's just that there is honest disagreement and good debate on the best way to get there.
Art Swanson and Scott Lind
©2012 ACM 1072-5220/12/0300 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.