Arnie Lund, Jean Scholtz, Nigel Bevan
We all realize that standards are good. For example, consider when something in your house breaks and you need to find a replacement part. What size does the part need to be? Will we be able to attach it successfully without needing a machine shop? Standards are great for mechanical thingsfor various car parts and in factories, to ensure that a failed component doesn't necessitate a whole new piece of equipmentbut why on Earth would we want standards in human-computer interaction?
The examples highlighted here are for very specific and enduring standards. If we created detailed standards for user interfaces, however, they would be outdated by the time the standards were completed, as it takes time to get them approved and in place. Moreover, wouldn't setting standards be contrary to the creativity we are always looking for in user-interface design? Do we even need standards? After all, the software industry has long produced its own style guides to bring consistency to applications based on its APIs and created default standards that human-computer interaction (HCI) research has built upon. The industry has also driven influential standards efforts, such as those from W3C, which continue to shape the Web experience and its accessibility.
Bevan  points out that standards in HCI typically are not of a precise nature but rather are concerned with principles that ensure interfaces meet the needs of users. He notes that these standards are of two types: top-down and bottom-up. The top-down approach looks at the concept of usability and at ways in which usability can be achieved in a product. Within this approach there exist two types of standards: those that specify usability requirements and assess whether they have been designed into the product, and those that specify how to incorporate usability into the design of the product. The bottom-up approach provides standards about the details of the appearance and the behavior of the user interface, guidance on the user interface design, and criteria for evaluating user interfaces. Ideally, these standards are derived from HCI and related behavioral science research and should be written to apply across specific technical implementations to ensure they are valuable as the field evolves. HCI standards apply to not just software, but also to hardware and to software-hardware interaction. They can also apply to the contexts in which the hardware and software are used (e.g., workstation design as specified in the ISO 9241 standard).
To better understand standards and how they apply to CHI researchers and practitioners, we need to know more about what standards currently exist in this area, what organizations are responsible for standards development, who can contribute to the creation of standards, and what, specifically, the CHI community can contribute.
The authors of this article have worked on standards that have had an impact on the CHI community. In the following paragraphs, Arnie Lund and Jean Scholtz share their experiences working on some standards for human-computer interaction in the U.S., and Nigel Bevan provides an international perspective from his work with ISO. We hope these experiences will help others in the CHI community better understand why and how they can contribute to standards efforts.
Over the years I've had the opportunity to work on several standards efforts. In the early 1980s, several people began to explore whether software standardization for user interfaces was even possible. Initially, I sent a representative from our team to the early meetings, and then I joined the effort myself. That work culminated in the first ANSI accessibility standard, as well as a standard for interaction techniques, speech interfaces, and visual presentation (e.g., the use of color). Unfortunately, it wasn't completed until August 2008 (ANSI/HFES 200). On the positive side, the research reviewed during the process of creating the standard served as the U.S. input into ISO standards efforts throughout the 1990s .
HCI in the communications industry has often focused on creating standards through the ITU Telecommunication Standardization Sector (ITU-T), formerly known as the International Telegraph and Telephone Consultative Committee (CCITT). In the early 1990s, telephone keypads were incomplete: They lacked the letters Q and Z, and of course it was not clear how punctuation should be entered. Following research led by Monica Marics, several of the HCI teams at various telecommunications companies worked together to support a change to international telephony standards . It is difficult to imagine how today's world of SMS would have happened without that work.
In 1997 I was working at the National Institute of Standards and Technology (NIST). I attended a workshop and participated in a discussion with several attendees concerning how companies buying software could compare the usability of products they were considering purchasing. Thus the Industry Usability Reporting Project (IUSR) was born.
First, we assembled a group of usability professionals, including consultants, usability professionals from organizations, and researchers in universities and other laboratories. From this group we elicited best practices in conducting and reporting usability tests. Our goal was to produce a template that could be widely used, assuming that if numerous software companies used the same format for summative usability tests, it would provide potential customers with an understanding of the usability of products. Customers could easily understand what was tested and with what kinds of users and compare the usability metrics to determine if that was sufficient for their needs. Customers would request this information in their request for proposals (RFPs), and it could be communicated under nondisclosures if needed.
The template did not specify what was to be tested, but rather the test information that was to be reported. This includes the following:
- Demographics of the participants;
- The tasks participants were asked to perform;
- The resulting metrics of effectiveness, efficiency, and user satisfaction;
- How the test was conducted;
- How the data were collected; and
- How the metrics were computed.
Using this information, a potential customer could determine if the tasks tested were the tasks their employees did, if the participants in the test were similar to those in their organizations, and if the usability results were acceptable in their domain. If not, negotiations with the software vendor might result in running a new test with different tasks and/or different participants.
It is worth noting the time frame for this effort. The original discussion, which was the concept for the standard, occurred in 1997. NIST held two workshops in 1998, to which a number of CHI researchers and practitioners were invited. A 1998 white paper (http://zing.ncsl.nist.gov/iusr/documents/WhitePaper.html) outlined the template for the Common Industry Format (CIF) for reporting usability information and outlined a pilot study that would start in 1999 and run for two and a half years to determine if the template contained the appropriate information and if it was useful for both consumers and producers of software products. The CIF became an ANSI/INCITS standard in 2001 and an international standard, ISO/IEC 25062, in 2005. ISO is continuing to develop other common industry formats. In the meantime, the Common Industry Specification for Usability-Requirements, also developed by IUSR, is available for download from NIST (http://nist.gov/iusr/).
I first started working on an international standards committee for keyboard layouts in the 1980s when I was at the National Physical Laboratory in the U.K. In the early 1990s, as a result of pressure from countries such as the U.S. (as explained by Arnie), one of our tasks was to work with CCITT to obtain international agreement on the position of the letters Q and Z on the numeric keypad, as different countries used different layouts on their phones (some without the letters Q and Z). This was an essential prerequisite for consistency of user interfaces, but few HCI standards are this specific.
In 1988 I started work on a standard for usability, which eventually resulted in the ISO 9241-11 usability standard. The original intention had been to specify the contents of usability assurance statements that would consist of test reports giving results for effectiveness, efficiency, and satisfaction in a specified context of use . Unfortunately, some leading companies at that time did not want to be forced to make usable products. For example, a large U.S. company threatened to use its influence to ensure the standard was rejected unless it was redrafted as guidelines rather than requirements.
So, 10 years later, I was delighted to have the opportunity to contribute to Jean's initiative. In this project, I collaborated with a group of U.S. companies frustrated by the low profile of usability in product procurement to develop a Common Industry Format for usability test reports (which was also tried in Europe ).
In 2000 I started work with another ISO group to develop a standard to provide usability assurance for machines used by the public and for consumer products. ISO 20282 was published as a preliminary standard for review in several parts in 2006 and 2007 (but not before a large German company lobbied hard to prevent publication of a standard that could possibly regulate the usability of consumer products). The ISO 20282 series of standards specifies a method for measuring the effectiveness, efficiency, and satisfaction of walk-up-and-use systems and consumer products. The intent is to provide a reliable method for measuring usability that produces results that are, for example, suitable for reporting in the Common Industry Format. This method could be used by consumer organizations to compare the usability of different products, or it could form the basis for claims made in advertising.
How widely this method will be adopted remains to be seen. It is difficult for usability testing to cover all relevant uses and users of a product without testing very large numbers of users . ISO 20282-3 is currently being revised to improve reliability when tests are made with 10 or 12 users. But an alternative approach to product certification has been taken by HFI, based on qualified staff using an adequate set of methods, standards, and tools to complete a proscribed set of activities designed to ensure good usability . Hopefully, we will see wider exploration of the potential value of the different approaches.
ISO 9241 also has many parts that contain user interface guidelinesfor example, ISO 9241-151: Guidance on World Wide Web User Interfaces, has 141 guidelines. While these types of compilations of guidelines provide a good reference set, designers find them difficult to use. Fortunately, the U.S. Department of Health and Human Sciences (HHS) produced the Research-Based Web Design & Usability Guidelines , with a similar scope to ISO but in a much easier-to-use format that would be beyond the resources of ISO . Although the HHS set contains even more guidelines (196), they are prioritized by importance, making it possible to extract subsets for use in particular project environments.
One of the most useful parts of ISO 9241 is part 210: Human-Centered Design (HCD) for Interactive Systems. Intended as a manager's guide, it is one of the best available concise introductions to applying HCI in the design process. The new version (superseding the original ISO 13407) provides specific requirements for the activities that must take place to develop a system that meets the needs of the users. It is supported by the ISO 18529 standard (currently being revised as ISO 9241-220), which contains more specific human-centered lifecycle process descriptions using categories derived from ISO 9241-210, such as the following:
- Ensure HCD content in system strategy;
- Plan and manage the HCD process;
- Specify the stakeholder and organizational requirements;
- Understand and specify the context of use;
- Produce design solutions;
- Evaluate designs against requirements; and
- Introduce and operate the system.
The detailed descriptions provided for the activities in these categories can be used for implementing, evaluating, and improving organizations' human-centered design processes.
The goal of following this process is to produce a usable system, but what exactly does that mean? (I have been pondering this for many years, e.g., [11,12,13]). The ISO 9241-11 standard includes the popular definition of usability: the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. But software quality standard ISO/IEC 9126 had already provided a competing definition of usability in terms of good user interface design, so I helped with the revision of ISO/IEC 9126 to introduce the broader view of usability as effectiveness, efficiency, and satisfaction under the synonym "quality in use" . More recently, ISO/IEC 9126 has been replaced by ISO/IEC 25010, in which we worked to harmonize the definitions of usability while recognizing the two perspectives: ease of use as a contribution to the higher-level goal of quality in use. Quality in use adds two additional objectives to effectiveness, efficiency, and satisfaction: minimizing the potential risks that could be a consequence of use and producing a product that is usable in all of the intended contexts of use.
With an increasing understanding of the relationship between usability, quality in use, user experience (defined in ISO 9241-210 as a person's perceptions and responses resulting from the use and/or anticipated use of a product, system, or service), and accessibility (defined in ISO 9241-210 as the usability of a product, service, environment, or facility by people with the widest range of capabilities), consultation has just begun on producing a new version of ISO 9241-11 that explains the concept of usability from different perspectives.
Why is this of interest to the CHI community? Many of you will work at companies that adhere to these standards. Some of you will develop products that you want to sell to particular industries that use these standards. For new designs to have the most impact, they will often need to work together (e.g., natural user interfaces) or leverage knowledge that users already have as they move from device to device and application to application. Sometimes you may need to provide an authoritative reference to good practice in HCI. And some of you are doing research in new technologies and in the application of technology in new domains. Enabling full accessibility for a global and diverse marketplace will require well-supported standards delivered in a timely and effective way.
The Usability Professionals' Association (UPA) has recently decided to liaise with ISO TC159/SC4, which produces the ISO 9241 standards, in order to receive input from experts who are UPA members.
More standards will be coming that relate to new types of interfaces, new types of interactions, new mobile devices, and new insights to good practice in HCI. Another example is the design of electronic records in various domains. The National Institute of Standards and Technology recently hosted a workshop on electronic health records (EHRs; http://www.nist.gov/health-care/usability/usability-technical-workshop.cfm). It is essential that these records preclude the kinds of errors that could lead to severe consequences for patients and healthcare providers. The design of such records must be considered not just from a U.S. perspective but from a global perspective, given the amount of international travel that occurs. If the current ISO standards are not sufficient to achieve this, then additional standards for assuring usable EHRs may appear.
International standards can be purchased as PDFs on the Web, either from ISO or your national standards body. These are listed on the ISO website, http://www.iso.org/iso/iso_members.
If you want to get involved in developing standards, contact your national standards organization. The U.S. member body, ANSI, delegates the development of ergonomics standards, including the ISO 9241 standards, to the Human Factors and Ergonomics Society (http://www.hfes.org/web/standards/standard-sISO.html).
New standards are based on the best practices and research of the community. Standards should be a representation of what we know, and they should be delivered in a way that has measurable impact. Questions arise when new standards are created; they include whether the level of detail included is sufficient for a standard, whether the standard will be of benefit to anyone, and whether the time is right for creating a standard for a particular process. Those best qualified to answer these questions are those who are working in the areain other words, those in the CHI community.
2. Reed, P., Holdaway, K., Isensee, S., Buie, E., Fox, J., Williams, J., and Lund, A. User interface guidelines and standards: Progress, issues, and prospects. Interacting with Computers 12, 2 (1999) 119142.
4. Butler, K., Wichansky, A., Laskowski, S.J., Morse, E.L., Scholtz, J.C. The Common Industry Format: A way for vendors and customers to talk about software usability. Computer-Human Interaction Conference (Bath, England). 2003.
5. Butler, K., Wichansky, A., Laskowski, S.J., Morse, E.L., Scholtz, J.C. Quantifying Usability: The Industry Usability Reporting Project. Proc. of Human Factors and Ergonomics Society Conference. 2002.
6. Bevan, N. and Holdaway, K. User needs for user system interaction standards. In User Needs for Information Technology Standards, Evans. Meek, and Walker, eds. Butterworth Heinemann, Woburn, MA, 1993.
7. Bevan, N., Claridge, N., Maguire, M., and Athousaki, M. Specifying and evaluating usability requirements using the Common Industry Format: Four case studies. Proc. of IFIP 17th World Computer Congress (Montreal, Canada). Kluwer Academic Publishers, 2002, 133148.
8. Schaffer, E.M. and Weinschenk, S. Certified Usable Designs. Products, Applications, and Web Sites. August 2011; http://www.humanfactors.com/downloads/documents/CUD.doc
9. HHS (U.S. Department of Health and Human Sciences) Research-Based Web Design & Usability Guidelines. 2006; http://www.usability.gov/guidelines/
12. Bevan, N. Usability is quality of use. In Symbiosis of Human and Artifact: Proceedings of the 6th International Conference on Human-Computer Interaction (vol. 2). Y. Anzai, K. Ogawa, and M. Hirohiko, eds. Elsevier, Amsterdam, 1995.
Arnie Lund is a principal user experience lead at Microsoft. He is a member of the SIGCHI Academy and is an HFES Fellow. On the HFES Executive Council he ran the HFES Institute where he oversaw HCI standards work, and was president of the board of directors for the Board of Certification in Professional Ergonomics.
Jean Scholtz is a chief scientist at the Pacific Northwest National Laboratory, where she works in user-centered evaluation of visual analytics environments. In her prior work at the National Institute of Standards and Technology, she focused on standards for usability testing and for human-robot interaction in both urban search and rescue and explosive ordnance disposal.
Nigel Bevan is an independent usability consultant with wide industrial and research experience. He has been editor of several international standards. Bevan leads the UPA Usability. Body of Knowledge project, and has worked on many professional projects, including the National Academy of Science Committee on Human-System Design Support for Changing Technology.
The following international organizations are responsible for the development of standards relevant to the CHI community:
- ISO: The International Organization for Standardization. This organization has developed more than 18,500 international standards. One hundred sixty-two countries are represented in this organization, with one member per country. National and liaison organizations appoint experts to work on various technical committees to arrive at a consensus on standards. Stakeholders with requirements for standards work through their national standards organization to get standards developed. http://www.iso.org/iso/home.htm
- IEEE-SA: The Institute of Electrical and Electronics Engineers Standards Association is part of the IEEE. Leaders in more than 160 countries work through this organization to build consensus on standards. It now coordinates its systems and software engineering standards with ISO. http://standards.ieee.org/about/index.html
- ITU-T: The ITU Telecommunication Standardization Sector (ITU-T) coordinates standards for telecommunications. Until 1993 it was known as CCITT (International Telegraph and Telephone Consultative Committee). http://www.itu.int/ITU-T/
- W3C: This international organization develops standards, including accessibility standards, to facilitate Web innovation. W3C is open to all organizations and individuals who wish to participate. http://www.w3.org/
Each country has its own national standards bodies. A list of ISO member bodies can be found at http://www.iso.org/iso/about/iso_members.htm
The following organizations are responsible for standards in the U.S.:
- ANSI: The American National Standards Institute. Founded in 1918, this nonprofit organization works to promote and facilitate standards in the U.S. Development of standards is delegated to organizations such as HFES and INCITES. http://www.ansi.org
- HFES: The Human Factors and Ergonomics Society. This group is involved in both U.S. and international standards. It maintains a technical advisory group to the ISO Technical Committee on Ergonomics. http://www.hfes.org/web/Standards/standards.html
- INCITES: International Committee on Information Technology Standards. Creates and maintains IT standards in the U.S. http://www.incits.org
- NIST: The National Institute for Standards and Technology. This U.S. government agency does not develop standards but works with other federal agencies and industry and participates in standards activities both domestically and internationally. http://www.nist.gov/director/sco/index.cfm
©2012 ACM 1072-5220/12/0100 $10.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2012 ACM, Inc.