The International Organization for Standardization (ISO) subcommittee on systems and software engineering approved the Common Industry Format for Usability Test Reports as an international standard at their plenary in Helsinki, May 2005. The new standard, "ISO/IEC 25062: Software EngineeringSoftware Product Quality Requirements and Evaluation (SQuaRE)"common industry format for usability test reportswill be published shortly. Information concerning the publication date will be available at www.nist.gov/iusr.
What Is the Common Industry Format? The Common Industry Format (CIF) is a standard method for reporting usability test findings. The format is designed for reporting results of formal usability tests in which quantitative measurements were collected and is particularly appropriate for summative/comparative testing. The CIF targets two audiences: usability professionals and stakeholders in an organization. Stakeholders can use the usability data to help make informed decisions concerning the release of software products or the procurement of such products.
Why Is a Common Format Needed? A format providing common information is needed because there are many possible ways to report usability results. The purchase team may require an evaluation of the validity and relevance of any test that it uses to support its decision making. A common format for reporting the test and its results facilitates evaluation of the test and the interpretation of its results. It also reduces misinterpretation of the test results. A common format for reporting usability tests therefore benefits both suppliers and consumers of software products.
What's Included in a CIF? It is important to note that the CIF assumes that best practice is used in designing and conducting a usability test. The CIF does not tell you what to do; it tells you how to report on what you did. The CIF is based on the usability definitions of ISO 9241-11, and as such is appropriate for use when the test procedures follow both ISO 13407 for Human Centered Design and ISO 9241-11.
The report format includes the following sections:
The description of the product. As some products have several releases and versions, the product description should include this information. The description should explain basic functionality of the product and the intended users of the product.
The goals of the test. User testing may be performed to accomplish a variety of goals, including problem identification or diagnosis, design alternatives comparison, or to complete a summative test. The goal(s) of the reported test should be clearly stated.
The test participants. This section should include information on the number of users who participated and the criteria by which they were selected.
The tasks the users were asked to perform. This section should list the specific tasks that participants were asked to perform during the study.
The experimental design of the test. This section should explain the logical configuration of the test conditions, including independent variables, what comparisons, if any, are intended between groups, and how conditions that might contaminate the results are brought under control.
The method or process by which the test was conducted. This section should report the sequence of events that was actually employed to instruct the test users, how well they followed it, any intervention such as coaching or assists, and materials used to give instructions or ask questions of them.
The usability measures and data collection methods. Usability measures may include objective measures of effectiveness, efficiency, and how much effort is required to learn to use the product successfully. Subjective data on user satisfaction should also be collected.
The numerical results. This section should report the data analysis procedures and summary data. The results may contain summary statistics such as the mean, median, range, standard deviation, and standard error of the estimate. Graphical methods are also to be included as appropriate.
Historical Perspective. The CIF has its origins in the Industry Usability Reporting (IUSR) effort, started in 1997 by the National Institute of Standards and Technology (NIST) with the objective of increasing the visibility of software usability. The participants, representatives from industry and academia, worked through a series of workshops to develop the format; then they ran several pilot studies to show the utility of using the format. In December 2001, the American National Standards Institute (ANSI) approved the CIF as a recognized standard, ANSI-NCITS 354-2001.
National Institute of Standards and Technology
About the Author:
Mary Frances Theofanos is a computer scientist in the Visualization and Usability Group at NIST, where she is working on the Industry Usability Reporting Project developing standards for usability. Previously, she was the manager of the National Cancer Institute's Communication Technologies Research Center, a state-of-the-art usability testing facility that included an extensive research program on the intersection of accessibility and usability. Mary spent 15 years as a program manager at the DOE's Oak Ridge National Laboratory complex.
Susan Dray & David A. Siegel
Dray & Associates, Inc.
2007 Kenwood Parkway
Minneapolis, MN 55405, USA
©2005 ACM 1072-5220/05/0900 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2005 ACM, Inc.