Chauncey Wilson, Kara Coyne
Most development organizations track software bugs and their severity in a corporate database, which is shared with product development and tech support teams. We find, however, that these same organizations seldom have a standard method, if any, of tracking usability issues. Usability practitioners communicate usability problems through reports, highlight tapes, and formal briefings, but are these methods adequate for tracking usability problems through successive cycles of product development?
We diverge in our specific approaches to tracking. Chauncey believes that it is essential to track usability issues in the official corporate bug database. Bugs that don't get into the bug database will not be considered "official" bugs or be visible to the product development community. Although Kara agrees that it is possible to track usability issues in a bug database, she believes that usability problems are better tracked in a different system that is controlled by the usability team. This article recommends suggestions for successfully tracking usability issues and discusses our differing views on how to track them.
Most medium to large corporations live and die by their daily bug reports. These reports become powerful sources of anxiety as a product gets closer to its ship date. I spent the last two years as a product development manager and found that usability problems that weren't in the database got much less attention than nonusability bugs that caused users less pain. I made a strong stand that usability bugs were just as important as any other bug (including "crash" bugs).
There are several reasons for treating usability problems as real bugs and encouraging everyone to enter them into the corporate bug database:
- Development managers are often responsible for deciding what bugs get fixed. If usability bugs are not in the bug reports, they are less likely to be fixed.
- Usability issues are logged with the other software issues. People do not have to search different databases to find the usability issues. Formal usability bugs are permanent, are entered into official bug counts, and can be used to persuade product teams to fix usability problems.
- Treating usability bugs like all other bugs puts usability into the mainstream of development.
- Usability issues are advertised and seen by the developers as much as any other issues are.
- You can begin to develop official usability metrics.
My stand is that usability problems must be entered as usability bugs just as other types of bugs so they can be tracked over time and become part of the formal development process.
The method, resources, and process for fixing usability issues are more complex than those for fixing nonusability bugs. For example, a functional bug report might read, "When I click the New e-mail button, nothing happens." The developers know that they are supposed to make this button open the e-mail form to write a message. Conversely, a usability issue could be, "Users don't know what will happen when they click the Send button. They don't know if it will save a copy of the e-mail or not." Whereas functional bugs usually need the attention of a developer, a quality engineer, and possibly a writer, decisions about usability issues require all these people plus the interface designers, usability professionals, and often product management and marketing.
Comparing usability issues in the context of programming bugs is dangerous. A level-one crash bug will get more attention than a level-one usability bug, whereby the user can't complete a task. Thus, usability problems can be overlooked if logged with functional bugs.
Separate entries of related usability issues are difficult to group and consider as a whole. For example, if you have 300 severity three or four usability bugs (see sidebar) related to alignment of controls and keyboard access, they may not be given a high priority, but together they could have major impact.
Bug databases often lack a clearly defined usability category and rules for assigning levels of severity for usability. Thus, usability bugs often end up in a catchall category with a low severity rating and can easily slip to the bottom of the list of things to fix. Adding an appropriate usability category may require extensive changes to a database table or functional design. It can also be a political process and require much training to deal with the change.
A short usability issue description often cannot stand on its own among a mass of bugs. Usability problems need longer explanations, discussions, and even observation of users to be fully understood and evaluated.
Most software applications and Web sites have so many bugs that usability issues can be diluted and lost among the myriad nonusability bugs.
Kara's point that usability bugs require a team effort to fix is well taken, but that doesn't negate the usefulness of tracking usability bugs in a corporate bug database. Usability practitioners should review the bug list several times a week to identify usability problems and work with product teams to achieve satisfactory solutions. Developers can handle many usability issues without involving an interdisciplinary team.
Kara noted that a serious usability problem won't be viewed the same as a crash, and I have to agree that it is a hard sell to get people to agree to this, but it can be done if you create a usability severity scale that parallels the severity scale for other bugs. The severity scale that we describe later is a starting point for getting people to accept that usability problems are just as important as any other type of problem.
The problem of many small usability problems is a difficult one. What is the impact of 200 minor aesthetic, consistency, and layout problems on product quality? I believe that many low-severity bugs can add up to a moderate or critical bug. So one approach would be for the usability team to track these bugs and then enter a composite bug of a higher priority.
Kara makes a good point about bug databases not supporting usability problems. The approach that I've used to make changes is to enlist quality assurance (QA) directors as allies and get them to push changes to the corporate bug database.
The point about usability bugs being diluted among all other bugs is valid, but in my experience, the number of usability bugs often makes up 20 to 50 percent of the overall bugs; this figure can be a powerful metric for those who say that usability is not a major issue.
A usability issues database can be useful for a team that needs to track many usability issues over several product releases. Although you will have the initial overhead and continued maintenance of yet another application, it can be fruitful to track usability issues in their own database for the following reasons:
- The form can be customized to work best for usability issues and automatically take data from usability reports without affecting the corporate bug database.
- All usability issues are weighed against each other and not against programming bugs.
- You don't need to deal with the politics of establishing usability bugs as equivalent to other types of bugs.
- The usability team has control of the database and how the issues are managed. They can create graphs and report and call meetings with critical staff when necessary.
The presence of a separate usability bugs database puts the burden on product development team members to look there in addition to the corporate bug database they already use. Similarly, usability professionals need to integrate this new system with the process and culture of the organization.
We urge you to investigate both reporting usability issues in the corporate bug database as well as other tracking systems and see which method or combination of methods works best for your usability group and the product team. If you do choose to report usability issues in your corporate bug database, we recommend taking the following actions:
- Add a distinct usability category to the bug-tracking database.
- Define what a usability bug is.
- Create a usability severity scale that parallels the severity scale for programming bugs.
- Evangelize the tagline usability bugs are real bugs.
- Train engineering, quality assurance, tech support, and documentation teams in how to recognize and rate the severity of usability bugs.
- Attend bug review meetings and provide input on usability bugs.
- Begin tracking the number and severity of usability bugs and use these data as tangible usability metrics.
- Cross-reference bug reports with other usability reports or documents.
Nielsen  recommends having several evaluators rate usability problems and then take the mean of the ratings. Although this might be the best way to determine severity, multiple usability specialists are not always available and when anyone with access to the bug tracking system is allowed to submit usability bugs, this level of rigor is not possible.
A severity scale for usability bugs should have the same levels as your existing bug scale, with equivalent definitions of severity. If your programming bug scale is from one to four, the usability scale should use an equivalent numerical scale. The following usability severity-rating scheme is similar to ones adopted by several companies .
The concept of formal usability bugs will be new to many companies, and some training on how to define and rate them is essential. Don't make the mistake, though, of training the QA team first. When Chauncey trained more than 100 QA engineers in the concept of usability bugs, they got so excited about this new skill that they entered 200 new bugs on a major product (about to go to beta) the day after the course! Naturally, the development managers panicked when they got their daily bug charts and saw this giant spike. (Chauncey was not popular with this group of managers.) The moral of the story is to present the facts first to the audience that will be affected by a sudden rise in their bug counts.
A usability report typically gives context to a problem found during a study. It discusses the method, tasks, situation, and the users involved. Usability reports are certainly valuable; following are some ways to use them with a bug database or a usability issues database.
- Link to reports. People who are very involved with a feature will usually be interested in more of the details behind the usability issues, not just the short problem report.
- Create bug reports for all the details. Test reports often summarize many usability problems. Every single problem should be logged in the bug database or those problems will be quickly lost if they are logged only in the written report.
- Predict the future. Because you know that issues will be added to the bug database or usability issues database, write summaries and severity ratings in the report. Some programs will allow you to easily export or automatically create bug descriptions from the report.
Chauncey asserts that logging usability problems in formal bug tracking databases should be mandatory for usability specialists because it improves the visibility of usability to the entire corporation. Usability bug tracking also ensures that problems will not be lost in the chaos of development. Although he strongly advocates formal usability bugs tracking, he does not suggest that usability reports and separate usability databases can be eliminated.
Chauncey and Kara agree that, although slightly divergent, their two methods of tracking bugs can be complementary. Bugs from a usability database can be entered into the formal corporate bugs database, and links from the corporate database to usability reports can provide details on complicated usability problems. Whatever way works well for your organization, you should begin now to track usability issues. Usability reports, slide shows, highlight tapes, and walls of Post-it notes packed with usability problems are useful for your immediate project, but these methods of reporting problems are less persistent than entries in a bugs database. Whether we like it or not, many usability problems aren't fixed until several product versions later. Whichever kind of database you choose, tracking the unfixed bugs in a database provides the key to future success.
Chauncey E. Wilson
Bentley College Design and Usability Testing Center
175 Forest Drive
Waltham, MA 02452
+1 (781) 891-2608
Kara Pernice Coyne
Nielsen Normal Group
48921 Warm Springs Boulevard
Fremont, CA 94539
+1 (646) 613-1122
Chauncey Wilson is director of the Bentley College Design and Usability Testing Center. He has spent 20 years as an engineering psychologist, user interface designer, usability engineer, and product development manager. His academic background includes a physics degree and graduate training in social psychology, human factors engineering, and statistics. Chauncey has taught courses in user-centered design for the last 10 years. For relaxation, Chauncey collects cool electronic gadgets and Beanie Babies, a registered trademark of Ty, Inc.
Kara Pernice Coyne is a senior user experience specialist at Nielsen Norman Group. She has started and managed usability programs at Lotus/IBM, Iris, and Interleaf and is chair of the Usability Professionals' Association 2000 and 2001 conferences. Kara has an MBA from Northeastern University and a BA from Simmons College. She loves Brady Bunch trivia, and in her first and only time target shooting she connected 15 out of 15 times. Kara works in NY, NY.
Whiteboard Column Editor
Senior Principal Engineer
Computer Sciences Corporation
15245 Shady Grove Road
Rockville, MD 20850
©2001 ACM 1072-5220/01/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2001 ACM, Inc.