Joseph Dumas, Rolf Molich, Robin Jeffries
Do you often feel like your usability reports aren't taken seriously? Perhaps part of the problem is that engineers have trouble acting on your problem descriptions. Writing effective, usable descriptions is a skill that all usability professionals should have mastered.
Evidence that we sometimes miss the mark when we describe usability problems and solutions comes from an examination of the usability comments from the fourth Comparative Usability Evaluation (CUE-4). In CUE-4, 17 teams of experienced usability specialists independently conducted either an expert review or a usability test of the reservation Web site for the Hotel Pennsylvania in New York and wrote a report of the evaluation.
Each of the team reports contains a table of usability comments describing the usability issues that the teams found in their evaluation and in many cases an accompanying recommendation for fixing the problem. This database of issues contains 647 individual comments. While many of these comments are competent and professional, there are enough that are not to make us consider whether our communication style as usability professionals could use some polish.1
In this article, we look at some of these comments in the spirit of self-examination. As usability specialists we don't often get to see the comments of our colleagues and we don't often receive constructive criticism of our own comments. In CUE-4, the 17 teams knew that they were writing their reports to the development team, a team that they did not know. They also knew that their report was the only way to communicate their comments. This situation is similar to a consulting arrangement in which usability specialists do not have the opportunity to explain their comments and negotiate their recommendations after the report is delivered.
We have grouped our selection of comments into the following sections:
- Emphasize the positive.
- Express your annoyance tactfully.
- Avoid usability jargon.
- Be as specific as you can.
We end with some advice for communicating more effectively.
We all have been taught to find good things to say about an interface in addition to our criticisms of it. But the teams seemed reluctant to do so. One of the requirements of CUE-4 was for the teams to include positive comments on the usability of the site in the executive summary. In addition, there was a code assigned for positive comments. In a technology that has been adopted by hundreds of hotels, one would imagine that there must be many positive issues to bring up. But the average number of positive comments was only 4.7 per team, less than 15 percent. Here is an example. [The values in brackets refer to the team letter and the comment number from the CUE-4 reports]:
- Reservation invites users to explore different possibilities.
- Test participants appreciated the possibility to explore all available options at once. They told us that this was the kind of feature they had been waiting for. They were ready to forgive most of the shortcomings of the site because of the well-functioning reservation system. [O-06]
Would our recommendations be more palatable if we were more positive? The answer may depend on our audience and how well we know them. But in the CUE-4 case, the teams did not know the developers. While we recognize that the primary purpose of a usability evaluation is to find problems, the reception those problems receive from developers may be enhanced by our recognition of what they have done well.
In several cases, even the positive comments have a negative one appended.
- Positive: The "Cancel" process is simple, although the link to cancel is buried at the bottom of the page. [P-24]
- Internationalization: International versions of the "About the Hotel" page available but for room reservations you need to go back to the English version. [O-09]
Our recommendation: Make positive comments purely positive. Put any reservations you might have in a separate comment. A negative remark at the end of a positive comment leaves a bitter aftertaste and destroys some of the psychological effects of the positive comment.
In reading through the comments, some issues elicited strong emotional reactions from certain teams, though the problems were not exactly "show stoppers"; that is, they did not result in loss of data or task failure.
One of the issues that provoked the strongest reaction was the presence of typos and/or grammatically incorrect sentences. Here are some examples:
- Muddled text.
- This nonsense text appears in the General Terms and Conditions: "For all reservations guarantee to valid form payment is required at booking."[C-21]
Here the words "muddled" and "nonsense" convey the annoyance of the expert, but what do they communicate? Here is another comment from an expert reviewer who does not hesitate to be blunt:
- User prompts and instructions: poor grammar makes an impression
- On the second panel, the prompt "Click a room from the list below will show its availability on the calendar" makes it sound like someone with English as a second language did the screen design and gives a poor impression of the hotel. REC: Check for proper grammar in all information. [D-23]
While the expert may be correct about the negative impression poor grammar makes, some developers may feel insulted by this comment. There are at least two points-of-view about this type of comment:
- Developers need to be told that that grammatically incorrect text is unacceptable. Being blunt helps convey this message in a way that calling it a typo would not.
- Developers will feel threatened by suggestions that they do not know the language. Being blunt is not helpful; it is simply rude.
Another design practice in this site that sparked an emotional response from the evaluators was the use of underscores in the names of types of rooms. The assumption of the experts in this case seemed to be that the developers were either sloppy or lazy by letting the end-users see the underscores. They took it to mean the developers did not care about their users. Here is a comment from a tester:
- Unprofessional text presentation showing 'coding' type text
- The area on the reservations page with the room names has underscores between the words (e.g. Superior_King_Bed). This was noted by half the participants and although would not stop their using it, it was seen as unprofessional and sloppy, one person saying it seemed as if it was coding exposed. Suggestion: Remove the underscores, include spaces between words. [A-19]
Notice the use of the word, "unprofessional" here, though the evaluator observes that the problem did not detract the user from making a reservation. Here is another comment on the same issue:
- Besides, the nerdy way of writing the selections make them look like variable names in a C-program (apparently Flash code). [O-25]
Does "nerdy" here step over the bounds of civility? Why do these issues provoke some of our strongest language?
Here is a more neutral approach:
- Problem: Underlying system visibility
- Description: The underscores for the room descriptions are not necessary. This seems like an implementation detail that should not show up on the user interface.
- Recommendation: Remove the underscores. [B-02]
It was interesting to notice that emotionally charged comments were used regardless of the evaluation methodbe it an expert review or a usability test. In some cases, the usability testers admitted the issue was not a major one for participants. Are the usability specialists expressing their own agendas here, but hiding behind both real and imaginary users to emphasize their point?
Clear communication in a usability evaluation requires speaking in the readers' language. Many of the teams' comments contained terms and concepts that only other usability professionals were likely to understand. Furthermore, the comments often included a rule or heuristic that confused rather than clarified the comment.
Here are some examples:
- Form & function imbalance. Rest of the site looks like it was designed by another team. Reservation page works better than it looks like. Recommendation: Graphical design of the website should also be consistent, let a usability consultant to evaluate the site. [O-36]
It is not clear what "Form and function imbalance" adds to this issue. Note also the backhand complement, "Reservation page works better than it looks like."
The following comment is from an expert review:
- Help and documentation
- Help is conceptual, not contextual. The same help appears whenever the user clicks on it during the process. Users may not see the scroll bar to scroll through help. [F-05]
This comment includes three separate issues. Will developers understand the distinction between conceptual and contextual help? The implication seems to be that conceptual help is always bad.
Another comment with a rule appended:
- Aesthetic and minimalist design.
- Reservation confirmation page (table): The heavy lines in the table make the borders stand out as much as the text. Lighten the lines or eliminate them. [B-07]
Here the problem and solution are clear but what information does the header add to the comment for a developer? If the usability specialist is trying to convey a heuristic here, stating it in plain English would be more effective.
In our comments to developers, we frequently accuse them of using technical jargon. Do usability specialists sometimes do the same to them?
Usability specialists were frequently vague in their comments. By "vagueness" we mean that developers would not necessarily understand the problem being addressed and/or the fix being proposed.
A typical problem that produced some of the least specific comments was in the use of colors and fonts. It is ironic, too, because it is precisely this kind of issue that developers assume usability specialists are experts in, often acknowledging our ability to "make things pretty." They may, as a consequence, feel that we have failed them when we offer only general solutions. Here is a comment on this issue from a usability tester with a recommendation:
- Reservation area text and background colors difficult to read
- One of the first things users noticed on entering the reservation area was the black text on the dark green background, which they found hard to read. The same color combination was used for the message 'Book your Reservation Today at the Hotel Penn!' in the top right of the reservations area, and several commented on the difficulty that people with poor eyesight would have. There were also comments on the difficulty in reading the text which was generally seen as too small, particularly in the Help area of reservation. Suggestion: Use a color with a better contrast to the background and increase the font clarity. [A-16]
The evaluator raises several issues about color and font in this comment, but stops short of making a recommendation. In the problem statement, he mentions font-size, but barely states more than the obvious with the suggestion: "increase the font clarity." Would the developer who created this site be able to make better choices from these suggestions?
Seven teams used screen shots to illustrate problems. Here is a comment with a screen shot that clarifies an issue, but notice how the recommended fix is still left to the developer's judgment:
- The contrast between text and background colors on the Reservations page is not sufficient (e.g., black text on green background, black text on dark tan background and red text on green background). Sixty percent (9/15) of the participants commented about reading difficulty on the Reservations page due to low contrast. We also observed many participants leaning closer to the screen to distinguish text on the interface.
- Recommendation: Change the colors of the reservation page to ensure high contrast between the text and the background color. [M-15]
In defense of the usability teams in CUE-4 some of the contrast problems were caused by the fact that the site used branding colorsblack text on a brown background. Usability specialists might have been reluctant to make specific suggestions without talking to developers. But by including such a disclaimer usability specialists would be filling in useful information, as the developers have already shown that they may not understand what "high contrast" means.
As we mentioned earlier, usability specialists often don't receive feedback about the effectiveness of our work. When we work in a consulting context in which a final report of a review or a test may be our only way to communicate issues and recommendations, we are often left out of the loop when it comes to knowing how or if our suggestions were implemented. In an internal organizational context, we may wonder why some of the problems are not fixed or not fixed properly. As the examples above show, one of the causes of these omissions may be our communication style and the attitude toward developers that our comments and recommendations reveal.
As usability professionals with the goal of being more effective in our work, we need to be more positive, clear, precise, and respectful in our communications about problems and solutions.
1. Molich, R., Bevan, N., Curson, I., Butler, S., Kindlund, E., Miller, D., & Kirakowski, J., (1998), Comparative Evaluation of usability tests, Paper presented at Usability Professionals' Association Annual Meeting.
Joseph S. Dumas, Ph.D.
Robin Jeffries, Ph.D.
Sun Microsystems, Inc
1All of the CUE studies are described at www.dialogdesign.dk/cue.html and the results of the first two studies have been previously published (Molich, et at, 1998, 2004). The technical reports of CUE-4 will appear shortly. The anonymous reports from each of the teams who participated in CUE-4 are at www.dialogdesign.dk/ tekster/cue4/all_cue4_reports.pdf
©2004 ACM 1072-5220/04/0700 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2004 ACM, Inc.