Authors: Rolf Molich
Posted: Wed, April 03, 2019 - 11:01:26
Should usability testers be certified?
Or rather: YES!
Let me ask two counter questions: Would you trust a doctor who was not certified? Would you want to fly with a pilot who was not properly certified?
Just like me, you hopefully want usability work, especially usability testing, to be taken seriously by your stakeholders, in particular your development team and management.
To be taken seriously, usability testers must insist on being measured. Usability testers must establish and follow generally accepted rules. Usability testing has matured to the extent where it should be regarded as a standard process, not a piece of art created by unruly artists. If you deviate from the standard process, you introduce unnecessary risks. Most likely, your management does not like unnecessary risks.
Deviations from established practice
Here are 10 deviations from established practice in usability testing that I often see:
- Test tasks are too simple
- Test tasks contain unintended clues
- Moderator helps the test participant too early
- Moderator explores the product together with the test participant
- Moderator manages the available time for the usability test session badly, for example by allowing the test participant to stray from the given task or by exceeding the time limit agreed with the test participant
- Moderator pays attention to test participants’ opinions rather than focusing on what they are actually able to accomplish
- Usability test reports include findings that are based on inspection rather than what the test participant did
- Usability test reports are unusable, because they are too long
- Usability test reports are unusable, because the most important findings are hard to find
- Usability test reports are unusable, because they are inconsistent, for example two reports written by the same person or by the same company that have widely differing formats.
My knowledge about these deviations is based on real-world data. The data comes from three sources:
- CUE studies
- Reviews of usability tests
- Usability test certification
CUE studies. Since 1998, I have conducted 10 Comparative Usability Evaluation studies, CUE , with more than 140 participating professional teams and a few motivated students. These studies have produced unique insights into how experienced UX professionals do usability testing.
In a CUE study, teams simultaneously and independently usability test the same product, most often a state-of-the art website. All of the teams are given the same test scenario and objectives for the test. Each team then conducts a usability test using their preferred procedures and techniques. After each team has completed its study, it submits the results in the form of an anonymous report. In a subsequent one-day workshop, all participants meet and discuss the reports, the differences, the reasons for the differences, and how to improve the test process. The differences are often stunning.
Most of the anonymous CUE reports are freely available
Reviews of usability tests. A few mature companies are relying so heavily on usability testing that they hire neutral experts to check the quality of usability tests carried out by their employees and subcontractors.
I have carried out such reviews. In many cases, the people that I reviewed did an excellent job. In some cases, I even learned important lessons from them. In other cases, subcontractors performed so badly that I recommended that they should either improve considerably or my client should find a more competent subcontractor.
Usability test certification. Here’s the approach to usability test certification that a candidate must follow if they want a CPUX-UT usability test certificate from our not-for-profit organization, the UXQB – User Experience Qualification Board:
- Obtain a foundation-level certificate (CPUX-F), where you prove your understanding of about 120 basic UX concepts like usability, user experience, contextual interview, persona, quantitative user requirement, prototyping, affordance, usability test plan, and usability testing.
- Study usability evaluation—that is, usability testing, inspection, and user surveys. The corresponding CPUX-UT curriculum is publicly available. Formal training is recommended but not required.
- Theoretical examination: Prove your knowledge of usability testing by answering 40 multiple-choice questions, each with six possible answers within 90 minutes. To pass, you must score at least 70%. Sample questions are available.
- Practical examination: Prove that you are able to conduct a simple usability test of a given website. You must recruit three usability test participants and write four usability test tasks for a simple website, for example, a weather website. You have seven days to carry out the usability test. The seven-day period can be placed whenever you have time. After the test has been completed, you submit raw videos of the three usability test sessions and your usability test report. The videos must show the screen and a picture of both the test participant and the moderator.
To pass, you must score at least 70%. The checklist used by the examiners for evaluating the practical examination is publicly available.
Pros and cons
Certification is not a shortcut to fame, wealth, and honor. Certification certainly will not make you a usability test expert in 10 days, no matter how hard you study and no matter how highly you score in the certification test.
Certification is cumbersome. Obtaining the CPUX-UT certificate takes up to 10 days: two-to-three days for the CPUX-F certificate, three days for the CPUX-UT training course including the theoretical certification test, and two-to-four days for the practical test.
On the pro side, curricula and certification are important steps toward a common UX language and a mature profession. Curricula define helpful activity models for usability testing, for example, Prepare utest, Conduct utest session, and Communicate findings; and Preparation of utest session, Briefing, Pre-session interview, Moderation, and Post-session interview. My research shows that these activity models are not known to all professionals. Such activity models help you understand when you deviate from established standards. They help the lone UX professional to understand: Am I doing it right? Another reason for certification is to let people who want to make usability testing their speciality demonstrate to themselves and to others that they are familiar with common knowledge in usability testing. Finally, certification shows that you care about keeping sharp in your profession.
Although many experienced professionals score highly on our certification test, we have seen interesting examples of experienced practitioners who had problems being certified. The most important deviations from established practice that caused problems for experienced practitioners are 1, 3, 5, 6, and 8 in the list above.
Is certification worth the effort? The real-world data I have shows that the practices of many professional usability testers need review, formalization, and a general tightening up. Since the teams I studied were professional, I suggest that everyone can benefit from having their practices reviewed.
Posted in: on Wed, April 03, 2019 - 11:01:26
View All Rolf Molich's Posts
@Bernard Rummel (2019 04 04)
Couldn’t agree more Rolf (how could I, since we’re working together on this!)
As for your opening statement - bad doctors and pilots can kill you. The consequences of bad usability tests are fortunately not quite as dire and direct. Nevertheless, I’d add three items to your list of blunders which can have a direct and severe negative impact:
1) data privacy violations,
2) violation of the participant’s dignity,
3) “burning” the participant base by inappropriate moderation behavior and tasks.
In our practical certification examinations, I have seen examples for all three, for instance: a moderator leaving the video recording on after the session, recording smalltalk about the participant’s wife’s sickness (1); failing to debrief a clearly distressed participant after task failure and explaining the solution instead (2); tasks unnecessarily asking the participant to adopt a political opinion (3).
Certification cannot prevent such issues, but we have at least an opportunity to test the testers. What’s even better, we have an opportunity to collect a body of best practices and common difficulties, and thereby gradually raise the professional standards of our trade. So is it’s definitely worth the effort - not only to get a certificate, but also to build certification programs.
Looking forward to further working with you on this,
@David Travis (2019 04 05)
Rolf knows more about usability testing than most people in UX. So I think we should take seriously his suggestion that usability testing needs to be standardised: practitioners “must establish and follow generally accepted rules”. Certification is certainly one way to achieve this and I agree it makes sense to insist that usability testing consulting firms should be certified. This will help clients appreciate the professionalism that’s needed and ensure they get a standard service.
But certification will have a revenge effect. Usability testing is a user centred design gateway drug for organisations with low UX maturity. Insisting that everyone that runs a usability test must be certified will put a barrier in their way. This could discourage these firms from ever trying out usability testing. At the very least, it will lead to these firms delegating their usability testing to consulting firms with the appropriate certification. And that means the (uncertified) development team are less likely to plan the test, less likely to observe the test and less likely to act on the results.
Certification will establish a minimum *quality* level for usability testing. But it will inevitably reduce the *quantity* of usability tests that are carried out. So the question then becomes: is a low quality usability test harmful? A low quality usability test at least has the benefit of getting the development team exposed to users. My experience is that exposure to users leads to empathy, which leads to more user research, which leads to firms growing in UX maturity. If we allow only certified people to run usability tests, I suspect that the real revenge effect will be that certification creates a world with a lower net usability than if we let anyone do it.
David Travis @userfocus