ACM Interactions

Should usability testers be certified?

Yes. 

Or rather: YES!

Let me ask two counter questions: Would you trust a doctor who was not certified? Would you want to fly with a pilot who was not properly certified?

Just like me, you hopefully want usability work, especially usability testing, to be taken seriously by your stakeholders, in particular your development team and management.

To be taken seriously, usability testers must insist on being measured. Usability testers must establish and follow generally accepted rules. Usability testing has matured to the extent where it should be regarded as a standard process, not a piece of art created by unruly artists. If you deviate from the standard process, you introduce unnecessary risks. Most likely, your management does not like unnecessary risks.

Deviations from established practice

Here are 10 deviations from established practice in usability testing that I often see:

1. Test tasks are too simple
2. Test tasks contain unintended clues
3. Moderator helps the test participant too early
4. Moderator explores the product together with the test participant
5. Moderator manages the available time for the usability test session badly, for example by allowing the test participant to stray from the given task or by exceeding the time limit agreed with the test participant
6. Moderator pays attention to test participants’ opinions rather than focusing on what they are actually able to accomplish
7. Usability test reports include findings that are based on inspection rather than what the test participant did
8. Usability test reports are unusable, because they are too long
9. Usability test reports are unusable, because the most important findings are hard to find
10. Usability test reports are unusable, because they are inconsistent, for example two reports written by the same person or by the same company that have widely differing formats.

Real-world data

My knowledge about these deviations is based on real-world data. The data comes from three sources:

• CUE studies
• Reviews of usability tests
• Usability test certification

CUE studies. Since 1998, I have conducted 10 Comparative Usability Evaluation studies, CUE [1], with more than 140 participating professional teams and a few motivated students. These studies have produced unique insights into how experienced UX professionals do usability testing.

In a CUE study, teams simultaneously and independently usability test the same product, most often a state-of-the art website. All of the teams are given the same test scenario and objectives for the test. Each team then conducts a usability test using their preferred procedures and techniques. After each team has completed its study, it submits the results in the form of an anonymous report. In a subsequent one-day workshop, all participants meet and discuss the reports, the differences, the reasons for the differences, and how to improve the test process. The differences are often stunning.

Most of the anonymous CUE reports are freely available

Reviews of usability tests. A few mature companies are relying so heavily on usability testing that they hire neutral experts to check the quality of usability tests carried out by their employees and subcontractors.

I have carried out such reviews. In many cases, the people that I reviewed did an excellent job. In some cases, I even learned important lessons from them. In other cases, subcontractors performed so badly that I recommended that they should either improve considerably or my client should find a more competent subcontractor.

Usability test certification. Here’s the approach to usability test certification that a candidate must follow if they want a CPUX-UT usability test certificate from our not-for-profit organization, the UXQB – User Experience Qualification Board:

• Obtain a foundation-level certificate (CPUX-F), where you prove your understanding of about 120 basic UX concepts like usability, user experience, contextual interview, persona, quantitative user requirement, prototyping, affordance, usability test plan, and usability testing.
  
  
• Study usability evaluation—that is, usability testing, inspection, and user surveys. The corresponding CPUX-UT curriculum is publicly available. Formal training is recommended but not required.
  
  
• Theoretical examination: Prove your knowledge of usability testing by answering 40 multiple-choice questions, each with six possible answers within 90 minutes. To pass, you must score at least 70%. Sample questions are available.
  
  
• Practical examination: Prove that you are able to conduct a simple usability test of a given website. You must recruit three usability test participants and write four usability test tasks for a simple website, for example, a weather website. You have seven days to carry out the usability test. The seven-day period can be placed whenever you have time. After the test has been completed, you submit raw videos of the three usability test sessions and your usability test report. The videos must show the screen and a picture of both the test participant and the moderator.
  
  To pass, you must score at least 70%. The checklist used by the examiners for evaluating the practical examination is publicly available.

Pros and cons 

Certification is not a shortcut to fame, wealth, and honor. Certification certainly will not make you a usability test expert in 10 days, no matter how hard you study and no matter how highly you score in the certification test.

Certification is cumbersome. Obtaining the CPUX-UT certificate takes up to 10 days: two-to-three days for the CPUX-F certificate, three days for the CPUX-UT training course including the theoretical certification test, and two-to-four days for the practical test.

On the pro side, curricula and certification are important steps toward a common UX language and a mature profession. Curricula define helpful activity models for usability testing, for example, Prepare utest, Conduct utest session, and Communicate findings; and Preparation of utest session, Briefing, Pre-session interview, Moderation, and Post-session interview. My research shows that these activity models are not known to all professionals. Such activity models help you understand when you deviate from established standards. They help the lone UX professional to understand: Am I doing it right? Another reason for certification is to let people who want to make usability testing their speciality demonstrate to themselves and to others that they are familiar with common knowledge in usability testing. Finally, certification shows that you care about keeping sharp in your profession.

Although many experienced professionals score highly on our certification test, we have seen interesting examples of experienced practitioners who had problems being certified. The most important deviations from established practice that caused problems for experienced practitioners are 1, 3, 5, 6, and 8 in the list above.

Is certification worth the effort? The real-world data I have shows that the practices of many professional usability testers need review, formalization, and a general tightening up. Since the teams I studied were professional, I suggest that everyone can benefit from having their practices reviewed.

Rolf Molich