I have always been skeptical of statistics. Maybe it’s just fear of the unknownmy brain is demonstrably not wired for math. But everyone knows that statistics can be made to lie, and can be tweaked and manipulated and interpreted to mean nearly anything, to support or justify nearly any position or goal. I’m even more skeptical of statistical methods applied to measurements of what are arguably qualitative factors. So why should statistics about usability or user-experience design or HCI be any different?
There has been plenty of discussion over recent years about how many users it takes to produce valid conclusions about usability. Despite my ignorance about statistical methods, I have no trouble understanding that you will get more valid information from tests with dozens or hundreds of participants than you will by having three to five users. I can see the value of small test samples during iterative development, when the goal is to locate any glaring usability issues. But you can’t convince me that such a test will reveal all the issues that a larger, more comprehensive test will find [5, 7].
Last year I had the opportunity to participate in an evaluation of some interface designs. The team agreed to quantify a set of heuristics and apply them to the designs, to gain some insight into which design might be best suited for use in a product. The evaluation was a dismal failure. As it turned out, each of the four participants applied the heuristics unevenly. We recorded our data in an unstable tool that provided inconclusive results, even after statistical analysis. The final report glossed over the statistical findings and focused instead on one team member’s qualitative evaluation of the interfaces’ appearance and features, with no reference to the quantitative values collected and analyzed. So much for quantifying heuristics.
Watching the recent Winter Olympics, I was reminded again of my pet peeve about sports: the problem of measuring beauty, of quantifying subjective evaluations. While I enjoy watching attractive people dancing on ice to beautiful music, I have trouble thinking of it as a sport equivalent to downhill ski racing or speed skating. The fastest skier or skater wins; that’s pretty simple. But judging beauty and calling it sport? That’s problematic. At least the scoring system for figure skating was changed after the scandal at the 2002 Olympics. But the new system still includes judging components for interpretation, choreography, and performance.
Does HCI have a similar problem? Are we measuring the unmeasurable? Can we seriously apply Six Sigma analytics on qualitative factors? Can Six Sigma provide indications that we can put the same confidence in qualitative judgments as in quantitative?
Don Norman has convinced me that emotion is an important factor in successful product design . Yet I suspect most measurement occurs on behavioral design, only one of Norman’s three aspects of emotional design. How do we measure joy? How about measuring fun? Does funology have metrics ? The name suggests that it’s the study of fun; how is fun measured? Is there a pleasure chart for emotional design, much like the pain-quantifying charts used in today’s hospitals? How do you quantify the pleasure of first opening your iPod’s packaging, or holding the gadget in your hands for the first time?
If we asked users to quantify their pleasure on a ten-point scale, how many would insist on "turning it up to 11?" (In the 1984 satirical film This Is Spinal Tap, one of the musicians famously, and naively, insisted that the volume knob on his amplifier go up to 11 so that it would be louder than those that stopped at ten.)
And what of the old management-theory adage that you manage what you measure, or the equally important corollary that employees (or designers?) pay attention only to what’s being measured (and thereby that for which they are rewarded or punished)? "What gets measured is what gets done" . If you can’t measure it, can you design for it? Will users find what you measure in usability tests to be important to them just because you measured it? Or will designers consider a factor important only if you measure it, or only if usability gurus make pronouncements about it because they managed to measure it? Are there very real but really unmeasurable qualities and factors that affect user experience, that don’t or can’t have metrics associated with them?
We know that when schools and teachers are evaluated on the basis of improvements in standardized test scores, they start teaching what the tests measure. Many educators feel, however, that such tests have the wrong effect: They’re producing successful test takers, not successful students. If you can’t measure how students learn to think for themselves, perhaps they won’t learn to think.
Be careful what you measure: It may be what you get.
On the other hand, I am gratified to see that Stanford researcher B.J. Fogg and colleagues, when evaluating the credibility of Web sites, count user responses but do not try to quantify aspects such as "design look" or "information focus" . Credible Web sites look good, just as we know that tall, attractive people are somehow seen as more credible than the rest of us. See, for instance, Malcolm Gladwell on the predominance of tall CEOs and the presidential good-looks (and dismal performance) of President Warren Harding . But please don’t insult me further by trying to quantify looking good; we all know that there are no tens. (Blake Edwards’ 1979 film 10 starred Bo Derek, whose beauty presumably qualified her as a ten on a scale of one to ten. Writer Shel Silverstein subsequently opined in song that "on a scale of ten to one, friend… there ain’t no tens.")
The September/October issue of <interactions> will focus on how we measure HCI and usability. Contact guest editor Jeff Sauro at firstname.lastname@example.org if you have something to say about measuring usability. Here’s your chance to convince me.
2. Fogg, B.J., Soohoo, C., Danielson, D., Marable, L., Stanford, J., Tauber, E. (2003). How Do Users Evaluate the Credibility of Web Sites? Proceedings of the 2003 conference on Designing for user experiences.
©2006 ACM 1072-5220/06/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.