Fresh: rant

XIII.6 November + December 2006
Page: 5
Digital Citation

Results are in

Jonathan Arnowitz, Elizabeth Dykstra-Erickson

back to top 

This issue is primarily about usability and metrics, measurements, and what we do with them. And thus our rant: deceiving with numbers.

At university you may have had the pleasure of a reading assignment instructing you to read and memorize a slim volume called How to Lie with Statistics [1]. This is a classic that presents the unimaginable—convincing people of something by manipulating unfamiliar concepts that sound scientific, and thus authoritative. One could, of course, achieve the same end via bullying or mesmerizing charisma. But lying with statistics is a venerable process engaged in by politicians, advertisers, and scientists—and even usability professionals.

One of the ingenious cheats of statistics is to "prove" that you are correct by accentuating the positive and eliminating the negative. To be a master of deception you do, of course, need tools. Not of the rabbit-and-the-magic-hat variety. A simple high-fidelity prototype can do the trick. We can call this type of deceiving with numbers High Fidelity Deception, and it goes something like this:

  • Reduce the overall feature set of a target product.
  • Design a prototype that covers some of the features.
  • Ensure that it is attractive enough to resemble a finished product with a dose of glamour: nice animations, transitions, professional color palette (for extra points, make it blue—you do know about that, right?), refined font, and upscale branding elements.
  • Show it to a few high-placed individuals who (this part is important) don't have the time or inclination to dive into the details.
  • Get their enthusiastic endorsement.
  • Proceed to formalize this into a final product regardless of the paucity of credible detail.

You will get excellent marks: high levels of user satisfaction, compliments on the prototype's refined sensibilities, and unequivocal approval to proceed. This prototype is, in fact, merely an attractive shell, but it passes muster, and you can quantify its success by repeating what percentage of users who have seen it were "wowed" by it (don't mention the actual number of users, though—after all, two out of two gives you 100 percent!).

Why does this happen?

A designer can impress stakeholders by producing an impressive-looking artifact. If it is, in fact, your product, but it isn't finished yet, you can call it a demo and all will be forgiven if your scripted click-through falls off track and your prototype crashes badly. Because it is lovely, it can even give the false sense of being usable.

Using early prototypes for usability evaluations is an important option for testing design concepts. However, if the fidelity of your prototype is too high, it may not be clear what you are really testing, and you risk gathering false positives on the concept. Sometimes the consequences are high stakes: You convince a customer that you are much further along than you actually are, and deadline negotiations for delivering the actual product reflect the customer's reasonable demand for early delivery while you struggle to meet impossibly close deadlines.

Is offering an unlovely collection of half-baked ideas to a user the only viable source of critique? One could say it's much better to go for a half-baked look without the bells and whistles, and simply test the concept with a low-fidelity prototype. It may not look nice, but it will result in a better product at the end of the day. But you can also do both: Show an attractive limited subset to illustrate the look and feel of the final product, and then get to work evaluating that drab gray collection of input fields and checkboxes.

In any case, the map is not the territory and the prototype is not the product. Confusing the two will result in an impossible entanglement that no quantitative data can unravel.—<eic>

back to top  References

1. Huff, D. (1954). How to Lie with Statistics. New York: W-W Norton and Company Inc.

back to top 

©2006 ACM  1072-5220/06/1100  $5.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.

Post Comment

No Comments Found