At CHI 2005, Eric Schaeffer, president of Human Factors International (HFI) who works from Mumbai, India, and Jared Spool, president of User Interface Engineering (UIE) in Middleton, Massachusetts, invited me to moderate their debate about usability. Approximately 700 people witnessed a "wrestling match" between the Maharajah of Mumbai and the Motormouth of Middleton. Each is a knowledgeable, passionate, and articulate professional. Eric has published his step-by-step guide Institutionalization of Usability (Addison-Wesley, 2004), and Jared runs a well-known user-interface tutorial series nationwide. I shall summarize the three segments of their debate.
Can usability groups get larger? Is larger better? Jared acknowledged many resources "proving" return on investment (ROI) for usability. However, citing a UIE study for the Gap's Web site, he showed that sometimes professionals don't know how to explain what's happening, which usability issues are most significant, or why things improved. He also noted that many large software companies have hundreds, even thousands, of usability professionals, but they do not necessarily have comparable increases in usability of their products when compared to smaller companies with smaller teams, but with extremely usable products like those of Google or Yahoo. His point: Investing more people and money may not bring about proportionally greater usability.
Eric pointed to our being in a transition to the "third wave of the Information Age." Anomalies may exist, but proof is available, e.g., in a Nielsen Norman Group report about the ROI of usability. He asserted: We know how to do it, we can do it at the larger scales needed for global enterprise software solutions, and we don't have a choice about adopting these practices. Fortunately, he feels that his own company's evolved practices demonstrate his precise, reproducible method for achieving usability.
The audience questioned some of Jared's numbers. Jared countered that there is simply insufficient careful proof of the connection between usability and business success. He even conjectured that usability might not be crucial at all, in comparison to money spent and earned when considering marketing and other costs. He pointed out that usability experts cannot give guarantees about the bottom-line effectiveness of their services. Others questioned Eric's notion of a "usability factory," as opposed to "Mom and Pop" groups today. The factory concept didn't sound appealing. Eric admitted it might not be the best term, but likened it to the model of information technology (IT); it expressed the larger-scale success of enterprise professionals, not village craftspeople.
How is usability to be accomplished at a global scale to meet development needs? Eric mentioned a Jakob Nielsen pronouncement that India required 60,000 usability experts to cope with their software development. Eric asked: How can this be accomplished without a factory process? He believes all jobs will not go to India, but a "global harmony" can emerge that puts locals together with global resources efficiently with regard to time and task sharing. He has put theory into practice at the HFI Mumbai, India, office, in which a majority of HFI's staff work at greatly reduced costs of resources, but also according to systematic methods of work.
Jared waved away the offshoring issue and pointed again to the difficulty of predicting results with usability groups. He pointed to an experiment involving seven usability teams, each of which found different, crucial problems in a usability review. Would the client have had to purchase the services of all seven groups to be sure that all significant errors were discovered? Jared challenged the audience: Are we a crafts group or an engineering profession? Why do different consultants get different answers? If we have such variability, what are we actually offshoring?
The audience challenged Eric with how one really could do usability evaluations that required deep understanding of local context. Eric countered that HFI had proved it could do all of its work from India, but admitted that best practice was a mix of Indian and local professionals, which could only work with a systematic usability process, not the craftsperson's methods. Another audience member challenged Jared's description of many professionals as craftspeople dependent upon their portfolio of past experiences by pointing out that many firms had sufficient depth in their staff skills that people could be swapped in or out of a project.
Continuing on his theme, Jared asked: Are usability professionals cobblers or engineers? Jared used the examples of graphic designers who rely on their portfolio vs. radiologists who have established basic knowledge required for practice. (No one mentioned the issue of "license to practice," but it was just around the corner.) What rankled Jared is that usability professionals seem to sell their services as engineers but actually conduct their practice like craftspeople. He faulted organizations like ACM/SIGCHI and Usability Professionals Association (UPA) for not addressing this issue clearly enough. He asked: Where does either organization discuss building a portfolio and how to sell services; or how and where to state the basic requirements of knowledge to be a professional?
Eric responded that we formerly sold our services as craftspeople, but we are shifting. He sees the need for certification, which the major organizations do not choose to address, and which software vendors cover only in terms of "certified tool users." For that reason, HFI decided to develop a certification program regarding its process and now has 272 certified professionals.
An audience member chided Jared for making a false distinction: Multi-layered professionals have many structures, terms, and processes. Hiring the best people means that a company can shift people or roles without a conflict between craft-oriented professionals and process-oriented engineers. In addition, CHI, UPA, and other groups have published some basic methods. Jared countered with the challenge: Do you sell a person or a process, an individual or a result? Can we really substitute one person for another and expect to get the same results? Clearly, he expected "no" to this last rhetorical question.
Another audience member critiqued Jared's implication that all engineers are equal; some of what they do is a craft, as when they are developing high-level designs. Jared claimed there was a qualitative difference in variability of results that made usability professionals seem far from engineers. What he demanded was honesty about how the profession sold its services. He used the example of magicians. They know their practices are "fake," and so do their audiences. At their conferences, they discuss techniques for putting on a show, an illusion. Perhaps this, he mused, should be SIGCHI's and UPA's approach to the profession: maintaining its ability to perform magic. Eric closed with the counter argument that many other fields have gone through the transition in which he now sees the usability practitioner, from the days of selling magic, to the era of selling a process. For him, the systematic, process-oriented method is the right road to travel.
After the session, many acknowledged that, even with the shouting (from at least one of the speakers and some of the audience members), the debate was a highlight of CHI 2005. Others commented that it seemed as hard to get two great egos to tango as it may be to scale up usability. Some may complain that definitions and facts were given some "spin." Nevertheless, Eric and Jared addressed vital issues facing the profession today. Everyone was stimulated to think hard about where he/she stood on these issues that confront us today, issues that won't go away anytime soon.
About the Author:
Aaron Marcus is the founder and president of Aaron Marcus and Associates, Inc. (AM+A). He has degrees from both Princeton University and Yale University, in physics and graphic design, respectively. Mr. Marcus has published, lectured, tutored, and consulted internationally for more than 30 years.
©2005 ACM 1072-5220/05/0700 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2005 ACM, Inc.