This column discusses two barriers to usability evaluation in the telecommunications industry, and a possible solution. The first problem is that carriers expect manufacturers to develop and market usable handsets, but manufacturers have little incentive to do user testing: It is carriers who provide the technical support and earn the incremental revenue from handset-based data sales. The second problem is the lack of cross-product/cross-company research that benefits both carriers and manufacturers; the competitive nature of the industry prevents cross-company cooperation.
The Structure of the Telecommunications Industry and Incentives for Usability. Let's consider the cost and benefits of usability testing. If carrier "X" has 20 million subscribers and five million of those subscribers buy a particular handset, say the "Venus," and most of those users have difficulty entering names into their handset-based phone books, the potential exposure for support calls is enormous. If five percent of the carrier's customers call just once, that's 250,000 calls costing, on average, $6.00 each (according to a client of mine at a U.S. carrier). The impact of poor usability adds up quickly: 250,000 calls could generate $1.5 million in costs. How much would usability testing and subsequent redesign cost? Maybe $100,000just seven percent of the potential damage.
Manufacturers' efforts are fragmented because they are spread across different mobile telephone models with different user interfaces, multiplying the costs of usability testing. Manufacturer "Y" sells 50 million mobile telephones each year, with 20 handset models world wide, but the models mostly differ by trim. In reality, they probably have four different active user interfaces. That is our first problem: Why design and support four different UIs? If only one UI were developed, only one-fourth the funds would be needed to conduct user testing, and the potential benefits of usability improvements would be shared across models.
The seemingly logical argument in favor of multiple user interfaces is actually a fallacy: They do not provide the customer with choice, since customers typically cannot try out a phone's UI before purchasing it. When one visits a mobile phone store, one must ask to try a live phone, since most often the demonstration models are inactive, with simulated displays. Live models simply get stolen. Even when live phones are available for trial, only a small subset of a store's models can be tried out. As a result, the usability of the UI doesn't highly influence the purchase decision, further weakening the case for usability testing.
The deeper problem has to do with the structure of the industry. The telecommunications industry in the U.S. has the unique challenge that the end user is the service carrier, not the handset user. A parallel in the computing world would be if end users purchased their PCs from Internet service providers. Since the "customer" is the carrier, but the end users experience the usability problems, the impact of poor usability on the manufacturer can appear to be negligible.
If we accept that the benefits of usability outweigh the costs, who pays to conduct the tests, and when should they occur? It would be most cost-effective for testing to occur before handsets are mass-produced, suggesting that the manufacturer conduct the user testing. However, it is the carrier who derives most of the benefits. The incentives just don't line up.
Is There a Way that Usability Would Benefit the Manufacturer? Carriers provide support for customers, so the old model of "reduce support costs" does not apply. Increasing revenue is the opportunity for the carrier, too. Increased voice and data do not benefit the manufacturer, except indirectly by increasing the likelihood that the carrier will buy more phones from the manufacturer whose phones generate the most revenue. Carriers can and do track these things, which is precisely why manufacturers need to worry about usability. If their products are found to be less usable, their customers, the carriers, will stop buying their products.
User testing by the manufacturer is an awkward selling point: They cannot market their handset to the carrier by saying, "we found 50 usability problems early in the design process," just like they cannot say, "our handset had a higher success rate in usability testing than competing handsets"no one would be likely to believe them. However, when a design problem gets released to the public, the carrier suffers immediately. The potential risk to the manufacturer is longer range: The manufacturer could suffer by eventually losing shelf space. All in all, manufacturers have less motivation to conduct user studies. However, the risk that a carrier will drop a manufacturer's handsets poses a high potential cost.
The Problem of Proprietary Research. Very little telecommunications-oriented usability research is published. The reason is simple: Usability research is very time-consuming and expensive to conduct, and when research is commissioned, clients want to keep it to themselves.
While keeping research proprietary is completely understandable, the irony is that it makes usability evaluation more expensive and less attractive. Each party has to conduct its own independent research and fund it independently, which inevitably duplicates efforts. Certain UI design challenges are inherent to the entire genre of mobile device user interfaces and functionality that are common in the industry.
Another Model for Funding Usability Testing. Another model for usability testing is for a trusted third party to conduct research and offer it for sale to multiple clients. My team has approached this strategy with success, though it has not been easy.
In early 2004, I was visiting a friend, and we were playing with his new mobile phone. I had never downloaded a ring tone before, and so we tried it out. After two lengthy calls to Support, we gave up. I later saw an article in The Economist's April 15, 2004 issue entitled "Ringing the Changes," where I first glimpsed statistics on ring-tone sales. In 2003, $3.5 billion in ring-tone sales were made worldwide, and yet only $80 million worth of ring tones were sold in the U.S. Was it due to poor usability? Incapable hardware? Lack of effective marketing? My team decided to develop a proof of concept to compare the usability of mobile phones for media download and installation.
For the next several months, we worked to build a data model that would enable us to analyze the results of 200 usability tests, 20 each for ten phones from all five major U.S. carriers. Through an iterative process to construct and refine our data logging and analysis, we built a solid data model that enabled us to visualize Time to Complete, Success Rates, Number of Attempts, and several sets of preference-based data. We plugged all our logging data into a database, built the charts, and completed the usability interviews day after day, week after week.
Our Media Download Usability study was our approach to helping the carriers and manufacturers make better decisions. Carriers who do not conduct usability research already could get a big jump on the competition by buying pre-packaged results. Manufacturers tend to conduct usability tests on only their own handset designsif they do it at allso our study provided them with significant extra benefit. The key challenge that remains is how to provide incentives to manufacturers to conduct user studies early in the design process. However, the fact that our study has sold well offers the promise that telephone models introduced in the next year will reflect the best practices that we presented. Our challenge is what to study next.
Usable Products Company
About the Author:
Scott Weiss is the principal of Usable Products Company, an ease-of-use agency that provides research and design services for interactive products. Scott's book, Handheld Usability (Wiley: 2002), is the definitive text on design, prototyping, and usability for mobile telephones and PDAs.
Susan Dray & David A. Siegel
Dray & Associates, Inc.
2007 Kenwood Parkway
Minneapolis, MN 55405, USA
©2005 ACM 1072-5220/05/0700 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2005 ACM, Inc.