Business

XI.5 September + October 2004
Page: 22
Digital Citation

The myths of usability ROI


Authors:
Daniel Rosenberg

Why has this never been necessary? Have I just been lucky in my choice of employers? Did these companies all have CEOs so enlightened about usability that no such analysis was necessary? I suspect not. Rather, I believe that the ROI analyses we are reading in the literature just don’t fit the real world, at least not the world I live in. Of course, like most practitioners in the HCI community, I define my professional goal as adding value to products by improving the user experience. However, the traditional ROI approach to defining and measuring the value of usability has never seemed to me to be an effective way of showing this added value. It is not in sync with the needs of the commercial software industry, and in my experience these metrics possess little or no appeal to upper management.

My current perspective is that of the enterprise software industry where large complex software suites are created for use by companies worldwide, both large and small, to handle financial, manufacturing, and sales data. Development teams in this segment of the industry employ thousands of programmers and their support and sales operations are similarly large and complex. Each individual sale of a system may account for millions of dollars in revenue, and each installation may be deployed to thousands of concurrent users with a wide range of skills and business roles.

The scale and requirements of the enterprise software business are very different from those of internal IT projects or typical consulting engagements. Yet it is from the latter domains that the usability ROI literature has emerged. Perhaps it is the shorter project time frames and/or internal corporate focus in these areas that leads to the current approach. In my experience, this approach, unfortunately, is not directly applicable to the largest segment of the software industry, where annual revenues are estimated at over $100 billion [2].

Although I acknowledge that classic ROI arguments may carry weight in some areas of the industry, our profession seems to be overly focused on these analyses. They have become what I would classify as generally accepted "myths," frequently over-quoted without critical review of their specific applicability. A side effect of this overuse is that it can also contribute to a general lack of credibility for the entire profession, which we hear about year after year, in this and other UE publications.

Before I suggest an alternative approach let’s look at some of the most prevalent ROI myths, where they come from, and how they propagate.

Myth #1. There is a lot of empirical data supporting the ROI claims for usability

Contrary to this myth, it looks like we are treating claims—that go well beyond the existing data—as if they were truisms. The fundamental rationale behind usability ROI calculations begins with the common sense thesis that good usability improves specific business metrics and that poor usability increases specific costs. The main categories of claimed impact are listed in Table 1.

Various studies have attempted to financially quantify a limited set of these variables and contrast them to the amount of money spent on UE activities within that particular case study. The most stringent studies would of course provide a before and after comparison. While there are many published articles, the number of empirical case studies with original and detailed financial data is only about a dozen. These empirical studies tend to be older ones, such as those contained in the landmark book Cost Justifying Usability [1]. This book did an excellent job opening the door to this complex topic for the profession. Unfortunately there has not been a significant number of rigorous follow up studies published recently. Rather, the same old data tend to be cited over and over.

A comprehensive online catalog of recent articles from a range of sources was created by Rashimi Sinha [9]. Yet of the 43 articles listed on this site only one provides new empirical data. And that specific article concluded that the "study failed in its intent to document a persuasive ROI." [6]

Whether more data even exist is an interesting question. My own experience publishing a book chapter on the financial cost/benefit for a common user interface standards project at Eastman Kodak in the late 80’s suggested that corporate legal departments have a tendency to block the publication of research that contains development cost data [8].

Myth #2. ROI calculation from the producer’s perspective is sufficient

Another typical line of discussion promises cost savings and schedule reductions as typified by the following quote:

A study of software engineering cost estimates showed that 63% of large software projects significantly overran their estimates ... four reasons rated as having the highest responsibility were related to usability engineering. Proper usability engineering methodology will prevent most such problems and thus substantially reduce cost overruns in software projects [5].

ROI calculations in published studies are based most often on cases where the cost and benefit for a given project can be measured internally. While this approach can be accurate and valuable, it is not representative of an industry where the cost to the user can far exceed the cost to purchase the product. These costs include such things as installation, training, and support costs. More importantly, this additional cost does not show up in the producers’ balance sheet, so a usability-driven reduction of the user’s costs may not be much of a priority for the producer. In fact, quite the opposite can occur in industries that have a paid support model, including the entire Open Source movement.

Myth #3. ROI claims can ignore other contributing factors and overgeneralize

Another type of claim we often see in the ROI literature is the following:

Revenues for one DEC product that was developed using UCD techniques increased 80% for the new version ... and usability was cited as the second most significant [improvement.] [10]

What is missing in a statement like this is an explanation of what the primary factor was, since usability was cited as the second most significant factor. What about the many other things that were going on at the same time in an effort to improve revenues? For example, the other factors could have been a change in price, a doubling in size of the sales force, a particularly successful marketing campaign, an increase in the general market for this type of product, bad press for a competitor, etc. Obviously another data point that would be relevant is exactly how much of the 80 percent growth could be directly attributable to the UCD process. Of course, corporate legal departments may block disclosure of the ancillary data that are needed to fully interpret an ROI claim like this. However, the lack of these details calls into question the utility of the claim as a justification for increased UCD expenditure. Even with additional data, this is likely to remain a flaw in usability ROI analysis because in the real world it is so hard to isolate the specific contribution of any one product attribute, such as usability, to commercial success.

Myth #4. Comparisons to analogous domains are not required

In a 1998 Alertbox, Jakob Nielsen, citing a Forrester Research report, stated that the cost of bad Web design is the "Loss of approximately 50% of potential sales from the site as people can’t find stuff."[4]

In this quote there is an obvious error of omission. What is the relevant comparison to the real world that will allow us to determine whether 50 percent is a good or bad number? How often do users leave bricks and mortar stores without purchasing because they can’t find what they want, don’t know what they want, or can’t locate a knowledgeable sales person to help them? Certainly, this is a pretty typical shopping experience. Furthermore, in the brick and mortar world, it is assumed that people will look many times before buying, so entering a store and leaving without purchasing is not automatically assumed to represent a failure.

Myth #5. Executives can be fooled by voodoo economics

Consider that there are one billion users on the Internet. Assume that only half of them come to your site, but, of these, 80 percent leave without buying. If the average cost of an abandoned shopping cart is $20 you will lose $8 billion a year in sales. With a $5,000 heuristic review, you could reduce abandoned shopping carts by at least 50 percent, thus increasing annual revenues by $4 billion, an ROI of 80,000,000 percent.

While this is obviously a parody, it is not that different from the overly creative claims that permeate the ROI online discussion: it is typical of the rhetoric surrounding the role of usability in electronic commerce. Just take a look at the 96 links Google returns to UI consulting Web sites when you search on "Usability ROI" for real examples of claims like the one above. No MBA schooled in the mathematics of market share, product branding, and merchandise positioning would ever take this kind of analysis seriously.

Myth #6. Decision makers will be so impressed by usability ROI, that they won’t weigh it against other investments

The usability ROI literature typically treats the cost/benefit tradeoffs for usability in a vacuum. It does not take into account the opportunity costs of investing funds in usability as opposed to other places where they might have an even bigger impact. This is a major disconnect with the way business decision makers have to think. It is often the case that automation or other technology changes can reduce complexity and/or increase revenues more dramatically than an improved UI design can.

For example, consider the home networking market. The introduction of the DHCP protocol made it possible to network a computer without going through the arduous process of manually configuring an IP address and domain name server location. This allowed individuals with less technical skill to install a computer on an existing network. This is certainly a more fundamental product improvement than any user interface solution for entering an IP address can provide. Yet from a traditional usability ROI perspective, how would you measure this, since no formal or discreet UI effort was applied, as we would classically define it? When they weigh ROI arguments for funds to support discreet UI usability activities, executives have to weigh these against potential order of magnitude improvements that can come from such technological advances. Conversely, as Gitte Lindgaard showed [3], sometimes assuming that every user problem needs to be solved by UI improvements can lead to solving the wrong problem and limiting our impact.

Another investment variable is that globalization has reduced product support costs, which in turn can lead to a reduced incentive to make usability improvements. Moving support offshore may be viewed as having a much bigger short-term impact than improving any aspect of product quality. This of course is only the producers’ perspective as noted in myth #2. From the consumers’ perspective any product flaws still exist. However the traditional usability ROI arguments do not fair well against this trend. Philosophically, our field opposes the idea that hiring lower cost support staff could be a more rapid way to reduce support costs than usability enhancements, but a truly compelling cost-justification must address such comparisons.

A More Fertile Approach? Customer Total Cost of Ownership (TCO)

So if belief in the power and relevance of ROI arguments is based on myths, what is the alternative? A clue comes from identifying the implicit theme that runs through my critique of many of these myths. A fundamental source of the disconnect between ROI arguments and the executive mindset is that the ROI arguments are at best tactical in nature, while the business challenges that upper management focuses on are inherently strategic. This calls for showing how we contribute to more strategic indicators of product value.

One of the key characteristics of the commercial software business is that vendors rely on a long-term relationship with their customers. Typical product lifecycles including field-testing, production deployment and upgrades can last as long as ten years. Over the lifecycle of product usage the operational costs can easily exceed the original purchase. In this context one of the most accurate measures of both success and competitiveness is customer "total cost of ownership" (TCO). This is a measure that technology executives understand and also one that has been studied in great detail by numerous analyst firms trusted by executives, such as Gartner and the MetaGroup.

TCO is a broad measure. In the area of implementation there are the costs of installation, configuration, administration, and training. This training is provided by the company that purchases the product for its employees and does not show up as a cost to the producer. Additional factors that contribute to TCO once a system is operational are the internal labor costs (a correlate of productivity) associated with specific business processes. Some examples include the number of orders that can be processed per hour, the time to manufacture a product, or the efficiency with which it can be distributed.

Total cost of ownership is both an external metric and a strategic one. It does not take into account product development cost or time savings during the development cycle, both of which form the backdrop of the current usability ROI discussion. By default, it does take into account whether you built a product that people will buy, because you can only measure TCO if someone is actually using the product. It also supports the underlying engineering truth that it is better to get the design and requirements right upfront than to have to fix it later, which is one of the main areas that the UCD process contributes to. Yet it measures the global end result, not an isolated intermediate step in the product lifecycle process.

In the business world, saving money is generally a tactical endeavor, while making money is a strategic one. In the enterprise software market, if the customers of your product are not successful you will go out of business. This is why most software companies fail and a small number of products dominate total software industry revenue. For those companies and products that do fail, there is no market financial return for investors; therefore, any "usability ROI" that could be measured is not truly relevant.

More importantly, TCO is related to the real value the product provides. A component of this external value is the user experience and, as noted at the beginning of this article, adding this value is our collective professional goal. It is within this context I suggest we look for better measures and a new research agenda for studying usability ROI and communicating it to our business associates and boardroom executives. This will involve shifting our focus away from short-term arguments that rely on numbers with low credibility, and which look at factors in isolation and without reference to context, and towards modes of analysis more in sync with the thinking of executives who have to conceptualize product value strategically and over the long-term.

References

1. Bias R. & Mayhew, D.J. (Eds.). (1994). Cost Justifying Usability. San Francisco: Morgan Kaufmann.

2. Capterra Enterprise Software Center. (n.d.). The need for a marketplace in the enterprise software industry. Retrieved July 7, 2004, from, www.capterra.com/enterprise-software.tcl

3. Lindgaard, G. (2004, May-June). Making the business our business: one path to value-added HCI. Interactions 11[3].

4. Nielsen, J. (1998, October 18). Failure of corporate websites, Alertbox. Retrieved July 7, 2004, from, www.useit.com/alertbox/981018.html

5. Nielsen, J. (1993). Usability engineering. San Francisco: Morgan Kaufmann.

6. Ostrander, E. (2000, October).Usability testing of documentation has many benefits of unknown value. Usability Interface, 7(2).

7. Rosenberg, D. (2003, October 14). 7 myths of usability ROI, Talk given at BayCHI Xerox PARC, Palo Alto, CA.

8. Rosenberg, D. (1989). A cost benefit analysis for corporate user interface standards. In J. Nielsen (Ed.), Coordinating User Interfaces for Consistency. Boston: Academic Press. pp. 21-34.

9. Sinha, Rashimi. (2004, February). ROI of usability: A collection of links. Retrieved July 7, 2004, from www.rashmisinha.com/useroi.html

10. Wixon, D. and Jones, S. (1992). Usability for fun and profit: A case study of the design of DEC RALLY, version 2. Digital Equipment Corporation.

Author

Daniel Rosenberg
daniel.rosenberg@oracle.com

Tables

T1Table 1.

©2004 ACM  1072-5220/04/0900  $5.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2004 ACM, Inc.

Post Comment


No Comments Found