Waits & measures

XIII.6 November + December 2006
Page: 38
Digital Citation

Making the fuzzy parts of ROI clear

John Sorflaten

Most managers must show dollar returns that justify the extra time it takes to incorporate usability methods into R&D. Many of those who are still new to usability resist changing their former development practices beyond lip service such as "oh, yes, do a usability test tomorrow when we have version one up and running." (Usability professionals will see the humor in this statement.)

Luckily, many ROI estimates require only simple mathematical skills to make the point, as we’ll see. No real statistics are necessary. But, following our demonstration below, we’ll show how the normal ROI may be fuzzier than we’d like. Consequently, we’ll show you how to clarify the fuzzy parts of ROI with concepts like "margin of error."

ROI Seems So Clear. Imagine that you have design responsibility for a flagship customer-service application used to manage call-ins for a large telecommunications firm. Usability enhancements will save time—and, thus, labor costs. You can modify the following six-step ROI calculation for other design goals such as increased "conversions" (purchases or requests for follow-up on your e-commerce Web site), decreased help-desk activity, or decreased training, etc.

Step 1. Identify your savings scope. Here, we apply our estimates to 5,000 customer-service representatives (CSRs) that handle billing questions, service complaints, up-sells, etc. Five thousand end users is not far-fetched for many organizations. (Think about intranet usage among your thousands of fellow employees.) Also, check out Daniel Rosenberg’s article "The Myths of Usability ROI" (<interactions> XI.5, September 2004, pages 23-29) for best practices in ROI scoping. Rosenberg recommends focus on "Total Cost of Ownership" of systems including internal labor costs that we discuss here.

Step 2. Calculate your savings rate. We need the average loaded labor rate for your end users. "Loaded" means the cost of salary plus the cost of benefits (including vacation), training, supervision, computer tools, and rent on cubicle and aisle space.

If you can’t get detailed information, usability ROI pundit Deborah Mayhew suggests using a loaded labor rate of double the worker’s before-tax salary (see Mayhew and Bias Cost-Justifying Usability, Second Edition, 2005, page 55).

So, given a $40,000 average CSR salary, we could estimate an $80,000 yearly loaded labor rate. Given a 2,000-hour year, this implies a $40-per-hour loaded labor rate ($80,000 divided by 2,000 hours). Note that a real loaded labor rate from your management could be much more or less.

Given an average of ten calls per hour, the $40 hourly loaded labor rate means each call costs $4 ($40 divided by ten calls = $0.666), or just under 67 cents per minute ($40.00 divided by 60 minutes), or approximately one cent per second ($0.666 divided by 60 seconds = $0.0111).

Step 3. Identify your savings source. We estimate the time savings from some usability improvement you want to make. For example, you guess that on average, a CSR could save 20 seconds per call after your redesign and usability testing. Twenty seconds multiplied by $0.0111 per second gives you just over 22 cents in savings per call.

For the hourly per-employee savings, we multiply the 22 cents of savings per call, times ten calls per hour to get $2.22 per hour.

For the yearly per-employee savings, multiply $2.22 in savings per hour by 2000 hours per year to get $4,440 per year.

Step 4. Calculate total yearly savings. Multiply the per-person savings of $4,440 by 5,000 employees to get $22.2 million in annual savings. The 20 seconds’ savings per call implies $1.11 million in savings per second. Large numbers for calls and CSRs make very large numbers for potential savings. This is the return component of ROI. You can duplicate the steps above in a spreadsheet using your own numbers.

Step 5. Identify cost of improvements. This is the "investment" part of ROI. What does it cost to save those 20 seconds per call? Let’s say that your chief information officer estimates that the programming effort will cost $2 million, including documentation efforts. Of course, such estimates are not easily given, and you must outline enough specifics to make the estimate possible. Even the estimate will cost you money. Let’s say that is part of the $2 million.

Meanwhile, your own usability efforts entail task analysis, redesign, formative usability testing, and prototyping, all with several iterations. You give $200,000 as your reasonable estimate.

Don’t forget retraining of your CSRs. Your training manager estimates a $100,000 cost for development and delivery of training by staff. Course time will be about two hours for each CSR, which, when multiplied by 5,000 CSRs, makes 10,000 hours. Multiply that by the $40 hourly loaded labor rate to get $400,000 in lost-time costs. Adding these two estimates makes $500,000 for retraining.

Step 6. Calculate your final ROI. Subtract the anticipated cost from the anticipated savings, and then divide by the cost to get a ratio of return on investment. The total cost appears to be $2.7 million ($2 million for programming and documentation, plus $200,000 for usability work, plus $500,000 for retraining).

Subtracting this from the gross savings of $22.2 million leaves you a net savings of $19.5 million. Dividing this return by your investment gives you a 7.22 times return ratio ($20.5 million divided by $1.7 million).

To convert this to a percentage ROI, multiply it by 100 percent to get a 722 percent ROI (100 percent multiplied by 7.22) for the first year. Not bad for an investment. If you anticipate the changes to have a lifetime of two or three years, add that to the return part of your calculations and recalculate the ROI.

Are we done? I’m sure you can add more to this illustrative listing of returns and investments. But now you see how it works in principle. Let’s get to the fuzzy part of our ROI analysis.

The Fuzzy Part of ROI. Many ROI estimates look like "job costing." We expect that the programming effort, the training development, and even your usability work allow for fudging during the project. Activities that take more time will be compensated by activities that turn out to take less time. People get good at such estimates. Therefore they keep their job as a project planner.

However, at least one element of our ROI still holds considerable uncertainty or fuzziness. For our scenario above, we might ask, "How do we know that our design provides the 20 seconds’ savings prior to coding and release?" This is the fuzzy part of your test measurements.

Use Margin of Error for Averages. Formative usability testing with a prototype can help answer this hard question. But remember, your test results merely give the average savings for that sample of test participants. If you extrapolate to your entire CSR group of 5,000, you must take steps that allow you to do so. You must take into account a margin of error.

For example, newspapers typically report a margin of error for voting polls as plus or minus three percent. You can consider this a relatively low margin of error if it results from interviewing 1,067 telephone respondents—a large sample.

Luckily, you won’t need anywhere near that many participants with time-savings data because it uses a different mathematical approach.

Large Consequences of Missing the Investment Goal. The stakes are high when making savings estimates from usability testing. Because our ROI analysis leverages the 20 seconds very highly, small misses in reaching that goal greatly affect the planned return on investment.

For example, each second saved gets multiplied by ten calls per hour. That result is multiplied by 2000 hours. And that result gets multiplied by 5,000 CSRs.

Doing the math shows that, for each second by which you miss your goal of 20 seconds’ savings, you have a collective multiplier of 100 million (ten calls per hour times 2,000 hours per CSR times 5,000 CSRs = 100 million).

Consequently, with each second you miss your target, your first-year ROI suffers. Note that your ROI drops rapidly because the investment (or costs) remains the same regardless of time savings (see Figure 1). Therefore, if we lose ten seconds of savings, our ROI drops dramatically, to 352 percent from the anticipated 727 percent total ROI. You can see we need to calculate a margin of error to account for the impact of small changes in the number of seconds saved.

A margin of error lets you demonstrate that you have attained the 20 seconds’ savings within a stated level of confidence.

Establishing a Confidence Level. Typically, establishing a confidence level requires computing a plus or minus range or specific "margin of error" within which your time savings will fall 95 percent of the time. This implies that if you make 100 such tests, the average time savings will fall within the given range 95 times. The margin of error we previously faced was the fuzzy part of our ROI. The 95 percent confidence level we now illustrate lets us clarify this fuzzy element.

Figure 2 shows that our first usability test of 20 participants provides an average savings of 18.67 seconds. The 95 percent confidence level is illustrated by the margin of error bars that fall above and below the average (mean) time savings. By the way, for time-on-task averages, statisticians recommend using the "geometric mean," not the "arithmetical mean."

See www.measuringusability.com/time_intervals.php for discussion and a calculator on this topic. You can duplicate these numbers at the above link using raw values of 29, 32, 16, 15, 8, 20, 17, 19, 28, 20, 32, 8, 26, 20, 7, 24, 30, 18, 17, 20 seconds. These are the time savings in used in the hypothetical usability test mentioned above. For each participant, these numbers represent the old time-on-task minus new time-on-task equals time savings. Also, for the "Confidence Level" requested on the page, select "90%." Since we are only interested in the bottom of the margin of error bar, for mathematical reasons, we use a 90 percent to calculate a 95 percent confidence level. Technically, this lets us conduct a "one-tailed test" using a calculator set up for "two-tailed" confidence intervals.

Evaluating Your Test Results. Have we met our goal of saving 20 seconds?

Unfortunately, no. This is because even though the top end of the margin of error falls at 22.2 seconds saved, we see that the bottom end of our margin-of-error bar falls at 15.7 seconds. Given this range for our margin of error, we know that we will not find our 20 seconds’ savings in 95 out of 100 such tests.

You must send your usability team back to work. Of course, you previously budgeted for additional iterations.

Exceeding the Target to Get the Target. So, how do you know you have achieved our 20 seconds’ savings with 95 percent confidence? You must keep improving your application until your test scores show that the bottom of the confidence interval meets or exceeds your savings goal, since the mean isn’t always the center of a confidence interval.

See Figure 3 for how your team may have progressed over several iterations.

On the fifth iteration, your team scores success, as the bottom of the 95 percent confidence interval falls at the targeted 20.08 seconds’ savings. (Note that the mean falls at 23.02 seconds savings—an illusory measurement.)

However, imagine that your team decides they want a "slam dunk" for the project. So you do another iteration. Your sixth iteration puts the mean at 24.08 seconds. And most importantly, it puts the bottom of the 95 percent confidence interval at 21.67 seconds, well above your savings goal of 20 seconds.

Now your team’s design is a clear winner. You know it’s a winner because you made the fuzzy part of ROI analysis clear.

Using a Web Calculator or Excel to Create Confidence Intervals. You can calculate the geometric mean and confidence intervals using Jeff Sauro’s calculator at http://www.measuringusability.com/time_intervals.php.

Additionally, you can use your spreadsheet to make these calculations. For instructions on this procedure and creating charts showing the "Y-error bars" (margins of error), visit this Human Factors International Web page: http://www.humanfactors.com/services/tasktimes.asp.


John Sorflaten
Human Factors International

About the Author:

A certified professional ergonomist, John Sorflaten has consulted in user-centered design and taught it for over 18 years at Human Factors International, Inc. He serves as a project director and unofficial visionary. Upon seeing Jeff Sauro’s Web site www.measuringusability.com and then inviting Jeff for dinner one evening, John started delivering on the promise of "quantitative usability." See John’s article When Discount Usability Misleads Management—a Solution at www.humanfactors.com/downloads/jul05.asp.


F1Figure 1.

F2Figure 2.

F3Figure 3.

©2006 ACM  1072-5220/06/1100  $5.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.


Post Comment

No Comments Found