**Authors:**

John Sorflaten

Most managers must show dollar returns that justify the extra time it takes to incorporate usability methods into R&D. Many of those who are still new to usability resist changing their former development practices beyond lip service such as "oh, yes, do a usability test tomorrow when we have version one up and running." (Usability professionals will see the humor in this statement.)

Luckily, many ROI estimates require only simple mathematical skills to make the point, as we’ll see. No real statistics are necessary. But, following our demonstration below, we’ll show how the normal ROI may be fuzzier than we’d like. Consequently, we’ll show you how to clarify the fuzzy parts of ROI with concepts like "margin of error."

**ROI Seems So Clear.** Imagine that you have design
responsibility for a flagship customer-service application used
to manage call-ins for a large telecommunications firm. Usability
enhancements will save time—and, thus, labor costs. You can
modify the following six-step ROI calculation for other design
goals such as increased "conversions" (purchases or
requests for follow-up on your e-commerce Web site), decreased
help-desk activity, or decreased training, etc.

**Step 1. Identify your savings scope.** Here, we apply our
estimates to 5,000 customer-service representatives (CSRs) that
handle billing questions, service complaints, up-sells, etc. Five
thousand end users is *not* far-fetched for many
organizations. (Think about intranet usage among your thousands
of fellow employees.) Also, check out Daniel Rosenberg’s article
"The Myths of Usability ROI"
(<*interactions*> XI.5, September 2004, pages 23-29)
for best practices in ROI scoping. Rosenberg recommends focus on
"Total Cost of Ownership" of systems including internal
labor costs that we discuss here.

**Step 2. Calculate your savings rate.** We need the
average loaded labor rate for your end users. "Loaded"
means the cost of salary plus the cost of benefits (including
vacation), training, supervision, computer tools, and rent on
cubicle and aisle space.

If you can’t get detailed information, usability ROI pundit
Deborah Mayhew suggests using a loaded labor rate of double the
worker’s before-tax salary (see Mayhew and Bias
*Cost-Justifying Usability, Second Edition*, 2005, page
55).

So, given a $40,000 average CSR salary, we could estimate an
$80,000 yearly loaded labor rate. Given a 2,000-hour year, this
implies a $40-per-hour loaded labor rate ($80,000 divided by
2,000 hours). Note that a *real* loaded labor rate from your
management could be much more or less.

Given an average of ten calls per hour, the $40 hourly loaded labor rate means each call costs $4 ($40 divided by ten calls = $0.666), or just under 67 cents per minute ($40.00 divided by 60 minutes), or approximately one cent per second ($0.666 divided by 60 seconds = $0.0111).

**Step 3. Identify your savings source.** We estimate the
time savings from some usability improvement you want to make.
For example, you guess that on average, a CSR could save 20
seconds per call after your redesign and usability testing.
Twenty seconds multiplied by $0.0111 per second gives you just
over 22 cents in savings per call.

For the hourly per-employee savings, we multiply the 22 cents of savings per call, times ten calls per hour to get $2.22 per hour.

For the yearly per-employee savings, multiply $2.22 in savings per hour by 2000 hours per year to get $4,440 per year.

**Step 4. Calculate total yearly savings.** Multiply the
per-person savings of $4,440 by 5,000 employees to get $22.2
million in annual savings. The 20 seconds’ savings per call
implies $1.11 million in savings per second. Large numbers for
calls and CSRs make very large numbers for potential savings.
This is the return component of ROI. You can duplicate the steps
above in a spreadsheet using your own numbers.

**Step 5. Identify cost of improvements.** This is the
"investment" part of ROI. What does it cost to save
those 20 seconds per call? Let’s say that your chief information
officer estimates that the programming effort will cost $2
million, including documentation efforts. Of course, such
estimates are not easily given, and you must outline enough
specifics to make the estimate possible. Even the estimate will
cost you money. Let’s say that is part of the $2 million.

Meanwhile, your own usability efforts entail task analysis, redesign, formative usability testing, and prototyping, all with several iterations. You give $200,000 as your reasonable estimate.

Don’t forget retraining of your CSRs. Your training manager estimates a $100,000 cost for development and delivery of training by staff. Course time will be about two hours for each CSR, which, when multiplied by 5,000 CSRs, makes 10,000 hours. Multiply that by the $40 hourly loaded labor rate to get $400,000 in lost-time costs. Adding these two estimates makes $500,000 for retraining.

**Step 6. Calculate your final ROI.** Subtract the
anticipated cost from the anticipated savings, and then divide by
the cost to get a ratio of return on investment. The total cost
appears to be $2.7 million ($2 million for programming and
documentation, plus $200,000 for usability work, plus $500,000
for retraining).

Subtracting this from the gross savings of $22.2 million leaves you a net savings of $19.5 million. Dividing this return by your investment gives you a 7.22 times return ratio ($20.5 million divided by $1.7 million).

To convert this to a percentage ROI, multiply it by 100 percent to get a 722 percent ROI (100 percent multiplied by 7.22) for the first year. Not bad for an investment. If you anticipate the changes to have a lifetime of two or three years, add that to the return part of your calculations and recalculate the ROI.

Are we done? I’m sure you can add more to this illustrative listing of returns and investments. But now you see how it works in principle. Let’s get to the fuzzy part of our ROI analysis.

**The Fuzzy Part of ROI.** Many ROI estimates look like
"job costing." We expect that the programming effort,
the training development, and even your usability work allow for
fudging during the project. Activities that take more time will
be compensated by activities that turn out to take less time.
People get good at such estimates. Therefore they keep their job
as a project planner.

However, at least one element of our ROI still holds
considerable uncertainty or *fuzziness*. For our scenario
above, we might ask, "How do we know that our design
provides the 20 seconds’ savings prior to coding and
release?" This is the fuzzy part of your test
measurements.

**Use Margin of Error for Averages.** Formative usability
testing with a prototype can help answer this hard question. But
remember, your test results merely give the *average*
savings for *that sample* of test participants. If you
extrapolate to your entire CSR group of 5,000, you must take
steps that allow you to do so. You *must* take into account
a margin of error.

For example, newspapers typically report a margin of error for voting polls as plus or minus three percent. You can consider this a relatively low margin of error if it results from interviewing 1,067 telephone respondents—a large sample.

Luckily, you won’t need anywhere near that many participants with time-savings data because it uses a different mathematical approach.

**Large Consequences of Missing the Investment Goal.** The
stakes are high when making savings estimates from usability
testing. Because our ROI analysis leverages the 20 seconds very
highly, small misses in reaching that goal greatly affect the
planned return on investment.

For example, each second saved gets multiplied by ten calls
per hour. That result is multiplied by 2000 hours. And
*that* result gets multiplied by 5,000 CSRs.

Doing the math shows that, for each second by which you miss your goal of 20 seconds’ savings, you have a collective multiplier of 100 million (ten calls per hour times 2,000 hours per CSR times 5,000 CSRs = 100 million).

Consequently, with each second you miss your target, your first-year ROI suffers. Note that your ROI drops rapidly because the investment (or costs) remains the same regardless of time savings (see Figure 1). Therefore, if we lose ten seconds of savings, our ROI drops dramatically, to 352 percent from the anticipated 727 percent total ROI. You can see we need to calculate a margin of error to account for the impact of small changes in the number of seconds saved.

A margin of error lets you demonstrate that you have attained the 20 seconds’ savings within a stated level of confidence.

**Establishing a Confidence Level.** Typically,
establishing a confidence level requires computing a plus or
minus range or specific "margin of error" within which
your time savings will fall 95 percent of the time. This implies
that if you make 100 such tests, the average time savings will
fall within the given range 95 times. The *margin of error*
we previously faced was the *fuzzy* part of our ROI. The
*95 percent confidence level* we now illustrate lets us
clarify this fuzzy element.

Figure 2 shows that our first usability test
of 20 participants provides an average savings of 18.67 seconds.
The *95 percent confidence level* is illustrated by the
*margin of error* bars that fall above and below the average
(mean) time savings. By the way, for time-on-task averages,
statisticians recommend using the "geometric mean,"
*not* the "arithmetical mean."

See www.measuringusability.com/time_intervals.php for
discussion and a calculator on this topic. You can duplicate
these numbers at the above link using raw values of 29, 32, 16,
15, 8, 20, 17, 19, 28, 20, 32, 8, 26, 20, 7, 24, 30, 18, 17, 20
seconds. These are the time savings in used in the hypothetical
usability test mentioned above. For each participant, these
numbers represent the old time-on-task minus new time-on-task
equals time savings. Also, for the "Confidence Level"
requested on the page, select "90%." Since we are only
interested in the *bottom* of the margin of error bar, for
mathematical reasons, we use a 90 percent to calculate a 95
percent confidence level. Technically, this lets us conduct a
"one-tailed test" using a calculator set up for
"two-tailed" confidence intervals.

**Evaluating Your Test Results.** Have we met our goal of
saving 20 seconds?

Unfortunately, no. This is because even though the *top*
end of the margin of error falls at 22.2 seconds saved, we see
that the *bottom* end of our margin-of-error bar falls at
15.7 seconds. Given this range for our margin of error, we know
that we will not find our 20 seconds’ savings in 95 out of 100
such tests.

You must send your usability team back to work. Of course, you previously budgeted for additional iterations.

**Exceeding the Target to Get the Target. So, how do you know
you have achieved our 20 seconds’ savings with 95 percent
confidence? You must keep improving your application until your
test scores show that the** *bottom* of the confidence
interval meets or exceeds your savings goal, since the mean isn’t
always the center of a confidence interval.

See Figure 3 for how your team may have progressed over several iterations.

On the fifth iteration, your team scores success, as the bottom of the 95 percent confidence interval falls at the targeted 20.08 seconds’ savings. (Note that the mean falls at 23.02 seconds savings—an illusory measurement.)

However, imagine that your team decides they want a "slam dunk" for the project. So you do another iteration. Your sixth iteration puts the mean at 24.08 seconds. And most importantly, it puts the bottom of the 95 percent confidence interval at 21.67 seconds, well above your savings goal of 20 seconds.

Now your team’s design is a clear winner. You know it’s a winner because you made the fuzzy part of ROI analysis clear.

**Using a Web Calculator or Excel to Create Confidence
Intervals.** You can calculate the geometric mean and
confidence intervals using Jeff Sauro’s calculator at
http://www.measuringusability.com/time_intervals.php.

Additionally, you can use your spreadsheet to make these calculations. For instructions on this procedure and creating charts showing the "Y-error bars" (margins of error), visit this Human Factors International Web page: http://www.humanfactors.com/services/tasktimes.asp.

**Author**

John Sorflaten

Human Factors International

john@humanfactors.com

**About the Author:**

*A certified professional ergonomist, John Sorflaten has
consulted in user-centered design and taught it for over 18 years
at Human Factors International, Inc. He serves as a project
director and unofficial visionary. Upon seeing Jeff Sauro’s Web
site www.measuringusability.com and then inviting Jeff for dinner
one evening, John started delivering on the promise of
"quantitative usability." See John’s article* When
Discount Usability Misleads Management—a Solution *at
www.humanfactors.com/downloads/jul05.asp.*

**Figures**

**©2006
ACM 1072-5220/06/1100 $5.00**

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.

## Post Comment

No Comments Found