Aaron Marcus, Jim Gasperini
In June 2004 the San Jose, California, Police Department (SJPD) rolled out a new mobile, in-vehicle communication system for police officers. This roll-out completed the entire replacement of a dispatch and mobile response system developed by PRC in 1990. Working with PRC, the SJPD had spent many years perfecting their own system, which was a customized, closed suite of applications. SJPD had learned much about what made a very usable system for their officers, sergeants and lieutenants who manage officers, dispatchers, and dispatch managers. They considered themselves among the more knowledgeable police departments in the US, having had many years of experience debugging and using their system. However, it gradually became apparent that a system based on a long-obsolete operating system (Windows 95) was in need of overhaul or replacement. Among other factors, connecting with new communications technologies, such as wireless phones, was not possible with the current system.
SJPD appointed a committee to investigate technology strategies and sent out a request for proposals that may have emphasized the dispatcher functions, not the mobile response functions. The committee was comprised of the department's managers, information technology (IT) representatives, managers, at least one dispatcher, but no police officers. The committee reviewed finalists from many submissions and concluded in their then-current estimates that upgrading the closed, customized system of PRC would be more costly than using "off-the-shelf" components from a commercial vendor. This decision was a significant shift in philosophy.
After further consideration, they selected Intergraph's I/CAD (Intergraph/Computer-Aided Dispatch System). The new system for the police officers, which is only a part of the total system, is a customized version of I/Mobile, a product of Intergraph Public Safety. I/Mobile is designed to complement and work with I/Dispatcher, another module of I/CAD, which the SJPD had also recently installed for the use of dispatchers.
Customization and debugging of the system had been an ongoing process since initial roll out, involving teams from Intergraph, the SJPD's IT department, and other parts of the SJPD. During the same period, another major roll-out was in the works: Automated Field Reporting (AFR) and Report Writer (RPW) software, both products of Data911. Roll out of this additional software was scheduled for September 2004.
Early experience with the I/Mobile software prompted many concerns, including safety, on the part of many SJPD officers and supervisors:
- The system for police officers had crashed upon first roll-out.
- The officers were given only three hours of training, not the 16 recommended by the software vendor.
- Map data appeared to show incorrect or missing data.
- Officers were finding the system so difficult to use that some were turning it off and reverting to only audio mobile communication, which was limited and placed additional burdens on the dispatch staff.
- Some officers became confused by the system and were performing poorly.
- The training had to be done on desktop PC systems in a training classroom instead of the actual touch-screen LCDs in the vehicle, or at least on actual units in a classroom setting.
- Versions were constantly being rolled out among partial vehicle fleets, which meant extra complexity in keeping track of which users were using which system in which vehicles, in terms of bug reporting, evaluations, etc.
From the onset, the acquisition and set up of the system seemed to be a case study in how not to do user-centered design. Consequently, some of the results seemed not too surprising. Among other factors that seemed to contribute to the low level of initial success were the following:
- Usability did not seem to be a priority for system acquisition evaluation (as opposed to functionality or cost). This is a typical oversight by technical people, business managers, and marketers. The focus is on the issue, "Does the system do function X?" rather than, "If the system does function X, what is the general usability of that function by itself but, more importantly, in relation to typical task scenarios A, B, and C for users L, M, and N under conditions E, F, G?" Of course this query is more complex and expensive to evaluate. If vendors were to provide data for these, presumably, typical functions, users, and tasks, it would make life simpler for customers to evaluate software; but of course they don't, for obvious cost and marketing reasons.
- The system acquisition team did not consist of at least one user representative (officer or dispatcher), but rather managers of those users, who might be presumed to make reasonable decisions about the users they were representing, but who might also have management bias. For unknown reasons, perhaps communication or bureaucratic slip-up, the police officers had been invited to send representatives to the committee, but none seem to have been sent.
- The system acquisition team did not query users in evaluating the finalists or in setting up initial conditions for customization for the initial rollout. In other words, user focus groups were not established for the purpose of elucidating user preferences, use tasks, and use scenarios.
- In fact, the entire development of detailed user-interface development, including customization of functions, layouts, colors, typography, textual content, symbols/icons, etc., was carried out, as far as the authors could determine, with little or no significant input from actual users, especially police officers.
- No one of the development team constructed user profiles and use scenarios, even though these would have been relatively easy to construct given the detailed knowledge of all available subject-matter experts and the somewhat limited, predictable, and closed world of most-typical circumstances (essentially business rules). Best-practice knowledge might have been collected, organized, and archived, and made available to all involved in UI development.
- The SJPD decided to use maps from the city's Department of Public Works (DPW) as the basis for geo-location functions. While this choice may yet prove to be sound in the long run, the short-term consequences were quite negative. Technical differences between the software vendor's system for handling geographic information and the city DPW's system for handling the same information resulted in dramatic display and accuracy problems.
- For cost reasons, the SJPD software roll-out team decided to reduce training to a minimum amount and did not establish good feedback loops from initial users in order to tune the system faster and better.
- The onsite Intergraph team initially assigned to the SJPD project seemed to have either little incentive or little knowledge of client-relationship best practices. They seemed not as cooperative as they might have been under the circumstances in offering and implementing improvements. Eventually, executive management became aware of the situation, made several visits, and replaced most of their team with more cooperative staff.
- Other installations of the software had been problematic, as reported in legal documents that the authors discovered later, but these may not have been available to the SJPD.
Unfortunately, a conflux of circumstances led to less-than-best practices. Because the SJPD, their managers, and IT people were not necessarily familiar with user-centered design, and because the software vendor did not supply that guidance, the SJPD was able to proceed only with good intentions.
At this point, more than $4 million had been spent on the software, and it was performing poorly, with many usability problems that even novice usability analysts, and certainly the dispatchers and police, could not help but notice. Reports of problems were leaking to the press, San Jose's City Council, and the police officers' union, the San Jose Police Officers Association (SJPOA). The SJPOA was becoming involved as tempers rose and delays escalated. At this point, the authors' firm was called in to evaluate the software.
Originally, the SJPOA, acting on behalf of the police officers and concerned about possible life-threatening dangers of the software to the officers (in a stage preliminary to considering lawsuits on behalf of the officers against the SJPD and/or the software vendors), invited Ms. Alison Heller-Ono, president of Worksite International, to conduct an ergonomic analysis of the in-vehicle devices. She quickly realized that more than physical human factors were involved and suggested that specialized user-interface analysts (UI) be called in. At her suggestion, SJPOA invited the authors' firm to conduct a usability evaluation of the just the UI of the mobile communications and report-preparation software provided by Intergraph (and also by Data911, a separately contracted firm whose software the authors did not directly view/review). These applications were called collectively the Mobile Dispatch Computer/Computing system (MDC).
This situation for a usability analysis was quite startling because of the emotions and politics involved, and the funds/reputations at stake, to say nothing of the usability issues. As someone quipped, it is somewhat unusual when your client is heavily armed and very upset. As we became more involved, we learned how complex the situation was and had to give more effort than anticipated to interview users, learn about the software, and slowly discover some of the historical roots of the situation.
In August and September 2004, we analyzed the software using our own heuristics, developed over many years, which incorporate aspects of published heuristics. At the time of Aaron Marcus & Associates' (AM+A) investigation, users had several months' experience with the software, enabling AM+A to interview six officers for approximately two hours each. The officers varied in experience with technology and history on the police force. The subjects included both officers and their managers. The project goals for this analysis were the following:
- Evaluate the software UI according to standard usability criteria, including judgments about the seriousness of any defects.
- Given certain "facts on the ground," make preliminary "best practice" suggestions about how the SJPD should move forward to repair or improve the usability of the software.
At one point in our interviews, one sergeant when discussing his management style, distinguished between "mistakes of the mind" and "mistakes of heart." To him, mistakes of the mind (honest misjudgments based on erroneous facts or thinking) are inevitable, and, within reason, should be forgiven. However, he was much less tolerant of mistakes of the heart (during which one knows what he is doing is wrong but proceeds anyway). The sergeant felt some of the usability problems resulted from this latter kind of error.
Everyone with whom we talked at the SJPD agreed that mistakes, some quite serious, were made during the MDC roll-out. It was our impression that they were all mistakes of the head, not of the heart. Our role was to point out where, from the user's point of view, improvement was needed, and to make preliminary suggestions for improvement. The I/Mobile system as implemented during our visit in late August 2004 posed serious usability concerns. Police officers seemed quite justified in voicing concern about the new system's effects on their productivity and the dangers that aspects of the system posed to their safety.
The main body of the 60-page report (plus 40 pages of appendices) we prepared and organized issues into 29 items of concern. The most severe issues were the following:
- The difficulty of completion of numerous important tasks while driving. Running a license plate, for example, is a frequently-performed procedure that enables officers to determine whether they have cause to stop a vehicle. Officer safety issues were of paramount concern to the SPJOA.
- Reliance on indirect rather than direct controls. Officers were accustomed to command-line entry in the previous system. Although that system required the memorization of arcane codes, once learned, the system allowed one-hand performance of many powerful functions without looking at the keyboard at all. Although the new system sometimes gave officers many options to perform the same function (touchpad, touch screen, keyboard combinations, etc.) these techniques usually required the officer to be aware of more than one part of the UI at the same time, resulting in serious cognitive distraction.
- UI not optimized for touch screen. Although touch-screen technology is potentially one of the strongest advantages of the new system, the UI relied almost exclusively on Windows conventions. While such techniques as pull-down menus have their place in the relative calm of an office setting, they are not optimal for drivers of a two-ton vehicle traveling at speed. The one part of the UI designed with touch screen in mind, a row of buttons, was laid out confusingly and labeled poorly and inconsistently.
- Poor filtering of important information. Officers rely, for example, on "event histories" for succinct summaries of key facts and actions taken by parties involved in a police "event." The new system buried key information in useless and redundant detail. Useful dispatcher and officer comments, and relevant responses from queried databases, appeared interspersed with voluminous content that rendered the relevant facts obscure.
- Information poorly laid out. An example of this problem was "unit summaries," which officers and supervisors use to get a quick overview of the location and status of other officers. Entries were displayed in a manner that did little to distinguish between major conceptual categories. Certain key information was missing, while useless information was presented with unwarranted prominence.
- Problems with sending and receiving messages. The process of sending a message involved too many steps, in particular, frequent forced tabbing through seldom-used fields. Officers found it hard to distinguish between important and unimportant messages, and hard to review important sets of exchanges between themselves and other members of their team.
- Awkwardness when units cross district borders. When a police officer's vehicle moved from one geographic area to another (e.g., to assist in an emergency situation elsewhere in the city), his vehicle might disappear inappropriately from screens of a supervisor or dispatcher.
- Numerous issues with mapping and routing. Symbols, colors, and text appeared in a confusing display. Maps were not set to show the most useful level of data. For example, rather than showing the location of a new "event" or the officer's car in relation to that event, it first showed the location of the officer's car, presumably something already known. Frequent encounters with erroneous data undermined officers' confidence in the system, and in at least one case led to a dangerous misunderstanding.
Certain key information was missing, while useless information was presented with unwarranted prominence.
Problems occurred in part because a Microsoft-windows-like UI with detailed menus and small data fields and labels was provided to officers who would primarily be interacting with a screen via a touch panel, sometimes while driving, in varying ambient light circumstances, and sometimes under impending emergency conditions. Text was sometimes not legibly or usefully displayed; insufficient attention had been given to details of type size, color, naming conventions, and organization of lists and tables of data. In addition, sufficient quick keyboard or screen-button command access had not been provided to do some of the most routine tasks. This last is a commonly-encountered situation: The functions are there, but not optimally available for typical tasks, because sufficient task analysis has not been carried out.
To summarize one of the most serious problems: At times, the officers required intense focusing on driving or on a scene in front of them. They were essentially operating the in-vehicle system "as if blind," only able to take their eyes off the scene in front of them for a second to check a key piece of data. For example, they might need to keep an eye on a suspect while also running a license-plate check to determine if the vehicle had been reported stolen or if the vehicle or its occupants matched any characteristics of persons with warrants for arrest, etc. This data would determine in advance the caution an officer would take when approaching occupants of a vehicle.
Although the authors are not software developers per se and were unfamiliar with what was "under the hood" of the Intergraph Public Safety system, we were confident that many problems could be corrected, or at least reduced, through the application of user-centered design techniques to the continuing customization process. Best practice in the software design process, or in this case perhaps the design customization process, was to undertake early on a rigorous task analysis. By soliciting input from subject matter experts, users, and administrators, the task analysis would seek to create a prioritized list of the most important processes or tasks that would be performed using the software. Though ideally such an analysis takes place before choosing a software vendor, or at least before beginning to customize the software, the old adage "better late than never" still applied.
Over the next few months, following the delivery of our report to our client, the emotional and political situation changed dramatically. We kept in close contact with the legal counsel of the SJPOA to ascertain what was happening and what we might be called upon to do.
- Disgruntled police officers and dispatchers wrote letters to management that were also circulated to the city council and to the press.
- City council members began more closely supervising what was occurring in the police department and requested/demanded reports on solving the crucial problems and the expenses involved. In particular, they asked for deadlines by which time the problems were expected to be solved.
- An article appeared on the front page of the San Jose Mercury News revealing the extent of the problems and the possible peril to officers .
- An article subsequently appeared on the front page of The New York Times' "Circuits" section, which provided even greater depth of coverage about the problems and quoted both the report and one of the authors commenting on usability problems .
- Intergraph, sensing the severity of the problem with one of its customers, replaced the team of people on site and improved its stance of cooperation, offering to make significant improvements under the current contract. The division president even visited the site to understand better the circumstances and to determine how things might be made right.
- The SJPD IT management group that was supervising the roll-out made significant efforts to include officers and lieutenants in review teams, recognized that they had a morale and PR problem, not just a usability challenge, and began to be more attentive to users, their concerns, their training, and their feedback, which they incorporated into improvements.
AM+A subsequently conducted an interim audit of the improvements being made against the usability problems that had been identified. In general, we found significant measures were being undertaken, but some items were still in progress. In checking with the San Jose Police and the SJPOA over many months afterwards, we discovered that the City of San Jose had required an independent audit of the usability problems, because we had been hired by the union and therefore might conceivably be biased. This study went on for many months. To our knowledge, this report confirmed the challenges that we had outlined in greater detail. It was not until July 2005 that we could confirm that the problems with the police-officers system had been rectified to the satisfaction of the police officer in charge of system improvements. The SJPOA seemed to concur. The situation settled down, with no lawsuits, and everyone a bit older and wiser.
The case was closed...or was it? At about that same time, a local television station announced that the separate dispatchers' system was causing problems and was a subject of scrutiny. These users represented a completely different union and had not been part of our study. Perhaps the cycle of study and correction would begin again.
In the end, the SJPD had to spend additional sums to complete its software customization of off-the-shelf solutions for its needs. The end-costs, to our knowledge, were comparable with what was estimated by the previous vendor for pure custom-solution. In retrospect, the debate about which path to take seems still debatable.
Fortunately in this case, improved processes and improved designs were accomplished. What lessons can be learned from this experience? One is the terrible consequences of not undertaking a user-centered design process. The toll is extensive: possible dangers to officers, including life-threatening risks; annoyed and angry users; involvement and time of other agencies and organizations (such as the union, the city council, and the press), to say nothing of the problems for dispatchers, who had equally serious concerns; and the time/expense required by the software vendor to address and solve as many of the usability issues as possible.
Another lesson, which frustrates us as professionals, is this: How could this situation have been avoided? The SJPD did not know they did not know. The software vendors have limited ability, or incentives, to inform their customers about a process that may complicate things in the short run, even if it might benefit everyone in the long run. They may also, in some cases, be partially or fundamentally unaware of best practices for user-centered design.
It seems clear that UI designers and analysts, especially usability analysts and/or user-experience analysts, must continue and perhaps redouble their outreach efforts to contact and educate government groups, industry leaders, and researchers about the user-centered UI development process. Although organizations like SIGCHI, UPA, HFES, STC, AIGA, and others have made considerable progress educating clients over the years, this particular case study indicates that there is much room for improvement.
There are many police officers associations for every major metropolitan area across the US, and hundreds, perhaps thousands, more dispatcher and emergency response centers. At the least, these groups might be informed about our profession's best practices to help improve current and future systems. The outreach effort lies clearly ahead. The benefits will accrue to all professional organizations, these communications centers, software providers, and, of course, to the end users.
The authors acknowledge the assistance of the following persons in conducting their original usability evaluation and in evaluating the content of this document: Gabe Atiya, AM+A designer/analyst; Don Demers, president, SJPOA; Alison Heller-Ono, president and usability analyst, Worksite International; Jannette M. Oliveri, legal administrative assistant, SJPO; John Tenet, legal counsel, SJPOA; Capt Walt Tibbet, SJPD.
Aaron Marcus and Associates
About the Author:
Aaron Marcus is the founder and president of Aaron Marcus and Associates, Inc. (AM+A). He has degrees from both Princeton University and Yale University, in physics and graphic design, respectively. Mr. Marcus has published, lectured, tutored, and consulted internationally for more than 30 years.
For more than four years, Jim Gasperini has worked with AM+A clients with a particular focus on user and needs analysis, writing/documentation, and conceptual design. At AM+A, he has completed several projects for Visa and completed user research and analysis for a major initiative of Wells Fargo Bank. He has also conducted usability analysis for McKesson, the San Jose Police Officers Association, Kaiser Permanente, and FileMaker. Prior to working with AM+A, Jim's credits include, among others, Lead UI Designer for Electronic Arts' SimCity 2000.
©2006 ACM 1072-5220/06/0900 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.