William Yurcik, Ramona Thompson, Michael Twidale, Esa Rantanen
System administrators are users too! While computer security usability research has largely been focused on end user issues such as password authentication, browser transactions, and Email handling, the needs of system administrators (sysadmins) have been relatively ignored . The size and complexity of the services and issues that sysadmins manage continues to grow, involving yet more components and interdependencies between systems. In addition, sysadmins are responsible for a wide range of tasks requiring disparate skill sets ranging from installation and configuration to monitoring, patching, and debugging. While some system administration is automated, far more of the work involves manual intervention than many people assume . Sysadmins are still very much in the loop, especially during emergencies, and as we consider next, during hacking attacks.
Imagine you are put in charge of protecting 1000 users on an organization’s network from Internet hackers. What tools would you like at your disposal to help you in this task? You would think you should have tools to answer basic questions like: Is there a problem? Where is the problem? What is the problem? The need is for both a broad overview of the overall state of the system and the activity within it as well as for detailed information about particular threats, both potential and actual. State-of-the-art tools to help answer these basic questions either do not currently exist or if they do exist they may be so unreliable as to be ineffective. Because of this, the job of a security sysadmin is similar to a detective piecing together clues with decision-making under uncertainty. This situation is less than ideal, particularly as we would prefer to prevent attacks rather than reacting to a security event after the fact.
There are effective security visualization tools, but they do not seem to get used. Why? Empirical results show that visual interfaces consistently outperform text, auditory, and tactile interfaces . There have been three "Visualization for Computer Security (VizSEC)" workshops dedicated primarily to visual tools  and there are visual tool contributions consistently at the annual Usenix Large Installation System Administration (LISA) conference . Many of these visual tools have been designed with user-centered best practices based on Shneiderman  and Tufte . Some of these visual tools have undergone usability studies with results showing their effectiveness in security administration tasks.
However, field studies of different types of sysadmins have consistently found that they effectively ignore visual tools in favor of home-brewed perl scripts and cryptic command-line tools such as low-level unix commands [1,3]. This is naturally frustrating to the developers of these visual interfaces, given that there nevertheless appears to be a clear need for such tools. Current tools view only a portion of the network, for example, bit patterns in packet data fields (signature-based intrusion detection systems), syslog messages from individual machines, vulnerability assessments from system scanners, and blocked connections from firewalls logs. Security sysadmins are trying to address this problem by independently developing their own specialized tools.
The problem is that sysadmins do not design tools based on user-centered principles: (1) specialized tools typically have textual input and output (which is, admittedly, far easier to develop), when good visual interfaces may be more effective; (2) typically several tools are used simultaneously in order to correlate information but current tools are not designed to integrate; (3) each tool has its own cryptic command set, creating a steep learning curve; and (4) new tools render previous ones obsolete, so there is continuous relearning.
We believe we have developed a visual tool that will make a differencebut acceptance has been slow. VisFlowConnect-IP was developed during 2003-2004 based on requirements gathered from security-system administrators . The tool design was successful as a research activity, generating several peer-reviewed papers, it has been packaged for point-and-click installation, results from pilot studies with targeted users are very encouraging, and we are spreading the word at conferences and via mailing lists. VisFlowConnect-IP answers the basic question: Who is connecting to whom on my networkinformation which is highly relevant for most security events. The tool is free and downloadable from the Internet. So why aren’t more system administrators using our tool?
Old Familiar Ways
Security-system administrators prefer to use the old tools they know and have experience with rather than learn new tools. In our working with security-system administrators we consistently find that while old command-line sysadmin tools may be less effective and difficult to learn for novices, unix system commands and home-brewed scripts in the hands of experts with years of experience may be more effective (or perceived as more effective). For example, sysadmins are able to work quickly with unix shell commands (e.g. grep, text editors) to filter through data. Even if the input or outputs are long, they can benefit from shell capabilities to auto complete commands with the tab key or cut-and-paste with a mouse. We are currently performing usability studies now to test whether the perception of greater speed is matched by actual superior performance.
In addition to the sysadmins’ preference for command-line interface for tool usage and data manipulation, visual tools may pose another barrier for their acceptance. Visualizationwhen the raw data is not directly visually perceivablenecessitates some amount of processing and hence represents a certain level of automation. Automation, in turn, raises questions on trust and reliance. The question the sysadmins therefore ask is what data is the graphical presentation based on, what information might be lost in the translation, and whether they should trust what they see and rely on it in their decision-making (i.e., whether an alert was false or based on a real threat). This perception of seeing only part of the data as if through a "keyhole" may indeed be warranted at higher levels of automation. It also poses a special challenge to the designers of visualization tools who should preserve their transparency with respect to underlying raw data.
So if we can’t get security-system administrators to change from their old text-based tools then let’s take the text-based tool and attach a visual interface to it which they may ignore initially but eventually may investigate. This led to the development of VisTextFlow-IP which combined a text shell interface with our visual tool, VisFlowConnect-IP, in a small dashboard/console interface . In the first iteration, VisTextFlow-IP was simply a merging of the two separate interfaces into onenot much different than having the two tools in separate windows. The current design has commands typed into the text shell window automatically highlighting the corresponding areas in the visual window and vice versa (areas of interest highlighted in the visual window causes the text shell window to display corresponding data). Figure 1 shows a working prototype of VisTextFlow-IP created with Java Swing.
Is a combined text/visual tool better than using each separately? VisTextFlow-IP is very different from having VisFlowConnect-IP (or any visual tool) opened and a command-line tool opened at the same timethe actions of one are tied to the other so that the user can see both detail and overview by interacting with just one of the tools. There is also the reduced cost of switching between two separate tools. We do not force the security-system administrator to use either tool, but allow them to naturally migrate to use that is most comfortable to them. Beyond our specific application for security-system administration, this looks to be an exciting new way to introduce visual tools in a non-obtrusive manner for any expert domain with legacy command-line text tools. With a combined text/visual interface, users can continue using command- line text tools but interest can be spurred in visualization introducing them to simple things that they can do with the visualization that may change how they interact with the command line and spiraling learning may occur.
Where do we go from here? We want to know more about why sysadmins are so conservative in technology adoption, a characteristic often ascribed to far less technologically adept groups of users. How much is it the cost of learning versus integrating the tool into current work practices? How much is due to doubts that the tool will be effective enough or reliable enough for time critical real-life emergency use? With closer studies of sysadmins’ current work practices we hope to uncover more information and use this to develop functionality and interfaces that enable both adoption and effective use.
The dashboard solution of presenting both raw data in text format and visual representation of them simultaneously allows sysadmins to take advantage of the visual tool’s power in making subtle patterns and trends in large amounts of data perceivable very quickly, while furnishing the associated raw data for immediately verification of the patterns gleaned from the graphical interface. The dashboard also allows for adoption of flexible strategies in response to situational demands; at times of low time pressure sysadmins may take advantage of the textual data to examine them in great detail while under emergencies or when critical decisions must be made quickly they may rely primarily on the visual interface.
1. Barrett, T., Kandogan, E., Maglio P.P, Haber, E.M., Takayama, L.A., & Prabaker, M. (2004). Field Studies of Computer System Administrators: Analysis of System Tools and Practices. ACM Conference on Computer Supported Cooperative Work (CSCW).
2. Barrett, R., Chen, Y-Y M. & Maglio. P.P. (2003). System Administrators are Users, Too: Designing Workspaces for Managing Internet-Scale Systems. ACM CHI Workshop: System Administrators Are Users, Too: Designing Workspaces for Managing Internet-Scale Systems.
3. Kandogan, E. & Haber, E.M. Security Administration Tools and Practices. chapter within Security and Usability: Designing Secure Systems People Can Use. (2005) edited by L .F. Cranor and S. Garfinkel, O’Reilly Press.
University of Illinois at Urbana-Champaign
Ramona Su Thompson
University of Illinois at Urbana-Champaign
Michael B. Twidale
University of Illinois at Urbana-Champaign
Esa M. Rantanen
University of Illinois at Urbana-Champaign
About the Authors:
William Yurcik is a manager security R&D and senior-systems security engineer for the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign. His research interests include visualization tools for security administration and threat modeling. He organized and chaired/co-chaired the three Visualization for Computer Security (VizSEC) Workshops in 2004, 2005, and 2006.
Ramona Su Thompson is currently an MS student in the Human Factors Department at the University of Illinois at Urbana-Champaign (UIUC). She received an MS in Computer Science, with a focus on HCI, from UIUC in August 2005.
Michael B. Twidale is associate professor in the Graduate School of Library and Information Science at the University of Illinois at Urbana-Champaign. His research interests include computer-supported cooperative work, computer-supported collaborative learning, museum informatics, and usability engineering. He is particularly interested in developing rapid lightweight techniques to support use analysis and interface design for collaborative learning environments, ubiquitous computing applications and open source software.
Esa M. Rantanen is an assistant professor of Human Factors at the Institute of Aviation at the University of Illinois at Urbana-Champaign. His principal areas of scholarship are human factors in aviation systems, human performance measurement and modeling, mental workload, decision making, and human error and reliability. He is particularly interested in human timing of actions, temporal decision making, and the effects of time pressure and temporal uncertainty on workload and performance.
©2007 ACM 1072-5220/07/0100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2007 ACM, Inc.