XVIII.1 January + February 2011
Page: 51
Digital Citation

Visual analytics and human-computer interaction

Richard Arias-Hernández, John Dill, Brian Fisher, Tera Green

back to top 

Defined as "The science of analytical reasoning supported by interactive visual interfaces" [1], visual analytics provides a scientific perspective on human interaction with complex graphical displays. Its goal is to support human decision making in situations characterized by complex "wicked problems" in diverse application domains. This article discusses the ways in which visual analytics research contributes to ongoing efforts in human computer interaction that address cognitive task performance and how it is affected by highly interactive "human-information discourse" with visualization of data, information, and knowledge. Our conclusion is that the challenges posed by visual analytics will require HCI practitioners and researchers to expand their collaboration with cognitive scientists and visualization and computation researchers.

Consider the following scenario: When the hurricane came ashore, it destroyed both physical and information infrastructure. Emergency managers were overwhelmed by conflicting reports from media and the public, variability in weather prediction and estimates of the probability of infrastructure failure and its human cost. New concerns arose about the increasing dependency of physical infrastructure—transportation, power, and medical—on cellular, Internet, and related information and communication technologies. For all of these information sources and concerns, there was no effective way to confidently filter accurate and relevant information from data that might be inaccurate, outdated, or irrelevant. Overwhelmed by volumes of complex and uncertain data, analysts were unable to formulate an accurate model of the events that were transpiring and to formulate effective plans of action. When decisions were made, there was no way to ensure that commands were effectively communicated to the field personnel or to accurately monitor and direct their operations. Emergency workers had valuable information but were unable to communicate it to a specific team or to coordinate their operations with nearby police, fire, medical, and infrastructure workers. With this breakdown in information management, emergency workers did the best they could with what they knew, and hoped that their best would be enough.

Scenarios like this one occur to some degree during any natural disaster near a population center. Less dramatic examples of the impact of information volume, uncertainty, and complexity on decision making can be found in a broad range of human activities. What is common among all of these situations is that there is a glut of heterogeneous information that must be analyzed and acted upon in a timely fashion. In many cases the information and its analysis are distributed across multiple decision makers who must work together to coordinate their activities. The combination of cognitive complexity, broad range of applications, large data sets, and new display and interaction technologies has driven a reevaluation of development methods for interactive visualization systems. This reevaluation gave rise to the new field of visual analytics. While methods for creating interactive visualizations are reasonably well understood, visual analytics takes a novel, interdisciplinary approach to directly address the process of human reasoning with the aid of interactive visualization tools. Visual analytics combines scientific investigation of human perception, cognition, and interaction with computational, mathematical, and statistical approaches to processing massive data sets that may contain uncertain or erroneous data. It integrates statistical and modeling analyses with human decision making through the use of interactive visualization to address real-world "wicked" problems characterized by large and uncertain data and pressures of time, limited resources, and potentially profound consequences.

The visual analytics research agenda was developed by visualization researchers from academia, industry, and government labs on the request of the U.S. Department of Homeland Security. Its objective was to address analytical challenges in intelligence analysis and public safety posed by the threat of terrorism and natural disasters. The scope and application of visual analytics has expanded over the years [2], but it maintains its focus on the design, implementation, and evaluation of human interaction with visual representations of information to support individual and collaborative analysis and decision making.

A key tenet of the visual analytics approach is the realization that technological solutions are insufficient to deal with the complexities of information and the need for novel solutions. Human analysts must utilize information technology to achieve insight and creative solutions to new and evolving problems characterized by massive and complex data. In defining visual analytics as a science of analytical reasoning facilitated by interactive visualization technologies, the founders of visual analytics placed equal emphasis on advancing our understanding of human cognitive processes and on the design, implementation, and evaluation of models and algorithms for processing (e.g., machine learning), displaying, and interacting with information.

From this perspective visual analytics and HCI share much common ground. Both take the human-computer system as a meaningful unit of analysis as they seek to understand the complexity and diversity of users' abilities and activities in technological contexts. In the case of visual analytics, the focus is more precisely tuned to human perceptual, cognitive, and collaborative abilities as they relate to the dynamic, interactive visual representations of information in the context of analytical approaches and strategies in the target domain. Visual analytics places particular emphasis on cognitive tasks, and specifically tasks that are ill-defined, e.g., "to detect the expected and discover the unexpected" [1]. The emphasis on complex cognition, hypothesis formation, and evaluation in visually rich settings requires visual analytics researchers to integrate their approach and methods with evolving theories of cognitive expertise, creativity, pattern recognition, and visual reasoning from cognitive science.

In making the transition to visual analytics work, traditional HCI methods such as protocol analysis, cognitive task analysis, and activity theory must be adapted to shed light on the use of complex visual displays in analytic processes. The science of visual analytics is unusual in that it was created in dialog with a particular approach to solving complex problems, the use of interactive and sometimes mixed-initiative human-computer visualization environments. This dialog between science and application creates constraints as well as opportunities for both. For understanding and advancing human interaction with visualization systems, challenges to HCI might be:

  • Graphical validity. Researchers must address complex perceptual environments and tasks. This requirement eliminates the vast majority of current laboratory studies in perception and cognition.
  • Systems thinking. Research teams must extend the social science model of description of human abilities in natural situations to understanding how human/technology systems will perform their tasks.
  • Situational constraints. Researchers must extend their understanding of human performance in novel situations, under time pressure etc. to enable "satisficing" performance of the human operator under those conditions.
  • Individual differences. Research must address performance differences within a population of users, e.g., cultural differences, levels and types of expertise (especially the unique characteristics of highly skilled individuals, "thinking styles," etc.).
  • Strong prediction. A science of interaction must move beyond interface-design guidelines to provide specific predictions of performance. While inferential hypothesis testing has its place, mathematical and computational models whose free parameters can be fitted and evaluated for a given user and context of use provide more explanatory power and should be emphasized.

For example, in our own visual analytics research, we have sought to understand the impact of individual differences in the perceptual and cognitive abilities of analysts on their capacity to use specific visual analytics systems. Empirically validated evaluation metrics could help to determine what effects individual differences (both innate and acquired through experience with interactive visualization) in perceptual and cognitive abilities might have on the ability to gain insight through interaction with different visual representation of information. In pursuit of these predicative models, our research in the Personal Equation of Interaction (PEI) looks at potential predictors of analytic success that include Trevarthan's Two Visual Systems theory from cognitive neuroscience; Pylyshyn's FINST theory of spatial attention; and commonly accepted psychometric measures such as Rotter's Locus of Control, the "Big Five" personality factors, and Cacioppo's Need for Cognition.

A related thread of investigation in our laboratory examines the social and organizational use of interactive visualization systems. This approach is informed by the HCI tradition of qualitative research—building on sociological and anthropological models and methods to address human-technological systems. Many take a "pair analytics" approach, with an expert visual analytics tool user (a so-called "VA Expert" or VAE) working with a subject-matter expert to perform an analytic task with real data. Our pair-analytics work examines aircraft-safety analysis with the Boeing Company. In collaboration with these analysts, we identified several analytical problems to test the efficiency of visual analytic tools. Our strategy relied on using our own visual analytic experts, who collaborated closely with Boeing safety analysts to solve a real-world problem. This close collaboration circumvented one of the obstacles for technological transition, namely the costs in time and effort for an expert safety analyst to learn a new software application and a new approach to visualizing and interacting with safety data. Video of analytic collaboration could then be analyzed using Grounded Theory and Joint Activity Theory approaches.

Of equal importance for visual analytics are the social and organizational aspects of technology acceptance. Our work with real-world visualization applications demonstrates analytical competence is often distributed across organizational, social, and cultural settings. Technological support for these processes must be adapted to their distributed nature. The focus of attention here expands from the human-technological systems emphasis of pair analytics to socio-technical systems that include deeper analysis of social and cultural interactions. Several challenges, some of them well known to HCI practitioners, are to be addressed here; among them: resistance to technological innovations, cognitive biases reinforced by social structures, and the influence of cultural differences on interpretation of visual representations.

Technological transition strategies to introduce visual analytics into organizations and reduce resistance must be further enhanced. What appears to work best is to first understand organizational culture and goals, then build user engagement in the developing and testing of prototypes, followed by the focused application of existing and prototype visual analyticstools and techniques to relevant analytical problems and realistic data sets. Identifying lead innovators in the organization whose analytical questions are not currently being addressed by available technologies can develop research collaborators and advocates for organizational involvement. This has been the strategy of application-oriented visual analytics developers, such as those in the Charlotte Visualization Center at the University of North Carolina. Its WireVis application focuses on helping expert wire-fraud analysts detect patterns in wire-transfer data that might indicate fraud. In the development of these interfaces, visual analytics scientists work closely with both expert analysts and technology developers to address relevant problems with actual users and real data.

As we have seen, visual analytics has ambitious goals: to support complex individual analytic cognition and social and cultural distributed cognition with one-to-one and one-to-many HCI, and to support these complex cognitive processes under the strain of time, overwhelming data, and limited resources. In pursuit of these goals, cognitive science researchers examine individual cognitive and perceptual processes and the interactive visual images that facilitate them, and interaction and visualization designers create collaborative interfaces that allow for easier sharing and analysis of information between contributing parties. Computational visual analytics researchers develop mathematical and computational tools for mixed-initiative systems, in which the computer can initiate certain tasks or processes in the pursuit of a goal or generated hypotheses. Grounding the entire process are ongoing user tests and user involvement in development processes.

If successful, the technology of visual analytics will test and validate the scientific findings. If the systems it creates can demonstrate quality improvement of cognitive processing that can be brought to bear on real-world problems, we can confirm the value of the science. Analysis of the use of VA systems will generate new scientific questions about the nature of cognitive processing in technological environments that can be addressed through laboratory experimentation. Through this multidisciplinary translational research approach [3], visual analytics seeks to build more predictive cognitive models, science-aware interaction and visualization design methods, and focused interdisciplinary software development processes.

The final challenge that visual analytics faces is perhaps the most difficult to surmount. To return to our disaster scenario, we can see addressing the complexity of this situation will require designers and builders of information and technology to be aware not only of the requirements of the task and the situation of use, but also of the impact of new display and interaction technologies on the perceptual, cognitive, and communicative processes that these users must operationalize if their decisions are to be effective in preserving life and property. We believe this capability can be best achieved with the active involvement of HCI researchers and practitioners in the visual analytics effort. Conferences and workshops in visual analytics can be found at IEEE Visweek, the Hawaii International Conference on Systems Sciences, and the annual VAC Consortium. Active participation of HCI researchers and practitioners in these and other events will be critical for the success of this vital and important approach to the creation of information and communication technologies to meet the needs of society and its institutions.

back to top  References

1. Thomas, J. and Cook, K., eds. Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE Press, 2005.

2. Kielman, J. and Thomas, J., eds. Special issue: foundations and frontiers of visual analytics. Information Visualization 8, 4 (2009).

3. Fisher, B., Green, T.M., and Arias-Hernández, R. Visual Analytics as a Translational Science. Topics in Cognitive Science. (in press)

back to top  Authors

Richard Arias-Hernandez holds a Ph.D. in science and technology studies from Rensselaer Polytechnic Institute. He is a postdoc fellow at the SCIENCE lab, School of Interactive Arts and Technology, Simon Fraser University. His current research focuses on applied psycholinguistics to solve coordination problems in collaborative visual analytics.

John Dill is a Professor Emeritus in the School of Interactive Art and Technology at Simon Fraser University. He received his Ph.D. in philosophy from Caltech, his research focuses on visual analytics. He spent the first part of his career in industry (GM Research) returning to academia in the School of Engineering Science at SFU.

Brian Fisher is an associate professor of Interactive Arts and Technology and Cognitive Science at Simon Fraser University and associate director of the Media and Graphics Interdisciplinary Centre at University of British Columbia. He is co-chair of the 2010 IEEE VAST conference and for the HICSS 44 visual analytics minitrack.

Tera Marie Green is a Ph.D student and graduate research assistant at the School of Interactive Arts and Technology, Simon Fraser University. Her research involves applied cognitive science for visual interfaces, with emphasis on the impact of individual differences on cognition and development of cognitive models for mixed initiative interfaces.

back to top  Footnotes

DOI: http://doi.acm.org/10.1145/1897239.1897249

back to top  Figures

UF1Figure. Canadian sailors assigned to the patrol frigate HMCS Toronto sort boxes of relief supplies at a Salvation Army warehouse managed by FEMA. Canadian Navy and U.S. military forces worked together to aid in Hurricane Katrina disaster relief efforts.

UF2Figure. The destruction of a watertower and buildings in Buras, Louisiana after Hurricane Katrina.

UF3Figure. Widespread devastation after Hurricane Katrina in Port Sulphur, Louisiana.

UF4Figure. Visual analytics as a multidisciplinary intersection of technical and behavioral science and philosophy.

UF5Figure. The multiple of views of the visual analytics tool CzSaw, which keeps a history of analysts interactions for future use through graphical tools and thumbnails. Visual views like the Data View (bottom left) and Dependency View (top middle) are coordinated, meaning that changes in one generates the related change in the other.

back to top  Sidebar: For more information on work being done in visual analytics:

IEEE Conference on Visual Analytics Science and Technology (VAST)

Visual Analytics Community Site

Visual Analytics Digital Library

VisMaster, an European Coordination Action Project

US National Visual Analytics Centre (Pacific Northwest National Laboratory)

Scalable Visual Analytics: Interactive Visual Analysis Systems of Complex Information Spaces

UK Visual Analytics Consortium (UKVAC)

Visual Analytics for Command, Control and Interoperatibiliy Environments (VACCINE)

Integrated Visual Analytics Community

back to top 

©2011 ACM  1072-5220/11/0100  $10.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2011 ACM, Inc.

Post Comment

No Comments Found