Forum: timelines

XIII.2 March + April 2006
Page: 46
Digital Citation

The GUI shock: computer graphics and human-computer interaction


Authors:
Jonathan Grudin

"Everything changed—the computer graphics community got interested in realism, I remained interested in interaction, and I eventually found myself doing HCI."
—William Newman

Someone recently wrote that SIGCHI was formed by computer scientists. This was an understandable error, but in addition to erasing several years of history, it glided over arguably the most wrenching transformation in human-computer interaction (so far). The transition from interfaces based on commands, forms, and full-page menus to graphical user interfaces based on windows, icons, and the mouse created opportunities, presented challenges, and wreaked havoc with some dominant HCI research programs of the early 1980s. Another such disruption is entirely possible.

Ivan Sutherland’s Sketchpad, initiated in 1960 on the MIT Lincoln Labs TX-2 to make computers "more approachable," launched computer graphics. To do so, Sutherland used the $10 million machine as a personal computer. (All prices in this article are converted to 2006 US dollars [1].) Sutherland’s appealingly informal 1963 thesis, beautifully digitized by Alan Blackwell and Kerry Rodden [2], introduced fundamental work on iconic representations, object-oriented techniques, constraints, interaction techniques, and approaches to animation. Sutherland noted economic as well as conceptual possibilities for digital cartoon creation 22 years before Toy Story. His goal of applying graphics to engineering practice was in his words "a big flop," but the refreshingly frank analysis of that experience helped subsequent researchers develop computer-assisted design, adding to the value of a wondrous thesis.

The TX-2 was very limited. Since then, aspects of computing capability have increased almost a billion-fold. This staggering change is easily overlooked when we focus on conceptual development, which has advanced much more slowly, but it is at the heart of how and when computer graphics and HCI research came together.

HCI and computer graphics looked at hardware change differently. HCI research and development emphasized widely used or potentially widely used systems and applications. Therefore, it focused on relatively modestly priced systems or terminals. To HCI researchers, hardware advances meant greater capability at a fixed low price. Computer graphics research focused on very high levels of computation. Hardware advances meant lower cost for a fixed high level of computation. The details in Figure 1 (page 45) can be debated, but the contrast is clear.

The difference in perspective had two effects. Computer graphics researchers were drawn to a handful of heavily funded centers—MIT, University of Utah, NYIT, Xerox PARC. This is not unusual: Leading physicists clustered around major observatories and accelerators; neuroscientists gathered in Positron Emission Tomography (PET) scan centers. But in contrast, useful human-computer interaction research could be conducted by anyone with a computer. The second effect was that the two fields remained largely apart until the price of computation declined.

Computer Graphics Through 1976: Visions, Prototypes, and Algorithms

In 1968, when Sutherland joined David Evans at the University of Utah, a fully equipped PDP-1 or PDP-7 cost on the order of $1 million. The list price of a preferred high-resolution display was over $100,000. Understandably, computer graphics research had little bearing on widespread human-computer interaction of the time.

In the 1960s, visionary writers and engineers including Sutherland, J.C.R. Licklider, Douglas Engelbart, and Ted Nelson described a future of discretionary computer use by professionals. In the early 1970s, some members of a second generation of computer graphics researchers grappled in earnest with how interaction might occur in this future of "augmented intellect." Many had been influenced by Sutherland and the others as students in the late 1960s and spent time at Xerox PARC in the mid-‘70s.

Ron Baecker, James Foley, and William Newman made significant contributions to graphics in the ‘70s before shifting to human-computer interaction more broadly a decade later. Foley and Wallace’s paper in the 1974 Proceedings of the IEEE special issue on computer graphics began with a call for user-centered approaches: "The design of interactive graphic systems whose aim is good symbiosis between man and machine involves numerous factors…." [3, p. 462]

Microcomputers attracted notice in the mid-1970s. Home computers and game consoles (e.g., Altair 8800, 1975; Atari 2600, 1977) were not powerful enough to interest most computer graphics researchers, but the Alan Kay and Adele Goldberg [4] vision of a personal computer was within range. At PARC, Larry Tesler and Tim Mott saw that the more powerful (and expensive—initially $80,000 to manufacture) Alto could support an interface accessible to untrained, nontechnical people. They employed user-centered methods in 1974 in designing the GYPSY text editor that, together with Butler Lampson’s and Charles Simonyi’s Bravo editor, preceded Microsoft Word.

As noted by William Newman in the quotation starting this article, the graphics research community was dividing. Many continued computation-intensive research that focused on photo-realism. Others were primarily interested in supporting interaction design. The latter group attended SIGGRAPH—18 papers in the first conference in 1974 had "interactive" or "interaction" in their titles. Then, in 1976, some met independently at a two-day SIGGRAPH-sponsored workshop in Pittsburgh.

UODIGS 1976

Participants in the "User Oriented Design of Interactive Graphics Systems" workshop included Jim Foley, William Newman, John Bennett, and Tom Moran. J.C.R. Licklider and Nicholas Negroponte presented, perhaps making this the symbolic end of the visionary period. UODIGS’76 was an idea whose time had not yet come. It was not repeated, and the 150-page proceedings were ignored. Not until 1981 was another such "user-oriented design" conference held, after which they have been held every year.

This work still required machines too expensive for widespread deployment. HCI research that focused on pre-GUI interaction driven by commands, forms, and full-screen menus gathered momentum elsewhere—the Computer Systems Technical Group of the Human Factors Society formed in 1972-1973 and became the society’s largest Technical Group. The curves in Figure 1 remained far apart, but not for much longer. At UODIGS’76 [5], Licklider saw it clearly:

"Interactive computer graphics appears likely to be one of the main forces that will bring computers directly into the lives of very large numbers of people during the next two or three decades. Truly user-oriented graphics of sufficient power to be useful to large numbers of people has not been widely affordable, but it will soon become so, and, when it does, the appropriateness and quality of the products offered will to a large extent determine the future of computers as intellectual aids and partners of people."

1981-1985: The Spread of Graphics Research and Discretionary Computer Use

In the span of a few years, the Xerox Star, IBM PC, Symbolics and LMI Lisp Machines (all 1981), Apollo (1981), Sun Microsystems (1982) and Silicon Graphics (1984) workstations, and the Apple Lisa (1983) and Macintosh (1984) were launched. This coincided with the minicomputer peak—Digital, Wang Laboratories, and Data General had undermined the mainframe market.

High-end workstations (around $100,000), although not mass-market items, freed the graphics researchers who focused on high performance from the need to be in heavily financed labs. (Similar dispersals of research capability occurred recently in neuroscience when fMRI scanners became commodities, and to some extent in physics, through Web-based sharing of data collected at major centers.)

The CHI focus on discretionary hands-on use emerged. SIGCHI formed in 1982, building on the 1981 SIGSOC conference in Ann Arbor and the independently organized 1982 Gaithersburg conference.

When CHI’83 met in December 1983, several commercial GUI systems were on the market, including the Star and Lisa. All were failing. Some were priced too high; others were too weak to devote sufficient cycles to the interface to do justice to a GUI. As suggested in Figure 1, the Star and Lisa were not powerful enough to support high-end research or inexpensive enough for the mass market. Their internal cost to Xerox and Apple would put them on the curve that supported internal research in HCI, following the Alto.

Major CHI’83 topics included command languages, text editors, psychology of programming, cognitive models, documentation, prototyping, and evaluation. Only one of the 58 papers described a GUI; the 58 papers cited 633 works, none by Sutherland—or Vannevar Bush or Douglas Engelbart. Nor was their influence simply taken for granted: Many researchers present, myself included, were not yet aware of their contributions as we worked on pre-GUI interface issues.

Who was cited? Gordon Bower, Jerome Bruner, Herb Clark, Philip Johnson-Laird, Annette Karmiloff-Smith, George Miller, Ulric Neisser, Eleanor Rosch, Roger Schank, Roger Shepard, S.S. Stevens, Anne Treisman, Endel Tulving, and Amos Tversky, among others. CHI’83 mainly comprised cognitive psychologists and sympathetic human-factors researchers. Psychology publisher Lawrence Erlbaum was chosen for the first CHI-supported journal, Human Computer Interaction, edited by Tom Moran and appearing in 1985.

1985: A GUI Succeeds

The Macintosh was launched with a January 1984 Super Bowl minute that some consider the most effective advertisement in history [6]. But not effective enough: The Mac appeared to be failing and taking Apple down with it a year and a half later. Macs by then had four times as much RAM and an optional LaserWriter to support software that appeared in 1985: Microsoft Word and Excel for Mac, Aldus Pagemaker, and Adobe PostScript. Consumer interest rose and desktop publishing was established on the Macintosh platform.

The success of graphical user interfaces now seems to have been inevitable, but it wasn’t seen that way at the time in the HCI community. GUIs had disadvantages: An extra level of interface code increased development complexity and created reliability challenges; GUIs consumed processor cycles and distanced users from the underlying system, which, many believed, experienced users would have to learn eventually. In 1986, Jack Carroll and Sandra Mazur published a study showing that people familiar with existing interfaces found GUIs confusing and problematic [7]. The same year, in an influential essay on direct manipulation interfaces, Ed Hutchins, Jim Hollan, and Don Norman concluded that that "it is too early to tell" how GUIs would fare [8]. GUIs could well prove useful for novices, they wrote, but "we would not be surprised if experts are slower with Direct Manipulation systems than with command language systems." Most human-computer interaction research had focused on expert use, so this valid insight seemed significant. In a 1985 focus group, a high-level manager stated emphatically, "There will never be a mouse at the Ford Motor Company!" But consumer emphasis shifted to first-time use, hardware and software advances ameliorated other difficulties, and GUIs were here to stay.

The Late 1980s: CHI Embraces Computer Graphics

The curves in Figure 1 intersected: SIGGRAPH participants interested in interaction design found a home. In 1987, the CHI conference was cochaired in conjunction with the leading Canadian graphics conference by Ron Baecker and Bill Buxton. They (and Jim Foley) had been involved with CHI from the start, but not strictly as graphics researchers. Along with William Newman, who became involved with CHI in 1988, they had for some time engaged with related aspects of interaction, including psychology of programming, user interface management systems, and managing and guiding design.

Technical work thrived in SIGGRAPH as realism found markets in games and film, but less HCI research was showcased. SIGGRAPH’86 saw only one paper with "interactive" or "interaction" in the title—by Brad Myers and Buxton, both of whom subsequently devoted most of their attention to CHI. Baecker presented several times at SIGGRAPH between 1976 and 1981, then shifted to CHI.

The GUI Shock

Human Factors and Information Systems researchers had five years to prepare for the graphical user interface to succeed in Microsoft Windows, a platform significant to their principal government and industry clients. In CHI, where the appeal of the interaction style was easily confirmed, the impact of the 1985 success was immediate and profound.

Technical complexities that had caused CHI behavioral scientists to distrust GUIs now led them to embrace computer science with more passion than before. As it did so, CHI loosened other ties: Many academic cognitive scientists who had been actively involved were gone by 1986 and not replaced (e.g., Don Foss, Roger Schvaneveldt, John Black, Misha Pavel). So were most human factors and ergonomics researchers (e.g., Richard Pew, Robert Williges, Brian Shackel, Paul Green), leaving only the CHI conference subtitle, "Human Factors in Computing Systems," as a reminder of what began as a partnership.

Nevertheless, cognitive psychologists, mostly working in industrial research labs, continued to exert an influence. Discomfort among CHI leaders with computer science is reflected in the decade’s delay in launching ACM Transactions on Computer-Human Interaction and in the movement of several senior researchers into schools of information science. Another legacy of behavioral science is CHI’s slow acceptance of visual design despite its importance to discretionary use.

By vastly expanding the design space, GUIs created great opportunities and introduced a wealth of research issues. They also abruptly terminated interest in other research issues, with disruptive consequences for many of us. The dream of building a comprehensive theoretical foundation for HCI was shattered for most who had entertained it. Years of research had been invested in tasks such as command naming, but papers and books on such topics would not be read or built upon. A central skill that psychologists had brought, formal experimentation, was less effective in addressing large design spaces. "Discount" and qualitative methods became more prominent.

The Underestimated Role of Technology Change

Progress in conceptual development has been gradual. Ivan Sutherland outlined many graphical interface concepts in 1963. Effects of nonlinear or exponential hardware growth have at times not been gradual. Even with 20 years’ advanced notice, few of us were ready when hardware change made the GUI practical in 1985. Effects may be imperceptible for decades, but when change comes, it comes with blinding speed. For HCI, it happened in 1985. By fully understanding that experience, perhaps you can be better prepared when it happens again. It may happen tomorrow. It may be happening now.

A Tip of the Hat…This timeline was inspired by a conversation with Turner Whitted, who noted the computer graphics research focus on the cost of computation. William Newman and Ron Baecker helped improve earlier drafts. Significant assistance was also provided by David Lenorovitz.

References

1. ftp://ftp.bls.gov/pub/special.requests/cpi/cpiai.txt

2. http://www.cl.cam.ac.uk/TechReports/UCAM-CL-TR-574.pdf

3. James D. Foley and Victor L. Wallace, The Art of Natural Graphic Man-Machine Conversation. Proceedings of the IEEE, Vol. 62, No. 4, pp. 462-471, 1974. The special issue is at http://ieeexplore.ieee.org/xpl/tocresult.jsp?isYear=1974&isnumber=31179&Submit32=Go+To+Issues

4. Alan Kay and Adele Goldberg: Personal Dynamic Media, IEEE Computer, Vol. 10, No. 3, pp. 31-42, 1977.

5. http://dx.doi.org/10.1145/1024273

6. To view the ad, follow links at http://en.wikipedia.org/wiki/1984_%28television_commercial%29

7. John M. Carroll and Sandra A. Mazur: Lisa Learning, IEEE Computer, Vol.19, No.11, pp. 35-49, 1986.

8. Edwin L. Hutchins, James D. Hollan, and Donald A. Norman: Direct manipulation interfaces. In D.A. Norman and S.W. Draper (eds.), User Centered System Design. Mahwah, NJ: Lawrence Erlbaum Associates, 1986, pp. 87-124.

Author

Jonathan Grudin
Microsoft Research
jgrudin@microsoft.com

About the author

Jonathan Grudin is a senior researcher in the Adaptive Systems and Interaction group at Microsoft Research. He has been active in CHI and CSCW since each was established. His Web page is http://research.microsoft.com/~jgrudin.

Figures

F1Figure 1. Computers that were focus of research communities, in 2006 US dollars.

©2006 ACM  1072-5220/06/0300  $5.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.

 

Post Comment


No Comments Found