The mystery of product development

XVII.4 July + August 2010
Page: 9
Digital Citation

The research-practice gap

Donald Norman

back to top 

"Oh, East is East, and West is West, and never the twain shall meet."
        (Rudyard Kipling, 1892. Barrack-Room Ballads)

There is an immense gap between research and practice. I'm tempted to paraphrase Kipling and say "Oh, research is research, and practice is practice, and never the twain shall meet," but I will resist. The gap between these two communities is real and frustrating. Sometimes the gap is deliberate. Some researchers proudly state they are unconcerned with the dirty, messy, unsavory details of commercialization while also complaining that practitioners ignore them. And some practitioners deride research results as coming from a pristine ivory tower—interesting perhaps, but irrelevant for anything practical. Sometimes the gap is accidental, caused by a misunderstanding on both sides of the requirements and goals of the other. I have heard researchers, who would like their ideas to impact practice, complain that when their ideas do get used, the practitioners do so incorrectly, omitting (or messing up) the most critical aspects. Practitioners, in turn, complain that research results, even if relevant, never exist in any form that can readily be translated into practice.

The gap between researchers and practitioners extends to the professional societies. The major societal home for many researchers is the Association for Computing Machinery's Special Interest Group on Computer-Human Interaction—ACM SIGCHI—the same group that brings you this magazine. Although CHI pretends that it is home for both researchers and practitioners, that is largely a delusion. In the major conferences, most especially its flagship conference, CHI proudly proclaims that it includes people from both universities and industry. Although this is true, the people from industry are seldom the developers and practitioners. Instead, they are primarily researchers who work for industrial research labs. Researchers in companies tend to be far more closely attuned to their academic brethren than to the people within the product divisions of their own companies. This close connection to research and separation from practice are a hallmark of the research community and the CHI conferences. interactions has made a valiant and reasonably successful attempt to bridge the gap, but the fundamental distinction remains strong. I know this problem well because I faced it when I headed a large research group in a product company (the Advanced Technology Group at Apple).

The gap between research and practice is fundamental. The knowledge and skill sets required of each group differ. Consider the research community within design: the area called design research. This community attempts to understand basic patterns of human and social behavior and how technology affects both. Most of the studies focus upon problems and difficulties, in part because they are far easier to study than benefits and changes in work and life patterns, but also because new technologies are mostly accompanied by problems, and most benefits do not show up for a long time, perhaps decades. Other researchers probe the technological boundaries, demonstrating new potential capabilities and new experiences. Both kinds of research are valuable. Both produce insights. But both are far removed from the intense attention to detail, reliability, and robustness that characterize products or the concern with how and what people actually buy, with the cost structure of potential products and the resulting profitability. Practitioners do not have time to deal with debates about the problems and difficulties that people face with new technologies. They do, however, want to hear about the benefits and the new product directions to pursue. Studies of technology-induced alienation or concerns (or lack of concern) over privacy might be of great importance for society, but not for driving the next product cycle.

Even when some research demonstration excites the product side of a company, it is seldom ready for release. Transforming a research demonstration into a practical product that can be sold profitably in the marketplace is a complex and demanding job, a job for which the research community does not have the skills, patience, or interest. The skill sets that make for a creative, insightful researcher are very different from the skills required of development engineers to make something work reliably and inexpensively or of marketing teams who must determine not what people actually need (which is where researchers tread), but what they will actually purchase. Product people have to worry about sales and profitability, reliability and cost. These issues are seldom of interest to researchers and, moreover, are not within their normal skill sets.

I emphasize the difference in required skills between research and practice to ensure that these comments are not taken as criticism. They are meant to reflect the reality that it is rare for a single individual to have the breadth and depth of knowledge and skills to understand business plans and marketing strategy, to know how to lead a team of perhaps hundreds of developers to produce a reliable, bug-free set of code with millions of lines of instructions that can work across the many platforms and perturbations of equipment and applications found in the real world, who can simultaneously make use of all the advanced research learnings of the multiple relevant disciplines: the social sciences, business, and technology. This is why I think the research-practice gap is so universal and so difficult to overcome.

back to top  Reexamining the Basics of Design

The research-practice gap is only one of many problems facing the design profession. One other issue is that many of our basic beliefs about how to develop and design are built upon a shallow, insecure foundation. In the many years I have been writing this column, I have reconsidered some highly cherished beliefs in the practice of design and found many of the principles wanting. We know surprisingly little about how to do design. There is no science of the practice in the same sense that there is a science to the structural analysis of buildings and bridges, or to the building of circuits. Design is still an art, taught by apprenticeship, with many myths and strong beliefs, but incredibly little supporting evidence. We do not know the best way to design something. The real problem is that we believe we do. Beliefs are based more on faith than on data.

This is a problem that confronts all professional disciplines: law, art, music, business, medicine, and design. Each of these disciplines often has some scientific field behind it (e.g., art and music have perceptual psychology; interaction design has well-established psychological roots; many parts of business have a basis in decision theory, economics, and finance; and medicine has biology and chemistry). But even in the fields with a substantive scientific basis, the practical applications to the daily practice are very limited. Thus, although biology is important as a foundation for medicine, it gives no guidance regarding patient-doctor interaction, patient histories, or diagnosis, and it has nothing to say about patient empathy or best hospital practices. In business, finance and economics provide a rationale for investment decisions, but where do the best management principles come from? In law, what science underlies jury selection or presentations? There is a great deal of music theory, but very little is directly relevant to music performance. In the end, practical disciplines are all taught through apprenticeship, internship, residency, and long periods of training.

In science there are clear links among hypotheses, conclusions, and evidence. But in the practices of most professions, the links are tenuous at best. Instead, there is much reliance upon "best practice," where "best" is often defined by short-term measurements, usually of variables that are easy to measure as opposed to those that are the most significant. Long-term measures are seldom taken; methods are seldom compared. Note: It is not easy to figure out how to do these studies or comparisons. Once again, this is not meant as criticism, but rather as a description of the current state of affairs. Scientists usually operate in what has been called "white room" conditions, carefully forming abstract characterizations of the phenomena under consideration and studying them in a controlled research environment or the clean precision of the laboratory. Similarly, the theories are of necessity simplified and abstracted to a pristine form of mathematical or simulation models. Science works best when all the variables are understood and controlled. But the real world is complex and messy, with uncontrolled variables, sometimes behaving in ways that contradict the neat, tidy, logical assumptions of the scientist. No wonder there is a gap.

The lack of scientific studies of practice is due to two things: First, practitioners are not trained in scientific research. They do not understand the need for experimental controls, nor do they understand statistical variability and experimental biases. Moreover, they don't wish to—they want to get on with their work. Second, even when researchers well versed in experimental methods attempt to study practices, they discover the very nature of a practical discipline throws in so many idiosyncratic variables that rigor is simply not possible.

In the field of design, many researchers end up studying the designers themselves. "How do designers think?" is a standard research question. I have seen many studies comparing individuals with groups, or people in one culture with those of another. All of these studies make for interesting reading, but I find them of little value in helping us know how designers ought to work or how they ought to think. As a result, we have many myths about the power of design research, brainstorming, rapid prototyping, iterative test and design, but zero evidence.

When researchers try to collect evidence, they often take a bunch of students, do some simple manipulations, and then try to state a general conclusion. The entire study lasts a few hours or at most an academic quarter or semester. I am continually amazed that the research community believes the study of naive, unskilled students tells us anything at all about the practical problems of design in a large company, with multiple constraints and requirements, working in teams, with highly practiced and accomplished skills. Moreover, real design projects take months and sometimes years. The difference between the researchers' notion of the design setting and reality is immense.

back to top  Misunderstandings: The Case of "Technology First, Needs Last"

In my March + April 2010 column, "Technology First, Needs Last," I stated that design research was quite effective at improving existing products both for their intended uses and also to move them into unexpected application areas, but it played little or no role in original invention. New technology occurred first, I argued, with inventors doing their thing without the benefit of any design research, with few people besides the inventors believing there was a need. This blind, research-free invention often fails, but it is also the source of our major innovations. This approach has led to such breakthroughs as the telephone, phonograph, radio, automobile, Internet, CD, portable music player, and camera. Design researchers had nothing to do with the initial developments. Instead, the researchers have come along afterward and sometimes made valuable contributions, demonstrating how the product could be improved for its intended usage and, more important, by noticing needs not satisfied by the existing product and identifying how it might serve a vastly different audience from the one for which it was originally intended.

The traditional folklore of research and development holds that there is a smooth, steady chain from pure, basic research to more applied research, to advanced products, to commodity products. As a large number of studies of research and development have shown, this nice, logical progression does not exist. Yet the myth persists.

Donald Stokes's book, Pasteur's Quadrant, provides a nice antidote. Here, Stokes argues the most effective research can be seen in what Pasteur did when he developed the smallpox vaccine. He started with a real, practical problem, realized it needed some fundamental scientific advances before it could be solved, did the science, and then applied it back to the problem. In other words, the research was done in search of a solution to a real problem: what Stokes calls "use-inspired basic research." Stokes argued that research can often be characterized along two dimensions. The first dimension is about the kind of knowledge that is sought: fundamental or practical. The second dimension concerns consideration of use: whether it is a search for pure knowledge without consideration of use or whether it is aimed at some fundamental, practical problem.

The two dimensions give rise to four quadrants. Those who seek fundamental knowledge without consideration of how it might be used fit the general view of the impractical longhair mathematician or scientist. Inventors such as Thomas Edison fit the quadrant of searching for relevant knowledge to solve an applied problem, but without any attempt to expand our general understanding of phenomena. Hence, Edison's classic search for the material that would improve the already existing lightbulb, allowing it to function more efficiently for greater duration, is a classic example. Although he succeeded in his quest, he did not advance our understanding of science or engineering. Edison provides an example of someone who did not attempt to add to our fundamental understanding but was consumed with making sure his inventions were practical and useful. He did read the scientific literature, but he did not try to add to it. A third quadrant is filled with tinkerers who produce inventions that neither add to fundamental understanding nor have any use.

For Stokes, the most powerful quadrant is not that of the pure scientist. Rather, it is the quadrant occupied by Pasteur, the quest for fundamental knowledge within a specific use context. This is where the biggest payoffs lie, at least according to Stokes.

In the four quadrants formed by the axes of pure versus applied science, usages versus no use, researchers most often play in the fun quadrant, finding lovely problems to work on without regard for whether anyone cares outside of their fellow research in-group. This is one reason for the research-practice gap. I recommend aiming at Pasteur's quadrant—fundamental research geared to solving important applied problems.

But even if researchers aim at the solution to practical problems, they still face the fundamental differences in the knowledge and skill sets required by those who conduct the research and those who attempt to translate those results into practical, reliable, and affordable form.

back to top  Translational Development

Between research and practice a third discipline must be inserted, one that can translate between the abstractions of research and the practicalities of practice. We need a discipline of translational development. Medicine, biology, and the health sciences have been the first to recognize the need for this intermediary step through the funding and development of centers for translational science. This intermediate field is needed in all arenas of research, and it is of special importance to our community. We need translational developers who can act as the intermediary, translating research findings into the language of practical development and business while also translating the needs of business into issues that researchers can address. Notice that the need for translation goes in both directions: from research to practice and from practice to research.

Translational developers are needed who can mine the insights of researchers and hone them into practical, reliable, and useful results. Similarly, translational developers must help convert the problems and concerns of practice into the clear, need-based statements that can drive researchers to develop new insights. Neither direction of translation is easy.

Great innovations can come from anywhere, anyplace. Usually they come about when a new technology is unleashed upon the world and inventors and technologists scurry to find something they can do with it. Most of these attempts fail, but a few stick. The researchers come aboard after the technology has been unleashed. But this is precisely when they can be most effective, because it is now that they can play Pasteur's game: starting with a real need, figuring out what the scientific requirements are, doing the science, and then feeding the results back to a practitioner community desperately awaiting those findings.

There is a huge gap between research and practice. To bridge the gap, we need a new kind of practitioner: the translational developer. The gap is real, but it can be bridged.

back to top  References

Stokes, D. E. Pasteur's Quadrant: Basic Science and Technological Innovation. Washington D.C.: Brookings Institution Press, 1997.

back to top  Author

Don Norman wears many hats, including cofounder of the Nielsen Norman Group, professor at Northwestern University, visiting professor at KAIST (South Korea), and author. His latest book is Living with Complexity. He lives at

back to top  Footnotes


back to top 

©2010 ACM  1072-5220/10/0700  $10.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2010 ACM, Inc.

Post Comment

@Mohamadou M. Amar (2013 03 22)

I am a Doctoral student in I/O Psychology with Touro UW and need to access your articles.

@Mohamadou Amar (2013 03 22)

Need access for Doctoral Research