In Part I (”Design Without Research?” interactions May + June, p. 68) we looked at some different approaches to design that do or do not succeed by omitting research. Here, we examine some of the limitations of doing research without design.

A startup approached us just before it began manufacturing and asked us to do a last-minute check with target customers to verify that it was going in the right direction. The company was developing an iPod accessory that would provide protection and allow connections to a range of other devices. Not until the kickoff meeting was I able to see what our participants would be evaluating: a raw, unfinished model made of thin plastic, painted a flat gray. There was a visible gap between the various pieces, and it squeaked when the parts moved. It contained no electronics, so it weighed only a few ounces; in short, it was a far cry from what the startup intended to manufacture.

Not surprisingly, when people looked at it next to their glossy iPods, they were unimpressed. It wasn’t attractive, it gave no appearance of sturdiness (especially when the pieces separated as I handed it to them!) and no amount of reassurance—”the final won’t look like this” and “the actual product will be made of sturdier plastic”—was successfully persuasive. We even brought in renderings of the final design, showing the level of finish, but once participants had held this development artifact in their hands, this aspect of the design was impossible to discuss seriously.

The research was far from worthless; it was effective in surfacing issues around the possible role of the device. Seeing this rough prototype (although our client corrected me in one user session by explaining that in fact this was an “appearance model,” not a prototype) led people to question just what this thing was and why they would want it. Indeed, our client had articulated only a specification, not a use case, and the reactions pointed clearly to the work that we needed to do: come up with the story for this product. While we were able to gain some usability feedback, we were mostly testing the design and implementation of the prototype, not the product. It was a good lesson learned, and fortunately, we were able to identify the most important (if previously unasked) question: What is this thing, and why do I want one? As of this writing, the client is trying to license its product out to another manufacturer, so we await the final answer to that question.

Our client had the right idea—get feedback on something unfinished in order to improve the finished product. Unfortunately, aspects of the object were so unfinished that people were unable to make the leap from the prototype (excuse me, appearance model) to the real thing, and the outcomes shifted away from usability and aesthetics toward high-level concept validation. Given that, there’s always the opportunity to create something specifically to provoke people around the deeper issues we want to explore. Imagine a mobile phone that is the size of your thumbnail; while not easily manufacturable or usable, as a concept intended to gather a reaction, it can be remarkably effective. In this engagement, we might have chosen different prototypes to better explore the questions our client was trying to address.

Another client approached us last year with an interesting challenge: It had developed a set of alternative designs for an installer application, and the company wanted to understand which one was easier to use. Additionally, in order to appropriately allocate development resources, the client wanted us to identify some measure of how much easier to use one was over another. We quickly negotiated a methodology, a budget, and a timeline, and set about preparing the research activities. When we asked for the design artifacts to be tested, we received a set of raw materials: the existing software, an interactive prototype from previous research, and a highlevel narrative describing the key differences between each version. We pushed the client hard to clarify exactly how each of these solutions should look and work. Extracting these details was such an extensive process that this was one of the few times we’ve ever had to tell a client that it exceeded the agreed-upon scope and we had to revisit our budget.

By the time we had a barebones representation for each of these alternative designs, we were down to the wire for conducting the actual research, and even though there were some obviously quirky aspects of the sample screens, there was no opportunity for further refinement. We showed the designs to people, and the sessions weren’t for naught—we learned some interesting things (for example, the presumptive best design wasn’t always preferred). We also identified some key principles for designing a piece of software: provide feedback, make the choices clear, provide information to support choices, and other heuristic-style feedback. We uncovered no-brainer principles that any good UI designer already knows.

Our original research question was to identify the usability differences between several alternative designs. But the unaddressed, “low-hanging fruit” UI shortcomings distracted people from that key question and introduced bias (as these shortcomings were not consistently distributed across the various prototypes).

Ultimately, we recommended that in the future the client put in some design effort before going into user research. The pushback was quite strong. Since the next step was an $80,000 design project with a big-name agency, the client didn’t want to spend that money without knowing more about what people wanted.

A good point, but we didn’t recommend this company spend $80,000. How about spending five percent of that amount to solve some of the obvious problems up front? That idea did not resonate; it seemed that doing “design” meant the full-meal deal, rather than a skill set that could be applied at many points along the development process.

In other situations like this, when we’re brought in to help refine an existing design, our clients frequently expect that we’ll peel away an artifact from their development process and show that same artifact to people in order to see what they do or don’t like about it. The classic paper about this issue is Stephanie Houde and Charles Hill’s “What Do Prototypes Prototype?” (http:// www.viktoria.se/fal/kurser/ winograd-2004/Prototypes.pdf). The authors explain there are three key testable aspects of a prototype: the role this thing could play in people’s lives; how usable this thing is; and the appeal of the visual, aesthetic, tactile, and so on. Houde and Hill’s major point is that any prototype can embody two of those three aspects at most. Trying to test all three with one artifact is not effective.

Lately, we’ve found that some clients we work with expect to ask research participants complete feature specification questions. Rather than our normal approach, which is to help people engage with a concept broadly on their own terms, identify desired features, and then prioritize those features, we’re being asked to essentially interrogate people to directly answer the engineering question that the client’s team faces. We’re big advocates for understanding the difference between the question you want to answer and the question you choose to ask. “What is the minimum wireless range for which you’d still consider buying this product at your preferred price point?” is not very navigable by someone who is seeing a new idea for the first time. While we don’t ask questions like that, the misguided belief that we should can impact how our findings are accepted.

In a recent project, we showed a number of alternative designs for a future product. One was high-end enough to be almost sci-fi; another was familiar to people using mobile devices; and another was a kludgey workaround. When we showed the third alternative, some people said nothing at all, while others changed the topic. Both responses seemed like a pretty clear value assessment, but when we met with our client, he challenged us to identify all the individuals who explicitly rejected this solution, as if that would be the proof the idea was not viable. We pointed to the passion that other features evoked and suggested this was the best indicator of what people wanted, but we had clearly different ideas about how the insights from the research were going to inform the design and development efforts. We’d hope to see a design effort translate the insights from the research into useful and buildable solutions, rather than take technically achievable solutions and use research to design a feature specification.

In each of these examples, the absence of design from the research process hampered the impact of the research. When we’re using research to understand whether or not a concept is going to address people’s needs, we need design to create the best representation of that concept, and we need design to translate the output from that research into the next iteration of that concept. We can conclude that research needs design, before and after. Rather than treat research and design as separate activities (sometimes performed by siloed departments or vendors), I would encourage all the stakeholders in the product development process to advocate for an integrated approach in which design activities and research activities are tightly coordinated and aligned.