As Internet-based services have become daily digital consumables for millions of people, opportunities to learn more about users have exploded. The so-called big data deluge has been going on for nearly a decade. The avalanche of human behavioral information has already been effectively exploited by many large Internet companies such Google, Facebook, and Yahoo. This has resulted in several impactful academic studies (e.g., ) as well innumerable changes to the design of the services themselves.
Wide use of data among digital service companies indicates a move toward an evidence-based design approach. Large-scale experimenting and conducting A/B tests have become standard practices. This trend of data-informed decision making has repeatedly revealed that designer intuition, as trialed in experiments, is wrong most of the time . This approach thus forces designers to embrace failure and have a receptive mindset toward continuous design iteration.
The term data-driven design pops up frequently in association with these approaches  but has not yet been defined. To me, data-driven design (DDD) means tools that assist in design research by automating data collection and analysis. Occasionally they may also offer new ideas or help us make decisions. In the near future, I expect to see an emerging cluster of assistive and agentive types of generative design tools.
The assistive tools will help designers to work more effectively by helping in creative tasks. Agentive tools will delegate design tasks to artificial intelligence and minimize human involvement. The relationship between generative and mainstream DDD tools is not yet clear. Today generative tools play a small role, mostly inside other systems, such as in algorithms that control how users are assigned to experiments. How a big disruption will happen in the future is still unknown.
Across application domains, the process of DDD contains some necessary steps. First you collect data; then you aggregate it, combine it with other data, and analyze it. Once you’ve begun to make interpretations, which preferably are hypothesis-guided, you are ready to make data-informed decisions with regard to your design and business questions. Visualizations can be useful for presenting insights to decision makers. How this all happens in detail can vary dramatically in terms of tools and formal processes. But it always begins with data.
Web interaction design is crowded with DDD solutions. The existence of dozens of commercially available tools validates market demand for such solutions. Additionally, major Internet companies commonly build their own experiment-management platforms to match their specific needs. These enable them to run hundreds of experiments in parallel, thanks to automation.
I classify data-collection and interpretation solutions as either active or passive solutions. This refers to whether the collection requires additional user effort or just records whatever the user normally does with the service.
The next important difference is whether the data collected provides answers to why or what questions with regard to user behavior. You could also see this as the difference between qualitative and quantitative data (or thick and big data). The perspectives are complementary, providing different types of evidence, despite the emphasis on quantitative data in the context of data-driven design.
Active data collection. Active data collection refers to techniques for requesting user input. As a variation of qualitative research, it can answer both how and why questions. While this approach is inherently more ethical than passive recording, a major issue is non-response, as most users are reluctant to participate. Successful contemporary targeted website surveys, such as Qualaroo or Hotjar, are commonly used for presenting simplified measures of user experience, such as the Net Promoter Score (NPS) . Unfortunately, these metrics are of little use for design because they are insensitive, lagging indicators of customer brand loyalty.
There are solutions for co-designing and testing digital services with users, but for the most part they don’t fit within the definition of DDD because of scaling. For instance, using private Facebook groups as online focus groups is an effective, but labor-intensive research method.
Some applications succeed in replacing traditional usability and user experience research methods in a scalable way. For instance, solutions for conducting card sorting for hierarchical menu design have been developed by Optimal Workshop and UsabiliTEST (Figure 1). These types of tools can be utilized without a functional product. They also reduce human effort by automating the analysis of user patterns, making them excellent examples of DDD tools.
|Figure 1. Web-based card-sorting tool OptimalSort enables hundreds of users to be tested quickly.|
Passive tracking. Passive tracking describes any kind of under-the-surface recording of user activity. These tools usually solve the challenges of automated collection, aggregation, and analysis all in one.
Passive tracking is the de facto standard practice. It has been estimated that nearly 70 percent of Fortune 500 companies use market leader Google Analytics to track online behavior.
Plunging deeper into subtleties of passive tracking, I identify four categories of tools. In order of decreasing usefulness for design research, they are:
- Screen recordings
- Heat maps
- Experiment management
- Descriptive behavioral analytics.
The categories and their properties are listed in Table 1. It includes an evaluation of the degree of insight provided, relative scalability, time cost, and the main drawbacks of each approach.
|Table 1. Four categories of functionality offered by tools for data-driven Web design.|
For a deeper understanding, I recommend that everyone with administrative access to a live website try them. Most tools offer a free trial and are affordable (at a maximum couple of thousand dollars a year), although some premium solutions are suitable only for enterprise use (e.g., Google Analytics 360 costs more than $100,000 a year).
User screen recordings. Remote usability tests pioneered the capture of user interactions. This first happened in real time (i.e., synchronously), when the technology simply connected one screen with another. But with screen recordings it has evolved so that the user sessions—with views, clicks, and all—can be recorded and stored as videos for asynchronous viewing later on. Currently, services such as WhatUsersDo, Crazy Egg, and ClickTale offer this type of functionality.
Session replays from screen recordings provide a very detailed view into user activities and even user intentions! In my work, I have used screen recordings in combination with A/B tests. This has enhanced my understanding of why the numbers describing the experimental outcomes turn out the way they do.
For instance, I have witnessed conflicting performance indicators, such as improved conversion rate for purchases despite an increased transaction time. Screen recordings revealed that instead of streamlining, the new single-page checkout design made it easier for users to try out different shipping and billing combinations. Therefore, they took more time to complete a purchase.
The biggest concern with screen recordings is privacy. The screen recordings present a new level of intrusion that can make even the designer feel uncomfortable, as disclosure of personal details is sometimes unavoidable.
Heatmaps. A popular technique to reveal how users interact with a single Web page, heatmaps visualize the amount of clicks made on a website, as well as how much users scroll down from the initial view. The latter metric is known as the scroll depth. Hotjar and SumoMe are services providing this approach. Hotjar is shown in Figure 2.
|Figure 2. Click heat maps such as the one from Hotjar shown here illustrate a user’s interaction with a Web page.|
The main virtue of heatmaps is that they allow intuitive interpretation of the data. For example, a heatmap clearly shows:
- The absence of clicks in elements expected to be clicked
- The presence of clicks in elements not expected to be clicked.
The scroll depth helps to explain the distribution of clicks. It is evident that nobody will click on targets they do not perceive.
While data from these tools is relatively easy to interpret, it is appalling that visual inspection is often the only available analysis method. Practice and experience help in interpreting the maps, but the tools fall short when it comes to assisting the researcher in discovering and quantifying insights.
Another issue is that the heatmaps aggregate user behavior across individual pages. Aggregation is good for privacy, but often every page requires its own configuration. Even aggregating several pages of similar layout but different content, say, YouTube videos or a main navigation element, is not standard functionality.
Experiment management. Experiment-management tools enable the running of different types of experiments. The most famous of them are A/B tests that pit two design variations against each another, one of them usually being the old or the control version. More complex tests, known as ABX or multivariate tests, include several variations, possibly varying across multiple dimensions.
Experiment-management tools control aspects such as the assignment of subjects to conditions, experiment triggering conditions, and goal states. Microsoft’s Ronny Kohavi has produced several excellent presentations (see, for instance [2,5]) about how to arrange experiments properly and how Microsoft’s in-house platform works. Its platform automates many aspects of testing, providing supervised ramp-up and shutdown according to specified key metrics.
Commercial experiment-management systems such as Optimizely and Visual Website Optimizer (Figure 3) fall somewhat behind with regard to automation, but they are designer-friendly and offer WYSIWYG editing so designers can configure UI-level experiments without a developer.
|Figure 3. Web-experiment manager Visual Website Optimizer includes a WYSIWYG editor.|
Experiments are great for allowing large organizations to fail fast with new design ideas. Their shortcoming is that the numbers themselves tell us little about why ideas fail. Kohavi’s publications include several examples of how tricky the results may be to understand and explain—particularly when the number of metrics is measured in hundreds instead of a handful.
Behavioral analytics. Behavioral analytics are the precursor to all current data-driven design solutions. They have been possible since the first Web server started to create usage logs. Analytics services collect data about user interactions with a Web service, aka the click-stream data, along with all contextual data related to the visit, and aggregate it for reporting. The outcomes can usually be summarized as two-dimensional tables, time-series diagrams, or user-flow funnels or similar visualizations.
There are numerous solutions out there, although Google Analytics dominates the market (Figure 4). Competition includes Adobe Analytics, KISSMetrics, and IBM Tealeaf.
|Figure 4. Typical descriptive analytics report obtained from free Google Analytics.|
While these types of tools have generally been adequate for marketing and business use, they tend to be less actionable from a design perspective. In my work, I’ve found that the most useful descriptive statistics are those that describe the current audiences, their preferred devices, and technology in general.
But the changing technology is also a peril. For example, recently popular single-page applications have been poorly supported by analytics solutions that expect websites to follow a standard “URL equals page” structure. Another issue relates to usability and ownership. The analytics products are aimed at analysts and may be poorly accessible for designers technically (poor usability) and politically (require special rights).
In the new design research era, the research questions still matter the most, or are possibly even more important than ever. DDD comes with a strong incentive for designers to adopt a hypothesis-driven approach to design research. While there are techniques that enable the exploration of data masses (e.g., machine-learning clustering techniques), clear hypotheses afford clarity in analysis and decision making.
Interpretation problems are not absent, say, from A/B test results, but having one clear question helps you to pick the most important results out of hundreds of potentially interesting comparisons and tests.
The age of data-driven design draws heavily upon designer morale, integrity, and candor. Opportunities to torture data are numerous. Staying true to your mission is more important than ever to maintain direction and progress.
The horizon. Data-driven design is still in its youth. It has the potential to grow and meet the professional requirements of designers and their organizations. The research-tool-based review here shows the numerous choices in the field. It also avoids identifying the approach with any particular method or design tool.
The tools of DDD must bend and be exploited in the hands of designers. As I see it, the tools that we can expect to stand the test of time will:
- Couple seamlessly with design processes
- Be adaptive to change (data, environment, product)
- Be compatible with other design tools
- Show maturity in addressing ethical questions.
I believe there will be tools suited to the different parts of the interaction design process (business, service, user interface, visual, and so forth) as well as to the different lifecycle phases within digital products. However, due to the necessity of possessing the data to start with, many organizations will find the approach easiest with released products and in fine-tuning their designs rather than with testing new concept designs.
Which tool should I pick? The Web-design tools shown here usually combine several features. For instance, Google Analytics includes not only aggregate descriptive statistics but also selected views into individual (anonymized) users and page-specific heat maps. With the recently released Google Optimize, it also handles A/B tests.
In fact, it appears that most tools are packed with overlapping features, apparently trying to keep up with the competition but lacking a good vision of who their users are.
To succeed, the data-driven interaction designer must feel comfortable by experimenting with a number of tools. There are, for better or worse, several tools for every use case that might do the job, each one a bit differently. It is therefore wise to seek support from developers and data scientists.
Future directions for data-driven design. Is data-driven design something designers will work with in the future? I think it is inevitable that designers will encounter and embrace more quantitative data in the upcoming years. This will be possible only with tools that provide scalability and research questions that help highlight where to look in the vast sea of answers.
The DDD era is just a first step toward automation inside interaction design. Whether generative or algorithmic, design will introduce assistive and agentive tools that do things with us and for us. A good metaphor is autonomous driving, applications of which are gradually relieving us from burdens we don’t miss. Take adaptive cruise control, a magical technology that relieves both feet (agency), but still forces you to keep your hands on the wheel (assistance).
Professional applications of generative design are just starting to emerge. As I’ve recently discussed , advances in artificial intelligence will soon result in new features for the tools designers already love. Google Slides is one of the forerunners with its suggested layouts feature. Or these advances may emerge in radically new products.
I believe the designers of tomorrow will have a symbiotic relationship with intelligent generative design systems. They will establish a strong bond in the fashion of distributed cognition, thinking that leans strongly on the environment, even to the extent of being fully dependent on it.
A similar development has happened in chess. There, the “post-Deep Blue” times have witnessed combinations of machines and humans battling side by side. The development of agentive, generative technologies is more disruptive and difficult to predict. These developments will sweep away many of the tasks that occupy the interaction designers of 2017.
Consent dilemma. The ethical questions shouldn’t be forgotten if we as interaction designers want to preserve high moral standards. The present situation in Web experiments resembles the Wild West, in which the most daring data prospectors do what they will and reap the rewards for potential breaches of user privacy.
The biggest ethical question of DDD is what you tell the user about your data acquisition. In academia, specific consent has been for long considered the hallmark of ethical research. In Web-based data collection, consent can be a) generic, b) specific, or c) absent. Most Internet companies have chosen a generic consent that is embedded inside a generic terms of service that gives users minimal information about any particular case. Some companies—for instance, Netflix—allow users to specifically opt out of experimentation, but this is uncommon.
The matter seems too important to be left to the technologists to solve. However, the political interventions are also difficult. So let’s solve the issue before it becomes an issue.
I want to believe that DDD tools don’t hurt people—and designers shouldn’t either.
1. Bond, R.M., Fariss, C.J., Jones, J.J., Kramer, A.D.I., Marlow, C., Settle, J.E. and Fowler, J.H. A 61-million-person experiment in social influence and political mobilization. Nature 489, 7415 (2012), 295–298.
3. Lomas, J.D., Forlizzi, J., Poonwala, N., Patel, N., Shodhan, S., Patel, K., Koedinger, K., and Brunskill, E. Interface design optimization as a multi-armed bandit problem. Proc. of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2016, 4142–4153.
5. Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., and Pohlmann, N. Online controlled experiments at large scale. Proc. of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2013, 1168–1176.
6. Liikkanen, L.A. Applications of machine learning for designers. Smashing Magazine (2017); https://www.smashingmagazine.com/2017/04/applications-machine-learning-designers/
Lassi A. Liikkanen is an adjunct professor at Aalto University and a data-driven design specialist at SC5 (Finland). His post-doctorate period included an extended research visit to Stanford University and appearances around Asia. His research interests lie in interaction design practices. firstname.lastname@example.org
©2017 ACM 1072-5520/17/09 $15.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2017 ACM, Inc.