ACM Interactions

Authors:
David Cronin

---

 

If you think about it, musical instruments provide really intriguing examples of user interface design. While it can take years of training and no small amount of aptitude, an instrument in the right hands can provide highly nuanced control over the many aspects of sound that come together to form one of the highest forms of human expression. And even for those of us who will never achieve such virtuosity, merely using such a "user interface" can result in a great sense of enjoyment, immersion, and fulfillment (what is often referred to as a state of "flow").

Music is almost universally important to human culture, but instruments are not strictly "useful," and it seems strange to think of them as mere tools. That said, from the first bone flutes and stone tools of the Paleolithic era, the evolution of musical instruments has closely paralleled that of more utilitarian technology. As inventor and futurist Ray Kurzweil puts it, "[musical expression] has always used the most advanced technologies available [1]."

Not surprisingly then, as with so many other things, the dramatic increase of processor speeds has brought about a revolution in the way people use computers to make music. But while computational power has been a critical enabling factor in this revolution, at least equally as important has been the ongoing evolution of the user interfaces of these new digital instruments. This article aims to tell the story of this evolution and explore the lessons and opportunities it presents to interaction designers of all kinds of products.

  A Recent History of Musical Technology and Interactivity

As with the broader universe of technology, musical instruments have coevolved with the practice of music. New technologies are often first introduced as a way of replicating and incrementally improving upon a previously established way of doing things, and then they may eventually point the way to something entirely new. (Take for example the first cars, which were designed as "horseless carriages.") This is certainly true in the world of musical innovation—by and large synthesizers were first looked to as a means to emulate the sounds of acoustic instruments, and it took decades before electronic sounds became aesthetically appealing in their own right.

Before we examine the advances in the new wave of musical user interfaces, it will be useful to define some terminology and touch on a bit of historical context behind different aspects of sonic technology. The first thing to keep in mind is that where in an acoustic live setting the qualities of sound are entirely determined by the musician, the instrument, and the acoustics of the room, in the contemporary world of recorded music, the number of factors that influence the sound grow exponentially to include all flavors of recording and distribution technologies. While the importance of the musician has not necessarily diminished (though this is, I suppose, debatable), what is done with sound after it leaves the instrument has become dramatically more important. While this is not a new development—the concept of the studio as an instrument goes back at least to the '60s and the pioneering work done at Abbey Road, as well as by Brian Wilson and Phil Spector—it is at the core of recent evolutions in digital music technology.

Starting with the concept of the instrument, of course we still have all the traditional kinds of soundmakers, from percussion to strings to brass to woodwinds. And then we also have the first generation of their plugged-in counterparts: electric guitars and pianos, for example. The next step from there was the synthesizer, which brings us closer to where we are today. Synthesizers are electronic instruments that employ a variety of techniques to allow users to explicitly control different qualities of the sound, including pitch, harmonic content, duration, and how the sound changes as it is played. These are typically controlled by a piano keyboard for pitch, as well as other hardware controls (such as knobs, sliders, dials, and buttons) for other aspects of the sound.

From a user interface perspective, synthesizers provided an interesting leap forward. Where a violinist must train for years to gain the dexterity to adjust the timbre of their sound, someone playing an analog synthesizer is able to make such an adjustment by twisting a knob or two. By making this control explicit and direct, it greatly flattens the learning curve (though to be fair, it has proven no shortcut to the highly nuanced styling of a master instrumentalist).

Most recently, there have been a couple significant evolutionary steps in the world of synthesis. As the processing power of personal computers increased through the '90s, it became possible to create synthesizers that run as applications in Windows or Mac OS. These "softsynths" are particularly useful because they can be seamlessly integrated into a computer-based production environment (often called a DAW, or Digital Audio Workstation), eliminating the piles of cables that course through traditional studios, as well as the attendant concerns about what each cable is connected to. Also, because they are able to rely on significant computing power and large graphical user interfaces, softsynths can present both novel methods of synthesis (such as physical modeling, which uses principles of physics to model the sound emitted by things like plucked strings), and most significantly for this article, novel methods of control, which I'll discuss in the next section.

With the evolution of the synthesizer came the sequencer. The idea behind a sequencer, whose heritage lies in the paper rolls and mechanisms of the player piano, is a way of recording, playing back, and editing a control signal that tells a synthesizer or sampler what notes to play, as well as other information about how they should be played. From an interaction perspective, this was quite revolutionary. A musician or producer could record the notes to be played in song, and while playing them back with the sequencer, continue to modify and adjust the sonic characteristics without ever having to make an audio recording.

At first glance, mucking around with a sequencer is not a terribly musical activity, and it can be a great distraction from the actual playing of music. This strikes at the heart of a typical conundrum faced by today's product designers: Adding sophisticated and potentially useful capabilities to a person's toolset can also add the significant overhead of managing those capabilities. Many musicians complain that if their equipment setup is too complex it becomes easy to lose track of their musical ideas because they have to spend so much energy managing technology. However, for many electronic musicians, the sequencer itself has become an instrument in its own right—after hard practice it is possible to achieve an impressive level of virtuosity in live performance.

The final bits of the contemporary music-technology equation are the components around the recording and manipulation of sounds. These include the sampler, which plays back recorded sound in a musical fashion; the mixer, which combines sounds from multiple sources while adjusting the volume of each source as well as shaping the sounds (by way of an equalizer); the multitrack recorder, which records several sound sources at once while maintaining their independence (ultimately to be mixed together later); and effects devices, such as distortion and echo boxes, which change the character of the played or recorded sounds. Once again, while there are certainly many prominent producers who stand by their '60s-era analog mixing desks and still prefer to record to 1/2" tape, here again increased processing power has meant that all of these capabilities can be found on a personal computer.

In fact, today, Digital Audio Workstation (DAW) software such as Ableton Live, Apple Logic, Steinberg Cubase, and Digidesign ProTools provide the capabilities of synthesizers, samplers, sequencers, mixers, effects, and recorders, all through a single, integrated environment. While these are not universally simple to use, in many cases the utility and usability of these applications is a big improvement over the technology that came before them (though there are still plenty of folks who debate this point). This fact, combined with the relative cost-effectiveness, has meant that a lot of musicians of all stripes have brought computers into their bedrooms, studios, practice rooms, and even on stage, and in the process they are redefining the way that recorded music is made. No longer does a band need to work according to a schedule in an expensive studio to record their ideas, but rather they're able to record as the feeling strikes, wherever and whenever that happens to be. This transformation hasn't been universally easy or fruitful, but the computer's new status as a musical instrument surely has some lessons for us.

  Designing for Fluency

Creating music requires a musician to achieve a sense of fluency with his or her tools. This is why musicians have to practice so much. It is incredibly detrimental to the musical experience to stop and think about how to operate a piece of equipment; the fleeting and elusive ideas behind a song are easily lost to technical distractions. For a musician to get an idea out of her head, the use of her instruments must be effortless. And this effortless isn't just a state of minimum exertion and maximum efficiency. The experience must allow for playfulness and spontaneity, the enablers of the best of human capabilities.

So it seems that these musicians and producers who are successfully composing music and producing recordings using their computers are able to achieve a sense of fluency (or even virtuosity) that is an ideal we should have for all kinds of users of technology, from surgeons to stock traders. The question then is, what about these musical user interfaces contributes to this sense of fluidity?

Simplicity. One of the most important as pects of a good musical interface is simplicity. Guitars, pianos, and trumpets are all quite straightforward in the way they present their capabilities. Obviously, the less interface a user is confronted with, the more mental bandwidth they can devote to hearing and playing music. With fewer choices the user is both able to explore a nuanced way of using each aspect of an instrument and is ultimately more able to play notes decisively and meaningfully.

To be clear, for a user interface to be simple, it is not necessarily true that it must also be basic or unsophisticated. Yes, a drum is quite straightforward and direct in the way its capabilities are presented, but it is obviously capable of being played with nuance and subtlety, though this takes years of practice. For an instrument to facilitate a feeling of fluidity, it must be learnable but not necessarily focused on novice use.

While not always successful, the drive for simplicity in the creation of digital instruments has always been there. Greg Hendershott, the creator of Cakewalk, one of the first computer-based sequencers, explains the driving force behind his design decisions: "My design philosophy for the early Cakewalk software was to make simple things simple and to have more complicated things require a little more effort. So anything that you'd be doing frequently... I tried to keep that very up front in the user interface—a single keystroke—and things that were a little less frequent I tried to have buried in a dialog box or away from view. I think that's one thing that we've tried to keep up, although that [gets] more challenging as you have more and more functionality and more features to the software [2]."

Unfortunately, as Hendershott alludes, Cakewalk and other early DAWs quickly grew top-heavy under the weight of features, and it became so that these tools lost their simplicity and started to impose their way of working on their users. Ultimately, a new player emerged with renewed dedication to simplicity.

In 2001, Live was released by Ableton, a small Berlin-based company focused on creating a simple tool for creative music performance. As Thomas Bangalter of French pop act Daft Punk puts it, "Live is by definition one of the most transparent and simple [pieces of] creative software I have ever used. Its interface is streamlined, and its features, though with a minimal approach, offer endless possibilities. The whole process is easy, simple, and fun, yet with accessible possible sophistication and professional results [3]."

There are several things that contribute to Live's feeling of simplicity. First, the feature set of the product is fairly limited. Where the more bloated DAWs provide every function that could possibly be required in any production situation, the Ableton design team has very intentionally omitted features that could contribute to complexity without adding to its capability as a creative performance tool. Music-related information (such as notes and control data) is best input through performance (e.g., via a keyboard), rather than numerically or through traditional notation in a score (which other DAWs provide).

Live's user interface also contributes to its sense of simplicity. Two main screen states contain the entire interface; one is optimized for improvisational performance, the other for composition and more-structured performances. A musician is typically able to spend an entire usage session in one view or the other, never with a thought to "navigation" (a distinctly unmusical concept). Both screens follow the same structural patterns, which provide access to files, instruments, and all music-related parameters without dialog boxes or overlapping windows. Further contributing to the simplicity are the control elements (such as "knobs" and "faders"), which are rendered in a flat style that minimizes the amount of pixels devoted to ornamentation. This makes it easy for users to understand the state of all the controls through effective use of modeless visual feedback, without the visual noise of faux 3-D rendering.

Redefinition of the instrument. Another important design decision made by the Ableton team was to respond to and accommodate the changing definition of "musical instrument." Here we return to the idea of the studio as an instrument, first explored by rock and R&B producers in the '60s and then fully embraced by hip-hop and electronic musicians in the '80s and '90s. As Robert Henke, member of seminal electronic group Monolake and cofounder of Ableton, explains: "Software sequencers and hard-disk recording applications were originally designed as studio tools, replacing tape machines. Historically, they were more aimed toward sound engineers than toward musicians. The underlying idea of timeline-based editing is construction and sculpturing, not so much performing. As a result, those tools fail onstage or in any context where improvisation or interaction with musicians is essential. So all the software Gerhard and I wrote for our own purposes enabled us to interact with the music in real time. We liked the concept of step sequencers and drum computers because they allowed for immediate control of structure by changing patterns in real time. Live is the result of applying some of these ideas to the world of sound files, thus providing more flexibility and easy access to an infinite pool of possible source material [4]."

To generalize, it seems that the lesson to be learned here is that many product designers (and business stakeholders) become unnecessarily constrained by a rigid product definition based upon existing categories, rather than a holistic understanding of user needs and mental models. Taking a step back to observe broader context may present opportunities to deliver something that people will find useful and desirable.

Modeless, visual feedback. Since music happens in real time, musicians must be able to quickly understand what's going on with their instruments. Most traditional instruments don't maintain state (and if they do, it's visibly and audibly noticeable). In the world of a computer-based production environment, it is quite easy for things to get so complex that it isn't obvious from the sound exactly what is going on with various components. This is where visual feedback can really help. Visual representations of knobs and faders are effective devices here—a knob not only allows direct manipulation, but also is a way to quickly understand the current value of the parameter without touching the control or requesting the information (and is therefore modeless).

Recent innovations in musical user interfaces have broken from metaphors referring to our mechanical past to achieve many novel ways of providing visual feedback. Many of the instruments that ship with Reaktor, Native Instrument's modular audio application construction environment, provide ingenious graphical mechanisms for controlling sounds and musical compositions.

Physical control. While elegant visual interfaces can greatly contribute to a musician's sense of fluency, focusing too much on the visual can be a problem. After all, music is an auditory experience. Visualizations of music are always an abstraction, and in many ways can stand in the way of a musician trusting his or her ears and personal perception of tone and time, which is truly at the heart of music.

One of the things that helps allow musicians to take their eyes off the screen is physical control surfaces. It's very difficult to interact with an onscreen knob and a mouse if one isn't looking at the screen. Also, music is traditionally a dexterous two-handed affair—there's usually more than one thing going on at once, and it helps to use two hands to keep everything moving. Computer operating systems offer only direct manipulation through the mouse pointer in a single place.

As musician and composer Brian Eno explains, "I want to use the whole of my body, not just my index finger. One of the problems with a lot of software systems is that they expect you to type, but it uses a part of my brain that I don't always want to be in the music process. I don't want to shift between being a musician and being a secretary [5]."

Quite a number of physical control surfaces have been developed to solve this problem. These range from surfaces with dedicated controls for common functions (such as faders for mixer channels and knobs for equalizers) to banks of knobs and faders that can be flexibly assigned to any aspect of a synthesizer, sequencer, or mixer. Not only are there standard piano keyboards, but there are also pads for triggering samples and even gestural controllers (including interfaces with Nintendo's Wii Remote). One of the most exciting control surfaces is Jazzmutant Lemur, which provides a multitouch screen (beating the iPhone to market by years) that can be flexibly configured to provide a number of different types of touch and gestural controls.

Balance between structure and experimentation. One of the hallmarks of the best of modern music, from jazz to pop, is a simultaneous reference to and departure from conventional forms. In most cases, either slavishly following the rules or blindly breaking them will bore the audience. Similarly, for musicians to work effectively, they require a fair amount of structure so that they don't need to reconsider first assumptions every time they play a note, but they also have the flexibility to reconfigure their instruments to chase that sound they hear in their minds.

Most DAWs allow for this with a straightforward architecture for recording and sequencing that also accommodates more arcane, cross-modulated routings. Perhaps the best example of this balance between structure and experimentation comes with Native Instrument's Reaktor, a modular musical toolkit. The product ships with an arsenal of highly innovative synthesizers, samplers, drum machines, and other musical inventions, but the inner workings of each are accessible and drastically modifiable through a visual construction environment. Users also build and share their own devices either from primitives or components borrowed from various other devices.

What makes this experience quite effective is that one can start from something useful (and even inspirational) and make small adjustments to come closer to an imagined sound or way of working. This is a great approach for all kinds of interactive tools. Often customization is used as a proxy for design—I can't count the number of times someone has told me, "We can't possibly imagine how someone will want to use this, so we should just let them customize the user interface." The fallacy here is that while some people can and do tweak products to get them just right, the starting place is hugely important.

Looking Forward. Clearly, the evolution of musical technology isn't over. If anything, it's accelerating and changing course. Dissatisfaction with the computer as a musical platform has led to renewed interest in musical instruments as physical devices. Yamaha's Tenori-On is perhaps the most dramatic of recent inventions in this arena. Designed by the renowned Toshio Iwai, this beautiful device provides an LED-lit grid of buttons as a novel user interface for musical improvisation and composition. Even game consoles provide new avenues for innovation. Electroplankton, a game for the Nintendo DS (incidentally, also designed by Iwai), allows users to create music by interacting with small onscreen "organisms," each of which has a unique control idiom and resulting sound.

The application of these inventions to more utilitarian digital products may not be universally obvious, but there's a great potential to follow the lead of musical technology. Many attributes of a successful musical experience can be applied to all kinds of venues where critical decisions must be creatively made in real time. With appropriate application, the design strategies discussed here can help us create products that better allow for the kind of fluency and virtuosity that represent our best abilities.

  References

1. Kurzweil, Ray. "The Future of Music in the Age of Spiritual Machines." Highlights of the Richard C. Heyser Memorial Lecture to the 115th Annual Convention of the Audio Engineering Society, New York, October 11, 2003. <http://www.kurzweilai.net/meme/frame.html?main=/articles/art0597.html>

2. Kirn, Peter. "Interview: Cakewalk Founder Greg Hendershott, 20 Years On." Create Digital Music. November 12, 2007. <http://createdigitalmusic.com/2007/11/12/interview-cakewalk-founder-greg-hendershott-20-years-on>

3. "Daft Punk Thomas Bangalter & Ableton Live." MacMusic.com. October 27, 2005. <http://www.macmusic.org/articles/view.php/lang/en/id/104/>

4. Henke, Robert. "monolake: the Genesis of Ableton." M-Audio.com. <http://www.m-audio.com/news/en_us-4.html>

5. Brian Eno, interview by John Alderman, MoreDarkThanShark.org July 1996. <http://www.moredarkthanshark.org/eno_int_hwire-jul96.html>

  Author

David Cronin
Cooper
[email protected]

About the Author

Dave Cronin is the director of interaction design at Cooper, where he's worked since 1999, leading design projects ranging from computer-assisted surgery tools to an institutional investment management platform to a museum information system. He's also the co-author of About Face 3.

  Footnotes

DOI: http://doi.acm.org/10.1145/1353782.1353800

  Figures

Figure. Yamaha's Tenori-on is an example of what's to come when music technology moves from the computer to actual physical devices.

Figure. The desktop as a musical instrument has evolved from basic softsynths to fully integrated audio environments.

Figure. Musical user interfaces offer simplicity and physical control for recording and sequencing.

 

©2008 ACM  1072-5220/08/0500  $5.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2008 ACM, Inc.