Launch any software product these days, and you can easily find a whole host of issues you wish the developer or designer would have better anticipatedwhether it is to optimize the interaction, organization of information, or further develop tools before releasing the product to the marketplace. I can't go a day without thinking to myself, "This would be a better software product if...." Anyone with a critical eye towards building quality product feels the need to critique, the nice form of complaining, either publicly (in email, blogs, or forums) or privately (to friends, coworkers, etc.).
In the development of a product, therefore, it's useful to consider building a beta version (a pre-release of the actual product). It can be an incredibly valuable way of determining issues before the product reaches its ultimate audience.
The earliest judge of how well a product will do in the marketplace is often the public reviews, so they can be used as measuring stick. First, start with a very simple theory:
Good product reviews are good for business
Bad product reviews are bad for business
Bad product reviews that are provided early, in a controlled manner, and turned into good reviews by removing user issues are great for business
In practice, it will require understanding the market drivers that lead to a perceived "win" with reviewers, customers, and the public.
When Walt Mossberg (of The Wall Street Journal) or David Pogue (of The New York Times) raves about your product, you have a leg up on the marketing and sales of your product. That happens for a very small percentage of the high-tech market, and you should count yourself lucky if you've made it that far.
Bad reviews, whether from any of the major industry pundits, or just on the C|Net messages boards, can dramatically affect you ability to convince prospective purchasers that you have a valuable, usable, and enjoyable product. In this case, you cannot underestimate the power of the formal and informal press.
However, this is where prototyping, and the business of betas, can help anticipate reviews, provide time for you to address the user issues, and seed the market with interesting new products to spur interest from users, reviewers, evangelists, and even investors.
Beta programs are both very well-controlled processes that have a defined input (early product design) and a defined output (feedback). They can be rousing successes, leading to shipping a compelling, focused product, or they can lead to product failures, ending up in great learning experiences. For success, you must decide on your goals, support an iterative feedback process, and plan accordingly. Failure in a beta program is much like learning to snowboard for the first time: You must first learn to fall, and not be scared of falling, in order to snowboard well.
Planning for your first or first successful beta program involves using several ideas that have been well documented in the basics of business strategic planning:
Define the people that will use your product, and just as importantly, the people who won't use your product.
Tip: Don't send your high-end WiFi laptop to rural Nebraska, it won't test well.
Example: Companies like Centercode can help you target a specific audience through profiles they keep in their beta programs participant database. Well-targeted audiences will generally provide more feedback and help tailor the specifics of your product rather than derail your product intentions.
Follow the lead of people like Guy Kawasaki (early Product Evangelist for Apple, look him up on Amazon) who have made a business of creating Evangelists that can speak from the heart about your product and how it is good and bad for your market.
When you find someone that is passionate about the problem space, and can give you good, objective feedback, they will help drive improvements as well as end up as your greatest asset in "word-of-mouth" marketing (read The Tipping Point for more details).
Tip: Never have your fashion displayed on Joan Rivers "Worst Dressed List." In the fashion industry, it's customary to display your designs on a real model, presenting them to the industry and fashion mavens early to rate and purchase the wares long before they hit the consumer market.
Example: Apple employs Mac "Geniuses" at their Apple stores to assist the average consumer with purchases. These are generally highly knowledgeable Apple enthusiasts that can now help in improving Apple's business from within.
There are plenty of forums, people, and entities that should have their hands on your betas. Best bets are user groups, interested public forum participants, employees of companies that may see your "vision" for the product, or pundits that have a wide reach in the press.
Tip: Ebert and Roperwell-known US movie reviewersare in business because people listen. I don't, but I think Adam Sandler is funny. Go figure. However, movies that aren't reviewed before release and don't have those fabulous commentaries are left to run on their own marketing power.
Example: Magazines like Wired and blogs like Engadget routinely print articles of upcoming technology and products that are still in beta that they're excited about. This ultimately markets your product before it's even available, prepping the market for your product and getting early conjectures from prospective customers.
The best product feedback I've received was from executives of companies to whom I wanted to sell my product eventually. They felt very involved in the design and development, and ultimately, became "tied" to the success of the product.
Time and effort are your friends, and it's good to have lots of both.
Tip: A Boeing 777 pilot would never try and take off from an aircraft carrier. The length and stability are not there to get it off the deck, and the same holds true for your product. If you skimp too much on the length of time and the strength of your effort to make the changes necessary, your customers will feel the pain.
Example: Product lifecycles are very tight in the "Web" world, but Yahoo! and Google still release early betas and focus on usability and feedback well before they launch the product. Never as much as they like, but that's life!
Worry about the Big Problems, and Plan for the Less Important Problems.
The key to successfully using a beta test to drive product development is to prioritize the feedback. Users and critiques can be harsh and somewhat uncaring (especially when they're not sitting face-to-face in a room with you), but you have to know the relative priorities to address the most important issues and get the product to market.
Tip: Governments would fail if they acted on all criticisms by their constituents, because it's lots of complaining and often conflicting desires. They prioritize the things they can change immediately for the most dramatic improvements for their public, and plan to get to the less important issues later down the road (if at all). The US government still hasn't gotten to my request for an "Adam Sandler Appreciation Day," but I'll keep lobbying.
Example: Mozilla and other Web browser products provide all types of browsing experiences for the Web, but if they slip on the major standards or introduce security risks these are considered "Big Problems." Updates are made quickly and often to address these issues, and less frequently on "Smaller Problems" around extra, added functionality.
Heeding these suggestions may help you take a fledgling prototype or beta from internal design to external success. Enticing the market, and developing a strong evangelist base to help sell your product, means you may create the ability to have another version down the road that gets even better.
About the author
Brian Frank specializes in product management, design, and usability.
©2006 ACM 1072-5220/06/0300 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.