Peter Froehlich, Lynne Baillie, Rainer Simon
Mobile computers are increasingly used as a link between the physical and the digital worlds. This innovation demands a more sophisticated multidisciplinary approach to the modeling of spatial interaction than has yet been developed. Our aim in this article is to open up the "black box" of mobile spatial interaction (MSI) and discuss some issues and possible approaches that could be taken.
Groundbreaking concepts that proposed how to combine virtual information with the user's direct surroundings arose as early as the 1990s. Two early examples include Egenhofer's Smart Compasses, which point and direct people to places of interest, and Geo-Wands, virtual geographic pointers for the selection of surrounding objects and attached services .
An early application of this type of technological advancement was implemented in Atlanta in the late 1990s; visitors to the Georgia Tech campus were able to experience enhanced "future visions" of spatial awareness .
Since then the concept of MSI has become increasingly feasible and desired. New technology, including digital cameras, GPS, accelerometers, digital compasses, and RFID chips, is constantly being built into mass-market mobile phones, thus enabling the user to interact with his or her surroundings in new ways. These innovations are inspiring industries to create new contactless applications, such as in the domains of ticketing and vending. Spatially related research, in the form of Web-based mapping and geo-browsing, is beginning to be transferred to commercial mobile devices. MSI is further boosted by virtual- or augmented-reality techniques, enabling the continuous interplay between virtual and spatial information.
Research in MSI has traditionally focused on three main categories, which have now reached a level of technical advancement that makes them ideal for technology transfer into commercial devices. The three categories are wayfinding, access and creation of spatial data, and augmented reality.
Orientation and Wayfinding. A good example of successful MSI is the "flight-mode" visualization in today's car navigation systems. The bird's-eye perspective facilitates an efficient match of digital information with the driver's view, thereby blocking out the task-irrelevant rear view and making closer parts of the road more prominent than farther-away ones. As exemplified with most wayfinding systems, MSI is frequently a secondary task that should not interfere with the primary activity. This imposes strong constraints on the interaction design.
Outside the car MSI is quickly evolving as a means of supporting mobility in pedestrian navigation, outdoor sports, and traveling. However, more work is required in this area to better support the user undertaking these activities. For example, Scott Counts from Microsoft Research proposed that the "route" should be viewed as a spatio-temporal document type . His group is currently investigating how runners can author their own routes, which can be automatically annotated with sensor data for outdoor sports.
Accessing and Creating Spatial Data. A common theme found in MSI research is the ability to access digital information that has been attached to physical places called points of interest (POI). The Geo-Wand is a research concept that enables tourists to point their mobile phone at a historic building and ask "What is this?"
Beyond physical spatiality, MSI includes other aspects, such as our sense of space and of social interaction. For example, researchers at Fraunhofer produced a prototype of "StreetBeat" , a location-based mobile application that aims to make people feel part of the cultural experience of alternative-music clubs in Berlin by playing excerpts from the previous night's band as they wander by.
Companies looking to implement academic MSI research in consumer products may latch on to these types of social experiences, and thus this area is one that can benefit from extended research and development.
In addition to accessing preexisting information and services, people are also increasingly creating and geo-referencing their own multimedia content. The challenge for MSI is to make this development fruitful for mobile contexts. Several concepts in recent years, such as GeoNotes, have demonstrated the feasibility and attractiveness of attaching digital information to real-world coordinates, with metaphors such as virtual post-its or graffiti. Some of the most interesting MSI activities here are concerned with the socially motivated interaction forms of sharing and tagging. One example is the Zonetag mobile application, which allows for the context-aware upload of photographs from a camera phone .
Zonetag aims at lowering the effort of tagging on mobile devices, by automatically suggesting tags based on the user's social network, his previous entries, as well as names of nearby places in the real world. However, this tagging could quickly get out of hand with a high number of postings in popular areas. Lemmelä and Korhonen from Nokia Research provide an interesting solution to this, a visualization method, which indicates the density of postings by overlaying semitransparent heat maps over the conventional map view and automatically displaying the "hot" topics .
Augmented Reality. For a tourist in a historic city, relevant information about a specific area is not always easily accessible. A spatially aware mobile device could serve as a window to virtual information, such as the historic past of a building .
Virtually enhanced exploration is also useful as a professional tool. Schall developed an application that enables users to view the subsurface structure of a city square, including water pipes and electricity, to assist urban planners and infrastructure service providers .
Getting MSI through the Innovation Funnel. MSI adds a sense of orientation and meaningfulness to conventional location awarenessand it is quickly stepping out of its research stage. The ftw. project Point-to-Discover demonstrated the high technical accuracy and attractiveness of accessing nearby points of interest with a spatially aware mobile phone, featuring a combination of GPS, a digital compass, and 3D-accelerometers . Japan, where mass-market phones with integrated GPS and compass have been available since 2006, is the first test bed for commercial MSI. The phones can provide information on nearby restaurants and shops . It is only a question of time before spatially aware handsets are sold globally.
To realize MSI applications and services, the strongest challenge is how to conceptualize and design applications that are tailored to the needs of mobile users. In this respect, the following things need to be considered:
Dealing with uncertainty. MSI applications are subject to many kinds of uncertainties: GPS, sensor inaccuracies, or incorrect geotags. There are also technical and design issues provoked when the user moves from indoors to outdoors. These certainties cannot always be avoided, and there is a need to ensure transparency and communicate these inaccuracies to the user.
Display of spatial information. Given the plethora of different standards and communication interfaces for mobile devices, as well as a range of varying computing power available in the devices, it will be quite difficult to design in global terms. Presenting spatial information and various details about the location and the people in the location without provoking cognitive overload is an interesting and challenging interaction-design task. To increase the bandwidth and flexibility of information display, the speech and sound-output capabilities of mobile devices should be exploited to a larger degree.
Social disclosure and privacy. There are challenges in how we go about protecting individual privacynot only in meeting legislative and safety requirements, but also in being sensitive to what users want and do not want to make available about their present location. Designing with these issues in mind is complex. For example, a shop in the vicinity wants to send you a money-off voucher (deny and delete); your buddy is nearby and wonders if you want a coffee (accept, audio alert); the train you need to catch is delayed (accept, vibrate).
Spatial content. The challenge here is in the acquisition and access to contextual datato understand and act appropriately on the infrastructure that is available when billions of people start to contribute MSI data every day. We will need new systems that can store, search, and mine geo-spatial data. This will require interdisciplinary and hybrid research across different fields to enable the successful collection and searching of such data in order for it to be useable, useful, and used.
Identifying the business value. The proven feasibility and attractiveness of conceptual research scenarios like wayfinding, POI access, and exploration should now motivate the definition of more commercially oriented application concepts. In-depth user research is one of the strongest success factors to identify the real needs of specific user groups, such as tourists in a variety of mobile situations. This market-oriented research should feed into the definition of business and role models including all stakeholders of MSI: end-user communities, service providers, mobile operators, content providers, and handheld manufacturers.
We see MSI as a major opportunity to make mobile internet services useable, because interaction styles such as pointing are closely matched to situational needs of mobile users. MSI will affect both how we interact with existing information (e.g., tourist attractions) and create new applications (e.g., augmented-reality games). As the technical advancements move from a research stage to production, new visualization methods and multimodal interaction concepts are needed. It is important to highlight the importance of this topic and drive joint initiatives to make MSI transition successful .
Telecommunications Research Center (ftw.)
Telecommunications Research Center (ftw.)
Telecommunications Research Center (ftw.)
About the Authors
Peter Fröhlich is a senior HCI researcher at Telecommunications Research Center Vienna (ftw.). He manages the project Point-to-Discover (p2d), which is co-funded by mobilkom austria, Siemens Austria, and the Austrian competence center kplus. The project develops the foundations for mobile spatial interaction: interaction techniques, modeling of spatial information, GIS data integration, and content aggregation, as well as hardware prototyping. Together with Lynne and Rainer, he has initiated the MSI initiative.
Lynne Baillie has a Ph.D. in HCI from Napier University in Scotland. She has worked as a senior researcher for the Telecommunication Research Center Vienna (ftw.). Since 2002 she has investigated existing HCI methods for their applicability in the mobile domain and as a result has developed new methods in order to design more usable applications for mobile devices. She has also undertaken studies in the home and investigated how we can improve and extend user-centered development methodologies to enable codesign with families.
Rainer Simon is a researcher at Telecommunications Research Center Vienna (ftw.) and a research fellow at the Vienna University of Technology Research Group for Industrial Software. He has several years of experience in the field of mobile application research and currently works with major mobile network operators on novel location-aware applications and gesture-based interaction methods for mobile phones.
©2008 ACM 1072-5220/08/0100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2008 ACM, Inc.