XXI.4 July + August 2014
Page: 54
Digital Citation

Tangible augmented reality for air traffic control

Jean-Luc Vinot, Catherine Letondal, Rémi Lesbordes, Stéphane Chatty, Stéphane Conversy, Christophe Hurter

back to top 

The boundaries between 'the digital' and our everyday physical world are dissolving as we develop more physical ways of interacting with computing. This forum presents some of the topics discussed in the colorful multidisciplinary field of tangible and embodied interaction. —Eva Hornecker, Editor

Air traffic control (ATC) guides aircraft to maintain safe distances and to optimize traffic fluidity. ATC has been a life-size HCI laboratory for many decades, as witnessed by the introduction of touchscreens in the 1960s, digital emulations of paper in the 1990s [1], and research on augmented reality [2]. This history reveals a succession of compromises between physical and digital interactions, of which tangible user interfaces might be the synthesis.

What makes ATC special is the combination of efficiency and safety that shapes its evolution. While efficiency is the driving force for technical innovation, safety is a selection force: Only those innovations for which air traffic controllers can safely adapt their working methods are put into operation. This co-evolution of technologies and working methods is a strong asset of ATC. However, it also makes life hard for engineers because of the difficulty in establishing exactly how a technology will contribute to safety before proposing it as an alternative to current technologies.

back to top  Insights


An example of this challenging evolution process is the switch from paper to digital media, fostered by the civil aviation agencies since the mid-1980s, but still not achieved. Almost everywhere, controllers monitor traffic through real-time radar displays and communicate with pilots by radio. In addition, to anticipate the traffic, manage their decisions, and program their actions, many controllers around the world still use paper flight strips. Printed on each strip is the planned route of an aircraft through a controller's sector of responsibility. Controllers annotate, grasp, move, and organize their paper strips on a stripboard, using these tangible interactions to organize their mental picture [2].

Unfortunately, paper strips are a poor connection between the physical and digital worlds: Once printed they cannot be updated, nor can the instructions given to pilots be used as input to the system. This impedes the development of more automated ATC systems in which controllers would be provided with useful alerts and hints computed from their instructions.

However, universal solutions for replacing paper with fully digital systems have yet to be found. The proposal to use graphical interfaces on the sole radar display met with mixed results. More sophisticated variants of "electronic stripping" were then proposed, culminating with combinations of rich graphics, animation, touch, and gestures [1]. These have encountered some success, but in many cases, particularly in airports, there is still a need for the better capturing properties of paper strips. Combining the benefits of paper with those of digital through tangible interaction is a promising direction for meeting these demands.

back to top  The Strip'TIC Prototype

We have designed Strip'TIC (Stripping Tangible Interface for Controllers), a prototype that combines augmented paper and digital pens on a multitouch glass stripboard, using vision-based tracking and augmented rear and front projection [3]. The paper strips, the stripboard, and the radar screen are all covered with Anoto Digital Pen patterns (DP-patterns), small patterns used by the pen's infrared digital camera to compute the location of the pen on the dotted surface. The controllers' actions with the digital pen are sent in real time to the software. They can annotate paper strips, and the resulting strokes are both physical (inked) and digital (projected). They can use the pen to point at symbols on the radar display or names on the paper strips. The stripboard itself is semi-opaque. This enables both a bottom projection on the stripboard and strip tracking, thanks to AR patterns printed on the back of the strips. A second projector located above the device displays graphics on top of the stripboard and the strips.

Designed in close collaboration with controllers, Strip'TIC allows them to manage traffic and workload with their current rules and practices while bringing new benefits. Most important, the prototype leverages the advantages related to the embodied use of physical objects in physical space. Let's observe two controllers working together to handle aircraft arriving at the Paris-Orly airport and see how their use of the system illustrates the beneficial properties of tangibility in such a critical activity.

In the scenario that follows, Paul is a planning controller. His role is to coordinate with adjacent sectors and prepare the flow of arriving aircraft for Tessa, a tactical controller, sitting next to him. Tessa's role is to manage traffic in her sector, monitoring it on her radar display, and using radio communication to give pilots clearances for new altitudes, headings, or speeds. Paul and Tessa share a set of paper strips arranged on a stripboard. Printed on each strip is the planned route of an aircraft through their sector.

Digital feedback for physical interactions. The system provides rich feedback, which increases the controllers' confidence in the system, as it means the system "knows" what is happening.

It's Friday at 4 P.M. Paul hears the strip printer and grabs a strip for the Turkish Airlines THY1825 flight, about to enter the sector and land at Orly. He puts the strip onto the stripboard and quickly analyzes the situation with this new incoming aircraft. The system displays a rectangular halo underneath the paper strip, colored in red to indicate the aircraft is an arrival (blue for departures). It also overlays the logo of the airline onto the strip, making it easier to locate it later on the stripboard. Some time afterward, the pilot of THY1825 calls. Tessa then underlines the aircraft call sign on the strip, to visually indicate the flight is now her responsibility. Animated feedback is displayed on the radar screen, with a concentric circle designating the flight ...

Connecting physical and digital information. Strip'TIC connects the radar and stripboard views by establishing a visual link between the strips and the radar symbols.

The prototype leverages the advantages related to the embodied use of physical objects in physical space.

Paul observes that THY1825 is inbound to ODRAN, one of the three main arrival beacons for Orly. A light aircraft, call sign F-GFPS, is estimated to transit at the same time. Paul wants to inform Tessa as quickly as possible: He takes the strip off the board to get her attention, and points with his pen at the ODRAN label on the strip. This highlights all the representations of the flights crossing this beacon in the near future. Tessa immediately detects the potential conflict over ODRAN and decides to monitor it closely. Another aircraft is now calling: "Good afternoon, this is Lufthansa 8950, descending Flight Level 90 (approx. 9,000 ft.), inbound ODRAN." Tessa cannot find the corresponding strip on the board—the planner may have put it in an unexpected place. Fortunately, the flight is highlighted on the radar display. She points at it to have the strip highlighted too.

Real-time paper strips. Ink printed on paper is not dynamic, which forces controllers to reprint strips when they want updated information. Besides, real-time data is available only on the radar display, which requires the controller to mentally consolidate the available information.

Thanks to the real-time projection of aircraft altitudes onto strips, Tessa notes that THY1825 is reaching its assigned flight level 90. She decides to allow the aircraft to descend farther, contacts the pilot, and gives a new altitude: 3,000 ft. She records this clearance by circling the pre-printed 3,000 ft. label on the strip. The system acknowledges this order and signals it as consistent by highlighting the drawn circle in green (the feedback would have been red in case of an inconsistent clearance, for instance a new level above 90).

Enabling input from paper through OCR. Tessa also asks THY1825 to turn to heading 340. She writes 340 on the strip.

The system recognizes the entered values and can use it to check that there is no risk of collision with other aircraft.

Fostering embodied cognition. With Strip'TIC, controllers can keep interacting in the physical space and rely on externalization to decrease their cognitive load [4]. For instance, spatial arrangements of strips help manage their workload [2,5]. Some tacit guidance on the given clearance level may also have been provided through physical manipulations of the strip.

Tessa is right-handed and must hold the strip with her left hand to write on it. Therefore she knows instinctively that she has circled a high level, as its value is located at the bottom right of the strip.

By contrast, since virtual strips do not need to be held, they do not trigger this type of embodied memory.

Now the lowest aircraft in her sector, easyJet EZY4262, is five nautical miles from the runway. She needs to transfer control to the tower controllers, but more urgently, she must give an interception heading clearance to another aircraft to let it rejoin the landing pattern. As a physical reminder, she holds the EZY4262 strip in her left hand so she won't forget to send it to the tower controller.

Augmented strips as a mix of virtual and physical media. The system provides a hybrid artifact, in which paper and digital media are identical and have equal importance. When one controller holds a paper strip, the other controllers can use the digital pen to interact with its virtual twin: They can write on it or move it by dragging on its border. When repositioning the paper strip on the board, the virtual strip is aligned under it. The then obscured handwritten notes of the virtual strip are projected onto the paper strip. The system lets the controller extend the physical strip virtually.

The system provides a hybrid artifact, in which paper and digital media are identical and have equal importance.

Thirty minutes later, the traffic gets quite intense, so Tessa's sector needs to be split up. Two colleagues are on standby in the control room. Tessa initiates the degrouping mode in the system. She decides to hand off the area around the MOLEK beacon, picks up the strips of the flights that are in this area, and gives them to Patrick, the planner of this new sector. During this phase, the virtual twins of the removed strips remain displayed on her board. Until Patrick tells her that he is ready to control the corresponding aircraft, she continues to control them with their virtual strips, rearranging them on the stripboard and writing clearances as necessary. The entered information is simultaneously projected onto the corresponding physical strips on Patrick's stripboard. Later, the crew of a recently departed aircraft calls to signal a sick passenger and report they must return to the airport. Tessa wants to note the problem on the corresponding strip, but it is already quite full. She chooses to enlarge the physical strip with a virtual writing area by pointing at the right side of the virtual strip underneath.

Tangible computation and cognition. Currently, controllers need to perform error-prone calculations and take significant time to check their results.

Since the traffic is really intense, Tessa had to put arriving planes in holding patterns. The strips are organized in two stacks at approach beacons MOLEK and ODRAN, but our tactical controller wants to optimize the arrival sequence in order to fill potential gaps. As she lays the strips down, the system automatically calculates the stack exit time—that the pilot needs to know—and displays it beside each strip. She can rearrange the strips as needed to produce the optimal sequence.

This tangible calculation illustrates how computing support can complement physical manipulations, rather than replace them.

back to top  Tangible Interactive Workspace

Strip'TIC gave us the opportunity to explore mixed interaction in a context where effective and safe physical interactions already exist. Unlike usual TUI approaches that rely on the mapping of physical objects to virtual objects, coherence in Strip'TIC relies on the mapping of virtual objects to physical objects. Both physical objects and their associated manipulations have an inner coherence due to their cognitive role as external representations—for controllers, the means of traffic planning and the programming of actions. A consequence is that physical objects represent cognitive objects rather than virtual ones [6]. Additionally, the familiar handling of paper strips on the stripboard space allows controllers to manage a mixed representation space with a perceptual fusion of both physical support and marks (printed and inked), and digital data (projected) in the same movement. Finally, the manipulation of digital paper strips on the stripboard supports externalization of cognitive concerns, since the strips and their spatial arrangements offload the controller's cognitive processes related to their responsibilities toward the flights crossing their sector [2,4,5,7].

Designed with concrete goals, Strip'TIC emerged as an example of what we call tangible augmented reality. In terms of augmented reality, dynamic top and bottom projections augment the real physical strips by providing real-time data and feedback. As for tangibility, the strips are also efficient and graspable interfaces that control the projection, a sort of tangible window manager. In fact, by combining externalization and lightweight and embodied interactions in physical space, together with concurrent access, non-fragmented visibility, collaborative awareness, and reconfigurability, Strip'TIC illustrates several of the main themes of tangible interaction [8] and distributed cognition [7].

Strip'TIC is a research prototype, and even though our experimental results are very promising, experience shows that it is difficult to predict its operational fate. What is sure is that it improves our understanding of the properties and new potential of paper strips. It demonstrates that the transition from a disconnected to a fully connected digital system, beyond the opposition between paper and digital, can now occur. It also suggests that augmented reality and tangible interaction can be a worthy solution for ATC.

back to top  References

1. Mertz, C., Chatty, S., and Vinot, JL. The influence of design techniques on user interfaces: The DigiStrips experiment for air traffic control. Proc. of HCI-Aero. 2000.

2. Mackay, W.E. Is paper safer? The role of paper flight strips in air traffic control. ACM Trans. Comput.-Hum. Interact. 6, 4 (December 1999).

3. Hurter, C., Lesbordes, R., Letondal, C., Vinot, JL., and Conversy, S. Strip'TIC: Exploring augmented paper strips for air traffic controllers. Proc. of AVI'12. ACM, 2012.

4. Kirsh, D. The intelligent use of space. Artificial Intelligence (1995).

5. Harper, R.H.R. and Hughes, J.A. What a f-ing system! Send 'em all to the same place and then expect us to stop 'em hitting": Making Technology Work in Air Traffic Control, Technical Report EPC-1991-125.

6. Letondal, C., Hurter, C., Lesbordes, R., Vinot, JL., and Conversy, S. Flights in my hands: Coherence concerns in designing Strip'TIC, a tangible space for air traffic controllers. Proc. of CHI'13. ACM, 2013.

7. Hollan, J., Hutchins, E., and Kirsh, D. Distributed cognition: Toward a new foundation for human-computer interaction research. ACM Trans. Comput.-Hum. Interact. 7, 2 (June 2000), 174–196.

8. Hornecker, E. and Buur, J. Getting a grip on tangible interaction: A framework on physical space and social interaction. Proc. of CHI'06. ACM, 2006.

back to top  Authors

Jean-Luc Vinot is an associate professor at the Interactive Computing Laboratory at ENAC in Toulouse. He is a graphic designer and an HCI expert in the fields of ATC and aeronautics. His current research focuses on the design of safety-critical systems, graphical user interfaces, and tangible interaction.

Catherine Letondal is a researcher at the Interactive Computing Laboratory at ENAC in Toulouse. After completing a Ph.D. on participatory programming, she studied the use of the digital pen by contemporary music composers. Her current research themes are tangible interaction and adaptation of software design processes to various contexts.

Rémi Lesbordes is both an air traffic controller and an HMI expert for the French civil aviation authority (DSNA). Co-founder of the Strip'TIC project, he also works on 3D and augmented reality applied to the integration of drones in real traffic.

Stéphane Chatty is the head of the Interactive Computing Lab ( at Ecole Nationale de l'Aviation Civile (ENAC, Toulouse, France) and a scientific advisor at IntuiLab (, Labège, France). He specializes in engineering methods, tools, and theories for the design of highly interactive and critical systems.

Stéphane Conversy is a professor of computer science at the University of Toulouse - ENAC. His research focuses on the design and programming of interactive tools. He received his Ph.D. from University of Paris-Sud and his Habilitation from the University of Toulouse.

Christophe Hurter is an associate professor at the Interactive Computing Laboratory at ENAC in Toulouse. His research interests lie in the areas of information visualization and HCI, with focus on visualization of multivariate data in space and time, the design of scalable visual interfaces, and the development of pixel-based algorithms.

back to top  Figures

UF1Figure. Strip'TIC mixes paper and electronic strips. The system tracks controller actions and pen input simultaneously (here filtering on a beacon started from an interaction with the paper strip).

UF2Figure. A controller collaboratively optimizes an arrival sequence, using a paper strip, numeric pen, and projected information. Free-hand writing and drawings convey key information.

back to top 

©2014 ACM  1072-5220/14/07  $15.00

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2014 ACM, Inc.

Post Comment

No Comments Found