Authors:
Victor Cheung
The science fiction genre has been the prime platform showcasing some of the most creative and otherworldly computer user interfaces (UIs), particularly in movies, TV shows, and games. With the help of visual effects and computer graphics, these UIs play a crucial role in convincing the audience that the world the story is set in, often a futuristic scene, is plausible. By extension, the stories are believable [1]. One way to achieve this is to incorporate interfaces that only exist in research labs (e.g., gestural UIs from the 2002 movie Minority Report).
The incorporation of unseen but promising interfaces and technologies, however, has a more subtle yet practical motive: to reach the general public and affect their perception, thus promoting acceptance and advancements (e.g., artificial heart technology [2]). In fact, much of HCI research (e.g., shape-changing interfaces [3]), as well as the DIY community—for example, the Cortana appliance project [4] based on the Halo franchise—has been inspired by the UIs in sci-fi movies and video games (the reverse is also true [5]).
As an educator in computer science and HCI, I believe sci-fi UIs can be more than fun video clips that generate some oohs and aahs in a classroom. They can be used as a powerful vehicle in encouraging CS students to examine the relationship between technologies and humans and to flex their technical knowledge. This belief led to a recent upper-division undergraduate course that I designed and taught in which the term-long project was to re-create a sci-fi UI in real life. (In comparison, MIT's Sci Fab had a different focus on the futuristic artifacts and how they inform today's technologies; https://bit.ly/4gLw59k.)
The objectives of the course were the following:
- To introduce students to a wide range of UIs, novel interaction styles, and their associated technologies
- To be able to read, present, and discuss literature from the HCI community on topics related to the course
- To be able to design, implement, and evaluate a novel computer UI.
To achieve these goals, the project had multiple phases, each presented in a different format to further train the students on illustrating their work:
- A proposal with a sci-fi computer UI selected and a plan on how to implement it (e.g., deciding which hardware and software to use, determining which tasks it supports)
- A low-fidelity prototype that demonstrates progress and elicits feedback/suggestions from the class
- A presentation that provides details about the selected UI, showcases the refined prototype, and reports the evaluation results of the prototype
- A report on the project (using the pictorial format in the HCI community) that includes evaluation of the design, recommendations for improvements, and reflection (lessons learned).
Figure 1 shows an example of a student group project that implements the digital scope of a sniper rifle from the 2015 movie Hitman: Agent 47 (http://bit.ly/41aMjTI).
 | Figure 1. An overview of a project from the course: a) the sketch of the prototype, including the UI of the digital scope taken from the movie Hitman: Agent 47; b) the finished prototype; c) a student testing the prototype; and d) the re-creation of the UI. |
The student group used several tools, including 3D printing, Arduino and sensors, Weather and Google Maps API, and React Native software, to re-create a UI that allowed users to zoom in and out via a touch sensor and provided various hints and real-time feedback, such as bullet drop prediction, rifle stability, wind direction, and speed.
Sci-fi UIs can be used as a powerful vehicle in encouraging CS students to examine the relationship between technologies and humans.
Among the pleasantly surprising outcomes from the course were the following:
- Students came back with a variety of UIs. Each of the six groups found a different UI from either a sci-fi game, TV show, or game.
- Students were motivated to apply and learn different technical skills to implement the UIs. For example, they used Unity 3D for heads-up display (HUD), React for multidevice communication, and ARToolKit for volumetric projection, many of which were not part of the current curricula.
Course Lessons
The course was successful in inspiring students to pursue disciplines in related topics (one, for example, decided to take an embedded systems course and later applied for a master's program in computer engineering) and left them with something they can be proud of and show their peers. Nevertheless, I realized a few things that will improve the delivery of the content and the impact of the course.
Offer clear framing and description of the project requirements. Since students could choose from a sci-fi movie, TV show, or game, the UIs were vastly varied, thus making it difficult to evaluate their effort, and for them to evaluate their own progress. Some criteria of what constitutes an acceptable UI—for example, must have a display, must have an input, must support X number of distinct tasks—should be provided, with examples of acceptable and unacceptable UIs. It might also be helpful to choose a theme (e.g., HUDs, large interactive displays) so that there is greater synergy among projects.
Provide stronger support and guidance for the technical aspects of the projects. Due to the open nature of the projects, students might have unrealistic expectations of what they could create (e.g., one group wanted to re-create the object recognition part of the Iron Man HUD but realized that required more than they could implement). Help should be provided to the students to narrow their project down to a feasible scope in terms of time and resources, as should pointers and suggestions on where and how to acquire the materials and knowledge to re-create the UIs (e.g., what software libraries they could use, how to build a low-cost volumetric display or mock up a large display).
Encourage students to think beyond just implementing the UI. While the primary deliverable of the course was the UI implementation, students should also be encouraged to reflect on the UI design, especially its impact on the real world. As argued by Daniel Russell and Svetlana Yarosh [6], sci-fi might "provide a good grounding for discussion" but would not be suitable for predicting the future due to "common tropes" and "shortcuts." To train students to be responsible and thoughtful designers of the future, questions that make them reflect on the design can be asked. These could include: How feasible are the UIs (e.g., space/computational/cognitive requirements)? What are the limitations if they are used in real life (e.g., ergonomics if used for an entire day)? (For more examples, see Russell and Yarosh's five heuristics for design fiction writers [6]).
To conclude, re-creating a sci-fi computer UI in real life as the underlying theme made this course an appealing opportunity for students to apply their technical knowledge in an engaging way and get a glimpse into some active areas of HCI research. With the described improvements, it is my hope that the course will be more than a technical CS course and that it will equip students with the ability and foresight to design UIs for the future.
References
1. Shedroff, N. and Noessel, C. Make It So: Interaction Design Lessons from Science Fiction. Rosenfeld Media, 2012.
2. Kirby, D. The future is now: Diegetic prototypes and the role of popular films in generating real-world technological development. Social Studies of Science 40, 1 (2010), 41–70; https://doi.org/10.1177/0306312709338325
3. Troiano, G.M., Tiab, J., and Lim, Y.-K. SCI-FI: Shape-changing interfaces, future interactions. Proc. of the 9th Nordic Conference on Human-Computer Interaction. ACM, 2016, Article 45, 1–10; https://doi.org/10.1145/2971485.2971489
4. Archer, J. Holographic cortana appliance: Working concept. Unt1tled; http://unt1tled.com/blog/cortana-hologram-working-concept
5. Dourish, P. and Bell, G. "Resistance is futile": Reading science fiction alongside ubiquitous computing. Personal and Ubiquitous Computing 18, 4 (2014), 769–778; https://doi.org/10.1007/s00779-013-0678-7
6. Russell, D.M. and Yarosh, S. Can we look to science fiction for innovation in HCI? Interactions 25, 2 (2018), 36–40; https://doi.org/10.1145/3178552
Author
Victor Cheung is a lecturer at Simon Fraser University. He has a Ph.D. degree in systems design engineering from the University of Waterloo. His teaching and research interests involve designing and evaluating technologies that mediate communication between humans and computers, resulting in improved user experience and user empowerment. [email protected]
Copyright 2025 held by owner/author
The Digital Library is published by the Association for Computing Machinery. Copyright © 2025 ACM, Inc.
Post Comment
No Comments Found