Authors:
Mahan Mehrvarz
The rapid development of generative AI, particularly since the advent of products such as ChatGPT, has underscored the importance of prompting. Large language models have transformed our interactions with computers through text, but their potential to control physical objects remains largely unexplored. This led me to ask: What if we could use conversational AI to interact with physical objects? The Prompting Realities project envisions bridging this gap by connecting conversational AI to tangible, interactive artifacts. This would enable people with no coding experience to control objects simply by talking to them, with AI translating their words into actions.
Prompting Realities is a pipeline designed to leverage the capabilities of LLMs to create an extendable system that mediates between end users and physical, interactive artifacts. This kind of system enables users to build new relations with artifacts, allowing them to reappropriate, manipulate, and alter the functionalities of digital, tangible objects.
The pipeline architecture relies on three primary components: a chatbot, an LLM (via OpenAI's Assistants API), and a microcontroller (ItsyBitsy M4) connected to an IoT protocol via WiFi. This configuration creates an iterative, tangible loop—from the user's prompt to the resulting physical actuation—shifting the role of system designers from direct programming to describing system behavior. The setup opens up new possibilities, enabling users to define what they want the system to do in their own words.
The project demonstrates this pipeline through a physical prototype: a landscape featuring three windmills, each with unique characteristics. Users control these windmills through natural language commands, with the system interpreting various descriptive references—from spatial indicators, such as "the one on the left," to stylistic descriptions (e.g., "the old-fashioned windmill"). This flexibility in handling references showcases the system's ability to connect natural human communication with precise physical control, while also embracing the occasional ambiguity or misinterpretation. Ultimately, it encourages playful development of prompting techniques for this specific tangible artifact.
Prompting Realities illustrates how LLMs can reshape our relationships with physical artifacts. I scaled models of windmills—that spin on command rather than wind—to offer a familiar object in an unexpected context, distanced from the usual repertoire of smart devices, such as lights, doors, and curtains. This choice allows users to focus on the open speculations around new relations with artifacts like "calling them names" or "asking them to dance together" without being constrained by preconceptions of voice-controlled automation. With this prototype, I'm interested in exploring new possibilities for end user programing of tangible products, where the boundary between users and programmers blurs and the threshold of computer literacy dissolves.
 | Three windmills that are different in style, position, and direction, on a scaled topographical model. |
 | A chat interface is used to communicate with the physical object using both text and voice message features. |
 | The assistant agent interprets initial instructions, and uses an LLM and JSON schema to generate responses and variables, which update the microcontroller, resulting in modified functionalities and tangible interactions. |
Author
Mahan Mehrvarz is a Ph.D. candidate at AI Futures Lab at the Delft University of Technology. He is a technology architect and researcher with a background in interaction design, media art, and architecture. His work focuses on the ways experiential DNA of design can drive meaningful insight and change, particularly in addressing the psychosocial impacts of digital technology. [email protected]
Copyright 2025 held by owner/author
The Digital Library is published by the Association for Computing Machinery. Copyright © 2025 ACM, Inc.
Post Comment
No Comments Found