Authors:
Becca Glowacki
In 2016, the U.K. media watchdog Ofcom published a report from a three-year study that focused on how children use media [1]. Their findings suggest that younger children tend to use mobile devices for passive media consumption. Children are motivated to use mobile devices such as iPads or tablets to engage in online social activity beginning in their early teens, but before that age, their media usage mostly involves watching content and playing games alone.
Challenging this kind of solitary media activity is important because children develop through social interactions. It is widely understood that learning is a social activity, and many contemporary approaches to learning, such as expansive or situated learning, have their roots in social-learning theories. Lev Vygotsky, an educational psychologist from the early 20th century, argued that cognitive ability grows when learners are guided by a more knowledgeable other [2]. Vygotsky indicates in his work that social interactions and joint engagement are the foundation needed for learning to take place. He suggests that the role of a teacher is to support social interactions through scaffolding social experiences for learners.
Insights
We have identified two possibilities for exploring more social ways of engaging with digital media: creative play with stories and using media with tangible objects. In this project, we explored how the crossover of both tangible interaction and storytelling could scaffold joint engagement with mobile media, a term known as joint media engagement [3]. We draw upon storytelling in our study because it has affordance to be a social activity: Telling stories is a method of communication and an outlet for creative expression. We coupled storytelling with tangible media because we wanted to explore the potential for joint media engagement when playing in physical space. Paul Dourish explores the crossover between tangible and social interaction, arguing that tangible interfaces encourage active participation between media participants in real space [4].
Tangible Tales
Building on tangible storytelling tools such as Storymat and Animal Blocks (developed in the Gesture and Narrative Language group at the MIT Media Lab) and Jabberstamp and TOK, which combine multiple sensory experiences to develop interactive stories, we set out to develop an app that invites children to use tangible objects to make stories. We were exploring ways in which a mobile device could prompt joint media experiences, much like the imaginary play that friends take part in with a puppet set, figure toys, or a doll house. As a starting point, we developed a number of prototypes, where objects made from accessible and familiar materials, such as card stock and ink, were augmented by characters, animations, and sound from the mobile device. For example, a housing block made with paper is animated with activity that you can see through the windows; digital leaves fall from the branches of a small cardboard tree; a door physically opens with creaking sounds (Figure 1). These initial experiments developed into a children's storytelling app called Bear Abouts.
 |
Figure 1. Initial tests. a) A physical door opens to a creek. b) Digital leaves and paper branches. c) An animated bear peeps out the window of a physical house. |
Bear Abouts is a storytelling platform where you use tangible objects to interact with digital stories. With it, you have the possibility of making your own objects and making your own stories. We designed a system that was able to learn and calibrate custom cardboard sensors that use touch points as their unique identifiers. The current platform recognizes the touch pattern that is encoded onto the bottom of each object with a conductive material. The system uses machine learning to recognize an object (it can be calibrated to avoid variation error) and recalls audio, text, and animations that are conceptually connected to it. By attaching a custom sensor to a tangible object, it is possible to integrate physical elements into the system using an animator built into the app. The platform can be used as a place to make stories and also as a space to bring stories to life (Figure 2).
 |
Figure 2. The Bear Abouts platform. a) The system uses machine learning to recognize each object. It can be calibrated to avoid variation error. b) The storymaking part of the app in use. c) A sensor made from card and copper, working with the app. |
To test out our ideas, we developed the app in a number of stages over the course of six months. The stages were as follows: 1) to build a working prototype of the app and explore how children played with it; 2) to explore the kinds of stories that can be developed with the app and experiment with ways of storytelling between the physical and digital; and 3) to find out what kinds of tangible objects children make with the app.
Testing in Schools
The first Bear Abouts story was based on a migrant bear who has a bag of useful objects to aid him on his journey. Inside the bag are sunglasses, a sandwich, a compass, and a woolly hat. In the initial steps of this project, we tested this story with a group of children ages 5 to 7 in a primary school (Figure 3). We found that children were using the app to develop stories both with the system and also through dialogue with each other. They mostly used the objects together, either taking turns to pass the objects back and forth, thus sharing the storytelling; or by co-creating narratives together as they went along. The children also developed verbal narratives together, and when asked to recall their version of the stories, they used the objects as props to retell them.
 |
Figure 3. Animating a paper pair of sunglasses from the first school test. |
Making Stories
In the next step of the project, we expanded on ways of storytelling and investigated the kinds of stories that could be made between the device and children using the Bear Abouts app. To do this, we worked with professional story makers to explore what kinds of stories could emerge with this type of interaction (Figure 4). We invited story makers with different backgrounds to a number of creative labs geared around the technology. Selected story makers then worked with the app for a couple weeks to develop ways of telling stories. We also worked with a children's book illustrator, a puppeteer, and an immersive storyteller. Each maker brought a different dimension to the process.
 |
Figure 4. Creative lab exploring the process of making stories with an app with professional story makers. |
The Search
The puppeteer started to make sensors through combining ideas around navigation and optical illusion—seeing beyond where you can see right now in order to go beyond reality in an imaginative way. She approached the tangible objects as ways of seeing digital spaces. The story that emerged was co-created in real time, between clues from the objects, prompts from the device, and exploration by the children participating in the story. Accompanying the story was a giant fabric map that you could literally step into as part of the experience.
We tested the story at the same primary schools as before with children ages 5 to 7 (Figure 5). When testing the story, two children, unprompted, took their shoes off before entering the map. They interacted with the story by physically standing inside the space; one held onto the object, and one held onto the device. They were engaged in the story that emerged from the connection between object and device, moving about the entirety of the map underfoot, swapping sides, holding objects between them, and asking each other questions: "What can you see?" "I can see the sun. I can see the shooting stars." "Can you see the clouds? Can you see Saturn?" They then went on to describe their viewpoints.
 |
Figure 5. Children playing with the Search story on a giant fabric map. |
The meandering story that the children created took place by developing connections between the digital and physical space, and between the child and the device. They collaborated to develop their own narrative through determining the order in which they used the objects. By doing so, the children developed verbal narratives together, and in the process developed their own meta-story in which they played a part, as well as triggering a story that would emerge through using the app. They were both part of a story and creatively engaged in developing a story.
This Mine is Mine
The children's book illustrator developed a story based around geology and the mystery of underground. The device became a portal to explore things that live below our feet. The story she developed is about a little girl who discovers a magical shed at the back of her garden, which, in Narnia fashion, is the doorway to a mine that leads her on an adventure in an underground world. As the story goes on, the little girl finds out about the worms that live in the mud, meets moles, and gets gemstone gifts from curious creatures. She has an adventure, learns about bugs and creepy crawlies, and gets home in time for tea. The role of children when playing with the app is to support the little girl on her journey using objects that are connected to the story. The story is nonlinear and the objects drive the narrative. The order in which you place the objects on the screen determines the order of the narrative. Similar to the story about a bear with a backpack, piecing together elements of the story enables children to develop a narrative.
We also tested the story in a primary school with children ages 5 to 7. When trying out the story with children, we found that they interacted not only with the device through the story, but also with the objects. They referred to the physical properties of the objects: "Ooooh! It's heavy." They were also engaged in the real space by looking through the instruments at the real world and then placing them back on the screen to explore the story. One group we worked with spent a long time exploring the objects in the real world and making stories with each other, then applying this to the story emerging in the digital space.
Ground-O-Scopes
In the third part of this project, we explored how children responded to making their own objects. We set up a workshop to give children the opportunity to make their own tangible objects with the sensors. In the workshop we provided a range of materials, including sensors, crafting materials (such as card stock, glitter, felt-tip pens, and pompoms), and a number of mobile devices on which the app was pre-installed. We invited children and their families to use the app as a platform to make objects. We called the sensors ground-o-scopes: imaginary viewing instruments for finding insects and other creepy crawly creatures beneath the earth (Figure 6). With this prompt for making, some children made ground-o-scopes, while others developed creatures and other creations to be part of the story.
 |
Figure 6. Ground-o-scope sensors made by children in the workshop. |
Mixed-Place Spaces
One of the key things that we observed in our studies was the way that children engaged with people and space around them while using the app. In our tests, children play-acted with the objects in real life as well as playing with them in digital space. For example, when using the app, children would offer the objects to their peers or teachers: "See what happens when you look through this." "Have a go." "Let me show you what happens."
There were also offers of play that were not connected to the app: Through the tangible objects, children had the possibility to play beyond the boundaries of the content programmed into an app, extending the digital story reality into their physical reality. For example, in one instance two children explored the ways that a magnifying glass lens could shift shapes around the room (Figure 7). The magnifying glass was connected to the app both thematically (in the story) and physically (with the sensors), but for a while the children's play took place between the two of them only. They passed it between each other to take turns in looking through it, before returning it to the device and using it with the app.
 |
Figure 7. Children playing with the magnifying glass from This Mine Is Mine. |
These examples suggest that the presence of tangible objects invited imaginative play in a triangulation between device and child, and between child and child. The way in which children used the objects to retell and reframe narratives with each other, as well as tell stories in digital space, demonstrates the power of tangible interaction in transcending these spaces. The story could be part of a reality in the app, but also a part of a reality in physical space. The objects were relevant in the story, or in their own right, or they were relevant at the time and place that children were participating in the story experience. For example, a telescope was used as part of the story with the app; it is also used as a relic of maritime exploration; and it was used as an object of play between the participants in the space.
The tangible objects afforded the possibility to physically share the experience, and through sharing, children became facilitators to the action in real life. The objects were agents of the story in real life, as well as agents of the stories in digital space. They transcended the physical and digital into this mixed-play space, giving children the opportunity to be the creators through physical facilitation and allowing them to be creative agents of the story, in collaboration with each other and the device. In our tests we found that through augmenting the story with the objects, rather than operating as a gatekeeper to a story, the device became a facilitator to a playful social experience that could be acted out through movement and language.
More Knowledgeable Devices
Our preliminary research has involved tests with small groups, but another direction would be to explore how children use the app when they are not in groups. Further exploration could be made with children using the app on their own. A 2016 study from Northwestern also recognizes the problems of solitary use of mobile devices, suggesting that there are developmental issues with the way in which media is consumed by children [5]. The authors expand some of the Vygotskian ideas around social learning and make an analogy between learning with a device and learning with a more knowledgeable other. They suggest that if a more knowledgeable other is a mobile device (as is the case with many young children using mobile media on their own), then the way in which learning is designed for mobile media is critical to children's development. From our pilot studies, we have seen how tangible media can facilitate social interaction between groups of children. Further research could be carried out into how children co-create with the platform when on their own, and it could also explore the role of the platform in the co-creation process.
Making it Yourself
From the start, we set out to make a low-cost sensor that could be used with current mobile devices. To keep costs to a minimum, a significant part of our design process has been concerned with developing sensors and software that connect paper with tablet devices without the use of external electronic hardware. The addition of external hardware can be expensive, whereas in the U.K., more than 50 percent of children already have access to a tablet [6]. We developed a system of custom sensors that have the potential to be handmade with a printed template and some household items such as cards, tape, and aluminium foil. One further space for inquiry would be into how to make these at home. The platform does not use any external electronic hardware, because it relies on the device "seeing" touch patterns. It has the potential to be a low-cost option for tangible interactions; there is room to explore how participants could print their own sensors and build tangible objects from scratch.
Acknowledgments
We would like to thank the following people and organizations for their contribution or continued support in the development of Bear Abouts: Mark Wonnacott (software development), Sarah Warden (creative producer), Amy Rose (immersive story maker), Emma Powell (puppet maker), Liv Bargman (illustrator), Filton Ave Primary School, Pervasive Media Studio and the Watershed, DCRC, Innovate UK, UWE, Nick Willshire (CAD), Joshua Gaunt and Jack Offord (film documentation).
References
1. Ofcom. Children's Media Lives—Year 3 Findings. 2016; https://www.ofcom.org.uk/__data/assets/pdf_file/0015/94002/Childrens-Media-Lives-Year-3-report.pdf
2. Vygotsky, L. Mind in Society: The Development of Higher Psychological Processes. Harvard Univ. Press, Boston, 1978.
3. Takeuchi, L. and Stevens, R. The New Coviewing: Designing for Learning through Joint Media Engagement. The Joan Ganz Cooney Center at Sesame Workshop, New York, 2011.
4. Dourish, P. Where the Action Is: The Foundations of Embodied Interaction. The MIT Press, Cambridge, MA, 2001.
5. Wartella, E. et al. What kind of adults will our children become? The impact of growing up in a media-saturated world. Journal of Children and Media 10, 1 (2016), 13–20. DOI: 10.1080/17482798.2015.1124796
6. Ofcom. Half of UK homes turn to tablets - in just five years. 2015; https://www.ofcom.org.uk/about-ofcom/latest/media/media-releases/2015/five-years-of-tablets
Author
Becca Rose Glowacki is an artist-educator-researcher. Her work explores ways of learning with technology. She is a senior lecturer in media design at the University of the West of England, a member of the Digital Culture Research Centre, and a resident at the Pervasive Media Studio, Bristol, U.K. [email protected]
Footnotes
Copyright held by author. publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.
Post Comment
No Comments Found