Tovi Grossman, Fanny Chevalier, Rubaiat Kazi
A picture is worth a thousand words. Scientists have long recognized this idea, and have been including charts, drawings, and photographs in their research articles even before the availability of desktop computers and document-editing software. Similarly, we would argue that a video or animation can, in some cases, be more effective at expressing a concept than a static image. This is especially the case in research areas such as human-computer interaction (HCI) and computer graphics, which deal with animation and interactive systems.
Surprisingly, scientific papers with embedded videos or animations are rarely published. Instead, venues may allow authors to submit a separate video that accompanies their article. This is somewhat primitive, equivalent to submitting a deck of figures that accompany a paper instead of embedding the figures directly into the article. While the technology to embed and render animated content within a PDF document has not been readily available until recently, it is now possible and fairly easy to embed videos and interactive content into a PDF—the status quo digital format for scholarly publications. New Web-based multimedia technologies such as HTML5 allow us to further think more broadly about the presentation of scientific articles that break the current constraints of existing distribution formats. Importantly, the consumption of scientific content has transitioned away from physical paper toward modern electronic-based media, which can display inline animated content. Given the ubiquity of these technologies, it is time to rethink how we disseminate scholarly research, and in particular, research that involves interactive content or dynamic systems.
While this topic is relevant to all fields of scholarly research, we feel no community is better suited to pursue this new vision than the community of HCI researchers, UX practitioners, and interaction designers, whose roots are in the design and development of technologies that facilitate the communication of digital content between humans and machines. No community would better understand that extra care and precautions are needed to ensure an aesthetically pleasing reading experience. To that end, we propose the use of animated figures, continuously looping videos that minimize disruptions to the typical reading experience.
In our Draco research project , we broke from the standard mold of rigid, static research articles and submitted the first research paper in the ACM Digital Library that embeds this type of animated figure. Here , we first provide an overview of the history of publication formats and describe the concept of an animated figure and a set of possible use cases. We then discuss recommendations to the broader research community going forward.
Academic publishing has undergone profound changes over the decades as the writing, submission, and dissemination processes evolved from a fully physical form to one increasingly reliant on electronic media  (Figure 1).
The earliest scientific journal aimed at providing a public record of original contributions to knowledge, Philosophical Transactions of the Royal Society, was introduced in 1665. With it originated academic publishing, which turned the exchange of scientific and technical information from encrypted letters mailed between small groups of private correspondents to organized periodicals disseminating clear and transparent claims to the learned society.
During this era, scientists submitted hard copies of handwritten manuscripts by mail, which were transcribed and then reproduced with traditional printing techniques. When used, figures were replicated using etching or drypoint printmaking techniques, where the image is incised into a copper or zinc plate that serves as a stencil. With the advent of typewriters in the early 1900s, publishers began imposing rigid formats on authors to make the production process more efficient and guarantee presentation consistency across articles.
The development of electronic technology, in particular the emergence of word processors in the late 1960s, drastically changed the way in which manuscripts were prepared. Authors became free to revise their articles as many times as necessary without the burden of retyping a page. However, physical distribution, either through paper or electronic disks, held until bandwidth capacity and transfer protocols enabled electronic transmission of files in the late 1980s .
While digital submissions developed quickly, the production of scholarly literature long remained essentially paper-based, with steady releases of physical journals and proceedings. Constrained by the paper medium and associated extra costs for color printing, authors included full-color figures only in rare cases.
When bandwidth allowed quick download of files, the Journal of Image Guided Surgery (1995) was the first peer-reviewed journal to offer an electronic version of its articles, later followed by many other journals . With digitization, color figures became commonplace. However, in spite of such a shift—and contrary to early predictions —articles kept their print-based nature. Adobe's PDF emerged as an efficient way to electronically deliver print-based documents in the early 1990s. Similar to its predecessor PostScript, PDF is based on vector graphics and optimized for printers. PDF became the standard format thanks to its greater portability over HTML at the time and its widespread adoption in the production workflow .
In parallel, auxiliary material in the form of video figures developed in scientific branches such as computer graphics and HCI, which deal with interactive systems. Similar to paper submissions, multimedia material was initially mailed on VHS or disks. As digital video file formats matured, videos could be submitted using electronic Web-based systems. With rare exceptions, such as The Journal of Visual Experiments which (exclusively) publishes scientific contributions as videos, the vast majority of conference and journal publications detach the multimedia content from the actual articles, even though embedding rich media in PDF files has been available since as long as 10 years ago.
While no model has yet imposed itself, diverse scientific communities are putting forth efforts to make academic publications interactive and open access. The Optical Society of America and National Library of Medicine at NIH jointly developed Interactive Science Publishing (ISP), a software for authors to organize and publish source data in scientific articles. With this medium, 2D/3D data figures can be viewed and analyzed interactively by the reader using the software. In a similar vein, the Collage European research project allows e-scientists to author actual executable papers, where methods can be executed, analyzed, and evaluated by the reader.
In 2014, we published "Draco: Bringing Life to Illustrations with Kinetic Textures" , which was the first scholarly publication in the ACM Digital Library to embed an animated figure within the actual article. The following section defines this form of multimedia component, which we argue should be better utilized and welcomed within scientific articles.
While PDFs support the broad use of videos and interactive Flash objects, special care must be taken in the integration of such formats within a scholarly research article. In particular, there should be minimal disruption to the typical reading experience. This means a reader should not be burdened with complex UI controls. Equally important, readers should not be distracted by the animation when reading text. As such we propose the use of animated figures.
Figure 1. A brief history of academic publishing.
An animated figure is a short autonomous and continuously looping video that enhances what would otherwise be a static figure or set of figures (Figure 3).
The process for embedding an animated figure within a PDF is fairly straightforward, as we have previously outlined . For specifics steps and examples, we refer the reader to our online appendix . Some important considerations should be made:
Minimize distractions. Appropriate steps should be taken to make the loop from the end to the start of the video as seamless as possible (Figure 2). When this is not possible, the original video can be modified so that there is a short cross fade between the first and last frame to avoid an abrupt visual jump. This will minimize distraction when the animation is in the reader's periphery view while reading text.
Utilize playback controls sparingly. Playback controls should be provided only if the video is anticipated to be disruptive or distracting to the reading process. Readers should be focused on the content of the paper, not UI controls. If playback controls are utilized, the figure should have some visual indication, such as an overlaid icon, to indicate to the reader that they can click to view the animation (Figure 2).
Figure 2. Left: An animated figure that plays automatically. The figure illustrates an animation effect and loops continuously. Source: Schödl et al., SIGGRAPH 2000. Right: An animated figure that must be clicked to begin playback. The figure illustrates an interaction technique. Source: Mott et al., CHI 2014.
Include an alternate static image. A static figure should always be placed in the position where the animated figure will appear. This image should be an appropriate static representation of the animated figure, to ensure that the article is still self-contained when printed on paper, when the video is disabled, or if for some reason the video playback is not supported. In our initial article, we used a still frame of the video with callouts to the animation effects (Figure 4).
We believe there are many situations when animated figures will be a valuable addition to a research article. Below we categorize the primary use cases.
Demonstrating how an interaction technique works. Often papers that describe interactive systems or interaction techniques must use a sequence of static figures to represent the nature of the interactions being described. In such cases, an animated figure could better convey the actual dynamics to the reader (Figure 2, right).
Illustrating interactive visualizations. Animations may also be useful when illustrating interactive visualization systems. These could be especially useful where there are coordinated dynamics between two components of an interface or visualization, for example, when brushing one area of a visualization dynamically modifies the properties of a second area of the visualization (Figure 5, left).
Contrasting visual differences. An animated figure can also be useful for contrasting two static figures that may have subtle differences. This could be used when comparing the output of multiple rendering algorithms, or contrasting results generated from different conditions of a user study. Instead of displaying the results side by side, an animated figure could fade between the two images to make their differences more apparent (Figure 5, middle).
Figure 3. Animated banner image as found in .
Figure 4. The static representation of the animated banner image, as found in , includes callouts in place of the actual animation effects.
Displaying an animation. Maybe the most obvious use of an animated figure is to depict an animation effect that a research paper is describing (Figure 2, left). Depending on the nature of the animation, it may be difficult or impossible to adequately communicate such animation effects with only text and static images.
Visualizing a process or algorithm. Animation can also be useful to explain how a process or algorithm works. When such processes or algorithms involve sequential steps or spatial information, an animation could allow the author to more efficiently communicate the concept to the reader (Figure 5, right).
While paper-based media is still prevalent, we do seem to be slowly transitioning toward electronic-based media consumption. The prevalence of tablets and eBook readers has accelerated this transition. As such, we are in favor of authors using animated figures in place of static images, and conferences and journals explicitly encouraging the practice. However, this should be done only in situations where it will significantly benefit a reader in understanding the accompanying text of the paper.
Instructions and regulations. As of today, most conferences and journals, including the ACM CHI conference, where our article was published, have no published policy on the use of embedded animations within submitted articles. However, there are existing research venues that have started to encourage the embedding of interactive content within PDFs. The Journal of Neuroscience provides particularly good advice on when interactive media should be embedded into PDF files:
Most conferences and journals have no published policy on the use of embedded animations within submitted articles.
The Journal of Neuroscience will publish multimedia and 3D models embedded in the HTML and PDF versions of articles. Essential multimedia and models will be displayed in line with the article text, as is done for figures. Authors should be aware that multimedia and 3D models will not be available to readers who only access the printed version.
Authors should not use multimedia and 3D models simply to enliven or embellish their article. Because they substantially increase file sizes and are not available to all readers, multimedia and 3D models will be accepted only when they are needed to display findings that 1) are essential to defend the articles conclusions and 2) could not be presented adequately in the text, a table, or a figure .
Figure 5. Example use cases for animated figures. Left: Interactive visualization. Source: Matejka et al, UIST 2014. Middle: Contrasting visual differences. Source: Xie et al., UIST 2014. Right: Visualizing an algorithm. Source: Baecker 1998.
We stress the importance of such official instructions as a reference for both authors and reviewers, and hope these types of guidelines will soon be adopted by other research venues to encourage the appropriate use of interactive content, and in particular, animated figures, within submitted and published research papers.
Reliance on commercial software. It should also be noted that most, but not all, PDF viewers are able to display animated figures. The swf capabilities are part of the published PDF ISO standard (ISO 32000 Extension Level 3), but some viewers still do not support playback. This raises questions regarding the PDF proprietary format itself as being the appropriate format for publishing. In particular, HTML5 offers responsiveness and accessibility, and can support dynamic and interactive content (see McGill's SIGCHI HTML5 Paper Template ). We believe it is important for publishers to seriously consider these other types of publication formats.
Interactive figures. Embedding animated contents in scientific articles can be seen as a step toward a more active reading experience in digital media. This could be extended further with the use of interactive figures, which would allow readers to interact with and explore the figures embedded in a research article. Interactive figures invite exploration, allowing users to change conditions and see consequences immediately . In academic papers, interactive diagrams can be beneficial in many cases, such as navigating a 3D model, exploring interactive statistical data charts, and illustrating how a dynamic software or hardware system works. Allowing the reader to control the presentation of the content creates an active reading experience, in contrast to the passive reading experience of current academic papers. The resulting implications of this fundamental change warrant future investigations.
We have described the use of animated figures within research articles, outlined a set of use cases where animated figures could be beneficial, and discussed recommendations and caveats surrounding their use. We hope this work will elicit broader debate and discussion among the community of researchers and publishers about the future of dynamic scientific publication formats.
We thank the reviewers of the alt.chi 2015 article from which this article has been adapted .
1. Kazi, R.H., Chevalier, F., Grossman, T., Zhao, S., and Fitzmaurice, G. Draco: Bringing life to illustrations with kinetic textures. Proc. of the ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, New York, 2014, 351–360.
2. This article is adapted from an abstract originally published in the extended abstracts of the ACM CHI 2015 conference: Grossman, T., Chevalier, F., and Kazi, R.H. Your paper is dead!: Bringing life to research articles with animated figures. Proc. of the ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, New York, 2015, 461–475.
8. Mark McGill's SIGCHI HTML5 Paper template: https://www.youtube.com/watch?v=oka2XiMELyU
9. Victor, B. Explorable explanations. 2011; http://worrydream.com/ExplorableExplanations/
Tovi Grossman is a distinguished research scientist at Autodesk Research in Toronto. His research is in HCI, focusing on input and interaction with new technologies. He received a Ph.D. from the Department of Computer Science at the University of Toronto. email@example.com
Fanny Chevalier is a researcher at INRIA, France, in the Mjolnir team and affiliate assistant professor at the University of Toronto, Canada. Her research is in HCI, with a focus on animated visualizations and perception. She received a Ph.D. from the University of Bordeaux. firstname.lastname@example.org
Rubaiat Habib Kazi is an HCI and graphics researcher in the User Interface group at Autodesk Research (Toronto). His research interest lies in designing sketch-based interfaces that facilitate powerful ways of thinking, communication, and storytelling with playful experience. email@example.com
Copyright held by authors. Publication rights licensed to ACM.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2016 ACM, Inc.