Learning from Animations in Science Education: Innovating in Semiotic and Educational Research [1st ed.] 9783030560461, 9783030560478

Citation preview

Innovations in Science Education and Technology 25

Len Unsworth   Editor

Learning from Animations in Science Education Innovating in Semiotic and Educational Research

Innovations in Science Education and Technology Volume 25

Series editor Karen C. Cohen Weston, MA, USA

As technology rapidly matures and impacts on our ability to understand science as well as on the process of science education, this series focuses on in-depth treatment of topics related to our common goal: global improvement in science education. Each research-based book is written by and for researchers, faculty, teachers, students, and educational technologists. Diverse in content and scope, they reflect the increasingly interdisciplinary and multidisciplinary approaches required to effect change and improvement in teaching, policy, and practice and provide an understanding of the use and role of the technologies in bringing benefit globally to all. Book proposals for this series may be submitted to the Publishing Editor: Claudia Acuna E-mail: [email protected]

More information about this series at http://www.springer.com/series/6150

Len Unsworth Editor

Learning from Animations in Science Education Innovating in Semiotic and Educational Research

Editor Len Unsworth North Sydney Campus Australian Catholic University North Sydney, NSW, Australia

ISSN 1873-1058 ISSN 2213-2236 (electronic) Innovations in Science Education and Technology ISBN 978-3-030-56046-1 ISBN 978-3-030-56047-8 (eBook) https://doi.org/10.1007/978-3-030-56047-8 © Springer Nature Switzerland AG 2020 Open Access Chapter 1 is licensed under the terms of the Creative Commons Attribution 4.0 International License http://creativecommons.org/licenses/by/4.0/). For further details see license information in the chapter. This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Contents

Part I 1

A Multidisciplinary Perspective on Animation Design and Use in Science Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Len Unsworth

Part II 2

3

4

Introduction 3

Educational Semiotics and the Representation of Knowledge in Science Animation

A Functional Perspective on the Semiotic Features of Science Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yufei He

25

Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis of Ecology Animations for Children . . . . . . Mandy Hoi Man Yu, Dezheng (William ) Feng, and Len Unsworth

55

Multimodal Affordances of Immersive Virtual Reality for Visualising and Learning Molecular Interactions . . . . . . . . . . . Kok-Sing Tang, Mihye Won, Mauro Mocerino, David F. Treagust, and Roy Tasker

Part III

75

Learning From Viewing Science Animations

5

Using Animated Simulations to Support Young Students’ Science Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Garry Falloon

6

Promoting Scientific Understanding through Animated Multimodal Texts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Maximiliano Montenegro, Alejandra Meneses, Soledad Véliz, José Pablo Escobar, Marion Garolera, and María Paz Ramírez

v

vi

Contents

7

Using Animation in the Representation Construction Approach in Senior High School Chemistry . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Zeynep Yaseen

Part IV

Learning Through Creating Science Animations

8

Slowmation and Blended Media: Engaging Students in a Learning System when Creating Student-Generated Animations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Garry Hoban

9

Animation Construction as Cross-Modal Translation in Senior Biology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Peta J. White, Russell Tytler, and Wendy Nielsen

10

Creating a Digital Explanation in Preservice Teacher Education: Scientific Knowledge Represented in a Digital Artefact . . . . . . . . . . 229 Wendy Nielsen, Annette Turney, Helen Georgiou, and Pauline Jones

Part V

Using Animation in Assessing Students’ Science Learning

11

Animation in Online School Science Assessment: The Validation of Assessment for Learning and Individual Development Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 Jennifer English

12

Exploring Students’ Scientific Competency Performance on PISA Paper-Based Assessment and Computer-Based Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 Ya-Chun Chen, Zuway-R Hong, and Huann-shyang Lin

13

Towards more Valid Assessment of Learning from Animations . . . 301 Richard Lowe and Jean-Michel Boucheix

Contributors

Jean-Michel Boucheix University of Burgundy, Dijon, France Ya-Chun Chen National Sun Yat-sen University (NSYSU), Kaohsiung, Taiwan Jennifer English New South Wales Department of Education, Parramatta, NSW, Australia Pablo Escobar Pontificia Catholic University of Chile, Santiago, Chile Garry Falloon Macquarie University, Sydney, NSW, Australia Dezheng (William) Feng Hong Kong Polytechnic University, Hong Kong, China Marion Garolera Pontificia Catholic University of Chile, Santiago, Chile Yufei He School of English for International Business, Guangdong University of Foreign Studies, Guangzhou, China Garry Hoban University of Wollongong, Wollongong, NSW, Australia Zuway-R Hong National Sun Yat-sen University (NSYSU), Kaohsiung, Taiwan Huann-shyang Lin Australian Catholic University, Brisbane, QLD, Australia Ric Lowe Curtin University, Western Australia, Perth, Australia Alejandra Meneses Pontificia Catholic University of Chile, Santiago, Chile Mauro Mocerino Curtin University, Western Australia, Perth, Australia Maximiliano Montenegro Pontificia Catholic University of Chile, Santiago, Chile Wendy Nielsen University of Wollongong, Wollongong, NSW, Australia María Paz Ramirez Nüyen Organization, Santiago, Chile Kok-Sing Tang Curtin University, Western Australia, Perth, Australia

vii

viii

Contributors

Roy Tasker Purdue University, West Lafayette, IN, USA David Treagust Curtin University, Western Australia, Perth, Australia Russell Tytler Deakin University, Geelong, VIC, Australia Len Unsworth North Sydney Campus, Australian Catholic University, North Sydney, NSW, Australia Soledad Véliz Pontificia Catholic University of Chile, Santiago, Chile Peta J. White Deakin University, Geelong, VIC, Australia Mihye Won Curtin University, Western Australia, Perth, Australia Zeynep Yaseen University of Technology, Sydney, Ultimo, NSW, Australia Mandy Hoi Man Yu Hong Kong Polytechnic University, Hong Kong, ChinaChina

Part I

Introduction

Chapter 1

A Multidisciplinary Perspective on Animation Design and Use in Science Education Len Unsworth

1.1

Introduction

Animation is now pervasive in the practices of science and science education. Science research centres are employing science animators to explain discoveries at the frontiers of research (Anderson, 2013; Iwasa, 2014; Iwasa, 2015; Martin, 2011); animations are included in online editions of major science journals such as Nature (e.g. Sasihithlu, 2019) and Developmental Cell (e.g. Isabella & Horne-Badovinac, 2016) with more widespread and increased use anticipated as online journals become the norm and reader access to animated content improves (Grossman, Chevalier, & Kazi, 2016); and animation is also increasingly being deployed as a dimension of science research methodology (e.g. Nosch, Foppa, Tóth, & Joos, 2015; Tarshizi, Sturgul, Ibarra, & Taylor, 2015; Villa, Olsen, & Hansen, 2017). In science education, animations have been used as educational resources to support science teaching and learning for several decades (Smetana & Bell, 2012) and research interest in this topic seems to be on a growth trajectory. The number of articles mentioning animation in major science education journals in 2019 was double the number in 2010. In Research in Science Education the increase was from 9 to 24 and in the International Journal of Science Education from 20 to 40. In the construction and communication of scientific understanding, animation has become a major resource within the meaning-making mode of the moving image (kineikonic mode (Burn, 2013)), alongside and complementary to the established meaning-making modes of language and image. Tytler, Prain, and Hubber (2018) draw on research by Latour (1986) and Gooding (2006) to argue that the relation

L. Unsworth (*) North Sydney Campus, Australian Catholic University, North Sydney, NSW, Australia e-mail: [email protected] © The Author(s) 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_1

3

4

L. Unsworth

between theoretical scientific claims made in papers and their raw research data is not unitary but distributed across a sequence of representational re-description pathways. Hence, generating, promulgating and interpreting scientific knowledge and comprehending and deploying the specialized representational forms of the discourse of science are completely interconnected. Tytler et al. (2018) support the view that students need to be able to interpret and generate both the generic and the discipline-specific verbal and visual representational conventions of the scientific community if they are to become scientifically literate (Bazerman, 2009; Klein & Boscolo, 2016; Unsworth, 2001). Hence learning science and learning the specialized representational forms of the discourse of science are also completely interconnected. The use of animation in science education needs to be seen in this light. Approaches to the induction of students into the multimodal disciplinary literacy practices of science have differed most noticeably in the extent to which they incorporate the development of students’ explicit and systematic knowledge of the semiotics or meaning-making resources of language and image and the use as a pedagogic tool of a meta-language describing these resources. Arguably the most educationally influential delineation of the semiotics of the discourse of science has emerged from systemic functional linguistics (SFL) through research on language led by Halliday (Halliday & Martin, 1993), Martin (Martin, 2017; Martin & Veel, 1998) and Lemke (1990, 2004) and related systemic functional semiotic (SFS) research focusing more on images led by Kress (Kress, Jewitt, Ogborn, & Tsatsarelis, 2001; Kress & Ogborn, 1998; Kress, Ogborn, & Martins, 1998; Kress & van Leeuwen, 2006), Bateman (2008), O’Halloran (2003) and more recently Doran (2019). Such accounts of the multimodal discourse of science have clearly informed approaches to integrating disciplinary literacy development in science pedagogy such as the Representation Construction Approach (RCA) (Hubber & Tytler, 2017; Hubber, Tytler, & Haslam, 2010; Prain & Tytler, 2012; Tytler et al., 2018; Tytler & Hubber, 2010; Tytler, Prain, Hubber, & Waldrip, 2013) and Primary Connections (Aubusson et al., 2019; Hackling & Prain, 2008), but without their taking up the explicit teaching of knowledge about language and image and the associated metalanguage. On the other hand, disciplinary literacy pedagogies closely associated with SFL and SFS have emphasized the role of knowledge about language and image and have incorporated the explicit development of students’ knowledge of linguistic and visual semiotic metalanguage as a key resource in disciplinary literacy learning and teaching (Dreyfus, Humphrey, Mahboob, & Martin, 2015; Fang & Schleppegrell, 2010; O’Hallaron, Palincsar, & Schleppegrell, 2015; Polias, 2015; Rose & Martin, 2012; Schleppegrell, 2011, 2013). However, while a prodigious amount of research has accumulated on the use of animation in science education, little attention seems to have been given to the semiotics of animation either as a resource for informing pedagogy or as explicitly taught facilitative knowledge for enhancing animation interpretation and creation. Notwithstanding its expanding use in science education, animation has not yet been fully acknowledged as a significant dimension of the multimodal disciplinary literacy of science, which all students need to be able to both interpret and produce. Recent

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

5

publications on global developments in literacy research for science education do not mention animation (Tang & Danielsson, 2018), and from 2015 to 2019, animation appeared in the title of only three papers published in the International Journal of Science Education and did not appear in the titles of any papers published in Research in Science Education or Research in Science Teaching. It is now widely accepted among science education researchers that learning science entails students being inducted into the practices of scientific knowledge generation, validation and communication and that the distinctive representational forms of science discourse are integral to these practices (Tytler et al., 2018). Animation now functions as a significant representational mode embedded in the epistemic processes of science enquiry. For effective student induction into this functioning of animation to occur, multidisciplinary perspectives on optimizing the use of animation in science education need to be brought to bear. From a semiotic perspective it is important to specify how the distinctive meaning-making resources of animation may be deployed in the construal of the different kinds of meanings involved in the scientific explication of phenomena. From the perspective of developing students’ multimodal disciplinary literacy, we need to determine how students can develop competence and confidence in the critical interpretation and strategic creation of the specialized animated representations of discipline knowledge. From the viewpoint of managing day-to-day science pedagogy in the classroom, there are many issues including determining the suitability of animations and approaches for their use with students at different grade levels; the articulation of animation with other representational modes; incorporating animation into established pedagogic practices, and the practicalities of determining the means to enable students to create animations that are functional to their scientific inquiry and to the demonstration of their scientific knowledge. A further crucial perspective is the assessment of students’ science learning, which is increasingly involving animation. The chapters in this volume highlight examples of innovative developments in research from these various perspectives. While several chapters align with the theoretical approaches of SFL and SFS, others have a more cognitive orientation. This reflects the current discrete complementarity of different research traditions, whose common interests being brought together here may encourage greater transdisciplinary intersection in future studies in this field. The chapters are organized into the four main parts of the book, each of which is previewed and briefly discussed in the following sections of this chapter.

1.2

An Educational Semiotics for Science Animation

The shift to viewing the development of students’ literacy in science as essential to their induction into the epistemic processes of the discipline has coincided with greater attention to the multimodal and increasingly digital nature of the disciplinary literacy of science and with the burgeoning of social semiotic explications of the distinctive deployment of the resources of language and static images in science

6

L. Unsworth

discourse. The very limited attention to science animation from a semiotic perspective is advanced in the innovative research reported in Chaps. 2, 3 and 4 in this volume, all of which derive their distinctive contributions from the theoretical foundation of SFL and SFS. In this section the generative potential of these approaches for explicating how the affordances of animation construct and communicate scientific knowledge will be proposed. Firstly, we will briefly note the variety of scientific representational modes which have been described using SFS and outline the key features of the theoretical framework that produce this versatility. Then the new developments in research in science animation based on SFS in Chaps. 2, 3 and 4 will be introduced. The range of representational modes in science for which SFS approaches have detailed the nature of their meaning making resources range from language to static images, to graphs, symbols and gestures (Doran, 2019; Halliday & Martin, 1993; Kress et al., 1998; Kress & van Leeuwen, 2006; Lemke, 1990; Martin & Veel, 1998; O’Halloran, 2003). Further work has importantly extended to the intermodal orchestration of these in paper and digital media texts and in teachers’ shaping of scientific knowledge in the classroom (Danielsson, 2016; Kress et al., 2001; Lemke, 1998; Tang, Delgado, & Moje, 2014). This versatility of SFL and SFS approaches derives from the fundamental bases of the theory. The genesis of SFS theory was in Halliday’s SFL account of language as social semiotic (Halliday, 1978; Halliday & Matthiessen, 2004). SFL posits the complete ‘interconnectedness’ of the linguistic and the social. According to SFL, the structures of language have evolved (and continue to evolve) as a result of the meaning-making functions they serve within the social system or culture in which they are used. But an important aspect of the impressively seminal nature of Halliday’s work lies in his very early emphasizing that language is only one semiotic system among many, which might include forms of art such as painting, sculpture, music and dance and other modes of cultural behaviour such as modes of dress, structures of the family and so forth. All of these modes of meaning-making interrelate and their totality might be thought of as a way of defining a culture (Halliday & Hasan, 1985, p.4). This conceptualization of language as one of many different interrelated semiotic systems, and hence the assumption that the forms of all semiotic systems are related to the meaning-making functions they serve within social contexts, indicates the capacity of the theory underpinning SFL to be extended more broadly to SFS, applying to all semiotic modes and to the development of inter-semiotic theory. According to SFS any communicative context can be described in terms of three main variables that are important in influencing the semiotic choices that are made. FIELD is concerned with the social activity, its content or topic; TENOR is the nature of the relationships among the people involved in the communication; and MODE is concerned with the channel (graphic/oral) and medium (the role of language in the situation - as constitutive of or ancillary to the activity) of communication, and the ways in which relative information value is conveyed. These three situational variables that operate in all communicative contexts: FIELD, TENOR and MODE are related to the corresponding three main categories of meaning-making functions, or metafunctions, of all semiotic systems – the ideational, interpersonal and textual

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

7

metafunctions. Ideational meanings construe FIELD i.e. entities, events or relationships and their properties; interpersonal meanings construe TENOR, enacting attitudinal stance and participation roles in social relationships; and the textual metafunction construes MODE, organizing meaning elements in the text as a whole and their relative emphases. While these three kinds of meaning are always made simultaneously in all instances of all semiotic modes such as language and image, the means by which they are realized differs. For example, in language, ideationally, the meaning of a particular activity of a specified field is typically realized by an action verb such as ‘explode’ (or its nominalized form ‘explosion’), whereas in an image such a meaning would be conveyed by what Kress and van Leeuwen (2006) refer to as ‘vectors’ (action lines); interpersonally in language a direct interchange with an interactant is realized by choices in the mood system of statement, command or question and by the second person pronoun ‘you’, while in images direct engagement is realized by what Kress and van Leeuwen (2006) refer to a ‘demand’ image where the gaze of the participant in the image is directed straight at the viewer; textually in language information that is given or assumed is normally located at the beginning of the clause and what is new information is located at the end of the clause, whereas in images the kinds of emphases are realized by positioning at the left and right of some images with a grid layout and at the centre or periphery of images organized with what Kress and van Leeuwen (2006) refer to centre-margin layout. Of course, SFL and SFS accounts detail very comprehensively and systematically the extensive meaning potential within each of the ideational, interpersonal and textual metafunctions and also the various options for realizing these meanings in all modes such as language, image and gesture etc. The systems of options for meaning making (within the stratum of semantics) is mapped against the systems of options (available within the strata of grammar and phonology/graphology of language and the visual grammar and graphological options of images) for the realization of these meanings. This kind of mapping means that it is possible to specify how the various modes make particular meanings. However, as Tang (Chap. 4, this volume) points out the nature of this inter-stratal mapping is influenced by the material nature of the medium through which different modes realize meanings. For instance, the temporal characteristics of human sound both afford and constraint speech as a mode to make meanings in a sequential manner. On the other hand, the spatial characteristics of visual representations allow meanings to be made and interpreted simultaneously instead of in sequence, thus allowing a unique way of meaning making compared to speech (Chap. 4, this volume).

This kind of systemic theorizing and analysis facilitates understanding and discussion of the commonalities and complementarities of the meaning-making that is possible in different modes as well as how such modes may function collaboratively to construct meanings in multimodal texts. Hence it can also inform pedagogic agendas for the development of students’ multimodal disciplinary literacy. As yet however, there has been very little such theorizing and analyses applied to science animations. The contributing authors to Part I of this volume provide pioneering work in this area.

8

L. Unsworth

The ideational, interpersonal and textual analyses of science animations from TED-ed animations, study.com and the online portal of Explain Everything provided by Yufei He in Chap. 2, include new frameworks for the analysis of ideational and textual meaning in animation. She notes that from an ideational perspective a key affordance of animation is the dynamic portrayal of change and innovatively extends previous classifications (Lowe, 2003; Ploetzner & Lowe, 2012, 2017) to present a very detailed network of options for different kinds of change that can be manifested in animations, which she developed with Theo van Leeuwen (He & van Leeuwen, 2019). In this network (Fig. 1.1) there are four types of changes: inclusion, movement, illumination and transformation. Movement can be displacing (e.g. the earth orbits around the sun) or non-displacing (e.g. the earth rotates around its own axis, i.e. without changing position). Displacing or non-displacing movements can also be either rotational or translational (linear) or a combination of those two. Illumination includes change of colour or change of brightness. Transformation involves an element changing its size (rescale) or changing its shape (reshape). Each of these four types of change can be instantaneous or gradual. If gradual, the speed of change can be constant or variable. If the speed is variable, it may be increasing or decreasing and if constant the speed could be located on a continuum from fast to slow. He also provides a network of the verbal, pictorial or abstract elements (geometric forms, mathematical symbols, cues – such as locational arrows or frames) that can occur either separately or in combinations in animations. And any kind of element can undergo any kind of change. Such a detailed framework of meaning-making options enables very precise description of the nature of change depicted in animations, facilitating critical discussion of how apposite the depiction choices in particular animations are in relation to the concepts being represented. In relation to textual meaning He draws on previous work (Ploetzner & Lowe, 2012, 2017) to represent the organization of information flow in animations as a network of options (Fig. 1.2). In He’s terms organization flow is either accumulating or predicting. If accumulating the information flow may be either concurrent or retrospective. When the flow is accumulating and concurrent, the changed elements in animations construing activity are presented on the same screen (for example, using a split screen to show the molecular change as a substance changes from solid to liquid). When the accumulation is retrospective, the changed elements are presented on different screens (for example, the molecular view of a substance changed into the gaseous state would be represented on a subsequent screen to that representing the substance as a solid or a liquid). If, on the other hand the information flow is predictive, information is not aggregated over successive screens but rather, the aggregated information is presented as a whole and the sequence of changes is highlighted by cues such a colour highlighting or superimposed frames or arrows. The cognitive processing implications of such different forms of information flow have been discussed by Lowe and his colleagues (Lowe & Ploetzner, 2017). With respect to interpersonal meaning, He also discusses semiotic resources used in animations to construct viewer engagement and attitudinal meaning. In her corpus these are principally anthropomorphic devices as well as representing humans

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

Fig. 1.1 System for change (adapted from He & van Leeuwen, 2019) Fig. 1.2 The information flow system

9

10

L. Unsworth

depicted minimally and distortedly to humorous effect. The analysis framework demonstrated by He, encompassing the three dimensions of meaning proposed in SFS (ideational, interpersonal and textual) indicates the potential of this semiotic approach to productively complement cognitive orientated approaches and provide a highly comprehensive and integrated theoretical basis for the investigation, critical analysis and further development of the design features of animation, which currently receive so little attention in educational research. Chapter 3 (Yu, Feng and Unsworth) focuses on the interpersonal metafunction, and particularly on the communication of attitudinal stance in a corpus of 67 ecology animations for children collected from Youtube. The analysis involved three phases. The first is based on the SFL appraisal framework (Martin & White, 2005). This framework includes three interacting dimensions: attitude, engagement, and graduation. Attitude encompasses: i) affect (un/happiness; dis/satisfaction; in/security), ii) judgments of individual’s characteristics and capacities and the veracity and ethics of their behavior, and iii) appreciation of the significance and aesthetics of natural and artificial phenomena. Engagement addresses the sources of the evaluation expressed, the alignment of the author with these, and the extent to which the evaluations are portrayed as negotiable. Graduation deals with the amplification of meaning along a gradient of intensity of affect, judgment or appreciation (e.g., satisfied, indulged, satiated). The initial phase of the analysis identified all instances of attitude in the corpus and classified them according to the sub-categories of affect, judgment and appreciation. The second phase was based on a framework generated by the authors for the multimodal realization of attitude in animations (Fig. 1.3). The framework distinguishes attitude that is articulated through spoken or written language and attitude that is communicated through being embedded in the design of the animation characters or the animation’s narrative structure. Attitudes can be explicitly articulated literally or by metaphorical language. They can also be articulated implicitly by describing an eliciting condition such as waste plastic causing the death of sea life or by describing remedying action such as banning the use of plastic straws in favour of paper straws. Attitude embedded in character design can be realized by narrative processes in the images (what the characters do as well as their facial expressions and gestures) and by conceptual processes of attribution and

Fig. 1.3 The multimodal realisation of attitude

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

11

possession such as their clothing, colour and attributes such as symbols (e.g. green colour for environmental champions and black for agents of pollution). Attitude realized through narrative structure can involve opposing environmental hero characters with their anti-environmental villain counterparts or by showing gradual change in characters’ behaviours as a result of experience of the negative impact of anti-environmental practices. In the second phase of analysis, all instances of the various kinds of attitude in the corpus were cross classified with the means of their realization. These patterns of attitudinal meaning and their realization in the animations were then related to three different categories of pro-environmental value positions identified in the literature: egoistic (pro-environmental behaviours that benefit the individual such as saving on power costs); altruistic (environmental protection to benefit current and future generations of humans); and biospheric (environmental protection to benefit all forms of life on earth). While biospheric values were found to predominate, findings concerning the means of realizing the attitudinal meanings that construct this value position indicate that they are mainly realised by eliciting conditions and resultant remedying behaviour (often co-occurring) which mainly function to provide scientific explanations and recommend ways to solve the environmental problem concerned respectively. The chapter indicates the pedagogical value of the majority of online animations in the corpus in view of their balanced representation of knowledge and action-oriented aspects of environmental education. It also shows how this kind of semiotic analysis can contribute to an evidential basis for critical comparison of available animations of this kind. In Chap. 4, Kok-Sing Tang and his colleagues use a semiotic approach to examine the multimodal affordances of a new form of animation made possible by virtual reality (VR) technology and discuss how these affordances facilitate student learning of molecular interactions. Immersive VR uses a head mounted display that enables the wearer to see a three-dimensional animation as if s/he were immersed within the three-dimensional space of the animation. Chemistry undergraduates were recruited to work in pairs to complete three activities using the immersive VR technology: (1) building the molecular structure of acetylcholine by moving several objects (representing pre-assembled functional groups of atoms) until they formed a bond when they were placed in close proximity to one another; (2) watching a scripted animated sequence that showed how acetylcholine is broken into two smaller molecules – acetate and choline; (3) exploring the structure of acetylcholinesterase to identify a particular gorge that leads into the active site where the reaction of acetylcholine takes place. The students could explore by walking around the enzyme, rotating it, and changing its size to the point where the students could zoom inside the enzyme. They could also change views according to different models of the enzyme, namely surface, mesh, ball-and-stick, cartoon and ribbon models. The chapter identifies five affordances of immersive VR 3D animation that are compared with flat screen 3D animation and the use of physical models: (1) viewing; (2) sequencing; (3) modelling; (4) scaling; (5) manipulating. Viewing the animation in immersive VR is not constrained by screen size or perspective – the animation can be viewed from any angle, as is the case for physical models. Sequencing is the

12

L. Unsworth

concatenation of images so that they dynamically represent a sequence. This affordance is common to immersive VR and flat screen 3D but the combination of sequencing with the Immersive VR affordance of viewing provided a new means of students’ perceiving the represented activity. Modelling refers to the fact that the VR application was programmed to enable the display of different molecular models of acetylcholine and acetylcholinesterase, such as surface, mesh, ball-and-stick, cartoon and ribbon models. It is possible to switch from one model to another dynamically maintaining spatial correspondence of the different models. Scaling is the capacity to zoom in or zoom out, which is also possible with flat screen 3D but with the viewing affordance of immersive VR 3D and its high-level scaling, it is possible for the viewer to enter inside an object to view its constituents. Manipulating by moving or rotating animation elements is also possible with flat screen 3D but it is the combination with the viewing affordance of immersive VR 3D that enables the viewer/participant to engage in tactile manipulation while maintaining the threedimensional spatial orientation of the visual object relative to his/her point of view. Immersive VR 3D animation combines the advantages of flat screen 3D and physical models. Its great advantage is that the viewing affordance can combine with each of the other affordances. The chapter clearly shows how this facilitated student learning in relation to the three learning activities they undertook. From the perspective of SFS, the viewing affordance concerns interpersonal meaning (what Kress and van Leeuwen call interactive meaning in reading images) – the nature of the relationship between the viewer and the represented participants in the animation, which changes according to the viewing angle and social distance between the viewer and the animation participants. Sequencing or the representation of activity refers to ideational meanings. It is textual meaning that is involved in the affordances of modelling and scaling. These deal with the organization of the represented information. Modelling uniquely has the capacity to organize the presentation of different models dynamically while maintaining spatial correspondence and scaling organizes the size of representations relative to the viewer. Manipulating appears to involve ideational and textual meaning as it seems to entail sequencing in enabling different positional arrangements of elements in the animation. In SFS terms then, immersive VR augments the semiotic resources of animation through enhanced affordances related to interpersonal and textual meaning.

1.3

Learning from Viewing Science Animations

The chapters in Part III of this volume highlight student learning from animations in the beginning and later years of primary school and in senior high school. As well as contributing to an understanding of the nature of the impact of active viewing of animations on student learning, the first two chapters draw attention to the importance of further research attention to the semiotic design of science animation. The third chapter emphasizes student perception of the value of learning from expert

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

13

animations relative to their own exploratory animation creation as part of a guided enquiry approach in their senior high school chemistry class. In Chap. 5 Garry Falloon draws attention to the limited research on learning from science animation by children in the early years of school. He conducted two studies of five-year-old students learning about electric circuits from animated simulations in an elementary school in New Zealand. The research investigated the strategies and cognitive processes used by the children to build procedural knowledge (know how) and conceptual knowledge (know why) and the transfer of learning from their initial learning experiences with the simulations to application with different simulations as well as their transfer of leaning to undertaking practical tasks in building electric circuits. The results indicated that procedural knowledge was effectively learned from the simulations and transferred when using different simulations, but the transfer did not occur to the same extent in the students’ application to using practical materials to build electric circuits. There was limited development and transfer of the students’ conceptual knowledge. As well as emphasising the need for further research into the transfer of conceptual learning of young children from simulations to practical applications, the chapter also briefly notes the issue of animation design and the potential of some visual representations of concepts leading to misconceptions in students scientific understanding. The recommendations for critical review by teachers, increased trialling by app developers and improved app design, support the contributing role of semiotic analyses in complementing more cognitively oriented studies of science animation. Chapter 6 deals with the impact of specially constructed animated versions of a multimodal science text on the learning of fifth grade low-skilled readers. The multidisciplinary team of Chilean researchers led by Max Montenegro first constructed a static multimodal text explaining energy transfer in an eco-system. They then produced two different animated versions of the text. One version scaffolded scientific concepts through animations that visually elaborated complex processes that were succinctly presented in condensed form in the verbal text. At the beginning of every page, the aural text was presented simultaneously with the animated images, but without the written text. Students were able to access the written text with a mouse click. In the second version of the animation the emphasis was on the scaffolding of academic language. In this version initially the written and aural text were presented simultaneously, and students were then able to click on unfamiliar vocabulary and logical conjunctions to access assistance in the form of simplified explanations or animations designed to support understanding of the verbal representation of the scientific processes. Three groups from a total of 84 students each experienced one of the three forms of the text: (1) non-animated; (2) scientific concepts scaffolded; (3) academic language scaffolded. Measures of scientific understanding, reading comprehension and vocabulary knowledge derived from the target text showed that both of the groups that experienced the different versions of the animated texts performed better on all three measures than the group that experienced the non-animated text, but there were no statistically significant differences in the results of the two groups who used the animated versions. It seems that the inclusion of aural text with the animations can boost the learning

14

L. Unsworth

performance of low-skilled readers with limited background knowledge of topics involving complex processes such and the transfer of energy in eco-systems. In Chap. 7 Zeynep Yaseen discusses an intervention in senior high school chemistry in which student viewing and creation of animations of sub-microscopic representations of states of matter from solid to liquid to gas were incorporated into the well-known representation construction approach (RCA) to science pedagogy (Hubber & Tytler, 2017; Prain & Tytler, 2012; Tytler et al., 2013; Tytler et al., 2018). One class of year 11 students in Turkey working in pairs or groups of three used the K-sketch software program (http://www.k-sketch.org/) to create an animation of what they imagined could be seen with an impossibly powerful microscope looking into matter in the different states. The group-created animations were then shared and critiqued in a teacher guided class discussion as students reviewed and considered revision of their represented conceptualizations. Subsequently the students viewed animations created by experts using K-Stretch and other software and the students again reviewed and revised their conceptualizations. After the intervention the students and the teacher were interviewed about their opinions of the teaching/learning process using the animations. While post intervention assessment indicated substantial learning had occurred, the interviews indicated that very limited learning occurred during the students’ animation creation and subsequent classroom discussion and critique. The animation construction did make the students’ misconceptions explicit, but they perceived viewing the expert animations as most efficacious in developing their understanding. The students felt this viewing of expert animations was indispensable to their learning, although creating their own animations may have alerted them to attend closely to the conceptual representations in the expert versions. The study suggests that the pedagogic interface between students’ representation construction using animation and their guided analysis and critical interpretation of expert or canonical animations warrants further investigation. As animation creation was a novel experience for these students, an additional implication of the study may be consideration of the extent to which students’ familiarity and confidence in their knowledge and deployment of the options for constructing meaning with the resources of animation software may influence their capacity for both creation and critique of conceptual representation in science animations.

1.4

Learning through Creating Science Animations

The three chapters in Part III all involve applications of the ‘slowmation’ approach to student animation creation introduced in Chap. 8. The three chapters also illustrate the transdisciplinary approach of the authors, who are all science education academics, in drawing on key semiotic principles included in the outline of SFS in Sect. 1.2. While there has been extensive uptake of the ‘slowmation’ approach, the increasingly accessible general animation software (such as K-Stretch, see Chap. 7), and the increasing recognition of animation in the conduct and

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

15

communication of scientific research (see Sect. 1.1) indicates the need for further research into the pedagogic use of science animation creation from the early years of schooling through to tertiary science education. In Chap. 8 Garry Hoban draws attention to the essentially transdisciplinary nature of student generated digital media representations in science education that integrate skills in science, media, semiotics, computing and coding. In response to the need to improve the engagement and discipline knowledge of teacher education students in science and taking account of the critique of science education animations as providing information too quickly for effective learning from them, Hoban devised ‘slowmation’ (slow animation) - an approach to enabling students to generate stopmotion science animations paced slowly enough for them to explicate their detailed understanding of science concepts for novice audiences and without the need for sophisticated subject-specific software. This involves students making three-minute narrated stop-motion animations by using everyday materials, such as plasticine, cardboard, or paper, or existing plastic models that are digitally photographed as they are moved manually along with enhancements such as music or static images. Slowmation greatly simplifies the process of creating an animation enabling preservice teachers to: (i) make or use existing 2D or 3D models that may lie flat on a table or the floor; (ii) play the animation slowly at 2 frames per second requiring 10 times fewer photos than normal animation; and (iii) use widely available technology such as a digital still camera, a tripod, and free movie-making computer software. The take up of slowmation in science teaching and research studies on its use are now very extensive indeed. Further development of the approach to studentgenerated digital media representations involves blended media whereby students create a ‘media collage’ using media products such as slowmation, still images including downloads from Google Images, personally made video and/or video downloaded from Youtube, enabling students to mix and match media for particular purposes (Hoban, Nielsen & Shepherd 2016). Hoban emphasizes the importance of students being aware of the affordances of each digital mode in selecting the most appropriate to suit the purpose of the explanation, underlining the fundamental transdisciplinary competencies required for student-generated multimodal digital representation in science. In Chap. 9 Peta White, Russell Tytler and Wendy Nielsen describe a study in which 17 year 11 biology students worked in pairs or groups of three to produce a slowmation explanation for their peers of a topic in the senior high school biology curriculum. In a two-hour period, they were provided with iPads loaded with the app ‘Stop Motion’, a range of construction materials including papers of several weights and colours, scissors, pipe cleaners, paddle pop sticks, toothpicks, modelling clay, thumb tacks, glue, sticky tape, pins, coloured markers, and a collection of biology textbooks and other reference materials. The study investigated the processes of collaborative reasoning that occurred for one group of two students during the different phases of the construction of the slowmation animation and the learning opportunities that were opened up by the translation from text and static image to slowmation production. Through their re-representation of the intricacies of the digestive process (chemical and physical digestion with enzyme and organ detail)

16

L. Unsworth

the students needed to confront the fact that different representational modes support different, and necessarily partial, understandings of phenomena. Such constraints are shown to be productive of enhanced student understanding of phenomena as they engage in relating the different modes. The students’ reasoning about biological phenomena initiated through this cross-modal translation was revealed through analyses of video and audio records of the two-hour period during which the group worked on creating the slowmation, storyboard sketches used as planning tools, the final slowmation product, and artefacts generated during construction. The analyses illustrate how creation of the visual-temporal rendition in the animation offers a productive constraint on the students’ visualisation of the part-whole relations and temporal sequencing of the digestive processes. In cross-modal translation students are obliged to attend to the semiotic means by which biological processes and part-whole relations are represented in language (i.e through nominalisations such as ‘secretion’ or ‘mucus failure’ and classifer/noun structures such as ‘abdominal cavity’ or ‘parietal enzymes’), and they also need to be aware of the visual semiotic means by which such processes and relations are represented in both static and dynamic images. This raises the issue of whether the pedagogic effectiveness and potential of cross-mode translation may be impacted by the extent of student familiarity with these semiotic resources and in addition, whether a metalanguage for the description of such resources being shared by teachers and students might further facilitate the productive reasoning among students and in teacher-student discussion of cross-mode translation. In Chap. 10 Wendy Nielsen and her colleagues adopted the blended media approach discussed in Chap. 8 to generate a digital media creation task for pre-service primary teacher education students as part of their assessment in a course on primary science curriculum and pedagogy. The digital explanation creation task required students to explain how a car demister works using multimodal resources in a manner accessible to a primary school student in year six. The task was intended to assess the students’ knowledge of the subject matter, their knowledge and use of multimodal resources and the appropriateness of their artefacts for students in the final year of primary school. The student artefacts were assessed using a rubric based on the SFS contextual variables of field (representing science); tenor (attention to audience); and mode (media choices). While almost all students had very limited prior knowledge about how electric circuits functioned in car demisters, the analysis of the student artefacts (and interview analyses) indicated a wide range of demonstrated knowledge in the digital explanations as illustrated in a detailed analyses of two explanations illustrating the highest and lowest levels of knowledge. This range of field knowledge broadly correlated with the appropriateness and of their choices of media and the suitability of the digital explanations for the intended audience. An implication of the work presented in this chapter is the potential of student-generated digital media explanations as an engaging pedagogic device to integrate the development of teacher education students’ disciplinary knowledge and digital multimodal discourse of science as part of their preparation of pedagogic resources appropriate for their future primary school pupils.

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

1.5

17

Using Animation in Assessing Students’ Science Learning

The incorporation of animation into large scale government conducted tests of students’ science learning as well as in very widely administered international tests has drawn attention to practical challenges in the production and implementation of such tests as well as issues of concurrent validity related to students’ experience of this mode of assessment and construct validity related to the design of the animations and the assessment items based on them. The chapters in Part IV of this volume include discussion of such challenges and specific proposals for addressing them. In Chap. 11 Jennifer English describes the use of animation in the online science assessments that have been undertaken by all students in government schools in years six, eight and ten in the State of New South Wales (NSW) in Australia since 2014. This science assessment program is known as the Validation of Assessment for Learning and Individual Development (VALID) program. The online assessments include static and animated stimulus material and the assessment items include those requiring extended written responses as well as well as multiple choice, matching and drag and drop responses. The animations are of two main types: (1) those that are display and play similar to a video; and (2) interactive animations that allow students to select variables which produce variance in the array of graphic entities and/or activities. Interactive animations are primarily used to assess procedural knowledge concerning the nature and conduct of science investigations. Descriptions and illustrations from the different types of animations and assessment items in VALID are included along with explanations of the procedures employed by the NSW Government Department of Education personnel to establish the validity and reliability of each assessment item. As well as outlining the theoretical bases of the development of the assessments, the chapter also points out a range of practical considerations that need to be taken into account in the use of animation in this Statewide online science assessment. These include the costs of bespoke animations created specifically for the assessment program and of copyright costs if existing published animations are used. Open access online animations need to be checked for scientific accuracy and regardless of the source, the file size of animations needs to be carefully monitored as the online assessment needs to be accessible throughout the large State of NSW and in some regional, rural and remote areas where internet connections may be unreliable and admit only limited bandwidth. A further challenge is the sourcing of animations on curriculum specific topics that are suitable for students at the three different grade levels. Nevertheless, the successful development and implementation of this government funded assessment program since 2014, has substantially advanced understanding of issues involved in the use of animation for assessment and affords great opportunities for further such research and development. In Chap. 12 Ya-Chun Chen, Zuway-R Hong, and Huann-shyang Lin describe a study that compared the performance of students who completed science assessment items in the Program for International Student Assessment (PISA) 2015 Field Trial

18

L. Unsworth

test in Taiwan in three different formats: (1) static stimuli and paper-based assessment (PBA); (2) static stimuli and computer-based assessment (CBA) and stimuli including animation and simulation with computer-based assessment (ACBA). On the overall performance PBA did better than CBA, which did better than ACBA. Detailed further analyses considering format type in relation to high, mid and low-level achievers showed the same pattern for high achievers but no format effect for mid and low-level achievers. Relatively lower performance by the ACBA group was sustained when separate analyses addressed explaining science concepts; designing and evaluating science investigations; and interpreting data and evidence. The format effect seems quite clear, although it cannot be absolutely confirmed since there was no pre-test of science competency. The researchers collected data on the use and disposition of participants in relation to Information and Communication Technology (ICT). They found a weak but significant effect on student performance for participants’ ICT interest and perceived competence and autonomy. The relatively low performance by the ACBA group may have been due to experience in responding to animations as stimuli in science assessments, but it may also be related to the design of the animations and a mismatch between animation mode of the stimuli and the verbal response mode of the assessment items, albeit in a computerbased format. In Chap. 13 Ric Lowe and Jean-Michel Boucheix take up the matters of animation design and response modes to address issues of validity in assessment of science learning using animation. They note the advantage of animation in being able to provide a veridical presentation of spatiotemporal relations (i.e. processes and procedures), particularly in representing complex and often simultaneous processes, such as those involved in the operation of electric motors. Animation can faithfully represent such dynamic content with a high degree of realism despite the component entities being represented in an abstracted manner. Because of the inherent linear sequential nature of language being able to devise an adequate verbal characterization of the dynamics of these kinds of complex and interdependent sets of processes is extremely challenging exercise, even for a domain expert. Hence the validity issue is that when dynamic processes of this kind are represented using animation, the assessment of their comprehension by students is almost invariably through some form of verbal response perhaps in conjunction with static image representation(s). The challenge then, is to devise student response alternatives that avoid the need to re-frame understandings into the linear-sequential format required for verbally based assessment approaches. Several experiments seeking solutions are described. While those involving the creation, completion or interpretive annotation of static images partially address the issue, it is the presentation of animated response alternatives from which students select the correct version or choose from true/false options that seem to most closely align the representation modes of the stimulus and the assessment item response. While the authors do not claim that their experimental materials could be readily directly applied in the assessment of science learning, they describe a range of novel non-verbal measurement tools that could provide a basis for more valid ways of assessing learning from animation. This poses a very realistic

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

19

applied research and development agenda for educational assessment authorities and science animation developers.

1.6

Concluding Remarks

The studies featured in these chapters emphasize the growing importance of animation in many contexts of science education reflecting the increasing significance of animation in scientific investigation and communication. They highlight the inherent transdisciplinary nature of science animation and point to the value of bringing together cognitive, social semiotic, educational technology and pedagogic perspectives in addressing educational challenges of inducting students into this vital dimension of the disciplinary discourse of science and science education. As well as contributing to an improved understanding of current approaches to the use of animation in science education, the studies herein attest to the growing international crossing of disciplinary borders in science education research, which it is hoped will be a catalyst to further transdisciplinary initiatives enhancing learning in science through the pedagogic infusion of the evolving development of animation and associated forms of digital multimodal representation in scientific discourse.

References Anderson, Q. (2013). Column: Animated science. Nature, 502(7470), 259–260. https://doi.org/10. 1038/nj7470-259a Aubusson, P., Skamp, K., Burke, P., Pressick-Kilborn, K., Ng, W., Palmer, T., et al. (2019). Primary connections: Linking science with literacy stage 6 research evaluation final report. Sydney, Australia: University of Technology Sydney. Bateman, J. (2008). Multimodality and genre: A foundation for analysis. London: Palgrave Macmillan. Bazerman, C. (2009). Genre and cognitive development: Beyond writing to learn. In C. Bazerman, A. Bonini, & D. Figueiredo (Eds.), Genre in a changing world. Perspectives on writing (pp. 279–294). Fort Collins, CO: The WAC Clearninghouse. Burn, A. (2013). The Kineikonic mode: Towards a multimodal approach to moving image media. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis. London: Routledge. Danielsson, K. (2016). Modes and meaning in the classroom–the role of different semiotic resources to convey meaning in science classrooms. Linguistics and Education, 35, 88–99. Doran, Y. (2019). Building knowledge through images in physics. Visual Communication, 18(2), 251–277. Dreyfus, S. J., Humphrey, S., Mahboob, A., & Martin, J. R. (2015). Genre pedagogy in higher education: The SLATE project. New York: Palgrave Macmillan. Fang, Z., & Schleppegrell, M. (2010). Disciplinary literacies across content areas: Supporting secondary reading through functional language analysis. Journal of Adolescent and Adult Literacy, 53(7), 587–597. Gooding, D. (2006). From phenomenology to field theory: Faraday’s visual reasoning. Perspectives on Science, 14(1), 40–65.

20

L. Unsworth

Grossman, T., Chevalier, F., & Kazi, R. H. (2016). Bringing research articles to life with animated figures. Interactions, 23(4), 52–57. Hackling, M. W., & Prain, V. (2008). Impact of primary connections on students’ science processes, literacies of science and attitudes towards science. Canberra, Australia: Australian Academy of Science. Halliday, M. A. K. (1978). Language as a social semiotic: The social interpretation of language and meaning. London: Edward Arnold. Halliday, M. A. K., & Hasan, R. (1985). Language, context and text: Aspects of language in a social-semiotic perspective. Geelong, Australia: Deakin University Press. Halliday, M. A. K., & Martin, J. R. (Eds.). (1993). Writing science: Literacy and discursive power. London: Falmer Press. Halliday, M. A. K., & Matthiessen, C. (2004). An introduction to functional grammar (4th ed.). London: Arnold. He, Y., & van Leeuwen, T. (2019). Animation and the remediation of school physics–a social semiotic approach. Social Semiotics, 1–20. Hoban, G., Nielsen, W., & Shepherd, A. (Eds.). (2016). Student-generated digital media in science education: Leaning, explaining and communicating science. London, England: Routledge. Hubber, P., & Tytler, R. (2017). Enacting a representation construction approach to teaching and learning astronomy. In D. Treagust, R. Duit, & H. Fischer (Eds.), Multiple representations in physics education (pp. 139–161). Cham, Switzerland: Springer. Hubber, P., Tytler, R., & Haslam, F. (2010). Teaching and learning about force with a representational focus: Pedagogy and teacher change. Research in Science Education, 40(1), 5–28. Isabella, A. J., & Horne-Badovinac, S. (2016). Rab10-mediated secretion synergizes with tissue movement to build a polarized basement membrane architecture for organ morphogenesis. Developmental Cell, 38(1), 47–60. Iwasa, J. (2014). Crafting a career in molecular animation. Molecular Biology of the Cell, 25(19), 2891–2893. Iwasa, J. (2015). Bringing macromolecular machinery to life using 3D animation. Current Opinion in Structural Biology, 31, 84–88. Klein, P. D., & Boscolo, P. (2016). Trends in research on writing as a learning activity. Journal of Writing Research, 7(3), 311–350. Kress, G., Jewitt, C., Ogborn, J., & Tsatsarelis, C. (2001). Multimodal teaching and learning: Rhetorics of the science classroom. London: Continuum. Kress, G., & Ogborn, J. (1998). Modes of representation and local epistemologies: The presentation of science in education. SISC Working Papers, 2, 2–22. Kress, G., Ogborn, J., & Martins, I. (1998). A satellite view of language: Some lessons from science classrooms. Language Awareness, 7(2–3), 69–89. Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). London: Routledge. Latour, B. (1986). Visualization and cognition. Knowledge and Society, 6(6), 1–40. Lemke, J. (1990). Talking science: Language, learning and values. Norwood, NJ: Ablex. Lemke, J. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. In J. R. Martin & R. Veel (Eds.), Reading science: Critical and functional perspectives on discourses of science (pp. 87–113). London: Routledge. Lemke, J. (2004). The literacies of science. In E. W. Saul (Ed.), Crossing borders in literacy and science instruction: Perspectives on theory and practice (pp. 33–47). Newark, DE: International Reading Association. Lowe, R., & Ploetzner, R. (2017). Learning from dynamic visualization. Cham, Switzerland: Springer. Lowe, R. K. (2003). Animation and learning: Selective processing of information in dynamic graphics. Learning and Instruction, 13(2), 157–176. Martin, C. (2011). Drew Berry. The Lancet, 377(9769), 895.

1 A Multidisciplinary Perspective on Animation Design and Use in Science Education

21

Martin, J. R. (2017). Revisiting field: Specialized knowledge in secondary school science and humanities discourse. Onomazein, 111–148. Martin, J. R., & Veel, R. (Eds.). (1998). Reading science: Critical and functional perspectives on discourses of science. London: Routledge. Martin, J. R., & White, P. (2005). The language of evaluation: Appraisal in English. London/New York: Palgrave/Macmillan. Nosch, D. S., Foppa, C., Tóth, M., & Joos, R. E. (2015). Blink animation software to improve blinking and dry eye symptoms. Optometry and Vision Science, 92(9), e310–e315. O’Hallaron, C. L., Palincsar, A. S., & Schleppegrell, M. J. (2015). Reading science: Using systemic functional linguistics to support critical language awareness. Linguistics and Education, 32, 55–67. O’Halloran, K. (2003). Intersemiosis in mathematics and science: Grammatical metaphor and semiotic metaphor. In A.-M. Simon-Vandenbergen, M. Taverniers, & L. Ravelli (Eds.), Grammatical metaphor (pp. 337–366). Amsterdam/Philadelphia: John Benjamins. Ploetzner, R., & Lowe, R. (2012). A systematic characterisation of expository animations. Computers in Human Behavior, 28(3), 781–794. Ploetzner, R., & Lowe, R. (2017). Looking across instead of Back and forth: How the simultaneous presentation of multiple animation episodes facilitates learning. In R. Lowe & R. Ploetzner (Eds.), Learning from dynamic visualization (pp. 51–68). Cham, Switzerland: Springer. Polias, J. (2015). Apprenticing students into science: Doing, talking, writing and drawing scientifically. Stockholm: Hallgren and Fallgren. Prain, V., & Tytler, R. (2012). Learning through constructing representations in science: A framework of representational construction affordances. International Journal of Science Education, 34(17), 2751–2773. Rose, D., & Martin, J. R. (2012). Learning to write, reading to learn: Genre, knowledge and pedagogy across the curriculum. London: Equinox. Sasihithlu, K. (2019). Heat transferred in a previously unknown way. Nature, 576(12th December), 216–217. Schleppegrell, M. (2011). Supporting disciplinary learning through language analysis: Developing historical literacy. In F. Christie & K. Maton (Eds.), Disciplinarity: Functional linguistic and sociological perspectives (pp. 197–216). London: Bloomsbury. Schleppegrell, M. (2013). The role of Metalanguage in supporting academic language development. Language Learning, 63(1), 153–170. Smetana, L. K., & Bell, R. L. (2012). Computer simulations to support science instruction and learning: A critical review of the literature. International Journal of Science Education, 34(9), 1337–1370. https://doi.org/10.1080/09500693.2011.605182 Tang, K.-S., & Danielsson, K. (2018). Global developments in literacy research for science education. Cham, Switzerland: Springer. Tang, K. s., Delgado, C., & Moje, E. B. (2014). An integrative framework for the analysis of multiple and multimodal representations for meaning-making in science education. Science Education, 98(2), 305–326. Tarshizi, E., Sturgul, J., Ibarra, V., & Taylor, D. (2015). Simulation and animation model to boost mining efficiency and enviro-friendly in multi-pit operations. International Journal of Mining Science and Technology, 25(4), 671–674. Tytler, R., & Hubber, P. (2010). A representation-intensive signature pedagogy for school science. Paper presented at the Australian Association for Research in Education, Melbourne, Australia. Tytler, R., Prain, V., & Hubber, P. (2018). Representation construction as a Core science disciplinary literacy. In K.-S. Tang & K. Danielsson (Eds.), Global developments in literacy research for science education (pp. 301–318). Cham, Switzerland: Springer. Tytler, R., Prain, V., Hubber, P., & Waldrip, B. (2013). Constructing representations to learn in science. Dordrecht, The Netherlands: Springer. Unsworth, L. (2001). Teaching multiliteracies across the curriculum: Changing contexts of text and image in classroom practice. Buckingham, UK: Open University Press.

22

L. Unsworth

Villa, C., Olsen, K., & Hansen, S. (2017). Virtual animation of victim-specific 3D models obtained from CT scans for forensic reconstructions: Living and dead subjects. Forensic Science International, 278, e27–e33.

Len Unsworth is Professor in English and Literacies Education and research director of educational semiotics in English and literacy pedagogy at the Institute for Learning Sciences and Teacher Education (ILSTE), at the Australian Catholic University in Sydney, Australia. Len’s current research interests include systemic functional semiotic perspectives on multimodal and digital literacies in English and in curriculum area teaching and learning in primary and secondary schools.

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Part II

Educational Semiotics and the Representation of Knowledge in Science Animation

Chapter 2

A Functional Perspective on the Semiotic Features of Science Animation Yufei He

2.1

Introduction

Recent decades have witnessed an increased use of animation in education (Lowe & Ploetzner, 2017; Ploetzner & Lowe, 2012), especially for science education (Berney & Bétrancourt, 2016). The popularity of science animation is largely due to its advantage of dynamically visualizing complex systems and is supported by the advent of digital media. Technology has brought innovations to the traditional way of making animation, i.e. frame-by-frame production, which is laborious and timeconsuming. Nowadays, non-animation professionals can also make simple animations by using easily accessible and affordable computer software. The use of computer code also enables automatic generation of animations which present more realistic depictions of the subject matter (Berry, 2018). The popularity of using science animation in education has ignited a strong research interest in the efficacy of animation in fostering learning. Most studies adopt a cognitive perspective to investigate the processes involved in interpreting dynamic visualization (e.g. Lowe, 2003, 2004; Tversky, Morrison, & Betrancourt, 2002). Many studies use experiments to compare static images with animation (e.g. Adesope & Nesbit, 2013; Ayres, Marcus, Chan, & Qian, 2009) or compare animation with live-action video (e.g. Smith, McLaughlin, & Brown, 2012) to see whether animation is superior in facilitating students’ learning of science knowledge. These studies have produced contradictory findings in terms of the efficacy of animation. Meta-analyses of experimental studies conducted by Berney and Bétrancourt (2016) and Höffler and Leutner (2007) report an overall positive effect

Y. He (*) School of English for International Business, Guangdong University of Foreign Studies, Guangzhou, China e-mail: [email protected] © Springer Nature Switzerland AG 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_2

25

26

Y. He

of animation. However, many researchers are also concerned about the negative consequences of learning from animation due to its visual complexity (Lowe, 2003) – there may be a huge amount of information to be processed within a short excerpt of animation. This visual complexity is made more prominent by the transient nature of animation (Ainsworth & VanLabeke, 2004; Mayer, Hegarty, Mayer, & Campbell, 2005). Unlike static images, information in animations is constantly in motion (unless the viewer hits a pause button to ponder over one frame of an animation), which makes it even harder for information to be processed continuously. A common problem that may explain the contradictory findings is animation blindness (cf. Maton, 2014), i.e. neglecting the semiotic features of animation itself (Berney & Bétrancourt, 2016; Ploetzner & Lowe, 2012). It is true that animation is basically about change. However, change is by no means unitary. There are different types of change which have different impacts on students’ perception and learning. As Scheiter (2017, p. 238) notes, “the manifoldness of changes over time that can be portrayed in a visualization is also the reason why research that targets overly general questions, such as whether dynamic visualizations are more effective than static ones, is doomed to failure”. A more in-depth understanding of the nature of animation is crucial to the design of science animations and the research on their efficacy in enhancing learning. To this end, this chapter proposes a functional perspective on the semiotic features of science animation. Examples of science animation used in this study are collected from TED-ed animations, study.com (an education website) and the online portal of Explain Everything (https://explaineverything.com/), an interactive whiteboard application. Section 2.2 offers a review of relevant studies of the semiotic features of educational animation in different fields. Drawing on concepts from linguistics and semiotics, Section 2.3 examines the semiotic features of animation from a functional perspective, i.e. different functions animations fulfill and different strands of meaning they make – ideational meaning, interpersonal meaning and textual meaning. The proposed framework is applied to analyze two animations on the topic of static electricity in Section 2.4. Section 2.5 concludes by pointing out the implications for both researchers and science educators.

2.2

The Semiotic Features of Animation

Although the emphasis of research on science animations has been largely about learners’ behaviour, there are a few studies that do discuss animations per se, including the advantages and disadvantages of animation compared to static images, the constituents of animation and a detailed characterization of different types of animation.

2 A Functional Perspective on the Semiotic Features of Science Animation

2.2.1

27

Advantages and Disadvantages of Animation

Animation has been acknowledged in terms of its function of attracting learners’ attention (Berney & Bétrancourt, 2016) and directly conveying change in a dynamic system or structure (Lowe, 2003). However, its disadvantages have remained at the centre of concern for researchers. First, the fact that animation is played sequentially makes it hard for the information in the animation to be compared and contrasted (Ploetzner & Lowe, 2017). Second, the amount of information to be processed in animation is demanding, especially when one needs to perceive and understand simultaneous changes (Berney & Bétrancourt, 2016). Third, the transience of animation increases the burden for learners to internalize information extracted from the visual representation. Fourth, the split attention effect within the representation also brings problems to perception, as “full attention to one part of the display would result in neglect of information in other regions” (Lowe, 2003, p. 159). Although these are general comments about animation, they are issues that we need to consider when examining features of different animations.

2.2.2

Constituents of Animation: Element and Change

Animation is basically about changing visuals. What is important is to understand what visuals are changing and what types of changes they go through. Visuals that are involved in changes and changes that happening to the visuals are two basic constituents of animation. In their systematic characterisation of educational animations, Ploetzner and Lowe (2012) classify the visual representations in animation as iconic pictures, analytic pictures, symbols, formal notations, labels and text. The difference between iconic pictures and analytic pictures lies in their abstraction qualities. Iconic pictures are more realistic depictions of life experience, including schematic pictures, realistic pictures and photo-realistic pictures; analytic pictures are more abstract representations with a focus on analysing the hidden structure of life experience, including charts, diagrams, graphs and maps. This distinction is important as different representations tend to be used for different knowledges. However, the term ‘iconic’ is ambivalent here. Within the category of iconic pictures, there is a difference in terms of their iconicity levels, as realistic pictures are definitely more iconic than schematic pictures. Although working on different kinds of animation (animation in film opening titles), Leão (2012) offers a classification of the changing visuals which is of great relevance to this study. According to Leão (2012), changing visuals are termed elements, including verbal entities (texts), pictorial elements (depictions of things in nature) and abstract elements (simple geometric forms). Building on Leão’s (2012) system, He and van Leeuwen (2019) extend abstract elements to geometric forms, mathematical symbols and cueing. Cueing is an important attention signalling device and has long been proposed as an intervention to improve the efficacy of animation (e.g. De Koning, Tabbers, Rikers,

28

Y. He

Table 2.1 Classification of visuals by different researchers Researchers Ploetzner and Lowe (2012) Leão (2012) He and van Leeuwen (2019)

Classification of visuals Iconic pictures, analytic pictures, symbols, formal notations, labels and text Verbal entities, pictorial elements and abstract elements Verbal elements, pictorial elements and abstract elements (geometric forms, mathematical symbols and cueing)

& Paas, 2009; Lowe & Boucheix, 2017). He and van Leeuwen (2019) confine cueing to arrows/frames that appear and/or move. By contrast, many researchers (Berney & Bétrancourt, 2016; de Koning et al., 2009) propose a wider definition of cueing which will be discussed in the next section in terms of the scaffolding of animation. Table 2.1 shows an overview of the classification of visuals in animation by different researchers. These references develop our understanding of the visuals that are involved in changes in animation. Compared to the visuals that are changed, change itself is under-researched. Only a handful of studies propose a classification of the types of changes. Lowe (2003) classifies changes into three types: 1. form changes (change of the elements’ size, shape, property and texture), 2. position changes (movement of elements from one place to another) 3. inclusion changes (appear/disappear of elements). Among these changes, he concludes that form changes and position changes are more likely to attract viewers’ attention when they stand out from the context. Besides the different types of changes, the properties of changes (e.g. the speed and direction of changes) are equally important to consider, as for different knowledges, the general requirements to perceive from animations are different. For some knowledges, it is only important to understand the change regardless of how fast or slow the change is; while for others, the relative speed of change is also important. These semiotic differences also pose different tasks for the visualization of changes (Scheiter, 2017). Drawing on Lowe (2003) and Leão’s (2012) classification of changes, He and van Leeuwen (2019) provide a detailed system of change that captures both the types of changes and the properties of changes. A simplified system of change proposed by them is presented in Fig. 2.1. They model animation in the tradition of Systemic Functional Linguistics (hereafter SFL), using a curly bracket for ‘and’ relation and square bracket for ‘either or’ relation. A combination of square bracket and curly bracket is used to show a relation of both as alternatives and as simultaneous options. Figure 2.1 shows that a change can simultaneously select from the sub-system of TIME and TYPE1 of changes, as any change is a change happening in time. The TIME system shows that a change can be instantaneous (a change that happens immediately) or gradual (a change that happens across time). Gradual change can be constant or variable depending on whether there is a 1

According to SFL system network conventions, system names are written in small capital letters.

2 A Functional Perspective on the Semiotic Features of Science Animation

29

Fig. 2.1 System for change. (Adapted from He & van Leeuwen, 2019)

variation in speed. The slanting square bracket for [change: gradual: constant]2 means that the system of fast and slow is a cline instead of one with clear-cut categories. There are four types of change: inclusion, movement, illumination and

When options in a system are referred to in writing, they are written in square brackets. ‘Constant’, an option in the SPEED system, can be written as [constant] or [change: gradual: constant] to indicate its position in a system. 2

30

Y. He

Fig. 2.2 System for element. (Adapted from He & van Leeuwen, 2019)

transformation. Movement can be further classified as displacing or non-displacing. An example of [movement: displacing] is the earth orbiting around the sun (the earth changing position); while an example of [movement: non-displacing] is the earth rotating around its own axis (the earth moves without changing position). [movement], either displacing or non-displacing, also follow pathways which can be either [rotation] or [translation] or a combination of these two. [illumination] includes change of colour or change of brightness. [transformation] describes an element changing its size ([rescale]) or changing its shape ([reshape]). This detailed system for change serves as a basis for a systematic description of changes in animation. He and van Leeuwen’s (2019) system of element is presented in Fig. 2.2. Any type of element can undergo any type of change.

2.2.3

Characterization of Types of Animation

Unlike semiotic studies (He & van Leeuwen, 2019; Leão, 2012) that focus on a detailed description of elements and changes as constituents of animation, studies in the field of science education work more on a broader characterization of animation as a whole. However, it should be noted that these studies are still in the minority compared with the vast number of studies that examine animation solely from the learners’ perspective. Höffler and Leutner (2007) extend the notion of decorational pictures and representation pictures (Carney & Levin, 2002) to the description of animation. According to them, whereas representation animation depicts the learning content explicitly, decorational animation mainly functions to motivate learners. They only offer the two broad categories without mentioning the criteria to distinguish the two. Berney and Bétrancourt (2016) propose the abstraction quality of the visual representation as a factor to categorize the functions of animation. However, they fail to establish a clear relation between the two. It is unclear whether representation animation adopts a more abstract or concrete representation style. Moreover, the notion of representation and decorational animation is itself problematic. As

2 A Functional Perspective on the Semiotic Features of Science Animation

31

Table 2.2 The dimensions of presentation. (Ploetzner & Lowe, 2012, p. 789) (1). Representations employed (2). Abstraction (3). Explanatory focus (4). Viewer perspective (5). Spatiotemporal arrangement

(a). Visual: iconic pictures, analytic pictures, symbols, formal notations, labels, text (b). Auditory: sound, speech, narration Iconic, abstract Behaviour, structure, function Single, multiple (a). Spatial resolution and structure (b). Temporal resolution and structure

(6). Duration

(1) spatial resolution: Constant, variable (2) spatial structure: 2D vs. 3D; flat vs. hierarchical organization (1) temporal resolution: Discrete, continuous with pauses, continuous with cuts, continuous (2) temporal structure: Linear or cyclic; sequential or simultaneous

Presentation time

representation animations also have the potential to motivate learners, it is often difficult to draw a clear-cut boundary between the two. By extracting dimensions of analysis from a wide range of studies of educational animation, Ploetzner and Lowe (2012) propose a systematic characterization of educational animations. Their pioneering study reveals four major dimensions of animation: presentation, user control, scaffolding and configuration. According to Ploetzner and Lowe (2012), the dimension of presentation includes attributes that are inherent characteristics of the animations and the other three dimensions cover attributes that are external supplements to the animations. I will briefly review the dimensions of presentation and scaffolding as they are relevant to this study. As can be seen in Table 2.2, the dimension of presentation includes representations employed (visual and/or auditory), the abstraction of representation, explanatory focus, viewer perspective, spatio-temporal arrangement and duration. The two sub-dimensions of representations employed and their abstraction are discussed above in Sect. 2.2. The presentation of animations can vary in their explanatory focus: they can focus on the behaviour of entities, the structure of entities and/or the functions that the behaviour and structures fulfil. They can generate single or multiple viewer perspectives. In terms of spatio-temporal arrangement, Ploetzner and Lowe (2012) state that an animation has both a spatial and temporal dimension; each can be analysed in its resolution and structure. Spatial resolution refers to the number of visuals displayed on the screen. It is related to “the degree of detail that can be distinguished within a display” (p. 785); temporal resolution is measured by the number of frames displayed per second, which influences “the degree of transformation that can be distinguished between adjacent frames” (p. 785). Spatial and temporal structure are concerned with the way information is structured spatially and temporally.

32

Y. He

The dimension of scaffolding covers visual cues, written prompts and spoken prompts that are used to facilitate the learning process. Based on de Koning et al. (2009), visual cues are classified into selection cues, organization cues and integration cues in terms of their different functions. Selection cues function to guide viewers’ attention; organization cues lay emphasis on the major topics of instruction and their organization. While selection cues make element/s salient, integration cues attend to the relations between elements and facilitate an integration of those elements. In addition to changed arrows/frames, color-coding of elements/parts and fading of elements/parts also fall within the category of visual cues. Ploetzner and Lowe’s (2012) study offers a very important framework for examining the semiotic features of animation. However, studies like theirs are still in the minority. The semiotic features of science animation and their possible implications on knowledge building are still very much under-explored. Moreover, a crucial problem is how to analyse all the variations in a systematic and integrated way, as they do not function separately but are closely interwoven in one video. A possible solution is to examine these variations in terms of the different functions they fulfil. In the next section, building on the detailed system for the constituents of animation (element and change) and Ploetzner and Lowe’s broader characterization, a functional perspective will be introduced as a complement to the widely used cognitive approach to explore and interpret the variations of animations.

2.3

A Functional Perspective on the Semiotic Features of Science Animation

As the focus of this chapter is on the semiotic features of science animation, it is helpful to draw on insights from semiotic studies, particularly those in the tradition of social semiotics largely influenced by SFL. The definition of social semiotics is first raised by Halliday (1978) in his modelling of language. In his work Language as Social Semiotic, Halliday (1978) proposes a framework that interprets language within a sociocultural context, in which the culture itself is interpreted in semiotic terms. He emphasizes the intrinsic functionality of language, i.e. the functions language fulfils in social and cultural contexts shape the way it is. This functionality is conceptualized as different kinds of meaning known as metafunctions, including ideational, interpersonal and textual meanings (Halliday & Matthiessen, 2004). Ideational meanings construe our experience of the world; interpersonal meanings enact our personal and social relationships with others; textual meanings organize ideational and interpersonal meanings as an information flow. This concept of metafunction has been widely used in modelling various semiotic modes other than language, including image (e.g. Kress & van Leeuwen, 2006; O’Toole, 2011), gesture (Martin & Zappavigna, 2018), mathematics (Doran, 2018a, 2018b; O’Halloran, 2005) and architecture (McMurtrie, 2017; Stenglin, 2004). Animation also fulfils the three metafunctional meanings. Taking an image changing size as an

2 A Functional Perspective on the Semiotic Features of Science Animation

33

example, it can mean that the image becomes larger or smaller (ideational meaning) or function to increase the visual prominence of the image and direct viewers’ attention (textual and interpersonal meaning). We can therefore analyse an animation in terms of the three metafunctional meanings. For the ideational meaning, there are two questions at play: what kinds of experience are construed in an animation and how are these experiences construed? The interpersonal meaning is concerned with the relationship between viewers and characters in an animation, which can be analysed by examining the way an animation engages its viewers. The textual meaning is related to the organization of information flow in an animation. Although a changed element may only be textually important without construing any ideational meaning (e.g. change of arrows, in Ploetzner and Lowe’s (2012) term ‘scaffolding’), an animation fulfils all of the three metafunctional meanings. This again shows that it is problematic to draw a distinction between decorational animation and representational animation, as most animations fulfil both functions of construing the target knowledge (ideational meaning) and engaging the viewers (interpersonal meaning). In the rest of this section, the three meanings of animation will be introduced, with examples of science animation used to illustrate this analysis framework.

2.3.1

Ideational Meaning

As mentioned above, the ideational meaning of animation construes the experience of the world, of which two aspects can be explored -- what kinds of experiences are construed in animation and in which way are these experiences construed.

2.3.1.1

What Kinds of Experiences Are Construed

For the kinds of experiences that are construed in animation, it is useful to draw on the SFL concept of field which is concerned with the nature of the social activity realised through language. Field is defined as “sets of activity sequences oriented to some global institutional purposes” (Martin, 1992, p. 292). The concept of field has been used widely to study the construing of target knowledge by both language and image in science discourse. According to Doran and Martin (forthcoming), field is described as a resource for construing phenomena either statically as items or dynamically as activities. An activity can be either momented or unmomented. Unmomented activity refers to single activity; momented activity means activity sequences. An item can be either singular or taxonomized in terms of its classification or composition. The FIELD PERSPECTIVE system is presented in Fig. 2.3. The two field perspectives, activity and item, can also be propertied. Properties are gradable and measurable, often providing criteria for distinguishing items in a taxonomy (Doran & Martin, forthcoming). Examples of properties of items include the charge of particles (distinguishing a proton from an electron) and the temperature

34

Y. He

Fig. 2.3 The field perspective system. (Doran & Martin, forthcoming)

Fig. 2.4 An expanded system for field. (Doran & Martin, forthcoming)

of particles; properties of activities involve the frequency of the oscillation of light (distinguishing different regions of electromagnetic radiation) and the speed of the particles’ motion. The expanded system for field is presented in Fig. 2.4. The option ‘–’ in the PROPERTY system means that this system is optional, i.e. activity or item can be propertied or not. Studies exploring the role of language in building science knowledge show that the language of science is characterized by complex taxonomies and activity sequences which construe very precise and field specific meanings (Hao, 2020; Martin, 1993; Wignell, Martin, & Eggins, 1989). Similarly, images also play a crucial role in organizing science knowledge. One single image has the potential to present deep taxonomies, long activity sequences and multiple properties (Doran, 2018b, 2019). Animation can also construe different kinds of activity or item. Figure 2.5 shows two successive frames in an animation explaining the difference

2 A Functional Perspective on the Semiotic Features of Science Animation

35

Fig. 2.5 A man walking in a zig-zag manner. (Drawing after two frames from Velocity and speed) Fig. 2.6 Measuring temperature. (Drawing after two frames from Temperature)

between velocity and speed. By presenting a scenario of a man walking 20 meters in a zig-zag manner in 60 seconds, this animation construes an unmomented activity. Figure 2.6 shows two snapshots from the lesson video Temperature. It explains the principle of measuring temperature. A substance is made up of moving particles. The temperature of a substance depends on the kinetic energy of its particles. As particles are small, we need to use an indirect way to measure their kinetic energy. A common way to do that is to use thermometers which use the expansion of substance to indirectly measure its temperature. The first frame in Fig. 2.6 shows constantly moving molecules (small circles) in an object (the big circle). The second frame shows that after heat (crooked arrows) is added to an object, the molecules move faster which causes the object to expand. The animation from which the snapshots are taken construes a momented activity (activity sequence): molecules in an object are constantly moving ! add heat to an object ! molecules in the object move faster ! the object expands. What is important to the target knowledge here is not just this momented activity, but also the properties of activities – the speed of the

36

Y. He

Fig. 2.7 Change of position construing item of composition. (From The Skeleton of Human Hand)

moving particles before and after heat is added to an object. The properties of activities are construed by the TIME system of change, i.e. whether the change happens instantaneously or gradually, and if gradually, whether it happens fast or slow. In the above two examples of animation, change of position of elements construes activities. They can also construe items of classification and composition. Figure 2.7 presents two screenshots from a project on the skeleton of the human hand. The project is available on the online portal of Explain Everything as a template for teaching, i.e. teachers can download and edit the project and ask students to do exercises (matching the names and shapes of bones) by animating the hand bones. As can be seen in Fig. 2.7, the bones can be displaced from the right part of the screen to fit the middle part of the bone structure. The changes of position of the bones do not function to construe activities but the composition of the human hand. Figure 2.8 shows two frames in an excerpt of an animation that construes an item of classification. It explains that adhesives can be made from synthetic molecules and natural proteins and carbohydrates, including the vegetable starch dextrin, the milk protein casein and the terpenes in tree resin. The pictorial elements of corn and grain, milk and tree branch appear by displacing (hopping) from the right of the screen. Similar to the example of Fig. 2.7, pictorial elements’ changing position do not construe activities of vegetables, milk and tree branch. They function to introduce items that enter into a classification which is explained in the voiceover of this animation. The above analysis shows that the same change can construe different experiences. Change of position can construe activities or items; it can also construe propertied activities by showing a contrast between different activities. It is important to consider the meaning-making potential of changes in construing target knowledge. For changes that construe activities, both the change and the process of change are equally important. By contrast, for changes that construe items, only the result of change is important regardless of the process of change. As shown in Figs. 2.7 and 2.8, changes of position only function to present elements or relate elements together. The TIME system of change doesn’t play a role in construing target

2 A Functional Perspective on the Semiotic Features of Science Animation

37

Fig. 2.8 Change of position functions to introduce items. (Drawing after two frames from Which is Stronger: Glue or Tape, Cox, 2018)

knowledge. The limited meaning-making potential of changes that construe items is also made clear when we consider rendering them to static images. It is easy to turn animations which construe items to static images as shown in Figs. 2.7 and 2.8 without losing their original meanings. However, in order to make sense of Fig. 2.6, we need additional explanations. Although we may be able to imply changes of molecules, we don’t know the speed of changes. Changes that construe activities have more elaborate meaning-making potential than those that construe items. According to Mayer (2009, p. 267), an important principle of multimedia learning is coherence principle, i.e. “People learn better when extraneous words, pictures, and sounds are excluded rather than included”. This principle may also be extended to extraneous changes, i.e. people learn better when extraneous changes are excluded. This is a hypothesis that opens for future investigation. As the process of change is not important for animations that construe items, it may be better to keep these changes as simple as possible.

2.3.1.2

How Are Experiences Construed

After figuring out the kinds of experiences construed in animation (activity/item/ property of activity or item), it is also important to examine how these experiences are construed, which is concerned with the depiction style of activity/item. Previous studies only focus on the depiction style of items without considering the representation of activities. Moreover, the depiction style of items is reduced to a simple iconic/abstract dichotomy in many studies (e.g. Ploetzner & Lowe, 2012). Figure 2.9 shows two human characters with different depiction styles in two animations. It can

38

Y. He

Fig. 2.9 Human characters in two animations Picture 1: Drawing after a frame from Which is Stronger: Glue or Tape? (Cox, 2018); Picture 2: Drawing after a frame from The science of static electricity (Bhagwat, 2015)

be seen that the level of iconicity is a continuum rather than one alternative in a dichotomy. The human character in picture 1 is less iconic than a photo-realistic representation of human, as the human face is reduced to a simple sketch with all the details of body parts removed. However, it is more iconic compared to the human character in picture 2 which can be hardly recognized as a ‘human’. Martin and Unsworth’s (forthcoming) notion of congruence is relevant here to account for the differences. Congruence is concerned with how faithful the visual representation depicts reality. Stronger congruence means the representation resembles reality and weaker congruence signals a distance from reality. We can code the two human characters in Fig. 2.9 in terms of their congruence level, with ‘+’ indicating stronger level of congruence and ‘-’ as weaker level of congruence. The two human characters from left to right can therefore be coded as congruence + and congruence respectively. The example of human character discussed above is concerned with the representation of item. The notion of congruence also applies to the representation of activities. Considering two representations of a caterpillar’s movement, the one that only shows the caterpillar displacing is less congruent with reality than the one showing all the details of its wriggling. The fact that a representation is more congruent with reality doesn’t necessarily mean that it is more beneficial for knowledge building, as it is important that animations only provide information that matches the learning objectives (Scheiter, 2017; Tversky et al., 2002). Figure 2.10 presents two images from two animations about the atom. Both representations show the atom as consisting of a nucleus (protons and neutrons packed together) with electrons zipping around. In terms of the level of congruence, the representation in picture 2 is more distancing from reality than picture 1, since the real number of particles in an atom is far larger than that depicted in picture 2: it only shows three protons (circles with plus signs) and three electrons (circles with minus signs) in an atom. However, the target knowledge of the animation represented by picture 2 is that “the atom has no net charge as there are equal numbers of protons and electrons in an atom”. This representation

2 A Functional Perspective on the Semiotic Features of Science Animation

39

Fig. 2.10 Two representations of atom Picture 1: Drawing after a frame from What does an atom really look like; Picture 2: Drawing after a frame from Electric charge and force

highlights the target knowledge by reducing the number of particles to a manageable amount. As mentioned by Scheiter (2017, p. 239), the visualizations designed for learning should be “limited to what is deemed essential for mental model construction, while ‘irrelevant’ details are left out”. We can therefore propose a cline of essentialization to measure the degree of animation in representing the essential details relating to target knowledge (Martin & Unsworth, forthcoming). Different from the cline of congruence, it doesn’t have a fixed reference point (the objective reality); its benchmark varies from different contexts of study. For example, the representation in picture 1 is essential - in terms of the target knowledge of ‘equal numbers of protons and electrons’, but it is essential + if the target knowledge is ‘there are many electrons zipping around the nucleus’. Whether an animation only includes essentials in relation to target knowledge should be examined for both the elements and changes. In Fig. 2.8 discussed above, the pictorial elements of vegetable and grain, milk and tree branch appear by hopping onto the screen. As discussed above, the hopping movement has little meaning-making potential for construing the item. Therefore, it is considered as less essential for building the target knowledge. In summary, it is important to consider the kinds of experiences (item/activity) construed and the ways these experiences are construed. We can measure the depiction style of item/activity in terms of the cline of congruence with objective reality and the cline of essentialization relating to target knowledge.

2.3.2

Interpersonal Meaning

Many of the animations analysed earlier in the chapter include representations that are less essential for building target knowledge. If target knowledge is the whole story, why do animation designers still choose to include all the unnecessary details?

40

Y. He

Fig. 2.11 Personified non-human characters Picture 1: Drawing after a frame from The Science of Static Electricity (Bhagwat, 2015); Picture 2: Drawing after a frame from Which is Stronger: Glue or Tape (Cox, 2018)

To understand this, we need to turn to the interpersonal meaning of animation, i.e. how does an animation engage its viewers. In the last section, examples were used to show that the same item/activity can be construed in different ways. These different depiction styles have different visual impacts on viewers (Zibrek & McDonnell, 2014). For example, cartoon characters are found to be more appealing than the more realistic characters (McDonnell, Breidt, & Bülthoff, 2012). An examination of science animations shows similar findings, i.e. representations which are less congruent with reality elicit more amusement from viewers and engage them more. Typical examples of representations that have weaker congruence in animations are personified non-human characters, as shown in Fig. 2.11. Picture 1 shows personified electrons, protons and neutrons, with facial expression and gesture emphasizing their charges. Picture 2 shows inanimate things including corn, grain, milk and tree branch depicted with eyes. In addition to personified non-human characters, human characters can also be represented as less ‘person-like’ to elicit amusement from viewers. The two pictures in Fig. 2.12 show a human character that can be hardly recognized as a human. The character’s face is disproportionately large, represented by a hard-edged geometric shape with a long protruding nose. More unrealistic is the representation of the character being zapped by static electricity (the picture on the right) as he appears like a hedgehog with distorted black fingers. Both types of representations, personified non-human characters and unrealistic human characters, are coded as weaker congruence with reality. However, interpersonally they create a sense of cuteness and an ambience of popular science as opposed to serious science. A cline of humour can be proposed to examine whether the depictions elicit amusement from viewers (Martin & Unsworth, forthcoming). Comparing the representation of particles in the two pictures in Fig. 2.13, the less congruent representation (the personified depiction) is more humorous.

2 A Functional Perspective on the Semiotic Features of Science Animation

41

Fig. 2.12 A human character that is less ‘human-like’. Drawing after two frames from The science of static electricity. (Bhagwat, 2015)

Fig. 2.13 Two representations of particles Picture 1: Drawing after a frame from The science of static electricity (Bhagwat, 2015); Picture 2: Drawing after a frame from Electric charge and force

The same also applies to a comparison between the human characters in Fig. 2.9. The human character in picture 2 is less congruent with reality but more humorous. By examining the degree of humour in animations, we can understand the overall ‘ambience’ of different animations – whether they are orienting more towards ‘serious science’ or popular science, which helps us to choose appropriate animations for different target audiences.

42

2.3.3

Y. He

Textual Meaning

The ideational and interpersonal meaning of animation are organized by the textual meaning as a coherent information flow. In their systematic characterization of expository animations, Ploetzner and Lowe (2012) put great emphasis on the presentation of animations. One of the dimensions relating to the presentation of animation is its spatio-temporal arrangement which is related to the organization of information both spatially and temporally. This spatio-temporal organization of information is what we are going to explore in terms of the textual meaning of animation in this section. There are two major structures for the organization of information flow, accumulating and predicting. In accumulating, changed elements which construe activity/item get accumulated into a flow of information. These changed elements can be presented on the same screen (within the same space) or on different screens (in different spaces), with the former categorized as [accumulating: concurrent] and the latter as [accumulating: retrospective]. An example of [accumulating: concurrent] can be seen in Fig. 2.14 which is taken from an animation visualizing different movements of particles within solid, liquid and gas: it shows that the distance between particles in gas is greater than those in liquid which is in turn greater than those in a solid; in terms of the movement of particles, the interaction of particles in a solid is the strongest and the interaction of particles in gas the weakest. By using a split-screen, this animation tables the target knowledge within the same space which enables viewers to compare and contrast the movement of particles in different states of matter. This information structure (‘simultaneous presentation’ in Ploetzner and Lowe’s (2017) term) facilitates the building of a coherent and structured mental model of the subject matter by allowing “learners to move relatively seamlessly between different levels of relationship” (Ploetzner & Lowe, 2017, p. 55).

Fig. 2.14 Accumulating: concurrent. (Drawing after a frame from What is sound)

2 A Functional Perspective on the Semiotic Features of Science Animation

43

Fig. 2.15 Accumulating: retrospective. (Drawing after two frames from States of matter)

Fig. 2.16 Predicting. (Drawing after two frames from Changes in heat and energy diagram)

Different from [accumulating: concurrent] structure, viewers do not have an overall snapshot of the information in [accumulating: retrospective] as changed elements are presented on different screens. Instead they need to rely on their memory to refer back to the previous parts of animation to aggregate the information. Figure 2.15 shows two screenshots from another animation on states of matter. While the movement of particles within gas and liquid are shown on the same screen (see picture 1), what happens to particles within a solid is presented later on a different screen (see picture 2). To compare the movement of particles in the three states of matter, viewers need to relate the visualization of solid to that of gas and liquid retrospectively. This is an example of [accumulating: retrospective] structure. In addition to accumulating, information flow can also be organized through predicting. In predicting, information is not accumulated as a larger whole; instead a general picture is presented first, which is then decomposed into smaller information portrayals with the use of visual cues (the animation of arrows/frames/colour coding of elements). An example of a predicting structure can be identified in Fig. 2.16 which shows two screenshots in an animation explaining the diagram of changes in heat and energy. The diagram first appears as one whole portrayal of information (picture 1). Then a moving arrow is added to highlight different parts of the diagram as new information (picture 2). By making certain parts of the diagram more salient,

44

Y. He

Fig. 2.17 The information flow system

the changed arrow is used to direct viewers’ attention. It also functions to relate segments of the voiceover to parts of the diagram, making the animation a coherent whole. Predicting structure is often used to walk viewers through complex diagrams like this one. The use of arrows/frames breaks a single eyeful into manageable smaller information units which facilitates knowledge building. The organization of information in animations is presented as a system in Fig. 2.17. The flow of information in animation can be either [accumulating] or [predicting], with the former aggregating information as a larger one and the latter decomposing information into smaller ones. [accumulating] can be further classified as [concurrent] or [retrospective], depending on whether information accumulation happens within the same space or across different spaces.

2.4

Analysis of Two Animations on Static Electricity

In the last section, a functional perspective was proposed to examine the semiotic features of science animations in terms of their ideational, interpersonal and textual meaning. Ideational meaning is concerned with the kinds of experiences construed and the way these experiences are construed; interpersonal meaning deals with how an animation engages its viewers; textual meaning explores the organization of information in an animation. In this section, these trinocular lenses will be applied to analyse two animations on the topic of static electricity. The two animations under discussion are collected from TED-ed and the study. com website respectively. TED-ed Animations are produced by a collaboration between professional animators and educators. They are short videos free to the public, each focusing on a specific topic. Study.com is an education website that offers lesson videos on a wide range of subjects, including science, social science and arts. These lesson videos are only accessible to its members. Non-members only have a free 5-day trial of their accounts. The Study.com website states that by enrolling in the courses, students are able to transfer credits to over 1000 colleges and universities in America. The animation on static electricity is one of the lesson videos in the course of Intro to Natural Sciences. The animation from TED-ed is hereafter mentioned as Animation 1 and the one from study.com referred to as Animation 2. Both animations are short, with a total duration of less than 7 minutes. Both have voiceover. Animation 1 also has background music and sound effects. Information about the two animations on static electricity is illustrated in Table 2.3. The target knowledge in these two animations is the phenomenon of static electricity and why people would be zapped by it. Both animations start with

2 A Functional Perspective on the Semiotic Features of Science Animation

45

Table 2.3 Information about the two animations on static electricity Available on Name of the animation Duration Voiceover Music and sound effects

Animation 1 TED-ed The science of static electricity (Bhagwat, 2015) 3:38 Yes Yes

Animation 2 Study.com Electric charge and force 6:33 Yes No

examples of static electricity in daily life. Animation 1 shows a person zapped by static electricity when he reaches the door knob to open the door after walking across a soft carpet. In Animation 2, a boy rubs a balloon against his hair and finds that his hair stands straight up after he takes the balloon away. Both animations begin to explain static electricity by introducing the composition of atoms. Everything in the universe is made of atoms which contain three types of particles: protons, neutrons and electrons. The particles are distinguished in terms of their charge. Protons have a positive charge; neutrons have a neutral charge and electrons a negative charge. Most atoms have no net charge as there are equal numbers of protons and electrons. But there are many atoms that either give away electrons or accept more electrons with relative ease. The accumulation of electric charge on an object, such as on a balloon or a door knob, is called static electricity which can result in a rapid transfer of electrons between objects. This rapid transfer of electrons is the sudden spark. In terms of the ideational meaning, the field relations construed in the two animations are item of composition (composition of atoms), properties of items (electric charge of particles) and activity (the transfer of electrons). However, the composition of atoms is largely construed in the voiceover of Animation 1, with the visual representation only vaguely suggesting it. Table 2.4 shows the part of the animation that explains the composition of atoms. As in some instances, static visual representations are not adequate to reveal the true picture of animation, simple descriptions are added to fill in the missing parts. In terms of the information organization of this animation excerpt, it follows an [accumulating: retrospective] structure, as the item (atom) and its composition (three types of particles) are shown on different screens (picture 1 and picture 2). Although an overall picture of the atom is given later (see picture 3), the particles are shown as different representations which may confuse learners who have no prior knowledge. Moreover, the animation represented by picture 3 shows the blue minus signs quickly revolving inside the circle. However, the movement of electrons (zipping around the outside) is not mentioned in the voiceover, which could make it harder for learners to identify the blue minus signs as electrons in picture 3. Compared to Animation 1, Animation 2 construes the composition of the atom by using a [predicting] structure. As can be seen in Table 2.5, the item (atom) first appears as one whole information portrayal. Its composition, protons (circles with plus signs) and electrons (circles with minus signs), are part of this information.

46

Y. He

Table 2.4 Explaining the composition of atoms in Animation 1 No. 1

Visual representation

Description

Voiceover All matter is made of atoms

2

Facial expression and gestures reinforcing the charge of particles

That consist of three types of smaller particles: Negatively charged electrons, positively charged protons and neutral neutrons.

3

Blue minus signs quickly revolving inside the circle

Normally the electrons and protons in an atom balance out.

Then the protons and electrons change colour sequentially (indicated in the blackand-white drawings as circles in black), which is in synchrony with the voiceover of ‘protons’ and ‘electrons’. This helps learners identify the particles and emphasizes the composition of the atom – there are equal numbers of protons and electrons. The atom and its composition are also construed in different depiction styles in these two animations. The representation of atom in Animation 1 is less congruent with objective reality than that in Animation 2. The atom and its composition are personified as human characters in Animation 1 (see Table 2.4). By contrast, the composition of atom is reduced to a schematic representation in Animation 2 (see Table 2.5), with circles standing for particles: circles with minus signs represent particles with negative charge; circles with plus signs stand for particles with positive charge. In terms of the target knowledge in these excerpts of animation, learners need to know the three types of particles that make up atoms, and more importantly, the fact that most atoms have no net charge because there are equal numbers of protons and electrons. In this respect, the representation of the atom in Animation 2 has stronger essentialization as it shows the essential knowledge. By

2 A Functional Perspective on the Semiotic Features of Science Animation

47

Table 2.5 Explaining the composition of atoms in Animation 2 No. 1

Description

Voiceover In most atoms, there are equal numbers of

2

Blue circles with plus signs changing colour to yellow.

protons

3

Red circles with minus signs changing colour to yellow.

and electrons

4

Visual representation

So the atom itself has no net charge.

48

Y. He

Fig. 2.18 Transfer of electrons in the two animations Picture 1: Drawing after a frame from Animation 1; Picture 2: Drawing after a frame from Animation 2

contrast, what is shown in Animation 1 has weaker essentialization, as the balance between protons and electrons is depicted as a person sitting in meditation (see picture 3 in Table 2.4) which strays away from target knowledge. Another important field resource construed in these two animations is the transfer of electrons (an activity). Figure 2.18 shows different representations of the transfer of electrons in the two animations. The transfer of electrons in Animation 1 is construed as an activity sequence: personified electrons on the scissors vibrate violently before jumping onto a person’s hand. A group of electrons are shown as displacing in complex and quick movement (hopping instead of just changing position). Animation 2 on the other hand construes the transfer of electrons as single repeated activities by repeatedly showing single electrons displacing from one’s hair to a balloon. The movement of electrons is simple (only changing position) and slow. In summary, movement of electrons in Animation 1 is depicted with vivid detail. However, what is important to the target knowledge is change of position itself, i.e. electrons displacing from one item to another, which is adequately shown in Animation 2 without unnecessary details. This excerpt in Animation 2 is therefore considered as stronger in essentialization than Animation 1. Many other instances in Animation 1 also exemplify unessential changes. For example, clouds are depicted as contracting and expanding alternately when they are displacing. The contraction and expansion movement are considered gratuitous in a context when it is only important to know that the cloud is displacing. In terms of the interpersonal meaning, the characters (both human and non-human) in Animation 1 are depicted in a way that is incongruent with reality, with human characters appearing to be less ‘human-like’ and non-human characters taking on human features, as shown in Table 2.6. The unessential hopping

2 A Functional Perspective on the Semiotic Features of Science Animation

49

Table 2.6 Depiction styles in Animation 1 and Animation 2 Animation 1

Animation 2

Human character

Non-human character (particles)

Table 2.7 Summary of an analysis of metafunctional meanings in the two animations Metafunctional meaning Ideational Fields construed

Animation 1 Only voiceover construes the item of composition (atom) Property of item (charge of particles) Activity (transfer of electrons)

Depiction style Interpersonal Textual

Congruence -, essential Humour + [accumulating: retrospective]

Animation 2 Item of composition (atom) Property of item (charge of particles) Activity (transfer of electrons) Congruence +, essential + Humour [predicting]

movement of electrons discussed above also functions to personify the electrons. The depiction style in Animation 1 is considered as more humorous than Animation 2, which elicits greater amusement from viewers and encourages deeper engagement. The use of music and sound effects in Animation 1, although not the focus of this study, also adds to the interpersonal appeal of the animation. The analysis of metafunctional meanings in the two animations is summarized in Table 2.7. Both animations construe the composition of the atom (although it is

50

Y. He

largely construed in the voiceover of Animation 1), charges of particles and the activity of the transfer of electrons. But these experiences are depicted in different ways in the two animations, with representations in Animation 1 featuring weaker congruence with reality and weaker essentialization in terms of the target knowledge. Interpersonally, it includes more humorous representations which creates an overall ambience of popular science that targets more towards children. The different textual organizations in these two animations serve as a basis for the construing of target knowledge, with [accumulating: retrospective] less intuitive than [predicting] in building the composition of the atom. It should be noted that an animation can have different information structure in different parts of the video. [predicting] and [accumulating: retrospective] are structures used by the two animations in construing the composition of atom.

2.5

Conclusion

This chapter offers a functional perspective on the semiotic features of science animation as a complement to the prevalent cognitive approach that focuses on the behaviour of learners. Animation fulfils three metafunctional meanings of ideational, interpersonal and textual meaning. The metafunctional framework is summarized in Table 2.8. Ideational meaning is concerned with the construing of human experience. To explore the ideational meaning of animation, we need to understand what kinds of field resources (item/activity/property of item or activity) are construed in animation and how they are construed. The ways that field resources are construed are related to the depiction style of representations. The depiction style can be measured in terms of how congruent the representation is with reality and how essential it is relating to target knowledge. Interpersonal meaning is concerned with the way an animation engages its viewers. Many non-congruent and inessential representations function to create a closer relationship between the characters and viewers. Textual meaning organizes ideational meaning and interpersonal meaning as an information flow, with different organization structure construing different types of field resources. By introducing a functional perspective on science animation, we can sort different semiotic features of animation in a more systematic way in terms of the functions they fulfil and tease apart different meaning-making potentials of animation. An analysis of the two animations on static electricity shows that representations that are interpersonally engaging are not necessarily ideationally conducive to knowledge building. By distinguishing the different meanings of animation, we can be clearer in terms of the meaning we are talking about when discussing the pros and cons of animation. An important problem left unanswered in this chapter is the potential effects of different representations on viewers. For example, it still remains a question in terms of whether there is a trade-off between the ideational meaning and interpersonal meaning of animation, i.e. whether representations with limited potential in building

2 A Functional Perspective on the Semiotic Features of Science Animation

51

Table 2.8 Summary of the metafunctional framework of animation Metafunction Ideational

Meaning What experiences does an animation construe and how does it construe these experiences

Types of experiences Representation dimensions

Types Description Activity, item, property of item/activity Congurence

Essentialization

Interpersonal

How does an animation engage its viewers

Humor

Textual

How does an animation organize its information flow

Accumulating

Predicting

Congurence +: Animation depicts reality Congurence -: Animation distances from reality Essential +: Animation only represents the essential details relating to target knowledge Essential -: Animation represents the inessential details relating to target knowledge Humor +: Elicit amusement from viewers (congruence -) Humor -: doesn’t elicit amusement from viewers (congruence +) [concurrent]: Changed elements presented on the same screen [retrospective]: Changed elements presented on different screens a general picture is presented first and then decomposed into smaller information by using visual cues

knowledge always work to engage viewers. As mentioned above, many representations that stray away from reality (congruence -) and include unnecessary details (essential -) function to engage viewers, for example in Animation 1 on static electricity. However, this observation is based on previous literature and my own judgments, which is open for future investigations.

52

Y. He

From a semiotician’s point of view, to articulate the nuanced differences of semiotic features in different animations has implications for both researchers and science educators. For researchers, including both semioticians and science education researchers, this provides a basis for further examination of these differences on viewers’ perception and on knowledge building. For science educators, including both teachers and parents, this offers different lenses on the process of selecting educational animations from a huge resource of online animations that target different learners.

References Adesope, O. O., & Nesbit, J. C. (2013). Animated and static concept maps enhance learning from spoken narration. Learning and Instruction, 27, 1–10. Ainsworth, S., & VanLabeke, N. (2004). Multiple forms of dynamic representation. Learning and Instruction, 14(3), 241–255. Ayres, P., Marcus, N., Chan, C., & Qian, N. (2009). Learning hand manipulative tasks: When instructional animations are superior to equivalent static representations. Computers in Human Behavior, 25(2), 348–353. Berney, S., & Bétrancourt, M. (2016). Does animation enhance learning? A meta-analysis. Computers and Education, 101, 150–167. https://doi.org/10.1016/j.compedu.2016.06.005 Berry, D. (2018). The molecular machines that create your flesh and blood. Paper presented at Learning from Viewing and Creating Animations in School Science Research Symposium. Australian Catholic University, Sydney, 29 November 2018. Bhagwat, A. (2015). The science of static electricity [video file]. Retrieved from https://www. youtube.com/watch?v¼yc2-363MIQs&t¼1s Carney, R. N., & Levin, J. R. (2002). Pictorial illustrations still improve students’ learning from text. Educational Psychology Review, 14(1), 5–26. Cox, E. (2018). Which is stronger: Glue or tape? [video file]. Retrieved from https://www.youtube. com/watch?v¼HHuTrcXNxOk&t¼2s De Koning, B. B., Tabbers, H. K., Rikers, R. M. J. P., & Paas, F. (2009). Towards a framework for attention cueing in instructional animations: Guidelines for research and design. Educational Psychology Review, 21(2), 113–140. https://doi.org/10.1007/s10648-009-9098-7 Doran, Y. J. (2018a). Intrinsic functionality of mathematics, metafunctions in systemic functional semiotics. Semiotica, 2018(225), 457–487. https://doi.org/10.1515/sem-2017-0004 Doran, Y. J. (2018b). The discourse of physics: Building knowledge through language, mathematics and image. New York: Routledge. https://doi.org/10.4324/9781315181134 Doran, Y. J. (2019). Building knowledge through images in physics. Visual Communication, 18(2), 251–277. https://doi.org/10.1177/1470357218759825 Doran, Y. J., & Martin, J. R. (forthcoming). Field relations: Understanding scientific explanations. In K. Maton, J. R. Martin, & Y. J. Doran (Eds.), Studying science: Knowledge, language, pedagogy. London: Routledge. Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Arnold. Halliday, M. A. K., & Matthiessen, C. M. I. M. (2004). An introduction to functional grammar (3rd ed.). London: Hodder Arnold. Hao, J. (2020). Analysing scientific discourse from a systemic functional linguistic perspective. New York: Routledge. He, Y., & van Leeuwen, T. (2019). Animation and the remediation of school physics – A social semiotics approach. Social Semiotics, 1–20.

2 A Functional Perspective on the Semiotic Features of Science Animation

53

Höffler, T. N., & Leutner, D. (2007). Instructional animation versus static pictures: A meta-analysis. Learning and Instruction, 17(6), 722–738. Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). London/New York: Routledge. Leão, G. (2012). A systemic functional approach to the analysis of animation in film opening titles. Sydney, Australia: University of Technology. Lowe, R. (2003). Animation and learning: Selective processing of information in dynamic graphics. Learning and Instruction, 13(2), 157–176. https://doi.org/10.1016/S0959-47520200018-X Lowe, R. (2004). Interrogation of a dynamic visualization during learning. Learning and Instruction, 14(3), 257–274. https://doi.org/10.1016/j.learninstruc.2004.06.003 Lowe, R., & Boucheix, J. (2017). A composition approach to design of educational animations. In R. Lowe & R. Ploetzner (Eds.), Learning from dynamic visualization (pp. 5–30). Cham, Switzerland: Springer. Lowe, R., & Ploetzner, R. (2017). Learning from dynamic visualization: Innovations in research and application. Cham, Switzerland: Springer. Martin, J. R. (1992). English text: System and structure. Amsterdam, The Netherlands: Benjamins. Martin, J. R. (1993). Literacy in science: Learning to handle text as technology. In M. A. K. Halliday & J. R. Martin (Eds.), Writing science: Literacy and discursive power (pp. 166–202). London: Falmer Press. Martin, J. R., & Unsworth, L. (forthcoming). Reading images in school science and history: A systemic functional semiotic perspective on images and knowledge building. Routledge. Martin, J. R., & Zappavigna, M. (2018). Embodied meaning: A systemic functional perspective on paralanguage. Contemporary Rhetoric, 1, 2–33. Maton, K. (2014). Knowledge and knowers. Milton Park, Abingdon, Oxon: Routledge. Mayer, R. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press. Mayer, R. E., Hegarty, M., Mayer, S., & Campbell, J. (2005). When static media promote active learning: Annotated illustrations versus narrated animations in multimedia instruction. Journal of Experimental Psychology: Applied, 11(4), 256–265. McDonnell, R., Breidt, M., & Bülthoff, H. H. (2012). Render me real? Investigating the effect of render style on the perception of animated virtual humans. ACM Transactions on Graphics, 31 (4), 1–11. McMurtrie, R. J. (2017). The semiotics of movement in space: A User’s perspective. New York/ Abingdon, Oxon: Routledge. O’Halloran, K. (2005). Mathematical discourse: Language, symbolism and visual images. London: Continuum. O’Toole, M. (2011). The language of displayed art (2nd ed.). Abingdon, Oxon/New York: Routledge. Ploetzner, R., & Lowe, R. (2012). A systematic characterisation of expository animations. Computers in Human Behavior, 28(3), 781–794. https://doi.org/10.1016/j.chb.2011.12.001 Ploetzner, R., & Lowe, R. (2017). Looking across instead of back and forth: How the simultaneous presentation of multiple animation episodes facilitates learning. In R. Lowe & R. Ploetzner (Eds.), Learning from dynamic visualization (pp. 51–68). Cham, Switzerland: Springer. Scheiter, K. (2017). Design of effective dynamic visualizations: A struggle between the beauty and the beast. In R. Lowe & R. Ploetzner (Eds.), Learning from dynamic visualization (pp. 233–251). Cham, Switzerland: Springer. Smith, D., McLaughlin, T., & Brown, I. (2012). 3-D computer animation vs. live-action video: Differences in viewers’ response to instructional vignettes. Contemporary Issues in Technology and Teacher Education, 12(1), 41–54. Stenglin, M. K. (2004). Packaging curiosities: Towards a grammar of three- dimensional space. University of Sydney. Tversky, B., Morrison, J. B., & Betrancourt, M. (2002). Animation: Can it facilitate? International Journal of Human-Computer Studies, 57, 247–262. https://doi.org/10.1006/ijhc.1017

54

Y. He

Wignell, P., Martin, J. R., & Eggins, S. (1989). The discourse of geography: Ordering and explaining the experiential world. Linguistics and Education, 1(4), 359–391. Zibrek, K., & McDonnell, R. (2014). Does render style affect perception of personality in virtual humans? In Proceedings of the ACM symposium on applied perception (pp. 111–115). New York: ACM.

Yufei He is a lecturer at the School of English for International Business, Guangdong University of Foreign Studies. She received her PhD in linguistics from the University of Sydney. Her research focuses on multimodal discourse analysis, particularly on knowledge building in science animations. She has published relevant research of science animations on Social Semiotics and Semiotica.

Chapter 3

Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis of Ecology Animations for Children Mandy Hoi Man Yu, Dezheng (William) Feng, and Len Unsworth

3.1

Introduction

While science is traditionally seen as a neutral and objective discipline that does not routinely entail the adoption of value positions, many researchers have pointed to the social embeddedness of science (e.g. Matthews, 2015; York & Clark, 2010) and stressed the importance of incorporating values such as democracy and pro-environmental values in science education (e.g. Corrigan, Dillon, & Gunstone, 2007; Dillon & Reid, 2007; Gunstone, Corrigan, & Dillon, 2007). As Gunstone et al. (2007: 4) assert, “it is no longer acceptable to keep science distinct from the society in which it exists, and, consequently, consideration needs to be given to the relationship between the values inherent in society and the values embedded in science”. Against this background, many researchers advocate the integration of science education and environmental education. For example, Dillon & Scott (2002: 1112) argue that the value-laden nature of environmental education is seen as beneficial to science education: Environmental education is uniquely placed to offer science education a range of perspectives on knowledge and situated learning that assist those in the science education movement who wish to challenge existing orthodoxies. Through its multi-disciplinary origins and traditions, environmental education offers a conceptual richness that challenges current thinking in science education.

M. H. M. Yu · D. (W.) Feng (*) Hong Kong Polytechnic University, Hong Kong, China e-mail: [email protected] L. Unsworth North Sydney Campus, Australian Catholic University, North Sydney, NSW, Australia © Springer Nature Switzerland AG 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_3

55

56

M. H. M. Yu et al.

Science education is also widely regarded as a crucial site for the development of environmental awareness and the teaching of environmental values (e.g. Gough, 2002; Hadzigeorgiou & Skoumios, 2013; Littledyke, 2008). A major reason is the scientific basis of environmental issues, i.e. the promotion of environmental awareness and attitude often needs to be supported by scientific principles and evidence (Littledyke, 2008). Ashley (2000: 269) even maintains that ‘science probably offers the strongest justification for the adoption of pro-environmental behaviours and policies.’ Despite its importance, not much research has been done on how environmental values are incorporated in science education, especially in early childhood education; furthermore, most existing studies are concerned with science textbooks or other formal instructional materials (e.g. Chambers, 2008, 2009; Gola, 2017) and to our knowledge, none have analysed online multimedia education materials. With the development of digital technology in recent decades, the use of online environmental and science education materials is becoming increasingly prevalent both inside and outside the classroom, but our understanding of how they recontextualise scientific knowledge is rather limited. For science education, most studies were concerned with the attitude towards the use of online science videos in the classroom (e.g. Buzzetto-More, 2014; Eick & King Jr., 2012) or on the effectiveness of the use (e.g. Everhart, 2009; Jones & Cuthrell, 2011). Only a handful of studies could be found that actually analysed online science videos, and all of these used some form of content analysis. While some looked at the format/style of the videos (e.g. Gustafsson, 2013; Muñoz Morcillo, Czurda, & Robertson-von Trotha, 2016), others focused on different aspects of the content such as the who, where and what of the videos (Chmiel, 2013) and dimensions of science (e.g. pure or applied science) (Christensson & Sjöström, 2014). Chmiel’s (2013) research on teacher-produced science videos on TeacherTube, an online video sharing platform of instructional resources, is one of the few that involved systematic discourse analysis. She first conducted a content analysis on 254 science videos on TeacherTube to investigate a number of parameters such as topics covered, settings and people involved, followed by a more detailed discourse analysis (of both language and visuals). Her analysis revealed that the videos often bore a strong resemblance to popular media narratives for children, as seen, for example, by the use of anthropomorphism and voice acting. In environmental education, many researchers are also well aware of the growing use of online audio-visual resources and emphasise the need to investigate this significant aspect of environmental education. For instance, Ardoina, Clark, and Kelsey’s (2013) meta-study on future trends in environmental education studies interviewed 15 prominent scholars in the field and found that all of them identified social media and information technologies as a crucial direction for further research. However, despite this consensus, these authors observed from their content analysis on articles in environmental education journals a lack of research in that direction. Addressing the need to understand online science education videos, and in particular, how pro-environmental values are constructed in early childhood education through the deployment of linguistic and visual resources, the present study analyses a corpus of 67 online ecology animations for children using a multimodal

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

57

discourse analysis approach. The analysis is guided by the following questions: (1) What pro-environmental values are incorporated in the animations? (2) What attitudinal meanings are used to represent the values? (3) How are the attitudinal meanings realised through multimodal semiotic resources? In what follows, we will first give an overview of research into environmental education. This will be followed by an introduction to our analytical framework, data and methods of analysis. The data will then be analysed and the findings will be discussed in relation to early childhood environmental education.

3.2

Research into Environmental Education

Environmental education (or education for sustainable development) has rapidly grown in global prominence over the past decades, which is largely attributable to the active role of intergovernmental organisations, e.g. the United Nations Educational, Scientific and Cultural Organization (UNESCO) and the United Nations Environmental Programmes (UNEP), in promoting environmental education since the 1970s against the backdrop of pressing global environmental issues, e.g. climate change (see e.g. Biedenweg, Monroe, & Wojcik, 2013; Fauville, Lantz-Andersson, & Säljö, 2014). Environmental education not only aims to increase individuals’ understanding and awareness of the environment, but also to promote pro-environmental values and behaviours. According to UNESCO-UNEP (1976: 2), the goal of environmental education is: To develop a world population that is aware of, and concerned about, the environment and its associated problems, and which has the knowledge, skills, attitudes, motivations and commitment to work individually and collectively toward solutions of current problems and the prevention of new ones.

Due to the growing global prominence of environmental education, the past few decades have witnessed the flourishing of scholarship in the field; however, the teaching of environmental values remains an underexplored topic. The most significant finding our literature review reveals is that most studies on the topic focused on formal instructional materials (e.g. Chambers, 2008, 2009; Gola, 2017; Laçin Şimşek, 2011; Lemoni, Stamou, & Stamou, 2011) and they all came to the conclusion that the materials under examination did not promote environmental ethical values adequately. For example, Gola (2017) conducted a content analysis on primary science textbooks in Poland and found that the representation of the relationship between humans and nature was often underpinned by anthropocentrism, as shown, for example, by the foregrounding of the importance of animals and plants to humans, rather than their inherent values. In line with Gola’s findings, Lemoni et al.’s (2011) picture analysis of primary natural science textbooks in Greece revealed that the textbooks anthropocentrically represented humans as exerting control over natural environments and their interventions in nature as ‘normal’ and ‘expected’.

58

M. H. M. Yu et al.

Some research investigated environmental values from the perspective of critical discourse studies (e.g. Sharma & Buxton, 2015; Stibbe, 2004; Xiong, 2014), often focusing on the phenomenon of ‘shallow environmentalism’ in textbooks, i.e. ‘react [ing] to ecological destruction by addressing immediate physical symptoms (such as acid rain or depletion of the ozone layer), but refus[ing] to address the underlying cultural, political and psychological causes’ (Stibbe, 2004: 243). For example, Xiong (2014) analysed 28 state-sanctioned English language textbooks in China and identified three major features, namely (1) the obscuration of human agency in ecological destruction; (2) the overuse of the problem-solution structure, which often directed learners’ attention to trivialised individual or technological solutions; and (3) the positive construction of the Chinese government’s active participation in environmental protection. His critique emphasised that such shallow environmentalism could not help to foster pro-environmental behaviour. In line with Xiong’s findings, Sharma and Buxton’s (2015) study of the representation of the relationship between humans and nature in a seventh-grade science textbook in the US revealed that the textbook often obscured human agency in environmental degradation and avoided mentioning the serious impact of various environmental problems on marginalised communities. They urged a revision of the representation to better reflect the present understanding of nature. Apart from the teaching of environmental values, conspicuously neglected in environmental education scholarship is early childhood education (see e.g. Davis, 2009; Davis, Engdahl, Otieno, Pramling-Samuelsson, Siraj-Blatchford & Vallabh, 2008; Davis & Elliott, 2014), although it is widely believed that early childhood constitutes a critical period for the fostering of pro-environmental values and attitude (see Tilbury, 1994; Tsantopoulos, Skanavis, Bantoudi, Petkou, & Dalamagkidou, 2018). Hedefalk, Almqvist, and Östman’s (2015) meta-research on environmental education found that the topic was mostly investigated theoretically and called for more empirical research on the teaching and learning of environmental education in early childhood education.

3.3

Analytical Framework

Our analytical approach is fundamentally based on Martin and White’s (2005) attitude framework, a systemic framework set within their appraisal theory which models attitudinal meaning at the stratum of discourse semantics. Martin and White see attitude as ‘ways of feelings’ (ibid: 42) and distinguish three types of attitude, namely affect, judgement and appreciation. Affect is concerned with emotion, which can be subcategorised into dis/inclination, un/happiness, dis/satisfaction and in/security. Judgement is attitude to behaviours with five subtypes, namely normality (i.e. how special one is), capacity (i.e. how capable one is), tenacity (i.e. how dependable one is), veracity (i.e. how honest one is) and propriety (how ethical one is). Appreciation is attitude to objects/phenomena, which can be divided into reaction (i.e. whether something catches our attention or pleases us), composition

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

59

literal explicit articulated ATTITUDE

spoken/written language

figurative eliciting condition

implicit

resultant action

character design embedded multimodal resources

narrative design

narrative process conceptual process manichean structure graduated structure

Fig. 3.1 The multimodal realisation of attitude (The curly bracket denotes simultaneous choices, while the square bracket denotes either-or choices)

(i.e. whether something hangs together or is easy to follow) and valuation (i.e. whether something is significant). The system has proved a useful analytical tool and has been applied to various types of discourse, such as academic writing (e.g. Hood, 2010; Lee, 2015), translation texts (e.g. Munday, 2012) and social media (e.g. Zappavigna, 2012). It has also been extended to investigate the multimodal construction of attitudinal meanings in different genres such as films (Feng, 2012), TV shows (Pounds, 2012), newspapers (Economou, 2009), picture books (Unsworth, 2015) and textbooks (Feng, 2019). For example, Unsworth (2015), through analysing evaluative images in picture books and animated movies, distinguishes explicit visual inscription from a range of strategies for implicit invoking of judgement in images. Feng (2019) considers social values as categories of judgement and proposes a framework to model how they are constructed linguistically and visually in English language textbooks in Hong Kong. We also propose a framework to deal with the multimodal construction of attitude which brings together the resources of language, visual depictions and narrative design based on Feng (2012, 2016), as illustrated in Fig. 3.1. The model first distinguishes between articulated and embedded attitudes. The difference between the two lies in whether an attitude is expressed verbally by characters or is represented through characters’ actions or the story plot. The most explicit realisation of attitudes is attitudinal lexis such as ‘happy’, ‘generous’ and ‘important’. Aside from literal expressions, attitudes can be articulated through figurative language, in particular, metaphors. Metaphors make language a more powerful tool for encoding abstract attitudinal concepts, as is advocated by the conceptual metaphor theory (Lakoff & Johnson, 1980). Attitude can be expressed implicitly in two ways: by recounting or depicting events that elicit the attitude (eliciting condition) or by saying/doing something that is motivated by the attitude (resultant action). For example, by saying ‘human beings killed my daddy for his skin’, a baby crocodile implicitly expresses its feeling of sadness or anger, and its negative judgement on human behaviour. Appreciation of the value of water can be implicitly articulated by describing its functions (eliciting condition) and recommending ways to conserve water (resultant action). In Text 1, both explicit

60

M. H. M. Yu et al.

Fig. 3.2 Example of embedded affect and judgement (narrative process). (Drawing after the screenshot from ‘Why do animals go extinct’)

and implicit expressions can be identified. The surfer represents the eliciting condition (i.e. littering) that leads to the negative judgement of irresponsibility, which is also explicitly expressed with the attitudinal lexis ‘irresponsible’. The judgement also results in a verbal action which is realised by a directive – ‘We should all take better care of this beach’. Text 1: I was in the water ready to stand upon my surfboard, when I heard a lady surfer scream. To my surprise, there was litter everywhere. There was even some on my surfboard. It was gross! The lifeguards put up ‘No Surfing’ flags because irresponsible people littered. We should all take better care of this beach! (from ‘All the way to the ocean’)

Unlike articulated attitudes, embedded attitudes are represented through the character design or the narrative design. The character design refers to characters’ actions and reactions (narrative process) and the characterisation of characters through colours, size, clothing and so on (conceptual process). In terms of the narrative process, affect is most typically embedded in characters’ facial expressions, and judgement is often realised through various actions of characters (e.g. littering, killing animals, polluting, etc.). For example, in Fig. 3.2, the tiger’s fear is embedded in its facial expression and its action of hiding behind a rock. Meanwhile the action of the poacher may invoke judgement of impropriety from viewers. Judgement can also be embedded in the conceptual process. An example can be found in an animation about the story of Ecoman, the hero, and Dread It Man, the villain. The animation teaches children about pro- and anti-environmental behaviours by inter alia visually characterising Ecoman as a Superman in a green costume (signalling his green mission) and Dread It Man as a supervillain in a grey hood (signalling his anti-environmental behaviours) (see Fig. 3.3). In terms of appreciation, owing to the frequent use of anthropomorphism in our data, animals and objects often exhibit human qualities, and perform all sorts of human actions in which attitude and values

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

61

Fig. 3.3 Example of embedded judgement (conceptual process). (Screenshot from ‘Sustainable development for kids (by Grade 3 kids)’, reproduced with permission)

are embedded. For example, an animation evaluates greenhouse gases as harmful by visually characterising them as a monster-cum-devil (with horns and only one eye) (i.e. the conceptual process) and representing the creature as bullying the personified earth (i.e. the narrative process). Attitude, in particular values related to judgement, can also be realised by two types of narrative design: the Manichean structure and the graduated structure (Feng, 2012; Smith, 1995). The Manichean structure dichotomises characters into ‘good’ and ‘evil’ ones, with the former defeating the latter in the end. This can be illustrated by the abovementioned animation about Ecoman and Dump It Man. In the story, Dump It Man has created a series of environmental problems, e.g. air pollution, at a school, but every time Ecoman comes to the rescue of the pupils and teaches them a wealth of ecology knowledge, so that they can defeat Dump It Man. Unlike the Manichean structure, in the graduated moral structure, characters often exhibit mixed or changing moral attributes. For example, in an animation about water pollution, a boy throws away a plastic bottle into the storm drain without any idea of its environmental impact. After talking with a crane and seeing how rubbish thrown into the storm drain all goes straight to the ocean and impacts negatively on animal and marine life, he regrets what he did and proposes a clean-up campaign at his school.

62

3.4

M. H. M. Yu et al.

Data and Methods of Analysis

After describing our analytical framework, this section introduces our dataset and explains our data analysis. Our data consists of 67 online ecology animations for children (more than 4.5 h in total), all being collected from YouTube. In order to do justice to the broad scope of ecology, our data covers a wide range of topics related to ecology such as natural resources (e.g. trees), biomes (e.g. rainforests), ecological phenomena (e.g. animal extinction), environmental problems (e.g. climate change), and conservation (e.g. sustainable development). Most of the animations were produced by educational channels for children, but we also tried to include other types of animations, including user-generated videos, and those by non-governmental organisations and private companies (see Table 3.1). For videos not produced by channels for children, we only included those which provide clear clues that they are targeted at children. For example, some animation producers explicitly specify their target audience in the animation title, e.g. ‘Global warming video for kids’. Our data analysis involved two main steps. The first step was to identify all instances of attitude and categorise them as affect, judgement and appreciation, including their subtypes, based on Martin and White’s (2005) attitude system, and analyse their patterns of use. The second step dealt with how the attitude is realised through multimodal resources in the animations. We categorised all realisations of attitude based on the framework in Fig. 3.1 and examined how often and in what way each type of realisations is used.

3.5

Attitude in Ecology Animations for Children

This section will start by presenting some overall trends in the construction of attitude before proceeding to a more detailed analysis of the patterns of use of different types of attitude. Table 3.2 presents the overall distribution of appreciation, judgement and affect, including their subtypes. Two overall trends have been observed. First, all the animations contain attitudinal elements. That is to say, they Table 3.1 Sources of videos Source Educational channels Individuals NGOs Private companies Others

Examples Learning Junction, Smart Learning for All

No. 46

NA CAFOD, Greenpeace 25SDA (a speed drawing animation company), Natracare (a personal health and hygiene company) United Nations, Australian government, United States Agency for International Development

10 4 4 3

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

63

Table 3.2 Distribution of attitude Appreciation Valuation (+) Reaction (+) Reaction (
) Composition (
) Subtotal

78 38 58 1 175

Judgement Propriety (+) Propriety (
) Capacity (
) Tenacity (
) Subtotal

5 99 1 2 107

Affect Happiness (+) Happiness (
) Inclination (+) Inclination (
) Security (
) Subtotal

12 23 2 9 3 49

Total 331

not only teach children facts about the environment, but also construct pro-environmental attitude and values. Second, the most prominent types of attitude are positive appreciation of nature (116 instances) and negative judgement of human behaviours (102 instances). Appreciation is the most frequently represented category of attitude. The most remarkable feature of the use of appreciation is that in contrast to the overall dominance of negative attitudes, most instances of appreciation are positive (116 out of the 175 instances). A close reading of the instances shows that positive valuation, which accounts for almost half of the instances of appreciation, mainly functions to highlight the value of different parts of nature (28 cases), especially energy and resources, e.g. wind energy and sunlight, as well as living organisms (27 cases), e.g. trees and rainforests. For example, Text 2 represents the importance of the ozone layer. The other half of the instances of appreciation fall into the reaction subtype (58 negative and 38 positive instances). The use of negative appreciation of reaction centres upon the harmful and undesirable impact of various environmental problems such as pollution and animal extinction (e.g. ‘soil pollution is very harmful for everybody’). Positive appreciation of reaction, on the other hand, mainly functions to foreground the beauty of nature and the desirability of different forms of energy and resources, e.g. tidal power and wind. For example, Text 3 evaluates the sun as ‘pretty awesome’. Text 2: Life on Earth is protected from the UV rays by a layer in the stratosphere called the ozone layer. Ozone is a gas made up of three oxygen atoms. This layer is just about three to five millimetres thick. This thinly spread out gas has been protecting life on the Earth’s surface from UV rays for billions of years. (from ‘What is ozone layer?’) Text 3: And because it [the sun] is in the centre of our solar system, the light from the sun heats the earth. It’s pretty awesome. (from ‘Global warming for kids’)

Judgement also plays a crucial role in the construction of pro-environmental values. The most important finding in this regard is that all but three instances are of the propriety subtype (104 cases), among which 99 are negative. The predominance of judgement of propriety is unsurprising considering that pro-environmental values are largely underpinned by environmental ethics, which governs what we, as morally responsible beings, can(not) do to nature (see Taylor, 2011). The negative judgement of propriety either targets environmental disturbances caused by humans (66 cases), such as building factories, or personal habits (33 cases), such as littering. For

64

M. H. M. Yu et al.

example, Text 4 makes a judgement of impropriety on how different human activities have disturbed the carbon cycle. A noteworthy point is that in contrast to the obscuration of human agency in environmental destruction in many instructional materials for environmental education (see Sharma & Buxton, 2015; Xiong, 2014), our analysis shows that humans are explicitly held responsible for various environmental problems in 75 out of the 99 judgements of impropriety, as can be seen in Texts 4–5 and Fig. 3.2. Concerning the five positive judgements of propriety, they all co-occur with negative judgements of propriety and serve contrastive purposes, e.g. between good and bad behaviours/practices. For instance, as shown in Text 5, after seeing a hunter killing a tiger in a forest, a girl explains to her brother that animals are hunted because many people buy products made of animal skins to decorate their home. Her brother immediately evaluates the behaviour as ‘not good’ and suggests that we should never buy anything made of animal skins, which leads to the positive judgement of ‘very kind’ by his sister. Such co-occurrences of positive and negative judgement of propriety function to teach children to distinguish between acceptable and unacceptable behaviours. Text 4: Is it getting hotter? It’s because we humans are disturbing the carbon cycle. As we burn more and more fuel, coal, petrol diesel and gas, more and more of the locked-up carbon gets released in the air . . . (from ‘The carbon cycle, carbon dioxide-cycle’) Text 5: Girl: . . . These hunters kill tigers and lions too to get their skins. Many people buy these skins to decorate their homes. Boy: That is not good. We should never buy anything that is made of animal skins. Girl: You are very kind. (from ‘Endangered and extinct animals’)

Affect appears much less frequently in the animations (49 cases). A striking finding about the use of affect is that with the exception of five instances, all affect is appraised by non-human characters, mostly animals (e.g. pandas), and different parts of nature (e.g. trees). In most cases, such affect is represented as a response of non-human beings to human intrusion into their habitat or ecosystem, as shown by the larger proportion of negative affect (35 out of the 49 instances). Among the 35 instances of negative affect, 23 fall into the unhappiness subtype, with images of crying animals, trees, and so on being often found. Another major finding about the use of affect is that happiness and unhappiness often co-occur (9 cases) to contrast between how different non-human characters feel before and after human intrusion/ evolution or before and after the implementation of various eco-friendly practices. For example, an animation about air pollution features the personified earth recounting her life before and after human evolution. It visually contrasts a happy earth pre-human-evolution and an unhappy one post-human-evolution. Apart from unhappiness, insecurity is also sometimes drawn on (9 cases), and this is most often used to represent frightened animals and plants and to foreground the vulnerability and helplessness of animals and plants in the face of environmental changes and various human activities, e.g. cutting trees and poaching. Therefore, despite its relatively low proportion, affect serves as an important strategy to give voice to animals and other parts of nature and elicit feelings of moral obligation to protect them.

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

3.6

65

The Multimodal Realisation of Attitudes

This section elucidates how attitude is constructed with multimodal resources. Categorising all realisations of attitude based on the framework in Fig. 3.1 yielded the results in Table 3.3. Since the realisations of appreciation and judgement exhibit similar patterns, we will discuss them together before proceeding to those of affect. The most salient feature about the realisations of appreciation and judgement is that they are much more often articulated than embedded in characters’ attributes or the narrative design. Articulated appreciation and judgement are mostly constructed implicitly, especially through eliciting condition, with 94 out of the 175 appreciations and 79 out of the 107 judgements being constructed in this way. In these cases, eliciting condition mostly serves to provide scientific reasoning of the evaluations, more specifically, explaining why different parts of nature are valuable and desirable and why different environmental problems are harmful in terms of appreciation, and why various human behaviours or practices are (im)moral according to environmental ethics in terms of judgement. For example, Text 2 above explains what the ozone layer is and its functions, which elicits positive valuation of the ozone layer, whereas Texts 4 and 5 describe how humans disturb the carbon cycle and kill animals for their skins, which elicit judgement of impropriety. Apart from eliciting condition, resultant behaviour also forms an important part of the construction of appreciation and judgement. A major finding in this regard is that resultant behaviour tends to co-occur with eliciting condition. This pattern can be found in 31 out of the 48 appreciations and 43 out of the 51 judgements and can be illustrated by the resultant behaviour in Text 6, which is preceded by Text 2. Text 2, as mentioned, explains the functions of the ozone layer and elicits positive appreciation of valuation, which motivates the resultant behaviour in Text 6, i.e. suggesting ways to stop ozone depletion. This pattern of explanation and Table 3.3 Realisations of attitudes Articulated Explicit Attitudinal lexis Metaphors Implicit Eliciting condition Resultant behaviour Embedded Character attributes Narrative process (actions, change of conditions) Conceptual process (e.g. clothing) Narrative design Manichean moral structure Graduated moral structure

Appreciation

Judgement

Affect

84 0

10 0

10 0

94 48

79 51

3 2

10 6

13 3

39 0

2 2

1 5

0 0

66

M. H. M. Yu et al.

recommendations/instructions is prevalent in the construction of appreciation (17 cases) and judgement (15 cases). In line with this finding, it is found that resultant action is mainly in the form of directive speech acts and serves to recommend or suggest ways to protect the environment (appearing in 30 appreciations and 21 judgements). In other words, explaining and recommending are often combined to disseminate environmental knowledge to children on the one hand, and to make them adopt more eco-friendly practice on the other. Text 6: Can we stop the depletion of ozone layer? Yes, we can. All we have to do is to reduce the production of those chemicals that cause the destruction of ozone like CFCs and nitrogen oxides, so encourage your parents, relatives and friends to make sure their refrigerators and air conditioners do not have CFCs. (from ‘What is ozone layer?’)

Despite many similarities between how appreciation and judgement are realised, the two exhibit apparent differences when it comes to explicit articulation. This type of realisation, all in the form of attitudinal lexis, is used much more frequently in the construction of appreciation (84 out of the 175 instances), compared with that of judgement (10 out of the 107 instances). This might be due to the widely perceived neutrality of science. Some animations might deliberately avoid ascribing subjective qualities to human characters and opt for implicit articulation of judgement. The explicit articulation of appreciation mostly occurs when characters/narrators share their opinion on the value or beauty of different aspects of nature (e.g. using words such as ‘nice’, ‘awesome’, and ‘important’), or their reaction to various environmental issues (e.g. ‘serious’, ‘dangerous’, and ‘bad’). Compared with articulated attitude, embedded attitude plays a less important role in the construction of appreciation and judgement, with only 13 and 19 out of the 67 animations containing embedded appreciation and embedded judgement respectively. This low proportion is to a certain extent because some animations do not fully utilise the visual affordance of online videos. For example, an animation features several animals taking turns to talk about the importance of trees and relies only on the verbal content to construct attitudinal meanings. Within the small amount of embedded appreciation and judgement, a major finding is that this type of realisation, including all its subtypes, most typically shapes opinions on objects, phenomena or behaviours through stories or imagined scenarios. Most embedded appreciation and judgement rely on the construction of character attributes. We have identified 10 instances of appreciation and 13 instances of judgement embedded in the narrative process, and 6 instances of appreciation and 3 instances of judgement embedded in the conceptual process. In terms of appreciation embedded in the narrative process, half are embedded in the actions of personified animals/objects, e.g. evil greenhouse gases bullying the earth, whilst half are embedded in the change of conditions of different parts of nature. For example, one video visually depicts how soil pollution has changed the originally fertile land with healthy trees to a barren one with bare trees, which invokes the harm of soil pollution. In terms of judgement embedded in the narrative process, most instances depict (often visually) what humans have done to nature and its devastating consequences. For example, an animation visually represents a boy swimming in the sea and seeing how much it has

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

67

been polluted by human activities (e.g. household garbage and industrial waste) and how much the marine life suffers (e.g. ailing fish and crying water). Turning to the conceptual process, appreciation of this type all makes use of personification. Four out of the six instances visually characterise greenhouse gases as devils/monsters, whereas interestingly, all the three judgements of this type promote pro-environmental behaviours through the costume and heroic image of Superman, with two of them even named ‘Ecoman’. Lastly, only four instances of appreciation and six instances of judgement are embedded in the narrative design. The four instances of appreciation (all involving personified objects) are evenly distributed between the two moral structures. For example, an appreciation embedded in the graduated moral structure can be found in an animation about the advantages of electric cars and the disadvantages of petrol cars through the story of Buzz, an electric car living with other electric cars in an eco-friendly town called Hopetown. One day, Buzz accidentally drives to a polluted town full of petrol cars called Gloomsburg and is tempted to have its electric engine replaced with a gas engine and to roar like petrol cars, but later comes to the realisation of the negative impact this will have on the environment and itself and decides to return to Hopetown. As is clear from the story plot, while Buzz is an eco-friendly electric car, it once gets into bad company (petrol cars) and is almost converted to evil (an anti-environmental car). As regards the Manichean moral structure, it can, for example, be identified in an animation that metaphorically represents a war between UVs and CFCs (evil), and a troop of ozone defenders who protect humans from the sun’s rays (good). In the end, an ozone defender teaches a child about the danger of UVs and successfully makes him spread the word and stop his father from breaking his old CFC refrigerator and hence from weakening the ozone layer. Different from appreciation, the construction of judgement relies more on the graduated structure (5 instances), with only one judgement embedded in the Manichean structure. The two types of moral structure can be seen in the little boy who regrets his anti-environmental behaviour and takes initiative to protect the environment after talking to a crane, and the story of Ecoman and Dump It Man, as discussed when introducing the framework. The small number of instances of attitude embedded in the narrative design may be because this strategy seems to be more applicable to the evaluation of human characters (cf. Feng, 2012; Smith, 1995), while most animations focus on nature, rather than humans. Unlike appreciation and judgement, affect is most often constructed through embedding in the narrative process (39 instances), with all but two being realised through the facial expressions, gestures or actions of non-human characters (e.g. a crying sea turtle and a smiling planet). Embedded affect often elicits moral responsibility and empathy towards non-human characters. For example, in Fig. 3.2, the action of the poacher (i.e. shooting the tiger) invokes negative judgement of propriety and the tiger’s fear invokes empathetic feelings from viewers. Articulated affect is drawn on much less frequently, being found in only ten animations, all involving explicit articulation, either via affective lexis (7 instances), such as ‘sad’ and ‘love’, or emojis (3 instances). Such explicitly articulated affect mainly functions for human

68

M. H. M. Yu et al.

characters to express their unhappiness caused by pollution, love for nature and enthusiasm for environmental protection (9 instances).

3.7

Discussion and Conclusions

In this final section, we will first recapitulate our major findings and discuss them from the perspective of environmental education; after that, we will conclude our research by pointing to some directions for further research. Our attitudinal analysis showed that all the animations incorporate pro-environmental elements into the teaching of science. The construction of pro-environmental values relies most heavily on appreciation, especially the valuation and reaction subtypes, to highlight the value and desirability of nature and the harm of various environmental problems. In addition, negative judgement of propriety is often used to foreground how humans have destroyed the environment. Affect in the form of unhappiness is used to give voice to non-human characters to respond to human intrusion into their habitat/ ecosystem. The characters’ feelings before and after human intrusion, and before and after the implementation of eco-friendly practices are often contrasted to highlight the consequences of destructive behaviours and the value of pro-environmental behaviours. To further understand the pro-environmental values reflected in the attitudes, we map the attitudes identified in our analysis onto biospheric, altruistic and egoistic values proposed by Stern and Dietz (1994) (see also de Groot & Steg, 2009; Schultz, 2000). Egoistic values place self-benefits as the first consideration and lead to pro-environmental behaviours which have direct impacts on the individual. For example, one may save power in order to lower her/his electricity bill. Altruistic and biospheric values correspond to the human-centred (or anthropocentric) and lifecentred (or biocentric) systems of ethics in Taylor’s (2011) theory of environmental ethics, i.e. the beliefs that we are morally obliged to protect the environment for the well-being of other people (including our future generations) and that of animals and plants, respectively. The results of our analysis obviously point to the predominance of biospheric values (276 out of the 364 instances),1 which applies to all the three types of attitude. This means that the animations most often take a biocentric standpoint, representing humans as morally responsible for all living beings on earth by virtue of their inherent worth (Taylor, 2011). Biospheric values form the foundation of the majority of expressions of affect and judgement, with 42 out of the 49 instances of affect (85.7%) and 95 out of the 116 judgements (81.9%) taking a biocentric stance. As mentioned, affect is often employed to give voice to non-human beings to respond to humans’ cruel treatment of them, while judgement of propriety is frequently used to criticise human disturbances of nature based on

1 The total number of environmental values is different from the total instances of attitude because one instance of attitude does not necessarily map onto only one type of pro-environmental values.

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

69

environmental ethics, and to educate children about what they should (not) do to take care of the well-being of nature. In the case of appreciation, biospheric values also carry crucial weight (69.8% or 139 out of the 199 cases). Most biospheric appreciations draw attention to the inherent values and desirability of nature (e.g. Text 3). A much smaller proportion of attitude is driven by altruism. Only about 29.1% of appreciation (58 instances) exhibits altruistic values, mostly to highlight the importance of nature and the harmful and undesirable effect of different environmental problems from the perspective of humans. Altruistic judgement and affect account for even lower proportions, namely 17.2% (20 instances) and 14.3% (7 instances) respectively. Altruistic judgement mainly represents what should (not) be done in consideration of the well-being of humans (e.g. ‘when we cut trees, we get less fresh and clean air’), whereas altruistic affect mostly visually represents happy human characters enjoying nature or unhappy ones being disturbed by pollution. Lastly, there is only an extremely low proportion of egoistic values (2 appreciations and 1 judgement), probably because these kinds of pro-environmental values are all about how protecting the environment benefits us as individuals, mostly financially, and such financial concern is less relevant to children. Turning to the multimodal realisation of pro-environmental values, most appreciation and judgement are wholly or partially realised by eliciting conditions and resultant behaviour (often co-occurring), which mainly function to provide scientific explanations and recommend ways to solve the environmental problem concerned respectively. Within the small number of instances of embedded appreciation and judgement, almost all serve to shape opinions about objects, phenomena, or behaviours via stories or imagined scenarios. Affect, on the other hand, is predominantly embedded in the narrative process, mainly serving to elicit moral obligation and protective feelings towards nature. Affect is also sometimes explicitly articulated by human characters to express their unhappiness caused by pollution, love for nature and enthusiasm for environmental protection. The above findings clearly show the pedagogical value of online ecology animations for environmental education. First, many online ecology animations follow environmental ethics closely and highlight humans’ moral responsibility to protect and respect nature. This can be seen in the use of appreciation and judgement to foreground the inherent value of nature, as well as human disturbances and their consequences for nature. Non-human characters are also given voice to express their emotions and attitude towards human intrusion into their habitat/ecosystem. The frequent construction of animals and other parts of nature as agents of judgement and affect also serves to encourage children to think from a more ecological perspective and to elicit their feelings towards nature. Such biocentric representation contrasts sharply with the heavy reliance on anthropocentrism in many formal environmental education teaching materials, e.g. science textbooks (see e.g. Gola, 2017) and English language textbooks (see e.g. Xiong, 2014). We thus think that online ecology animations are more appropriate teaching and learning materials for environmental education from the environmental-ideological standpoint. We also regard online ecology animations as pedagogically sound materials for environmental education because many animations in our dataset successfully

70

M. H. M. Yu et al.

address both the knowledge- and action-oriented dimensions of environmental education. As indicated in our analysis of the multimodal realisation of attitude, the animations often support their construction of pro-environmental values with scientific explanations (via eliciting condition) to ensure that viewers understand why certain behaviour is (not) good and why different parts of nature are significant; in the meantime, many animations also seek to transform viewers’ behaviour by giving suggestions/instructions on how to solve or lessen various environmental issues (via resultant actions). This balance between the knowledge- and actionoriented dimensions of environmental education shows that many online ecology animations do justice to the goal of environmental education, i.e. increasing individuals’ understanding and awareness of the environment and fostering pro-environmental attitude and behaviours. Focusing more specifically on early childhood environmental education, our findings reveal the animations’ frequent use of anthropomorphism, which is widely viewed as beneficial to science/environmental learning for young children, e.g. fostering empathy for nature (Gebhard, Nevers, & Billmann-Mahecha, 2003) and engaging children (Geerdts, 2014). Anthropomorphism in our data is mostly used for the visual construction of embedded attitudes, which often explains abstract ideas or phenomena through narratives or imagined scenarios and makes them easier to understand for children, e.g. via the use of a devil-cum-monster character bullying the earth to represent the harmful effect of greenhouse gases. Furthermore, embedded affect is also often drawn on to give voice to non-human characters to emotionally respond to human intrusion into their habitat or ecosystem. Such emotional responses represent humans’ cruelty in an easily understood yet compelling way, invoking guilt and empathy for nature from young children. We therefore consider the use of anthropomorphism as an important strategy to construct embedded attitude in online ecology animations for children. However, despite the widespread use of anthropomorphism, embedded attitude is not used very widely in the animations, as mentioned above. We suggest that science education video producers should make better use of the visual affordance of the multimodal genre to facilitate children’s understanding and learning. Further research can explore how far online ecology videos facilitate children’s science learning and how far they affect children’s perception of nature. It would also be interesting to explore the construction of environmental values, or other values, in other online education genres, or for older students or adults.

References Ardoina, N. M., Clark, C., & Kelsey, E. (2013). An exploration of future trends in environmental education research. Environmental Education Research, 19(4), 499–520. https://doi.org/10. 1080/13504622.2012.709823 Ashley, M. (2000). Science: An unreliable friend to environmental education? Environmental Education Research, 6(3), 269–280. https://doi.org/10.1080/713664678

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

71

Biedenweg, K., Monroe, M. C., & Wojcik, D. J. (2013). Foundations of environmental education. In M. C. Monroe & M. E. Krasny (Eds.), Across the spectrum: Resources for environmental educators (pp. 9–27). Washington, DC: North American Association for Environmental Education. Buzzetto-More, N. A. (2014). An examination of undergraduate student’s perceptions and predilections of the use of YouTube in the teaching and learning process. Interdisciplinary Journal of E-Learning and Learning Objects, 10, 17–32. Chambers, J. M. (2008). Human/nature discourse in environmental science education resources. Canadian Journal of Environmental Education, 13(1), 107–121. Chambers, J. M. (2009). Critical discourse analysis: A research methodology for the analysis of environmental education materials. In D. B. Zandvliet (Ed.), Diversity in environmental education research (pp. 131–146). Rotterdam, The Netherlands: Sense Publishers. Chmiel, M. M. (2013). Science on TeacherTube: A mixed methods analysis of teacher produced video. (Doctoral thesis). George Mason University, Fairfax. Christensson, C., & Sjöström, J. (2014). Chemistry in context: Analysis of thematic chemistry videos available online. Chemistry Education Research and Practice, 15(1), 59–69. https://doi. org/10.1039/c3rp00102d Corrigan, D., Dillon, J., & Gunstone, R. (Eds.). (2007). The re-emergence of values in science education. Rotterdam, The Netherlands: Sense Publishers. Davis, J. M. (2009). Revealing the research ‘hole’ of early childhood education for sustainability: A preliminary survey of the literature. Environmental Education Research, 15(2), 227–241. https://doi.org/10.1080/13504620802710607 Davis, J. M., & Elliott, S. (2014). An orientation to early childhood education for sustainability and research: Framing the text. In J. M. Davis & S. Elliott (Eds.), Research in early childhood education for sustainability: International perspectives and provocations (pp. 1–17). Abingdon, Oxon; New York: Routledge. Davis, J. M., Engdahl, I., Otieno, L., Pramling-Samuelsson, I., Siraj-Blatchford, J., & Vallabh, P. (2008). The Gothenburg recommendations on education for sustainable development. Gothenburg, Gothenburg: Swedish International Centre for Education for Sustainable Development (SWEDESD); Chalmers University & Gothenburg Universities. de Groot, J. I. M., & Steg, L. (2009). Mean or green: Which values can promote stable pro-environmental behavior? Conservation Letters, 2(2), 61–66. https://doi.org/10.1111/j. 1755-263X.2009.00048.x Dillon, J., & Reid, A. (2007). Science, the environment and citizenship: Teaching values at the Minstead study Centre. In D. Corrigan, J. Dillon, & R. Gunstone (Eds.), The re-emergence of values in science education (pp. 77–88). Rotterdam, The Netherlands: Sense Publishers. Dillon, J., & Scott, W. (2002). Perspectives on environmental education-related research in science education. International Journal of Science Education, 24(11), 1111–1117. https://doi.org/10. 1080/09500690210137737 Economou, D. (2009). Photos in the news: Appraisal analysis of visual semiosis and visual-verbal intersemiosis. (PhD doctoral thesis), University of Sydney, Sydney. Eick, C. J., & King Jr., D. T. (2012). Nonscience majors’ perceptions on the use of YouTube video to support learning in an integrated science lecture. Journal of College Science Teaching, 42(1), 26–30. Everhart, J. (2009). YouTube in the science classroom. Science and Children, 46(9), 32–35. Fauville, G., Lantz-Andersson, A., & Säljö, R. (2014). ICT tools in environmental education: Reviewing two newcomers to schools. Environmental Education Research, 20(2), 248–283. https://doi.org/10.1080/13504622.2013.775220 Feng, D. (2012). Modeling appraisal in film: A social semiotic approach. (Doctoral dissertation). National University of Singapore, Singapore. Feng, D. (2016). Promoting moral values through entertainment: A social semiotic analysis of the spring festival gala on China central television. Critical Arts, 30(1), 87–101. https://doi.org/10. 1080/02560046.2016.1164387

72

M. H. M. Yu et al.

Feng, D. (2019). Infusing moral education into English language teaching: An ontogenetic analysis of social values in EFL textbooks in Hong Kong. Discourse: Studies in the Cultural Politics of Education, 40(4), 458–473. Gebhard, U., Nevers, P., & Billmann-Mahecha, E. (2003). Moralizing trees: Anthropomorphism and identity in children’s relationships to nature. In S. Clayton & S. Opotow (Eds.), Identity and the natural environment: The psychological significance of nature. Cambridege, MA: Massachusetts Institute of Technology. Geerdts, M. (2014). Anthropomorphic media and children’s biological knowledge. (Doctoral thesis). Rutgers, The State University of New Jersey, New Jersey. Gola, B. (2017). Is formal environmental education friendly to nature? Environmental ethics in science textbooks for primary school pupils in Poland. Ethics and Education, 12(3), 320–336. https://doi.org/10.1080/17449642.2017.1343619 Gough, A. (2002). Mutualism: A different agenda for environmental and science education. International Journal of Science Education, 24(11), 1201–1215. https://doi.org/10.1080/ 09500690210136611 Gunstone, R., Corrigan, D., & Dillon, J. (2007). Why consider values and the science curriculum? In D. Corrigan, J. Dillon, & R. Gunstone (Eds.), The re-emergence of values in science education (pp. 1–10). Rotterdam, The Netherlands: Sense Publishers. Gustafsson, P. (2013). How physics teaching is presented on YouTube videos. Educational Research for Social Change, 2(1), 117–129. Hadzigeorgiou, Y., & Skoumios, M. (2013). The development of environmental awareness through school science: Problems and possibilities. International Journal of Environmental and Science Education, 8(3), 405–426. https://doi.org/10.12973/ijese.2013.212a Hedefalk, M., Almqvist, J., & Östman, L. (2015). Education for sustainable development in early childhood education: A review of the research literature. Environmental Education Research, 21 (7), 975–990. https://doi.org/10.1080/13504622.2014.971716 Hood, S. (2010). Appraising research: Evaluation in academic writing: Evaluation in academic writing. London: Palgrave. Jones, T., & Cuthrell, K. (2011). YouTube: Educational potentials and pitfalls. Computers in the Schools, 28(1), 75–85. https://doi.org/10.1080/07380569.2011.553149 Laçin Şimşek, C. (2011). Investigation of environmental topics in the science and technology curriculum and textbooks in terms of environmental ethics and aesthetics. Educational Sciences: Theory and Practice, 11(4), 2252–2257. Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press. Lee, S. H. (2015). Evaluative stances in persuasive essays by undergraduate students: Focusing on appreciation resources. Text & Talk, 35(1), 49–76. https://doi.org/10.1515/text-2014-0029 Lemoni, R., Stamou, A. G., & Stamou, G. P. (2011). ‘Romantic’, ‘classic’ and ‘baroque’ views of nature: An analysis of pictures about the environment in Greek primary school textbooks – Diachronic considerations. Research in Science Education, 41(5), 811–832. https://doi.org/10. 1007/s11165-010-9191-4 Littledyke, M. (2008). Science education for environmental awareness: Approaches to integrating cognitive and affective domains. Environmental Education Research, 14(1), 1–17. https://doi. org/10.1080/13504620701843301 Martin, J. R., & White, P. R. R. (2005). The language of evaluation: Appraisal in English. Basingstoke, UK: Palgrave Macmillan. Matthews, B. (2015). The elephant in the room: Emotional literacy/intelligence, science education, and gender. In D. Corrigan, C. Buntting, J. Dillon, A. Jones, & R. Gunstone (Eds.), The future in learning science: What’s in it for the learner? (1st ed., pp. 193–212). Cham, Switzerland: Springer International Publishing. Munday, J. (2012). Evaluation in translation: Critical points of translator decision-making. Abingdon, Oxon/New York: Routledge.

3 Infusing Pro-Environmental Values in Science Education: A Multimodal Analysis. . .

73

Muñoz Morcillo, J., Czurda, K., & Robertson-von Trotha, C. Y. (2016). Typologies of the popular science web video. Journal of Science Communication, 15(4). https://doi.org/10.22323/2. 15040202 Pounds, G. (2012). Multimodal expression of authorial affect in a British television news programme. Discourse, Context & Media, 1(2–3), 68–81. https://doi.org/10.1016/j.dcm.2012.03.001 Schultz, P. W. (2000). Empathizing with nature: The effects of perspective taking on concern for environmental issues. The Journal of Social Issues, 56(3), 391–406. https://doi.org/10.1111/ 0022-4537.00174 Sharma, A., & Buxton, C. A. (2015). Human-nature relationships in school science: A critical discourse analysis of a middle-grade science textbook. Science Education, 99(2), 260–281. https://doi.org/10.1002/sce.21147 Smith, M. (1995). Engaging characters: Fiction, emotion, and the cinema. Oxford, UK: Clarendon Press. Stern, P. C., & Dietz, T. (1994). The value basis of environmental concern. Journal of Social Issues, 50(3), 65–84. Stibbe, A. (2004). Environmental education across cultures: Beyond the discourse of shallow environmentalism. Language and Intercultural Communication, 4(4), 242–260. https://doi. org/10.1080/14708470408668875 Taylor, P. W. (2011). Respect for nature: A theory of environmental ethics (25th Anniversary Edition). Princeton, NJ: Princeton University Press. Tilbury, D. (1994). The critical learning years for environmental education. In R. A. Wilson (Ed.), Environmental education at the early childhood level (pp. 11–15). Troy, Ohio: North American Association for Environmental Education. Tsantopoulos, G., Skanavis, C., Bantoudi, F., Petkou, D., & Dalamagkidou, A. (2018). Assessing preference of informal environmental education sources for Greek primary school students. International Journal of Environmental & Science Education, 13(3), 357–371. UNESCO-UNEP. (1976). The belgrade charter: A global framework for environmental education. UNESCO-UNEP Environmental Education Newsletter, 1(1), 1–9. Unsworth, L. (2015). Persuasive narratives: Evaluative images in picture books and animated movies. Visual Communication, 14(1), 73–96. Xiong, T. (2014). Shallow environmentalism: A preliminary eco-critical discourse analysis of secondary school English as a foreign language (EFL) texts in China. The Journal of Environmental Education, 45(4), 232–242. https://doi.org/10.1080/00958964.2014.943686 York, R., & Clark, B. (2010). Critical materialism: Science, technology, and environmental sustainability. Sociological Inquiry, 80(3), 475–499. https://doi.org/10.1111/j.1475-682X. 2010.00343.x Zappavigna, M. (2012). Discourse of twitter and social media: How we use language to create affiliation on the web. London/New York: Continuum.

YouTube Videos What is ozone layer? https://www.youtube.com/watch?v¼1b3-l4BxqGA Why do animals go extinct. https://www.youtube.com/watch?v¼2mlT0HeVLv4&t¼41s Sustainable development for kids (by Grade 3 kids). https://www.youtube.com/watch? v¼5ACfPVA-EE8 Endangered and extinct animals. https://www.youtube.com/watch?v¼lKvex-0x0fA All the way to the ocean. https://www.youtube.com/watch?v¼sZW2ByM623g Global warming for kids. https://www.youtube.com/watch?v¼Vh8XVkzsn1Y The carbon cycle, carbon dioxide-cycle. https://www.youtube.com/watch?v¼xFE9o-c_pKg

74

M. H. M. Yu et al.

Mandy Hoi Man Yu holds a PhD in linguistics from Lancaster University. She is currently a research associate at The Hong Kong Polytechnic University and an adjunct assistant professor at The University of Hong Kong. She has been involved in several discourse analytical projects relating to Chinese masculinities, media representation of China, science education and corporate branding. Her research interests include (critical) discourse analysis, pragmatics, language and gender, and media discourse. Dezheng (William) Feng, PhD, is Associate Professor and Associate Director of the Research Centre for Professional Communication in English at the Department of English, The Hong Kong Polytechnic University. His main research interests include multimodal discourse analysis and media studies. His recent works have appeared in Journal of Pragmatics, Discourse and Communication, and Visual Communication. Len Unsworth is Professor in English and Literacies Education and research director of educational semiotics in English and literacy pedagogy at the Institute for Learning Sciences and Teacher Education (ILSTE), at the Australian Catholic University in Sydney, Australia. Len’s current research interests include systemic functional semiotic perspectives on multimodal and digital literacies in English and in curriculum area teaching and learning in primary and secondary schools.

Chapter 4

Multimodal Affordances of Immersive Virtual Reality for Visualising and Learning Molecular Interactions Kok-Sing Tang, Mihye Won, Mauro Mocerino, David F. Treagust, and Roy Tasker

4.1

The Use of Virtual Reality in Education

Computer animations have been used as educational resources to support science teaching and learning for many decades (Smetana & Bell, 2012). As animations are built on dynamic and interactive forms of representations, they are particularly useful in helping students visualise complex real-world phenomena and processes, provided they are used in conjunction with appropriate instructional strategies (Clark & Jorde, 2004). In recent years, with the increasing commercialisation and sophistication of Virtual Reality (VR), new educational possibilities of using animations are now opened up for researchers and educators to create and explore. Not surprisingly, more researchers are currently investigating the potential of VR in mediating student learning (Southgate et al., 2018). In general, there are two kinds of VR: non-immersive and immersive (Freina & Ott, 2015). Non-immersive VR is a desktop-based technology that simulates a 3D virtual environment for a user to navigate and interact with. Although the rendered images look three dimensional, they are essentially rendered on a two dimensional flat or curved screen. Thus, a user who looks at the screen retains the perception that he or she is separated from the virtual environment. Many modern games and virtual worlds such as Second Life and Minecraft belong to this category. Science animations that are designed and rendered in a 3D virtual space are also considered a form of non-immersive VR. Immersive VR, on the other hand, provides an all-round

K.-S. Tang (*) · M. Won · M. Mocerino · D. F. Treagust Curtin University, Perth, Australia e-mail: [email protected] R. Tasker Purdue University, West Lafayette, IN, USA © Springer Nature Switzerland AG 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_4

75

76

K.-S. Tang et al.

perception of being physically present in the virtual environment (Slater & SanchezVives, 2016). This is typically achieved through the use of a head-mounted display (HMD) that simultaneously tracks a user’s position and gaze and projects images that correspond to where the user is looking at in the virtual space. In addition, the HMD renders a stereo pair of images for each eye in order to generate depth perception for 3D vision, similar to a pair of 3D glasses. Therefore, immersive VR delivers a more “immersed” spatial experience that evokes real and plausible feelings of presence (Slater & Sanchez-Vives, 2016). Research in non-immersive VR has been around for more than 30 years. Most studies have reported some positive effects and gains when games, simulations and virtual worlds were used in K-12 and university settings (e.g., Dalgarno & Lee, 2010; Merchant, Goetz, Cifuentes, Keeney-Kennicutt, & Davis, 2014). Comparatively, research in immersive VR is currently lacking as the technology was only made available for mass consumption in recent years. Although there has been a lot of excitement and hype on immersive VR, it is important to unpack what immersive means in terms of its learning affordances and how it supports the learning process. Specifically, few have investigated the difference between the use of a HMD in an immersive VR compared to a non-immersive VR (or 3D virtual environment) that is projected on a flat (2D) electronic screen. Therefore, the purpose of this study was to examine the affordances of science animation in an immersive VR environment compared to other available tools. In particular, the study used a semiotic perspective to analyse the multimodal potential of VR in terms of the interaction of images, words, sound, gaze, gesture and bodily movement that occurred in the VR environment. The research questions that guided the study were: 1. What are the affordances of an animation in an immersive VR environment compared to an animation on a flat screen and a physical model? 2. How do students use the affordances provided by an animation in an immersive VR environment to support their learning of science?

4.2

Theoretical Perspectives

The theoretical basis of this study was informed by social semiotics, which is a theory developed to understand how people use and develop various signs as semiotic resources to make meanings in a community (Kress & van Leeuwen, 2001). According to Halliday (1978), language is a semiotic system, or mode, that consists of an accumulation of culturally shaped signs (e.g., sounds, alphabets, logograms) that are used and moulded over time by a particular community into a meaning-making tool. In the same way, any sign can be used as a resource to make and communicate meanings for oneself and to others. In the history of human civilisation, multiple modes have been developed (e.g., language in the form of speech and writing, image, gesture, Braille, music) in different cultures and communities. All these modes function as meaning-making resources that are organised

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

77

and used for making three general types of meaning, namely ideational – for construing an event or relationship about the world, interpersonal – for enacting a stance and participation in the world and social relationships, and textual – for organising meaning elements together into a broader text or representation (Halliday, 1978). The meaning-making potential of any mode is greatly dependent on the medium that carries the sign of the mode. Every sign requires a material substance for it to be made and communicated (Peirce, 1986). By shaping and working on the medium culturally over time in order to meet certain practical and social requirements in a community, this is how every mode was developed. For example, speech and music as a mode is built on the medium of sound, while writing is traditionally built on some inscription devices (e.g., ink, paper). Some modes like gesture and dance rely on the movement of our bodies and hands to make various narrative and symbolic meanings. The inherent characteristics of the material determines the specific potentials and constraints of every mode, giving rise to the affordance of the mode (Kress, Jewitt, Ogborn, & Tsatsarelis, 2014). For instance, the temporal characteristics of human sound both afford and constraint speech as a mode to make meanings in a sequential manner. On the other hand, the spatial characteristics of visual representations allow meanings to be made and interpreted simultaneously instead of in sequence, thus allowing a unique way of meaning making compared to speech. The meaning making potential of gesture, by comparison, falls somewhere between speech and visual mode as it has both temporal and spatial elements (e.g., pace of gesture to represent speed or excitement, hand positions to represent displacement in a 3D space). As a mode is intrinsically linked to a medium, technology plays a crucial role in shaping the affordance of the mode. As technology develops, so does the aggregated meaning-making potential of a community. For instance, in the evolution of writing from the use of stone tablets, to ink and paper, to the digital page on modern devices, each technological improvement brings with it new ways of making meaning that was not available previously. In particular, the medium of the digital page brings about the possibility of embedding images and sound in conjunction with written text. Thus, writing as a mode in the digital age has evolved with technological development and becomes increasingly multimodal as it integrates other modes. According to Lemke (1998), the meaning-making potential of digital writing “multiplies” as it harnesses the affordance of each individual mode that it encompasses, such as written text, image, sound, and more recently, hyperlinks and emojis. In the same way, the mode of image has also evolved with technology that allows static images to be superimposed temporally, thus creating moving images as seen in animations and videos. The creation of moving images opens up a significant new way of meaning making. Burn (2013) coins the term kineikonic (from the Greek words kinein, to move, and eikon, image) to describe this unique mode of making meaning with moving images. The affordance of a mode is not entirely dependent on the technological advancement of the medium, but it is also shaped by social practice as well in terms of how people within a community use the medium for various purposes. As Jenkins (2008)

78

K.-S. Tang et al.

explains, this is related to a set of cultural norms and conventions that evolved alongside the use of a new technology. For example, the development of SMS abbreviations and emojis are recent cultural phenomena that change the affordance of text messaging. In this sense, the cultural and technological development of text messaging as a mode co-developed simultaneously; just as technology makes it possible to insert emojis, the different cultural ways of using emojis has prompted developers to improve and facilitate its use accordingly. As for the use of animations in teaching and learning, its predominant use as a visualisation tool has given rise to the cultural and technological development of certain features commonly seen in educational animations, such as navigational buttons (e.g., stop, play, step forward, step backward) to aid teachers and students to playback the animation in sequential stages. Building on these ideas of mode, medium and affordance, what can we theorise about the affordance of immersive VR? Without the use of a HMD and hand controllers, the main experience is akin to a virtual world projected on an electronic screen. In that sense, VR possesses similar affordances to a kineikonic (moving image) mode. It also has the same affordance of any digital page where sound and written text can be inserted. With the HMD and controllers, new ‘immersive’ affordances will be added to the existing multimodal affordances. Technologically, these affordances revolve around a new visual mode facilitated by the HMD tracking and a gestural mode facilitated by hand controllers. How the affordances of these two modes manifest and how they interact with one another as well as the existing affordances without the VR will be a point of interest to be investigated in this study. At the same time, the affordances of VR are not determined by technology alone, but also by the cultural ways users tap into the technological affordances of VR to make meaningful actions according to specific tasks and purposes. As such, this requires user testing that will be conducted in this study.

4.3

Methods

This study is part of a larger project that uses a design research approach (Collins, Joseph, & Bielaczyc, 2004) to investigate how students learn science through an immersive VR environment. Design research consists of several iterations of design and testing in order to understand students’ learning experiences and to re-design/ improve the VR application at the same time. Currently, we have designed and built a VR application around a series of learning activities and completed the first cycle of data collection based on user testing of the VR application that was designed. The study reported in this chapter is based on the first cycle of data collection and analysis. The VR platform used in the study was HTC Vive, which consists of a HMD connected to a computer and two wireless motion controllers. Although HTC Vive is a high-end platform that is currently not feasible in today’s classroom (compared to the cheaper and more portable devices such as Google Cardboard or Samsung Gear

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

79

VR), the rationale for choosing this platform is due to its accuracy, reliability and multi-user interactivity for our research purpose. These are crucial factors not only for the interactions that are required in the learning activity, but also for the detailed multimodal analysis in this study. The VR application was developed using the programming software Unity® for two users who can interact with one another as well as animated objects in the virtual space. Although both users can be remotely connected from different locations, we chose to put them in the same physical space so that they could feel the physical presence of their peers as they interacted with one another in the virtual space. The VR application consisted of several activities designed to support students’ understanding of molecular interactions. For the purpose of this chapter, we describe three activities that are relevant to understand the subsequent analysis to be presented. The first activity involved the students building the molecular structure of acetylcholine by moving several objects (representing pre-assembled functional groups of atoms) until they formed a bond when they were placed in close proximity to one another. In the second activity, the students watched a scripted animated sequence that showed how acetylcholine is broken into two smaller molecules – acetate and choline. The animated sequence was presented in stages and the students could continue or playback the animation at each stage. In the third activity, students explored the structure of acetylcholinesterase. They could do so by walking around the enzyme, rotating it, and changing its size to the point where the students could zoom inside the enzyme. They could also change views according to different models of the enzyme, namely surface, mesh, ball-and-stick, cartoon and ribbon models. The objective for the students was to explore the structure of acetylcholinesterase and identify a particular gorge that leads into the active site where the reaction of acetylcholine takes place. The participants in this study were recruited from first-year undergraduates studying in a chemistry-related unit at an Australian university. Twenty-two volunteers (12 female and 10 male) took part in the study. These students had varying degrees of background knowledge in chemistry. Most of the students did not have much prior experience in using VR. In pairs, the students registered and participated in a user study session that consisted of three parts. The first part was a pre-activity interview to assess the students’ background knowledge and their familiarity in using a ball-and-stick physical model to construct the molecular structure of acetylcholine. In the second part, both students experienced the VR application and worked together to complete the learning activities. On average, they took about an hour to complete all the activities in the VR. The third part was an interview with the two students immediately after the VR activity to assess their learning experiences. All user study sessions were facilitated and observed by at least two researchers, including the first and second authors. The researchers also recorded their observations and thoughts with field notes during the session. The primary data sources for the study consist of: (1) video files from a tripodmounted camera to capture the students’ interaction in the physical space, (2) audio files from recorders attached to each user in order to capture their voice in close range, and (3) video streams from each user’s HMD in order to capture what they

80

K.-S. Tang et al.

were looking at in the virtual space. These video and audio files were synchronised and merged into a single video file using Adobe Premiere® in order to facilitate the next phase of analysis. The merged video from each pair of students during the VR activity was viewed in its entirety and discrete episodes were identified according to the shifts in participant structure or nature of the activity (Erickson, 1992). Based on the video viewing and field notes, a brief description of the events was tagged in each episode. Subsequently, a content listing (Jordan & Henderson, 1995) indexing the description of every episode with their respective timestamp in the video was generated. This content listing provided an overall contextual basis for subsequent analysis as well as facilitate a purposeful identification of interesting and relevant episodes or telling cases (Mitchell, 1983). The selection of these episodes were guided by the research questions focusing on the affordances of VR as well as noticing revealing events as characterised by rich multimodal actions from the students. Multimodal transcripts (Bezemer & Mavers, 2011) from these episodes were subsequently generated to show not only the students’ utterances, but also their visual perception in the virtual space as well as their bodily movement and orientation in the physical space. These transcripts were used to facilitate a micro-analytic interpretation focusing on how the various modes (e.g., words, images, body movement) were coordinated in conjunction with their chemistry meaning-making. This involved a line-by-line analysis of their utterances (from audio recorders), hand gestures and body orientation (from camera’s video) and virtual images they were looking at (from HMD’s video streams). As seen from the transcripts presented in the next section (see Transcripts 4.1 to 4.5), these three sources of data were juxtaposed in vertical columns in a table, and lineated according to either a turntaking switch between the participants (e.g., Transcript 4.1, line 1–2) or a significant shift in the images as seen by the participants (e.g., Transcript 4.1, line 4–7). In every line, the interpretation of the students’ meaning-making was made by contextualising the verbal utterance with the image, gesture and body orientation (Lemke, 1998). Based on this analysis, the affordances of VR were inductively generated and iteratively grouped them into meaningful categories.

4.4

Findings

In this study, we found and documented five major affordances that immersive VR provided for the learners. These affordances are viewing, sequencing, modelling, scaling and manipulating. Table 4.1 shows the affordances of animation in an immersive VR in relation to an animation on a flat screen animation and a physical model. In what follows, each of these categories will be elaborated and illustrated by representative examples from the multimodal analysis in order to show how students used these affordances to support their learning of science.

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

81

Table 4.1 Summary of the affordances across three different tools (√ ¼ high affordance,  ¼ no or low affordance)

Viewing (see Transcript 4.1) Window size Multiple perspectives Three dimensions Sequencing (see Transcript 4.2) Animating image sequences (e.g., moving, rotating) Modelling (see Transcript 4.3) Changing scientific models Scaling (see Transcript 4.4) Zooming in/out Manipulating (see Transcript 4.5) Manipulating object Tactile manipulation

4.4.1

Immersive VR Animation

Flat Screen Animation

Physical Model

√ √ √

  

√ √ √

√

√



√

√



√

√



√ √

√ 

√ √

The Affordance of Viewing

The first affordance of viewing examines the possibilities for the user to see visual objects in terms of window size, perspective and dimensionality. One of the key differences in a flat screen is that the viewing experience is limited to the window size of the screen. In an immersive VR, there is no window as the HMD provides a viewing experience that is 360
all around. This does not only allow users to see more objects more easily in the environment, but crucially, to see them from a different perspective depending on their spatial positions and orientations. This is something that cannot be realised in a flat screen as the orientation of an object is fixed regardless of where the user is viewing from in relation to the screen. However, in an immersive VR, the orientation changes depending on where a user is standing. Furthermore, multiple users in the VR environment will not be looking at the same thing as the perspective is always relative to their spatial positions. This difference between the two environments is what gives rise to the different dimensionality of the viewing experience. Although many computer-based science animations can be viewed and manipulated in a virtual 3D space, they are rendering 3D images on a flat screen. The viewing experience is essentially two dimensional without the affordance of multiple perspectives. A more realistic 3D viewing experience on the other hand, as afforded by immersive VR, provides the critical multiple perspectives and dimensionality just like our experiences and interactions with objects in the real world. As for the comparison with a physical model, there is not much difference between immersive VR and a physical model, as there is no viewing window involved in our viewing and interaction with physical objects in the real world.

82

K.-S. Tang et al.

Transcript 4.1 Multimodal interaction illustrating the VR affordance of viewing S/ N 1

Time 19:50

Speaker Josh

2

20:02

Anna

3

20:08

Anna

I found it before. I think

4

20:14

Josh

Oh

5

20:15

Josh

6

20:17

Josh

There

7

20:18

Josh

That one

Utterance (Reading from the screen) look around the surface of the protein, identify the gorge, and place the yellow sphere on it. . . I think that. . . oh. . .

Virtual Space (View from Josh’s HMD)

Physical Space

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

83

Transcript 4.2 Multimodal interaction illustrating the VR affordance of sequencing S/ N

Time

Speaker

Utterance

1

14:18

Daniel

Are you ready?

2 3

14:19 14:20

Kate Daniel

4

14:21

Computer

5

14:40

Computer

6

14:46

Daniel

Yup (Clicks the start button) As a new bond between the cardinal carbon and the hydroxyl oxygen is formed, the single bond between the carbon and the oxygen in acetylcholine is broken. . . The oxygen from the choline part then forms a bond with the hydrogen, as the bond between the hydrogen and the oxygen from the acetate part is broken That’s actually really useful like. Seeing how it happens

7 8

14:51 14:52

Kate Daniel

9

14:57

Kate

Virtual Space (View from Kate’s HMD)

Physical Space

Yah Cos normally in like text and stuff, they just show you what the products are, without showing what parts are actually reacting I mean we have started doing the

(continued)

84

K.-S. Tang et al.

Transcript 4.2 (continued) S/ N

Time

Speaker

Utterance

Virtual Space (View from Kate’s HMD)

Physical Space

mechanism of the arrows Yah. But even so, the arrows are only showing like where things are going. But this shows you the movement of the molecules and everything Yah I wonder how stereochemistry will affect it? Oh no, we can’t tick those

10

15:01

Daniel

11 12

15:10 15:12

Kate Kate

13

15:16

Kate

Should we restart it and have a look?

14

15:18

Daniel

15

15:26

Computer

Yeah. (Clicks the start button) (Repeats the narration. . .)

The viewing experience in terms of window size, perspective and dimensionality is an important affordance for science learning as students often have to imagine complex structures and phenomena in three dimensions and their interactions. To understand the impact of this affordance, consider the following interaction when Josh and Anna were examining the protein structure of acetylcholinesterase. The task given was to identify a particular narrow gorge in the protein that leads into the active site where the reaction of acetylcholine could take place. In the multimodal transcript above, as the computer program was narrating the instruction to “look around the surface of the protein”, Josh was looking at the written instruction to his left while Anna had already started searching for the gorge (line 1). In the beginning, they were standing and looking at the acetylcholinesterase

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

85

Transcript 4.3 Multimodal interaction illustrating the VR affordance of modelling S/N

Time

Speaker

Utterance

1

22:14

Facilitator

2

22:20

Daniel

Do you see any corresponding relationship between the different representations? All the empty spaces in this one

3

22:21

Daniel

Correspond to object in the uh.. Mesh one

4

22:25

Daniel

The mesh one, you have this object

5

22:28

Daniel

And that seems to oh, wrong one

6

22:33

Daniel

The space, you see there is a big object here

7

22:36

Facilitator

8

22:38

Daniel

So do you understand why? Uh..

9

22:43

Daniel

No

Virtual Space (View from Daniel’s HMD)

Physical Space

(continued)

86

K.-S. Tang et al.

Transcript 4.3 (continued) S/N

Time

Speaker

Utterance

10

22:45

Kate

The mesh will be showing.. Regions of electron density

11

22:50

Daniel

So why would they do it like that then?

12

22:56

Kate

Maybe because if you look at it as not a thing and a thing box

13

23:00

Kate

The surface.. The outside of this is where the molecules are

14

23:06

Daniel

Yah. I guess.. So let’s look at

15

23:08

Daniel

Oh wait, so there’s an oxygen there.

16

23:10

Daniel

And yah, look

17

23:11

Daniel

Okay, oxygen

Virtual Space (View from Daniel’s HMD)

Physical Space

(continued)

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . . Transcript 4.3 (continued) S/N

Time

Speaker

Utterance

18

23:12

Daniel

And then.. Right.

19

23:15

Daniel

And would uh.. Near these yellow ones. . .

20

23:20

Daniel

Let’s see. Yah, look

21

23:22

Kate

Okay, so the region down there

22

23:24

Daniel

Yah. And you can see all on there. And like on down there as well

23

23:29

Kate

Electron density. This is a massive molecule

24

23:32

Daniel

Yup. What about the cartoon one?

Virtual Space (View from Daniel’s HMD)

Physical Space

87

88

K.-S. Tang et al.

Transcript 4.4 Multimodal interaction illustrating the VR affordance of scaling S/ N

Time

Speaker

Utterance

1

26:05

Kate

Can you see over there? That’s like.. Double bonds things?

2 3

26:10 26:12

Daniel Kate

Yah I see those Are they double bonds between carbon or.. it’s kind of avoiding its space?

4

26:16

Daniel

Let’s look at the ball-andstick. . . can you tell?

5

26:19

Kate

6

26:33

Kate

Is that another molecule? Just in there because of intermolecular forces? It’s something bonded to oxygen and a carbon Can we make the molecule smaller and get closer?

7 8

26:35 26:47 26:56

Daniel Daniel Daniel

Yeah Gosh There we go

9

26:56

Kate

Oh, there we go

Virtual Space (View from Kate’s HMD)

Physical Space

(continued)

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . . Transcript 4.4 (continued) S/ N

Time

Speaker

Utterance

10

27:06

Facilitator

Seeing them in different scale, does it change the way you looked at it?

11

27:10

Kate

12

27:10

Daniel

Yes, it’s make it much easier Yah, definitely, you can look into the intricacy of it

13

27:16

Facilitator

14

27:21

Daniel

15

27:35

Kate

16

27:39

Daniel

17

27:41

Kate

So why do you need to zoom in and zoom out of it? Going in, you get to see the specifics of the chemistry behind it. Like the molecules and how they are interacting, and form shapes. And when you look at the overview, you get to see the shape they do form and how they can be useful in biological reactions What is this thing?

The orange one? It’s a molecule. It’s not bonded to anything

Virtual Space (View from Kate’s HMD)

Physical Space

89

90

K.-S. Tang et al.

Transcript 4.5 Multimodal interaction illustrating the VR affordance of manipulating S/ N 1

Time 26:13

Speaker Anna

Utterance Ah, there we go.. Oh, it’s meant to be in the hole though isn’t it?

2

26:15

Josh

3

26:16

Anna

Yah. Bring it to the hole Like that?

4 5 6

26:18 26:19 26:28

Josh Anna Anna

No, the red bit Oh No. oh, there?

7

26:31

Josh

(Takes control of the object)

8 9

26:35 26:36

Facilitator Josh

10

26:39

Josh

Go deeper Deeper? Er. Ah There we go

11

26:41

Josh

Virtual Space (View from Josh’s HMD)

Physical Space

We got to find the collision point

(continued)

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

91

Transcript 4.5 (continued) S/ N 12

Time 26:44

Speaker Josh

Utterance I was thinking that way. Do you reckon?

13

26:47

Josh

And then see does it fit as well as it fit? Does it?

14

26:48

Anna

Oh yah. I don’t’ know

15

26:54

Josh

Uh. Like in that one.

16

26:56

Josh

And it goes through like that

Virtual Space (View from Josh’s HMD)

Physical Space

structure at about right angles to each other. Both of them then spent about 40 sec looking around in two ways: (i) they grabbed and turned the acetylcholinesterase using their hand controllers and (ii) they moved and tilted their heads to look from a different angle. Although they did not walk around to search for the gorge, some other groups did that more frequently compared to using their controllers to move the acetylcholinesterase instead. In line 4, Josh noticed something in the acetylcholinesterase and he immediately approached to grab it (line 5) and rotated until he saw the gorge in front of him (line 6). He then pointed at the gorge and uttered “That one” to his partner (line 7). As he did that, notice that Anna shifted her position as she moved towards Josh in order to see the acetylcholinesterase from his point of view. Thus, both of them were now looking at the gorge from a parallel perspective.

92

K.-S. Tang et al.

This brief example illustrates the viewing affordance in terms of threedimensional viewing from multiple points of view. Initially, both Josh and Anna were looking at the protein structure orthogonally to each other and they were seeing different things. As Josh found the gorge from his point of view, he then rotated the virtual object and showed Anna where it was. This divide-and-conquer approach aided the collaboration between both of them as they could spread out to search for the gorge. This is something that cannot be done with any program that is projected on a flat screen.

4.4.2

The Affordance of Sequencing

The second affordance is sequencing, which is the potential to juxtapose static images together so that they form a moving image, in what Burn (2013) calls a kineikonic mode. This is the basic method of creating an animation, and both immersive VR and flat screen animations have that affordance. Sequencing is crucial for viewers to examine the dynamic processes and changes that frequently occur in natural phenomena. For this reason, sequencing is frequently used in science animations. The next interaction shows another pair of students, Daniel and Kate, watching an animated sequence that showed the reaction of acetylcholine molecule breaking down into acetate and choline. As they started the animated sequence at the beginning, both Kate and Daniel were standing at right angles to each other. Thus, they were looking at the reaction from a different perspective. Kate was looking from an angle that saw the water molecule on the left and the acetylcholine on her right (see line 1). After the sequence ended, Daniel made an interesting remark in line 6 and 8 of how the animated sequence was useful in showing the entire reaction in terms of the constituent parts (individual atoms and bonds) that were reacting. Kate responded that this was similar to the use of arrows (“mechanism of the arrow” in line 9) that were typically used in traditional texts to show chemical reactions. But Daniel rebutted, saying that the arrows in texts could only show “where things are going” but not the movement of the molecules in the same way as the animated sequence (line 10). From line 12, Kate raised an interesting question of how stereochemistry would affect the reaction. Stereochemistry is the branch of chemistry that deals with threedimensional arrangement of atoms in molecules and how this has an effect on chemical reactions. Kate’s response suggested that she was not just looking at the molecule movement and reaction, but she was also concerned with why or how the reaction was affected by the spatial arrangement of the acetylcholine molecule. Prompted by this question, she then tried to interact with the virtual molecules with her controllers before realising they were not programmed to respond to her action. In line 13, she suggested to re-watch the animated sequence, presumably to address her question regarding the stereochemistry of acetylcholine.

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

93

As the sequence was replayed, Kate shifted her position to look at the reaction from a different angle. When initially she was watching the water molecule moving from left to right (see line 1), she was watching it moving forward in the replay (see line 15). This position thus allowed her to see a different perspective of the threedimensional arrangement of acetylcholine compared to where she was standing initially. This example provides further evidence to support the viewing affordance that was illustrated earlier. However, in this case, the viewing affordance was combined with the sequencing affordance that was provided by watching and replaying the animated sequence in the immersive VR environment.

4.4.3

The Affordance of Modelling

The third affordance – modelling – is the potential to dynamically change the scientific model of an object. The VR application in this study was programmed to generate different molecular models of acetylcholine and acetylcholinesterase, such as surface, mesh, ball-and-stick, cartoon and ribbon models. Each model shows a different molecular representation and highlights specific aspects of a molecule. For instance, the ball-and-stick model displays the three-dimensional positions of the constituent atoms and bonds. The surface model shows the relative electron density on an isosurface of the molecule (with red indicating an electron-rich area and blue showing an electron-deficient area). The mesh model shows a surface region that has an equal electron density. As a user switched between these molecular models, he or she could compare across the different representations and made meanings about the underlying characteristics of the atoms or molecules. Modelling is an affordance that is not exclusive to an immersive VR but is also available in any computer animation that involves computer-generated models. As for a physical model, it is primarily designed and manufactured according to a particular model. The ball-and-stick model, for example, is often presented as a physical model so that students can physically examine and manipulate it in three dimensions. However, there are two important limitations to the use of physical models. First, for very large and complex molecules, the use of a physical model is unwieldy, floppy and impractical. Second, although it is possible to make two or more physical models of the same molecule (e.g., ball-and-stick and surface model), it is not impossible to change from one model to another dynamically so as to allow a user to compare different models seamlessly. This is an important affordance of modelling that could aid learning, and this will be illustrated in the next interaction. In the following interaction, Daniel and Kate were exploring the different models of acetylcholinesterase. There are three interesting parts in this interaction that collectively show the students’ progressive understanding as they toggled backand-forth the different models of acetylcholinesterase. The first part occurred when Daniel observed and stated a corresponding relationship between an oval-shaped object in the mesh model and an empty space in the ball-and-stick model (line 1 to 6). The second part began when they were prompted to explain this corresponding

94

K.-S. Tang et al.

relationship. Initially, Daniel could not explain the relationship, but Kate stepped in to give an insightful comment regarding the electron density of the mesh model (line 7 to 13). The third part showed Daniel gaining an understanding as a result of Kate’s comment and he was able to subsequently apply this understanding to a similar relationship at a different region in the acetylcholinesterase molecule (line 14 to 22). In terms of the affordance of modelling, what supported the students’ learning was the spatial correspondence that occurred when the students toggled across the different models. As a particular visual object in one model was replaced by something else in another model within the exact spatial location, this enabled them to notice the corresponding relationship from a specific part of a model to the same part of another model. We call this relationship spatial mapping. For example, Daniel noticed at the beginning that a region of empty space in the balland-stick model (see line 2 and 6) was replaced by a large oval-shaped object as he switched to the mesh model, and vice versa (line 3 and 4). His observation of this spatial mapping was evident from his deictic (pointing) gesture and deictic pronouns (e.g. this, that, there, it, this one, here). As shown in the above transcript, he frequently used his hand controller to point at a specific spatial region or object as he uttered “this one” (line 2), “mesh one” (line 3), “this object” (line 4), “here” (line 6), and so on. The simultaneous use of deictic gestures and pronouns was repeated 16 times by Daniel and Kate as they switched back and forth the ball-and-stick and mesh models a total of 10 times within this two-minutes interaction. Initially, Daniel only noticed the relationship between the two models through the spatial mapping, but he could not explain why (line 7 to 9). In particular, he was puzzled why the oval object (as a thing) in the mesh model would fit into the empty space in the ball-and-stick model (line 11). Inferring from Kate’ comment in line 12, the problem arose because Daniel was interpreting the mesh as a standalone object, rather than a surface that was folded into a volume of empty space. Thus, Kate told Daniel to “look at it [oval object] as not a thing and a thing box” (line 12), but as a “surface” where the molecules are located outside (line 13). In other words, Kate saw and understood that the surface in the mesh model, as “regions of electron density” (line 10), was mapped to the atoms rather than the empty space in the balland-stick model. She also understood that the oval shape in the mesh model was due to the electron density of the atoms that surrounded it. When Kate uttered “the outside of this is where the molecules are” in line 13, she used her hand controller to gesture the region of space outside the oval shape. After Kate’s comments, Daniel began to show some understanding of the relationship, as seen from his actions from line 15 to 22. Again, the spatial mapping across the two models were instrumental for Daniel to notice the relationship between the atoms and the surface mesh. First, he saw the position of an oxygen atom (red sphere) corresponded spatially to a specific region outside the oval (line 15 and 18), as he pointed at the exact location while switching the models. This was repeated again when he did the same for a yellow sphere in line 19 to 22. These instances demonstrate how the affordance of modelling provided by the immersive VR could enable the students to make connections between electron surface density and molecular arrangement.

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

4.4.4

95

The Affordance of Scaling

The fourth affordance is scaling which examines the potential to change the size of the viewing object in relation to the viewer by zooming into and out of the object. The change in scale typically occurs at an order of magnitude of 100 times or more, thus allowing the viewer to virtually ‘enter’ into an object and see its constituent parts. The ability to change the scale at a large order of magnitude is a common technique employed in computer animations. It is particularly useful in revealing the arrangement and constituent parts of a complex system, ranging from microscopic objects like an atom, molecule or cell to celestial objects like a solar system or galaxy. Scaling is an affordance that can be programmed in an immersive VR or flat screen animation, but it is something that cannot be provided in a physical model. To see the impact of scaling affordance, the next interaction shows Daniel and Kate who continued to explore the various models of acetylcholinesterase. As Kate saw something interesting (a double bond) in the ribbon model, they switched to the ball-and-stick model to get another perspective. When Daniel and Kate switched to the ball-and-stick model to examine the double bond, Kate noticed another molecule (“something bonded to oxygen and a carbon”, line 4) that was separated from the surrounding acetylcholinesterase molecule. As she could not see it clearly, she suggested in line 5 to zoom out of the acetylcholinesterase (“make the molecule smaller”) so that she could move closer to it (“get closer”). After Daniel zoomed out in line 8, Kate then moved to the other side of acetylcholinesterase from line 10 to 13 and rotated it from line 14 to 17 until she could see the isolated molecule that she noticed earlier. It is puzzling why at line 6, Kate wanted to zoom out of the acetylcholinesterase, walked around it and then moved closer to the isolated molecule when she could simply walk straight towards the isolated molecule. One possible interpretation is that the isolated molecule was deep inside the acetylcholinesterase and she found it too disorientating to “walk into” the acetylcholinesterase, and thus wanted to circumvent it and approach the isolated molecule from the opposite side. While Kate was navigating to the isolated molecule, Daniel was responding to the facilitator’s question of how changing the scale changed their point of view (line 10). He responded that “seeing the molecules in different scale” has helped him to “look into the intricacy” of the molecular structure (line 12). He also elaborated what he could see when “going in”, particularly the “specifics of the chemistry” like the interaction and shape that are formed by the molecules (line 14). As he talked about “how the molecules are interacting”, he used his controller to point at a particular bond. He contrasted this affordance with “looking at the overview”, which allowed him to see the collective shape formed by all the individual bonds. Although scaling is an affordance that can also be programmed in any flat screen animation, the viewing experience is very different compared to that in an immersive VR. Using the protein structure of acetylcholinesterase as an example, seeing the individual atoms and bonds on a flat screen is quite challenging, as can be seen from the screenshot in Fig. 4.1. One of the reasons is because without the resolution of

96

K.-S. Tang et al.

Fig. 4.1 A 2D screenshot of the atoms and bonds in the ball-and-stick model

depth in a 2D viewing, the numerous atoms and bonds surrounding the object of interest will simply overlay on the viewer’s line of sight, making it difficult to distinguish an individual atom or bond from the surrounding. Thus, the resulting view looks very messy. In an immersive VR, the viewer is spatially situated inside the protein structure with a sense of depth such that the position of every atom and bond in relation to the object of interest is realistically portrayed as the viewer zooms into or out of the protein structure. Another reason why the scaling affordance provided in an immersive VR is different has to do with the viewing affordance that was explained earlier. Combining both affordances of viewing and scaling, immersive VR provides a different experience that allows students to visualise better compared to a flat screen animation. This combination of affordances will be elaborated further in the discussion.

4.4.5

The Affordance of Manipulating

The last affordance is manipulating, which is the potential to allow users to manipulate virtual or real objects. Many animations allow users some forms of manipulation or control, such as moving and rotating an object. However, in a flat screen animation, the manipulation is done using a mouse from a computer, or if it is a touchscreen panel, through a stylus pen or our fingers. This is not an intuitive way of moving objects as the spatial and bodily orientation is lost in the manipulation. In an immersive VR environment, the use of hand controllers mirrors some kind of tactile manipulation through our hand movements. For example, if we rotate an object to see the back of it using a mouse, it is difficult to get an intuitive sense of how much the object has been turned spatially. Using the hand controller to grab and rotate an object by rotating your hand, just like in the real world, it is easier to retain the spatial orientation during the rotation. In the next interaction, the task was to manipulate an acetylcholine molecule until its position and orientation will allow it to enter the narrow gorge of an

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

97

acetylcholinesterase enzyme. Due to the arrangement of the molecules, only a specific orientation will allow the acetylcholine to enter and subsequently react with the active site inside the acetylcholinesterase. The students therefore had to figure this out as they explored and manipulated the acetylcholine until they found the right position and orientation. The interaction shows how Josh and Anna completed this task. In the beginning, Anna was moving and turning the acetylcholine at the opening of the gorge. She had some difficulty in getting the orientation right despite some verbal help from Josh (line 1 to 6). From line 7, Josh decided to give it a try. He had some difficulty grabbing the acetylcholine initially as he did not reach far enough to touch the object. Once he managed to grab the acetylcholine (line 10), he moved and turned it until it could fit the gorge. Notice his hand movement from line 10 to 13 as he rotated the acetylcholine over by about 180
. As shown in line 13, when he bent his head to look at the acetylcholine, he could see the gorge directly ahead of it. This seamless movement through the intuitive coordination of his hand movement and his spatial point of view allowed him to position the acetylcholine easily (in less than 10 sec). From line 14 to 16, Josh then dragged the acetylcholine through the gorge into the active site, where he could see the catalytic triads inside the acetylcholinesterase (see line 16). Throughout the process of tactile manipulation, Josh could maintain the three-dimensional spatial orientation of the visual object relative to his point of view, in terms of what is up, down, front, back, outside and inside. This is something that is not impossible in a flat screen animation.

4.5

Discussion & Implications

As an emerging technology for teaching and learning, many educators are keen to explore and utilise the use of immersive VR. It is thus imperative for researchers to learn about its affordances and how it supports learning in order to make recommendations about its use. Many current studies tend to overly focus on the excitement and motivation of students as they experienced VR (Merchant et al., 2014), but reveal few insights into why or how immersive VR enriches the learning experience in terms of the rich animation and multimodal resources that are made available to them. At the same time, recent advances from social semiotics studies – as applied in the science education – have gained new understanding on how students make scientific meanings by combining verbal and visual modes of representation (e.g., Ge, Unsworth, Wang, & Chang, 2018; Kress et al., 2014; Tang, Delgado, & Moje, 2014). This chapter presents one of the first studies that bridge these two research areas by applying a semiotic perspective and analysis to understand the multimodal affordances of VR, both theoretically and empirically. Theoretically, it was hypothesised that immersive VR combines the affordances of a kineikonic mode from the moving images and a gestural-kinaesthetic mode as facilitated by the HMD tracking and hand controllers.

98

K.-S. Tang et al.

Empirically, through the user testing study, this chapter has illustrated there are five key affordances (viewing, sequencing, modelling, scaling, manipulating) that an immersive VR provides for students to learn through science animation. Flat screen animation and physical models are alternative tools that have some of these affordances as well, but not all of them. As shown in Table 4.1, an important consideration is that the affordances between flat screen animation and physical model are almost opposite to each other. That is, the advantages of using an animation on a flat screen are precisely the limitations of a physical model, and vice versa. An immersive VR, however, incorporates all the advantages from both of these tools. In this sense, we can say that immersive VR combines both the affordances of a flat screen animation and a physical model. The combination of affordances from both a flat screen animation and a physical model is an important consideration in choosing when to use immersive VR as a learning tool. If the use of a particular animation in an immersive VR environment only harnesses one affordance (e.g., sequencing), then this is not a good use of educational technology because there is an existing alternative that not only provides the same affordance but is cheaper and easier to use. However, if the animation in an immersive VR utilises, say, the affordance of viewing (an advantage of physical model) with the affordance of sequencing, modelling or scaling (an advantage of flat screen animation), then this will harness the key advantage of using immersive VR that is not possible with other tools. Most of the examples presented in this chapter have illustrated this advantage of combining the affordance of viewing with the rest of the affordances. For instance, when Kate and Daniel were watching the animated sequence of the reaction of acetylcholine (transcript 4.2), it was not just the sequencing of moving images that facilitated their learning. When they replayed the animated sequence, Kate walked to a new position to look at the sequence from another angle, thus utilising the viewing affordance of multiple perspectives provided by an immersive VR. This combination of viewing affordance was also clearly observed in the other examples that illustrate the affordances of scaling and manipulating. In the example on scaling, Kate zoomed out of the acetylcholinesterase (scaling affordance) and walked around it (viewing affordance) in order to find the isolated molecule that she noticed earlier. In the example on manipulating, Josh shifted his perspective a number of times (viewing affordance) as he was rotating the acetylcholine (manipulating affordance) in order to see how the molecule could orientate and fit into the gorge.

4.6

Conclusion

Understanding the multimodal affordances of an immersive VR is crucial if we are to make use of this new tool for educational purposes. As digital technology continues to advance, it is foreseeable that more science animations involving complex realworld phenomena and processes will be viewed through an immersive VR or related forms of technology, such as Augmented Reality (AR) and Mixed Reality (MR).

4 Multimodal Affordances of Immersive Virtual Reality for Visualising and. . .

99

Therefore, it will be increasingly important to find new ways to study the multimodal affordances of these emerging tools in facilitating the teaching and learning of science. Acknowledgement This project was supported by the Research and Innovation Support Program (RISP17-02) at the School of Education, Curtin University. Adam Mathewson and Jesse Helliwell contributed to the development of the VR application. Jianye Wei and Vergel Mirana contributed to running the us-er study sessions. Hyerin Park synchronised the video and audio files for analysis. Curtin HIVE (Highly Immersive Visualisation eResearch) provided technical support and hosted the user trial sessions for the project.

References Bezemer, J., & Mavers, D. (2011). Multimodal transcription as academic practice: a social semiotic perspective. International Journal of Social Research Methodology, 14(3), 191–206. https://doi. org/10.1080/13645579.2011.563616 Burn, A. (2013). The Kineikonic mode: Towards a multimodal approach to moving image media. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis. London: Routledge. Clark, D., & Jorde, D. (2004). Helping students revise disruptive experientially supported ideas about thermodynamics: Computer visualizations and tactile models. Journal of Research in Science Teaching, 41, 1), 1–1),23. https://doi.org/10.1002/tea.10097 Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical & methodological issues. Journal of Learning Sciences, 13(1), 15–42. Dalgarno, B., & Lee, M. J. (2010). What are the learning affordances of 3-D virtual environments? British Journal of Educational Technology, 41(1), 10–32. Erickson, F. (1992). Ethnographic microanalysis of interaction. In M. D. LeCompte, W. Millroy, & J. Preissle (Eds.), The handbook of qualitative research in education (pp. 201–225). New York: Academic. Freina, L., & Ott, M. (2015). A literature review on immersive virtual reality in education: State of the art and perspectives. In Proceedings of eLearning and software for education (eLSE). Ge, Y.-P., Unsworth, L., Wang, K.-H., & Chang, H.-P. (2018). Image design for enhancing science learning: Helping students build taxonomic meanings with salient tree structure images. In K. S. Tang & K. Danielssson (Eds.), Global developments in literacy research for science education (pp. 237–258). Cham, Switzerland: Springer. Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Arnold. Jenkins, H. (2008). Convergence culture: Where old and new media collide. New York: New York University Press. Jordan, B., & Henderson, A. (1995). Interaction analysis: Foundations and practice. The Journal of the Learning Sciences, 4(1), 39–103. Kress, G., Jewitt, C., Ogborn, J., & Tsatsarelis, C. (2014). Multimodal teaching and learning: The rhetorics of the science classroom (2nd ed.). New York: Bloomsbury Academic. Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold. Lemke, J. L. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. In J. Martin & R. Veel (Eds.), Reading science (pp. 87–113). London/New York: Routledge. Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education, 70, 29–40. https://doi.org/10.1016/j. compedu.2013.07.033

100

K.-S. Tang et al.

Mitchell, J. C. (1983). Case and situation analysis. The Sociological Review, 31, 187–211. Peirce, C. S. (1986). Writings of Charles S. Peirce: A chronological edition. Bloomington: Indiana University Press. Slater, M., & Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Frontiers in Robotics and AI, 3, 74. https://doi.org/10.3389/frobt.2016.00074 Smetana, L. K., & Bell, R. L. (2012). Computer simulations to support science instruction and learning: A critical review of the literature. International Journal of Science Education, 34(9), 1337–1370. https://doi.org/10.1080/09500693.2011.605182 Southgate, E., Buchanan, R., Cividino, C., Saxby, S., Eather, G., Smith, S. P., et al. (2018). What teachers should know about highly immersive virtual reality: Insights from the VR School study. Scan, 37, 4. Tang, K. S., Delgado, C., & Moje, E. B. (2014). An integrative framework for the analysis of multiple and multimodal representations for meaning-making in science education. Science Education, 98(2), 305–326. https://doi.org/10.1002/sce.21099

Kok-Sing Tang is an Associate Professor and Discipline Lead of the STEM Education Research Group at Curtin University. He received a BA and MSc in Physics from the University of Cambridge and a MA and PhD in Education from the University of Michigan. His research examines the role of language, discourse and representations in supporting science teaching and learning. He is a co-founder and coordinator of the SIG Languages & Literacies in Science Education formed within the European Science Education Research Association (ESERA). He is also an Associate Editor of Pedagogies and an editorial board member of Science Education. Mihye Won is a senior lecturer in the Science and Mathematics Education Centre at Curtin University in Perth, Western Australia. Mauro Mocerino is an Associate Professor and Coordinator of Post-graduate coursework in Chemistry at Curtin University. He was the Director of Undergraduate Studies in Chemistry who oversaw the development and introduction of a completely revised chemistry program at the university. He is a Co-editor of the Australian Journal of Education in Chemistry and co-author of the leading first year chemistry text in Australia and NZ. David F. Treagust is Professor of Science Education in the Science and Mathematics Education Centre at Curtin University. He taught secondary science for 10 years in schools in England and in Australia prior to working in universities in the USA and Australia. David is the author of over 240 science education articles in refereed journals, several books and chapters in books and has presented over 250 papers at international conferences. His research interests are related to understanding students’ ideas about science concepts and how these ideas can be used to enhance the design of curricula and teachers’ classroom practice. Roy Tasker is a Professor of Chemistry at Purdue University and an Adjunct Professor at Western Sydney University. He teaches freshman chemistry and graduate courses in chemistry education, and his research interests are in how and what students learn through visualization of the molecular world underpinning observable phenomena (see VisualizingChemistry.com).

Part III

Learning From Viewing Science Animations

Chapter 5

Using Animated Simulations to Support Young Students’ Science Learning Garry Falloon

5.1

Introduction

This chapter investigates the use of animated simulations to support young students’ science learning. Principally, it describes and reports methods and outcomes from two related studies that were completed in New Zealand, specifically focused on the outcomes and learning processes of 5 year olds using a series of animated simulations on iPads to learn about simple electrical circuit design, construction and operation. Both studies explored for evidence of learning transfer, and sought understanding of the cognitive processes the students applied during transfer. The first study investigated conceptual and procedural knowledge development and transfer between simulations, and the second explored this from the simulations to equivalent, equipment-based tasks. Motivation for the studies was two-fold. First, there is limited research available probing outcomes from present trends in primary schools towards Bring Your Own Device (BYOD) classrooms, where low cost apps of mixed educational value are being used extensively across the curriculum, on the assumption that they are beneficial for students’ learning. Second, increasingly digital resources including apps and websites are incorporating simulated experiences as learning assets. While a significant body of literature exists on the effectiveness of these for adult training (e.g., Ganier, Hoareau, & Tisseau, 2014) and in secondary and college education (e.g., Evagorou, Korfiatis, Nicolaou, & Costas, 2009; Hulshof & de Jong, 2006; Huppert, Lomask, & Lazarowitz, 2002) very little work has been completed with younger students, and almost none in the context of science learning. This seems surprising, given the reported benefits from using simulations with older students for science content knowledge development (e.g.,

G. Falloon (*) Macquarie University, Sydney, NSW, Australia e-mail: [email protected] © Springer Nature Switzerland AG 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_5

103

104

G. Falloon

Rutten, van Joolingen, & van der Veen, 2012), process skills (e.g., Huppert et al., 2002), conceptual change (e.g., Chen, Pan, Sung, & Chang, 2013; Olympiou & Zacharia, 2011) and learning engagement (e.g., Noorhidawati, Ghazel, & Ghalebandi, 2015). Also, given well-documented issues with low general science literacy, declining levels of school science achievement and engagement with STEM study and careers, and the importance of early years education in forming attitudes towards STEM (e.g., Timms, Moyle, Weldon, & Mitchell, 2018), it is timely to investigate whether readily-available animated simulations offer potential to help teachers address some of these challenges. To position this chapter within existing literature, a review is presented that briefly details studies into the use of simulations in science education, dating back to the 1980s. This is supplemented by discussion of some of the limited number of available studies investigating their use in early years science learning, followed by an introduction to the foci, context, methods, procedures and outcomes from the two studies introduced above. The chapter will conclude by presenting implications for teachers considering using animated simulations in science curricula, and point to new areas of research needed to optimise potential benefits from their use for procedural and conceptual knowledge building with younger learners.

5.2 5.2.1

A Review of Literature Simulations in Science Education

Media-controlled simulation use in science education is not new, with documented studies dating back to the early 1980s (e.g., Lunetta & Hofstein, 1981). Computer simulations can be defined as “a program that contains a model of a system – natural or artificial (e.g., equipment), or a process” (de Jong & van Joolingen, 1998, p. 2). Research over a number of years has signalled multiple benefits for science learning from using simulations, including enabling students to investigate hypothetical scenarios, interacting with and learning about science processes and procedures in a risk-free manner, changing the time scale of events, and practicing and repeating experiments without the need for expensive or difficult to obtain materials (van Berkum & de Jong, 1991). More recently, research has also identified significant cognitive benefits from students’ use of simulations. These relate to the operationalisation of higher order thinking capabilities as students build and test abstract theories within simulations, and given appropriate tasks, attempt to transfer these to working with concrete materials. While this work is in its early stages, initial outcomes appear promising (Falloon, 2019a, 2019b). Traditionally, simulations have been used in science curricula as means of developing content knowledge, preparing students for practical laboratory work – especially experimental procedures and techniques, and as a useful resource for challenging and addressing students’ misconceptions (e.g., Foti & Ring, 2008; Kumar & Sherwood, 2007; Marshall & Young, 2006; Stieff & Wilensky, 2003).

5 Using Animated Simulations to Support Young Students’ Science Learning

105

Some successful work also been completed investigating effects of simulation use for affective aspects of science learning, including engagement in courses and general interest in science (e.g., Baltzis & Koukias, 2009; Limniou, Papadopoulos, Giannakoudakis, Roberts, & Otto, 2007). Much of this research has focused on the capacity of simulations to support student content, procedural or process knowledge and reflection, contributing to greater accuracy in lab and experimental techniques when used to augment practical work. Generally, these studies reported on experiments using pre-and post-test designs comparing traditional classroom or lecture delivery methods with ones substituted or supported by simulations. In appraising outcomes from these studies, Smetana and Bell (2012) concluded that: . . . simulations are equally, if not more effective than traditional instructional methods in teaching science concepts (and that they) hold several advantages, including flexibility, time efficiency, and the ability to focus student attention on the essential aspects of an experiment, eliminating extraneous variables (p. 1342).

However, other work has been more speculative, highlighting a number of factors influencing outcomes from using simulations. These include when in the learning sequence the simulation is introduced (Klahr, Triona, & Williams, 2007), whether it is used to supplement physical or practical experience (Zacharia, 2005; Zacharia, Olympiou, & Papaevripidou, 2008), the level and nature of instructional support (Fund, 2007), and if it is used by individuals or in collaboration with others (Manlove, Lazonder, & de Jong, 2009). Some studies suggest simulations by themselves may actually be more effective than physical experiences for building conceptual understandings, as they “allow students to perform and repeat an experiment faster, experience it more, and also devote more time to the conceptual aspects” (Zacharia et al., 2008, p. 1022), However, perhaps unsurprisingly, the most consistent message from this research is that simulations are more effective when used in conjunction with practical lab work, usually before the lab experience to introduce students to concepts or procedures they will encounter in practical work, or following, to reinforce developed conceptual understandings. Regardless of positioning, integral to achieving any benefits from using simulations is how they are supported instructionally, either by the pedagogical and organisational strategies of the teacher, and/or via design aspects, such as the presence and nature of built in scaffolds (e.g., instructions, cognitive tools) or how concepts are represented (e.g., static, dynamic, concrete, idealised). Representation appears particularly relevant for conceptual development, where several studies argued for simulations to include multiple representations of concepts to support deeper and more enduring knowledge development (e.g., Prain, Tytler, & Peterson, 2009). van der Meij and de Jong (2006) suggest multiple representations require students to make links between the different representations, supporting more robust and transferable conceptual understandings. Moreover, Rutten et al. (2012) claim that concrete representations are more effective for building conceptual understandings used within the simulation, while idealised representations are of greater benefit when transfer to more abstract but related scenarios is the goal.

106

G. Falloon

Research on the learning benefits of animated visualisations in simulations has reported generally favourable results, when aligned with specific outcomes and purposes. For example, Höffler and Leutner (2007) concluded dynamic visualisations are more effective than static only when used in a representational (move or change form to enhance understanding) rather than decorative manner. They also commented that dynamic visualisations appear more effective for developing procedural motor knowledge – skills and procedures used in performing manual tasks, than for deeper understanding of more abstract procedural or declarative knowledge. Additionally, other studies have shown that when observing animations learners tend to focus mainly on their most salient components, which can therefore provide opportunities to ‘cue’ learning towards particular information or knowledge (Lowe, 2003). They could focus attention on, for example, concepts that change with time, the action of the earth’s magnetic field, or the direction of electrical current in a circuit. However, findings indicate this is most relevant to scenarios where animations do not allow for high levels of learner control (for example, manipulating speed, sequence or information delivery), or where limited other scaffolding is available to focus learners’ attention to relevant information (de Koning, Tabbers, Rikers, & Paas, 2007). Effectiveness is also strongly influenced by the nature of guidance learners receive (Plass, Homer, & Hayward, 2009). This applies both to feedback and domain specific knowledge, instructions or advice provided within the simulation, and also the pedagogical, instructional and support strategies of teachers using them in classrooms. de Jong (2005) highlights the importance of feedback that is explanatory rather than simply corrective, for supporting learning that is retained and transferable to other contexts. Explanatory feedback, “where learners, in addition to being told whether or not they are correct, are also given a domain relevant explanation of why their answer is correct, or not” (Plass et al., 2009, p. 46) is argued as being most beneficial to concept formation and transfer, and particularly important to novice and younger learners. While it is important learners maintain levels of control, autonomy and self-direction when using simulations, pedagogical models such as guided discovery are preferred to pure discovery, to reduce the cognitive load associated with organising, assimilating and integrating new information in knowledge construction (Plass et al., 2009). This conclusion is particularly relevant to the current study given the young age of the students, and the fact that they had no prior experience or conceptual knowledge of circuits to draw on.

5.2.2

Simulations in Young Students’ Science Learning

Although some studies have been completed with young students investigating simulations for literacy and general language development (Lieberman, Bates, & So, 2009), geometry (Steen, Brooks, & Lyon, 2006), and mathematical computation (Burns & Hamm, 2011; Reimer & Moyer, 2005), very little appears to have been completed in science education. Smetana and Bell (2012) identify this as a key area

5 Using Animated Simulations to Support Young Students’ Science Learning

107

for research, commenting “while computer simulations exist for elementary-aged learners, more research is needed to determine what impacts these tools have on young students’ content knowledge acquisition, science process skill development, and conceptual understanding” (p. 1361). Additionally, while some work has been carried out exploring learning transfer from simulations in adult training, schoolbased research on learning transfer is practically non-existent. Although there is some research attention to the relevance of higher order and metacognitive thinking in early years learning (e.g., Bryce & Whitebread, 2012) and a little exploring this in science education (e.g., Georghiades, 2000), historical assumptions about the limited capacity of young students to function cognitively at levels required to abstract from simulations, has restricted investigation in this area (Bransford, 2000). Notwithstanding these challenges, a few studies are emerging probing outcomes from simulation use for science learning with young students. Perhaps the most relevant of these, given the age of the participants, is Wang and Tseng’s (2018) comparative study of physical (equipment) and virtual manipulatives (simulations), and a combination of both, for building conceptual knowledge about molecular level phase changes in water. Their work compared eight and nine year olds who used simulations containing animated visualisations, with those who completed lab work only, and others who combined both. ANCOVA analysis indicated statistically significant gains for students who used the simulations before the lab work, but notably, also by themselves. While combining both physical and virtual experience yielded the greatest benefits, significantly, the simulations by themselves out-performed lab experiences. Wang and Tseng (2018) concluded the simulations aided understanding by making abstract concepts visible to students, thereby assisting them to form mental models explaining the molecular changes. Simulations also allowed students to quickly and conveniently slow and repeat procedures, helping focus attention on the most salient aspects of the molecular change process. Another study by Zacharia, Loizou, and Papaevripidou (2012) involving kindergarteners (4–6 year olds) learning about mass using a beam balance simulation and physical equivalents, concluded no significant difference in ability to predict and explain results before and after testing objects using either physical equipment or a simulation, for children who held reasonably correct prior knowledge of how beam balances worked. Their study focused on physicality as a necessary element for young children’s learning through experimentation – in this case, whether physical manipulation of objects of different mass was a prerequisite for understanding what a beam balance does. Interestingly, for children who held existing knowledge of beam balances through prior manipulation or testing of objects using one, physicality was less influential, while for those with no prior knowledge, it was. This finding suggests that for very young children physical manipulation of objects may be important, if more abstract concepts related to specific properties of materials introduced in simulations alone, are to be fully understood. This conclusion is of relevance, because the studies detailed in this chapter investigated if concepts learnt in simulations were able to be transferred successfully to physical tasks, without any prior knowledge of the materials being used. Acknowledging differences in the properties of materials in both examples, and possibly their relative importance in

108

G. Falloon

terms of transferable concept development, the conclusion by Zacharia et al. (2012) suggests there may be conceptual development challenges to using simulations with younger children who lack related prior physical experiences.

5.2.3

Learning Transfer from Simulations

Recent research has investigated the extent to which learning developed within virtual environments such as simulations, can be successfully transferred to ‘real world’ scenarios. To date, this has mainly centred on adult training, particularly focusing on manual procedural skills such as tank maintenance (Ganier et al., 2014) or machine assembly and disassembly operations (Rodríguez, Gutiérrez, Sánchez, Casado, & Aguinaga, 2006). According to Bossard, Kermarrec, Buche, and Tisseau (2008), transfer “could well be the best way to prove how effective virtual environments are for learning or training” (p. 152). However, determining learning transfer is complex. The extent and nature of transfer is influenced by multiple factors, including the type of learning targeted, specific features of the simulated environment supporting transfer (or not), and similarities and differences between the virtual and ‘real world’ scenarios. Studies indicate the latter of these factors aligns with the type of knowledge being targeted for transfer, and is related to the task towards which the transferred knowledge is to be applied. In this respect, Bossard et al. (2008) identifies two distinct forms of transfer – vertical transfer and horizontal transfer. Vertical transfer is associated with the application of acquired knowledge to new scenarios that possess the same or similar attributes as the environment within which the knowledge was originally constructed. According to Bossard et al. (2008), vertical transfer is often used synonymously with ‘near’ transfer, and is mostly associated with procedural knowledge transfer, that is, “tasks include a sequence of operational steps and the sequence is repeated every time the task is performed” (p. 153). Subedi (2004) adds that while procedural transfer rates can be high, it is unlikely this type of knowledge can be adapted to different situations, or changed conditions or environments. Horizontal transfer, on the other hand, represents more complex learning, signalling an ability to abstract beyond the simulation through applying and evaluating more conceptually-based understandings, in new situations and environments. Horizontal transfer may apply to general scenarios (applicable to different knowledge disciplines) or specific (applicable to a close or related knowledge discipline), but the key consideration is the ‘distance’ or extent of similarity between the initial learning and target application contexts. Horizontal transfer is closely aligned with ‘far’ transfer (Bossard et al., 2008), requiring learners to build generalised, theoretical understandings - in this case from simulations, that can be successfully transferred to problems or scenarios with dissimilar attributes or characteristics. Horizontal transfer has been associated with the development and exercise of higher order and metacognitive thinking processes, as learners activate complex thinking skills in forming transferable mental models.

5 Using Animated Simulations to Support Young Students’ Science Learning

109

In terms of science learning, Georghiades (2000) comments that metacognitive processes are central to establishing robust and transferable science concepts, but to date, “the greatest volume of work has been undertaken on transfer of science process skills, with too little attention paid to transfer of science concepts” (p. 123). Some of this lack of research attention can possibly be attributed to the complexity of determining transfer, and the dearth of empirically-validated indicators of learning transfer within classroom contexts. In an attempt to respond to the challenge, this chapter outlines methods and results from two related studies that focused on identifying specific transfer events, and the cognitive processes underpinning them, in data collected from five year olds using app simulations and physical materials to build simple circuits. While detailed methods and results for each study have been published elsewhere (Falloon, 2019a, 2019b), the chapter provides a synthesis of the main outcomes from both studies, and identifies specific recommendations for teachers considering using simulations and simulation-to-real transfer tasks with their students.

5.3

Research Goal and Questions

The principal goal of both studies was to learn more about the ability of younger students to build basic procedural (know how) and conceptual (know why) knowledge from animated simulations, and transfer this to practical tasks. The first study specifically focused on knowledge development and transfer within simulations, with data being collected and analysed responding to this question: 1. To what extent can science simulations help young students learn simple circuit concepts, construction procedures, and the function of circuit components? The second study then sought evidence of the students’ capacity to transfer this knowledge to physical equipment tasks of similar designs. Data for the second study were collected responding to this question: 2. What evidence exists that young students can transfer procedural and conceptual knowledge of simple circuit components, construction and operation, from simulations to equipment tasks? Both studies also investigated the thinking skills and strategies students applied while learning using the simulations, and during the transfer tasks. To facilitate this, analytical frameworks were developed based on Kolb’s (1984) Experiential Learning Theory (study 1) and an integration of Georghiades (2000) metacognitive strategy indicators and Krathwohl’s (2002) cognitive dimensions (study 2).

110

5.4

G. Falloon

Research Background

Both studies were completed in a New Zealand primary school in 2017. Data were collected from 40, five year olds attending a 550 student regional school located in the Waikato district. The class was team taught by three teachers – one experienced junior school teacher (18 years), one lesser experienced teacher (5 years) and a beginning teacher (1 year). While this study was the first involving these students, the author had been involved in studies with other classes at the school since 2013 (e.g., Falloon, 2013; Falloon & Khoo, 2014; Falloon, 2014; Falloon, 2015; Falloon, 2016; Falloon, 2017). The overall research focus aligned with the school’s curriculum and learning space redevelopment, that included building several new, large flexible teaching spaces and implementing Bring Your Own Device (BYOD) programs from years 3–6. This was associated with a re-visioning of the school’s purpose and mission, and along with it, curriculum and pedagogy transformations considered to better reflect the future-focused skills and competencies needed by its students. This resulted in the establishment of the school’s COGs and learning virtues framework, that underpinned and strongly informed curriculum design, pedagogy and competency development in all classrooms (Fig. 5.1). The framework comprised the learning virtues of respect, honesty, cooperation, self-discipline, creativity and excellence; and the core learner goals (COGs) of effective thinkers, effective communicators, technologically – capable learners, active learners and capacity to make a difference (to class, school, community etc). Representing them as intermeshing cogs signalled their interrelationship, and how they all worked together to define the qualities, capabilities and characteristics of learners at the school. With younger students, there was a strong focus on essential knowledge mastery (literacy/language, numeracy) developed through programs that blended traditional teacher-directed methods with more student-focused, problembased learning.

Fig. 5.1 The school’s learner virtue and competency model (COGs)

5 Using Animated Simulations to Support Young Students’ Science Learning

111

The unit data were collected from was part of the class’s science and technology program, and emphasised the COGs of effective thinking, technology-capable and active learning.

5.4.1

The Learning Unit and Teaching Approach

The unit was entitled ‘Building Simple Circuits’ and was taught in two stages (studies 1 and 2) over a 5 week period. The first study involved students using 4 app-based animated simulations that introduced simple circuit building procedures and electricity concepts, and the function of different circuit components (bulbs, cells, switches, wires etc.). This unit and study lasted 2½ weeks (2 x 40–45 minute lessons per week) during which time the students transitioned from the templated (scaffolded) simulation, ‘Electronics for Kids’, to the more open design, breadboard simulation, Exploriments - ‘Simple Circuits’. The simulations used are described briefly in Table 5.1. Both studies used the same task design during lessons, with image changes made reflecting differences between virtual and physical representations (Fig. 5.2). Circuit building tasks followed a ‘Can You?’ challenge approach, but were structured by restricting student access to only the resources needed to complete each task. The second study was of a similar duration and design, except that physical equipment was used in place of the simulations. The teaching approach for both studies was identical and based on guided discovery pedagogy. Typically, each lesson began with a short revision of the previous lesson followed by an introduction to the app or its physical equipment equivalent. This was followed by 30–35 minutes of practical exploration during which time the teachers circulated among the pairs (study 1) and groups of 3 (study 2), using open questioning, strategic prompting (to focus attention on problems and possible solutions), and redirecting (suggesting alternatives) to guide the students’ learning. At no time did the teachers intervene physically (i.e., construct circuits or fix problems) or use direct instructional methods. Each lesson concluded with a plenary discussion when the students fed back on their activities and what they had learnt.

5.5

Data Methods and Coding Frameworks

Primary data for study 1 comprised iPad display and audio recordings, and in study 2, video supplemented by stimulated recall interviews (SRI) was used. SRI enabled the collection of additional data elaborating on specific events of note identified in the video recordings. The interviews investigated reasons behind students’ observed actions, confirming or modifying interpretations and gathering more information about their thinking and decision making processes. Simulation video for study

112

G. Falloon

Table 5.1 The animated circuit builder simulations. (Adapted from Falloon, 2019a) Simulation Electronics for kids (age rated 4+)

Description This simulation comprised a series of circuit templates that students built using a drag and drop resource toolkit. Resistance, simple uncontrolled, controlled, series and parallel circuits were introduced. Appliances included bulbs and a siren.

DC circuit builder (age rated 4+)

This breadboard simulation comprised a ‘snap to’ grid for connecting drag and drop wires and appliances. Voltage could be varied by tapping the cell terminal. Students could construct circuits of any configuration, using minimal appliances. A resistor was introduced, and resistance levels could be varied.

Exploriments – Simple circuits (age rated 4+)

This simulation had 2 user modes. ‘Teaching’ mode comprised a series of circuit diagrams (series, parallel, controlled, uncontrolled, single and multiappliance) students could replicate on the breadboard using drag and drop resources from the toolbox. ‘Free form’ mode allowed them to construct circuits of any design. The number of connector points varied according to the breadboard selected.

(continued)

5 Using Animated Simulations to Support Young Students’ Science Learning

113

Table 5.1 (continued) Simulation Parallel bulbs (age rated 4+)

Description This simulation was of breadboard design and could be used in the same teaching and ‘free form’ modes as Exploriments. Circuits constructed were represented schematically on the blackboard. Detailed instructions and explanations about the constructed circuits were provided by a voice over. Language used was technical and beyond the comprehension of most students.

Fig. 5.2 A sample simulation and transfer task. (From Falloon, 2019b)

1 was recorded using a display and audio capture app installed on a class set of iPads, while data for the second study were recorded using the iPad’s video camera. In study 2, students took turns at video recording while others in the group built the circuits, meaning all contributed equally to tasks across the lessons. Both datasets were analysed using Studiocode video analysis software. Studiocode supports the identification, logging, annotation and analysis of video data through mapping recordings on timelines, where specific events of interest can be coded. Export of quantitative information summarising total event counts per code, average event duration, percentage of run time by event etc. provides valuable insights into data patterns, such as, in this case, the relative contribution of different thinking skills in the execution of the tasks. While both studies used Studiocode for analysing data, each employed different theoretical and coding frameworks, reflecting their different emphases. The first study applied a coding scheme developed from Kolb’s (1984) Experiential Learning

114

G. Falloon

Theory (ELT) to help understand how the young students engaged qualitatively different thinking types as they both built and transferred conceptual and procedural knowledge between the simulations (near transfer). The second study focused more on far transfer, that is, it required students to abstract from the simulations to complete equivalent tasks using physical equipment. Both studies used the same concept definitions relating to circuit properties and component functions, these being generated by the author and a science faculty member based on identification of the main circuit concepts introduced in the simulations. However not all concepts were coded against in both studies as some could not be replicated due to equipment unavailability, or the absence of relevant information in the simulations. The ‘in common’ concepts used were: • • • •

operating circuits are closed (circuits need continuous current); switches control current in circuits (interrupt or facilitate current flow); resistance affects current (effect of multiple resistors in a single circuit); series circuits and voltage drop (single path for current, resistors ‘share’ current in series circuits).

In addition, conceptual understandings relating to voltage and current pathways in parallel circuits were investigated in study 1, while data were also gathered about knowledge of component placement and function in study 2. Table 5.2 summarises the circuit concepts investigated in both studies. Understanding the students’ cognitive processes and thinking skills while learning using the simulations and later when transferring to the equipment tasks, was

Table 5.2 Summary of the circuit concepts investigated in both studies Circuit design or electricity concept Operating circuits are closed (uninterrupted current) Series circuits (voltage drop, resistors in single circuit)

Parallel circuits (equal voltage, resistors in ‘separate’ circuits)

Resistance (effect on current, resistor performance) Controlling current in circuits (circuit components) Components in circuits Component function

Simulation study Yes

Transfer study Yes

Yes

Yes

Yes

Yes

Yes

No

Yes

No

Yes

Yes

Switches control current

Yes

Yes

Positioning of components can affect circuit performance Components serve particular functions in circuits

No

Yes

No

Yes

Circuit knowledge (codes) Circuits must be continuous Current has only one path in a series circuit Appliances ‘share’ voltage in series circuits Current has more than one path in a parallel circuit Appliances get the same voltage in a parallel circuit Resistance affects current

5 Using Animated Simulations to Support Young Students’ Science Learning

115

more complex. Table 5.3 briefly summarises the codes and definitions developed from Kolb’s (1984) ELT model that were used in study 1. Following the model, coding focused on data revealing insights into the students’ reflective and descriptive observation and thinking, as they built and transferred learning within and between the simulations. For the second study, emphasis was on data indicating the presence of transfer, and understanding the types of thinking students engaged in during transfer. Table 5.4 summarises the coding framework used for this, illustrating how each Table 5.3 Code labels and descriptions used in the simulation study. (From Kolb, 1984) Code label Concrete experience Descriptive observation & thinking Reflective observation & thinking Conceptualising

Experimenting

Description Introducing students to the tasks. Teacher-led focus activity. Students describe observed events (what happened?) with no speculation/explanation of reason, or expressed intent to find out (why?). Students question observed events. Evidence of seeking explanation or reason, with explicit or implied reference to prior learning. Tentative generalised ideas or theories about how components function and/or how circuits should be built (procedures), and/or conceptual ideas or theories about why operating circuits need to be constructed in particular ways. Applying tentative theories and testing ideas within and between simulations.

Table 5.4 Code labels and descriptions used in the transfer study. (Adapted from Falloon, 2019b) Cognitive Strategy Indicator (Georghiades, 2000) Revisiting learned processes and concepts

Cognitive dimension (Krathwohl, 2002) Remember

Code label

Description

Recalling learned concepts or procedures Understanding the function and purpose of materials and components Applying learned procedures, concepts and materials Analysing and debugging

Recalling learned concepts or procedures associated with circuit design and/or construction Demonstrating understanding of the function and purpose of circuit components

Evaluating and comparing the applicability of procedures, concepts and component knowledge to a new situation Systematic application of concepts, materials and procedures to create new circuit designs

Identifying the use and purpose of learned concepts and materials

Understand

Applying learned materials and processes in different circumstances Being aware of and analysing difficulties and differences

Apply

Making comparisons between prior and current conceptions

Evaluate

Evaluating and comparing

Handling learned materials or processes in different ways

Create

Creating new designs

Analyse

Applying learned concepts, materials or procedures to circuit design and/or construction Analysing, debugging, fault-finding components and designs in non or incorrectly operating circuits

Code colour (sample data)

116

G. Falloon

code was aligned with Georghiades’s (2000) cognitive indicators (to identify transfer events) and Krathwohl’s (2002) cognitive dimensions (to understand thinking applied during the events).

5.5.1

Data Coding

Figure 5.3 illustrates a sample Studiocode timeline and code template from study 1. The same method was used to code data in study 2, although the code template differed. Data aligned with the codes detailed in Tables 5.3 and 5.4 were logged on the timelines through activation and deactivation of code buttons on the template (see right side of Fig. 5.3). Due to the volume of recorded data across both studies (in excess of 60 hours) a purposive sample from each was selected for analysis, based on teacher recommendations for groups reflecting a range of abilities and levels of general learning engagement. In total, data from 9 groups were analysed from study 1 and 12 from study 2. The lesser number in study 1 was due to licence

Fig. 5.3 A sample Studiocode timeline and video, with coding framework (right)

5 Using Animated Simulations to Support Young Students’ Science Learning

117

Fig. 5.4 Sample data from the transfer study (study 2). Adapted from Falloon, 2019b

restrictions on the recording software installed on the iPads, while no such restrictions existed in study 2. Regardless, data from the same students were selected for analysis from each study, to enable tentative comparisons to be made. The author and a research assistant independently coded 20 hours and 11 hours of recordings (study 1 and 2 respectively), after which inter-rater agreement calculations were performed. Kappa scores ranged from 0.604–0.865, indicating good to very good inter-rater agreement across both studies (Landis & Koch, 1977). The stimulated recall interviews (SRI) and audio from the video recordings were transcribed and integrated into data tables, supported by stills taken from the videos. An example from study 2 is shown in Fig. 5.4. The coloured highlighted sections in the example indicate data coded against the relevant transfer strategy indicators as described in Table 5.4, while (P) denotes data coded as procedural knowledge and (C) as conceptual knowledge.

118

5.6 5.6.1

G. Falloon

Findings Conceptual and Procedural Learning and Transfer

The first research question investigated students’ procedural and conceptual learning using the four simulations in study 1, while the second sought evidence of their ability to transfer this to equipment tasks in study 2. Knowledge included understanding how to build circuits of different designs, the characteristics, requirements, and control of operating circuits (e.g., continuous current, effect of multiple resistors in series circuits), and the function of different circuit components (e.g., switches, wires). ‘In common’ conceptual data from both studies have been combined and charted below (Fig. 5.5). Doing this provided a single representation of total event counts from each study. However, it is important to reiterate the rationale for combining them in this way is not to calculate the ‘learning value’ of simulations based on comparing the number of event counts. Instead, the purpose of the chart is simply to communicate the number of occurrences coded as concept development (in study 1) and concept transfer (in study 2). Table 5.5 summarises data from the studies that were not ‘in common’. Specifically, these relate to conceptual understandings of the characteristics of parallel circuits in study 1, and procedural knowledge of the placement or function of components in study 2.

Fig. 5.5 ‘In common’ circuit concept event counts Table 5.5 Summary of concepts not ‘in common’ Study Simulation-only (study 1) Simulation-only (study 1) Transfer (study 2)

Concept Parallel circuits: Appliances get the same voltage Parallel circuits: Current has more than one path Component placement or function

Event count 37 26 104

5 Using Animated Simulations to Support Young Students’ Science Learning

119

Reviewing results for the ‘in common’ concepts, it is apparent most sampled students found transferring concepts from the simulations to the physical tasks, very challenging. Notwithstanding the difference in the volume of data analysed in each study, only 84 events were identified that suggested some conceptual transfer had occurred. This contrasted with 362 events indicating conceptual learning within the simulation-only study. While it was not the intention of these studies to base assessment of the educational worth of simulations on quantitatively measured transfer, these results do suggest the young students found conceptual development and transfer much easier within and between the simulations (near transfer), than from the simulations to the equipment tasks (far transfer). However, while conceptual transfer was comparatively rare, data aligned with procedural transfer – that is, knowledge of how to build operating circuits, were far more prevalent. In total, 263 events were coded as indicating basic procedural transfer in study 2, with the majority (125) of these aligned with how to correctly connect components in circuits of different designs, and 104 with the positioning and operation of components (their function). Only 34 events were identified as procedural knowledge being applied to debugging or fault-finding in non-operating circuits. Almost all procedural events were direct transfer in nature. Video, often supported by oral evidence, indicated the students simply recalled or remembered learnt procedures from the simulations and copied them to the equipment tasks, as illustrated by the green (recall) and grey (apply) highlighted data in the sample. Very few procedural transfer events were associated with evidence of conceptual understanding indicating knowledge of why particular procedures needed to be followed, or the results from doing so.

5.6.2

Cognitive Processes

Both studies focused on the cognitive processes students engaged in when solving problems, specifically investigating the contributions of different thinking types to knowledge building, problem resolution, and learning transfer between the simulations, and from the simulations to the equipment tasks. As each study adopted a different theoretical framework aligned with its principal objective (process of knowledge development vs process of knowledge transfer), results for each are presented separately.

5.6.2.1

The Simulation Study (Study 1)

Figure 5.6 charts data from study 1 against the codes developed from Kolb’s (1984) ELT model (Table 5.3). The red bars represent event counts for thinking coded as descriptive, and stem from students’ responses to the results of their actions when building their circuits using the simulations. The blue bars represent the same for thinking coded as reflective. Descriptive thinking referenced data where students simply described what happened, while with reflective thinking they both described

120

G. Falloon Thinking types, Conceptualising and Experimenting Event Counts (developed from Kolb, 1984)

200

180

160

140

Event counts

120

100

80

60

40

20

0 Descriptive Reflective

Descriptive Reflective

Thinking type (when observing phenomena)

Conceptualising (generating theories from thinking types)

Experimenting

Experiential Learning Theory (ELT) element

Fig. 5.6 Study 1 event counts for descriptive and reflective thinking. (From Kolb, 1984)

what happened and offered explanations for why it happened (tentative theory building). The first set of bars denote the thinking types associated with students’ observations of outcomes following their attempts at circuit construction. The second set report the extent to which the two thinking types supported conceptualising - that is, their relative contribution to the generation of basic explanatory theories for why outcomes occurred. The brown bar represents event counts for data coded as experimenting. These data align with the final stage in

5 Using Animated Simulations to Support Young Students’ Science Learning

121

Kolb’s ELT model, and indicates evidence found of students testing and evaluating tentative theories generated mainly from reflective thinking, within and between the simulations. Data coded as experimenting required additional confirmatory evidence to minimise the chance of random trial and error or ‘playing’ being interpreted as informed experimentation. Evidence usually came from recorded oral statements or exchanges between the students, that suggested the deliberate application of tentative theories to decision-making relating to circuit design and construction. As illustrated in Fig. 5.6, there was significantly more translation of reflective thinking into conceptualisation, despite the predominance of descriptive thinking in students’ interactions with the simulations. The better translation rate by students who reflected on and attempted to make sense of what they observed, contributed to higher levels of theory generation and testing between the simulations, compared to those who only described what they observed. While both thinking types played a role in experimentation, the degree of ‘fading’ in translation to experimenting was much greater for students who offered no explanatory ideas. Interestingly, data indicated a much stronger association of descriptive thinking with procedural transfer, and reflective thinking with conceptual transfer between the simulations. Students whose data were coded as mainly representing descriptive thinking, tended to simply copy designs between the simulations, offering no conceptual explanations of why circuits needed to be constructed as they did. Conversely, those whose thinking included that coded as reflective provided conceptual explanations in some form, generally supporting or justifying their circuit-building decisions. Of note, however, is that not all of the students’ explanations or emerging theories were scientifically accurate. In fact, as will be discussed later, how some students interpreted the content in some simulations - particularly the animated visual representations, actually appeared to trigger the formation of misconceptions.

5.6.2.2

The Transfer Study (Study 2)

Data from study 2 were coded against the framework summarised in Table 5.4. The codes aligned Georghiades (2000) metacognitive strategy indicators of transfer with Krathwohl’s (2002) cognitive dimensions, to identify data signalling transfer and the type of thinking activated during transfer. Table 5.6 reports event counts by group against each code, with the coloured indicators corresponding to the illustrative data samples in Fig. 5.4. In selected data, a total of 384 events across all groups and lessons were coded as indicative of transfer of some form. Data coded as recalling concepts or procedures and applying learned procedures, concepts and materials was predominantly associated with students’ direct recall of learnt procedures from the simulations and their application to the equipment tasks. This direct transfer interpretation is supported by oral data that makes explicit reference to construction procedures and circuit designs being copied from the simulations, as shown in row

122

G. Falloon

Table 5.6 Cognitive strategy event counts by group. (From Georghiades, 2000) Group

Recalling learned concepts or procedures

Evaluating and comparing

Analysing and debugging

Understanding the function and purpose of materials and components

Creating new designs

Applying learned procedures, concepts and materials

T&A

12

5

4

11

2

21

C&S

6

5

2

10

0

24

J&D

6

7

5

15

1

23

P&A

5

5

4

13

0

18

H&T

4

7

3

9

1

19

L&M

2

6

3

12

0

19

J&S

3

7

6

10

1

27

J&P

4

6

2

9

0

20

Total

42

48

29

89

5

171

2 of the sample. In terms of Krathwohl’s (2002) cognitive process dimensions, these represent the basic thinking skills of remembering and application. Although the majority of transfer events (56%) were ‘recall and apply’ in nature, solid evidence was found of students’ transfer of basic understanding of what particular components do in operating circuits (switches, wires, cells etc.). While in scientific and conceptual terms many understandings were incomplete or naively stated, they did suggest some transfer of knowledge about the role and placement of components in operating circuits. An example of this can be seen in the purple highlighted sample data, where students J and D are discussing the positioning of wires on their knife switch, ‘so the power can go’. These data aligned with Krathwohl’s (2002) Understand(ing) cognitive dimension, and although representing greater thinking complexity than that engaged in direct transfer, limited evidence was found associating these data with conceptual understanding – for example, how switches control current in operating circuits. Data aligned with Krathwohl’s (2002) higher order cognitive dimensions of evaluating, analysing and creating were comparatively scarce (n ¼ 82). Data coded as evaluating and comparing were associated with emerging explanatory theories about how the different circuits work – scientifically accurate or otherwise, and how particular components functioned. These data were qualitatively different to that coded as evidence of more basic understanding, as they were supported by additional details indicating tentative conceptual understandings about how circuits worked, such as why multiple bulbs in series circuits were dim (e.g., Fig. 5.7). Data coded as Analysing generally aligned with students’ problem solving and debugging of non or incorrectly operating circuits. However, only 29 events were identified as students attempting to analyse and correct faults. Interestingly, it appeared most students lacked systematic strategies for analysing and rectifying problems, choosing instead to dismantle their circuits and start again. Data highlighted the potential value of teaching systematic fault-finding strategies, as most issues related to poor alligator clip connections, loose fitting cells in holders, or

5 Using Animated Simulations to Support Young Students’ Science Learning

123

Fig. 5.7 Example of the emergence of explanatory theories about series circuits. (From Falloon, 2019b)

Fig. 5.8 Students T&A extended series circuit

bulbs that had not been fully screwed in – rather than incorrect circuit design. Finally, five events were coded as Creating, but only two of these were considered to result from the deliberate application of learnt concepts. The most significant of these events were the actions of students T&A, who displayed solid conceptual knowledge of voltage drop in multi-bulb series circuits, addressing this by the unprompted addition and correct placement of an extra cell (Fig. 5.8).

5.7

Discussion

This chapter synthesised two studies investigating knowledge development and transfer within and from animated simulations, designed to introduce young students to simple circuit-building procedures and concepts. The studies also investigated strategies students applied to solving design and construction problems presented by different circuit challenge scenarios, with specific emphasis on the thinking skills operationalised in each.

124

5.7.1

G. Falloon

Procedural and Conceptual Development and Transfer

Earlier-reviewed literature indicated generally positive outcomes from studies on simulation use for adult training and in older student education, particularly for the development of transferable procedural skills or knowledge associated with replicable processes (e.g., Ganier et al., 2014; Rodríguez et al., 2006) or scientific methods (e.g., Kumar & Sherwood, 2007; Marshall & Young, 2006). Results from the current studies confirm similar benefits for younger students, although data clearly indicate that students found learning and transferring procedural knowledge between the simulations (near or vertical transfer) much easier than the far or horizontal transfer tasks involving physical materials. This finding is consistent with the work of Bossard et al. (2008) and Subedi (2004), whose research concluded close alignment between procedural transfer and near (or vertical) transfer scenarios. Notwithstanding this, and despite most students adopting ‘direct transfer’ methods, sufficient evidence was present supporting the use of simulations for teaching students of this age basic technical procedures (know how), that appeared reasonably robust and transferable to ‘real world’ scenarios. While basic procedural learning within and transfer from the simulations was solid, developing and transferring conceptual understandings proved far more challenging. Conceptual knowledge development was evaluated using evidence that indicated the students held more abstract understandings about why operating circuits needed to be constructed in particular ways, and in some instances, could apply these across the simulations, and/or transfer them to the equipment tasks. As for procedural learning, significantly more occurrences coded as conceptual learning were observed in study 1, as students quickly learnt and transferred simple concepts between the simulations. Over two-thirds of these linked to operating circuits needing to be closed, and the role of switches. However, it should be noted that while these have been identified as conceptual in study 1, most were judged on visual evidence alone, such as students’ careful checking of wire connections or switch position. Generally, any oral evidence associated with these events was simplistic in nature – such as, ‘I know . . . the wire isn’t touching . . . it isn’t touching the battery’ (student T, transcript, Exploriments: Simple Circuits) rather than indicating deeper, more scientific understandings for why connections must be sound (e.g., continuous current). As detailed previously there were exceptions to this, with a few students reflecting on and offering explanatory reasons (scientifically accurate or otherwise) for their observations. Interestingly, the visual representations in some simulations appeared to influence the accuracy of concept formation, with one in particular (Electronics for Kids) appearing to trigger possible misconception development through progressively diminishing the number of visible electron sprites circulating in the wires, as more bulbs were added. Three groups associated this with the popular misconception of power from the cell being ‘used up’ or consumed. It could reasonably be argued that difficulties these students experienced in conceptual development and transfer could be expected, given their young age. However, while not common, evidence was present that the simulations helped

5 Using Animated Simulations to Support Young Students’ Science Learning

125

seed simple science concepts that a few groups managed to transfer to the equipment tasks. This suggests more research needs to be done to elevate simulation use in schools beyond predominantly procedural learning, to investigate optimal pedagogies and learning designs supporting their use for conceptual development. This conclusion is consistent with Georghiades’s (2000) call for transfer research in science education to move beyond basic skill transfer, to learn more about how conceptual transfer can be achieved – in this example, supported by animated simulations. The results of these studies also add weight to arguments presented by researchers such as Bransford (2000), who comment that young students, given appropriate support, are capable of the thinking at levels necessary to support more abstract conceptual transfer.

5.7.2

Cognitive Strategies and Process Development

Findings from both studies support the value of technology, in this case simulations, for scaffolding development of higher order thinking capabilities in young students. This is consistent with the work of Wang, Kinzie, McGuire, and Pan (2010), who comment on technology’s value for supporting “various aspects of children’s learning, including conceptual and cognitive development, literacy skills (and) mathematics knowledge and competence” (p. 382). Kolb’s (1984) Experiential Learning Model provided a valuable lens through which to understand the type of thinking strategies and processes students engaged in study 1. While it revealed a predominance of descriptive thinking from observations that mostly aligned with direct procedural transfer, links were also identified to reflective thinking that contributed to concept development, which, on occasions, resulted in the testing of emergent theories through experimentation. While some ‘fading’ occurred as students moved from observing to conceptualising and then to experimenting, good examples of higher order reflective and analytical thinking were present in some data, as students constructed tentative explanations for their observations. Moreover, although most thinking was descriptive in nature and could be considered lower order, interpreting and describing events from observations is still an important and worthwhile skill for young students to develop. In addition, descriptive thinking also held some association with concept formation, although this was not as strong as for students who displayed more reflective thinking capability. While a small number of students were able to convert descriptive thinking into testable theories, this was not as robust as for those who displayed more reflective capacities, and resulted in significant ‘slippage’ as students reverted back to basic reproduction of learnt procedures. Similarly, when examining the cognitive strategies and thinking skills operationalised in study 2, the predominance of basic ‘remember and apply’ thinking could be interpreted as a poor result. However, developing these thinking skills is still important for young students, and pragmatically, how they were used could be

126

G. Falloon

viewed as a completely reasonable response to achieving the tasks they were set. Also, methodological limitations in data analysis due to the requirement to have both oral and visual evidence supporting interpretation as ‘higher order’ thinking in study 2 (analysing, evaluating, creating), is likely to have led to the under-reporting of conceptual transfer and/or its interpretation as procedural transfer. That is, it is quite possible some students held solid conceptual knowledge about the circuits and their design, but did not express them. Notwithstanding these limitations, enough evidence existed of conceptual transfer occurring – albeit naïve in many cases, and that these events triggered higher order thinking as the students abstracted from the simulations to the equipment tasks. This finding is consistent with adult studies that point to a positive relationship between learning transfer and metacognition (e.g., Scharff et al., 2017).

5.8

Conclusions and Implications for Practice

The proliferation of educational digital resources that embed animated simulations has undoubtedly opened new potentials for educators to bring to their classrooms previously difficult-to-access learning experiences. This has been supported by the roll out of device provisioning programs in schools such as BYOD, and the advent of resource repositories such as Apple’s AppStore and Google’s Play, where literally tens of thousands of low or no cost, so-called educational apps – many of them containing animated simulations, can be accessed. While this is positive overall, very few studies have been carried out exploring what and how students learn from these resources, and even less investigating whether any learning can benefit ‘real world’ tasks. While it is challenging to fully report the results of two substantial studies in one chapter, some implications can be drawn from this synthesis useful to both teachers and content designers. On the positive side, the simulations were generally effective for teaching these young students replicable and transferable procedural knowledge. While more evidence of this existed for near transfer between simulations, this was robust enough to transfer to working with different materials, albeit on similar tasks. Similarly, evidence was also present of some conceptual development and transfer, although in several examples generated theories lacked scientific accuracy. This finding is significant, as it behoves teachers using simulations to monitor students closely, and be prepared to intervene in situations where misinterpretations may lead to misconceptions. Research in science education over a number of years highlights difficulties changing students’ conceptions, once they have been established (e.g., Tao & Gunstone, 1999). Related to this, developers of simulation-based digital resources would be well advised to spend more time trialling their products with students of the target age, to ensure that they are appropriately designed to as much as possible support accurate concept development. Presently, app repository star or

5 Using Animated Simulations to Support Young Students’ Science Learning

127

number rating systems tend to communicate popularity, paying little regard to educational quality. Therefore, teachers should review simulations very carefully before selecting them to use with their students, and not assume that students will interpret with scientific accuracy, phenomenon represented in them. Findings from both studies showed some alignment with similar work involving older students and adults with regards to the engagement of higher order thinking processes. Predictably given their age, the sophistication and frequency of this was not at the same level as the adult studies, although enough evidence was present to argue a case for simulation-based learning and transfer tasks as useful experiences for young students’ thinking skill development. Importantly for teachers, findings indicate the benefits of learning collaboratively with simulations, with the oral exchanges between students serving to enhance problem solving and ‘ramp up’ the levels of thinking as they discussed and debated possible solutions. Teacher adoption of a guided discovery approach and use of open-ended questions and prompts as they circulated around the groups was integral to thinking skill development, supporting the students to focus on areas where potential issues may exist, without providing them with solutions. Understanding was also enhanced by short plenary sessions held at the end of each lesson, when the students shared outcomes and their initial ideas about how the circuits worked. These sessions were also good opportunities for teachers to gauge their students’ learning, and consolidate and at times clarify emerging ideas relating to current flow and how components such as switches, work. This final stage was supported by whole class activities where teachers modelled more abstract concepts such as electron flow in wires, using learning experiences from the National Geographic education resource website.1 Finally, given the increasing volume of educational apps with embedded simulations specifically targeting younger learners, more empirical studies are needed examining what and how concepts are learnt from simulations. This knowledge can contribute to the development of pedagogies and learning designs that optimise the inherent potential of simulations for building understanding in previously difficultto-access areas of learning, and to make these accessible to younger learners. Additionally, further work is needed investigating learning transfer from simulations and other app-based resources containing visual representations – particularly in foundational learning areas such as literacy and numeracy, to ensure any knowledge derived from them is robust and accurate. While results of these studies generally indicated positive cognitive outcomes in terms of thinking skill development, it is equally important that these translate to correct understandings. The studies outlined in this chapter should be viewed only as a starting point to addressing these important research agendas.

1

See https://www.nationalgeographic.org/activity/circuits-friends/

128

G. Falloon

References Baltzis, K., & Koukias, K. (2009). Using laboratory experiments and circuit simulation IT tools in an undergraduate course in analog electronics. Journal of Science Education and Technology, 18(6), 546–555. Bossard, C., Kermarrec, G., Buche, C., & Tisseau, J. (2008). Transfer of learning in virtual environments: A new challenge. Virtual Reality, 12, 151–161. Bransford, J. (2000). Learning and transfer. In J. Bransford, A. Brown, & R. Cocking (Eds.), How people learn: Brain, mind, experience and school (pp. 51–78). Washington, DC: National Academy Press. Bryce, D., & Whitebread, D. (2012). The development of metacognitive skill: Evidence from observational analysis of young children’s behavior during problem-solving. Metacognition and Learning, 7(3), 197–217. Burns, B., & Hamm, E. (2011). A comparison of concrete and virtual manipulative use in third and fourth grade mathematics. School Science and Mathematics, 111(6), 256–261. Chen, Y.-L., Pan, P.-R., Sung, Y.-T., & Chang, K.-E. (2013). Correcting misconceptions on electronics: Effects of a simulation-based learning environment backed by a conceptual change model. Educational Technology and Society, 16(2), 212–227. de Jong, T. (2005). The guided discovery principle in multimedia learning. In R. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 215–229). Cambridge, UK: Cambridge University Press. de Jong, T., & Van Joolingen, W. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68(2), 179–201. de Koning, B., Tabbers, H., Rikers, R., & Paas, F. (2007). Attention cueing as a means to enhance learning from an animation. Applied Cognitive Psychology, 21, 731–746. Evagorou, M., Korfiatis, K., Nicolaou, C., & Costas, C. (2009). An investigation of the potential of interactive simulations for developing system thinking skills in elementary school: A case study with fifth-graders and sixth-graders. International Journal of Science Education, 31(5), 655–674. Falloon, G. W. (2013). Young students using iPads: App design and content influences on their learning. Computers & Education, 68, 505–521. Falloon, G. W. (2014). What’s going on behind the screens? Journal of Computer Assisted Learning, 30(4), 318–336. Falloon, G. W. (2015). What’s the difference? Learning collaboratively using iPads in conventional classrooms. Computers & Education, 84, 62–77. Falloon, G. W. (2016). An analysis of young students’ thinking when completing basic coding tasks using scratch Jnr. On the iPad. Journal of Computer Assisted Learning, 32(6), 576–593. Falloon, G. W. (2017). Mobile devices and apps as scaffolds to science learning in the primary classroom. Journal of Science Education and Technology, 26(6), 613–628. Falloon, G. W. (2019a). Using simulations to teach young students science concepts: An experiential learning theoretical analysis. Computers & Education, 135, 138–159. Falloon, G. W. (2019b). From simulations to real: Investigating young students’ learning transfer from simulations to real tasks. British Journal of Educational Technology, 1–20. https://doi.org/ 10.1111/bjet.12885 Falloon, G. W., & Khoo, E. (2014). Exploring young students’ talk in iPad-supported collaborative learning environments. Computers & Education, 77, 13–28. Foti, S., & Ring, G. (2008). Using a simulation-based learning environment to enhance learning and instruction in a middle school science classroom. Journal of Computers in Mathematics and Science Teaching, 27(1), 103–120. Fund, Z. (2007). The effects of scaffolded computerised science problem-solving on achievement outcomes: A comparative study of support programs. Journal of Computer-Assisted Learning, 23, 410–424.

5 Using Animated Simulations to Support Young Students’ Science Learning

129

Ganier, F., Hoareau, C., & Tisseau, J. (2014). Evaluation of procedural learning transfer from a virtual environment to a real situation: A case study on tank maintenance training. Ergonomics, 57(6), 828–843. Georghiades, P. (2000). Beyond conceptual change learning in science education: Focusing on transfer, durability and metacognition. Educational Research, 42(2), 119–139. Höffler, T., & Leutner, D. (2007). Instructional animation versus static pictures: A meta-analysis. Learning and Instruction, 17(6), 722–738. Hulshof, C., & de Jong, T. (2006). Using just in time information to support scientific discovery learning in a computer-based simulation. Interactive Learning Environments, 14(1), 79–94. Huppert, J., Lomask, S., & Lazarowitz, R. (2002). Computer simulations in the high school: Students’ cognitive stages, science process skills and academic achievement in microbiology. International Journal of Science Education, 24(8), 803–821. Klahr, D., Triona, L., & Williams, C. (2007). Hands on what? The relative effectiveness of physical versus virtual materials in an engineering design project by middle school children. Journal of Research in Science Teaching, 44(1), 183–203. Kolb, D. (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice-Hall. Krathwohl, D. (2002). A revision of Bloom’s taxonomy. Theory Into Practice, 41(4), 212–218. Kumar, D., & Sherwood, R. (2007). Effect of a problem based simulation on the conceptual understanding of undergraduate science education students. Journal of Science Education and Technology, 16(3), 239–246. Landis, J., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. Lieberman, D., Bates, C., & So, J. (2009). Young children’s learning with digital media. Computers in the Schools, 26(4), 271–283. Limniou, M., Papadopoulos, N., Giannakoudakis, A., Roberts, D., & Otto, O. (2007). The integration of a viscosity simulator in a chemistry laboratory. Chemistry Education Research and Practice, 8(2), 220–231. Lowe, R. (2003). Animation and learning: Selective processing of information in dynamic graphics. Learning and Instruction, 13(2), 157–176. Lunetta, V., & Hofstein, A. (1981). Simulation in school science. Science Education, 65(3), 243–252. Manlove, S., Lazonder, A., & de Jong, T. (2009). Collaborative versus individual use of regulative software scaffolds during scientific inquiry learning. Interactive Learning Environments, 17(2), 105–117. Marshall, J., & Young, E. (2006). Preservice teachers’ theory development in physical and simulated environments. Journal of Research in Science Teaching, 43(9), 907–937. Noorhidawati, A., Ghazel, S., & Ghalebandi, S. (2015). How do young children engage with mobile apps? Cognitive, psychomotor and affective perspectives. Computers & Education, 87, 385–395. Olympiou, G., & Zacharia, Z. (2011). Blending physical and virtual manipulatives: An effort to improve students’ conceptual understanding through science laboratory experimentation. Science Education, 96(1), 21–47. Plass, J., Homer, B., & Hayward, E. (2009). Design factors for educationally effective animations and simulations. Journal of Computing in Higher Education, 21, 31–61. Prain, V., Tytler, R., & Peterson, S. (2009). Multiple representations in learning about evaporation. International Journal of Science Education, 31(6), 787–808. Reimer, K., & Moyer, P. (2005). Third-graders learn about fractions using virtual manipulatives: A classroom study. Journal of Computers in Mathematics and Science Teaching, 24(1), 5–26. Rodríguez, J., Gutiérrez, T., Sánchez, E., Casado, S., & Aguinaga, I. (2006). Training of procedural tasks through the use of virtual reality and direct aids. In C. Lányi (Ed.), Virtual reality and environments (pp. 43–68). Rijeka, Croatia: InTech Open Science.

130

G. Falloon

Rutten, N., van Joolingen, W., & van der Veen, J. (2012). The learning effects of computer simulations in science education. Computers & Education, 58, 136–153. Scharff, L., Draeger, J., Verpoorten, D., Devlin, M., Dvorakova, L., Lodge, J., et al. (2017). Exploring metacognition as support for learning transfer. Teaching & Learning Inquiry, 5(1), 1–14. Smetana, L., & Bell, R. (2012). Computer simulations to support science instruction and learning: A critical review of the literature. International Journal of Science Education, 34(9), 1337–1370. Steen, K., Brooks, D., & Lyon, T. (2006). The impact of virtual manipulatives on first grade geometry instruction and learning. Journal of Computers in Mathematics and Science Teaching, 25(4), 373–391. Stieff, M., & Wilensky, U. (2003). Connected chemistry: Incorporating interactive simulations into the chemistry classroom. Journal of Science Education and Technology, 12(3), 285–302. Subedi, B. (2004). Emerging trends of research on transfer of learning. International Education Journal, 5(4), 591–599. Tao, P.-K., & Gunstone, R. (1999). Conceptual change in science through collaborative learning at the computer. International Journal of Science Education, 21(1), 39–57. Timms, M., Moyle, K., Weldon, P., & Mitchell, P. (2018). Challenges in STEM learning in Australian schools. Camberwell, VIC: Australian Council for Educational Research. van Berkum, J., & de Jong, T. (1991). Instructional environments for simulations. Education & Computing, 6, 305–358. van der Meij, J., & de Jong, T. (2006). Supporting students’ learning with multiple representations in a dynamic simulation-based learning environment. Learning and Instruction, 16, 199–212. Wang, F., Kinzie, M., McGuire, P., & Pan, E. (2010). Applying technology to inquiry-based learning in early childhood education. Early Childhood Education Journal, 37, 381–389. Wang, T., & Tseng, Y. (2018). The comparative effectiveness of physical, virtual, and virtualphysical manipulatives on third-grade students’ science achievement and conceptual understanding of evaporation and condensation. International Journal of Science and Mathematics Education, 16(2), 203–219. Zacharia, Z. (2005). The impact of interactive computer simulations on the nature and quality of postgraduate science teachers’ explanations in physics. International Journal of Science Education, 27, 1741–1767. Zacharia, Z., Loizou, E., & Papaevripidou, M. (2012). Is physicality an important aspect of learning through science experimentation among kindergarten students? Early Childhood Research Quarterly, 27(3), 447–457. Zacharia, Z., Olympiou, G., & Papaevripidou, M. (2008). Effects of experimenting with physical and virtual manipulatives on students’ conceptual understanding in heat and temperature. Journal of Research in Science Teaching, 45(9), 1021–1035.

Garry Falloon is Professor of STEM Education and Digital Learning at Macquarie University. Previously he was Professor of Digital Learning in the Faculty of Education at Waikato University in Hamilton, New Zealand. His background includes 22 years teaching and leadership of primary and secondary schools in New Zealand, Education Foundation Manager at Telecom New Zealand, working with Microsoft in the Partners in Learning and Digital Learning Object projects, and as project lead for the New Zealand Government’s $10m Digital Opportunities Projects. His research interests include mobile learning, digital learning in primary and middle schools, online and blended learning, curriculum design, pedagogy and assessment in digitally-supported innovative learning environments, learning in primary science and technology, and educational research methods.

Chapter 6

Promoting Scientific Understanding through Animated Multimodal Texts Maximiliano Montenegro, Alejandra Meneses, Soledad Véliz, José Pablo Escobar, Marion Garolera, and María Paz Ramírez

6.1

Animated Text and Reading Comprehension in Fifth Grade

While science education increasingly includes hands-on inquiry and activities, reading comprehension remains a fundamental skill for knowledge building (Patterson, Roman, Friend, Osborne, & Donovan, 2018; Pearson, Moje, & Greenleaf, 2010). However, reading in science is a challenging process for elementary school students because of the abstract scientific concepts involved and also because these scientific ideas are organized and logically connected by an academic discourse that they are not familiar with (Fang, 2006; Snow, 2010). In fact, McNamara, Ozuru, and Floyd (2011) contributed empirical evidence to the so-called “fourth-grade slump” by comparing students’ reading comprehension according to prior knowledge (low vs. high), type of discursive genre (literary narrative vs. scientific explanatory) and degree of textual cohesion (low vs. high). The results showed that comprehension of science texts among fourth-grade students is significantly lower than in narrative texts in all measures, both receptive and productive. Coincidentally, the 2016 results of the Progress in International Reading Literacy Study (PIRLS) showed statistically significant differences in fourth-grade students performance in reading comprehension between expository scientific texts and narrative literary texts, mainly in favor of the latter; specifically, Chilean students’ performance in expository scientific texts was fifteen points lower than in narrative literary texts (Mullis, Martin, Foy, & Hooper, 2017).

M. Montenegro (*) · A. Meneses · S. Véliz · J. P. Escobar · M. Garolera Pontificia Universidad Católica de Chile, Santiago, Chile e-mail: [email protected] M. P. Ramírez ONG Neyün, Santiago, Chile © Springer Nature Switzerland AG 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_6

131

132

M. Montenegro et al.

Although scientific texts are multimodal –the construction of meaning occurs in the interaction of verbal and visual modes– research on reading comprehension of science texts has instead focused on the role of verbal language (Kloser, 2013, 2016; McNamara et al., 2011) and less research has been done that addresses the role of visual language in reading comprehension. McTigue and Slough (2010) propose orientations for the construction of more accessible scientific texts for students, and they emphasize not only the degree of concreteness of verbal language and a voice to engage the reader, but also a high-quality visual language that relates with verbal language. Ge, Unsworth, and Wang (2017) explored the effect of redesigned biological images with explicit visual cues through a quasi-experimental study of on-line reading. Their research showed that tree-structured diagrams significantly increase understanding of scientific concepts about taxonomy and categorization, but the use of arrows for representing the direction of energy transference in the food chain did not show any effect. They also found that prior knowledge was related to the effect of diagrams on the understanding of scientific concepts. Meneses, Escobar, and Véliz (2018) explored the effects of multimodal texts to scaffold the reading comprehension of fifth-grade Chilean students. The researchers redesigned a text on energy transfer, taking into consideration lexical-grammatical resources, explanatory structure, the function of images, and the relations between image and text to support the understanding of a scientific process. The results of their quasi-experimental study showed that there are no statistically significant differences when comparing the means of the group that read the text with low scaffolding in relation to the group that read the text with high scaffolding. However, significant differences were found when examining the effect of multimodal texts with high scaffolding on students with low comprehension skills. This study provides evidence on how multimodal scaffolded texts, in which there is an enriched interplay between verbal and visual languages, can boost the comprehension and learning of students who have lower reading comprehension. Therefore, this study poses new questions about types of scaffolds for diverse readers and how multimodality, in particular, can increase opportunities for understanding scientific processes. This is important for groups that have traditionally struggled in science because of their minimal prior knowledge and low reading comprehension skills. Research of this type can contribute to reducing inequalities by proposing criteria to guide the construction of texts that support the learning of all students. Less research has been done to explore whether animated multimodal texts can promote understanding for students who have not consolidated their reading comprehension skills. Dalton, Proctor, Uccelli, Mo, and Snow (2011) investigated how digital scaffolded texts focused on reading strategies and vocabulary could improve reading comprehension for monolingual (English-speaking) and bilingual fifthgrade students. Students read a digital narrative and explanatory text under three conditions: (1) strategies adapted from the reciprocal teaching proposal (Palincsar & Brown, 1984) that included predicting, asking questions, clarifying, summarizing, visualizing, and feeling; (2) interactive vocabulary learning of Spanish-English cognates; and, (3) integration of both elements. The results indicated that groups in the last two conditions performed significantly higher than the strategy-focused

6 Promoting Scientific Understanding through Animated Multimodal Texts

133

group. Also, monolingual students performed better in reading comprehension of explanatory texts and in vocabulary achievement compared to bilingual students. Dalton and Palincsar (2013) developed an intervention using narrative and explanatory digital texts to compare the effects of three forms: a static version, an interactive version requiring manipulation of a diagram, and an interactive version that included coaching. The fifth-grade students who read any of the interactive texts performed significantly higher than in the static version, providing evidence on how interactivity allows readers to establish the relation between diagrams and verbal information. However, the study did not find a reduction in the comprehension gap between struggling and more advantaged readers. The findings in this study suggest that there is a need for investigating how animated multimodal reading can scaffold the comprehension and learning of low skilled readers. Our research aims to provide empirical evidence on how animated multimodal texts contribute to understand scientific concepts in the context of energy transfer in an ecosystem for Chilean fifth-grade students using different scaffoldings. Therefore, we seek to provide criteria for designing effective animated text versions, particularly for students with minimal prior scientific knowledge and low reading comprehension skills so that no student is left behind when it comes to disciplinary literacy.

6.1.1

Multimodality and Science Understanding

Understanding and building scientific knowledge involves the learning of a specialized language, which is different from the language used in everyday life (Evagorou & Osborne, 2010; Lemke, 2004). From a Systemic Functional Linguistic (SFL) point of view, Halliday and Martin (1993) posit that scientific discourse seems obscure to students because they are neither familiar with technical terms nor with grammatical constructions that condense information and establish logical causeand-effect relations in explaining scientific processes. From the perspective of science education, Evagorou and Osborne (2010) argue that scientific language is characterized not only by the use of technical terms but also by the construction of logical relations conveyed through discursive connectors. Indeed, scientific language shares discursive and linguistic resources that configure the academic language register used at school in learning and assessment contexts (Schleppegrell, 2004; Snow & Uccelli, 2009; Uccelli, 2019). However, scientific language encompasses more than written text since visual and mathematical languages are also used for building knowledge (Evagorou & Osborne, 2010; Lemke, 1998, 2004). Scientific discourse is multimodal because there is an interplay among these three languages through different genres, like in the classifications of living and non-living beings at different scales, on explanations of complex dynamic processes, or for reporting scientific experiments (Jewitt & Kress, 2003; Kress, 2009; Lemke, 1998, 2004). As Lemke (1998) suggests, scientific knowledge is constructed by the integration of written text, graphics, photographs, diagrams, tables, and, equations,

134

M. Montenegro et al.

among others. Therefore, multiple languages are used to represent and explain a diversity of natural phenomena. However, these visual and mathematical languages also operate in different and specialized ways than the verbal language used in everyday interactions or for other communicative purposes. From the perspective of school literacy, the multimodal language of science is challenging for elementary school students because disciplinary literacy in science calls for the ability to identify the roles that these different languages play in science communication and knowing how to use them simultaneously and coherently (Ge et al., 2017; Kress, Jewitt, Ogborn, & Charalampos, 2001; Tang, Delgado, & Moje, 2014; Tang & Moje, 2010; Tytler, Prain, Hubber, & Waldrip, 2013; Unsworth, 2004). The difficulties that many students of elementary education face when reading science texts are mainly due to the specialized language that science employs and that it is different from the language of everyday interactions (Lemke, 1998, 2004; Schleppegrell, 2004; Snow, 2010). In effect, Tang and Moje (2010) emphasize that students deal with a series of demands and difficulties related to the specific and precise use of the verbal and visual resources for knowledge building in science. As noted, previous studies have shown that students understand science texts less well than narrative texts (McNamara et al., 2011) and that students who have more prior knowledge and a higher level of comprehension learn more from scientific texts (Ozuru, Dempsey, & McNamara, 2009). Therefore, it is crucial to know how to scaffold science comprehension to improve science learning of students with low prior knowledge or reading skills to foster educational equity. In the last decade, studies have been developed that demonstrate the importance of images in the comprehension and production of science texts (Chang, 2012; Cox, 2005). Consistent with this finding, research from a cognitive psychology perspective highlights the relevant contribution of images in the learning process (Mayer, 2005). On the other hand, literacy research has shown that academic language skills are a significant predictor of reading comprehension (Uccelli, Phillips Galloway, Barr, Meneses, & Dobbs, 2015). However, less research has been done showing the pedagogical potential of multimodality (Meneses, Escobar, & Véliz, 2018; Townsend, Brock, & Morrison, 2018; Wilson & Bradbury, 2016), and, especially, the use of animated texts to scaffold comprehension and learning in science.

6.1.2

Static and Animated Texts in Digital Environments for Science Understanding

Several studies in the last decade, mainly focused on cognitive psychology and emphasizing information processing, have demonstrated the importance of images in the understanding of scientific concepts (Levin & Mayer, 1993; Mayer, 2005; Mayer & Gallini, 1990). However, these studies were challenged by research on seductive details, elements in a text that hinder rather than facilitate students’ understanding (Ardasheva, Wang, Roo, Adesope, & Morrison, 2018; Moreno & Mayer, 2000). The

6 Promoting Scientific Understanding through Animated Multimodal Texts

135

theory of seductive details posits that processing non-relevant visual information may arouse interest in the reader but does not contribute significantly to a coherent construction of what has been read because it overloads the working memory with unnecessary demands. Indeed, Ardasheva et al. (2018) found that there were no statistically significant differences in reading comprehension of adolescent English learners when faced with a traditional text or visually supported text, although those reading the traditional text showed a slightly superior performance. Clearly, the inclusion of visual information does not necessarily improve comprehension, and additional research should identify visual supports that can favor learning in science, particularly for English language learners. Most studies of multimodal reading comprehension in science have been conducted using printed texts with static images. As pointed out earlier, taking a semiotic approach to science education, Meneses, Escobar, and Véliz (2018) redesigned texts to study whether increased verbal and visual language supports could facilitate reading comprehension for monolingual fifth-grade Spanish-speaking students. The results showed no significant differences in the understanding of the scientific concepts between the groups that read the high multimodal scaffolding text and those that read the low scaffolding text. However, among students with lower reading comprehension skills, the text with high verbal and visual scaffolding fostered the understanding of scientific concepts even up to the level of readers with medium abilities. Nevertheless, more research is required on animating multimodal scientific texts to support the understanding of scientific concepts of low-skilled comprehenders. Although animation has been used for decades in the construction of literature, only recently has there been more systematic and empirical research on how animation can foster learning in other disciplinary areas, particularly science. With the rise of digital literacy, the concept of the textbook is being redefined, calling into question the role of static verbal and visual language and the tension between moving images and more traditional ways of meaning-making. Bétrancourt and Chassot (2008) studied an instructional multimedia document defining animations as a series of pictures that advance temporarily according to the purposes of a designer (Bétrancourt & Tversky, 2000). Although some argue that the use of animations can promote the understanding of dynamic processes, Bétrancourt and Chassot (2008) suggest that the key questions are when and why animations can promote learning and they connect these questions to the construction of mental models from processed verbal and visual information (Mayer, 2005; Schnotz & Bannert, 2003). In their review of literature on animation and learning, Tversky, Morrison, and Bétrancourt (2002) concluded that animations are beneficial when representing dynamic processes, when animating fundamental information for the construction of mental models, and when explaining inferences that are not included in static images. However, most of the studies that support multimedia theory have been conducted with undergraduate students in controlled experiments, and there has been less research done on children in authentic contexts, despite the fact that animations can complement the verbal abilities of schoolchildren and facilitate the construction of their mental models (Bétrancourt & Chassot, 2008).

136

M. Montenegro et al.

Using the social semiotic approach, there have been empirical studies of the textimage relation, especially of printed science texts and the comprehension attained by school students (Ge et al., 2017; Meneses, Escobar, & Véliz, 2018). However, little research has compared static texts with the effects of animation on comprehension achieved by school students. Unsworth (2004) examined science explanations in school textbooks with digital formats. One of that study’s conclusions was that the multimodal organization of dynamic texts requires specific reading practices on the part of students. In addition, Unsworth (2004) pointed out that more research is needed to understand how different students read animated texts for various purposes. For this study, following Ainsworth’s (2008) proposal, animation is defined as a form of dynamic representation constructed with moving images that change in time and are frequently used to represent processes that are not easy to observe directly. Different factors that indicate that animations are beneficial for learning because of the engagement of students with the processes explained (Rieber, 1991), the impact on the working memory, and the mobilization of a dynamic mental model. From the perspective of social semiotics, animation not only benefits from spatial composition but also incorporates temporality to decompose processes. As Kress and van Leeuwen (2006) suggest, animation can help the understanding of abstract processes and stimulate an aesthetic dimension. The purpose of this study was to gather evidence of the ways in which animated multimodal science texts with different types of mediations (non-animated, scientific concepts animation, and academic language animation) can scaffold science understanding for Chilean fifth graders.

6.2

Development Process and Criteria for the Design of Animated Multimodal Versions

A team comprising science experts, linguists, psychologists, teachers, illustrators, designers, programmers, and psychometricians was brought together for an interdisciplinary and collaborative project to design multimodal animated resources in order to examine how these resources would affect students with low prior knowledge of science and low reading comprehension skills. Figure 6.1 shows a diagram illustrating the different phases that was divided the development process of multimodal animated scaffolded texts.

6 Promoting Scientific Understanding through Animated Multimodal Texts

137

Scaffolding Decisions Identification of scientific concepts and academic language resources for the multimodal scaffolds.

Topic Selection Definition of energy transfer as the topic for text and anticipations of students' initial models.

Design of Animated Texts Full development of animated multimodal versions of scientific concepts’ and academic language’s texts.

Verbal Text Development Propositions development for verbal text and experts’ review.

Animations Prototyping Initial development of animated multimodal texts. Experts’ review and usability tests of the prototype.

Fig. 6.1 Development process of multimodal animated scaffolded versions

6.2.1

Development Process

6.2.1.1

Phase 01: Topic Selection

Following the revision of the grade five science curriculum, we decided to work on energy transfer. In a previous project, we had determined the effects of printed multimodal texts on this scientific process with high and low scaffolding (Meneses, Escobar, & Véliz, 2018). Therefore, we considered it appropriate to move towards animated multimodal versions. We decided to continue working with energy transfer for two reasons. First, the ecosystem is a disciplinary core idea (National Research Council, 2012). Indeed, the U. S, framework for K-12 Science Education proposes that this core idea explore the interactions between organisms and identify factors that determine how an ecosystem functions. Furthermore, energy and matter correspond to one of the crosscutting concepts of the same framework (National Research Council, 2012). Second, energy transfer is a complex issue for elementary school students, as their daily identification of energy as power or force differs from the scientific concept. Students tend to associate energy with food or fuel, and it is difficult for them to understand the model for representing the scientific process of energy transfer in an ecosystem (National Research Council, 2012).

6.2.1.2

Phase 02: Verbal Text Development

To explain energy transfer, we chose the Altiplano ecosystem to ensure greater validity and connection to students’ experience. The Altiplano is a high altitude plain located at the intersection of Peru, Bolivia, Argentina, and Chile in South America. Given its height above sea level and aridity, it is characterized by steppe flora and fauna that endure extreme conditions. The verbal text was constructed with an

138

M. Montenegro et al.

explanatory sequence in which students could recognize an initial constatation phase, which introduced the scientific phenomenon: energy allows organisms to carry out their vital processes. Later, a problematization was established (How is the energy transferred from the Sun to the Earth’s organisms?). Subsequently, the resolution presented types of living beings according to the way they obtain energy and then indicated how energy is transferred between living beings. Finally, a conclusion showed the interdependence between living beings. The total length of the text is 830 words with an average of 17.66 words per sentence.

6.2.1.3

Phase 03: Scaffolding Decisions

This phase involved two types of research decisions. The first of these was to determine the style of images to represent the scientific process of energy transfer in an ecosystem and its participants. The team produced different proofs of layout and colors to represent the Altiplano, a Chilean ecosystem, in a scientific and engaging way. The second decision concerned the type of scaffolding for each version and how they would be sequenced. For the scientific concepts animated version, the team focused on common misconceptions and fundamental concepts that students struggle to understand most, as reported by the literature (Özkan, Tekkaya, & Geban, 2004) and in the map for energy transfer and interdependence from the Science Literacy Maps (http://strandmaps.dls.ucar.edu/) of the National Science Digital Library (NSDL, https://nsdl.oercommons.org/). Special attention was given to the dynamic processes of the energy transfer across organisms by making explicit how the links between the different entities were represented; for example, by illustrating the journey of the energy from the Sun to Earth, the use of energy for organisms’ vital processes, and the transfer of energy from the Sun to plants and then to animals. For the academic language animated version, the team selected technical terms (energy, organism, vital processes, transference) and logical connectors (however, on the other hand, therefore, due to) with which students were unfamiliar. User-friendly and simply formulated explanations helped students access verbal and visual aids that they could click on to understand the meaning of these linguistic elements.

6.2.1.4

Phase 04: Animations Prototyping

After the research team decided on style and types of scaffolding, it built prototypes for each version where verbal language (aural and written) was integrated with images, prioritizing the composition of each frame over elements’ details. The first versions were revised by experts in science, linguistics, and psychology to verify the comprehensibility of the texts and the scientific precision with which the process of transference was visually represented. In addition, usability tests were conducted with fifth-grade students to decide about composition as well as how they would

6 Promoting Scientific Understanding through Animated Multimodal Texts

139

navigate the digital version and access the aids in each form. Feedback from experts and users was analyzed and incorporated into the development of the final versions.

6.2.1.5

Phase 05: Design of Animated Texts

After the scripts for the animated digital versions were developed, media production for each version ensued. This involved audio recording, image animation, multimodal assembly text, and programming digital texts for interaction. In addition, a static multimodal version was created for the control group.

6.2.2

Criteria for the Design of Animated Multimodal Versions

Different criteria from a variety of theoretical approaches were used for the design of the animated multimodal scaffolded versions in order to build a resource with specific educational purposes and robust semiotic composition. Each of the criteria used is briefly explained below.

6.2.2.1

Key Ideas for the Construction of the Mental Model

One of the criteria informed by both research on the multimedia approach (Bétrancourt & Chassot, 2008; Mayer, 2005; Schnotz & Bannert, 2003) as well as from the perspective of social semiotics (Unsworth, 2004), is the strategic use of animation to help construct a coherent representation of an abstract dynamic process. Therefore, a fundamental milestone was the segmentation of the text into key ideas to support reading comprehension (Dalton et al., 2011; Dalton & Palincsar, 2013). The 830-word verbal text was segmented into 20 fragments following the intrinsic explanatory structure, and preserving the relevant ideas (analysis of propositions), from which 20 frames were created: • Initial constatation (2 frames): Frames about the arrival of the solar energy on Earth and its use for the vital processes of the organisms. • Problematization (5 frames): A direct question about how energy is transferred from the Sun to organisms was presented, followed by scenes showing explicitly the tension between plants that can capture the energy from the Sun directly vs. other organisms that obtain solar energy indirectly through their alimentation. • Resolution (6 frames): Across these frames, organisms other than plants are defined as consumers, and they are classified according to their types of consumption. These frames introduce decomposers as a relevant participant in an ecosystem and explain energy transfer from the Sun to the different organisms.

140

M. Montenegro et al.

• Synthesis (2 frames): The problematizing question is reintroduced, and the transfer process is verbally explained in a synthetic way using discursive organizers such as first, then, finally. • Resolution (3 frames): Frames where the food chain is introduced and explained as a visual representation of the energy transfer process with particular emphasis on the meaning of the arrows, their size, and direction. • Conclusion (2 frames): Final frames where the interdependence between living beings is presented through a new question and its precise answer, which explains that the decreasing of one organism’s population would affect the rest of the ecosystem’s living beings.

6.2.2.2

Image Style

As proposed by Kress and van Leeuwen (2006), two types of images were produced for the energy transfer text: a covert taxonomy to represent the different types of consumers (primary, secondary and tertiary) according to the way in which they obtain energy; and, in addition, a process of conversion to represent the way in which energy is transferred from the Sun to the decomposers. We used the method proposed by Martin (2017) for condensing information through images that refer to entities that, in ideational terms, correspond to technicality. In the latter case, we represented how the energy that comes from the Sun is distributed across the Altiplano ecosystem, in which different types of organisms are found (Fig. 6.2). The ecosystem’s participants organize themselves into a taxonomic representation that ends up showing the food chain, and food web (Fig. 6.3), making the relations between organisms more complex. The degree of abstraction and the schematic representations increase through the use of arrows of different sizes.

Fig. 6.2 The Altiplano ecosystem as represented in the multimodal text

6 Promoting Scientific Understanding through Animated Multimodal Texts

141

Fig. 6.3 From left to right, Consumers Taxonomy, Food Chain, and Food Web

Fig. 6.4 Sequence of energy representation

Fig. 6.5 Unpacking of the sequence of activities that allows the transference of energy across participants of the ecosystem

One of the main challenges in visual production was the decision of how to represent the abstract concept of energy. We used triangles for two reasons: (1) not to confuse energy with atoms or molecules, which may have occurred if we have chosen circles and (2) to show that energy is something captured by producers and stored in their tissues (Fig. 6.4).

6.2.2.3

Text-Image Relation

On the interface between languages, concurrent relations are established for the verbal and the visual representation of the participants, since both codes depict the same entities of the Altiplano ecosystem (shrub, vizcacha, fox, puma, fly, and worm) plus the Sun. However, the animation provides additional information explaining the relations between these entities and making visible how energy is transferred between organisms, an issue that is minimally explained in the verbal code (“In this way, energy is transferred from the primary consumer to the secondary consumer through food”). Thus, the animation unpacks and explains the sequence of activities between different entities that result in the transfer of energy across the ecosystem, as it is shown in Fig. 6.5. From a multimedia perspective, the criteria for the dynamic information design of Narayanan and Hegarty (2002) were considered: (1) early presentation of the different entities to reinforce their representation (creation of a cover with all participants), (2) explanation using spatial and logical

142

M. Montenegro et al.

connections to reduce the role of prior knowledge (animated sequence of images where each consumer feeds on another animal for energy), (3) construction of the mental model by making reference chains with different representations for a common referent (energy is represented first as yellow triangles and then as yellow arrows), (4) possibility of constructing the mental model actively by the reader (use of rhetorical questions). Therefore, in these multimodal assemblages, there is a variety of relations between verbal and visual languages. In addition to the movement of images, the presence of aural and written verbal language is added. For the composition, the principles of multimedia design were considered (Mayer, 2005) in which redundancy is postulated as fundamental, that is, to elaborate an explanation visually and with words, but not to use the aural and the written modes at the same time. Therefore, for the scaffolded version of scientific concepts, it was decided to first present the aural text with the animated images and, later, through a click, the students could access the verbal text. Hence, in this version, the scaffolding focused on understanding the scientific process and then supporting the reading of the text. On the other hand, in the scaffolded version of academic language, students had access to written verbal text accompanied by reading aloud (aural mode). Students could then click on the highlighted linguistic resources and could access animated or verbal-visual aids to understand the meaning of the vocabulary or the logical relations established by connectors. Thus, in this version, the construction of meaning from verbal language was accessed, and then the animated images more deeply supported the understanding of the scientific process.

6.2.2.4

Scaffold Type

The last criterion considered for the design of multimodal animated versions was the type of scaffolding. Based on research on digital reading (Dalton et al., 2011; Dalton & Palincsar, 2013) and previous research (Meneses, Escobar, & Véliz, 2018), decisions were made on the type of navigation through the texts. For the scaffolding version of the scientific concepts, it was decided to use a content-based subjectmatter knowledge strategy. The integration of the animated images and the aural language supported the dynamic construction of the model of representation, to facilitate the reading of the text later once each segment of it has been understood. Figure 6.6 shows the scaffolding sequence to illustrate that every organism needs energy for its vital processes: first, an animation showing different living beings in activities that require energy (represented by a yellow halo in the left side image) accompanied by an aural version of the text; then, the verbal representation of the text is presented (right side image). On the other hand, a more academic-language-based knowledge strategy was used for the academic language version. After reading the written text supported by audio with reading aloud, the student could click on the most prominent linguistic resources (academic vocabulary or connectors) to obtain further help. The aid was user-friendly verbal explanations or animations designed to assist in understanding

6 Promoting Scientific Understanding through Animated Multimodal Texts

143

Fig. 6.7 Academic language scaffolding

Fig. 6.6 Scaffolding of scientific concepts

the scientific process. Figure 6.7 shows the academic language scaffolding sequence for the same concept than in Fig. 6.6: first, the verbal text is simultaneously presented with its aural version (left image), with the highlighted linguistic resources that can be clicked for help; then, the animation is shown (right image).

6.2.3

Study Purpose

The purpose of this study was to determine the effect of animated multimodal texts in scaffolding academic vocabulary, reading comprehension, and science learning of students with low prior science knowledge and low reading comprehension. Two research questions guided this study: RQ1: Are there significant differences in student performance in academic vocabulary, reading comprehension, and science learning according to condition (non-animated, animated scientific concepts, animated academic language)? RQ2: How much of the differences in academic vocabulary, reading comprehension, and science learning performance can be explained by students’ prior knowledge (reading fluency, general academic vocabulary, general reading comprehension, and general science knowledge)?

144

6.3

M. Montenegro et al.

Methods

A total of 326 fifth graders attending four subsidized schools in Santiago, Chile, was the initial sample in this study. The schools chosen were low and middle-low socioeconomic levels because our study focused on low comprehenders, who require specific pedagogical resources that can help them overcome their educational disadvantages. All students were assessed at the end of the school year on general science knowledge, general reading comprehension, general academic vocabulary, and reading fluency. For all assessments, except reading fluency, student results are reported as a percentage of achievement for ease of understanding. Reading Fluency Reading fluency was measured using the Spanish rapid segmentation task (Elosúa et al., 2012) adapted culturally for Latin American contexts. The participants identified three words in a string of words without spaces. For example, in cocheperapino (carpearpine), the student should divide as coche/pera/pino (car/ pear/pine). The total score was calculated as the number of correct words identified in 90 sec. General Academic Vocabulary (α = .80) Cross-disciplinary academic vocabulary was assessed with the multiple-choice Spanish Academic Vocabulary Test (Meneses, Ucelli, Santelices et al., 2018). This test is composed of 15 items that evaluate vocabulary knowledge from words taken from a list of words frequently used in academic texts (Coxhead, 2000). The test was translated and functionally and culturally adapted from the English Word Generation Academic Vocabulary Test (Hwang, Lawrence, Mo, & Snow, 2015). Words for Spanish were revised in the Spanish corpus and in school textbooks. Items are evaluated with 1 point for a correct answer and with 0 points for incorrect answers. The maximum score is 15 points. General Reading Comprehension (α = .82) Reading comprehension was evaluated with the PROLEC-SE group application test (Ramos & Cuetos, 2011). Students read two general knowledge explanatory texts, and a total of 20 literal and inferential comprehension questions are answered. The correct answers are scored with 1 point and the incorrect ones with 0 points. The maximum total score is 20 points. General Science Knowledge (α = .85) Science knowledge was evaluated through an instrument that measures the knowledge and skills reached by 4th-grade students and aligned with the school science curriculum. This instrument is group administered and contains 40 multiple-choice items. Correct items are scored with 1 point, while incorrect items are scored with 0 points. The maximum total score is 40 points. For the purposes of this study, any students with a score equal to or lower than the 50% of the maximum score for the general comprehension test was classified as a low comprehender, reducing the initial sample to 117 students. These 117 were distributed in three statistically equivalent groups of 39 students by using propensity score matching (PSM). PSM is an analytical technique that matches participants between groups by estimating their probability of belonging to the group given their characteristics, and it can be used for obtaining equivalent groups. In our case, we

6 Promoting Scientific Understanding through Animated Multimodal Texts Table 6.1 The distribution of the sample by sex and condition

Conditions Non-animated Scientific concepts Academic language Total

145

Female N %

Male N

%

Total

14 12 15 41

12 18 13 43

46 60 46 51

26 30 28 84

54 40 54 49

used sex, reading fluency score, general academic vocabulary score, general reading comprehension score, and general science knowledge score as covariates for the grouping and the Matchit R Package (Ho, Imai, King, & Stuart, 2011) for the analysis. Ultimately, from this subsample, only 84 students were assessed in this quasi-experimental study, with 49% females and 51% males; all students were monolingual Spanish-speaking. The study was a between-subject design with three conditions: non-animated version (26 students), animated version scaffolding scientific concepts (30 students), animated version scaffolding academic language (28 students). In Table 6.1, the distribution of the sample by sex and condition is displayed.

6.3.1

Text Conditions

As explained in the previous section on the creation of the digital versions, three versions of the energy transfer text were designed, and each of them was assigned to one of the three statistically equivalent groups. Condition 1, Non-Animated Version (26 students). This static text explains through images and written text about how energy is transferred through the trophic chain. Condition 2, Scientific Concepts (30 students). This is the animated version to scaffold science concepts. This version was created from the previous multimodal text by adding animation to its images and an aural version of the text. The animations point to the visualization of those complex and abstract concepts that are present and need to be understood in the text. At the beginning of every page, the aural text was presented simultaneously with the animated images, but without the written text; then, students could access the written text with a mouse click. Condition 3 Academic Language (28 students). This is the animated version of scaffolding academic language, specifically, for support of cross-disciplinary vocabulary and logical connectives. The same animated images and aural text added to the original text were used in this condition, but the order of the presentation was changed. For this version, the aural text and the written text were presented at the beginning of each page, simultaneously; then, students could access the animated images and some user-friendly explanations of difficult words with a mouse click.

146

6.3.2

M. Montenegro et al.

Procedures

Each student participated in just one of the conditions; they read the assigned energy transfer version of the text in a quiet space within the school, but different from the classroom. Trained personnel assisted the students during the reading of the digital science text on a computer to ensure the understanding of the reading task. Each student had the necessary time to read the text, and, at the end of the readout, a set of tests as proximal measures were applied.

6.3.3

Proximal Text’s Measures

The following proximal measures refer to tests based on the target texts read by the students and that they were applied after reading it. Academic Vocabulary (α = .76) A 22-item multiple-choice test was designed to measure knowledge of cross-disciplinary academic vocabulary present in the multimodal text (e.g. organism, transfer, community). Each word is presented in a sentence, and the student must select the word that adequately replaces the underlined word. Distractors correspond to a phonetically similar word, an academic word from another domain, and a more general word (e.g., “The energy is transferred to the animals. (a) it is transmitted; (b) it is located; (c) is delivered; (d) is to be passed on”). Reading Comprehension (α = .73) A 20-item multiple-choice test was designed to measure the literal (e.g., “Which organisms directly obtain energy from the Sun to produce nutrients?”) and inferential level (“What does the arrow on a food chain indicate?”) of reading comprehension achieved from the read text about energy transfer. Science Learning (α = .69) A 26-item multiple-choice test was designed to measure the understanding of the scientific process and the transfer of this knowledge to a new context (classification of living being by food, classification of living beings by level in the food chain, production of food chain in another ecosystem, interdependency relations, decomposers and location in the food chain).

6.3.4

Analysis

An ANOVA and Tukey multiple comparisons of means were carried out to determine if there were statistical differences between the three conditions. Furthermore, a regression analysis was performed, where academic vocabulary, reading comprehension, and science learning were the dependent variables and the initial measures of academic vocabulary knowledge, general reading comprehension, and general

6 Promoting Scientific Understanding through Animated Multimodal Texts

147

science learning were the independent variables; the objective of this analysis was to determine how much predictive power prior knowledge has on the text’s comprehension in every condition in order to discover how helpful every text version was for the students.

6.4 6.4.1

Results Comparison among Multimodal Non-animated and Animated Texts

Table 6.2 shows the mean and the standard deviation for the initial measures of academic vocabulary, general reading comprehension, and general science learning before reading the digital texts. Likewise, Table 6.2 reports the results of the students’ processing of the digital text in terms of academic vocabulary, reading comprehension, and science learning applied once the digital texts have been read. The ANOVA analysis for the initial measures showed that there was no statistical difference among groups for general academic vocabulary score (F(2,80) ¼ 0.97, p < 0.384), and general reading comprehension score (F(2,81) ¼ 2.46, p < 0.092). However, there were significant differences between groups for reading fluency (F (2,81) ¼ 10.64, p < 0.000) and general science knowledge score (F(2,81) ¼ 31.11, p < 0.000). The non-animated group has a statistically lower reading fluency than the other two groups, which do not have any difference; however, for general science knowledge, all three groups have statistically different scores. These differences among groups come from the unplanned reduction of the initial subsample at the time of the final evaluation due to some student were not present at the time of application, especially for the non-animated and academic language groups.

Table 6.2 The mean and standard deviation for the initial and proximal measures per condition

Initial measures Reading fluency General academic vocabulary General Reading comprehension General science knowledge Proximal Text’s measures Academic vocabulary Reading comprehension Science learning

Conditions Non-animated Mean SD

Scientific Concepts Mean SD

Academic Language Mean SD

30.4 49% 38% 30%

9.7 19% 11% 12%

40.9 53% 43% 51%

7.1 13% 8% 7%

36.9 47% 38% 43%

8.7 15% 10% 9%

58% 47% 33%

22% 16% 12%

70% 63% 49%

14% 13% 18%

67% 57% 44%

15% 18% 21%

148

M. Montenegro et al.

Given the dependence of the initial measures on the group, we performed an ANCOVA analysis for the proximal measures taking into account these initial measures as covariates. It was found that there were significant differences among groups in all three text’s measures: academic vocabulary (F(2,75) ¼ 5.454, p < 0.006), reading comprehension (F(2,76) ¼ 9.341, p < 0.000), and science learning (F(2,75) ¼ 6.306, p < 0.003). From Table 6.2, we can see that the nonanimated version group underperformed the academic language scaffolding group, that in turn, underperformed the science concepts scaffolding group, for every proximal measures. However, a post-hoc Tukey comparison showed that not all the scores’ differences were statistically different: the group that experienced the non-animated version statistically underperformed in relation to the science concepts scaffolding group in all three measures ( p < .00), whereas it only statistically underperformed in relation to the academic language scaffolding group in the academic vocabulary ( p < .05) and reading comprehension ( p < .02), with a marginally non-statistical difference in science learning ( p < .07). There were no statistically significant differences between the science concepts scaffolding and academic language scaffolding groups in any of the measures, so we cannot attribute the difference of their scores to solely the different versions that they faced. The results indicate that the animated versions of the texts enabled low comprehenders to improve their reading comprehension, which in turn allowed them to acquire more technical vocabulary and science knowledge, especially in the case of the text that had scaffolding of science concepts. However, given the fact that both science concept and academic language scaffolding groups did not perform quite differently despite their scaffolding’s purposes, our results show that both scaffoldings can increase science learning, academic vocabulary and reading comprehension, with a little overperformance for part of the science concepts scaffolding version. Considering that our subsample is composed of low comprehenders, students who perform under 50% on reading comprehension achievement, the equivalent improvement on both areas may be due to our students relying mainly on the aural version of the text and the animations, given that the complementary help on scientific concepts or academic language required a further reading of the written version. The next section will investigate this issue further.

6.4.2

Effects of Multimodal Non-animated and Animated Texts

To further explore the effect of the animated texts, we performed a regression analysis of the proximal measures as a function of the initial measures. In the previous section we showed that our animated texts, especially the one with science concepts scaffolding, generated a significant difference in learning between groups, but, due to the subsample’s lack of homogeneity, it is not clear whether or not those difference were due mainly to the students’ prior knowledge. If we can show that

6 Promoting Scientific Understanding through Animated Multimodal Texts

149

those learning differences cannot be explained from student’s prior knowledge and that they are mainly due to the scaffolding type of the animated text, we will have evidence that multimodal animated texts can really help disadvantage students, not only those students with low reading comprehension but also with lower science comprehension, to overcome their learning gaps: multimodal animated texts could be a powerful tool to address educational inequality. The object of this section’s analysis was to determine to what extent our animated texts were able to increase student’s learning despite their prior knowledge. Table 6.3 shows the regression coefficients and statistics of the proximal measures in vocabulary, reading comprehension, and science learning in function of reading fluency, general academic vocabulary, general reading comprehension, and general science knowledge for the three conditions. As can be seen from the table of the nine regressions, three were not significant, revealing that in our case, not all the acquired knowledge was due to prior knowledge. The first one, text’s vocabulary learning for the science concept group, was not fully explained by prior knowledge (F(4,23) ¼ 1.98, p < 0.132). The other two correspond to the text’s science learning for the non-animated and the scientific concepts groups (F(4,18) ¼ 1.26, p < 0.323, and F(4,23) ¼ 0.64, p < 0.642, respectively). Moreover, in these three cases, general science knowledge had negative (non-significant) regression coefficients, which implies that students were misled by their prior knowledge. These results provide evidence that the original non-animated science text was supportive in mediating students’ science learning acquisition and that the animated version of it with science scaffolding even improved its effect, with low prior knowledge dependence in both cases. Of the six significant regressions, two belong to the proximal measure text’s vocabulary. 58% of text’s vocabulary variance for the non-animated group was explained by our regression analysis (F(4,18) ¼ 6.12, p < 0.003) and 54% of it for the academic language group (F(4,17) ¼ 5.08, p < 0.07). In both groups, general academic vocabulary was a significant predictor of the acquired vocabulary, with small coefficients, and general reading comprehension was only a significant predictor for the non-animated group, although with the largest regression coefficients. Reading fluency and science general knowledge were not significant predictors, with smaller regression coefficients. These results confirm the relevance of prior vocabulary knowledge and, primarily, reading comprehension over prior scientific knowledge for the acquisition of science vocabulary. Moreover, given the lower regression coefficients for reading fluency and reading comprehension for the academic language group in relation to the non-animated group, academic language scaffolding was able to help students to overcome their reading comprehension limitations partially and not as effectively as science concepts scaffolding. The only proximal text’s measure that was significantly predicted by prior knowledge for all three groups was reading comprehension, with the regression explaining 41% of the variance in the non-animated group, 33% in the scientific concepts group, and 51% in the academic language group. General reading comprehension’s coefficients are larger and significant for the science concepts and academic language groups, showing the relevance of mastering language skills to

*p < .005

Proximal Measures Text’s vocabulary Non-animated Scientific concepts Academic language Text’s Reading comprehension Non-animated Scientific concepts Academic language Text’s science learning Non-animated Scientific concepts Academic language 0.475* 0.296 0.182

0.008* 0.006 0.015*

0.003 0.003 0.007

0.004 0.005 0.009

0.060 0.063 0.466

0.118 0.242 0.212

0.209 0.591 0.592

0.195 0.245 0.218

0.264 0.521* 0.524*

General Reading Comprehension

Regression coefficients Reading General Academic Fluency Vocabulary

4 4 4 4 4 4

0.081 0.509 0.231

4 4 4

Par.

18 23 17

18 23 17

18 23 17

df

1.26 0.64 3.60

3.11 2.89 4.36

6.12 1.98 5.08

F

0.323 0.642 0.027

0.041 0.045 0.013

0.003 0.132 0.007

p

Regression statistics

0.062 0.363 0.429

0.129 0.221 0.005

General Science Knowledge

Table 6.3 Regression coefficients and statistics of the proximal text’s measures as a function of the initial measures

0.22 0.10 0.46

0.41 0.33 0.51

0.58 0.26 0.54

R2

150 M. Montenegro et al.

6 Promoting Scientific Understanding through Animated Multimodal Texts

151

understand science texts. The academic language group has the lowest (no significant) reading fluency coefficient and the largest general reading comprehension coefficient, showing that academic language scaffolding helped students in decoding the text, but not entirely in comprehending it.

6.5

Discussion and Conclusions

Regarding both animated versions of the text, our findings have shown that adequate scaffolding, one that helps students grasp science concepts or to manage academic language, can be an useful way to improve their understanding of science. The scaffolded science concept’s version overperformed in a statistically significant way the static version in every proximal measure, namely academic vocabulary, reading comprehension and science learning, and the scaffolded academic language’s version only statistically overperformed the statics’ version in academic vocabulary and reading comprehension. Moreover, these significant differences between the animated and the non-animated versions were not entirely explained by students’ prior knowledge. The regression-analysis’ results showed that the scaffolded science concepts’ version of the text was able to improved students’ vocabulary and science learning with a non-significant support of prior science knowledge or vocabulary acquisition, helping students in disadvantage. Also, they showed that mastering language skills is relevant to understand science texts, but that academic language scaffolding can also help students in disadvantage. The inclusion of animated images and aural text can boost student performance significantly, especially for low-comprehenders, as in our case. As we will show in the next section, our study suggests some guidelines for the construction of effective multimodal text that can help to achieve science understanding for all students, even those in educational disadvantage.

6.5.1

Science Digital Multimodal Texts: From Still to Dynamic Images

This study provides evidence in Spanish about the differences between static and dynamic texts when it comes to an understanding of abstract scientific processes, namely energy transfer in an ecosystem. Starting from the assumption that science texts are multimodal, this study was aimed at determining the effects of the textimage animated relation for students with low prior knowledge and low reading skills. This research is based on the design principles of multimedia theory (Mayer, 2005) and incorporates verbal and visual scaffolds from a functional semiotic view of language as a mediator of learning (Schleppegrell, 2004; Snow, 2010; Uccelli, 2019; Unsworth, 2004). The effects of the animated multimodal scaffolded versions show that they were beneficial for fifth-grade students. This finding coincides with

152

M. Montenegro et al.

previous research on the effects of static multimodal texts with quasi-experimental design (Ge et al., 2017; Meneses, Escobar, & Véliz, 2018) and advances our understanding of the dynamic dimension provided by animation. In regard to studies on digital reading comprehension scaffolding, our research shows that animated digital texts are more likely to be beneficial to the understanding of scientific concepts than static versions. Dalton et al. (2011) also found that scaffolded digital texts with a focus on strategies and vocabulary increased the reading comprehension skills of both monolingual and bilingual fifth-grade students. Given that their research focused on narrative and expository texts, they did not explore the effects of science-specific scaffolded digital texts. Dalton and Palincsar (2013) also compared the effects of static narrative and expository texts with the interactive digital versions and found that animated versions fostered the comprehension of fifth-grade students; however, gaps in comprehension between different types of readers could not be reduced in their work. The study of Dalton and Palincsar (2013) also was not able to advance our knowledge of how animated digital versions can promote understanding and learning in science. In contrast, our study converges with research derived from multimodality in science, specifically, the study of static texts, with research in the field of reading comprehension and digital scaffolds to support elementary school students’ comprehension and learning.

6.5.2

Animations and Scaffolding Affordances for Diverse Learners in School Science Contexts

One of the main contributions of our study has been to demonstrate how digital animated texts in science can benefit the reading comprehension and understanding of scientific concepts of students with low prior knowledge and weak reading comprehension skills. Our study was conducted with a sample of students identified as needing the most support to address learning differences within the classroom, distributed across different conditions using propensity score analysis to assure homogeneity among groups. It gave evidence that animated digital text versions can promote learning and understanding of this group of disadvantaged students. Consistent with this work, previous research on static multimodal texts in science (Meneses, Escobar, & Véliz, 2018) also found that scaffolded static text versions with high multimodality improved reading comprehension, in particular, of fifthgrade students with low reading comprehension and enabled them to reach an achievement level equivalent to students with medium comprehension. From an educational research point of view, the visual, verbal, and visual-verbal criteria used to design the digital animated versions of this work show in an exploratory way how educational resources can support the learning of specific students. More research is needed inside classrooms to understand how adaptive digital resources can support learning for different groups of students.

6 Promoting Scientific Understanding through Animated Multimodal Texts

6.5.3

153

Disciplinary Literacy and the Challenges of Science Reading Comprehension

This study of animated scaffolded science texts and their effects on readers with low prior knowledge and reduced reading skills has provided empirical evidence with implications for disciplinary literacy practices. Tang and Moje (2010) draw attention to a number of demands and challenges students face that are linked to the different, specific, and precise use of verbal and visual resources in the science domain due to lack of exposure in home contexts. Therefore, animated texts offer students scaffolding not only in the understanding of scientific concepts but also in supporting the development of their comprehension and academic language skills, which confirms the hypothesis formulated by Bétrancourt and Chassot (2008). The scientific practices and specialized languages of science challenge equally students and teachers. Studies from science teaching state that students need to learn a metarepresentational competence; this means that they need to manage a varied repertoire of scientific representations, criteria for their validation, as well the development of interpretation and production skills of scientific representations (Prain & Tytler, 2012; Tytler et al., 2013). On the other hand, research has shown that the integration of multiple representations to explain scientific phenomena is challenging since it is necessary to complement information from diverse sources and different forms of representation (Cook, 2008; Tang & Moje, 2010; Waldrip, Prain, & Carolan, 2010). The individual is expected to have the capacity to perceive, recognize, and coherently interrelate the different representations in a structure of knowledge (Cook, 2008; Mayer & Moreno, 2003). In this study, animated texts were constructed to scaffold the understanding of an abstract scientific process, although more research is required to understand the learning trajectories that science texts without scaffolds promote in school students, as well as semiotic and discursive resources used in authentic scientific practices. As Lemke (1998) states, science corresponds to a set of practices constructed by specialized languages that are learned within the scientific community. Therefore, the science promoted at school needs to explicitly adopt a multimodal approach so that students can increase their learning (Prain & Waldrip, 2010). However, teachers have not developed a specific awareness and knowledge to reveal the contribution of these languages in science learning, and students have not been taught in a way that makes their functions visible in the construction of scientific knowledge. Our interdisciplinary study, in which different approaches converge, provides criteria to consider in the creation of digital texts that may promote the understanding of complex scientific processes, particularly for students who require more in-depth support. We have provided evidence showing how teaching resources and, in particular, texts operationalized from a perspective of digital and disciplinary literacy, can promote learning opportunities in a differentiated way to respond to the considerable classroom performance heterogeneity, as a way to overcome educational inequalities.

154

M. Montenegro et al.

Acknowledgments This work was supported by Pontificia Universidad Católica de Chile, Vicerrectoría de Investigación under Grant Proyecto Basal de Centros de Investigación Interdisciplinaria, and ANID/CONICYT, FONDECYT Regular 1190990. The authors would like to thank Diego Urzúa for his work as a research assistant, Iván Orellana, and Ignacio Zamorano for the development of the animated texts and Isidora Rodríguez for her data collection work. Finally, we would like to express our sincere gratitude to the students, teachers, and principals who participated in the study.

References Ainsworth, S. (2008). How do animations influence learning? In D. Robinson & G. Schraw (Eds.), Current perspectives on cognition, learning, and instruction: Recent innovations in educational technology that facilitate student learning (pp. 37–67). Charlotte, NC: Information Age Publishing. Ardasheva, Y., Wang, Z., Roo, A. K., Adesope, O. O., & Morrison, J. M. (2018). Representation visuals’ impacts on science interest and reading comprehension of adolescent English learners. The Journal of Educational Research, 111(5), 631–643. https://doi.org/10.1080/00220671. 2017.1389681 Bétrancourt, M., & Chassot, A. (2008). Making sense of animation: How do children explore multimedia instruction? In R. Lowe & W. Schnotz (Eds.), Learning with animation: Research implications for design (pp. 149–164). New York, NY: Cambridge University Press. Bétrancourt, M., & Tversky, B. (2000). Effect of computer animation on users’ performance: A review. Le Travail Humain: A Bilingual and Multi-disciplinary Journal in Human Factors, 63 (4), 311–329. Chang, N. (2012). What are the roles that children’s drawings play in inquiry of science concepts? Early Child Development and Care, 182(5), 621–637. https://doi.org/10.1080/03004430.2011. 569542 Cook, M. (2008). Student’s comprehension of science concepts depicted in textbooks illustration. Electronic Journal of Science Education, 12(1), 39–54. Cox, S. (2005). Intention and meaning in young children’s drawing. International Journal of Art & Design Education, 24(2), 115–125. https://doi.org/10.1111/j.1476-8070.2005.00432.x Coxhead, A. (2000). A new academic word list. TESOL Quarterly, 34(2), 213–238. https://doi.org/ 10.2307/3587951 Dalton, B., & Palincsar, A. S. (2013). Investigating text–reader interactions in the context of supported etext. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 533–544). New York, NY: Springer. https://doi.org/10.1007/ 978-1-4419-5546-3_34 Dalton, B., Proctor, P. C., Uccelli, P., Mo, E., & Snow, C. E. (2011). Designing for diversity: The role of reading strategies and interactive vocabulary in a digital reading environment for fifthgrade monolingual English and bilingual students. Journal of Literacy Research, 43(1), 68–100. https://doi.org/10.1177/1086296X10397872 Elosúa, M.-R., García-Madruga, J.-A., Gómez-Veiga, I., López-Escribano, C., Pérez, E., & Orjales, I. (2012). Habilidades lectoras y rendimiento académico en 3 y 6 de primaria: aspectos evolutivos y educativos. Estudios de Psicología, 33(2), 207–218. https://doi.org/10.1174/ 021093912800676411 Evagorou, M., & Osborne, J. (2010). The role of language in the learning and teaching of science. In J. Osborne & J. Dillon (Eds.), Good practice in science teaching: What research has to say (2nd ed., pp. 135–157). New York, NY: Open University Press/McGraw-Hill. Fang, Z. (2006). The language demands of science reading in middle school. International Journal of Science Education, 28(5), 491–520. https://doi.org/10.1080/09500690500339092

6 Promoting Scientific Understanding through Animated Multimodal Texts

155

Ge, Y.-P., Unsworth, L., & Wang, K.-H. (2017). The effects of explicit visual cues in reading biological diagrams. International Journal of Science Education, 39(5), 605–626. https://doi. org/10.1080/09500693.2017.1297549 Halliday, M. A. K., & Martin, J. R. (1993). Writing science: Literacy and discursive power. Pittsburgh, PA: University of Pittsburgh Press. Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8), 1–28. https://doi.org/10. 18637/jss.v042.i08 Hwang, J. K., Lawrence, J. F., Mo, E., & Snow, C. E. (2015). Differential effects of a systematic vocabulary intervention on adolescent language minority students with varying levels of English proficiency. The International Journal of Bilingualism, 19(3), 314–332. https://doi. org/10.1177/1367006914521698 Jewitt, C., & Kress, G. R. (Eds.). (2003). Multimodal literacy. New York, NY: Lang. Kloser, M. (2013). Exploring high school biology students’ engagement with more and less epistemologically considerate texts. Journal of Research in Science Teaching, 50(10), 1232–1257. https://doi.org/10.1002/tea.21109 Kloser, M. (2016). Alternate text types and student outcomes: An experiment comparing traditional textbooks and more epistemologically considerate texts. International Journal of Science Education, 38(16), 2477–2499. https://doi.org/10.1080/09500693.2016.1249532 Kress, G. (2009). Multimodality: A social semiotic approach to contemporary communication. London: Routledge. Kress, G., Jewitt, C., Ogborn, J., & Charalampos, T. (2001). Multimodal teaching and learning: The rhetorics of the science classroom. London: Bloomsbury Publishing. Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design. New York, NY: Routledge. Lemke, J. L. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. In J. Martin & R. Veel (Eds.), Reading science: Critical and functional perspectives on discourses of science (pp. 87–113). London: Routledge. Lemke, J. L. (2004). The literacies of science. In E. W. Saul (Ed.), Crossing borders: Literacy and science instruction: Perspectives on theory and practice (pp. 33–47). Newark, DE: International Reading Association. Levin, J. R., & Mayer, R. E. (1993). Understanding illustrations in text. In B. K. Britton, A. Woodward, & M. Brinkley (Eds.), Learning from textbooks (pp. 95–113). Hillsdale, NJ: Erlbaum. Martin, J. R. (2017). Revisiting field: Specialized knowledge in secondary school science and humanities discourse. Onomázein Número especial LSF y TCL sobre educación y conocimiento, 111–148. https://doi.org/10.7764/onomazein.sfl.05 Mayer, R. E. (2005). The Cambridge handbook of multimedia learning. New York, NY: Cambridge University Press. Mayer, R. E., & Gallini, J. K. (1990). When is an illustration worth ten thousand words? Journal of Educational Psychology, 82(4), 715–726. https://doi.org/10.1037/0022-0663.82.4.715 Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52. https://doi.org/10.1207/s15326985ep3801_6 McNamara, D., Ozuru, Y., & Floyd, R. G. (2011). Comprehension challenges in the fourth grade: The roles of text cohesion, text genre, and readers’ prior knowledge. International Electronic Journal of Elementary Education, 4(1), 229–257. McTigue, E., & Slough, S. W. (2010). Student-accessible science texts: Elements of design. Reading Psychology, 31(3), 213–227. https://doi.org/10.1080/02702710903256312 Meneses, A., Escobar, J. P., & Véliz, S. (2018). The effects of multimodal texts on science reading comprehension in Chilean fifth-graders: Text scaffolding and comprehension skills. International Journal of Science Education, 40(18), 2226–2244. https://doi.org/10.1080/09500693. 2018.1527472

156

M. Montenegro et al.

Meneses, A., Uccelli, P., Santelices, M. V., Ruiz, M., Acevedo, D., & Figueroa, J. (2018). Academic language as a predictor of reading comprehension in monolingual Spanish-speaking readers: Evidence from Chilean early adolescents. Reading Research Quarterly, 53(2), 223–247. https://doi.org/10.1002/rrq.192 Moreno, R., & Mayer, R. E. (2000). A coherence effect in multimedia learning: The case for minimizing irrelevant sounds in the design of multimedia instructional messages. Journal of Educational Psychology, 92(1), 117–125. https://doi.org/10.1037/0022-0663.92.1.117 Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2017). PIRLS 2016 International results in reading. Resource Document. TIMSS & PIRLS International Study Center website. http:// timssandpirls.bc.edu/pirls2016/international-results/. Accessed July 15th, 2019. Narayanan, N. H., & Hegarty, M. (2002). Multimedia design for communication of dynamic information. International Journal Human-Computer Studies, 57(4), 279–315. https://doi.org/ 10.1006/ijhc.1019 National Research Council (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Committee on a conceptual framework for new K-12 science education standards. Board on science education, division of behavioral and social sciences and education. Washington, DC: The National Academies Press. Özkan, Ö., Tekkaya, C., & Geban, Ö. (2004). Facilitating conceptual change in students’ understanding of ecological concepts. Journal of Science Education and Technology, 13(1), 95–105. https://doi.org/10.1023/B:JOST.0000019642.15673.a3 Ozuru, Y., Dempsey, K., & McNamara, D. S. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction, 19(3), 228–242. https://doi.org/10.1016/j.learninstruc.2008.04.003 Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1(2), 117–175. Patterson, A., Roman, D., Friend, M., Osborne, J., & Donovan, B. (2018). Reading for meaning: The foundational knowledge every teacher of science should have. International Journal of Science Education, 40(3), 291–307. https://doi.org/10.1080/09500693.2017.1416205 Pearson, P. D., Moje, E., & Greenleaf, C. (2010). Literacy and science: Each in the service of the other. Science, 328(5977), 459–463. https://doi.org/10.1126/science.1182595 Prain, V., & Tytler, R. (2012). Learning through constructing representations in science: A framework of representational construction affordances. International Journal of Science Education, 34(17), 2751–2773. https://doi.org/10.1080/09500693.2011.626462 Prain, V., & Waldrip, B. (2010). Representing science literacies: An introduction. Research in Science Education, 40, 1–3. https://doi.org/10.1007/s11165-009-9153-x Ramos, J. L., & Cuetos, F. (2011). Evaluación de los procesos lectores: PROLEC-SE Manual. Madrid, Spain: TEA. Rieber, L. P. (1991). Animation, incidental learning, and continuing motivation. Journal of Educational Psychology, 83(3), 318–328. Schleppegrell, M. J. (2004). The language of schooling: A functional linguistic perspective. Mahwah, NJ: Lawrence Erlbaum Associates. Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representation. Learning and Instruction, 13(2), 141–156. https://doi.org/10.1016/S0959-4752 (02)00017-8 Snow, C. E. (2010). Academic language and the challenge of reading for learning about science. Science, 328(5977), 450–452. https://doi.org/10.1126/science.1182597 Snow, C. E., & Uccelli, P. (2009). The challenge of academic language. In D. R. Olson & N. Torrance (Eds.), The Cambridge handbook of literacy (pp. 112–133). New York, NY: Cambridge University Press. Tang, K.-S., Delgado, C., & Moje, E. B. (2014). An integrative framework for the analysis of multiple and multimodal representations for meaning-making in science education. Science Education, 98(2), 305–326. https://doi.org/10.1002/sce.21099

6 Promoting Scientific Understanding through Animated Multimodal Texts

157

Tang, K.-S., & Moje, E. B. (2010). Relating multimodal representations to the literacies of science. Research in Science Education, 40(1), 81–85. https://doi.org/10.1007/s11165-009-9158-5 Townsend, D., Brock, C., & Morrison, J. D. (2018). Engaging in vocabulary learning in science: The promise of multimodal instruction. International Journal of Science Education, 40(3), 328–347. https://doi.org/10.1080/09500693.2017.1420267 Tversky, B., Morrison, J. B., & Bétrancourt, M. (2002). Animation: Can it facilitate? International Journal of Human-Computer Studies, 57(4), 247–262. https://doi.org/10.1006/ijhc.2002.1017 Tytler, R., Prain, V., Hubber, P., & Waldrip, B. (2013). Constructing representations to learn in science. Rotterdam, The Netherlands: Sense Publishers. Uccelli, P. (2019). Learning the language for school literacy. In V. Grøver, P. Uccelli, M. L. Rowe, & E. Lieven (Eds.), Learning through language. Towards an educationally informed theory of language learning (pp. 95–109). New York, NY: Cambridge University Press. https://doi.org/ 10.1017/9781316718537.010 Uccelli, P., Phillips Galloway, E., Barr, C. D., Meneses, A., & Dobbs, C. L. (2015). Beyond vocabulary: Exploring cross-disciplinary academic-language proficiency and its association with reading comprehension. Reading Research Quarterly, 50(3), 337–356. https://doi.org/10. 1002/rrq.104 Unsworth, L. (2004). Comparing school science explanations in books and computer-based formats: The role of images, image/text relations, and hyperlinks. International Journal of Instructional Media, 31(3), 283–301. Waldrip, B., Prain, V., & Carolan, J. (2010). Using multi-modal representations to improve learning in junior secondary science. Research in Science Education, 40(1), 65–80. https://doi.org/10. 1007/s11165-009-9157-6 Wilson, R. E., & Bradbury, L. U. (2016). The pedagogical potential of drawing and writing in primary science multimodal unit. International Journal of Science Education, 38(17), 2621–2641. https://doi.org/10.1080/09500693.2016.1255369

Maximiliano Montenegro is an Assistant Professor of Science Education at the Facultad de Educación, Pontificia Universidad Católica de Chile and a member of the interdisciplinary research group Factoría Ideas. With both a Ph.D. in Physics and in Education, in the last ten years, he has been deeply involved in Science Teachers’ formation and in the science curriculum’s development. His research seeks to identify variables that deter students from understanding science and how to promote this understanding through the design of evidence-based STEM interventions. He is currently leading a 4-year project aimed to understand how multimodality can foster students’ science learning. Alejandra Meneses is an Associate Professor of Language and Literacy at the Facultad de Educación, Pontificia Universidad Católica de Chile. She is an educational linguist and researcher at Factoría Ideas. Her research is focused on the development of academic language across the school and its relations with reading and learning in subject matters, especially in Science, and multimodal reading comprehension to improve understanding. Also, she led a four-year grant aimed to increase students’ opportunities to learn Science through explicit academic language instruction for low-income 4th grade students. She is convinced that teachers are agents of social change. Soledad Véliz is a Ph.D. candidate in education at the Pontificia Universidad Católica de Chile. She is a psychologist with a Master of Arts (Children’s books illustration) and an M.Sc. in Forensic Mental Health Science. Her Ph.D. work is focused on challenging, unusual, or disturbing topics in picturebooks’ narratives and how they stimulate, alter, and test the practices of literary education and reading in educational contexts. She is currently working at two research centers: the Development Centre of Inclusion Technologies (CEDETi UC), and the Centre for Educational Justice, both at Pontificia Universidad Católica de Chile.

158

M. Montenegro et al.

José Pablo Escobar is a psychologist with a Master’s degree in Educational Psychology and a Ph. D. in Psychology from the Pontificia Universidad Católica de Chile. He is currently working at the Development Centre of Inclusion Technologies of the Pontificia Universidad Católica de Chile (CEDETi UC) as a research associate. His professional interests are related to the understanding of cognitive processes associated with the development of reading skills and how technology can support their acquisition in children with disabilities. Marion Garolera is a psychologist currently attending a Master’s degree in Gender and Cultural Studies at Universidad de Chile. She works as a research assistant at the Development Centre of Inclusion Technologies (CEDETi UC) of the Pontificia Universidad Católica de Chile. Her research field of interest is on diversity in education and queer desire in erotic narratives. María Paz Ramírez is an elementary teacher with a Master in Education. She is currently working at the NGO Neyün, that promotes socio-emotional well-being in schools in Chile, through educational programs based on simple, friendly and easy to apply Mindfulness dynamics.

Chapter 7

Using Animation in the Representation Construction Approach in Senior High School Chemistry Zeynep Yaseen

7.1

Introduction

Mastery of chemical concepts involves understanding three levels of chemistry: macroscopic, submicroscopic and symbolic (Johnstone, 1993). Chemistry educators and researchers have long been seeking effective ways to help students to develop their understanding of chemical concepts at those levels of representation (Ainsworth, 1999; Carolan, Prain, & Waldrip, 2008; Danish & Phelps, 2010; Tytler et al., 2006; Wu, Krajcik, & Soloway, 2001). Visualisation tools can help the students’ conceptual understanding in chemistry by making connections across those levels to represent aspects of the same phenomena at different levels (Ferk, Vrtacnik, Blejec, & Gril, 2003; Kelly & Akaygun, 2016). In particular, multiple visualisations (Rapp, 2007) in the learning of chemistry make learning active for students, increase student engagement with chemistry, improve learning outcomes and promote interactivity among students, all of which contribute to the construction of mental models during the learning process (Hubber, 2017; Hubber, Tytler, & Haslam, 2010; Tytler, Prain, Hubber, & Waldrip, 2013). Animation has been used by students to create multiple visual representations in science learning. There are different sorts of student-generated animations. Clay animation is one of the most frequently used student-generated animation tools in schools, but it is also time consuming and difficult to use when trying to show the movement of the models (Barab, Hay, Barnett, & Keating, 2000; Chang & Quintana, 2006; Schank & Kozma, 2002). Another way to construct animations is to use computers as animation tools. They promote active learning, visualisation of chemical processes, interpretation and reasoning (Papert, 1991). In Hoban, Macdonald,

Z. Yaseen (*) University of Technology, Sydney, Ultimo, NSW, Australia e-mail: [email protected] © Springer Nature Switzerland AG 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_7

159

160

Z. Yaseen

Ferry, and Hoban’s (2009) study with slowmation (abbreviated from ‘Slow Animation’) elementary school pre-service teachers created their own science animations. Data were collected from interviews with students, their concept maps and created animations. The results showed that the majority of pre-service elementary teachers greatly increased their scientific content knowledge. Hoban and Nielsen (2010) have also studied the Slowmations made by the pre-service teachers. Results show that creating animations helped students to learn science concepts through facilitating their thinking about the concepts in multiple ways. Even so, Hoban and Nielsen (2010) acknowledge their research to date has not provided a sufficient basis for understanding how constructing animations influences students’ learning. They call for further research ‘to better understand how students learn about particular concepts by videoing students over several hours as they create a Slowmation and encouraging them to think aloud (p. 37). While the contributions of animations are helpful in developing students’ understanding in science, the use of dynamic animations are an important tool for dealing with particular science concepts that include motion (Akaygun, 2016; Yaseen & Aubusson, 2018). However, there is very little research on student-generated animations that include the dynamism of chemical concepts especially in senior high school (Akaygun, 2016; Hoban et al., 2009; Hoban & Nielsen, 2010, 2014; Udo & Etiubon, 2011). The study reported in this chapter sought to further investigate the use of animation using different software and in the context of senior high school chemistry where teachers were implementing the Representation Construction Approach (RCA) to science pedagogy, which had previously been shown to be effective with primary school and early junior high school students (Tytler, Prain, et al., 2013).

7.2

Pedagogy: Representational Construction Approach

Many researchers have focused on the effectiveness of visualization with multiple representations to develop conceptual understanding in science (Ainsworth, 1999; Akaygun, 2016; Carolan et al., 2008; Hadenfeldt, Xiufeng, & Neumann, 2014; Saul, 2004; Wu et al., 2001). Waldrip and Prain (2013) state that understanding different representations and their translations into one-another and understanding their appropriate usage are critical to science education. Students represent and translate their learning with multiple representations in order to reach conceptual understanding (Ben-Zvi, Eylon, & Silberstein, 1986; Harrison & Treagust, 2000). While there is extensive research on students’ learning from multiple expert representations (Barak, Ashkar, & Dori, 2011; Barnea & Dori, 1996; Dori, Barak, & Adir, 2003; Kozma & Russell, 1997), the body of work investigating students’ own representations for learning is less well-developed. The representational construction approach provides a contribution to knowledge about students’ engagement with their own representations in science learning (Hubber, 2017; Prain & Tytler, 2013).

7 Using Animation in the Representation Construction Approach in Senior High. . .

161

Waldrip and Prain’s (2013) approach uses multiple representations of conceptual understanding for effective science learning. A key feature for effective learning they identify is ‘reaching general classroom negotiation of representational understanding’ (p. 16). Students need to explain their representations to reveal their understanding of concepts. They are expected to develop key science ideas through multiple representations, and these are reported and scrutinized in classroom discussions. Waldrip and Prain (2013, p. 16–17) conclude: Students learn most effectively in science and are engaged more, where they were challenged to develop meaningful understandings, where individual learning needs and preferences were catered for, where a range of assessment tasks were used, where the nature of science was represented in its social, personal and technological dimensions.

These findings are consistent with earlier research conducted by Tytler, Waldrip, and Griffiths (2004). Representation construction is teacher guided inquiry pedagogy. Representational challenge initiates and is central to the representational construction approach. It includes construction of students’ own representations, discussion and improvement of these representations in a class environment, and discussion of the role of representation in learning and knowing. This involves a new organisation or synthesis of existing representations and concerns the ways in which a concept or phenomenon is related to the variety of viable ways in which a concept or phenomenon could be represented and explained. It also acknowledges individual differences in the ways students transfer their thinking to their representations and explain their representations. The first part of a representational challenge occurs when students create their representations, explain them or comment on what they did or did not represent, and how their representations are related to each other. All these are parts of the ‘challenge.’ Students are challenged to make a reasoned claim about the phenomena they are studying. It is important that students learn to put forward not only their reproductions of known representations, but also the challenges they will face in the interpretation, synthesis and organisation of representations of new phenomena (Tytler, Prain, et al., 2013). The second part of representational challenge includes ‘class or group discussions.’ In most cases, representational challenges are group work; they compare, contrast and discuss the group’s responses to a challenge. They involve the relationship between students’ representational responses in small groups or as individuals. The teacher plays an active role during the discussion by questioning, shaping and assessing students’ representations. These teacher-guided discussions in the classroom environment ‘generate and clarify meaning’ (Tytler, Prain, et al., 2013, p. 61) and develop ‘shared understanding of the appropriateness and efficacy of various representations, and their role’ (p. 51). While students need to explain key ideas through multiple representational modes linked to science phenomena and share their personal work and experiences with fellow students in their classroom, teachers too are challenged to develop their understandings of the science concepts when planning and organising representational work. They need to decide what representational abilities would be

162

Z. Yaseen

satisfactory for students’ learning and to provide scaffolding to students (Fraser, Seddon, & Eagleson, 1982) for their active roles in the construction of multiple representations (Waldrip & Prain, 2013). Tytler, Hubber, Prain, and Waldrip (2013) developed principles for the representation construction approach, which include a learning process for both teachers and students. The four principles of the representation construction approach are outlined below. 1. ‘Teaching sequences are based on sequences of representational challenges’ (Tytler, Hubber, et al., 2013, p. 34). Students actively create representations to explore and make claims about phenomena. Teachers explain key concepts, important ideas and their representations to guide students in the beginning stage of a topic. Students are supported with guided questions to explore before they are given the canonical representational forms. They are supported to solve problems by generating their own explanatory representations, which they seek to improve through peer discussion and teacher guidance. There is a back-and-forth between student-generated representations and teacher- introduced representations that encourages and challenges students to improve and organise their understanding. 2. ‘Representations are explicitly discussed’ (Tytler, Hubber, et al., 2013, p. 35). Teachers’ scaffolding and classroom discussions are necessary both during and after constructing representations. Teachers scaffold the class discussions to support student representation construction and to share, compare and critique examples of student representations. In these discussions, students are helped to understand that multiple representations are necessary when using different aspects of a concept, and that in a guided process, group agreement on ‘generative representations’ (p. 35) also assists the resolution of a problem. 3. ‘Meaningful learning involves representational/perceptual mapping’ (Tytler, Hubber, et al., 2013, p. 35). Students undertake a task in strong perceptual contexts. They are supported in mapping or reasoning between visible features of objects, likely implications and representations. Student engagement in perceptual contexts is important. Activity sequences need to engage students in learning as personally meaningful and challenging, according to students’ interests and values. 4. ‘Formative and summative assessment is ongoing’ (Tytler, Hubber, et al., 2013, p. 35). Students and teachers continuously evaluate the process to assess the sufficiency of representations and their explanations of them. Formative and summative assessments allow students to generate further representations. Support is also essential for students to coordinate and re-present multiple modes during the learning process. This chapter reports on one pedagogical intervention as part of a larger research project funded by the University of Technology Sydney that dealt with the adoption of a representational construction approach (RCA) in senior high school science classrooms. It focusses on the ways in which RCA is adapted in a particular chemistry classroom. A distinctive aspect of this study is the use of animation as

7 Using Animation in the Representation Construction Approach in Senior High. . .

163

the form of students’ representations. In previous studies of the representational construction approach (Tytler, Prain, et al., 2013), the dimension of ‘usefulness’ evolved as students compared their representations with those of peers, rather than direct comparison with established canonical representations. This study addresses new dimensions of RCA research by investigating how students compare their representations with both the expert and peer animations. The research questions are: 1. In what ways do student created animations influence conceptual learning with senior high school chemistry students? 2. How do students perceive the relative effectiveness of learning through creation of science explanation animations and critical comparison with those of peers and learning from expert/canonical science animation explanations? The site for the investigation was the teaching and learning about the states of matter in a year 11 chemistry class. As in many topics in science education the research literature documenting student misconceptions is quite extensive (Hadenfeldt et al., 2014; Kind, 2004) and in states of matter this is especially the case (Çalık & Ayas, 2005; Jones, 1984; Osborne & Cosgrove, 1983; Tatar, 2011; Tsai, 1999). In this study, the implementation of the representational construction approach sought to take this into account. The study was concerned with how teachers selected and adapted the representational construction approach and in particular how the use of animation could be accommodated by the approach.

7.3

Methodology

This investigation took the form of a case study involving one teacher and her students. The participants were 28-Year 11 Science students in Istanbul taught by the same experienced chemistry teacher in two classes. The concept the students were learning was states of matter. The researcher worked collaboratively with the teacher. The researcher planned the lessons and the teacher implemented the lesson sequence in the classroom as a professional learning experience. The teacher carried out all activities after negotiation with the researcher. The researcher sought to understand the learning process through working with the teacher and through observation of the students’ participation in the learning experiences. The K-sketch software program (http://www.k-sketch.org/) was used by students to construct the animations in the representation creation process. The program was designed to create basic animations from sketches. It is a free drawing and design program rather than a science-tools program and there are no directions to students about scientific animations such as molecular shape. An intention of this research was to see if this software could be used as a visual scaffolding tool to help students to understand the molecular world and to make links between what they can see and not see in chemistry. The software enables students to show the movement of particles in the different states of matter.

164

Z. Yaseen

The K-sketch animation program was taught to students by the researcher during two lesson periods, following which they designed their own animations in small groups. Students used the K-sketch animation guide (summary of guide: Appendix). All students created three non-science related animations, one in each tutorial from the K-sketch animation guide, during the two lesson periods. It was presumed that the students had then learned the program sufficiently well to create their own animations using K-sketch software program. There were 4 tutorials, each of which were prerequisite for progressing to the next tutorial i.e. students cannot progress to tutorial two before creating the animation in the first tutorial. Since the students successfully created non-science animations in each tutorial, it was concluded that they learned the animation skills needed to apply the animation software to the representation of particle movement in the different states of matter. Those skills included: translate, scale, rotate, set timing, move relative, appear, disappear, trace, repeat motion, copy motion, physically simulate, interpolate, move forward, move back, deform, move limb, orient to path. During the chemistry lessons the students created their representations of states of matter in the animation software program as partners or in groups of three. The intervention in this study drew on the RCA to produce a teaching sequence for investigation that consisted of interdependent elements including students creating their own animation representations, teacher guidance, watching expert animations and discussions to critique the animations. First lesson in the teaching sequence refers to the pre-kinetic molecular theory (KMT) activities, The teaching sequence covered 4 lessons with each involving about 40 min of class time. The sequence of lessons in the intervention is outlined below and summarized in Table 7.1. 1. In the first week, this time was used to explain the aim of the study and the animation program was taught to each class. The researcher used an active learning process to teach the program during two class periods (80 min). Students used their own laptops and they followed the user guide step by step with researcher to learn how to use the program. They created 4 non-scientific animations: Count Down, The Car Jumped Over the Plane, Rolling Down, Disappearing of Small Balls (Appendix). All of these animations involved the students’ use of the specific tools of this animation program including the representation of motion. This facilitated their creating of multiple representations dealing with the dynamic processes of molecules. 2. In the second week, the students were given representational challenges where they were to create their own representations (animations) about the states of matter using the animation program (40 min). (a) Imagine that you could see into the matter with an impossibly powerful microscope to see what matter was made up of and what it was doing. (b) Create an animation of what you imagine. They had explicit discussions of their representations with their partners before they were provided with canonical forms of the explanations. Those discussions

7 Using Animation in the Representation Construction Approach in Senior High. . .

165

Table 7.1 The framework of interventions Period 1st week

Time Two class hours, 80 min

Interventions Aim of the study, Pre-tests drawings Teaching of K-Sketch animation program Producing science animation

2nd week

One class hour, 40 min

3rd week

4th week

One class hour, 40 min 10– 15 min

Follow-up discussions Post-test drawings Student interviews

4th week

1 class hour

Teacher interview

Instrument K-Sketch software program

Analysis methods Two class hours, 80 min SOLO taxonomy, statistical quantitative analysis

K-sketch software program Observations Audio and video recordings Observations video recordings Expert animations

Coding the students’ videotaped and audiotaped dialogues with partner and the teacher, qualitative data

Semi-structured stimulated recall interview questions Students’ K-Sketch animations Semi-structured interview questions

Coding the students’ videotaped dialogues, qualitative analysis SOLO taxonomy, statistical quantitative analysis Analysis of transcript data, qualitative analysis

Analysis of transcript data, qualitative analysis

included seeking understanding of their partners’ claims, exposing their own ideas and sometimes advising and convincing the partner. 3. In the following (third) week (40 min), students participated in a class discussion in order to discuss their animations explicitly with their classmates. The teacher showed each group’s animation to the whole class by projecting it onto a screen so that the whole class could watch it. The members of each group then explained their animations to the whole class. The teacher invited students to comment on and critique others’ animations, and to ask questions about anything they thought was unclear in the animation or was inconsistent with their own ideas. The teacher then showed canonical forms of the animations. The students watched expert animations created in the K-sketch and in other software programs (eg: https:// www.chem.purdue.edu/gchelp/liquids/character.html). All of these expert animations were visualising the states of matter. The teacher led a discussion that provided an opportunity for students to think more about the changes of state and to see the similarities and differences between their animations and the expert animations. 4. In the following week, after completing the intervention, the researcher interviewed all the students as pairs (or groups of three) and also interviewed the teacher.

166

7.3.1

Z. Yaseen

Data Collection

The data collected included: 1. 2. 3. 4.

Pre and post-test drawings Video recording of classroom discourse, Artefacts (students’ representations as animations), Transcripts of semi-structured interviews with the teacher and students.

All the data recordings and transcriptions were in Turkish and then translated into English by researcher. • Pre and post- test Drawings: All students answered the following question on paper as pre and post-tests. “Imagine an ice block in a beaker at -20
C and you can also see the particles of that ice block. Suppose that you are increasing the temperature until 100
C and still you can see the particles. Please draw what you imagine you would see as the temperature changes.”. • Videotaped and audiotaped recordings: Conversations between the teacher and students and interactions among the students were videotaped during the students’ animation creating process. Classes 11-A and 11-B were video recorded as they were using the K-Sketch animation program, creating their own animations on that program, and in the follow-up discussion classes. To record these activities in the classroom at the same time, the researcher set up one camera at the front of the room and another at the back of the room. As well, in each of the two groups a volunteer group used a close-focus camera to record conversations. The researcher also had a camera in her hand to record conversations as she walked among groups while they created the animations. However, when the researcher was circulating, she didn’t capture the all of the context of discussion for all groups. • K-Sketch animations in pairs: The animations were retained for classroom discussion by peers and with the teacher and for subsequent examination by the researcher. • Interviews: Semi-structured, stimulated recall interviews (McMillan & Schumacher, 2006; Slough, 2001) were held with all student groups and also with the teacher. The students were interviewed after the follow-up discussion class. The aim of interviewing was to gain the students’ insights about the concepts that were studied and the effectiveness of whole animation creating process. It was also to find out what the students said they were thinking or feeling while they were doing the animation task. The researcher stopped the animation at some points and asked the students questions related to the animations. She also asked the groups about teacher support, partner interaction, and for their overall ideas about the whole learning experience. Interviews with each student group took about 15–20 min, with each interview session audiotaped and transcribed. In addition, the researcher interviewed the teacher for about an hour to

7 Using Animation in the Representation Construction Approach in Senior High. . .

167

ask about the advantages, and disadvantages of using student-generated animations in science classes. She also talked about each group’s animations in both Class 11-A and 11-B when watching them again during interview session. The teacher’s interview session was also audiotaped and transcribed.

7.3.2

Data Analysis

Students’ answers to the pre/post-test question were categorised according to simplified elements of ‘The Particle Theory of Matter: The Kinetic Molecular Theory’. The Kinetic Molecular Theory (KMT) emphasises the movement of particles. It is based on the following five postulates: 1. All matter consists of extremely small particles. 2. All particles of one substance are identical. 3. The spaces between particles are very large compared with the size of particles themselves (in gases). 4. The particles in matter attract one another. 5. All particles of matter are constantly in motion (NSW Department of Education and Communities 1999–2011). Each student’s drawing was categorised according to whether or not it included representations of the KMT postulates. The students’ interviews and the teacher interview were transcribed from audiotaped recordings. Qualitative analysis was used so that participants’ perspectives and actions could be examined in depth in natural situations. An interpretivist epistemology was used to seek understanding of student perceptions about their animations through interviews and observations (Angen, 2000; Creswell & Plano Clark, 2007; Mackenzie & Knipe, 2006). This followed the general process of inductive data analysis outlined by McMillan and Schumacher (2006, pp. 364–377), including establishing text extracts from interview transcripts that exemplified identified patterns of response. The student animations were not formally analyzed because they were regarded as a heuristic to facilitate the students’ depictions of their envisioning of the particles of matter in the different states rather than as an indicator of students’ learning outcomes. Nevertheless, the animations were critically examined during classroom work as part of discussion of the students’ representations among the students and between the students and the teacher. Subsequently, the researcher reviewed the animations in the light of the transcripts of the class discussions to determine the extent to which they revealed the kinds of misconceptions students demonstrated, taking into account that in some cases the nature of animation depictions may have reflected students’ inexperience with the animation software rather than misconceptions they held.

168

7.4 7.4.1

Z. Yaseen

Results Pre-test and Post-test Results

The Wilcoxon signed-rank test was used to compare the sum of the postulates in the drawings of pre- and post-test answers. The p-value was .001 ( 0.05) (Table 7.2). Figure 7.2 shows the percentage of students’ drawings that included the second postulate, ‘All particles of one substance are identical’, was 36.8% in the pre-test and 73.7% in the post-test. There large increase was statistically different in the Wilcoxon signed-rank test, with the p-value of .035 ( Group 3

Group 1 > Group 2 > Group 3

Post hot test Group 1 > Group 2 > Group 3

b 2

Group 1: Paper-pencil test with trend items; group 2: Computer-based assessment with trend items; group 3: Computer-based assessment with new items η ¼ eta-squared effect size: small ¼ 0.01, medium ¼ 0.06, large ¼ 0.14 (Cohen, 1988) ***: P < 0.001

a

Interpret data and evidence scientifically

Evaluate and design scientific enquiry

Explain phenomena scientifically

Scientific competency Total

Table 12.2 One-way ANOVA results of students’ scientific competencies with different assessment methods

292 Y.-C. Chen et al.

12

Exploring Students’ Scientific Competency Performance on PISA Paper-Based. . .

293

Table 12.3 One-way ANOVA results of different levels of students’ scientific competency performance among groups Level of scientific competency High

Medium

Low

Groupa 1 2 3 1 2 3 1 2 3

N 260 341 496 209 304 588 146 284 660

M 621.60 610.56 599.38 510.20 511.01 508.19 386.31 385.28 379.21

SD 37.21 34.94 29.21 29.62 28.59 29.03 43.74 44.12 50.14

F 39.843

p < 0.001***

1.058

0.347

2.367

0.094

ES (η2)b 0.07

Post hoc test Group 1 > Group 2 > Group 3

a

Group 1: Paper-pencil assessment with trend items; group 2: Computer-based assessment with trend items; group 3: Computer-based assessment with new interactive items b 2 η ¼ eta-squared effect size: small ¼ 0.01, medium ¼ 0.06, large ¼ 0.14 (Cohen, 1988) ***: P < 0.001 Table 12.4 Correlation matrix, means, and standard deviations of the variables for Taiwan PISA 2015 FT dataset (N ¼ 1305) Variables 1. ICT use for schoolwork activities 2. ICT interest 3. Perceived ICT competence 4. Perceived autonomy 5. Scientific competency Mean Standard deviation

1 1 0.08** 0.14*** 0.12*** 0.11*** 15.04 4.93

2

3

4

5

1 0.44*** 0.47*** 0.27*** 15.44 2.66

1 0.66*** 0.04 13.98 2.60

1 0.17*** 14.65 2.49

1 493.33 97.45

***: P < 0.001; **: P < 0.01

different assessment methods. The results revealed that the high achievers in PBA outperformed their counterparts in CBA and Interactive CBA Groups while the CBA also performed significantly better than the Interactive CBA (F ¼ 39.843, p < 0.001, ES ¼ 0.07). However, there was no testing mode effect on medium and low achievers’ scientific competency performance (F ¼ 1.058, 2.367, respectively, p > 0.05). RQ2. How are the constructs of ICT familiarity related with students’ scientific competency? Correlation matrix, means, and standard deviations of the four ICT-related independent variables and the dependent variable of scientific competency for the Taiwan PISA 2015 FT dataset are given in Table 12.4 The correlation analysis results indicated that scientific competency was significantly correlated with ICT use for schoolwork activities (r ¼ 0.11, p < 0.001), ICT interest (r ¼ .27, p < 0.001), perceived autonomy related to ICT (r ¼ 0.17, p < 0.001). However, there was no

294

Y.-C. Chen et al.

ICT interest .55*** .60***

.08*

.40*** Perceived ICT competence

e1 -.41***

Scientific competency

.77*** Perceived autonomy related to ICT use

.17***

R2=.18

.24***

.12***

.15*** ICT use for schoolwork activities

Fig. 12.2 The tested model of ICT constructs and scientific competency with the standardized coefficients

significant correlation between scientific competency and perceived ICT competence (r ¼ .04, p > 0.05). The hypothesized model (Fig. 12.1) of the study was explored by analysing the paths among ICT use for schoolwork activities, ICT interest, perceived ICT competence, perceived autonomy related to ICT, and scientific competency. Several fit indicators were used to assess the fitness of the datasets to this model, such as goodness-of-fit index (GFI ¼ 0.924), the comparative fit index (CFI ¼ 0.919), the standardized root mean square residual (SRMR ¼ 0.056), and the root mean square error of approximation (RMSEA ¼ 0.055). The indices of GFI and CFI larger than 0.90 represented a reasonable fit. A value of SRMR less than 0.08 was considered a good fit, and a value of RMSEA less than 0.08 represented a reasonable fit (Schumacker & Lomax, 2010). The test results supported the quality of the model for the data from the participating students. In Fig. 12.2, curved arrows represent correlation among variables and the straight arrows indicate the path strengths (ß) for the dependent variable predicted by individual independent variables. The results of SEM analysis revealed that the four ICT-related variables had direct effects on Taiwanese 15-year-old students’ scientific competency. All of the path standardized coefficients were statistically significant. ICT interest has the strongest positive direct effect on students’ scientific competency (ß ¼ 0.40), followed by perceived autonomy related to ICT (ß ¼ 0.24), and ICT use for schoolwork activities (ß ¼ 0.12). The negative direct effect on scientific competency was from students’ perceived ICT competence with a statistically significant effect of 0.41. Regression results (R2) presented the percentage of variances explained for the dependent variable by the combined independent

12

Exploring Students’ Scientific Competency Performance on PISA Paper-Based. . .

295

variables. The total variances explained for scientific competency by the four independent variables was 18% (R2 ¼ 0.18).

12.5

Discussion

One of the noteworthy findings of this study is that we provide empirical evidence to confirm the potential effect of different testing modes (i.e., paper-basedvs. computer-based assessment) on student performance. Our research findings reveal that Group 1 PBA students significantly outperformed their counterparts in Groups 2 and 3 CBA on overall scientific competency performance. Interpreting this result as a potential effect is mainly because there are other threats for a solid conclusion of testing mode effect. For example, as described in the method section, even though the participating students were randomly assigned into each group for performance assessment, without a pretest to compare initial differences among groups, we were not able to ensure that the scientific competency of the three groups were equally matched. Similar testing mode effect on student learning achievement was revealed by Backes and Cowan (2019); they investigated a same examination being administered in online and offline formats and concluded an online test penalty ranging from 0.10 to 0.25 standard deviations for the mathematics and English tests, respectively. Without any additional evidence for inference, we were not able to identify what factors contributed to the testing mode effect. One possible reason could be the lack of experience and practice for students taking achievement tests through computer-based assessment. In addition, while responding to questions in the paper-pencil test, it might be easier for students to find their own written errors than in the computer-based assessment tasks. Moreover, most schoolwork, assignments, and important examinations have been conducted with paper-pencil assessment for avoiding possible cheating on the test. The ANOVA results revealing students in both the PBA and CBA static-item assessment modes performed significantly better than those of the CBA interactiveitem assessment mode deserve a special note from science educators and test developers. As mentioned in the literature review section, although dynamic and interactive representations have the potential to support students’ understanding and learning of science (Ainsworth, 1999, 2006; Plass et al., 2012), students may struggle to infer such dynamic processes or develop alternative ideas and conceptions (Chabalengula et al., 2012; Chang & Tzeng, 2018). In addition, Unsworth and Chan (2009) concluded that different types of image/language relations in hard copy or computer texts differ in the degree of difficulty they pose for students. Consequently, all the above factors are likely to contribute to the lower achievement performance of the students who took the computer-based interactive assessment. Apart from the results of Backes and Cowan’s (2019) study that the testing mode effect is more pronounced for low achievers on their test performance of mathematics and language, we found that the variation of high achievers’ scientific competency performance on different testing modes was much more noticeable than

296

Y.-C. Chen et al.

medium and low achievers’ variation of performance. Given that scores of national or international studies are used for accountability purposes, the inconsistency of testing mode effects for different achievement levels of students on different subject matters and with various assessment strategies deserves attention from educators, teachers, and policymakers. Future studies are recommended to examine the factors associated with testing mode effects and explore technical strategies to minimize the testing mode effect on student performance. Another noteworthy finding of the study is that there is a weak but significant effect of students’ scientific competency explained by their ICT familiarity. A total of 18% of variance of students’ scientific competency was explained by the variables of their ICT interest, perceived ICT competence, perceived autonomy related to ICT use, and ICT use for schoolwork activities. According to Moore, Notz, and Flinger (2013), R2 value less than 0.3 is generally considered a nil or very weak effect size. More importantly, there is a negative and significant predictive effect (.41) on the subscale of students’ perceived ICT competence on their scientific competency. The low variances of students’ scientific competency explained by their ICT familiarity along with the low predicting effect of student perceived ICT competency subscale indicate that student overall ICT familiarity is not a strong determinant of scientific competency. In other words, students simply equipped with a high level of ICT familiarity have limited advantage of outperforming their counterparts of low ICT familiarity when other significant determinants of scientific competency are controlled. Overall, the data analyses of the PISA 2015 Field Trial dataset allowed us to infer that students’ ICT familiarity has a very weak effect on their scientific competency performance. As the computer has become a major venue not only for learning and teaching but also for assessment, the empirical evidence of this study seems to support the feasibility and validity of using computer-based assessment in largescale studies. However, the negative testing mode effects on high achievers’ scientific competency performance serve to remind educators and policymakers that the change from static representation of paper-based assessment to interactive computerbased assessment should be cautiously considered when students’ scores are used to serve as accountability purposes. Additional studies, techniques, or longer time of practice for students to get used to the relatively unfamiliar interactive computerbased assessment might minimize the testing mode effect.

References Ainsworth, S. (1999). The functions of multiple representations. Computers & Education, 33(2), 131–152. Ainsworth, S. (2006). DeFT: A conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183–198. Al-Balush, S. M., & Al-Hajrib, S. H. (2014). Associating animations with concrete models to enhance students’ comprehension of different visual representations in organic chemistry. Chemistry Education Research and Practice, 15, 47–58.

12

Exploring Students’ Scientific Competency Performance on PISA Paper-Based. . .

297

Areepattamannil, S., & Santos, I. M. (2019). Adolescent students’ perceived information and communication technology (ICT) competence and autonomy: Examining links to dispositions toward science in 42 countries. Computers in Human Behavior, 98, 50–58. Backes, B., & Cowan, J. (2019). Is the pen mightier than the keyboard? The effect of online testing on measured student achievement. Economics of Education Review, 68, 89–103. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall. Barak, M., & Hussein-Farraj, R. (2013). Integrating model-based learning and animations for enhancing students’ understanding of proteins structure and function. Research in Science Education, 43, 619–636. Blanco-Lopez, A., Espana-Ramos, E., Gonzalez-Garcia, F. J., & Franco-Mariscal, A. J. (2015). Key aspects of scientific competence for citizenship: A Delphi study of the expert community in Spain. Journal of Research in Science Teaching, 52(2), 164–198. Bong, M., & Skaalvik, E. M. (2003). Academic self-concept and self-efficacy: How different are they really? Educational Psychology Review, 15(1), 1–40. Chabalengula, V. M., Sanders, M., & Mumba, F. (2012). Diagnosing students’ understanding of energy and its related concepts in biological context. International Journal of Science and Mathematics Education, 10(2), 241–266. https://doi.org/10.1007/s10763-011-9291-2 Chang, H. Y., & Tzeng, S. F. (2018). Investigating Taiwanese students’ visualization competence of matter at the particulate level. International Journal of Science and Mathematics Education, 16(7), 1207–1266. Cheema, J. R., & Hang, B. (2013). Quantity and quality of computer use and academic achievement: Evidence from a large-scale international test program. International Journal of Education and Development using Information and Communication Technology, 9(2), 95–106. Chi, S., Wang, Z., Liu, X., & Zhu, L. (2017). Associations among attitudes, perceived difficulty of learning science, gender, parents’ occupation and students’ scientific competencies. International Journal of Science Education, 39(16), 2171–2188. Chiu, J. L., DeJaegher, C. J., & Chao, J. (2015). The effects of augmented virtual science laboratories on middle school students’ understanding of gas properties. Computers & Education, 85, 59–73. Chiu, M.-H., & Duit, R. (2011). Globalization: Science education from an international perspective. Journal of Research in Science Teaching, 48(6), 553–566. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Academic. Csapo, B., Ainley, J., Bennett, R. E., Latour, T., & Law, N. (2012). Technological issues for computer-based assessments. In P. Griffin, B. McGraw, & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 143–230). New York: Springer. de Jong, T., Linn, M. C., & Zacharias, Z. C. (2013). Physical and virtual laboratories in science and engineering education. Science, 340(6130), 305–308. DeBoer, G. E. (2011). The globalization of science education. Journal of Research in Science Teaching, 48(6), 567–591. Dickhäuser, O., & Stiensmeier-Pelster, J. (2003). Gender differences in choice of computer courses: Applying an expectancy-value model. Social Psychology of Education, 6, 173–189. Dohn, N. B. (2013). Upper secondary students’ situational interest: A case study of the role of a zoo visit in a biology class. International Journal of Science Education, 35(16), 2732–2751. Dolin, J. (2007). Science education standards and science assessment in Denmark. In D. Waddington, P. Nentwig, & S. Schanze (Eds.), Making it comparable: Standards in science education (pp. 71–82). Munster: Waxmann. European Council. (2006). Key competences for lifelong learning – European reference framework. Luxembourg: Author. Available from http://www.britishcouncil.org/sites/default/files/youth-inaction-keycomp-en.pdf Fallon, G. (2019). Using simulations to teach young students science concepts: An experiential learning theoretical analysis. Computers & Education, 135, 138–159.

298

Y.-C. Chen et al.

Fensham, P. (2009). Real world contexts in PISA science: Implications for context-based science education. Journal or Research in Science Teaching, 46(8), 884–896. Fraillon, J., Ainley, J., Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for life in a digital age. Singapore: Springer. Freeman, B., Marginson, S., & Tytler, R. (Eds.). (2014). The age of STEM: Educational policy and practice across the world in science, technology, engineering and mathematics. New York: Routledge. Gale, J., Wind, S., Koval, J., Dagosta, J., Ryan, M., & Usselman, M. (2016). Simulation-based performance assessment: An innovative approach to exploring understanding of physical science concepts. International of Science Education, 38(14), 2284–2302. Goldhammer, F., Gniewosz, G., & Zylka, J. (2016). ICT engagement in learning environments. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds.), Assessing contexts of learning: An international perspective (pp. 331–351). Singapore: Springer. Gordon, I. J. (1968). Test manual for the how I see myself scale. Gainesville: Florida Educational Research and Development Council. Gumus, S., & Atalmis, E. H. (2011). Exploring the relationship between purpose of computer usage and reading skills of Turkish students: Evidence from PISA 2006. TOJET – Turkish Online Journal of Educational Technology, 10(3), 129–140. Guzmán, A., Oliveros, D., & Mendoza, M. (2017). Scientific competencies: A mechanism to favour the inclusion of working market professionals. Journal of Baltic Science Education, 16(2), 175–187. Hagger, M. S., Sultan, S., Hardcastle, S. J., & Chatzisarantis, N. L. D. (2015). Perceived autonomy support and autonomous motivation toward mathematics activities in educational and out-ofschool contexts is related to mathematics homework behavior and attainment. Contemporary Educational Psychology, 41, 111–123. Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. Hong, Z.-R., & Lin, H.-s. (2011). An investigation of students’ personality traits and attitudes toward science. International Journal of Science Education, 33(7), 1001–1028. Hu, X., Gong, Y., Lai, C., & Leung, F. (2018). The relationship between ICT and student literacy in mathematics, reading, and science across 44 countries: A multilevel analysis. Computers & Education, 125, 1–13. Jaakkola, T., Nurmi, S., & Veermans, K. (2011). A comparison of students’ conceptual understanding of electric circuits in simulation only and simulation-laboratory contexts. Journal of Research in Science Teaching, 48(1), 71–93. Jack, B., Lin, H.-s., & Yore, L. D. (2014). The synergistic effect of affective factors on student learning outcomes. Journal of Research in Science Teaching, 51(8), 1084–1101. Krathwohl, D. (2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice, 41(4), 212–218. Kuo, C.-Y., & Wu, H.-K. (2013). Toward an integrated model for designing assessment systems: An analysis of the current status of computer-based assessments in science. Computers & Education, 68, 388–403. Li, N., & Black, J. B. (2016). Inter-level scaffolding and sequences of representational activities in teaching a chemical system with graphical simulations. Journal of Science and Technology, 25, 715–730. Lin, H.-s., Hong, Z., & Chen, Y. (2013). Exploring the development of college students’ situational interest in learning science. International Journal of Science Education, 35(13), 2152–2173. Lin, H.-s., Hong, Z.-R., & Huang, T. (2012). The role of emotional factors in building public scientific literacy and engagement with science. International Journal of Science Education, 34 (1), 25–42. Lin, H.-s., Lawrenz, F., Lin, S., & Hong, Z.-R. (2012). Relationships among affective factors and preferred engagement in science related activities. Public Understanding of Science, 22(8), 941–954.

12

Exploring Students’ Scientific Competency Performance on PISA Paper-Based. . .

299

Lin, H.-s., Shiau, B., & Lawrenz, F. (1996). The effectiveness of teaching science with pictorial analogies. Research in Science Education, 26(4), 495–511. Liu, W. C., Wang, J. C. K., & Ryan, R. M. (2016). Building autonomous learners: Perspectives from research and practice using self-determination theory. Singapore: Springer. Ministry of Education. (2015). Curriculum outlines of “nature science and living technology” for K–12. Taipei: Author. Moore, D. S., Notz, W. I., & Flinger, M. A. (2013). The basic practice of statistics (6th ed.). New York: W. H. Freeman. Murphy, S., MacDonald, A., Danaia, L., & Wang, C. (2018). An analysis of Australian STEM education strategies. Policy Futures in Education, 17(2), 122–139. National Research Council. (2000). Inquiry and the National Science Education Standards: A guide for teaching and learning. Washington, DC: National Academies Press. National Research Council. (2012). A framework for K–12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press. Naumann, J. (2015). A model of online reading engagement: Linking engagement, navigation, and performance in digital reading. Computers in Human Behavior, 53, 263–277. Nieswandt, M. (2007). Student affect and conceptual understanding in learning chemistry. Journal of Research in Science Teaching, 44(7), 908–937. Nikou, S. A., & Economides, A. A. (2016). The impact of paper-based, computer-based and mobile-based self-assessment on students’ science motivation and achievement. Computers in Human Behavior, 55(Part B), 1241–1248. OECD. (2015). Students, computers and learning: Making the connection. Paris: OECD Publishing. OECD. (2016). PISA 2015 results (Vol. I): Excellence and equity in education. Paris: PISA, OECD Publishing. https://doi.org/10.1787/9789264266490-en OECD. (2017). PISA 2015 assessment and analytical framework: Science, reading, mathematics, financial literacy and collaborative problem solving. Paris: OECD Publishing. Olympiou, G., Zacharias, Z. C., & de Jong, T. (2013). Making the invisible visible: Enhancing students’ conceptual understanding by introducing representations of abstract objects in a simulation. Instructional Science, 41, 575–596. Palmer, D. H. (2004). Situational interest and the attitudes towards science of primary teacher education students. International Journal of Science Education, 26(7), 895–908. Petko, D., Cantieni, A., & Prasse, D. (2017). Perceived quality of educational technology matters: A secondary analysis of students’ ICT use, ICT-related attitudes, and PISA 2012 test scores. Journal of Educational Computing Research, 54(8), 1070–1091. Plass, J. I., Milne, C., Homer, B. D., Schwartz, R. N., Hayward, E. O., Jordan, T., et al. (2012). Investigating the effectiveness of computer simulations for chemistry learning. Journal of Research in Science Teaching, 49(3), 394–419. Renninger, K. A. (2000). Individual interest and its implications for understanding intrinsic motivation. In J. M. Harackiewicz & C. Sansone (Eds.), Intrinsic and extrinsic motivation: The search for optimal motivation and performance (pp. 373–404). San Diego: Academic. Rutten, N., van Joolingen, W. R., & van der Veen, J. T. (2012). The learning effects of computer simulations in science education. Computers & Education, 58(1), 136–153. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68–78. Ryan, R. M., & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness. New York: Guilford. Ryoo, K., & Bedell, K. (2017). Supporting linguistically diverse students’ science learning with dynamic visualization through discourse-rich practices. Journal of Research in Science Teaching, 56(3), 270–301. Schumacker, R. E., & Lomax, R. G. (2010). A beginner’s guide to structural equation modeling (3rd ed.). New York: Routledge.

300

Y.-C. Chen et al.

Senkbeil, M. (2018). Development and validation of the ICT motivation scale for young adolescents. Results of the international school assessment study ICILS 2013 in Germany. Learning and Individual Differences, 67, 167–176. Singh, K., Allen, K. R., Scheckler, R., & Darlington, L. (2007). Women in computer-related majors: A critical synthesis of research and theory from 1994 to 2005. Review of Educational Research, 77(4), 500–533. Skryabin, M., Zhang, J., Liu, L., & Zhang, D. (2015). How the ICT development level and usage influence student achievement in reading, mathematics, and science? Computers & Education, 85, 49–58. Smith, K. C., & Villarreal, S. (2015). Using animations in identifying general chemistry students’ misconceptions and evaluating their knowledge transfer relating to particle position in physical changes. Chemistry Education Research and Practice, 16, 273–282. Taasoobshirazi, G., & Sinatra, G. M. (2011). A structural equation model of conceptual change in physics. Journal of Research in Science Teaching, 48(8), 901–918. Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81(1), 4–28. United Nations Environment Programme. (2012). 21 issues for the 21st century: Result of the UNEP foresight process on emerging environmental issues. Nairobi: Author. Unsworth, L., & Chan, E. (2009). Bridging multimodal literacies and national assessment programs in literacy. Australian Journal of Language and Literacy, 32(3), 245–257. Vekiri, I. (2010). Socioeconomic differences in elementary students: ICT beliefs and out-of-school experiences. Computers & Education, 54, 941–950. Velasco, M. S. (2014). Do higher education institutions make a difference in competence development? A model of competence production at university. Higher Education, 68(4), 503–523. Wang, T., & Tseng, Y. (2018). The comparative effectiveness of physical, virtual, and virtualphysical manipulatives on third-grade students’ science achievement and conceptual understanding of evaporation and condensation. International Journal of Science and Mathematics Education, 16(2), 203–219. Yaseen, Z. (2018). Using student-generated animations: The challenge of dynamic chemical models in states of matter and the invisibility of the particles. Chemistry Education Research and Practice, 19, 1166–1185. Zylka, J., Christoph, G., Kroehne, U., Hartig, J., & Goldhammer, F. (2015). Moving beyond cognitive elements of ICT literacy: First evidence on the structure of ICT engagement. Computers in Human Behaviour, 53, 149–160.

Ya-Chun Chen received a PhD degree in science education from National Sun Yat-sen University (NSYSU), Taiwan. Currently she is an assistant professor at the Institute of Education, National Chiao Tung University, Taiwan. Her research interests focus on systems thinking, inquiry-based teaching and assessment. Zuway-R Hong is a Distinguished Professor in the Institute of Education at the NSYSU and the Center for General Education, Kaohsiung Medical University, Taiwan. Her research fields include gender education, moral and character education, educational psychology and science education. Huann-shyang Lin is a Chair Professor at the Si Wan College of NSYSU and a professor at the Australian Catholic University. He is also the Editor-in-Chief of International Journal of Science and Mathematics Education. His research concerns science education, teacher professional development, scientific literacy teaching and assessment.

Chapter 13

Towards more Valid Assessment of Learning from Animations Richard Lowe and Jean-Michel Boucheix

13.1

Introduction

The burgeoning use of animation in education is a relatively recent development that has been largely driven by the information technology revolution. Before the advent of sophisticated graphics-capable computing, animations were time-consuming, difficult, and expensive to produce then distribute. They therefore found little application in mainstream classroom contexts where traditional textbooks and teaching methods dominated. However, today’s computing hardware and software have made the production and distribution of instructional animations relatively quick, easy and cheap, even for non-specialists (Lowe, 2017). As devices such as portable computers, tablets and smart phones have gained acceptance as part of the everyday educational landscape, explanatory animations have become increasingly ubiquitous across a range of teaching areas. A key educational advantage posited for animations over more traditional forms of information presentation is their capacity to provide direct, veridical presentation of the subject matter’s spatio-temporal aspects (i.e., processes and procedures). Whereas conventional textbooks rely on representations that are either linear-sequential (i.e., textual explanations) or unchanging over time (i.e., static pictures), animation is a type of representation that can be both visually analogous to the subject matter’s spatial structure and explicit about its dynamics (Lowe & Schnotz, 2008, 2014). The distinctive attributes of animation as means of faithfully representing dynamic content have found particular application in science

R. Lowe (*) Curtin University, Bentley, WA, Australia e-mail: [email protected] J.-M. Boucheix University of Burgundy, Dijon, France © Springer Nature Switzerland AG 2020 L. Unsworth (ed.), Learning from Animations in Science Education, Innovations in Science Education and Technology 25, https://doi.org/10.1007/978-3-030-56047-8_13

301

302

R. Lowe and J.-M. Boucheix

teaching where the understanding of processes and competent performance of procedures are generally regarded as important educational outcomes. Although educators have generally been keen to incorporate explanatory animations into their teaching programs, there has been little accompanying consideration of the assessment approaches they use to measure the learning that results. Typically, learning from animation is measured using the same established forms of assessment that have long prevailed in traditional classroom contexts. Verbal information plays a central role in these assessment approaches, a situation that reflects the historical dominance of written and spoken text as vehicles for instruction. Well known examples include test items presented to the student in the form of written text (with perhaps occasional accompanying pictorial material) that are to be responded to by the student’s production of a written answer, or multiple choice tests requiring selection of an answer from a set of text-based alternatives. When well designed and appropriately used, these approaches are widely considered to provide valid measures of learning achievement. However, are they equally valid for assessing learning that results from studying animation rather than more conventional resources? In former times when instruction was largely based on conventional resources such as textbooks and teachers’ verbal explanations, the utility of such assessments went essentially unquestioned. This ready acceptance was likely because of the close correspondence between the types of representations involved in teaching and testing. For example, if learners studied a textbook account of how an electric motor works, such an account would likely consist of a subject matter expert’s carefully constructed written explanation of the processes involved in the motor’s operation accompanied by a static diagram or two. In this situation, common approaches to assessment could involve the learner generating a written explanation of how such a motor works (plus a suitable sketch) or selecting the best answers to a set of multiple choice questions on the motor’s operation. In both cases, assessments would be measures of the extent to which the learner’s responses were consistent with the explanations given in the original learning resources. However in the present era, conventional textbooks are no longer as central to teaching and learning as they were in the past. Consider an hypothetical situation where the main resource provided for learning about how an electric motor works is an animated explanation rather than a textbook account. In order to highlight some notable differences between the textbook and animation cases, we will assume that the animation in question is not accompanied by a text but relies instead on a clear pictorial explanation of the operational processes. With such an animation, the learner has no ready-made expert verbal characterization of the motor’s various sub-processes that together are responsible for its operation. Rather, it is up to the learner to characterize these processes. This raises the issue of the types of information that are provided by an animation. For the purposes of this chapter, we define an animation as a constructed dynamic depiction of a nominated referent system (i.e., the targeted subject matter). Animations are therefore distinguished from video recordings that, although also dynamic representations, are captured rather than constructed. There are two basic aspects of the depicted subject matter that must be addressed when assessing learning from an

13

Towards more Valid Assessment of Learning from Animations

303

Fig. 13.1 Frame from an animation depicting the operation of an electric motor (left) and labelled diagram showing the names of the motor’s components (right)

animation: (i) visuospatial (the appearance and arrangement of the system’s constituent entities and subsystems), and (ii) spatiotemporal (how those visuospatial aspects change over time). If, as a result of studying the animation, learners are able to internalize this visuospatial and spatiotemporal information into their mental representations of the to-be-learned subject matter, they have a foundation for understanding the depicted content. However, this information on its own is not sufficient. There is another crucial requirement for comprehending how and why the system portrayed in the animation actually works. A learner’s internal representation of the subject matter must also include knowledge of the functional relationships that exist amongst these foundational visuospatial and spatiotemporal aspects. In particular, the attribution of causality (what causes an observed effect) is central to an in-depth understanding of a dynamic system. Proper assessment of animationbased learning should therefore also be able to tap a learner’s knowledge of the relationships that account for the dynamics exhibited by the system. To illustrate these assessment requirements, we again consider an animation that depicts the operation of an electric motor. A single frame from the animation is shown on the left of Fig. 13.1 while the names of the motor’s components are given on the right. This system is composed of various components (field magnets, coil, armature, shaft, commutator, and brushes) that work together to make the shaft of the motor rotate when a battery is connected. The animation uses an array of simple graphic entities to portray the real-life components, their configuration, and their behaviour. Now let us move from the animation’s external representation of the motor to the internal representation that needs to be developed in the mind of the learner in order to understand this system. An adequate mental representation of the motor’s visuospatial aspects should include not only internal tokens that stand for each of these components (including their respective shapes, sizes, constitution, etc.), but also knowledge of how they are spatially configured (i.e., the motor’s structure). For example, the two field magnets have a curved shape, are big enough to fit around the other components, and are separated by a gap within which are enclosed the coil, armature and shaft with a north pole on one side and a south pole on the other. An adequate mental representation of the motor’s spatiotemporal aspects would need to include knowledge that the field magnets and brushes are

304

R. Lowe and J.-M. Boucheix

static components but that the other components (coil, armature, commutator, and shaft) are dynamic in that they jointly rotate around the motor’s central axis. However, possession of these visuospatial and spatiotemporal knowledge sets alone would not be sufficient for a learner to comprehend how and why the motor functions. An understanding of the motor’s functionality requires additional knowledge about the various relationships involved between its components and how they explain why connecting the battery causes the observed dynamic effect. For example, learners need to know that when a battery is connected to the brushes, it causes electricity to flow through the commutator to the coils that are wrapped around the magnetisable armature. This in turn causes the central parts of the motor (rotor) to become temporarily magnetized which sets up a situation where attractive and repulsive forces result from magnetic interaction between the rotor and field magnets. As the shaft turns due these forces, the attached commutator switches the electricity to change the magnetic polarity of different coil-armature sections. The net effect is that the alternating magnetic forces cause the shaft to rotate continuously. If learners who have studied such an animated explanation are faced with a conventional verbal form of assessment as mentioned above, they may be disadvantaged in two important ways. First, the written text used to present the assessment questions will not have been ‘pre-figured’ by a text-based expert encapsulation of the processes involved in the motor’s operation (as would have been the case had the learner studied a textbook account). The learner would therefore not be able to leverage the textbook author’s coherent, refined account of the essentials involved in these processes as the basis for comprehending the assessment question. As a result, it may be that the learner’s capacity to answer the question effectively is compromised simply because the question’s characterization of the motor’s operation is not sufficiently understood. Second, even if the learner does understand the question, it may be very difficult for the learner to construct a satisfactory verbal account of the motor’s operation essentially ‘from scratch’. It is perfectly possible for a learner actually to comprehend the motor’s operational principles in a non-verbal way (in terms of the visual, spatial and temporal relationships involved) without having formed an internal verbal representation of this information. Being able to devise an adequate verbal characterization of the dynamics of a complex and interdependent set of processes (such as are responsible for those in an electric motor’s operation) can be an extremely challenging exercise, even for a domain expert. How realistic then is it to expect a novice to cope with the sophisticated challenge of converting an understanding represented internally in terms of visual, dynamic information into an external representation that is verbal and static? The differences between these two types of representation are likely to compromise the validity of assessments that effectively ignore their distinctive characteristics.

13

Towards more Valid Assessment of Learning from Animations

13.2

305

Depictive Versus Descriptive Representations

The distinction between depictive and descriptive representations discussed by Schnotz (2001) helps to explain why conventional verbally-oriented assessment approaches are likely to be sub-optimal for measuring the learning that results from explanatory animations. Depictive representations (such as animations) are those that portray their referent subject matter in an analogical manner. In other words, there is a substantial degree of referent-representation resemblance with respect to appearance and structure. This means that it is relatively straightforward to make a direct point-to-point mapping between the referent and its representation. In the case of animations, the directness with which such mapping can be performed applies not only to the visuospatial aspects of the subject matter, but also to its spatio-temporal aspects. Indeed, most educational animations actually privilege the subject matter’s dynamics, even when the appearance and structure of the depicted system have been somewhat modified to produce a more diagrammatic rendition. Animations such as the electric motor example discussed above therefore exhibit a high degree of behavioural realism despite their visuospatial aspects being represented in an abstracted manner. Descriptive representations (such as written text) are based on a sign system that is very different from that used for depictive representations. Instead of employing analogical portrayals of the subject matter, descriptive representations bear essentially no visuospatial resemblance to their referents. Rather, they rely on discrete strings of arbitrary symbols (letter groups/words) that are arranged in a directed linear sequence according to various conventional rules of assembly. Whereas depictive representations can use two dimensions for their portrayal of spatial information, this is not possible within the one-dimensional structure of a descriptive representation. Because of this constraint, a text-based explanation of a system such as an electric motor is obliged to rely on a variety of generic terms (words such as ‘curved’ to indicate the shape of a motor’s field magnets, and ‘separated by a gap’ to indicate their spatial arrangement) to represent visual and spatial aspects of the subject matter. However, there is a considerable lack of precision in such terms and so the information they provide is generally far more indeterminate than what is available from a corresponding depictive representation. A similar lack of precision also applies to descriptive representations of spatio-temporal information, such as the dynamics exhibited within an animation. This problem is particularly acute for dynamics of even modest complexity, especially when the changes concerned involve a degree of simultaneity. Because it is structurally impossible for text’s linear sequential structure to capture such simultaneity in a veridical manner, attempts to convey co-occurring events via the written word can be both clumsy and difficult to interpret. An unfortunate consequence of the central role that verbal information has played in teaching and learning through the ages is that too many educators implicitly assume that comprehension is something that necessarily involves text. A

306

R. Lowe and J.-M. Boucheix

repercussion of this assumption is that unless a student can either produce or recognise an adequate verbal account of the subject matter, comprehension is concluded to be lacking – the term ‘comprehension’ is in effect a proxy for ‘text comprehension’. In a world suffused with information technology where non-verbal representations abound, it is perhaps high time that this view was challenged. The many educators who have this limited view of comprehension need to realize that many types of content can be satisfactorily comprehended in a non-verbal way. Exploring new, alternative approaches to assessment that are capable of tapping such understandings offers an opportunity for the educational community to move beyond the current reliance on verbal questions and answers.

13.3

What Is Being Assessed?

The ideas presented in the rest of this chapter originated with insights gained during our research into learning from animation (rather than having arisen from contemplations about classroom practice). We consider it is important to signal this from the outset because we do not claim to offer well-worked out assessment tools that are immediately applicable in classroom contexts. Instead, our goal is to stimulate inquiry about how the range of prevailing verbally-oriented approaches to assessment might usefully be complemented by other somewhat different approaches. Ultimately, this could provide educators with techniques that are better suited to measuring the distinctive learnings that can come from the study of explanatory animations. Rather than searching for more valid ways to assess learning from animation in classroom contexts, our purpose as researchers has been to obtain better evidence about what learners are doing while studying animations. This process orientation was motivated by empirical research that found animations were not in fact the educational panacea many believed them to be. Contrary to prevailing orthodoxies about the intrinsic benefits of animation for learning, those findings showed that animations could pose substantial challenges to students, especially if they portrayed subject matter that was complex and unfamiliar. A substantial proportion of the experimental work that underpins this chapter investigated learning from an animation of a traditional piano mechanism (chosen because of its complexity and lack of familiarity to most people). In the early stages of those investigations, we adopted the same types of verbally-oriented learning measures that were then the standard approach used across the field of multimedia learning research (e.g. Mayer & Anderson, 1992; Mayer & Moreno, 2002). Most experiments in that field were concerned with comparing the effectiveness of different media combinations or presentation conditions in terms of learning outcomes. However, because our research soon became more process-oriented, we were interested not so much in the products of various alternative educational interventions but in what perceptual and cognitive activities learners were engaging in when they studied an animation. Ultimately, we realized that verbally-oriented approaches were limited in what they

13

Towards more Valid Assessment of Learning from Animations

307

could reveal about such processing, and perhaps even flawed in some respects. They proved to be particularly lacking in what they could reveal about where learners’ animation processing was deficient or inappropriate. This prompted us to devise what we hoped would be more revealing ways of probing the processing that was taking place during learning from animation. Most of the research referred to in this chapter has been carried out within the framework of our theoretical perspective on the process of learning from animation and the mental outcomes of such learning as encapsulated in the Animation Processing Model (APM) (Lowe & Boucheix, 2008, 2012; Lowe, Boucheix, & Menant, 2018). To appreciate why we were motivated to devise alternative non-verbal techniques of assessment, it is necessary to be aware of the types of processing activity that our investigations targeted. We posit that learning from animation involves cumulative, constructive processing that encompasses both the extraction of information from the external stimulus (the animation) and progressive mental assembly of that internalized information into hierarchical knowledge structures. Both bottom-up (data-driven) and top-down (knowledge-driven) processing play a role in such extraction, internalization and knowledge construction (Kriz & Hegarty, 2007). The Animation Processing Model characterizes learning from animation in terms of five interrelated phases. These phases can be broadly divided as follows: • Decomposition of the external animation into event units (entities plus their associated dynamics) that can be readily extracted and internalized by the learner (APM Phase 1), and • Composition of the extracted event units via relation formation into higher order knowledge structures that mentally represent the depicted subject matter (APM Phases 2 to 5). In terms of the APM, the ideal outcome of studying an animation is a high quality mental model (Johnson-Laird, 1983) of the target content (i.e., one that is appropriate, accurate, and sufficiently comprehensive). However, such an ideal can be achieved only if the learner deals successfully with all five APM processing phases. Our research demonstrates that for more challenging animated subject matter, the learning outcomes can be far from ideal. Rather than composing a coherent and well elaborated mental model, learners too often develop fragmentary, partial understandings that contain serious functional errors regarding the subject matter’s dynamics. In order to track down the reasons for these deficiencies, it was necessary to examine in detail the distinctive types of processing activities that were posited for each of the APM’s five phases. A succession of empirical investigations indicated that errors arose from processing failures in either decomposition of the externally presented animation or composition of the internalized information extracted from the animation. These problems were attributed in large part to a mismatch between (i) the design features of conventional animations and (ii) the characteristics of human perceptual and cognitive processing. Subsequent experiments found that a novel animation design approach in which this mismatch was substantially reduced

308

R. Lowe and J.-M. Boucheix

Fig. 13.2 Piano mechanism showing the damper in contact with string and the hammer poised ready for launching towards string in order to sound the note once the piano key is pressed

produced significantly better learning outcomes than conventionally designed animations (Lowe & Boucheix, 2016). One of the most serious deficiencies encountered in the verbal data we collected was its lack of specificity regarding how learners processed dynamic aspects of the subject matter, especially dynamic relationships. To clarify and elaborate on the nature of these deficiencies, we will consider the animation of a traditional piano mechanism’s internal workings that we used in a number of our experiments (Fig. 13.2). The animation presented a somewhat simplified two-dimensional depiction of how these hidden mechanics function when a piano key is pressed to produce a musical note. In essence, the mechanism consists of two causal chains of interacting levers whose coordinated movements operate (i) the hammer which strikes the piano string to set it into vibration, and (ii) the damper which stops this vibration as required. Because correct functioning of the mechanism requires a number of events to occur simultaneously or in a rapid cascade, the processing demands imposed on the viewer are considerable. With a varied set of spatially distributed components moving at the same time or in quick succession, there is competition for the viewer’s limited resources with respect to visual attention. The concept of event units is at the core of the APM. Rather than characterizing an animation primarily in terms of its constituent graphic entities (as has been the case in other research), the APM emphasises the importance of learners also extracting and internalizing the dynamics associated with such entities. Early in our investigations of learning from the piano mechanism animation, we devised a checklist for assessing participants’ written or spoken explanations (protocols) of how the mechanism operated. Although we continued to use the checklist, it became clear that crucial data about learner processing of dynamics were largely absent from

13

Towards more Valid Assessment of Learning from Animations

309

the protocols. It was therefore difficult to assess the extent to which information about event units was being extracted from the animation, internalized, and interrelated. This severely limited our opportunities for investigating the various decomposition and composition processes posited by the APM or for characterizing possible areas of processing deficiency. We hypothesised that verbal representation has systemic features that make it problematic as a means of clearly and parsimoniously expressing certain aspects of to-be-learned content. Although these features are well-suited to conventional narrative structures in which a series of events unfolds across time, they can work against clear, parsimonious expression when the target content involves substantial simultaneity. In light of the limitations of verbal measures as a means of illuminating learners’ animation processing activities, a range of alternative approaches that we have explored will now be discussed. This discussion is based on the assertion that greater consistency between representational characteristics of learning materials and assessment materials should be able to provide a better indication of actual learning outcomes. We consider that the principles involved in the more successful of these approaches could be adapted to the classroom situation to improve assessment of what students actually learn from educational animations. These approaches particularly targeted how learners processed animation’s dynamic aspects with the aim of addressing the deficiencies in dynamic data obtained from verbal measures alone. Of the various techniques we have tried, some have proven more useful than others. However, before examining these techniques, we need to be more specific about what is meant by the term ‘dynamics’. In a general sense, it refers to change over time but such change can be manifested in various ways. To properly assess learners’ understanding of the target subject matter’s dynamics, it is important to take account of the three main types of dynamic change that may be present in an animation (Lowe, 2004): • Translation whereby a graphic entity moves from one place to another, • Transformation whereby a graphic entity changes shape or size during the course of the animation, • Transition whereby a graphic entity enters or leaves the picture frame. These different types of behaviour can take place separately within an animation or in combination. Assessment of learning from animations needs to be capable of addressing these various possibilities. In the piano mechanism (in common with many such physical systems), there are no transformations involved (assuming we ignore the string’s displacement). Although the mechanism’s constituent components rotate on their respective axes, the shape and size of those components remains unchanged. In one sense, this makes the interactions between components relatively straightforward. In contrast, other types of system (biological, meteorological, etc.) exhibit transformational dynamics, often simultaneously with translations. This can make the task of characterising the spatio-temporal changes present in such subject matter particularly demanding and something that is well beyond the capacity of verbal explanation to capture adequately.

310

13.4

R. Lowe and J.-M. Boucheix

Probing Dynamics Directly

To begin with, we consider one of the more straightforward approaches devised to address the limitations of verbal protocols with respect to dynamic information. It was intended to probe in a highly targeted but non-verbal manner, what a participant had learned from the piano mechanism animation about the dynamics of the mechanism’s constituent entities. Each of the entities under consideration was displayed on a computer screen in isolation but with a cross marked on the region of interest with respect to the dynamics. Clicking the computer mouse on the cross allowed the participant to drag the cross in order to indicate the movement of that part of the entity. The computer automatically recorded information about the direction, angle and extent of the cross’s movement. Results from this cross task were compared with the corresponding values for the entity that were present in the actual animation. This approach provided more and higher quality information about dynamics than could be obtained from the verbal protocols. Although the cross task provides better information about the dynamics of individual entities, it does not address how the movement of two or more entities are coordinated with one another. This type of information is most important with respect to the latter phases of the APM which are concerned with how event units are combined during compositional processing in order to generate higher level relational structures. In the next example, the focus shifts from the local scale of dynamics associated with individual event units to the far broader scale of the concerted movements of all the piano mechanism’s components working together.

13.5

Manipulation of a Physical Model

Another assessment tool we devised for our experiments is a two-dimensional model of a piano mechanism that is closely based on the animated depiction (Fig. 13.3). This replica is constructed from plastic shapes (representing the piano’s operational components) that are pivoted so that they can be manipulated to demonstrate the movements presented in the animation. It is important to note that the replica was designed not to be ‘self-operating’ in the sense that moving the piano key down on the replica did not result in concomitant actions automatically occurring through the remaining components of the mechanism. Rather, successive components in the causal chains were arranged so that in order to make them move, the learner had to physically operate (manipulate) each component. Our intention was that after studying the piano animation, experimental participants’ demonstrations with the replica mechanism would reveal their level of understanding of the piano’s functional dynamics (Lowe & Boucheix, 2010). Although this assessment tool had the potential to provide valuable insights about how the relationships amongst the mechanism’s components contributed to the piano’s overall functionality, there were practical challenges to be overcome in the data collection process (c.f. Lowe, Boucheix, &

13

Towards more Valid Assessment of Learning from Animations

311

Fig. 13.3 Replica piano mechanism that allows understanding of the piano’s functionality to be assessed via movements of the components performed during manipulation of the replica

Fillisch, 2017). A core challenge was posed by the fact that proper operation of the piano mechanism involves a high degree of simultaneity in which all the components move at the same time. Given that this manipulable model was made up of six main components, it was therefore essentially impossible for someone operating the model with just two hands to perform all the movements as they actually occurred in real time. As expected, pilot testing confirmed that participants could accurately execute movements of only two (or rarely three) components at a time. In other words, the normal limitations on human manipulative capacity prevented them from performing a veridical demonstration of how the full set of piano components moved in a concerted fashion. We addressed this deficiency by training students to accompany their largely pairwise demonstrations with verbal elaborations specifying that these partial demonstrations in fact occurred simultaneously. However, it is important to note that the role of these verbalizations was essentially ancillary to the core assessment process in which the participant’s movement of replica components was the primary evidence for what had been learnt from the animation. Video recordings of the participants’ demonstrations were therefore scored largely on how faithfully they reproduced the component behaviours present in the animation. Thus, the verbalizations produced during the assessment task functioned as a minor complementary clarification for the demonstration rather than (as in more traditional assessment approaches) the main source of information about what had been learnt.

312

13.6

R. Lowe and J.-M. Boucheix

Sequencing Animation Frames

In common with the complex dynamics present many biological systems, kangaroo locomotion (hopping) relies on coordinated transformations of various parts of the animal’s body to translate the kangaroo through space, both vertically and horizontally. Developing an understanding of how such locomotion occurs requires the learner to correctly characterise and properly integrate these different aspects. An experiment by Lowe, Schnotz, and Rasch (2011) compared the effectiveness of three different ways of presenting information about kangaroo locomotion. Participants studied a computer-based display of either a dynamic representation of the content or one of two static versions (Fig. 13.4). The three stimulus conditions were as follows: (i) a standard looped eight-frame animation of kangaroo locomotion (dynamic condition), (ii) the sequence of eight frames as contained in the animation but presented one at a time (successive), (iii) the same sequence but presented all at once (simultaneous). Exposures were controlled so that all participants viewed the presented information for the same time period. To determine the relative effectiveness of these different presentation formats, participants completed a post-test in which all eight kangaroo images that had been used in the stimulus materials were presented on the computer screen, but in random order. Their task was to use the computer mouse to rearrange these images into a linear ordering that correctly showed their sequence of occurrence during a kangaroo hop. In order to produce a correct sequence, several attributes of the hopping action needed to be considered together (e.g., elevation of the kangaroo’s body, configuration of the body as a whole, orientation of specific body parts, etc.). If participants took account of a subset only of these attributes (such as elevation alone), an incorrect sequence was likely to result. This is because configuration and orientation during the ascending part of a hop are different from those during the kangaroo’s descent. Analysis of the sequences produced by participants in the different presentation conditions indicated that the animated version was least effective for learning about the kangaroo hop. This counter-intuitive result suggests that animation’s veridical depiction of subject dynamics is not unequivocally beneficial. In this

Fig. 13.4 Frames from kangaroo animation. Correct sequencing of the animation’s eight frames requires both translations and transformations of the kangaroo’s body to be properly considered. Note differences in elevation (translation) and shape of rear leg (transformation)

13

Towards more Valid Assessment of Learning from Animations

313

case, the transitory quality of the presentation that is part of its valued faithfulness to the dynamics has a downside. Because each frame is exposed for only a fraction of a second, the viewer has insufficient time to conduct the type of detailed analysis that is necessary for picking up differences in how the kangaroo body transforms in the ascending versus the descending part of a hop. The results of this experiment highlighted the importance of taking account of different types of dynamics in assessing learning from animation (not only translations, but also transformations).

13.7

Learner-Generated Drawings

It is not uncommon for traditional assessment approaches to rely partly on learners sketching aspects of the subject matter they have studied. Typically, these sketches are based on existing static depictions, such as textbook diagrams or teachergenerated drawings. With respect to animations, learners can also be asked to generate sketches that capture the subject matter presented in these dynamic depictions. Sketching has the potential to provide a means of assessment that is more representationally consistent with the nature of the original learning stimulus than would be the case for a text-based response. For this approach to be effective, the sketches produced would need to contain adequate information not only about the form of the entities involved and their configuration, but also about how these change over time. The quality of the drawing generated by a learner is a crucial determinant of its suitability for assessment purposes. This aspect of quality needs to be carefully defined in the context of assessment. For example, it is not a simple matter of how realistically the drawing depicts the subject matter that is at issue here (i.e., the result of artistic flair). Rather, the criteria for judging learner-generated drawings should be based on what it indicates about the quality of the mental model that the learner has constructed during study of the animation (in particular, its appropriateness, accuracy and comprehensiveness). Using learner-generated drawings to assess learning from animation is likely to be an attractive option for teachers because it is a low tech approach that is easy to implement in the classroom. However, as noted elsewhere (Lowe & Mason, 2017), learner generation of drawings is not without its potential problems. In particular, the subject matter needs to be relatively easy to draw so that learners with modest drawing skills are not disadvantaged. Learner generated drawings were used as part of an experiment designed to improve learning from animation by supplementing viewing with an ancillary task intended to increase depth of processing (Mason, Lowe, & Tornatora, 2013). The animation used in this study portrayed the real-life behaviour of a Newton’s Cradle device (rather than a frictionless idealization). The Newton’s Cradle depicted in the animation consisted of five balls, each suspended by its own string from a horizontal bar. The animation showed what happened from the time a displaced end ball was released until everything came to a standstill. Across this time course, the device exhibited a series

314

R. Lowe and J.-M. Boucheix

Fig. 13.5 Frames from Newton’s Cradle stimulus animation showing configurations that occur early, mid-way and late in the animation’s time course. (Reflecting progressive energy degradation)

of dynamic patterns as energy was progressively dispersed from the system to its environment (Fig. 13.5). Participants (12-year olds) were asked to produce six drawings that represented what they had observed in the Newton’s Cradle animation. Minimal artistic skill was required for this drawing task because the device could be depicted quite adequately using just lines and blobs. The most common approach participants used for portraying the Newton’s Cradle dynamics was to produce what was essentially a sequence of temporally-distributed ‘snapshots’ representing different configurations of the suspended balls. The information in these drawings was sufficient to assess the degree to which individual participants understood some very basic aspects of how the Newton’s Cradle dynamics changed over time. However, the considerable interparticipant variability across the sets of drawings produced limited more detailed comparisons. One aspect noticeably lacking from most of the participant drawings was any indication of how individual balls were moving between the six states shown in the snapshots. We suspected that this was because the children lacked the specialized graphic vocabulary necessary to portray these intermediate motions via static depictions. The effective portrayal of dynamic information via a static depiction is one that even professional artists can find challenging and much work has been done over the years to develop tailored techniques and conventions for this specific purpose. These include the use of multiple depictions of the target subject matter that act as snapshots summarizing main stages of the dynamics, multiple images within the same depiction (e.g. ghosting, dotted outlines), and the addition of specialized graphic symbols (e.g. arrows, motion lines) intended to indicate the dynamics associated with particular entities in the depiction. Nevertheless, static depictions of dynamic subject matter are not always clear (despite the artists’ best efforts), especially when many and varied changes need to be represented. If learners are ill-equipped to depict fine-grained dynamic information in the drawings they make from an animation, this severely restricts the effectiveness of this assessment technique. As indicated by the previous example, our efforts to use drawings for assessing the understandings learners develop from animations met with some limited success

13

Towards more Valid Assessment of Learning from Animations

315

Fig. 13.6 (a) Changes shown in the worm animation stimulus versus (b) drawn modifications to provided response item produced by high and low dynamic spatial ability participants

but only under very specific circumstances. In order to probe finer-grained details of how learners understood dynamics, we devised a more tightly constrained sketching situation that required respondents to make a small number of highly targeted modifications to a given drawing (rather than having to generate entire sketches on their own). This approach was used in an experiment in which participants studied a simplified animated depiction of earthworm locomotion (Boucheix, Lowe, Breyer, & Ploetzner, 2015). We were interested in comparing how well high spatial and low spatial ability learners extracted key dynamic information from the animation. Previous pilot investigations that gave participants rather open ended drawing tasks as a way of demonstrating what they had learned from the worm locomotion animation resulted in data so varied as to be essentially impossible to analyse. For this reason, we devised a far more constrained drawing activity in which participants were given most of the image as a starting point and had merely to sketch in a few closely specified aspects. Figure 13.6 shows (a) two frames from the animated learning materials on earthworm locomotion then (b) the drawing output produced by a high and a low dynamic spatial ability participant. The assessment item provided participants with a pre-existing depiction of a series of earthworm segments and their task was to indicate graphically what would happen to the two nominated segments in the next phase of locomotion. The high dynamic spatial ability participant correctly showed that the eighth segment underwent a transformation (becoming shorter and fatter) in that next phase. However, this transformation was not indicated in the low dynamic spatial ability participant’s modifications. This highly supported drawing activity greatly reduced the amount of information that participants had to draw and carefully focussed their responses on very specific

316

R. Lowe and J.-M. Boucheix

targets. In contrast to the far more open type of drawing requirement described earlier for the Newton’s Cradle example, we found that this tightly constrained task provided a way to probe fine-grained details about participants’ understanding of dynamics. Thus far, the examples presented in this chapter all rely to a greater or lesser extent on learner production in response to the assessment task. In some cases the production requirements were relatively modest (such as correctly sequencing a set of eight kangaroo locomotion frames) whereas in other cases, the requirements were more substantial (such as manipulation of a replica piano mechanism to demonstrate the functional dynamics of all its components). These non-verbal production tasks can be compared to their verbal counterparts which also vary in their requirements – from completing a sentence by filling in the missing word, to writing a complete explanatory essay. The final two examples in this chapter rely on learner recognition rather than production. For these assessment tasks, learners were provided with stimulus materials related to the studied subject matter and asked to make judgements about them based on what they had learnt from the animation. These non-verbal recognition-based assessments are somewhat comparable with verbalbased assessments such as multiple choice or true-false items.

13.8

Movement-Type Judgement

In an experiment on learning to identify fish species from their distinctive movement patterns (as opposed to the standard approach of relying on their physical morphology), participants studied animations of fish locomotion then were then tested on their recognition capacity (Boucheix & Lowe, 2017). The motivation was to teach participants to identify fish despite the sub-optimal viewing conditions (turbid water, difficult viewing angle, incomplete view, etc.) that are often encountered by conservation researchers when doing fish species counts from underwater video records. The different training conditions used instructional animations of trout locomotion that showed either full or partial fish bodies. Following the training sessions, participants completed a fish identification task that used specially developed animations that were constructed as follows. Swimming patterns exhibited by two fish with very different types of locomotion (an eel and a trout) were applied to a top view of the same fish body (trout) to produce a trout-shaped fish that swam exactly like an eel and a trout-shaped fish that swam exactly like a trout (Fig. 13.7). In addition to these 100% eel and 100% trout examples, a further six identically troutshaped fish were then developed whose swimming patterns were mathematically blended to contain varying intermediate percentages of eel-like and trout-like locomotion patterns. For the fish identification task, participants were presented with the eight motion variants in random order and asked to nominate whether or not the swimming pattern was that of a trout. We found this assessment approach to be effective in discriminating between learners who had studied two different types of animated fish locomotion training materials (those who studied animations of partial

13

Towards more Valid Assessment of Learning from Animations

317

Fig. 13.7 Frames from animation of a trout-shaped fish (a) swimming like a trout with movement largely confined to the tail, and (b) swimming like an eel with movement involving most of the body

fish bodies performed better on the recognition task than those who studied animations of whole fish bodies). A notable feature of this example is the very high level of consistency between the characteristics of the representation used in the learning materials and the characteristics of the representation used for the assessment. Animations were used in both cases and these were very similar visuospatially and spatio-temporally. This representational consistency across teaching and testing is far greater than was the case for some of the previous examples presented. For example, we saw that although student-generated drawings offer a relatively straightforward way to avoid some of the more fundamental problems with verbally-oriented assessments, they may still not be well suited to capturing learners’ understandings of dynamics. However, the ideal of a close matching between the animated representations used in learning materials and the types of representations used for assessment likely is difficult to achieve in current classroom contexts. In the case of our fish locomotion experiment, we had access to expert animators who were able to tailor the learning materials and assessment tools to our very specific requirements. Classroom teachers are of course in a very different situation. Perhaps one way forward in terms of this representational consistency issue would be for educational resource developers who produce animated learning materials for classroom use to be responsible for also developing matched assessment items.

13.9

Split Animation

In a recent experiment (Boucheix & Lowe, 2016; Lowe & Boucheix, 2017), we investigated learning from an animation of a Scotch Yoke mechanism (a device for interconverting rotary and reciprocal motion). We were particularly interested in the acquisition of understandings about the dynamic relationships that are at the core of

318

R. Lowe and J.-M. Boucheix

Fig. 13.8 The Scotch Yoke is a device for interconverting between rotary motion (wheel) and reciprocal motion (bar). Its two components are linked by constraining the pin within the yoke

Fig. 13.9 Wheel-only and bar-only versions of split animation items. After presentation of each initial stimulus, participant was shown a correct or incorrect animation of the missing component then indicated if this complementary depiction was consistent or inconsistent with the stimulus

this device’s functionality. As shown in Fig. 13.8, the Scotch Yoke consists of two main parts (i) a wheel onto which a pin is mounted, and (ii) a slotted bar into which the pin is fitted. The wheel is mounted on an axle and the bar is directed along a horizontal axis by guides at either end. If the wheel is rotated, the action of the pin within the slot causes the bar to oscillate to and fro within its guides. Conversely, if the bar is moved back and forth, the action of the slot on the pin causes the wheel to rotate. The mechanism can be made to perform a range of different movements according to the extent and direction of rotary or reciprocal actions. Because the wheel and bar components are intimately connected via the pin and slot, their respective movements are always related to each other. A proper understanding of this device and its fundamental relational nature therefore makes it possible to predict the movement of one component (wheel or bar) from the known movement of the other (bar or wheel). One of the ways we measured learning outcomes was to use a variant of conventional true-false items in which the stimulus materials were a set of 24 partial animations of the device in operation. In 12 of these items, the animated sequence began showing just the wheel alone moving, while the other 12 began showing just the moving bar (Fig. 13.9). In each item, this initial stimulus phase was repeated three times so that participants had sufficient exposure to characterize the movement properly. After these three repetitions, a visual mask was presented to prevent the possibility of perceptual retention. Finally, two repetitions showing movements of the component missing from the stimulus phase were presented that were either correct or incorrect. After each item, the participant indicated if the two component movements were consistent or inconsistent (same or different). All the items differed in some respects from the original animation that participants had studied during the

13

Towards more Valid Assessment of Learning from Animations

319

learning phase of the experiment (e.g. for the wheel, partial rather than full rotations; starting points part-way around the rotation; different directions of movement; changes in movement direction, etc.) and varied in their level of difficulty. This meant that simplistic recall of the original animation alone was insufficient to correctly answer the questions – rather, correct answering required a deeper understanding of the relational interaction between the two components. In contrast to conventional text-based assessments, this approach prioritized consistency between the type of representation used for the learning materials and the type used to measure learning outcomes. Both were spatially faithful depictions of the target subject matter and both were intrinsically dynamic in nature. With a simple ‘same’ or ‘different’ response being employed for each item, reliance on text for evidence of understanding was essentially absent. For the more complex items, it would be relatively challenging to generate a comprehensive, accurate, and appropriate text-based account of the movements performed by the ‘missing’ component in each case. However, there was no such challenge in responding satisfactorily to the animated true-false items used in this experiment. We suggest that this generic approach could readily be applied for assessing learning from animation across a variety of scientific disciplines wherever some form of dynamic contingency was involved.

13.10

Conclusion

This chapter raises the possibility that the prevailing verbally-oriented approaches to assessment may need to be reconsidered with respect to learning from animations. Our concerns about the limitations of these existing approaches arose from the inadequacies they exhibited as tools for measuring such learning in experimental studies. These concerns prompted us to devise new measurement tools for assessing learning that did not rely primarily on verbal representations. The various non-verbal techniques surveyed in this chapter exemplify the range of alternative approaches we have investigated in an effort to find more valid and informative ways of assessing animation-based learning. We do not suggest that these techniques are suitable for use in classroom settings as they stand. Rather, our intention is to stimulate practitioners to explore better approaches for assessing learning from animation in the classroom. Conventional verbally-oriented assessment can be ill-suited to determining learner comprehension of the dynamic subject matter depicted in animations because of fundamental inconsistencies between (i) the type of representation studied by the learner and (ii) the representational system used for assessment. We maintain that this mismatch works against obtaining valid measures of learning outcomes and so the scores resulting from such assessments are likely a poor indication of what has in fact been learned. Substituting alternative assessment techniques based on representations that are more consistent with the essential nature of animation could offer teachers better insights into animation-based learning outcomes. These alternatives

320

R. Lowe and J.-M. Boucheix

can avoid the need for learners to reframe their understandings into the linear sequential format that is required for verbally-oriented assessment approaches. The examples presented in this chapter differ in the type and extent of learner input that is required. At one extreme, the assessment can be very open-ended and unsupported, with learners being entirely responsible for generating the required information (as is the case when they are asked to produce drawings that capture multiple events that occur throughout the studied animation). At the other extreme, the assessment tasks can be far more constrained and tightly targeted (as is the case when learners are asked to demonstrate how just one specific component of a system behaves). Between these two extremes, various intermediate assessment approaches are possible that give different levels of support to the learner. These include the provision of ready-made drawings in which the learner has merely to alter specified aspects (so that minimal artistic skill is required) or a physical model of the animation’s subject matter that the learner can manipulate in order to demonstrate an overall understanding of the dynamics involved. There are also assessment techniques that, rather than involving the learner in some type of production task, are instead based on recognition. These can perhaps be considered as analogues of traditional true-false items or multiple choice tests but where the stimuli are provided not in the form of verbal representations, but as animations. Classroom adaptations of the non-verbal assessment approaches we have discussed in this chapter are potentially valuable tools for evaluating the overall learning outcomes that result from studying an animation. However, their potential is not limited to summative evaluation. They could be even more useful in formative evaluation because of the very direct matching that is possible with such approaches between a student’s response to an assessment item and the correct answer (as contained in the original animation). In this role, they could offer a powerful means of helping to shape the course of the learning process by greatly facilitating students’ detection of aspects in the presented animation they may have initially misunderstood. In contrast with verbal assessment techniques, these alternative approaches provide a very straightforward way to compare external and internal representations because no inter-representational conversions are required of the student. The high degree of representational consistency between the external animation and an individual student’s interim internal representation (as embodied in an erroneous response) should help make misunderstandings readily apparent. Once detected, the animation can be interrogated further to clarify the problematic aspects so that the learner’s developing mental representation can be updated with correct information. We acknowledge that although in principle many of the examples presented in this chapter have the potential to be adapted for the classroom, there are practical barriers to such adaptation. While it would be relatively easy to introduce drawingbased assessment approaches into mainstream classrooms, this would not realistic to rely solely on teachers to implement some of the other approaches that are far more technologically demanding and time-consuming. Instead, it is perhaps time that the educational authorities and commercial developers who are the main producers of

13

Towards more Valid Assessment of Learning from Animations

321

educational animations were also challenged to devise appropriate measuring tools that allow the learning that results from their products to be validly assessed.

References Boucheix, J. M., & Lowe, R. K. (2016, July, 11–13). ‘Acting-out’ dynamics: Can physical manipulation foster building of mental models from animation? In J. Désiron, S. Berney, M. Bétrancourt, & H. Tabbers, H. (Eds.), EARLI SIG 2 conference: Comprehension of text and graphics, learning from text and graphics in a world of diversity (pp. 18–21). Geneva: University of Geneva. Boucheix, J. M., & Lowe, R. K. (2017). Generative processing of animated partial depictions fosters fish identification skills: Eye tracking evidence. Le Travail Humain, 80, 367–394. Boucheix, J. -M., Lowe, R. K., Breyer, B., & Ploetzner, R. (2015, August). How self-generated drawing may impact learning from animation. Paper presented at The 16th biennial EARLI conference for research on learning and instruction, Limassol, Cyprus. Johnson-Laird, P. N. (1983). Mental models. Towards a cognitive science of language, inference, and consciousness. New York: Cambridge University Press. Kriz, S., & Hegarty, M. (2007). Top-down and bottom-up influences on learning from animations. International Journal of Human Computer Studies, 65, 911–930. Lowe, R. K. (2004). Interrogation of a dynamic visualization during learning. Learning and Instruction, 14, 257–274. Lowe, R. K. (2017). Designing static and animated diagrams for modern learning materials. In A. Black, P. Luna, O. Lund, & S. Walker (Eds.), Information design research and practice (pp. 361–376). Oxon: Routledge. Lowe, R.K., & Boucheix, J.M. (2017, August). Demonstration as an aid to learning from animation. Paper presented at 17th biennial EARLI conference for research on learning and instruction, Tampere, Finland. Lowe, R. K., & Boucheix, J.-M. (2008). Learning from animated diagrams: How are mental models built? In G. Stapleton, J. Howse, & J. Lee (Eds.), Diagrammatic representation and inference (pp. 266–281). Berlin: Springer. Lowe, R. K., & Boucheix, J.-M. (2010). Manipulatable models for investigating processing of dynamic diagrams. In A. K. Goel, M. Jamnik, & N. H. Narayanan (Eds.), Diagrammatic representation and inference (pp. 319–321). Berlin: Springer. Lowe, R. K., & Boucheix, J.-M. (2012). Dynamic diagrams: A composition alternative. In P. Cox, B. Plimmer, & P. Rogers (Eds.), Diagrammatic representation and inference (pp. 233–240). Berlin: Springer. Lowe, R. K., & Boucheix, J.-M. (2016). Principled animation design improves comprehension of complex dynamics. Learning and Instruction, 45, 72–84. Lowe, R. K., Boucheix, J.-M., & Fillisch, B. (2017). Demonstration tasks for assessment. In R. K. Lowe & R. Ploetzner (Eds.), Learning from dynamic visualization: Innovations in research and application (pp. 177–201). Berlin: Springer. Lowe, R. K., Boucheix, J. M., & Menant, M. (2018). Perceptual processing and the comprehension of relational information in dynamic diagrams. In P. Chapman, G. Stapleton, A. Moktefi, S. Perez-Kriz, & F. Bellucci (Eds.), Diagrammatic representation and inference (pp. 470–483). Berlin: Springer. Lowe, R. K., & Mason, L. (2017). Self-generated drawing: A help or hindrance to learning from animation? In R. K. Lowe & R. Ploetzner (Eds.), Learning from dynamic visualization: Innovations in research and application (pp. 309–331). Berlin: Springer. Lowe, R. K., & Schnotz, W. (Eds.). (2008). Learning with animation: Research implications for design. New York: Cambridge University Press.

322

R. Lowe and J.-M. Boucheix

Lowe, R. K., & Schnotz, W. (2014). Animation principles in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 513–546). New York: Cambridge University Press. Lowe, R. K., Schnotz, W., & Rasch, T. (2011). Aligning affordances of graphics with learning task requirements. Applied Cognitive Psychology, 25, 452–459. Mason, L., Lowe, R. K., & Tornatora, M. C. (2013). Self-generated drawings for supporting comprehension of a complex animation. Contemporary Educational Psychology, 38, 211–224. Mayer, R. E., & Anderson, R. B. (1992). The instructive animation: Helping students build connections between words and pictures in multimedia learning. Journal of Educational Psychology, 84, 444–452. Mayer, R. E., & Moreno, R. (2002). Animation as an aid to multimedia learning. Educational Psychology Review, 14, 87–99. Schnotz, W. (2001). Sign systems, technologies, and the acquisition of knowledge. In J.-F. Rouet, J. J. Levonen, & A. Biardeau (Eds.), Multimedia learning: Cognitive and instructional issues (pp. 9–29). London: Pergamon.

Ric Lowe is an Adjunct Professor at Curtiwen University, Australia, and a Visiting Professor at the University of Burgundy, France. From an original interest in learning from static scientific diagrams, he developed a research focus on the perceptual and cognitive processes involved in comprehension of animations that represent complex technical content. His process-oriented empirical investigations with Jean-Michel Boucheix explore the activities individuals engage in during learning from animation and have resulted in the Animation Processing Model (APM), a first comprehensive theoretical account of such learning. He is co-editor of two seminal edited books on learning from animation. Jean-Michel Boucheix is a Professor in Cognitive Psychology at the University of Burgundy, France. He began his research on the comprehension of technical documents in professional learning. His research interest later became focused on the topic of learning from and with animations and dynamic visualizations (including videos and virtual reality). He investigates the perceptual, attentional and cognitive processing of dynamic information in learning. Together with Professor Richard Lowe (Australia), he uses experimental methods and eye tracking techniques to test their jointly developed Animation Processing Model. He has co-authored a series of scientific articles on animation comprehension and design.