Multimodality across Communicative Settings, Discourse Domains and Genres [1 ed.] 9781443862653, 9781443811071

Citation preview

Multimodality across Communicative Settings, Discourse Domains and Genres

Multimodality across Communicative Settings, Discourse Domains and Genres Edited by

Veronica Bonsignori and Belinda Crawford Camiciottoli

Multimodality across Communicative Settings, Discourse Domains and Genres Edited by Veronica Bonsignori and Belinda Crawford Camiciottoli This book first published 2016 Cambridge Scholars Publishing Lady Stephenson Library, Newcastle upon Tyne, NE6 2PA, UK British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Copyright © 2016 by Veronica Bonsignori, Belinda Crawford Camiciottoli and contributors All rights for this book reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the copyright owner. ISBN (10): 1-4438-1107-6 ISBN (13): 978-1-4438-1107-1

TABLE OF CONTENTS List of Figures............................................................................................ vii List of Tables ............................................................................................ viii Foreword .................................................................................................... ix Theo van Leeuwen Introduction ................................................................................................. 1 Belinda Crawford Camiciottoli and Veronica Bonsignori Part I - Multimodality in Pedagogical Settings Chapter One ............................................................................................... 14 Multimodal Listening Skills: Issues in Assessment and Implementation Mari Carmen Campoy-Cubillo Chapter Two .............................................................................................. 37 Teaching Specialised Vocabulary to Dyslexic Adult Learners: A Proposal for Multimodal Lexical Input Processing Enhancement Gloria Cappelli and Sabrina Noccetti Chapter Three ............................................................................................ 65 A Multimodal Analysis of Interpersonal Features in Academic Lectures: A Case Study Belinda Crawford Camiciottoli Chapter Four .............................................................................................. 93 The Distinctive Multimodal Nature of Pragmatic Competence: Bridging the Gap between Modes Vicente Beltrán-Palanques Chapter Five ............................................................................................ 116 Dialogue Interpreting as a Multimodal Activity in Community Settings Elena Davitti

vi

Table of Contents

Part II - Multimodality in Specialised Discourse Domains and Genres Chapter Six .............................................................................................. 146 Gestures in Motion in TED Talks: Towards Multimodal Literacy Silvia Masi Chapter Seven.......................................................................................... 166 Gender-related Communication in Political Interviews: A Multimodal Analysis in Meaning-Making in the Obama/Clinton Interview of 2013 Silvia Bruti Chapter Eight ........................................................................................... 189 Analysing Political Discourse in Film Language: A Multimodal Approach Veronica Bonsignori Chapter Nine............................................................................................ 212 A Multimodal Analysis of Discourse in the South African Courtroom: The Oscar Pistorius Case Daniele Franceschi Chapter Ten ............................................................................................. 236 How Idiomatic are Disney Animated Films? The Multimodal MeaningMaking of Idioms Gianmarco Vignozzi Contributors ............................................................................................. 265

LIST OF FIGURES 1-1 Mode interaction in a multimodal sequence (1): an educational psychologist giving advice to parents about their child 1-2 Mode interaction in a multimodal sequence: an employer checks employee’s wellbeing or asks an employee about possible problems at work 1-3 Multimodal Listening Micro-skills 1-4 Conversation flow and verbal/non-verbal input interpretation. 1-5 Non-verbal behaviour categories (based on Knapp, 1972) 2-1 Run the RAN: Familiarisation phase 2-2 Run the RAN: Training phase 3-1 Question: What are the problems that I’m referring to? 3-2 Question: What is a regime? 3-3 Comprehension check: Yes? 3-4 Phrasal verb: look up to 3-5 Imperative: Think of that 3-6 Idiomatic expression: pie in the sky 5-1 Mutual gaze between INT and T in PTM 5-2 Seating arrangements of each PTM in the pedagogical dataset 5-3 Seating arrangement in PTM1 5-4 Multimodal details of lines 10-11 of Extract 1 6-1 Example of deictic social pattern 6-2 Example of metaphoric-referential pattern 7-1 Key domain cloud for Clinton 7-2 Key domain cloud for Obama 7-3 Gestures attributing ‘voice’ 7-4 a. and b. Indexical gestures 7-4 c. Iconic gesture 7-5 CIFP key domain cloud 7-6 CIMP key domain cloud 8-1 Multimodal analysis of parallelism from clip 1 – political speech 8-2 Multimodal analysis of Q&R from clip 2 – political interview 8-3 Multimodal analysis of clip 2 – camera angle on audience 8-4 Multimodal analysis of a tricolon from clip 3 – political speech 10-1 The distribution of idioms in the DNF corpus 10-2 Idioms in the corpus 10-3 Idioms with multimodal implications

LIST OF TABLES 3-1 7-1 7-2 7-3 7-4

Distribution of co-occurring verbal and non-verbal features Frequency lists CIFP composition CIMP composition Frequency lists in Clinton’s and Obama’s turns and in the CIFP and CIMP 8-1 Example of the annotation framework created in ELAN from clip 3 8-2 Summary of gestures from clip 1 – political speech 8-3 Summary of gestures from clip 2 – political interview 8-4 Summary of gestures and head movements from clip 3 – political speech 9-1 The five phases of a criminal trial 9-2 Mr Roux’s reformulated objection 9-3 Other segments of Mr Roux’s examination 9-4 Segment of Mr Nel’s cross-examination 10-1 Films in the DNF corpus 10-2 A multimodal transcription of the idiom “cat got your tongue” in Snow White and the Seven Dwarfs 10-3 A multimodal transcription of the idiom “use your head” in Hercules 10-4 A multimodal transcription of the idiom “you, birdbrain” in Robin Hood 10-5 A multimodal transcription of the idiom “save your skins” in 101 Dalmatians 10-6 A multimodal transcription of the idiom “time is up” in Aladdin 10-7 A multimodal transcription of the idiom “hit the bottom” in Aladdin

FOREWORD THEO VAN LEEUWEN So far, many studies in multimodality have focused on semiotic artefacts, or what Sigrid Norris has called “cultural tools” (2012)—magazine advertisements, children’s picture books, textbooks, websites, and so on. But increasingly there is also a focus on interaction, on ‘live’ communicative events, and this volume goes further in that direction than most of the edited volumes on multimodality that have appeared so far, with papers on lectures, interpreting, courtroom interaction, Ted talks and so on. With this increasing interest in live interaction comes an increasing interest in what, traditionally, and perhaps rather logocentrically, has been called ‘non-verbal communication’—i.e., facial expression, gaze, gesture, posture, proxemics and so on. While there is, in the literature on these communicative modes, still a degree of psychological universalism, the papers in this volume begin to show that ‘non-verbal communication’ also has its registers, that it is differently realised in different settings, for instance in lectures and courtrooms, and that it has its own scales of formality. All this is important. In the age of the new orality, with its mixture of the formal and the informal, the public and the private, it is important to understand how informal communication works, in all its dimensions. Traditional linguistic approaches, which, as we now understand, described formal rather than everyday informal language, are not of much help here, as can be seen from many of the contributions to this volume, which highlight the rise of idiom, the importance of pragmatic competence, and, above all, the importance of a focus on the role of non-verbal communicative modes in effective communication in many different contexts, from language learning to winning elections. Equally important is this volume’s focus on the relevance of multimodality for the study of professional practices, in chapters on dyslexia, interpreting, language learning and so on. Multimodality matters. It is not just an add on, a matter of style. It partakes meaningfully in the representational as well as in the interpersonal and textual functions of communication. And understanding how it does so is of practical

x

Foreword

importance in many domains, especially when online practices inevitably reduce the richness of live interaction and therefore require a critical eye on what might be lost, and hence needs to be compensated for. This is not to say that there are no challenges ahead for the multimodal study of live interaction. Moving linguistic and pragmatic approaches such as conversation analysis, pragmatic analysis and corpus linguistics into the age of multimodality, as all the papers in this volume do, is complex, and sometimes still does not get much further than asserting the importance of multimodality, while remaining theoretically and methodologically safely anchored in linguistics. When that is reversed, as for instance in the chapter on interpreting, we can begin to see that ‘non-verbal’ communication, far from being ‘prosodic’ or ‘paralinguistic’, in fact provides the fundamental structure of the interaction, in which language, along with other modes, then finds its place, just as in contemporary writing it is often layout which provides the fundamental structure, in which words, along with images and other graphic images, then find their place. In all this I would like to take the opportunity to stress the as yet insufficiently recognised role of rhythm as the lifeblood of all live interaction, the single element that integrates the multiple modes at play as they unfold in time, that frames the multimodally realised communicative moves which provide the functional structure of the interaction, and that synchronises the bodies of the participants, when they are speaking as well as when they are listening (cf. e.g., Van Leeuwen 2011, 2014). The analysis of rhythm can therefore help bring out how different semiotic modes are orchestrated, including the integration of the dynamics of facial expression, gaze, gesture, posture and proxemics with the dynamics of all the dimensions of voice quality and articulation, in short, with the music of speech. Too often, the literature has treated these semiotic modes in isolation from each other, segmenting the unity of the body into parts while in reality the body works as a whole. This also raises questions for the way multimodal communication is modelled in transcription templates, whether computer-mediated or not. Multimodal theory and method is, as yet, not advanced enough to take these for granted and use them for purposes of convenience. It is important to ask just which modes they select for analysis and which they do not, and how they represent the way these modes integrate. Every new ‘applied’ study of the way multimodality works in specific contexts will add or modify multimodality theory—and also contribute to our understanding of the roles—I use the plural deliberately—language may play in different kinds of multimodal interaction.

Multimodality across Communicative Settings, Discourse Domains and Genres

xi

In all this it will, finally, also be important to keep revisiting the pioneering work of the 1960s on non-verbal communication and its relation to verbal communication, as is indeed done in the introduction to this volume. Although the work of these pioneers is now more than half a century old, the current volume is, in its own way, also a pioneering work, as it determinedly introduces multimodality in a range of linguistic and discourse analytical approaches, and in the study of a range of professional practices. For this the editors and contributors should be congratulated.

References Norris, Sigrid, ed. 2012. Multimodality in Practice. Investigating Theoryin-practice-through-methodology. London: Routledge. Van Leeuwen, Theo. 2011. “Rhythm and multimodal semiosis.” In Semiotic Margins. Meaning in Multimodalities, edited by Shoshana Dreyfus, Susan Hood and Maree Stenglin, 168–176. London: Continuum. —. 2014. “Towards a semiotics of listening.” In Critical Multimodal Studies of Popular Discourse, edited by Emilia Djonov and Sumin Zhao, 250–263. London: Routledge.

INTRODUCTION BELINDA CRAWFORD CAMICIOTTOLI UNIVERSITY OF PISA, ITALY

VERONICA BONSIGNORI1 UNIVERSITY OF PISA, ITALY

This collection of scholarly research originates from a workshop entitled “Multimodal Perspectives on Language Teaching and Research” held in May 2015 at the University of Pisa. The aim of the workshop was to explore multimodality from both analytical and practical perspectives, with contributions grounded in discourse analysis, conversation analysis, pragmatics, and corpus linguistics, as well as applications for teaching and learning. The workshop was sponsored by the Department of Philology, Literature, and Linguistics, and organised by the Corpus Research Unit of the University of Pisa Language Centre, whose recent initiatives include the Pisa Audio-Visual Corpus Project (cf. Crawford Camiciottoli and Bonsignori 2015),2 and a partnership with the Language Center of the University of California at Berkeley to collaborate in the development of the Library of Foreign Language Film Clips (LFLFC), an ongoing project aiming to promote the learning of language and culture through films.

1

Belinda Crawford Camiciottoli wrote the introductory paragraph and the section “Multimodality and Multimodal Studies”, while Veronica Bonsignori wrote the section “Overview of the Volume”. 2 Veronica Bonsignori was the recipient of a three-year research scholarship from the University of Pisa Language Centre (2013-2016) focusing on the use of audiovisual texts to teach English as a second language in multicultural contexts. During the last year she cut and annotated more than 70 clips from American films and TV series of different genres, which are now published on the LFLFC website (cf. http://blcvideoclips.berkeley.edu/).

2

Introduction

Multimodality and Multimodal Studies The notion that other communicative modes beyond speech and writing are crucial components of human interaction is not a new one. As far back as late 19th century, Darwin (1890/1989) studied the patterns of facial expressions in humans. Moving into the 20th century, the social scientist Efron’s (1941/1972) milestone research on gestures used by Jewish and Italian immigrants in New York led to the method of naturalistic observation that still today represents the fundamental approach to analysing gestures and other forms of non-verbal bodily communication. In the 1960s, the anthropologist Hall (1966) proposed the concept of proxemics, or how people use spatial and body positioning to communicate non-verbally, also revealing strong cross-cultural differences. Birdwhistell (1970) developed the notion of kinesics, coining the term kineme to indicate a unit of gesture, similar to the way we use morpheme or lexeme to indicate a unit of language. In the 1980s, an important framework for analysing the contribution of different semiotic modes was proposed by Poyatos (1982), with the Basic Triple Structure that sees communication as a combination of language (verbal), paralanguage (prosodic or nonverbal/vocal), and kinesics (extra-linguistic or non-verbal/non-vocal). Ekman (1980) and McNeill (1992) formulated classification schemes to analyse gestures, including underliners or vague rhythmic hand movements to accent words, deictic gestures to point to some referent, iconic gestures to describe objects or actions, metaphoric gestures that represent ideas or abstract concepts, and emblems that carry meaning on their own and are not necessarily accompanied by a verbal message (e.g., the ‘thumbs up’ gesture). With particular reference to language-oriented research, some scholars of conversation analysis expanded their approach to include other communicative modes beyond speech. Goodwin (1981) studied the role of gaze direction, showing how it contributes to reshaping the meaning of utterances as they unfold during conversation. Focusing on body movements, Kendon (1990) found that they tend to be synchronised with speakers’ utterances, for instance, when speakers shift their posture slightly forward before responding to interlocutors. Similarly, Heath (1984) observed that both gaze and posture shifts interact with speech to display recipiency towards interlocutors. In the late 1990s, multimodal studies began to emerge as a field of scientific enquiry in its own right, starting with Kress and van Leeuwen’s (1996) and Lemke’s (1998) ground-breaking work on the contribution of visual modalities to the construction of meaning. With the concept of

Multimodality across Communicative Settings, Discourse Domains and Genres

3

social semiotic multimodality, Kress and van Leeuwen (1996) showed how people make use of modal resources beyond the verbal in social practices, describing a grammar of visual design that looks at how colour, perspective, and composition encode meanings in the same way as functional grammar in language. Other approaches utilised in multimodal studies include 1) multimodal discourse analysis based on multimodal transcription systems incorporating verbal and non-verbal features that illustrate their meanings and functions, and how they interact (Baldry and Thibault 2006), and 2) multimodal interactional analysis that focuses on situated interaction and how participants express themselves and react to the discourse of others, particularly from the perspective of intercultural communication (Scollon and Scollon 2001). These approaches have some points in common and do not always have well-defined boundaries, and most current multimodal research draws on their fundamental principles to some extent. In recent years, interest in multimodality has been further heightened by the rapid developments in the area of digital technology, which have profoundly changed the way we communicate and interact in our social practices (Hyland 2009). This applies especially to educational settings. As O’Halloran, Tan, and Smith (2016: 256) aptly point out: Changes in higher education, especially in the use of digital technology, have revolutionized traditional academic practices, with an increasing recognition of the need for students and teachers to develop multimodal competencies across a range of communicative platforms.

This trend is closely linked to the rise of multimodal literacy, meaning the ability to construct meanings through “reading, viewing, understanding, responding to and producing and interacting with multimedia and digital texts” (Walsh 2010: 213). It is now beyond question that educators must actively foster the acquisition of multiliteracies amongst learners (Jewitt and Kress 2003), also to successfully leverage their ever-growing inclination to expertly use multisemiotic digital resources outside the classroom (Street, Pahl, and Rowsell 2011). To achieve this goal, it is necessary to implement new pedagogic practices to raise awareness of the role of multimodality in learning from two different perspectives. In the first case, the focus is on effectively exploiting the multimodal aspects of teacher-student interaction in the classroom, as well as the materials utilised for learning, e.g., texts, images, websites, and audiovisual resources. In the second case, the emphasis is on teaching students explicitly how multiple semiotic resources interact to construct meanings. This approach is associated with the New Literacy Studies paradigm,

4

Introduction

which refuted the traditional interpretation of literacy in terms of the capacity to read and write (Gee 1996), and the New London Group, which encouraged the teaching of multimodal discourse by means of new technologies (New London Group 1996). With particular reference to language teaching, the multimodal approach not only helps learners to better understand and produce texts in the target language (O’Halloran, Tan, and Smith 2016), but also enhances their awareness of the target culture as reflected in diverse approaches to non-verbal communication. Numerous studies have demonstrated the benefits of exposing language learners to input that integrates non-verbal modes such as gestures and facial cues (cf. Busá 2010; Sueyoshi and Hardison 2005; Harris 2003, inter alia). This research has highlighted the importance of being able to exploit different semiotic resources in language teaching and learning.

Overview of the Volume The contributions to this volume focus on multimodality in various communicative settings, with special attention to how non-verbal elements reinforce and/or add meaning to verbal expressions. They reflect a variety of methodological approaches that are grounded in both quantitative and qualitative techniques, including multimodal discourse analysis, multimodal transcription, and multimodal annotation software capable of representing the interplay of different semiotic modes, i.e., speech, intonation, direction of gaze, facial expressions, hand/arm gesturing, and spatial positioning of interlocutors. The ten chapters in the volume are structured into two parts. The five chapters in the first part explore issues related to the use of multimodal resources in educational activities and interactions. In the five chapters of the second part, multimodality is investigated as a key component of communication that takes place in different specialized domains (e.g., political discourse, legal discourse, economic discourse) and genres (e.g., fictional genres such as live action and animated films vs. authentic forms of communication, such as political interviews, courtroom trials, and TED Talks). In Chapter 1, “Multimodal listening skills: Issues in assessment and implementation”, Mari Carmen Campoy-Cubillo discusses how different communicative modes may enhance various comprehension issues related to the listening construct. In this sense, multimodal listening skills may be considered a multi-faceted construct composed of a number of micro-skills that entail being able to understand and interpret the inner connection of

Multimodality across Communicative Settings, Discourse Domains and Genres

5

several modes within a communicative unit. Bearing in mind the complex nature of multimodal communication, she addresses the implications for assessing multimodal listening tasks. In this regard, Campoy-Cubillo suggests that different criteria for multimodal listening task design should consider: (1) the purpose and meaningfulness of the selected multimodal input; (2) the difficulty of the assessment questions, considering that the learner may be asked to infer information from a number of co-occurring modes; (3) question types; and (4) task administration. She focuses in particular on the last two points, highlighting their important role in multimodal listening implementation. Finally, she suggests that devising a listening construct that accounts for a variety of modes encountered in real communication may lead to a better understanding of the listening process, where an awareness of available communicative modes may enhance both comprehension and communication on the part of the language learner. This may also imply a change in the way listening is taught in the classroom and the nature of learners’ active listening responses. Chapter 2, “Teaching specialised vocabulary to dyslexic adult learners: A proposal for multimodal lexical input processing enhancement”, by Gloria Cappelli and Sabrina Noccetti, deals with the nature of vocabulary learning and suggests how to enhance lexical input processing in adult students with dyslexia through multimodal and multisensory learning activities. More specifically, it presents a case study comparing the learning outcome of specialised vocabulary instruction of English as a second language in two groups of Italian learners with dyslexia. The experimental group carried out highly multimodal and multisensory activities and also received an adapted training application to develop first-language lexicon that is used with dyslexic children. The control group was instead taught following the guidelines discussed in the literature on foreign language teaching to dyslexic learners (cf. Nijakowska 2010; Kormos and Smith 2012). The results of this preliminary study seem to confirm the beneficial impact of multimodal and multisensory teaching methods for specialised vocabulary acquisition in learners with dyslexia. Also using a case-study approach, Chapter 3, “A multimodal analysis of interpersonal features in academic lectures: A case study”, by Belinda Crawford Camiciottoli, aims to shed light on the interplay between verbal and non-verbal modes during interpersonal episodes in an academic lecture, and how this may work towards enhancing understanding. On the verbal level, the analysis focuses on the lecturer’s use of interactional devices, i.e., comprehension checks, imperatives, idioms, and puns, while on the non-verbal level, the co-occurrence of prosodic stress, gaze

6

Introduction

direction, and hand/arm gesturing with interactional language were investigated. The data consist of the digital video recording and corresponding transcript of a political philosophy lecture available on Yale University’s Open Courses website. The multimodal annotation software ELAN (Wittenburg et al. 2006) was implemented to identify the cooccurrence of verbal and non-verbal elements during interpersonal episodes in which the lecturer engages the audience. With particular reference to gesturing, the study adopts an analytical framework based on description and function (Kendon 2004; Weinberg et al. 2013). The multimodal analysis of interpersonal features in lecturer-audience interaction contributes to a better understanding of how verbal and non-verbal features can work synergistically to reinforce meanings, thus improving comprehension and promoting a learning-friendly classroom atmosphere. In Chapter 4, “The distinctive multimodal nature of pragmatic competence: Bridging the gap between modes”, Vicente BeltránPalanques discusses the nature of pragmatic competence, with particular attention to its multimodal dimension, in contrast to traditional language teaching approaches that have focused mainly on the development of pragmalinguistic competence. Beltrán-Palanques maintains that the development of learners’ communicative competence depends not only on pragmalinguistic competence, but also on other competencies involving different semiotic modes that allow learners to communicate successfully in the target language. However, the communicative competence model has rarely taken into account the multimodal nature of communication, with the exception of some works (Celce-Murcia 2007; Royce 2007). Indeed, it may be argued that a multimodal approach for teaching pragmatic competence in a second/foreign language (SL/FL) is crucial since communication is multimodal by nature, and learners need to become aware of the different modes that coexist in a communicative event. Thus, he proposes a ‘revisitation’ of the communicative competence model from the perspective of multimodal pragmatic competence. Chapter 5, “Dialogue interpreting as a multimodal activity in community settings”, by Elena Davitti, demonstrates how a multimodal approach can contribute to a more in-depth understanding of the interactional dynamics of Dialogue Interpreting (DI), through an analysis of a selection of extracts from authentic interpreter-mediated data in naturally-occurring pedagogical settings, with comparisons to interactions in other professional settings (i.e., medical and legal). In particular, the paper explores various multimodal practices displayed by participants to co-construct meaning, manage dynamic participation frameworks, and monitor and coordinate the interaction. Different practices may lead to changes and reconfigurations of

Multimodality across Communicative Settings, Discourse Domains and Genres

7

the participation framework, distribution of tasks and responsibilities, and ultimately frame the interaction as more or less collaborative. Conclusions highlight (1) how this type of research can truly enrich our understanding of the communicative and interactional dynamics of DI, thus contributing to its conceptualisation as socially situated and embodied interaction; (2) how this type of research can increase the ‘multimodal literacy’ of professional interpreters, thus informing everyday practice; and (3) how this approach and findings can be integrated into interpreter education. The second part of the volume opens with Chapter 6 by Silvia Masi entitled “Gestures in motion in TED talks: Towards multimodal literacy”, serving as a ‘bridge’ between the first part of the volume dedicated to pedagogical settings and the second part that focuses on specialised discourse domains and genres. She starts from the assumption that, as a relatively new genre, TED Talks have become a useful resource in foreign language teaching for the development of listening comprehension skills and the teaching of non-verbal behaviours. This chapter addresses precisely the issue of the use and understanding of gestures in this genre of popularisation that is now increasingly exploited in educational settings. The study presents a qualitative analysis of a selection of examples from three talks on socioeconomic topics, with the aim of exploring and categorising the interplay between verbal signals and arm and hand gestures. In doing so, it sheds light on how different gestures may assist in the comprehension of meanings in an international context, as represented by TED Talks. In the following Chapter 7, “Gender-related communication in political interviews: A multimodal analysis of meaning-making in the Obama/Clinton interview of 2013”, Silvia Bruti investigates the contribution of gestures and body language (including facial expressions) to meaning-making in political interviews, distinguishing between female and male styles. For this purpose, a multimodal approach is used to carefully analyse the joint interview with President Obama and then Secretary of State Hillary Rodham Clinton on the CBS News programme 60 minutes that aired in January 2013. In particular, the analysis makes some distinctions between typical female and male non-verbal behaviours and highlights the association between gestures and body movements with the linguistic elements identified as typical concomitants of gender-related language use. To this end, comparisons were drawn between Clinton’s and Obama’s speech and those of other female and male politicians during interviews by referring to two small self-compiled comparable corpora. Finally, both Clinton’s and Obama’s gestural behaviours in this interview were compared to the gestural styles of other politicians in the same genre, with a view to identifying their peculiar traits.

8

Introduction

Chapter 8, “Analysing political discourse in film language: A multimodal approach”, by Veronica Bonsignori, explores the ways in which non-verbal elements co-occur with certain rhetorical strategies (i.e., parallelism) often employed in political discourse. She utilises films for the analysis as they represent the perfect multimodal product due to their audiovisual nature. In political communication, non-verbal elements, such as gestures, gaze, facial expression, head movements, and body posture, are widely acknowledged as performing important communicative and persuasive functions. Extracts from two political drama films The Ides of March (2011, George Clooney) and The Iron Lady (2011, Phyllida Lloyd) were analysed, first by selecting and creating video clips where relevant communication exchanges were present. These were then investigated with the multimodal annotation software ELAN (Wittenburg et al. 2006) to determine how various semiotic resources work together to construct meaning in political discourse. Particular attention was paid to the interplay of verbal and non-verbal signals. The chapter also offers some reflections on how such video resources can provide useful tools for both research and teaching in ESP contexts focusing on political discourse. Chapter 9, “A multimodal analysis of discourse in the South African Courtroom: The Oscar Pistorius case”, by Daniele Franceschi, provides an investigation of trial language from a multimodal perspective, combining the analysis of the lexical-semantic and socio-pragmatic features of this specific type of spoken legal discourse with the non-verbal elements associated with the speech. The data used for the analysis consist of authentic audio-visual excerpts from the trial of Oscar Pistorius, the South African Paralympic athlete who was on trial for the murder of his girlfriend. The focus here is on both defence and prosecuting lawyers’ questioning techniques and speaking styles during the examination and cross-examination phase of the accused. The ultimate aim is to advance theoretical research in the field of legal discourse by utilising a wider, multi-semiotic approach, while extending this type of analysis to the South African context. This is because courtroom discourse studies have thus far relied almost exclusively on U.S. or U.K. cases. At the same time, the results obtained may be useful for developing new teaching materials that allow ESP learners studying law to be exposed to various sources that integrate verbal with non-verbal signals. Finally, Chapter 10, “How idiomatic are Disney animated films? The multimodal meaning-making of idioms”, by Gianmarco Vignozzi, delves into the role and representation of idiomatic expressions in animated films. The analysis makes use of Disney films, which are amongst the most successful audiovisual products of all time and widely cherished by both

Multimodality across Communicative Settings, Discourse Domains and Genres

9

children and grown-ups. As illustrated by numerous studies (cf. LippiGreen 1997; Chiaro 1998, inter alia), such success is not only the result of complex and winsome plots, but also of multi-layered and well-planned dialogues and images, which are carefully contrived to be entertaining to an audience. Vignozzi implemented a framework developed by Wildfeuer (2003) to perform a multimodal analysis of the idioms occurring in a corpus of five Disney animated films, covering a wide time span and different themes. His analysis shows how and to what extent the codeployment of different semiotic components of a shot is crucial for the effective meaning-making of idioms in animated movies. From different perspectives and using a variety of analytical techniques, the various contributions to this volume have highlighted the increasingly important role of multimodality in communication across different communicative contexts and different genres. It is hoped that the volume will offer new insights about how to exploit multimodal resources to enhance the learning of English for both general and specific purposes.

References Baldry, Anthony and Paul J. Thibault. 2006. Multimodal Transcription and Text Analysis. A Multimedia Tool Kit and Coursebook. London and New York: Equinox. Birdwhistell, Ray L. 1970. Kinesics and Context: Essays on Body Motion Communication. Philadelphia: University of Pennsylvania Press. Busà, Maria Grazia. 2010. “Sounding natural: Improving oral presentation skills.” Language Value 2(1):51–67. Celce-Murcia, Marianne. 2007. “Rethinking the role of communicative competence in language teaching.” In Intercultural Language Use and Language Learning, edited by Eva Alcón and Maria Pilar Safont, 41– 57. Dordrecht: Springer. Chiaro, Delia. 1998. “Talking Disney: Reflections on what children watch.” In British/American Variation in Language, Theory and Methodology, edited by Carol Taylor Torsello, Louaan Haarman and Laura Gavioli, 97–104. Bologna: CLUEB. Crawford Camiciottoli, Belinda, and Veronica Bonsignori. 2015. “The Pisa Audio-visual Corpus Project: A multimodal approach to ESP research and teaching.” ESP Today 3(2):139–159. Darwin, Charles. 1890/1989. The Expression of Emotions in Man and Animals. London: Pickering. Efron, David. 1941/1972. Gesture, Race and Culture. The Hague: Mouton.

10

Introduction

Ekman, Paul. 1980. “Three classes of nonverbal behaviour.” In Aspects of Nonverbal Communication, edited by Walburga von Raffler-Engel, 89–102. Lisse: Swetz and Zeitlinger. Gee, James Paul. 1996. Sociolinguistics and Literacies: Ideology in Discourses, (2nd ed.). London: Taylor & Francis. Goodwin, Charles. 1981. Conversational Organization: Interaction between Speakers and Hearers. New York: Academic Press. Hall, Edward T. 1966. The Hidden Dimension. Garden City, NY: Anchor Books. Harris, Tony. 2003. “Listening with your eyes: The importance of speechrelated gestures in the language classroom.” Foreign Language Annals 36:180–187. Heath, Christian. 1984. “Talk and recipiency: Sequential organization of speech and body movement.” In Structures of Social Action: Studies in Conversation Analysis, edited by J. Maxwell Atkinson and John Heritage, 247–265. Cambridge: CUP. Hyland, Ken. 2009. Academic Discourse: English in a Global Context. London: Continuum. Jewitt, Carey, and Gunther Kress, eds. 2003. Multimodal Literacy. New York: Peter Lang. Kendon, Adam. 1990. Conducting Interaction: Patterns of Behaviour in Focused Encounters. Cambridge: CUP. —. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Kormos, Judit, and Anne Margaret Smith. 2012. Teaching Languages to Students with Specific Learning Differences. Bristol: Multilingual Matters. Kress, Gunter, and Theo van Leeuwen. 1996. Reading Images. The Grammar of Visual Design. London and New York: Routledge. Lemke, Jay L. 1998. “Multiplying meaning: Visual and verbal semiotics in scientific text.” In Reading Science: Critical and Functional Perspectives on Discourses of Science, edited by J.R. Martin and Robert Veel, 87–113. London: Routledge. Lippi-Green, Rosina. 1997. English with an Accent: Language, Ideology and Discrimination in the United States. London/New York: Routledge. McNeill, David. 1992. Hand and Mind: What the Hands Reveal about Thought. Chicago: University of Chicago Press. Nijakowska, Joanna. 2010. Dyslexia in the Foreign Language Classroom. Bristol: Multilingual Matters.

Multimodality across Communicative Settings, Discourse Domains and Genres

11

New London Group. 1996. “A pedagogy of multiliteracies: Designing social futures.” Harvard Educational Review 66(1):60–92. O’Halloran, Kay L., Sabine Tan, and Bradley A. Smith, 2016. “Multimodal approaches to English for Academic Purposes.” In The Routledge Handbook of English for Academic Purposes, edited by Ken Hyland and Philip Shaw, 256–269. London & New York: Routledge. Poyatos, Fernando. 1982. “New perspectives for an integrative research of non-verbal systems.” In Nonverbal Communication Today: Current Research, edited by Mary Ritchie Key, 121–138. Berlin: Walter de Gruyter & Co. Royce, Terry. 2007. “Multimodal communicative competence in second language contexts.” In New Directions in the Analysis of Multimodal Discourse, edited by Terry D. Royce and Wendy L. Bowcher, 361– 390. Mahwah, Jew Jersey: Lawrence Erlbaum Associates, Inc. Scollon, Ron, and Suzanne Wong Scollon. 2001. Intercultural Communication. Malden, MA: Blackwell. Street, Brian, Kate Pahl, and Jennifer Rowsell. 2011. “Multimodality and new literacy studies.” In The Routledge Handbook of Multimodal Analysis, 191–200, edited by Carey Jewitt. London: Routledge. Sueyoshi, Ayano, and Debra Hardison. 2005. “The role of gestures and facial cues in second language listening comprehension.” Language Learning 55:661–699. Walsh, Maureen. 2010. “Multimodal literacy: What does it mean for classroom practice?” Australian Journal of Language and Literacy 33(3):211–223. Weinberg, Aaron, Tim Fukawa-Connelly, and Emilie Wiesner. 2013. “Instructor gestures in proof-based mathematics lectures.” In Proceedings of the 35th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, edited by Mara Martinez and Alison Castro Superfine, 1119. Chicago: The University of Chicago Press. Wildfeuer, Janina. 2013. Film Discourse Interpretation. Towards a New Paradigm for Multimodal Film Analysis. New York: Routledge. Wittenburg, Peter, Hennie Brugman, Albert Russel, Alex Klassmann, and Hans Sloetjes. 2006. “Elan: A professional framework for multimodality research.” In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation, Genoa, Italy, May 24–26. Accessed October 5, 2015. http://www.lrecconf.org/proceedings/lrec2006/pdf/153_pdf.pdf

PART I – MULTIMODALITY IN PEDAGOGICAL SETTINGS

CHAPTER ONE MULTIMODAL LISTENING SKILLS: ISSUES IN ASSESSMENT AND IMPLEMENTATION1 MARI CARMEN CAMPOY-CUBILLO UNIVERSITAT JAUME I, SPAIN

1. Introduction Assessment criteria for listening comprehension tests (Buck 2001; Nation and Newton 2009) have traditionally focused on understanding audio input. However, real instances of communication are multimodal in nature. When we listen to someone, we do so in a specific context in which other communicative modes such as gestures, facial expression, visual context, or interpersonal distance participate in the communicative act, producing a given multimodal message (Jewitt 2009). The development of multimodal listening skills would then imply focusing not only on audio input, but also on all types of non-verbal input. This chapter analyses how different communicative modes may enhance different comprehension issues related to the listening construct. In this sense, in the case of the listening macro-skill, multimodal listening skills may be considered a complex construct composed of a number of micro-skills that entail being able to understand and interpret the inner connection of several modes within a communicative unit. Bearing in mind the complex nature of multimodal communication, implications for assessing multimodal listening tasks have recently been discussed (Campoy-Cubillo and Querol-Julián 2015). In this regard, the different criteria for multimodal listening task design should consider 1

The research conducted in this article is part of the Innovation in Education Research Project: 2954/14 Projecte d’Innovació Educativa: Anàlisi i millora de les estratègies per a escoltar i interpretar textos orals en llengua anglesa sobre la base d'objectius específics, Universitat Jaume I.

Multimodal Listening Skills: Issues in Assessment and Implementation

15

issues such as (1) the purpose and meaningfulness of the selected multimodal input, (2) the difficulty of the assessment questions, taking into consideration that the learner may be asked to infer information from a number of co-occurring modes, (3) question types, and (4) task administration. The last two aspects are particularly relevant. The way questions are posed and how they are sequenced may determine to a large extent the possibility of dealing with non-verbal issues. If a learner is asked to pay attention to non-verbal information and is requested to answer a question at the same time, unless the task is administered in a way that the listening input contains pauses in the right places, it may not achieve its intended purpose. Thus, question type and listening administration procedures will have an important role in multimodal listening implementation.

2. Multimodal Listening Configuration In past decades, communicative interaction has been influenced by the rapid increase in new technologies and the resulting applications and digital resources that create new multimodal communicative configurations (e.g., virtual communities or social networks, vodcasts, blogs, chat rooms, online dictionaries, etc.). With the advent of online and off-campus studies, there has been a notable shift in the way we teach and in the wide variety of (multimodal) resources that we may use when doing so. If we think about how lectures were taught in the past, we can agree on the fact that a good lecturer would use at least two of the main non-verbal communicative features: kinesics (including mostly gestures, head nods, and facial expressions) and paralanguage (above all prosodic features). The lecturer could also provide the students with hand-outs in order to follow the lecture or complement the presentation. In such a situation, we can identify at least six different communicative modes: the utterances produced in this particular speech event, prosodic features, gestures, head nods, facial expressions, and a written text. Some of these modes (in the same way as written and spoken modes) may in turn be divided into different sub-categories. For instance, when examining facial expressions, researchers say that they may fall into different categories which may be indicative of an emotion: anger, happiness, surprise, fear, sadness, or disgust (Ekman 1982). Moreover, these categories may be represented by a combination of different micro-expressions. Surprise, for instance, may be indicated by brow raising and jaw dropping, among others. If we shift a lecture situation to the present times, we can assume that, given an adequate economic situation, a lecturer may also use a computer

16

Chapter One

when delivering a presentation, and thus introduce audio material (different from his/her own voice), images, and other types of contextual visual information. Finally, if we take into consideration off-campus formats, or situations where m-Learning (i.e., using mobile devices for learning on accessible portable platforms) or any type of e-Learning is fostered, there are other communicative modes that may come into action, thus further complicating the multimodal construct, for example by facilitating student-teacher interaction through online discussion boards. What all of these options mean is that the lecture as a multimodal genre may include more modes today than it did a couple of decades ago. It also means that being able to encode and decode this information implies having developed digital skills (Van Dijk 2006), both on the part of the lecturer/teacher (Rangel Baca 2015) and the audience that is expected to interpret the combination of modes provided in a given lecture. In the teaching and learning environment, it was the design of computer-mediated input that made experts in different fields pay attention to the advantages that the new technologies could offer to the educational world. Material that was offered in CD or online format had a higher level of acceptance among learners, particularly because they felt more motivated by the multimodal format (Handley 2008; Laurillard 2013; Meyers, Erickson, and Small 2013; Blake 2016). However, the impact of new technologies in education has also been recognised by teachers, researchers, and language testers (Chappelle 1997, 2016; Blake 2016; Chun 2016; García-Laborda 2007, 2013; Martín-Monje, Elorza, and García-Riaza 2016). This is because the new “multimodal ensembles” (Jewitt 2009: 301) require new interpretations of co-occurring modes that are not present in non-technical communicative modes. Moreover, working with multimodal tools to create teaching materials also entails analysing in which ways multimodal input may affect students’ output, and how that input should be manipulated when creating multimodal testing materials. This has important implications for multimodal elearning formats and in making decisions on what and how to teach when using multimedia resources in an effective way. However, as Blake (2016) rightly points out, although new technologies are widely diffused and used, little use is made of these technologies in order to enhance nonverbal communication awareness and train students in understanding nonverbal features as a part of communication. Blake (2016: 129) states that: Most L2 instructors implement their curriculum with an eye to improving the four skills: speaking, listening reading and writing. Absent in this vision are notions of pragmatic, sociolinguistic, and multicultural competences.

Multimodal Listening Skills: Issues in Assessment and Implementation

17

What is needed is a detailed analysis of non-verbal features and how they interact among each other and with verbal features in a particular situation. This interaction needs to be studied within a specific event or situation because non-verbal communication is mainly ‘situation-based’ or ‘situation-activated’. Pragmatic features of communication, however, have not been fully incorporated into teaching and learning practices as much as they could have been. Multimodal texts are a sound starting point where some or all of these features may be used—when present—as part of language learning instruction. The use of these features requires a sound teaching theory for how to teach them, when to teach them, and also how much is enough for a given lesson or as part of a syllabus. In this regard, an interesting point is made by Blake (2016: 133) when he observes that implications for teaching may be related to pre- and post-activities, and in the case of listening tasks, these provide the opportunity to introduce or frame these activities within an adequate knowledge and context background: Pre-listening activities are a sine qua non in order to frame authentic videos with the necessary cultural background and, in turn, deal with the illocutionary intent of the authors. Pragmatic considerations (something almost never taught in the first or even second year of instruction) also deserve explicit attention when preparing students to listen to authentic videos.

There are three communicative modes—namely gestures, facial expressions and prosodic features—that have gained special attention within the fields of Linguistics and Multimodal Communication. Since humans have always relied on the use of gestures and facial expressions together with prosodic features as primary modes to convey their messages, these modes naturally combine in all instances of oral communication. Thus, relating verbal content with paralinguistic and kinesic features in video samples is a natural way to start analysing how co-occurring modes combine and interrelate in spoken interaction or other oral communicative situations (Campoy-Cubillo and Querol-Julián 2015). In the analysis of multimodal texts, it is essential to focus on mode diversity and how each mode contributes to the message. In this sense, we need to analyse, first, the different components of the multimodal text, and second, their interaction. Concerning this interaction, some modes may be more clearly and directly interconnected than others in a particular situation, that is, the part of the meaning that they bring to a particular

18

Chapterr One

message maay be more relevant r than the part of thhe meaning that t other modes may provide.

Fig. 1-1. Modde interaction in i a multimodal sequence: an educational psychologist giving advicee to parents abouut their child

Fig. 1-1 exemplifies how, when uttering the pphrase And this t is so important fo for your childd, an educatio onal psycholoogist is makin ng use of intonation too signal impoortance of thee message annd is at the saame time supporting hhis/her messaage with a bro ow rise. All of this happeens at the psychologistt’s office, whhich reflects th he context thaat tells us how w relevant this phrase may be for thhe decoders of o the messagge, i.e., parentts. In this particular evvent, we have added the fig gure of the lear arner, who is seen s as an observer of this event in a video. It is expected e that, in order to un nderstand all the moddes in this tassk, the learner has been reeceiving somee kind of training in pprevious classrroom practice. In selecting this piece of video, v the instructor neeeds to make sure that thee chosen situaation is not misleading m and that the different moddes interacting g within it maay be clearly discussed d in the classsroom, thus making it possible to raaise awarenesss of the importance of the diffeerent channelss or modes of communiccation in clarifying orr adding inform mation to a message/event. m

Multimodal Listening Skills: Issues in Assessment and Implementation

19

Fig. 1-2. Mode interaction in a multimodal sequence: an employer checks an employee’s wellbeing or asks an employee about possible problems at work

In Fig. 1-2, a work environment and an employer-employee short interaction are exemplified. As an answer to the employer’s question, the employee shows a tight lip and scratches his arm as part of the response. The verbal answer, Yes, I’m OK, contradicts the non-verbal answer where scratching one’s arm indicates general anxiety—i.e., self-touching belonging to the non-verbal category of MAPs (manipulators, adaptors, pacifiers)—and a tight or flattened lip may show distress, nervousness, anxiety, or a repressed desire to speak. The training, practice, and observed repetition of non-verbal behaviour seem to be important not only because these phases are part of any learning process, but also because, in the case of recognising and interpreting non-verbal communication, they help people to recognise instances of communication accurately and quickly. In a study conducted by Calvo et al. (2014: 563) regarding the facial expression of emotions, it is stated that “frequency accounts for the recognition of differences among expressions”. These authors analyse repeated exposure to gestures in the same environment where observers learn to interpret non-verbal communication more accurately in social encounters due to prolonged or

20

Chapter One

repeated observation of people’s non-verbal messages as regards facial expression. As exemplified in Figs. 1-1 and 1-2, communicative modes are described in relation to how they create meaning within an utterance. But the meaning, connotations and implications of each separate mode have also been analysed by a number of researchers—sometimes in and sometimes out of a communicative context—mostly in Natural Language Processing, where it is the interpretation of a gesture or a facial expression by a potential service or game user that matters most. Different authors try to point out how a particular mode or modes are manifested as part of a multimodal message. For instance, Kendon (2004: 108) talks about gesture units and gesture phrases where these phrases are understood as “units of visible bodily action identified by kinesic features which correspond to meaningful units of action such as pointing, a depiction, a pantomime, or the enactment of a conversational gesture”; they are also conceived by most people as “part of what the person is trying to say” (Kendon 1996: 8). McNeill (2005) describes gesture and “the actual motion of the gesture itself” as “a dimension of meaning” (2005: 98). Prosodic features have been examined by authors like Bäzinger, Patel, and Scherer (2014). These authors have devised an instrument to measure and assess perceived voice and speech features as part of the emotion communication process. In a different study, Bänziger and Scherer (2005) analyse vocal emotion portrayals based on a quantitative corpus prosodic analysis, proposing a coding system for the role of intonation in the expression of emotions. Another interesting study was conducted by Loehr (2004), where the relationship and alignment of gesture, intonation, and rhythm are analysed and it is suggested that gesture and intonation are related in terms of timing, structure, and meaning. A remarkable finding of this research is the identification of seven pragmatic functions of cooccurring gesture and intonation: completeness, incompleteness, information status, focus, emphasis, contrast, and visual status of discourse (on-going or not). Research on the use of facial expressions and the functions of an emotion is connected to the semantics of the utterance as attitude or stance (Ekman 1982; Russell 1994), thus the expression of attitudes and feelings seems to correspond to facial expressions that may be considered to express a particular emotion or attitude as a non-verbal response. Ekman (2007) talks about the “emotional profile” (2007: 233) of a person, the frequency (speed, length in time, and frequency) and manner of our emotional episodes, our different degrees of control over emotion, and our clarity when signalling specific emotions to others.

Multimodal Listening Skills: Issues in Assessment and Implementation

21

One way of coding emotion is through the FACS (Facial Action Coding System, Ekman and Friesen 1978; Hager, Ekman, and Friesen 2002). Action Units are “the smallest visually discriminable facial movements” (Cohn, Ambadar, and Ekman 2007: 203). Cohn, Ambadar, and Ekman (2007) define an event as “a set of action units that overlap in time (i.e., co-occur) and appear to define a perceptually meaningful unit of facial action” (2007: 211). It has also been argued that facial expressions may be determined by society and may be culturally controlled (Kita 2009; Archer 1999), and they may also change in meaning from one context or setting to another. Thus, Russell (1994: 123) clearly exemplifies how the meaning of a combination of gestures and especially facial expressions may yield different interpretations or meanings depending on the context: (…the expresser’s context) is useful in interpreting the facial expression. Thus, whereas the smile in the context of just having received a gift might be interpreted as a sign of pleasure, a smile in the context of just having spilled soup might be interpreted as a sign of embarrassment, and a smile in the context of greeting an adversary might be interpreted as an act of politeness. To refer to a specific facial expression as a signal requires that the expression communicate the hypothesized message not only when the face is seen alone, but when seen embedded in a reasonable range of naturally occurring contexts.

What all these authors have in common is the idea that communicative modes are not independent, but rather co-dependent on other modes, among which contextual information is a relevant one. To summarise, the actual meaning of an utterance is created when all the dimensions of meaning expressed through the different selected modes efficiently interact to form an intended message in a given context. Which are, then, the implications of research on non-verbal communication for language teaching? Since in this chapter we are focusing on listening skills and video input as a multimodal classroom text, we need to arrange and structure the possible modal options that we may come across while preparing a listening task and its subsequent testing. Thus, each mode will be an element to consider in the task design, and it will also be an element to be identified when analysing a potential video that we want to use in a language classroom. In the same way that we need to recognise and interpret modes, we will expect our students to understand these modes. However, students will not be able to do so unless we provide them with adequate information and feedback. Fig. 1-3 summarises the skills that are needed in order to understand and interpret a multimodal listening comprehension task. Each subskill

22

Chapter One

identifies one communicative mode and the ability to comprehend that mode.

Fig. 1-3. Multimodal Listening Micro-skills

Fig. 1-3 represents how in a multimodal listening event the listener should, first, try to understand the different verbal and non-verbal information (i.e., what we have called listening micro-skills); second, interpret that understood information within a listening context; and third, provide a verbal or non-verbal response. Thus, by response we mean any time or instance in which the listener returns communication, and this includes all reactions to a stimulus. Accordingly, multimodal listening skills may be considered a complex construct composed of a number of micro-skills that entail being able to understand and interpret the inner connection of several modes, as well as being able to provide a response to all or some of the information modes.

Multimodal Listening Skills: Issues in Assessment and Implementation

23

3. Criteria for Multimodal Listening Task Design Bearing in mind the complex nature of multimodal listening, the careful design of listening tasks is a crucial issue. Since the listener is receiving all types of stimuli, we need to clarify what we expect them to do with that information, so that they can organise themselves and be able to process the information in a way that will allow them to provide the expected answers. In order to do so, the listeners need to know why they are asked to listen to that event and for what reason. Related to this issue is the importance of providing meaningful listening situations. Language teachers and testers will need to take into account the different levels of difficulty not only of the multimodal format, but also of the task questions. In order to make this possible, there needs to be a clear question typology from which to choose questions, depending on the multimodal density of a determined listening sequence. A final important criterion is to plan for an adequate task administration. This entails being aware of the physical and technological resources and their quality, as well as sequencing the task and/or its different parts in an efficient way. The different criteria for multimodal listening task design should consider the following issues: (1) the purpose and meaningfulness of the selected multimodal input, (2) the difficulty of the assessment questions taking into consideration that the learner may be asked to infer information from a number of co-occurring modes, (3) question types, and (4) task administration.

3.1. Purpose and meaningfulness of multimodal input A number of purposes for listening are identified in the literature on listening skills (Nation and Newton 2009; Weir and Vidakoviü 2013). Among the main listening purposes, listening for understanding the primary information (i.e., general comprehension) is the most frequently demanded skill in the language learning environment, together with the ability to understand or recognise specific detailed information. Different listening tasks may also imply recognising different degrees of complexity in the information that is being processed, and this is also related to the learners’ proficiency level. In some tasks, the purpose is to listen for accuracy, and to recognise sounds and words. There are also tasks that are designed to infer information from the context of the listening situation. When it comes to multimodal listening, we should bear in mind that we need both to listen to and to watch the different types of input in order to interpret non-verbal content in the different input modes.

24

Chapter One

Describing and analysing the purpose of a task that implies an understanding of different communicative modes is perhaps one of the most challenging issues in the field of multimodality and language learning. Teaching and learning multimodality may be seen from a receptive or a productive approach (i.e., teaching and learning how to interpret modes vs. teaching and learning how to produce a coherent and cohesive multimodal message). While receptive multimodal skills (i.e., understanding the different modes can be taught because the multimodal message can be analysed to a greater or lesser extent, using that input to generate an output (i.e., productive multimodal skills) is not only controversial, but also a highly demanding and ambitious task. Non-verbal features may not only be culture-bound, but may also change from one individual to another, that is, people have their own nonverbal idiosyncratic choices (Archer 1997; Kita 2009). Gestures and facial expressions are also spontaneous in everyday communication and, for this reason, trying to teach learners how to use them may seem inadequate in the sense that it would yield an unnatural, forced production. However, although the training of non-verbal behaviour is seen as controversial within the field of language acquisition, the observance and training of non-verbal behaviour is seen as a positive trait of that training in some professional settings. Two clear examples are the business and the medical professions. Caris-Verhallen et al. (1999) examine non-verbal instances in an analysis of nurse-patient non-verbal behaviour in two care settings: home nursing and home care for the elderly. They identify four features that are important to generate a good relationship with the patient: patient-directed eye gaze, smiling, affirmative head nodding, and affective touch. However, the observation of non-verbal expression in medical or health science settings is carried out not only to highlight the characteristics of effective professional communication, but it is also investigated in relation to patients’ possible disorders. In this vein, Philippot, Feldman, and Coats (2003) studied nonverbal communication in clinical settings. The authors paid attention to therapeutic relations and processes and to specific clinical disorders where non-verbal deficits, as well as misuse or misinterpretation of non-verbal behaviour are indicative of a particular disorder. Other professional settings have been examined by Philippot, Feldman, and McGee (1992). In this book, different authors examine the impact of non-verbal behaviour in different social contexts, such as business, law, health, education, and society, or different social relationship types. In a chapter on the importance of non-verbal research in the educational context, Philippot, Feldman, and McGee (1992) discuss how non-verbal

Multimodal Listening Skills: Issues in Assessment and Implementation

25

behaviour skills form part of social competence, since they may aid in regulating the verbal part of the message by modifying, adapting, or stressing the relevant parts of a message. Non-verbal communication may also be in charge of regulating turn-taking by means, for instance, of a change in the direction of gaze or gestures signalling the beginning or the end of a turn. This message regulation and the ability to become skilled in controlling adequate non-verbal expression in an interaction are part of the process of mastering the social rules of encoding and decoding multimodal messages. Consider the following example: I am giving a presentation at a business meeting. I have several colleagues around me, but one of them is the most important person for me because this person will have to evaluate my work and this will affect me professionally. If after presenting my data the person’s facial expression is neutral and the muscles around the jaw are a little bit tense, I will have to interpret that information in order to continue with my presentation. In such a context, interpretation will probably go towards the idea that ‘I am not doing a great job’ or that ‘the person was expecting something else’ or that ‘the person does not agree with me or is not convinced’. There could be other interpretations like assuming that the person is not listening and is thinking about something else, or something else is going through his/her mind, or the person did not rest properly before the meeting and feels uncomfortable. The issue here is not whether or not my interpretation is correct, but that I need to choose an interpretation in order to carry on with my presentation. If I opt for thinking that the person’s mind is somewhere else and, consequently, do not change the course of my presentation, but then again I get tense facial expressions as non-verbal response together with postural shifts expressing nervousness, or biting lips, or shifting the jaw to one side or lip pursing, I will have to readjust my interpretation and act accordingly. What this means is that there is no such thing as an instantaneous correct interpretation in all communication instances, and that listeners need to learn how to interpret multimodal messages, which, as mentioned above, calls for training and familiarisation with the speaker. There are instances, however, where non-verbal behaviour may be clear as in the case of deictic gestures (pointing), some emblems (like thumbs up) or some metaphoric gestures. Listening purpose, then, in a multimodal task, is determined not only by the content of the event (as in most listening comprehension tasks), but also on the interpretation of how the use of non-verbal modes informs and regulates conversation and interaction among people (see Fig. 1-4). This is

26

Chapter One

an essential skill both as part of an individual’s personal and professional development.

Fig. 1-4. Conversation flow and verbal/non-verbal input interpretation

3.2. Question difficulty Knapp (1972) distinguished seven different categories when discussing non-verbal behaviour in communicative situations. These categories are related to the different communicative modes that may be activated in a communicative event. To address the key issue of question difficulty when dealing with non-verbal language learning input, it is essential to start with the non-verbal types of information that may form part of our input. Fig. 1-5 below summarises the main categories and examples.

Multimodal Listening Skills: Issues in Assessment and Implementation

27

Fig. 1-5. Non-verbal behaviour categories (based on Knapp 1972)

In the design of multimodal comprehension questions, we should remember that we are dealing with either videos or other devices that include a number of modes. In the case of videos, we need to bear in mind that we are not only listening to information but also watching it. While watching, we need to observe non-verbal behaviour so as to decide whether any possible non-verbal feature is relevant in that communicative event or is interesting for introduction in the classroom as part of the learning process. Thus, for instance, if we are working with a video that is recorded in an airport, questions related to environmental factors may include paying attention to the physical setting and recognising or learning about the terminals, gate agents, or baggage claim area. If we are watching a video with a conversation between two people, we may need to pay attention to proxemics and physical contact in order to determine and better understand the relationship between the participants involved in the

28

Chapter One

conversation. We may need to pay close attention to paralanguage in a video that shows people interacting in a work environment where voice volume, gaze, or gestures manifest the relationship they have and their attitudes towards a specific issue. Physical characteristics may sometimes determine the relationship among people or how people react to a particular physical trait. Artefacts or adornments may also be indicative of roles in a work environment. Finally, kinesics may be present in all of the above and relates to the way people speak with their body, or the position and activity carried out through body language in people’s every day and context-related behaviour. We may also talk about question complexity in relation to the listening purpose. In this sense, for instance, the purpose of a listening activity may be simply to understand general information in a given situation. This listening instance is the type of comprehension that may be carried out by any individual engaged in a listening experience, where the speaker engages in most of the conversation and the listener has mainly a decoding role. The main purpose in this listening situation is to understand what is being said. However, a listener may also be required to perform some kind of critical thinking and thus analyse, evaluate, or create some form of meaning. In this sense, the individual is not only implicated in the listening process in order to understand, but he or she is also required to understand the social dimension of what is heard and to process the information accordingly in a critical way. In this type of critical listening, we expect the individual to make inferences, to engage in independent and reflective thinking, and to provide a response that is the fruit of judgement and reflection, which could lead to taking some sort of action. While in critical listening both the individual and the social part of the listening event are important, in empathetic listening it is the social part that is the most important one. Empathetic listening may be a complex listening type in the sense that the person is also expected to evaluate the emotions of another person who will demand an immediate response, as in medical professions, or in personal situations where feelings have an important role in the communicative situation. The listener in empathetic listening is also required to be supportive, and for this he or she will need to interpret not only what other people say, but also what they do, their facial expressions, changes in intonation in relation to what they say, the gestures they use when saying something, etc. The listener is then required to demonstrate that he or she is listening (i.e., by providing a supportive response, verbal or nonverbal), and do so in an immediate way while

Multimodal Listening Skills: Issues in Assessment and Implementation

29

processing all the different modes that the speaker is using, which are sometimes in contradiction with each other. In this and the previous sections, I have discussed the different categories of non-verbal behaviour and the purpose of the listening task in relation to the complexity of information processing options. Linked to these two important complexity factors is the density of information being processed and message abundancy (i.e., the number of modes and their saliency). These two issues, information and mode complexity, are related to the amount of information that is being processed and may be analysed in terms of quantity. The closing concern in this section is related to task design. An aspect that may create more complexity within the question typology is: (1) how we ask questions in a task—i.e., question design—and the types of questions thereby produced, and (2) how we sequence those questions, (i.e., question typology and question sequencing). These issues will be dealt with in the following sub-sections: question types and task administration.

3.3. Question types Audio-based listening tasks include questions that are organised around content, general and pragmatic knowledge, syntactic knowledge, lexical and phonological features, and several kinds of paralinguistic features (i.e., voice quality and type, differentiators). The video input brings in additional elements or components. These are the extra-linguistic features: visual context, visual and aural information, kinesics, and proxemics. Deciding what type of question to ask in relation to all these types of information is not an easy task, and a reflection on which is the most important mode according to a particular communicative purpose needs to be the guiding principle in the selection of adequate questions. Moreover, in an ideal situation, we should also try to formulate authentic questions, that is, questions that we would ask ourselves or that we may be asked as a result of involvement in a listening event. In this sense, it is of paramount importance to be aware of the types of settings that serve as multimodal listening input, as well as the role that each participant has in the aforementioned setting. As Rozelle, Drucman, and Baxter (1997: 67) state: Non-verbal communication can be understood best in relation to the settings in which it occurs. Settings are defined both in terms of the varying roles taken by actors within societies and the diverse cultures in which expressions and gestures are learned.

30

Chapter One

Thus, questions related to a setting may be part of the non-verbal question types. Non-verbal input questions may also relate to the functions of nonverbal behaviour. For example, gestures of proximity may show relationships between people and may be part of the information to infer the kind of relationship they hold. Spacing in relation to a client in a business situation may provoke a reaction of disgust if the spacing is too small, and we may question our students about this disgust reaction. Patterson (1983, 1988) suggests different functions for non-verbal behaviour and indicates that these functions are activated if a communicative situation is generated in an interaction between two or more participants. These functions are summarised as follows: (1) Providing information (2) Regulating interaction (development, maintenance, and termination of a communicative event) (3) Expressing intimacy (showing the distance or the degree of union between people) (4) Expressing social control (gaze patterns or vocal intonation to make another person accept one’s point of view) (5) Presentational (how people present themselves in front of others, for instance holding hands) (6) Affect management (demonstrative processes such as shame, embarrassment, or turning away from another person) (7) Facilitating a service or goal (those related to a service or profession such as spacing between seller and client) Finally, we may want students to interpret either very explicit verbal or non-verbal information or to interpret the co-occurrence of both. For instance, interpreting a “thumbs up” gesture in a given setting would be a non-verbal only interpretation question (i.e., where thumbs up and other non-verbal issues such as context may be at play). Interpreting or in some way analysing contradictory speech and facial expressions would be a mode co-occurrence type of question. Another example would be the interpretation of a lecture where the lecturer’s explanations are interpreted in conjunction with lecture notes or slides.

3.4. Task administration There are several issues that need to be taken into account in the task administration process. The first one is to be aware of the multimodal

Multimodal Listening Skills: Issues in Assessment and Implementation

31

format(s) that we use with our students in the classroom, intending whether we use a particular type of video/vodcast input that can be defined by: (1) Its length, i.e., how many minutes it lasts, and whether it is shown in full or sequenced in a number of shorter sequences. If in sequences, what do the “sequence cuts” or selections respond to? For example, we may break a lecture into opening, introduction, body, etc., and explain the purpose of each move so that students associate specific features with each part. (2) Its (multimodal) genre, whether we are dealing with a lab experiment, a lecture, a tutorial, an interaction such as a service encounter or a patient-doctor interaction, etc. Genre awareness is important because each genre is used in a particular context, has a general purpose and audience, has a set of lexical choices, and may use a specific language structure (i.e., we have genre expectations). (3) Its content, which should occupy a place in the syllabus and be adequate for the course/test for which it has been chosen. It also determines other aspects such as lexical choice. It may also determine non-verbal choices (for example, in a sociology video lecture called “Big Data in the Social Sciences” we may encounter statistical information (i.e., statistical images in the visual mode). (4) Recurrent appearances of multimodal co-occurrences. Following the statistics example, in presenting statistical information in a video we may expect the statistical graphics, other images representing statistical information, use of voice/intonation to highlight important data, and the speaker’s facial expression and gestures accompanying the information being presented. This format choice should determine the assessment format. If we use multimodal formats for teaching and assessing listening, as teachers or language testers we need to be aware of these formats and their multimodal features, so that we may have a clear typology not only of the text type we use (e.g., a short video lecture), but also of the question typology that may accompany this listening practice in an effective way. Being effective means that questions are seen as natural by students (resembling real life situations), that they can be implemented in the specific level we are teaching/testing (proficiency level), that these questions are not misleading and we know the kind of answer that students may provide (including open answers), and that the questions adequately and efficiently test the type of knowledge that we want to test (for example, understanding a specific gesture in a given context).

32

Chapter One

The other important issue in task administration for multimodal listening is how the questions are posed: whether there are questions or specific input before listening, whether questions are answered while listening, whether the video/vodcast needs to be stopped in order to answer a specific type of multimodal question that requires a lot of information processing that cannot be retained easily, and whether some questions need to be answered after listening to the whole recording. We believe that there are few questions than can be easily answered simultaneously if we truly want to test more than one mode. In this sense, deferred questions seem to be most adequate, since the student is not processing a variety of modes and trying to provide an answer at the same time. Some teachers say that the video is a distracting tool in a listening task, but this is precisely because these teachers are not aware of the multimodal complexity and the need to postpone the questions and ask them in the right timing. For some genres—particularly if they are long texts—it can be a good idea to stop the video/vodcast for questions after each textual move. Finally, there are questions that may be answered after watching the video/vodcast, since only then will the students be able to process complex types of information, as well as information that may have to be processed and understood only after the whole event has been observed and analysed.

4. Concluding Remarks As we have seen in this chapter, the shift from audio only to multimodal listening is not a simple one. It entails understanding how the modes work and create meaning in a specific event or situation, and then being able to relate this multimodal functioning to language teaching and learning practices. Thus, devising a listening construct that accounts for a variety of modes encountered in real communication may lead to a better understanding of the listening process where awareness of available communicative modes may enhance both comprehension and communication on the part of the language learner. It may also imply a change in the way listening is taught in the classroom (Rost 2011) and the quality of learners’ active listening responses (Weger, Castle, and Emmet 2010). It has been shown how a number of issues have to be taken into consideration in order to define a multimodal listening construct. The first and most important general question is how to teach and test non-verbal issues in language learning environments. Second, we need to consider what kind of teaching material and explanations we need to design in order to present multimodal information to our students. This should allow them

Multimodal Listening Skills: Issues in Assessment and Implementation

33

to have a clear idea of how important non-verbal communication is and also make it possible for them to create some kind of schemata to which they can relate future multimodal tasks or situations in which they may be involved. We also need to make multimodal analysis a current practice so that students can develop their multimodal competence. In order to do so, we need to decompose a multimodal structure into micro-skills that will be related to the understanding of each mode and its significance within a context or environment. Another important aspect is to find a clear purpose for the practice of multimodal listening/watching. This includes not only the purpose that the teacher/tester has in mind, but also how meaningful that purpose is to the student or test-taker. Finally, this chapter has also shown how complexity is a major issue in teaching and learning multimodal listening skills. There are different levels of complexity: text and mode complexity, question complexity, task complexity, and task administration processes. This means that the planning of multimodal lessons and their assessment should be carefully planned. What all the above issues point out is the need to better define the aspects of multimodal communication that we want to teach and why (an explicit purpose) and to devise assessment methods that may adequately link what has been taught with what the students understand about multimodal communication and how they respond to the different input modes in different situations.

References Archer, Dane. 1997. “Unspoken diversity: Cultural differences in gestures.” Qualitative Sociology 20(1):79–105. Bänziger, Tanja, and Klaus R. Scherer. 2005. “The role of intonation in emotional expressions.” Speech Communication 46(3):252–267. Bänziger, Tanja, Sona Patel, and Klaus R. Scherer. 2014. “The role of perceived voice and speech characteristics in vocal emotion communication.” Journal of Nonverbal Behavior 38(1):31–52. Blake, Robert. 2016. “Technology and the four skills.” Language Learning & Technology 20(2):129–142. Accessed May 18, 2016. Buck, Gary. 2001. Assessing Listening. Cambridge: Cambridge University Press. Calvo, Manuel G., Aida Gutiérrez-García, Andrés Fernández-Martín, and Lauri Nummenmaa. 2014. “Recognition of facial expressions of

34

Chapter One

emotion is related to their frequency in everyday life.” Journal of Nonverbal Behavior 38(4):549–567. Campoy-Cubillo, Mari Carmen, and Mercedes Querol-Julián. 2015. “Assessing multimodal listening.” In Multimodal Analysis in Academic Settings: From Research to Teaching, edited by Belinda Crawford Camiciottoli and Inmaculada Fortanet-Gómez, 193–212. London: Routledge. Caris-Verhallen, Wilma MCM., Ada Kerkstra, and Jozien M. Bensing. 1999. “Non-verbal behaviour in nurse-elderly patient communication.” Journal of Advanced Nursing 29(4):808–818. Chapelle, Carol A. 1997. “Call in the year 2000: Still in search of research paradigms?” Language Learning & Technology 1(1):19–43. Chapelle, Carol A., and Erik Voss. 2016. “20 years of technology and language assessment in Language Learning & Technology”. Language Learning & Technology 20(2):116–128. Accessed May, 18, 2016. http://llt.msu.edu/issues/june2016/chapellevoss.pdf. Chun, Dorothy M. 2016. “The role of technology in SLA research.” Language Learning & Technology 20(2):98–115. Cohn, Jeffrey F., Zara Ambadar, and Paul Ekman. 2007. “Observer-based measurement of facial expression with the Facial Action Coding System.” In The Handbook of Emotion Elicitation and Assessment, edited by James A. Coan and John J.B. Allen, 203–221. Oxford: OUP. Ekman, Paul, ed. 1982. Emotion in the Human Face. Cambridge: University of Cambridge Press. Ekman, Paul. 2007. Emotions Revealed. Recognizing Faces and Feelings to Improve Communication and Emotional Life. 2nd edition. New York: OWL Books. Ekman, Paul, and Wallace V. Friesen. 1978. Facial Action Coding System. Consulting Psychologists Press. Palo Alto: California. Hager, Joseph C., Paul Ekman, and Wallace V. Friesen. 2002. Facial Action Coding System. Salt Lake City, UT: A Human Face Garcia Laborda, Jesús. 2007. “From Fulcher to PLEVALEX: Issues in interface design, validity and reliability in internet based language testing.” CALL-EJ Online, 9(1). Accessed February 10, 2016. http://callej.org/journal/9-1/laborda.html García Laborda, Jesús. 2013. “Alternative assessment in English for tourism through Web 2.0.” In Teaching Foreign Languages for Tourism, edited by Gloria Bosch and Torsten Schlak, 89–106. Bern: Peter Lang. Handley, R. 2008. “Using technology to motivate student learning.” In Proceedings of the Emerging Technologies Conference 2008, and

Multimodal Listening Skills: Issues in Assessment and Implementation

35

edited by Ian Olney, Geraldine Lefoe, Jessica Mantei and Anthony Herrington. Wollongong: University of Wollongong. Accessed February 10, 2016. http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1009&context=etc08. Jewitt, Carey. 2009. “Glossary.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 293–308. London: Routledge. Kendon, Adam. 1996. “An agenda for gesture studies”. The Semantic Review of Books 7(3):8–12. Kendon, Adam. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Kita, Sotaro. 2009. “Cross-cultural variation of speech-accompanying gesture: A review.” Language and Cognitive Processes 24(2):145– 167. Knapp, Mark L. “The field of nonverbal communication: An overview.” In On Speech Communication: An Anthology of Contemporary Writings and Messages, edited by Charles J. Steward and Bruce Kendall, 57–71. New York: Holt, Rinehart & Winston. Laurillard, Diana. 2013. Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology. London: Routledge. Loehr, Daniel P. 2004. Gesture and Intonation. PhD diss., Georgetown University. McNeill, David. 2005. Gesture and Thought. Chicago: University of Chicago Press. Meyers, Eric M., Ingrid Erickson, and Ruth V. Small. 2013. “Digital literacy and informal learning environments: An introduction.” Learning, Media and Technology 38(4):355–367. Martín-Monje, Elena, Izaskun Elorza, and Blanca García-Riaza, eds. 2016. Technology-Enhanced Language Learning for Specialized Domains: Practical Applications and Mobility. London: Routledge. Nation, I.S.P., and John Newton. 2009. Teaching ESL/EFL Listening and Speaking. New York: Routledge. Patterson, Miles L. 1983. Nonverbal Behaviour: A Functional Perspective. New York: Springer Verlag. Patterson, Miles L. 1988. “Functions of nonverbal behavior in close relationships.” In Handbook of Personal Relationships: Theory, Research and Interventions, edited by Steve Duck, Dale F. Hay, Stevan E. Hobfoll, William Ickes, and Barbara M. Montgomery, 41– 56. Oxford: John Wiley & Sons.

36

Chapter One

Philippot, Pierre, Robert S. Feldman, and Erik J. Coats, eds. 2003. Nonverbal Behavior in Clinical Settings. Oxford University Press. Philippot, Pierre, Robert S. Feldman, and Gail McGee. 1992. “Nonverbal behavioral skills in an educational context: Typical and atypical populations.” In Applications of Nonverbal Behavioral Theories and Research, edited by Robert S. Feldman, 191–213. New York: Psychology Press. Rangel Baca, Ariadna, 2015. “Competencias docents digitales: Propuesta de un perfil.” (Digital teaching skills: a profile). PIXEL-BIT-Revista de Medios y Educación 46:235–248. Rost, Michael. 2011, Teaching and Researching Listening (2nd ed.). Abingdon, Oxon: Routledge. Rozelle, Richard M., Daniel D. Druckman, and James C. Baxter. 1997. “Non-verbal behaviour as communication.” In The Handbook of Communication Skills, edited by Owen Hargie, 67–102. New York: Routledge. Russell, James A., 1994. “Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies.” Psychological Bulletin 115(1):102–141. Van Dijk, Jan A.G.M. 2006. “Digital divide research, achievements and shortcomings.” Poetics 34(4):221–235. Weger Jr, Harry, Gina R. Castle, and Melissa C. Emmett. 2010. “Active listening in peer interviews: The influence of message paraphrasing on perceptions of listening skill.” The International Journal of Listening 24(1):34–49. Weir, Cyril, and Ivana Vidakoviü. 2013 “The measurement of listening ability 1913-2012.” In Measured Constructs: A History of Cambridge English Examinations, 1913-2012, edited by Cyril J. Weir, Ivana Vidakoviü and Evelina D. Galaczi, Vol. 37, 347–444. Cambridge: Cambridge University Press.

CHAPTER TWO TEACHING SPECIALISED VOCABULARY TO DYSLEXIC ADULT SECOND-LANGUAGE LEARNERS: A PROPOSAL FOR MULTIMODAL LEXICAL INPUT PROCESSING ENHANCEMENT GLORIA CAPPELLI AND SABRINA NOCCETTI1 UNIVERSITY OF PISA, ITALY

1. Introduction Second language (L2) vocabulary development is a widely explored area. Interest is on both retaining and retrieving newly learnt words. For a long time, vocabulary was not considered central in language acquisition. Focus was on the generative component. However, it is now widely accepted that the lexical component of the language system is of paramount importance in the language learning process, and that structure develops from formulaic lexical bundles. Moreover, vocabulary knowledge is linked to reading and inferential skills, as well as to academic achievement (Laufer 1997). This is particularly true of specialised vocabulary. Many adult learners of English learn the language for professional development. Others have to use it in higher education settings as the lingua franca of specialised instruction: they read and attend lectures and conferences in some type of English for specific purposes (ESP) or another. It is therefore reasonable to assume that domain specific vocabulary, with varying degrees of specialisation, plays a central role, not only in L2 acquisition, but also in broadening learners’ knowledge of a certain field (Gablasova 1

The research was carried out jointly by the two authors. Gloria Cappelli wrote sections 2 and 4; and Sabrina Noccetti wrote section 3. Sections 1, 5 and 6 were written together.

38

Chapter Two

2014). In spite of this, specialised vocabulary acquisition is relatively unexplored in second language acquisition studies at large, with very few works focusing mostly on incidental learning contexts, e.g., in CLIL classrooms. Learners with dyslexia have difficulties in developing literacy and literacy-related abilities, including learning and retrieving new words. This means that they have limited access to the advantages offered by incidental vocabulary learning (e.g., through reading). Given the low frequency of most technical terms, dyslexic learners might, in fact, never acquire them. As a consequence, they might never be as proficient as their peers in processing domain specific texts and developing new technical knowledge. It is therefore necessary to investigate the factors that can potentially favour specialised vocabulary acquisition in learners with dyslexia and develop effective inclusive teaching techniques. The positive effects of multimodal resources for vocabulary teaching are indeed uncontroversial. Studies converge on the conclusion that exploiting more than one semiotic mode improves the learning outcome (Barcroft 2015). Extending research on multimodal learning to the development of L2 vocabulary teaching resources for adults with specific learning differences seems therefore the natural next step. This is not a new idea in itself, since other approaches to foreign language teaching have advocated the use of multisensory resources in the past.2 Several studies (Schneider and Crombie 2003; Nijakowska 2010; Kormos and Smith 2012) on dyslexia and foreign language learning have recently attempted to integrate the suggestions of the Orton-Gillingham multisensory structured learning approach (Gillingham and Stillman 1997) into guidelines for effective L2 teaching. This has been done mostly in an intuitive way, and none of these attempts has focused on ESP. This paper attempts to bring together suggestions from a varied body of research on L2 vocabulary acquisition and dyslexia and discusses the particular nature of specialised vocabulary learning. It provides a tentative explanation for the beneficial effects of multimodal vocabulary teaching to dyslexic learners. It suggests ways in which activities can be adapted in a multimodal perspective to maximise input enhancement on the basis of the current models of lexical input processing. Finally, it presents a case study comparing the learning outcome of specialised vocabulary instruction in 2

Multisensoriality and multimodality are sometimes used as synonyms. However, whereas multimodality deals with the semiotic modes used to create meanings, multisensoriality revolves around the different sensory channels through which human beings perceive the world. The two terms involve complementary perspectives.

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

39

two groups of learners with dyslexia, one of which carried out highly multimodal and multisensory activities.

2. Vocabulary Acquisition and Knowledge Vocabulary acquisition has received much attention over the past 50 years. Studies have focused on the development of the lexical system in L1 and in L2, and have tried to provide cognitively plausible models of the organisation of the mental lexicon and of the processes underlying its development. In spite of this vast body of research, a comprehensive theory of vocabulary acquisition is, however, not yet available (Schmidt 2010). And yet, investigating vocabulary learning is crucial to any theory of second language acquisition (SLA), as it is now widely acknowledged that lexis and grammar are interdependent (Römer 2009), and that the development of the morphosyntactic system is strictly connected to the acquisition of lexical-grammatical patterns and lexical bundles (Tomasello 2003; Hoey 2005; Römer 2009). Moreover, research in foreign language teaching and learning has produced a wealth of data which demonstrate that vocabulary knowledge is a reliable predictor of successful reading comprehension (Milton 2010; Dóczi and Kormos 2016). Barcroft (2015: 29) writes that “like language acquisition in general, the story of vocabulary acquisition is a story of form, meaning, and formmeaning mapping”. Learning a new lexical item ultimately consists in converting input into intake and creating memory traces. Vocabulary acquisition is first and foremost allowed by exposure to relevant comprehensible input, that is, to samples of the target language. Input provides the necessary information on the formal and semantic properties of words and the mapping of ‘labels’ and conceptual material. Each successive encounter with a word increases “the strength of the mapping between each word form and its different meaning and uses” over time (Barcroft 2015: 35). Exposure to the input is, however, no guarantee of successful acquisition. The input must be processable. Richly meaningful contexts offer the best opportunities for the development of lexical knowledge. Studies over the past two decades have uncovered specific conditions which can favour or hinder lexical input processing by changing the allocation of the limited cognitive resources that the human brain has for this task (cf. Barcroft 2015 for a comprehensive review). This type of research can contribute to the development of effective methodologies for vocabulary teaching (cf. section 5.2).

40

Chapter Two

There is still no agreement on where and how lexical knowledge is stored in the human mind. The existence of a mental lexicon seems to be widely accepted, even though, over the years, new theories of language and language acquisition have stretched its boundaries to the point that some have started to wonder whether such a construct is actually needed in order to understand word knowledge (cf. Dóczi and Kormos 2016 for a thorough discussion). Studies agree, however, that lexical knowledge is organised in a structured way. The most debated aspect concerns the distinction between semantic and conceptual knowledge. Most accounts seem to converge on the idea that lexical concepts can be connected to imagery, background knowledge, and non-linguistic information (Roelofs 2000). This wealth of conceptual material available to the speaker is seen as “multimodal-information, which includes imagery, schemas, motor programs, auditory, tactile and somatosensory representations, based on experiential world knowledge” (Pavlenko 1999: 211). A thorough discussion of the nature of the mental lexicon and of its place in the architecture of the human brain would certainly exceed the limits of the present study. The present study adopts Dóczi and Kormos’s (2016: 22) view of vocabulary knowledge as encompassing “both mental representations encoded in memory as well as ability and control in the usage of this knowledge in producing and comprehending language”.

2.1. Vocabulary acquisition in L1 and L2 Much of the seminal research on vocabulary acquisition has focused on the way in which memory traces are developed when we encounter new words and on how the various aspects of lexical knowledge are encoded in actual communication (cf. Aitchison 1994; Schmidt 1994; Laufer and Nation 1995; Meara 1997; Jiang 2000; Robinson 2003; Ellis 2006; Barcroft 2015). In addition, over the past 30 years, a vast literature has highlighted both important similarities and important differences in L1 and L2 vocabulary acquisition (Dóczi and Kormos 2016). In both L1 and L2 vocabulary acquisition, learners need to first and foremost construct a formal representation of the word (aural or orthographic). Then, they need to map the form to the conceptual material the word must encode. This process requires time and undergoes revision and adjustments. Finally, the word’s morphosyntactic and collocational properties must be acquired. There is no agreement on whether these steps are sequential, but most studies support the idea that processing of the formal, semantic, and mapping aspects proceeds in a parallel fashion (Barcroft 2015). Like

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

41

children, L2 learners might also have to face the challenge of segmentation and understanding of word boundaries in oral production. Moreover, remembering novel word formats might also represent a difficulty, especially if they do not follow the phonetic rules of their first language. In this sense, the processes involved in dealing with a new word in an L2 are the same as those involved in dealing with a non-word under experimental conditions. Different words come with a different “learning burden” that is, “the amount of effort required to learn it (Nation 2001: 23). Some factors affecting the learning burden are intrinsic and others are extrinsic. Thus, in both L1 and L2, word length can affect word learnability. Short words are easier to learn. Other word-based factors influencing learnability are phonological and orthographic similarity (e.g., cognate words are easier to learn in L2 than false cognates). Extrinsic factors include aspects which enhance input and make it more easily processable. Regardless of whether learning is incidental or intentional, contrary to L1 learners, L2 learners come endowed with pre-existing conceptual and semantic information about the world as well as about lexical items they have already acquired in their L1. Therefore, learning a foreign word is often a matter of learning new forms and mapping them to available concepts. Of course, in order to say that an L2 word has been learnt, its semantic space must be refined and adjusted. This is a process that happens over time. It is, however, widely accepted that when learning a second language, we make assumptions about new target words on the basis of our experience with our L1. All these aspects, which cannot be discussed in greater detail given the limits imposed by the nature of this work, are fundamental in order to understand L2 vocabulary development and to identify factors which may favour or hinder it. This, in turns, has repercussions on how vocabulary is taught.

2.2. Learning specialised vocabulary Specialised vocabulary acquisition represents an interesting domain to investigate L2 word learning processes. In many cases, learners of languages for specific purposes may already have acquired technical knowledge in their L1 (i.e., medical, financial, botanical concepts). However, in professional or academic contexts, specialised notions and L2 words may be learnt at the same time. In other words, learning specialised vocabulary items in an L2 context may involve processes similar to those involved in learning new words in the L1 context, from

42

Chapter Two

the segmentation of the input, to the acquisition of the formal, semantic, and form-meaning mapping features of novel technical words. This is often the case in content and language integrated learning, in international higher education, or professional contexts. Specialised vocabulary is mostly acquired incidentally in these cases, usually through reading and lectures (Lessard-Clouston 2006, 2009). Knowledge of the subject matter in the L1 may facilitate L2 specialised vocabulary acquisition. Gablasova (2014) points out that specialised texts tend to provide richly meaningful contexts that facilitate the retrieval of difficult referents. They take into account that readers might not be familiar with both specialised terms and contexts and, for this reason, they ‘assist them’ in the interpretation of the text by providing rich contextual information, (e.g., paraphrases or examples). In this sense, specialised texts should make it relatively easy for learners to process the input and infer the meaning of unknown technical terms, in spite of their low frequency. On the other hand, incidental (inferential) vocabulary learning is directly linked to the breadth of one’s vocabulary knowledge, as well as to the knowledge of the subject matter. This might represent a challenge for low proficiency learners with a limited lexical repertoire. They might not be able to take advantage of the contextual information provided and of the increased frequency of the specialised words in domain specific texts. Specialised vocabulary learning and teaching remains, to our knowledge, is a largely unexplored area, even though it has the potential to highlight the positive and negative effects of contextual factors on vocabulary acquisition.

3. Dyslexia and Vocabulary Acquisition Vocabulary learning is particularly challenging for learners with dyslexia due to the deficits involved in their condition. Over the past few years, several studies have been published discussing dyslexia and foreign language learning and teaching (cf. Schneider and Crombie 2003; Nijakowska 2010; Kormos and Kontra 2008; Kormos and Smith 2012). Specialised vocabulary has, however, received no attention. And yet, most adults with dyslexia enrolled in higher education programs or having to read in English in the work environment will be faced with the challenge of understanding and using specialised lexical items. Research on dyslexia has made considerable progress in recent years due to the increasing interest shown by psychologists and educators. Moreover, the understanding of dyslexia has deepened by dint of a parallel understanding of the neuronal architecture and brain functions. Nonetheless, a

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

43

widespread agreement on what dyslexia is and how it can be defined does not exist. This is probably due to the fact that there are major differences between dyslexic people that make it difficult to generalise about their behaviour, problems, and cognitive capacity (Thomson 2009). Dyslexia manifests itself with a variety of symptoms ranging both in quality and degree of severity (Frith 1999; Ramus 2004). In addition, the observed differences between the brains of dyslexics vouch for more than a single neurobiological model of dyslexia (Ramus 2004). Bearing in mind that definitions are not specific and usually are the result of a compromise (Nicolson and Fawcett 2008), dyslexia can be broadly described as a deficit characterised by difficulties in developing literacy and literacyrelated abilities. In 2002, the International Dyslexia Association (IDA) adopted a consensual definition of dyslexia, defining it as a “specific learning disability, neurological in origin, characterised by difficulties with accurate and/or fluent word recognition and by poor spelling and decoding abilities”. Such difficulties were ascribed to “a deficit in the phonological component of the language that is often unexpected in relation to other cognitive abilities and the provision of effective classroom instruction”. This definition also highlighted that there may be “secondary consequences” including “problems in reading comprehension and reduced reading experience that can impede growth of vocabulary and background knowledge”.3 The IDA’s definition combines various levels of description, mainly focusing on the behavioural and cognitive levels, ascribing the difficulties in acquiring literacy (the behavioural level) to deficits in phonological processing (the cognitive level). It also recognises the neurological origin of dyslexia (the biological level), but it describes it as “an ‘either-or’ condition” (Kormos and Smith 2012: 24), while it is now widely recognised that the deficit may vary in degree of severity. In 2006, the British Dyslexia Association (BDA) provided a more comprehensive definition, in which both weaknesses and strengths of the dyslexic learners are highlighted. Dyslexia, in fact, is defined as a “combination of abilities and difficulties” in one or more reading, spelling, and writing skills.4 In the BDA definition, low speed of processing, shortmemory deficits, problems with organisation, sequencing, spoken language and motor skills, as well as difficulties in auditory and visual perception are recognised as possible weaknesses of such a condition. On

3 4

https://dyslexiaida.org/definition-of-dyslexia/ http://www.bdadyslexia.org.uk/dyslexic/definitions

Chapter Two

44

the contrary, creativity, good interpersonal skills, and strong oral skills are identified as possible strengths. Some important aspects of dyslexia emerge from the above statements. First, dyslexia is seen more as a learning and processing difference than a deficit, which has substantial consequences for the educational approach adopted. Second, it is recognised that the signs of dyslexia are not the same for all dyslexic learners, and that both weaknesses and strengths may vary between individuals. Third, such a condition is a permanent one and is caused by weaknesses involving aspects other than phonological processing. Lastly, dyslexia may occur independently of individual cognitive abilities, socio-economic background, and education. In the following section, some of the most widely recognised causes and symptoms of dyslexia will be briefly discussed, highlighting how they can affect L2 learning, including vocabulary learning.

3.1. Causes and symptoms of dyslexia Morton and Frith (1995) identify four levels of description of dyslexia: biological, cognitive, behavioural, and environmental. Here, the focus will be on the biological, cognitive, and behavioural levels that affect L2 learning, as they are the most relevant for the purpose of the present discussion, while the environmental level will not be tackled here. A separate section will be devoted to memory deficits, as it is the domain which has the most impact on storing and retrieving L2 vocabulary. 3.1.1. Biological level At the biological level, there are differences between the brains of dyslexic and non-dyslexic people (Ramus 2004), although all anomalies are not universally present in all dyslexics. Such differences have been correlated to the symptoms manifested during reading, spelling, and learning activities. Research has shown absence of left-right asymmetry of the planum temporale (PT) of the language area (Galaburda, Rosen and Sherman 1989) in dyslexics. The PT, normally larger in non-dyslexics, is located in the heart of Wernicke’s area, which plays a role in auditory processing and representation of sounds and is therefore involved in the correct articulation of phonemes (Warren et al. 2005). It is evident that the discrimination of the incoming acoustic stimuli and their representations are fundamental to building a correct template of the sounds both in L1 and L2 learning. An incomplete or wrong representation and lack of motor

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

45

constraints might cause problems not only in language comprehension, but also in language production. Other scholars have identified deficits in the magnocellular pathways, both visual and auditory, which are much smaller than normal in dyslexic people and are involved in the processing of visual and auditory incoming stimuli. Stein and Walsh (1997) have correlated the anomalies in this network to letters that blur and move around during reading activities. Dyslexics have difficulties with the detection of rapidly changing visual stimuli during reading. The auditory pathway controls the perception of auditory stimuli received at various time-intervals. Deficit in the auditory network would affect perception. When two sounds are produced one after the other at a small interval, the dyslexic is said to have problems in processing them (Tallal, Miller, and Fitch 1993). This correlates to less precision in phonological input/output and, consequently, to an inadequate phonological representation.5 Studies on the cerebellum of dyslexic children show that it is abnormally activated in 80% of cases (Nicolson, Fawcett, and Dean 2001). According to these authors (2001), the weak activation of the cerebellum is responsible for the full spectrum of deficits displayed by dyslexics (Nicolson, Fawcett, and Dean 2001), and correlates to motor skills and articulatory deficits, with severe consequences on writing, and negative effects on reading and spelling. The difficulty in performing two tasks at the same time, for example reading aloud and understanding, is also attributable to the insufficient activation of the cerebellum. However, one of the most important functions of the cerebellum is the role it plays in the automatisation of learned skills (Nicolson, Fawcett, and Dean 2001). The lack of automatisation affects both phonological learning and language learning in general, especially when implicit and non-focussed learning is demanded. 3.1.2. Cognitive level The description at the cognitive level is the functional description of the brain’s activity (Morton and Frith 1995). At this level, dyslexia can be described as a phonological deficit, a speed processing deficit, and an automatisation deficit. These are all aspects that have a relevant role in vocabulary acquisition.

5

There is no general agreement on the magnocellular deficit theory because the deficit is present in only one third of the dyslexic population, and the correlation to reading and cognitive problems is not clear.

46

Chapter Two

Dyslexic learners often show problems with phonological awareness. They display difficulties in counting the syllables of a word, tapping them, making rhymes, working with syllable onsets and rhymes, and repeating non-words. Processing words in an L2 can be compared to processing nonwords, both for their phonotactic structure and lack of an already associated meaning. A difficulty of this type can, therefore, severely affect L2 vocabulary learning. This is not a pure phonological deficit, though, as the working memory deficit also plays a role in keeping the phonemic materials in mind for processing (Baddeley 2000, 2006; Alloway et al. 2009; Diamond 2013). Another important deficit is a speed processing difference, which manifests itself in terms of speed of naming (Wolf and Bowers 1999). According to these authors, there are two independent sources for reading problems: a phonological deficit and a naming speed deficit. A naming speed deficit may or may not accompany a phonological deficit, so that some dyslexic people can present problems in phonological processing but not in naming speed. However, if the deficit is present, there is evidence of difficulties in speed processing for all kinds of stimuli (Nicolson and Fawcett 2008). Dyslexics with this deficit are thus slower than control groups in accessing and retrieving verbal labels. Denckla and Rudel (1976) created a test, called RAN (Rapid Automatized Naming), with the aim of measuring the speed of naming and found correlations between reading and naming deficits. RAN is now used as a training tool for dyslexic and SLI children, as it can foster and automatise verbal access and retrieval (cf. section 5.2.1.). Finally, dyslexia is characterised by an automatisation deficit, or procedural memory deficit. This defect is a general learning problem and affects not only language, but also motor skills and balance (Nicolson, Fawcett, and Dean 2001). Dyslexic learners suffer problems in fluency for all acquired behaviours that normally become automatic through extensive practice, and thus are severely challenged in language learning. 3.1.3. Behavioural level Developmental dyslexics are generally believed to demonstrate primarily reading and spelling problems.6 Usually they are slower and less accurate than control groups. Phonological problems are characterised by processing differences, with difficulties in phoneme coding and decoding. 6

Comorbidities will not be taken into account as they do not characterise the pure dyslexic type.

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

47

Dyslexics read slowly and are less accurate. For example, they make reversal errors, such as reading was instead of saw, and confuse b with d (Reid 2009). They are unable to discriminate individual sounds in words or build rhymes, and they show limited phonological awareness. It is evident that all these difficulties make vocabulary learning a challenging task for dyslexic learners. Dyslexia is a permanent condition, but adults are generally able to compensate for the lack of fluency with instruction and reading strategies, although some signs of dyslexia persist in adulthood (e.g., spelling problems and low phonological awareness). The adult learner struggles to remember new vocabulary, lists of irregular verbs, and grammar rules. They have difficulties understanding written texts, in addition to storing, accessing, and retrieving information from memory. These symptoms are the effects of automatisation and memory deficits, which influence, among other things, L2 learning at large. 3.1.4. Memory deficits Ullman and Pierpont (2005) suggest that two memory systems account for acquisition: declarative memory and procedural memory.7 People with dyslexia have shown an insufficient activation of the cerebral regions correlated to procedural memory (Temple 2002). In their widely accepted Procedural Deficit Hypothesis, Ullman and Pierpont (2005) suggest that the procedural memory system is impaired in dyslexics. As a consequence, dyslexics are unable to learn implicitly and perform automatic tasks, and they struggle to retrieve words from memory and to process grammar. Therefore, in order to compensate for the insufficient activation of the procedural network, dyslexics rely more on declarative memory. Ullman (2004) suggests that there is a sort of “see-saw effect” (Ullman 2004: 243). It consists of the enhanced compensatory activation of one of the two systems when the other is insufficiently activated. Dyslexics thus rely more on their declarative memory system because their procedural memory is insufficiently activated to learn. Other studies have focused on the role of working memory in successful learning (Alloway et al. 2009; Alloway and Alloway 2014). Learners with poor working memory fail in remembering lengthy 7 According to Ullman and Pierpont (2005), declarative memory is a network with associative-like features that supports explicit and focussed learning. Procedural memory is responsible for the instinctive and not conscious acquisition process of cognitive and sensory-motor skills, as well as for their control once they are acquired.

48

Chapter Two

instructions and in performing simultaneous processing, tending to miss letters and words. People with dyslexia present difficulties in reading, in articulatory suppression tests (Cornoldi, Re, and Martino 2013), and in word segmentation. Moreover, they feature slow word retrieval, problems in keeping in memory verbal material, and a shorter working memory span (Jeffries and Everatt 2004; Kormos and Smith 2012). A reduced capacity working memory leads to numerous difficulties which inevitably impact L2 vocabulary acquisition. Among them, problems in concentration and focusing attention, word access and retrieval, new vocabulary storage, repetition of non-words, sequential repetition of numbers and items, and holding pieces of information in mind are detrimental to L2 learning and need to be taken into account by educators as specific weaknesses of this kind of learner.

4. Multimodality and Specialised Vocabulary Learning Explicit specialised vocabulary instruction occupies a small space in ESP materials. It often consists in a few exercises focusing on specific words encountered in reading comprehension texts. In ESP course books, learners are usually asked to match specialised lexical items and their definitions, to identify collocations, to group target words according to various formal or semantic criteria, and to complete ‘fill-in-the-gap’ exercises with target words. Definitions are either explicitly provided or questions following a reading task ‘force’ learners to focus on the meaning of a specific word in the text. Most frequently, however, specialised vocabulary must be learnt incidentally, by inferring the meaning of specific words in a text. The difficulties in phonological processing, automatisation, and memory associated with dyslexia make these types of activities highly challenging and ineffective. Only rarely do we find exercises that incorporate non-verbal features, for example matching specialised lexical items and images. However, a multimodal approach to specialised vocabulary teaching would seem to have significant potential to address and overcome the main difficulties observed in learners with dyslexia. Multimodality refers to the contribution of different semiotic resources to the construction of meaning. The application of a multimodal approach to learning involving visual, aural, tactile, and kinesic resources is not in itself a new thing. It is, however, only with the evolution of technology, with research on multiliteracy and with the seminal work of Kress and van Leeuwen (1996) that research on multimodality has developed in a systematic way. Since then, the benefits of a multimodal approach to

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

49

learning have been explored in a number of studies, some of which focus on L2 development and teaching. Models of multimodal learning such as those proposed by Mayer (2001), Schnotz, Bannert, and Seufert (2002), and Schnotz and Baadte (2008) have highlighted the use of multiple presentation modes in teaching to enhance understanding and result in deeper learning. These contributions focus mostly on the combination of visual and written input, which is said to favour knowledge construction and, consequently, memorisation. Moreover, multimodal input allows learners to find their own path in processing information, and to become “an active sense maker who experiences multimedia presentation and tries to organise and integrate the presented material into a coherent mental representation” (Mayer 2001: 13). Schnotz, Bannert, and Seufert (2002) discuss the use of pictures and texts for text comprehension. Their approach builds on Baddeley’s (2000) theory of working memory, which consists of a phonological loop, a visuo-spatial sketchpad, an episodic buffer, and an executive function. Each of these components is responsible for the storage, integration, and retrieval of verbal or visuo-spatial information. The central executive is responsible for the selection of relevant information, for focussed attention, and for the association and comparison between new and old information and its activation and retrieval from long-term memory (Alloway et al. 2009). According to Schnotz, Bannert, and Seufert (2002), exploiting different channels allows learners to learn concepts and to organise them through different resources. Thus, text comprehension together with image interpretation and comprehension are complementary but distinct ways to create mental representations, therefore leading to conceptual organisation through different paths. This is especially relevant for dyslexic learners who have a deficit in the phonological path (Ramus 2004; Reid 2009; Kormos and Smith 2012). Through multiple semiotic resources, they can in fact use these diverse “external representations as information sources in order to construct internal (mental) representations of the learning content in working memory and store those representations in long-term memory” (Schnotz and Baadte 2008: 27, cited in Farias, Obilinovic and Orrego 2011). Over the past two decades, this field of research has expanded and several studies have highlighted the potential of multimodal instruction for L2 teaching and learning. The first systematic attempt to apply models of multimodal learning to L2 development is Plass and Jones’s (2005) adaptation of Mayer’s (2001) model. They discuss the use of visual and verbal input in terms of input apperception and processing enhancement. Some of the most interesting studies have focused on vocabulary

50

Chapter Two

acquisition and retention (cf. Farías, Obilinovic and Orrego 2011 for a review). However, research on ESP learning and teaching from a multimodal perspective is scarce, with some notable exceptions (cf. Crawford Camiciottoli and Bonsignori 2015; Crawford Camiciottoli and Fortanet-Gómez 2015). It is widely acknowledged that learners with dyslexia benefit from multisemiotic input presentation. A long-standing tradition of research on the pedagogical treatment of dyslexia has investigated the ways in which teaching materials and methodologies can create truly inclusive learning environments capable of accommodating individual differences (cf. Nijakowska 2010). The main methodology applied in developing inclusive materials for dyslexic learners is the multisensory structured learning approach, which is inherently a multimodal methodology. Based on the Orton-Gillingham instructional approach (Gillingham and Stillman 1997), it has generated a vast amount of theoretical studies and applications, mostly to teaching reading and writing to dyslexic children (cf. Nijakowska 2010 for a review of the methodology and its applications). The method resorts to the simultaneous activation of different sensory channels (e.g., auditory, tactile, visual, and kinaesthetic) to create a compensation effect in the learner. “[...] the information integrated via unaffected routes can lead to the development of written language skills. The multimodal perception is usually more advantageous than mono-modal with regard to both the amount of remembered material and the pace of learning [...]. The simultaneous presentation of linguistic material with the use of as many sensory channels as possible benefits individual with dyslexia, in fact, it is assumed that the more modalities involved in the learning process, the more effective it appears to be.” (Nijakowska 2010: 125).

It should be evident that a multimodal approach to vocabulary teaching can enhance acquisition in all learners, but it is essential in dyslexic learners. They need extra work in order to process the input correctly, and to memorise and automatise information. ESP poses further challenges to these learners because specialised vocabulary usually includes lowfrequency items and long, morphologically complex words. Moreover, the etymology of specialised lexical items is usually rooted in different language families (e.g., Romance languages, Ancient Greek, Arabic, etc.). For this reason, learners often cannot rely on their knowledge of the general language. Monoreferentiality also hinders the inferential derivation of the meaning of specialised lexical items in a text, which may already be impaired to a certain extent in some people with dyslexia.

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

51

A multimodal approach to vocabulary teaching can provide the opportunity for input enhancement and increased exposure to the target words through more than one mode. This would favour noticing processes (Schmidt 1994), described by Robinson (2003) as rehearsal in working memory that plays a central role in vocabulary acquisition. By increasing the frequency of multisemiotic presentation of target words, multimodality would prompt input rehearsal—a fundamental mechanism in vocabulary acquisition (Hulstijn 2001)—thus decreasing the load on working memory. By providing a richly meaningful context, multimodal materials can favour the transformation of input into intake, and prompt the development of both declarative and procedural knowledge. Moreover, by providing opportunities for meaningful multisemiotic output, multimodal L2 vocabulary instruction can favour automatisation processes, recognition, and recall of target words.

5. A Case Study This section describes the results of a case study designed to test the effects of different multimodal vocabulary teaching techniques with ESP learners with dyslexia. The study was carried out at the Linguistic Centre of the University of Pisa between January and June 2016.

5.1. Research questions and methodology The participants in the study were 26 learners between 19 and 25 years of age with developmental dyslexia. They were enrolled in different degree programmes and participated in an 80-hour course of English offered by the university to learners with specific learning differences (developmental dyslexia). At the beginning of the course their proficiency level was assessed with an adapted placement test corresponding to the A2 CEFR level (elementary). They attended the course to reach the B1 CEFR level (intermediate), which is the minimum level of English required in order to graduate from any undergraduate programme of the University of Pisa. Given the low proficiency level of the participants, the course programme mostly focused on general English structures and vocabulary. However, given that the majority of the participants were enrolled in the general or applied sciences programmes, a module of the course was dedicated to medical and scientific English. More specifically, a number of activities were devised in order to help learners develop receptive language skills, so that they could read a scientific text or listen to a lecture or presentation focusing on scientific topics. The present discussion focuses

52

Chapter Two

only on some of the activities devised to teach the students the words for body parts. Given the low proficiency level of the learners, activities focused both on the general words for body parts in English, with various degrees of detail (e.g., head, chin, waist, groin, etc.) and on specialised terminology connected to organs and body parts (e.g., bowels, intestines, liver, hepatic, etc.). The participants were divided into two groups: Group A and Group B. Group A was taught with a range of highly multimodal activities, including some derived from the pedagogic adaptation of rehabilitation materials. Group B was taught following the guidelines discussed in the literature on foreign language teaching to dyslexic learners (cf. Nijakowska 2010; Kormos and Smith 2012), and some multisensory materials. The case study was designed with two research questions in mind. The first question concerned the impact of using exclusively multimodal techniques vs. a mix of traditional and multimodal techniques on specialised vocabulary learning. Secondly, we wanted to assess the impact of adapting therapy strategies for multimodal teaching of vocabulary on specialised word recognition and recall. Finally, we wanted to verify whether developing receptive vocabulary knowledge could favour specialised text comprehension in dyslexic learners. Reading comprehension is a notoriously challenging task for dyslexic learners. This is due first and foremost to their well-known difficulties in decoding the written text. However, even when provided with the vocal reading of a text, they struggle to understand the meaning. Various hypotheses have been put forward to explain this, among which the fact that their vocabulary breadth and depth and their general knowledge is somewhat limited compared to that of their peers with an equal level of instruction. This is probably because they read little, and therefore have fewer opportunities for vocabulary development. This problem is obviously amplified when they read a text in a foreign language. The participants in the study were given a pre-test and a post-test. The pre-test was a B1 level reading comprehension exercise built around a text about angina. The post-test included a B2 level reading comprehension task, a word recognition task, and a spoken word recall task. The test was adapted for dyslexic learners and was administered in a multimodal format (i.e., written test with multiple choice questions including images and vocal reading of all the linguistic material and hotspot drag-and-drop word-image matching) so as to eliminate the obstacles posed by their condition as much as possible.

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

53

5.2. Theoretical assumption and multimodal adaptation of materials The activities developed and implemented were based on the theoretical assumptions relative to the processes at work in vocabulary acquisition. Special attention was devoted to making lexical input more easily processable. We assumed that this would give learners with dyslexia a better chance to transform it into intake and eventually acquire long-term lexical knowledge. We also assumed that such knowledge would eventually be available for automatic processes. In line with Barcroft (2015) and earlier research, we assumed that semantic, structural, and mapping-oriented processing are largely dissociable and that different tasks can have a positive or negative impact on lexical input processing. More specifically, we assumed that tasks requiring one specific type of processing would favour the learning of that specific feature of target words and be detrimental to the acquisition of other features. This translated into keeping tasks simple, clear, and progressive, so as to avoid depleting learners of their resources. Only after they had had a chance to process all three levels of target word knowledge, were more global activities proposed. Since we know that learners with dyslexia have difficulties with implicit learning, all vocabulary activities involved explicit instruction and intentional learning. Explicit vocabulary instruction preceded reading or listening comprehension tasks. The first step was always the presentation of the target words through the visual and auditory channels. In order to reproduce the L1 formmeaning mapping process as much as possible, we resorted to the visual and oral presentation of the target word in order to help learners familiarise with the phonological form while developing semantic representations for it. Slideshows with an image of the target words’ referents were presented and the word was heard by the learners. Since input with phonetically relevant acoustic variability results in better lexical representation and favours vocabulary learning (Barcroft and Sommers 2014), each target word was pronounced by at least three different native speakers and by two different non-native speaker instructors. Occasionally, short videos with people uttering the target words were presented. Sommers and Barcroft (2013) have demonstrated that referent token variability in word-image pairing tasks has negative effects on vocabulary learning. For this reason, the images selected to represent the referents of

54

Chapter Two

target words were kept the same throughout the course and across activities, including testing. Vocabulary acquisition is more successful if the number of exposures to the target word is higher than if the time of exposure is longer (Barcroft 2015). Each target word was therefore presented five times for 3 seconds, before moving to the next target word. This first phase of the lexical training was assumed to favour semantic and phonological vocabulary rehearsal. Failing to perform the latter task is one of the causes for poor foreign vocabulary acquisition in dyslexics. Moreover, presenting specialised words through visual input was assumed to make processing monoreferential lexical items easier and to make processing resources for segmentation of complex forms available. Phonological processing is favoured by repetition. Therefore, after a first silent phase, learners were asked to repeat the words after the audio input and while looking at the image. Despite the fact that choral repetition has been found detrimental to vocabulary learning (Barcroft 2015), there is, to our knowledge, no research investigating the effects of repetition on the phonological processing of words in dyslexic learners. Moreover, the negative effects have only been observed in the case of output without access to meaning (e.g., listen and repeat tasks). The multimodal presentation of the input, though, provided semantic information. In addition, the learners were only asked to repeat the target words after having been exposed to the visual and auditory input and, therefore, after meaning-mapping processing. Output with access to meaning has, in fact, positive effects on vocabulary acquisition (Barcroft 2015). According to Slamecka and Graf (1978), it is easier for human beings to remember words when they are required to generate them on their own. This is known as the generation effect. In other words, learners better remember words that they are required to retrieve. For this reason, the next step consisted in projecting the images again and in asking learners to pronounce the word before hearing the audio. Each target word was presented five times at short intervals. Finally, the orthographic input was added. Target words were projected twice, five times each with their pronunciation and their orthographic form in English. The first set of five repetitions was only accompanied by the presentation of the written form of the word. The second set was presented with the written form of the word in a colourcoded format which highlighted the morphological composition of the word in a consistent way across stimuli (e.g., similar suffixes in the same colour). Both Group A and Group B received this instruction.

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

55

After this phase, learners in both groups were given flash cards with the same images used in the slideshow and other cards with the corresponding written form. These flashcards were used for several vocabulary learning ludic activities, such as matching activities, memory games, and puzzles. Group A was also involved in more creative and active tasks which involved the activation of the tactile channel and kinaesthetic activity. They were asked to mime the meaning of some of the target words presented, to cut and paste the images on a body shape, and to create moulds of organs and body parts with modelling paste. Moreover, simple songs for children were presented which focused on body parts. They were shown on video, and then the learners were asked to learn the song, and to sing it first while listening to the video and then without audio, to force word retrieval from memory. When words designating body parts had been learnt, we presented specialised anatomical terms through the same types of activities. Domain specific breadth and depth was developed through matching exercises involving generic and technical body part lexical items. Lexical properties of the target words were taught by presenting morphologically complex derived words (e.g., pancreas – pancreatic) through colour-coded written words and audio files stressing the derivative process. Group A was asked to change word classes by shaping the suffixes with modelling paste to exploit the tactile channel. Finally, the learners in Group A were asked to stand in a circle and to tape cards with names of the organs of human body on the position of the latter on their classmates. This forced meaning retrieval from reading, which is one of the most challenging tasks for learners with dyslexia. Finally, learners were presented with short video clips and texts where the target words were used. They were asked to identify words associated with the target words without specification of the type of semantic relation (e.g., synonymy, antonymy, or thematic associations). This activity was meant to promote semantic processing, the wiring of the newly acquired words in the lexical network, and the expansion of the vocabulary breadth and depth, and also give the learners opportunities to practise inferencing of new word meaning.8 An activity was developed to improve rapid naming of specialised vocabulary. It was built on an application (Run the RAN) used to train

8

Learners were engaged in productive communicative activities as well (e.g., doctor-patient meetings). Such activities are not described in the present work because they did not focus on specialised vocabulary learning per se, but on the development of more general communicative skills in specialised contexts.

56

Chapter Two

dyslexic and SLI children to foster and automatise verbal access and retrieval. RAN (Rapid Automatized Naming) was initially created to measure the speed of naming and to identify naming and automatisation deficits (see 3.1.2). Naming is ontogenetically as well as philogenetically a primary activity connected to learning and speaking, and precedes reading. Reading recycles the neuronal structure which is devoted to visual word form area and face recognition (Dehaene and Cohen 2011). Naming is, therefore, an activity that precedes reading letters and words, but that follows object detection and form recognition. Naming is an automatised cognitive behaviour, which is impaired in dyslexics, but it is possible to train the brain and make word-retrieval less slow. RAN is a training activity to speed up naming. A computer application was developed in 2013 by the software house Anastasis after the idea of Chiara Pecini of the IRCC Stella Maris.9 The application, called Run the RAN, is meant to allow SLI and dyslexic children to train at home on a daily basis. The RAN task reproduces a naming activity which facilitates reading because it uses a sequence of simple pictures of referents that have to be named one after the other in a given time span. The application was developed for Italian, so the number of word-syllables was also taken into account. It is not meant for L2 instruction. First, the child has to familiarise with the labels for each picture. The pictures appear on the screen in two different ways. Initially, they pop up one at a time, in a row. Then, they are presented all together in a row and then they disappear. The pictures are presented from left to right, following the direction of the eyes when reading. The task shares some features with reading: selective visual attention, the maintenance of phonological and lexical information in memory, and output planning and control. RAN training has been proven to develop word and non-word reading ability and rapidity and lexical representation of words, and to speed up the retrieval of phonological labels corresponding to visual stimuli (Pecini 2015). Run the RAN was adapted to enhance specialised English vocabulary learning. Only Group A was taught with the adapted application. The same medical terms presented in the activities described in 5.2 were included. Learners were first presented with five items at a time on the screen for the familiarisation phase. The pictures were accompanied by the written and 9

The Scientific Institute for Research, Hospitalisation and Health Care Stella Maris is a centre for children with SLI and other cognitive disorders in Pisa, Italy.

Teachingg Specialised Voocabulary to Dysleexic Adult Seconnd-Language Leaarners

57

spoken form m. Differentlyy from the Itaalian SLI childdren, the lexiical items presented w were new (e.g.,, presented in the previous llesson with th he method discussed beefore) or not known k (i.e., prresented for thhe first time). Learners had to famiiliarise with both b the pictu ure and the pphonological label l (see Fig. 2-1). F For words thaat had already y been presennted, the sam me images were used.

Fig. 2-1. Runn the RAN: Fam miliarisation phaase

The trainning phase foollowed. Each h stimulus wass presented att least 15 times in subbsequent slide presentation on the screen (see Fig. 2-2) and had to be nameed in one seecond. In eveery lesson, 110 to 15 wo ords were presented. Each woord was then introduced i in its written annd spoken forrm in two separate listts. Both the written w and spoken s word lists containeed all the words preseented in a lesson. The two modalities w were added in n order to enhance andd strengthen the acquisitio on of each ittem, which should s be repeated annd recognisedd starting frrom visual aand oral mo ode. The multimodal task adopted favoured a mu ultisensorial aanchoring.

58

Chapter Two

Fig. 2-2. Runn the RAN: Traiining phase

The objeective of the taask was differrent from the ooriginal Run the t RAN. First of all,, it was desiigned to prov vide further opportunities to learn specialised vocabulary inn a multimod dal way. Seccondly, it allo owed the continuous repetition of the items, further fu increassing the freq quency of exposure to the input woord. Thirdly, itt partially repproduced the favouring f ostensive coontexts in whicch L1 word-laabels are learnned (Tomaselllo 2001).

5.3. Results Learners w were administeered a pre- and a post-teest. The pre--test was administeredd before thhe beginning of the mu multimodal vo ocabulary instruction aand consisted of a reading comprehensio c on activity. A short text about anginna and its sym mptoms was presented in a multimodall adapted format (B1 CEFR level)). Learners were w asked too answer six true/false questions. T The questions were aimed at testing undderstanding which w was dependent oon vocabularyy knowledge. Both B groups sshowed similaar results, obtaining ann average globbal score of 58 8.9%. The posst-test was addministered after a 10 weekks of instrucction and included a ssimilar reading comprehenssion exercise,, a word-imag ge pairing exercise, andd an oral wordd recall test. The text in the reading comp prehension exercise waas slightly moore difficult (B2 CEFR levvel) than the previous one. On ann average, both b groups showed signnificant imprrovement, answering ccorrectly 73.66% of the tim me, in spite oof the higherr level of

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

59

general proficiency presupposed by the task. However, differences between the two groups were observed. Group A performed better than Group B. The average improvement in the performance of the participants included in Group A was 18.5%, two points higher than the average improvement observed in Group B. Group A also scored better than Group B in the matching task and in the word recall task. However, whereas the difference between the two groups in the first exercise was not marked (95% vs. 93.75%), the difference in the word recall task—notoriously challenging for dyslexic learners—was considerably greater (67% vs. 58.7%), with an advantage of 8.3% for Group A. The results of this preliminary case study seem, therefore, to confirm that multimodal teaching has a beneficial impact for specialised vocabulary acquisition in learners with dyslexia.

6. Conclusions Researching vocabulary acquisition in learners with specific learning differences is a complex task that necessarily requires an interdisciplinary effort. The literature discussed throughout this paper shows that the beneficial effects of a multimodal approach in vocabulary teaching observed in previous research can indeed be explained with reference to the cognitive architecture of the human mind and to the processes at work in second language vocabulary development. Learners with dyslexia can especially benefit from explicit vocabulary teaching and a truly multimodal presentation of the input. L2 teaching is inherently multimodal. However, instruction that involves multiple communicative modes (e.g., images, sounds, gestures) is more effective than instruction that relies only on images and sounds. The case study presented shows that, through the activation of multiple channels, multimodality contributes to enhancing the input, making it more easily processable for learners with impairments which impact their ability to exploit one or the other semiotic mode. Multimodality allows teachers to create richly meaningful contexts in which learners can select the best pathway to retrieve and store information. The case study also provides evidence for the possibility of adapting rehabilitation and training tools for use in the language classroom. A multimodal theory of learning is the ideal starting point for any such adaptation.

60

Chapter Two

References Aitchison, Jean. 1994. Words in the Mind. An Introduction to the Mental Lexicon. Malden, Basil, UK: Blackwell. Alloway, Tracy P., Susan E. Gathercole, Hannah Kirkwood, and Julian Elliot. 2009. “The cognitive and behavioural characteristics of children with low working memory.” Child Development 80(2):606–621. Alloway, Tracy P., and Ross G. Alloway. 2014. The Working Memory Advantage: Train Your Brain to Function Stronger, Smarter, Faster. New York/London/Toronto/Sidney/New Delhi: Simon & Schuster. Baddeley, Allan D. 2000. “The episodic buffer: A new component of working memory?” Trends in Cognitive Sciences 4:417–423. —. 2006. “Working memory: An overview.” In Working Memory and Education, edited by Susan J. Pickering, 1–31. Burlington, MA: Academic Press. Barcroft, Joe. 2015. Lexical Input Processing and Vocabulary Learning. Amsterdam: John Benjamins. Barcroft, Joe, and Mitchell Sommers. 2014. “A theoretical account of the effects of acoustic variabilityon word learning and speech processing.” In The Processing of Lexicon and Morphosyntax, edited by Vincent Torrens and Linda Escobar, 7–24. Newcastle-upon-Tyne: Cambridge Scholars. Cornoldi, Cesare, Anna Maria Re, and Maria Grazia Martino. 2013. “Le implicazioni dell’incompleta automatizzazione della scrittura in studenti universitari che hanno una storia di dislessia.” In Dislessia in età adulta, edited by Enrico Ghidoni, Giacomo Guaraldi, Elisabetta Genovese, and Giacomo Stella. Trento: Edizioni Centro Studi Erikson. Crawford Camiciottoli, Belinda, and Veronica Bonsignori. 2015. “The Pisa Audio-visual Corpus Project: A multimodal approach to ESP research and teaching.” ESP Today 3(2):139–159. Crawford Camiciottoli, Belinda, and Inmaculada Fortanet-Gómez, eds. 2015. Multimodal Analysis in Academic Settings: From Research to Teaching. New York: Routledge. Dehaene, Stanislas, and Laurent Cohen. 2011. “The unique role of the visual word form area in reading.” Trends in Cognitive Sciences 15(6):254–262. Denckla, Martha B., and Rita G. Rudel. 1976. “Rapid automatized naming (R.A.N.): Dyslexia differentiated from other learning disabilities.” Neuropsychologia 14(4):471–479. Diamond, Adele. 2013. “Executive functions.” Annual Review of Psychology 64:135–168.

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

61

Dóczi, Brigitta, and Judit Kormos. 2016. Longitudinal Developments in Vocabulary Knowledge and Lexical Organization. Oxford: Oxford University Press. Ellis, Nick. 2006. “Selective attention and transfer phenomena in L2 acquisition: Contingency, cue competition, salience, interference, overshadowing, blocking and perceptual learning.” Applied Linguistics 27:164–194. Farias, Miguel Angel, Katica Obilinovic, and Roxana Orrego. 2011. “Pontos de possível diálogo entre aprendizado multimodal e ensinoaprendizado de línguas estrangeiras”. Trabalhos em Linguística Aplicada 50(1):133–151. Frith, Uta. 1999. “Paradoxes in the definition of dyslexia.” Dyslexia 5:192–214. Gablasova, Dana. 2014. “Learning and retaining specialized vocabulary from textbook reading: Comparison of learning outcomes through L1 and L2.” The Modern Language Journal 98(4):976–991. Galaburda, Albert M., Glenn D. Rosen, and Gordon F. Sherman. 1989. “The neural origin of developmental dyslexia: Implications for medicine, neurology, and cognition.” In From Reading to Neurons, edited by Albert M. Galaburda, 377–388. Cambridge Massachusetts: The MIT Press. Gillingham, Anna, and Bessie Stillman. 1997. The Gillingham Manual. Cambridge, MA: Educators Publishing Service. Hoey, Michael. 2005. Lexical Priming: A New Theory of Words and Language. London: Routledge. Hulstijn, Joris. 2001. “Intentional and incidental second language vocabulary learning: A reappraisal of elaboration, rehearsal and automaticity.” In Cognition and Second Language Instruction, edited by Peter Robinson, 258–286. Cambridge, UK: Cambridge University Press. Jeffries, Sharman, and John Everatt. 2004. “Working memory: Its role in dyslexia and other specific learning difficulties.” Dyslexia: An International Journal of Research and Practice 10:196–214. Jiang, Nan. 2000. “Lexical representation and development in a second language.” Applied Linguistics 21:47–47. Kormos, Judit, and Edit Kontra, eds. 2008. Language Learners with Special Needs: An International Perspective. Bristol: Multilingual Matters. Kormos, Judit, and Anne Margaret Smith. 2012. Teaching Languages to Students with Specific Learning Differences. Bristol: Multilingual Matters.

62

Chapter Two

Kress, Gunther R., and Theo van Leeuwen. 1996. Reading Images: The Grammar of Visual Design. London/ New York: Routledge. Laufer, Batia. 1997. “The lexical plight in second language reading: Words you don’t know, words you think you know and words you can’t guess.” In Second Language Vocabulary Acquisition, edited by James Coady and Thomas Huckin, 20–34, Amsterdam: John Benjamins. Laufer, Batia, and Paul Nation. 1995. “Vocabulary size and use: Lexical richness in L2 written production.” Applied Linguistics 16:307–322. Lessard-Clouston, Michael. 2006. “Breadth and depth specialized vocabulary learning in theology among native and non-native English speakers.” Canadian Modern Language Review 63:175–198. —. 2009. Specialized Vocabulary Learning in Theology: Native and Nonnative English-speaking Students in a Graduate School. Köln: Lambert Academic Publishing. Mayer, Richard. 2001. Multimedia Learning. New York: Cambridge University Press. Meara, Paul. 1997. “Towards a new approach to modelling vocabulary acquisition.” In Vocabulary: Description, Acquisition and Pedagogy, edited by Norbert Schmitt and Michael McCarthy, 109–121, Cambridge: Cambridge University Press. Milton, James. 2010. Measuring Second Language Vocabulary Acquisition. Bristol: Multilingual Matters. Morton, John, and Uta Frith. 1995. “Causal modelling: A structural approach to developmental psychopathology.” Manual of Developmental Psychopathology 1:357–390. Nation, Paul. 2001. Learning Vocabulary in Another Language. Cambridge: Cambridge University Press. Nicolson, Roderick I., Angela J. Fawcett, and Paul Dean. 2001. “Developmental Dyslexia: The Cerebellar Deficit Hypothesis.” Trends in Neurosciences 24(9):508–511. Nicolson, Roderick I., and Angela J. Fawcett. 2008. Dyslexia, Learning, and the Brain. Cambridge Massachusetts: The MIT Press. Nijakowska, Joanna. 2010. Dyslexia in the Foreign Language Classroom. Bristol: Multilingual Matters. Pavlenko, Aneta. 1999. “Bilingualism and cognition”. PhD diss. Cornell University. Pecini, Chiara. 2015. “Trattamento della dislessia evolutiva.” Paper presented at the workshop Dislessia e apprendimento della lingua inglese, Pisa, Italy, October 23.

Teaching Specialised Vocabulary to Dyslexic Adult Second-Language Learners

63

Plass, Jan, and Linda Jones. 2005. “Multimedia learning in second language acquisition.” In The Cambridge Handbook of Multimedia Learning, edited by Richard Mayer, 467–488. Cambridge: Cambridge University Press. Ramus, Franck. 2004. “Neurobiology of dyslexia: A reinterpretation of the data.” Trends in Neurosciences 27(12):720–726. Reid, Gavin. 2009. Dyslexia. A Pratictioner’s Handbook. Oxford: WileyBlackwell. Robinson, Peter. 2003. “Attention and memory during SLA.” In The Handbook of Second Language Acquisition, edited by Catherine Doughty and Michael H. Long, 631–678. Oxford: Blackwell. Roelofs, Ardi. 2000. “Word meanings and concepts: What do the findings from aphasia and language specifity really say?” Bilingualism: Language and Cognition 3:19–21. Römer, Ute. 2009. “The inseparability of lexis and grammar. Corpus linguistic perspectives.” Annual Review of Cognitive Linguistics 7(1):140–162. Schmidt, Richard. 1994. “Implicit learning and the cognitive unconscious: Of artificial grammars and SLA.” In Implicit and Explicit Learning of Languages, edited by Nick C. Ellis, 165–209, San Diego: Academic Press. —. 2010. Researching Vocabulary: A Vocabulary Research Manual. Basingstoke, UK: Palgrave Macmillan. Schneider, Elke, and Margaret Crombie. 2003. Dyslexia and Foreign Language Learning. Abingdon: David Fulton Publishers. Schnotz, Wolfgang, Maria Bannert, and Tina Seufert. 2002. “Toward and integrative view of text and picture comprehension: Visualization effects on the construction of mental models.” In: The Psychology of Science Text Comprehension, edited by José Otero, José Antonio León, and Arthur Graesser, 385–416. Mahwah, NJ: Lawrence Erlbaum. Schnotz, Wolfgang, and Christiane Baadte. 2008. “Domain learning versus language learning with multimedia.” In Aprendizaje Multimodal/Multimodal Learning, edited by Miguel Angel Farias and Katica Obilinovic, 21–49, Santiago de Chile: Publifahu USACH. Slamecka, Norman, and Peter Graf. 1978. “The Generation Effect: Delineation of a phenomenon.” Journal of Experimental Psychology: Human Learning and Memory 4(6):592–604. Sommers, Mitchel, and Joe Barcroft. 2013. “Effects of referent token variability on L2 vocabulary learning.” Language Learning 63(2):186– 210.

64

Chapter Two

Stein, John, and Vincent Walsh. 1997. “To see but not to read; the magnocellular theory of dyslexia.” Trends in Neuroscience 20(4):147– 152. Tallal, Paula, Steve Miller, and Roslyn Holly Fitch. 1993. “Neurobiological basis of speech: A case for the preeminence of temporal processing.” Annals of the New York Academy of Sciences 682(1):27–47. Temple, Elise. 2002. “Brain mechanisms in normal and dyslexic readers.” Current Opinion in Neurobiology 12:178–183. Thomson, Michael. 2009. The Psychology of Dyslexia: A Handbook for Teachers with Case Studies. West Sussex, UK: John Wiley and sons Ltd. Tomasello, Michael. 2001. “Perceiving intention and learning words in the second year of life.” In Language Acquisition and Conceptual Development, edited by Melissa Bowerman and Stephen C. Levinson, 132–158. Cambridge: Cambridge University Press. —. 2003. Constructing a Language: A Usage-Based Theory of Language Acquisition, Harvard: Harvard University Press. Ullman, Michael T. 2004. “Contributions of memory circuits to language: The declarative/procedural model.” Cognition 92:231–270. Ullman, Michael T., and Elisabeth I. Pierpont. 2005. “Specific language impairment is not specific to language: The procedural deficit hypothesis.” Cortex 41(3):399–433. Warren, Jane E., Richard J.S. Wise, and Jason D. Warren. 2005. “Sounds do-able: Auditory-motor transformations and the posterior temporal plane.” Trends in Neurosciences 28(12):636–643. Wolf, Maryanne, and Patricia Greig Bowers. 1999. “The double-deficit hypothesis for the developmental dyslexias.” Journal of Educational Psychology 91(3):415–438.

CHAPTER THREE A MULTIMODAL ANALYSIS OF INTERACTION IN ACADEMIC LECTURES: A CASE STUDY BELINDA CRAWFORD CAMICIOTTOLI UNIVERSITY OF PISA, ITALY

1. Introduction Academic lectures have long been the hallmark of higher education. As the main oral instructional channel in universities, lectures fall under the umbrella of pedagogic discourse, which refers to the set of specialized practices that are designed to transmit knowledge and skills from experts to novices (Bernstein 1986). Although traditional lectures have often come under criticism as an outdated teaching method that leads only to passive learning (cf. Di Piro 2009), they nonetheless remain the most practical way to deliver education to large numbers of university students. Lectures also continue to be popular with both instructors and students in universities around the world (Bordieu and Passeron 1994; Parini 2004; Clay and Breslow 2006; Jones 2007). The predominance of the lecture format in higher education has been further consolidated in the twenty-first century, which brought significant changes and options in terms of how learners may experience lectures. More specifically, learners can now access authentic lectures delivered in university settings in digital format on Internet platforms. Rendering highquality education freely available to whomever desires to learn and advance their knowledge was the primary objective of the Massachusetts Institute of Technology that first pioneered digital education, known as Open CourseWare (OCW) in 2002. Since then, the OCW movement has grown dramatically and digital video recordings of lectures are the core component of the OCW experience (Vladoiu 2011). Today it is possible to access lectures offered by accredited universities located in many different

66

Chapter Three

countries. In addition to accessing lecture videos from the OCW websites of individual universities (e.g., Yale University, which is the source of data for the present study), the Open Education Consortium operates a portal which provides links to thousands of courses and related materials offered by a wide range of educational providers. A vast assortment of ‘virtual’ lectures is also freely available on iTunes U, a digital content resource that allows free access to university lectures. With respect to the traditional face-to-face lecture, the digital counterpart affords users with great flexibility as to when and where they can listen to the lecture, as well as the possibility to review it later for better understanding (McGarr 2009; Santosa 2004). In fact, thanks to technological innovation in the delivery of education, it is often no longer strictly necessary to attend live lecture events, thus removing potential temporal and physical barriers to higher education. However, even students who have no impediments to attending traditional lectures consider their digital versions to be an important resource for study and review purposes (Copley 2007). Thus, OCW lectures can be a valid component of the “‘out-of-school’ experience” that can contribute to education in a meaningful way (van Leeuwen 2015: 583). Lectures in digital format can be particularly beneficial to students who are not native speakers of English, as they provide opportunities for extensive and repeated exposure to language. Sun and Yang (2012) found that higher proficiency Chinese students learning English as a foreign language demonstrated improved proficiency and comprehension in posttesting after viewing MIT and Yale University OCW lectures. Similarly, in Pierce and Scutter’s (2010) study, non-native speakers of English perceived benefits from listening to podcast lectures, particularly as a way to fill gaps in understanding and thereby reinforce learning. Considering the rising demand for English-medium instruction within a process of globalisation in higher education (Coleman 2006), digital lectures have become an important resource for learning. As the leading oral instructional genre in universities, lectures have received considerable attention from discourse analysts. Language studies dedicated specifically to lecture discourse have highlighted numerous linguistic features used by lecturers not only to explain disciplinary concepts, but also to interact and engage with students in order to facilitate comprehension and create a classroom atmosphere that fosters learning. Organizational features of lecture discourse have been extensively explored in the literature. On a macro level, various types of structural patterns have been identified, also influenced by the particular disciplinary content of the lecture. For example, problem-oriented structures in which

A Multimodal Analysis of Interaction in Academic Lectures

67

lecturers explain problems and then suggest possible solutions are frequently found in scientific and technical fields, such as engineering and computer science (Olsen and Huckin 1990; Allison and Tauroza 1995). Genre analysis has been applied to identify functional moves in lecture discourse, including announcing the topic, outlining the structure and presenting objectives (Thompson 1994). Young (1990) analysed lectures across different disciplines that were instead characterised by intermittent phases that appeared and re-appeared in no particular order throughout the communicative event. She distinguished these phases according to their focus on content, exemplification, or discourse structuring. With particular reference to the latter, a number of studies have focused specifically on various types of discourse markers (e.g., Today I want to talk about, Now let’s turn to) used as ‘signposts’ by lecturers to guide listeners through the content (Chaudron and Richards 1986; DeCarrico and Nattinger 1988, among others).1 Lectures are also characterised by some more explicitly interactional features with respect to the discursive aspects described above, which allow speakers to engage with the audience in a more direct way. For example, lexical items such as Okay?, Right?, Yes? posed with a rising intonation as elliptical questions have been described as comprehension checks (Young 1994; Thompson 1998; Morell 2004), used by lecturers to verify understanding on the part on the audience. However, because lecturers typically continue speaking without pausing for feedback after uttering comprehension checks, they rarely trigger any verbal response from the audience (Crawford Camiciottoli 2007), and may actually be a “symbolic rather than real invitation to the audience to enter the interaction” (Thompson 1998: 141). Syntactically complete questions also play a key role in lectures and have been classified by Thompson (1998) as audience-oriented, when a response from the audience is plausible but not necessarily forthcoming, or content-oriented when no response from the audience seems expected and is often provided immediately afterward by the lecturer (Bamford 2005). In a corpus of business lectures, Crawford Camiciottoli (2008) identified some functions of questions in lectures as focusing information, stimulating thought, eliciting response and soliciting agreement, which cut across their audience vs. content orientations. Other interactional features found in lectures are imperative verb forms used as directives. Deroey and Taverniers (2012) found that imperative patterns (e.g., notice, bear in mind, don’t forget) were by far the most frequent type 1

For a recent comprehensive overview of research on discourse markers in lectures, see Rodgers and Webb (2016).

68

Chapter Three

of relevance marker in a corpus of lectures from the British Academic Spoken English (BASE) corpus. In business lectures with a strong theoretical slant, imperatives (e.g., suppose, consider, assume) served to highlight the hypothetical and model-based nature of the content (Crawford Camiciottoli 2007). The trend towards lectures that are not only more interactional but also quite conversation-like is now well consolidated in the literature (Benson 1994; Swales 2004; Crawford Camiciottoli 2007; Morell 2004, 2007). As a consequence, lectures now present an increasing level of informality as seen in features that reflect a colloquial approach to interaction between lecturers and students. Corpus-based studies have shown that lectures contain the same types of discourse dysfluencies and informal lexical features that are associated with everyday conversation (cf. Pérez-Llantada 2005; Biber 2006; Crawford Camiciottoli 2007). Among the latter are phrasal verbs which are largely “colloquial in tone” and therefore strongly associated with informal spoken language (Biber et al. 1999: 127). Phrasal verbs often have the same meaning as single verb forms of Latinate origin, e.g., get rid of vs. eliminate (Swales and Feak 2004). Such informal options (e.g. pick up on, get into, end up with) have been identified in the speech of academic lecturers (Liu 2003; Crawford Camiciottoli 2004), where they seemed to ‘lighten up’ the discourse and thus reduce the social distance between speakers and audiences. Idiomatic expressions have also been analysed in academic lectures. According to McCarthy (1998), idioms have an interpersonal function as they must exploit shared social and cultural knowledge between interlocutors in order to arrive at understanding, and are thus typical of informal spoken interactions. Both Simpson and Mendis (2003) and Crawford Camiciottoli (2007) investigated idiomatic expressions in university lectures. They were not found in particularly high frequencies, but they nonetheless seemed to permeate the discourse due to their marked variety, with many different idioms appearing only once or twice. Moreover, idiomatic expressions served multiple functions, including evaluation, description, emphasis and paraphrasing, which were not always easy to distinguish from each other. Alongside linguistics features of informality, previous research has highlighted other types of interpersonal devices used by lecturers, such as more or less lengthy digressions from the topic. Both Strodt-Lopez (1991) and Flowerdew and Miller (1997) noted that lectures sometimes interrupt the main lecture content with asides or self-reference that serve to build rapport with the audience. Similarly, Dyer and Keller-Cohen (2000) analysed narratives of personal experience integrated into lectures that portrayed lecturers as ordinary people. This was sometimes done in a self-

A Multimodal Analysis of Interaction in Academic Lectures

69

deprecating manner, apparently to mitigate the lecturer’s authoritative role and become more ‘equal’ to students, reflecting a favoured approach among North American academics. More recently, Nesi (2012) looked at lecturer-initiated laughter across three corpora of university lectures: the BASE corpus, the parallel MICASE (Michigan Corpus of Academic Spoken English) and a corpus of engineering lectures. In this setting, laughter appeared to perform various functions, including “maintaining social order, building rapport, relieving tension, and modelling academic and professional identities” (Nesi 2012: 79). During lectures, the verbal features of interaction in lecture discourse discussed in the preceding paragraphs are often accompanied by other semiotic resources beyond speech. These include visual images and the lecturer’s non-verbal cues while speaking that comprise prosodic signals, facial expressions, direction of gaze, hand/arm gestures, and body positioning and movement, all of which may all contribute to comprehension (Sharpe 2006). The role of multimodality in pedagogic settings has been a topic of research at the levels of primary and secondary education (cf. Kress et al. 2001). However, relatively few studies have addressed the contribution of semiotic resources beyond the verbal mode in university settings, and particularly how these resources interact with each other in order to construct interpersonal meanings between lecturers and students (Fortanet-Gómez and Crawford Camiciottoli 2015). Using a case-study approach, this chapter aims to shed light on the interplay between verbal features of interaction and accompanying nonverbal cues during an academic lecture, as well as how this interplay may work towards enhancing understanding. On the verbal level, the analysis focused on the lecturer’s use of selected interactional features, specifically questions, comprehension checks, imperatives, idiomatic expressions and phrasal verbs. On the non-verbal level, prosodic stress, gaze direction, and hand/arm gesturing that co-occurred with interactional language were investigated. The following research questions were addressed: (1) Which verbal interactional elements are present in the lecture? (2) Are the verbal interactional elements accompanied by non-verbal features and, if so, which ones? (3) How do the verbal and non-verbal features interact to construct interpersonal meaning and reinforce learning?

Chapter Three

70

2. Methodology 2.1. Data, setting and participants As a case study, the data refer to a single communicative event, in this case an academic lecture delivered at Yale University during the fall semester of 2006. The lecture was entitled “Introduction: What is Political Philosophy?” and was the first of 24 lectures that comprised the semesterlong undergraduate course Introduction to Political Philosophy. The lecture was video recorded and made available on Yale University’s Open Courses website,2 as well as on YouTube and iTunes. According to the information found on the website, Introduction to Political Philosophy aims to introduce students to the major figures, texts, and concepts associated with Western political thought. In addition to non-edited digital recordings of the lecture in both video and audio formats, the website also offers numerous other resources for those who follow the course. These include the syllabus, an overview of the content of each lecture, a list of textual references with direct access links for each lecture, an interactive survey for course participants, and a transcript of the lecturer’s speech. The latter resource proved to be particularly useful for the analysis, as will become evident in the next sub-section. From various Internet sources, it was possible to collect information to characterise the lecturer (male and middle-aged) as a highly experienced and esteemed scholar of political science.3 The lecturer’s delivery style appeared to be a hybrid between “reading style” and “conversational style” lectures (Dudley-Evans 1994: 48). The lecturer often gazed down towards the podium, probably consulting written notes. The discourse was characterised by the presence of highly varied vocabulary to encode abstract concepts and well-formed syntactic structures illustrated in the transcript excerpt in example (1), which was also delivered slowly, deliberately, and smoothly, unlike typical spontaneous speech production. (1)

2

These doctrines have not simply been refuted, or replaced, or historically superceded; they remain in many ways constitutive of our most basis outlooks and attitudes […] Furthermore, it would seem that the study of

http://oyc.yale.edu/ Steven B. Smith, Introduction to Political Philosophy (Yale University: Open Yale Courses), http://oyc.yale.edu/ (Accessed June 13, 2014). License: Creative Commons BY-NC-SA. (http://oyc.yale.edu/terms): http://creativecommons.org/licenses/by-nc-sa/3.0/us/

3

A Multimodal Analysis of Interaction in Academic Lectures

71

the great books or great thinkers of the past can easily degenerate into a kind of antiquarianism, into a sort of pedantry.

However, there were also stretches of discourse that seemed to be quite spontaneous due to the presence of some dysfluencies such as pause fillers (i.e., uh), repetitions, and false starts typical of unplanned speech production (Biber et al. 1999). In this particular lecture, there were no supporting visual resources and there was no direct verbal interaction with students in the audience. Viewing of the other 23 recorded lectures and corresponding transcripts for this course confirmed that most of them followed this format associated with the traditional frontal and largely monologic lecture (cf. Morell 2004), although a few lectures included video resources and some limited interaction between the lecturer and students. In one lecture, the speaker began on a humourous note, joking and asking the audience whether they had seen a particular comic film. The video camera was positioned exclusively on the lecturer who was positioned behind a podium in front of a blackboard (which he did not use) in what appeared to be a relatively large classroom. This was surmised from the recording which showed students passing in front of the video camera while taking their seats and the presence of various types of audience noise (e.g., coughing, seat adjustments, etc.) throughout the lecture, which lasted 37 minutes and 5 seconds. Moreover, the recording of the final lecture of the course concluded with a forceful appreciative applause for the lecturer from the students, which suggested a relatively large number of participants in the course.

2.1. Analysis The analysis was undertaken in various phases in order to identify both verbal features of interaction and possible accompanying non-verbal signals. This process is explained step-by-step below. The first step was to verify the accuracy of the lecture transcript (4,285 words) as compared to the recorded speech. The transcript was produced by means of standardised orthographic transcription (Thompson 2005), with no prosodic or phonetic mark-up. Italics or double quotes had occasionally been inserted to signal stress or emphasise certain lexical items. The comparison of the transcript with the audio input revealed a largely verbatim representation of the verbal speech, with the exception of the omission of one aside in which the lecturer commented on needing to take a sip from a cup sitting on the podium. All pause fillers (e.g., uh, um) had been eliminated and only one false start was indicated by a string of three periods (I hesitate to... I don’t hesitate to say). The transcript

72

Chapter Three

contained standard punctuation to facilitate reading and the text had been structured into four topical sections called “Chapters”, each with its own sub-title and timeline indication of the minutes/seconds where it started. Any additional information, such as a bibliographical reference for the point being discussed by the lecturer, was distinguished from the running speech by the use of square brackets. Thus, the transcript appears to have been carefully prepared in order to facilitate the viewer’s learning experience to the maximum.4 The second step was an in-depth analysis of the verbal content of the lecture. The transcript was carefully read to identify elements of verbal interaction, drawing on the findings of the literature discussed in section 1. I opted to include only elements with at least two occurrences across the lecture, excluding those that were used only once as potentially isolated features that may be less representative of the lecturer’s personal interactional style. Because there was only one aside (described in the previous paragraph) and one instance of a humorous pun which, however, did not evoke any audible laughter from the audience, asides and humour were excluded from the analysis. The following interactional elements were identified for subsequent analysis: questions, comprehension checks, imperatives, phrasal verbs, and idioms. These items were manually coded in the transcript as follows: QU (questions), CC (comprehension checks), IM (imperatives), PV (phrasal verbs) and ID (idioms). They were then tallied for frequency of occurrence. The third step was dedicated to the analysis of non-verbal signals that co-occurred with the verbal features of interaction described above. On a preliminary level, I first viewed the complete video recording of the lecture to understand which types of non-verbal features could be systematically observed. In this case, the positioning and distance of the video camera allowed for clear observation of body movement/posturing, hand/arm gesturing, and whether the lecturer’s gaze was directed outward towards the audience or downwards towards the podium. It was instead not possible to clearly observe and describe facial expressions due to lack of close-ups on the speaker. The quality of the audio input was sufficient to enable prosodic stress on certain lexical items to be distinguished. The video file was then loaded into the software ELAN (Wittenburg et al. 4

Further evidence of this objective was found in a summary of an online interview with Yale Open Course developers. They explained that the transcripts had been produced by real people (rather than machines) with the aim of providing multiple formats (i.e., video, audio and textual) for users to access the lecture content. Available at http://christytucker.wordpress.com/2007/12/11/open-yale-coursesgood-media-traditional-pedagogy/

A Multimodal Analysis of Interaction in Academic Lectures

73

2006).5 This tool makes it possible to annotate multimodal digital resources in a way that simultaneously captures and displays the contribution of multiple semiotic modes. This is especially useful to shed light on intersemiotic complementarity, meaning how different modes “complement each other in the ways that they project meaning” (Royce 2007: 63). Specifically, the tool provides for multiple layers (or “tiers”) of user-defined annotations which are created, inserted, and synchronised with a streaming video. For this lecture, the following tiers were created in the software: x Transcript: the tier in which corresponding words were inserted under the sound waveform in synchronisation with the streaming video; x Interactional devices: the tier in which the options question, comprehension check, imperative, idiom, or phrasal verb were inserted under the corresponding lexical items; x Prosody: the tier that marked the presence of prosodic stress on lexical items or the presence of relatively long pauses; x Gaze: the tier that indicated whether gaze is directed downwards or outwards on corresponding lexical items; x Gesture_description: the physical/dynamic description of hand and arm gestures (Kendon 2004); x Gesture_type: the type of gesture based on McNeil’s (1992) taxonomy that includes beats, iconic, metaphoric, and deictic. The lecture was viewed again while streaming in the software, following along with the transcript previously marked up to code verbal features of interaction. For each verbal feature, the video was paused to allow for the insertion of the corresponding text from the transcript and the insertion of any co-occurring non-verbal cues as described above. In this way, the software was able to provide a visual representation of the interplay or intersemiotic complementarity (Royce 2007) of both verbal and nonverbal features of interaction during the lecture.

5

ELAN was developed at the Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen, The Netherlands. It is freely available at http://tla.mpi.nl/tools/tla-tools/elan/.

Chapter Three

74

3. Results and discussion Table 3-1 reports the frequency counts of each type of verbal interaction identified in the transcript, any accompanying non-verbal cues and their percentages of co-occurrence. Gaze was not analysed separately as it only rarely occurred without some form of gesturing. Tot

Questions Comprehension checks Phrasal verbs Imperatives Idioms Totals

None

+ Stress

Gaze + Gesturing

N 57 19

N 8 4

% 14.1 21.1

N 6 2

% 10.5 10.5

N 10 4

% 17.5 21.1

13 7 6 102

4 3 1 20

30.7 42.9 16.7

6 1 15

46.2 16.7

1 3 18

7.7 42.9 -

Stress + Gaze + Gesturing N % 33 57.9 9 47.3 2 1 4 49

15.4 14.2 66.6

Table 3-1. Distribution of co-occurring verbal and non-verbal features Before discussing the results in Table 3-1, I will first provide an overall description of the lecturer’s non-verbal communication style. Because of the highly individual nature of extra-linguistic cues which largely depend on personal characteristics of speakers and their perceived roles in various situations (Galloway 1972), it is important to take this into account. The lecturer’s speech was often marked by prosodic stress and seemingly deliberate silent pauses. His gaze was frequently directed outward towards the students, but he also often looked downward towards the podium, likely to consult notes as previously mentioned. The lecturer maintained a stable and natural body stance behind the podium, but with a very limited range of movement that did not stray from this position. He occasionally leaned into the podium, placing both hands on either side. Gesturing was frequent but on a small scale using his hands in a relatively contained space in front of the body, without larger movements involving his arms and with a limited variation in terms of gesture type. In fact, the vast majority of the gestures could be classified as beats, or vague rhythmic movements of the hands that correspond to certain words or stretches of speech (McNeil 1992). Other types of gestures—iconic to depict actions or objects, metaphoric to represent ideas or abstract concepts, and deictic gestures to point to some referent—were infrequent.

A Multimodal Analysis of Interaction in Academic Lectures

75

Thus, the lecturer can be described as having a relatively reserved nonverbal style. Table 3-1 shows that questions were by far the most frequent type of verbal interaction, accounting for over half (N=57) of the total number of 102 verbal interactional features, followed by comprehension checks, phrasal verbs, imperatives, and idioms, respectively. In addition, almost half (N=49) of the verbal features were accompanied by a combination of prosodic stress, gaze and gesturing, followed by those with no accompanying non-verbal cues (N=20), with gaze and gesturing only (N=18), and with prosodic stress only (N=15). These results suggest that the interplay between the verbal and the non-verbal modes was an important component of the interactional dimension of this lecture. In the following sub-section, I discuss this phenomenon in each of the five features of verbal interaction.

3.1. Questions Questions and the act of questioning were the main thrust of this introductory lecture, which explains why they were the most frequent type of verbal interaction. Indeed, the overall structural organization of the lecture appeared to be essentially ‘question-driven’, similar to problemdriven macrostructures identified in previous research (Olsen and Huckin 1990; Allison and Tauroza 1995). The primary aims of the lecture were to introduce students to the concepts that would be explored throughout the course and to stimulate the students to begin thinking about them. To do so, the lecturer clearly preferred a question format. However, although the questions corresponded in form to Biber et al.’s (1999) taxonomy—Whquestions, yes/no questions and alternative questions—none were actually questions which required an answer from the students in the audience. Moreover, the lecturer often did not even provide the answer himself as shown in example (2):6 (2)

6

Doesn’t the study of old books, often very old books, risk overlooking the issues facing us today? What can Aristotle or Hobbes tells us about the world of globalization, of terrorism, of ethnic conflict and the like? Doesn’t political science make any progress? So why then does political science, apparently uniquely among the social sciences, continue to study Aristotle, Locke and other old books? These are all real questions, and I

Technically, such questions also cannot be interpreted as so-called rhetorical questions as these are defined by Quirk et al. (1985: 825) as questions that do not expect an answer because it is either irrelevant of non-existent (e.g., Who knows?).

76

Chapter Three raise them now myself because they are questions I want you to be thinking about as you do your reading and work through this course.

The above example illustrates the lecturer’s use of multiple questions posed one after another to create a dynamic sequence that builds momentum with each question. This seemed to emphasise the inquisitive nature of the lecture and, at the same time, function to focus learner’s attention and stimulate thought, in line with previous studies of both spoken and written pedagogically-oriented academic discourse (Hyland 2002; Crawford Camiciottoli 2008). Non-verbal cues systematically accompanied the questions. In fact, they were absent in only 14.1% of occurrences (see Table 3-1). Figures 31 and 3-2 reproduce screenshots from ELAN to illustrate the multimodal analysis of two questions. In Figure 3-1, the question (Interactional device tier) is accompanied by prosodic stress (Prosody tier), as evident from the corresponding waveform under which the words are positioned. The lecturer’s gaze is directed out towards the audience (Gaze tier). His gesture is annotated with the descriptive abbreviation PalmsOpApUD, meaning palms open and apart moving up and down (Gesture_description tier), and classified as a Beat (Gesture_type tier), functioning to underline the importance of the question. In Fig. 3-2, unlike the question analysed in Fig. 3-1, prosodic stress does not accompany the question as illustrated by the relatively flat soundwave form. However, the lecturer uses an interesting metaphoric gesture of touching fingers (FingTch) which seems to perform the abstract function of focusing the audience’s attention on the particular question as one of the most fundamental that will be addressed in the course. This same question also serves as the sub-title of one “Chapters” into which the transcript was structured (see section 2.1). The question can therefore be interpreted as a type of discourse marker (cf. Chaudron and Richards 1986; DeCarrico and Nattinger 1988) to signal which direction the lecture is about to take, reflected both in the lecturer’s speech and in the corresponding written transcript.

A Multimodal Analysis of Interaction in Academic Lectures

Fig. 3-1. Question: What are the problems that I’m referring to?

77

78

Chapter Three

Fig. 3-2. Question: What is a regime?

From the above analyses, the prominent role of questions in this lecture appears to be enhanced by intersemiotic complementarity (Royce 2007). In fact, multiple modes were used by the lecturer to construct interpersonal meanings beyond semantic content by drawing the audience’s attention to particular important points.

3.2. Comprehension checks Similarly to questions, comprehension checks were also mostly accompanied by non-verbal cues, with only 21.1% occurring exclusively

A Multimodal Analysis of Interaction in Academic Lectures

79

on a verbal level (see Table 3-1). In this lecture, the verbal expressions used for this purpose were Okay?, Right?, Yes? and So Far?, sometimes combined together. Interestingly, in one instance, the lecturer used the French expression N’est pas?, although neither the video nor the transcript offer any clues as to the reason for this apparently idiosyncratic choice. Fig. 3-3 shows the multimodal analysis of the comprehension check Yes?. In this case, it is accompanied by prosodic stress, gaze clearly directed towards the audience, palm up moving up and down (PalmUpUD) in a Beat gesture to underline the act of checking comprehension. Thus, the reinforcement of comprehension checks with other semiotic modes seems to heighten their interactional role to verify understanding.

Fig. 3-3. Comprehension check: Yes?

Chapter Three

80

3.3. Phrasal verbs As can be seen in Table 3-1, phrasal verbs were accompanied most frequently either by prosodic stress (46.2% of instances) or without any co-occurring non-verbal signals (30.7% of instances). With respect to questions and comprehension checks, phrasal verbs were less frequently accompanied by gaze and gesturing. A possible explanation is that phrasal verbs are, in effect, very brief linguistic elements embedded within a larger syntactic structure. For this reason, they may not often be afforded the same prominence during speech as a complete interrogative clause or a stand-alone comprehension check. The following phrasal verb forms were identified in the lecture, with several occurring multiple times: end up, build into, rule out, look up to, look to, stand for, open up to, run into, wither away. Fig. 3-4 shows one of the few cases where the phrasal verb look up to was accompanied by both prosodic stress, as well as gaze and gesturing. This particular item can be interpreted as an informal alternative to the single verb forms admire or value of Latinate origin (cf. Swales and Feak 2004). The Beat gesture with palm open and moving up and down (PalmOpUD) serves to underline the verbal expression. However, what is particularly interesting is the use of prosodic stress with distinct pauses between the verb and the particles as is also evident from the corresponding sound waveform. The lecturer’s use of multiple non-verbal modes to emphasise his choice of a colloquial phrasal verb seems to reflect the special importance of the concept which addresses fundamental ethical issues in society, as shown in example (3). (3)

And this regime - in this sense the regime describes the character or tone of a society. What a society finds most praiseworthy, what it looks up to.

A Multimodal Analysis of Interaction in Academic Lectures

Fig. 3-4. Phrasal verb: look up to

81

82

Chapter Three

3.3. Imperatives With only seven instances, imperatives were among the least frequent types of verbal interaction. The imperative forms identified were Let me start, Let me try to say, Let’s just say that, Consider and Think of that (3 occurrences). The three forms based on the hortative let and co-hortative let’s had a mitigating effect so that the imperative actually functions as a suggestion (Jiang 2006). These forms are therefore more inclusive and are relatively common in spoken academic discourse, where they may also function as discourse markers to introduce topics or to shift to new topics (cf. Webber 2005; Crawford Camiciottoli 2007). As indicated in the literature review, imperatives are characteristic of academic lectures (Crawford Camiciottoli 2007; Deroey and Taverniers 2012), so it is not surprising that they also appear in this lecture. However, in this case, the multimodal analysis revealed a type of usage that seems rather unique. Contrary to what might be expected given the pragmatic function of imperatives as directives, here they were accompanied only once by prosodic stress (together with gaze and gesturing). Moreover, three out of seven occurrences were not accompanied by any non-verbal signals, suggesting that this lecturer’s use of the imperative form was mitigated on multiple levels. I would like to concentrate on the form Think of that, which occurred three times. In all cases, it was distinguished prosodically not by stress, but by silent pauses both before and after the utterance. Fig. 3-5 illustrates the multimodal analysis of this imperative form which integrates silence, a fixed gaze and a deictic gesture pointing a finger outward toward the audience (FingUpOut). This seemed to be a highly interactive strategy on the part of the lecturer, who preferred to use a subtle means to direct the audience’s attention, perhaps attempting to downplay his authoritative role in an interpersonal strategy that has also been found in other university lectures (cf. Dyer and Keller-Cohen 2000).

A Multimodal Analysis of Interaction in Academic Lectures

83

Fig. 3-5. Imperative: Think of that

3.5. Idiomatic expressions Idiomatic expressions were identified in the transcript on the basis of McCarthy’s (1998: 130) broad definition, which characterizes them as strings of more than one word whose syntactic, lexical and phonological form is to a greater or lesser degree fixed and whose semantics and pragmatic functions are opaque and specialised, also to a greater or lesser degree.

84

Chapter Three

Narrower definitions of idioms place greater emphasis on the aspect of semantic opacity. For example, according to Moon (1998), idiomatic expressions are “fixed and semantically opaque or metaphorical” (1998: 4). However, for the purposes of this case study, I adopt a broader interpretation with less rigidity in relation to opaqueness to achieve the most exhaustive analysis possible. Idiomatic expressions were the least frequent of all the types of verbal interaction in this lecture, counting only six instances: putting the cart before the horse, heavy hitters, grain of truth, steering the ship of state, streak of cruelty, and pie in the sky. While some of these are simply fixed expressions (i.e., grain of truth and streak of cruelty), others have a metaphorical or cultural nature (i.e., steering the ship of state and heavy hitters inspired by the American sport of baseball), and others are relatively opaque in relation to the meanings of their individual lexical items (i.e., putting the cart before the horse and pie in the sky). In terms of accompanying non-verbal signals, no clear patterns emerged from the analysis, perhaps due to the small number of occurrences. However, four out of six occurrences were accompanied by prosodic stress, gaze, and gesturing, pointing to their interactional role in this lecture. Figure 3-6 illustrates the multimodal analysis of the expression pie in the sky, which co-occurs with prosodic stress, gaze directed outward, and a beat gesture with palms open and apart moving up and down (PalmsOpApUD). Corroborating Simpson and Mendis (2003), this idiomatic expression appeared to serve both emphatic and evaluative (negative) functions, as can be deduced from the complete utterance: this is a very common prejudice against political philosophy, that it is somehow uniquely sort of “pie in the sky” and detached from the world. Interestingly, the expression is preceded by the hedge sort of, a type of usage which Poos and Simpson (2002) also found in academic lectures where it performed a metapragmatic function to draw listeners’ attention to “particularly sophisticated or jargon words” (2002: 16). This also appears to be the case in this lecture.

A Multimodal Analysis of Interaction in Academic Lectures

85

Fig. 3-6. Idiomatic expression: pie in the sky

To conclude this discussion, it is worth pointing out another result that emerged from the multimodal analysis of the lecture: the presence of clusters of verbal interactional devices with co-occurring non-verbal cues. More specifically, at times verbal interactional devices were used concomitantly, rather than in isolation. For example, the phrasal verb look up to, illustrated in Fig. 3-4 was immediately followed in the streaming video by the comprehension check Okay?. Thus, this stretch of discourse features two verbal interactional devices: a phrasal verb and a comprehension check accompanied by prosodic stress, outward gaze, and gesturing. The imperative form Think of that (Fig. 3-5) was not only offset prosodically by a deliberate pause and accompanied by direct gaze at the

Chapter Three

86

audience and a distinctive deictic gesture, but it was also preceded by two thought-stimulating questions (cf. Crawford Camiciottoli 2008) shown in examples (4) and (5): (4)

Will the best regime be a small closed society that through generations has made a supreme sacrifice towards self-perfection?

(5)

Or will the best regime be a large cosmopolitan order embracing all human beings, perhaps even a kind of universal League of Nations consisting of all free and equal men and women?

The presence of multiple types of verbal and non-verbal interaction is further testimony to the phenomenon of intersemiotic complementarity (Royce 2007), which seems to perform an important pedagogic function by highlighting particularly important ideas in the lecture.

4. Concluding Remarks The analysis presented in this study has provided some insights into the interplay between verbal and non-verbal features of interaction used by a lecturer in a university setting. On the verbal level, the lecturer’s speech contained many of the linguistic features that are associated with academic lectures, such as questions, imperatives and comprehension checks, corroborating previous studies on lecture discourse (cf. Bamford 2005; Young 1994; Thompson 1998; Morell 2004; Deroey and Taverniers 2012). Despite the relatively formal register and setting, elements of informality also emerged in the lecturer’s speech, specifically phrasal verbs and idiomatic expressions. These findings are also in line with past research on academic lectures (cf. Liu 2003; Simpson and Mendis 2003; Crawford Camiciottoli 2004, 2007). However, the most interesting findings emerged at the non-verbal level. More specifically, a range of different semiotic resources were often found to accompany the verbal expressions of interaction. In this way, the speaker appeared to skilfully integrate verbal and non-verbal elements to reinforce the most important points of the lecture. This may have served to compensate for the constraints imposed by the frontal lecture that lacked dialogic interaction in which students could request clarification about the content to ensure understanding. The findings of this case study can alert us to possible patterns in the co-occurrence of verbal and non-verbal features found in lecture settings and how they may be used for instructional purposes of the analysis, even

A Multimodal Analysis of Interaction in Academic Lectures

87

if they are not generalizable to other lecturers or lecture events.7 In fact, on a verbal level, lecturers may have very different styles, largely driven by individual proclivities and idiosyncrasies (Crawford Camiciottoli 2007). In addition, as mentioned previously, a person’s non-verbal communication style is highly individual and influenced by a range of personal and social characteristics (Galloway 1972). For a deeper understanding, it would be useful to carry out multimodal analyses of lecture discourse on a larger scale, possibly with several speakers and across different disciplines. With the growing use of technology for educational purposes, the number of universities offering access to OpenCourseWare lectures is destined to increase, and this would facilitate larger scale research on the multimodal dimension of lectures. In addition, as I have attempted to demonstrate in this analysis, increasingly sophisticated instruments for multimodal analysis make it easier to systematically investigate the “performed” nature of the lecture as a multimodal speech genre characterised by the integration of language and action (cf. van Leeuwen 2004). The emerging insights from such research could find useful application in academic listening courses for non-native speakers of English to help them learn to capitalize on non-verbal cues to improve comprehension.

References Allison, Desmond, and Steve Tauroza. 1995. “The effect of discourse organisation on lecture comprehension.” English for Specific Purposes 14(2):157–173. Bamford, Julia. 2005. “Interactivity in academic lectures: The role of questions and answers”. In Dialogue Within Discourse Communities: Metadiscursive Perspectives, edited by Julia Bamford and Marina Bondi, 123–145. Tubingen: Max Miemeyer. Benson, Malcom J. 1994. “Lecture listening in an ethnographic perspective.” In Academic Listening. Research Perspectives, edited by John Flowerdew, 181–198. Cambridge: Cambridge University Press. Bernstein, Basil. 1986. “On pedagogic discourse.” In Handbook of Theory and Research for the Sociology of Education, edited by John G. Richardson, 205–240. New York: Greenwood Press.

7

Following Scherer and Ekman (1982: 15), the value of small-scale research, such case studies, is to provide “discovery” and “illustration”, rather than “proof”.

88

Chapter Three

Biber, Douglas, Stig Johansson, Geoffrey Leech, Susan Conrad, and Edward Finegan. 1999. Longman Grammar of Spoken and Written English. Essex: Longman. Biber, Douglas. 2006. University Language: A Corpus-Based Study of Spoken and Written Registers. Amsterdam: John Benjamins. Bourdieu, Pierre, and Jean Claude Passeron. 1994. “Introduction: Language and relationship to language in a teaching situation.” In Academic Discourse, edited by Pierre Bourdieu, Jean Claude Passeron and Monique De Saint Martin, 1–34. Cambridge: Polity Press. Chaudron, Craig, and Jack C. Richards. 1986. “The effect of discourse markers on the comprehension of lectures.” Applied Linguistics 7(2):113–127. Clay, Tom, and Lori Breslow. 2006. “Why students don’t attend class.” MIT Faculty Newsletter XVIII:6–7. Accessed July 2, 2015. http://web.mit.edu/fnl/volume/184/breslow.html. Coleman, James A. 2006. “English-medium teaching in European higher education.” Language Teaching 39:1–14. Copley, Jonathan. 2007. “Audio and video podcasts of lectures for campus-basedsStudents: Production and evaluation of student use.” Innovations in Education and Teaching International 44:387–399. Crawford Camiciottoli, Belinda. 2004. “Interactive discourse structuring in L2 guest lectures: Some onsights from a comparative corpus-based study.” Journal of English for Academic Purposes 3(1):39–54. —. 2007. The Language of Business Studies Lectures. A Corpus-Assisted Analysis. Amsterdam: John Benjamins. —. 2008. “Interaction in academic lectures vs. written text materials: The case of questions.” Journal of Pragmatics 40(7):1216–1231. DeCarrico, Jeanette, and James R. Nattinger. 1988. “Lexical phrases for the comprehension of lectures.” English for Specific Purposes 7(2):91– 201. Deroey, Katrien L., and Miriam Taverniers. 2012. “Just remember this: Lexicogrammatical relevance markers in lectures.” English for Specific Purposes 31(4):221–233. DiPiro, Joseph T. 2009. “Why do we still lecture?” American Journal of Pharmaceutical Education 73(8): Article 137. Accessed June 15, 2015. doi: 10.5688/aj7308137. Dudley-Evans, Tony. 1994. “Variations in the discourse patterns favoured by different disciplines and their pedagogical implications.” In Academic Listening. Research Perspectives, edited by John Flowerdew, 146–158. Cambridge: Cambridge University Press.

A Multimodal Analysis of Interaction in Academic Lectures

89

Dyer, Judy, and Deborah Keller-Cohen. 2000. “The discursive construction of professional self through narratives of personal experience.” Discourse Studies 2(3):283–304. Flowerdew, John, and Lindsay Miller. 1997. “The teaching of academic listening comprehension and the question of authenticity.” English for Specific Purposes 16(1):27–46. Fortanet-Gómez, Inmaculada and Belinda Crawford Camiciottoli. 2015. “Introduction.” In Multimodal Analysis in Academic Settings. From Research to Teaching, edited by Belinda Crawford Camiciottoli and Inmaculada Fortanet-Gómez, 1–14. New York: Routledge. Galloway, Charles M. 1972. “An analysis of theories and research in nonverbal communication”. Retrieved from ERIC Database (ED059988). Accessed February 22, 2016. http://eric.ed.gov/?id=ED059988. Hyland, Ken. 2002. “What do they mean? Questions in academic writing.” Text 22:529–557. —. 2002. “Directives: Argument and engagement in academic writing.” Applied Linguistics 23(2):215–239. Jiang, Xiangying. 2006. “Suggestions: What should ESL students know?” System 34:36–54. Jones, Steve E. 2007. “Reflections on the lecture: Outmoded medium or instrument of inspiration?” Journal of Further and Higher Education 31(4):397–406. Kendon, Adam. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Kress, Gunther, Carey Jewitt, Jon Ogborn, and Charalampos Tsatsarelis, eds. 2001. Multimodal Teaching and Learning: The Rhetoric of the Science Classroom. London: Continuum. Liu, Dilin. 2003. “The most frequently used spoken American English idioms: A corpus analysis and its implications.” TESOL Quarterly 37(4):671–700. McCarthy, Michael. 1998. Spoken English and Applied Linguistics. Cambridge: Cambridge University Press. McGarr, Oliver. 2009. “A review of podcasting in higher education: Its influence on the traditional lecture.” Australasian Journal of Educational Technology 25(3):309–321. McNeill, David. 1992. Hand and Mind: What the Hands Reveal About Thought. Chicago: University of Chicago Press. Moon, Rosamund. 1998. Fixed Expressions and Idioms in English. Oxford: Clarendon Press.

90

Chapter Three

Morell, Teresa. 2004. “Interactive lecture discourse for university EFL students.” English for Specific Purposes 23(3):325–338. —. 2007. “What enhances EFL students’ participation in lecture discourse? Student, lecturer and discourse perspectives.” Journal of English for Academic Purposes 6(3):222–237. Nesi, Hillary. 2012. “Laughter in university lectures.” Journal of English for Academic Purposes 11(2):79–89. Olsen Leslie A., and Thomas N. Huckin. 1990. “Point-driven understanding in engineering lecture comprehension”. English for Specific Purposes 9:33–47. Parini, Jay. 2004. “The well-tempered lecturer.” The Chronicle of Higher Education, January 16. Accessed June 3, 2015. http://chronicle.com/article/The-Well-Tempered-Lecturer/7998/ Pearce, Karma, and Sheila Scutter. 2010. “Podcasting of health sciences lectures: Benefits for students from a non-English speaking background.” Australasian Journal of Educational Technology 26(7):1028–1041. Pérez-Llantada, Carmen. 2005. “Instruction and interaction in an American lecture class. Observations from a corpus.” The ESPecialist 26:205–227. Poos, Deanna, and Rita Simpson. 2002. “Cross-disciplinary comparisons of hedging: Some findings from the Michigan Corpus of Academic Spoken English.” In Using Corpora To Explore Linguistic Variation, edited by Randi Reppen, Douglas Biber, and Susan Fitzmaurice, 3–23. Amsterdam: John Benjamins. Quirk, Randolf, Sidney Greenbaum, Geoffrey Leech, and Jan Svartvik. 1985. A Comprehensive Grammar of the English Language. London/New York: Longman. Rodgers, Michael P. H., and Stuart Webb. 2016. “Chapter 13. Listening to lectures.” In The Routledge Handbook of English for Academic Purposes, edited by Ken Hyland and Philip Shaw, 309–322. London: Routledge. Royce, Terry D. 2007. “Intersemiotic complementarity: A framework for multimodal.” In New Directions in the Analysis of Multimodal Discourse, edited by Terry D. Royce and Wendy L. Bowcher, 63–109. Mahwah, NJ: Lawrence Erlbaum. Santosa, Paulus Insap. 2004. “Student’s attitude toward webcast lecture: An online survey result.” In Proceedings of the Sixth International

A Multimodal Analysis of Interaction in Academic Lectures

91

Conference on Information Integration and Web-Based Applications Services, Jakarta, Indonesia, September 27–29. https://www.researchgate.net/publication/221237896_Student's_Attitu de_toward_Webcast_Lecture_An_Online_Survey_Result. Scherer, Klaus R., and Paul Ekman. 1982. “Methodological issues in studying nonverbal behavior.” In Handbook of Methods in Nonverbal Behaviour Research, edited by Klaus R. Scherer and Paul Ekman, 1– 44. Cambridge: Cambridge University Press. Sharpe, Tina 2006. “Unpacking scaffolding: Identifying discourse and multimodal strategies that support learning.” Language & Education: An International Journal 20:211–231. Simpson, Rita. C. and Dushyanthi Mendis. 2003. “A corpus-based study of idioms in academic speech.” TESOL Quarterly 37 (3): 419–441. Strodt-Lopez, Barbara. 1991. “Tying it all in: Asides in university lectures.” Applied Linguistics 12 (2):117–140. Sun, Yu-Chih, and Hui-Chi Yang. 2012 “Do language proficiency and lecture comprehension matter? OpenCourseWare lectures for vocabulary learning.” Calico Journal 29(4):663–678. Swales, John M. 2004. “Evaluation in academic speech: First forays.” In Academic Discourse. New Insights Into Evaluation, edited by Gabriella Del Lungo Camiciotti and Elena Tognini Bonelli, 31–53. Bern: Peter Lang. Swales, John M., and Christine Feak. 2004. Academic Writing for Graduate Students. Essential Tasks and Skills. Second Edition. Ann Arbor, MI: The University of Michigan Press. Swales, John M., Ummul K. Ahmad, Yu-Ying Chang, Daniel Chavez, Dacia F. Dressen, and Ruth Seymour. 1998. “Consider this: The role of imperatives in scholarly writing.” Applied Linguistics 19(1):97–121. Thompson, Paul 2005. “Spoken language corpora”. In Developing Linguistic Corpora: A Guide To Good Practice, edited by Martin Wynne, 59–70. Oxford: Oxbow Books. Accessed February 6, 2016. http://ota.ox.ac.uk/documents/creating/dlc/. Thompson, Susan. 1994. “Frameworks and contexts: A genre-based approach to analyzing lecture introductions.” English for Specific Purposes 13(2):171–186. —. 1998. “Why ask questions in a monologue? Language choice at work in scientific and linguistic talk.” In Language at Work. Selected Papers from the Annual Meeting of the British Association of Applied

92

Chapter Three

Linguistics University of Birmingham, September 1997, edited by Susan Hunston, 137–150. Clevedon: Multilingual Matters. Van Leeuwen, Theo 2004. “Ten reasons why linguists should pay attention to visual communication”. In Discourse and Technology. Multimodal Discourse Analysis, edited by Philip LeVine and Ron Scollon, 7–19. Washington, DC: Georgetown University Press. —. 2015. “Multimodality in education: Some directions and some questions”. TESOL Quarterly 49(3):582–589. Vladiou, Monica. 2011. “State-of-the-art in Open Courseware initiatives worldwide.” Informatics in Education 10:271–294. Webber, Pauline 2005. “Interactive features in medical conference monologue.” English for Specific Purposes 24(2):157–181. Weinberg, Aaron, Tim Fukawa-Connelly, and Emilie Wiesner. 2013. “Instructor gestures in proof-based mathematics lectures.” In Proceedings of the 35th Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, edited by Mara Martinez and Alison Castro Superfine, 1119. Chicago: The University of Chicago Press. Wittenburg, Peter, Hennie Brugman, Albert Russel, Alex Klassmann, and Hans Sloetjes. 2006. “Elan: A professional framework for multimodality research.” In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation, Genoa, Italy, May 24–26. Accessed October 5, 2015. http://www.lrecconf.org/proceedings/lrec2006/pdf/153_pdf.pdf Young, Lynne. 1990. Language as Behaviour, Language as Code. A Study of Academic English. Amsterdam: John Benjamins.

CHAPTER FOUR THE DISTINCTIVE MULTIMODAL NATURE OF PRAGMATIC COMPETENCE: BRIDGING THE GAP BETWEEN MODES1 VICENTE BELTRÁN-PALANQUES UNIVERSITAT JAUME I, SPAIN

1. Introduction The most widely accepted approach for language teaching is the so-called “communicative competence” (Hymes 1972) model, which encompasses knowledge of grammar, language use in social contexts, and the sociolinguistic norms of appropriacy. This trend has been followed by several researchers who have advanced various communitive competence models and have described the components of this construct (cf. Canale and Swain 1980; Canale 1983; Savignon 1983; Bachman 1990; CelceMurcia, Dörnyei and Thurrell 1995; Bachman and Palmer 1996, 2010; Usó-Juan and Martínez-Flor 2006; Celce-Murcia 2007). Most communicative competence models are, to the best of my knowledge, typically verbally-oriented, thereby not including explicitly the interplay among modes. An exception could be the work of Celce-Murcia (2007), who emphasised the combination of various modes in her definition of interactional competence. Therefore, there seems to be a need for more discussion about the multimodal nature of communicative competence. One type of communicative competence that has received particular attention among researchers is pragmatic competence. Broadly speaking, pragmatic competence involves pragmalinguistic and sociopragmatic knowledge (Leech 1983; Thomas 1983), which enables speakers to 1

Innovation in Education Research Project: 2954/14 Projecte d’Innovació Educativa: Anàlisi i millora de les estratègies per a escoltar i interpretar textos orals en llengua anglesa sobre la base d'objectius específics, Universitat Jaume I.

94

Chapter Four

communicate successfully in different communicative encounters. The development of learners’ pragmatic competence is essential for increasing their overall communicative competence, since it may help them to become aware of the social norms governing language use, and how to use language appropriately in specific contexts. Considering the importance of promoting pragmatic competence in the language classroom, several researchers, as I will discuss later (c.f. section 4.1.), have advanced different pedagogical approaches that account mainly for awarenessraising and production activities. While these proposals might be applied to teach the verbal side of pragmatic competence, they do not explicitly address the interplay between modes, which is indeed needed since communication is multimodal (Jewitt 2013). To address this issue, in this chapter I first provide an overview of the notion of communicative competence. Secondly, I discuss different communicative competence models, pointing particularly to pragmatic competence and its multimodal nature, i.e., multimodal pragmatic competence. Following this, I present a revision of different pedagogical frameworks designed to integrate speech acts in the language classroom within a pedagogical approach, and then conclude with some final remarks and suggestions for future research directions.

2. The Notion of Communicative Competence Dell Hymes proposed the term communicative competence in 1972 as a reaction against Noam Chomsky’s (1965) notion of competence, which related to the rules of grammar in isolation, without taking into account contextual appropriateness. Moreover, Chomsky’s (1965) competence was based on an ideal speaker-listener with perfect linguistic knowledge, which seems not to be affected by cognitive and situational factors during linguistic performance. Chomsky’s (1965) approach to performance involved “the actual use of language in concrete situations” (1965: 4). For Hymes (1972), competence referred to both tacit knowledge and ability to use language, thereby including non-cognitive factors within the notion of competence. Hymes (1972), among others such as Habermas (1970), reacted against this vision and argued that the Chomskyan view could not account for real-life communication. More specifically, Hymes (1972) argued that his view did not involve any reference to aspects related to language use in social contexts and the appropriateness of utterances in a specific situation. In his view, communicative competence takes into account a sociolinguistic perspective, which consists of both grammatical competence, as in

The Distinctive Multimodal Nature of Pragmatic Competence

95

Chomsky (1965), as well as sociolinguistic competence. Thus, Hymes’s (1972) communicative competence involved not only knowledge of grammatical aspects, but also knowledge of the sociocultural rules of appropriate language use. In this sense, Hymes (1972) understood the notion of communicative competence not only from a theoretical viewpoint, but also taking into account practical needs. Concerning this, the author pointed out that “It is not that there exists a body of linguistic theory that practical research can turn to and has only to apply. It is rather that work motivated by practical needs may help build the theory that we need” (1972: 269). The author rejected the dichotomy of competence and performance, claiming that performance is the observable part and competence refers to the ability to produce such performance (cf. Rickheit, Strohner, and Vorwerg 2008 for a review). Hymes’s (1972) perspective has greatly influenced the field of second/foreign language (SL/FL) studies, not only because his work was regarded as a theoretical background for developing a new language teaching approach and language teaching materials, but also because several researchers followed his tenet, presenting various communicative models and the different components that the construct of communicative competence might involve. Hence, drawing on Hymes’s (1972) notion of this construct, several theoretical models emerged, such as those provided by Canale and Swain (1980), Canale (1983), Savignon (1983), Bachman (1990) Celce-Murcia, Dörnyei, and Thurrell (1995), Bachman and Palmer (1996, 2010), Usó-Juan and Martínez-Flor (2006) and Celce-Murcia (2007). Although various communicative competence models may be found in the literature, I limit the present literature review to those models that, to the best of my knowledge, have been widely recognised by several authors due to their influence within the realm of SL/FL teaching and assessment.

3. Communicative Competence 3.1. Proposed communicative competence models The model proposed by Canale and Swain (1980) may be regarded as a pioneering work that included the basis of the communicative approach to language teaching. Their interest in communicative competence appeared to be related to their need to develop a language test to measure learners’ knowledge of and proficiency in language use. This model, clearly influenced by Hymes’s (1972) approach, consisted of three different competencies: grammatical competence, sociolinguistic competence, and

96

Chapter Four

strategic competence. This model was further developed by Canale (1983), who added a new competence, namely discourse competence. Grammatical competence refers to knowledge of the language system, including knowledge of lexis, morphological rules, syntax, phonology, and semantics. Sociolinguistic competence involves the knowledge of the sociocultural rules of use. Strategic competence relates to the knowledge of how to use verbal and non-verbal communication strategies to overcome communicative breakdowns or to reinforce speakers’ communicative intention. Finally, discourse competence involves learners’ understanding and production of texts in different modes (i.e., listening, speaking, reading, and writing) and it is related to coherence and cohesion in different text-types. The two models proposed by Canale and Swain (1980) and by Canale (1983), while very influential, are not without criticism. For example, Bachman and Cohen (1998) indicated that these two models of communicative competence were to some extent static, as there was little discussion on how the different components interacted with each other, or how language users coped with the characteristics of the context of situation in a given speech event. Moreover, Schachter (1990) claimed that the two communicative models did not provide an accurate description of pragmatic competence. In fact, Schachter (1990) raised the questions “Where does pragmatics fit into the Canale and Swain framework? Is it assumed not to exist? Or is it thought to be coextensive with discourse competence?” (1990: 42). Pragmatic competence was, however, not included as an independent element within the communicative model until Bachman (1990) advanced a communicative language ability model that included pragmatic competence in its own right. Drawing on the works of Hymes (1972), Canale and Swain (1980), and Widdowson (1989), Bachman (1990) provided a framework for communicative language ability that was described as “consisting of both knowledge, or competence, and the capacity for implementing, or executing that competence in appropriate, contextualised communicative language use” (1990: 84). This model maintained the components proposed by Canale and Swain (1980) and Canale (1983), but expanded the nature of strategic competence. Moreover, the author interpreted pragmatic competence as an independent component. Bachman’s (1990) approach focused more on language assessment than on SL/FL teaching. Specifically, this author, along with Bachman and Palmer (1996, 2010), presented a model of communicative language ability. This model includes three components: (1) language competence, which is “a set of specific knowledge components that are utilised in communication via language” (Bachman 1990: 66); (2) strategic competence, or “the mental capacity for

The Distinctive Multimodal Nature of Pragmatic Competence

97

implementing the components of language competence in contextualised communicative language use” (1990: 67); and (3) psychophysiological mechanisms, namely “the neurological and psychological processes involved in the actual execution of language as a physical phenomenon” (1990: 67). Language competence as proposed by Bachman (1990) is therefore further divided into two main components: organisational competence and pragmatic competence. Organisational competence consists of grammatical competence, which relates to the knowledge of lexis, phonology, morphology, and syntax, and textual competence, which involves the knowledge of the norms to join utterances to form a text, either spoken or written, that is coherent and cohesive. Pragmatic competence is further divided into illocutionary competence and sociolinguistic competence. This division seems to be associated with Leech’s (1983) and Thomas’ (1983) distinction between pragmalinguistics and sociopragmatics. Hence, illocutionary competence refers to the ability to perform and interpret utterances, and sociolinguistic competence is concerned with the knowledge of the conventions of language use to perform utterances appropriately in a given context. This competence appears to be similar to Canale and Swain’s (1983), Celce-Murcia, Dörnyei, and Thurrell’s (1995), and Celce-Murcia’s (2007) sociocultural competence. In addition, the author also suggested that strategic competence,2 first introduced by Canale and Swain (1980), involves three different components: assessment, planning, and execution. The concept of strategic competence appears to go beyond Canale and Swain’s (1980) perspective in the sense that this term was used by Bachman (1990) “to characterize the mental capacity for implementing the components of language competence in contextualized communicative language use” (1990: 84). Although Bachman (1990) provided an expanded version of the model proposed by Canale and Swain (1980) and Canale (1983), some problems were identified. In Bachman’s (1990) model, as in those advanced by Canale and Swain (1980) and Canale (1983), the relationship among the different components is not explicitly described as, for example, in CelceMurcia, Dörnyei, and Thurrell (1995). Bachman and Palmer (1996) revisited Bachman’s (1990) model, showing how language knowledge, metacognitive strategies, and affect interacted with each other during language use. This model also showed the interaction between the components of communicative language ability and the contextual features 2

See the works of Jordan (2004) and Phakiti (2008) for further information on strategic competence.

98

Chapter Four

of language use, the test task, as well as mental schemata. The authors also replaced the name ‘illocutionary competence’ with ‘functional knowledge’, arguing that functional knowledge “enables us to interpret relationships between utterances or sentences and texts and the intentions of language users” (Bachman and Palmer 1996: 69). Similarly, they replaced ‘competence’ with ‘knowledge’, including, for example, ‘pragmatic knowledge’ instead of ‘pragmatic competence’. Celce-Murcia, Dörnyei, and Thurrell (1995) proposed a communicative competence model that accounts for the existing relationship between the different components of the communicative model. This model includes the following competencies: (1) discourse competence, (2) sociocultural competence, (3) linguistic competence, (4) actional competence, and (5) strategic competence. Discourse competence refers to the selection, sequencing, and arrangement of sentences to obtain a unified spoken or written text. This competence might be associated to Bachman’s (1990) and Bachman and Palmer’s (1996) textual competence. Sociocultural competence involves the speakers’ knowledge of how to convey appropriate messages in a particular social and cultural context, thereby corresponding to sociolinguistic competence as described by Canale and Swain (1980) and Bachman (1990). Linguistic competence might be seen as similar to Canale and Swain’s (1980) grammatical competence and Bachman’s (1990) grammatical competence. In this case, however, linguistic competence also relates to basic elements of communication, including sentence patterns and types, structure, morphological inflections, phonological and orthographic systems, and lexical resources. Actional competence, similar to Canale and Swain’s (1980) sociocultural competence and Bachman’s (1990) pragmatic competence, involves performing and understanding communicative intent, thus relating the actional intent with a linguistic form that carries illocutionary force, i.e., speech act sets.3 Finally, the authors propose the strategic competence, which is also observed in Canale and Swain (1980) and Bachman (1990). All the different components of the model are influenced by strategic competence, referring to the knowledge of communication strategies and how to employ them. The strategies identified involve three functions of strategy used from different perspectives, specifically, the psycholinguistic perspective, the interactional perspective and the communication continuity/maintenance perspective. Furthermore, strategic competence involves the following communicative strategies: avoidance or reduction 3

A speech act set comprises the set of realisation patterns that are typically used by native speakers of a target language (Olshtain and Cohen 1983).

The Distinctive Multimodal Nature of Pragmatic Competence

99

strategies; achievement or compensatory strategies; stalling, or time-gaining strategies such as fillers, hesitation devices, gambits, and repetitions; selfmonitoring strategies; and interactional strategies. In this model, as pointed out by Celce-Murcia (2007), although it implies “a step forward with respect to Canale and Swain (1980) and Canale (1983)” (2007: 45), there are some gaps that the author tried to solve (Celce-Murcia 1995, 2007). In 2007, Celce-Murcia presented a revised version of the models proposed in 1995 (Celce-Murcia, Dörnyei, and Thurrell 1995; CelceMurcia 1995). This model involves six different competencies: discourse competence, (2) sociocultural competence, (3) linguistic competence, (4) interactional competence, (5) formulaic competence, and (6) strategic competence. The revised model also locates discourse competence in the central position of the model, where it involves selection, sequencing, and arrangement of sentences to obtain a unified spoken or written text, as in Celce-Murcia (1995). The author further distinguished various areas of discourse competence including cohesion, deixis, coherence, and generic structure. Sociocultural competence has to do with speakers’ pragmatic knowledge and involves the knowledge of language variation as regards the sociocultural norms of the target language. Interactional competence includes three different competencies: actional competence, conversational competence, and non-verbal/paralinguistic competence. Actional competence, as in Celce-Murcia (1995), refers to how to perform speech acts and speech act sets. Conversational competence is related to the turntaking system (Sacks, Schegloff, and Jefferson 1974), as well as to other dialogic genres, for example, how to open and close conversations; collaborate and backchannel; and get, hold and relinquish the floor, among others. The last interactional competence is non-verbal/paralinguistic competence, which relates to kinesics, proxemics, haptic behaviour, and non-linguistic phenomena, i.e., silence and pauses. Linguistic competence, as in Celce-Murcia, Dörnyei, and Thurrell (1995), includes phonological knowledge, lexical knowledge, morphological knowledge, and syntactic knowledge. Formulaic competence refers to the chunks of language that speakers use in everyday interactions such as routines, collocations, adjectives, idioms, and lexical frames. Finally, strategic competence, similarly to Celce-Murcia, Dörnyei, and Thurrell (1995), refers to the knowledge of communication strategies and how to employ them. In this regard, Celce-Murcia (2007)—building on Oxford’s (2001) strategies for language learning and language use—indicates that cognitive, metacognitive, and memory-related strategies are important features in this new model. Furthermore, Celce-Murcia (2007) also mentioned that other

100

Chapter Four

strategies crucial for this model were those included in Celce-Murcia, Dörnyei, and Thurrell (1995), namely achievement, stalling or time gaining, self-monitoring, interacting, and social strategies.

3.2. Multimodal pragmatic competence Apart from the competencies described in the above models, other aspects should be taken into account when dealing with communicative competence. More specifically, the multimodal perspective is also crucial, considering that communication is multimodal by nature (Jewitt 2013), and that language learners need to master some specific non-verbal competencies to develop their communicative competence. Speakers use several paralinguistic and extra-linguistic sources together with linguistic sources to communicate either in the spoken or written mode. Speakers produce multimodal output and receive multimodal input constantly. Therefore, approaches to SL/FL learning should integrate both multimodality and communicative competence in order to recognise the importance of using various modes to communicate within the domain of language teaching, learning, and assessment. In line with this notion and following Royce (2007), I also argue for the need to move from the traditional perspective of communicative competence towards a multimodal communicative competence. I suggest that the design of a multimodal communicative competence should go beyond, for example, the development of visual literacy, digital issues, and the interrelation of different semiotics modes in spoken or written communication. Rather, multimodal communicative competence should be discussed drawing on previous research on communicative competence and taking into account the different competencies that have already been described. Thus, in developing a multimodal communicative approach, we should take into account different competencies, the way they are interrelated, and how their development affects the overall multimodal communicative competence. If we do not take into account these aspects, it might be problematic to specify how this approach can be integrated in the language classroom and which route is better to assist language learners’ development of communicative competence. Royce (2007) also posits that a multimodal view of communicative competence “makes a number of important assumptions that are derived from the SFL [systemic functional linguistics] model” (2007: 373). Accordingly, it seems that multimodal communication is constructed from the perspective of exchanging, projecting, or sending meanings in a particular social context. Royce (2007) also argues that social meaning

The Distinctive Multimodal Nature of Pragmatic Competence

101

selections might depend on the cultural context. Moreover, the author suggests that the way people communicate in different visual and verbal modes is the outcome of their choice of particular semiotic systems. Concerning these aspects, Royce pointed out that that “These meanings and choices, realized in differing modes, will necessarily have a message and interactional-focus, and will draw on the textual or compositional conventions appropriate to the mode” (2007: 374). In a similar way, Stenglin and Iedema (2001) indicate that multimodal communicative competence also involves how learners may become competent in interpreting and constructing meaning from a multimodal perspective. My intention here is not to provide a definition of the term ‘multimodal communicative competence’, but to open a discussion on the vision of communicative competence from a multimodal perspective. The phenomenon of multimodal communicative competence requires further investigation in order to examine which competencies should be integrated and how they should be dealt with from both theoretical and practical perspectives. Rather, my endeavour here is to reflect on the multimodal aspects of one of the competencies described in the aforementioned models, particularly, pragmatic competence. The rationale behind this selection is based on the need to integrate pragmatic competence in the SL/FL classroom in order to provide learners with opportunities to develop their communicative competence, with particular attention to illocutionary and sociolinguistic knowledge, or in Leech’s (1983) and Thomas’s (1983) words, pragmalinguistics and sociopragmatics. One the one hand, pragmalinguistics refers to the study of “the particular resources which a given language provides for conveying particular illocutions” (Leech 1983: 11). Therefore, it is concerned with the different linguistic resources speakers might select for a particular context, such as pragmatic strategies, pragmatic routines, and modification devices (Rose and Kasper 2001; Barron 2003). The term of sociopragmatics involves the “sociological interface of pragmatics” (Leech 1983: 10). Hence, sociopragmatics entails the conditions that constrain and govern speakers’ language use. Specifically, it focuses on how different social variables such as power or status, social distance, and rank of imposition, or severity of offense (Brown and Levinson 1987) affect language use. Finally, as indicated by Thomas (1983), sociopragmatics involves “the social conditions placed on language in use” (1983: 99). Roever (2011) points out that sociopragmatics and pragmalinguistics are “tightly connected, as a speaker’s sociopragmatic analysis of a situation (in terms of politeness, possible meanings, and cultural norms and prohibitions) is linguistically encoded through pragmalinguistic

102

Chapter Four

choices” (2011: 2). Furthermore, Thomas (1983) claims that, in order to perform a pragmatically appropriate utterance, speakers should make two main adjustments, specifically the pragmalinguistic assessment of the pragmatic force of the linguistic features, and the sociopragmatic judgement regarding the social factors involved in the social encounter. Hence, as pointed out by Roever (2009), “it is essential that both aspects of pragmatic knowledge are developed and accurately mapped onto one another” (2009: 560). If a language user understands the sociopragmatic norms of a particular language but lacks the pragmalinguistic resources, he/she will not succeed in the communicative event and a pragmatic failure may occur. Hence, the areas of pragmalinguistics and sociopragmatics are critical during communication and are of special interest for SL/FL. As a key component of communicative competence models (cf. Bachman 1990), pragmatic competence has become an object of inquiry in different disciplines such as linguistics, applied linguistics, anthropology, sociology, psychology, communication research, and cross-cultural studies (Taguchi 2009). In the field of SL/FL research, pragmatic competence is commonly regarded as the ability to perform and understand utterances that are appropriate for a particular sociocultural context (Rose and Kasper 2001; Thomas 1983). Specifically, Thomas (1983) defines pragmatic competence as “the ability to use language effectively in order to achieve a specific purpose and to understand language in context” (1983: 92). Pragmatic competence can be viewed “as the knowledge of the linguistic resources available in a given language for realising particular illocutions, knowledge of sequential aspects of speech acts and finally, knowledge of the appropriate contextual use of the particular languages’ linguistic resources” (Barron 2003: 10). This is also supported by Taguchi (2009), who argues that pragmatic competence is “broadly defined as the ability to use language appropriately in a social context” (2009: 3). Thus, pragmatic competence can be seen as the knowledge of the linguistic resources that speakers of a given language have available in a language in order to perform specific utterances and the knowledge of the contextual factors affecting language use (i.e., knowledge of pragmalinguistic and sociopragmatic features). However, pragmatic competence can also be viewed from a multimodal perspective, involving not only verbal performance but also other elements, such as paralanguage and extra-linguistic features, which are employed to communicate. In this sense, I would argue for the use of the term ‘multimodal pragmatic competence’, which would involve not only pragmalinguistic and sociopragmatic knowledge, but also kinesic, paralanguage, proxemics, haptic knowledge, and non-linguistic phenomena—similar to Celce-Murcia’s

The Distinctive Multimodal Nature of Pragmatic Competence

103

(2007) interactional competence —as well as visuals used when writing, such as emoticons, capitalisation, etc. Pragmatic features are present during interaction, whether spoken or written, and therefore all the different modes that operate at the level of pragmatic competence should be taken into account as they all convey meaning. This is why I suggest the use of the term multimodal pragmatic competence, since we should not ignore the non-verbal and extra-linguistic components that are involved in a communicative encounter. For example, when complaining, several paralinguistic and extra-linguistic elements might be observed, such as facial expressions, gestures, haptics, and paralanguage, which might play a paramount role, since they might offer the other participant further information about message being conveyed beyond the words. The same can occur in the case of responding to a complaint. For example, the complainee’s facial expressions and gestures could determine to some extent how he/she feels when the other person is accusing him/her of some wrongdoing. The negotiation of a complaint situation may be also affected by the paralinguistic and extra-linguistic elements in the sense that the complainer might, for instance, reduce the intensity of the complaint by taking into account the paralinguistic and extra-linguistic output of the complainee. In short, I argue for a multimodal perspective of pragmatic competence, based on the premise that communication involves more than words and, therefore, all the elements that interact when communicating should be considered as part of the communicative event as they convey meaning. This is also of paramount interest for language learners because they need to become aware of the different resources they have at their disposal to communicate. Likewise, becoming aware of the importance of other elements in communication, such as paralinguistic and extra-linguistic elements, learners can become better communicators and better understand their interlocutors.

4. Bridging the Gap between Modes: The Case of Speech Acts The pedagogical implications that I present in this section derive from the theoretical framework presented above in relation to multimodal pragmatic competence. However, before dealing with that, it is important to review some previous research on the pedagogical implications of developing pragmatic competence. One pragmatic feature that has received particular attention is the speech act. Performing a speech act appropriately requires a command of pragmatic expertise in order to succeed in different

104

Chapter Four

communicative encounters (Martínez-Flor and Beltrán-Palanques 2014), which, as aforementioned, involves the knowledge of pragmalinguistics and sociopragmatics. My concern here is to provide a framework that attempts to integrate multimodal communicative competence as a way to foster multimodal pragmatic competence in the language classroom. The focus will be on speech acts in a general sense, without focusing on a specific speech act, which would go beyond the scope of the present chapter.

4.1. A pedagogical framework for teaching speech acts The literature makes reference to different pedagogical approaches for the teaching of specific speech acts, e.g., requests, refusals, and apologies, among others (c.f. Félix-Brasdefer 2006; Usó-Juan and Martínez-Flor 2008; Beltrán-Palanques 2011; Martínez-Flor and Beltrán-Palanques 2013, 2014). However, for the purpose of this chapter, I limit my review to those frameworks that I have followed to develop the multimodal pragmatic conversational approach presented in this chapter. One of the pioneering pedagogical models for the teaching of speech acts is advanced by Olshtain and Cohen (1991). The model involved different steps, specifically, (1) the diagnostic assessment, which serves to determine learners’ level of awareness of speech acts in general and, more particularly, the specific speech act under study by means of acceptability rating tests and oral or written tests; (2) the model dialogue, in which learners are presented with natural samples of the speech act studied in order to increase awareness of the social and pragmatic factors that influence the speech act; (3) the evaluation of a situation whose aim is to reinforce learners’ awareness of the factors that affect the pragmalinguistic realisation of the specific speech act; (4) role-play activities, in which learners are provided with opportunities to practise the speech act taught; and (5) feedback and discussion. Broadly speaking, in this approach, learners are first provided with opportunities to be exposed to the typical realisation patterns of the chosen speech act and encouraged to reflect on its use by taking into account the factors involved in the selection of one specific form rather than other. Then, they are provided with output opportunities to practise the speech act and then receive feedback on their performance. The importance of feedback is also noticed by Judd (1999), who proposes a framework for teaching pragmatics that takes into account the specific conditions of each classroom, specifically, whether it is a SL or a FL context, whether the teacher is a native speaker of the target language

The Distinctive Multimodal Nature of Pragmatic Competence

105

or not, as well as learners’ needs to learn the target language and the materials available. Judd’s (1999) model consists of five steps: (1) teacher’s analysis of the speech act, which serves to relate the content of what is taught with learners’ needs; (2) learners’ development of cognitive awareness skills, by which learners are presented with a particular speech act in order to make them understand the appropriate linguistic realisations that may be employed to perform it; (3) receptive, integrative skills, which involve learners’ recognition of the realisation patterns of the speech act at the discourse level; (4) controlled productive skills, by which learners practise the speech act taught; and (5) free integrated practice, which relates to learners’ practice, not only of the speech act studied, but also of other features of conversation. Drawing on Olshtain and Cohen (1991) and Judd (1999), MartínezFlor and Usó-Juan (2006) propose a framework for the teaching of requests and suggestions (6Rs Approach), which consists of six steps: (1) researching, where learners are provided with pragmatic instruction and they are asked to collect examples of requests and suggestions in their mother tongue and provide sociopragmatic information about the examples; (2) reflecting, in which learners work on the data they have collected and complete some awareness-raising questions that focus on pragmalinguistics and sociopragmatics; (3) receiving, in which learners are provided with explicit instructions on the pragmalingusitic realisation of requests and suggestions, and also encouraged to compare them with the data they previously collected; (4) reasoning, which involves learners’ completion of awareness-raising activities to deepen their sociopragmatic understanding; (5) rehearsing, which focuses on learners’ provision of output opportunities, specifically by means of controlled and free activities; and (6) revising, in which learners receive the teacher’s feedback. Focusing also on request-mitigating devices, Usó-Juan and Martínez-Flor (2008) present a learner-based method to teach requestmitigating devices which is divided into three main stages. In the first stage, learners’ exploration, learners are provided with opportunities to examine pragmalinguistic and sociopragmatic features that influence the performance of request-mitigating devices by means of awareness-raising activities. In the second stage, learners’ production, learners are encouraged to practise request-mitigating devices in the oral and written mode. Finally, in the third stage, learners’ feedback, learners are provided with feedback from their peers about their performance. A different speech act is addressed by Beltrán-Palanques (2012), namely refusals. The instructional approach consists of five different phases. In the first phase, comparing, the teacher explains the notions of

106

Chapter Four

social distance and social status and how these aspects might affect language use. Then, learners are asked to reflect on the realisation of refusal strategies in the target language and in their mother tongue in order to draw some comparisons. The second phase, exploring, focuses on learners’ exposure to pragmatic input, for example, film scenes. In this phase, leaners are also expected to answer some questions regarding the input presented. This might be followed by a whole class discussion on the information collected. In the third phase, selecting, learners are provided with examples of refusals at the discourse level in order to reflect on sociopragmatic features. The fourth phase involves producing, in which learners take part in written and oral communicative activities. Finally, the last phase, providing feedback, involves a whole class discussion and individual interviews. By revisiting the above pedagogical frameworks, we might come to the conclusion that one of the distinguishing features is the adoption of consciousness-raising activities that are then transformed into production activities. Nevertheless, features of interaction and negotiation seem not to be addressed in some of the frameworks reviewed, or at least, explicitly addressed. Concerning this, Félix-Brasdefer (2006), in line with BardoviHarlig and Hartford (2005) and Kasper (2006), argues for teaching pragmatic aspects, particularly speech acts, at the level of discourse following a conversation analytic approach. Félix-Brasdefer (2006) proposes a pedagogical framework that involves three pedagogical units: communicative actions and cross-cultural awareness, doing conversation analysis in the classroom, and communicative practice and feedback. The first part attempts to develop learners’ cross-cultural awareness of the act of refusing in Spanish and English. Then, in the second part, the author focuses on the structure of conversation, thereby pointing to specific characteristics of conversation such as sequences, actions, and linguistic and non-verbal expressions. Finally, learners are engaged in communicative spoken activities, specifically role-plays. This framework points to the importance of conversation features, in which both verbal and non-verbal aspects are included. However, additional approaches that explicitly integrate pragmatic aspects with paralanguage features and extra-linguistic elements would be useful. It is also important to note that further research into teaching speech acts from a conversation analytic approach is needed to expose learners to actual language use, and more specifically, speech acts (cf. Cohen and Ishihara 2010). Although the pedagogical frameworks of Martínez-Flor and BeltránPalanques (2013, 2014) do not follow a conversation analytic approach, they point to the importance of interaction and negotiation of speech acts.

The Distinctive Multimodal Nature of Pragmatic Competence

107

Moreover, as previously reported, Judd’s (1999) proposal also emphasises the importance of taking into account the performance of speech acts at the level of discourse. Martínez-Flor and Beltrán-Palanques (2013) present a pedagogical framework for the teaching of refusals from an inductive and deductive approach, which included four different phases. The first phase involves pragmalinguistic awareness, in which learners’ attention is drawn to pragmalinguistic nature of refusals. In so doing, learners can be engaged in awareness-raising activities that attempt to make them reflect on different linguistic resources that can be used to perform refusals. The second phase, sociopragmatic awareness, involves learners’ attention to sociopragmatic features of refusals. Specifically, learners should centre on how sociopragmatic features such as context, participants’ role, gender, and politeness aspects affect pragmalinguistic choice and use. In the third phase, pragmatic production, learners are asked to put into practice the aspects covered in the previous phases. Specifically, they perform a set of role-play activities that are designed drawing on film scenes. After acting out the roles, learners participate in retrospective verbal report sessions in order to gain further insights into their pragmatic competence development, similarly to the interview suggested by Beltrán-Palanques (2012). The last phase, feedback on pragmatic production, involves the teacher’s and peer’s feedback on performance. A similar approach is adopted by Martínez-Flor and Beltrán-Palanques (2014), focusing on the speech act of apologies at the discourse level, taking into account the role of politeness. The authors divide their teaching approach into five different stages. In the introductory phase, introduction: input presentation, learners are provided with authentic-like input, namely film scenes in which apologies are performed in different situations, and they are asked to reflect on the context and the utterances employed in those examples. The second stage, awareness-raising activities, focuses on learners’ attention to how sociopragmatic features affect pragmalinguistic realisation. After this, metapragmatic explanations are provided for pragmalinguistic and sociopragmatic features associated with an apology. Then, learners are engaged in two different communicative practice activities, specifically a written activity, in which they are asked to elaborate a dialogue, and an oral activity, i.e. a role-play. The last stage, final revision, feedback, relates to learners’ feedback on performance. Other salient features of the pedagogical frameworks reviewed above are the nature of input and the context of instruction. Concerning input, it is important to note that it is the quality (not the quantity) of exposure that might have an effect. In fact, learners’ exposure to authentic input or authentic-like input that is of good quality and appropriate for them is

108

Chapter Four

necessary to provide them with opportunities to focus on the use of pragmatic features. Another important aspect to take into account is the context of instruction, either SL or FL. In SL contexts, learners might have more opportunities to be exposed to the target language, but also to employ it in different communicative encounters, differently from FL contexts, where input and output opportunities are limited mostly to the language classroom. Moreover, output opportunities could be also determined by the type and the quality of the activities that learners accomplish, as well as whether they are relevant and meaningful for learners. Finally, feedback is another important issue that has been emphasised, which can be also combined, for example, with discussions (Félix-Brasdefer 2006), interviews (Beltrán-Palanques 2012), or retrospective verbal reports (Martínez-Flor and Beltrán-Palanques 2014).

4.2. A multimodal approach for teaching speech acts The pedagogical models above presented, although of paramount interest for the teaching of speech acts, do not account for a multimodal perspective of pragmatic competence. Drawing from the above literature and the nature of multimodal pragmatic competence, I argue not only for incorporating pragmatic competence following a conversation analytic approach (Félix-Brasdefer 2006; Cohen and Ishihara 2010), but also for exploring pragmatics from a multimodal perspective, thereby integrating not only the linguistic side of pragmatics, but also paralanguage and extralinguistic elements. Hence, I present a multimodal pragmatic conversational framework that attempts to promote learners’ multimodal pragmatic competence in the FL context which consists of six stages: (1) prior knowledge, (2) explanation, (3) speakers’ intentions, (4) revision and explicit instruction, (5) communicative spoken activities, (6) retrospective verbal reports, and (6) feedback and discussion. In the first stage, prior knowledge, teachers should focus on learners’ prior knowledge of pragmatic knowledge in general and, in particular, the speech act in question (Olshtain and Cohen 1991), as well as the different modes that can be used to communicate in written and spoken language. This stage might be seen as a diagnostic stage, since teachers obtain information concerning both pragmatic knowledge and insights into different modes of communication. This can be performed orally in the form of a whole class discussion. Once teachers have posed some questions and learners have provided their responses, they can move to the second stage, exploration, which focuses on how people perform the pragmatic feature in the target language, similarly to Usó-Juan and

The Distinctive Multimodal Nature of Pragmatic Competence

109

Martínez-Flor (2008). To do so, teachers might use either authentic data or authentic-like data, for example, from audio-visual materials (c.f. BeltránPalanques 2011, 2012; Martínez-Flor and Beltrán-Palanques 2013, 2014). In this stage, learners should be asked to notice how people organise their conversation, focusing not only on the speech act chosen, but also on the different sequences performed to understand how communication is constructed and developed over the different turns, and on speakers’ paralinguistic and extra-linguistic elements. After exploring the data, teachers should engage learners in a whole class discussion. If necessary, they may guide the discussion in order to draw learners’ attention to relevant aspects, such as turn-taking, pauses, filled pauses, backchannels, politeness, paralinguistic elements, or gestures. The third stage is speakers’ intentions, which also involves learners’ provision of input. The purpose is to examine speakers’ intentions by focusing on what is verbally said and non-verbally conveyed in order to see whether there is any kind of interplay among modes. To do so, different scenes could be projected, for example, two scenes in which various social parameters are portrayed, such as interaction between two friends vs. interaction between a boss and his/her employee. Learners should focus on sociopragmatic and pragmalinguistic features to understand how these two elements are combined to convey the speech act studied, as well as the interplay between modes by pointing out paralanguage and extra-linguistic elements. This stage can be performed orally and in pairs or small groups, with a final whole class discussion on the issue. The fourth stage, revision and explicit instruction, focuses on recapitulating what has been done in the previous stages to ascertain whether learners have properly understood the aspects that might affect speech act performance, and how other elements that are not verbally produced take part in the communicative event. At this point, explicit instruction regarding the sociopragmatic and pragmalinguistic features of the speech act chosen would be required. In addition, teachers might explicitly focus on the nature of paralinguistic and extra-linguistic elements by pointing out the role that they play in communication. It is worth noticing that these elements might vary across cultures and that they could be affected, for example, by the social norms of the target language. These activities are then followed by the production, introspection, and feedback stages. Communicative spoken activities involve the spoken production of the pragmatic aspects chosen by means of role-play tasks (cf. Judd 1999; Beltrán-Palanques 2011, 2012; Martínez-Flor and BeltránPalanques 2013, 2014). In order to understand how social norms affect language use, teachers might design at least two scenarios involving

110

Chapter Four

different social parameters. The scenarios could, for instance, involve the following situations with different parameters of social status and power: (1) two friends, who share the same social status and power; and (2) a university student who participates in an internship programme and his/her supervisor. The first role-play might be acted out by two learners, who both perform the role of friends. However, for the second role-play scenario described, I would argue for an interaction between a learner and an unknown person, for example, a native speaker or native-like speaker of the target language, or another language teacher. If this situation is not possible, two learners might complete the task together. My purpose in having an unknown person or a teacher is to expose learners to a more authentic-like situation; if the person performing the role of the supervisor is someone different, learners might perhaps elaborate their discourse in a different manner, rather than if they interacted with a peer. However, this does not imply that a communicative activity among peers is not appropriate or meaningful. The spoken performance should be done in different time slots, so that each pair performs the two situations in a more relaxed manner. Spoken interactions should be video-recorded, in order to examine both the verbal performance and the non-verbal aspects. After completing the role-play task, learners should take part in an introspection activity with their teacher, i.e., retrospective verbal reports, in which each pair visualises their performance and comments on their performance. To do so successfully, teachers should prepare beforehand some specific questions regarding learners’ pragmatic, paralinguistic, and extra-linguistic production. If the second role-play task is performed by a person who is not a learner, he/she might also take part in the retrospective verbal reports, as his/her data would be also significant to understand the performance. Once learners have completed the oral communicative activity and the introspection stage, they take part in the final stage: feedback and discussion. In this case, feedback consists of teachers’ and peer’s feedback. This can be done either in the written form, by providing learners with a sheet in which they are asked specific questions about learners’ performance, or orally by means of a whole class discussion. It is important to note that, in some cases, learners’ consent might be required not only to record them, but also to show the data collected to their peers or other teachers. The purpose of the feedback stage is to explore learners’ performance by focusing particularly on how they organise the conversation pragmatically, pointing out aspects such as turn-taking, backchannels, speech acts, and their responses, and non-verbally, by referring to paralinguistic and extra-linguistic elements where attention

The Distinctive Multimodal Nature of Pragmatic Competence

111

should be paid to the interplay with verbal language and how these elements are performed taking into account social norms and politeness. Therefore, this stage might serve to provide learners with feedback about their multimodal pragmatic competence, as special emphasis is given to pragmatic competence both from a purely linguistic view and from a nonverbal view. Finally, there could be a whole-class discussion dealing with the aspects covered in the different stages and learners’ interest concerning multimodal pragmatic competence. The above-described pedagogical framework suggests a set of stages that might be followed to integrate not only speech acts in the language classroom to promote pragmatic competence, but also to make learners become aware of the importance of other aspects such as paralinguistic and extra-linguistic elements that are part of the communicative event. The first three stages, prior knowledge, explanation, and speakers’ intentions, are characterised as awareness-raising activities. The fourth stage, revision and explicit instruction, serves to recapitulate the aspects covered in the previous stages and to provide learners with explicit instruction. The fifth stage, communicative spoken activities, is the production phase, and finally, the two remaining stages, retrospective verbal reports and feedback and discussion, refer to learners’ introspection of performance and peers’ and teachers’ feedback. Following a conversation analysis perspective, learners can become aware of how language is structured and what the role of speech acts in communication is. Furthermore, this approach also serves to explore paralinguistic and extra-linguistic elements, since learners can explore, over the different turns, how these elements interplay with verbal performance.

5. Final Remarks This chapter has presented an overview of the nature of communicative competence by discussing several of its most influential models. As shown, it seems that there is a need to revisit the communicative competence model from a multimodal perspective in order to integrate all the different modes that are employed to communicate, beyond only words. I have focused particularly on pragmatic competence, which involves both pragmalinguistic and sociopragmatic features (Leech 1983; Thomas 1983), and which, broadly speaking, focuses on how people employ language according to social norms. This is of paramount interest for language teaching, since learners should become aware of how to use language socially. Moreover, I have pointed out that it is also important to account for paralinguistic and extra-linguistic elements, as they are part of

112

Chapter Four

the communicative event, thereby arguing for a multimodal view of pragmatics, in other words, multimodal pragmatic competence. This perspective integrates elements that are linguistically-oriented in addition to the paralinguistic and extra-linguistic elements of communication. When performing speech acts, or other pragmatic aspects, speakers go beyond words to utilise other resources that are at their disposal. Considering all these aspects, I have presented a multimodal pragmatic conversational approach for teaching speech acts. As shown, this approach involves six different stages that focus on awareness-raising activities, explicit instruction, production, introspection and feedback. Although I focused particularly on speech acts, this approach might be followed to integrate different pragmatic aspects, such as implicatures or pragmatic formulas.

References Bachman, Lyle. F. 1990. Fundamental Considerations in Language Testing. Oxford: Oxford University Press. Bachman, Lyle F., and Andrew D. Cohen. 1998. “Language testing - SLA interfaces: An update.” In Interfaces between Second Language Acquisition and Language Testing Research, edited by Lyle F. Bachman, and Andrew D. Cohen, 1–31. Cambridge: Cambridge University Press. Bachman, Lyle. F., and Adrian S. Palmer. 2010. Language Assessment in Practice. Oxford: Oxford University Press. —. 1996. Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press. Bardovi-Harlig, Kathleen, and Beverly S. Hartford. 2005. Interlanguage Pragmatics: Exploring Institutional Talk. Mahwah: Lawrence Erlbaum Associates. Barron, Anne. 2003. Acquisition in Interlanguage Pragmatics: Learning how to do Things with in a Study Abroad Context. Amsterdam: John Benjamins. Beltrán-Palanques, Vicente. 2012. “Integrating the speech act refusals in the instructed setting.” Babel Afial: Aspectos de Filología Inglesa y Alemana, 85–100. —. 2011. “Analysing refusals in films.” In Current Trends in Anglophone Studies: Cultural, Linguistic and Literary Research, edited by Javier Ruano, María Jesús Fernández, Miriam Borham, María José Díez, Santiago Bautista, Pedro Álvarez, and Blanca Garcia-Riaza, 70–84. Salamanca: Aquilafuente, Ediciones Universidad de Salamanca.

The Distinctive Multimodal Nature of Pragmatic Competence

113

Brown, Penelope, and Stephen C. Levinson. 1987. Politeness: Some Universals in Language Usage. Cambridge: Cambridge University Press. Canale, Michael. 1983. “From communicative competence to communicative language pedagogy.” In Language and Communication, edited by Jack C. Richards and Richard W. Schmidt, 2–27. London: Longman. Canale, Michael, and Merril Swain. 1980. “Theoretical bases of communicative approaches to second language teaching and testing.” Applied Linguistics 1(1):1–48. Celce-Murcia, Marianne. 2007. “Rethinking the role of communicative competence in language teaching.” In Intercultural Language Use and Language Learning, edited by Eva Alcón and Maria Pilar Safont, 41– 57. Dordrecht: Springer. —. 1995. “The elaboration of sociolinguistic competence: Implications for teacher education.” In Linguistics and the Education of Language Teachers: Ethnolinguistic, Psycholinguistic, and Sociolinguistic Aspects, edited by James E. Alatis, Carolyn A. Straehle and Maggie Ronkin, 699–710. Washington D.C.: Georgetown University Press. Celce-Murcia, Marianne, Zoltán Dörnyei, and Sarah Thurrell. 1995. “A pedagogical framework for communicative competence: A pedagogically motivated model with content specifications.” Issues in Applied Linguistics 6(2):5–35. Chomsky, Noam. 1965. Aspects of the Theory of Syntax. Cambridge, MA: MIT Press. Cohen, Andrew D., and Noriko Ishihara. 2010. “Discourse, interaction, and language corpora.” In Teaching and Learning Pragmatics. Where Language and Culture Meet, edited by Noriko Ishihara and Andrew D. Cohen, 166–185. London: Routledge. Félix-Brasdefer, J. César. 2006. “Teaching the negotiation of multi-turn speech acts. Using conversation-analytic tools to teach pragmatics in the classroom.” In Pragmatics and Language Learning, Vol. 11, edited by Kathleen Bardovi-Harlig, J. César Félix-Brasdefer and Alwiya S. Omar, 165–197. National Foreign Language Resource Center: University of Hawai’i at Manoa. Habermas, Jürgen. 1970. “Towards a theory of communicative competence.” In Recent Sociology, No. 2, edited by Hans Pete Dreitzel, 115–148. New York: MacMillan. Hymes, Dell. 1972. “On communicative competence.” In Sociolinguistics, edited by John B. Pride and Janet Holmes, 269–293. Harmondsworth: Penguin.

114

Chapter Four

Jordan, Geoff. 2004. Theory Construction in Second Language Acquisition. Amsterdam: John Benjamins. Jewitt, Carey. 2013. “Multimodal teaching and learning.” In The Encyclopedia of Applied Linguistics, edited by Carol A. Chapelle, 4109–4114. Oxford: Blackwell. Judd, Elliot L. 1999. “Some issues in the teaching of pragmatic competence.” In Culture in Second Language Teaching and Learning, edited by Eli Hinkel, 152–166. Cambridge: Cambridge University Press. Kasper, Gabriele. 2006. “Speech acts in interaction: Towards discursive pragmatics.” In Pragmatics and Language Learning, Vol. 11, edited by Kathleen Bardovi-Harlig, J. César Félix-Brasdefer and Alwiya S. Omar, 281–314. National Foreign Language Resource Center: University of Hawai’i at Manoa. Leech, Geoffrey. 1983. Principles of Pragmatics. London: Routledge. Martínez-Flor, Alicia, and Vicente Beltrán-Palanques. 2014. “The role of politeness in apology sequences: How to maintain harmony between speakers.” Elia: Estudios de Lingüística Inglesa Aplicada 43–66. —. 2013. “Teaching refusal strategies in the foreign language classroom: A focus on inductive-deductive treatments.” Journal of English Studies 11:41–67. Martínez-Flor, Alicia, and Esther Usó-Juan. 2006. “A comprehensive pedagogical framework to develop pragmatics in the foreign language classroom. The 6R approach.” Applied Language Learning 16(2):36– 64. Olshtain, Elite, and Andrew D. Cohen. 1991. “Teaching speech act behaviour to nonnative speakers.” In Teaching English as a Second or Foreign Language, edited by Marianne Celce-Murcia, 154–165. Boston: Heinle & Heinle. —. 1983. “Apology: Speech act set.” In Sociolinguistics and Language Acquisition, edited by Nessa Wolfson and Elliot L. Judd, 18–35. Rowley MA: Newbury House. Oxford, Rebecca L. 2001. “Language learning styles and strategies.” In Teaching English as a Second or Foreign Language, edited by Marianne Celce-Murcia, 539–366. Boston: Heinle & Heinle. Phakiti, Aek. 2008. “Construct validation of Bachman and Palmer’s (1996) Strategic Competence Model over time in EFL reading tests.” Language Testing 25(2):237 272. Rickheit, Gert, Hans Strohner, and Constanze Vorwerg. 2008. “The concept of communicative competence.” In Handbook of Communication

The Distinctive Multimodal Nature of Pragmatic Competence

115

Competence, edited by Gert Rickheit and Hans Strohner, 15–62. Berlin: Mouton de Gruyer. Roever, Carsten. 2011. “Testing of second language pragmatics: Past and future.” Language Testing 28(4):463 481. —. 2009. “Teaching and testing pragmatics.” In Michael H. Long and Catherine J. Doughty, The Handbook of Language Teaching, 560–577. Oxford: Blackwell Publishing Ltd. Rose, Kenneth, and Gabriele Kasper. 2001. Pragmatics in Language Teaching [Cambridge Applied Linguistic Series]. New York: Cambridge University Press. Royce, Terry D. 2007. “Multimodal communicative competence in second language contexts.” In New Directions in the Analysis of Multimodal Discourse, edited by Terry D. Royce and Wendy L. Bowcher, 361– 390. Mahwah, Jew Jersey: Lawrence Erlbaum Associates, Inc. Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson. 1974. “A simplest systematics for the organisation of turn-taking for conversation.” Language 50(1):696–735 Savignon, Sandra J. 1983. Communicative Competence: Theory and Classroom Practice. Reading, MA: Addison-Wesley. Schachter, Jacquelying. 1990. “Communicative competence revisited.” In The Development of Second Language Proficiency, edited by Birgit Harley, Patrick Allen, Jim Cummins and Merril Swain, 39–49. Cambridge: Cambridge University Press. Stenglin, Maree, and Rick Iedema. 2001. “How to analyze visual images: A guide for TESOL teachers.” In Analysing English in a Global Context. A Reader, edited by Anne Burns and Caroline Coffin, 194– 208. London: Routledge. Taguchi, Naoko. 2009. “Pragmatic competence in Japanese as a second language: An introduction.” In Pragmatic Competence, edited by Naoko Taguchi, 1–18. Berlin: Mouton de Gruyter. Thomas, Jenny. 1983. “Cross-cultural pragmatic failure.” Applied Linguistics 4(2):91–112. Usó-Juan, Esther, and Alicia Martínez-Flor. 2008. “Teaching learners to appropriately mitigate requests.” ELT Journal, 62(4):349–357. —. 2006. “Approaches to language learning and teaching: Towards acquiring communicative competence through the four skills.” In Current Trends in the Development and Teaching of the Four Language Skills, edited by Esther Usó-Juan and Alicia Martínez-Flor, 3–25. Berlin: Mouton de Gruyter. Widdowson, Henry.1989. “Knowledge of language and ability for use.” Applied Linguistics, 10(2):128–137.

CHAPTER FIVE DIALOGUE INTERPRETING AS A MULTIMODAL ACTIVITY IN COMMUNITY SETTINGS ELENA DAVITTI UNIVERSITY OF SURREY, UK

1. Introduction Dialogue Interpreting (DI), i.e., face-to-face interpreter-mediated activity performed bidirectionally in a variety of settings (e.g., public services, business), has been the object of analytical scrutiny over the last couple of decades. Empirical research based on authentic data has soon come to recognise the importance of shifting the focus away from the interpreter to also encompass how all parties-at-talk contribute to the co-construction of meaning, thus achieving a more refined understanding of the communicative dynamics characterising these scenarios. Nevertheless, most emphasis has been placed on the verbal dimension of this type of interaction and little is known about how the integrated use of verbal and embodied resources (e.g., gaze, gesture, proxemics, etc.) affects the unfolding of interpretermediated events. Irrespective of the specific research traditions in which they are grounded, approaches to the study of DI relying entirely on talk to investigate specific interactional practices will always provide a partial, if not distorted, view of what is actually happening during an interpretermediated encounter. In turn, this will continue to perpetuate the myth of DI as a language-transfer activity, which is still widespread among laypeople as well as some practitioners, where talk is considered to be the dominant resource being used by participants to co-construct meaning and reach common understanding. Building on these premises, the present paper argues that the adoption of a multimodal approach to the study of DI is not only necessary but indispensable to its advancement both as a professional practice and as a field of enquiry. The facts that (monolingual) “communication is inherently

Dialogue Interpreting as a Multimodal Activity in Community Settings 117

multimodal” (O’Halloran 2011: 21) and that “human social interaction involves the intertwined cooperation of different modalities” (Stivers and Sidnell 2005: 1) are widely recognised and uncontroversial. When it comes to interpreter-mediated interaction, however, a relatively limited body of research has adopted a more holistic approach to its investigation. While the reasons behind this may vary (for instance, limited access to video-recorded data, methodological complexity of multimodal analysis, etc.), the time has come to pave the way for a multimodal turn in DI studies. To this end, it is necessary to clarify, first and foremost, why a multimodal approach can contribute to a more refined and systematic framing and understanding of DI dynamics. Following on from this will be the question of how this can be done in the most efficient way possible. The latter question addresses more methodological issues that are not discussed in this paper. To start with, interpreter-mediated dialogue situations will be characterised as multiparty and multimodal activities. Through reference to selected extracts and relevant literature, this paper will attempt to exemplify and systematise the merits of broadening the scope of analysis of DI ‘beyond the verbal’. Conclusions will reflect on how a more nuanced and enriched view of interactional dynamics obtained through multimodal lenses can inform everyday practice and be integrated into interpreter education and training.

2. DI as a Special Instance of Multiparty Interaction Interpreter-mediated interaction in dialogic situations has started to be scrutinised in research only in recent times, namely towards the end of the 1990s. Thanks to the conceptualisation of these events as triadic exchanges (Mason 2001), interpreters have come to be recognised as active and visible parties in the interaction taking part in a communicative pas de trois (Wadensjö 1998), and handling not only differences in language practices, but also in cultural and institutional norms, values, and expectations. In particular, thanks to empirical analysis of real-life interpreter-mediated interaction, it has become clear that the presence of an interpreter in a communicative event has an impact on its structure and on the way it unfolds. This has led to a reconceptualization of the interpreters’ role as going beyond that of simple language machines to integrate translation, coordination, and intercultural mediation (cf. Wadensjö 1998; Davidson 2000; Roy 2000; Mason 2001; Angelelli 2004; Baraldi and Gavioli 2012).

118

Chapter Five

In the light of this, interpreter-mediated interactions in dialogic settings can be considered as instances of multiparty interaction, where interpreters have agency and interact with other parties like interdependent cogs of an engine (Turner et al. 2009), thus actively contributing to the unfolding of the interaction. Language transfer is only one of the many layers of their multitasking activity, which also includes coordinating the communicative encounter, with implications for the negotiation of participation frameworks and responsibilities. The merits of empirical research on DI in unveiling that interpreter’s positioning (Mason 2009) is not static and predetermined, but dynamic and shifting on a moment-by-moment basis according to the interactional behaviour of all participants are beyond dispute. Nevertheless, all claims made are constrained by the fact that, irrespective of the conceptual and methodological framework adopted, they are mostly reliant on the analysis of vocal-aural resources alone. As a result, however in-depth and valuable, findings only provide us with a partial snapshot of what is actually happening in their context of occurrence. It is therefore timely to revisit such phenomena with a view to gaining a more holistic understanding of how they come about, as well as of their interactional implications. The urge to broaden the scope of analysis to encompass visuo-spatial resources also comes from the need to revisit codes of conduct, which are often based on a narrow and prescriptive view of interpreters’ do’s and don’ts in interaction. Issues such as the seating arrangement that is most conducive to an ‘illusion’ of direct interaction between the other participants in a meeting are at times discussed, but without being solidly grounded in any empirical observation of real-life events (e.g., NCIHC 2004; NHANES 2006; IMIA 2007—see also Pokorn 2015). There is therefore an urgent need to enrich current studies and carry out new ones that account for resources other than talk. This paper will attempt to highlight some of the merits that the adoption of a multimodal approach has yielded when applied to authentic interpreter-mediated interaction.

3. On the Multimodal Nature of DI Multimodality as a term refers to a number of theoretical and methodological approaches that look at “representation, communication and interaction as something more than language” (Jewitt 2013: 1). The integration of verbal and embodied resources to investigate meaning-making in (monolingual) human interaction has been scrutinised in a wide range of settings and through different methodological lenses. Different frameworks for multimodal

Dialogue Interpreting as a Multimodal Activity in Community Settings 119

analysis exist and they can be linked to three main paradigms: Discourse Analysis (cf. Hodge and Kress 1988; Kress and van Leeuwen 1996, 2001; Baldry and Thibault 2001, 2006; O’Halloran 2004); Interaction Analysis (cf. Scollon and Scollon 2004; Norris 2004; Norris and Jones 2005); and Conversation Analysis (cf. Goodwin 2000; Kendon 2004; Deppermann 2013; Hazel, Mortensen, and Rasmussen 2014; Mondada 2014). The common denominator across all these studies is that meaning is considered to be the result of the interplay between a number of resources, namely the participants’ verbal and embodied resources as well as the physical environment in which they interact. These are key variables in that they add new layers of complexity and suggest new meanings and interpretations that an analysis focused exclusively on participants’ verbal output would not allow. Building on analysis of authentic interaction, these studies have provided enriched snapshots of actual practice and unveiled different ways in which communication breakdowns or collaborative solutions to interactional problems are brought about. DI, as a special instance of multiparty interaction, is likely to be managed multimodally due to its socially situated and multilingual nature (Pasquandrea 2011), where participants share the same physical—or virtual in video-mediated interpreting—interactional space. Empirical studies on DI using micro-analytical approaches to the investigation of audio-recorded data have revealed a number of frequent phenomena occurring in interpreter-mediated interaction and have provided evidence that interpreting is not a word-for-word language transfer activity. Among these phenomena are, for instance, expanded renditions, omissions, code switching, minimal responses, and hesitations. Nevertheless, research on how such phenomena are constructed multimodally and what their impact is on the unfolding of the interaction and on the participatory framework is relatively scant. With the exception of some pioneer studies (cf. Lang 1978; Apfelbaum 1998), DI literature has seen the development of research adopting a multimodal approach only since the early 2000s. Existing studies on multimodal DI have been carried out in a variety of settings (e.g., medical, pedagogical, police, and legal) and have borrowed analytical tools from various multimodal frameworks linked to the three paradigms presented above: for instance, Ticca (2010), Pasquandrea (2011, 2012), Davitti (2012), Davitti and Pasquandrea (2013, 2014, 2016); Licoppe, Licoppe, and Veyrier (2016) have built on multimodal Conversation Analysis; Krystallidou (2014) has combined Conversation Analysis with Norris’ framework for multimodal Interaction Analysis; Monteoliva García (forthcoming) has accounted for multimodal features in a framework

120

Chapter Five

combining Conversation Analysis and Relevance Theory. This has contributed to a sketchy and unsystematic landscape where it is difficult to establish comparability across findings. Nonetheless, despite several methodological shortcomings, which will not be addressed in this paper, each study has highlighted interesting aspects of interpreter-mediated interaction, thus showing how enriching a multimodal perspective can be for our understanding of DI dynamics. The merits of adopting a more holistic approach will be discussed and exemplified in the next section.

4. Merits of a Multimodal Approach to DI Studies adopting a multimodal approach to DI to date have looked at how a multimodal perspective can provide more in-depth and comprehensive insights into phenomena that have been found to characterise interpretermediated interaction. In particular, they have focused on how embodied resources can be used to complement, integrate, and replace talk with a view to triggering, disambiguating, and modulating specific conversational moves or the lack thereof (e.g., in the case of gaps in conversation). Studies encompassed under this body of literature can be arranged according to two main axes: on the one hand, studies that, while accounting for the activities of the triad, shed light on the nuances of interpreting processes, by clarifying the import of specific moves performed by the interpreter, such as interpreter-initiated expansions (Davitti 2012, 2013, 2015). On the other hand, there are studies that have focused on broader communicative dynamics involving all participants and how they vary in relation to different set-ups, spatial distributions, handling of artefacts, and visual ecologies (cf. Davitti and Pasquandrea 2016 and Ticca and Traverso 2016 on the (non-)collaborative achievement of specific actions in pedagogical and social services settings, respectively). Ultimately, a common denominator across all studies is their attempt to investigate the implications of the specific phenomena analysed on the participatory framework (e.g., inclusion/exclusion of the parties-at-talk), which is key for an efficient and smooth unfolding of the interaction. A multimodal approach can therefore be useful to highlight challenges for the interpreting process (i.e., translation dimension), as well as for the management of communication (i.e., coordination dimension).

Dialogue Interpreting as a Multimodal Activity in Community Settings 121

4.1. Reversing the perspective: What embodied features can reveal about interpreting dynamics This section aims to temporarily reverse the perspective on data analysis that has been traditionally adopted by empirical research on DI. In other words, instead of using the verbal dimension as the main springboard for analysis of interpreter-mediated interaction, the focus will be on the embodied dimension only, leaving temporarily aside what is actually being said. Adopting an approach that discards the verbal dimension of interaction will, on the one hand, show that the same issues and limitations will arise as when focusing on the verbal dimension only. On the other hand, however, this exercise will show the interesting and diverse patterns revealed by a close look at the embodied dimension of interaction, thus supporting the argument that integrating both dimensions would clearly enhance and refine our understanding of such dynamics. To this end, the present section shows the results of the comparative analysis of three studies which have focused on gaze and head orientation in three different interpreter-mediated scenarios. The rationale for focusing on such embodied dynamics is that gaze and head shifts play a key role in interaction as contextualisation cues (Gumperz 1982) and offer new positionings that can be taken up, rejected, or ignored by each participant. In particular, the findings analysed are from Davitti (2012), who has worked on instances of parent-teacher interaction in pedagogical settings.1 For comparative insights, they are discussed with reference to the research of Bot (2005), who has devoted some sections of her doctoral thesis to gaze behaviour in mediated mental healthcare encounters between therapists and patients, and Mason (2012), who has worked on mediated immigration hearings involving immigration officers and asylum seekers. The expression ‘gaze patterns or dynamics’ is used to encompass also the general head/body orientation of the parties. The various patterns emerging from the different datasets are presented and contrasted from the perspective of each one of the parties-at-talk.

1

The corpus consists of three video-recorded, authentic parent-teacher meetings for a total of approximately 4h (see Fig. 5.2). PTM1 and PTM3 were recorded in Italy, while PTM2 was recorded in the UK. These encounters feature one professional interpreter, one mother from a migrant background and two teachers. The language pair in all cases is English and Italian. The extract analysed in the second half of the paper is from PTM1. The English gloss for the Italian turns was provided by the researcher.

122

Chapter Five

4.1.1. Overall orientation dynamics compared: The ‘primary’ parties To start with, the focus will be placed on the ‘primary’ parties in interaction, namely the institutional party (i.e., teacher, therapist, and immigration officer) and the non-institutional one (i.e., mother, patient, and asylum seeker). Each stakeholder will be looked at separately to highlight some orientation patterns when in speaking and listening mode. It will ultimately be shown that such patterns deviate from the default assumption that participants look at each other throughout the encounter, thus suggesting interesting dynamics of inclusion and exclusion in relation to the interpreter. Starting from the institutional party, in the pedagogical dataset it was noticed that teachers rarely maintain sustained gaze with mothers when speaking; rather, they tend to either constantly shift their gaze from the mother to the interpreter or to exclusively address the latter (see Fig. 5-1). This embodied behaviour is accompanying and, at times, enhancing what happens at the verbal level, where teachers are found to use devices such as third person pronouns to talk about the mother while addressing the interpreter directly.

Fig. 5-1. Mutual gaze between INT and T in PTM

Despite the relatively small dataset, this multimodal pattern emerged fairly clearly across all interactions in the pedagogical setting analysed. It may be due to a number of reasons: for instance, the lack of an established common practice about how to communicate through an interpreter,

Dialogue Interpreting as a Multimodal Activity in Community Settings 123

which, on the contrary, may characterise other (more formalised) settings. Furthermore, the semi-formal and collaborative nature of parent-teacher meetings combined with the perception of interpreters’ role as mediators may create the favourable conditions for teachers to naturally address interpreters directly, and treat them as their direct interlocutors, thus multimodally ratifying interpreters as fully-fledged participants. This pattern differs considerably from Mason’s (2012) and Bot’s (2005) studies, where the immigration officer and the therapist show a tendency to look at the asylum seeker and at the patient, respectively, when speaking, and their gaze is only punctuated by looks away or brief glances at the interpreter. This suggests a certain ’effort’ on the part of the institutional parties in these interactions to display continuous engagement with the ‘laymen’, and disengagement with the interpreter, contrary to what was found in the parent-teacher meetings dataset. No mutual gaze was found to be normally established between mothers and teachers when the latter is in listening mode, either during mothers’ turns, while being interpreted, and while listening to the interpreter rendition of mothers’ talk. In the first case (i.e., during the mother’s turns), mutual gaze is at times established when the mother starts talking, but then either the teacher would constantly shift her gaze from the interpreter to the mother, or would look away; the teachers often engaged in schisms or parallel activities, such as talking to their colleague if present in the room or reading the school report, and thus at least partially self-excluding from the participatory framework. In the second and third cases (i.e., while being interpreted and listening to the interpreter rendition of the mother’s talk), the teachers in the dataset were found to mostly look at the interpreter, often waiting for a signal that they could take or resume the floor (which is understandable considering the frequently multi-turn nature of their contributions—cf. section 4.2 for a more in-depth discussion). These gaze configurations may also be influenced by the fact that, in the data, the mothers mainly gaze at the interpreter both when speaking and listening to the interpreter rendition of teacher talk, thus treating the interpreter, rather than the teacher, as their privileged interlocutor. Once again, this scenario is totally different from Mason’s (2012) and Bot’s (2005) studies; when in listening mode, the predominant pattern is that the immigration officer and the therapist tend to look at the asylum seeker and at the patient, respectively, while listening to their turns and while being interpreted. In the first case (i.e., when the asylum seeker and the patient are talking), the immigration officer’s and the therapist’s behaviour resemble ordinary monolingual patterns, where listeners focus mainly on the speaker, casting occasional glances at the other participants.

124

Chapter Five

In the second case (i.e., while the institutional representative is being interpreted), the only non-institutional data available are from Bot, who points out that the therapist ends up gazing at a party who is listening to someone else, given that the patient mostly looks at the interpreter while the latter is interpreting the therapist’s talk. Bot (2005: 136) suggests that gaze held towards a party who is looking at someone else is “unnatural”, according to Goodwin’s (1981) observations on gaze patterns in monolingual interaction, and has two main interactional implications. Firstly, the therapist “treats the interpreter as a disengaged non-participant, different from the other participants in the dialogue” (Bot 2005: 137); secondly, this gaze pattern may be “felt as an intrusive act” (2005: 137), given that the therapist seems to watch the patient while the latter is listening to someone else in order to monitor his/her responses and reactions to their words. Last but not least, my analysis of the pedagogical dataset also reveals that the teachers tend to gaze at the interpreter while listening to their rendition of the mother’s talk, thus displaying a gaze pattern which is closer to natural, ordinary monolingual conversation, when mutual gaze is normally established between speakers and recipients. This is similar to Mason’s (2012) study, where the immigration officer engages more, at least through embodied resources, with interpreters during their rendition of asylum speakers’ speech, thus ratifying interpreters as fully-fledged participants. However, Mason notices that, even though the orientation of the interpreter’s head is mostly towards the immigration officer when either speaking or listening to him/her, direct gaze at him/her is strictly limited. This behavioural pattern is similar to Lang’s (1978: 235) finding that his court interpreter “signalled his status as neutral”. Conversely, Bot (2005) points out that therapists tend not to reciprocate interpreters’ gaze when the latter are interpreting the patient’s turns, thus displaying a certain tendency to disengage from the interpreter. As suggested by Mason’s (2012) and Bot’s (2005) studies, the immigration officer and therapist’s behaviour in speaking and listening mode may be the result of training guidelines or, more simply, of a regulated behaviour which has gained consensus within their community of practice. This point is also shared by Wadensjö (2001), who investigated the impact of various seating arrangements on mediated therapeutic encounters. In her study, “three interpreters mentioned they had been specifically instructed by the therapist to avoid looking at the patient and to avoid moving their head back and forth between the parties to mark the anticipated exchange of turns” (Wadensjö 2001: 83). Conversely, my data show less mutual engagement between the primary

Dialogue Interpreting as a Multimodal Activity in Community Settings 125

parties than previous studies, which suggests the lack of a similar regulated behaviour, and a tendency to display gaze/orientation patterns which are closer to the dynamics of spontaneous (monolingual) multiparty interaction. Shifting our attention now to the non-institutional party, mothers mainly look at the interpreter when speaking, listening to the interpreter rendition of teacher talk, and listening to the interpreter rendition of their own turns; in this case, they would also at times look away. Instances of mothers gazing at teachers directly are, therefore, very limited in my dataset. This is only partially in line with previous studies. Because Mason (2012) works with what he calls “impoverished data” (2012: 180), where no recordings are provided for the vulnerable, non-institutional party, no data are available for asylum seekers’ behaviour. However, patients in speaking mode were reported by Bot (2005) to sometimes address the therapist, sometimes address the interpreter, or look away, especially when talking about stressful issues. Furthermore, patients who normally held mutual gaze with the interpreter when the latter is rendering the therapist’s talk showed a tendency not to treat interpreters as an ordinary speaker when the latter was rendering their talk, but to focus on the listener (i.e., the therapist) or to turn their gaze away. Having highlighted some of the main patterns emerging from the ‘primary parties’’ behaviour, attention will now shift onto the interpreters and their embodied behaviour. In particular, the comparative analysis will show how findings in the pedagogical setting differ from findings from studies in the other settings in that interpreters physically display a very pronounced involvement, mostly through the establishment of direct, mutual gaze with the other parties. 4.1.2. Overall orientation dynamics compared: The interpreter Shifting our focus to interpreters, they display very engaged gaze behaviour in the pedagogical dataset by looking directly at the party who is talking or to whom they are listening. This seems to support the argument that they play an active role in and carefully monitor what is going on in the interaction. It also suggests that their attitude may not be constrained by the institution in which they are working. In his study, Mason (2012) shows that such a communicative style displayed by interpreters is an exception to the more general ‘rule’, according to which interpreters, both in listening and speaking mode, tend to look away and avert mutual gaze. As Mason points out, asylum seekers especially do not receive much sign of engagement from either the immigration officer or the interpreter. In my study, mothers equally do not receive much sign of

126

Chapter Five

engagement from teachers, while interpreters display direct engagement with them by establishing mutual gaze. Gaze held towards participants while talking suggests that the interpreter is acting as a ratified participant, as is the case in ordinary monolingual conversation, where a speaker is expected to gaze at their interlocutor (Goodwin 1980). The effect is that interpreters, whether deliberately or not, seemingly compensate for the lack of embodied engagement shown by the institutional representatives, thus increasing the interactional proximity with the mother. Interpreters’ embodied way of addressing the parties directly when in speaking or listening mode is closer to Bot (2005)’s findings, where they mainly gaze directly at their interlocutor when rendering patient or therapist talk, or shift their gaze from one to the other, possibly in the attempt to simultaneously facilitate mutual engagement between the two primary parties. Considering that interpreting does not happen in a vacuum, elements related to the specific ecology of action (Heath et al. 2002) in which the meeting takes place need to be integrated in this type of observation. In particular, the specific seating arrangement of the parties may contribute to facilitating or hindering certain gaze dynamics. In the pedagogical setting analysed, the amount of mutual gaze between one of the parties and the interpreter largely exceeds the amount of held mutual gaze between teachers and mothers, irrespective of whether the interpreter sits between the mother and the teachers, as in PTM1 and PTM3 (Fig. 5-2), or next to the mother and opposite the teachers, as in PTM2 (Fig 5-2). As a result, the patterns highlighted do not seem to be significantly affected by the participants’ seating arrangement, thus suggesting that the very engaged attitude displayed by interpreters may not be directly due to contingent ‘environmental’ factors. Nevertheless, different seating arrangements made gaze shifts more or less visible markers of turn organisation and monitoring. Shifts from the mother to the teachers and vice versa can be more easily pinpointed in the encounters where the interpreter is sat inbetween the parties (i.e., PTM1 and PTM3), given that they entail head movement. In contrast, in PTM2, where the interpreter occupies a more lateral position, gaze shifts from one party to the other are often simply performed by eye movement.

Dialogue Interpreting as a Multimodal Activity in Community Settings 127

Fig. 5-2. Seating arrangements of each PTM in the pedagogical dataset

This also seems to be the case in Mason (2012)’s studies, where the seating arrangement is very similar across the five interviews recorded. The primary parties sit opposite each other; interpreters are close to the immigration officer’s desk, their chair is slightly angled and, if they look ahead, they are gazing midway between the institutional representative and the asylum seeker, while being physically closer to the latter. On the contrary, Bot’s (2005) data present a different seating arrangement per group of sessions (six sessions in total), thus gaze behaviour may have been more affected by the different configurations. In particular, the seating arrangement of what Bot calls group A is that of an equilateral triangle with the patient in the middle. Group C is characterised by an equal distance between participants with, in one case, the patient in the middle, while in another case, the interpreter in the middle. The most different seating arrangement is the one of Group B, where the interpreter sits slightly behind the therapist, opposite the patient. Although the issue of how seating arrangement may affect gaze patterns is raised in both studies, it is, however, not explored in any depth. To my knowledge, Wadensjö (2001) is the only study which has focused on this research issue, finally suggesting that “interpreters seem to be more ‘out of the way’ when they are present in a communicative radius, than if they are present in the room but without access to, and not immediately available in a common, triadic, focused interaction” (2001: 83). More recently, Davitti and Pasquandrea (forthcoming) have also compared two sequences from pedagogical settings in which participants perform the same action involving the handling of an artefact, i.e., reading and signing a school report, in different seating arrangements, with very different implications in terms of management of the participatory framework. In particular, the article has provided empirical evidence of how participation “is a dynamic concept, subject to inherent shifts towards more or less collaborative configurations and affected by different spatial arrangements, object availability and by a range of multimodal and multilingual resources mobilised by participants” (Davitti and Pasquandrea, forthcoming).

128

Chapter Five

To sum up, this section has singled out the embodied dimension of interpreter-mediated interaction, highlighting some gaze and head orientation configurations which emerged as typical across a dataset of pedagogical mediated interaction as opposed to previous studies on asylum seeking and therapeutic settings. The claims made so far are based on the observation of the gaze behaviour displayed by all the participants in the pedagogical sequences selected. This is in line with the interactionist approach embraced by recent empirical research on DI, which stresses the importance of accounting for the behaviour of all parties-at-talk in order to gain a comprehensive picture of what is happening in interaction and a better understanding of the interactional impact of specific verbal and embodied moves. Furthermore, the comparison of recurrent gaze patterns across datasets gathered from different settings provides empirical evidence of how engagement and visibility can also be displayed through resources other than talk. Such macro-level observation is useful to highlight the variability of gaze patterns and show the existence of dynamics which seem to be rejected a priori by many codes of conduct, such as, for instance, the tendency displayed by primary parties not to look at each other but at the interpreter, thus treating him/her as a fully-fledged participant. Nevertheless, in order to explore in detail what positionings are taken up by interpreters and how they shift during the interaction, the analysis of gaze/head/body orientation, as well as of any other embodied resources, needs to be combined with the analysis of verbal behaviour to achieve a more thorough understanding of the interactional impact of specific interpreters’ moves.

4.2. Enriching the perspective: How the integration of verbal and embodied resources can fine-tune interactional analysis As mentioned previously, the starting point for my analysis was the impression, stemming from the data, that interpreters overall display a very engaged attitude during the parent-teacher meetings in which they are involved. This emerged from their embodied behaviour, as suggested in section 4.1.2, which showed different patterns with respect to those unveiled by studies focusing on other settings. Nevertheless, looking at embodied features alone did not allow us to gain any more insights into specific shifts in positioning on a moment-by-moment basis. A more finegrained, multimodal look is therefore necessary to pinpoint exactly those moments where such an attitude manifests itself more prominently, how this comes about, and its implications on the unfolding of the interaction.

Dialogue Interpreting as a Multimodal Activity in Community Settings 129

4.2.1. Interpreter-mediated parent-teacher meeting: A micro-analytical look at the verbal dimension of interaction To this end, it is worth going back to an analysis of the verbal dimension of interaction to see what results it may yield, and subsequently ‘enhance’ it by integrating it with other layers of analysis. In my dataset, one of the ways in which active engagement comes to the surface at the verbal level is through interpreter self-initiated expansions. Expansions are a common phenomenon inherent in the interpreting process: there is a body of research that has shown what/where/how interpreters add and their impact on the unfolding of the interaction (cf. Wadensjö 1998; Angelelli 2012; Baraldi 2012; Merlini and Favaron 2005). However, little attention has been devoted to their investigation in conjunction with embodied features to try and capture the nuances and interactional implications of these moves (among the exceptions are Davitti 2012, 2013, 2015, as pointed out in section 3). To characterise their nature, the expansions under discussion present some regularities. Firstly, they are of an evaluative nature (i.e., offering some kind of assessment on the point under discussion) and secondly, such expansions are mostly produced towards the end of the interpreter rendition, thus acting as turn-increments (Schegloff 2001), which were not originally uttered by the teacher. The multimodal analysis conducted to gain more insights into this phenomenon will focus on an extract from the same corpus of authentic video-mediated PTMs analysed above in relation to gaze/head orientation dynamics, which has constituted the basis of my research on interactional moves in DI. In the particular encounter analysed, the participants are sat around a table in a classroom as per Fig. 5-3.

Fig. 5-3. Seating arrangement in PTM1

Chapter Five

130

Extract 1 below shows the transcript of the verbal component of the sequence under scrutiny, where one of the teachers (T1) is providing an assessment of the overall child performance, particularly in relation to his language proficiency in Italian (see Appendix for transcription conventions). This is potentially a delicate point in the encounter, as the child comes from a migrant family and has very little knowledge of the language of the host country, which causes him to struggle at school. Given the collaborative nature of the encounter, in order to succeed in her task, T1 needs to establish and maintain a common ground the mother (M), balancing concern and support, criticism, and encouragement. The interpreter (INT) is therefore presented with the very complex task of producing a rendition that enables the parties to communicate efficiently and achieve agreement while preventing conflicts and misunderstandings. 1

T1

qui: è questo è quell- (.) praticamente abbiamo de-: detto adesso here is this is what

2

practically what we have

said now

che è stato inserito all’inizio dell’anno che ha he was included at the beginning of the year that he

3

instau[rato buoni rapporti* anche con noi: (.) è sempre molto established good relationships also with us

4

(he) is always very

[ah °buoni rappor-°

INT

(he) has good relation-

5

T1

carino non conosceva nessuna parola di italiano ma è migliorato nice (he) did not know any words of Italian

6

but he has improved

ma rimangono difficoltà (.) non accetta di svolgere attività but there are still difficulties (he) does not accept to do activities

7

[diver*se dai compagni (.) eh different from his mates

8 9

INT T1

[ah: segue le attività della classe aiutato dalle insegnanti e le sue he follows the activities of the class with the help of the teachers and his

10

difficoltà non sono nella comprensione ma nella conosce[nza* difficulties are not in understanding

11

INT

12

T1

quindi una volta che lui arriverà a conoscere la lingua (.) cioè so

13

but in knowledge

[enza once

he manages to know the language I mean

non ha nel sen- non ha problemi di apprendimento o di (he) does not have any problems in learning

14

or

comprensione perché le cose che già adesso comprende understanding because the things he already understands

Dialogue Interpreting as a Multimodal Activity in Community Settings 131 15

le fa [bene (he) does them well

16

T2

17 18

INT

[fa bene does well

so here it’s (.) written that actually he has been (.) this is the first year he is in Italy and is the first year that he is in this class but he

19

has (.) he gets along (.) really good with his mates (.) and (.) with

20

the teachers as well (.) when he came of course he didn’t know (.)

21 22

any words (.) of Italian so he had (.) he had difficulties with the language but now (.) the knowledge of the language is getting

23

better (.) [is improving* (.) but there are still many

24 25

difficulties [in the* communication [si yes

26 27 28 29 30 31 32

M M (1.49) INT

[exactly mhm: even though he has this kind of problems he doesn’t wanna (.) he doesn’t accept to do (.) different activities from his mates (.) so: the teacher (.) they will help him in a special way but he will: (.) actually do (.) what the others do (.) with the help of course of the teachers (.) an:d (.) their they (.) eh the teacher here she just (.)

33 34

point out that his problem (.) is not (.) a comprehension problem in terms of (.) learning (.) but (.) in communication and (.)

35

language (.) so once he will have overcome the language

36

problem he won’t have any problem and he will be good (.) he

37

(certainly) will catch up with all (.) with th- with the

38

children of his age

39

M

°(nodding)°

An initial exploration, exclusively drawing on the verbal dimension of interaction, would enable us to gain some insights into the unfolding of the sequence, but would not make us fully appreciate the interactional dynamics at play. Some observations can be made: (1) the structure of the teacher’s talk consists of a multi-unit turn framing a potentially negative assessment of the child in a positive way, possibly in the attempt to prevent conflicts and present issues in a constructive manner (for a comparison between the structure of

132

Chapter Five

monolingual and interpreter-mediated PTMs, see Davitti 2015). T1’s delivery is interspersed with the interpreter’s acknowledgment tokens (e.g., line 8) and other-turn completion morphemes (e.g., line 11) which signal understanding of what is being said but do not act as turn-taking devices. Turn shift only happens at line 17, when the interpreter self-selects to take the floor and start her rendition. It is impossible to establish from the transcript how this transition happens, as there is no verbal element signalling it; (2) at line 27, a gap of 1.49 seconds is produced. This silence can be considered significant and is normally dispreferred in conversation in that it delays the completion of the action and needs to be repaired by other-selection (Schegloff 1988; Heritage 1984). At this point, in the sequential development of talk, the teacher or the mother could have self-selected to take the floor, although what actually happens is that the interpreter holds the floor to continue the rendition. As the teachers do not speak English, it seems unlikely that they did not take the floor to let the interpreter finish the rendition which, at that point, is incomplete. Another question therefore arises concerning how the interpreter manages to maintain the floor so smoothly. (3) the interpreter rendition (lines 17-39) is constructed as a formulation of the source utterance; in terms of interactional content, it could be considered complete at lines 34-35 (…and language). However, it is independently expanded in rendition final position through a series of evaluative assessments that had not been originally produced by the teacher. The positive and reassuring tone of such expansion has the effect of upgrading the source utterance (Davitti 2013) and to produce new, conclusive meaning that is attributed to the teacher (lines 37-38). A number of questions arise here as well; one is about the appropriateness of the explicitation that the child will definitely overcome all the problems and catch up with his peers. While a descriptive, microanalytical framework does not allow an investigation of the intentions behind specific actions, it seems plausible to argue that the integration of participants’ embodied behaviour into the analysis could enrich our understanding of how the sequence unfolds, as well as of the interactional implications of the moves described.

Dialogue Interpreting as a Multimodal Activity in Community Settings 133

4.2.2.Interpreter-mediated parent-teacher meeting: interactional phenomena through multimodal lenses

Revisiting

In this section, I will revisit the three interactional moments identified above through multimodal lenses with a view to establishing what (if any) insights can be gained from such enhanced analysis. Regarding point (1), the verbal transcript only lets us infer the presence of a school report. This emerges verbally from the deictic used by the teacher at line 1 (qui… è questo è quell- praticamente abbiamo detto) as well as from the interpreter’s start of the rendition at line 17 (so here it’s written that…). As discussed in Davitti and Pasquandrea (forthcoming), the presence of an artefact and the way it is handled in interaction in combination with other multimodal resources (i.e., speech, gaze, and posture) “can be actively constructed as a sense-making device and exert a different influence on the unfolding interaction, thus leading to different shifts and reconfigurations of the participation framework” (Davitti and Pasquandrea, forthcoming). Between lines 1-16, the teacher is reading aloud and commenting on a part of the school report that presents a written, general assessment of the child’s attitude and performance in the classroom. The actual artefact, namely a school report, is laying at the centre of the table, and the teacher is alternating moments in which she reads out from it with spontaneous comments that elaborate upon the content of the report (particularly 1216). This is clearly marked multimodally by gaze/head shift from the artefact to the interpreter, who is sat between the teacher and the mother. The artefact here becomes the centre of collective attention (Mondada 2007), with the teacher pointing at the relevant parts in the documents and expanding upon its content.

Fig. 5-4. Multimodal details of lines 10-11 of Extract 1

134

Chapter Five

While this way of handling the artefact is conducive to the teacher retaining interactional control (Davitti and Pasquandrea, forthcoming), one possible downside of such practice is shown by the teacher’s tendency to produce long turns, which may be challenging for the interpreter to handle. Such long turns require chunking, meaning that the interpreter needs to find a way to (temporarily) gain the floor, insert his rendition to the mother, and give the floor back to the teacher without disrupting the flow of the conversation. This task is inherent in the interpreter activity as a coordinator and involves constant decision-making about where and how to intervene. In the extract analysed, turn management is handled entirely through embodied resources by the interpreter: the overlap between the teachers at lines 15-16 seems to be used by the interpreter as a transition relevant place (Schegloff 1992) to produce a clear head turn and gaze away shift to the mother, thus signalling embodied self-selection for the floor. This embodied practice characterises the whole event, where head/gaze turn are effectively used by the interpreter to coordinate turn management with the teacher. This multimodal practice results in a mostly seamless alternation of turns between the two parties, resembling a sort of dance-like act which was not negotiated a priori, but seemingly resulted from mutual monitoring and contextual adjustment. The teacher’s way of handling her turns, alternating reading out from the report with personal remarks, also increases the multitasking activity of the interpreter, who not only has to sight translate the content of the report, but also has to keep track of the additional information produced by the teacher. Regarding point (2) (i.e., the gap in conversation at line 27), we can gain insights from multimodal analysis about how the interpreter manages to retain the floor. At this point, the interpreter is looking down at the report, pointing to the relevant part she needs to sight translate for the mother. This embodied behaviour seems to enhance the filled pause produced at line 28 (mhm:), which can be interpreted as a turn-holding device while gathering her thoughts before starting the second part of the rendition. At this point, the teacher is looking at the interpreter, monitoring what is going on. The mother is looking down at the report. The tacit way of handling turns multimodally through gaze/body is momentarily suspended here, but the multimodal behaviour displayed by the interpreter shows a clear coordinating act on her part to handle the long turn originally uttered by the teacher through retaining the floor while chunking. Last but not least, regarding point (3), (i.e., the expansion produced by the interpreter), we can get further insights by looking at the embodied behaviour displayed by all participants. The verbal transcript did show

Dialogue Interpreting as a Multimodal Activity in Community Settings 135

some acknowledgment tokens produced by the mother (lines 25 and 26) during the interpreter’s renditions of the first chunk of the teacher’s talk (lines 17-24). After the gap, no more acknowledgment tokens are verbally produced by the mother. At line 35, however, the interpreter reorients her gaze towards the mother just before starting her expansion (so once he will have overcome…). This is a recurrent pattern found in the data: when combined with the production of this specific type of expansion, held gaze seems to do more than display engagement. It also seems to serve the expressive function of reassuring while acting as a device to achieve inclusion and to pursue a favourable response from the mothers, thus enabling the interaction to unfold smoothly. This is ultimately achieved at line 39, when the mother nods, thus displaying acknowledgement of what is being said. The verbal component of this interpreter-promoted move seems to exert an explicitating and reassuring function on the mother. However, multimodal analysis accounting for embodied behaviour suggests another possible function of such move, i.e., trying to elicit a positive response— which is perceived as missing—while enabling the interaction to move on. As discussed elsewhere (Davitti 2012, 2013), the interactional implications can be diverse, and only multimodal analysis can help us disambiguate them. In some cases, it was noticed how expansions independently produced by interpreters in this specific sequential environment, coupled with gaze away shifting back to the teacher at the end of their production, can signal the closing of a large, multi-turn unit. If explicit agreement is not established between the parties before such a shift occurs, the move can actually result in ‘stifling’ the mother’s voice instead of empowering her by creating opportunities to voice her opinions and concerns. This confirms that multimodal resources can be in actual conflict with one another, suggesting contrasting meaning, and therefore need to be analysed as an integrated whole in order to better understand the import of specific interactional moves. In other cases, such as the one analysed in this paper, the interpreter independently creates an opportunity for the mother to express her thoughts by asking whether she has understood everything and she has any questions. The question (not reported here) follows the mother’s slight nodding (line 39) and leads to the latter taking the floor and expressing her concern about other issues related to her child’s performance. In extract 1, the extremely engaged attitude and coordinating role performed by the interpreter emerge very prominently through her verbal behaviour and are enhanced by embodied display (e.g., by addressing the mother directly while holding mutual gaze with her). The subtle apparatus by which

136

Chapter Five

interpreters regulate turn taking by using head and gaze shifts as turnyielding devices in this case does not create an opportunity for the teacher to intervene, and shows the tacit and subtle but forceful power of embodied features in shaping the unfolding of a communicative event. To recap, analysis of this extract through multimodal lenses has provided a more nuanced view of how interaction is co-constructed, which is often the result of tacit agreement, as is the case in the data where the interpreter has never worked with the same teachers or mother before. It has also shed light on silent moments of interaction, disambiguating gaps in conversation where a response is perceived as missing, where verbal analysis only would probably lead to a misleading interpretation of what is going on and, consequently, of how tasks and responsibilities are shared. Finally, the analysis has also shown how different modalities work together to constitute coherent courses of action, how they are adjusted to the specific ecology of action in which participants operate, and what relations can be established among them, which vary on a continuum from mutual enhancement to conflict.

5. Conclusion This paper has attempted to exemplify some of the merits of a multimodal approach to gain insight into interpreter-mediated interaction. In particular, through analysis of authentic data and reference to existing literature that has looked at interpreter-mediated interaction through multimodal lenses, the paper has pointed out some of the opportunities brought about by a more holistic look at these communicative events which integrates features other than talk. One of the main points to stress in order for DI studies to fully embrace this approach is to start considering interaction as inherently multimodal, and meaning as the result of the integration of embodied resources bearing the same weight and interactional force. In other words, the supremacy of the verbal component over other dimensions of interaction needs to be questioned and the dichotomy between vocal-aural and visuo-spatial resources needs to be reconceptualised as a complex union where different resources displayed by different parties-at-talk interacting with one another and constituting an integrated system to be analysed as a whole. In order for research in this field to take a step forward and achieve a more comprehensive understanding on interpreter-mediated interaction, a solid methodological and analytical framework is needed. Discussion about how to bring multimodality into dialogue with other disciplines continues to produce “provocative issues for qualitative research

Dialogue Interpreting as a Multimodal Activity in Community Settings 137

methodology” (Dicks et al. 2011: 227). When it comes to DI in particular, existing multimodal studies have adopted very different approaches, which have resulted in difficulties when comparing and contrasting findings. A serious methodological discussion is therefore vital in order to develop a multimodal approach that can adjust to and shed light on the specificities of triadic exchanges. The ultimate goal of multimodal analysis of DI interaction is not to discover universally valid patterns of multimodal behaviour, but rather to revisit existing practices and perhaps enrich the repertoire with new ones. For instance, gaze/head orientation has been found to play a key regulatory function in monolingual interaction (cf. Kendon 1967, 1990; Goodwin 1981; Rossano 2012) and is often indicated in training as a ‘device’ that interpreters can use strategically to regulate turn-taking. Nevertheless, the implications of such moves need to be analysed in context, based on the exact point in interaction in which they are performed, what are the other resources at play, what happens before, and what comes next (cf. Davitti and Pasquandea 2014, for a discussion of how to use multimodal analysis of authentic interpreter-mediated interaction in the classroom). Descriptive studies accounting for the same practice across different datasets are therefore needed to collect evidence of how different moves are performed in local practices, what may contribute to bringing them about, and what their interactional impact is. Such a body of research, which is still in its infancy, can greatly contribute to increased self-awareness and can have long-term implications in terms of training and education by enhancing the multimodal literacy of all the stakeholders involved in an interpretermediated event, with a view to learning how to work efficiently in such encounters. Revisiting codes of conduct on the basis of findings from empirical data can also be another important achievement of this literature, for instance, in relation to seating arrangement and spatial positioning, which remain hotly debated topics in the field. Beyond a pure analytical interest in deepening our knowledge of the dynamics of such complex encounters, multimodality applied to DI can have a real impact on practice. To conclude, van Leeuwen (2015: 588) has recently put forward a set of questions in relation to multimodality and the practice of education which are useful to frame the steps ahead to develop such integration. These questions can be rephrased and adjusted to multimodality and DI, and can provide the same roadmap for future work: .

x

To which degree and how can we use findings from multimodal analysis to educate future professionals and clients for critical awareness as well as for access to the specific skills needed to get by in interpreter-mediated communication?

Chapter Five

138 x x

What multimodal skills can be developed and passed on to practitioners to enrich training? How can this renewed awareness be used in education to preserve as well as innovate skills, knowledge, and forms of creative practice?

These questions are not easy to answer, especially if we consider the current gap between research and education in interpreting studies, but they need to be addressed if we aim to foster the progress of DI, both as research and practice.

Appendix: Transcription Conventions (1.5)

(.)

:

pause (in seconds and tenths of seconds) lengthened sound

°text°

quiet volume

°°text°°

micro-pause (shorter than 0.5 seconds) abrupt interruption of talk very quiet volume

[

start of overlapping talk

*

end of overlapping talk

(text)

dubious text

-

References Angelelli, Claudia V. 2004. Revisiting the Interpreter’s Role: A Study of Conference, Court, and Medical Interpreters in Canada, Mexico and United States. Amsterdam: Benjamins. —. 2012. “Challenges in interpreters’ coordination of the construction of pain.” In Coordinating Participation in Dialogue Interpreting, edited by Claudio Baraldi and Laura Gavioli, 244–260. Amsterdam: Benjamins. Apfelbaum, Birgit. 1998. “Instruktionsdiskurse mit Dmolmetscherbeteiligung. Aspekte der Turnkonstruktion un Turnzuweisung.” In Neuere Entwicklungen in der Gesprächforschung, edited by Alexander Brock and Martin Hartung, 11–36. Tübingen: Narr. Baldry, Anthony, and Paul J. Thibault. 2001. “Towards multimodal corpora.” In Corpora in the Description and Teaching of English, edited by Guy Aston and Lou Burnard, 87–102. Bologna: CLUEB. —. 2006. Multimodal Transcription and Text Analysis. A Multimedia Toolkit and Coursebook. London and New York: Equinox.

Dialogue Interpreting as a Multimodal Activity in Community Settings 139

Baraldi, Claudio. 2012. “Interpreting as dialogic mediation: The relevance of expansions.” In Coordinating Participation in Dialogue Interpreting, edited by Claudio Baraldi and Laura Gavioli, 297–326. Amsterdam: Benjamins. Baraldi, Claudio, and Laura Gavioli. 2012. Coordinating Participation in Dialogue Interpreting. Amsterdam: Benjamins. Bot, Hanneke. 2005. Dialogue Interpreting in Mental Health. Amsterdam and New York: Rodopi. Davidson, Brad. 2000. “The interpreter as institutional gatekeeper: The sociolinguistic role of interpreters in Spanish-English medical discourse.” Journal of Sociolinguistics 4(3):379–405. Accessed: July 25, 2016. doi:10.1111/1467-9481.00121 Davitti, Elena. 2012. “Dialogue interpreting as intercultural mediation: Integrating talk and gaze in the analysis of mediated parent-teacher meetings”. PhD diss., University of Manchester, UK. —. 2013. “Dialogue interpreting as intercultural mediation: Interpreters’ use of upgrading moves in parent-teacher meetings.” Interpreting 15(2):168–199. Accessed August 1, 2016. doi: 10.1075/intp.15.2.02dav. —. 2015. “Interpreter-mediated parent-teacher talk.” In Linking Discourse Studies to Professional Practice, edited by Lubie Grujicic-Alatriste, 176–200. Clevedon: Multilingual Matters. Davitti, Elena, and Sergio Pasquandrea. 2013. “Interpreters in intercultural communication: How to modulate the impact of their verbal and nonverbal practices?” In Best Practice Catalogue. Accessed: August 19, 2016. http://bridge-it.communicationproject.eu/Publications.htm. —. 2014. “Enhancing research-led interpreter education: An exploratory study in Applied Conversation Analysis.” The Interpreter and Translator Trainer 8(3):374–398. Accessed August 15, 2016. doi: 10.1080/1750399X.2014.972650. —. Forthcoming. “Embodied participation: What multimodal analysis can tell us about interpreter-mediated encounters in pedagogical settings.” Special Issue of Journal of Pragmatics. Accessed August 21, 2016. http://dx.doi.org/ 10.1016/j.pragma.2016.04.008. Deppermann, Arnulf. 2013. “Conversation analytic studies of multimodal interaction.” Special Issue of Journal of Pragmatics, 46(1):1–7. Dicks, Bella, Rosie Flewitt, Lesley Lancaster, and Kate Pahl. 2011. “Multimodality and ethnography: Working at the intersection.” Introduction: Qualitative Research Special Issue: Multimodality and

140

Chapter Five

Ethnography: Working at the Intersection 11(3):227–237. Accessed: August 21, 2016. doi: 10.1177/1468794111400682 Goodwin, Charles. 1980. “Restarts, pauses, and the achievement of mutual gaze at turn-beginning.” Sociological Enquiry 20(3-4):272–302. doi:10.1111/j.1475-682X.1980.tb00023.x. —. 1981. Conversational Organization. Interactions Between Speakers and Hearers. New York: Academic Press. —. 2000. “Action and embodiment within situated human interaction.” Journal of Pragmatics 32(10):1489–1522. doi:10.1016/S0378-2166(99)00096-X. Gumperz, John J. 1982. Discourse Strategies. Cambridge: Cambridge University Press. Hazel, Spencer, Kristian Mortensen, and Gitte Rasmussen. 2014. “A body of resources – CA studies of social conduct.” Special issue of Journal of Pragmatics 65:1–9. Heath, Christian, Marcus Sanchez Svensson, Jon Hindmarsh, Paul Luff, and Dirk Vom Lehn. 2002. “Configuring awareness.” Computer Supported Cooperative Work 11:317–47. Heritage, John. 1984. Garfinkel and Ethnomethodology. Cambridge: Polity Press. Hodge, Robert, and Gunther Kress. 1988. Social Semiotics. Cambridge: Polity Press. IMIA. 2007. “Medical interpreting standards of practice.” Accessed August 15, 2016. www.imiaweb.org/uploads/pages/102.pdf 2007. Jewitt, Carey. 2013. The Routledge Handbook of Multimodal Analysis. London: Routledge. Kendon, Adam. 1967. “Some functions of gaze direction in social interaction.” Acta Psychologica Psychologica 26:22–63. —. 1990, Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press. —. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Kress, Gunther, and Theo van Leeuwen. 1996. Reading Images: The Grammar of Visual Design. Routledge, London. —. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold. Krystallidou, Demi. 2014. “Gaze and body orientation as an apparatus for patient inclusion into/exclusion from a patient-centred framework of communication.” The Interpreter and Translator Trainer 8(3):399– 417. Accessed: August 15, 2016. doi:10.1080/1750399X.2014.972033.

Dialogue Interpreting as a Multimodal Activity in Community Settings 141

Lang, Rainer. 1978. “Behavioural aspects of liaison interpreters in Papua New Guinea: Some preliminary observations.” In Language, Interpretation and Communication, edited by David Gerver and Wallace H. Sinaiko, 231–244. New York: Plenum Press. Licoppe, Christian, and Clair-Antoine Veyrier. Forthcoming. “How to show the interpreter on screen? The normative organization of visual ecologies in multilingual courtrooms with video links.” Special Issue of Journal of Pragmatics. Mason, Ian. 2001. Triadic Exchanges. Studies in Dialogue Interpreting. Manchester: St. Jerome. Mason, Ian. 2009. “Role, positioning and discourse in face-to-face interpreting.” In Interpreting and Translation in Public Service Settings: Policy, Practice, Pedagogy, edited by Raquel de Pedro Ricoy, Isabelle A. Perez and Christine W.A. Wilson, 52–73. Manchester: St. Jerome. —. 2012. “Gaze, positioning and identity in interpreter-mediated dialogues.” In Coordinating Participation in Dialogue Interpreting, edited by Claudio Baraldi and Laura Gavioli, 177–200. Amsterdam: Benjamins. Merlini, Raffaela, and Roberta Favaron. 2005. “Examining the ‘Voice of Interpreting’ in speech pathology.” Interpreting 7(2):263–302. Accessed: August 15, 2016. doi: 10.1075/intp.7.2.07mer. Mondada, Lorenza. 2007. “Multimodal resources for turn-taking: Pointing and the emergence of possible next speakers.” Discourse Studies 9(2):195–226. Accessed: August 15, 2016. doi: 10.1177/1461445607075346 —. 2014. “The local constitution of multimodal resources for social interaction.” Journal of Pragmatics 65:137–156. Accessed: August 15, 2016. doi: http://dx.doi.org/10.1016/j.pragma.2014.04.004 Monteoliva García, Patricia. Forthcoming. The impact of language transparency upon communication and participation in interpretermediated police interviews with Spanish-speaking suspects in Scotland, PhD Thesis, Heriot-Watt University. NCIHC. 2004. “A national code of ethics for interpreters in health care.” Accessed August 15, 2016. http://www.ncihc.org/assets/documents/publications/NCIHC%20Natio nal%20Code%20of%20Ethics.pdf. NHANES. 2006. “National Health and Nutrition Examination Survey – Interpretation guidelines.” Accessed August 18, 2016. https://www.cdc.gov/nchs/data/nhanes/nhanes_07_08/Interpretation_G uidelines.pdf.

142

Chapter Five

Norris, Sigrid. 2004. Analyzing Multimodal Interaction: A Methodological Framework. London: Routledge. Norris, Sigrid, Rodney H. Jones. 2005. Discourse in Action: Introducing Mediated Discourse Analysis. London: Routledge. O’Halloran Kay L. 2004. Multimodal Discourse Analysis. London/New York: Continuum. Pasquandrea, Sergio. 2011. “Managing multiple actions through multimodality: Doctors’ involvement in interpreter-mediated interactions.” Language in Society 40:455–481 Accessed August 18, 2016. doi:10.1017/S0047404511000479 —. 2012. “Co-constructing dyadic sequences in healthcare interpreting: A multimodal account.” New Voices in Translation Studies 8:132–157. Pokorn, Nike K. 2015. “Positioning.” In Routledge Encyclopedia of Interpreting Studies, edited by Franz Pöchhacker, 312–314. London: Routledge. Rossano, Federico. 2012. “Gaze in conversation.” In The Handbook of Conversation Analysis, edited by Jack Sidnell and Tanya Stivers, 308– 329. Malden, MA: Wiley-Blackwell. Roy, Cynthia B. 2000. Interpreting As a Discourse Process. New York: Oxford University Press. Schegloff, Emanuel. 1988. “On an actual virtual servo-mechanism for guessing bad news: A single case conjecture.” Social Problems 35(4):442–457. —. 1992. “To Searle on conversation: A note in return.” In (On) Searle on conversation, edited by Herman Parret and Jef Verschueren, 113–128. Amsterdam: John Benjamins. —. 2001. “Conversation Analysis: A project in process – ‘Increments’”, Forum Lecture in the 2001 LSA Summer Institute, University of California, Santa Barbara. Scollon, Ron, and Suzie Wong Scollon. 2004. Discourse in Place. Language in the Material World. London: Routledge. Stivers, Tanya, and Jack Sidnell. 2005. “Multimodal interaction” Semiotica 156(1-4):1–20. Ticca, Anna Claudia. 2010. “Configurazioni diadiche nel dialogo mediato medico-paziente.” In La Mediación Lingüística y Cultural y su Didáctica: Un Nuevo Reto Para la Universidad, edited by Maria Carreras i Goicochea and María Enriqueta Pérez Vázquez, 102–119. Bologna: Bononia. Ticca, Anna Claudia, and Véronique Traverso. Forthcoming. “Forms of participation in bilingual interactions: The organization of activities

Dialogue Interpreting as a Multimodal Activity in Community Settings 143

around translating, interpreting, and mediating documents in a social center.” Special Issue of Journal of Pragmatics. Turner, Graham, Kyra Pollitt, Andrew John Merrison, Bethan Davies, and Gary Quinn. 2009. “The pragmatics of dialogue interpreting between sign and speech.” Paper presented at the 11th IPrA - International Pragmatics Conference, Melbourne, July 12-17, 2009. Van Leeuwen, Theo. 2015. “Multimodality in education: Some directions and some questions.” TESOL Quarterly 49(3):582–589. Accessed August 19, 2016. doi: 10.1002/tesq.242 Wadensjö, Cecilia. 1998. Interpreting As Interaction. London: Longman. —. 2001. “Interpreting in crisis: The interpreter’s position in therapeutic encounters.” In Triadic Exchanges: Studies in Dialogue Interpreting, edited by Ian Mason, 71–85. Manchester: St. Jerome.

PART II – MULTIMODALITY IN SPECIALISED DISCOURSE DOMAINS AND GENRES

CHAPTER SIX GESTURES IN MOTION IN TED TALKS: TOWARDS MULTIMODAL LITERACY SILVIA MASI UNIVERSITY OF PISA, ITALY

1. Introduction Gestures are used spontaneously, and often unconsciously, to illustrate or emphasise what is said (Hostetter 2011). As a consequence, the potentially facilitating role of gestures in foreign language learning and teaching has attracted increasing interest over the last decade (cf. Crawford Camiciottoli 2007; Gullberg and McCafferty 2008; Busá 2010; Walsh 2010; Littlemore et al. 2012; O’Halloran, Tan, and Smith 2016). These research avenues are also supported by experimental evidence from a psycholinguistic perspective (Macedonia and von Kriegstein 2012), which has highlighted, for example, the important role of gestures in facilitating memorisation in foreign language learning. The relation between gesture and metaphor (Cienki and Müller 2008) and the role of metaphoric gestures in educational contexts have also been extensively investigated (cf., among others, Littlemore and Low 2006; McCafferty and Stam 2008). Metaphors are pervasive in language and thought (Lakoff and Johnson 1980), they vary across languages, and the gestures accompanying them may reflect such differences (Cienki 2008). Littlemore et al. (2012), for instance, underline the importance of metaphoric gestures used by native English instructors in academic tutorials addressed to international students, while also showing that they may be difficult for these students to understand, sometimes even leading to misunderstandings. Thus, there is a clear need to develop multimodal literacy, which involves an awareness of non-verbal semiotic resources, including gestures. This notion is gaining more and more importance in the contemporary society of screen-mediated communication and in educational settings in

Gestures in Motion in TED Talks: Towards Multimodal Literacy

147

particular (McCafferty and Stam 2008; Taleghani-Nikazm 2008; Street, Pahl, and Rowsell 2011; Crawford Camiciottoli and Bonsignori 2015). An increasingly popular form of screen-mediated communication is the TED Talk,1 a genre which has recently begun to be studied from different perspectives (Laudisio 2013; Caliendo and Compagnone 2014; Compagnone 2014; Scotto di Carlo 2013, 2014a, 2014b, 2014c, 2015; Rasulo 2015; D’Avanzo 2015; Meza and Trofin 2015; Mattiello forth.), including the pedagogical and multimodal ones.2 The talks represent a hybrid and innovative tool of popularisation, addressing both experts and non-experts in synchronous or asynchronous (web-mediated) modes. Among their features are the use of an informal and familiar register, humour, personal narratives, and anecdotes, reflecting a style that appeals to emotions and values. This favours the direct involvement of the audience and encourages discussion of the interesting ideas, rather than passive reception. In addition, due to wide accessibility, TED Talks represent a suitable vantage point for the exploration of the interplay of different semiotic resources in popularisation in the digital era. The most viral TED Talkers have indeed been found to use gestures extensively (van Edwards 2015). TED Talks have also recently become a useful resource in EFL teaching (Dummett, Stephenson, and Lansford 2016). Suggestions have been made envisaging their exploitation to develop, for example, listening comprehension skills (Takaesu 2013) and methods for teaching learners how to use gestures (Carney 2014). This chapter addresses precisely the need for a more detailed coding of gestures used in this genre of popularisation which is becoming increasingly exploited in educational settings. The analysis presents a selection of examples from three TED Talks on socio-economic topics, with the aim of exploring and categorising the interaction between speech and arm and hand gestures. In doing so, the investigation aims to shed some light on the ways different gestures may assist in meaning comprehension in international contexts such as those where TED Talks take place. 1

TED Talks is a non-profit organisation established in 1984. It began as a conference where Technology, Entertainment and Design converged; its slogan is ‘ideas worth spreading’. TED Talks (lasting 18 minutes or less) are stored as online videos and cover topics from science, to business, to global issues in more than 100 languages. 2 This chapter builds on preceding work on TED Talks as part of a broader corpusdriven study of multimodal approaches to ESP research and teaching, which involved different genres in the discourse domain of political science (Crawford Camiciottoli, Bonsignori, Bruti, Cappelli and Masi 2015).

Chapter Six

148

The next section introduces the data and the methodology used for the multimodal analysis. Section 3 focuses on a detailed description of a range of examples which are representative of different categories and functions of gestures from my data. A final section proposes a summary of findings and some concluding remarks, also envisaging possible developments of future research.

2. Data and Methodology of Multimodal Analysis Three different TED Talks were taken into account for this study. All of them were retrieved from the topic section on economics of the TED Talks website3 and feature male speakers with a seemingly native American English accent. The speakers talk from a large stage on a red carpet or central area, with a large screen at the back and the audience at the front. The talks are mainly monologic, sometimes involving some audience discussion. The first talk, entitled “The death of innovation, the end of growth” (filmed February 2013, duration 12:02), is by Robert J. Gordon, who is introduced as one of the most influential macroeconomists in the world. The talk includes the use of slides projected at the speaker’s back, to which he makes reference several times. Overall, Gordon appears to be more stationary than the other two speakers. The second talk’s title is “Why we shouldn’t trust markets with our civic life” (filmed June 2013, duration 14:18), by Michael Sandel, who is presented as a teacher of political philosophy at Harvard. The talk contains a dialogic part (i.e., a brief exchange with two members of the audience), which is reflected by the speaker’s more interactional style, also in terms of his gestures. The third talk, “A conservative’s plea: Let’s work together” (filmed February 2016, duration 14:05), is by Arthur Brooks, a social scientist and author. The speaker is rather exuberant in his use of gestures and other non-verbal signals (e.g., facial expressions). The talk starts with a personal narrative and contains some extent of enacted dialogism. Both his speech and gestures tend to be highly expressive and intense. Despite the different layout and styles of the talks, arm and hand gestures are used extensively in all of them, also displaying recurrent patterns that appear to affect different levels or dimensions of communication, viz. the ideational level (i.e., illustrating content), as well as the textual and interpersonal ones that assist in discourse structuring and 3

http://www.ted.com/.

Gestures in Motion in TED Talks: Towards Multimodal Literacy

149

enhancing the engagement of the audience, respectively. Indeed, the qualitative analysis of such patterns was based on a multimodal transcription accounting for and interpreting the relations between speech and gestures, inspired by Baldry and Thibault (2005). The description of data followed an integrated method (Lazaraton 2004), which led to the identification of different functions of gestures that were interpreted. As for gesture categories, reference was made to McNeill’s (1992) classification, covering 1) beats/underliners (i.e., repeated vague hand movements providing emphasis and regulating speech flow), 2) emblems, which tend to be culturally-specific and carry conventional meaning independently of speech, 3) deictic gestures, pointing at referents, either physically present or not, and 4) iconic and metaphoric gestures, representing concrete and abstract notions. Iconic gestures depict a concrete shape or describe a movement or action, and their form is conditioned by a perceived similarity with the object or action in the real world to which they refer. Metaphoric gestures involve an increasing degree of abstraction. The similarity between a source domain, in the concrete gesture, and a target domain, or abstract concept, is established on the basis of image-schemas, namely the generalised conceptual structures that reveal patterns of reasoning unconsciously informed by our habitual bodily experience (perceptual and kinaesthetic) with objects, people, and events in the world (Cienki 2008). In fact, metaphoric gestures also represent the source domain of a metaphor iconically to some degree (Cienki 2008). Metaphorically conceptualised models and processes may be more or less conventionalised within a given culture (McNeil 2008), and have been found to guide, to a high degree, the understanding of gestural illustrations used in teaching abstract knowledge domains (Mittelberg 2008). In line with this, it is here assumed that iconic and metaphoric gestures, even those that are built on less culturally-shared schemas, can enhance comprehension in international settings, either via a link with reality (in the case of iconic gestures) or via the often unconscious reflection of the more basic conceptual structure on which metaphors are built. Although plenty of beats and several iconic gestures were present in my data, the main focus of the analysis was on deictic and metaphoric gestures, which appeared to involve quite a varied combination of functions. While iconic gestures are obviously important devices for assisting comprehension, due to their imagistic nature and direct link with the concrete world, metaphoric gestures, in particular, were regarded as more interesting, as many crucial concepts in the field of economics are abstract. More precisely, the analysis attempted to put forward some

150

Chapter Six

hypotheses about the likely types of conceptual schemas depicted by gestures for abstract notions (Cienki and Müller 2008). The main focus was then on the relations between the schemas represented and word meanings to determine the functions of gestures in context. The relations revealed varying degrees of conventionality or relevance of schemas to what was said, thus presumably having a different potential impact on understanding. As far as gesture functions are concerned, a first set taken here into account (viz. indexical, representational, social) was inspired by the classification of gestures in classroom discourse proposed in Weinberg, Fukawa-Connelly and Wiesner (2013). Reference was also made to the pragmatic functions found in Kendon (2004) and those in Müller (2008). However, in the present study, deictic and metaphoric gestures are regarded, respectively, as inherently indexical (indicating a position) and representational (depicting entities or ideas). Both types of gestures may serve further pragmatic functions: 1) social (i.e., increasing the speaker’s immediacy with the audience and regulating turns within talks), 2) performative (i.e., illustrating a speech act), 3) parsing (i.e., demarcating different discourse units), and 4) modal functions (i.e., modulating certainty/uncertainty). Metaphoric gestures, in particular, may represent abstract entities or ideas on a pragmatic level or on a semantic-referential level, thus having a referential function (Müller 2008). The description of gestures largely hinges upon some of the most widely used coding parameters in the relevant literature (McNeill 1992, 2005; Kendon 2004; Webb 1996, inter alia): hand presence (left hand and/or right hand; if both are involved, their reciprocal position), hand shape (e.g., open as tray, cup, lid or closed as fist), palm orientation (e.g., upwards, downwards, both palms facing each other), movement (e.g., trajectory, and type or manner), location (the gesture space relative to the position and posture of the speaker). Another meaningful variable in my data is the possible presence of tension in the gesture.

3. Gestures in Motion: Selected Examples and Some Interpretive Hypotheses Below are the most basic type-function patterns that emerged from the analysis of deictic and metaphoric gestures across the three talks:4 4

It is worth noting that there were many more interesting examples covering more complex patterns, as well as some ambiguous cases, that could not be dealt with here for reasons of space.

Gestures in Motion in TED Talks: Towards Multimodal Literacy

x x x x x

151

deictic-spatial (i.e., pointing at slides, only in Gordon’s talk); deictic-social deictic-parsing deictic and/or metaphoric-performative metaphoric-referential

As will be shown, clear-cut distinctions are sometimes difficult to make; however, for the sake of clarity, the following sections attempt to tackle the two macro-categories of gestures separately.5 In the examples in 3.1 and 3.2, the stretches of text accompanied by relevant gestures are in bold, followed by the description of the gesture, the speaker’s name, and the time at which the segment occurs within it (in parentheses).

3.1. Deictic gestures Deictic gestures may help locate concrete items in space, as in the case of slides in Gordon’s talk, or virtual entities and ideas that are part of discourse. They tend to co-occur with verbal indices and their basic indexical function appears to serve more specific functions, which make clear which levels of discourse they primarily indicate. The following examples illustrate the social function of some deictic gestures (see Fig. 6-1 for the corresponding screenshot of example 2): (1) Let’s have a discussion (Both arms are half-raised and extended to front, open hands, palms facing one another); (Sandel, 06:48) (2) What would you guess? (Right arm extended to front, open hand, palm up, slightly slanting leftwards); (Sandel, 00:44)

The gestures are addressed to the audience and try to engage their attention and even elicit a response of some type. They also assist in understanding the speech acts involved in each case (i.e., the invitation to join the discussion in the first example and the question in the second), as well as promoting interpersonal involvement.

5

As for the modal function, in my data several beats, among other gestures involving tension, could probably be associated with the function of intensifying the certainty of what is said (especially in Brooks’ talk). As far as hedging is concerned, there were few examples where other non-verbal signals conveyed this function more directly than gestures (although deictic and metaphoric gestures combined to make it more salient).

152

Chapter Six

Fig. 6-1. Example of deictic-social pattern

The next extracts exemplify the deictic-parsing pattern. In addition to several cases involving enumeration, there were many instances of combined deictic gestures emphasising binary contrasts between entities, concepts, or situations by locating them in different spaces through different arm and hand gestures and their directions, usually on a transversal axis. In fact, it is possible to conceive of them as reflecting an underlying ontological metaphor, whereby the source domain is a distinct physical space and the target domain is the speaker’s mental representation of discourse structure (Cienki 2008). As such gestures are metaphoric on a pragmatic or metanarrative level (Cienki 2008), they benefit the addressee in that they can make the logical organisation of ideas physically clear. Furthermore, the distinctions on the transversal (left-to-right) axis are thought to be inspired by the schema of symmetry. This is because the axis corresponds to the frontal plane and imposes the image of symmetry of all bodies viewed from the front and being separated into two parts, with a change of hand expressing distinction of entities and a change of direction expressing distinction of qualities (Calbris 2008). As a result, metaphors can be cumulative (Calbris 2008), which makes it more likely for gestures to be meaningful beyond the stylistic-pragmatic level, for example, by complementing what is said with further semantic meaning. In example (3), arm choice and direction reflect a culturally shared association of left and right-wing political positions with the respective arms: the left arm represents the liberals’ position, whereas the right one represents the conservatives’. In fact, the gestural information provided

Gestures in Motion in TED Talks: Towards Multimodal Literacy

153

complements the utterance, as the denomination is verbal and the qualification of the political positions named is gestural: (3) […] Jon Haidt has shown that liberals care about poverty 59 percent more than they care about economic liberty (Left arm half-raised on the left – vision of gesture partially hindered due to close-up). And conservatives care about economic liberty 28 percent more than they care about poverty (Right arm half-raised on the right); (Brooks, 09:41)

In the following case, the left-to-right trajectory appears to be in line with research that has highlighted a strong correlation of this type of physical progression with several symbolic meanings (Calbris 2008). Among them, example (4) seems to instantiate the representation of the logical course of actions along the time line, which is normally conceived to go from left to right in western cultures, with ‘past’ and ‘cause’ being typically located on the left, and ‘future’ and ‘effect’ being located on the right:6 (4) […] or may it lead them to read maybe for the wrong reason initially (Right arm half raised, index pointing at front) but then lead them to fall in love with reading for its own sake? (Right arm moves rightwards, hand opens in loosely-shaped cup, palm facing left); (Sandel, 09:34)

In extract (5), rather than the expected left-to-right trajectory, the binary opposition contains a gesture that points at the speaker himself. Once again, the gesture dichotomy is not merely stylistic, but semantic, as it reveals information that is not communicated on the verbal level, namely that the speaker’s ideology is based in love, in opposition to others’, but only by changing hands and correlated directions: (5) That’s what psychologists call the phenomenon of assuming that your ideology is based in love (Left open hand touching speaker’s chest) but your opponents’ ideology is based in hate (Right arm extended towards right, open hand); (Brooks, 08:18)

Excerpt (6) shows a more complex pattern, where deictic gestures perform both parsing and social functions:

6

The schema of the logical sequence of events derives from the transverse axis being a privileged vantage point for visually tracking moving bodies; it is also the axis of reading and writing in western cultures (Calbris 2008).

154

Chapter Six (6) […] I’d like to award an Oscar to the inventors of the 20th century, the people from Alexander Graham Bell to Thomas Edison to Wright Brothers, (Enumeration with left hand); I’d like to call them all up here, (Both hands move in parallel backwards from front, palms facing speaker) and they’re going to call back to you. Your challenge is, can you match what we achieved? (Both hands push towards audience, repeatedly); (Gordon, 11:51)

The gestures appear to demarcate and involve different elements, namely the inventors of the past in contrast with the audience. Overall, the different directions of pointing seem to act as stage directions regulating speech turns and guiding understanding. The repeated pushing gesture at the end, in particular, places emphasis on its social function and puts the deictic shift on the verbal level in the background (we in what we achieved?). In doing so, it possibly supports the interpretation of the last segment as an instance of free direct speech, as the speaker is enacting fictional characters (i.e., the great inventors of the past). Another relevant pattern in the data is that of deictic gestures with a performative function, often accompanied by adverbs or demonstratives as verbal indices with a cataphoric, introductory role. Once again, the speaker’s mental spaces about the organisation of discourse are represented in the form of physical spaces. In example (7), the segment in bold announcing the speaker’s analysis of data is physically located within a complex representation of growth as a vertical axis (i.e., at the point where ‘growth is cut in half’); indeed the whole passage has an illustrative function assisting the speaker’s presentation: (7) They are powerful enough to cut growth in half. So we need a lot of innovation to offset this decline. And here’s my theme: because of the headwinds, if innovation continues […] (Right hand is holding remote control at front, and pointing twice leftwards, perpendicularly to the middle of hypothetical vertical axis delimited at the top by half raised left arm, open hand, palm down); (Gordon, 01:20)

The explicit reference to the ‘physical location’ of the speaker’s theme makes its relevance to his argumentation more evident. In the case (8), the speaker’s epiphany is virtually placed right in front of the audience for consideration: (8) This is the epiphany (Quick movement of right arm half raised at front, loosely open hand, palm facing the audience); (Brooks, 06:15)

Gestures in Motion in TED Talks: Towards Multimodal Literacy

155

Example (9) shows a deictic component, through pointing, and also a metaphoric representation, as a sort of loop in the air, of the debate that has just taken place within the talk. The circular kinetographic gesture, with initial point, progression, and return to the initial position, appears to have a recap function of the different arguments discussed in the talk: (9) Now, what this, (Right index pointing rightwards) even this brief debate, brings out is something that many economists overlook (Right index moves to front, drawing a circular shape); (Sandel, 09:43)

3.2. Metaphoric gestures Metaphoric gestures also performed a range of functions. The performative function is actually served more often by metaphoric gestures than by deictic ones. Example (10) illustrates a rather conventional form of metaphoric gesture associated with speakers’ presentation of new ideas, which is based on the conduit metaphoric model (McNeill 1992, 2008) and the containment schema, in which ideas are thought of as if they were objects, and communication as if it were a simple transfer of ideas via a container from one person to another: (10) But here’s the truth. Here’s the epiphany that I had that changed my thinking (Right arm is half raised and extended to front, hand opens, palm up, loosely cupped); (Brooks, 04:14)

While the opening of the hand in example (10) stands for disclosure, its shape and position can indeed be viewed as the source domain for the container of the speaker’s idea (i.e., as the target domain). Metaphoric gestures can also illustrate other types of speech acts besides announcing the presentation of a new idea. The case in (11), for instance, is from the final part of Brooks’ talk and displays quite a conventional gesture immediately following the act of thanking and reinforcing that expressive function in the closing: (11) Thank you (Prayer gesture over speaker’s mouth); (Brooks, 14:05)

Examples (12) and (13) reveal other forms of rather conventional metaphoric gestures, for example, a hand on the head standing for an idea in the mind or for the act of thinking or remembering: (12) But more than that I said, “What an opportunity.” (Right open hand on speaker’s forehead) (Brooks, 07:25)

156

Chapter Six (13) […] and I was visiting my parents for Christmas (Right index touches speaker’s forehead); (Brooks, 1:40)

Interestingly, in both cases the mental acts are underscored only by gestures and, on a more global level, appear to contribute cohesion to the lively narrative, intertwining with argumentation, of the speaker. In a similar way, the hand gestures and head movements accompanying the reformulation/explanation in the passage in example (14) do not depict the actions referred to on the verbal level. In contrast, they appear to foreground an idea of approximation whose perception, in context, may lead to interpret the segment as an act of reluctant concession after a selfironic understatement, which causes laughter on the part of the audience: (14) …I took a route that was not exactly what my parents had in mind. When I was 19, I dropped out of college - dropped out, kicked out, splitting hairs (Half-raised arms apart at front, open hands slightly tilting sideways repeatedly; head movement in between nodding and gently rocking sideways); (Brooks, 00:46)

Example (15) shows another case where a metaphoric gesture appears to complement what is said in subtle ways. In fact, the full impact of the gesture on understanding (e.g., the ensuing laughter on the part of the audience) requires consideration within a wider context. The gesture takes place after the ironic tale of Brooks’ upbringing—that he had taken a route, namely a gap decade that did not comply with what his parents had in mind, although they never complained—and in the context of enacted dialogism, whereby the speaker reports a preceding conversation with his mother. In this case, the gesture focused on actually marks one of the mother’s reported speech turns, the third segment in bold in the example: (15) And she [mother] was quiet, too quiet. Something was wrong. And so I said, “Mom, what’s on your mind?” (Left arm half raised on the left, loosely open hand, palm up). And she said, “Your dad and I are really worried about you.” (Both hands at front, interlaced fingers, close to chest) […] “I want you to be completely honest with me: have you been voting for Republicans?” (Right arm half raised, fingers closed in a ring, knuckles facing down); (Brooks, 01:52)

The gesture under analysis can possibly be regarded as metaphorically representing a repressed negative emotion, thus revealing the facethreatening quality of the directive act involved, which seems more a demand for an explanation rather than a request. In this way, it also emphasises a humorous contrast between the suspense, created through the

Gestures in Motion in TED Talks: Towards Multimodal Literacy

157

narrative, about the possible sources of the mother’s worry and the final revelation of the nature of the worry itself. Indeed, the whole effect is supported by the contrast in characterisation through the different gestures (supported by different face expressions) employed for the different turns, the different hand configurations used, and especially the contrast between the open and relaxed deictic gesture marking Brooks’ reported turn vs. the closed configuration marking the reported turn of his mother. Metaphoric gestures were also often used to represent different types of processes and actions on the ideational level, especially in connection with different nouns and verbs and via more or less expected image schemas. Example (16), from the already mentioned illustrative explanation in Gordon’s talk, shows the canonical representation of growth as a vertical axis, on the basis of the experiential correlation between quantity and vertical elevation: (16) They’re powerful enough to cut growth in half (Left arm half raised, open hand, palm down, representing the top extreme of axis); (Gordon, 01:14)

However, example (17) highlights that a similar notion may be represented by recruiting a different source domain: (17) […] I think can bring us progress (Right arm extended to front, hand closed in a fist, knuckles facing down); (Brooks, 06:40)

Emphasis seems to be placed, this time, on physical force, rather than on vertical increase, possibly representing the effort to be made to achieve progress. In example (18), the same type of gesture is used by the same speaker, possibly with the same type of meaning of intensity and strength, but in connection with a concrete noun: (18) …the moral obligation to be a warrior for the poor (Right arm extended rightwards, hand closed in a fist, some tension is possibly involved); (Brooks, 13:14)

Example (19) displays a more complex configuration of gestures used to complement the meanings of verbs with this same idea of strength and effort, rather than with schemas representing their more ‘literal senses’: (19) But when it comes to lifting people up who are starving and need us today, those are distractions (Arms half raised and apart, both hands

158

Chapter Six are closed in fists, knuckles facing down, slightly moved up and down, repeatedly, tension involved); (Brooks, 11:30)

In fact, verbs expressing actions and processes are often accompanied by kinetographic gestures with different trajectories. The latter may involve both hands and underline, for instance, separation, as in example (20): (20) […] to see how they differ (Half-raised arms moving apart at front, loosely open hands, palms up); (Brooks, 09:30)

More often, only one hand is used, whose path may reveal different perspectives taken by the speaker vis-à-vis the actions described and the audience. In extract (21), the direction of the movement indeed projects the world far from the speaker and into the audience: (21) [...] accompanied by economic freedom spreading around the world (Right arm extends to front, loosely open hand with palm down); (Brooks, 10:14)

In the passage in (22), the pushing gesture seems to highlight a causative component through the idea of an action exerted on something (cost inflation), which is moved away from the speaker and, as a result, likely to be perceived as smaller or insignificant from his vantage point: (22) In college, we’ve got cost inflation in higher education that dwarfs cost inflation in medical care (Left open hand pushing forward); (Gordon, 04.38)

Examples (23) and (24), instead, show how the same action, mentioned twice by the same speaker, is represented differently in each case: (23) Now these are controversial questions, and so we tend to shrink from them. (Right arm half raised at front, open hand, palm facing the speaker, then pulls towards the speaker, slightly downwards, and hand closes); (Sandel, 11:48) (24) For fear of disagreement, we shrink from these questions (Right hand, loosely open, palm towards audience, moving from left to right following an upward semi-circular path); (Sandel, 12:09)

The first occurrence of shrink from (23) is marked by a trajectory that appears to be congruent with the representation of the ‘move back/away’ sense of the verb, also complying with the inclusive we as subject. The

Gestures in Motion in TED Talks: Towards Multimodal Literacy

159

second occurrence in (24), although with the same sense, is instead accompanied by a different path, both in terms of direction and shape of motion, possibly involving the idea of jumping and avoidance. Example (25) displays a different sense of shrinking, namely ‘becoming smaller in amount’, represented by the vertical trajectory of the gesture: (25) Now hours per person are shrinking (Left open hand, palm down, moves down); (Gordon, 04:10)

The excerpt in (26) offers another example in which gesture trajectories underscore different directions complying with their meanings: (26) […] with some social goods and practices, when market thinking and market values enter, (Right open hand, loosely open, palm facing left, moves from right to left) they may change the meaning of those practices and crowd out attitudes and norms worth caring about (Right hand moves outward from the body, from left to right, loosely open hand facing audience); (Sandel, 04:55)

The left-to-right trajectory of the right hand accompanying crowd out, in particular, involves motion away from the central part of the speaker’s body (opposed to the trajectory accompanying enter). It may be viewed as reflecting a physical progression as a logical sequence of events (see above), combining with a spatial relation in which the trajector or focal element which follows the trajectory (i.e., the direct objects of the verb) is exterior to a bounded landmark (i.e., the speaker/experiencer), with the consequence of being excluded from his experience, a condition which is regarded as unfavourable.7 The contrast between the directions of the two gestures makes each of them more salient, thus also emphasising their contribution to the comprehension of the potentially more opaque phrasal verb. Example (27) shows a different gesture representing the exclusion sense of another phrasal verb with the particle out, still involving motion away from the speaker/experiencer (as the centre of the transversal axis): (27) Let’s start with those of you who object, who would rule it out even before trying (Both hands are open at front, palm facing audience, and move apart); (Sandel, 06:54) 7

On the use of trajector and landmark for the description of the polysemy network associated with out (and the Exclusion Sense therein) see Tyler and Evans (2003: 200).

160

Chapter Six

The final example (28) displays a different sense of out, namely a configuration in which the trajector (larger moral meaning) is not in its default location, represented by a bowl-shaped hand as an empty bounded landmark (cf. Fig. 6-2):8 (28) […] when market reasoning and market thinking have gathered force and gained prestige, our public discourse during this time has become hollowed out, empty of larger moral meaning (Right hand at front, palm up in the shape of cup); (Sandel, 12:02)

Fig. 6-2. Example of metaphoric-referential pattern

4. Summary and Concluding Remarks Overall, the analysis has highlighted the important contribution of nonverbal signals, particularly deictic and metaphoric gestures co-occurring with abstract notions, to the meaning of the three TED Talks. Deictic and metaphoric gestures were especially found to perform a varied combination of functions, which were determined through examining the relation of the gestures with the words they co-occurred. Deictic-social gestures were found to increase interpersonal engagement, as an important function in this type of popularisation where asymmetry 8

Cf. the Not In Situ Sense of out in Tyler and Evans (2003: 202-203).

Gestures in Motion in TED Talks: Towards Multimodal Literacy

161

between expert speakers and a non-expert audience is substantially reduced. Deictic-parsing and deictic and metaphoric gestures with a performative function appeared to enhance the salience of textual organisation and the organisation of the ideas therein. Performative gestures appeared to contribute to the visualisation of where (deicticperformative) and what (metaphoric-performative) the speaker was doing in terms of speech acts. They may alert the audience to what is said, reinforce the illustration of what is said, or accompany it with different information communicated only via gestures. In the latter case, metaphoric gestures acted as cues to subtle mental and emotional components whose detection was the prerequisite not only for achieving involvement, but also for enhancing text cohesion and coherence on a more global level. Finally, metaphoric gestures representing notions on a referential-ideational level appeared to have the potential to guide the interpretation of senses of polysemous lexical items, a case in point being adverbial particles in phrasal verbs, whose metaphoric meaning is often notoriously opaque for learners of English. In addition, the analysis revealed different degrees of conventionality and relevance in the relationship between speech and the schemas depicted by metaphoric gestures. Gestures either reinforced the senses of the cooccurring words, or complemented what was said in various ways, thus presumably having different potential consequences for comprehension on the part of the audience. It is possible to hypothesise a correlation between decreasing degrees of conventionality and a higher level of complexity of the implied meanings and associations that must be worked out to make sense of the gesture-speech occurrence. In an international context, this correlation is also likely to be affected by cultural differences. Further data exploration and experimental studies are obviously necessary to corroborate this hypothesis. Nevertheless, a fine-grained analysis along the lines proposed in this paper, also in a cross-linguistic-contrastive perspective, could provide some explanatory insights for implementing conscious multimodal literacy. This entails educating learners to notice the contribution of gestures used for the creation of a shared common ground in popularising/educational contexts involving an international audience.

References Baldry, Anthony, and Paul J. Thibault. 2006. Multimodal Transcription and Text Analysis. London: Equinox. Busà, Maria Grazia. 2010. “Sounding natural: Improving oral presentation skills.” Language Value 2(1):51–67.

162

Chapter Six

Calbris, Geneviève. 2008. “From left to right…: Co-verbal gestures and their symbolic use of space.” In Metaphor and Gesture, edited by Alan Cienki and Cornelia Müller, 27–54. Amsterdam/Philadelphia: John Benjamins. Caliendo, Giuditta, and Antonio Compagnone. 2014. “Expressing epistemic stance in university lectures and TED talks: A contrastive corpus-based analysis.” Lingue e Linguaggi 11:105–122. Carney, Nat. 2014. “Is there a place for instructed gesture in EFL?” In JALT 2013 Conference Proceedings, edited by Nozomu Sonda and Aleda Krause, 413–421. Tokyo: JALT. Cienki, Alan. 2008. “Why study metaphor and gesture?” In Metaphor and Gesture, edited by Alan Cienki and Cornelia Müller, 5–26. Amsterdam/Philadelphia: John Benjamins. Cienki, Alan, and Cornelia Müller, eds. 2008. Metaphor and Gesture. Amsterdam/Philadelphia: John Benjamins. Compagnone, Antonio. 2014. “Knowledge dissemination and environmentalism: Exploring the language of TED Talks.” In Popularisation and the Media, edited by Eleonora Chiavetta, Silvana Sciarrino and Christopher Williams, 7–25. Bari: Edipuglia. Crawford Camiciottoli, Belinda. 2007. The Language of Business Studies Lectures. A Corpus-Assisted Analysis. Amsterdam: John Benjamins. Crawford Camiciottoli, Belinda, and Veronica Bonsignori. 2015. “The Pisa Audiovisual Corpus Project: A multimodal approach to ESP research and teaching.” ESP Today 3(2):139–159. Crawford Camiciottoli, Belinda, Veronica Bonsignori, Silvia Bruti, Gloria Cappelli, and Silvia Masi. 2015. “The Pisa Audio-Visual Corpus Project: A multimodal approach to ESP research and teaching.” Paper presented at the XXXIII AESLA International Conference, Multimodal Communication in the 21st Century: Professional and Academic Challenges, Madrid, 16-18 April 2015. D’Avanzo, Stefania. 2015. “Speaker identity vs. speaker diversity: The case of TED talks corpus.” In Languaging Diversity: Identities, Genres, Discourses, edited by Giuseppe Balirano and Maria Cristina Nisco, 279–296. Newcastle upon Tyne: Cambridge Scholars Publishing. Dummett, Paul, Helen Stephenson, and Lewis Lansford. 2016. Keynote. National Geographic Learning, CENGAGE Learning. Accessed April 23, 2016. http://www.eltkeynote.com/. Gullberg, Marianne, and Steven G. McCafferty. 2008. “Introduction to gesture and SLA: Toward an integrated approach.” Studies in Second Language Acquisition 30:133–146.

Gestures in Motion in TED Talks: Towards Multimodal Literacy

163

Hostetter, Autumn B. 2011. “When do gestures communicate? A metaanalysis.” Psychological Bullettin 137:297–315. Kendon, Adam. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Laudisio, Adriano. 2013. “Popularization in TED talks: A contrastive analysis of expertise asymmetry.” Paper presented at the CLAVIER 13 Conference Discourse in and through the Media. Recontextualizing and Reconceptualizing Expert Discourse. Modena, 6-8 November 2013. Lazaraton, Anne. 2004. “Gesture and speech in the vocabulary explanations of one ESL teacher: A microanalytic inquiry.” Language Learning 54(1):79–117. Lakoff, George, and Michael Johnson. 1980. Metaphors We Live By. Chicago: University of Chicago. Littlemore, Jeannette, and Graham Low. 2006. Figurative Thinking and Foreign Language Learning. Basingstoke: Palgrave MacMillan. Littlemore, Jeannette, Fiona MacArthur, Alan Cienki, and Joseph Holloway. 2012. How to Make Yourself Understood by International Students: The Role of Metaphor in Academic Tutorials. ELT Research Papers 12-06, London: British Council. Macedonia, Manuela, and Katharina von Kriegstein. 2012. “Gestures enhance foreign language learning.” Biolinguistics 6.3-4:393–416. Mattiello, Elisa. forth. “The popularisation of science via TED Talks.” International Journal of Language Studies 2016. McCafferty, Steven G., and Gale Stam, eds. 2008. Gesture. Second Language Acquisition and Classroom Research. New York: Routledge. McNeill, David. 1992. Hand and Mind: What the Hands Reveal about Thought. Chicago: University of Chicago Press. —. 2005. Gesture and Thought. Chicago: University of Chicago Press. —. 2008. “Unexpected metaphors.” In Metaphor and Gesture, edited by Alan Cienki and Cornelia Müller, 155–170. Amsterdam/Philadelphia: John Benjamins. Meza, Radu, and Constantin Trofin. 2015. “Between science popularization and motivational infotainment: Visual production, discursive patterns and viewer perception of TED Talks videos.” Studia Universitatis Babes-Bolyai – Ephemerides 60(2):41–60. Mittelberg, Irene. 2008. “Peircean semiotics meets conceptual metaphor: Iconic modes in gestural representations of grammar.” In Metaphor and Gesture, edited by Alan Cienki and Cornelia Müller, 115–154. Amsterdam/Philadelphia: John Benjamins.

164

Chapter Six

Müller, Cornelia. 2008. “What gestures reveal about the nature of metaphor.” In Metaphor and Gesture, edited by Alan Cienki and Cornelia Müller, 219–145. Amsterdam/Philadelphia: John Benjamins. O’Halloran, Kay L., Sabine Tan, and Bradley A. Smith. 2016. “Multimodal approaches to English for academic purposes.” In The Routledge Handbook of English for Academic Purposes, edited by Ken Hyland and Philip Shaw. London: Routledge. Rasulo, Margaret. 2015. “TED culture and ideas worth spreading.” In Languaging Diversity: Identities, Genres, Discourses, edited by Giuseppe Balirano and Maria Cristina Nisco, 262–278. Newcastle upon Tyne: Cambridge Scholars Publishing. Scotto di Carlo, Giuseppina. 2013. “Humour in popularisation: Analysis of humour-related laughter in TED talks.” European Journal of Humour Research 1(4):81–93. —. 2014a. “The role of proximity in online popularizations: The case of TED Talks.” Discourse Studies 16(5):591–606. —. 2014b. “Ethos in TED Talks: The role of credibility in popularised texts.” Facta Universitatis – Linguistics and Literature 2:81–91. —. 2014c. “New trends in knowledge dissemination: TED Talks.” Acta Scientiarium. Language and Culture 36(2):121–130. —. 2015. “Stance in TED talks: Strategic use of subjective adjectives in online popularisation.” Ibérica 29:201–222. Street, Brian, Kate Pahl, and Jennifer Rowsell. 2011. “Multimodality and new literacy studies.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 191–200. London: Routledge. Takaesu, Asako. 2013. “TED Talks as an extensive listening resource for EAP students.” Language Education in Asia 4(2):150–162. Taleghani-Nikazm, Carmen. 2008. “Gestures in foreign language classrooms: An empirical analysis of their organization and function.” In Selected Proceedings of the 2007 Second Language Research Forum, edited by Melissa Bowles, Rebecca Foote, Silvia Perpiñán and Rakesh Bhatt, 229–238. Somerville, MA: Cascadilla Proceedings Project. TED Talks, retrieved from http://www.ted.com/. Accessed January 30, 2016. Tyler, Andrea., and Vyvyan. Evans. 2003. The Semantics of English Prepositions. Spatial Scenes, Embodied Meaning and Cognition. Cambridge: Cambridge University. Van Edwards, Vanessa. 2015. “20 hand gestures you should be using.” Science of People. Accessed May 10, 2016. http://www.scienceofpeople.com/2015/08/20-hand-gestures-using/.

Gestures in Motion in TED Talks: Towards Multimodal Literacy

165

Walsh, Maureen. 2010. “Multimodal literacy: What does it mean for classroom practice?” Australian Journal of Language and Literacy 33(3):211–223. Webb, Rebecca. 1996. “Linguistic features of metaphoric gestures.” In Proceedings of WIGLS. The Workshop on the Integration of Gesture in Language and Speech, edited by Lynn Messing, 79–95. Delaware: Applied Science and Engineering Laboratories Newark. Weinberg, Aaron, Tim Fukawa-Connelly, and Emilie Wiesner. 2013. “Instructor gestures in proof-based mathematics lectures.” In Proceedings of the 35th annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, edited by Mara Martinez and Alison Castro Superfine, 1119. Chicago: University of Illinois at Chicago.

CHAPTER SEVEN GENDER-RELATED COMMUNICATION IN POLITICAL INTERVIEWS: A MULTIMODAL ANALYSIS OF MEANINGMAKING IN THE OBAMA/CLINTON INTERVIEW OF 2013 SILVIA BRUTI UNIVERSITY OF PISA, ITALY

1. The Hybrid Genre of Interviews In the most relevant literature on interviews (cf., among many, Jucker 1986; Ekström and Lundell 2009; Gubrium 2012) the majority of scholars have started from investigating this genre as a kind of communicative activity type as defined by Levinson (1979). The concept is especially suitable in the case of interviews, which are in themselves a hybrid genre, designed in advance and planned to a greater or lesser extent, and offering different media of communication, i.e., written in absentia interviews or oral face-to-face interviews that can be broadcast via radio or television. In activity types, all class members, regardless of their more or less prototypical position, have to meet certain requirements, including ‘framing’, meaning that the interactional situation is defined in terms of possible types of behaviour, speech roles, and related aims. The type of interview that is at the core of this study, a political interview designed to reach the largest possible audience, is just one among several subgenres. Mediated political interviews have been labelled as “accountability interviews” (Montgomery 2007: 145), also because they feature responsible politicians or people who occupy important power positions (Clayman and Heritage 2002), have been granted much attention over the last few years, although during on-air interactions (i.e., those

Gender-related Communication in Political Interviews

167

intended for an overhearing audience), or have had the lion’s share of interviewing, in part because of their large traceability. There are in fact interviews that are scarcely available to the analyst, as it is not possible to tape them and subsequently transcribe the dialogue. In the case of TV interviews, as will be shown later on, all the semiotic channels are craftily designed to make meaning. The nature of interviews is best described through reference to their formal and functional features. Being a type of institutional discourse, the roles, functions, and causes behind interviews are ruled by their institutional setting. The mediatisation process is responsible for the double frame of interviews: there is a first-frame interaction between the interviewer and the interviewee, but also a second one between both interviewer and interviewee and their audience, either in a TV studio or at home (Fetzer 2000; Furkó and Abuczki 2014). From a structural viewpoint, interviews have a rather fixed, repetitive, and asymmetrical structure, as the exchange is managed by interviewers, who initiate turns, select the next speaker, and sometimes also point out if answers sound irrelevant to questions. This rigid turn-taking adherence, together with the strict limitations in topic management and the strategic avoidance of straight answers, distinguishes interviews from spontaneous conversation (Ekström and Lundell 2009; Furkó and Abuczki 2014). As mentioned above, Montgomery (2007) identifies the political interview as one subtype of the news interview and one which has received much attention, to the point that it has often been identified with the whole genre. The four subtypes Montgomery distinguishes are the accountability interview, the experiential interview, the expert interview, and the affiliated interview. Subtypes can be distinguished on the basis of four parameters: whether the interview is affiliated with the institution; whether the interviewee is an actor or a responsible agent; whether the interviewee has direct, first-hand knowledge of the matter at issue; and, finally, the alignment (o lack of it) with the audience. In the political interview, or the accountability type, the interviewee is a public figure who is not affiliated with the institution to which the interviewer belongs, and who knows about and is often a primary agent of the facts being discussed. As Montgomery (2007) explains, “interviewees are public figures in the sense that they hold institutional positions and by their official status are treated as ‘having some locus’ on the matter at hand” (2007: 148). Lorenzo-Dus (2009) has recently shown that more banal television genres such as talk shows and more serious ones such as political debates and interviews share one feature: a pronounced tendency towards the

168

Chapter Seven

staging of conflict. In political debates conflict unfolds “through actionopposition sequences that are arbitrated by an institutional agent, [… while in interviews it] follows a question answer format, which is said to be bounded by a demand for impartial yet critical inquiry” (2009: 121). To reach this aim, both journalists and politicians try to achieve “formal neutralism”, but the majority of accountability interviews display more or less overtly adversarial features. This is not the case in the Clinton/Obama interview under investigation, as shall be seen in what follows. The interview is in fact aimed at showing a relaxed and close relationship between President Obama and Hillary Rodham Clinton. Obama lavishes praise on his former political rival, now Secretary of State, as an experienced professional diplomat who has also become a close friend and advisor. Clinton says she did not expect to be offered such a post and said she was originally uncertain about accepting it. In the following turns Obama cited some international political successes to be ascribed to his first term, e.g., the end of the war in Iraq, the winding down of the war in Afghanistan, the removal of Libyan dictator Muammar Gaddafi, and the undoing of Al Qaeda. Then, both Obama and Clinton agree on a note of caution that accounts for the non-involvement in Syria.

2. The Clinton/Obama Interview The twenty-minute interview was recorded in 2013 in a room at the White House (as is indicated by the U.S. flag next to the door, behind Obama’s chair) where the two interviewees, President Barack Obama and Secretary of State Hillary Clinton, sit side by side in front of the interviewer, journalist Steve Kroft, and are seen in a three-quarter profile. In what follows, I describe the analysis that was carried out in different stages. A first analysis was made with the software Wmatrix (Rayson 2008) to identify both the most frequent words and the key semantic domains in both Clinton’s and Obama’s speech (respectively amounting to 1,400 and 1,564 words). This was followed by a multimodal analysis focusing especially on gestures to highlight the synergic effect of wording and kinesics. Clinton’s and Obama’s gestures in this interview were compared with their gestural behaviour in another two interviews, i.e., Brianna Keilar’s interview with Hillary Clinton for CNN in 2015 and Nina Totenberg’s interview with the President for NPR in 2016. Finally, Clinton’s and Obama’s speech is compared to the speech of female and male political figures in the same genre in order to see whether the two political leaders adopt typical gendered speech styles or distinctive and idiosyncratic ones. Two small corpora of twenty political interviews each

Gender-related Communication in Political Interviews

169

were assembled for this purpose: the first one is a corpus of interviews given by American and British female politicians (the Corpus of Interviews of Female Politicians, from now on CIFP); the second contains interviews by American and British male politicians (the Corpus of Interviews of Male Politicians, from now on CIMP). For the time being, the analysis is confined to the verbal component, as a full multimodal transcription of all the material is still in progress. In future, comparison will be extended to the non-verbal codes. It is especially for this reason that British material was included in the corpus, because if in specialized verbal discourse no major stylistic differences can be observed across the two varieties (apart from references to different situations and people), it is likely that different preferences in non-verbal behaviour may emerge. The interviews in the CIFP and CIMP took place between 2010 and 2015 with important political figures belonging to different political parties in various TV shows. The corpus includes only the lines spoken by the female politicians belonging to different parties, whereas the interviewers’ turns have been deleted. A second one contains, instead, interviews given by male politicians in the same time period, following the same criteria, i.e., including leaders of several political parties and removing the lines spoken by journalists. This allows for several types of comparisons: first, between the language employed in the Clinton/Obama interview by Clinton and the speech of female politicians; second, overall, between the language of female and male politicians; and finally, between Obama’s speech and other male politicians.

2.1. An overall linguistic analysis In the course of the Clinton/Obama interview, both interviewees appear relaxed and smiling. Before analysing gestures and movements from a multimodal perspective, I briefly describe a few dominant features in both interviewees’ speech. Table 7-1 contains lists of the most frequent words in Clinton and Obama’s lines in raw frequency counts. CLINTON I and the to we you_know it

Raw frequency 71 65 43 39 34 32 27

OBAMA the and that to of we I

Raw frequency 56 51 44 32 32 32 30

Chapter Seven

170 that a of but was have in you ’s what do president

27 24 20 18 18 17 17 17 17 17 14 13

you_know a it our is Hillary was in ’s going_to be you

26 25 17 15 14 14 14 14 14 12 12 12

Table 7-1. Frequency lists The most frequent word in Clinton’s speech is I, which instead occurs in Obama’s speech as seventh in rank with half the number of occurrences, whereas the most frequent pronoun form employed by the U.S. President is we, whose collaborative and collective nature is further reinforced by the use of the possessive our. We appears instead as fifth in rank in Clinton’s speech (cf. similar remarks in a comparison of Clinton’s and Sanders’s use of pronouns on CNN Politics, Tapper and Savage, CNN 2016). This means that Secretary Clinton takes responsibility for her actions and does not avoid using references to her person. The remainder of the expressions included in the frequency list is made up—mostly—of grammatical words or key verbs such as be and have. Interestingly, considering the low number of words in the interview, both speakers use the discourse marker you know quite often, i.e., 32 times by Clinton and 26 by Obama. As Furkó and Abuczki (2014) point out, in this genre you know has two main functions, either as a marker of evidentiality or shared knowledge, or as a ventriloquizing element with which another ‘voice’ is introduced in the text. An example of the latter offered by Furkó and Abuczki (2014) is the following: “I understand that and I know there’s a lot of concern because people say well look, you know … get rid of all the targets for waiting lists and our life would be easier” (BBC Newsnight, 7 February 2003) (2014: 58). This function is not very prominent in either Clinton’s or Obama’s speech, with respectively 4 and 1 occurrences out of 32 and 26 instances. In the following example, Secretary Clinton attributes speech to somebody else, which is signalled by the discourse marker you know: the one thing he did mention was he basically said, “You know, we’ve got this major economic crisis that may push us into a depression.”

Gender-related Communicatiion in Political Interviews

171

Drawingg on Wmatrixx (Rayson 20 008), both pooliticians’ speech were compared too similar languuage, in this case c the incorrporated BNC C Sampler CG (Spokenn) institutionaal, which asseembles politiccal speeches, sermons, local and nnational goveernmental pro oceedings. Fiigs. 7-1 and 7-2 are screenshots of key semaantic domain clouds generaated for Clintton’s and Obama’s sppeeches, respeectively, show wing the dom mains that occcur with statistically higher frequeency when com mpared to BN NC reference corpus c by the softwaree. Domains shown in larger fonts are the most significaant.

Fig. 7-1. Keyy domain cloud for Clinton

Fig. 7-2. Keyy domain cloud for Obama

Both cloouds show rathher clear referrences to topiics and themees that are tackled duriing the interviiew, i.e., ‘Gov vernment’, ass most of the discourse revolves aroound governm mental decisions, and ‘Thhe_universe’, as many references aare made to world w issues. In n Obama’s sppeech, howeveer, clearer mention is m made of speccific U.S. poliicies on Iran, Israel, Russiia, China, Egypt, and Libya, whichh is reflected in the label ‘‘Geographicall_names’. Features thaat instead diistinguish onee cloud from m the other are quite interesting. Obama resortts more often n to ‘Personal__names’, in particular, p referring too his co-inteerviewee calliing her Hilla lary, whereass Clinton

172

Chapter Seven

employs a huge quantity of pronouns, notably to refer to herself, while she refers to Obama through the noun phrase the president, included under the label ‘Government’. Obama recurrently refers to issues related to ‘Crime’ and describes people or decisions being taken with strong adjectival expressions (e.g., the label ‘Tough/strong’). On the other hand, Clinton intersperses her turns with references to danger, risk, unexpectedness (e.g., the labels ‘Danger’, ‘Unexpected’), towards which she always maintains a positive attitude, displayed by labels such as ‘Understanding’ (verbs such as figure out, understand), and ‘Content’, which refers to her positive feelings (satisfaction, pride, thankfulness). Furthermore, Clinton’s turns also display a rather insistent use of boosters such as amplifying adverbs of degree or adverbial constructions (e.g., “I would have desperately wanted him to be in my cabinet”, “I care deeply about what’s going to happen”, “it’s more complex than it’s been”). The literature on gender talk has highlighted a certain difference in the use of boosters and intensifiers (Wright and Hosman 1983; Holmes 1990), but works such as Bradac, Mulac, and Thompson (1995) have also shown that boosters are used more by females in same-sex interactions, whereas more hedges are employed in mixed-sex talk. Overall, the choice of hedging and boosting devices can be ascribed to a complex web of interrelated decisions, among which are the topic being debated, the status of the politician(s), the current political scene, and the power of the interviewee (Roth 2014). The influence of gender, gender roles, and gendered language features might play a less prominent role, considering also the fact that modern politicians are surrounded by teams of very skilled experts who work at their campaigns, images, and speech styles to convince the widest possible audience.

2.2. Towards a multimodal analysis The interview was recorded in a room at the White House and the two interviewees are seen in a three-quarter profile while answering in turn Steve Kroft’s questions and sometimes interacting with each other. In order to carry out a multimodal analysis, several aspects have been considered (cf. Baldry and Thibault 2006; Wildfeuer 2013), among which are types of shots and their editing, as well as kinesics including movements and gestures in particular. Aural elements other than dialogues are not relevant in this communicative context, as speech needs to be heard clearly, so music and contextual sounds were absent. Most of the shots are medium shots that focus on Clinton’s and Obama’s bodies, with some close-ups of their faces at specific moments

Gender-related Communication in Political Interviews

173

and some long shots of the whole room, which are used to portray all the three protagonists of this unprecedented event and to provide the audience at home with a view of the context. Sometimes Kroft is also focused upon, but his turn at speech and his non-verbal behaviour have not been considered in this analysis. Concerning the editing, as is typical in a conversation, the organising strategy is the shot/reverse shot pattern, in which two consecutive shots focus on two different actors, closely following the turn-taking system. Since both interviewees are seated, they keep their hands folded in their laps when they are silent, Clinton with her left hand on top of the right, and Obama with palms touching and intertwined fingers. The interviewees’ position strongly influences their freedom of movement, to which another conditioning factor needs to be added, i.e., the camera work, as sometimes the camera lingers on their faces, leaving their arms and hands out of the frame. Their body posture is rather stable throughout the interview, although Obama often turns toward Clinton. Overall, as the atmosphere is quite relaxed, both of them smile and even laugh more extensively than politicians normally do in political interviews. Their gaze is most of the time directed at the interviewer, with some moments in which they turn to each other; in close-ups, they are shown in a threequarter profile, and again their gaze is most often directed at Kroft. To analyse gestures, I refer to McNeil’s (1992, 2005) classification of their functions. As Crawford Camiciottoli has pointed out (2015), gestures that accompany speech are quite varied and often escape easy classification. Given the nature of the genre at issue, I believe that modal and performative functions—two of the main purposes that gestures can have according to Kendon (2004)—are scarcely relevant, as the speakers do not need to convince or press points, but simply to expose and describe them. Furthermore, since many different topics, public and private, are quickly touched upon, there is no time to delve into strong and deep argumentation during the typical political interview. The types of gestures identified by McNeil (1992, 2005) are the following: iconic, metaphoric, deictic, and beats. Iconic gestures exhibit images of concrete objects or actions. Metaphoric gestures usually represent abstract content, giving an image to what usually is not thought of in these terms. Deictic gestures aim at locating objects or actions in space or time against some reference point. Finally, beats are called in this way because the hand makes a regular, paced movement (e.g., up and down, back and forth, etc.). Beats are speech-related but also have discourse functionality, highlighting what the speaker thinks or feels is important in comparison with the larger context.

174

Chapter Seven

Clinton uses many different combinations of hand gestures involving both hands, and often two hands alluding to her relationship with President Obama. Many of her gestures combine different functions, merging, for instance, a representational function with emphasis. As McNeil himself (1992, 2005) claims, beats, in fact, often overlap with other gesture types. For example, Clinton sometimes uses movements that describe either iconically or metaphorically some words: when saying I’d be going back to the Senate, she rotates her left hand, or when referring to depression (this major economic crisis that may push us into depression), she rotates her right hand downwards. In this case, her gestures allude to the semantics of the expressions she uses and emphasize crucial points that she wants to press in her speech. Similarly, she uses her hands to signal different voicing or to ventriloquize different speakers, in this case President Obama himself (Lauerbach 2006; Furkó and Abuczky 2014). Ventriloquizing is a typical feature in interviews which can be indicated either verbally or gesturally, or, in some cases, by both modes. The strategy aims at “personalizing and dramatizing political discourse and implicitly construct[s] identities and relations in the interplay between interviewer and interviewee” (Furkó and Abuczky 2014: 56). In answering Kroft’s question What did he [=Obama] say that made you [accept]?, Clinton oscillates between what Obama said and what she thought: We could never figure out what we were different on. Yeah, we worked at that pretty hard. And so I really thought I’d be going back to the Senate, where I would be supporting the president on all of those issues. And what surprised me is he said, “Well, I want you to come to Chicago.” And honestly, at the time, I thought, “Well, you know, that’s a very nice gesture. And maybe he wants to ask me about some people that might serve in the administration.” So when I got to Chicago and he asked me if I would consider being his Secretary of State, I immediately said, “Oh, Mr. President, there’s so many other people. Let me give you some other names.” Because it just took me by surprise. But he is pretty persuasive, I’ll tell you that much. And he kept saying, “Well, I want you to think about it again. I want you to… wait a minute, don’t make… don’t give me a final answer.”

In Fig. 7-3 Clinton uses gestures to pinpoint what she said and what Obama said when he invited her to Chicago to offer her to be his Secretary of State.

Gender-related Communication in Political Interviews

175

Fig. 7-3. Gestures attributing ‘voice’

In President Obama’s turns, gestures are also abundant, with a preference for beats that bear a strong pragmatic significance, some of which also share an iconic or indexical function, in that they paraphrase or allude to some of the expressions that the President employs. This happens, for example, when Obama enumerates some features counting the items with his fingers, when he moves his hands forward to convey the idea of ‘pushing’ or when he forms a square space with both hands to indicate the word ‘screen’ etc. The gesture of enumerating occurs three times in the interview, the first towards the beginning, in his second answer, when he details Hillary Clinton’s qualities, as illustrated in Figs. 7-4a and 7-4b: I was a big admirer of Hillary’s before our primary battles and the general election. You know, her discipline, her stamina, her thoughtfulness, her ability to project, I think, and make clear issues that are important for the American people, I thought made her an extraordinary talent.

Figs. 7-4a and 6-4b. Indexical gestures

Fig. 7-4c. Iconic gesture

The most idiosyncratic gesture, an iconic one, used by Obama, is a reaction to Kroft’s pressing question about “the expiration date on this endorsement”, which anticipates what has actually happened, i.e., Clinton running in the presidential elections of 2016. He answers provocatively and jocularly by saying You know, Steve, I gotta tell you, the… you guys in the press are incorrigible. I was literally inaugurated four days ago. And you’re talking about elections four years from now. While uttering the expression literally inaugurated four days ago, he touches his watch (see Fig. 7-4c). The gesture is in itself iconic, alluding to the passing of time, in

176

Chapter Seven

this case the very recent past, but it is meant to accentuate Obama’s humorous quip. An investigation into Obama’s gestures in political speeches revealed that his manner of gesticulating changed a great deal from 2008 and 2012. A YouTube video (YouTube 2012) shows a collage of extracts from Obama’s speeches in the two campaigns for the presidential elections of 2008 and 2012, which is quite revealing: although Obama employs the same wording, his gestures are radically different. He seems to have moved from more emphatic gesticulation, sometimes even pedantically highlighting all the points that needed special attention, to a more strategic one, in which he controls his gestures and uses them more sparsely. The most used gestures in the 2008 campaign were either pointed fingers to crucially underline points in his speech or Kendon’s “finger bunch” to draw some conclusions. In the 2012 campaign, instead, the most frequent gesture is palms sideways moving up to underline crucial moments: Obama’s gesturing seems to have acquired a more controlled, but still highly involving style.

2.3. Clinton’s and Obama’s gestures in other interviews In what follows a brief comparison is drawn between the gestures Clinton and Obama used in other television interviews, with a view to establishing if the ones they used in their joint interview are in keeping with their usual gestural behaviour. The interviews that have been chosen are Brianna Keilar’s interview with Hillary Clinton for CNN, which took place while she was campaigning in Iowa in 2015 (CNN 2015), and Nina Totenberg’s interview with the President for NPR after he nominated Judge Merrick Garland to the Supreme Court (YouTube 2016). The Clinton interview by Brianna Keilar for CNN was the first national interview (lasting approximately twenty minutes), a kind of reintroduction of Clinton to the American public in the middle of the 2016 campaign. The journalist presses Clinton for about 20 minutes on several thorny issues, such as her rival for the democratic nomination Bernie Sanders, the Clinton/Bush dynastic race, trust, the Clinton foundation, Donald Trump, and other topics. All the questions could be anticipated, so Clinton surely was prepared to answer them, even though she sometimes is quite evasive and moves rapidly on. Keilar, on her part, does not dig for details and moves on to the next topic, so the interview is overall easy and relaxed, although not very precise. From a behavioural point of view, there are in fact no traces of tension or embarrassment on either side, as the two women sit calmly across from each other and interact politely. There are a

Gender-related Communication in Political Interviews

177

few long shots that show them sitting as if engaged in a conversation, in a library room, with a small table with water and drinks. The majority of shots are instead medium close-ups, which do not always let viewers see Clinton’s arms, so a quantitative analysis would not be possible. The same applies to Obama’s interview, so the analysis takes into account their overall gestural behaviour. In this interview, Hillary Clinton has a very rich gestural repertoire, both quantitatively and qualitatively. She sits with her hands folded and she returns to this position, but she moves her arms and hands a great deal, performing a wide array of gestures. Most of them are beats that punctuate important stretches of her discourse, so they accompany keywords or pivotal expressions: sometimes, however, in addition to signalling the importance of the topic, they also reinforce the semantic dimension of what the speaker is saying, but also representing a concept, either iconically or metaphorically. So, for example, when she pronounces the adjective committed, she positions her hands in front of her bosom, forming a kind of box, which suggests a stable position, and hence, metaphorically, commitment. Likewise, she forms the same kind of square space with her hands and then moves them from left to right to convey the idea of a movement “across”. She also uses another movement quite frequently, i.e., palms forward moving forward, with the same function of emphasising points but also representing a concept: she employs this gesture when uttering expressions such as because I wanted to do above and beyond, in which the movement mirrors the prepositions, or in trust me to have a plan, in which case, instead, the movement represents a plan metaphorically, as a projection of today’s ideas into the future. Another interesting gesture Clinton utilises is the ‘stop’ sign, i.e., her hand lifted in vertical position, which appears in conflictual sequels, for example when mentioning being “attacked”, or making reference to the email scandal that the Republicans brought up. In the latter case, words and gestures go different ways, as Clinton says this is blown up with no basis in law, in fact, that’s fine, I get it, which sounds conciliatory enough, whereas her ‘stop’ gesture signals clearly that she is determined to put an end to this matter (see Bateman 2014 on this). Another strong and assertive gesture with which she punctuates her turns is the ‘finger bunch’, usually emphasised by repeated forward movements which go hand in hand with parallelism in the verbal code. In particular, for example, the ‘finger bunch’ movement accompanies parallel structures or lists of points (the so-called bicolons or tricolons, cf. Partington and Taylor 2010; Bonsignori, this volume), as when she says The law, the regulation did not ever stand in my way or it’s good economically, it’s good for the taxes, it’s

178

Chapter Seven

good for the children. The production of gestures is indeed very rich, but although Clinton uses different movements, gestures are performed to carry out the two functions pinpointed above, i.e., signalling crucial points in her speech and lending them more strength, and representing ideas or things that are being mentioned. Sometimes the two functions very conveniently overlap, making both meaning and illocutionary force explicit at the same time. The interview with Obama by Totenberg is quite long (28 minutes) and tackles a thorny issue, i.e., the President’s nomination of Judge Merrick Garland to the U.S. Supreme Court. The positive reviews about Garland from Republican senators were a significant factor in Obama’s decision to nominate him for a seat on the Supreme Court, hoping to restore some cohesion and to increase America’s faith in the judicial system. The interview took place in a room at the White House and the two interlocutors sit one in front of the other, separated by their microphones. The atmosphere is quite relaxed, with both participants indulging in witty remarks and humorous lines, despite the quite serious topic being tackled. The reason for choosing this interview is that it is quite recent, so Obama’s non-verbal behaviour can be evaluated against that of 2013 and previous years. The shots used in the recording of the interview often leave Obama’s arms out of the frame. Yet it is quite evident that he employs a limited spectrum of movements, consisting mainly of four types (i.e., raised thumbs, one or both palm(s) moving forward, and using his fingers to enumerate points in his speech). The neutral position is with hands folded in his lap with interlaced fingers. From time to time, Obama raises his thumbs to underline important words, and from this position he may also move his palms forward with the same intention of stressing or emphasising a concept, or driving some point home. These are quite clearly ‘beats’, but with exactly the same movement that he uses in the 2013 joint interview with Clinton. Another gesture he frequently uses in this interview is a round open space formed with both hands, sometimes moving them forward, again for the purpose of underlining a point. An identical gesture to the 2013 interview, performing an indexical function, occurs when Obama enumerates something and at the same time counts the points on his fingers. Similarly, he uses this gesture quite often in the Clinton/Obama interview, three times in both interviews. Overall, his nonverbal behaviour appears controlled and the repertoire of gestures quite limited, especially if compared to his non-verbal behaviour in the 2008 campaign. The gestural style in the latest interview is very similar to the one displayed in the 2013 interview with Hillary Clinton, suggesting that

Gender-related Communication in Political Interviews

179

Obama’s initially quite exuberant style has left space for a more sparse but still significant way of gesticulating.

2.4. Clinton vs. female politicians and Obama vs. male politicians To compare Clinton’s and Obama’s speech with speech used by female and male politicians in the same genre, two small corpora of political interviews were assembled, namely the Corpus of Interviews of Female Politicians (CIFP) and the Corpus of Interviews of Male Politicians (CIMP), with 20,566 and 35,588 words, respectively. As explained in section 2, the investigation is limited to the verbal component, but it will be extended to include other semiotic modes in future research. Each corpus contains the transcripts of twenty interviews of British and American politicians participating in different TV shows in a time span from 2010 to early 2016. They only contain the turns uttered by the politicians, as those by the interviewers have been excluded. The details of corpus composition are specified in the Tables 7-2 and 7-3. Nationality

Name

American American American American American American American American American American British British British British British British British British British British

Bailey Hutchison Clinton Jarrett Murkowski Ayotte Fallin Pelosi Feinstein Fiorina Lynch Carwyn Jones Cooper Lucas May Bennett Miller Maria Widdecombe Rudd Stuart Sturgeon

Table 7-2. CIFP composition

Political orientation Republican Democratic Democratic Republican Republican Republican Democratic Democratic Republican Democratic Green Labour Green Tory Green Tory Tory Tory Labour Scottish Party

Date of interview 6 November 2010 21 November 2010 24 January 2010 19 September 2010 9 June 2013 2 February 2013 10 March 2013 22 November 2015 9 September 2015 6 December 2015 14 February 2010 5 December 2010 11 April 2010 31 October 2010 10 February 2013 3 November 2013 9 June 2013 13 December 2015 14 February 2016 21 February 2016

Chapter Seven

180 Nationality

Name

American American American American American American American American American American British British British British British British British British British British

Biden Cantor Clinton Bill McCain Cruz Kerry McDonough Rubio Sanders Trump Cameron Clegg Gove Miliband Blair Farage Redwood Falconer Corbyn Hunt

Political orientation Democratic Republican Democratic Republican Republican Democratic Democratic Republican Democratic Republican Tory Liberal Democrats Tory Labour Labour Independent Tory Tory Labour Labour

Date of interview 14 February 2010 7 November 2010 19 September 2010 5 September 2010 22 September 2013 24 November 2013 8 September 2013 31 January 2016 6 December 2015 10 January 2016 3 October 2010 17 January 2010 14 February 2010 21 January 2010 3 February 2013 5 May 2013 19 May 2013 15 November 2015 27 September 2015 6 December 2015

Table 7-3. CIMP composition In what follows, there are comparisons of word frequency lists (cf. Table 7-4) and key domain clouds (cf. Figs. 7-5 and 7-6). 1 2 3 4 5 6 7 8

Clinton I and the to we you_know it that

71 65 43 39 34 32 27 27

9 10 11 12 13 14

a of but was have in

24 20 18 18 17 17

Obama the and that to of we I you_ know a it our is Hillary was

56 51 44 32 32 32 30 26

CIFP the that and to we I of is

993 638 570 550 494 405 393 364

CIMP and that to we a of I in

1146 1005 940 833 698 691 640 624

25 17 15 14 14 14

a it in ’s what do

329 292 274 263 196 182

is it ’s you do are

618 523 389 355 339 326

Gender-related Communication in Political Interviews 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

you ’s what do president be is our so going_to this n’t me ’m he some been ’ve my for about ’re are

17 17 17 14 13 13 12 12 12 12 11 10 10 10 10 9 9 8 8 8 8 8 8

in ’s going_to be you for her do but are ’re think one had n’t have did as i_think world were there around

14 14 12 12 12 11 10 10 9 9 8 7 7 7 7 7 7 6 6 6 6 6 6

are have this but be for think you people they not our well so was on n’t he there about with ’re if

179 178 175 175 164 153 150 140 138 130 128 124 120 119 119 114 112 110 109 106 102 100 72

38 39 40 41 42 43 44 45 46 47

just would as because had people there very hard world

7 7 7 7 7 7 7 6 6 6

so where just ’ve she on things not by all

6 6 5 5 5 5 5 5 5 5

will very can would been who an were need no

48 49 50

think him well

6 6 6

very they can

5 5 5

51

on

6

he

5

just because you_ know which

181 317 309 293 278 267 262 261 256 251 229 198 197 185 182 179 176 174 173 169 162 153 152 152

69 67 67 66 64 62 59 59 57 56

have this be not people they what for but on with there well ’re think n’t very about was if so he going_ to our who been will can no want which all as

56 55 54

has because know

109 107 107

54

country

104

148 140 139 130 121 119 117 116 114 113

Table 7-4. Frequency lists in Clinton’s and Obama’s turns and in the CIFP and CIMP

182

Chapter Seven S

As is evvident from Table T 7-4, th he contingent situation of the joint interview haas repercussioons on word frequency, ass president is the most frequent lexxical word in Clinton’s C turns and Hillary is the most frrequent in Obama’s. A All groups shaare the word think, and thhree the word d people, apart from Obama who uses world more m frequenntly. In additiion, male politicians uuse the modal want most offten, whereas female politiicians use need more fr frequently.

Fig. 7-5. CIFP P key domain cloud c

The key domain clouud—as for Cliinton and Obaama calculateed against the BNC Saampler CG (S Spoken) Instiitutional—for the CIFP in Fig. 7-5 shows a few w similarities to and many differences ffrom Clinton’s speech. Apart from m predictablee topics su uch as ‘Pollitics’, ‘Goveernment’, ‘Geographiccal_names’ annd ‘People’, the t main dom mains in the CIFP are ‘Location_annd_direction’, ‘Mental_objecct’, ‘Sensory:__Sight’, ‘Toug gh/Strong’ and ‘Weathher’. The first one dependss on the stronng tendency to t anchor discourse deeictically throough spatial pronouns, p adjeectives or adv verbs; the label ‘Menntal_object’ innstead refers to the desscription of measures employed iin the politiccal action deescribed by tthese politiciians, i.e., solution, appproach, style etc., whereas the label ‘Sennsory’ refers to t careful analysis of critical situatiions through verbs of visuual perception n like see, look, and waatch. The most unexpected d label is ‘Weeather’, which h provides evidence off concern not only for the precarious p clim mate conditio ons of our planet, but aalso for speciific weather disasters d that ooccurred in th he United States. Finallly, the label ‘Tough/Stron ng’, which is ssimilarly typiccal in the CIMP (Fig. 7-6), accounts for an emph hatic way of deescribing even nts.

Gender-related Communicatiion in Political Interviews

183

MP key domain cloud Fig. 7-6. CIM

In the keey domain clooud for the CIIMP illustrate d in Fig. 7-6, the most relevant labeels are ‘Cause//Effect’, ‘Chan nge’, ‘Degree:__Boosters’, ‘H Hindering’, and ‘Strongg_obligation_oor_necessity’. The first onne is motivateed by an expository style aimed at describing or explainning events, therefore adhering to a consequenttial logic. Thee other labels are all instrum mental to persuasion: politicians talk about corrections of present problems (‘Change’) very often em mploying mo odals of neceessity (need, ought to, should, havee to), nouns such s as comm mitment, and nnecessity, and boosters, to reinforcee the strengthh of their speech (e.g., verry, really parrticularly, strongly). Innterestingly, the t label ‘Hin ndering’ is alsso quite typiccal in the field of poliitical affairs, as politics itsself includes nnot only goveernmental activities buut also initiativves taken by organizations o (e.g., political parties), or people w who try to haave their say in the way a country is managed (Wilson 20003; van Dijk 2002, 2 2006). This often enntails opposition and is reflected in words havinng to do with fighting. M Metaphors of war and conflict are very often reesorted to in order o to descrribe everyday y political confrontatioons (Howe 19888). With resspect to the keey semantic do omains that em merged from the CIFP and the CIM MP, what disttinguishes Cliinton’s turns iin the interview is the emphasis onn the ‘Unexppected’, as sh he decides to talk about surprising s things ratherr than expecteed or likely ev vents. This is vvery much in line with other domaains, e.g., ‘Content’, whicch expresses her enthusiaasm (I’m proud, thannkfully), and the t use of ‘D Degree_Boostters’, i.e., inttensifying adverbs suchh as very, reaally, deeply, in ncredibly, andd a lot. Obam ma’s turns are characteerised by a wider w array of labels, even though the number of words is sliightly less nuumerous than n that of Clinnton’s part (1,400 vs. 1,564). In pparticular, sim milarly to Clin nton, he talks about many important i issues that cconcern not onnly the United States but tthe entire worrld, hence

184

Chapter Seven

the presence of the label ‘The_Universe’. The President projects the local onto the global scene, and mentions ‘Crime’ but within a discourse where there are also references to ‘Change’, ‘Confidence’ and ‘Success’, therefore redressing the balance in favour of a positive picture. A comparison of key domain clouds of the CIMP (Fig. 7-5) and the CIFP (Fig. 7-6) discloses both similarities and differences. An interesting feature in the CIMP is the presence of labels that point at areas that balance each other out, such as ‘Helping’ and ‘Hindering’, and of strong language in general, represented by labels such as ‘Boosters’, ‘Tough/Strong’, ‘Size:_Big’, ‘Evaluation’ and ‘Positive_Evaluation’, ‘Trying_hard’, ‘Violent/Angry’, etc. In the CIFP, more attention is instead granted to domains such as ‘Mental_object:_Means,_Method’, ‘Expected’ and ‘Work_and_Employment’: the two former encompass references to a positive attitude towards the solution of problems and towards hope and expectations, with allusions to a brighter future; the latter points to a special concern for the current job situation. Expressions of degree are also present, but two categories can be observed, namely maximizers and boosters, whereas in the CIMP boosters are better represented. Interestingly, Clinton’s speech shares some features of the CIFP and CIMP, that is, respectively, concern for people’s working conditions, and reference to failure—especially her own—and the choice of emphasising her speech with boosters. This finding is in line with one of the features that has been indicated as typical of female speech-style, i.e., supportiveness. This is manifested here in the choice of topics that show orientation towards others and refer to personal experience (Herring 2001). Similarity between Obama’s turns and the CIMP is scanty: Obama’s lines display in fact some semantic domains that appear with a certain frequency in the CIFP, namely, ‘Work_and_Employment’, ‘Mental_object:_means,_method’ and ‘Crime’. Overall, Obama’s speech does not share the assertive and adversarial nature of male speech-style (e.g., the use of boosters and epistemic and deontic modals.)

3. Conclusions The verbal and kinesic behaviours adopted by Clinton and Obama in their joint interview do not appear to correspond to what would be considered prototypical for members of their gender communities. In the abundant literature on gender-related patterns of talk, agonism is identified as a typical trait of male speech, whereas females seem to seek intimacy and bonding (cf. among many, Tannen 1990; Coates 1996; Kendall and Tannen 2001).

Gender-related Communication in Political Interviews

185

Clinton often refers to herself (71 instances of I) and only secondarily to groups to which she belongs with an inclusive we (34, a few referring to herself and Obama, the majority to different entities, such as the US, the government, the Department of State, etc.), whereas Obama’s most frequently employed pronoun form is we, closely followed by I. After five years of presidency, Obama appears to refer more to the nation and to his institutional role than to himself as a person. Pronominal usage (I vs. we) in the CIFP and CIMP is also revealing of Clinton’s atypical and dominant speech style: both female and male politicians use we more frequently than I. Clinton balances this tendency by using the term people quite often. Indeed, it is the second noun for frequency of occurrence after president, in line with both the CIFP and CIMP. Clinton’s contribution in the analysed interview is quite enthusiastic, a trait which is conveyed both by her verbal and gestural codes: boosters and emphasisers appear alongside a whole array of beats that punctuate her discourse. Conversely, Obama’s gestures, rather stunted in comparison with the gestural behaviour he used in his first presidential campaign, not only highlight important concepts, but are also meant to facilitate the audience’s understanding in the form of iconic, indexical and metaphorical gestures. Overall, even though the object of this investigation was quite limited, comparison of data from similar genres has suggested that the topic of discourse and the circumstances of enunciation prevail over gender-related features. Similarly, the gestural behaviour in the various interviews seem to be more dependent on the narrative of discourse and personal tastes than on gender preferences: Obama has some preferred gestures, for example enumerating or representing items in his speech visually; Clinton has a wider repertoire, with an abundance of beats, even when verbal argumentation does not require her to press a point. Parameters that I have not yet taken into account but which seem crucial in creating rapport in interaction and deserve further investigation are the timing of gestures, i.e., whether they are sudden, brisk movements or slow ones, and size, i.e. whether they involve finger movements or whole arm sweeps. Finally, a brief look at Obama’s gestures in time, in the two presidential campaigns, indicates that the role of image consultants and communication experts must have left an influence on his public behaviour. As Clinton is currently involved in the U.S. presidential election, there is much scope for further investigation in these directions, both in case she wins and in case she loses.

186

Chapter Seven

References Baldry, Anthony, and Paul J. Thibault. 2006. Multimodal Transcription and Text Analysis: A Multimedia Toolkit and Coursebook. London: Equinox. Bateman, John. 2014. “Multimodal coherence research and its applications.” In The Pragmatics of Discourse Coherence: Theories and Applications, edited by Helmut Gruber and Gisela Redeker, 145– 177. Amsterdam: Benjamins. Bradac, James J., Anthony Mulac, and Sandra A. Thompson. 1995. “Men’s and women’s use of intensifiers and hedges in problem-solving interaction: Molar and molecular analyses.” Research on Language and Social Interaction 28:93–116. Clayman, Stephen, and John Heritage. 2002. The News Interview. Journalists and Public Figures on the Air. New York: Cambridge University Press. CNN. 2015. “Hillary Clinton: The full interview.” Accessed June 26, 2016. http://edition.cnn.com/videos/tv/2015/07/07/hillary-clinton-cnnexclusive-wxdc.cnn. —. 2016. “Clinton’s ‘me’ vs. Sanders’ ‘us’.” Accessed June 26, 2016. http://edition.cnn.com/2016/02/12/politics/hillary-clinton-berniesanders-pronouns/. Coates, Jennifer. 1996. Women Talk: Conversations with Women Friends, Oxford and Cambridge, MA: Blackwell. Crawford Camiciottoli, Belinda. 2015. “Elaborating explanations during OpenCourseWare humanities lectures: The interplay of verbal and nonverbal strategies”. In Multimodal Analysis in Academic Settings. From Research to Teaching, edited by Belinda Crawford Camiciottoli and Inmaculada Fortanet-Gómez, 144–170. London: Routledge. Ekström, Mats, and Åsa Kroon Lundell. 2009. “The news interview: Diversity and hybridity in the communicative activities of broadcast interviews.” Media Culture and Society 30(1):61–80. Fetzer, Anita. 2000. “Negotiating validity claims in political interviews.” Text 20(4):1–46. Furkó, Péter, and Ágnes Abuczki. 2014. “English discourse markers in mediatised political interviews.” Brno Studies in English 40(1):45–64. Accessed June 26, 2016. doi: 10.5817/BSE2014-1-3. Gubrium, Jaber F. 2012. The Sage Handbook of Interview Research: The Complexity of the Craft. London: Sage.

Gender-related Communication in Political Interviews

187

Herring, Susan C. 2001. “Computer-mediated discourse.” In The Handbook of Discourse Analysis, edited by Deborah Schiffrin, Deborah Tannen and Heidi E. Hamilton, 612–634. Oxford: Blackwell. Holmes, Janet. 1990. “Hedges and boosters in women’s and men’s speech.” Language and Communication 10(3):185–205. Howe, Nicholas. 1988. “Metaphor in contemporary American political discourse.” Metaphor and Symbolic Activity 3(2):87–104. Jucker, Andreas H. 1986. News Interviews: A Pragmalinguistic Analysis. Amsterdam/New York: John Benjamins. Kendall, Shari, and Deborah Tannen. 2001. “Discourse and gender.” In The Handbook of Discourse Analysis, edited by Deborah Schiffrin, Deborah Tannen and Heidi E. Hamilton, 548–567. Oxford: Blackwell. Kendon, Adam. 2004. Gestures: Visible Action as Utterance. Cambridge: Cambridge University Press. Lauerbach, Gerda. 2006. “Discourse representation in political interviews: The construction of identities and relations through voicing and ventriloquizing.” Journal of Pragmatics 38:196–215. Levinson, Stephen C. 1979. “Activity type and language”. Language 17:365–399. Lorenzo-Dus, Nuria. 2009. Television Discourse. Analysing Language in the Media. Basingstoke: Palgrave Macmillan. McNeill, David. 1992. Hand and Mind: What the Hands Reveal about Thought. Chicago: The University of Chicago Press. McNeill, David. 2005. Gesture and Thought. Chicago: Chicago University Press. Montgomery, Martin. 2007. The Discourse of Broadcast News. London, Routledge. Partington, Alan, and Charlotte Taylor. 2010. Persuasion in Politics. A Textbook. Milano: LED. Rayson, Paul. 2008. “From key words to key semantic domains.” International Journal of Corpus Linguistics 13(4):519–549. Roth, Stephanie S. 2014. “Power, politics, and gender-related epistemic modality in interview discourse. A case study of TV/video interviews with Canadian politicians.” BA thesis, University of Gothenburg. Accessed June 26, 2016. https://gupea.ub.gu.se/bitstream/2077/37146/1/gupea_2077_37146_1.p df. Tannen, Deborah. 1990. You Just Don’t Understand: Women and Men in Conversation. New York: Ballantine. Van Dijk, Teun A. 2002. “Political Discourse and Political Cognition”. In Politics as Text and Talk. Analytical Approaches to Political

188

Chapter Seven

Discourse, edited by Paul Chilton and Christina Schäffner, 204-236. Amsterdam: John Benjamins. —. 2006. “Discourse and Manipulation”. Discourse in Society 17(2):359383. Wildfeuer, Janine. 2013. Film Discourse Interpretation: Towards a New Paradigm for Multimodal Film Analysis. New York: Routledge. Wilson, John. 2003. “Political discourse.” In The Handbook of Discourse Analysis, edited by Deborah Schiffrin, Deborah Tannen and Heidi E. Hamilton, 398–415. Oxford: Blackwell. Wright, J.W. II, and Lawrence A. Hosman. 1983. “Language style and sex bias in the courtroom: The effects of male and female use of hedges and intensifiers on impression information.” The Southern Speech Communication Journal 48(2):137–152. YouTube. 2016. “President Obama’s interview with NPR’s Nina Totenberg - March 2016.” Accessed June 26, 2016. https://www.youtube.com/watch?v=BIoBwblIxws. —. 2012. “A side-by-side comparison of Obama’s 2008 and 2012 speeches.” Accessed June 26, 2016. https://www.youtube.com/watch?v=-C_x5JmHa7E.

CHAPTER EIGHT ANALYSING POLITICAL DISCOURSE IN FILM LANGUAGE: A MULTIMODAL APPROACH VERONICA BONSIGNORI UNIVERSITY OF PISA, ITALY

1. Introduction When trying to give a definition of politics, Chilton (2004) states that “however politics is defined, there is a linguistic, discursive and communicative dimension” (2004: 4). The central role of language in the political context is, in fact, recognised by many scholars (cf. Fairclough 1989; Schäffner 1997; Wilson 2015, inter alia). It is indeed nearly impossible to refer to any kind of political action that does not inevitably involve language (Partington and Taylor 2010). Political discourse is here intended as a class of genres defined by the domain of politics, but considered in a more limited and specific sense as the range of activities engaged by politicians, i.e., the “discourse of politicians” stricto sensu (van Dijk 2002: 20; cf. also van Dijk 2006; Fairclough and Fairclough 2012). As Wilson (2015) states, political discourse consists of “a piece of extended text or talk produced for or by political actors” (2015: 775). Therefore, political discourse is intended in its professional dimension as “a form of institutional discourse” (van Dijk 2002: 20; cf. section 4.2.), that is a kind of discourse produced by politicians in institutional settings, such as parliaments and political parties. Since ancient times, political discourse has always had a strong connection with rhetoric, as the main aim of the use of language in politics is for persuasion, characterised by Cook (1989: 26) as “seeking to affect the behaviour of the addressee”. This can be achieved with verbal communication that involves various types of linguistic strategies and rhetorical techniques such as parallelism, oxymoron, binomials, metaphors

190

Chapter Eight

and the like, as well illustrated by Partington and Taylor (2010). However, the paramount importance of non-verbal elements has also been acknowledged, as gestures, gaze, facial expression, head movements, and body posture express various communicative functions and can have a relevant persuasive import, especially in political communication. Several scholars have investigated this aspect, giving a general overview of the role of bodily behaviour in politics (cf. Atkinson 1984; Poggi et al. 2013), focussing on intonation and pauses (Bull 1986), or mainly on gestures (Poggi and Pelachaud 2008; Streeck 2008) and gaze (Poggi and Vincze 2008a, 2008b). More specifically, for instance, in their studies, Poggi and Vincze (2008a, 2008b) have singled out those gestures that convey meaning relevant to persuasion in political communication when analysing electoral debates of Italian and French politicians. Moreover, Streeck (2008) studied the use of gestures of the democratic presidential candidates during the U.S. 2004 primary campaign. His analysis highlighted the reiteration of one specific hand gesture (i.e., the so-called “finger wag”) used by the candidate Howard Dean that was perceived as a “hierarchical act” by the audience, expressing presumption and supposedly costing him the election. Gestures and gaze, as well as all the other non-verbal elements, can either contribute to expressing meaning along with speech or can convey meaning per se. Either way, different modalities are exploited in order to persuade others, thus producing “multimodal persuasive discourses” (Poggi and Vincze 2008a: 47). For this reason, it is important to use an annotation scheme that allows the analyst to take into account all these signals and their functions. Indeed, in the present work, the theoretical frame of reference is multimodal discourse analysis (cf. O’Halloran 2004; Scollon and Levine 2004; Norris 2004), which centres on the ways in which the various semiotic resources are intertwined and make meaning in a given situational context, so that both aural and visual elements are fundamental in order to fully interpret the message. As pointed out by Jewitt (2009), “language is therefore only ever one mode nestled among a multimodal ensemble of modes” (2009: 15). The fact that all semiotic resources, although in different ways, contribute to making meaning in combination is a key concept in multimodality from the first and pioneering work by Kress and van Leeuwen (1996) up until more recent studies (cf. Jewitt, Bezemer, and O’Halloran 2016).

Analysing Political Discourse in Film Language

191

2. The Film as a Multimodal Text Audiovisual products have widely proven to be ideal material for multimodal studies, due to their intrinsic multisemiotic nature in which each mode contributes to the construction of meaning. In particular, as Bateman (2013) states, “film is an obvious candidate for consideration as a multimodal artefact” (2013: 4030), due to its complex inner nature, as both audio and visual channels are involved. For this reason, several recent studies focus on films, with the aim of analysing the film text multimodally in order to understand how each semiotic resource contributes to conveying meaning (cf., inter alia, Bateman and Schmidt 2012; Wildfeuer 2013). Everything plays an important role in the film construct, nothing is left to chance, and “everything in the frame has been put there for a reason” (Kaiser and Shibahara 2014: 3), ranging from elements pertaining to the mise en scène, colours, sounds, costumes, to more technical details such as camera angles, zooming, types and length of shots, etc. For instance, with reference to film rhythm, van Leeuwen (1985: 216) states that it is “created by the filmmaker, more specifically by the film editor”, describing it as follows: It is a view which sees film rhythm as resulting from an alternation between what could be called ‘upbeat’ and ‘downbeat’ shots which are rhythmically accented (and hence have greater perceptual conspicuity and ‘psychological’ impact) and shots which are not. The perceptual conspicuity which causes the shot to be perceived as ‘accented’ is seen as created by the director’s or photographer’s choice of size of frame, and by the editor’s manipulation of the duration of shots: film rhythm is articulated by the technical manipulation of material. (1985: 217)

Therefore, when analysing film dialogue, it is fundamental to take into account all these elements, as they are part of the narrative, more or less explicitly. Moreover, the film text provides a complete set of tools that allow the analyst to fully understand each sequence: the communicative exchange can be situated in a precise situational context and language can be studied along with gestures, prosodic features, kinesics, proxemics and the like, since “when analysing communication dynamics, it is fundamental to take into account not only what is said, but also how it is said” (Bonsignori 2013: 1). As previously stated in the introductory section of this chapter, authentic political discourse is strongly characterised by bodily behaviour (Poggi and Pelachaud 2008), which has also a persuasive import in the communication. Therefore, it seems particularly interesting to study

192

Chapter Eight

political drama films, as they represent a good opportunity to analyse a quite faithful representation of this type of specialised discourse. Indeed, even though the fictional character of film language in general is supported by many scholars (cf., inter alia, Gambier and Soumela-Salme 1994; Chaume 2004), recent studies have demonstrated the strong similarities of this language genre and spontaneous conversation (cf. Kozloff 2000; Forchini 2012; Bonsignori 2013). The present paper intends to explore the ways in which non-verbal elements co-occur with certain rhetorical strategies often employed in political discourse, such as parallel structures (Beard 2000; Partington and Taylor 2010), paying particular attention to the use of non-verbal elements and how they are used to make meaning in different political contexts. For this purpose, some extracts from two political drama films—namely The Ides of March (2011, George Clooney) and The Iron Lady (2011, Phyllida Lloyd)—will be analysed adopting a multimodal approach to gain insights into how various semiotic resources work together to construct meaning in political discourse.

3. Data and Methodology 3.1 Data The data used for the present study consist of some selected sequences taken from two famous political drama films: namely The Ides of March (2011, George Clooney; screenplay by George Clooney, Grant Heslov, Beau Willimon) and The Iron Lady (2011, Phyllida Lloyd; screenplay by Abi Morgan). Both films are quite accurate in the representation of political discourse on the one hand and political contexts on the other. More specifically, The Ides of March1 is an adaptation of the critically acclaimed Broadway play Farragut North (2008), written by the American playwright and screenwriter Beau Willimon, and is loosely based on his personal experience as press aide for Howard Dean’s presidential campaign in 2004 (CUE 2014). Prior to Dean’s campaign, Willimon had also worked on campaigns for Charles Schumer, Bill Bradley, and Hillary Clinton, and his expertise was so solid that, immediately after The Ides of March, he was contacted by producer David Fincher, who invited him to work on the American remake of the UK political miniseries House of Cards (CUE 2014). The Ides of March is about insider politics and tells 1 For more detailed information on The Ides of March, see http://www.imdb.com/title/tt1124035/?ref_=nv_sr_1

Analysing Political Discourse in Film Language

193

the story of a junior press secretary, Stephen Meyers (Ryan Gosling), for Pennsylvania Governor Mike Morris (George Clooney), a democratic presidential candidate who is competing against Senator Ted Pullman in the Democratic primary. As Willimon himself states in an interview about Farragut North, but which can also be extended to its cinematographic adaptation, “(t)he play is about people who become so entrenched in the political process that they ultimately find themselves working day in and day out for the pure joy of winning, not necessarily for the platform that their candidate is propagating” (Wallenberg 2010). There was some speculation at the time of the film’s release about who mostly inspired the character of Governor Morris. Apart from the obvious reference to Howard Dean, Barack Obama was also mentioned, as Morris is described as “a soulful and articulate Obama-in-2008-esque candidate who is promising a new kind of politics” (Gleiberman 2012), not to mention the strong similarities between Morris’ “Believe” campaign billboard in the film and Obama’s “Hope” poster. The Iron Lady2 is a biopic based on the life of British and first woman Prime Minister Margaret Thatcher, interpreted in a masterly manner by Meryl Streep, who won the Academy Award for Best Actress for this role in 2012. It is reported that the actress watched several videos and broadcasts of Margaret Thatcher to prepare for the role and learn her way of speaking and mannerisms.3 For instance, there is a scene in the film where Margaret Thatcher/Meryl Streep arrives at 10 Downing Street for the first time as Prime Minister in 1979 and gives a public speech to journalists and people who went there to celebrate her for her election and she recites, paraphrasing them, the words of St. Francis of Assisi.4 If we compare such a scene with the real videos of the time, it is astonishing how well Meryl Streep portrayed her, not only by faithfully reproducing her tone, tempo and pronunciation, but also in the way of moving her head and directing her gaze, as she even paused and looked down while reciting St. Francis’s words in the same way as real Margaret Thatcher did.

2

For further information on The Iron Lady, see http://www.imdb.com/title/tt1007029/?ref_=nv_sr_1 3 See the trivia section of the film on the Internet Movie Database. 4 The paraphrased words from St. Francis’s prayer are: “Where there is discord may we bring harmony, where there is error may we bring truth, where there is doubt may we bring faith, where there is despair may we bring hope”.

194

Chapter Eight

3.2. Methods of analysis As the initial step of the analysis, both films were carefully viewed in their entirety to select some meaningful scenes based on the presence of relevant verbal rhetorical strategies typically employed in political discourse for persuasion, such as parallelism, metaphor, simile, etc. (Beard 2000; Partington and Taylor 2010). Clips were created for each of the selected scenes (10 clips from The Ides of March and 9 clips from The Iron Lady). Then, the dialogues in each clip were transcribed and each clip was thoroughly analysed taking into account any crucial non-verbal signals that could be clearly observed more or less occurring with speech, including gestures, gaze, facial expression, head movements, and proxemic features, as well as paralinguistic stress. The analysis was carried out with the help of the multimodal annotation software ELAN in its latest version 4.9.4. (Wittenburg et al. 2006),5 which allows the researcher to investigate how verbal and non-verbal elements interact to generate meaning (cf. Baldry and Thibault 2006; Wittenburg et al. 2006; Wildfeuer 2013) within the theoretical framework of multimodal discourse analysis (cf. section 1). The analytic framework created in the ELAN software is multi-levelled and consists of the following tiers: Transcription, Gesture_description, Gesture_function, Gaze, Face, Head, Prosody and, finally, Notes. The first tier labelled Transcription contains the exact words uttered by the speaker and is synchronised with the streaming video. If there is more than one speaker in the clip, different transcription tiers can be created and labelled with the characters’ names. This tier is very important because it helps visualise the linguistic material and the strategies used by the speakers. The following tier is Gesture_description, which hosts all the gestures performed by the speaker and placed in correspondence with the words with which they co-occur. The various types of hand gestures are labelled with abbreviations and refer to Querol-Julián’s (2011) descriptive framework: for example, “PsUMfss”, corresponding to “palms up moving from side to side”; however, those that were not present in her analysis were invented ad hoc following the same system. The third tier is Gesture_function, referring to the function expressed by the gesture in question. Two main frameworks were used in this case, namely those developed by Kendon (2004) and Weinberg, Fukawa-Connelly, and Wiesner (2013). Kendon (2004) identifies three main pragmatic functions 5

ELAN was developed at the Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen, The Netherlands. It is freely available at http://tla.mpi.nl/tools/tla-tools/elan/.

Analysing Political Discourse in Film Language

195

of gestures: modal, to express certainty, performative, to show the kind of speech act, and parsing, to mark different units within an utterance, while Weinberg, Fukawa-Connelly, and Wiesner (2013) distinguish the following functional gestures: indexical, to indicate a referent, representational, to represent an object or idea, and social, to emphasise or highlight the importance of the message or to involve the audience more. The Gaze tier is associated with various directions such as “up”, “down”, “out”, “left”, “right” and “back”, while the Face tier includes only the two annotations “smiling” and “frowning”. The Head tier, which comprises annotations such as “QN” for “quick nod”, and “MSsts” for “moving slowly from one side to the other”, was included only in the analysis of clip 3 (cf. Section 4.3.). Finally, in the Prosody tier the annotation “stress” is used whenever specific words are uttered by the speaker with a paralinguistic stress in order to give particular emphasis to certain key concepts in the political speech. Indeed, voice quality with all its components such as loudness, pitch range, etc., is used to make meaning as well as gestures (van Leeuwen 2009). One last comment is in order and it refers to the last tier, namely the Notes tier, where annotations regarding camera angles or the audience’s behaviour are inserted. As we shall see in the section devoted to the analysis of the three clips, this tier is very important when analysing multimodal products like films. Table 8-1 below illustrates all the tiers created in the software, followed by examples of the descriptions of their components and the corresponding abbreviations. Tiers Transcription Gesture_description

Gesture_function

Gaze

Description

Abbreviation

fist up moving forward

FiMf

open hand moving forward to the right to indicate position to express certainty to mark different units within an utterance to indicate the kind of speech act to represent an object/idea to emphasise/ highlight importance up down back left right out looking at the discussant

OHMfr indexical modal parsing performative representational social up down back left right out Ldis

196 Face Head

Prosody Notes

Chapter Eight frowning smiling up & down quick nod left to right moving down to one side paralinguistic stress description of camera angles / audience

frowning smiling UD QN LR Mds stress

Table 8-1. Example of the annotation framework created in ELAN from clip 3

4. Analysis In the following section, three clips will be analysed: two from The Ides of March and one from The Iron Lady. They portray different political contexts, ranging from a public speech to a political interview, and with different kinds of audiences—e.g., supporters of the candidate speaking, members of Parliament, etc.—thus giving a more general overview of how political discourse can be represented in a film.

4.1. Clip 1: The political speech Clip 1 is taken from The Ides of March and lasts 32 seconds. The full scene shows a political debate with two adversary candidates for the Ohio democratic primary, namely Governor Mike Morris and Senator Ted Pullman, in the auditorium of Miami University, Ohio. They are on a stage and each one speaks to the audience, full of students, from behind a lectern. Opposite, in front of the audience, sits a moderator. However, this clip was classified as an example of monologic speech because it is centred on Governor Morris’s answer to a provocative question asked by his opponent, which he turns into a real speech. As Beard (2000) states, “political speeches are often written for the speakers, with leading politicians having teams of speech writers to prepare their material for them” (2000: 37). This system is faithfully reproduced in the film, since the opening scene of The Ides of March features Governor Morris’s junior press secretary, Stephen Meyers, rehearsing the same exact speech on the same stage, while checking technical details with the director just before the debate. The question asked by Sen. Pullman is “Would you call yourself a Christian?” and the following is the transcription of Gov. Morris’s answer/speech:

Analysing Political Discourse in Film Language

197

I am not a Christian or an atheist. I’m not Jewish or a Muslim. What I believe, my religion, is written on a piece of paper called the Constitution. Meaning that I will defend until my dying breath your right to worship whatever god you believe in, as long as it doesn’t hurt others. I believe we should be judged as a country by how we take care of people who cannot take care of themselves. That’s my religion. If you think I’m not religious enough, don’t vote for me. If you think I’m not experienced enough or tall enough, then don’t vote for me, because I can’t change that to get elected.

As can be seen, this speech is characterised by one of the most common rhetorical strategies used in political discourse for evaluation and persuasion (Partington and Taylor 2010), namely parallelism. Parallel structures can take on various forms, according to the number of phrases or words repeated: when there are two, they are called bicolons, whereas when there are three, tricolons. Generally, in the three-part list, the last element is longer, in order to reach climax and be more effective. In this example, Gov. Morris’s speech begins with a bicolon and ends with a tricolon. The screenshot in Fig. 8-1 shows the multimodal analysis of the final part of the last tricolon. As shown in the Transcription tier, some words in the parallel structure are stressed; therefore the annotation “stress” appears in the Prosody tier below in correspondence with those words: experienced and tall. These adjectives, together with the adjective religious mentioned in the first part of the tricolon, are used by Gov. Morris to diminish the importance of the question asked by his opponent, an effect that is strengthened by the redundancy of the parallel structure itself. In addition, these key-words are highlighted also by the repetition of the same gesture, annotated in the Gesture_description tier with the label “PUMf”, corresponding to “palm up moving forward”, as shown in the still image of the streaming video. These gestures all perform a social function, with the only aim to attract the audience’s attention and to paradoxically give emphasis to those words. The effect of ridiculing Sen. Pullman’s question is finally achieved at the end of the tricolon, when Gov. Morris pronounces the last words smiling, as annotated in the Face tier and as is visible in the still image in Fig. 8-1.

Chapter Eight

198

Fig. 8-1. Multimodal analysis of parallelism from clip 1 – political speech

Table 8-2 illustrates the number and the type of gestures used by Gov. Morris while giving his speech; the words in bold in the transcript column bear paralinguistic stress. It is surprising to notice that in a 32-second speech two different parallelisms are used and both are accompanied by some gestures (see the rows in light grey). In effect, out of the total 18 gestures employed, nearly half of them (N=8) occur with the parallel structures. Moreover, there are ten different types of gestures, of which three types are used more than once, and they perform all functions, even though the social function appears to be dominant. N

Transcript

1

2

I am not a Christian or an atheist. I’m not Jewish

3

or a Muslim.

4

What I believe,

5

my religion,

Gesture - detailed description palm up, thumb & forefinger forward

Gesture abbrev PU-TFf

Gesture function parsing

palm up moving back & forth palm up moving up & down palm up moving up & down palm up moving up & down

PUMbf

parsing

PUMud

parsing

PUMud

modal

PUMud

social

Analysing Political Discourse in Film Language 6

called the Constitution.

7 8

Meaning I believe we should be judged

9

as a country

10

by how we take care of the people

11

who cannot take care of themselves. That’s my religion.

12 13 14

15

16 17 18

If you think I’m not religious enough, don’t vote for me.

If you think I’m not experienced enough or tall enough, then don’t vote for me, ’cause I can’t change that to get elected.

closing palm up, forefinger & thumb forming a ring, moving up & down forefinger up closing palm up, forefinger & thumb forming a ring, moving up & down closing palm up, forefinger & thumb forming a ring, moving up & down closing palm up, forefinger & thumb forming a ring, moving up & down fist down, moving left & right fist moving down, forefinger forward forefinger pointing out open hand moving from one side to the other palm up moving forward palm up moving forward palm up moving forward palm up moving forward

199

CPFTRud

social

FU CPFTRud

social social

CPFTRud

social

CPFTRud

parsing

FdMlr

social

FMdFf

social

FP

indexical

OHMfso

representational

PUMf

social

PUMf

social

PUMf

social

PUMf

social

Table 8-2. Summary of gestures from clip 1 – political speech

4.2. Clip 2: The political interview Clip 2 is also taken from The Ides of March, but it depicts a different political context, in this case a political interview. Partington and Taylor (2010) define political language as “one form of institutional discourse”

Chapter Eight

200

(2010: 114) of which the political interview is a spoken type in the form of questions and responses. The more general term “response” is preferred to “answer” in accordance with Harris’s (1991) idea that a question is usually followed by a response that is not necessarily able to successfully satisfy the questioner and, thus, cannot be considered a real answer. One important reason for the “evasiveness of politicians” (Harris 1991: 77) can be found in the development of modern media which permit the recording, storage, and reproduction of information so that politicians have to be extremely careful with what they say (Beard 2000). In addition, there has been a change in the relationship between political figures and the press, which tends to be represented also by journalists who have achieved a high profile and respectability, and which wishes to show its independence and reliability to the public, often preferring confrontational and hostile questioning (Jones 1996). The types of questions that can be asked are basically three: 1) yes/no questions, with the variant of tag questions, which tend to be rather “conducive” (Hudson 1975) in this context, 2) wh-questions, and 3) the socalled “alternative questions” (Beard 2000: 99), which contain the conjunction or, thus offering the chance to give alternative responses. However, questions can be more difficult for the politician when they contain an assumption and a presupposition (Beard 2000; Partington and Taylor 2010). An assumption is an open statement, while a presupposition entails taking for granted that what the questioner asserts is the position shared by the interviewee, thus challenging him to admit it. As Partington and Taylor state, “criticism is often hidden in the question’s presuppositions” (2010: 117) and this is why the politician is forced to be evasive in these cases. Turning now to clip 2, this is a one-minute long episode and shows in medias res a part of an interview for the primary elections between Governor Mike Morris and an actual and widely known journalist, Charlie Rose, who made a cameo in the film, thus creating a more authentic flavour. The interview takes place in a hotel conference room turned into an interview space, and the two participants are surrounded by members of Gov. Morris’s staff and by the TV crew. The following is the transcription of the communicative exchange of clip 2, with each turn numbered: 1 GOV. MORRIS

2 CHARLIE ROSE

Is it a difficult decision? I have to believe it is. Eh… Would I do it? No. But… I can’t see myself or anyone, certainly not a government, telling a woman what she should do with her body. So, you would appoint a judge?

Analysing Political Discourse in Film Language 3 GOV. MORRIS 4 CHARLIE ROSE 5 GOV. MORRIS 6 CHARLIE ROSE 7 GOV. MORRIS 8 CHARLIE ROSE 9 GOV. MORRIS 10 CHARLIE ROSE 11 GOV. MORRIS 12 CHARLIE ROSE 13 GOV. MORRIS 14 CHARLIE ROSE 15 GOV. MORRIS 16 CHARLIE ROSE 17 GOV. MORRIS

201

I would consider it arrogant to judge anyone until I’ve walked in their shoes. But you’re against the death penalty? Mm-mm. Because of what it says about us as a society. Suppose, governor, it was your wife. And she was murdered. What would I do? It gets more complicated when it’s personal. Sure. Well, if I could get to him, uh… I would find a way to kill him. So you, you, governor, would impose a death penalty! No, I would commit a crime for which I would happily go to jail. Then why not let society do that? Because society has to be better than the individual. If I were to do that, I would be wrong. What about guns? Isn’t it time for a commercial? (laughing) This is public television. We don’t have commercials. (laughing) That’s unfortunate!

As can be seen, various types of questions are present, ranging from wh-questions (e.g., why not let society do that? in turn 12) to implicit yes/no questions, as in turns 2 and 4, when journalist Charlie Rose makes a statement with a final rising tone, typical of questions, as a way to ask for confirmation. Interestingly, some of them are anticipated by Gov. Morris himself—e.g., in turn 1, Is it a difficult decision?, Would I do it?—as a way to show the journalist that he is prepared and that he definitely knows what they are talking about, i.e., the death penalty. Moreover, at the end of the sequence, in order to avoid answering a really tough and blunt question about guns (turn 14)—an extremely delicate matter in the United States— Gov. Morris replies with a question, namely a negative interrogative (turn 15), to change topic with a joke, which makes the audience and the journalist himself laugh in a more relaxed atmosphere. This is an example of the evasive politician. Another way to be evasive is by using nonlexical backchannels such as mm-mm in turn 5, instead of the clearer and straightforward answer yes, as if the interviewee did not want to expose himself too much. Conversely, the interviewer often uses the vocative governor (turns 6 and 10) in order to underline a sort of “paternity” of the interviewee’s statements, thus alleging responsibility. Indeed, Fig. 8-2 shows the multimodal analysis of turn 10, when the journalist quite aggressively indirectly asks Gov. Morris to confirm his assumption with a

202

Chapter Eight

challenging tone by repeating twice the second person pronoun you and using the vocative governor. As shown in the Prosody tier, these three words, together with the following impose and death, bear a paralinguistic stress—except for governor—and are accompanied by the same gesture, namely “OPDhT”, corresponding to “open palm down hitting the table”, performing a social function. Considering the previous part of the exchange, this indirect question constitutes the climax of the interview, since it proves to be the real question the journalist wanted to ask Gov. Morris in the first place.

Fig. 8-2. Multimodal analysis of Q&R from clip 2 – political interview

It is worth pointing out that in the Notes tier we can find information about camera angles; in this case the camera is on Gov. Morris’s back and Charlie Rose’s front, in medium shot. These details are extremely useful when analysing audiovisual products such as films: the camera angles and the types of shots are integral part of the film because they contribute to the making of the narrative. For example, as Fig 8-3 shows, while the two interlocutors are speaking during the interview, at the end of Charlie Rose’s question So, you would appoint a judge?, which overlaps with Gov. Morris’s initial part of his response I would consider it arrogant […]—see the two Transcription tiers—the camera moves away from them and frames Gov. Morris’s staff, who, despite being in the same room, are watching the interview on various screens, in order to check the outcome of the interview and especially the potential effects on the public.

Analysing Political Discourse in Film Language

203

Fig. 8-3. Multimodal analysis of clip 2 – camera angle on audience

As a consequence, the two interlocutors are off-screen, but we can still hear them talking, which of course has an impact on the analysis of the communicative exchange. In effect, referring to Table 8-3, which summarises the analysis of gestures in clip 2, we can notice that six types of gestures are employed, for a total of 10 in the whole sequence. This is a very low number compared with the 18 gestures in clip 1, which is even shorter. However, it is true that in clip 2 there are many sequences in which the two speakers are off-screen, which may explain the lower frequency of gestures in this clip. Table 8-3 shows that Gov. Morris uses three gestures, each one of a different type, whereas journalist Charlie Rose uses seven gestures of three different types. The main functions performed by gestures are modal for Gov. Morris and social for Charlie Rose. This can be explained by the different role of the two interlocutors, as the journalist mainly uses gestures to give the floor to the interviewee in order to get an answer, while Gov. Morris uses them to express a certain self-confidence so as to appear as much persuasive and convincing as possible.

Chapter Eight

204 N

Transcript

1

(GM) Eh… Would I do it? (GM) No! (GM) certainly not a government, (CR) So you would appoint a ju(dge)? (CR) It gets more complicated when it’s personal. (CR) So you, you, Governor, would impose a death penalty.

2 3 4 5

610

Gesture - detailed description opening palms up

Gesture abbrev OPsU

Gesture function social

closing palms down opening palm up towards interlocutor palm up moving forward joined hands to the chin

CPsD OPUtI

modal modal

PUMf

indexical

JHsC

modal

open palm down hitting the table

OPDhT (x5)

social

Table 8-3. Summary of gestures from clip 2 – political interview

4.3. Clip 3: The political speech in the House of Commons The last clip to be analysed is taken from The Iron Lady and lasts one minute and nine seconds. Clip 3 shows a sequence of a parliamentary sitting in the British House of Commons. The Prime Minister, Margaret Thatcher, stands at the dispatch box, surrounded by her own supporters and by the members of all the opposition parties, led by Michael Foot, who is seated opposite in the first row. It is 1982 and what follows is the transcription of the speech made by the Prime Minister to celebrate the victory in the Falkland Islands against Argentina: We congratulate the men and women of our armed forces for their skill, bravery, and loyalty to this country. We were faced with an act of unprovoked aggression, and we responded as we have responded in times past: with unity, strength, and courage! Sure in the knowledge that though much is sacrificed, in the end, right will prevail over wrong. And I put it to the Right Honourable Gentleman opposite that this is not a day for him6 to carp, find fault, demand inquiries. They will happen, I can assure him of that, for we have nothing to hide. No. This is a day to 6

As Beard (2000) points out, “the pronoun him, rather than you, is a product of the rules of parliamentary address, which say that speakers must refer to other members of the house in the impersonal third person” (2000: 107). In this speech, him is used twice and refers to the leader of the opposition party, Michael Foot.

Analysing Political Discourse in Film Language

205

put differences aside, to hold one’s head high and take pride in being British!

It is worth pointing out that the first sentence is pronounced by Margaret Thatcher, but she is off-screen and we can only hear her voice in the background, while real archive images of the Plymouth Victory parade and of people saluting the task force are shown, accompanied by the music played by the marching band. Overall, the speech is clearly planned as it is characterised by some recognisable rhetorical strategies: the use of two antonyms, at the end of the second paragraph in the phrase right will prevail over wrong, but most of all, four parallel structures in the form of tricolons, highlighted in bold in the text. As can be seen, even in these cases the third element is often longer than the preceding two (cf. Section 4.1.) as a way to heighten the rhetorical effect. Fig. 8-4 shows the multimodal analysis of the second tricolon.

Fig. 8-4. Multimodal analysis of a tricolon from clip 3 – political speech

Interestingly, in this case the tricolon is not accompanied at all by any gestures, which occur instead at the beginning of the following sentence, with “FiMf” (i.e., “fist up moving forward”), visible in the still image,

Chapter Eight

206

used in correspondence with Sure in the knowledge and performing a social function. However, when the three words of the three-part list are uttered (i.e., unity, strength and courage) in the Prosody tier the annotation “stress” is present, but above all the annotation “QN” standing for “quick nod” appears in the Head tier. This is particularly interesting since, referring to the overall multimodal analysis of clip 3, gestures are only used twice, while head movements are used 15 times, as shown in Table 84 below. N 1 2 N 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Transcript Sure in the knowledge This is a day to put differences aside

Gesture - detailed description fist up moving forward open hand moving forward to the right

Transcript We were faced with an act of unprovoked aggression, as we have responded in times past: with unity, strength, and courage! right will prevail over wrong that this is not a day for him to carp, They will happen, I can assure him of that, for we have nothing to hide. No. and take pride in being British!

Gesture abbrev FiMf

Gesture function social

OHMfr

representational

Head - detailed description up & down

Head - abbrev UD

quick nod

QN

quick nod quick nod quick nod moving down & to one side up & down up & down up & down

QN QN QN Mds

up & down up & down up & down left to right up & down up & down

UD UD UD LR UD UD

UD UD UD

Table 8-4. Summary of gestures and head movements from clip 3 – political speech Therefore, it seems that in clip 3 gestures are somehow replaced by head movements, which are generally used to mark different units in the utterance but, at the same time, also to give them a certain emphasis. As

Analysing Political Discourse in Film Language

207

shown by the light grey rows in Table 8-4, some elements of the various tricolons—if not all—are accompanied by head movements instead.

5. Concluding Remarks The present study has shown how political discourse as a type of specialised discourse can be analysed in a particular type of audiovisual product, namely the film. Indeed, due to their multisemiotic nature, films provide a wider set of elements that allows the analyst to gain a complete understanding of the whole product by adopting a multimodal approach. Particular attention was paid to the interplay of verbal and non-verbal signals (e.g., gaze direction, hand/arm gestures, body positioning, etc.) in three different situational political contexts (i.e., a political speech during a debate, a political interview, and a political speech in parliament). It was observed that hand gestures tend to co-occur with certain persuasive strategies (i.e., parallel structures) and are often accompanied by prosodic stress. Only in the last case regarding the British Prime Minister, Margaret Thatcher, hand gestures were replaced by head movements. Interestingly, viewing some British politicians giving a speech in parliament in clips available on the web7—thus in a similar situational context to the one shown in clip 3—despite the gender of the speaker, head movements appear to be a privileged non-verbal form of communication. Therefore, when comparing the film portrayal of American and British politicians, this may suggest that this difference could be cultural. However, of course, in order to ascertain this assumption, further investigation is needed. Another important point when analysing films relates to bearing in mind that technical details play a fundamental role in meaning-making and also in the film construct. Therefore camera angles may limit, so to say, the analysis of a speech event, as revealed in the case of clip 2, where the number of gestures was in fact far lower than in clip 1 from the same film. Finally, films in general and political films in particular can be useful both for research purposes and for teaching. In effect, despite their fictional character, films have often been used in the context of language learning (see among the most recent works, Kaiser and Shibahara 2014; Bruti 2015) because they are actually considered as “an authentic source material (that is, created for native speakers and not learners of the language)” (Kaiser 2011: 233). Consequently, the multimodal analysis of

7

See, for example, Prime Minister Tony Blair making his case for the Iraqi war in 2003 (https://www.youtube.com/watch?v=Pg9aEV9bcxs).

208

Chapter Eight

political films could provide useful tools for research and teaching in ESP contexts (Crawford Camiciottoli and Bonsignori 2015).

References Atkinson, Max. 1984. Our Master’s Voices. The Language and Body Language of Politics. London: Methuen. Baldry, Anthony, and Paul J. Thibault. 2006. Multimodal Transcription and Text Analysis. London: Equinox. Bateman, John A. 2013. “Multimodality and film”. In The Encyclopedia of Applied Linguistics, edited by Carol A. Chapelle, 4030–4033. Oxford: Blackwell. Bateman, John A., and Karl-Heinrich Schmidt. 2012. Multimodal Film Analysis: How Films Mean. London: Routledge Beard, Adrian. 2000. The Language of Politics. London: Routledge. Bonsignori, Veronica. 2013. English Tags. A Close-up on Film Language, Dubbing and Conversation. Newcastle-upon-Tyne: Cambridge Scholars Publishing. Bruti, Silvia. 2015. “Teaching learners how to use pragmatic routines through audiovisual material”. In Multimodal Analysis in Academic Settings. From Research to Teaching, edited by Belinda Crawford Camiciottoli and Imaculada Fortanet-Gómez, 213–236. London: Routledge. Bull, Peter. 1986. “The use of hand gestures in political speeches: Some case studies.” Journal of Language and Social Psychology 5(2):103– 118. Chaume, Frederic. 2004. Cine y traducción. Catedra: Signo e Imagen. Chilton, Paul. 2004. Analysing Political Discourse: Theory and Practice. London: Routledge. Cook, Guy. 1989. Discourse. Oxford: OUP. Crawford Camiciottoli, Belinda, and Veronica Bonsignori. 2015. “The Pisa Audio-visual Corpus Project: A multimodal approach to ESP research and teaching.” ESP Today 3(2):139–159. CUE Editor. 2014. “Beau Willimon on The Ides of March”. Columbia University Entertainment. March, 10. Accessed August 2, 2016. http://www.cue-network.org/sitefiles/index.php?option=com_content& view=article&id=136:beau-willimon&catid=20:alumniinterviews&Itemid=38. Fairclough, Norman. 1989. Language and Power. London: Longman. Fairclough, Isabela, and Norman Fairclough. 2012. Political Discourse Analysis. London: Routledge.

Analysing Political Discourse in Film Language

209

Forchini, Pierfranca. 2012. Movie Language Revisited. Evidence from Multidimensional Analysis and Corpora. Bern: Peter Lang. Gambier, Yves, and Eija Suomela-Salme. 1994. “Subtitling: A type of transfer.” In Transposes Culturales: Literatura, Cine, Traducción, edited by Federico Eguíluz, Raquel Merino, Vicky Olsen, Eterio Pajares and José Miguel Santamaría, 243–252. Vitoria-Gasteiz: Universidad del País Vasco. Gleiberman, Owen. 2012. “Review of The Ides of March”. Entertainment Weekly. January, 9. Accessed August 2, 2016. http://www.ew.com/article/2012/01/09/ides-march-review. Harris, Sandra. 1991. “Evasive action: How politicians respond to questions in political interviews”. In Broadcast Talk, edited by Paddy Scannell, 76–99. London: Sage. Hudson, Richard A. 1975. “The meaning of questions”. Language 51:1– 31. Jewitt, Carey. 2009. The Routledge Handbook of Multimodal Analysis. London: Routledge. Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran. 2016. Introducing Multimodality. London: Routledge. Jones, Nicholas. 1996. Soundbites and Spin Doctors: How Politicians Manipulate the Media and Vice-versa. London: Cassell. Kaiser, Mark. 2011. “New approaches to exploiting film in the foreign language classroom.” L2 Journal 3(2):232–249. Accessed August 2, 2016. http://escholarship.org/uc/item/6568p4f4. Kaiser, Mark, and Chika Shibahara. 2014. “Film as a source material in advanced foreign language classes.” L2 Journal 6(1):1–13. Accessed August 2, 2016. http://escholarship.org/uc/item/3qv811wv. Kendon, Adam. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Kozloff, S. 2000. Overhearing Film Dialogue. Berkeley: University of California Press. Kress, Gunther, and Theo van Leeuwen. 1996. Visual Images. The Grammar of Visual Design. London: Routledge. Norris, Sigrid. 2004. Analyzing Multimodal Interaction: A Methodological Framework. London: Routledge. O’Halloran, Kay L., ed. 2004. Multimodal Discourse Analysis: Systemic Functional Perspectives. London: Continuum. Partington, Alan, and Charlotte Taylor. 2010. Persuasion in Politics. A Textbook. Milano: LED.

210

Chapter Eight

Poggi, Isabella, and Catherine Pelachaud. 2008. “Persuasive gestures and the expressivity in humans and machines.” In Embodied Communication in Humans and Machines, edited by Ipker Wachsmuth, Manuela Lenzen, and Günther Knoblich, 391–424. Oxford: Oxford University Press. Poggi, Isabella, and Laura Vincze. 2008a. “The persuasive import of gesture and gaze.” In Proceedings of Workshop on Multimodal Corpora, LREC, Marrakech, 46–51. —. 2008b. “Persuasive gaze in political discourse.” In Proceedings of the Symposium on Persuasive Agents, vol. 3, AISB, Aberdeen. Accessed August 2, 2016. http://aisb.org.uk/convention/aisb08/proc/proceedings/03%20Persuasiv e%20Technology/Final%20vol%2003.pdf#page=61. Poggi, Isabella, Francesca D’Errico, Laura Vincze and Alessandro Vinciarelli, eds. 2013. Multimodal Communication in Political Speech: Shaping Minds and Social Actions. Berlin: Springer. Querol-Julián, Mercedes. 2011. Evaluation in Discussion Sessions of Conference Paper Presentation: A Multimodal Approach. Saarbrücken: LAP Lambert Academic Publishing. Schäffner, Christina. 1997. “Editorial: Political speeches and discourse analysis.” In Analysing Political Speeches, edited by Christina Schäffner, 1–4. Clevedon: Multilingual Matters. Scollon, Ron, and Philip Levine, eds. 2004. Multimodal Discourse Analysis as the Confluence of Discourse and Technology. Washington, DC: Georgetown University Press. Streeck, Jürgen. 2008. “Gesture in political communication: A case study of the democratic presidential candidates during the 2004 primary campaign.” Research on Language and Social Interaction, 41(2):154– 186. Van Dijk, Teun A. 2006. “Politics, ideology and discourse.” In Elsevier Encyclopedia of Language and Linguistics. Volume on Politics and Language, edited by Ruth Wodak, 728–740. Oxford: Elsevier. —. 2002. “Political discourse and Ideology.” In Anàlisi del Discurs Politic, edited by Clara Ubaldina Lorda and Montserrat Ribas, 15–34. Barcelona: Universitat Pompeu Fabra, IULA. Van Leeuwen, Theo. 2009. “Parametric systems: The case of voice quality.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 68–77. London: Routledge. —. 1985. “Rhythmic structure of the film text.” In Discourse and Communication, edited by Teun A. van Dijk, 216–232. Berlin: De Gruyter.

Analysing Political Discourse in Film Language

211

Wallenberg, Christopher. 2010. “A political drama with powerful ambitions.” The Boston Globe. April, 25. Accessed August 2, 2016. http://archive.boston.com/ae/theater_arts/articles/2010/04/25/playwrig ht_beau_willimon_talks_about_farragut_north_opening_friday_in_bos ton/. Weinberg, Aaron., Tim Fukawa-Connelly, and Emilie Wiesner. 2013. “Instructor gestures in proof-based mathematics lectures.” In Proceedings of the 35th annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, edited by Mara Martinez and Alison Castro Superfine, 1119. Chicago: University of Illinois at Chicago. Wildfeuer, Janina. 2013. Film Discourse Interpretation: Towards a New Paradigm for Multimodal Film Analysis. London: Routledge. Wilson, John. 2015. “Political discourse.” In The Handbook of Discourse Analysis, edited by Deborah Tannen, Heidi E. Hamilton, and Deborah Schiffrin, 775–794. London: Blackwell. Wittenburg, Peter, Hennie Brugman, Albert Russel, Alex Klassmann, and Han Sloetjes. 2006. “ELAN: A professional framework for multimodality research.” In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation Accessed August 2, 2016. http://www.lrec-conf.org/proceedings/lrec2006/pdf/153_pdf.pdf.

CHAPTER NINE A MULTIMODAL ANALYSIS OF DISCOURSE IN THE SOUTH AFRICAN COURTROOM: THE OSCAR PISTORIUS CASE DANIELE FRANCESCHI UNIVERSITY OF PISA, ITALY

1. Introduction Legal discourse has primarily been analysed in its written form (cf. Tiersma 1999, 2001, 2006; Tiersma and Solan 2012) through the examination of the recurrent features of various text types, e.g., contracts, wills, codes of law, etc., and often within the context of broader studies on specialised languages (Gotti 2008; Bhatia, Sánchez Hernández, and PérezParede 2011). Spoken legal discourse, on the other hand, has received far less attention and it has generally been viewed as a pure reflection of written legal language. Indeed, spoken legal English may also be ‘wordy’, ‘unclear’, ‘pompous’, and ‘dull’ (Mellinkoff 1963; Lindsey 1990). However, it presents its own idiosyncrasies and it thus needs to be addressed more in-depth with respect to its various forms and manifestations. One form of spoken legal discourse is trial language, whose distinctiveness has only relatively recently been recognised as a separate legal sub-genre (cf. Levi and Graffam Walker 1990; Stygal 1994; Heffer 2005, 2008, 2015; Anesa 2012). Studies on this micro-language are typically based either on U.S. or U.K. cases though, disregarding the immense diversity of trial systems in other English-speaking countries around the world. For instance, there appears to be just one main study on trial discourse in South Africa (Moeketsi 1999), which would seem to require greater attention due to its various peculiarities. The present paper therefore intends to fill this gap by providing a first in-depth analysis of communicative practices in a South African courtroom.

A Multimodal Analysis of Discourse in the South African Courtroom

213

Studies on trial language so far have typically investigated the lexicalsyntactic features and rhetorical strategies of this specific discourse domain (Heffer 2012, 2013), without considering that extra-linguistic factors (e.g., gaze, gestures and body movements) also contribute to meaning and thus represent important elements in specialised discourse analysis. As Kress and van Leeuwen suggest (2006), the visual, like other semiotic modes, has important communicational and representational functions.1 This paper, therefore, is an initial investigation of trial discourse from a multi-modal perspective, based on the examination of verbal and non-verbal strategies used in the courtroom during the Oscar Pistorius trial, which took place in Pretoria, South Africa in 2014. Since trial discourse is neither uniform or homogeneous by nature (i.e., not only does it consist of both monologic and dialogic phases, but each of these phases is also characterised by its own distinct features) for reasons of space the focus of attention here is confined to the examination and crossexamination of the accused, i.e. to the defence and prosecuting lawyer’s questioning techniques and speaking styles. In this way, I hope to provide a finer-grained description of at least one sub-domain instead of a more comprehensive but inevitably superficial analysis of trial language in general. The paper is structured as follows. Section 2 provides a description of the South African legal system and illustrates the different stages of a criminal trial; section 3 briefly summarises the recent literature on courtroom language; section 4 presents the story and case of the Paralympic athlete Oscar Pistorius, who was eventually convicted for the murder of his girlfriend, Reeva Steenkamp; section 5 outlines the methodology used for the analysis of the data; section 6 presents the core findings of the paper: first it addresses the strictly linguistic features of the examination and cross-examination of Pistorius by the defence and prosecuting lawyers, and then examines pictographs and kinetographs to see how and to what extent non-verbal elements support communication during this specific stage of the trial. Finally, section 7 summarises the results of the study. 1

The objects and elements appearing in visual images, i.e., the participants, are of two distinct types. “Interactive participants” are directly involved in the act of communication and are the ones who “speak and listen or write and read, make images or view them”; “represented participants”, on the other hand, are those “who constitute the subject matter of the communication; that is the places, people and things (including ‘abstract’ things) represented in and by the speech or writing or image, the participants about whom or which we are speaking or writing or reproducing images” (Kress and van Leeuwen 2006: 48).

214

Chapter Nine

2. The South African Legal System The South African legal system (cf. Zimmermann and Visser 1996; Du Bois 2007) is known for being very hybrid, reflecting the country’s history of successive colonial governance. Still today it is strongly influenced by Roman-Dutch law, especially in the fields of family and property law. Although in the Netherlands Roman-Dutch law was replaced by the Napoleonic Code at the beginning of the 19th century, it still survives in South Africa, where superior courts often cite the works of preeminent Dutch jurists in their judgements, such as Grotius, Van Leeuwen, and Voet. The influence of Roman-Dutch law is mostly evident in substantive private matters. When South Africa passed under the sovereignty of Great Britain between the end of the 18th and the beginning of the 19th century, various aspects of South African law had to be integrated with the concepts of English law. In particular, the legislative, executive, and judicial branches of the state relied on English precedents and English became the official language of the courts and of all legal procedures. This mixed legal system, however, often created tension, because certain Roman-Dutch institutions, which remained the foundations of South African law, did not necessarily have counterparts in the English law. During both periods of Dutch and English governance, certain minor matters, e.g., concerning marriage and succession, continued to be dealt with under indigenous law, known as African customary law, provided that it did not conflict with the rules of the colonial state. African customary law is preserved today, especially in some rural parts of the country, but it must conform to the current South African Constitution, which often bans previously accepted practices and traditions. South Africa has had a supreme Constitution and a Bill of Rights since the end of the apartheid regime in 1997. As far as South African criminal trials are concerned (cf. Kemp et al. 2012), the courts that adjudicate them may be: 1) district magistrates’ courts for minor criminal offences (other than treason, murder and rape) and imposing lighter punishments; 2) regional magistrates’ courts for all crimes, except treason, leading to more severe sentences and fines; and 3) high courts, which have jurisdiction over all offences and impose sentences up to and including life imprisonment. The latter courts can also hear appeals, which may then have to be heard by the supreme court of appeals and by the constitutional court for the final decision. All criminal trials consist of five distinct structural phases (see Table 9-1 below). From the discursive perspective, the beginning and end are monologic, while the

A Multimodal Analysis of Discourse in the South African Courtroom

215

middle stage or body of the trial is necessarily dialogic. Phase one (arraignment) is the moment when the defendant is called upon for the crime he/she has been accused of committing and is required to plead either guilty or not guilty to the charge. Phase two (opening of the case) begins only if the defendant pleads not guilty to the charge made against him/her. At this stage the prosecutor presents the charge and introduces the evidence for the benefit of the magistrate, who, as a neutral participant is—at least theoretically—not aware of the contents of the prosecutor’s docket or of the defence file. Phase three (examination) represents the ‘core’ of the trial, during which the litigants, (i.e., the state prosecutor and the defence lawyer) question the defendant and the witnesses to put their evidence before the court. Like phase two and three, phase four (closing argument) is also monologic in that the prosecutor and the defence, after adducing the evidence, address and try to convince the court of the plausibility of their versions. Finally, in phase five (verdict), the magistrate/judge, after hearing the evidence and considering all the issues raised in phases one to four,2 renders a verdict and imposes a sentence as prescribed by the law. Phase 1 Phase 2 Phase 3

Phase 4 Phase 5

Arraignment Case for the prosecution Opening of the state’s case Examination Examination-in-chief/direct examination Cross-examination Re-examination Closing addresses, by prosecution and defence Summation, verdict and sentencing by the magistrate

Table 9-1. The five phases of a criminal trial

3. Courtroom Language The two main features of courtroom language are its hybrid nature and intrinsically narrative structure (Heffer 2015). The reasons for its hybridity reside in the asymmetric relations among the participants involved in the trial, which are reflected in the choices made primarily at the lexical and rhetorical level. The latter are to a large extent dictated by the highly institutionalised, standardised, and complex context of the trial itself, but 2

Unlike in the U.S.A. and U.K., criminal trials in South Africa are not decided by jury, since the jury system was abolished in 1969.

216

Chapter Nine

they also need to be viewed as an expression of power on the part of the legal professionals operating in the courtroom. Judges and lawyers generally adopt a formal style and use legal jargon, which may be perceived as opaque or even unintelligible to the witnesses, and the defendant. While prosecutors are more likely to resort to ambiguous as well as ‘violent’ language, defence lawyers are instead generally clearer and more intelligible. Stygal (1994) takes this point much further: in her analysis of the language used by trial lawyers working on a tort law case in the United States, she argues that unintelligibility is the product of an intentional discursive practice aimed at ‘changing’ the final verdict. Stygal views the ambiguous, incoherent, and alienating performance of the plaintiff’s counsel as eventually causing the defence lawyer to ultimately lose the case. Although bad lawyering may certainly affect the course of a trial, Stygal’s conclusion is hardly generalizable because it is based on just one case and is therefore idiosyncratic (Ainsworth 1995). Heffer (2005) confirms that judges have a tendency to rely on legalese, but his analysis of over 100 criminal jury trials in the United Kingdom also shows an extraordinary ordinariness of the language used in the courtroom, especially at the word and sentence level. Lawyers appear to prefer legal terminology only in opening and closing speeches when explaining the charges, but otherwise they want to be as clear and simple as possible in order to be readily understood. Complex legal language remains prevalently in judges’ instructions to juries, while lawyers address juries in a more ordinary and plain way. Heffer (2012, 2013) examines various components of trial language, i.e., not just the judges and lawyers’ rhetorical strategies, but also how witnesses present testimony, for instance. As for the examination and cross-examination phase, Heffer observes that in direct examination lawyers usually follow tightly controlled steps aimed at putting together a chronological summary of the facts and ask open or gently guiding questions. By contrast, crossexamining lawyers need to test the evidence and thus adopt a series of rhetorical devises aimed at intimidating and discrediting the witness before the jury and the judge. This is typically achieved through repeated or reformulated questions, the use of presuppositions, strategic silence, etc. In addition, cross-examiners tend to make a lot of declarative statements (e.g., You lied), often reinforced by question tags (e.g., You lied, didn’t you?) (Biscetti 2006). Matoesian (2001) has also shown that the language of the crossexamination phase can be particularly aggressive, taking the form of coercive questioning and sometimes leading to a repeated ‘victimisation’ of the defendant. Such speaking style is without doubt an expression of

A Multimodal Analysis of Discourse in the South African Courtroom

217

power on the part of the prosecutor, who peremptorily tries to demonstrate that the defendant is guilty of the charges brought against him/her. When defence lawyers question the accused and the witnesses instead, they tend to adopt a ‘mixed’ speaking mode, characterised on the one hand by the typical features of forensic discourse, e.g., the use of formulaic clichés and legal terms, and, on the other hand, by a more general conversational style. The hybridity of courtroom language is largely due to the presence and interdependence of written and spoken modes. During the questioning of the witnesses and the defendant, for instance, lawyers typically rely on a series of prepared questions, which they then need to adapt to the requirements of the communicative exchanges they engage in. Therefore, there is a combination of relatively planned and unplanned elements, producing a formal and technical register in certain circumstances and a more ordinary one in others, with several instances of oversimplification as well. Despite the highly formalised context of jury trials, lawyers inevitably add an individual and personal “flavour” to it. Cotterill (2003: 10) argues that there is an “overwhelming supremacy of the oral over the written in court”, because the dexterity of the lawyer “becomes a significant factor in the presentation of credible testimony”. The rhetorical mode par excellence in trial discourse is that of narration (Cotterill 2003; Anesa 2012; Heffer 2005, 2012, 2013), even if it does not necessarily follow a linear or precise structure. The reconstruction of past events, which is the result of a complex phenomenon of coauthorship, always entails some form of ‘emplotment’, be it narration proper or a more general narrative practice, involving story negotiation (i.e., making sure that some key elements of the case discussed are admitted into court and taken into due consideration) and the maintenance of character identity. The whole trial has an intrinsic, albeit rudimentary, narrative pattern, and may thus be seen as a macro-narrative, consisting of a series of micro-narratives provided by the different interactants involved in the communicative event. As Heffer (2015) puts it, “trial lawyers present their ‘master narrative’ of the case during their opening statement and then ‘navigate’ that narrative through the trial” (2015: 7). During the examination and cross-examination phase, for instance, the questions they pose and the answers they receive make it possible for different types of narratives to emerge. According to Heffer (2015: 2-6), defence lawyers tend to prompt “evidential narration”, whereas prosecutors are associated with “antagonist narration”, marked by a higher degree of unidirectionality. Although “it is evident that in the context of a trial verbal communication is always intertwined with non-verbal aspects” (Anesa 2012: 111), and that narration is produced through various means and

218

Chapter Nine

strategies, past studies have entirely disregarded how narrators within the courtroom intertwine a multiplicity of modalities. Therefore, this aspect will be brought more into the fore in the following sections. In addition, an initial account of the language used in a South African criminal court will be provided in that it presents different peculiarities compared to the ones discussed in the literature so far, which has been almost exclusively concerned with communicative practices and interactions within American and/or British courtrooms. Moeketsi (1999) deals with the structure of South African courtroom discourse, distinguishing it from that of everyday conversation, but she discusses primarily the role relations between the interlocutors and does not provide a strictly linguistic analysis of the various sub-phases of the trial.

4. The Oscar Pistorius Trial The trial of Oscar Pistorius lasted for over two years and resulted in two distinct convictions, the final one of which was released in July 2016. The presiding judge in the trial was Thokozile Matilda Masipa from the Gauteng Division of the High Court of South Africa. The prosecution was conducted by Gerrie Nel, while the lead defence lawyer in the case was Barry Roux.

4.1. The case3 Oscar Pistorius was born without calf bones and at the age of 11 his parents decided that he should have both legs amputated below the knees, so that he could be fitted with prosthetic ones. This made it possible for him to play sports while growing up, including rugby, water polo, tennis, and wrestling. He started running in 2003, and in 2004, at the age of 18, he smashed the 200-metre world record at the Athens Paralympics. He then won other titles in the Beijing Paralympic Games and later decided to launch a battle to take part in able-bodied athletics, overcoming arguments that his prosthetic legs gave him an unfair advantage. He made history both in 2011 and 2012 as the first double-amputee who competed at both the Paralympics and Olympics.

3

All the necessary background information concerning this case was obtained from international newspapers and magazines, easily accessible online at http://www.ipl.org/div/news/. There is also an interesting blog entirely dedicated to famous trials, including the one involving Oscar Pistorius: https://juror13lw.com.

A Multimodal Analysis of Discourse in the South African Courtroom

219

His image and reputation as a sports star suddenly changed on the 14th February 2013, when he shot and killed his girlfriend Reeva Steenkamp at his home in Pretoria. He admitted killing her, but said that his act had not been intentional because he had mistaken her for an intruder. He thought that she was in bed with him when he heard noises in the bathroom, which made him fire shots through the bathroom door. The trial started in March 2014 and in September of the same year he was convicted of manslaughter. The following year prosecutors appealed this conviction and the verdict was overturned. Pistorius was then released on bail and lived at his uncle’s home for some time. He was finally convicted on appeal of murder in December 2015. Although the statutory minimum punishment for murder in South Africa is fifteen years in prison, the sentence was reduced to six years on account of his disability, mental condition, and of the fact that he had already served one year of the original five-year prison sentence for manslaughter. The prosecution has recently announced they will appeal the six-year prison term, claiming that it is too lenient, especially in view of the likelihood that Pistorius will be released on parole after serving only three years. The case is therefore not yet closed.

5. Data and Methodology The linguistic data used for the analysis consist of the transcription of the defence lawyer and prosecutor’s questioning sessions during the first trial of Oscar Pistorius that led to his conviction of manslaughter. The data, which consist of approximately 170,000 words, were integrated with video clips taken during the examination and cross-examination.4 In order to examine both verbal and non-verbal elements in parallel, images extracted from the video clips were included in most of the transcription of the data, following the technique proposed by Baldry (2000), Thibault (2000), and Baldry and Thibault (2006), which brings together verbal text and visual data with a description of the function of non-verbal signs. Due to the size of the data set, it was not possible to add paralinguistic and extra-linguistic information to the entire transcription. This is because such manual annotation is an extremely labour-intensive and time-consuming task that is feasible only with relatively small amounts of data. However, the 4

The transcription of the athlete’s examination and cross-examination is publicly available online at https://s3-eu-west-1.amazonaws.com/pmb-portals/behind-thedoor/downloads/transcripts/Oscar+Pistorius.pdf. The corresponding video clips, instead, can be easily found on YouTube.

Chapter Nine

220

exchanges between the lawyers and Oscar Pistorius were studied in minute detail in order to identify their crucial features, also with respect to nonverbal elements. Some of the most representative moments for the purposes of the present study are provided in the Tables in section 6 below.

6. Questioning the Defendant There is a marked difference from several points of view between the questioning style of the defence lawyer, Mr Roux, and that of the prosecutor, Mr Nel. Such difference is observable both at the level of linguistic and rhetorical choices, as well as in the use of non-verbal elements. There is one feature, instead, which both the examination and crossexamination phase have in common. When Mr Roux and Mr Nel question Oscar Pistorius, there is constantly a process of triangulation (example 1), whereby the questions are posed directly to the defendant, who then replies to the judge (and only very rarely to the lawyers), with an extremely deferential form of address: (1) MR ROUX:

Mr Pistorius, when I explained to you that your evidence would be presented by dealing with your background and moving on, you mentioned that there is something that you want to do first? PISTORIUS: Yes. MR ROUX: What is that Mr Pistorius? PISTORIUS: My Lady, if I may please just start off with my evidence by tendering an apology.

The expression My Lady is extremely abundant, and it is used not just by Pistorius himself (example 2), but also by the two lawyers (examples 34) to address Judge Masipa. Pistorius only rarely uses the form My Lord to reply to the lawyers directly: (2) MR NEL:

And you then said words to the effect “It is a lot softer than brain”. Am I right? PISTORIUS: That is correct, My Lady. MR NEL: Referring to? PISTORIUS: Referring to the watermelon. MR NEL: But “It is like a zombie stopper”. Referring to? PISTORIUS: I guess referring to a firearm that would be used to stop the zombie, My Lord.

A Multimodal Analysis of Discourse in the South African Courtroom

221

(3) MR ROUX:

Thank you for your indulgence, M’Lady. I have no further questions.

(4) MR NEL:

As the court pleases, My Lady.

These forms of address have their roots in the English institution of the aristocratic ‘peers of the realm’ i.e., those members of the peerage in Great Britain who have the right to sit in the House of Lords. Such expressions are some of the remaining signs of the colonial era. Although they come across as amusingly old-fashioned, they are still very much in use in South African courts today.5

6.1. The examination phase Mr Roux, the defence lawyer, generally uses a highly formal register with a two-fold aim. On the one hand, he intends to show the utmost respect for the judge by accepting her authoritativeness and his inferior position in the decision making process (example 5). His use of hedged, long-winded language, on the other hand, appears as a sign of ‘protection’ for the defendant, who is always addressed in a very kind manner that shows sympathy (example 6): (5) MR ROUX:

Thank you for your indulgence, My Lady. I have no further questions.

(6) MR ROUX:

Mr Pistorius, I know you want to face Ms Steenkamp when you apologise, but there is one difficulty and that is that the court must hear you, must be in a position to hear you, and the only way we can do it, is if you talk in the direction of Her Ladyship.

Thank you for your indulgence, My Lady in example 5 comes across as almost obsequious and the term indulgence as serious, and perhaps unclear to the uneducated layperson. As is often the case with Latinate words and phrases, it adds formality and opacity to the sentence, which could be more simply rephrased here as ‘Thank you for your patience/understanding/kindness’. As for example 6 instead, this is an example of a request turned into a rather tangled sentence as a way of avoiding the use of a straightforward question or assertion. The sentence could certainly be much shorter and less repetitive though, e.g., ‘Mr 5 The judges of the Labour Court formally insisted in their 2013 practice manual upon being addressed as “My Lord” and “My Lady”.

Chapter Nine

222

Pistorius, please talk in the direction of Her Ladyship because the court cannot hear you’, without sounding too assertive. There is a constant attempt on the part of Mr Roux to reduce the relational distance between the court and Pistorius. This is reflected in the use of questions, which are never asked too directly (examples 7-8) and are preferably substituted by cautious statements, as we have seen in example 6 above, or by assertions of fact in search of support, typically via question tags or formulas, like the one in example 9, which are often employed by lawyers in the courtroom (cf. Biscetti 2006): (7) MR ROUX:

Mr Pistorius, if I may ask you…if you would mind [going] to the door… Would you now take your prostheses off?

(8) MR ROUX:

Mr Pistorius, can you speak up a little bit if possible, please?

(9) MR ROUX:

You sent a Whatsapp to Reeva, is that right?

The conditional clauses in example 7 are hedges (Lakoff 1972; Frazer 2010), mitigating the impact of the request that the lawyer is making. There is a sort of climax in the language that Mr Roux employs. He typically starts off with indirect, tentative requests (If I may ask you…; If you would mind…), culminating in explicitly formulated and firmer questions (Would you now take your prostheses off?). Other hedging elements may be observed in example 8, where a little bit and if possible again work as markers of politeness (Brown and Levinson 1987; Miskovic-Lukovic 2009), softening and downplaying the lawyer’s request. Mr Roux sometimes also anticipates the defendant’s answers (example 9) and simply asks him to confirm the truth of a certain statement ([…], is that right?). This happens when the lawyer is going over and summarising the case facts. When questions alternate with monologic passages addressing the judge, especially in response to a particular aspect brought up by the prosecuting lawyer, Mr Roux’s speaking style becomes excessively verbose and opaque (example 10). This is reflected in the choice of lexical items, which are often repeated, as well as in the length, structure, and complexity of the sentence: (10) MR ROUX: It is also on the understanding that we understand the indirect and derivative effect of this but that we would make the submission that this should not be seen as any further opportunity for the state to introduce in crossexamination evidence which should have and could have formed part of the state’s case.

A Multimodal Analysis of Discourse in the South African Courtroom

223

This long and complicated utterance, for instance, contains several elements of redundancy and many subordinated clauses that complexify both its structure and meaning. The first part (It is also on the understanding that […]) could simply have been omitted and a pause made after the demonstrative pronoun this, i.e., we understand the indirect and derivative effect of this. Then, a new sentence beginning with an adversative conjunction, e.g. however, could have been added. It looks as if Mr Roux’s ‘heavy’ and intricate style had the function of concealing his emotional reaction to a problematic moment in the examination phase, thus giving him the chance of lengthening the time he needs to come up with an appropriate response, while his interlocutors are at the same time trying to make sense of what he is saying. Non-verbal communication, however, seems to attenuate the level of complexity of Mr Roux’s language, which is often hard to understand even for the judge herself. For example, after the following exchange with Judge Masipa (example 11), Mr Roux decides to reformulate his ideas more clearly and to support his speech with several hand gestures, in order to clarify and add incisiveness to his line of argumentation (Table 9-2): (11) MR ROUX: I think with respect, My Lady, the difficulty in the question is “There are other things”. If he puts to him a certain aspect and say “Is that reconstruction or is that what you remember?” then I can understand. But once the witness is confronted with “other things”, then he must expect a long answer, because you try to cover what “other things” may include and not include. So I think if the question is put differently, to say “This is what I say to you. Can you tell me, is it reconstructed or real?”, then we will not have what we have seen here. MR NEL: Without disrespect, My Lady… JUDGE: I am just… One minute, one minute. Mr Roux, I am not quite sure that I understand the objection.

Chapter Nine

224 Image frrame

Verba al text

Non-verbal behaviour and a interpretation

1

ady, the My La questio on put to Mr Pistorius was “Theree are other thingss. What was reconsstructed, what not?” We do not know what the “otherr things” is.

Raising rightt hand and moving it to the forces right (=reinfo the concept of o ‘other thingss’).

2

p If it is put pertineently, “What do you u say about A?”, “Was “ A reconsstructed or is it yourr independent memorry?”, then he can an nswer.

Moving righ ht hand forwards and d downwards (=stresses th he concept of assking a question properly).

3

But thee moment you ask a question… q

Raising both h hands and bringing g them together with h fingertips tou uching, as if shieldin ng something (= =refers to the conten nts of the question)).

A Mulltimodal Analyssis of Discoursee in the South A African Courtro oom

225

4

…abou ut “other thingss”…

Opening both hands (=marks the difference from the idea of a pertinent, well-formulaated question to a vague one), mouth slightly open (=suggests surprise and disapproval).

5

n the witness …then will an nswer as widelyy as possible.

h hands Keeping both wide and opeen (=reproducess the concept of ‘w wide’), slightly frow wning, (=worried gaaze).

6 Table 9-2. M Mr Roux’s reeformulated objection o

Mr Rouxx’s body langguage is both iconic and m metaphoric. Itt conveys meaning thrrough a sort of o ‘pictorial’ resemblance to the conten nts of the words assocciated to his geestures. In imaage 1, for insttance, the mov vement of his right haand to one sidde mirrors thee concept of ‘other’, whilee keeping both hands open and awaay one from the t other in im mage 5 reprod duces the notion of ‘w width’. In imaage 2, instead, the movemeent of the han nd, which points to thee physical ‘heere and now’ provided p by tthe context off the trial, metaphoricaally indicates what is releevant and perrtaining to th he matter under discusssion. Similarrly, images 3 and 4 presennt instances off gestures figuratively reproducing and a expanding g on the conceepts expressed d through words. Mr Roux seems to be holding g or protectinng something with his hands in im mage 3, by extension e suggesting that he is referrin ng to the 6

In Tables 99-2 and 9-3, thee words, phrases, or sentencees in bold are those t with which the gesstures are synchhronised.

Chapter Nine

226

contents of the questionn; keeping haands wide apaart with palm ms facing upwards in iimage 4, insteead, indicates surrendering tto the vagueness of Mr Nel’s questiion. After Mr Roux’s reform mulated versioon of example 11 with supporting hhand gesturess, the judge iss able to folloow his line of thought and the lawyyer resumes his examination n. Some m more exampless of Mr Roux’s body languuage working as metadiscourse caan be observedd in Table 9-3. Imagee frame

Verba al text

Non-verbal behaviour and a interpretation

6

You’ree a little boy withou ut legs. You experience daily that diisability.

l Aggressive look with protrud ding chin and slightly open mouth showiing lower teeth (=expresses anger and rage), lo ooking straight at th he judge and furrowin ng eyebrows tog gether (=tries to con nvince her), pointing index finger and jaabbing it (=delivers an n important po oint).

7

You kn now I cannot run aw way.

Moving righ ht hand vertically (=supports concept of ‘not being g able to’) and then n slightly to th he right (=reproducess idea of movemen nt).

A Mulltimodal Analyssis of Discoursee in the South A African Courtro oom

227

8

Why would w he want help fo or her if he just triied to kill her? Why would w he want her to be safe […]?

Moving righ ht hand forwards and d then downwards (=supports th he concept of s ‘wanting’), slightly bending forw wards (=stresses th he idea of going ahead with a decision).

9

This man m must be locked d up. You know what w he did to me.

Index finger he left pointing to th (=indicates t someone in the courtroom).

10

as desperate to He wa save her.

Right hand reaching r to the head (=indicates a mental state).

Table 9-3. O Other segmen nts of Mr Rou ux’s examinaation In imagee 6, both the laawyer’s gaze and his pointiing finger rein nforce his aggressive ttone and servve the purposse of foregroounding one particular p aspect in his speech, i.e.,, the sheer an nd brutal fact that Oscar Piistorius is physically ddisabled and thhat this condittion had a bigg impact on his life as a young boy. The movemennt of the hand d in image 7 ffiguratively reeproduces the idea of running, whhile the action n of bending forwards in image 8

Chapter Nine

228

suggests ‘going ahead’, i.e., making a decision. Image 9 is a clear example of indexical behaviour accompanying an indexical utterance beginning with a demonstrative, i.e., this. In image 10, bringing the hand towards the head emphasises the concept that desperation resides first of all in the mind. During the examination by Mr Roux, body language appears as an important component that runs parallel to and better illustrates certain verbal elements by replicating their semantic content and emphasising the salient points. Increased clarity eventually guarantees the smooth flow of conversation in the courtroom.

6.2. The cross-examination phase Mr Nel’s cross-examination style is varied, alternating contrasting modalities in the use of both single lexical items and rhetorical strategies. First of all, there is a great abundance of Latin words (example 12) and phrases (example 13), as well as technical terms, especially from police jargon. These lexical items are often followed by an explanation of their meaning (example 14) and are typically interposed in rather colloquial and informal exchanges: (12) MR NEL:

But nothing that is in it that one can read in your affidavit for the bail application is untrue?

(13) MR NEL:

Mr Pistorius, let us just put on record what we have seen in the video. You are on a shooting range with, inter alia, Mr Derick Housgaardt. Is that correct?

(14) MR NEL:

Tampering is if you change something on the scene with the intention to let it look different.

Therefore, they increase the register level of Mr Nel’s examination, which is otherwise rather chatty in tone and rich in colloquial expressions (examples 15-16): (15) MR NEL:

(16) MR NEL:

You see, I know that you would brush it off, but it came from you. You said you remembered it, you said you thought it was a Saturday. You had to go train on a Saturday. I know you would brush it off. But it is not that easy. Are you done?

A Multimodal Analysis of Discourse in the South African Courtroom

229

The prosecuting lawyer’s main aim is that of being as direct and easily understood as possible in order to present clear evidence proving that Pistorius is guilty of the murder of his girlfriend. His style is characterised by aggressive questions (example 17) and many straightforward assertions (example 18), challenging the defendant with force (example 19): (17) MR NEL:

Why do you not just admit that you shot at the watermelon to see [what] the effect would be if you hit someone in the head, in the brains?

(18) MR NEL:

I have already indicated that you are not telling the truth.

(19) MR NEL:

I do not understand why you plead not guilty.

For the purpose of being clear, Mr Nel has a tendency to switch to the historic present to guide Pistorius through his narrative (example 20), which is instead presented using the more natural simple past: (20) MR NEL: You sitting on the passenger side, where is your gun? PISTORIUS: It was on my hip in the holster, M’Lady. MR NEL: And the metro cops then stop you. PISTORIUS: They stopped us on two occasions.

While the defence lawyer, Mr Roux, strives to establish rapport with Pistorius during the examination phase, Mr Nel’s questioning mode always creates and maintains distance with him. The prosecutor even avoids calling Pistorius by name and uses impersonal forms of address (example 21), which apart from having a distancing effect also appear to add sarcasm (example 22): (21) MR NEL:

Do you have page 64, Sir?

(22) MR NEL:

It is the same thing as the watermelon! You had it now in practice, Mister.

Irony (example 23), sarcasm (example 24) and insinuations (example 25) are scattered throughout Mr Nel’s cross-examination. There are many instances of statements meaning the exact opposite of what the lawyer is in fact saying (irony), or of bitter remarks intended to discredit Pistorius without necessarily being ironic (sarcasm); insinuations alternate with explicit inferences (example 26), which appear mostly when Mr Nel is drawing conclusions after a number of questions and answers on a certain topic:

Chapter Nine

230 (23) MR NEL:

So the day in Tashas when you…when the gun went off miraculously, you had your own gun with you?

(24) MR NEL:

I know it is always easy to be wise after the event.

(25) MR NEL:

What was the purpose of this whole incident?

(26) MR NEL:

So, am I correct in saying that you were shooting at the watermelon to see what the effect would be if you shoot somebody in the brain. Or not? Because this is the inference I am drawing.

In example 23, the adverb miraculously is used to mean ‘intentionally’ and the whole statement in example 24 makes an obvious point, which however expresses contempt in this specific context; example 25 is a rhetorical question requiring no answer, i.e., what Mr Nel intends to say is that the whole incident in question has no purpose; example 26 is a deduction made after examining the facts and aimed at inducing Pistorius to admit his guilt in open court. Unlike Mr Roux’s language, which does not entail implicit presuppositions or speech acts, Mr Nel’s questions and assertions are highly allusive (example 27), they tend to have a performative function, e.g., they often consist of reproaches (example 28), and they typically express judgements or opinions (example 29): (27) MR NEL:

Your evidence in court was also true?

(28) MR NEL:

You should not have taken it off and you should not have left it in the car.

(29) MR NEL:

I think you’re wrong!

The verbal dimension is central to Mr Nel’s cross-examination and it is only minimally supported by non-verbal elements. Body language plays a role only in highlighting the prosecuting lawyer’s confidence, but hand gestures are avoided or reduced to a minimum (Table 9-4). This is a marked difference compared to Mr Roux’s speaking style, which instead relies systematically on the use of hand gestures to support his line of argumentation. Mr Nel is more static, but verbally very skilled and engaging.

A Mulltimodal Analyssis of Discoursee in the South A African Courtro oom Imagee frame

231

Verba al text

Non-verbal behaviour and a interpretation

11

-

g, while Just listening resting his heead on his left hand, elbow on the table (=shows selffconfidence th hrough a bored and despising atttitude).

12

You wanted to see the efffect on the waterm melon […]. And th hen you said words to the effect “It is a lot softer than brains”

Slightly look king right, in the direction of Pistorius.

13

Is it no ot a reconsstruction of everyth hing you heard and you read? And that’s why yo ou gave the eviden nce?

h Taking off his glasses, restiing right hand on his chest c (=shows selffconfidence and a a sense of conttrol).

232 14

Chapter Nine Let us just answer the questions. I know you y want to say a lot l of things and… It’s interessting. You’re arguin ng, you’re not answering. Why are you arrguing and not answering?

Resting rightt hand of his left forrearm, slightly look king right in the direction d of Pistorius.

Table 9-4. S Segment of Mr M Nel’s crosss-examination n In imagee 11, Mr Nel is listening to o Pistorius’s aanswer with a know-itall attitude tthat passively communicatees disapprovaal. Most of thee time, he keeps his arrms down andd does not mo ove that muchh (image 12). His most frequent gessture consists in taking off his glasses annd looking forrwards in the directionn of the judgee, while drawing some sortt of conclusio on (image 13). He nevver loses his composure c or raises his vooice and alwaays shows self-control even when he h is attacking g the defendaant’s behaviou ur (image 14). There aare no repeated associationss between thee use of speciffic lexical items and haand gestures or other body movements. m M Mr Nel’s comm municative and questionning style esseentially relies on the verbal medium.

7. Conclusions The analysiis above hass identified tw wo different speaking sty yles. The examinationn phase appeaars to be marrked by a hiigh level of formality, f which is refflected in the extensive e use of hypotaxis aand not so mu uch at the level of lexiical choices. It has been observed that seentence compllexity has a sort of ‘shhielding’ functtion, allowing Pistorius’s deefence lawyerr to resort to a hedgedd questioningg modality, consisting c off indirect and d politely formulated rrequests that aim a to protect the accused aand reduce thee distance with him. W When excessivve verbosity causes c opacityy and unintellligibility, reformulatioon is accompaanied by a pro onounced usee of hand gesttures that support, reinnforce, and back b up whatt Mr Roux iss saying. On the other hand, the prrosecuting law wyer adopts a blunt tone, rresulting in aggressive questions annd assertions challenging and discreditiing the accussed. Nonverbal elem ments in the crross-examinattion phase do not play a siignificant role and onnly rarely suppport Mr Nel’’s speaking sstyle, which is already

A Multimodal Analysis of Discourse in the South African Courtroom

233

clear and forcible in itself despite his repeated use of allusions and insinuations. In addition, the prosecuting lawyer often employs irony and sarcasm as a means of more subtly attacking the accused, thus coming across as the least empathetic participant in the courtroom. Although this is a case study with all its limitations and idiosyncrasies, it represents a first investigation of trial language from a wider, multisemiotic perspective in an attempt to examine the function of non-verbal behaviour in this particular setting and map its possible links with lexicalsyntactic and rhetorical strategies. It is worth investigating these connections more extensively through the analysis of a larger amount of authentic data in order to see whether they are generalizable or just context-specific. This is the reason why future studies on trial discourse in English necessarily need to be based on several court cases in different legal systems and jurisdictions around the world.

References Ainsworth, Janet. 1995. Review of Gail Stygall, “Trial Language: Differential Discourse Processing and Discursive Formation”. Forensic Linguistics 2(2):194–199. Anesa, Patrizia. 2012. Jury Trials and the Popularization of Legal Language. Bern: Peter Lang. Baldry, Anthony. 2000. “English in a visual society: Comparative and historical dimensions in multimodality and multimediality”. In Multimodality and Multimediality in the Distance Learning Age, edited by Anthony Baldry, 41–89. Milan: Edizioni Unicopli. Baldry, Anthony, and Paul J. Thibault. 2006. Multimodal Transcription and Text Analysis. A Multimedia Toolkit and Coursebook. London and New York: Equinox. Bhatia, Vijay Kumar, Purificación Sánchez, and Pascual Pérez-Paredes, eds. 2011. Researching Specialized Languages, Amsterdam/ Philadelphia: John Benjamins. Biscetti, Stefania. 2006. “Tag questions in courtroom discourse.” In Studies in Specialized Discourse, edited by John Flowerdew and Maurizio Gotti, 209–238. Bern: Peter Lang. Brown, Penelope, and Stephen C. Levinson. 1987. Politeness: Some Universals in Language Usage. Cambridge: Cambridge University Press. Cotterill, Janet. 2003. Language and Power in Court: A Linguistic Analysis of the O.J. Simpson Trial. Basingstoke, UK: Palgrave. Du Bois, François. 2007. Wille’s Principles of South African Law. Cape

234

Chapter Nine

Town: Juta & Co. Frazer, Bruce. 2010. “Pragmatic competence: The case of hedging.” In New Approaches to Hedging, edited by Gunther Kaltenböck, Wiltrud Mihatsch, and Stefan Schneider, 15–34. Bingley: Emerald Group Publishing Ltd. Gotti, Maurizio. 2008. Investigating Specialized Discourse. Bern: Peter Lang. Heffer, Chris. 2005. The Language of Jury Trial: A Corpus-aided Analysis of Legal-Lay Discourse. Basingstoke: Palgrave Macmillan. —. 2008. “Judgement in court: Evaluating participants in courtroom discourse.” In Language and the Law: International Outlooks Vol. 16, edited by Barbara Lewandowska-Tomaszczyk, Tomasz Pludowski, and Dolores Valencia Tanno, 145–179. Frankfurt am Main/New York: Peter Lang. —. 2012. “Narrative navigation: Narrative practices in forensic discourse.” Narrative Inquiry 22(2):267–286. —. 2013. “Revelation and rhetoric: a critical model of forensic discourse.” International Journal for the Semiotics of Law 26(2):459–485. —. 2015. “Jury trial discourse”. In The International Encyclopaedia of Language and Social Interaction Vol. 1, edited by Tracy Karen, Ilie Cornelia and Todd Sandel, 1–10. Oxford: Wiley-Blackwell. Kemp, Gerhard, Robin Palmer, Shelley Walker, Dumile Baqwa, Christopher Gevers, Leslie Brian, and Anton Steynberg. 2012. Criminal Law in South Africa. Oxford: Oxford University Press. Kress, Gunther, and Theo van Leeuwen. 2006. Reading Images: The Grammar of Visual Design. Second Edition. London: Routledge. Lakoff, George. 1972. “Hedges: A study in meaning criteria and the logic of fuzzy concepts.” Papers from the Eighth Regional Meeting of the Chicago Linguistic Society, 183–228. Levi, Judith and Anne Graffam Walker. 1990. Language in the Judicial Process. New York: Plenum. Lindsey, John. 1990. “The legal writing malady: Causes and cures”. New York Law Journal, December 12. Matoesian, Gregory. 2001. Law and the Language of Identity: Discourse in the William Kennedy Smith Rape Trial. Oxford: Oxford University Press. Mellinkoff, David. 1963. The Language of the Law. Boston: Little, Brown and Co. Miskovic-Lukovic, Mirjana. 2009. “Is there a chance that I might kinda sort of take you out to dinner? The role of the pragmatic particles kind

A Multimodal Analysis of Discourse in the South African Courtroom

235

of and sort of in utterance interpretation.” Journal of Pragmatics 41:602–25. Moeketsi, Rosemary. 1999. “Discourse structure in a criminal trial of a magistrate’s court.” South African Journal of South African Languages 19(1): 30–38. Stygal, Gail. 1994. Trial Language: Differential Discourse Processing and Discursive Formation. Amsterdam: John Benjamins. Thibault, Paul J. 2000. “The multimodal transcription of a television advertisement: Theory and practice.” In Multimodality and Multimediality in the Distance Learning Age, edited by Anthony Baldry, 311–385. Campobasso: Palladino Editore. Tiersma, Peter. 1999. Legal Language, Chicago, IL: University of Chicago Press —. 2001. “Textualizing the law”. Forensic Linguistics 8(2):73–92. —. 2006. “The language of legal texts.” In Encyclopaedia of Language and Linguistics Vol. 6, edited by Keith Brown, 549–556. 2nd edition. New York: Elsevier. Tiersma, Peter and Lawrence Solan, eds. 2012. The Oxford Handbook of Language and Law. Oxford: Oxford University Press. Zimmermann, Reinhart, and Daniel Visser. 1996. Southern Cross: Civil Law and Common Law in South Africa. Oxford: Clarendon Press.

CHAPTER TEN HOW IDIOMATIC ARE DISNEY ANIMATED FILMS? THE MULTIMODAL MEANING-MAKING OF IDIOMS GIANMARCO VIGNOZZI UNIVERSITY OF PISA, ITALY

1. Introduction The purpose of this paper is to cast light on how the screenwriters of animated films adapt the complexity of meaning of idioms (i.e., literal and figurative, see section 3) to the polysemiotic nature of animated films. More specifically, idiomatic expressions are evaluated from a multimodal point of view, so as to observe how different semiotic codes interact to create a certain effect on the audience. The study takes into account a diachronic selection of Disney animated films, which are not only some of the most successful and long-running animated audiovisual products, but are also semiotically and linguistically elaborate. Early childhood literacy is nowadays based on an increasingly commercialised and globalised textual landscape that includes popular media, such as television programmes, films, video games, and websites, as well as franchises with dolls, toys, collectibles, apparel, beauty products, food, school supplies, and other everyday consumer goods (Kress 2003). In this scenario, it is impossible not to mention the pivotal role covered by Walt Disney cultural products, which have been dominating child entertainment industry for almost a century. Nowadays, “Walt Disney Studios” is the film production house that is responsible for the creation and distribution of all Disney films (either animated or not). However, since the company owns an extensive series of subsidiary companies, including Touchstone, Miramax, Pixar, Lucas Film (responsible

How Idiomatic are Disney Animated Films?

237

for the Star Wars film series) and several other large productions, it is difficult to quote the total number of movies released so far. According to some data dating back to 1995 (Giroux 1995), Disney products are consumed by millions of people every year. For example, approximately 200 million people watch a Disney film per year, 395 million people watch a Disney-produced television show each week, 212 million people listen to Disney produced music each week, 50 million people visit a Disney theme park per year, and 42 million people per year make a purchase at a Disney store. These figures, therefore, clearly point out the massive propagation of the Disney Corporation and its hegemonic status on child-oriented media. Since its inception, the creative talent of Walt Elias Disney, the father of the company, gave rise to a series of everlasting successes, becoming synonymous with “the best values and virtues of American society” (Robb 2014: 56). As a consequence, generations of children have been entertained with a cinematographic tradition of hyper-idealised versions of the American Dream, where the notions of good triumphing over evil, the awarding of the honest, the punishment of the wicked and, most especially, love conquering all, are the core values. Nevertheless, given Disney films’ frequent debt to canonical historical children’s literature (e.g., the works by the Brothers Grimm, Hans Christian Andersen, and so forth), their basis in American moral tradition, and their archetypal standing as medium of highly colourful, enjoyable, and politically innocuous children’s entertainment, Disney films are still highly valued by families, representing everything a parent could want from the “Third Educator”, i.e., the media (Buckingham 2007). Re-watching these films as a grown-up not only engenders a feeling of nostalgia in the viewer, but also permits him/her to notice “A Whole New World” (to quote Aladdin’s well known theme song) of hidden meanings, references, and peculiarities that are likely to be overlooked by children, who perhaps cannot discern the complex net of cultural references, jokes, and allusions used by screenwriters to render the films more charming for adults as well. In my opinion, it is thanks to these nested levels of interpretation that these works are enjoyable for a larger global audience, irrespective of epochs. In fact, they are often defined as “classics that can stand the test of time” (Robb 2014: 32). But what kind of world do these movies portray? How can they be so appealing to children and adults alike? How do screenwriters manage to achieve such an effect? What it is undeniable, and what sparked off my interest for this topic, is this sort of unique universal recognition, also evident at the box office, of everything released by Disney. In light of that, I believe that we may

238

Chapter Ten

infer that this great fortune is not only the result of compelling and wellcrafted scripts that draw on well-worn devices of mythical structure (i.e., those presented by canonical fables and fairy tales), but also of the joint effort of the diverse semiotic components at work in animated movies, among which verbal language (i.e., dialogues) is just one element. In this perspective, the reasons for studying the meaning-making of idioms with a multimodal approach are manifold. Firstly, since idiomatic expressions are highly frequent in any kind of discourse, it is worth investigating whether this also applies to dialogues in animated films, which are supposed to be a mimesis of naturally occurring language (Gregory and Carroll 1978: 42; Baños-Piñero and Chaume 2009; Chaume 2012); secondly, the pragmatic complexity of such expressions, meaning that their interpretation is subject to the addressee’s ability to draw inferences, and thus to his/her capacity to detect what the speaker pragmatically implies on the basis of what is literally said (Ruiz de Mendoza and Pérez Hernández 2003). This pragmatic complexity is particularly interesting in child-oriented audiovisual texts in that children’s understanding of idiomatic expressions may not be complete (Wray 2002). It is thus useful to investigate the ways scriptwriters try to ease decoding complexity. Thirdly, the multimodal nature of film language (Chaume 2004) explicitly allows scriptwriters to account for the complex meaning of idioms—figurative vs. literal—by exploiting the different channels and codes making up the shot. Consequently, in order to expand on this issue, a multisemiotic evaluation of idiomatic expressions in a corpus of Disney animated films is undertaken in the next sections of this paper, which is organised as follows: in section 2, I provide an account of Disney animated films as a specialised kind of audiovisual text. Section 3 is devoted to idiomatic expressions in English and to the aim of this work. The data under investigation are, instead, the subject of section 4. Section 5 presents the methodology I have used and the analysis that has been carried out, illustrating both quantitative data and the multimodal codification of some examples. Finally, some concluding remarks and questions for future research are sketched in section 6.

2. Disney Animated Films as a Sort of Specialised Genre 2.1. Animated films and ‘Disney’s prototype’ As Wells (1998, 2002) maintains, an animated film is a distinctive film form that offers a wide range of meaning-making potentialities that are unavailable to live-action filmmakers. Generally speaking, animation is

How Idiomatic are Disney Animated Films?

239

thought of as a technique in which individual drawings that minimally differ from each other are photographed frame by frame and then played in a quick session (24 frames per second) to create the illusion of motion and shape change (Wells 1998). It is thus self-evident how the potentialities of this craft can be exploited to realise a wide range of audiovisual texts spanning over different genres. Despite the Eastern tradition of producing full-length animated films (e.g., anime, manga, and Hayao Miyazaki films) that are mainly geared towards adults, in the Western tradition animation techniques have been mostly used as a filmmaking practice for the realisation of children’s films. Drawing on the famous prototype theory, formulated in the 1970s by Rosch and according to which for any given category,1 there are central examples (i.e., particularly representative examples of members of that category), secondary examples (less good examples), and peripheral examples (generally not very good examples), it seems plausible to classify Disney feature-length animated movies as the ‘prototype’ of Western animated movies. In fact, considering the enormous popularity and spread of Disney animations, three main ingredients have contributed to the planetary success of these movies, so much so that I believe they have become a prototypical model for Western animated products, from which other animated movies take inspiration, e.g., Rio (2011, Saldanha), from 20th Century Fox Animation, or Shark Tale (2004, Bergeron, Jenson, Letterman) from Dream Works Animation, among others. These three ingredients are: 1) their recurrent backbone structure, i.e., fables, fairy tales, and stories of cultural resonance, 2) their values, e.g., good vs. evil, and 3) their themes, i.e., love, friendship, etc.

2.2. Disney animated films: An established cultural force With the release of Snow White and the Seven Dwarfs (1937, Cottrell, Hand, Jackson, Morey, Pearce, Sharpsteen), the first full-length Technicolor sound-synchronised animated cartoon feature, the Walt Disney Productions (founded in 1923 by Walt Disney and his brother Roy) marked a new era in the history of animated movies and established its role as a leading company in the field. After branching out into feature films with Snow White, grossing more than $1.7bn, the new flagship 1

Categories are not discrete entities but are best described as having ‘fuzzy’ edges; one person’s central example may be another’s secondary example (McIntyre 2012).

240

Chapter Ten

animation studio continued to deliver a long series of successes such as Pinocchio (1940, Ferguson, Hee, Jackson, Kinney, Luske, Roberts, Sharpsteen), Dumbo (1941, Armstrong, Ferguson, Jackson, Kinney, Roberts, Sharpsteen, Elliotte), Bambi (1942, Algar, Armstrong, Hand, Heid, Roberts, Satterfield, Wright), Cinderella (1950, Geronimi, Jackson, Luske), which chiefly address younger audiences, but that may also appeal to adults. As Watts (1997) explains, the pioneering series of feature-length animated films produced by the Disney Studios during its early years established many of the conventions that have dominated children’s films for decades, and still influence the genre today, casting them in the prototypical role within the category of animation. These films, in fact, feature storylines where characters overcome hardships through hard work, calling to mind the 31 narrative functions (i.e., absentation, interdiction, violation, etc.) outlined by Propp in Morphology of Folktale (1928). The atmosphere, instead, tends to be light and fun, with several musical numbers incorporated into the plot to enhance the aura of innocence characterising the genre. To put it simply, the plot usually revolves around a good-over-evil scenario with a stereotypical happy ending, carrying the implicit message that a better future is always possible, which also recalls the U.S. ethos of the ‘American Dream’. In the last few decades, with the production of great successes such as Shrek (2001, Adamson, Jenson), The Incredibles (2004, Bird), Up (2009, Docter, Peterson), and Frozen (2013, Buck, Lee), which were realeased after Disney’s merger with Pixar Animation Studios—a pioneer in digital animation—the company has attempted to produce films with a more cross-generational appeal by portraying more sophisticated plots that also tackle everyday problems and have a clear dual-sided humour centred on sarcasm. Notwithstanding, all Disney films are still mainly intended to be broadly comedic adventures scattered with kid-pleasing elements and goofy anthropomorphic animals and creatures which provide comic relief to the story. In fact, although in more recent productions stories and themes have a higher degree of variability, their goal is still constant, that is, getting kids to laugh while also attracting and entertaining their parents, thus achieving a massive success at the box office.

2.3. Disney and linguistics Given the enormous success and influence of Disney products over society, its films have often been investigated from many different viewpoints, including a linguistic one. Most of this language-oriented research focuses on the function of language variation to depict characters

How Idiomatic are Disney Animated Films?

241

in a quick and efficient manner, and on the consequent implications of reproducing and reaffirming certain ethnic, social, and local stereotypes (cf. Lippi-Green 1997; Di Giovanni 2003; Bruti 2009, inter alia). Overall, what has been noticed is that in these apparently innocent and lighthearted films, scriptwriters create a sort of ‘illusion of reality’ by means of dialogues and images that also give rise to—or reaffirm—a series of stereotypes, most of the time specific cultural ones, which may have a heavy impact on the way children perceive people and identities. In this regard, Lippi-Green (1997) hypothesised that children tend to assign values (i.e., good vs. evil) on the basis of how certain linguistic features are associated with a race, an ethnicity, or a homeland. It is thus evident that Disney’s reach into the lives of children is quite strong; that is why Chiaro (1998) talks about a sort of “Disney ideology” (1998: 98), which shapes children’s and also, to some extent, adults’ view of the world. Recently, in line with the “growing interest in the domain of audiovisual studies” (Díaz Cintas 2009: 2), films have more frequently started to be evaluated as plurisemiotic (i.e., multimodal) text-types, where meaning is engendered by the interplay between two channels of communications, i.e., acoustic and visual, and two codes, i.e., verbal and non-verbal (Wildefeur 2013). As a result, also semiotic and mediumspecific approaches have been applied to animated films, even though they tend to be considered a trivial genre, whose status can never be placed on a par with that of live action films. Hence, studies on Disney animated films from a multimodal point of view are still not very numerous. Most of them (Chiaro 1998; González Vera 2015, inter alia) deal with the creation of linguistic humour, showing how effective results strongly depend on the joint effort of the different semiotic components making up the animated movie. The main semiotic layers that are taken into account when analysing audiovisual texts are those pinpointed by Gottlieb (2005): 1) the verbal auditory, i.e., dialogues, background voices, soundtrack lyrics, 2) the non-verbal auditory, i.e., music, natural sounds, sound effects, 3) the verbal visual, i.e., written signals, labels, newspapers headlines, and 4) the non-verbal visual, i.e., gestures, movements, colours, and picture flow.

3. Idioms Idiomatic expressions are very abundant in English and, as Jackendoff (1997) points out, more than 25,000 idioms can be counted. Their massive occurrence in discourse has also been stressed by Searle (1975), who states that speakers seem to obey to the implicit rule to “speak idiomatically unless there is some special reason not to” (1975: 50). In the OED, an

Chapter Ten

242

idiom is defined as A form of expression, grammatical construction, phrase, etc. used in a distinctive way in a particular language, dialect or language variety; spec. a group of words established by usage as having a meaning not deducible from the meaning of the individual words” (OED, s.v. idiom, sense 3)

Hence, it appears that idioms encompass non-literal or figurative expressions, such as metaphors, metonymies, similes, and proverbs, and belong to the broad category of formulaic language (for an overview, see Wray 2000, 2002). What differentiates them from the aforementioned instances of figurative language is that idioms have a unique meaning that can adapt to various contexts. Metaphors, instead, can change their meanings on the basis of the situation (Cacciari and Papagno 2012). The sentence John is an elephant could, in effect, mean either that he is big or that he is clumsy and bumbling, according to the context of use. In order to elucidate their role within the category of language formulae, Glucksberg (2001) explains that “what sets idioms apart from most other fixed expressions is their ‘non-logical’ nature, that is, the absence of any discernible relation between their linguistic meanings and their idiomatic meanings” (2001: 68). Furthermore, as Cooper (1998) says, idioms are also “the most frequently formulaic expressions used in discourse” (1998: 155). He presents statistics showing that language users—either when speaking or writing—use “about 1.08 novel figures of speech and 4.08 idioms per minute”, indicating that “over a lifetime of 60 years, a person would use about 20 million idioms” (1998: 155). What prototypically defines idioms is idiomaticity. This feature is not exclusive to idioms but is common to all figures of semantic idiosyncrasy, e.g., those mentioned above (Fernando 1996). Idiomaticity is reflected in non-compositionality, i.e., the individual meanings of the words do not contribute to the meaning of the idiom as a whole (Sinclair 1991), and in the lack of formal flexibility, meaning that idioms allow for only a restricted set of morphosyntactic changes. In the words of Wray (2002), “the feature ±2 idiom could be a defining variable in a typology of formulaic sequences along a continuum from fully bound to fully free” (2002: 34). Therefore, in this sense, idiomaticity is a graduated phenomenon, as some idioms are more idiomatic than others. A thorough account of the multidimensional and intricate nature of idioms is provided by Nunberg, Sag, and Wasow (1994), who single out six properties ascribable to this class of expressions, namely: 1) inflexibility 2

This symbol has been used by the author meaning ‘more or less idiomatic’.

How Idiomatic are Disney Animated Films?

243

(see above), 2) conventionality, i.e., synonym of non-compositionality, 3) figuration, i.e., every idiom involves figuration to some extent, 4) proverbiality, i.e., idioms usually refer to common and recurrent situations by means of scenes involving familiar things and actions that indirectly represent them, 5) informality, i.e., idioms are usually associated with informal registers, and 6) affect, i.e., they often imply an affective evaluation of reality. Within the vast panorama of attempts to classify idioms in systematic categories, Glucksberg (2001) provides a model based on “idioms’ semantic complexity” (2001: 73-74), mostly relying on the previously mentioned notions of compositionality and transparency. In line with these principles, Glucksberg also maintains that idioms can be non-compositional, partially compositional, or fully compositional. In non-compositional idioms, the constituents are not semantically related and the idiom’s meaning is opaque, as in the idiom kick the bucket, which means ‘to die’. On the other hand, the figurative meaning of partially compositional idioms can normally be discerned from its constituents. In partially compositional idioms, at least one word has a literal sense, for example in the expressions it’s raining cats and dogs the notion of ‘raining’ also remains in the idiomatic meaning, thus making it less opaque and easier to decode. Finally, fully compositional idioms map directly onto the idiomatic referents, resulting in mostly transparent expressions, as in pop the question. Here the verb pop and the noun phrase the question are directly associated, for native speakers, with the idiomatic meaning of sudden marriage proposal. It is therefore undeniable, and has been proved by corpus-based studies (Moon 1998), that all kinds of idioms, no matter their degree of transparency and compositionality, have two possible realisations—the literal and the idiomatic—which could be treated as homonymous since they have identical constituents but different meanings. What is also evident is the potential difficulty for children to decode expressions that “make use of mental pictures to provide an image of what is being said” (Cacciari and Levorato 1999: 63). Research shows that the understanding of idioms begins in early childhood and gradually improves throughout the school-age years, adolescence, and well into adulthood (Nippold and Taylor 2003). In the light of such complexity, as is evident from this brief literary survey on idioms, and given the polysemiotic and nature of audiovisual texts, the analysis carried out in the next sections attempts to describe how verbal and non-verbal elements in animated films jointly contribute to the appreciation of idiomatic expressions.

Chapter Ten

244

4. Data The analysis in this paper is based on a corpus of Disney films (hereafter referred to as the DNF corpus) which counts 52,274 words and includes five Disney animated films (see Table 10-1 below).3 The reasons behind the choice of these particular movies are twofold: firstly, they encompass a wide time span (from the beginning of the Disney era up to the 1990s), thus allowing me to pinpoint possible diachronic changes; secondly, they portray a wide variety of different themes, movie genres, characters and cultural aspects of Disney production, in order to have a sample that is as representative as possible. Table 10-1 lists in chronological order all the films under investigation, together with the year of release (in the United States), the film director(s), the running time, the adapted source, and the genre tagset4. Film title

Year of release

Director(s)

Running time

Adapted source

Genre tagset

Snow White and the Seven Dwarfs

1937

David Hand

83”

Schneewittchen (1812), Brothers Grimm Fairy tale

Animation, Family, Fantasy, Musical, Romance

101 Dalmatians

1961

79”

1973

The Hundred and One Dalmatians (1956), Dodie Smith Novel English Legend of Robin Hood (14th century first written attestation) Legend/folklore

Animation, Adventure, Comedy, Family, Music

Robin Hood

Clyde Geronimi, Hamilton Luske, Wolfgang Reitherman Wolfgang Reitherman

3

89”

Animation, Adventure, Comedy, Family, Romance

The data were extracted from a larger corpus that I collected and analysed for my Master’s dissertation (Vignozzi 2015). 4 Genres have been defined according to http://www.imdb.com.

How Idiomatic are Disney Animated Films? Aladdin

1992

Ron Clements, John Musker

90”

Aladdin and the Magic Lamp (18th century) Fairy tale

Hercules

1997

Ron Clements, John Musker

89”

Myth of Heracles Greek mythology

245 Animation, Adventure, Comedy, Family, Fantasy, Musical, Romance Animation, Adventure, Comedy, Family, Fantasy, Musical, Romance

Table 10-1: Films in the DNF corpus Snow White and the Seven Dwarfs and Aladdin are examples of fairy tale adaptations. The first is based on a German fairy tale, Schneewittchen, from the collection Kinder- und Hausmärchen, written by the Brothers Grimm in 1812. Aladdin is the audiovisual adaptation of the Arab folktale Aladdin and the Magic Lamp from One Thousand and One Nights. This movie belongs to the so-called “Disney Renaissance”, that is, a phase of great innovation of the company, after a period (the 1980s) in which Disney reached a deadlock, which began in 1989 with The Little Mermaid. It is during this renewing era that Disney plots started to deal with more up-to-date issues, such as promoting equal opportunities for men and women, e.g., by portraying rebel heroines like Ariel, Jasmine, Pocahontas, etc. Slightly differently, Robin Hood draws inspiration from the famous folk English legend of Robin Hood and is set in Middle Age England. What is peculiar to this adventure movie is that the characters are all anthropomorphic animals. Conversely, One Hundred and One Dalmatians is the screen adaptation of a contemporary children novel (The Hundred and One Dalmatians, written by Dodie Smith in 1956). The result of this transposition is slightly different from that of fairy tales: it is in fact a comedy that shows everyday situations where there is not a real hero or heroine, even though the opposition between good and evil still strongly underlies the plot. Another different case is that of Hercules. This movie is loosely based on the story of the legendary Greek mythological hero Heracles, with the addition of a pure ‘Disneyan’ ending, depicting the triumph of love. It is interesting to notice that Hercules shares the same directors as Aladdin; both movies show, in fact, a clear texture of humorous and sarcastic references to American mass culture.

246

Chapterr Ten

5. A Mu ultimodal Analysis A of Idioms in tthe DNF Corpus 5.1. Som me quantitatiive considerrations All the aforementioned films fi were colllected, first, iin digital videeo format (.avi) and then orthogrraphically traanscribed in order to acccess the dialogues. A After this phasse of data gath hering and preeparation, all idiomatic expressions were located and then evalluated throughh careful readiing of the transcripts. A As for the iddentification of o idioms, I reelied upon th he criteria described inn detail in secction 3. From this preliminnary investigaation, 126 idioms coulld be counteed. Fig. 10-1 displays theeir distributio on in the corpus.

Fig. 10-1. Thee distribution of o idioms in the DNF corpus

On the bbasis of these data, it appeaars that idiom ms are present in all the films in the corpus. By looking at thesee graphs, and aalso on the baasis of my experience aas an average moviegoer, we w may appreeciate that thee usage of idioms seem ms to be havee been steadilly increasing over time. Th his could testify to a tendency for the language in more receent movies to be more informal andd colloquial, closer c to the sp peech of the m man in the streeet. Another interesting treend is that theey seem to be used more ex xtensively in more lauughter-provokiing comedies (i.e., Herculees and Aladdiin, which are also desscribed with the exact sam me genre labeels, as can bee seen in Table 10-1)). This tendeency could be b related to the fact thaat idioms “decorate laanguage use” (Wood 2010:: 7), and thuss make it witttier, more colourful, annd funnier. Starting from the prem mise that the minimum m unitt of analysis in i movies are ‘shots’, i.e., a series of o frames thatt run for an uuninterrupted period of time (Sklar 1990) rather than ‘words’’, after the exxploratory qu uantitative

How Idiomatic are Disney Animated Films?

247

analysis of idioms across the corpus, a multimodal transcription of all the semiotic resources involved in the meaning-making of the idiomatic expressions was carried out. The model for the transcription was adapted from work on film discourse by Wildfeuer (2013) and includes: 1) a shot of the scene, 2) the shot description, 3) kinesics and proxemics, 4) sounds and music, 5) spoken language. Due to the multidimensional nature of idioms, this semiotically integrated transcription is particularly relevant in that it brings to the fore how the diverse components of film shots deal with the meaning-making of such expressions and also permits us to discern their overall pragmatic functions.

5.1. Idioms as humour boosters The first interesting aspect that has emerged thanks to the multimodal analysis is that some idioms (34 out of 126) occur in humorous scenes— for an introductory account of the notion of linguistic humour, see Attardo (1994), and for linguistic humour on screen, see Chiaro (2006), Chiaro, Heiss, Bucaria (2008), inter alia. Tables 10-2, 10-3, and 10-4 illustrate the multimodal transcriptions of selected idioms, showing how the pragmatic phenomenon of humour creation stems from the intersection of different semiotic elements involved in the meaning-making of idioms. The first column of each table lists the movie from which the idiom comes from and the idiom taken into account. The second column contains the consecutive shots, which are described in the third column according to the framework proposed by Lacey (2005). In the fourth column, kinesics (i.e., the study of body language) and proxemics (i.e., the use of space of the characters on the screen) are described. The fifth column is devoted to all the sound elements and, if present, to soundtrack music. Soundtrack lyrics have been considered as spoken language; thus, they have been described in the sixth column, which contains the orthographic transcription (for an overview of film transcription methodology, see Bonsignori 2009).

Snow White kindly bows to Grumpy with a very sympatheticc facial expression. The dwarf standss cross-armed, with h his back turned. His facial expression is contrite

Medium two--shot of Snow Whitte and the dwarff Grumpy

Medium two--shot of Snow Whitte and the dwarff Grumpy

(1)

(2)

Snow Whitee and the Sevven Dwarfs “Cat got your tongue” (40:33 - 40::34)

X

X

d) Sounds / music

X

Snow White

X

Well, aren’t you going to wash? What’s the matter? Cat got your tongue?

e) Spokeen language

Table 10-2.. A multimodal transccription of the idiom ““cat got your tongue” ” in Snow White and th he Seven Dwarfs

Grumpy responds to Snow White’s sarcastic question n by sticking his tongue out towards her. Snow White is astonished

c) Kinesics and proxemics

b) Shot description

a) Shot

Chhapter Ten

Movie and idiomatic expression

248

b) Shot description Medium shot of Philoctetes and Pegasus

Medium close up of Hercules running towards the enemy

a) Shot

(1)

(2)

Movie and idiomatic expression

Hercules “Use your head” (33:13 - 33:17)

Hercules, very distant from Philoctetes, acknowledges through facial expression that he has understood the message

Philoctetes shouts at Hercules and puts his hands close to his mouth to amplify the sound

c) Kinesics and proxemics

Non-lexical vocalization produced by Hercules to suggest that he has understood Philoctetes’ order

X

d) Sounds / music

How Idiomatic are Disney Animated Films?

X

Philoctetes

Concentrate, use your head!

e) Spoken language

249

Long shot of Hercules running towards the centaur

Long shot of Hercules hitting the enemy

(3)

(4)

Hercules hits his enemy with his head, while he is holding Megara

Hercules runs towards the monster recalling a battering ram

Chapter Ten

Collision sound

Effort sound

Table 10-3. A multimodal transcription of the idiom “use your head” in Hercules

250

X

X

Long shot of the Sheriff of Nottingham aand Nutsy, a guarrd

(1)

Robin Hood d “you birdbrain” (39:17 - 39::19)

The Sheriff of Nottingham gives an order to a guard by pointing g his finger somewhere. The guard (=bird)’s face expresses resignation and fear

c) Kinesics and proxemics X

d) Sounds / music

Table 10-4.. A multimodal transccription of the idiom ““you birdbrain” in Ro obin Hood

b) Shot description

a) Shot

Movie and idiomatic expression

How Idiomatic aree Disney Animated Filmss?

Sheriff

You heard him, Nutsy! Get goin’! Move it, you birdbrain.

e) Spokeen language

251

252

Chapter Ten

What clearly stands out from these transcriptions is that the comic element is triggered by the double interpretation of the idiomatic meanings. The first example (Table 10-2), taken from Snow White and the Seven Dwarfs, illustrates a scene in which Snow White asks Grumpy if a cat got [his] tongue. She uses a partially compositional idiomatic expression—nowadays considered a bit archaic—that is normally directed at someone, mostly a child who is quiet when he/she would be expected to speak to someone who is suspiciously unobtrusive. The image shows that, although Snow White has a very kind and genuine tone of voice, demonstrating affect in line with idiom properties highlighted in section 3, Grumpy takes her metaphoric and affective question literally and reacts uncooperatively, by showing her that he still has got his tongue, sticking it out with scorn. In this case, we may notice that the shot that follows the one in which Snow White utters the idiomatic expression works at two levels. On the one hand, it contributes to the creation of fun, in that Grumpy’s disrespectful and uncooperative gesture is perceived as both amusing and unexpected. In fact, Snow White, who in this scene embodies the archetype of the mother, is astonished. On the other hand, the visual representation of the idiom unveils its literal meaning, thus pre-empting any possible decoding difficulty: also very young children, who may not completely understand this expression, and thus cannot notice the clash between literal and figurative meaning, can make most of its literal and funny visual realisation, without wasting much of the effect created. A similar example is displayed in Hercules (Table 10-3), where the superhero, who is in danger while fighting a monster, takes his mentor’s suggestion use your head, an idiom (partially compositional) meaning to think carefully, to understand something, or to avoid making a mistake, literally. Therefore, after confirming through his facial expression that he has understood Phyloctetes’ advice, Hercules runs towards the centaur and physically hits him with his head. In this example, the clash between the two meanings of the idiom is clearer in that the divide between hitting someone with the head (i.e., the illocutionary force of the utterance that Hercules perceives) and reflecting carefully (i.e., the actual, idiomatic, illocutionary force meant by Phyloctetes) is evident. What triggers laughter here is that the context would naturally favour the figurative interpretation, i.e., the idiomatic meaning, as more plausible and appropriate in that situation, besides being more common in the real world in general. In order to emphasise this funny clash of interpretations, the wrong interpretation of the idiom’s meaning is reinforced and rendered more manifest by different semiotic components such as facial expression (Hercules’ acknowledgement), sounds (collision) and, obviously, images,

How Idiomatic are Disney Animated Films?

253

which jointly contribute to making the scene more effective, and thus more amusing and convincing. In the third example from Robin Hood (Table 10-4), a different case is displayed. The wicked Sheriff of Nottingham gives an order to Nutsy, a clumsy guard at his command. In order to reinforce his utterance, he insults the guard by saying you bird-brain, which is an informal and derogatory (literal) idiomatic expression denoting a stupid person or someone who behaves in a silly way. What is peculiar to this case is that the image alters the figurative meaning of the spoken expression, in that Nutsy is actually a bird, more specifically a vulture; therefore, both the literal and the figurative interpretation of the idiom are plausible here. This duplicity provides the scene with comic features that can be, in my opinion, appreciated by children and adults alike. For reasons of space, I could include only these examples which appeared particularly interesting, but other similar cases are traceable in the corpus of films under investigation (Vignozzi 2015).

5.2. Idioms as visual puns Even though some idioms are willingly used for entertainment purposes (34 out of 126 as described above), they also occur in other contexts, when humour is not directly manifest. Interestingly, 41 idioms occurring in nonhumorous situations share a similar multimodal behaviour, where the verbal and non-verbal seem, to a certain extent, to cooperate in the meaning-making, generating some sorts of witty visual puns (for an overview of puns and visual puns see Chiaro 1998, Partington 2006, inter alia). In Tables 10-5, 10-6 and 10-7, the same multimodal codification used in the previous paragraph is employed to describe some of these idioms occurring in the dialogues of the movies in the corpus.

X

Sergeant is among some Dalmatian puppies with an astonished faciall expression. The puppies look surprised

Medium clooseup of the catt Sergeant andd some Dalmatians

(1)

101 Dalmattians “Save your skins” (50:15 – 50:16)

Sergeant

You’d better get out of here if you want to save your skins ͒

e) Spoken n language

Table 10-5.. A multimodal transccription of the idiom ““save your skins” in 101 1 Dalmatians

d) Sounds / music

c) Kinesics and proxemics

b) Shot description n

a) Shot

Chhapter Ten

Movie and idiomatic expression

254

b) Shot description Close up of Jafar

Long shot off Jasmine

a) Shot

(1)

(2)

Movie and idiomatic expression

Aladdin “Time is up p” (56:45 – 56:48)

Jasmine is hit by Jafar’s spell

Jafar is casting a spell with his magic scepter. His expression is fanatic and evil

c) Kinesics and proxemics

How Idiomatic aree Disney Animated Filmss?

X

X

d) Sounds / music

Jafar

Jafar

Your time is up!

Ah, ah, ah, princess

e) Spoken language

255

(3)

Long shot off Jasmine in a hourglass Jasmine is trapped in a twirling hourglass

Chhapter Ten

One of the dancers hits Aladdin with her bottom, after he has barged into their room from the window

Long shot off Aladdin amongst twoo Arabian dancers

(1)

Aladdin “Hit the bottom” 54) (07:53-07:5

Background music

d) Sounds / music

X

Table 10-7.. A multimodal transccription of the idiom ““hit the bottom” in Aladdin

c) Kinesics and proxemics

b) Shot description

a) Shot

Movie and idiomatic expression

Table 10-6.. A multimodal transccription of the idiom ““time is up” in Aladdiin

256 Jasmine!

Arabian dancer

Aladdin hit the bottom

e) Spoken n language

Aladdin

How Idiomatic are Disney Animated Films?

257

Differently from the examples collected in Tables 10-2, 10-3 and 10-4, these idioms do not appear to be used by scriptwriters with the primary intention of prompting laughter. They instead produce some double entendres, making the scene more winsome. In the first instance from 101 Dalmatians (Table 10-5), Sergeant, a cat that is on a mission to rescue the kidnapped Dalmatian puppies, tells them that they have to escape with him in order to save [their] skins. This idiom, meaning to avoid getting into trouble, has, in this scene, an apparent dual-sided realisation. Since the protagonists of the story are dogs whose lives are endangered because the villain of the movie, Cruella De Vil, wants to make a fur coat out of their skins, save your skin can either be taken literally or metaphorically. Sergeant’s worried look and the dogs’ expressions of disbelief testify to the cruelty of the literal interpretation of the expression. The puppies evidently take this expression literally, as they heard Cruella’s lackeys say that she wanted to kill them to get their precious fur for a new coat. The imaginary setting of the movie makes such a cruel interpretation plausible without being inadequate for children. The second example is taken from Aladdin (Table 10-6). Here Jafar, the evil sorcerer who appears in the first close-up shot, is using wizardry to put a spell on princess Jasmine. While casting the spell, he says to Jasmine your time is up. Interestingly, this idiomatic expression, indicating metaphorically that the allotted time has run out, here again is illustrated literally, in that Jafar’s sorcery (displayed in shot 3) puts Jasmine into an hourglass, showing the flow of time. Finally, the last example, still from Aladdin (Table 10-7), illustrates a scene of an escape sequence, where the protagonist is running away from the royal guards after ripping off a piece of bread. In the shot taken into account, Aladdin bursts into a room where three Arabian women are dancing. The Middle-Eastern background music and the mellow tone of voice of the three women, together with the costumes and the setting, suggest that it is a harem. The dancers, on seeing Aladdin in danger, say out loud, chanting, that Aladdin hit the bottom. They use an idiomatic expression, more frequently found as ‘to hit rock bottom’, meaning that he reached the worst or lowest possible point. As can be noticed by looking at the screenshot and by evaluating kinesics, while the idiom is uttered, one of the dancers actually hits Aladdin with her bottom. Hence, the literal meaning of the expression corresponds to what is shown by the visual channel. This last example is slightly different from the other two, being a sort of hybrid between humorous and non-humorous. Overall, the situation depicted may appear farcical, given the fact that normally people do not hit each other with their backside. However, since the scene is very fast-

Chapterr Ten

258

paced—just one shot—aand the danceers are singingg and not tallking, the funny connootation is not, in my opinion n, as marked aas it is in the examples that I have cclassified as deeliberately humorous.

5.3. Findings an nd discussioon

Fig. 10-2. Idiooms in the corppus

Fig. 10-3. Idiooms with multiimodal implicattions

As show wn in Fig. 100-2, the majo ority of idiom ms (75 out off 126, or 59.52%) cann be interpretted as having clear multiseemiotic impliccations as far as their meaning-makking is conceerned. A mulltimodal evalu uation of idioms has made it poossible to deetect a markked tendency in their representatioon. In fact, 344 idioms (26..98%) deliberrately contribu ute to the creation of hhumorous situuations (cf. Fig. 10-3). Thee idioms coveering this pragmatic fuunction appeaar to converge towards a sim milar direction n: humour is, in fact, triggered by the double meaning m (seee section 3), which is adapted to thhe multimodaal nature of audiovisual prodducts. The exaamples in Tables 10-2, 10-3 and 10--4 illustrate th hat the visual cchannel tends to depict

How Idiomatic are Disney Animated Films?

259

the literal meaning of idioms, whereas the words uttered (the acoustic channel) would logically refer to the figurative one. More specifically, it seems that in these cases of audiovisual humour, the perlocutionary outcome (i.e., laughter), originates from the contradiction between the locutionary act (i.e., the verbal performance of the idiom) and its deeper illocutionary meaning, mostly conveyed by kinesics. The other 41 idioms (32.53%) do not manifestly create comic relief, even though, as the example in Table 10-7 displays, some cases can be considered liminal. In fact, whether and to what extent something (or someone) is humorous mainly lies in personal value judgements. However, a recurrent pattern of interplay between the audible and visible can be singled out in all these examples. A sort of witty double entendre effect, in fact, clearly stems from something more than the words uttered. Finally, as for the remaining 51 idioms (40.47%) that have marked effects only on one semiotic level (i.e., verbal), their function appears to bestow naturalness and spontaneity on the fictional dialogues.

6. Concluding Remarks The purpose of this research was to shed light on the multimodal implications of the meaning of idioms in a corpus of Disney animated films. The findings showed that idiomatic expressions can be found in all the films in the corpus under investigation, but their usage seems to be foregrounded in more recent and explicitly humorous movies, namely Aladdin and Hercules. The multimodal analysis of the idiomatic expressions that I have located in the corpus suggests that in many cases scriptwriters consciously adapt idioms to the multifaceted nature of animated movies in order to arouse laughter. The dual-sided humour created in these contexts is evidently functional to the vast and diversified audience of Disney animations. In effect, even though children may not comprehend the collision of meanings, as their understanding of idioms may not be fully developed (see section 3), they may enjoy the entertainment provided by the odd actions depicted (i.e., the literal realisations of idioms, which are usually extraordinary actions) as is evident from the visual-non-verbal channel (i.e., the images, which are reinforced by the sounds). Adults, instead, can appreciate the cheerful wit lying behind the sophisticated jokes created by combining the double nature of idioms with the multilayered semiotic framework that characterises animated films. In line with Zabalbeascoa (2008)’s intuition, I agree that “in fictional movies, characters can easily defy the rules of nature and physics” (2008: 64),

260

Chapter Ten

without causing any loss of credibility. This suspension of disbelief, characterising this peculiar genre, makes animated films a privileged site for this kind of audiovisual humour, which does not adhere to realism but boasts of entertainment. Conversely, the idioms whose multimodal meaning-making creates visual puns undoubtedly enrich the film with a sort of wit that does not directly prompt laughter, but makes the movie more winsome and captivating also for grown-ups, who can enjoy the peculiar way in which visual aspects are combined with dialogues. On the whole, the way in which many idioms tend to be represented seems to be functional to ease their decoding, which, as shown above, can be problematic. In fact, either the literal (mostly) or the figurative meaning is represented by the visual channel, which is the most intelligible one for the general audience. From a multimodal point of view, the most meaningful semiotic elements involved in the construction of the meaning of idiomatic expressions are the kinesics and the spoken language. The particular kinds of shots and the sounds, instead, do not seem to carry particularly relevant implications. As a further step in this research and to achieve more precise and satisfactory conclusions, it would be interesting to carry out a reception study, involving both children and adults, in order to test 1) how the audience perceives idiomatic expressions to ascertain whether they are more emotionally engaging than their literal counterparts and whether they do enrich the use of language, 2) whether, and to what extent, they constitute a source of difficulty in decoding, 3) which semiotic components are perceived as pivotal for their meaning-making in animated films, and 4) why/if some scenes involving idioms are more amusing than others.

References Attardo, Salvatore. 1994. Linguistic Theories of Humour. Berlin: Mouton de Gruyter. Baños-Piñero, Rocio, and Frederic Chaume. 2009. “Prefabricated orality. A challenge in audiovisual translation.” Intralinea, Special Issue Online: The Translation of Dialects in Multimedia, edited by Michela Giorgio Marrano, Giovanni Nadiani and Chris Rundle. http://www.intralinea.org/specials/article/1714. Bonsignori, Veronica. 2009. “Transcribing film dialogue. From orthographic to prosodic transcription.” In Analysing Audiovisual Dialogue. Linguistic and Translational Insights, edited by Maria Freddi and Maria Pavesi, 185–200. Bologna: CLUEB. Bruti, Silvia. 2009. “From the US to Rome passing through Paris: Accents

How Idiomatic are Disney Animated Films?

261

and dialects in The Aristocats and its Italian dubbed version.” Intralinea, Special issue Online: The Translation of Dialects in Multimedia, edited by Michela Giorgio Marrano, Giovanni Nadiani and Chris Rundle. http://www.intralinea.org/specials/article/1713. Buckingham, David. 2007. Media Education: Literacy, Learning and Contemporary Culture. Cambridge: Polity Press. Cacciari, Cristina, and Levorato M. Chiara. 1999. “Idiom comprehension in children: Are the effects of semantic analyzability and context separable?” European Journal of Cognitive Psychology 11(1):51–66. Cacciari, Cristina, and Costanza Papagno. 2012. “Neuropsychological and neurophysiological correlates of idiom understanding: How many hemispheres are involved.” Neuropsychology of Language, 368–385. Wiley-Blackwell. Chaume, Frederic. 2004. “Film studies and translation studies: Two disciplines at stake.” Meta 49(1):12–24.͒ —. 2012. Audiovisual translation: Dubbing. Manchester: St. Jerome. Chiaro, Delia. 1998. “Talking Disney: Reflections on what children watch.” In British/American Variation in Language, Theory and Methodology, edited by Carol Taylor Torsello, Louaan Haarman and Laura Gavioli, 97–104. Bologna: CLUEB. —. 2006. “Verbally expressed humour on screen: Reflections on translation and reception.” JoSTrans 6. Chiaro, Delia, Christine Heiss, and Chiara Bucaria. eds. 2008. Between Text and Image. Updating Research in Screen Translation. Amsterdam: John Benjamins. Cooper, Thomas. 1998. “Teaching idioms.” Foreign Language Annals, 31(2):255–266. Di Giovanni, Elena. 2003. “Cultural otherness and global communication in Disney films at the turn of the century.” The Translator 9(2):207– 224. Díaz Cintas, Jorge. ed. 2009. New Trends in Audiovisual Translation. Bristol: Multilingual Matters. Fernando, Chitra. 1996. Idioms and Idiomaticity. Oxford: Oxford University Press. Giroux, Henry. 1995. “Animating youth: The Disneyfication of children’s culture.” Socialist Review 24(3):23–55. Glucksberg, Sam. 2001. Understanding Figurative Language: From Metaphor to Idioms. Oxford: Oxford University Press. González Vera, M. Pilar. 2015. “When humour gets fishy: The translation of humour in animated films.” In Audiovisual Translation: Taking

262

Chapter Ten

stock, edited by Jorge Díaz Cintas and Josélia Neves, 123–139. Newcastle-upon-Tyne: Cambridge Scholars Publishing. Gottlieb, Henrik. 2005. “Multidimensional translation: Semantics turned semiotics.” In Challenges of Multidimensional Translation: MuTra 2005 Conference Proceedings, edited by Heidrun GerzymischArbogast and Sandra Nauert, 33–61. Saarland: Saarland University. Gregory, Michael, and Susanne Carroll. 1978. Language and Situation: Language Varieties and their Social Contexts. London/New York: Routledge. Jackendoff, Ray. 1997. The Architecture of the Language Faculty. Cambridge: MIT Press. Kress, Gunther. 2003. Literacy in the New Media. London/New York: Routledge. Lacey, Nick. 2005. Introduction to Film. Basingstoke: Palgrave Macmillan. Lippi-Green, Rosina. 1997. English with an Accent: Language, Ideology and Discrimination in the United States. London/New York: Routledge. McIntyre, Dan. 2012. “Prototypical characteristics of blockbuster movie dialogue: A corpus stylistic analysis.” Texas Studies in Literature and Language 54(3):402–25. Moon, Rosemund. 1998. Fixed Expressions and Idioms in English: A Corpus-based Approach. Oxford: Clarendon Press. Nippold, Marylin, and Jill Taylor. 2003. “Mental imagery and idiom comprehension: A comparison of school-age children and adults.” Journal of Speech, Language, and Hearing Research 46(4):788–799. Nunberg, Geoffrey, Ivan A. Sag, and Thomas Wasow. 1994. “Idioms.” Language 70(3):491–538. OED. 2015. Oxford English Dictionary. Oxford: Oxford University Press. http://www.oed.com Partington, Alan. 2006. The Linguistics of Laughter: A Corpus-Assisted Study of Laughter-Talk. London/ New York: Routledge. Propp, Vladimir. 1928/1968. Morphology of the Folktale. Translated by Laurence Scott. 2nd ed. Austin: University of Texas Press. Robb, Brian. J. 2014. A Brief History of Walt Disney. London: Constable & Robinson. Ruiz de Mendoza Ibáñez, Francisco, and Louis Pérez Hernández. 2003. “Cognitive operations and pragmatic implication.” In Metonymy and Pragmatic Inferencing, edited by Linda L. Thornburg and Klaus-Uwe Panther, 23–51. Amsterdam: John Benjamins.

How Idiomatic are Disney Animated Films?

263

Searle, John. 1975. “Indirect speech acts.” In Syntax and Semantics Speech Acts, edited by Peter Cole and Jerry L. Morgan, 59–82. New York: Academic Press. Sinclair, John. 1991. Corpus, Concordance and Collocation. Oxford: Oxford University Press. Sklar, Richard. 1990. Film: An International History of the Medium. London: Thames and Hudson. Vignozzi, Gianmarco. 2015. “Instances of Formulaic Language in Disney Movies: A Case Study on Idioms and Conversational Routines.” MA diss., University of Pisa. Watts, Steven. 1997. The Magic Kingdom: Walt Disney and the American Way of Life. Boston: Houghton Mifflin. Wells, Paul. 1998. Understanding Animation. London: Routledge. —. 2002. Animation: Genre and Authorship. London/New York: Wallflower Press. Wildfeuer, Janina. 2013. Film Discourse Interpretation. Towards a New Paradigm for Multimodal Film Analysis. New York: Routledge. Wood, David. 2010. Formulaic Language and Second Language Speech Fluency: Background, Evidence, and Classroom Applications. London/New York: Continuum. Wray, Alison. 2000. “Formulaic sequences in second language teaching: principle and practice.” The Journal of Applied Linguistics 21(4):463– 489. —. 2002. Formulaic Language and the Lexicon. Cambridge: Cambridge University Press. Zabalbeascoa, Patrick. 2008. “Translating dialogues in audiovisual fiction.” The Translation of Fictive Dialogue 35(1):63–78.

Filmography 101 Dalmatians. 1961. Clyde Geronimi, Hamilton Luske, Wolfgang Reitherman, USA. Aladdin. 1992. Ron Clements, John Musker, USA. Bambi. 1942. James Algar, Samuel Armstrong, David Hand, Graham Heid, Bill Roberts, Paul Satterfield, Norman Wright, USA. Cinderella. 1950. Clyde Geronimi, Wilfred Jackson, Hamilton Luske, USA. Dumbo. 1941. Samuel Armstrong, Norman Ferguson, Wilfred Jackson, Jack Kinney, Bill Roberts, Ben Sharpsteen, John Elliotte, USA. Frozen. 2013. Chris Buck, Jennifer Lee, USA.

264

Chapter Ten

Pinocchio. 1940. Norman Ferguson, T. Hee, Wilfred Jackson, Jack Kinney, Hamilton Luske, Bill Roberts, Ben Sharpsteen, USA. Rio. 2011. Carlos Saldanha, USA. Robin Hood. 1973. Wolfgang Reitherman, USA. Shark Tale. 2004. Bibo Bergeron, Vicky Jenson, Rob Letterman, USA. Snow White and the Seven Dwarfs. 1937. William Cottrell, David Hand, Wilfred Jackson, Larry Morey, Perce Pearce, Ben Sharpsteen, USA. The Incredibles. 2004. Brad Bird, USA. Up. 2009. Pete Docter, Bob Peterson, USA.

CONTRIBUTORS

Vicente Beltrán-Palanques is a PhD student in Applied Linguistics at Universitat Jaume I (Castelló, Spain). He is a member of the GRAPE research group (Group for Research on Academic and Professional English). His research interest focuses mainly on interlanguage pragmatics, language teaching methodology, and multimodality. His work has been published in different journals such as ELIA: Estudios de Lingüística Inglesa Aplicada, Journal of English Studies, Language Value, among others. His work has also appeared in different edited collections published by Routledge and Cambridge Scholar Publishing. Veronica Bonsignori holds a PhD in English Linguistics (2007) from the University of Pisa, where she teaches English Language and Linguistics as Adjunct Professor. She was the recipient of research scholarships at the University of Siena (2010-2011) and at the Linguistic Centre of the University of Pisa (2013-2016). Her interests are in the fields of pragmatics, sociolinguistics, audiovisual translation, and multimodality. She has contributed to national and international conferences and has published several articles, focusing on the transposition of linguistic varieties in Italian dubbing (2012, Peter Lang), and on linguistic phenomena of orality in English filmic speech in comparison to Italian dubbing, as in the paper co-authored with Silvia Bruti on introductions and wishes (2016, Equinox). She also authored the monograph English Tags: A Close-up on Film Language, Dubbing and Conversation (2013, Cambridge Scholars Publishing). Silvia Bruti, PhD in English from the University of Pisa, is Associate Professor of English language and linguistics at the University of Pisa. Her research interests include topics such as discourse analysis, (historical) pragmatics, corpus linguistics, audiovisual translation, and language teaching. She has published widely in these areas and contributed to national and international conferences. She has recently investigated issues in intercultural pragmatics and audiovisual translation, e.g. the translation of compliments, conversational routines and terms of address in interlinguistic subtitles and dubbing. One of her most recent works is a monograph on the translation of politeness (2013, Pisa University Press)

266

Contributors

and she co-edited with Elisa Perego the volume Subtitling Today: Shapes and their Meanings (2015, Cambridge Scholars Publishing). Mari Carmen Campoy-Cubillo, PhD, is Senior Lecturer at the Department of English Studies, Universitat Jaume I, Spain. Her main research interests are in the areas of lexicography, language testing and the application of corpus linguistics to the teaching of foreign languages. She has co-edited with P. Safont Computer-Mediated Lexicography in the Foreign Language Learning Context (2004, Publicacions de la Universitat Jaume I), and with B. Bellés-Fortuño and M.LL. Gea-Valor Corpus-Based Approaches to English Language Teaching (2010, Continuum), a compilation of research exploring different ways to apply corpus-based and corpusinformed approaches to English language teaching. She is the author of the textbook Working with Technical and Scientific English. She has published research articles in the International Journal of Lexicography (OUP) and Lexicography: Journal of Asialex. Recent publications include a chapter with M. Querol-Julián entitled “Assessing multimodal listening” in B. Crawford Camiciottoli and I. Fortanet-Gómez (eds.) Multimodal Analysis in Academic Settings: From Research to Teaching (2015, Routledge). Belinda Crawford Camiciottoli is Associate Professor of English Language and Linguistics at the University of Pisa. Her research focuses mainly on interpersonal, pragmatic, and multimodal features of discourse found in both academic and professional settings, with particular reference to corpus-assisted methodologies. She has published in leading international journals, including Intercultural Pragmatics, Journal of Pragmatics, Discourse & Communication, Discourse, Context and Media, English for Specific Purposes, Text & Talk, and Business and Professional Communication Quarterly. She recently co-edited with Inmaculada Fortanet-Gómez the volume Multimodal Analysis in Academic Settings: From Research to Teaching (2015) in the Routledge series Studies in Multimodality edited by Kay O’Halloran. Gloria Cappelli is a Lecturer in English Language at the University of Pisa. Her research focuses on lexical semantics and pragmatics, English for specific purposes, foreign language teaching to learners with specific learning differences and on the complex adaptive systems approach to language acquisition. She is the author of a book on the lexical and pragmatic features of English verbs of cognitive attitude (2006) and of a book on the written genres of English tourism discourse (2007). She was

Multimodality across Communicative Settings, Discourse Domains and Genres

267

co-editor of a volume on lexical complexity (2007, Pisa University Press). She has published on specialised vocabulary (2016, Lingue e Linguaggi) and language development in national and international journals and in collective research volumes. Elena Davitti is a Lecturer in Translation Studies at the University of Surrey, Centre for Translation Studies. She holds a PhD in Translation and Intercultural Studies from the University of Manchester and an MA in Conference Interpreting from the University of Bologna (Forlì). Her research has focused mostly on naturally-occurring interpreter-mediated interaction, which she has analysed through multimodal lenses. Her interests include technologies applied to interpreting, particularly videomediated and mobile interpreting, as well as interpreter and translator education. She has published on several different outlets, including the journals Interpreting (John Benjamins) and Interpreter and Translator Trainer (Routledge) and she has recently co-edited a Special Issue of the Journal of Pragmatics (Elsevier) on participation in interpreter-mediated interaction. Daniele Franceschi holds a Ph.D. in English Linguistics (2008) awarded by the University of Pisa (Italy), where he currently works as an Adjunct Instructor. He teaches ESP at both the undergraduate and graduate levels, and specialises in the fields of law, business, economics and medicine. He has published several papers on domain-specific Englishes (2013, Canadian Center of Science and Education; 2015, Pisa University Press), with publications also in the area of lexical/temporal semantics, pragmatics, language learning, and cognitive studies (2014, Benjamins). He authored the monograph Ingressive and Egressive Verbs in English: A Cognitive-Pragmatic Approach to meaning (2015, Cambridge Scholars Publishing). He has presented papers at a number of international conferences and is an active member of ESSE (European Society for the Study of English), AIA (Associazione Italiana di Anglistica) and SLE (Societas Linguistica Europaea). Silvia Masi is Associate Professor of English Language and Linguistics at the University of Pisa, Department of Philology, Literature and Linguistics. She has published in national and international journals and collective research volumes and is the author of a monograph on the polysemy of some English spatial particles in a contrastive perspective with Italian (2011, Pisa University Press). Other publications focus on formulaic language in film dubbing (in contributions with Veronica

268

Contributors

Bonsignori and Silvia Bruti, 2011, Peter Lang, and 2012, Rodopi), culturebound items such as food in dubbing and in the translation of children’s literature, and metadiscourse in scientific popularisation. More recent research interests include translation strategies in guidebooks for children and multimodality. Sabrina Noccetti is Lecturer of English Language and Linguistics at the University of Pisa. Her research focuses mainly on first language acquisition and second language learning with particular reference to noun and verb morphology, verb tense and aspect and to adjective acquisition in English and Italian. Recently, her interest has focussed on the learning of English as a second language in adult students with dyslexia. She has published several papers in books, including Mouton de Gruyter (2003, 2009), Benjamins (2007), and journals, including First Language (2013) and Linguistics (2015). She co-authored a paper on vocabulary learning in dyslexic students (to be published by Erikson). She was co-editor of a book on the acquisition of adjectives in first languages in a cross-linguistic perspective (2015, Cambridge Scholars). Gianmarco Vignozzi is a PhD candidate in English Linguistics at the University of Pisa. His research interests lie in the area of discourse analysis and corpus linguistics, ranging from political discourse (focus of his PhD research) and filmic discourse, to corpus stylistics and audiovisual translation. He co-authored one article on the representation and pragmatic functions of conversational routines in period dramas (2016, Odoya) and another on the dubbing of language variation from English into Italian in animated movies and period dramas (2016, Status Quaestionis).