The Palgrave Handbook of Intermediality 303128321X, 9783031283215

Citation preview

Jørgen Bruhn Asun López-Varela Azcárate Miriam de Paiva Vieira Editors

The Palgrave Handbook of Intermediality

The Palgrave Handbook of Intermediality

Jørgen Bruhn • Asun Lo´pez-Varela Azcárate • Miriam de Paiva Vieira Editors

The Palgrave Handbook of Intermediality With 92 Figures and 4 Tables

Editors Jørgen Bruhn Linnaeus University Växjö, Sweden

Asun López-Varela Azcárate Complutense University of Madrid Madrid, Spain

Miriam de Paiva Vieira Federal University of São João del-Rei São João del Rei, Brazil

ISBN 978-3-031-28321-5 ISBN 978-3-031-28322-2 (eBook) https://doi.org/10.1007/978-3-031-28322-2 © Springer Nature Switzerland AG 2024 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Palgrave Macmillan imprint is published by the registered company Springer Nature Switzerland AG. The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Paper in this product is recyclable.

This Handbook is dedicated to the memory of our friend and colleague and inspirator Lars Elleström. Lars Elleström has been an academic but also personal role-model for the editors as well as for many junior and senior colleagues worldwide. One chapter in the Handbook specifically explores his influential theoretical work and highlights some of his international and national institutional achievements. By dedicating the Handbook to his memory, we wish to acknowledge and honor his lasting contribution to the field of Intermedial studies. Jørgen Bruhn (Växjö, Sweden) Asun López-Varela Azcárate (Madrid, Spain) Miriam de Paiva Vieira (São João del-Rei, Brazil) Editors

Preface

Intermediality is understood broadly as the study of interrelations and interactions among all forms of media, including transmedial phenomena. With a certain emphasis on artistic media, this Handbook primarily offers extensive overviews over traditional and emerging research areas within the realm of intermediality that will guide scholars, newcomers to the field, and general audiences in discovering the many facets of intermedial theories and practices, without excluding more innovative perspectives such as the new-materialist approaches to the notion of medium. The Handbook is conceived at a moment of increasing interdisciplinarity and media convergence, in which it is ever more vital to think and understand debates across old disciplinary boundaries. Intermediality studies already do this on some level and by their nature is a prime area in which to model cross-disciplinary and comparative work. Therefore, the Handbook is not just about demonstrating the range, variety, and conceptual differences within the field, but about placing different perspectives and approaches in dialogue with one another, thus demonstrating the permeability and perhaps even inadequacy of disciplinary, historical, and other boundaries that have traditionally been used to organize knowledge production. With Europe being the historical center of intermediality-focused scholarship, the Handbook also looks outward toward scholarship about or rooted in other geographical and cultural contexts. It is an invitation to a decidedly global field of intermedial studies that acknowledges worldwide flows and counterflows of media and knowledge production, considering intermediality across multiple spatiotemporal dimensions involving inter-actions of all components. It works centripetally, seeking to organize ideas and concepts, and at the same time centrifugally, broadening the discussion of various geographically and culturally defined theories and practices. Växjö, Sweden Madrid, Spain São João del-Rei, Brazil November 2023

Jørgen Bruhn Asun López-Varela Azcárate Miriam de Paiva Vieira Editors

vii

Acknowledgments

We would like to thank our main Palgrave editorial contact Aysha Shameem for her tireless patience and support, as well as the generosity and collaborative partnership demonstrated by all the contributing authors.

ix

Contents

Volume 1 1

Introduction to the Palgrave Handbook of Intermediality . . . . . . . Jørgen Bruhn, Asun López-Varela Azcárate, and Miriam de Paiva Vieira

Part I Histories and Schools, Theories, and Methods of Intermedial Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

3

1

11

Intermediality: Introducing Terminology and Approaches in the Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marina Grishakova

13

An Updated Survey of Early Interart and Intermediality Roots: Claus Clüver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Solange Ribeiro de Oliveira and Thaι__s Flores Nogueira Diniz

31

4

Ekphrasis: Intermedial and Anglophone Perspectives . . . . . . . . . . Sofie Behluli and Gabriele Rippl

47

5

Intermediality and Medium Specificity . . . . . . . . . . . . . . . . . . . . . . Jarkko Toikkanen

73

6

Intermedialities, Societies, and Power Histories . . . . . . . . . . . . . . . Juergen E. Mueller

97

7

Montreal School of Intermediality: Beyond Media Studies . . . . . . Rémy Besson

135

8

Case Studies as a Heuristic of Intermediality . . . . . . . . . . . . . . . . . Marion Froger and Caroline Bem

159

9

Linnaeus University Center for Intermedial and Multimodal Studies and the Legacy of Lars Elleström . . . . . . . . . . . . . . . . . . . Jørgen Bruhn and Beate Schirrmacher

185

xi

xii

Contents

10

Intermediality in Brazil: A Diachronic Survey . . . . . . . . . . . . . . . . Camila Augusta Pires de Figueiredo, Miriam de Paiva Vieira, Ana Cláudia Munari Domingos, and Érika Viviane Costa Vieira

203

11

An Overview of Intermedial Studies in China . . . . . . . . . . . . . . . . Rong Ou

225

12

Intermediality, Semiotics, and Media Theory . . . . . . . . . . . . . . . . . Tomáš Chudý and Richard Müller

253

13

Intermediality and/in Translation . . . . . . . . . . . . . . . . . . . . . . . . . . Marta Kaźmierczak

285

14

Visual Citation in Intermedial Relations Ana Luiza Ramazzina-Ghirardi

....................

321

15

Reformulating the Theory of Literary Intermediality: A Genealogy from Ut Pictura Poesis to Poststructuralist In-Betweenness . . . . . . Bowen Wang

339

16

Transmedial Narratology and Transmedia Storytelling . . . . . . . . . Raphaël Baroni, Anaïs Goudmand, and Marie-Laure Ryan

365

17

The Narrator: A Transmedial Device . . . . . . . . . . . . . . . . . . . . . . . Liviu Lutas

391

18

Intermediality, Teaching, and Literacy . . . . . . . . . . . . . . . . . . . . . . Ana Cláudia Munari Domingos, Érika Viviane Costa Vieira, Miriam de Paiva Vieira, and Camila Augusta Pires de Figueiredo

415

19

Intermedia, Multimedia, and Media . . . . . . . . . . . . . . . . . . . . . . . . Ken Friedman and Lily Díaz

433

20

Citational Aesthetics: For Intermediality as Interrelation . . . . . . . Mieke Bal

461

Part II Intermedial Perspectives on Media Until the Nineteenth Century: A Living Legacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since the Eleventh Century . . . . . . . . . . . . . . . . . . . . . . . . . . . Rong Ou

22

The Anchor and the Dolphin: A History of Emblems . . . . . . . . . . Heidrun Führer, Cecilia Victoria Muszta, and Viktor Ferdinand Kovács

23

The Age of Wonder and Entertainment: An Introduction to Intermedial Networks in Baroque Culture . . . . . . . . . . . . . . . . . . . Massimo Fusillo and Mattia Petricola

493

495 521

551

Contents

24

25

26

27

xiii

Intermediality in Seventeenth-Century Baroque Celebrations in Hispanic America: Commissions, Poetry, and Ephemeral Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mauricio Vásquez Arias and Andrés Burbano Valdés

577

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dominika Bugno-Narecka

603

Crossing Media Borders: From Intermedial Shakespeares to Shakespearean Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Víctor Huertas-Martín

627

Metareference in the Nineteenth-Century Pictorial Press and Beyond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sonya Petersson

663

28

Picturing Music in the Nineteenth Century . . . . . . . . . . . . . . . . . . Bálint Veres

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht) and Their Resonances in Contemporary Performance . . . Chiel Kattenbelt and Robin Nelson

693

723

Volume 2 Part III Intermedial Perspectives on Media in the Twentieth Century: New Mediascapes in a Growing World . . . . . . . . . . . . . . . . 30

Intermediality and Liveness at the Turn of the Twentieth Century . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mary Simonson

749

751

31

The Sonification of Modernist Fiction: A Critical Review . . . . . . . Niklas Salmose

769

32

Adaptation and Sound Kate Newell

..................................

803

33

Music Transformation in Literature . . . . . . . . . . . . . . . . . . . . . . . . Beate Schirrmacher

833

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Márcia Arbex-Enrico

865

35

Anthropophagic Appropriation and Intermediality . . . . . . . . . . . . Anna Stegh Camati

899

36

Late Twentieth-Century Intermedia Poetry in the Americas . . . . . Rebecca Kosick

921

xiv

Contents

37

Photojournalism and Beyond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Baetens and Domingo Sánchez-Mesa

38

Media Borders in a Post-Media Age: The Historical and Conceptual Co-evolution of Cinema, Television, Video, and Computer Screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrea Virginás

39

The Qualified Medium of Computer Games: Form and Matter, Technology, and Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ida Kathrine Hammeleff Jørgensen

Part IV Intermedial Perspectives on Media in the Twenty-First Century: Challenges in Contemporary Society . . . . . . . . . . . . . . . . .

955

979

999

1031

40

The Ecological Crisis and Intermedial Studies . . . . . . . . . . . . . . . . 1033 Jørgen Bruhn, Matilda Davidsson, and Niklas Salmose

41

Simulated Climate in Ecological Games: Mediating Climate Change to Endow Players with Transformative Agency . . . . . . . . 1061 Péter Kristóf Makai

42

Intermediality in Theme Parks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1089 Florian Freitag

43

Interactive and Participatory Sound Vadim Keylin

44

Intermediality and Computer Simulation . . . . . . . . . . . . . . . . . . . . 1135 Jens Schröter

45

Intermediality and Digital Fiction . . . . . . . . . . . . . . . . . . . . . . . . . . 1147 Bartosz Lutostański

46

Intermediality and Metamediality: From Analog Representations to Digital Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1169 Klaus Bruhn Jensen

47

The Recommended Experience: Engaging Networked Media Platforms with Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1195 Maja Bak Herrie

48

Posthuman Intermedial Semiotics and Distributed Agency for Sustainable Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1213 Asun López-Varela Azcárate

. . . . . . . . . . . . . . . . . . . . . . . 1115

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1241

About the Editors

Jørgen Bruhn, PhD., is professor of Comparative Literature, Linnæus University, Sweden. He has written and co-edited or edited several books and special journal issues, his three latest monographs being The Intermediality of Narrative Literature: Medialities Matter (Palgrave Macmillan 2016), with Anne Gjelsvik, Cinema Between Media: An Intermedial Approach (Edinburgh UP, 2018), and, with Niklas Salmose, Intermedial Ecocriticism: Mediations of the Climate Crisis across Media (forthcoming 2023, Lexington Books). His main research areas are literary theory, intermediality and media studies, ecocriticism, and environmental humanities. He recently co-edited two volumes related to intermedial studies: Multispecies Storytelling in Intermedial Practices (Punctum Books, eds Jørgen Bruhn and Ida Bencke, 2021) and Intermedial Studies. Meaning Making across Media (Routledge, eds Jørgen Bruhn and Beate Schirrmacher, 2022). Jørgen Bruhn uses Facebook and, to a limited degree, Academia and ResearchGate to promote and disseminate his research. Asun López-Varela Azcárate is Associate Professor at the Department of English Studies, Universidad Complutense de Madrid. Her research interests are Comparative Literature, Cultural Studies, as well as Cognitive and Intermedial Semiotics. Since 2007, she has been coordinating the research program Studies on Intermediality and Intercultural Mediation SIIM. In 2013, she was awarded a Fulbright Visiting Scholarship at Harvard University, Department of Comparative Literature. A proactive member of the profession, xv

xvi

About the Editors

currently, López-Varela is Vice-Chair at European Commission Unit REA.A2, Marie Skłodowska-Curie European Postdoctoral Fellowships, Social Sciences and Humanities (SOC). López-Varela is also Network Coordinator at New Directions in the Humanities. In order to strengthen relations between Europe and Asia, López-Varela coordinates an annual seminar series on cross-cultural dialogue and sustainability funded by the Eurasia Foundation. She is honorary member of the Poetry Award Committee of Beijing Literature and ArtNetwork. For López-Varela’s activities as editor and member of scientific committees in various academic journals, please see https://www.ucm.es/siim/ asun-lopez-varela. Her academic publications can be seen at https://www.ucm.es/siim/lopez-varela-publica tions and at https://orcid.org/0000-0003-1616-5830 Miriam de Paiva Vieira is a Professor at the Department of Letters, Arts, and Culture at Universidade Federal de São João del Rei, Brazil. Her research interests are Comparative Literature and Intermedial Studies, with a focus on ekphrasis and the relations between literature and architecture. She has been granted funding from the Brazilian Council for Scientific and Technological Development Scholarship in their Research Productivity Program (CNPq/PQ2 2022–2025). She did a visiting professor fellowship at John Paul II Catholic University of Lublin, Poland (spring 2023), and a postdoctoral fellowship funded by CNPq at UFMG, Brazil (2018). Vieira holds a doctorate degree in Literary Studies (2016) and a bachelor’s in Architecture (1991). She is a board member of the International Society for Intermedial Studies and a member of IAWIS/AIERTI, CRIalt, and Grupo Intermídia (CNPq). Besides publishing articles, Vieira has edited journal dossiers and the book anthologies Escrita, som, imagem, V.1 (2020) and V.2 (2019), with Arbex, Diniz, Figueiredo, Lima. Her academic publications can be accessed at https://helder. academia.edu/MiriamVieira and https://orcid.org/00000001-9851-0217

Contributors

Márcia Arbex-Enrico Universidade Federal de Minas Gerais – UFMG/CNPq, Belo Horizonte, Brazil Jan Baetens Cultural Studies, KU Leuven, Leuven, Belgium Maja Bak Herrie Postdoc at School of Communication and Culture at Aarhus University, Aarhus, Denmark Mieke Bal Amsterdam School of Cultural Analysis (ASCA), Amsterdam, The Netherlands Raphaël Baroni University of Lausanne, Lausanne, Switzerland Sofie Behluli English Department, University of Bern, Bern, Switzerland Caroline Bem Saint Paul University, Ottawa, ON, Canada Rémy Besson University of Montreal, Montreal, QC, Canada Jørgen Bruhn Linnaeus University, Växjö, Sweden Klaus Bruhn Jensen Department of Communication, University of Copenhagen, Copenhagen, Denmark Dominika Bugno-Narecka Institute of Literary Studies, John Paul II Catholic University of Lublin, Lublin, Poland Andrés Burbano Valdés Faculty of Arts and Humanities, Open University of Catalonia, Barcelona, Spain Anna Stegh Camati Centro Universitário Campos de Andrade (UNIANDRADE), Curitiba, Brazil Tomáš Chudý Czech National Bank, Prague, Czech Republic Matilda Davidsson Linnaeus University, Växjö, Sweden Camila Augusta Pires de Figueiredo Universidade Federal de Minas Gerais, Belo Horizonte, Brazil xvii

xviii

Contributors

Miriam de Paiva Vieira Universidade Federal de São João del Rei, São João delRei, Brazil Lily Díaz Art and Media Department, Aalto University, Espoo, Finland Florian Freitag University of Duisburg-Essen, Essen, Germany Ken Friedman College of Design and Innovation, Tongji University, Shanghai, China Marion Froger Université de Montréal, QC, Canada Heidrun Führer Lund University, Lund, Sweden Massimo Fusillo Department of Human Sciences, University of L’Aquila, L’Aquila, Italy Anaïs Goudmand Sorbonne University, Paris, France Marina Grishakova University of Tartu, Tartu, Estonia Ida Kathrine Hammeleff Jørgensen Department of Media, Design, Education and Cognition, University of Southern Denmark, Kolding, Denmark Víctor Huertas-Martín Universitat de València, Valencia, Spain Chiel Kattenbelt Utrecht University, Utrecht, The Netherlands Marta Kaźmierczak Institute of Applied Linguistics, University of Warsaw, Warsaw, Poland Vadim Keylin Institute for German Language and Literature, Universität Hamburg, Hamburg, Germany Rebecca Kosick University of Bristol, Bristol, UK Viktor Ferdinand Kovács Lund University, Lund, Sweden Asun López-Varela Azcárate English Studies, Comparative Literature, Semiotics, Universidad Complutense, Madrid, Spain Liviu Lutas Linnaeus University, Växjö, Sweden Bartosz Lutostański University of Warsaw, Warsaw, Poland Péter Kristóf Makai Rachel Carson Center for Environment and Society, München, Germany, and Kazimierz Wielki University, Bydgoszcz, Poland Juergen E. Mueller (Professor emeritus) University of Bayreuth, Bayreuth, Germany Richard Müller Institute of Czech Literature of the CAS, Prague, Czech Republic Ana Cláudia Munari Domingos Universidade de Santa Cruz do Sul, Santa Cruz do Sul, Brazil

Contributors

xix

Cecilia Victoria Muszta Lund University, Lund, Sweden Robin Nelson Royal Central School of Speech and Drama, University of London, London, UK Manchester Metropolitan University, Manchester, UK Kate Newell Savannah College of Art and Design, Savannah, GA, USA Thaι__s Flores Nogueira Diniz Universidade Federal de Minas Gerais, Belo Horizonte, Brazil Rong Ou Hangzhou Normal University, Hangzhou, China Sonya Petersson Department of Culture and Aesthetics, Stockholm University, Stockholm, Sweden Mattia Petricola Department of Human Sciences, University of L’Aquila, L’Aquila, Italy Ana Luiza Ramazzina-Ghirardi UNIFESP, São Paulo, Brazil Solange Ribeiro de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brazil Gabriele Rippl English Department, University of Bern, Bern, Switzerland Marie-Laure Ryan Independent Scholar, Bellvue, CO, USA Niklas Salmose Linnaeus University, Växjö, Sweden Domingo Sánchez-Mesa Literary Theory, Universidad de Granada, Granada, Spain Beate Schirrmacher Linnaeus University, Växjö, Sweden Jens Schröter University of Bonn, Bonn, Germany Mary Simonson Colgate University, Hamilton, NY, USA Jarkko Toikkanen University of Oulu, Oulu, Finland Mauricio Vásquez Arias School of Arts and Humanities/Creation Area, Universidad EAFIT, Medellín, Colombia Bálint Veres Moholy-Nagy University of Art & Design, Budapest, Hungary Érika Viviane Costa Vieira Universidade Federal dos Vales do Jequitinhonha e Mucuri, Diamantina, Brazil Andrea Virginás Faculty of Theatre and Film, Babeş-Bolyai University, ClujNapoca, Romania Bowen Wang Trinity College Dublin, Dublin, Ireland

1

Introduction to the Palgrave Handbook of Intermediality Jørgen Bruhn , Asun Lo´pez-Varela Azcárate Miriam de Paiva Vieira

, and

Abstract

In this introductory chapter, the authors outline the background of the handbook and the sad reasons why Lars Elleström could not finalize the project that he had sketched. Following this, a few of the “slogans” of intermedial studies are briefly mentioned and discussed, before all chapters in the handbook receive an opening comment. Keywords

Intermediality · Intermedial studies · Handbook in intermedial studies

The Palgrave Handbook of Intermediality aims to offer an updated overview of intermedial studies. Intermediality is broadly understood in this handbook as the study of interrelations and interactions among all forms of communicative media types, including transmedial phenomena, but with a certain emphasis on artistic media types. The handbook offers overviews of traditional research areas within the realm of intermediality. It also includes emerging and innovative perspectives on the field considering, for instance, neomaterialist approaches that extend the notion of medium and materiality. The authors represent a very broad range of scholarly

J. Bruhn (*) Linnaeus University, Växjö, Sweden e-mail: [email protected] A. López-Varela Azcárate English Studies, Comparative Literature, Semiotics, Universidad Complutense, Madrid, Spain e-mail: [email protected] M. de Paiva Vieira Universidade Federal de São João del Rei, São João del-Rei, Brazil e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_1

1

2

J. Bruhn et al.

disciplines and geographical contexts, and features contributions from established authorities as well as promising young researchers. When Lars Elleström was approached by Palgrave Macmillan in 2020 with the request of editing a handbook on intermedial studies, he accepted after some hesitation, because he understood that it would be an enormous task to sketch the outlines of a handbook as well as to access and later edit all the material that such a handbook would demand. Elleström accepted the invitation and then gathered an editorial group of scholars from North and South America as well as Europe, namely Jørgen Bruhn, Asun López-Varela Azcárate, Miriam de Paiva Vieira, and Mary Simonson, and the group started working on the overall layout and sending out call for papers as well as addressing individual scholars with particular research topics that should be covered. Lars Elleström and the editorial group agreed on having an overview that covered intermediality as a deep historical and geographically broad phenomenon, because we wished to end the tendency to contemplate intermediality as something contemporary and centered in Europe and North America. It was vital to cover both artistic and nonartistic phenomena, which is an important trend in contemporary intermedial studies – and we were eager to provide some kind of overview of intermedial studies as an academic endeavor. In the middle of the productive editorial process, Lars Elleström unexpectedly and shockingly passed away in December 2021, leaving family, friends, and colleagues in sorrow and distress. The new editorial group, namely Jørgen Bruhn, Asun López-Varela Azcárate, and Miriam de Paiva Vieira, accepted the request from Palgrave to continue the editing process. The handbook offers extensive overviews of classical and emerging research areas within the realm of intermediality. The goal is to guide readers in discovering the many facets of intermedial theories and practices, without excluding innovative perspectives. The volume is conceived at a moment of increasing interdisciplinarity and media convergence, and it is ever more vital for scholars to think and understand debates across disciplinary boundaries. Intermedial studies already do this on some level, and it is in their very nature to model cross-disciplinary and comparative work. Therefore, the handbook is not just about demonstrating the range, variety, and conceptual differences within the field, but about placing different perspectives and approaches in dialogue with one another, and demonstrate the permeability, and perhaps even the inadequacy of disciplinary, historical, and other boundaries that have traditionally been used to organize knowledge production. This includes discussing how intermedial studies engage in or depart from adjacent but distinct areas of inquiry such as media history, media studies, media archaeology, and image studies, among others. Historically, Europe was probably the initial center of intermedial research, but the handbook looks toward scholarship about or rooted in other geographical and cultural contexts, where interest in the field has been growing for decades and where crucial work is being done. The book is thus an invitation and an introduction to a global field of intermedial studies that acknowledges worldwide flows and counterflows of media and knowledge production, and the contributions consider intermediality across multiple spatiotemporal dimensions involving intra-actions

1

Introduction to the Palgrave Handbook of Intermediality

3

and inter-actions of all components. The framework is organized centripetally, seeking to organize ideas and concepts, and at the same time centrifugally, broadening the discussion of various geographically and culturally defined theories and practices. As claimed by Kamilla Elliot when discussing the field of adaptation studies, the “best places to find larger issues addressed today and tomorrow are field companions and handbooks, as a brief survey of their contents attests” (Elliott 2017, n.p.). Earlier works do exist that explicitly or implicitly have attempted to give an overview of intermedial studies, such as Changing Borders: Contemporary Positions in Intermediality (2007, Eds.: Arvidson; Askander, Bruhn, Führer), Media inter Media: Essays in Honor of Claus Clüver (2009, Ed. Glaser), parts of the important series Studies in Intermediality (2006–2019, Ed. Bernhart et al). Two recent volumes are perhaps particularly important to mention: In 2015, Gabriele Rippl edited the ambitious and comprehensive Handbook of Intermediality. Literature – Image – Sound – Music (Rippl 2015). In the “Introduction,” the editor gives a thorough and engaging introduction to discussions and terminology as well as many of the thorny questions in the field. After the “Introduction” follows 34 chapters, often by established scholars, divided into three parts: first “Text and Image” (the largest section), followed by “Music, Sound, and Performance,” and finally a short section on “Intermedial Methodology and Intersectionalities.” Its focus is intermediality and literature but in a wide variety of forms which is not surprising given Rippl’s research interests. It also reflects that the handbook was the first volume in a De Gruyter series, Handbooks of English and American Studies: Texts and Theory. Less comprehensive, but covering a wider field of communicative forms, Jørgen Bruhn and Beate Schirrmacher’s Intermedial Studies: An Introduction to Meaning Across Media (Bruhn and Schirrmacher 2021) offers an entrance to the field of intermedial studies, very much based on the multimodal framework of intermediality constructed by Lars Elleström and colleagues at the Linnaeus University Center for Intermedial and Multimodal Studies in Sweden. Conceptualized more as a kind of advanced textbook and less as a theoretical or historical overview, Bruhn and Schirrmacher’s book is meant to facilitate intermedial analysis of anything from ekphrasis and video games to rock music performances and climate change communication. The goal of The Palgrave Handbook of Intermediality – as hinted above – is to offer an outline of the productive position of intermedial studies today. We both hope and believe that this handbook testifies to this. The purpose of this short introduction is not to offer an overview of intermedial studies or intermedial phenomena as it stands today or how the field developed. Instead, it might be useful to point out a few simple guiding principles that are perhaps best covered by way of some of the slogans or catchphrases that for some reason has been part and parcel of the development of media and intermedial studies. The earliest one arguably being McLuhan’s notion from the 1960s that “media are the extensions of man” and that we, because of the new medial conditions, live in a “global village” (some of the ideas are collected in McLuhan 2001), which opened up media studies to study the extremely broad field of anything that in some sense

4

J. Bruhn et al.

facilitates human beings’ relation to the world. Later on, another North-American scholar, W.J.T. Mitchell, coined the intermedial slogan that “all media are mixed media” (2005), basically expressing opposition to Clement Greenberg’s archmodernist idea of pure medial forms, and thus carving out an aesthetic and ideological field of intermedial studies. In a distinctly more European tradition of thinking, Friedrich Kittler offered a darker war metaphor that “media define our situation” where “situation” referred to a “military strategic” discourse (as discussed in Kittler et al. 1999; Mitchell and Hansen 2010). McLuhan’s understanding of media as “extensions of man” marks the increasingly widespread practice of seeing a conventional and delimited understanding of media as unproductive. Instead, media and intermedial studies are not limited to fictional or artistic works but can deal with basically all imaginable medial expressions. Mitchell’s oft-repeated catchphrase “all media are mixed media” is an argument for seeing all communicative forms as being mixed and worked as something of a rallying cry in intermedial studies in the 2000’s. Today the slogan appears less provocative, but the idea that intermedial studies, in parallel with multimodal insights, offer a privileged vision of the hybridity of all imaginable meaning production is still pertinent. Kittler’s war rhetoric should remind us, like Mitchell did and still does, that media practices and histories are always embedded in power relations that need to be untangled, as several writers in this handbook do. More recently, in what turned out to be his last interview, Lars Elleström, in an uncharacteristically broad and almost casual phrase, claimed that “everything is intermedial” (Elleström et al. 2020). Apparently, this phrase cumulates the other slogans into one suggestive research program: why not see all the human and more than human relationships to the world as extended, medially mixed relations to a world embedded in power relations but probably also exhibiting some relatively consistent though historically changing structures? This, in a sense, is what the many brilliant contributions to this handbook, seen as a collective effort, manage to do. The handbook follows an outline of four parts, where the first includes broader, systematic overviews of general questions, and the three following parts are very roughly divided into a chronological sequence. Part I, “Histories, Schools, Theories, and Methods of Intermedial Research,” offers accounts of the development of the field of intermediality and its core issues. It contains articles on histories, theories, and methods of intermediality. Because intermedial studies have grown out of a variety of cultural roots and research fields, there are several stories to tell about the historical development and theoretical positions. An overview of intermedial studies is offered by ▶ Chap. 2, “Intermediality: Introducing Terminology and Approaches in the Field” delineated by Marina Grishakova, who aims at both conveying and opening the idea of intermediality as it is understood in recent and contemporary research literature. In ▶ Chap. 3, “An Updated Survey of Early Interart and Intermediality Roots: Claus Clüver,” Thaïs Diniz and Solange Oliveira trace the progress of Claus Clüver’s work on intermedial studies and his recent concerns with the future of the discipline. In ▶ Chap. 4, “Ekphrasis: Intermedial and Anglophone Perspectives,” Gabriele Rippl and Sofie Behluli give a concise overview of the research field in the Western

1

Introduction to the Palgrave Handbook of Intermediality

5

tradition and highlight newer trends that bring ekphrasis in conversation with, for instance, ecocriticism, the cognitive sciences, and digital humanities. Also from a historical perspective, Jarkko Toikkanen surveys in ▶ Chap. 5, “Intermediality and Medium Specificity” through its core concept. In ▶ Chap. 6, “Intermedialities, Societies, and Power Histories,” Jürgen E. Müller tackles a selection of paradigmatic intermedial power plays based on several historical cases, ranging from prehistorical caves, medieval mystery plays, tapestries, nodes of intermedial networks of colporteurs/pedlars in modern times, TV news, and internet platforms to postmodern documentaries. The chapter’s major challenge lies in the development of historical and stable categories for the reconstruction of social functions of intermedial phenomena. Relating to some historically strong environments, Rémy Besson presents the intermedial perspective developed by the so-called Montreal School of Intermedial Studies in ▶ Chap. 7, “Montreal School of Intermediality: Beyond Media Studies.” Including the work of Despoix, Mariniello, Méchoulan, and Villeneuve, the work related to the Montreal School aims less at the study of media for its own sake, than at a reflection of political and philosophical issues connected to media and mediation. Also taking as a starting point the Montreal-based, international, and peer-reviewed journal Intermédialités/Intermediality, ▶ Chap. 8, “Case Studies as a Heuristic of Intermediality,” by Marion Froger and Caroline Bem, explores how case studies published between the journal’s inception in 2003 have given rise to a heuristic of intermediality. By delving into this archive, the chapter performs itself as a case study. Written by Beate Schirrmacher and Jørgen Bruhn, ▶ Chap. 9, “Linnaeus University Center for Intermedial and Multimodal Studies and the Legacy of Lars Elleström,” provides an overview of the main contributions of the so-called Växjö school to the field of intermedial studies. The chapter also presents Elleström’s rich contribution to the field. To encompass the reach of intermedial studies in non-European centers, Camila Figueiredo, Miriam Vieira, Ana Domingos, and Érika Vieira offer ▶ Chap. 10, “Intermediality in Brazil: A Diachronic Survey,” while Rong Ou makes in ▶ Chap. 11, “An Overview of Intermedial Studies in China.” From a theoretical position, Tomáš Chudý and Richard Müller argue for a reversal in weighing the importance of the cognitive versus the technical dimension of the medium as this can be observed in intermedial studies. The consequence has been that intermedial research has tended to lose sight of the internal logic of media evolution. ▶ Chapter 12, “Intermediality, Semiotics, and Media Theory,” presents a reconceptualization that contributes to the qualification of the medium where technics and semiotics are studied together. In ▶ Chap. 13, “Intermediality and/in Translation,” Marta Kaźmierczak discusses intersemiotic transformation into a different medium and its similarities with interlingual translation. In ▶ Chap. 14, “Visual Citation in Intermedial Relations,” Ana Luiza Ramazzina-Ghirardi offers a critical overview of the different uses of visual citations and examines its working when applied to different media and intermedial processes. Meanwhile, in ▶ Chap. 15, “Reformulating the Theory of Literary Intermediality: A Genealogy from Ut Pictura Poesis to Poststructuralist In-Betweenness,” Bowen Wang argues that the notion of intermediality should be

6

J. Bruhn et al.

reformulated as a conceptual and ideological extension of ut pictura poesis framed within a poststructuralist in-betweenness. In ▶ Chap. 16, “Transmedial Narratology and Transmedia Storytelling,” Raphaël Baroni, Anaïs Goudmand, and Marie-Laure Ryan explore the differences in the notion of transmediality, exemplified by transmedial narratology and transmedia storytelling. The chapter describes the evolution of narratology from a languagecentered to a transmedial field of study as well as the various manifestations of transmedia storytelling and its theoretical implications. Liviu Lutas proposes that narration is a transmedial phenomenon in ▶ Chap. 17, “The Narrator: A Transmedial Device,” as a mental construction in the process of communication at the basis of several media types. ▶ Chapter 18, “Intermediality, Teaching, and Literacy,” by Ana Domingos, Érika Vieira, Miriam Vieira, and Camila Figueiredo, assembles and comments upon methods of intermediality to discuss the issue of literacy and the systematic teaching of intermediality. ▶ Chapter 19, “Intermedia, Multimedia, and Media,” by Ken Friedman and Lily Diaz-Kommonen aims at recovering the lost history of intermedia with a focus on the contribution by Dick Higgins and the international laboratory for experimental art, design, and music, known as Fluxus (of which Friedman was himself a member). To close the first part, in ▶ Chap. 20, “Citational Aesthetics: For Intermediality as Interrelation,” Mieke Bal brings intermediality in contact with a concept of inter-temporality called “preposterous history,” in which quotation and citation are specific ways of integrating older and newer media products and their connections to different media. The chapter develops a variety of forms of citation through a close look at cases from recent media history. The integration of theory and the practice of analysis of concrete cases is also an element of the chapter’s insights regarding the teachability of intermediality. Part II, “Intermedial Perspectives on Media Until the Nineteenth Century: A Living Legacy,” and Part III, “Intermedial Perspectives on Media in the Twentieth Century: New Mediascapes in a Growing World,” explore intermedial facets of communication from ancient times and to the turn of the twenty-first century. The parts demonstrate the broad applicability of intermedial theories, methods, and concepts. Importantly, each part includes articles on a broad variety of cultural settings (not only Western), media types (including nonartistic as well as artistic), and themes and topics (what the media communicate). Following the terminology of Elleström, “media integration” perspectives as well as “media transformation” perspectives are applied, or in other words, synchronic as well as diachronic analyses of intermedial relations are covered. In her ▶ Chap. 21, “Traditional Chinese Painting: An Intermedial Play of Sister Arts Since the Eleventh Century,” Rong Ou explores the complex interaction of different media in traditional Chinese painting. In ▶ Chap. 22, “The Anchor and the Dolphin: A History of Emblems,” Heidrun Führer, Cecilia Mustaf, and Viktor Kovács explore the history of emblems in relation to tropes such as symbol, ekphrasis, or allegory, and deconstruct the common narrative of emblems. With early modern and contemporary examples, the chapter demonstrates the historical,

1

Introduction to the Palgrave Handbook of Intermediality

7

technological, and sociocultural background directing the fluid expectations on this genre. In ▶ Chap. 23, “The Age of Wonder and Entertainment: An Introduction to Intermedial Networks in Baroque Culture,” Massimo Fusillo and Mattia Petricola study the baroque mediascape in Europe from the perspective of intermedial studies, including baroque aesthetics and the emotional responses that baroque media products strived to elicit. ▶ Chapter 24, “Intermediality in Seventeenth-Century Baroque Celebrations in Hispanic America: Commissions, Poetry, and Ephemeral Architecture,” by Mauricio Vásquez Arias and Andrés Burbano Valdés, is concerned chiefly with a set of early forms of media convergence and intermediality in the HispanicAmerican context, associated with what is now defined as curation, poetry, and ephemeral architecture. As for ▶ Chap. 25, “Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon,” Dominika Bugno-Narecka considers cabinets of curiosities as intermedial composites, with objects connected to the significance, power, and intellectual abilities of the collector. The author claims that, despite the changes, the media type still convey the image of the world and communicate states of knowledge of a particular era and a specific collector. Moreover, Víctor HuertasMartín discusses the relations between Shakespeare’s dramas and intermediality on stage, screen, and in social media. ▶ Chapter 26, “Crossing Media Borders: From Intermedial Shakespeares to Shakespearean Intermediality” shows that in this case, intermediality depends on creative interpretive decisions and receivers’ perceptions. ▶ Chapter 27, “Metareference in the Nineteenth-Century Pictorial Press and Beyond” by Sonya Petersson explores metareference as a medial and semiotic phenomenon within the nineteenth-century pictorial press and art culture. In particular, the chapter examines the Swedish journal Ny illustrerad tidning (1865–1900). The chapter’s metareferential examples are highlighted as tools to reexamine established discourses on media, mediation, and representation. In ▶ Chap. 28, “Picturing Music in the Nineteenth century,” Bálint Veres discusses the interplay of music and its surrounding verbal and visual discourses, such as its depiction in a painting, considering also early examples of the audiovisual experience and their aesthetic and historical preconditions in eighteenth and nineteenth centuries. Chiel Kattenbelt and Robin Nelson present a threefold proposal in ▶ Chap. 29, “Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht) and Their Resonances in Contemporary Performance.” Firstly, they identify historical prototypes of interrelations between mediums from the late nineteenth to early twentieth centuries, which sought new intermedial forms of artistic expression aimed at particular ends. Secondly, the authors focus on the effects and affects intended to be achieved in their historical contexts. Finally, the chapter updates the trajectories of these prototypes as they have impacted the late twentieth and early twenty-first centuries. Part III opens with ▶ Chap. 30, “Intermediality and Liveness at the Turn of the Twentieth Century,” by Mary Simonson. It surveys scholarship on intermedial performance at the turn of the twentieth century focusing particularly on its invocation in interdisciplinary studies of both live theatre and pre- and early cinema technologies. Niklas Salmose contextualizes the historical and aesthetic era of high

8

J. Bruhn et al.

modernist fiction (1918–1939) in ▶ Chap. 31, “The Sonification of Modernist Fiction: A Critical Review” by concentrating specifically on its multimodal and sonorous qualities. The chapter briefly discusses conceptualizations and taxonomies of musico-literary theory as well as musico-literary criticism. Kate Newell, in ▶ Chap. 32, “Adaptation and Sound,” considers intermedial relations in terms of sound technologies and the related adaptive engagement, as well as the critical and historical conversation generated in response to those relationships. Beate Schirrmacher explores in ▶ Chap. 33, “Music Transformation in Literature” how intermedial relations with music transform literary narration and expression. Elleström’s concept of media transformation and structures of transmediation are also explored. From a visual perspective, Márcia Arbex-Enrico studies collage from its emergence in the avant-garde (from cubism, dadaism, and futurism to surrealism). Thus, ▶ Chap. 34, “Collage as a Creative Act: Emergence, Displacement, and Re-signification” relates to intertextual practices, such as citation, by sharing the principles of borrowing and of appropriation as well as hybrid forms of media combination, such as the object-poem and the collage-novel. In ▶ Chap. 35, “Anthropophagic Appropriation and Intermediality,” Anna Camati argues that anthropophagic appropriation or cultural anthropophagy is a form of non-subaltern intertextuality and that, seen from a diachronic perspective, it is a specific mode of creative media transformation that can be located within the broader scholarly field of intermediality. In ▶ Chap. 36, “Late Twentieth-Century Intermedia Poetry in the Americas,” Rebecca Kosick relies on Irina Rajewsky’s work to demonstrate the breadth of intermedial practices within the Americas and the elasticity of recent American poetry by drawing examples primarily from the interval between two peaks of intermedia fervor, roughly from the 1960s to the 1990s. ▶ Chapter 37, “Photojournalism and Beyond,” by Jan Baetens and Domingo Sánchez-Mesa, considers photojournalism as an intermedial practice and tackles historical and theoretical questions, both intermediality as the copresence and interaction of different media within a single medium as well as transmediality as the distribution of a story in a different medium. ▶ Chapter 38, “Media Borders in a Post-Media Age: The Historical and Conceptual Co-evolution of Cinema, Television, Video, and Computer Screens,” by Andrea Virginás, offers an overview of literature theorizing the current condition defined by electronic screens, often called a post-cinema age, the age of expanded or fragmented cinema, or indeed named the spatial turn in the analysis of electronically mediated audiovisual communication. By taking computer games as a qualified medium situated in an intermedial network of other qualified and technical media and media cultures, ▶ Chapter 39, “The Qualified Medium of Computer Games: Form and Matter, Technology, and Use,” by Ida Kathrine Hammeleff Jørgensen, offers an overview of the state of the art of gaming studies and sheds light on the relations between games and other media. Part IV, “Intermedial Perspectives on Media in the Twenty-First Century: Challenges in Contemporary Society,” is the last in the handbook, and it is devoted to contemporary intermedial perspectives, including recent technological developments,

1

Introduction to the Palgrave Handbook of Intermediality

9

with a special emphasis on urgent societal issues. This last part introduces the theme of ▶ Chap. 40, “The Ecological Crisis and Intermedial Studies” in which Jørgen Bruhn, Matilda Davidsson, and Niklas Salmose argue that there is a need for a method of intermedial comparison that can encompass different disciplines and media types dealing with the ecological crisis. In the chapter, an intermedial toolbox is combined with the field of ecocriticism. Understanding video games as tools for representing complexity, ▶ Chap. 41, “Simulated Climate in Ecological Games: Mediating Climate Change to Endow Players with Transformative Agency,” by Péter Kristóf Makai and Kazimierz Wielki, introduces the reader to the challenges of using games as media for communicating climate change and how to use ecocritical analysis done on climate change games in order to raise awareness. In ▶ Chap. 42, “Intermediality in Theme Parks,” Florian Freitag provides readers with a broad overview of the plurimediality of theme parks and their multisensory, immersive experiences. The chapter also studies transmedia conglomerates and the distribution of transmedia franchises, as well as the paramediality associated with theme parks. Vadim Keylin, in his turn, offers an overview of contemporary research into technological and aesthetic aspects of interactive audio from the fields of sound studies, video games studies, media studies, and, to a lesser extent, music sociology and HCI. His ▶ Chap. 43, “Interactive and Participatory Sound” concentrates on three principal domains: interactive and participatory sound art; sound in immersive media (video games and VR); and sound practices of online participatory cultures. Jens Schröter relates to crucial phenomena in digital media culture in ▶ Chap. 44, “Intermediality and Computer Simulation” by doing a close analysis of a popular movie, Monsters, Inc. (2001), to show how computer simulation allows for new forms of “transmaterial” intermediality. As for ▶ Chap. 45, “Intermediality and Digital Fiction,” Bartosz Lutostański presents an overview of the evolution of digital environments from hypertext to apps and digital fiction in connection to intermediality. From historical analysis to elaborations on the relationship between metamedia and metacommunication, ▶ Chap. 46, “Intermediality and Metamediality: From Analog Representations to Digital Resources,” by Klaus Bruhn Jensen, explores the ethical and political implications of metamedia. Meanwhile, the spatiotemporal and sensorial affordances of Spotify’s musical advice to the complex semiotic modalities of the “Popular Highlights” in the Kindle interface is discussed in Maja Bak Herrie’s ▶ Chap. 47, “The Recommended Experience: Engaging Networked Media Platforms with Intermediality.” Lastly, the chapter by Asun López-Varela Azcárate, ▶ Chap. 48, “Posthuman Intermedial Semiotics and Distributed Agency for Sustainable Development,” introduces intermediality in a discussion on the possible scaling of agentive capacities beyond the human and onto the material medium itself. The chapter summarizes main ideas in the neomaterialist mindmatter continuum of philosophical posthumanism and their relation to Charles S. Peirce to offer a theorization of semiotic process ontology that aligns intermedial studies with sustainable development. The handbook’s editors hold a strong belief that Lars Elleström would have deeply appreciated the final outcome of the project he initiated. We feel he would

10

J. Bruhn et al.

have cherished the project’s high-quality and diverse contributions, especially knowing many of the contributors personally. Elleström would probably have been even happier to see that a host of new, talented scholars were interested in contributing to the collective work. He would have been enthusiastic about the fact that there is now a thriving international community that, with both divergent and convergent views as well as recognizable terms, discusses what had come to be his grand academic task, to both organize, rethink, to participate in intermedial studies making it a blooming field within contemporary humanities.

References Arvidson, Jens, M. Askander, J. Bruhn, and H. Führer, eds. 2007. Changing borders: Contemporary positions in intermediality. Lund: Intermedia Studies Press. Bernhart, Walter, ed. 2006–2019. Studies in intermediality: Systematic and historical perspectives on intermedial, transmedial, and multimodal theory and practice. https://brill.com/display/serial/ SIIM Bruhn, J., and B. Schirrmacher, eds. 2021. Intermedial studies: An introduction to meaning across media. London: Routledge. Elleström, Lars, Massimo Fusillo, and Mattia Petricola. 2020. Everything is intermedial: A conversation with Lars Elleström. Between X.20 (Novembre/November 2020). https://doi.org/10. 13125/2039-6597/4443. Elliot, Kamilla. 2017. How do we talk about adaptation studies today? Literature/Film Quarterly 45 (2). https://lfq.salisbury.edu/_issues/first/how_do_we_talk_about_adaptation_studies_today. html Glaser, Stephanie A., ed. 2009. Media inter media: Essays in honor of Claus Clüver. Amsterdam/New York: Rodopi. Kittler, F.A., G. Winthrop-Young, and M. Wutz. 1999. Gramophone, film, typewriter. Stanford: Stanford University Press. McLuhan, M. 2001. Understanding media: The extensions of man. London: Routledge. Mitchell, W.J.T. 2005. There are no visual media. Journal of Visual Culture 4 (2): 257–266. Mitchell, W.J.T., and M.B.N. Hansen. 2010. Critical terms for media studies. Chicago: The University of Chicago Press. Rippl, G. 2015. Handbook of intermediality: Literature – Image – Sound – Music. Berlin: De Gruyter Mouton.

Part I Histories and Schools, Theories, and Methods of Intermedial Research

2

Intermediality: Introducing Terminology and Approaches in the Field Marina Grishakova

Contents Word, Image, and Beyond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Medium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality as a Material Practice and Artistic Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality in the Digital Age . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15 17 19 21 23 26 26

Abstract

The chapter discusses “intermediality” and related concepts. It offers a short overview of the history of intermediality, from ancient visual poetry to new media formats; lists various conceptions of “medium” dominating in the field; and explores a range of approaches to intermediality, from literary-oriented to digitally oriented. It zooms in on the understanding of intermediality as a material practice and artistic event and provides a short outline of the further fields of research on intermediality. Keywords

Image and word · Medium · Intermediality · Gesamtkunstwerk · Metarepresentation · Narrative environments

M. Grishakova (*) University of Tartu, Tartu, Estonia e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_12

13

14

M. Grishakova

“Intermediality” is a famously rich and multifaceted concept. It retains a connection with the avant-garde understanding of a work of art as an experiment, happening, or performance and with the “intermedia” as a fusion of two or more art forms in a new, innovative work. In the twentieth century, Wagner’s “total work of art” (Gesamtkunstwerk) has been reconceived as a variation of intermediality (Schröter 2012; Fusillo and Grishakova 2020) – a particularly intensive interplay and blending of material, perceptual, and aesthetic aspects of media. Intermediality embraces traditional topics of interart studies (literature and music, sculpture, painting, or photography), but it also operates within and between semiotic modes (image, sound, graphic signs); material media (paint, ink, electronic signals); and aesthetic media or systems of expression. It has always been and still is an incentive for experimentation with new materials and cultural forms. It works across media borders and disciplines, and, as such, it could be defined as a research perspective or a method of creation, reception, and study of artefacts and artistic practices, revealing the yet unexplored and undiscovered resources of media. In a broad sense, “intermediality” covers any possible relations between media. Transformation, transgression, and negotiation of media borders, as highlighted by various authors, distinguish intermediality from similar phenomena, such as “transmediality” (a spread of content or its features – narrative, stylistic, generic – across media and platforms); “multimediality”; and “multimodality” (a co-existence of multiple media or semiotic modes in a single work, genre, or message). For example, in his 2002 essay on the musicalization of fiction, Werner Wolf points out that music as a literature’s Other fosters “experimental transgressions of established aesthetic boundaries,” supports meta-reflexivity and self-referentiality, introduces sensual and emotional effects, intensifies (postmodernist) feeling of increasing complexity and fragmentation of existence, and responds to the (post-)modernist distrust of mimetic storytelling (2018, 47–60). In her insightful interpretation of Laurie Anderson’s short multimedia performance White Lily, Christina Ljungberg discusses intermediality as a transgression and negotiation of borders of media (Ljungberg 2010). White Lily, a restaging of a scene from Rainer Maria Fassbinder’s film Berlin Alexanderplatz based on Döblin’s 1929 novel, develops a dialogue with both the novel and the film, switching between various media, combining visual animation, electronic music, lyrics, and gesture. Ljungberg concludes that such instances of intermediality as White Lily are “radically performative, as we are confronted with hybrid forms that generate something new and unique,” “strongly self-reflexive, since they focus attention both on their own mode of production and on their own semiotic specificity,” and “a highly effective communication strategy, as they give readers, viewers and listeners access to different levels of meaning” (Ljungberg 2010, 83). In sum, as Stephanie Glaser observes, “the term ‘intermediality’ is today used in a variety of ways, one of which is to describe creative works that resist classification into the ‘pure’ or conventional categories of literature, music, or the visual arts” (Glaser 2009: 12). Intermediality works against any essentialist conceptions of media (Rippl 2015, 16): it conveys a sense of transformation and renewal of practices, perceptions, and meanings. Jan Baetens and Domingo Sánchez-Mesa point out that “intermediality [. . .] is not only the general term that defines the

2

Intermediality: Introducing Terminology and Approaches in the Field

15

relationships between autonomous media, it is also the term that identifies the internal plurality of each medium” (Baetens and Sánchez-Mesa 2015: 292). Intermediality as an inherent heterogeneity of the verbal medium (cf. Jørgen Bruhn’s “heteromediality” – “the internal mixed character” as “a condition of any text,” 2010) may include verbal imagery (metaphors, vivid descriptions); ekphrasis (situated in-between verbal and visual representation); or even percepts, memory images, and dreams evoked by verbal texts.

Word, Image, and Beyond The enigmatic nature of the relationship between word and image, inextricably entwined but distinct, complementary but competing (paragonal), attracted considerable attention in interart and intermedial studies – from Simonides’ definition of painting as “mute poetry” and poetry as “speaking picture” to W. J. T. Mitchell’s “rift between the seeable and the sayable” (Mitchell 1994, 12). The complementarity of visual and verbal medium manifests in the aspiration of the verbal towards the immediacy of sensory presence (in verbal imagery, metaphor, ekphrasis, and hieroglyphic writing) and the aspiration of the visual towards distinctness and articulation. Articulation makes elements of a visual work similar to combinations of verbal signs which the viewer is invited to “read” (in abstract painting or collage). The complementary-rival relations between word and image were thematized in visual or pattern poetry in Ancient Greece and in Europe during the Middle Ages and later, for instance, in Hellenistic Greek texts in the shape of urns, altars, axes, wings, and other forms from the third and fourth century BC that integrated pictorial and verbal patterns. Twentieth-century avant-gardes and postmodernisms perceived the opportunity of integrating visual and verbal elements in their works as an artistic challenge and merged various media to modify and disturb conventional uses of the space of the page or canvas, to foreground the materiality of the verbal, and to intensify the effects of porousness and metamorphosis of media. In avant-garde visual collages, combinations of drawings with matter pasted to the surface of the canvas (newspaper or magazine cuttings and texts, music scores, pieces of paper, cloth, and other materials) challenged viewers’ perceptual and cognitive habits and offered multiple ways of “reading” the artwork. The concept of ekphrasis, of historically varying scope and meaning, also illustrates tension and amalgamation between word and image. Initially defined as a rhetorical exercise, a vivid description, and further as “a verbal representation of a visual representation” (Heffernan 1993, 3), such as the description of the painting attributed to Pieter Brueghel in William Carlos William’s poem “Landscape with the Fall of Icarus” (1960), ekphrasis was extended to music, film, architecture, and other arts in the twentieth century. Understood either as a form of contest and rivalry or as an “encounter” of media (Kennedy and Meek 2019) and a hybrid of two modes of representation (Keefe 2011), ekphrasis conveys a sense of difference in similarity and is not reducible to a simple imitation of one medium by another. As Murray Krieger puts it, “ekphrastic aspiration. . . must come to terms with two impulses, two

16

M. Grishakova

opposed feelings about language: one is exhilarated by the notion of ekphrasis and one is exasperated by it.” This double feeling is evoked by the inability of language to freeze into a spatial form and, simultaneously, its pursuit “to recover the immediacy of a sightless vision built into our habit of perceptual desire since Plato” (Krieger 1992, 10). For Mitchell, the ekphrastic impulse propagates on the scale between ekphrastic indifference (the awareness of the impossibility of ekphrasis), ekphrastic hope, and ekphrastic fear (fear of obliterating the specificities of each medium; 1994, 156). From this perspective, the extension of the concept of ekphrasis to other media – film, photography, music, and architecture (see, e.g. Pethö 2011) – beyond the binary of the verbal and visual, and a resulting broader view of ekphrasis as a resonance, a creative reflex and transformation, rather than a transposition of an image from one medium to another, have been emancipating for ekphrastic studies. Recent perspectives include stylistic, cognitive, and embodied approaches to ekphrasis. For example, Siglind Bruhn discusses John McCabe’s orchestral piece The Chagall’s Windows (1974), a response to Chagall’s stainedglass windows in the Synagogue at the Hadassah-Hebrew University Medical Centre, Jerusalem, with reference to style, rhythm, and moods that the ekphrastic patterns evoke (Bruhn 2000). In his chapter on “cognitive ekphrasis” in Living Artefacts (2022), Terence Cave illustrates the embodied-mimetic foundation of ekphrasis, in its primary meaning of a “vivid description” conjuring up an image of a real thing, by referring to Guilemette Bolens’ kinesic analysis and her idea of a perceptual simulation of sensorimotor events by readers or viewers. Perceptual simulation triggers affective and imaginative inferences and makes complex meanings accessible through sensory experience (Bolens 2022). In this way, Cave suggests, ekphrasis serves as an epitome of the “as-if-ness of literary mimesis” at large and extends to the “macro-ekphrases” instantiating whole fictional worlds (Cave 2022, 88, 94). Finally, digital ekphrasis revives the ancient concept of energeia inherent in the ekphrastic “vivid description” by involving readers in physical interaction and evoking multisensory (visual, tactile, auditive) experiences. The concepts of iconotext and imagetext have been coined to describe tighter wordimage blends. Both concepts apply to relationships between text and image that, due to their complementary-antagonistic nature, have always been of central importance in artistic practices. Illustrated books, where images are part of the narrative and shape the textual whole, for instance, or images that include writing, are iconotexts or imagetexts, given their verbal and visual parts cannot be separated without distorting or impoverishing their meaning – the texts whose rhetoric “depends on the co-presence of words and images” (Wagner 1996, 16; see also Montadon 1990, Louvel 2011). The volume Icons – Texts – Iconotexts edited by Peter Wagner includes such examples of iconotexts as James Gillray’s and William Dent’s caricatures (Wagner 1996). Mitchell’s examples of imagetexts are the illustrated books of William Blake, aspiring for a magnificent synthesis of textual, sensory, and spiritual elements, and photo-essays, such as Agee’s Let Us Now Praise Famous Men (1941), which use tightly integrated image-text structures (Mitchell 1994). Mitchell’s “metapictures” (Mitchell 1994) or pictorial paradoxes call the very possibility of purely visual or verbal representation into question. The destabilizing

2

Intermediality: Introducing Terminology and Approaches in the Field

17

effect of a metapicture, such as Magritte’s famous painting This Is Not a Pipe (1929), with its “wildness” and resistance to interpretation demonstrates “the impossibility of a strict metalanguage,” capturing the essence of the metapicture, and reveals “the imbrication of visual and verbal experience” (Mitchell 1994, 83). I extended Mitchell’s idea of metapictorialism to verbal representations by introducing the concept of “intermedial metarepresentation,” branching into “metaverbal” and “metavisual” representations, that is to say, the features of verbal texts that evoke images and the features of visual representation that call for verbalization, whereas the verbal and visual counterparts as such appear to be discrepant or inadequate to fathom the meaning of the whole (Grishakova 2010, see also López-Varela and Khaski (2013) on intermedial representations as “self-reflexive nesting structures”). Such representations, similar to Leonardo da Vinci’s definition of painting as mute poetry and poetry as blind painting, convey a sense of incompleteness and inability of a single medium to capture the multimodality and complexity of natural perception. Many hybrid media – comics, animation, films, and videogames – feature combinations of image and word with various proportions of “telling” (words) and “showing” (images). In his classic Understanding Comics, Scott McCloud lists such types of combinations as picture-specific, word-specific, “duo-specific,” additive (with words amplifying or elaborating on the image or vice versa), parallel, montage (with words integrated into an image), and interdependent (1994, 152–161). The spread of comics and graphic novels was given a boost by the development of print cultures and the massive reproduction of images in the nineteenth century. By adding movement and sound to images and allowing, in Bazin’s words, for the first time “the image of things” to coincide “with the image of their duration” (1960, 8), cinema allowed also for meaningful conjunctions and disjunctions, shifts, and discrepancies between images and sounds of various types and origins (on-screen and off-screen, diegetic and extra-diegetic, synchronic and asynchronic), producing striking perceptual, emotional, and cognitive effects skilfully exploited by Welles, Godard, Antonioni, Hitchcock, and many other celebrated filmmakers. However, whereas “the dominant position of the visual at the expense of the verbal is definitely stronger” in film (Baetens 2012, 98), comics develop new formats, such as the underground comix and the graphic novel, where text is more sophisticated and complex than it is supposed to be in a captioned cartoon – an original basic form of comics. Further, the blending of comics with television, film, animation, and videogames and the rise of digital comics bring to the fore visual and interactive modalities of the medium.

Medium Before defining and discussing the concept of intermediality, it is necessary to pinpoint what is meant by “medium.” Word and image are minor or, to use Lars Elleström’s term, “basic” media. As Marie-Laure Ryan pointed out, the meanings and scope of the concept of “medium” vary across fields and professions:

18

M. Grishakova Ask a sociologist or cultural critic to enumerate media, and he will answer: TV, radio, cinema, the Internet. An art critic may list music, painting, sculpture, literature, drama, the opera, photography, architecture. A philosopher of the phenomenological school would divide media into visual, auditory, verbal, and perhaps gustatory and olfactory (are cuisine and perfume media?). An artist’s list would begin with clay, bronze, oil, watercolor, fabrics, and it may end with exotic items used in so-called mixed-media works, such as grasses, feathers, and beer can tabs. An information theorist or historian of writing will think of sound waves, papyrus scrolls, codex books, and silicon chips. (Ryan 2005, 14).

Ryan succinctly summarizes three types of meanings of “medium” adopted by various authors (2005, 14–16): • Materials and technologies (clay, stone, the human body; technologies such as writing, print, and digital encoding) • Semiotic (or, rather, perceptual-semiotic) media: verbal, visual, auditory, olfactory, gustatory, tactile, or – based on the dominant type of semiotic signs – iconic (classical painting), indexical (music, abstract painting), symbolic (verbal arts) • Cultural media (literature, film, TV, radio, etc.) as both practices and institutions I would add a fourth type to Ryan’s schematic: “media” as aesthetic expressive systems that involve sets of specific features, conventions, and forms. Discussions and studies of intermediality may be based on any of these four meanings. Apparently, distinctions between the four meanings of “medium” are not clear-cut: these meanings can also be viewed as four levels of mediation or medium-ness present in any intermedial works. Rather than distinguishing between the types of media, Lars Elleström (2010) refers to the “modalities of media.” He distinguishes between the material, sensorial, spatiotemporal, and semiotic modalities, ranging on a scale between the tangible, perceptual, and conceptual features of any medium. Elleström defines the material modality as “the latent corporeal interface of the medium” (such as images on a flat surface and sound waves in film); the sensory modality as “the physical and mental acts of perceiving the present interface of the medium through sense faculties”; the spatiotemporal modality as “the structuring of the sensorial perception of sensedata” in perception and conceptualization; and the semiotic modality as “the creation of meaning in the spatiotemporally conceived medium” by the way of thinking and interpretation (17–22; see also Bruhn and Schirrmacher (2022) for the application of Elleström’s conception to various media). Elleström’s approach recruits the semiotic hierarchy of embedded signs, from icon and index to symbol, and thereby establishes a continuity between the natural and cultural modalities of media. In this way, Elleström concludes, Mitchell’s idea of “mixed media” can be revised: the “mixed” nature of each medium is not a result of shared modalities of media, but of shifting modes of the modalities, understood broadly, in Kress and Leeuwen, as semiotic resources and vehicles of meaning in various communicative contexts: texts, images, sounds, gestures, etc. The publication of Raymond Williams’ influential book Television: Technology and Cultural Form (1974) opened new vistas in the study of media as cultural forms,

2

Intermediality: Introducing Terminology and Approaches in the Field

19

emerging at the intersection of and in response to cultural, social, and economic practices and needs. From this perspective, the spread of television was prompted by the rise of the suburban population moving between private homes and public workplaces. Television combines centralized entertainment with domestic consumption, as a form of mediated sociality, and performs the functions of social orientation and cohesion in the fragmented and changing world. The forces or practices that contributed to the development of television are, then, technology (apparatus, infrastructure); consumers or public; producers (editors, distributors, funders, etc.); and uses or kinds of mediated experience. In his Visible Fictions (1982, with many reprints) John Ellis builds on Williams’ concepts of cultural form and televisual “flow” and explores television and cinema and the uses of the cinematic medium (film), which is split between these two cultural forms: the film watched in cinema is a public event, whereas the film watched on TV is a domestic event, with a limited number of spectators. In the cinema, it is projected on a big screen and requires hyper-attention. In the second case, it is a less attention-demanding event and often alternates with other domestic doings or involves multitasking. Although Williams’ and Ellis’ observations do not always extend to contemporary technologies and their uses, such as online film watching, Williams’ perspective on medium as a cultural form is still influential. It resonates in Siegfried J. Schmidt’s conception of media systems and four aspects of media as communication (semiotic) tools, technologies, institutional systems, and media products (2008). Unlike Williams’ and Schmidt’s broad view of media as cultural or social forms, neat academic taxonomies are not always able to grasp the complex life of media and to identify the distinctions and specificities of each medium. Nevertheless, exploring historically changeable features, modalities, and configurations of media through the lens of intermediality understood as a cultural practice and an analytical tool allows for flexibility in approach without losing the sharpness of detail.

Intermediality In the 1980s–1990s, academic research on intermediality was largely informed by literary theories and concepts. Aage Hansen-Löve, who introduced the German term Intermedialität in (1983), Claus Clüver, and German scholars (Jürgen Müller, Werner Wolf, and Irina Rajewsky) considered intermediality as an extension or a new version of intertextuality. Werner Wolf argued in favour of a literature-centred study of contacts between literature and other media and the ways in how other media shape fiction through various historical periods: landscape painting in the late eighteenth to the early nineteenth century or photography and film in subsequent media history (pictorialization, musicalization, or filmization of literature, Wolf 2018, 136, see also Lewis 2020). In her 2005 chapter, Rajewsky proposed to distinguish the concept of intermediality, as used by herself and other German scholars, from the American term “intermedia” {referring to Dick Higgins’ [1984 (1966)] coinage}. Whereas the former was employed as an analytical tool, the latter originates in experimental

20

M. Grishakova

artistic practices of the twentieth century. However, the distinction between intermedia and intermediality largely conceived as amalgamations of various media or artistic forms, even avant la lettre, before the respective concepts emerged, was not, in fact, so neat. In the Italian theatre of the Renaissance, a musical or theatrical interlude shown between the parts of the main play was called an “intermedio.” The term “intermedium” was introduced by the English poet Samuel Taylor Coleridge in his lecture on Edmund Spencer (1812) to refer to the works that fall between traditional genres or media. Coleridge calls narrative allegory in literature and visual arts an “intermedium” between “person and personification” (Coleridge 1936, 33), that is between an image of reality and a symbolic quality attributed to it. More recently, Fluxus artist Dick Higgins applied both “intermedium” and “intermedia” to experimental artistic practices. For Higgins (1984 [1966]), “intermedia” was a distinctive feature of Fluxus works. Fluxus was an international community of experimental artists, including, among others, John Cage, Joseph Beuys, Alison Knowles, and Yoko Ono, who propagated an understanding of artistic practice as an experiment, happening, or performance: in other words, an exciting encounter of artistic forms and media and their fusion in a new, innovative work. Similarly, for McLuhan, “the moment of the meeting of media is a moment of freedom and release from ordinary trance and numbness imposed by them on our senses” (McLuhan 2003 [1964]: 81). Higgins distinguished “mixed media” such as opera or song, which demonstrate complementary relations, from “intermedia,” where the media are fused and inextricably entwined, for example, visual poetry as both a graphic image and a printed text. “Intermedia” also embraces any kind of conceptual art (e.g. John Cage’s music) and writing (Roland Barthes’ essays) that fuse art with philosophy or the social sciences (Higgins 1984 [1966]: 26). “Intermedium,” as Higgins puts it, is the “uncharted land that lies between” different media: intermedial works are “not governed by rules; each work determines its own medium and form according to its needs” (Higgins 1984 [1966]: 22). Higgins also defines intermediality in terms of its experiential and mental effect, as a holistic mental experience. The idea that media, working together towards a unique artistic event, produce a specific cognitive-perceptual effect and involve the viewer or user in the production of this effect through somatic and perceptual circuits was prominent in the theoretical and philosophical conceptualizations of new forms of intermediality, from McLuhan’s view of media as extensions of human senses (1964) to Youngblood’s concept of “expanded cinema” as extended human consciousness (1970). The term “intermediality” engendered a plethora of related concepts and typologies. Among the literary-informed frameworks, Rajewsky’s (2005; see also Wolf 2018) three types of intermediality proved to be influential: • Medial transposition (e.g. in film adaptation), where intermediality is an aspect and principle of text production: the intermedial quality “has to do with the way in which a media product comes into being, i.e., with the transformation of a given media product (a text, a film, etc.) or of its substratum into another medium” (Rajewsky 2005, 51)

2

Intermediality: Introducing Terminology and Approaches in the Field

21

• Media combination (film, opera, comics, performance, etc.), where intermediality is the principle of a media product’s or text’s constitution • Transmedial reference (e.g. the evocation of film’s structural features or techniques in literature or the evocation of visual features in literature) The transmedial or intermedial reference can be explicit (for instance, a discussion of a film in a novel) or implicit, which includes structural similarity of imitation of one medium by another (Wolf 2005). Schröter’s (2012) four types of intermediality are more inclusive as compared with literary-oriented concepts. They manifest in artistic practices and critical talk around these practices, rather than only in scholarly writing. • Synthetic intermediality involves the fusion of several media into a new medium and is seen as the triumph over monomedia and as a way of social liberation. It is a Gesamtkunstwerk-type intermedia, with a social-utopian potential that involves a merger of art and life in a single liminal experience, as described in Yalkut’s (Fluxus artist’s) Understanding Intermedia (1967). • Formal (transmedial) intermediality is based on structural homologies or transmedial features, for instance, forms and structures shared by painting, photography, and cinema or narrative features found across media. Media-specific features are difficult to pinpoint, but hybrid categories such as “media narrating cinematically or literarily” may help identify further, more sophisticated cross-media distinctions and similarities. • Transformational intermediality as a representation of one medium by another overlaps with Grusin and Bolter’s (1999) concept of “remediation” and brings to the fore media-specific differences. • Ontological intermediality is viewed as a hypothetical synthetic condition (archor ontomediality) preceding separate media and undermining the idea of clearly separated “monomedia”: to see a specific character of any medium the latter should be juxtaposed to a whole range of other media. In this way, ontomediality serves as a prerequisite for the existence of specific media. At some points, Schröter’s schematic overlaps with Rajewsky’s and other types of intermediality, which is no surprise given intermediality is generally understood by various researchers as a relationship or interaction between different media. Nevertheless, Schröter’s taxonomy has the advantage of embracing the social and cultural functions of media.

Intermediality as a Material Practice and Artistic Event The history of intermedia includes bold experiments that have changed conventional views of media, cultural practices, and book cultures. In my chapter in the Routledge Companion to Literary Media (forthcoming), I defined intermediality as a material practice and artistic event and, in Higgins’ vein, as a resulting fusion of two or more

22

M. Grishakova

media in a new, innovative work or cultural form. Intermediality reveals and intensifies material, semiotic, and expressive (aesthetic) features of these media, fosters their proliferation and instantiation in various cultural forms, or inscription of materiality in the symbolic orders of language and culture (see Tötösy de Zepetnek et al. 2011). This approach acknowledges that new cultural forms instantiate and integrate material and semiotic features of earlier media into the new wholes: new media integrate, refashion, and repurpose old media (see, e.g. Bolter and Grusin 1999). In this way, for example, cinema appropriates features of photography, theatre performance, painting, and other media, and the computer absorbs features of the cinematic medium. Katherine Hayles coins the concept of “comparative textual media” (Hayles and Pressman 2013), referring to the fact that the digital format made the materiality of print books visible in hindsight. Intermediality as a material practice and artistic event, evoking rich and transformative perceptual and cognitive experiences, is not historically new, nor is it typical exclusively of twentieth-century avant-garde cultures. Richly illustrated and ornamented medieval and Renaissance books can be seen as both books and artefacts, collages of jewels, metalwork, calligraphy, and painting. Botler and Grusin refer to such phenomena as examples of “hypermediacy” – a co-existence of multiple heterogeneous media representations competing for the viewer’s attention (1999, 327–28). The Baroque aesthetics of exuberance and abundance in church and palace interiors, theatre, and poetry also involved fusions of the arts and media. More recently, intermediality has been associated with Eisenstein’s “montage of attractions,” with Kandinsky’s Bühnenkompositionen (stage compositions), where different arts affect each other (Kattenbelt 2008), and with Wagner’s concept of Gesamtkunstwerk. The concept of Gesamtkunstwerk or a total work of art – coined by Richard Wagner as a means of reshaping musical theatre and of recovering the synthesis of the arts at the core of Greek tragedy – played a prominent role in the practices of Symbolism and Aestheticism. Following Adorno’s critique of Wagner’s total work of art as a commodity, Matthew Smith considers the twentieth-century syntheses of the arts and entertainment (e.g. in attraction parks, such as Disneyland, or the advertisement industry) as a variety of the Wagnerian Gesamtkunstwerk transformed into “total entertainment” (see Smith 2007; Fusillo and Grishakova 2020). Intermediality as both a concept and a practical principle culminates in neo-avantgarde and postmodernist experimental cultures. Whereas the avant-garde invested in an authored artistic event or performance, postmodernism downplays the role of the author and celebrates the event of hybridization and intermediation: “Since the seventies, the very idea of an avant-garde, or of individual genius, has fallen under suspicion. Combative, collective movements of innovation have become steadily fewer, and the badge of a novel, self-conscious ‘ism’ ever rarer. For the universe of the postmodern is not one of delimitation, but intermixture – celebrating the crossover, the hybrid, the pot-pourri” (Anderson 1998: 93). Experimentation with page space and text segmentation in postmodernist fiction, including the use of blank spaces, short text segments, boldface or upper-case type, and coloured pages, was disruptive of the usual perception of textuality, foregrounded the materiality of the

2

Intermediality: Introducing Terminology and Approaches in the Field

23

book, and fostered crossovers and metamorphoses of the verbal into the visual and vice versa. As McHale notes (1987, 182), narrative discontinuity in postmodernist texts has a physical “objective correlative” in the organization of page space, for instance, the spacing of the text or “extremely short chapters or short paragraphs separated by wide bands of white space” in Brautigan’s, Barthelme’s, and Vonnegut’s books. Neo-avant-garde and postmodernist experimentation is often considered as the anticipation of the potentialities of digital media. Such phenomena as visual or pattern poetry, collage, cine-novel, and comics appear in new digital guises, revealing and developing the opportunities offered by new media. Visual print poetry – with its ambiguous status as both image and text – functioned as a resource for digital poetry, particularly through its ability to support multilinear or multi-sequential reading, to disrupt smooth and transparent signification, and to probe the limits of media. Digital provides additional ways of expression: time, movement, and interaction. The movement, shift, and displacement thematized and illustrated by visual print poetry are animated in digital poetry and highlight the process of its production. Techniques of remix and mashup enjoy new possibilities made available by digital cut-copy-and-paste technologies. As modes of appropriation and re-shaping of pre-existing materials (verbal, visual, and audio), they had their predecessors in print poetry, fiction, and essayistic writing, such as Walter Benjamin’s collection The Arcades Project (1927–40). The Oulipo experiments with constrained writing or the tradition of dictionary novels, combining the “database” and “narrative” principles (Grishakova 2018), in turn, anticipated digital hypertexts and other forms of electronic literature. Novelty has been introduced through the computational aesthetics and participatory affordances of new media that integrate participatory-interactive verbal works in everyday vernacular practices and destabilize the traditional roles of the author and reader.

Intermediality in the Digital Age Digital culture has been seen as a challenge and threat to traditional art practices, formats, and to the very existence of an autonomous work of art. As Kiene Brillenburg Wurth puts it, a literary scholar faces the alternative: either to defend the book page “as the last trace of a material humanist tradition against the incursions of a network of digital distraction, or to promote digitization as liberation from the material constraints of paper and print” (2012, 2). Circulation of media content across different media systems, integration of media products into franchises, interchangeable roles of readers and authors in digital environments, and volatility of the digital signs seemed to downplay the old conceptions of authorship, the autonomy of a work of art, and devalue the distinctions between the original text and a derived text or a copy. Arguably, Jenkins’ concept of the convergence of media through digital coding and computer processing, in contrast to their earlier differentiation and individuation, makes studies of separate media and their interactions obsolete. Jenkins is interested primarily in the spread (transmediation) of narratives and images across media and in the franchising strategies of media products in

24

M. Grishakova

convergence culture, within the flow of content across various media platforms (2006). The expansion of franchise transmedial snowball narrative worlds (Baetens and Sánchez-Mesa 2015) can be associated with medium insensitivity (Baetens 2011). A massive ongoing process of the technological production of images, their remediation, and adaptation in popular culture – a global transmedial transfer – is guided by the logic of marketing and the user’s instant gratification. The efficiency of the transfer depends on the mimetic recognizability of brand stories and popular characters, rather than on a sophisticated interplay of media. Alternatively, a media-sensitive adaptation from one medium to another can be seen as a complex event, retaining a dialogical relationship with source texts and images. Gaudreault and Marion (2005) highlight the fact that each medium has its own specific configurations and forms of resistance, or, otherwise, its own inertia and force of gravity, which serve as both the limitations and productive challenges in the adaptation process. From this perspective, adaptation is something more than a simple transfer of content from one medium to another; rather, it is a new communicative event (Jost 2005) that involves an interplay and new configuration of materials and media and their fusion into a new artefact (see Grishakova 2014). According to Ryan (2005; Grishakova and Ryan 2010; Ryan and Thon 2014), different media afford for different forms of narrativity (that is to say, transmediality, or a transfer of content and its features and properties across media, always involves intermediality). Ryan (2005) lists media characteristics that affect the experience of narrative (spatio-temporal, kinetic, communication channels, and sensory channels), yet she also points out that specific forms and types of narrativity in various media are contingent on the cultural uses of the medium. For example, the alternation of broadcasting with advertisement as well as the adaptation of TV formats and programs to user’s daily routines and producers’ economic strategies prompted the emergence of the segmented structure of broadcasting, where episodes of the serials last between half an hour to an hour and films around two hours. Many authors of the volume Intermediality and Storytelling explore how such uses and contexts impact narration in different media. The necessity to maintain coherence and to hold viewer’s attention shapes TV formats, for example, prime time serials that use redundancies, repetitions, recaps, diegetic retelling, voiceover narration, flashbacks, and replays to keep even erratic viewers on track and to trigger their memory mechanisms (Mittel 2010). In musical films the story is told through dialogue, music, singing, dancing, and acting, without priority given to any single medium (Hansen 2010). The management of blank spaces in photonovels and gutters in comics defines narrative features of these genres (in Baetens and Bleyen’s, McHale’s, and Kuskin’s chapters in the same volume). Gameplay, that is, the coordination of the player’s actions with what happens in the game world as a result of these actions, affects the narrativity of videogames (Ciccoricco 2010, see also Ciccoricco 2015). In this way, forms of narrativity in various media are contingent on the interplay, uses, affordances, and limitations of these media: transmedial transposition and transfer of a story across media never leave the story unchanged.

2

Intermediality: Introducing Terminology and Approaches in the Field

25

By opposing the idea of homogenization and convergence of media, other researchers pointed out that new media formats recruit and repurpose (or “remediate”) the formats and features of old media and often rely on traditional forms of aesthetic expression. McLuhan saw the medial turn as a new stage in the evolutionary process of development and growing complexity of media. In reality, it is a retroactive evolution where earlier and later stages reciprocally affect one another, while producing stochastic effects. Remediation and adaptation shed new light on the old media and arts and activate their latent potentialities. These processes convey a cultural memory of old media, genres, and forms (for instance, weblogs and blogs as remediated diaries, web pages reminiscent of print magazine pages, etc.) or even make these old formats more visible and prominent in new media ecology. For example, a “techno-text” reveals that “the physical form of the literary artifact always affects what the words and other semiotic components mean” (Hayles 2002, 25). Rather than seeing techno-text as “more developed” as regards to printed or written text, Hayles considers it as a means to prompting us to reconceive and reinvent print cultures in hindsight and thereby casting a fresh light on the history of the book. Hayles critically recasts Bolter and Grusin’s “remediation” into “intermediation” – the concept which refers to the knowledge and memory of old media “transformatively incorporated and animated in new media” (Brillenburg 2012, 14) and resisting “media numbness.” The computational dimension of literary writing prompts readers to reflect on the possibilities and conditions of literary production and the ways the digital medium affects them. Finally, in the digital age, “intermediality” is reconceived as the process and product of digitization. The digitization perspective is critical in countering Jenkins’ conception of media convergence (2006), as it demonstrates that media environments foster variation, transformation, and proliferation of media features and formats along various trajectories. Rather than constituting a convergence, “digital media [is a] pattern made up of a set of interlaced devices, markets, aesthetics and practices woven together—a textile metaphor underlining the fact that today’s digital media are an interwoven environment where the original elements are still recognizable but constitute now a new and distinctive object” (Balbi and Magaudda 2012, 154). In offering “a less linear explanation of what has been happening to digital media in recent years” (Balbi and Magaudda 2012, 156), intermediality thus helps to identify trajectories of proliferation and hybridization of various media features. It appears in condensed but also fuzzier forms in digital environments. The environment binding together tools, formats, and platforms with human interaction in co-constructing these environments is a crucial factor in the development of new forms of intermediality. In Narrative Complexity: Cognition, Embodiment, Evolution (2019), Maria Poulaki and I adopted Gubrium and Holstein’s concept of “narrative environments” to refer to the material spaces where storytelling takes place. Different narrative environments foster different types of storytelling through developing various participation frameworks and formats, communication channels, and settings that mediate storytelling and determine participants’ roles. The settings, in which the boundless stories develop, in turn, shape – from the bottom up – their narrative environments. Finally, a nostalgic return to analogue formats or to paper books that adopt digital techniques [such as Jonathan Safran Foer’s Extremely Loud and Incredibly Close

26

M. Grishakova

(2005), Mark Danielewski’s House of Leaves (2000), Graham Rawle’s Woman’s World (2005), and S. (2013) by J.J. Abrams and Doug Dorst], combining digital and analogue formats in various art forms and performances, eco-art, Ambient Literature and Walking Art projects are signs of intermediality permanently reinventing itself, rather than disappearing or being levelled and homogenized in new media. As Sarah Wyman notes (2011), the participatory dynamics of new media is, in a way, reminiscent of human navigation in the natural world: “a look at the pixeled image in several examples of contemporary art — geometric painting, concrete poetry, and the mosaic — provides a counterpart to our own experience of reading the broader reality around us. The two dynamics of fragmenting form and of building parts (or pixels) into aesthetic wholes both correspond to the act of visual perception itself.” But the apparent naturalness of art always involves an artifice: framing and focusing our attention, conjuring illusionary imaginative dimensions of perception and cognition, and including a perceiver into a self-reflexive process of aesthetic play.

Conclusions In the section on further fields of intermedial research in her introduction to (2015) De Gruyter’s Handbook of Intermediality, Gabriele Rippl envisaged the opportunities of integrating intermedial research with postcolonial, transcultural, and cosmopolitan perspectives in the study of literature and research on visual cultures, multimodality, and transmediality in the study of culture. Today, we extend the intermedial inquiry into media ecology as part of natural ecology, with reference to the issues of sustainability, suffering, environmental concerns (Bruhn 2021), selforganization and networking, advantages and dangers of technologies, and the emancipatory functions of art and aesthetics in ordinary life. Links between intermediality and natural perception and navigation in the world are promising in view of contemporary naturalistic and embodied approaches in the study of art and culture and in artistic practices, development of eco- and bio-art beyond technoutopias, and considering the need for a more attuned and perceptive engagement with environments and finding a balance between personal circumstances and functioning of states and societies, between private and public spheres. Multisensory environments, where we navigate in our everyday life, are complex, messy, and difficult for the neat and clear-cut categories to capture. Approaching intermediality as a practice and event, as part of human everyday experience, may help discover its new artistic, sociocultural, and ecological aspects.

References Anderson, Perry. 1998. The Origins of Postmodernity. London-New York: Verso. Baetens, Jan. 2011. World literature and popular literature: Toward a worldless literature? In The Routlegde Companion to World Literature, ed. Theo D’haen, David Damrosh, and Djelal Kadir, 336–344. Routledge: London and New York.

2

Intermediality: Introducing Terminology and Approaches in the Field

27

———. 2012. Words and images in the contemporary American novel. In ReBlurring the Boundaries, ed. Bernd Herzogenrath, 92–110. Hanover: Dartmouth College Press. Baetens, Jan, and Domingo Sánchez-Mesa. 2015. Literature in the expanded field: Intermediality at the crossroads of literary theory and comparative literature. Interfaces: Image – Text – Language 36: 289–304. https://doi.org/10.4000/interfaces.245. Balbi, Gabriele, and Paolo Magaudda. 2012. A History of Digital Media: An Intermedia and Global Perspective. New York and London: Routledge. Bazin, André. 1960. The ontology of the photographic image. Translated by Hugh Gray. Film Quarterly 13 (4): 4–9. Bolens, Guillemette. 2022. Kinesic intelligence, medieval illuminated psalters, and the poetics of the psalms. Studia Neophilologica: 1–24. https://doi.org/10.1080/00393274.2022.2051733. Bolter, Jay David, and Richard Grusin. 1999. Remediation: Understanding New Media. Cambridge: MIT Press. Brillenburg Wurth, Kiene, ed. 2012. Between Page and Screen: Rethinking Literature through Cinema and Cyberspace. New York: Fordham UP. Bruhn, Jørgen. 2010. Heteromediality. In Media Borders, Multimodality and Intermediality, ed. Lars Ellestrom, 225–236. Basingstoke and New York: Palgrave Macmillan. ———. 2021. Towards an intermedial ecocriticism. In Beyond Media Borders, vol. 2. Intermedial Relations among Multimodal Media, ed. Lars Elleström, 117–148. Basingstoke and New York: Palgrave Macmillan. Bruhn, Jørgen, and Beate Schirrmacher, eds. 2022. Intermedial Studies: An Introduction to Meaning Across Media. London and New York: Routledge. Bruhn, Siglind. 2000. Musical Ekphrasis: Composers responding to poetry and painting. Pendragon Press. Cave, Terence. 2022. Live Artefacts. Literature in a Cognitive Environment. Oxford: Oxford UP. Ciccoricco, David. 2010. Games of interpretation and a graphophiliac god of war. In Intermediality and Storytelling, ed. Marie-Laure Ryan and Marina Grishakova, 232–257. Berlin: De Gruyter. ———. 2015. Refiguring Minds in Narrative Media. Lincoln and London: University of Nebraska Press. Coleridge, Samuel T. 1936. In Miscellaneous Criticism, ed. T.M. Raysor. Cambridge: Harvard University Press. Elleström, Lars. 2010. The modalities of media: A model for understanding intermedial relations. In Media Borders, Multimodality and Intermediality, ed. Lars Elleström, 11–48. Basingstoke and New York: Palgrave Macmillan. Ellis, John. 1982. Visible Fictions: Cinema – Television – Video. London: Routledge. Fusillo, Massimo, and Marina Grishakova, eds. 2020. Gesamtkunstwerk as a Synergy of the Arts. Bruxelles-Bern-Berlin-New York-Oxford-Wien: Peter Lang. Gaudreault, André, and Philippe Marion. 2005. Transécriture and narrative mediatics. The stakes of intermediality. In A Companion to Literature and Film, ed. Robert Stam and Alessandra Raengo, 58–69. Wiley-Blackwell: Hoboken, NJ. Glaser, Stephanie A. 2009. Dynamics of intermedial inquiry. In Media inter Media. Essays in Honor of Claus Clüver, ed. Stephanie Glaser, 11–36. Amsterdam-New York: Rodopi. Grishakova, Marina. 2010. Intermedial metarepresentations. In Intermediality and Storytelling, ed. M. Grishakova and M.-L. Ryan. Berlin-New York: De Gruyter. ———. 2014. “Unruly fictions”: literature, mimesis, and media. In Intersections, Interferences, Interdisciplines: Literature with Other Arts, ed. Gerald Gillespie and Haun Saussy, 213–227. Bern-Berlin-Brussels-Frankfurt am Main-New York-Oxford-Wien: Peter Lang. (New Comparative Poetics 30). ———. 2018. The abecedarian imagination: On dictionary structures in literature. In Literatur in Wissenschaft und Unterricht, 1-2. Special issue “Lists and Narrative,” ed. by E. Neven, B. Vervaeck, and L. Herman, 93 103

28

M. Grishakova

———. forthcoming. Intermediality as a material practice and artistic event. In Routledge Companion to Literary Media, ed. Bronwen Thomas, Julia Round, and Astrid Ensslin. London-New York: Routledge. Grishakova, Marina, and Maria Poulaki, eds. 2019. Narrative Complexity: Cognition, Embodiment, Evolution. Lincoln: University of Nebraska Press. Grishakova, Marina, and Marie-Laure Ryan, eds. 2010. Intermediality and Storytelling. Berlin-New York: De Gruyter. Hansen, Per Krogh. 2010. All talking! All singing! All dancing! Prolegomena: On film musicals and narrative. In Intermediality and Storytelling, ed. Marie-Laure Ryan and Marina Grishakova, 147–164. Berlin: De Gruyter. Hansen-Löve, Aage. 1983. Intermedialität und Intertextualität: Probleme der Korrelation in Wortund Bildkunst: Am Beispiel der russischen Moderne. In Wiener Slawistischer Almanach, vol. 11, ed. Dialog der Texte, 291–360. Wolf Scmid and W.-D. Stempel. Hayles, N. Katherine. 2002. Writing Machines. Cambridge, Massachusetts: MIT Press. Hayles, N. Katherine, and J. Pressman. 2013. Comparative Textual Media: Transforming the Humanities in the Postprint Era. Minneapolis: Minnesota University Press. Heffernan, James A.W. 1993. Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery. Chicago: University of Chicago Press. Higgins, Dick. 1984. Horizons, the Poetics and Theory of the Intermedia. Carbondale, IL: Southern Illinois University Press. Jenkins, Henry. 2006. Convergence Culture: Where Old and New Media Collide. New York: New York University Press. Jost, François. 2005. The look: From film to novel. An essay in comparative narratology. In A Companion to Literature and Film, ed. Robert Stam and Alessandra Raengo, 71–91. Hoboken, NJ: Wiley-Blackwell. Kattenbelt, Chiel. 2008. Intermediality in theatre and performance: Definitions, perceptions and medial relationships. Cultura, Lenguaje y Representación/Culture, Language and Representation VI: 19–29. Keefe, Anne. 2011. The ecstatic embrace of verbal and visual: Twenty-first century lyric beyond the ekphrastic paragone. Word & Image 27 (2): 135–147. https://doi.org/10.1080/02666286.2010. 516891. Kennedy, David, and Richard Meek, eds. 2019. Ekphrastic Encounters: New Inter-disciplinary Essays on Literature and the Visual Arts. Manchester: Manchester University Press. Krieger, Murray. 1992. Ekphrasis: The Illusion of the Natural Sign. Baltimore and London: The Johns Hopkins UP. Lewis, Cara L. 2020. Dynamic Form: How Intermediality Made Modernism. Ithaca: Cornell University Press. Ljungberg, Christina. 2010. Intermedial strategies in multimedia art. In Media Borders, Multimodality and Intermediality, ed. Lars Ellestrom, 81–95. Basingstoke and New York: Palgrave Macmillan. López-Varela Azcárate, A., and C. Khaski Graglia. 2013. Intermedial metarepresentation in serial novels: Dickens’ The Pickwick Papers. CLCWeb: Comparative Literature and Culture 15: 7. https://doi.org/10.7771/1481-4374.2390s. Louvel, Liliane. 2011. Poetics of the Iconotext. Farnham-Burlington, VT: Ashgate Publishing Ltd. McCloud, Scott. 1994. Understanding Comics: The Invisible Art. New York: Harper Perennial. McHale, Brian. 1987. Postmodernist Fiction. London-New York: Routledge. McLuhan, Marshall. [1964] 2003. Understanding Media: The Extensions of Man. Berkeley: Ginko Press. Mitchell, William John Thomas. 1994. Picture Theory. Chicago: Chicago University Press. Mittel, Jason. 2010. Previously On: Prime Time Serials and the Mechanics of Memory. In Intermediality and Storytelling, ed. Marina Grishakova and Marie-Laure Ryan, 78–98. Berlin: De Gruyter. Montadon, Alain. 1990. Iconotextes. Paris: C.R.C.D-OPHRYS.

2

Intermediality: Introducing Terminology and Approaches in the Field

29

Pethö, Ágnes. 2011. Cinema and Intermediality: The Passion for the In-Between. Cambridge Scholars. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités/Intermediality 6: 43–64. Ryan, Marie-Laure. 2005. Foundations of transmedial narratology. In Narratology Beyond Literary Criticism: Mediality, Disciplinarity, ed. Jan-Christoph Meister, 1–24. Berlin-New York: De Gruyter. Ryan, Marie-Laure, and Jan-Noël Thon, eds. 2014. Storyworlds Across Media: Toward a MediaConscious Narratology, 1–21. Lincoln: University of Nebraska Press. Rippl, Gabriele. 2015. Introduction. In Handbook of Intermediality: Literature – Image – Sound – Music, ed. Gabriele Rippl, 1–31. Berlin-Boston: De Gruyter. Schmidt, Siegfried. 2008. Medienkulturwissenschaft. In Metzler Lexikon Literatur- und Kulturtheorie, ed. Ansgar Nünning, 472–474. Stuttgart: Metzler. Schröter, Jens. 2012. Four models of intermediality. In ReBlurring the Boundaries, ed. B. Herzogenrath, 15–36. Hanover: Dartmouth College Press. Smith, Matthew Wilson. 2007. The Total Work of Art: From Bayreuth to Cyberspace. London/New York: Routledge. Tötösy de Zepetnek, Steven, Asunción López-Varela, Haun Saussy, and Jan Mieszkowski. 2011. Introduction to new perspectives on material culture and intermedial practice. CLCWeb: Comparative Literature and Culture 13: 3. https://doi.org/10.7771/1481-4374.1783. Wagner, Peter, ed. 1996. Icons – Texts – Iconotexts. Essays on Ekphrasis and Intermediality. BerlinNew York: De Gruyter. Williams, Raymond. 1974. Television: Technology and Cultural Form. London: Fontana. Wolf, Werner. 2005. Intermediality. In Routledge Encyclopedia of Narrative Theory, ed. David Herman, Manfred Jahn, and Marie-Laure Ryan, 252–256. London: Routledge. ———. 2018. In Selected Essays on Intermedialty (1992-2014), ed. Walter Bernhart. Leiden/ Boston: Brill Rodopi. Wyman, Sarah. 2011. Plotting the pixel in remediated word and image. CLCWeb: Comparative Literature and Culture 13(3). Thematic Issue: New Perspectives on Material Culture and Intermedial Practice. ed. Steven Tötösy de Zepetnek, Asunción López-Varela, Haun Saussy, and Jan Mieszkowski, https://doi.org/10.7771/1481-4374.1792 Youngblood, Gene. 1970. The Expanded Cinema. New York: Dutton & Co.

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver Solange Ribeiro de Oliveira and Thai__s Flores Nogueira Diniz

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Development of Clüver’s Œuvre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part I: The History and Theory of the Studies of Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part II: The Concepts of Ekphrasis and Intermedial Transposition and Related Texts . . . . . . The Concept of Medial Transposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part III: Other Intermedial Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32 33 33 39 41 42 44 45

Abstract

In order to illustrate the development of studies of intermediality, this chapter traces the progress of Claus Clüver’s academic career from his early work as a professor of Comparative Literature at Indiana University, USA, to his most recent publications. Moving from a description of early interart studies into intermedial studies, his œuvre virtually coincides with the history of the studies of intermediality. In this trajectory it focuses on general questions, such as his attempts at a set of prolegomena to a general theory of intermediality and a taxonomy of intermedial relations, while also exploring specific topics like theories of ekphrasis, transposition, and adaptation. His academic production is marked by a constant (re)evaluation of the methodological and theoretical questions involved – a fact corroborated by his recent concerns with the future of the discipline. Keywords

Intermediality · Ekphrasis · Medial transposition · Adaptation · Claus Clüver

S. Ribeiro de Oliveira · T. F. Nogueira Diniz (*) Universidade Federal de Minas Gerais, Belo Horizonte, Brazil © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_2

31

32

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

Introduction Adopted in the 1980s by scholars like Leo Hoek, Claus Clüver, and Eric Vos, the term “intermediality” still lacks a consensual definition. Nevertheless, however problematic, this indefinition presents positive aspects: it affords a transdisciplinary resonance and keeps open the possibility of new orientations for the debate on the terminological and conceptual dilemma. Intermediality has thus become a basic concept for the study of literature, theatrical performance, sequential art, history, philosophy, sociology, and the history of art – disciplines which, involving different intermedial configurations, generate specific approaches and definitions. The concept has also been used to deal with general relations between media: transformations of one medium into another, media combination, and phenomena inherent in them. Nowadays, intermedial studies have been dealing with what was once called “interarts studies” (music, literature, dance, the visual arts) and also with mass media (radio, television, newspapers) and the processes of production, distribution, and reception of the most diverse cultural objects. For interarts and intermedial studies, the importance of Claus Clüver’s œuvre can hardly be exaggerated. A groundbreaking leader in intermediality studies, he has greatly contributed to the propagation of theories and methods investigating the relations between the various arts and media. In his latest publications, which highlight developments in German and English academic discourse, Clüver takes a look back at the movements that culminated in studies of intermediality. The title of his 2019 essay “From the ‘Mutual Illumination of the Arts’ to ‘Studies of Intermediality’” offers an overview of the interdisciplinary study of the interrelations of the arts and media during the past 100 years. The essay traces a familiar path, pointed out throughout the author’s oeuvre. Starting from a focus on the binary relations of literature with the visual arts, music, and film, the work of Clüver and other international scholars led to interarts studies, which eventually came to include interrelations with nonverbal arts and configurations of a definitely nonartistic nature. In the 1990s, this would result in the reconception of the arts, nonapplied arts, and nonartistic genres as media and hence in the recognition of their interrelations as intermediality. Also to be mentioned are his attempts to construct a theoretical foundation for the study of intermediality (and transmediality) highlighting media combination, intermedial reference and intermedial transposition, especially adaptation. In an interview published in 2020 in a journal of the graduate program in the Visual Arts of the University of Brasilia, Clüver talks about how he began working in this field and his subsequent career. In 1964 he was invited to come to the University of Indiana in the USA to teach a new course called “Modern Literature and the Arts.” This line of teaching, which was not yet guided by any solid theoretical or methodological orientation, became the major area of his interest. He got involved with a research area which was known for a while in the USA as interarts studies. This was followed by an overall diachronic view of the development of new theories that redefined the areas of knowledge in the academy. New terminologies and concepts were introduced, including the development of

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver

33

theoretical branches of Comparative Literature, such as intermediality. The route thus taken is described in Clüver’s article “Inter textus/inter artes/inter media” (2001/ 2006). By that time, the field had begun to be reconceived as “intermedial studies.” Historically, the interest in word-and-image studies and in all of interarts studies originated with literary scholars, and the new (or revitalized) field of Comparative Literature appeared to be a logical home, although its international organization did not recognize it until the 1970s. “All these disciplines,” Clüver concludes, “sooner or later felt the impact of the overall turn toward critical theory that occurred in the late 1960s. It gave us the concept of paradigm shift and made us see all our cherished concepts such as “art” and “literature” as cultural constructs. This turn also profoundly affected interarts studies and ultimately favored the substitution of “media” for “arts.”

The Development of Clu¨ver’s Œuvre From its launching in the late 1950s to his most recent publications, Clüver’s work has been marked by a remarkable cohesion, looking simultaneously forward and backward, as each stage takes up and expands themes associated with the development of intermedial studies. In order to retrace these steps, this chapter will be divided into three parts: the history and theory of the studies of intermediality, the concepts of ekphrasis and intermedial transposition, and other intermedial relations.

Part I: The History and Theory of the Studies of Intermediality It consists of five essays, composed over a time of almost two decades, from 1991 to 2010. The essays, chosen by Clüver himself, may be taken to represent an important part of the development of his work as a whole. In his own words, they “are at best viewed as historical documents, each showing how the developing and changing discourse appeared to me at the time of its composition. When I wrote the first, the field was just beginning to be labeled “interarts studies,” having gone before under such labels as “literature and the other arts” or “comparative arts.” But very soon the new label began to be questioned and these essays show my assessment, at the various times of their composition, of the developments that would finally lead to the designation “studies of intermediality” or “intermedial studies.” This part starts with an essay, first published in Swedish in 1992 and translated into Portuguese in 1996, that remained unpublished, circulating only among colleagues, until made available in 2009 with the title “Interarts Studies: An Introduction.” Clüver here emphasizes the increasing difficulty of defining “the arts” and “works of art.” From the perspective of our time, the interest of this text goes far beyond the merely historical. The essay is a landmark in the author’s career – a kernel containing the gist of what was to be developed in later publications. It reads like an excursion along the meandering paths of the studies of intermediality, raising a number of crucial questions along the way. Among such questions Clüver

34

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

repeatedly reminds the reader of the provisional character of all critical paradigms, which are the results of cultural practices and thus subject to change. This involves the now quite common realization that aesthetic qualities are not inherent in certain artefacts but conferred upon them by the artworld. So, “[a]s long as the distinction between ‘art’ and ‘non-art’ is held to serve some purpose, it will be best to define a ‘work of art’ as a text in any sign system or medium that our interpretive communities authorize or oblige us to read as a ‘work of art’” (2009, p. 503). Hence, Clüver’s warning: “an art’s discourse can be viable and vital only if it also accomodates contemporary phenomena” (2009, p. 503), including those deemed “nonartistic.” A crucial passage in this essay discusses the ways in which the term intermediality can be understood: it may refer to the interrelations among the media as well as to specific forms of media transfer; it may most adequately be defined as the combination of media in multimedia, mixed media and intermedia texts. Clüver also makes a point of establishing the difference between studies of intermediality and interarts studies, a topic to which he will return again and again: “both approaches operate within a transdisciplinary perspective; but the disciplines involved are individual media studies on one hand, and the traditional ‘arts’ discipline on the other, and the paradigmatic assumptions and practices operative in each of the arts differ considerably from those governing media studies” (2009, p. 523). The approach is historical, as it sketches the progression of such topics as “the mutual illumination of the arts” via the “arts and their interrelations” and the “comparative studies of the arts” to interarts studies and studies of intermediality. It also describes the emergence of Comparative Literature, its emphasis on the analysis of relations, connections, analogies, and differences among the individual arts – which ended up in studies of intermediality. In this connection, Clüver in referring to the “new institutional sites where different arts, sign systems and media were studied collectively in various combinations and with different interests and objectives, they usually went under such labels as Media Studies, Communication Studies or Cultural Studies. In these institutional sites, references to ‘art’ or ‘the arts’ seldom turned up, and still rarer was the appearance of ‘Interarts Studies’” (2009, p. 522). From these remarks, Clüver moves on to the hopeful conclusion that “just as Comparative Literature turned from an interdisciplinary discipline with its own degree diplomas, manuals and theoretical foundations, so is the study of Intermediality on its way to becoming institutionalized as a program with its own course of studies leading to academic degrees” (2009, p. 523). This belief has proved well-grounded. In the interview published in the aforementioned journal at the University of Brasília in 2020, Clüver remarks that the field of intermedial studies has established itself independently in such places as Växjö, Sweden, and Graz, Austria, and in interdisciplinary research groups such as the one at the Federal University of Minas Gerais in Brazil. He adds that scholars and students in these institutions frequently bring along the baggage of their original discipline with its own interpretive community, which will inevitably guide their approach and the questions they will ask. “Estudos Interartes: Conceitos, Termos, Objetivos” (Clüver 1997b) (Interarts Studies: Concepts, Terms, Objectives), the second one, is among those whose

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver

35

interest, according to Clüver himself, is also historical – unsurprisingly, given the date of publication: 1997. However, as the professor himself observes, this history is partly that of intermedial studies. In fact, the essay belongs to the archeology of these studies and thus goes back to the days when the relation between literature and the other arts was associated with the notion of literary texts as autonomous and selfsufficient entities and were lumped together with extrinsic analyses – biographical, psychological, or historical. To argue against such views, Clüver compares “Sainte,” a poem by Stéphane Mallarmé, to the painting that probably inspired it. He then uses the comparison to demonstrate the importance of artistic questions like production, reception, the role of the reader, and the author in the context of the poem. Similar conclusions work together with the investigation of forms of representation and conventions in the reconstruction of the text. Next, questions like word-image relations, the concepts of ekphrasis, intersemiotic translation, transculturation, and the role of illustration are investigated. Deserving of the same attention is the notion of adaptation, defined as the study of conversions of novels into theatre plays, and of the latter into operas, or of fairy tales into ballets, and short stories into films. The third text of Part I, “Intertextus/Interarts/Intermedia,” translated from German (Clüver 2006) into Portuguese (2006), deals with the reconception of the term “interarts studies” (replaced by “intermedial studies”) and with the construction of a theoretical basis for these studies, especially in relation to the concepts of media and the current senses for “intermediality.” In the section called “Intertexts,” Clüver refers to source and target texts and their intertextual contact. Based on the idea that any work is a sign structure, whose meaning and function can be translated, the fact that they could be of different media does not prevent the possibility of translation from a text in a sign system into another of another system. Clüver moves on to consider Leo Hoek’s classification of intermedial relations and then advances his own views. He classifies cinematographic or operatic adaptations, symphonic poems, ekphrases, and ballet reviews as instances of transmedial relations. In the same line of argument, he regards emblems and titles as forms of a multimedia discourse, to which he adds a new category, mixed media. Another important addition is the insistence on the need for intermedial studies to consider contextual data as well as noncanonical forms of representation and production. When this essay was composed, Clüver had in mind the development of a common basis for studies involving not only verbal but all kinds of texts, including those combining different media and sign systems. Only in this way, he argued, would scholars of the new generation be equipped to deal with the largest part of contemporary art. In the section dedicated to interarts, Clüver discusses a number of issues to which he will return in successive efforts to lay a foundation for the criticism and interpretation of intermedial works – especially the indispensable conceptual basis and the corresponding terminology. He notes the appearance of the term intermediality in 1995 and the substitution of “studies of intermediality” for the labels “comparative arts” and “interarts studies.” He then discusses the several stages of the research in the field: it was initially confined to texts acknowledged as “works of art.” The distinction was undermined by certain decisive phenomena: the appearance of the

36

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

ready-made, the ensuing difficulty of distinguishing “art” from “nonart” and, finally, the acceptance, as a promising objects of study, of texts not considered “artistic” in the traditional sense. This process led to the recognition of the unsuitability of the label “interarts studies,” which was replaced by denominations like “word and image studies.” The study of the processes of intersemiotic transposition and parallel procedures turned to the linguistic representation of nonverbal texts and the transposition of literary texts to other arts and media, including those not always accepted as works of art. But, as Clüver adds, the term “intermediality” also proved quite suitable for the works still thought of as art: music, literature, dance, painting, architecture, opera, cinema, and traditional media like TV, radio, video, electronic, and digital media. Here also belong texts like livres d’artiste and logos. In the section dubbed “Intermedia,” Clüver considers the classification of texts into three different forms, namely, intermedia or intersemiotic, multimedia, and mixed media. But now his interest moves on to the role of the reader as the performer of the text. He also considers other related matters: the importance of post-texts and paratexts, the awareness that intertextuality always implies intermediality, and the recognition that an intermedial component also exists in works that represent aspects of sensorially apprehensible reality. Other models for the study of intermediality are considered, such as the one proposed by Werner Wolf in 1999. Even though it was based on the relation between literature and music and did not take into consideration textual production and reception, Clùver thinks Wolf’s system an advancement, particularly his distinction between “direct/open intermediality,” which requires no media transformation, and “hidden, indirect intermediality,” which does. Although most word- and image-related works can be approached as interart studies, there are many for which only intermediality proved to be an adequate concept, such as livre d’artiste, a logotype, a painting translated into a sculpture and other products. Another seminal essay, which stands out in the group of five mentioned in Part I, is “intermediality and interarts studies,” dated 2007, which reads like a sequel that expands and complements the issues advanced in “Interart Studies: Concepts, Terms, Objectives.” The new essay resumes the historical approach favored in the previous text, while, in an attempt to standardize specific terms used in intermedial research, it simultaneously deals with theoretical questions. From the historical standpoint, Clüver recalls the genesis of studies of intermediality from earlier stages like interarts studies and “literature and the other arts,” as well as such influential tenets as Oskar Walzel’s views on the “mutual illumination of the arts” and the development of film studies. He also recalls how influential the birth of Comparative Literature was in the mid-twentieth century as an interdisciplinary program designed to deal with the interrelations among modern literatures housed in individual disciplines. Clüver remarks that by the time this essay was composed (2007), the term intermediality was still comparatively new, and only a few institutions offered courses and carried on research under this label. One of the earliest “intermedial studies” programs mentioned was the intermedia analyses (first called interarts

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver

37

studies) at the Department of Cultural Sciences of Lund University, created by Hans Lund in 2001. In fact, interarts studies, an interdisciplinary area of the humanities dominated by investigations into the interrelations of literature and the other arts, receives special attention from Clüver. As he notes, this kind of research involved aspects of intermedial connections between the visual arts, music, dance, performance arts, theatre, film, and architecture, where the word plays only a secondary role or is even ignored. However, the focus of interarts studies still tended to favor those objects that could be accepted as works of art, with aesthetic concerns being most important. This was part of a basically formalist discourse that took the ontological status of “art” for granted; it was engaged in defining the essence of each of the individual arts and tended to restrict their study to specific academic disciplines. Intermittently, this essay refers to the transdisciplinary discourse on intermediality that incorporated the traditions of “interart(s) studies.” The discussion was often carried on within media studies in the context of communications. As the areas studied – radio, cinema, television, video, and print media – involve multiple media and often complex technological production processes, Clüver remarks that intermediality is an issue both within each of these media and in their interrelations with each other and with the “arts.” Besides historical considerations, theoretical questions often take the forefront in “intermediality and interarts studies.” Clüver recalls the developments that led to the recognition of intermediality as the central concern in the comparative study of the arts. As a concept and a label, the term was widely accepted to designate a discourse that not only far exceeded the parameters of the more traditional interarts studies but also introduced new objects and objectives, interests and concerns, criteria, and methods. Nevertheless, the concept of “art,” however understood, had certainly not been abandoned, even though quite a few of the cultural productions within the purview of “intermedial studies” would not be considered by an “art”-oriented discourse. Further on in the essay, Clüver returns to the three categories of word-image relations, carefully distinguishing transmedia, multimedia, and mixed-media discourses. The classification relies on criteria of distinctiveness, coherence, polytextuality, simultaneous production, and simultaneous reception. Also considered is the process, which may imply transposition, juxtaposition, combination, union, or fusion – a classification represented in a table first devised by Eric Vos and synthesized in 1997, with Leo Hoek’s and Clüver’s contributions, like intermediality, combinations of media in multimedia, mixed media, and intermedia texts. Clüver furthermore considers other supremely important concepts like medium and genre and the questions involved in distinguishing between them. In this context, he adds that the term “medium” comprises diverse but interfused categories, besides involving questions of materiality and means of production. The last text of Part I, “Intermidialidade,” (Clüver 2008) comes close to indicating how Clüver saw the field lately as a crossing of borders. As a result of his work as a teacher and researcher, this crossing of borders between media begins by searching

38

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

its way for defining medium (as communication) considering its material aspects and conceiving arts as media. He alludes to the fact that the theorization of the various aspects of intermediality and its concepts, terms, and methods consider media collectively. However, as most researchers explore more specific relations in individual texts, Clüver presents three categories proposed by Irina Rajewsky for this kind of study: the combination of media, intermedial references, and medial transposition. Combination includes phenomena such as opera, film, theatre, performances, illuminated manuscripts, computer or sound art installations, comics, and so on, whose intermedial quality is determined by the media constellation constituting a given media product, which is to say the result or the very process of combination of at least two conventionally distinct media or medial forms of articulation” (Rajewsky, 51 (2005)). Intermedial references are “to be understood as meaning-constitutional strategies that contribute to the media preoduct’s overall signification,[..] using its own mediaspecific means either to refer to a specific, individual work produced in another médium or to refer to a specific medial subsystem or another media system” (Rajewsky, 51–52). Medial transposition in which “the intermedial quality has to do with the way in which a media product comes into being [...] with the transformation of a givem media product or its substractum into another médium” (Rajewsky, p. 51). As his later essays will abundantly make clear, Clüver’s main interest seems to lie in the notion of ekphrasis, a term reintroduced by Leo Spitzer in 1955 and succinctly defined as verbal representations of visual representations. Clüver proposes to reformulate the definition as “verbal representations of texts composed in non-verbal sign systems,” arguing that the objects of such representations need not themselves be representations, as they often consist in nonrepresentative creations like bridges and cathedrals. From this the essay moves to considerations of adaptation – a concept that continues to stimulate interest. One important angle of his essays is the attention dedicated to what were then new intermedial texts like electronic digital media and their increasing interactive character. The reader is reminded of their dependence on evolving technological possibilities, including the passing of performative tasks to the recipient and the transmission to receivers all over the globe, inviting the interaction of a global community. Also considered in the article is media poetry as a producer of visual, aural (including musical), and kinetic events in which the word in its graphic and sometimes vocal dimensions may play a leading role or, on the contrary, none at all. Their connection with earlier forms of visual poetry invites comparison of the old media with the new. Other contemporary intermedial phenomena come up in the discussion. Here belong the opening ceremonies of the Olympic Games, which engage all the traditional theatrical and ceremonial media, including light shows and fireworks, and rely heavily on modern technologies. Clüver considers many other such events – religious and civic rituals, processions and parades, rallies, and pop concerts – adding that they are entirely based on the interaction of many media and can only be adequately approached from the perspective of an intermedial

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver

39

discourse. The same is said of neglected though culturally significant genres like carmina figurata, emblems, broadsides, and even medieval plays. A concern with the future of intermedial studies to which Clüver will return in later texts is not neglected in the essay. It refers to a comprehensive theory of intermediality as an ongoing project, continually affected by the development of new media. In this connection, he wonders whether the emphasis will fall on the processes of communication or on the techniques of production, or further still on the increasingly important question of reception. The text ends up with a carefully annotated bibliography including a number of journals primarily devoted to intermedial relations. We now proceed to Parts II and III of this chapter: an overview of essays composed on the foundation built by those in Part I. They may be taken as prolegomena to a full-fledged theory of the field, reinforcing and expanding what was stated before and offering the author’s interpretation of the research as it turned into “studies of intermediality.”

Part II: The Concepts of Ekphrasis and Intermedial Transposition and Related Texts The Concept of Ekphrasis In this part, the essays adopt a spiraling approach, presenting basic themes and concepts in order to eventually take them up again. Rephrased, in amplified or deeper ways, they guide the reader to a progressive familiarity with complex questions that could not be dealt with in a single text. Set up in this way, this set of essays can be divided into two groups. The first engages in theoretical questions for the purpose of analyzing specific intermedial works in the second group. Both groups of essays – the one referring to conceptual questions as well as the one dedicated to case studies – offer a framework to understand configurations constructed in different sign systems. Closely intertwined, the two groups work to ensure cohesion and illuminate the vision of the whole. The section with theoretical questions basically dwells on the notion of ekphrasis. In consecutive texts, the concept is analyzed from several angles. The first one, “From Painting to Poetry,” (Clüver 1978) discusses the category of ekphrasis in the light of its historical evolution. By way of illustration, ekphrastic poems by several authors, from the nineteenth century to modern times, are critically perused. The second essay, “On Intersemiotic Transposition,” (Clüver 1989) complements and expands the analysis of the first. It considers the affinity between the concepts of ekphrasis and of intersemiotic translation, warning the reader that the considerations on the transposition from the verbal to the visual medium are just as good for the opposite phenomenon, which is precisely the object of another essay, “Ekphrasis Reconsidered: On Verbal Representation of Non-Verbal Texts (Clüver 1997a).” Here, Clüver recapitulates and discusses concepts considered so far and then offers his own definition. He argues that the existence of ekphrasis does not require that one of the objects studied belong to the arts of representation. Ekphrasis may therefore

40

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

include verbalizations of architectural objects, abstract music, nonnarrative dance, verbalizations of artistic objects, livre-objets, critical reviews, and configurations not necessarily considered works of art, such as posters and photographs. The next essay, “Quotation, Enargeia, and the Functions of Ekphrasis,” (Clüver 1998) studies from new perspectives the history of the term “ekphrasis,” recalling the fact that more recent studies just look back to manuals of ancient Greek rhetoric when they affirm that, to be considered ekphrastic, verbal and visual configurations must be endowed with “enargeia,” that is, the capacity to represent the extratextual reality clearly and distinctly. In conclusion, Clüver insists that etymologically and traditionally, ekphrasis is the suitable label for descriptions endowed with enargeia, which excludes the possibility of ekphrastic texts that consist of mere titles or allusions to works of art. In the following essay, “The Musikgedicht: Notes on an Ekphrastic Genre,” (Clüver 1997c) the focus of the discussion is dislocated to poems about musical compositions, underlining the deep differences between the transposition of an aural and a visual phenomenon, between predominantly temporal and spatial structures, and the codes and conventions of musical, nonpictorial composition. As the title itself indicates, the next essay, “Ekphrasis and Adaptation,” (Clüver 2017b) investigates the affinities and differences between the two forms of intersemiotic transposition. According to Clüver, adaptation examines the process (and the results) of adjusting a source text to the demands and possibilities of another medium, while parts of the source are preserved and incorporated in the target text. In the following essay, “On gazers’ encounters with visual art: Ekphrasis, readers, iconotexts,” (Clüver 2019b) he offers his own definition of ekphrasis as “the verbal representation of real or fictive configurations composed in a non-kinetic visual médium.” An important addition to this definition is Clüver’s firm assertion that the chief concern of the discourse on ekphrasis should be the need to consider readers’/spectators’ performances, the way they process verbal cues so that they become a mental or visual configuration. As may be seen from our overview of Clüver’s œuvre, his work often revisits certain crucial themes, whether to be reviewed, expanded, or discussed, in a new light. This procedure bears witness to his constant refusal to remain within closed critical positions. Such is the case of his aforementioned essays centered on the concept of ekphrasis. It is no surprise that in a recent article, “From the ‘Mutual illumination of the Arts’ (Clüver 2019a) to ‘Studies of Intermediality’” (2019b), he argues that many examples of ekphrasis, though not all, can be read as intermedial transpositions. He argues that there are many markers that allow the reader to receive some verbal texts as ekphrases – depending on the concept accepted by his interpretive community. Part of the definition proposed by Claus (2019b, p. 252), at the moment in which he wrote the quoted text, reads as follows: “Ekphrasis is an ‘enargetic’ description of non-kinetic visual configurations as semiotic objects” and goes on to say “[Ekphrasis] verbalizes perceptions of, or reactions to, characteristic features of configurations that actually exist or suggests the perceived existence of such configurations in virtual, or fictive, reality. Its materials are purely

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver

41

verbal.” [...] Adding a caveat to his definition, Clüver warns the reader that because it is a [...] “cultural construct” [...], his present definition is subject to change.

The Concept of Medial Transposition In short, Clüver will not give up his pursuit of the accuracy and comprehensiveness indispensable to serious intellectual work. As further evidenced, we would like to discuss in some detail the treatment of transposition, another form of intermediality to which he returned again and again. This second concept has been explored since Clüver’s initial work. In 1987, he stated that an adequate transposition of a considerable part of twentieth-century paintings into poetry could only be possible through the semiotic process, that is, the transfer of the literary material to a correspondent sign system. Two years later, following Roman Jakobson, he defined intersemiotic translation or transmutation as “the interpretation of verbal signs through non-verbal signs.” Based on the notion of interlingual translation and its consequent search for equivalents, he expanded the concept to include imitation, adaptation, pastiche, parody, interpretation, and discourse. In 1992, influenced by the concept of rewriting, Clüver concentrated his attention on two contrasting forms of contact between verbal and nonverbal texts: translation and adaptation. In his view, translation included semiotic transposition, verbal description, musicalization, visual illustration, the transcreation of program music characteristics, mime, and dance. Here, he also included the adaptation of verbal texts to compositional demands of other media, as well as the imitation of formal characteristics in different media. Later on, in 1997, Clüver again devoted his attention to the term intersemiotic transposition, this time involving the translation from one sign system to another system belonging to a different medium. He distinguished translation from transposition: translation involves signs (intersemiotic translation) and culture (cultural translation), while transposition is a more comprehensive process. As happens in interlingual translation, Clüver added some kinds of illustration to the category of intersemiotic transposition. He soon began to use the term medial transposition, establishing a difference between this concept and the adaptation of texts to a different medium. In this category, he included the transfer of elements of the font text to the target text; the same is applied to the transfer of narratives to the stage and the screen. Clüver also makes a point of distinguishing adaptation from transposition: as he sees it, adaptation consists in the transfer of a process or a configuration from one médium to another. Conversely, medial transposition deals with the process that results in configurations belonging to only one medium. That is, in different ways, they refer only to the medium in which they were composed. In his 2017 essay, “Intermediality and Interarts Studies (Clüver 2007a),” writing historically about these terms, Clüver agrees with Irina Rajewsky’s proposal to consider three categories of relations among media – media combination,

42

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

intermedial references, and transposition – adding to the latter the possibility of medial transformation. In his long involvement with these themes, Clüver came to discuss a series of interrelated notions, such as transmutation, with special attention to the relation between ekphrasis and adaptation, and the corresponding terminological changes. Another pair of interrelated concepts treated by him is adaptation x ekphrasis. The former is the process of adjusting a font text to the demands and possibilities of another, to which it is incorporated in such a way that its parts are preserved. By contrast, ekphrasis deals exclusively with the description of visual configurations.

Related Texts The second section of essays included in Part II turns to the study of specific works. “Rewriting Edward Hopper” consists of a series of readings of poems on Edward Hopper’s paintings (Clüver 1999). In a number of these poems, Clüver points out ekphrastic passages, explicit verbal representations of visual configurations. Next, “Body, Voice and Gaze” explores the relation between text and image in the illustrations of One Thousand and One Nights by three painters (Clüver and Sumi 2006). In his reading of the interaction between narrative and iconography in the illustrations, Clüver is particularly interested in the relations between discourse (storytelling), sight (seeing and looking), and (representations of) the feminine body. The essay “Devouring the Other: Antropophagy in Nelson Pereira dos Santos’s Como era gostoso o meu francês” (How tasty was my Frenchman) discusses Santos’ film, which narrates the experiences of a Frenchman captured and devoured by Indians of a Tupinambá tribe (Clüver 2001). This plot is used as a cue to explore the notion of “anthropophagy,” a powerful metaphor of intellectual and literary Brazilian life since 1928.

Part III: Other Intermedial Relations In contrast with the strategies adopted in the previous part, the essays we now discuss are not structured around a single controlling concept like ekphrasis or transposition. If in these texts we can detect a guiding thread, it lies in the attention to critical practices for the study of contemporary works, such as the analysis of different approaches to objects perceived as texts potentially decoded by the reader; the interdisciplinary discourse about mixed media, especially those resulting from the fusion of verbal and visual texts made feasible by new technologies; the discussion of the innovative character of the metareferentiality found in graffiti; a reflection on a basic assumption of contemporary creations; and the discussion of the concepts of literariness and musicality as qualities not inherent in the text, but conferred on it by cultural codes. These aspects of present-day criticism are contemplated and minutely illustrated in five essays about specific texts, three of which would not be considered “art” in the traditional sense. The first one, “Seeing Through the Camera Lens: Photo-Realism and Alain Robbe-Grillet’s Instantanés” compares Alain Robbe-Grillet’s short text, “L’escalier

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver

43

mécanique,” with “Downtown,” a painting by Richard Estes (Clüver 1992). The essay emphasizes the ways in which remarks about a verbal text may affect the reading of a painting and the other way round. The study of mixed media is the object of the essay “Incestuous liaisons: the sister arts in contemporary culture (Clüver 2000).” The article analyses animations, magazine covers, and comic strips, as well as coins, banknotes, and postage stamps. There follows the analysis of kinds of writings in paintings and concrete poems that aim at a maximum effect with a minimal manipulation of the verbal material. Similar attention is lavished on the creation of an iconic relation between signifier and signified. Clüver finally discusses the close relation between verbal and visual signs that was made possible by the advent of new technologies. The essay “On Modern Graffiti and Street Murals: Metareferential Aspects of Writings and Paintings on Walls” discusses graffiti, paintings developed with new materials and techniques that occupy the walls of urban spaces (Clüver 2011). Now accepted as a kind of popular art, they take a prominent position in the contemporary art scene. Clüver compares them to concrete poems: as in concrete poetry, the potential of the font and size and color of the letters of graffiti to attract the eyes. Even though he does not equate graffiti and concrete poems, the essayist emphasizes that both explore the potential of letters in a way that dissociates them from the formation of words. The essay likewise establishes a relation between graffiti, logos, and trademarks based on letters and also with imperial monograms and church symbols. Finally, Clüver comments on the ephemerality of graffiti: given the absence of their original context, even when preserved in archives or photographs, they eventually lose their meaning. The recycling or creative rewriting of preexisting texts is the object of the article “Luciano Berio’s Sinfonia: the Creative Flow of a Musico-verbal Collage (Clüver 2007b).” This essay analyses Berio’s “Sinfonia,” starting from a mixture of music and words, collages of musical fragments, and verbal texts recited by vocalists. There are musical citations and hardly intelligible verbal material, besides references to myths on the origin of water. Berio’s composition therefore presents itself as a symbol of a basic presupposition of making art in his day: an assemblage of preexisting verbal and musical texts composed by imitation, translation, parody, direct and indirect quotation, references and allusions, and the integration of motifs, phrases, fragments, and gestures inherent in language, social codes, and cultural léxicon. The question of literariness, in parallel with that of musicality, emerges in the fifth essay, “Musical Train Rides in the Classroom (Clüver 1991).” This essay narrates an experience of Clüver himself in one of his courses on Comparative Literature at Indiana University. Having presented some texts, he asked students if they could be read as literature. A similar question was asked after they had listened to Schaeffer’s Étude, a recording of noises/sounds by moving trains. As it used nonconventional sounds, the work was associated with cubism. Étude was contrasted with Honegger’s Pacific 231. This composition does not consist of the imitations of sounds by a locomotive, but, in a translation, by means of a musical construction, of a visual impression and the corresponding physical sensation. After listening to

44

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

the two compositions, the class came to the conclusion that, just as in literature, the status of musicality is conferred on a sound by its listeners. Clüver concludes his essays returning to a point that becomes increasingly important in his theoretical work: the advent of new technologies and new media has made possible the fusion of verbal and visual signs and the participation of the reader as co-creator of the text in the virtual space. He alludes to those texts that include sounds and challenge the illustration on the page and to verbal-visual environmental creations, typical of present-day works.

Conclusion Alongside a review of such long-contemplated issues, Clüver remains faithful to his lifelong habit of continuously refurbishing his theoretical armory. We might here mention his latest rephrasing of the concept of ekphrasis. As he puts it now, ekphrasis verbalizes the perception/reaction by a fictitious observer of real or fictitious nonkinetic visual configurations: In his words [e]kphrasis verbalizes a real or fictive viewer’s perceptionn of, or reactions to, characteristic features of non-kinetic visual configurations. It deals with configurations that actually exist, or suggests the perceived existence of such configurations in visual, or fictive reality. (Clüver 2017a, p. 30–31)

This does not render less useful his previous definitions of the concept. It simply looks at the phenomenon from a new perspective. On the other hand, as a counterpoint to his interest in the genealogy of studies of intermediality, Clüver, Janus-like, turns his gaze to the future. In his interview at the University of Brasília in 2020, he argues for several needs faced by the discipline. First of all, he emphasizes the urgency of a full-fledged theory of intermediality with a corresponding terminology. He also attaches great importance to teaching. Aware that courses in intermedial studies, when available in any format, have proven extremely attractive for students, he insists on the urgency of training teachers, who must be competent in dealing with the media involved, familiar with the research and the relevant bibliography, and so be able to equip learners with the basis for pursuing their interests. As an example, Clüver argues that, before analyzing how a musical setting relates to its lyrics, or how Haroldo de Campos’ concrete poem “branco” might be read as a transcreation of a painting by Malevich, a student has to know how to read the lyrics of a concrete poem and what to listen for in music or how to deal with a Suprematist painting. To make such teaching feasible, Clüver mentions the requirement of technical support, perhaps textbooks or eventual digital equivalents. In fact, since most presentations on relevant topics require somewhat complex technologies, pertinent conferences should offer sessions about sharing these. He also considers the advantages of scholars’ association with groups dedicated to studies of intermediality.

3

An Updated Survey of Early Interart and Intermediality Roots: Claus Clu¨ver

45

Another important possibility he considers is team-teaching courses, especially when they are meant to introduce the entire field, such as those taught at UFMG, where professors representing different media give individual presentations, with a coordinator providing indispensable coherence. Clüver also stresses the need for scholars’ familiarity with the constant flow of medial possibilities created by the new technologies – a need all the more pressing in the face of new academic practices resulting from the worldwide COVID-19 pandemic. Among the implications of such changes for the reception of art and the interaction with spectators in exhibition spaces, Clüver hopes that art museums and art galleries may succeed in finding ways to create virtual visits of new exhibitions and their regular collections. These and other considerations in Clüver’s recent work are an important part of his legacy to studies of intermediality.

References Clüver, Claus. 1978. Painting into poetry. Yearbook of Comparative and General Literature. 27: 19–34. ———. 1989. On intersemiotic transposition. Poetics Today 10 (1): 55–90. Duke University Press. ———. 1991. Musical train rides in the classroom. Indiana Theory Review 12: 163–185. ———. 1992. Seeing through the camera lens: Photo-realism and Alain Robbe-Grillet’s Instantanés. In I musernas sällskap: Konstarter och deras relationer, ed. Bernt Olsson, Jan Olsson, and Hans Lund, 200–234. Lund: Wiken. ———. 1997a. Ekphrasis reconsidered: On verbal representation of non-verbal texts. In Interart Poetics: Essays on the interrelations of the arts and media, ed. Ulla-Britta Lagerroth, Hans Lund, and Erik Hedling, 19–33. Amsterdam/London: Rodopi. ———. 1997b. Estudos Interartes: Conceitos, Termos, Objetivos. Literatura e Sociedade 2 (2): 37–55. ———. 1997c. The Musikgedicht: Notes on an Ekphrastic Genre. In Word and music studies: Defining the field, ed. Walter Bernhart, Paul Steven Scher, and Werner Wolf, 187–204. Amsterdam: Rodopi. ———. 1998. Quotation, “Enargeia”, and the functions of Ekphrasis. In Pictures into words: Theoretical and descriptive approaches to Ekphrasis, ed. Valerie Robillard and Els Jongeneel, 35–52. Amsterdam: Free University Press. ———. 1999. (Re) Writing Edward Hopper. In Das Visuelle Gedächtnis der Literatur, ed. Manfred Schmeling and Monika Schmitz-Emans, 141–165. Würzburg: Königshausen & Neumann. ———. 2000. Liaisons Incestueuses: The sister arts in contemporary culture. In Entre Artes e Culturas. ACT 2, ed. Helena C. Buescu and João F. Duarte, 9–37. Lisboa: Centro de Estudos Comparatistas, Faculdade de Letras. ———. 2001. Devouring the other: Antropophagy in Nelson Pereira dos Santos’s Como era gostoso o meu francês. Cadernos de Tradução. 7 (1): 31–51. ———. 2006. Intertextus/Inter artes/Inter media. Aletria: Revista de Estudos de Literatura 14: 9–39. ———. 2007a. Intermediality and interarts studies. In Changing Borders: Contemporary positions in intermediality, ed. Jens Arvidson, Mikael Askander, Jørgen Bruhn, and Heidrun Führer, 19–37. Turnhout, Berlgium: Intermedia Studies Press. ———. 2007b. The creative flow of a Musico-Verbal Collage: Section III of Luciano Berio’s Sinfonia. In Creations: Medieval rituals, the arts, and the concept of creation, ed. Sven Rune

46

S. Ribeiro de Oliveira and T. F. Nogueira Diniz

Havsteen, Nils Holger Petersen, Heinrich W. Schwab, and Eyolf Østrem, 237–254. Brepols Publishers. ———. 2008. Intermidialidade. Póς: Revista do programa de pós-graduaçao em Artes. UFMG 1: 5–20. ———. 2009. Interarts studies: An introduction. In Media Inter Media: Essays in Honor of Claus Clüver, Org. Stephanie A. Glaser, 497–526. Studies in Intermediality. Amsterdam/New York: Rodopi. ———. 2011. On modern Graffiti and Street Murals: Metareferential aspects of writings and paintings on walls. In The Metareferencial Turn in Contemporary Arts and Media: Forms, functions, attempts at explanation, ed. Werner Wolf, 279–303. Brill. https://doi.org/10.1163/ 978940120091_012. Print E-Book, ISBN: 9789401200691. ———. 2017a. A new look at an old topic: Ekphrasis revisited. Todas as Letras 19 (1): 30–33. ———. 2017b. Ekphrasis and adaptation. In The Oxford handbook of adaptation studies, ed. Thomas Leitch, 459–476. New York, USA: Oxford University Press. ———. 2019a. From the “Mutual Illumination of the Arts” to “Studies of Intermediality”. International Journal of Semiotics and Visual Rhetoric 3 (2): 63–74. https://doi.org/10.4018/ IJSVR.2019070104. ———. 2019b. On Gazer’s encounters with visual art: Ekphrasis, Readers, ‘Iconotexts’. In Ekphrastic Encounters: New interdisciplinary essays on literature and the visual arts, ed. Richard Meek and David Kennedy, 237–256. Manchester, UK: Manchester University Press. ———. 2020. Interview to Márcia Arbex and Miriam Vieira. VIS 19 (1): 2447–2484. Clüver, Claus, and A.M. Sumi. 2006. Body, voice and Gaze: Text and illustration in the Frame Story of the Thousand and One Nights. In The Arabian Nights and Orientalism, Perspectives from East & West, ed. Yuriko Yamanaka and Tetsuo Nishiko, 1st ed., 194–218. Londres: I. B. Tauris. Rajewsky, I.O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités / Intermediality 6: 43–64.

4

Ekphrasis: Intermedial and Anglophone Perspectives Sofie Behluli and Gabriele Rippl

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Histories of Ekphrasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Theories and Definitions of Ekphrasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Typologies of Ekphrasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Functions of Ekphrasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

48 50 53 58 60 66 67

Abstract

“Ekphrasis” (from the Greek ek ¼ out and phrazein ¼ to speak, meaning “to speak out” or “to show clearly and completely”) is one of the most adaptable and multifaceted modes of speaking and writing in the Western tradition, looking back on a rich history of nearly three millennia. In its broadest sense, ekphrasis is understood as an intermedial reference, for example, when a text refers to a photograph or a film to a painting. More narrow understandings of ekphrasis follow James A. W. Heffernan’s 1993 definition of it being “the verbal representation of visual representation.” Focusing mostly on word-image configurations, this chapter traces the long history of ekphrasis from Homer to Margaret Atwood; presents an overview of definitions, typologies, and theoretical approaches to ekphrasis against the backdrop of various cultural shifts and important aesthetic debates; collects a number of functions of ekphrasis that range from the paragonal to the ethical to the auraticizing; and features many literary examples from Anglophone literature to illustrate the ideas and theories at hand. The aim of this chapter is to give a concise overview of the research field in the Western tradition and to highlight newer trends that bring ekphrasis in conversation with, for instance, ecocriticism, the cognitive sciences, and the digital humanities. S. Behluli (*) · G. Rippl English Department, University of Bern, Bern, Switzerland e-mail: sofi[email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_64

47

48

S. Behluli and G. Rippl

Ultimately, this chapter demonstrates how important it is to actively think about ekphrasis in today’s media ecology because it is such a potent tool to question epistemologies, read and misread images, and reconstruct and subvert concepts of representation. Indeed, ekphrasis reveals, contests and rewrites deep-seated beliefs that go beyond visual culture. Keywords

Ekphrasis · Intermediality · Description · Enargeia · Digital Age

Introduction In its broadest sense, “ekphrasis” is today understood as an “intermedial reference” (Rajewsky 2005: 52, Wolf 2005: 254), that is, a subcategory of intermediality. As a mode of (often self-reflexive) speaking and writing that brings word and image together, it refers to visual media such as paintings, drawings, photographs, or sculptures, as well as to images transmitted and circulated by the mass media of our time, such as television, videos, the Internet, or social media such as Facebook and Twitter. Ekphrases in narrative fiction and poetry can refer explicitly or covertly to absent or present images or to actual or fictional ones. They can be either long and very detailed or short and nothing but mere traces and allusions, and they can take on different degrees of concreteness and may have a descriptive-static or narrative nature (cf. Yacobi 1995). Regarding today’s Anglophone literary market and the surprisingly high number of contemporary novels and poetry collections that are replete with a great variety of ekphrases, it becomes immediately clear how innovative and experimental ekphrastic writing practices are and how widespread this particular intermedial aesthetics is in our digital age. In spite of the easy accessibility and availability of images on the screens of computers, laptops, iPads, and mobile phones, ekphrasis is thriving while at the same time taking on new functions. Of course, one would assume that the rapid development and dissemination of images through the Internet and other mass media would make the description of images futile for an expanding and increasingly media- and Internet-educated audience, but this is not the case. The fascination with ekphrasis today is due to, among other things, its participation in our culture of images, its invitation to ask questions that belong to the most pressing themes of our audio-visual culture and its training of our visual literacy. Moreover, ekphrasis is a means to question epistemologies, read and misread images, and reconstruct and subvert concepts of representation while concomitantly fostering insights into the collaborations of words and images in our present media ecology. While this essay places its focus on ekphrasis in the Western and specifically Anglophone tradition, the practice and study of ekphrasis can be found across the globe, for instance, in Brazil, where several intermediality researcher work on ekphrasis (Vieira 2011, 2020; Ferreira and Vieira 2020; BugnoNarecka and Vieira 2020). Moreover, Anglophone literature in the Caribbean (Eckstein 2005; Neumann and Rippl 2020), Africa (Neumann and Rippl 2020;

4

Ekphrasis: Intermedial and Anglophone Perspectives

49

Rippl 2018), and India (Kortenaar 1997; Rippl 2015b; Neumann and Rippl 2020), to name just a few, celebrate ekphrastic aesthetics and have scholars engaged in studies of ekphrasis and vision beyond Western traditions and paradigms. While many scholars still base their work on James A. W. Heffernan’s 1993 definition – “ekphrasis is the verbal representation of visual representation” (3), according to which ekphrasis is a second-degree representation of a work of art, it must be noted, though, that his definition has come under scrutiny (cf. Behluli and Rippl 2017). The result of this was a considerable adaptation, modification, and expansion of the concept of ekphrasis with regard to (1) genres: poetry and narrative fiction, art historical writing, and other forms of art writing; (2) the visual objects referred to ekphrastically: according to many researchers these objects are no longer exclusively (canonical) works of art; (3) current definitions of “representation” (cf. below “Theories and Definitions of Ekphrasis”); (4) theoretical conceptualizations of ekphrasis within the semiotic paradigm: in accordance with Irina O. Rajewsky’s (2005) and Werner Wolf’s (2005) respective definitions, ekphrasis is now understood by many scholars as “intermedial reference,” that is within the wider cultural frame of intermediality (Rippl 2005, 2015a, b, 2018; Bruhn and Schirrmacher 2022); (5) the media of ekphrasis: according to some critics (e.g., Bruhn 2000; Rippl 2005), language is no longer considered the only medium to deliver ekphrases; opera, drama, film, etc., may also do so; and (6) today’s rapidly changing media landscapes with the ready availability of pictures: scholars working in the field of digital humanities have recently started to discuss ekphrasis as a digital phenomenon with specific affective and immersive potential. Taking the cue from Mitchell and his famous argument that “all media are mixed media” (1995: 94–95), many ekphrasis scholars today suggest overcoming the view of word and image as a dichotomy, as two different semiotic systems and media, instead putting a focus on networks and collaborations of the verbal and the visual (Neumann and Rippl 2020) and on ekphrasis’ performative force interacting with an audience (cf. Brosch 2018a: 226). These new definitions catapult borderline cases of ekphrasis into the attention of researchers. What Heffernan’s definition has always excluded is abstract, non-representational painting, as well as a staple example of traditional ekphrastic writing, namely, architectural descriptions, such as the descriptions of Roman villas and gardens in Statius’s (c. 45–96 CE) Silvae (probably composed between 86 and 96 CE). As Miriam Vieira has demonstrated, architectural ekphrasis and “the medial traits involved in the transmediation of architectural processes” (Vieira 2020: 121) still play an important role in contemporary literature, for instance, in Michael Sledge’s biographical narrative The More I Owe You (2010). Reminding readers of the ancient rhetorical notion of ekphrasis, which in the words of Ruth Webb “does not seek to represent, but to have effects in the audience’s mind that mimics the act of seeing” (2009: 38), Vieira underlines the importance of emotions and intense embodied visual experiences that the mental performative power of ekphrases of spaces and buildings help readers to make (2020: 122). Speaking of borderline cases, there are also many instances in contemporary Anglophone literature where (precious or mundane) household objects figure prominently via quasi-ekphrastic strategies, as is the case in Margaret Atwood’s many detailed

50

S. Behluli and G. Rippl

descriptions of quilts in her novel Alias Grace (1996), the artful descriptions of pieces of furniture and interior in Donna Tartt’s novel The Goldfinch (2013) or the verbal evocations of taxidermy objects such as a hummingbird in Jeff VanderMeer’s novel Hummingbird Salamander (2021). These references to and descriptions of objects, which are not exactly “visual representations” according to Heffernan’s definition, deserve the attention of ekphrasis scholars, as do ekphrastic assemblages in Leanne Shapton’s in her Important Artifacts and Personal Property from the Collection of Lenore Doolan and Harold Morris, Including Books, Street Fashion, and Jewelry (2009, cf. Fjellestad 2015). Future discussions of ekphrasis will need to ask the question of how to integrate such borderline cases into existing definitions and to ponder on the problems an expanded definition of ekphrasis might bring with it. Recent important publications such David Kennedy and Richard Meek’s edited volume Ekphrastic Encounters (2019) show that the research field on ekphrasis is thriving with new questions and approaches.

Histories of Ekphrasis During its long history, ekphrasis had periods of great attention and wide distribution, and other times when it was only a marginal phenomenon (for an extended overview of the history of ekphrasis cf. Rippl 2019b). Interestingly, since the late 1980s, ekphrasis has had another meteoric rise not only in literary studies (Krieger 1992; Heffernan 1993; Hollander 1995; Boehm and Pfotenhauer 1995; Wagner 1996) but in Anglophone literatures, too, where ekphrastic and other intermedial aesthetics have played an increasingly crucial role during the past three decades. One of the reasons why writers and scholars alike are fascinated by ekphrasis is that ekphrastic phenomena invite us to ask questions that belong to the most pressing themes of our audio-visual culture. In other words, they react to the ubiquitous presence of images and help to reconstruct concepts of representation and mimesis, which predominate in specific historical periods and cultures. Originally, “ekphrasis” is a Greek term, which etymologically simply means “to speak out” or “to show clearly and completely” (ek ¼ out, phrazein ¼ to speak) and which the OED records as first used in English in 1715. As a literary descriptive mode, ekphrasis is at least as old as Homer’s depiction of Hephaestos’s making of Achilles’s shield in Book 18 of The Iliad (eighth century BCE) and has thus close ties to the semantic field of the epic, the empire, and imperial interests. In late antiquity, the first “theory” of description was developed and termed ekphrasis (in Greek) or evidentia (in Latin) (sometimes also hypotyposis or descriptio), which consisted of a set of rules about how to describe objects, people, places, and times properly (Webb 1999, 2009; Rippl 2019, 2022). Description was considered as a mode of speaking and writing which – by aiming at enargeia (Anschaulichkeit) – was able to bring absent objects before the listeners’ mental eyes and served, according to Quintilian, to rouse their emotions by making them quasi-eyewitnesses. Classicist Ruth Webb has demonstrated how, for ancient rhetoricians, the term ekphrasis referred to any description of persons, objects, places, or times that was “enargetic,” while in many later definitions until

4

Ekphrasis: Intermedial and Anglophone Perspectives

51

today, the concept has been understood much more narrowly – a development that started during the Second Sophistic, when ekphrasis came to designate the description of works of art (cf. Philostratus the Elder’s Eikones [Imagines, third century CE], whose ekphrases have works of art as their subject). Ancient ekphrasis, in the broader sense of detailed and lively description, had a rich afterlife throughout the Middle Ages, the Renaissance, Neoclassicism, and even into the Romantic Age (Johnston et al. 2015; Barbetti 2011; Klarer 2001; Heffernan 1993). The classical traditions of literary ekphrasis as enargetic description thrived during the Middle Ages and influenced the work of Geoffrey Chaucer (c. 1340–1400), for example, his paragone-like ekphrastic descriptions in The Knight’s Tale (c. 1380), which have temples and their frescoes as their objects, or in The Parliament of Fowles (c. 1380), which contains descriptions of paintings on the walls of the temple of Venus. In his Ars poetica, Horace (65–8 BCE) referred to an influential formula ascribed to Simonides of Ceos (late sixth century BCE), ut pictura poesis, which has been translated “as in painting so in poetry” and implies that poetry and painting function according to the same rules. This formula was still influential in the Renaissance when painting and poetry were first referred to as “sister arts” (cf. Hagstrum 1958). However, the term “sister arts” hides the fact that the different art forms were increasingly understood to be in competition with each other. This competitive understanding of word-image relationships intensified in the Renaissance debate about the hierarchy of the arts, which found its most notable manifestation in painter, draughtsman, and inventor Leonardo da Vinci’s Paragone of Poetry and Painting (the Italian paragone literally means “comparison,” before it came to signify “competition”). Da Vinci (1452–1519) argues for the supremacy of sight and the superiority of the visual arts, which are capable of producing highly authentic and vivid representations of the world because they present their objects in a seemingly immediate, transparent manner and allow for an almost natural resemblance with the world. This competitive understanding of media as well as da Vinci’s notion of the visual arts as a seemingly immediate and natural aesthetic form have strongly influenced subsequent assessments of the “sister arts,” with theorists reevaluating the paragone as “rivalry” (instead of “comparison”) between the arts. During the English Renaissance, ekphrasis kept on being a common feature of poetry and drama. William Shakespeare’s narrative poem The Rape of Lucrece (1594), for example, with Lucrece’s contemplation of the fall of Troy represented by a painting, is a classic of Renaissance ekphrasis. Likewise, Thomas Kyd’s Spanish Tragedy (1587), Edmund Spenser’s Faerie Queene (1590) and Philip Sidney’s The New Arcadia (1590) are examples of the Renaissance paragone (Klarer 2001). The long, rich and multifaceted history of how poetry and painting, word and image, the visual and the verbal relate to each other, and how they interact and network, does not come to a halt in the Enlightenment. In fact, Addison’s essay “The Pleasures of the Imagination” (published in The Spectator in 1712), with its discussion of description and enargeia in connection with the question of mimesis and the paragone represents another important moment in the history of ekphrasis (Rippl 2015c). A common point of discussion was the comparison of poetry and visual art:

52

S. Behluli and G. Rippl

While poetry (and other discursive genres) works with non-iconic, arbitrary signs and is hence further detached from nature than sculpture or painting, its enargetic descriptions still have the power to forcefully affect the reader – a claim that is in line with ancient rhetoricians and illustrates their impact on eighteenth-century aesthetics (cf. Rippl 2015c, 2019, 2022). Gotthold Ephraim Lessing’s (1729–1781) influential 1766 essay “Laocoön: An Essay on the Limits of Painting and Poetry” differentiates between words/poetry and images/painting as two sign systems adhering to two radically different and independent modes of representation with specific semiotic and medial properties: while painting constitutes an art of space and therefore abides by the laws of simultaneity (painting is “a visible arrested action, the different parts of which develop side by side in space”), poetry is an art of time, which instead adheres to the principles of succession (poetry is “a visible continuous action, the different parts of which occur step by step in succession of time,” Lessing 1984: 98). This gives rise to a range of related binary oppositions, such as reason vs. body, voice/eloquence vs. muteness, passivity vs. activity, subject vs. object, masculinity vs. femininity, etc., which have structured Western concepts of the “sister arts” and epistemologies more generally. Ekphrasis, understood in the broad sense defined by ancient rhetoricians, had an important afterlife in the vivid descriptions of place and natural scenery in the works of British romantic writers such as William Wordsworth and Lord Byron (Koelb 2006). Indeed, the fully fledged ekphrastic poem in English that references a painting (notional or actual) was established by the Romantic poets. Among the canonical ekphrastic poems on the “timeless serenity of visual art” (Heffernan 1993: 93) belong classics such as John Keats’ “Ode on a Grecian Urn” (1820), William Wordsworth’s “Peele Castle” (1807) as well as Percy Bysshe Shelley’s “Ozymandias” (1818) and “On the Medusa of Leonardo da Vinci” (c. 1810). While these early Romantic poems present a paragonal conflict between word and image, late Romantic writer and classicist Walter Pater contested the separation and binary pairing of the arts: “each art may be observed to pass into the condition of some other art, by what German critics term as Anders-streben – a partial alienation from its own limitations, through which the arts are able, not indeed to supply the place to each other, but reciprocally to lend each other new forces” (Pater 1986: 85). More pertinent ekphrastic poetry was produced during the nineteenth century, in particular by Pre-Raphaelites such as poet-painter Dante Gabriel Rossetti (Sonnets for Pictures, 1850) and by Robert Browning (“Fra Lippo Lippi”, 1855, “My Last Duchess”, 1842). Late nineteenth-century aestheticism can boast with one of the most prominent examples of ekphrastic writing: Oscar Wilde’s The Picture of Dorian Gray (1890). Henry James’ novels, for instance, The Portrait of a Lady (1881), as well as short stories such as “The Real Thing” (1892), are replete with ekphrastic passages and respond to the media landscape of their time with photography establishing itself as one of the new art forms. James’s fiction is the start of modernism’s intense negotiation of the collaborations and entanglements of words and images. Modernists writers on both sides of the Atlantic, for example, James Joyce, Ezra Pound, Hilda Doolittle (cf. Rippl 2005), and Gertrude Stein (cf. Haselstein 2019), were

4

Ekphrasis: Intermedial and Anglophone Perspectives

53

engrossed in the visual and produced a wealth of ekphrases. Postmodernist and contemporary narrative fiction and poetry are often also replete with ekphrases that are no longer just focused on paragone: intermedial entanglements, configurations, collaborations, and processual networks characterize the digital age. Ekphrastic poetry (Denham 2010; Loizeaux 2008; Kennedy 2012, 2015), ekphrastic narrative fiction (Karastathi 2015), and the ekphrastic novel (Behluli 2021b) more concretely thrive in the twentieth and twenty-first centuries. Among the many often anthologized ekphrastic poems are W. H. Auden’s “Musée des Beaux Arts” (1938), which engages with Bruegel’s iconic painting Landscape with the Fall of Icarus (c. 1560s); Sylvia Plath’s “The Disquieting Muses” (1957), which references Giorgio de Chirico’s “Le Muse Inquietanti” (1916); and William Carlos Williams Pictures from Breughel (1962). These poems do not subscribe to slavish descriptions of images, rather, they verbally transform them, making readers see something they could not see before. In the USA, one particular group of writers, painters, dancers, and musicians named the “New York School” (1950s and 1960s), with Frank O’Hara as one of its leading figures, published an array of post/modernist ekphrastic poems. John Ashbery’s long and highly self-reflexive postmodernist ekphrastic poem “Self-Portrait in a Convex Mirror” (1973), contemplating a Renaissance painting of the same name by Francesco Parmigianino (1524) and the condition of the modern museum-goer, is one of the most well-known ones. In a typically postmodernist manner, the deconstructive skepticism of Ashbery’s poem questions representation and mimetic or illusionist aesthetics. Among ekphrastic narrative fiction, which has become a conspicuous feature of the literary market at the end of the twentieth and beginning of the twenty-first centuries, are Julian Barnes’ A History of the World in 10 ½ Chapters (1989), A. S. Byatt’s The Matisse Stories (1993), Salman Rushdie’s The Moor’s Last Sigh (1995), John Updike’s Seek My Face (2002), Teju Cole’s Open City (2011), Donna Tartt’s The Goldfinch (2013), NoViolet Bulawayo’s We Need New Names (2013), James Bradley’s Clade (2015), Ottessa Moshfegh’s My Year of Rest and Relaxation (2018), and Richard Powers’s The Overstory (2018). These contemporary narratives adhere to an experimental ekphrastic aesthetics, which invites readers to contemplate, among other things, the new functions of ekphrasis in the digital age.

Theories and Definitions of Ekphrasis For a long time, ekphrasis was considered an exotic literary subgenre or mode of speaking and writing, but its more recent stellar career as a much-debated concept in literary studies, cultural studies, intermedial studies, theater studies, art history, and other disciplines makes any attempt to give a single definition futile. In what follows, a range of seminal concepts and definitions of ekphrasis are discussed, and recent developments in the field summarized. The first twentieth-century literary scholar to define ekphrasis was Leo Spitzer in 1955: according to him, ekphrasis is “the poetic description of a pictorial or sculptural work of art” (72). Spitzer’s in-depth discussion launched a renewed interest in ekphrasis, which was followed in 1967 by Murray

54

S. Behluli and G. Rippl

Krieger’s seminal essay “Ekphrasis and the Still Movement of Poetry; or Laokoon Revisted” and in 1992 by his influential book-length study Ekphrasis: The Illusion of the Natural Sign. In the tradition of da Vinci and Lessing, Krieger considers poetry as a temporal art resting on the arbitrariness of signs and paintings as a spatial art of “natural signs.” According to him, ekphrastic poetry is characterized by “the semiotic desire for the natural sign” and works “toward the illusion that it is performing a task we usually associate with an art of natural signs” (Krieger 1992: 11), that is, with painting or sculpture. Krieger’s view was contested later by many scholars. Three years later, John Hollander’s The Gazer’s Spirit: Poems Speaking to Silent Works of Art (1995) offers an important contribution to ekphrasis research through its differentiation between “actual ekphrasis” – the description of an existing work of art – and “notional ekphrasis” – “the verbal representation of a purely fictional work of art” (Hollander 1995: 4). The most influential and broadly used definition of ekphrasis to date is the one James A. W. Heffernan presented in his Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery (1993): “ekphrasis is the verbal representation of visual representation” (1993: 3). Heffernan’s definition of ekphrasis as a second-degree representation understands ekphrasis as a representation of a work of art – a view that met with criticism due to its alliances with concepts of mimesis and referentiality. Researchers have suggested replacing the term representation with other terms such as “re-presentation” (Yacobi 1995), “re-representation” (Kafalenos 2012), and “evocation” (Rippl 2005). Another important intervention was made by Claus Clüver, who pleaded for an expansion of the concept of ekphrasis and defined it as “intersemiotic transposition” (1997: 21): “Ekphrasis is the verbal representation of a real or fictitious text composed in a non-verbal sign system. This definition uses the term ‘text’ as it is used in most semiotic discourses” (Clüver 1997: 26). In 2019, Clüver would come to the realization that this and other broad definitions make ekphrasis “less precise and also less functional” (2019: 238), restating his definition thus: “Ekphrasis is the verbal representation of real or fictive configurations composed in a non-kinetic visual medium” (Clüver 2019: 239). An important takeaway from Clüver’s revision, without going too much into detail here, is the understanding that the nonverbal target of ekphrasis is a central and often contested point in ekphrasis debates. From a historical point of view, it is important to understand that the majority of academic publications dealing with literature’s negotiation of visual phenomena before the 1990s commonly adhered to a semiotic paradigm and worked within the framework of ut pictura poesis. Research on ekphrasis was done under the umbrella of Interart Studies and Comparative Arts and its creed of a “wechselseitige Erhellung der Künste” (the title of a talk given by Oskar Walzel at the Berlin Kant Society in 1917), that is, the complementarity of the arts and a principal transferability of and translatability between the art forms literature, art, music, and film (Weisstein 1992; Clüver 2007). Since the late 1980s, however, the traditional approach of Interart Studies or Comparative Arts, as well as semiotic approaches to art forms, were challenged by the academic discussions around the materiality and inter/mediality of cultural products (Rippl 2005). In this context, Gabriele Rippl

4

Ekphrasis: Intermedial and Anglophone Perspectives

55

(2005) has criticized semiotic conceptualizations of ekphrasis by focusing on inter/ medial and material considerations. Like Siglind Bruhn, who expanded ekphrasis to refer to any “representation in one medium of a real or fictitious text composed in another medium” (2000: 8), Rippl claims that ekphrasis may evoke verbally, visually, or musically any other (highbrow or popular) cultural phenomena and works of art that are done in another medium, as is the case, for example, in Cindy Sherman’s History Portraits in which the artist uses her body and photography to discuss Western painting. In the same way that the nonverbal target of ekphrasis is debated, the source medium in which ekphrasis is produced is also highly contested: some scholars would limit it to verbal representations (e.g., Clüver 2019), while others argue that ekphrasis can occur in music (e.g., Bruhn 2000), film (e.g., Pethő 2009, 2011), or other media. The popularity and increasing importance of intermediality studies and other related fields can be attributed to the fact that in our digital age, many works of art, cultural artifacts, literary texts, and other cultural configurations either combine and juxtapose different media, genres, and styles or refer to other media in a plethora of ways. Ever since the 1990s, intermediality studies have become an important research field within literary studies and beyond (Rippl 2015b). Renowned theoreticians of intermediality like Werner Wolf (2005), Irina O. Rajewsky (2005), as well as Jørgen Bruhn and Beate Schirrmacher (2022) differentiate between a wide range of intermedial phenomena, one of them being ekphrasis. In Rajewsky’s typology of intermedial phenomena, ekphrasis belongs to the category of “intermedial references” (2005: 52), for instance, references in a literary text to a piece of music (the so-called “musicalization of fiction,” Wolf 1999); the imitation and evocation of filmic techniques such as dissolves, zoom shots, montage editing, etc.; descriptive modes in literature, like ekphrasis, which evoke visual effects or refer to specific visual works of art; etc. Narratologist Werner Wolf also defines ekphrasis as an intermedial relation/reference where the involvement with the other medium takes place covertly/indirectly. For Wolf, as for many other scholars, the mere thematization of another medium is not enough, and so they reserve the term “intermedial relation/reference” for an evocation of certain formal features of another medium (what Rajewsky calls the “as if” character and illusion-forming quality of intermedial references; they create the illusion of another medium’s specific practices; Rajewsky 2005: 54–55). Bruhn, Gutowska, Tornborg, and Knust define ekphrasis as “an example of a transmediation between a source media product and a target media product, belonging to different media types that are structured differently on the level of the semiotic modality” (2022: 148). It is important to note that in any process of transmediation, “narratives and ideas can only be transferred across media by being transformed” and that “transmediation and media representation are like two sides of the same coin” (Bruhn and Schirrmacher 2022: 104, 105). Ekphrasis research that subscribes to an intermedial paradigm was certainly propelled by W. J. T. Mitchell’s claim that “all media are mixed media” (1995: 94–95). Moreover, his focus on the social nature of ekphrasis as developed in “Ekphrasis and the Other” (2004) is important. Mitchell highlights the fact that

56

S. Behluli and G. Rippl

ekphrastic texts are often saturated with gender hierarchies (1994: 151–181). As suggested by Mitchell, the paragone or contest/struggle/confrontation between the media text and image, which some ekphrases display, goes back to Lessing’s Laocoön (1766) and can be read as the equivalent of the hierarchical power relations at specific historical times in Western cultures. Many ekphrasis scholars have built their understanding of ekphrasis on Mitchell’s pioneering work. Like him, they link their discussions of ekphrasis not only to questions of inter/mediality and gender but bring to the fore ekphrasis’ entanglement with issues such as class and race. In this tradition, Liliane Louvel, Elizabeth Bergmann Loizeaux, David Kennedy (2012), Renate Brosch (2018a), as well as Birgit Neumann and Gabriele Rippl (2020) plead for a functional approach to ekphrasis that underscores the importance of ekphrastic practices and investigates the cultural, sociopolitical, and affective work ekphrasis does (cf. below for “Functions of Ekphrasis”). While not expanding the concept of ekphrasis, Stephen Cheeke’s study Writing for Art: The Aesthetics of Ekphrasis (2008) is another important contribution to the field of ekphrasis research since it highlights aesthetic and affective aspects of ekphrasis: “The kinds of formal, rhetorical or generic characteristics [of ekphrases] cannot be separated from the notion of a reader, who may also be in this case a viewer of the artwork (as the poet has also been a viewer)” (Cheeke 2008: 3). Cheeke’s affect-oriented aesthetics of ekphrasis takes us back to the root meaning of aisthesis, which in Greek means an act of perception and a sensory response. With respect to the cultural work of ekphrasis and in addition to Mitchell’s pioneering work, Elizabeth Bergmann Loizeaux’s 2008 book-length study, Twentieth-Century Poetry and the Visual Arts, is of great importance, even if her definition of ekphrasis is a narrow one: she considers an ekphrastic poem as “the poem that addresses a work of art” (2008: 1). Taking contextual issues into account, such as the “growing familiarity of works of art among a broad reading public,” she claims that “ekphrasis has both increased and tapped the cultural currency of the images it engages, and helped to shape the debate about them,” a debate “fraught with mixed emotions about images” (Loizeaux 2008: 4). According to Loizeaux, “[l]ooking is not, never has been, ethically neutral, and ekphrasis stages relations lived under that fact. [. . .] The inherently social dynamics of ekphrasis and its possibilities for polyvocality made it especially attractive to a postmodernism alive to the multiplicity of the lyric subject and to racial, ethnic and gender differences” (2008: 8–10). Loizeaux further argues that ekphrasis “often stages an engagement with the foreign” (2008: 11), which is why “[t]he ekphrastic poem is all about [. . .] otherness, and about how one engages it” (2008: 9). In The Ekphrastic Encounter in Contemporary British Poetry and Elsewhere (2012), David Kennedy opens his discussion of ekphrasis by stating that “a revision of some aspects of the dominant representational model of ekphrasis is long overdue” (1), e.g., the aspect of paragone and the concept of representation itself (2). Kennedy underlines the importance of community “in which and for which such paintings were produced” and demonstrates in how far earlier as well as contemporary British poets writing in the ekphrastic tradition have ignored detailed descriptions of the paintings and focused on ethical issues: “the articulation of peculiarly

4

Ekphrasis: Intermedial and Anglophone Perspectives

57

twentieth- and twenty-first-century anxieties. We fear permitting art to be transcendent because it reminds us that we are not” (2012: 2–3). Kennedy thus links ekphrasis to anthropological and ethical issues such as death, moral questions, as well as issues of history/temporality. Like Kennedy, Renate Brosch argues for a revision of the representational model and diagnoses an urgent need for a revival of rhetorical and performative understandings of ekphrasis “that can augment theoretical conceptualizations and bring them into line with the participatory and hybrid practices of ekphrasis today” (Brosch 2018a: 225). She favors a cognitive approach to ekphrasis that includes embodied cognitive reader response and the interrogation of ways of seeing (2018a: 225). Brosch’s cognitive approach (2015) allows for the investigation of the cognitive effects and functions of ekphrases. Her focus on cognitive processing and visualization during the reading of literary texts and the question of what textual elements intensify the visual imagination is interested in reception processes, attention management, and heightened creative reader participation: “On the level of the individual reading processes it aids comprehension, memory, and emotional response. These individual benefits can also have larger socio-cultural repercussions when they impact the formation of cultural memory” (Brosch 2015: 343). More recently, important ekphrasis research has also been taking place in connection with postcolonial and transcultural Anglophone literatures. Writers like Salman Rushdie employ ekphrastic strategies to discuss sociopolitical and politico-religious issues such as the possibility of secular postcolonial democracy in India, the right-wing ideology of Hindutva, and the politically enforced Indian religious fundamentalism in an age of neoliberal globalization, the new media, and the digital networks of the worldwide communications systems. His novels The Moor’s Last Sigh (1995) or The Enchantress of Florence (2008), which ekphrastically reimagine those periods in Indian history characterized by pluralistic and inclusive visions of community, are projects in “ekphrastic ethics” or “ethical ekphrasis” (Rippl 2015b, c). By means of ekphrasis, postcolonial literatures critically engage with the colonial imagery and the “politics of vision” (Emery 2007: 3). Precisely because the relation between word and image is frequently staged as a conflict, even an antagonism, ekphrasis is particularly suited to engage the reader affectively in meaning-making processes, to negotiate cultural conflicts, ongoing sociopolitical and epistemological struggles in a world characterized by global polycentric networks and transcultural linkages (Neumann and Rippl 2020). An additional promising future research field is “intermedial ecocriticism” (Bruhn 2021; cf. also Elleström 2020). Many contemporary Anglophone narratives and poetry negotiate climate change and other uncanny events of the Anthropocene with the help of the concept of “eco-ekphrasis” (Rippl 2019). The environmental aesthetics of Canadian and American novelists such as Don DeLillo (Underworld [1997]), Margaret Atwood (The Year of the Flood [2009]; cf. Rippl 2019), and Richard Powers (The Overstory [2018]; Bewilderment [2021]) is characterized by frequent use of intermedial references to and descriptions of visual works of art. Ecoekphrases are verbal evocations of (fictive or actual) works of art and other images or artifacts, which are dedicated to ecological topics. They discuss ecological issues

58

S. Behluli and G. Rippl

and take on important cultural-critical, ethical, and sociopolitical functions; after all, ekphrasis is a mode of writing, which showcases epistemological frameworks and cultural hierarchies (Rippl 2015b, 2018) and provides readers with a visceral imaginary experience of an unsustainable future environment. There exist a few approaches to ekphrasis in the digital age (López-Varela 2015) and to digital (that is, hypertextually encoded) ekphrases (cf. Behluli and Rippl 2017; Mathieson 2018) and their numerous multimedial constellations: as recipients, the digital natives are “no longer simply reading an ekphrastic poem but engaged in an activity of reading, viewing, and listening, whereby ekphrasis becomes part of a multisensory ‘event’” (Jansson 2018: 299). Interestingly, more and more scholars today refer back to antiquity to describe digital ekphrases and discuss the significance of the ancient rhetorical definitions of ekphrasis that underline immersive qualities, immediacy, and tactility produced by enargeia (cf. Lindhé 2013). As time goes on, further theories will have to be developed to fully conceptualize digital ekphrasis and uses of ekphrasis in the digital age.

Typologies of Ekphrasis Throughout its long history, ekphrasis has not only undergone several definitory changes but has also been divided into various typologies by scholars. While a strict categorization of ekphrasis is not possible due to its fluid, ever-changing nature, an overview of existing typologies is helpful. One of the first subdivisions of ekphrasis was proposed by John Hollander in 1988, who suggested a differentiation between “actual” and “notional” ekphrases. Hollander’s two categories are given by the target of the ekphrasis: an actual ekphrasis evokes images that (used to) exist in the world, whereas a notional ekphrasis describes images that are fictional. This distinction is valuable to determine whether an ekphrasis is more interested in referentiality or in invention. In the 1990s, Tamar Yacobi suggested a heuristic differentiation according to the length of the ekphrasis, pointing out that they can range from lengthy descriptions to abbreviated references and mere allusions to an “ekphrastic model” (1995; see also Robillard 1998). Furthermore, noting that Hollander’s typology only considers a “one-to-one relation (numerical as well as mimetic) [. . .] between the artworks” (Yacobi 2013: 601), Yacobi offers three additional image-to-word relationships. These four “ekphrastic relations” include (1) one-to-one between source and target (one image, one ekphrastic response), (2) one-to-many (one image, many ekphrases), (3) many-to-one (cumulative ekphrasis), and (4) many-to-many (multiple visual sources and multiple ekphrastic responses) (Yacobi 2013: 602–603). Yacobi claims that the first two relations have received the most critical attention and that this bias has prevented scholars from fully understanding more complex ekphrases, including “ekphrastic models” and “ekphrastic double exposures” (subcategories of the many-to-one relationship; cf. also Yacobi 2013). In 2008, Laura M. Sager Eidt offered a typology of four qualitative categories with “increasing degrees of complexity” (45). These four types apply to ekphrases in literature and film: (1) “attributive ekphrasis,” which is characterized by “the

4

Ekphrasis: Intermedial and Anglophone Perspectives

59

smallest degree of involvement with the visual arts” but can become complex when it “contribute[s] to the signification of the text or film, or to the characterization of the protagonists” (Sager Eidt 2008: 45, 46); (2) “depictive ekphrasis,” where “images are discussed, described, or reflected on more extensively” (Sager Eidt 2008: 47; this type comes closest to Heffernan’s definition); (3) “interpretive ekphrasis,” a highly complex category, which can manifest itself “either as a verbal reflection on the image, or a visual-verbal dramatization of it in a mise-en-scène tableau vivant” (Sager Eidt 2008: 50); and, finally, (4) “dramatic ekphrasis,” the most complex and self-reflexive type where “the images are dramatized and theatricalized to the extent that they take on a life of their own. Thus, this category is the most visual of all four, and has a high degree of enargeia” (Sager Eidt 2008: 56). These four ekphrastic categories, which are appealing as heuristic tools but seem difficult to keep apart when applied to interpretations of actual texts or films, “become progressively selfreflective and independent, the paragone also increases” (Sager Eidt 2008: 218). Similarly but more simplistically, Emma Kafalenos suggested in 2012 that we divide ekphrasis into “describing” and “narrativizing” types to capture their complexity and their functions. While Kafalenos’s differentiation has its value, it is well-known by now that a “purity” of description and narration cannot be upheld and that, in addition, the notion of “mimesis” needs to be regarded with suspicion. Liliane Louvel has established influential typologies of ekphrasis at two different points of her career, first in her book Poetics of the Iconotext (2011; first published in French in 1998) and more recently in an article titled “Types of Ekphrases” (2018). In Poetics, she offers a sophisticated theory and typology of the myriad fusions of literary texts and the range of images they contain – ekphrasis being only one of them. Indeed, Louvel describes a whole spectrum of text/image relations according to pictorial saturation. This “typology of the pictorial” positions ekphrasis at the far end: (1) the painting-effect, (2) the picturesque view, (3) hypotyposis, (4) the tableau vivant, (5) the aesthetic arrangement, (6) the pictorial description, and finally (7) ekphrasis (Louvel 2011: 89–100). In Poetics of the Iconotext, Louvel states that ekphrasis “provides the highest degree of pictorialization of the text” (2011: 98), and in “Types of Ekphrases,” she further subdivides it into 11 categories. Louvel’s typology features the following types of ekphrasis: (1) narrativizing ekphrasis, (2) hermeneutic ekphrasis, (3) maieutic ekphrasis, (4) subversive (debunking) ekphrasis, (5) anachronistic ekphrasis, (6) monumental or elegiac ekphrasis, (7) affective ekphrasis, (8) pragmatic or technical ekphrasis, (9) creative ekphrasis, (10) art criticism and second-order ekphrasis, and (11) heuristic ekphrasis (Louvel 2018: 249–258). Louvel’s categories are influenced by the various functions ekphrases can fulfill (cf. “Functions of Ekphrasis” below) and sometimes overlap with the previously mentioned classifications. Following in the footsteps of Cecilia Lindhé (2013) and Johanna Hartmann (2015), both of whom revaluate ekphrasis against the backdrop of the digital age, Behluli and Rippl (2017) have identified four modes of ekphrastic writing that respond to the digital age: (1) “auraticizing ekphrases,” which place value on original, unique, material objects over (digital) copies; (2) “Wikipedia-inspired ekphrases,” which reproduce parts of Wikipedia entries to “debunk the notion of

60

S. Behluli and G. Rippl

originary ekphrasis” and exhibit that knowledge production and information retrieval “rely increasingly on global digital networks” (Behluli and Rippl 2017: 150); (3) “ekphrases born digital,” which further heighten the web of referentiality and collaboration between words and images in literary texts produced with and for digital media (cf. “digital born,” Hayles 2008); and (4) “critical ekphrases,” which are employed by scholars and critics “to convey the object under inspection to their readers,” especially when describing the complex word/image configurations found in electronic literature (Behluli and Rippl 2017: 166). Like every other typology that was laid out here, Behluli and Rippl’s budding classification of ekphrasis in the digital age has to be viewed with a critical eye and challenged when reading literary texts. More often than not, literary examples of ekphrases are borderline cases that resist simple categorization. Indeed, this fluidity and complexity is what makes ekphrasis so interesting and, as a mode of speaking and writing more broadly, so lasting. Instead of getting stuck on rigid typologies that should only be taken as heuristic guidelines, therefore, a stronger emphasis on ekphrastic functions is recommendable.

Functions of Ekphrasis While there are manifold ways to think critically about ekphrases, one of the more fruitful avenues is to reflect on their functions. Indeed, most scholars who have written about ekphrastic phenomena have asked, in one way or another, what they exactly do, what work they perform, or what functions they fulfill. The plethora of functions that critics have identified in response to these questions cover a vast range “from admonitory, moral, and instructive [...] to the production of presence and evidence, the creation of illusion of the fictive world, and the production of affects and emotions” (Rippl 2019, 2020: n. pag.). In its original function, when ekphrasis was used in ancient rhetoric, its primary function was to make “the listener ‘see’ the subject in their mind’s eye” (Webb 2009: 2) and thus convince them of the speaker’s arguments by turning them into a sort of “eyewitness.” In other words, the ekphrastic quality of enargeia (or “vividness”) was harbored as a rhetorical strategy for persuasion. This ekphrastic function clearly stresses the effects on the listeners (and readers) over the described object itself: “in the ancient definition the referent is only of secondary importance; what matters [. . .] is the impact on the listener” (Webb 2009: 7). It is important to stress here that the emotional intensity, which ekphrasis produces, is a crucial aspect of this “impact on the listener” and instrumental in the goal of persuasion. In spite of photoshop, social media filters, and other new technologies, the associations between sight and trust have survived to this day (to ‘see’ is to believe), which is why the persuasive function of ekphrasis is still important in modern usage. This can be seen, for example, in John Banville’s novel The Book of Evidence (1989), in which the protagonist wants to persuade the reader of a painting’s spellbinding force, which drove him to kill a servant girl: “You do not know the fortitude and pathos of her [the painting’s] presence. You have not come upon her suddenly in a golden room in a summer eve, as I have. You have not held

4

Ekphrasis: Intermedial and Anglophone Perspectives

61

her in your arms, you have not seen her asprawl in a ditch. You have not – ah no! – you have not killed for her” (Banville 1989: 75). In spite of the comedic undertone of this scene, which arises from the delusional personification of the painting as a real woman, the protagonist’s use of ekphrasis to persuade his readers is evident: if you could only see this painting with your own eyes, you would also kill for it. The next two functions we want to mention here often go hand in hand: ekphrasis halts time and creates presence. The former arises out of a by-now outdated dichotomy between narration and description and believes that ekphrasis, as a subcategory of description, “pauses” the narrative. This goes back to an ancient rhetorical tradition, as Murray Krieger points out, when ekphrasis was “a device intended to interrupt the temporality of discourse, to freeze it during its indulgence in spatial exploration” (Krieger 1992: 7). The latter ekphrastic function of creating presence profits from this interruption of narrative temporality to evoke a quasispatial “presence.” Both functions spring out of a Lessingian tradition that connects the linguistic sign system with time and the logic of succession on the one hand and the visual sign system with space and the logic of simultaneity on the other. In “Why Ekphrasis?” (2007), for example, Valentine Cunningham states that ekphrasis is a “pausing, in some fashion, for thought” (57). This pause for thought is required to create a sort of presence within the immaterial realms of language: “The ekphrastic encounter seeks [. . .] to resolve this ancient and continuing doubting by pointing at an allegedly touchable, fingerable, thisness” (Cunningham 2007: 61). David Kennedy makes a similar argument when he says that ekphrasis is “an attempt to bring art into the realm of our contingency” (2012: 6). Kennedy understands this linguistic present-making not primarily through the physicality of the object but through that of language itself: “we are intimately involved with language in a physical sense. It comes out of our mouths; we handle it when reading a book or a newspaper; and we move it about when we make a Word document” (Kennedy 2012: 6). Clearly, the ekphrastic functions of halting time and creating presence must be viewed as attempted approximations to spatial materiality – what W. J. T. Mitchell calls “ekphrastic hope” (1994: 157) – rather than as actualities. Here is a passage from Hanya Yanagihara’s queer novel A Little Life (2015), which is taken from a gallery exhibition scene toward the end of the novel, where we see this ekphrastic yearning for pause and presence: The rest of the floor is given over to images from “The Boys,” and he walks slowly through the rooms, looking at pictures of Malcolm, of him, of Willem. Here are the two of them in their bedroom at Lispenard Street, both of them sitting on their twin beds, staring straight into JB’s camera, Willem with a small smile; here they are again at the card table, he working on a brief, Willem reading a book. Here they are at a party. Here they are at another party. Here he is with Phaedra; here Willem is with Richard. (Yanagihara 2015: 676)

In this passage, so close to the novel’s end, the protagonist Jude looks at paintings of himself, his deceased partner Willem, and his friends JB and Malcolm. The anaphoric use of “here is / here are / here they are / etc.” signals that, although nothing is “happening” on the level of plot, this passage demands the reader’s special attention:

62

S. Behluli and G. Rippl

here, ekphrasis functions to convey a yearning for presence – both of the deceased partner and the material artworks. The passage from the end of Yanagihara’s novel compresses numerous paintings into one ekphrasis because the scene is set at an artist’s retrospective and, more importantly, because this is a metafictional moment in which the novel looks back at its own story. Analogous to how Jude looks at these paintings as a summary of his lived life, readers can understand this scene as the novel summarizing its own plot so far. This introduces another ekphrastic function that has been highlighted by scholars such as James A. W. Heffernan, Elizabeth Bergmann Loizeaux, and Liliane Louvel: a poetic, decorative or self-reflexive function. In this usage, ekphrasis emphasizes the artificiality, mediality, or “nature of representation itself” (Heffernan 1991: 304). Indeed, on top of everything that ekphrasis does, it always also points at itself. Louvel views this ekphrastic look back at itself as a form of aesthetic narcissism: “Embedded in the text and holding up a mirror to it, art plays its reflexive and narcissistic role to the full, interweaving otherness with sameness. In the duality of the infinite dialogue, the image in the text fully takes on the role of commentary on the work, on art in general, and on writing in particular” (Louvel 2011: 127). By evoking images and visual artifacts, ekphrasis reveres its own verbal roots. We can observe these playful, self-reflexive tendencies in the ekphrasis from A Little Life above. Proving that one ekphrasis can perform multiple functions at once, though, we can also spot a highlighted visualization and creation of reference in the passage. The advocate of the former function is Renate Brosch. She views intermedial references such as ekphrases as a “textual strategy for eliciting highlighted visualization” (Brosch 2015: 343). Ekphrases intensify visualization and support imagination. In line with a Shklovskian understanding of perception as something that easily becomes habitual and automated, Brosch thus understands ekphrases as items in text or speech that can defamiliarize readers and listeners. She foregrounds cognitive aspects by stating that ekphrasis “can serve to unsettle conventional schemata of representation,” trigger cognitive participation and “aid [. . .] comprehension, memory and emotional response” (Brosch 2015: 343). All of this is true for the ekphrastic passage from A Little Life above: the evocations of numerous images, which have been produced throughout the past few decades stimulate the reader’s memory and facilitate their recollection of the story they have read so far. Birgit Neumann (2015) has also stressed the alliance between ekphrasis and memory, claiming that ekphrasis is connected to cultural memory and the Western canon, social imaginaries, and postcolonial trauma. It is also because of this deep link to memories that ekphrasis manages to produce strong emotional responses in readers and listeners. The other ekphrastic function of creating a reference goes in a similar direction: ekphrases can establish connections to people, places, events, and other items elsewhere in the story or, importantly, outside of the fictional realm. This function is especially potent when the ekphrasis is an “actual” one, that is, of a real image in the world: “The referential context serves to create the illusion of reality, to authenticate and ‘anchor’ the text within a ‘real’, easily identifiable space and time for the reader; hence its connection with one of the functions of rhetoric: persuasion”

4

Ekphrasis: Intermedial and Anglophone Perspectives

63

(Louvel 2011: 103). Ekphrastic “reference[s] to the extra-text,” argues Liliane Louvel, thus serve to lend authenticity to the text and further strengthen its referential illusion (2011: 103). Here is an ekphrastic passage from Donna Tartt’s novel The Goldfinch (2013) to exemplify this referential function through which ekphrasis can anchor a text or speech within history: We spent some time in front of a Hals portrait of a boy holding a skull (“Don’t be mad, Theo, but who do you think he looks like? Somebody” – tugging the back of my hair – “who could use a haircut?”) – and, also, two big Hals portraits of banqueting officers, which she told me were very, very famous and a gigantic influence on Rembrandt. [. . .] I followed after her with a sort of dazed sense of lost time, delighted by her preoccupation, how oblivious she seemed of the minutes flying. (Tartt 2013: 26–27)

In this early scene, the protagonist, Theo, and his mother walk around an exhibition at the Metropolitan Museum of Modern Art and engage with Dutch art from the Golden Age. Although the titles of the evoked paintings are not explicitly mentioned, a simple Google search identifies them as Frans Hals’s Young Man Holding a Skull (Vanitas) (1626–8), Banquet of the Officers of the St George Militia Company (1616), and Banquet of the Officers of the Calivermen Civic Guard (1627). By hinting towards these canonical paintings, Donna Tartt grounds her novel in the “real” world. She describes images that everyone could see with their own eyes by googling them or traveling to their host galleries. In addition to creating a reference to the extra-textual world, two further ekphrastic functions can be found in this short passage: an educational and a revelatory one. The educational function can be comprehended in a straightforward sense, where the description of an image is also didactic. When Theo’s mother evokes the Frans Hals paintings above, for example, she is also communicating knowledge from art history by explaining that these particular images were influential for Rembrandt van Rijn. Elizabeth Bergmann Loizeaux explains that this “tutelary function of ekphrasis [is] present in the tradition since Philostratus tutored his young charge by showing him the family art collection” (2008: 23). This function rests on the belief that “the image has a lesson the ekphrasis delivers” (Loizeaux 2008: 23). In the same fashion that the educational function of ekphrasis can impart important historical details and lessons, the revelatory function turns to the image as a source of new or significant information. Simply put, ekphrasis can reveal new aspects about the story – “It subtly transgresses the treatment of time of the narrative when it proleptically plays a foreshadowing role or analeptically explains the past” (Louvel 2011: 118) – and it can convey subtle details about characters – it is an “‘indirect’ mode of characterization, which traditionally supplies a lot of clues about a character’s preferences, sensibilities, or biases” (Louvel 2011: 119). This revelatory function is important in the passage from The Goldfinch above, where the protagonist’s link to death (“boy with a skull”) and his fondness for his mother (he follows “dazed” and “delighted”) are expressed implicitly. This revelatory function also points to the deep alliance between ekphrastic descriptions and narrative. Indeed, another important function ascribed to ekphrasis

64

S. Behluli and G. Rippl

is that of generating narrative. Heffernan has laid out this argument most clearly in Museum of Words (1993), where he states that ekphrasis develops out of a “pregnant moment” (a mistranslation from the German prägnant): “The ‘pregnant moment’ of action is the arrested point which most clearly implies what came before the moment and what is to follow it. [. . . Ekphrasis] typically delivers from the pregnant moment of visual art its embryonically narrative impulse, and thus makes explicit the story that visual art tells only by implication” (Heffernan 1993: 5). Loizeaux has also stressed this ekphrastic impulse “to deliver the story out of the single moment so often depicted in western painting and sculpture” (2008: 22). Heffernan’s and Loizeaux’s statements reveal that they might not have had abstract art in mind when writing, as abstract art arguably does not tell a story or represent something concrete. Since ekphrases of abstract art do, in fact, exist, we have to take this narrative-generating function with a pinch of salt. We can also view this hyperfocus on representative, narrative images as a consequence of one of the most important ekphrastic functions: the paragonal one. In accordance with the paragone tradition, which pits poetry against painting (most notably by Lessing in 1766), this ekphrastic function accentuates the contest for dominance between word and image. As Heffernan states, ekphrasis “evokes the power of the silent image even as it subjects that power to the rival authority of language, it is intensely paragonal” (1993: 1). According to this logic, the word is in constant rivalry with the “silent” image; it evokes the image to dominate it. Some ekphrases – for example, Walter Pater’s 1873 description of the Mona Lisa and the “graceful mystery” and “secrets” he sees behind her smile – manage to shape the painting’s interpretation so lastingly that Stephen Cheeke calls them “parasitic,” “vampiric,” and “deadly” (Cheeke 2008: 177, 182). Originally, W. J. T. Mitchell also saw ekphrasis primarily in the light of paragone (he later revised this opinion to stress the mutually beneficial interaction of different media). He understood the contest between the arts to go far beyond a friendly rivalry and into the spheres of the political and cultural: ekphrasis “expose[s] the social structure of representation as an activity and a relationship of power/knowledge and desire – representation as something done to something, with something, by someone, for someone” (Mitchell 1994: 180). Ekphrasis stages this power struggle between word and image. This paragone discourse has birthed a slew of new research on ekphrasis that relates to its ethical, political, and sociocultural functions. These functions are significantly driven by W. J. T. Mitchell’s chapter on “Ekphrasis and the Other” in Picture Theory (1994). Stating that there are no neutral forms of representation, Mitchell demonstrates that ekphrastic descriptions stage an encounter with “the other” and thereby thematize power relations: “The ‘otherness’ of visual representation from the standpoint of textuality may be anything from a professional competition (the paragone of poet and painter) to a relation of political, disciplinary, or cultural domination in which the ‘self’ is understood to be an active, speaking, seeing subject, while the ‘other’ is projected as a passive, seen, and (usually) silent object” (Mitchell 1994: 157). Mitchell’s notion of the powerless “other” includes children, disabled people, colonized subjects, and women, to name just a few. Critics have further developed this ekphrastic function of staging an encounter with

4

Ekphrasis: Intermedial and Anglophone Perspectives

65

otherness to discuss a variety of hierarchies and ekphrastic strategies to level them out. Indeed, ekphrasis has thus come to play an ethical function: “Whether and how one can speak for others gets to the center of the ethical questions ekphrasis raises” (Loizeaux 2008: 24; cf. also Rippl 2015b). Numerous approaches, such as feminist, ecocritical, and transcultural ones, have harbored this ethical potential of ekphrasis to discuss asymmetric power relations. In her study on Twentieth-Century Poetry and the Visual Arts (2008), Elizabeth Bergmann Loizeaux takes the feminist function of ekphrasis seriously and investigates how female poets “put ekphrasis to feminist use, rewriting the social and epistemological dynamics implied in the notion of an active, masculinized word and a passive, feminized image” (25). Similarly, David Kennedy argues that there is “a distinctive female ekphrastic mode” that renegotiates “cultural norms of sexuality, gendered identity and male and female subjectivities” (2012: 90; cf. also Hedley, Halpern and Speigelman 2009). More recently, Sofie Behluli demonstrates that “Anglo-American women novelists use the work of art as an imaginary site to investigate the frictions between the personal and the political; between private experiences of selfhood, grief, trauma, abuse and sexism on the one hand, and public mechanisms of aesthetic value, artistic canon and institutional power on the other” (2021b: 3). As Behluli shows, women authors employ the ekphrastic staging of the gaze to make feminist interventions in the ways cultural value is produced and disseminated. Carol Anne Duffy’s poems “Pygmalion’s Bride” (1999) and “Standing Female Nude” (1985), which stage Galatea and the female nude talking back to Pygmalion and the male artist, respectively, are two famous examples that illustrate this. This idea of “talking back” is further taken up by Malaika Sutter, who also focuses on the feminist function of ekphrasis (cf. forthcoming article on the Tiny Pricks Project). Ekphrasis has also come under new scrutiny against the backdrop of the digital age, which, due to the ready availability of pictures on the Internet and hypertextually encoded literary texts/electronic literature, has come to fulfill new functions. One of them is to reflect on the relationship between originals and copies, as the technologies to replicate, disseminate, and even fake artworks have become much more sophisticated. Indeed, Liliane Louvel has noted that many literary texts nowadays “pose the question of falsehood and forgery, which is more directly linked to aesthetics and its presuppositions as well as to the world in which art manifests itself and is used” (Louvel 2011: 123). When texts such as Donna Tartt’s The Goldfinch (2013) or Michael Frayn’s Headlong (1999) juxtapose artworks with digital or even printed reproductions, they often favor the original over the copy. In an earlier article on “Ekphrasis in the Digital Age” (2017), we have identified this ekphrastic strategy as an auraticizing function. Derived from Walter Benjamin’s concept of “aura” (1935), which refers to a certain aesthetic quality that arises from an artwork’s inimitable authenticity, the auraticizing function of ekphrasis negotiates and redistributes cultural value: “it helps to create a uniqueness and intensity in the reading process which lingers with readers and satisfies their wish for originality in a world dominated by copy-and-paste techniques and ubiquitously present reproduction” (Behluli and Rippl 2017: 143–144). Viewing ekphrasis in conjunction with

66

S. Behluli and G. Rippl

aura is a particularly effective way to highlight its embodied, tactile, and affective effects. While these effects were crucial in the ancient uses of ekphrases, as critics such as Cécilia Lindhé (2013) and Jonna Hartmann (2015) have pointed out, they have gained new importance in the era of digitality, technology, and hyperreality. As this selective list of ekphrastic functions has shown, the abilities of ekphrasis to take on various shapes and produce innumerable effects are manifold. Covering a broad range of functions from the critical – ekphrasis as a mode “rewriting the artwork and in the process questioning accepted meanings, values, and beliefs, not just relating to the particular artwork in question but referencing the ways of seeing and the scopic regimes of the culture at large” (Brosch 2018a: 225) – to the enchanting – ekphrasis as a mode to “restore a sense of wonder to a world that has bowed down for much of the twentieth century to abstraction and theory” (Hepburn 2010: 16) – ekphrasis proves to be one of the most enduring and malleable modes of speaking and writing. Throughout nearly three millennia of cultural and technological developments, ekphrasis has not only survived but expanded its status in writing and speech. As long as words and images coexist, ekphrasis has no expiration date.

Conclusion Ekphrasis is one of the most adaptable, multifunctional, and dynamic modes of speaking and writing. It allows readers and listeners to “see” the tiny, perfect details of an object – the world in miniature, so to speak – and to imagine sights that could never exist in reality. In other words, ekphrasis channels attention and expands the imagination. Although it is most commonly viewed as a subcategory of description, ekphrasis must be placed on a continuum between description and narration. It occurs in oral and written communication, covering a wide range from traditional media such as books and newspapers to newer media such as podcasts and YouTube videos. Scholars of ekphrasis have studied it in poetry (Heffernan 1993; Loizeaux 2008; Hedley, Halpern and Speigelman 2009; Kennedy 2012), prose texts (Hepburn 2010; Karastathi 2015; Neumann and Rippl 2020; Behluli 2021b), film (Marcus 2007; Pethő 2009, 2011), music (Bruhn 2000), and architecture (Vieira 2020), to name just a few genres. Ever since its first recording in ancient Greek texts, ekphrasis has been a stable component of communication throughout all cultural periods – both in fiction and in non-fiction. In fact, ekphrasis has easily adapted to the recent increase in blurred boundaries between media, genres, disciplines, and ontological boundaries. Ekphrasis arises from, responds to, and reinforces these border-transgressing tendencies. Jaś Elsner, for instance, has argued that art history “is nothing other than ekphrasis, or more precisely an extended argument built on ekphrasis” (2010: 11). We have proposed that scholars of ekphrasis are often also practitioners of ekphrasis – especially in the context of digital or electronic literature (Behluli and Rippl 2017: 166); and that ekphrasis is increasingly employed, for example, by authors of life writing, to soften the divide between what is real and what is fictional (Behluli 2020, Forthcoming). This can be seen in explorative texts written by authors such as Ben Lerner, Olivia

4

Ekphrasis: Intermedial and Anglophone Perspectives

67

Laing, Laura Cumming, and many others. Ekphrasis is thus not only an intermedial phenomenon – located between different media and fostering ties across them – but it also actively creates borderlands, where fiction and non-fiction flow into each other; one genre melts into another; and disciplinary approaches are replaced by transdisciplinary thought. As already stated above, there is no foreseeable end to the millennia-long reign of ekphrasis. However, we may wonder about the shapes and functions it will take on, especially in response to Artificial Intelligence. Poems and novels that were fully created by AI software have already been published, and they are likely to become more sophisticated as time goes on. What does this mean for the future of ekphrasis? Can something as creative, affective, and meaningful as “looking” attentively at the world be replicated by computers? If yes, which sights will they draw our attention to, and which ones will they obscure and omit? We can envisage long books written by AI that are only ekphrasis (“What would Moby-Dick as pure ekphrasis look like?”), and ekphrastic descriptions of objects and creatures that no human has ever seen, imagined, or described before.

References Atwood, Margaret. 1996. Alias Grace. Toronto: McClelland & Stewart. Banville, John. 2014 [1989]. The book of evidence. London: Picador. Barbetti, Claire. 2011. Ekphrastic medieval visions: A new discussion of Interarts theory. New York: Palgrave Macmillan. https://doi.org/10.1057/9780230370531. Behluli, Sofie. 2021a. Framing lives as paintings. Women: A Cultural Review 32 (2 (June)): 119–139. https://doi.org/10.1080/09574042.2021.1914901. ———. 2021b. Women writing novels on art: Ekphrasis in twenty-first century fiction. Dissertation. University of Oxford. Oxford University Research Archive. ———. Forthcoming. Entangled lives: The use of visual art in contemporary life narratives. In Life writing, creativity, and the social in the Americas, ed. Wilfried Raussert and Susana Rocha Teixeira. Behluli, Sofie, and Gabriele Rippl. 2017. Ekphrasis in the digital age. In Digitalität und literarische Netz-Werke. Antikanon 2, ed. Christina Hoffmann and Johanna Öttl, 131–176. Wien: Turia + Kant. Benjamin, Walter. 1999 [1935]. The work of art in the age of mechanical reproduction. In Illuminations: Essays and reflections, 11–32. London: Pimlico. Boehm, Gottfried, and Helmut Pfotenhauer, eds. 1995. Beschreibungskunst – Kunstbeschreibung: Ekphrasis von der Antike bis zur Gegenwart. Munich: Fink. Brosch, Renate. 2015. Images in narrative literature: Visibility – Visualization – Description. In Handbook of intermediality: Text – Image – Sound – Music, De Gruyter handbook series, ed. Gabriele Rippl, 343–360. Berlin/Boston: De Gruyter. ———. 2018a. Ekphrasis in the digital age: Responses to image. Poetics Today 39 (2 (June)): 225–243. https://doi.org/10.1215/03335372-4324420. ———, ed. 2018b. Special issue Contemporary Ekphrasis of Poetics Today 39 (2 (June)). Bruhn, Siglind. 2000. Musical Ekphrasis: Composers responding to poetry and painting. Hillsdale: Pendragon Press. Bruhn, Jørgen. 2021. Towards an intermedial ecocriticism. In Beyond media borders, ed. Lars Elleström, vol. 2, 117–148. Cham: Palgrave Macmillan. https://doi.org/10.1007/978-3-03049683-8_5.

68

S. Behluli and G. Rippl

Bruhn, Jørgen, and Beate Schirrmacher, eds. 2022. Intermedial studies: An introduction to meaning across media. London/New York: Routledge. Bruhn, Jørgen, Anna Gutowska, Emma Tornborg, and Martin Knust. 2022. Transmediation. In Intermedial studies: An introduction to meaning across media, ed. Jørgen Bruhn and Beate Schirrmacher, 138–161. London/New York: Routledge. Bugno, Dominika, and Miriam de Paiva Vieira. 2020. Unveiling Ekphrasis on screen. In Revista VIS: Revista Do Programa De Pós-Graduação Em Artes Visuais, 19(2): 48–69. https:// periodicos.unb.br/index.php/revistavis/article/view/35393. Cheeke, Stephen. 2008. Writing for art: The aesthetics of Ekphrasis. Manchester: Manchester University Press. Clüver, Claus. 1997. Ekphrasis Reconsidered: On verbal representations of non-verbal texts. In Interart poetics. Essays on the interrelations of the arts and media, edited by Ulla-Britta Lagerroth, Hans Lund and Erik Hedling, 19–33. Amsterdam: Rodopi. ———. 2007. Intermediality and interarts studies. In Changing borders: Contemporary positions on intermediality, ed. Jens Arvidson, Mikael Askander, Jørgen Bruhn, and Heidrun Führer, 19–37. Lund: Intermedia Studies Press. ———. 2019. On Gazer’s encounters with visual art: Ekphrasis, readers, ‘Iconotexts’. In Ekphrastic encounters: New interdisciplinary essays on literature and the visual arts, ed. David Kennedy and Richard Meek, 237–256. Manchester: Manchester University Press. Cunningham, Valentine. 2007. Why Ekphrasis? Classical Philology 102 (1 (January)): 57–71. https://doi.org/10.1086/521132. Denham, Robert, D. 2010. Poets on paintings: A bibliography. London: McFarland. Eckstein, Lars. 2005. Ekphrastic memory in David Dabydeen’s A Harlot’s progress and the politics of aestheticist transfiguration. In Revisiting slave narratives I: Les avatars contemporains des récits d’esclaves, ed. Judith Misrahi-Barak, 327–348. Montpellier: Presses universitaires de la Méditerranée. Elleström, Lars. 2020. Representing the anthropocene: Transmediation of narratives and truthfulness from science to feature film. Ekphrasis 2 (December): 36–48. https://doi.org/10.24193/ ekphrasis.24.3. Elsner, Jaś. 2010. Art history as Ekphrasis. Art History 33 (1 (January)): 10–27. https://doi.org/10. 1111/j.1467-8365.2009.00720.x. Emery, Mary Lou. 2007. Modernism, the visual, and Caribbean literature. Cambridge: Cambridge University Press. Ferreira, Joicy Silva, and Miriam de Paiva Vieira. 2020. From brushes to lenses: An Ekphrastic stroll. In A line made by walking, Lublin studies in modern languages and literature, ed. Sara Baume, vol. 44, no. 2, 51–62. Lublin: Marie Curie Sklodowska University Press. Fjellestad, Danuta. 2015. Nesting – braiding – weaving: Photographic interventions in three contemporary American novels. In Handbook of intermediality, edited by Gabriele Rippl, 193–218. Berlin, Boston: De Gruyter. Hagstrum, Jean H. 1958. The sister arts: The tradition of literary Pictorialism and English poetry from Dryden to Gray. Chicago: The University of Chicago Press. Hartmann, Johanna. 2015. Ekphrasis in the age of digital reproduction. In Handbook of intermediality, ed. Gabriele Rippl, 113–127. Berlin/Boston: De Gruyter. Haselstein, Ulla. 2019. Gertrude Steins literarische Porträts. Göttingen: Wallstein. Hayles, N. Katherine. 2008. Electronic literature: New horizons for the literary. Notre Dame: University of Notre Dame. Hedley, Jane, Nick Halpern, and Willard Speigelman, eds. 2009. In the frame: Women’s Ekphrastic poetry from Marianne Moore to Susan Wheeler. Newark: University of Delaware Press. Heffernan, James A.W. 1991. Ekphrasis and representation. New Literary History: A Journal of Theory and Interpretation 22 (1 (Spring)): 297–316. https://doi.org/10.2307/469040. ———. 1993. Museum of words: The poetics of Ekphrasis from Homer to Ashbery. Chicago: University of Chicago Press.

4

Ekphrasis: Intermedial and Anglophone Perspectives

69

Hepburn, Allan. 2010. Enchanted objects: Visual art in contemporary fiction. Toronto: University of Toronto Press. Hollander, John. 1995. The Gazer’s spirit: Poems speaking to silent works of art. Chicago: University of Chicago Press. Jansson, Mats. 2018. Ekphrasis and digital media. Poetics Today 39 (2 (June)): 299–318. https:// doi.org/10.1215/03335372-4324468. Johnston, Andrew James, Ethan Knapp, and Margitta Rouse, eds. 2015. The art of vision: Ekphrasis in medieval literature and culture. Columbus: Ohio State University Press. Kafalenos, Emma. 2012. The polysemy of looking: Reading Ekphrasis alongside images in Calvino’s Castle of crossed destinies and Vargas Llosa’s In praise of the stepmother. Poetics Today 33 (1 (March)): 27–57. https://doi.org/10.1215/03335372-1505531. Karastathi, Sylvia. 2015. Ekphrasis and the novel/narrative fiction. In Handbook of intermediality, ed. Gabriele Rippl, 92–112. Berlin/Boston: De Gruyter. Kennedy, David. 2012. The Ekphrastic encounter in contemporary British poetry and elsewhere. Farnham: Ashgate. ———. 2015. Ekphrasis and poetry. In Handbook of intermediality: Text – Image – Sound – Music, ed. Gabriele Rippl, 82–91. Berlin/Boston: De Gruyter. Kennedy, David, and Richard Meek, eds. 2019. Ekphrastic encounters: New interdisciplinary essays on literature and the visual arts. Manchester: Manchester University Press. Klarer, Mario. 2001. Ekphrasis: Bildbeschreibung als Repräsentationstheorie bei Spenser, Sidney, Lyly und Shakespeare. Berlin/Boston: De Gruyter. Koelb, Janice Hewlett. 2006. The poetics of description: Imagined places in European literature. New York: Palgrave Macmillan. Krieger, Murray. 1967. Ekphrasis and the still movement of poetry; or Laokoon revisted. In The poet as critic, ed. Frederick P.W. McDowell, 3–26. Evanston: Northwestern University Press. ———. 1992. Ekphrasis: The illusion of the natural sign. Baltimore: Johns Hopkins University Press. Lessing, Gotthold Ephraim. 1984. Laocoön: An essay on the limits of painting and poetry. Translated by Edward Allen McCormick. Baltimore: The Johns Hopkins University Press. ———. 1985. Laocoon, or on the limits of painting and poetry. In German aesthetic and literary criticism: Winkelmann, Lessing, Hamann, Herder, Schiller, Goethe, ed. H.B. Nisbet, 55–133. Cambridge: Cambridge University Press. Lindhé, Cecilia. 2013. ‘A visual sense is born in the fingertips’: Towards a digital Ekphrasis. DHQ: Digital Humanities Quarterly 7 (1). http://www.digitalhumanities.org/dhq/vol/7/1/000161/ 000161.html. Accessed 29 Jan 2023. Loizeaux, Elizabeth Bergmann. 2008. Twentieth-century poetry and the visual arts. Cambridge/New York: Cambridge University Press. López-Varela Azcárate, Asunción. 2015. Transmedial Ekphrasis. From analogic to digital formats. International Journal of Transmedia Literacy 1 (1): 45–66. Louvel, Liliane. 2011. Poetics of the Iconotext, introduction by Karen Jacobs and translation by Laurence Petit. Farnham: Ashgate. ———. 2018. Types of Ekphrasis: An attempt at classification. Poetics Today 39 (2 (June)): 245–263. https://doi.org/10.1215/03335372-4324432. Marcus, Laura. 2007. The tenth muse: Writing about cinema in the modernist period. Oxford: Oxford University Press. Mathieson, Jolene. 2018. The body and the possible soul in digital Ekphrastic poetry. Poetics Today 39 (2 (June)): 359–382. https://doi.org/10.1215/03335372-4324505. Michael Sledge. 2010. The more I owe you. New York: Counterpoint. Mitchell, W.J.T. 1994. Picture theory: Essays on verbal and visual representation. Chicago: University of Chicago Press. Neumann, Birgit, and Gabriele Rippl. 2020. Intermedial aesthetics: Verbal-visual configurations in postcolonial literature. London: Routledge.

70

S. Behluli and G. Rippl

Pater, Walter. 2020 [1873]. The renaissance: Studies in art and poetry. Oxford: Oxford University Press. Pethő, Ágnes. 2009. Media in the cinematic imagination: Ekphrasis and the poetics of the in-between in Jean-Luc Godard’s cinema. In Media borders, multimodality and intermediality, ed. Lars Elleström, 211–225. Basingstoke: Palgrave Macmillan. ———. 2011. Cinema and intermediality. The passion for the in-between. Cambridge: Cambridge Scholars Publishing. Rajewsky, Irina O. 2005. Intermediality, Intertexuality, and remediation: A literary perspective on intermediality. Intermédialité 6 (Autumn): 43–64. https://doi.org/10.7202/1005505ar. Rippl, Gabriele. 2005. Beschreibungs-Kunst: Zur intermedialen Poetik angloamerikanischer IkonTexte (1880–2000). Munich: Fink. ———, ed. 2015a. Handbook of intermediality: Text – Image – Sound – Music. Edited by Gabriele Rippl. Berlin/Boston: De Gruyter. ———. 2015b. Postcolonial Ekphrasis in the contemporary Anglophone Indian novel. In Handbook of intermediality: Text – Image – Sound – Music, ed. Gabriele Rippl, 128–155. Berlin/ Boston: De Gruyter. ———. 2015c. Description and the production of presence: Literary debates in eighteenth-century England and Germany. In Präsenz und Evidenz fremder Dinge im Europa des 18. Jahrhunderts, ed. Birgit Neumann, 458–474. Göttingen: Wallstein. ———. 2018. The cultural work of Ekphrasis in contemporary anglophone novels. Poetics Today 39 (2 (June)): 265–285. Poetics Today 39 (2 (June)): 225–243. https://doi.org/10.1215/ 03335372-4324420. ———. 2019a. Sustainability, eco-ekphrasis and the ethics of literary description. In Cultural sustainability, ed. Torsten Meireis and Gabriele Rippl, 221–232. London: Routledge. ———. 2019b. Ekphrasis. In Oxford research encyclopedia of literature. Oxford: Oxford University Press. https://doi.org/10.1093/acrefore/9780190201098.013.1057. Robillard, Valerie. 1998. In pursuit of Ekphrasis (an intertextual approach). In Pictures into words: Theoretical and descriptive approaches to Ekphrasis, ed. Valerie Robillard and Els Jongeneel, 53–72. Amsterdam: VU University Press. Sager Eidt, Laura M. 2008. Writing and filming the painting: Ekphrasis in literature and film. Amsterdam/New York: Rodopi. Sutter, Malaika. Forthcoming. Hate speech in threads: Stitching and posting a resistance in the Tiny pricks project. Discourse: Journal for Theoretical Studies in Media and Culture. Tartt, Donna. 2013. The goldfinch. St. Ives: Abacus. ten Kortenaar, Neil. 1997. Postcolonial Ekphrasis: Salman Rushdie gives the finger back to the empire. Contemporary Literature 38 (2): 232–259. VanderMeer, Jeff. 2021. Hummingbird Salamander. New York: Picador. Vieira, Miriam de Paiva. 2011. Ekphrasis in Girl with a pearl earring. Scripta Uniandrade 9 (2): 11–29. ———. 2020. Architectural Ekphraseis: Unveiling a Brazilian wall-less house in contemporary fiction. In Transmediations: Communication across media borders, ed. Niklas Salmose and Lars Elleström, 118–135. New York: Routledge. Wagner, Peter, ed. 1996. Icons – Texts – Iconotexts: Essays on Ekphrasis and intermediality. Berlin: De Gruyter. Webb, Ruth. 1999. Ekphrasis ancient and modern: The invention of a genre. Word & Image 15 (1): 7–18. https://doi.org/10.1080/02666286.1999.10443970. ———. 2009. Ekphrasis, imagination and persuasion in ancient rhetorical theory and practice. Farnham: Ashgate. Weisstein, Ulrich. 1992. Literatur und Bildende Kunst. Ein Handbuch zur Theorie und Praxis eines komparatistischen Grenzgebietes. Berlin: Erich Schmidt. Wolf, Werner. 1999. The musicalization of fiction: A study in the theory and history of intermediality. Amsterdam/Atlanta: Rodopi.

4

Ekphrasis: Intermedial and Anglophone Perspectives

71

———. 2005. Intermediality. In Routledge encyclopedia of narrative theory, ed. David Herman, Manfred Jahn, and Marie-Laure Ryan, 252–256. London/New York: Routledge. Yacobi, Tamar. 1995. Pictorial models and narrative Ekphrasis. Poetics Today 16 (4 (Winter)): 599–649. https://doi.org/10.2307/1773367. ———. 2013. Ekphrastic double exposure and the museum book of poetry. Poetics Today 34 (1–2 (June)): 1–52. https://doi.org/10.1215/03335372-1894487. Yanagihara, Hanya. 2015. A little life. London: Picador.

5

Intermediality and Medium Specificity Jarkko Toikkanen

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 1: Medium as Essence or Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 2: Classical Views – The Sister Arts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 3: Lessing and Burke – Poetry over Painting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 4: Kantian Aesthetic Judgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 5: Greenbergian Essentialism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 6: McLuhan and Barthes – Messages and Meanings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 7: Early 2000s – Hypertext and Visual Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Section 8: Recent Debates on Transmediality and Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion: New Applications in Intermediality Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

74 75 75 77 78 80 82 84 86 89 91

Abstract

This chapter surveys the historical development of a key concept in intermediality research in the West: medium specificity. From its classical origins, the discussion progresses through the early modern period to the modern period and the present day. The timeline is divided into sections that focus on the key ideas concerning medium specificity, either directly or with crucial relevance, both in the historical period in question and the influence of the ideas on later periods. The sections are constructed around quotations from original and secondary sources interpreted and explained to forge a somewhat logical narrative on the tale of medium specificity through time. In a fundamental fashion, the historical journey has to do with this question: Is “medium” a physical thing fixed with essential qualities that set it apart from all other media and so make it “specific,” or is it a concept to be understood in a more active sense, not as a fixed thing but as a changing set of practices by which we interact with the physical world in specific ways? J. Toikkanen (*) University of Oulu, Oulu, Finland e-mail: jarkko.toikkanen@oulu.fi © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_13

73

74

J. Toikkanen

Keywords

Intermedial · Medium Specificity · Medium · Transmedial · Experience

Introduction For a long time now, medium specificity, alternatively rendered as medium specificity or media specificity, has been a much-debated concept in art and literary studies, as well as philosophy and media theory. It has its roots in critical ideas dating back to the Antiquity, and it has been passed down to the present day through many variations, some of which may not always have discussed the concept explicitly as “medium specificity,” but which nonetheless have influenced subsequent estimations of its significance and relevance across the range of views on the nature of media. This handbook chapter is devised in a chronological fashion in which, after introducing the topic in general, the origins of the concept of medium specificity are located in classical authors including Horace, Quintilian, and Philostratus. From them the discussion leaps forward by more than a good millennium to the closing stages of the early modern period with Lessing, Burke, and the rise of aesthetics. The transcendental idealism of Kant and its impact on the nineteenth century follows next. By then the history will have come to the point where Clement Greenberg begins to talk explicitly about the medium-specific qualities of Modernist painting. In the subsequent decades, the leverage of the concept is expanded from art research to media theory and cultural studies through influential figures such as Marshall McLuhan and Roland Barthes. At the turn of the 2000s, there are further twists in the tale as visual studies branch out into novel directions and interdisciplinary literary critics such as N. Katherine Hayles go from the medium specificity of physical objects in the real world – printed books, for instance – to digital environments where electronic hypertexts possess specific material qualities of their own. The journey through tradition eventually reaches a tentative conclusion in recent debates on the theoretical concepts of transmediality and adaptation, as well as some applications of medium specificity in English and literary studies that constitute my background in intermediality research. The discussion on the progressing timeline is divided into sections that focus on the key ideas concerning medium specificity in the West, either directly or with crucial relevance, both in the historical period in question and the influence of the ideas on later periods. At the level of text, the sections are constructed around quotations from original and secondary sources, interpreted and explained to forge a somewhat logical narrative on the tale of medium specificity through time. I definitely expect my audience to find flaws in the adopted approach and to disagree with any readings of the sources involved, but if this is the result, the chapter will surely have been a success and meet its brief when it leads to further talk and continued work on the concept of medium specificity. In the included list of references, there are a hundred sources to help in doing that.

5

Intermediality and Medium Specificity

75

Section 1: Medium as Essence or Practice Over the last couple of decades, Noël Carroll has made repeated declarations against the merit of medium specificity as a theoretical concept or an idea with practical worth in philosophy and art studies and most recently in an article succinctly titled “Medium Specificity” (2019). Clotilde Torregrossa took up Carroll’s famous maxim from 2003, “Forget the medium!”, as motivation for her 2020 study on medium specificity in film studies, to argue against Carroll’s denunciation of the concept for improved uses of it in the footsteps of contemporary critics such as Berys Gaut (2010). In a fundamental fashion, the conflict described here, and in the historical journey to follow, has to do with this question: Is “medium” a physical thing fixed with essential qualities that set it apart from all other media and so make it “specific,” or is it a concept to be understood in a more active sense, not as a fixed thing but as a changing set of practices by which we interact with the physical world in specific ways? Torregrossa (2020: 45) puts the question and her case against Carroll: If the unique constituents of a medium are set independently of any concrete instance of the medium, then an artist wishing to work within a particular medium must respect these constituents and focus exclusively on them. Pushing this idea a little further, we have to accept that an artist’s own use or practice of a medium has no determining power on the medium. Either the artist respects the unique constituents of the medium and produces an artwork instantiating the medium, or she does not respect these constituents and what she produces does not instantiate the medium she intended to work with. This second scenario is problematic precisely because the work produced is not a work of art as it intended to be. This point is crucial in Carroll’s overall critique of the medium specificity claim.

According to Torregrossa, for Carroll, it is useless to talk about medium specificity if an artwork does not “focus exclusively” on the specific qualities of the art medium in question, whether film or something else, and instead the artwork calls on qualities such as narrativity (Torregrossa 2020: 48) that are available in other kinds of art media too. In contrast, for Torregrossa, the benefit of medium-specific research is about focusing on how a film specifically employs narrativity, or another quality shared across art media, through “an artist’s own use or practice of a medium” (Torregrossa 2020: 45) – not on what is exclusive to film and available nowhere else. It is a feud between media as essences or practices, and I will explain how it has historical roots from classical and early modern views on the arts to new turns and applications of the concept of medium specificity across the disciplines. The strife is also relevant to the topical debate on the distinction between intermediality and transmediality as discussed in “Section 8: Recent Debates on Transmediality and Adaptation” near to the end of the journey.

Section 2: Classical Views – The Sister Arts As Judith Harvey (2002) claims, Horace (2022) would have been aware of the discussion preceding him, starting with Simonides of Ceos in the sixth century BCE, when he introduced the phrase “ut pictura poesis” (“as is painting, so is

76

J. Toikkanen

poetry”) in Ars poetica a couple of decades before the common era. The weight of Horace’s observation lies on understanding painting and poetry as “sister arts,” to suggest a term used much later by John Dryden in the preface to his 1695 English translation of Charles Alphonse du Fresnoy’s 1668 poem De arte graphica. Horace’s key point is that painting and poetry are endowed with the same essential qualities, and the artwork is there for pleasure, regardless of the specific medium in which the object is either viewed or read, and the effect may vary based on the degree of attention given to the artwork. Whereas Horace may not go into detail between the effects of painting and poetry, and how exactly observing an artwork might help analyze different replications of reality, Quintilian (2022) indeed goes this way in Institutio oratoria at the end of the first century. As a rhetorician and orator after Seneca, Quintilian is interested in practicing the verbal arts, but he also draws comparison between his efforts and those of authors involved with painting and sculpture (Book XII, Chap. 10): But these different kinds of work, of which I speak, are not merely the product of different authors, but have each their own following of admirers, with the result that the perfect orator has not yet been found, a statement which perhaps may be extended to all arts, not merely because some qualities are more evident in some artists than in others, but because one single form will not satisfy all critics, a fact which is due in part to conditions of time or place, in part to the taste and ideals of individuals.

In shifting the focus from the sister arts possessing qualities for pleasure that would stay essentially the same across the media, Quintilian trains his attention on the medium-specific causes of changes in pleasure between the works of different authors and the preferences of their audience members. Not only does he look for the distinctions between the verbal, visual, and plastic arts, but he also discusses differences of effect within kinds of verbal arts including spoken and written language. Highlighting the subjective experience of individuals, and why not everyone will have the same experience of an artwork despite the essential qualities allegedly shared or the degree of attention given to the object, Quintilian foreshadows the rise of modern aesthetics in the eighteenth century. The idea of the differences of effect within the verbal arts, and their power to compel readers and listeners into imagining sensory perceptions that might be, for instance, visual or auditory, was an important part of Quintilian’s rhetoric (see Webb 2009). A century later, Philostratus the Elder (2022) the Elder wrote Imagines that, in Norman Bryson’s words, remains “our most extensive account of what a Roman picture gallery, a Roman catalogue of pictures, and the Roman viewing of pictures may have been like” (Bryson 1995: 255, see Lehmann 1941). Bryson explains how Imagines became a prolific source of inspiration in the modern period, and the appeal was found in “the promise contained in the idea of resurrection” (Bryson 1995: 256), of bringing the past to life through the “strange, hallucinatory power” of descriptions that evoked “the capacities of words and pictures to describe the world” and created “another space where presence is alive to all the senses at once (sight, hearing, touch, taste)” (Bryson 1995: 273). The rhetorical device to achieve this intermedial effect in practice was ecphrasis, or ekphrasis, classically defined as poetry about paintings.

5

Intermediality and Medium Specificity

77

Studying works in the verbal arts with ekphrasis shows how, while poetry and painting both tell stories and create imagined realities, they do so in their own ways. The fact that reading words evokes sensory perceptions of other media does not signal a lack of specificity in the verbal medium because it is only the words of Imagines that produce the effect Bryson describes.

Section 3: Lessing and Burke – Poetry over Painting The debate on the Horatian “ut pictura poesis” in the Renaissance and postRenaissance era spread out from Italy and seminal figures such as Leonardo da Vinci across Europe. Many critics sustained the argument the visual and plastic arts could provide superior pleasure, thanks to their immediacy and naturalness, when compared to works in the verbal arts whose effect lagged because of the interpretive labor involved (Harvey 2002; see Lee 1940). With G. E. Lessing’s Laocoon (1766), not only were the tables turned on which art was preferred, but the theoretical underpinnings of the sister arts as essentially sharing the same qualities were put under scrutiny, contributing to the rise of modern aesthetics in Germany together with Alexander Baumgarten’s Aesthetica (1750). Lessing (2005: 92) defines the difference between painting and poetry in a key passage: Painting, in its coexistent compositions, can use but a single moment of an action, and must therefore choose the most pregnant one, the one most suggestive of what has gone before and what is to follow. Poetry, in its progressive imitations, can use but a single attribute of bodies, and must choose that one which gives the most vivid picture of the body as exercised in this particular action.

Lessing’s sympathies clearly lie on the side of poets because of their unique ability, or genius, in their verbal art “of so combining negative with positive traits as to unite two appearances in one” (2005: 60) that is impossible to painters who cannot simultaneously visualize two contradictory appearances in their static depictions – for a single work, the painter is forced to choose one. The range of describing conflicting actions and emotions is much broader for the poet who, if they are sufficiently skilled in their way with words, may evoke more complex and powerful experiences. (For an early twentieth-century comparison of the arts, see Babbitt (1910). For an application of the intermedial distinction between the classical rhetorical concepts of enargeia and energeia in relation to this issue, see Toikkanen (2013: 104).) The significant turn in Lessing, however, does not have to do with the territorial contest between painting and poetry, but the fact that the practice of one art over another may not be sourced in questions of attention and pleasure, but it can be recognized as medium-specific difference. In Britain, Edmund Burke published A Philosophical Enquiry into the Origin of Our Ideas of the Sublime and Beautiful in 1757 to produce a conceptually and

78

J. Toikkanen

empirically convincing account of the causes of sublime or beautiful experience in encountering either nature or art. He makes a strong case for poetry and verbal arts not dissimilar to Lessing (Part 5, Sect. 7): Now, as words affect, not by any original power, but by representation, it might be supposed, that their influence over the passions should be but light; yet it is quite otherwise; for we find by experience, that eloquence and poetry are as capable, nay indeed much more capable, of making deep and lively impressions than any other arts, and even than nature itself in very many cases.

The three causes by which Burke explains the special capabilities of poetry exhibit the superiority of words to “express all the circumstances of most passions,” to make a lasting impression on the reader with “many things” that “can seldom occur in the reality,” and to “make such combinations as we cannot possibly do otherwise” (emphasis in the original; see Virtanen and Toikkanen 2020). All of Burke’s three causes can be collated with Lessing on the range of verbal representation in contrast to visual depiction, and the third one – on how words create simultaneous combinations of conflicting actions and emotions to evoke strong experiences – makes it obvious how Lessing and Burke share the aesthetic concern with medium-specific difference. It is the practice of working with and responding to words that matters, not the essential qualities of an art or medium as such.

Section 4: Kantian Aesthetic Judgment Although Immanuel Kant might not be the first name on the list of famous thinkers of medium specificity, the impact and influence of his philosophy, known as transcendental idealism, is too great to ignore. On the side of his scientific and philosophical commitments, Kant stayed close to aesthetic concerns that eventually became a substratum for his system of transcendental idealism in the form of the third Critique, or the Critique of the Power of Judgment (1790). The third Critique was written to complete the trilogy with the analytical foundation outlined in the Critique of Pure Reason (1781/1787) and the moral principles set in the Critique of Practical Reason (1788). The third Critique builds on questions very similar to Burke’s on the sublime and beautiful, and how aesthetic judgments that rely on subjective experiences of taste may be systematically grounded and validated as objective evidence of the philosophical system in action. On that note, Kant is critical of Burke for engaging in “empirical psychology, which only with difficulty could ever lay claim to the rank of a philosophical science” (2000: 38), even when their eyes are drawn towards the same phenomena. What Kant sets out to establish are the rules of presenting thoughts and intuitions in one medium or another. How is it possible to ensure the connections made between concepts such as beauty and morality have objective validity, instead of being a matter of individual taste? In practice, Kant employs the rhetorical device of hypotyposis – which Quintilian defined as “any representation of facts which is

5

Intermediality and Medium Specificity

79

made in such vivid language that they appeal to the eye rather than the ear” (Book IX, Chap. 2) – in support of this principle (2000: 25; see Toikkanen 2015): All hypotyposis (presentation, subjecto sub adspectum), as making something sensible, is of one of two kinds: either schematic, where to a concept grasped by the understanding the corresponding intuition is given a priori; or symbolic, where to a concept which only reason can think, and to which no sensible intuition can be adequate, an intuition is attributed with which the power of judgment proceeds in a way merely analogous to that which it observes in schematization, i.e., it is merely the rule of this procedure, not of the intuition itself, and thus merely the form of the reflection, not the content, which corresponds to the concept.

The passage may flummox the unaccustomed reader. On the one hand, Kant is saying that when someone hears or reads the word tree, or sees a tree, and understands what a tree is based on their knowledge, a schematic hypotyposis will have been successfully completed. A direct connection between the sensible presentation and the concept is established. On the other hand, when someone hears or reads the word beauty, and sees or imagines seeing something, the connection between the sensible presentation and the concept is indirect, and the success of the symbolic hypotyposis must be ensured by other means. What I understand as beautiful based on my knowledge may not be your understanding at all, even when we are hearing, reading, or seeing the same thing – and so Kant must write hundreds of pages of transcendental philosophy to confirm the validity of his procedure. (For more on the Kantian hypotyposis as a rhetorical device in contrast to ekphrasis and their intermedial distinction as enargeia and energeia, see Toikkanen (2013: 36–42).) How is the Kantian enterprise related to the concept of medium specificity, and what is the scope of his influence on later thought? The last half of the passage – in which aesthetic judgment ultimately founds “merely the form of the reflection, not the content” in making connections between sensible presentations and concepts – is a telling conclusion. Reflecting on anything requires a medial form that may be auditory, verbal, or visual, and whereas the same content could be reflected on in any of these medial forms, it does not follow that the resulting understanding might be even similar. Ergo, there is nothing essential about what is presented since the Kantian power of aesthetic judgment falls on the specific practices of how things are presented in medial forms (see de Man 1996; Gasché 2003). During the Romantic period, many authors would strive to articulate, in the Wordsworthian idiom, “the spontaneous overflow of powerful feelings” (1800) where it becomes the duty of the poet to express what is pure about their own experience as they wish to speak to the humanity in general. It could be argued the Romantics, in their own way, perfected the art of forsaking the idea there was anything objective about things found out there, in nature, since everything was somehow intuited and presented as understood by the individual subject. As the post-Kantian author refuses direct connections between works of art and established concepts, new worlds are figured forth for poets and other artists to express themselves with authenticity through the media at their disposal. At the same time, as this activity can manifest in any medium and still be primed towards the same end, it is no longer so much concerned with the specific qualities of a particular art than with

80

J. Toikkanen

the essential idea of self-expression that transcends medium-specific difference. The Victorian critic John Ruskin did not idolize the Romantics, but he too would subscribe to the philosophy of the primacy of ideas before the arts in Modern Painters, Vol. III (1848, Part IV, Chap. 1): I come, after some embarrassment, to the conclusion, that poetry is “the suggestion, by the imagination, of noble grounds for the noble emotions.” I mean, by the noble emotions, those four principal sacred passions – Love, Veneration, Admiration, and Joy (this latter especially, if unselfish); and their opposites – Hatred, Indignation (or Scorn), Horror, and Grief,
this last, when unselfish, becoming Compassion. These passions in their various combinations constitute what is called “poetical feeling,” when they are felt on noble grounds, that is, on great and true grounds. (§13) It is only farther to be noticed, that infinite confusion has been introduced into this subject by the careless and illogical custom of opposing painting to poetry, instead of regarding poetry as consisting in a noble use, whether of colors or words. Painting is properly to be opposed to speaking or writing, but not to poetry. Both painting and speaking are methods of expression. Poetry is the employment of either for the noblest purposes. (§15)

For Ruskin, as arguably for the Romantics before him, poetry was not to be mainly understood as an art medium with specific qualities, but instead as an essential idea that figured forth “the noble emotions” and “poetical feeling.” Beyond an author’s use of “colors or words” in the practice of either painting, speaking, or writing poetry, the first and foremost criterion of their achievement was whether, through their “methods of expression,” they had been successful in employing them “for the noblest purposes.” As the concept of medium specificity enters the discussion explicitly in the Modernist period with Clement Greenberg and others, this nineteenth-century intellectual legacy traced through Kant, Wordsworth, and Ruskin is to be kept in mind about the primacy of ideas expressed over the media used to express them.

Section 5: Greenbergian Essentialism In Towards a Newer Laocoön (1940), Greenberg soon confirms the claim advanced above: To preserve the immediacy of the feeling it was even more necessary than before, when art was imitation rather than communication, to suppress the role of the medium. The medium was a regrettable if necessary physical obstacle between the artists and his audience, which in some ideal state would disappear entirely to leave the experience of the spectator or reader identical with that of the artist.

Greenberg adds that “painting suffered most at the hands of the Romantics” (1940: 299). The thrust of the painter’s argument is about spelling out how to return from expressing “ideas and notions” to expressing “with greater immediacy sensations, the irreduceable elements of experience” (1940: 303) along with the contemporary avant-garde and Modernist movements. It should not do for the practicing

5

Intermediality and Medium Specificity

81

artist, in painting or else, to merely think of their medium as something that stands in the way of the ideas they wish to convey, but instead the artist seeking “purity” should become aware “of the limitations of the medium of the specific art” and intentionally submit and begin to make use of those conditions: “It is by virtue of its medium that each art is unique and strictly itself. To restore the identity of an art the opacity of its medium must be emphasized” (1940: 305). It must be noted that Greenberg, in spite of his insistence on discovering what creates “the identity of an art,” does not here provide a catalogue of the essential qualities the arts should exploit to be recognized as successful works in medium specificity. (Only much later does he claim, in the 1962 essay “After Abstract Expressionism,” that “flatness” is among the definitive criteria of painting.) Instead, Greenberg says that such identity can only be found in “the opacity of its medium” – that is, in something that cannot be explained away. If there is an essence, it may not be known what it is. As I have argued, this lack is significant since “the irreducible experience of a specific medium remains” (Toikkanen 2020: 74) even when the experience cannot be tied to any particular quality of the medium. Greenberg adds that painting and sculpture, through their forms, are able “to affect the spectator physically,” whereas the medium of poetry or literature is “essentially psychological and sub- or supralogical” (1940: 305), but he does not make a checklist for achieving such results. It could be argued that, in 1940, Greenberg comes across as a very strange case of essentialism in which the essences are nothing but blanks in the artist’s consciousness as they work on their chosen medium with the task of intermedially affecting their audience: The poet writes, not so much to express, as to create a thing which will operate upon the reader’s consciousness to produce the emotion of poetry. The content of the poem is what it does to the reader, not what it communicates. [. . .] Painting and sculpture can become more completely nothing but what they do; like functional architecture and the machine, they look what they do. The picture or statue exhausts itself in the visual sensation it produces. There is nothing to identify, connect or think about, but everything to feel. Pure poetry strives for infinite suggestion, pure plastic art for the minimum. (1940: 306–307, emphasis in the original)

It is Greenberg’s view that poetry – and Modernist literature as “an impossible ideal” of separating “the poem as a unique object” from “referents outside the poem” – puts the mind endlessly at work, or “the reader’s consciousness to produce the emotion of poetry.” In contrast, painting and sculpture are immediately effective in their bare naturalness. On readers and viewers, Greenberg reiterates the Horatian view that the poem or painting is essentially there for pleasure, with the effect depending on the degree of attention to the “content” which is defined as what art “does” to the audience. For the practicing artist, however, the situation is very different as they must not – as is the exclamation raised by Greenberg against consumerist catering in “Avant-Garde and Kitsch” (1939) – churn out pleasurable works of just any sort. They need to become aware of the blanks at the core of their chosen medium and attach to them, even without knowing what the blanks might be.

82

J. Toikkanen

As the range of art media had been broadening since the 1800s, Walter Benjamin’s thoughts on photography can provide a productive contemporary analogue. In “The work of art in the age of mechanical reproduction” (1936), he claimed photography and film as revolutionary technologies that deprived the traditional arts, painting, and poetry among them of their unique sense of mystery and individual expression. Benjamin sought to politicize artistic representation by exposing its artificiality and context dependence, and he understood photography as the predecessor to film. Photographs could be worked on by technological means that manipulated reality, potentially for political purposes, even if the photographic item as a work of art was thought to represent reality as such and brought out details that might have otherwise gone unnoticed. When the item represented something other than “a human face,” such as landscape or milieu, it would lose its mysterious “cult value” and become historical evidence reproducible by medium-specific technological means. With Benjamin’s attention trained on collective sociopolitical issues beyond concerns with individual expression, the set of practices available to the artist was at once expanded through the new media at their disposal, as well as restricted due to the essential limitations of the technological devices and the kind of content that was intermedially brought to the fore in the works produced by them.

Section 6: McLuhan and Barthes – Messages and Meanings As the story goes, the Modernist and formalist schools of the first half of the twentieth century were occupied with the specific qualities of art media and the unique works of art produced. Since the 1950s, with the widening impact of the linguistic turn after Saussure and Wittgenstein, the focus began to shift from observing particular works towards establishing universal rules for how any work was constructed in the first place, out of what kinds of linguistic elements, and according to what kinds of cultural conventions. This systematic shift in critical attention would become known as structuralism. One of the main achievements of the change was to provide theories and methods of inquiry applicable across the sciences, from the study of languages, culture, and the arts to media studies, sociology, anthropology, education, and beyond. I will explore what happens to the concept of medium specificity in Marshall McLuhan and Roland Barthes, accounting for their influence. In Understanding Media (1964: 8–9), McLuhan makes this claim about electric light as a medium: When the light is being used for brain surgery or night baseball is a matter of indifference. It could be argued that these activities are in some way the “content” of the electric light, since they could not exist without the electric light. This fact merely underlines the point that “the medium is the message” because it is the medium that shapes and controls the scale and form of human association and action.

5

Intermediality and Medium Specificity

83

It strikes the reader that McLuhan, in reflecting on a technological medium such as electric light, turns away from understanding its significance from having to do with its use or “content.” It is not in enabling brain surgery or night baseball that the medium-specific qualities of electric light can be found because, as McLuhan continues, “the content or uses of such media are as diverse as they are ineffectual in shaping the form of human association.” The media critic’s attention is instead trained specifically on the “character of the medium” such as electric light as it affects the ways by which human beings associate and interact with one another. It consequently appears that McLuhan’s eye is indeed drawn towards the medium as such, but there is a somewhat surprising twist that both recalls the Greenbergian view and transcends it: The electric light escapes attention as a communication medium just because it has no “content”. And this makes it an invaluable instance of how people fail to study media at all. For it is not till the electric light is used to spell out some brand name that it is noticed as a medium. Then it is not the light but the “content” (or what is really another medium) that is noticed. The message of the electric light is like the message of electric power in industry, totally radical, pervasive, and decentralized. (1964: 9)

Whereas Greenberg stressed the essential “opacity” of the medium, together with the artist unaware of what made up their chosen medium at its core, McLuhan’s technological medium of electric light remains an essentially unconscious cultural phenomenon. While the medium enables diverse forms of human interaction, it constitutes an epistemological cipher, apart from the “message” it keeps sending out. Structurally the medium exists as blank matter that is severed from specific practices, turned into something that is everywhere yet uncontrolled and ungoverned except commercially. It is not too straining to spot the similarities between McLuhan’s ideas and those of Roland Barthes who, effectively, theorized the reality in which we live as a “text.” In the 1971 essay “From Work to Text,” Barthes defines the work as “a fragment of substance” (1977: 156) such as a book on the library shelf, while the “Text” (with a capital T) constitutes “a methodological field” (1977: 157). Basically, regardless of the art medium instantiated in a case being observed, he understands the work as a physical object and the text as the set of universal rules that structure the object. This understanding, however, does not reduce the text into anything abstract or unchanging: The Text is not a co-existence of meanings but a passage, an overcrossing; thus it answers not to an interpretation, even a liberal one, but to an explosion, a dissemination. The plural of the Text depends, that is, not on the ambiguity of its contents but on what might be called the stereographic plurality of its weave of signifiers (etymologically, the text is a tissue, a woven fabric). (1977: 159)

Similar to McLuhan’s electric light, the specific medium that enables communication, here likened to “an explosion,” holds no essential value but exists, in practice, to send forth meanings. However, in discussing the textuality of art media, ranging

84

J. Toikkanen

from the traditional to more recent forms, along with issues of contemporary consumer culture, Barthes maintains the specifically material aspects of the communication process – the Text is not something that is only structured by the mind but, quite literally, a coming together of many voices whom none may claim to control (1977: 164): The discourse on the Text should itself be nothing other than text, research, textual activity, since the Text is that social space which leaves no language safe, outside, nor any subject of the enunciation in position as judge, master, analyst, confessor, decoder. The theory of the Text can coincide only with a practice of writing.

The voices that speak and become visible through the Text are real voices in the world who are sometimes reading, sometimes writing, regardless of the medium with which they have chosen to work. Therefore, on the level of art media, Barthesian theory is essentially non-medium specific, exposing the privileging of any particular art as a historically politicized idea (see Arnheim 1974; Williams 1977). Then again, on the level of humans interacting in the world, the theory is attuned to the specific practices that engage with our senses differently. When Barthes famously describes the experience of being moved by a photograph as the “punctum,” or “this wound, this prick, this mark made by a pointed instrument” (1981: 26), it becomes intimated that mediating the Text is more than cerebral play. It is an intermedial aesthetic and cultural practice that is designed to affect the body. Although Diarmuid Costello only refers to Barthes in footnotes in discussing Michael Fried’s (1998) and Stanley Cavell’s (1971) post-Greenbergian views on painting and photography, his conclusion is very Barthesian as he claims medium specificity as “a function of the structures of intention underwriting a given practice” (2008: 311). It is not about the art medium as such, nor the specific practice enabled by the art medium for use by the individual. It is about the set of universal rules and collectively available conventions that structure what artists and audiences go looking and feeling for in objects coming into being as the cultural upshot of given artistic practices.

Section 7: Early 2000s – Hypertext and Visual Culture In “Print is Flat, Code is Deep” (2004), N. Katherine Hayles starts with recognizing the impact of the Barthes’s essay discussed above – she is “struck both by its prescience and by how far we have moved beyond it” (2004: 68). While Hayles is impressed with Barthes’s foresight in calling for “a movement away from works to texts” that has reduced the attention on medium-specific terms to do with “print culture” (2004: 68) such as books, she is looking for change in the new millennium on the trail of McLuhan and the topical theories of Jerome McGann (1991) and Jay David Bolter and Richard Grusin (2000): “Perhaps now, after the linguistic turn has

5

Intermediality and Medium Specificity

85

yielded so many important insights, it is time to turn again to a careful consideration of what difference the medium makes” (2004: 68). Hayles seeks to develop MSA (Media-Specific Analysis) for the study of “literary hypertext”: Attuned not so much to similarity and difference as to simulation and instantiation, MSA moves from the language of “text” to a more precise vocabulary of screen and page, digital program and analogue interface, code and ink, mutable image and durably inscribed mark, texton and scripton, computer and book. (2004: 69)

The attention of the media-specific analyst is trained on what Klaus Bruhn Jensen has defined as “material intermediality” (2016: 4–7), or the aspects of media products that can be distinguished from the “discursive” interpretations of their content and the “institutional” conditions by which media are organized as consumer environments in society. For Hayles, analyzing the specifically material qualities of digital media against and together with what they have in common with non-digital media, such as shared generic features, will help study “how medium-specific constraints and possibilities shape texts” that “must always be embodied to exist in the world” (2004: 69). The medium specificity of electronic hypertexts may not be found in their essential qualities as digital objects, but the ways in which the digital environments are used in creating media products for the new millennium: “Materiality always matters in some sense, but it matters most to humanists and artists when considered in relation to the practices it embodies and enacts” (2004: 72). The view expressed is firmly lodged in the tradition of medium specificity as a fashion of doing things. Another field of inquiry that raised the stakes on media-related matters, after W. J. T. Mitchell’s call for a “pictorial turn” (1994) in the early 1990s, was visual culture, or visual studies. Arriving at the 2000s, some of the most significant works since John Berger’s Ways of Seeing (1972) were written, including the first editions of Marita Sturken and Lisa Cartwright’s Practices of Looking (2001) and Gillian Rose’s Visual Methodologies (2001). For Dirk J. van den Berg, much concern was placed on the historical and contemporary effects of “ocularcentrism,” or “the shape of logocentrism in the field of visuality.” In the spirit of Foucault (1973) and Lyotard (1984), to combat “the pervasive ‘picturalising’ of non-pictorial image categories” that created “ideology-infected and embodied networks of images” (2004), a politicized understanding of the potential specificities and non-specificities of media in culture, whether visual or verbal, was required. To quote Sturken and Cartwright (2009: 4): It is important to keep in mind that in any group that shares a culture (or set of processes through which meaning is made), there is always a range of meanings and interpretations “floating about”, so to speak, with regard to any given issue or object at any given time. Culture is a process, not a fixed set of practices or interpretations.

It could be argued that, in order to bring out the dynamic nature of contemporary media culture, Sturken and Cartwright rehearse the Barthesian view of the critical eye turned away from the art medium as such, as well as from specific artistic

86

J. Toikkanen

practices. (For further elaboration on this idea of medium in visual culture, and the problems and solutions raised for intermediality research, see Toikkanen (2013: 33–36).) Instead, their attention is trained on the cultural “processes” of meaning making essentially conditioned by the conventions and resources available, or “floating about,” at the time and the place. Notably, the processes do not materialize as the result of individual work or self-expression, since “meanings are produced not in the heads of the viewers so much as through a process of negotiation among individuals within a particular culture” (2009: 4, emphasis in the original). You can only make use of what is out there. In focusing exclusively on the discursive processes of making meaning within a culture, in which the specific medium assumes less than a significant role, what seems to be lacking in Sturken and Cartwright, in contrast to Barthes and Hayles, is a decisive concern with materiality. As “meanings and interpretations” hover around, seemingly as Platonic ideas, and there is no specificity between how works of art or media formats actually materialize and mediate experience, what remains is a thoroughly idealist account of cultural production that is arguably the reverse image of Romantic idealism. The visual cultural perspective, as explained here, has indeed to do with the propagation (or “negotiation”) of traditional ideas such as expression and authenticity, but to the end of radically disbanding them by displacing the source from the individual subject to the collective culture. Further sources on relevant studies around the 1990s and 2000s include Cheeke (2008), Clüver (1997), Collins (1991), Heffernan (1993), Jay (2005), Jones (2005), Krauss (2009), Krieger (1992), Mirzoeff (1999), Soules (2002), Thorburn and Jenkins (2003), Wagner (1996), and Yacobi (1997).

Section 8: Recent Debates on Transmediality and Adaptation Among the most active recent debates involving talk on the topical relevance of medium specificity has been the one on transmediality. Building on Henry Jenkins’s concept of transmedia storytelling (2008), or how the same stories may be told and franchised across media formats and platforms, theorists such as Jan-Noël Thon and Lars Elleström have endeavored to develop the methods of transmedial narratology and concepts of transmediation in general to distinguish the elements and characteristics that traverse media boundaries. To use Thon and Marie-Laure Ryan’s terms, they can be “medium-free,” “medium-specific,” or fall somewhere in the middle: Solid candidates for the medium-free pole are the defining components of narrativity: character, events, setting, time, space, and causality. A good example of a transmedially valid yet not medium-free concept is interactivity. It is applicable to video games, improvisational theater, hypertext fiction, tabletop role-playing games, and even oral storytelling, if one considers the impact of the audience on the narrative performance, but not to literary narrative, print-based comics, and film. Medium-specific concepts, finally, are explicitly developed for a certain medium, but they can occasionally be extended to other media through a metaphoric transfer. For instance, the concepts of gutter, frame, and the arrangement of panels on a page are tailor-made for the medium of comics. (2014: 4)

5

Intermediality and Medium Specificity

87

Ryan and Thon are careful not to make sweeping claims about any medium essentially sporting the same qualities in each possible instance, nor about any medium being void of specific constraints in how it presents content. Instead, it is a “media-conscious” sliding scale of potential elements for producing narratives across media that can be observed case by case. One of the ruling theoretical frameworks in the study of languages, the arts, culture, and media in the humanities since the linguistic turn has been semiotics, or semiology, after whether Peirce or Saussure, often influenced by Barthes or Foucault. In the semiotic view, linguistic signs form into communicative discourses that shape the everyday production of culture. In the middle, there is the medium that participates in the process passively, as a physical conduit of information with essential qualities, or actively, as a set of practices for using media in specific ways. Thus, the options for the functions of media within the semiotic view are very much in line with the tradition of debates on medium specificity and specifically as forms of communication. As Thon confirms the theoretical consensus on the definition of “medium” (2016: 17): “The term is best understood as referring to a multidimensional concept, which usually emphasizes the semiotic structure and the communicative function(s) of different media.” Lars Elleström’s views on medial complexities are staunchly Peircean. Before his untimely passing, he was refining the concept of transmediation to support his modeling of the kinds of medialities and modalities within semiotics. In focusing on the transmedial distribution of media characteristics across media, and what happens to the characteristics in the process, he asserted (2019: 5): It is only a short step from the idea that represented media characteristics may be transmedial to different degrees to recognizing that media characteristics, because of their transmedial nature, can be understood as being transferred among different kinds of media. Inserting a temporal perspective, it very often makes sense to acknowledge not only that similar media characteristics are or may be represented by dissimilar media but also that media characteristics that can in some respect be understood as the same, recur after having appeared in another medium.

A normative dichotomy between similarity and difference appears in Elleström’s account of what constitutes transmediation – that something, whether media content or form, remains unchanged in the process. The concept of medium specificity is preserved in a covert fashion so that, whereas no semiotic medium has essential qualities to boot because media are always crossing into each other, and there are no specific medial practices that would be semiotically insulated from one another in the everyday production of culture, specific media must still exist because something changes and something stays the same in moving between them. The task of the scholar in transmediality is to identify the elusive “same,” in aimed inquiry within the general field of intermediality research. The volume Transmediations (2020) Elleström edited with Niklas Salmose divides approaches on studying transmedial processes into two main categories: transmediation and media representation. Transmediation, including transmedial narration and adaptation studies, is defined as the “repeated mediation of equivalent

88

J. Toikkanen

sensory configurations by another technical medium.” Meanwhile, media representation, including the use of rhetorical devices such as ekphrasis, is defined as a process in which “a medium, which is something that represents something in a context of communication, becomes represented itself” (2020: 4). There is no need here to detail the respective complexities, but it should be noted how the concern with medium-specific similarity and difference is pointed towards concepts such as “sensory configurations” in current research. In training the attention so, there are more than shades of aesthetic and rhetorical inquiry updated for critical purposes. Kamilla Elliott is presently making a related effort in adaptation studies. In Theorizing Adaptation (2020), she provides an extensive historical account of the concept of adaptation as having to do with works of art and media products invariably denigrated as secondary creations (2020: 5): Adaptation has been castigated both for failing Romantic originality and poststructuralist deconstructions of originals and copies; it has been excoriated both for violating aesthetic purity and medium specificity theory and for supporting those principles by postmodern and radical theorists; it has been decreed a semiotic impossibility under formalism, structuralism, and poststructuralism alike; it has been charged with political incorrectness by both conservative and radical scholars; it has been accused of philosophical untruth by both modernist and postmodern theorists.

Elliott’s report on adaptation can be read together with the historical journey of medium specificity as sketched in this chapter because the two concepts are intermedially involved. The issue in each case concerns what is essential, if anything, about a work in a specific medium; what survives when a work, or something in a work, is mediated, or “adapted,” into another through a specific practice; and what kinds of judgments of value and taste we make about the objects that come into being as the result. For Elliott, the problem with theorizing adaptation has been pinched on how “adaptation scholars continue to propound the myth of fidelity criticism against the evidence of history” (2020: 17), as if adaptation studies had to be only occupied with medium-specific questions of originality and purity instead of asking what was specific about the processes of adaptation as such, regardless of the media used in a given instance: “We have been too focused on theoretical specificities – medium specificity in particular – and insufficiently on adaptation specificities” (2020: 231). Elliott makes a contrast between Linda Cahir’s (2006) linguistically oriented approach to adaptation as a “purely formal field of intermedial translation” and Linda Hutcheon’s (2006) Barthesian approach in which adaptations such as “narratives are both cultural and formal constructs” and “other media forms are just as much contexts for adaptation as the cultural contexts that we more commonly nominate as such” (2020: 145–146). She expands on Hutcheon’s view that because “entities and environments cannot be carved into separate theoretical territories” (Elliott 2014 quoted in 2020: 272) to adapt between the media, “this is not only because adaptations are composites of entities and environments but also because media and art forms are just as much environments for other media and art forms as nations, cultures, and industries” (2020: 272). For Elliott, the media forms out there constitute more than cultural contexts as “environments” that adapt with the natural world: “Biological adaptation is evolving and its theories are evolving along with

5

Intermediality and Medium Specificity

89

it. So too adaptation in the humanities is adapting and our theories need to adapt to it” (Elliott 2012 quoted in 2020: 273). Elliott essentially assumes the evolutionary perspective for the future of adaptation studies, and although she does not resolve the question of how to employ a relevant methodology, she does claim that the set of practices should be informed by aesthetics and rhetoric in discovering the effects of sensory configurations, to use Elleström’s term, in adapting between medial environments: “synaesthesia can also synthesize rhetorical figures to inform the study of adaptation” (2020: 288). It is by combining theoretical legacies on aspects of medium specificity from across the ages that new directions are sensed. Recent sources on relevant studies around the last decade include Bao (2015), Blore (2018), Bristow (2016), Brownlee (2018), Bruhn (2016), Bruhn and Schirrmacher (2021), Bull and Cobussen (2021), Grennan and Hague (2018), Houwen (2017), Kinder and McPherson (2014), Király (2010), Lopes (2014), Louvel (2011), Noë (2015), Petersson et al. (2018), Pop (2020), Rancière (2011), Richon (2011), Rippl (2015), Schrock (2015), Schröter (2012), Smith (2018), Stewart (2010), Thoben (2020), Verstegen (2014), and Wolf (2011).

Conclusion: New Applications in Intermediality Research John Guillory sketched in “Genesis of the Media Concept” (2010) a timeline of his own for the root word “medium.” Although Guillory does not ponder on the question of medium specificity as such, he wonders about the fact how studies on media over the centuries still have not figured out “how to relate the theory of mediation to the fact of media” (2010, 359), or as the matter can be parsed too: “Why and how exactly is it that media mediate?” (Toikkanen 2020, 74). The challenge of medium specificity is indeed to explain how the technological and institutional apparatuses known as media are connected with the phenomenon of mediation, traditionally observed as a philosophical concept with its basis either in the objective – something universally or, more recently, culturally valid – or in the subjective, in the aesthetic judgment of the individual. (For perspectives in the discipline of media studies, see Krämer (2015), Meyrowitz (1999), Packer (2013), Parikka (2013), and Peters (2015).) Over the centuries, scholars have found answers in accounts of similarities and differences between the arts, with preferences on what is to be celebrated and preserved or renewed and revolutionized. In most cases, as suggested at the start, the choice has come down to deciding whether a “medium” is a physical thing fixed with essential qualities, or if it is a changing set of practices for interacting with the physical world, either on the level of actually doing the work or being guided by cultural conventions in thinking about it. In the last hundred years or so, the inquiry has been influenced by the aftermath of the linguistic turn and, in the twenty-first century, directed towards the potential coming through in innovative media platforms and products, as well as theorizations about changing forms of materiality and affectivity, among other issues. At the same time, in pursuing the concern with medium specificity in the new millennium, the critic should be careful in establishing leverage – historical, contemporary, and philosophical – for the concepts and questions, so as to thwart the criticism of authors such as Carroll who has chalked off the

90

J. Toikkanen

“praxeological” approach, the one based on practices, to studying medium specificity as “ultimately uninteresting” (2019: 36). As Torregrossa demonstrated in confronting Carroll, there are options, but it requires commitment and a solid knowledge of tradition in intermediality research, as Irina O. Rajewsky has proclaimed: In fact, the criterion of historicity is relevant in various ways: with regard to the historicity of the particular intermedial configuration itself, with regard to the (technical) development of the media in question, with regard to the historically changing conceptions of art and media on the part of the media’s recipients and users, and finally with regard to the functionalization of intermedial strategies within a given media product. In this approach, therefore, intermediality is not bound to a uniform, fixed function. It analyses individual instances in terms of their specificity, taking into account historically changing possibilities for the functionalization of intermedial practices. (2005: 50–51)

I close this handbook entry for now by introducing some examples from my recent research. I have been developing the concept of intermedial experience for longer than a decade outside the popular semiotic framework. A few years ago, I introduced the Finnish translation “välinemääräisyys” for “medium specificity” and launched a three-tier model of mediality for the study of literary texts and works in other media (Toikkanen 2017), drawing on traditions in phenomenology, aesthetics, and rhetoric, as well as topical questions in contemporary affect and media theory. Whereas semiotics finds the functioning of the “human sense receptors within a communicative situation” (Elleström 2019: 23) as a “presemiotic” condition by which “sensory configurations” (Salmose and Elleström 2020: 4) are mediated, my three-tier model of mediality, influenced by Hegelian phenomenology (1977), substitutes the study of communication for the study of experience. When the role of the senses is reduced to acting as presemiotic conduits of modally communicated information, as in semiotics, the understanding is at odds with the phenomenological awareness of the senses taking fundamental part in the production of experience. How sensory experience is, and what it feels like, is more than the kind of information the senses modally communicate in the form of sensory configurations. The challenge of this modeling of experience, and its intimate relation to medium specificity, has to do with defining the concept of medium, on the first tier, as “the basic sensory means by which we become aware of and perceive the environment” (Toikkanen 2020: 73). As I have argued with my colleague, sensing and perceiving are prerequisites to grasping and interpreting in the “whole process” of experience (Virtanen and Toikkanen 2020: 82) and do not only assume the non-specific role of passing on information. Instead, how something is perceived is specifically affected by the sensory medium, or media, activated on the first tier. Experience is medium specific to begin with. The second tier of the model recognizes that, for sensory engagement, perceptions are presented in some way, ranging from the simple (speech, writing, gesture) to the complex (art forms and media formats). The third tier consists of conceptual abstractions, such as ideas, that the second-tier presentations of the first-tier perceptions mediate. I have claimed “there is no experience at all without the medial interaction of these three tiers that, in practice, occur all at once but can be distinguished for analytical purposes” (Toikkanen 2022: 235). Studying

5

Intermediality and Medium Specificity

91

what I call intermedial experience is studying the interaction of each medial tier in specific medial environments: In this model, the first-tier sensory means are grasped as primarily mediating both what is imagined and what is not, whereas the second-tier ways of presenting mediate according to specific convention, and the third-tier conceptual abstractions mediate the ideas and judgments attached to the phenomena at hand. (2020: 73)

In the recent years, with the help of the three-tier model of mediality, I have analyzed the medium-specific “effect of the taking away of sight, and the kinds of imagined sensory perceptions mediated in the form of touch and other sensory images” in Edgar Allan Poe’s short fiction (2021: 19), how “intermedial experience specifically comes into being” between shifts from the verbal to the visual in William Wordsworth’s poetry (2019: 116), and, in another kind of medial environment with its own specificities, how paranormal reality television can engage with the viewer’s sense of touch: “When what is there is not seen or heard but felt or, more peculiarly, imagined to be felt, how does that affect the intermedial experience?” (2020: 71, emphasis in the original). In the future, the task for developing further applications of the concept of medium specificity is to keep a close eye not only on the history and tradition contributing to its sustained critical leverage, but to stay alert and sensitive to the new kinds of sensory engagements in medial environments such as virtual reality.

References Arnheim, Rudolf. 1974. On the nature of photography. Critical Inquiry 1: 149–161. Babbitt, Irving. 1910. The New Laokoon: An essay on the confusion of the arts. London and New York: Houghton Mifflin. Bao, Weihong. 2015. The trouble with theater: Cinema and the geopolitics of medium specificity. Framework: The Journal of Cinema and Media 56 (2): Article 6. Barthes, Roland. 1977. Image music text. Trans. Stephen Heath. London: Fontana Press. ———. 1981. Camera Lucida: Reflections on photography. Trans. Richard Howard. New York: Hill & Wang. Baumgarten, Alexander. 1750. Aesthetica. Traiecti cis Viadrum: J.C. Kleyb. https://archive.org/ details/bub_gb_dopKAAAAcAAJ/mode/2up. Accessed 9 September 2022. Benjamin, Walter. 1969 (1936). The work of art in the age of mechanical reproduction (Das Kunstwerk im Zeitalter seiner technischen Reproduzierbarkeit). In Illuminations, ed. Hannah Arendt, 217–254. Trans. Harry Zohn. New York: Schocken Books. https://www.marxists.org/ reference/subject/philosophy/works/ge/benjamin.htm. Transcript by Andy Blunden. Accessed 9 September 2022. Berg, Dirk J. van den. 2004. What is an image and what is image power? Image & Narrative: Online Magazine of the Visual Narrative. Issue 8 Mélanges/Miscellaneous. http://www. imageandnarrative.be/inarchive/issue08/dirkvandenbergh.htm. Accessed 9 September 2022. Berger, John. 1972. Ways of seeing. London: Penguin. Blore, Francis. 2018. The importance of medium specificity in writing and drawing. Drawing: Research, Theory, Practice 3 (1): 99–106. Bolter, Jay David, and Richard Grusin. 2000. Remediation: Understanding new media. Cambridge: The MIT Press.

92

J. Toikkanen

Bristow, Maxine. 2016. Pragmatics of attachment and detachment: medium (un)specificity as material agency in contemporary art. PhD thesis, Norwich University of the Arts. Brownlee, Shannon. 2018. Fidelity, medium specificity, (in) determinacy: Identities that matter. In The Routledge companion to adaptation, ed. Dennis Cutchins, Katja Krebs, and Eckart Voigts. London: Routledge. Bruhn, Jörgen. 2016. The intermediality of narrative literature: Medialities matter. Palgrave Pivot. Cham: Springer. Bruhn, Jörgen, and Beate Schirrmacher, eds. 2021. Intermedial studies: An introduction to meaning across media. London/New York: Routledge. Bryson, Norman. 1995. Philostratus and the imaginary museum. In Vision & Textuality, ed. Stephen W. Melville and Bill Readings, 174–194. Durham: Duke University Press. Bull, Michael, and Marcel Cobussen, eds. 2021. The Bloomsbury handbook of sonic methodologies. New York: Bloomsbury Academic. Burke, Edmund. 1990 (1757). A philosophical enquiry into the origins of our ideas of the sublime and beautiful. Oxford/New York: Oxford University Press. Cahir, Linda Costanzo. 2006. Literature into Film. Jefferson: McFarland. Carroll, Noël. 2003. Engaging the moving image. New Haven: Yale University Press. ———. 2019. Medium specificity. In The Palgrave handbook of the philosophy of film and motion pictures, ed. Noël Carroll, Laura T. Di Summa, and Shawn Loht, 29–47. Cham: Palgrave Macmillan. Cavell, Stanley. 1971. The world viewed: Reflections on the ontology of film. New York: Viking Press. Cheeke, Stephen. 2008. Writing for art: The aesthetics of Ekphrasis. Manchester: Manchester University Press. Clüver, Claus. 1997. Ekphrasis reconsidered: On verbal representations of non-verbal texts. In Interart poetics: Essays on the interrelations of the arts and media, ed. Ulla-Britta Lagerroth, Hans Lund, and Erik Hedling, 19–33. Amsterdam: Rodopi. Collins, Christopher. 1991. Reading the written image: Verbal play, interpretation and the roots of Iconophobia. University Park: The Pennsylvania State University Press. Costello, Diarmuid. 2008. On the very idea of a “specific” medium: Michael Fried and Stanley Cavell on painting and photography as arts. Critical Inquiry 34 (2): 274–312. Dryden, John. 1695. The art of painting. London, Printed by J. Heptinstall for W. Rogers. https:// quod.lib.umich.edu/e/eebo/A36766.0001.001?view¼toc. Accessed 9 September 2022. Elleström, Lars. 2019. Transmedial narration: Narratives and stories in different media. Palgrave Pivot. Cham: Springer. Elliott, Kamilla. 2012. The adaptation of adaptation: A dialogue between the sciences and humanities. In Adaptation and cultural appropriation, ed. Pascal Nicklas and Oliver Lindner, 145–161. Berlin: de Gruyter. ———. 2014. Rethinking formal-cultural and textual-contextual divides in adaptation studies. Literature/Film Quarterly 42 (4): 576–593. ———. 2020. Theorizing adaptation. New York: Oxford University Press. Foucault, Michel. 1973. Seeing and knowing. In The birth of the clinic: An archaeology of medial perception, 131–151. New York: Pantheon Books. Fried, Michael. 1998. Art and Objecthood. Chicago: The University of Chicago Press. Gasché, Rodolphe. 2003. The idea of form: Rethinking Kant’s aesthetics. Stanford: Stanford University Press. Gaut, Berys. 2010. A philosophy of cinematic art. Cambridge: Cambridge University Press. Greenberg, Clement. 1939. Avant-Garde and Kitsch. Partisan Review 6 (5): 34–49. http://archives. bu.edu/collections/partisan-review/search/detail?id¼283920. Accessed 9 September 2022. ———. 1940. Towards a Newer Laocoön. Partisan Review 7 (4): 296–310. http://archives.bu.edu/ collections/partisan-review/search/detail?id¼283925. Accessed 9 September 2022. ———. 1962. After abstract expressionism. Art International 6 (8): 24–32.

5

Intermediality and Medium Specificity

93

Grennan, Simon, and Hague, Ian. 2018. Medium, knowledge, structure: Capacities for choice and the contradiction of medium-specificity in games and comics. Image & Narrative. Text and Image in Children’s Literature/Texte et image dans la littérature de jeunesse. http://www.imageandnarrative. be/index.php/imagenarrative/article/view/1765. Accessed 9 September 2022. Guillory, John. 2010. Genesis of the media concept. Critical Inquiry 36 (2): 321–362. Harvey, Judith. 2002. ut pictura poesis. The University of Chicago, theories of media, keywords glossary. https://csmt.uchicago.edu/glossary2004/utpicturapoesis.htm. Accessed 9 September 2022. Hayles, N. Katherine. 2004. Print is flat, code is deep: The importance of media-specific analysis. Poetics Today 25 (1): 67–90. Heffernan, James A.W. 1993. Museum of words: The poetics of ekphrasis from Homer to Ashbery. Chicago: The University of Chicago Press. Hegel, Georg. W. F. 1977 (1807). Phenomenology of Spirit. Trans. A. V. Miller. Oxford/New York: Oxford University Press. Horace. Ars poetica. https://www.poetryfoundation.org/articles/69381/ars-poetica. Accessed 9 September 2022. Houwen, Janna. 2017. Film and video intermediality: The question of medium specificity in contemporary moving images. New York: Bloomsbury Academic. Hutcheon, Linda. 2006. A theory of adaptation. New York: Routledge. Jay, Martin. 2005. Songs of experience: Modern American and European variations on a universal theme. Berkeley: The University of California Press. Jenkins, Henry. 2008. Convergence culture: Where old and new media collide. New York: NYU Press. Jensen, Klaus Bruhn. 2016. Intermediality. In The international encyclopedia of communication theory and philosophy, ed. Klaus Bruhn Jensen, Robert T. Craig, Jefferson D. Pooley, and Eric W. Rothenbuhler, 3–12. Hoboken: John Wiley & Sons. Jones, Caroline A. 2005. Eyesight alone: Clement Greenberg’s modernism and the bureaucratization of the senses. Chicago/London: The University of Chicago Press. Kant, Immanuel. 2000. Critique of the power of judgment. ed. Paul Guyer. Trans. Paul Guyer and Eric Matthews. New York/London: Cambridge University Press. Original Kritik der Urteilskraft (1790). http://www.archive.org/details/kritikderurteils00kantuoft. Accessed 9 September 2022. Kinder, Marsha, and Tara McPherson, eds. 2014. Transmedia frictions: The digital, the arts, and the humanities. Berkeley: University of California Press. Király, Hajnal. 2010. The dance of Intermediality: Attempt at a semiotic approach of medium specificity and intermediality in film. In Media borders, multimodality and intermediality, ed. Lars Elleström, 199–210. Basingstoke: Palgrave Macmillan. Krämer, Sibylle. 2015 (2008). Medium, messenger, transmission: An approach to media philosophy. (Medium, Bote, Übertragung. Kleine Metaphysik der Medialität). Trans. Anthony Enns. Amsterdam: Amsterdam University Press. Krauss, Rosalind. 2009. The guarantee of the medium. In Studies across disciplines in the humanities and social sciences 5, ed. Tiina Arppe, Timo Kaitaro, and Kai Mikkonen, 139–145. Helsinki: Helsinki Collegium for Advanced Studies. Krieger, Murray. 1992. Ekphrasis: The illusion of the natural sign. Baltimore: The Johns Hopkins University Press. Lee, Rensselaer W. 1940. Ut Pictura Poesis: The humanistic theory of painting. New York: Norton. The Art Bulletin 22 (4): 197–269. Lehmann, Karl. 1941. The imagines of the elder Philostratus. The Art Bulletin 23 (1): 16–44. Lessing, Gotthold E. 2005 (1766). Laocoon: An essay upon the limits of painting and poetry. Trans. Ellen Frothingham. Mineola: Dover Publications. Lopes, Dominic McIver. 2014. Beyond art. Oxford: Oxford University Press. Louvel, Liliane. 2011. Poetics of the Iconotext. Trans. Laurence Petit. Farnham: Ashgate. Lyotard, Jean-Francois. 1984 (1979). The postmodern condition: A report on knowledge. Trans. Geoff Bennington and Brian Massumi, Minneapolis: Minnesota University Press.

94

J. Toikkanen

de Man, Paul. 1996. In Aesthetic Ideology, ed. Andrzej Warminski. Minneapolis/London: University of Minnesota Press. McGann, Jerome. 1991. The textual condition. Princeton: Princeton University Press. McLuhan, Marshall. 1964. Understanding media: The extensions of Man. London/New York: McGraw-Hill. Meyrowitz, Joshua. 1999. Understandings of media. et Cetera 56 (1): 44–52. Mirzoeff, Nicholas. 1999. An introduction to visual culture. London/New York: Routledge. Mitchell, W.J.T. 1994. Picture theory: Essays on verbal and visual representation. Chicago: University of Chicago Press. Noë, Alva. 2015. Strange tools: Art and human nature. New York: Hill & Wang. Packer, Jerome. 2013. Epistemology not ideology OR why we need new Germans. Communication and Critical/Cultural Studies 10 (2–3): 295–300. Parikka, Jussi. 2013. Afterword: Cultural techniques and media studies. Theory, Culture & Society 30 (6): 147–159. Peters, John D. 2015. The marvelous clouds: Toward a philosophy of elemental media. Chicago/London: The University of Chicago Press. Petersson, Sonya, Christer Johansson, Magdalena Holdar, and Sara Callahan, eds. 2018. The power of the in-between: Intermediality as a tool for aesthetic analysis and critical reflection. Stockholm: Stockholm University Press. Philostratus the Elder. Imagines. https://www.theoi.com/Text/PhilostratusElder1A.html. Accessed 9 September 2022. Pop, Doru. 2020. The gamification of cinema and the cinematization of games. In Transmediations: Communication across media Borders, Routledge studies in multimodality, ed. Niklas Salmose and Lars Elleström, 52–74. New York/London: Routledge. Quintilian. Institutio oratoria. http://penelope.uchicago.edu/Thayer/E/Roman/Texts/Quintilian/ Institutio_Oratoria. Accessed 9 September 2022. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités 6: 43–64. Rancière, Jacques. 2011. What medium can mean. Trans. Steven Corcoran. Parrhesia 11: 35–43. Richon, Oliver. 2011. Introduction: On literary images. Photographies 4 (1): 12. Rippl, Gabriele, ed. 2015. Handbook of intermediality: Literature – image – sound – music. Berlin: de Gruyter. Rose, Gillian. 2001. (Fourth edition in 2016). Visual methodologies: An introduction to the interpretation of visual materials. London: Sage. Ruskin, John. 1848. Modern Painters, Vol. III. London: Dent. https://www.gutenberg.org/files/ 59392/59392-h/59392-h.htm. Accessed 9 September 2022. Ryan, Marie-Laure, and Jan-Noël Thon, eds. 2014. Storyworlds across media: Toward a mediaconscious narratology. Lincoln: University of Nebraska Press. Salmose, Niklas, and Lars Elleström, eds. 2020. Transmediations: Communication across media borders. Routledge studies in multimodality. New York/London: Routledge. Schrock, Andrew Richard. 2015. Communicative affordances of mobile media: Portability, availability, locatability, and multimediality. International Journal of Communication 9: 1229–1246. Schröter, Jens. 2012. Four models of intermediality. In Travels in intermedia[lity]: ReBlurring the boundaries, ed. Bernd Herzogenrath, 15–36. Hanover: Dartmouth College Press. Smith, Anthony N. 2018. Storytelling industries: Narrative production in the 21st century. Cham: Springer. Soules, Marshall. 2002. Animating the language machine: Computers and performance. https:// web.archive.org/web/20060224032144/http://www.mala.bc.ca/~soules/animate.htm. Accessed 9 September 2022. Stewart, Garrett. 2010. Bookwork as demediation. Critical Inquiry 36 (3): 110–157. Sturken, Marita, and Cartwright, Lisa. 2001. (Second edition in 2009). Practices of looking: An introduction to visual culture. Second edition. Oxford: Oxford University Press.

5

Intermediality and Medium Specificity

95

Thoben, Jan. 2020. Media parasites, specificity and the unheard. Auditive Perspektiven. https:// edoc.hu-berlin.de/bitstream/handle/18452/21841/5-JThoben_kunsttexte2020AP.pdf. Accessed 9 September 2022. Thon, Jan-Noël. 2016. Transmedial narratology and contemporary media culture. Lincoln: University of Nebraska Press. Thorburn, David, and Henry Jenkins, eds. 2003. Rethinking media change: The aesthetics of transition. Cambridge: MIT Press. Toikkanen, Jarkko. 2013. The intermedial experience of horror: Suspended failures. Basingstoke: Palgrave Macmillan. ———. 2015. Transcendental puppets: Kant and Kleist. In The poetics of transcendence, Currents of encounter series, ed. Elisa Heinämäki, P.M. Mehtonen, and Antti Salminen, 159–178. Amsterdam: Brill Rodopi. ———. 2017. Välineen käsite ja välinemääräisyys 2010-luvulla. (The concept of medium and medium-specificity in the 2010s). Media & viestintä 40 (3–4): 69–76. ———. 2019. Intermedial experience and ekphrasis in Wordsworth’s “Slumber”. Partial Answers 17 (1): 107–124. ———. 2020. Feeling the unseen: Imagined touch perceptions in paranormal reality television. The Senses and Society 15 (1): 70–84. ———. 2021. At the cutting edge: Touch images in Edgar Allan Poe’s “The Pit and the Pendulum”. Connotations – A Journal for Critical Debate 30: 1–23. ———. 2022. Endless north: Intermedial experience of motion and balance in H. P. Lovecraft’s “The Dream-Quest of Unknown Kadath”. In Shaping the north through multimodal and intermedial interaction, ed. Juha-Pekka Alarauhio, Tiina Räisänen, Jarkko Toikkanen, and Riikka Tumelius, 233–246. Cham: Palgrave Macmillan. Torregrossa, Clotilde. 2020. Remember the medium! Film, medium specificity, and responsedependence. PhD thesis. University of St. Andrews. Verstegen, Ian. 2014. Dispositional realism and the specificity of digital media. Leonardo 47 (2): 159–163. Virtanen, Ira A., and Jarkko Toikkanen. 2020. Co-creating study of experience in dialogue. Research in Art Education 1: 74–92. Wagner, Peter, ed. 1996. Icons – Texts – Iconotexts: Essays on ekphrasis and intermediality, European cultures: Studies in literature and the arts. Berlin: de Gruyter. Webb, Ruth. 2009. Ekphrasis, imagination and persuasion in ancient rhetorical theory and practice. Farnham: Ashgate. Williams, Raymond. 1977. From medium to social practice. In Marxism and literature, 158–164. Oxford: Oxford University Press. Wolf, Werner. 2011. (Inter)mediality and the study of literature. CLCWeb: Comparative Literature and Culture 13: 3. https://docs.lib.purdue.edu/clcweb/vol13/iss3/2/. Accessed 9 September 2022. Wordsworth, William. 1800. Lyrical ballads, with other poems, volume 1. https://www.gutenberg. org/ebooks/8905. Accessed 9 September 2022. Yacobi, Tamar. 1997. Verbal frames and ekphrastic figuration. In Interart poetics: Essays on the interrelations of the arts and media, ed. Ulla-Britta Lagerroth, Hans Lund, and Erik Hedling, 35–46. Amsterdam: Rodopi.

6

Intermedialities, Societies, and Power Histories Juergen E. Mueller

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedialities, Societies and Power Histories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Some Brief Comments on Power(s) and Intermedialities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Power Plays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedialities and the Power Plays of Materialities: Prehistorical Caves, Paintings, Human Movements, and Sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Power Plays and Historical Genres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedialities and the Power of Economies of Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedialities and Political Power Plays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Power Nodes and Market Places . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedialities and the Struggle for Dominance of Symbolic Universes (“Sinnwelten”) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedialities, Social Institutions, and Power Plays of Social Memories . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Word at the End or Towards a History of Social Archetypes of Intermedial Functions and Power Plays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

98 98 102 105 105 107 110 116 120 121 125 127 127 129

Abstract

From their very beginnings intermedial phenomena and processes are closely connected with social power structures and various power plays. In our chapter we tackle a selection of paradigmatic intermedial power plays based on several historical cases, ranging from prehistorical caves, medieval mystery plays, tapestries, nodes of intermedial networks of colporteurs/pedlars in modern times, TV-news, internet platforms, to post-modern “documentaries.” The major challenge of our approach lies in the development of research axes and viable historical and stable categories for the re-construction of social J. E. Mueller (*) (Professor emeritus) University of Bayreuth, Bayreuth, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_6

97

98

J. E. Mueller

functions and power plays of intermedial phenomena. We undertake a double or triple methodological effort: after a brief clarification of our concept, as well as of the notion of power and of levels of intermedial power plays, we propose seven heuristic research axes. The application of these historiological axes or perspectives implies movements of zooming in to specific cases and processes and zooming out to broader groups of intermedial phenomena and practices. While doing this, we will make use of various “tools” of media research, of social research and sociology of knowledge, of economies of attention, but, also of “sound archaeology,” semiology, and theories of memory. This procedure will finally lead us to an “in-between” research level, oscillating between theoretical reflections and historiological reconstructions, which is constituted by bottom up and top-down movements. The chapter ends with a preliminary conclusion of what intermedial power plays “(can) do to us” under specific sociohistorical and technological conditions. In this sense, our contribution is another test run for the breadth and applicability of a historiological approach of a network history of intermedial functions and power plays. Keywords

Intermedial power histories · Social functions, materialities and genres · Economies of attention · Political powerplays · Social memories and alternative histories

Introduction Intermedialities, Societies and Power Histories “Everything Is Intermedial”: Some Preliminary Notes In a recent interview, Lars Elleström coined the phrase “Everything is intermedial.” (Elleström et al. 2020)

There is no doubt that our everyday life and our media-constructed realities are deeply constituted by intermedial phenomena and processes, and that is why Elleström’s statement cannot be contradicted, especially in a contribution to a handbook on intermediality. In this chapter, it will serve as an impulse for further reflections and investigations of the manyfold manifestations of the “everything,” of its historical and playful appearances, of its historical and social functions in general, as well as, in particular, of its power balances. If the main challenges for Elleström’s “system or research axis” lie in the task of a construction of a “Medium-Centred Model of Communication” (Elleström et al. 2020, 9), that is of a universal semiotic approach and a theory of cognition for the explanation of intermedial phenomena, the challenges of our approach lie in the development of research axes and viable historical and stable categories for the

6

Intermedialities, Societies, and Power Histories

99

reconstruction of intermedial phenomena and their social functions stretching from far prehistorical times until our current situation of media plenitudes. Our next steps will therefore imply procedures of broadening and narrowing our focus and perspective in the form of a double methodological approach; after a clarification of our concept, we will propose a historiological research axis of intermedialities and power (plays). This axis will then be exemplified by a selection of historical cases and will be characterized by movements of zooming in to specific phenomena and processes and zooming out to broader groups of intermedial practices. Our analytical steps will therefore imply procedures of broadening and narrowing our focus and perspective. The idea that intermediality is a highly evasive phenomenon and only accessible through the traces it has left in media products and in societies seems to be an important starting point in relation to the perspectives (Müller 2000), histories (Clüver 2019; Moser 2007; Müller 1998), and the status (Müller 1987, 2008a, 2015b) of intermedial studies. However, if we take a closer look at current models trying to operationalize the concept of “intermediality,” they turn out to be strongly “systematizing” in order to reach “closed” or “meta-disciplinary” models, which does not always seem to be very convincing (In fact, Elleström’s (2021), Wolf’s (2002, 2018), Pethő’s (2020), Bruhn and Gjelsvik’s (2018), and Helbo’s (2022), Rajewsky’s (2002, 2014), Clüver’s (2019), Guiyoba et al.’s (2022), concepts are among the exceptions in this field). In this light, “lost illusions” (Cisneros 2007) concerning the options of intermedial research axes would be above all the result of exaggerated expectations comparable to those we have known about the notion of “intertextuality,” about the systematizing force of semiotics, or of cognitive sciences. The contributions of our approach would hence be characterized rather by its capacities to rethink media histories than to provide a “theory of media theories” – a somewhat “naïve” idea which does not consider the complexity of the phenomena which is expressed, for example, in the exorbitant number of possible intermedia interactions. How to build a system for all these types of possible and realized media interactions? It would be preferable to conceive “intermediality” as an axis of historical, social, and functional rather than theoretical relevance. Taking another look at Elleström’s statement that “everything is intermedial” and remembering our comments about his – admittedly extremely broad – circumscription of intermedial fields, our initial question must be specified and differentiated. We should ask ourselves under what sort of social and historical conditions the “everything” does become “intermedial” and what sort of sociological and historiological perspectives can and will be useful for scholarly reconstructions of intermedial processes and corresponding social functions in history. Such kind of intermedial studies should be done under the awareness that human forms of live, human societies, always are interwoven with complex historical factors and social institutions, such as human needs and work, languages and modalities of communication, housing, food production, habituation, industrialization, technological developments, digitization, and power, to mention only a few central aspects. It is evident that an intermedial approach which would develop research axes and viable historical and stable tools for the reconstruction of intermedia phenomena

100

J. E. Mueller

stretching from very far prehistorical times until our current situation of media plenitudes is a great challenge, which involves a high degree of risk. The risks of such an enterprise are, for example, to be found in great theoretical and practical problems to apply sociological or semiological methods to very different societies and human social systems. We must be careful not to impose views and categories, often formed by representatives of Enlightenment or nineteenth century’s early sociologists on sociohistorical processes and structures, which diverge considerably from ours, for example, regarding the role of “personal property.” There is evidence that hunter-gather societies are characterized by immediate-return economies, which later, in farming and industrials societies, change into delayed-return ones (Suzman 2020, 152). In other words: in so-called “egalitarian” societies, economies, power plays, and corresponding social functions of media are to be seen as very different from our modern times (Woodburn 1982; Blurton-Jones 1987). Looking back at social systems that existed (hundreds of) thousands of years ago, we are confronted with the fact that, nowadays, they are only “present” in the form of extremely few traces – mostly of stony material, in the form of bones, footprints, artifacts, or in lucky cases, in the form of cave manufacturing or cave paintings. These traces or signs, which exist as very rare presences of absent lives, events, or intermedia power processes, can only be related by marked “extrapolations” to concrete historical and intermedial activities, which took place in specific prehistoric societies, in societies whose central values, systems of knowledge, patterns of action and relevant media can only be revitalized by very few signs or findings, such as special sites, fireplaces, caves, music instruments, etc. (By the way, these ages might better be called wooden ages, instead of stone ages, because of an intense human use of wooden material of which, unfortunately, there are only very few traces left.) The further we move back to prehistorical traces, the more difficult it becomes to cope with and to reconstruct (inter-) medial phenomena of hunter-gatherer societies, of first farming communities, of patriarchal or matriarchal social structures, etc. But there are some remarkable approaches and findings of lost and gone intermedialities, for example, reconstructed by sound archaeology (Till 2014). In any case, we must scrutinize the reliability of our historiological tools and results of our intermedial enterprises. We believe that phenomenological, semiological, and historiological approaches, as we can find them, for example, in the publications of Structures of Lifeworld (Schutz and Luckmann 1973), The Social Construction of Reality (Berger and Luckmann 1966), Semiohistory (Schmid 2000), Social Histories of Literature and Media (Gumbrecht 1992; Müller 1997, 2001), or Odin (1994) in his semio-pragmatics of films studies) can be essential building blocks of such an endeavor. These framing theories and approaches will have to be – depending on the focus of our historiological studies – completed by fitting tools, for example, of cultural studies (Hall 1973; Fiske 2010), of network theories (Hepp et al. 2008; Schüttpelz 2007; Steinmaurer 2015; Koubek 2008), of genre theories (Altman 1999), and of functional histories (Elias 2012) (Especially his work at the ZIF (Zentrum für interdisziplinäre Forschung) Bielefeld in the research group of “Funktionsgeschichte”). Ideally, focused historical descriptions should lead us to reconstructions and explanations of intermedial phenomena, as well as to their

6

Intermedialities, Societies, and Power Histories

101

interactions of power processes and power patterns, such as, for example, the power of economies of attention. Cornerstones of such an intermedial network archaeology would have to consider processes of ruptures, transitions, innovations, and encounters of cultural series – the latter inscribed in different sociohistorical contexts or situations – and while reconstructing these processes to consider the non-simultaneities of simultaneous series and media. Our axis of relevance hence leads to a rhizomatic history between the poles of technology, cultural series, historical mentalities, powers, and social practices, which will have to find itself in a field once considered the preserve of “general history.” However, this history cannot be limited to the so-called traditional, institutionalized, and established media of the past millennia or centuries, but must also include the imagination, utopias, and forgotten media practices, as well as the pictorial and pragmatic textual representations of “old,” “virtual,” and “new media.” We must stay aware that our current media histories are surrounded by an enormous number of unwritten histories of (not only pre-) historical visions of media and forgotten media, which would have to focus on phenotypes of intermedial interactions and on corresponding social functions. Visions of the intermedium “television” can, for example, be found in Tiphaigne de la Roche’s novel Giphantie (1760) and Robida’s novel Le vingtième siècle (1883), where they outlined and imagined technical characteristics of the intermedial dispositif (Baudry 1978, 2003; Müller 2003) of a “magic mirror” (Tiphaigne de la Roche) or “téléfonographe” (Robida), as well as – and this is with regard to the research interests of our approach even more intriguing – a kaleidoscope of social desires and wanted usages of a utopia of TV. Usages, which ranged from attending live performances at home, going along with sexually loaded male gazes, to live views of seascapes or the mass media power of big informing and advertising panels on editorial buildings/offices of newspapers (Müller 2001). In the following, the elaboration of our research axis will be illustrated by a selection of paradigmatic “cases” and processes, which we have been tackling so far (including some studies of colleagues, PhD students, or postdocs at the University of Bayreuth). While doing this, we should stay aware of the fact that, on the one hand, historiological intermedial research asks for a clarified motivation of the perspectives chosen in relation to specific questions and phenomena and, on the other hand, the fact that this choice always implies – in a phenomenological sense – a motivated moving of certain phenomena into the center and others into the horizon of our viewpoints. These movements and thematic focusing of attention to centers and horizons can be redirected, and during all this, elements of themes and horizons are permanently interacting. For reasons of research capacities and research economies, however, we are forced to focus on specific intermedial processes that justifiably catch our attention. We also should stay aware that historical functions and intermedial processes, even in our current times of avatars presuppose – in phenomenological terms – our corporeality and our senses as well as the central assumption of the reciprocity of perspectives. Generally, these central givens lie at the basis of all historiological research of the manyfold interplays between humans, power

102

J. E. Mueller

processes, and intermedia. While carrying out this historiological research, we are forced to focus on intermedial phenomena, which must have left readable traces for reconstructions of their intermedial functions. In a paradigmatic way, we will try to get back as far as possible to eroded and blurred traces of the prehistoric past of intermedial processes in societies and relate these traces to possible social functions. This procedure could be characterized as bottom-up and top-down movements leading us to an “in-between” research level, oscillating between theoretical reflections and historiological reconstructions. In this context, we finally must repeat that our proposal, as we understand it, includes social dimensions and social functions of intermedial processes and treats intermedialities in relation to these vectors. By the way, this axis of intermedial relevance should prevent us from propagating a pan-intermedialization of all social phenomena, extending from questions of gender relations to the migration movements of the nineteenth to the twenty-first centuries. This does not mean that our axis of relevance would not be able to provide impulses for research in the social field, but we should not forget that there are other concepts and (sociological, socialpsychological, or political sciences, etc.) disciplines that would be more useful for this kind of research. Before the background of Elleström’s methodological procedure (2010), our historiological approach (our intermedial network history) might help to develop historical profiles of modalities and of historical functions of intermedia processes ranging from specific patterns of aesthetic experience of the recipient to patterns of action and behavior of individuals or social groups taking place with and within the frames of certain dispositifs, as well as to associated power processes. Our description and explanation of historical intermedial phenomena implies various interplays between (formerly?) separated academic disciplines of humanities and social sciences and will have to make use of the best-suited tools and categories for our specific research axes applied to historical media landscapes. Thus, our research axis entails a rethinking of the dynamics and interactions between media, the materiality, and contents of the media, by taking into consideration the modalities of these interplays. Perspectives and vectors such as mediality, as well as social functions, networks, power, economies (also of attention, of cultural, and social capitals), ludifications, genres, social systems of knowledge will prove to be essential for our intermedial history (Müller 2010a). While carrying out this historiological effort, we should stay aware of the fact that the abovementioned vectors are to be found in permanent interactions. Arrived at this point of our calibrating introduction, some key aspects of power outlined by social scientists and media scholars are to be shortly summarized.

Some Brief Comments on Power(s) and Intermedialities What Do We Have to Understand Under “Power”? It is evident that there is not only one sort of “power” and that “power” influences the histories of media and intermedialities in manyfold ways. If we look, for example, at the Dictionary of Contemporary English, we find at least two or three dozen of first

6

Intermedialities, Societies, and Power Histories

103

denotative and second connotative meanings of the term “power,” ranging from “an ability that forms part of the nature of body or mind,” over “the ability to do something or produce a certain effect,” “force, strength,” “control over others” to an “apparatus provided with or worked by a motor.” Hence, regarding our short inquiry of the encounters and interactions between intermedia, social, sociopolitical, and economic powers, a further sociological and theoretical sharpening of this notion seems to be useful. This sharpening should make us aware of some central dimensions of the power plays of intermedial processes in societies. Max Weber’s definition of power as the “ability to control others, events, or resources; to make happen what one wants to happen in spite of obstacles, resistance, or opposition” (Weber 1922, 28 ff., own translation) circumscribes one of the main factors of this phenomenon. According to Weber, power would be held, coveted, seized, taken away, lost, or stolen, and it is used for the determination of adversarial relationships involving conflict between those with power and those without. Media, sociopolitical, and economic institutions of power have found themselves from their very beginnings in various conflicts. “Power” cannot be reduced to “pure and negative dominance”; rather, it should be considered as a permanent flow from a social system’s potential to coordinate human activities and resources in order to accomplish goals. Following this line – for example, by referring to Foucault (2008) – “power” would not only be repressive but also productive. Moreover, it would not only be a “simple” property of social institutions, governments, or the state, but something that it is always exercised throughout the social body. Powers as typical forms and processes (for example, sovereign, disciplinary, or pastoral) are omnipresent and correlate with the production of different types of knowledge and media; they create, constitute, and organize social realities as well as concepts of social realities (Berger and Luckmann 1966). Following this line, “traditional” and “digital” media are to be regarded as “power-players,” who are putting (concepts of) realities into sounds, movements, pictures, and texts. Taking another closer look at the notion of “power,” we can find several proposals for the analysis of media and power processes in the academic contexts of political sciences, cultural studies, sociology of knowledge, sociology of communication, and of contemporary history (as “Zeitgeschichte”). In his book Screens of Power. Ideology, Domination and Resistance in Informational Society, Luke adopted perspectives from semiotics and critical theory in order to apply them to a study of the “sources of mystification, power, and domination in the social production and consumption of meaning” (1989, 8). This enterprise is realized in the American sociohistorical context of the 1980s and should “provide some guideposts for the actual resistance groups always forming at the margins of society” (1989, 9). Within the frame of cultural studies, for example, Hall and Fiske gave some decades ago very valuable insights into power plays of twentieth century mass media, especially of radio and television (Hall 1974; Fiske 1993). They pointed to the interplays between media and political dominance, politics, and culture, as well as medial generations of social identities. Sociology of knowledge related power structures to opportunities and procedures to construct and control units or “bodies” of symbolic systems of knowledge (Berger, Luckmann), a statement that can be in a

104

J. E. Mueller

certain way be seen in some parallelism to concepts of power defined as a vector to own and control the means of sense-production (Fiske 2010, 14). In communication studies and sociology of communication, several (realized and desired) social functions of power have been identified (and also postulated): functions ranging from socialization, information, construction of dominant narratives, entertainment, political and ideological influencing, cultural norming, to information transfer. Curran (2002) published a useful survey of how power of and within media developed in different sociohistorical contexts and of how their interactions with political and social powers can be characterized. Even if his labeling and classification of sociological media research in categories such as “liberal” or “radical research traditions,” or “revisionist models of powers” (110,111) sometimes seem to overstress taxonomic thinking, his summaries of power histories of media give deep insights into specific phases, stretching from medieval and Christian times to twenty-first century media. He distinguishes between seven “principal ways in which popular forces can influence the media in liberal democracies,” ranging from “Cultural power” to “staff power” (151–155) and “eleven main factors that encourage the media to support dominant power interests,” ranging from “state censorship” to “dominant discourses” (148–151). He makes clear that media power can be seen as a vector, which dominates the audience, or as processes, which are under strong influence of audiences. In our perspective, both aspects are to be seen as relevant and will be considered on the basis of our intermedial research axis. Mitchell’s Picture Theory (1994) constitutes another cornerstone, which tackles in innovative and transdisciplinary ways, interplays between texts and pictures. This approach focuses on power dynamics of images in the second half of twentieth century and on the dynamics between technologies and conditions of our “human vision” (24), but also on criticisms of power (for example of suggestion of deception). His theory generates new insights into central social dimensions, functions, and power plays of images and media that imply complex transitions between several codes, conventions, and channels. As a particular position in this discursive field, an Austrian project of scholars of contemporary history should be mentioned: >kunst >kommunikation >macht (2004), which was the topic of the sixth “Österreichischer Zeitgeschichtetag 2003” and which presents a great variety of historiological reflexions and contributions concerning arts, media and power, as well as numerous paradigmatic studies. At the end of this very brief survey “taking place at dizzying speed,” we must finally mention an anthology about the “Power of the In-Between,” where the study of power aspects is investigated in a series of case studies, which target to go beyond the “unhappy divide between [. . .] the intermedial subfields of semiotically and formalistically oriented studies and [. . .] media-historical ones” (Johansson and Petersson 2018, 2). The relevance of such an enterprise can only be underlined. The Power of the In-Between contains a great variety of thorough studies, for example, by Jørgen Bruhn and Henriette Thune (2018) (“In between Life and Death: Sophie Calle’s Rachael, Monique” 25–48) or a conceptual proposal “Old and New Media: On the Construction of Media History” (Johansson (2018),

6

Intermedialities, Societies, and Power Histories

105

375–406), primarily based on Bolter’s and Grusin’s concept of remediation (1999) and Manovich’s media theory (2001). Compared to this background of studies of in-betweenness, our approach of an intermedia network history is aiming less at a combination of formalistically and historiologically based studies, but at transparent and plausible reconstructions of social functions of intermedial inter-betweenness. That is to say: of reconstructions of what sort of power plays intermedialities are involved in media and social history, or in other words, what intermedialites “do to us” and what “we do to intermedialities.” In this sense, we will put a somewhat stronger focus on intermedial processes, social vectors, and power plays. To sum up: the question of “power and media” has seen a rather long runtime in several disciplines; however, in the academic discourses on “intermediality,” it did, if at all, only show up in very few publications or in terms of “asides,” or footnotes. Also, in current debates of “convergence” or “transmedia storytelling” (Jenkins 2006, 2009, 2014), it is only implicitly referred to as a factor of (primarily) fixed media and not of intermedia processes and statuses. While realizing our research, we must stay aware that in most cases there are copresent several vectors of power plays, and that, usually, it is advisable to direct our research interest to the most dominant ones. The chosen (pragmatic) procedure of zooming in and zooming out will allow us to focus in greater detail on specific cases or power aspects and to merge several phenomena in broader historical periods in order to arrive at more general observations. The findings of our short paradigmatic and historiological studies might and should be related to comparable processes and phenomena in histories of intermedial power plays. Let us, finally, also keep in mind that the ensuing categories and findings are an open list and work in progress, i.e., that they will have to be modified or completed in future research (For further options and categories, see Müller (2022)).

Intermedial Power Plays Intermedialities and the Power Plays of Materialities: Prehistorical Caves, Paintings, Human Movements, and Sounds Some of the first paragraphs of this chapter are elaborated in greater detail in Müller (2020a, 2020b).

Caves, Materialities, and Intermedial Power Plays Some traces of power plays between materialities, intermedial processes, and prehistoric participants of actors have been reconstructed (see Müller 2022). In several caves, for example, in the cave of Cueva de las Chimeneas, prehistoric man used sound as an underground sonar as a hummed “mmmmmm” (due to laws of acoustics preferably a male voice) in narrow passages, where the most resonant spots were marked by red points in order to find back his way. These red points denote and connote complex intermedial power plays between rocky surroundings, human

106

J. E. Mueller

movements, noises, and sound qualities, as well as guides and points of orientation in a (in these passages) mostly dark environment (see Reznikoff 2010; Müller 2020a, 2020b), in an environment, which was and still is the prerequisite for very intensive multi-(In my article (Müller 2008b), I have proposed a semantic and theoretical differentiation between the terms “multimediality,” “hybridity,” and “intermediality.”) and intermedia experiences. These human experiences are based on the material powers of these places which left traces on human bodies that, today, unfortunately, are not present any more for our historiological research, but which still can be found in prehistoric paintings created in narrow interactions with structures of rock and stone configurations and recently also in soundscapes as sound archaeology (Reznikoff 2000, 2002; Till 2014; Jacobs 2017) has rediscovered. Prehistoric sounds and music can be performed in theses spaces, based on reconstructed instruments. The interactions between material powers of cave-spaces, spatial givens of caves, paintings, and sounds can also be experienced in Cave Kapova, Ural. At the right and left side of the ground of a part of the cave, where groups of horses and mammoth move towards a corner, there are two very intense sound niches/recesses which function as “acoustic resonators or receptacles in relationship to the paintings” [own translation, J.E.M.] (Reznikoff 2010, CD–49). So, the horses and mammoths can approach the listener in an intermedia and synesthetic way. In the context of their approach, Rupert Till and his colleagues (2014) have produced audiovisual clips of prehistoric performances in caves. Human voices and instruments (flutes, whistles, trumpets made from bones or horns) have been reconstructed on the basis of historical discoveries. Till’s videos are characterized by complex historical and intermedial transformations; they refer in an impressing way to the intermedial plays and ceremonies which very likely have taken place in the dispositif of these caves. His audiovisual reconstructions of prehistorical events make palpable again central effects of intermedial processes in mythical sound spaces, of interactions triggered between structure and materiality of rocks, drawings and sounds, between proper sounds of the underworld, songs, human voices, music instruments, echoes, and human bodies. These revitalizations and remediations of prehistoric thrilling spectacles give us an idea of archetypes of human actions and media interactions realized in caves some 40.000 years ago. Nowadays, some traces of that kind of early spectacles can be seen in the (often commercialized) retrievalboom of so called “places of power” (“Kraftorte”), which are rediscovered and “re-activated” in special spots, for example, of the Alps, by twenty-first century’s shamans. Caves, but also other exceptional places, such as the surroundings of sound rocks (even Stonehenge is regarded to be such a place) become spirited places of prehistoric man, i.e., “vibrant places” of dynamic and complex intermedial processes which constitute multi- and intermedial subgenres of performances, rites, and stories, for example, narrations of hunting, or also in dreamtime myths of the creation of living species on our planet. In the Australian Aboriginal Dreamtime creation story and myth, the sun goddess Yhi goes down into caves in order to make animals from rocks of the caves into which she brings bright light. According to this myth, rocks

6

Intermedialities, Societies, and Power Histories

107

are also the basis for a transformation of stony material into living animals, which form the material and spatial basis for cave paintings such as Chauvet. To a certain extent, Yhi’s presence in the darkness and her bringing light into it could be seen as a parallel to bring “movement” and “life” to animals in prehistorical caves. At the end of this short (pre-) historical tour, we might undertake two zooms out: firstly, a very short step into times of antique power plays between materialities, intermedial processes, and social, religious, political, as well as economic powers; and secondly, another very short step into the period of so-called early cinema. Babylonian clay tablets, for example, which originally served as tools for trade and later also for religious and political dedications, for orders, complaints, as well as the constructions and writings of myths, are inextricably interlinked with processes of material, economic, sociopolitical, and religious powers. They are constitutive factors for the creation of empires, and their materiality has a direct impact on the forms and possibilities of governance, power structures, and borders of empires, as the shift from clay tablets to papyrus rolls in the Egyptian period shows. Papyrus has a considerably minor weight than tablets and can be handled and transported much easier – even to the furthest cities and villages of empires. In this sense, the materiality of ancient intermedia is closely linked to economic, political, and religious activities and, at the same time, these media are also economic goods. They transmit and store information which is considered to be of high value. The medium “papyrus” enables those who are in control of it to communicate to privileged individuals or social groups in order to exercise, maintain, or expand power, and it allows these groups to preserve and permanently “rework”/“refine” the medium as well as their authority. Looking back at the early days of media and writing, we might ask ourselves in which ways comparable interactions between media and power are still to be seen as central factors in our (post-) modern societies. The spectacles of so-called early or Vaudeville cinema show many parallels with our above described intermedial cave performances and their specific spatial arrangements. To a certain extent, the man-made “cave,” the mechanical and technological givens and powers of the projector, and the movement of the film reel and its noise make necessary and imply complex multi- and intermedia power plays between the apparatus, the bonimenteur, or commentator, sounds, actions, and songs of the audience. In this perspective, silent cinema has never been silent but was an intermedia performance based on powers of the material and the cinematic apparatus (“dispositif ”), which, in the history of film and media theories, shows several similarities with Plato’s allegory of the cave. In the late nineteenth and early twentieth centuries, the powers of this apparatus demanded sympraxis (i.e., coacting, Kloepfer 1999) from its participants, which might again be seen as a parallel to prehistorical cave spectacles.

Intermedial Power Plays and Historical Genres Intermedial Genres and Historical Power Plays If we agree that genres are to be characterized as typical patterns of action between producers, texts/intermedia, and consumers/prosumers, they execute undeniable

108

J. E. Mueller

power on more or less all cultural products or types of communication. In this sense, the concept of “genre” is a relevant category for medial and intermedial research on dispositifs, processes, and products of analogue and digital media. So, let us take a brief look at this term and at its relevance for intermedial histories. Communicative genres offer more or less effective and binding ‘solutions’ to specific communicative ‘problems’ and constitute specific components of our social stock of knowledge. In everyday social life, we permanently make consistent use of different genres to bring certain recurrent problems to a communicative solution. Their social function is to organize these solutions, to routinize them, and to let them proceed in a more or less obligatory way. Thomas Luckmann proposed in his “Prolegomena to a social theory of communicative genres” (1989) an analytical and – in a broad sense – empirical model for the sociological analysis of the profiles and social functions of “genres” in our societies. This model was developed against the academic interdisciplinary background of the encounters between sociology and literary studies of the School of Constance, which prepared an excellent breeding ground for crossover developments. In this light, the transfer of the literary category of “genre,” to the sociology of everyday life, points not least to the chances of transdisciplinary theory building and historical intermedial research. Genre patterns react to and give answers to specific communicative questions and form the communicative “household” of a society, which includes routinized, historical, ethnic, or group-specific forms of uses and functions. The theoretical and methodological benefit of Luckmann’s proposal lies in the opening of a wide field of research on intermedial phenomena, on intermedial recycling, as well as on the de- and recoding of everyday communicative genres and corresponding social functions. Hence, despite some current theoretical tendencies of merging or fusing all media into a digital “pabulum,” the notion and concept of “genre” still is very valuable for intermedial research and intermedial histories. Following Altman “genres” could be characterized as: “as blueprint, as a formula that precedes, programmes and patterns of industry production; as structure, as the formal framework on which individual films [or intermedial products, J.E.M.] are founded; as label, as the name of a category central to the decisions and communications of distributors and exhibitors; as contract, as the viewing position required by each genre film of its audience [or intermedial product and its users, J.E.M.]” (Altman 1999, 14). What do the above statements imply for intermedia studies? We think that – in spite of the undisputable relevance of materiality – we cannot and should not forget the important role of genres. Also, in our digital age, genres have to be considered as essential factors of intermedia processes, interacting in specific historical, technological, and social contexts with vectors of power, interests, horizons of expectation of producers and recipients/prosumers. In these days, where documentaries become interactive, participatory, transmedia formats, genres undergo radical changes in terms of producing and consuming, of action and types of participation as well as of fundamental generic patterns, we believed to be more or less stable.

6

Intermedialities, Societies, and Power Histories

109

The Genre of “Documentary” The terms “documentaries” or “documentary value” were coined towards the end of the 1910s in the form of the Francophone designation of travel films as “documentaries,“later in the form of the anglophone paraphrase of the “documentary value” of a film in the New York Sun of February 8, 1926. These terms and concepts are closely linked to media practices and traditions as they have developed in the audiovisual field since the end of the nineteenth century. The indicated (analogue and historical) medial practices imply more or less “clearly delineated” processes of medial creation of the instances of producers, medial products, and recipients and constitute the frame of reference of almost all “traditional” theoretical approaches to the “documentary.” If we had to offer a short résumé of the numerous “traditional” theoretical approaches to documentary audiovisual media or to documentary film, we would have to mention categories or characteristics such as: “realism,” “special relations of resemblance to reality,” “testimonial character of contemporary and historical events,” “concealment of the sign-like nature of reality,” “use of found footage and ‘non-aesthetic’ material,” “low degree of narrativity,” “situability, localization of observed events in space and time,” “construction of a ‘preferential relationship’ between conceptions of reality and media product” (Noguez 1979; Guynn 2014; Müller 1995), “specific modes of production of meaning, of significance and affects as well as institutionalized communicative pact between producers and recipients” (Odin 1994), or with regard to TV, a “television reality” (Jost 2009). This list of “traditional” categories or approaches to the determination of the documentary could be extended by several further items/points. Before the background of the functional vector of our intermedial archaeology, however, we should not forget to point to the usually intended function of a “suspension of disbelief” (which should go along with the reception of fictional texts or media products), where we or the readers/recipients should be reassured by generic and semiological patterns of the “reality-status” of that what we can hear and see. Documentaries are embedded in and interacting with various social, political, and generic networks, which very often have to secure intended functions, as we currently can note in numerous “documentaries” of wars, which are produced for wars of images.

Power Plays of Documentaries in the Digital Age In our digital age, we encounter a diffusion of the contours and fixed points of documentary creation, which makes numerous “core statements” about documentary seem obsolete or in need of revision. Interactive documentaries of the twenty-first century are characterized by an enormous range of media and social manifestations, which can extend from experimental forms of digital coproductions by numerous actors and institutions to complex intermedial rhizomes. In these days, where documentaries become interactive, participatory, transmedia formats (Wiehl 2019), genres undergo radical changes in terms of producing and consuming, of acting and types of participation, as well as of fundamental generic outlines we believed to be stable. They oscillate between traditional and new networks. Let us

110

J. E. Mueller

take a short look at a small selection of historical power plays of genres in intermedial networks, at so-called (new) documentaries. Wiehl has examined very carefully the case of Public Secrets (Daniel 2007), which we would like to refer to as an example for a strong impact of an intermedial product on communities of artists as well as of academics in the United States. Public Secrets was “co-authored with incarcerated women (imprisoned in the Central California Women’s Facility, the largest women’s prison in the United States) and focuses on the specific conditions of injustice perpetrated against women in the criminal punishment system” (Daniel 2007). It questions “truth claims” of documentary networks, reframes “the real, “authorship,” and “networked/networking”), and gives way to an interactive documentary which develops audio(
visual) “sedimentations of networking processes” in layered networks. This documentary oscillates between hybrid forms of art, scholarship, and activism, and offers a digital nonlinear storytelling, which is constituted by the activities of its numerous authors and prosumers. The detainees are not objects of documentary processes, but they are subjects and authors of complex interactions, where the power of production has to a great extent shifted on their side and is oscillating between several producers. We can see that social and generic functions of these new documentaries relate to former networks and corresponding social intermedial power plays, and they open up interlocking of media-philosophical-political reflections and new media practices. Accordingly, this interactive documentation must be understood as a self-reflexive experiment in which numerous practices, including those of art, science, and politics, are interconnected in a complex networked and networking field (Wiehl 2019, 185), practices that imply a fundamental shift and development of “traditional-linear” documentations in the direction of networking/networked. The new documentary Public Secrets gives as a sort of counterpower voice to the voiceless female inmates of a Californian prison, and its intermedial dissolution of generic patterns goes along with a dissolution and undermining of central political powers, which manifest themselves in social modes of punishment.

Intermedialities and the Power of Economies of Attention Intermedial Power Plays of Economies of Attention: A Brief Introduction There is a further relevant vector of intermedia power plays that we have so far only mentioned in a marginal way: the economies of intermediality, i.e., economies in a general sense and in a particular sense of an economy of attention. As scholars of humanities, we tend to overlook the influence of economies on intermedia processes and intermedia histories. Before this background, a research axis of an intermedia economy would fill a big gap in an intermedia network history, which focuses on the historical development of interferences and intersections between economic usages and typical patterns of action of users/prosumers/fans. Nowadays, these economic power plays go in many cases along with processes of ludification (Huizinga 1949). The powers of an economy of attention create intense interplays between intermedial representations and recyclings of audiovisual materials with a market of attention.

6

Intermedialities, Societies, and Power Histories

111

The permanent maintaining of the highest possible level of attention of the recipients (cf., for example, Franck 2018), users, or prosumers reveals itself to be crucial for the success intermedial products in their multiple forms of representation. A historical flashback into the academic history of film and media studies turns us to the fact that the study of “media economy” is not a new preoccupation of the twenty-first century scholars. Already in 1985, Allen and Gomery outlined in their volume on Film History, an “economic film history,” and in the early 1990s, Gomery refocused in his book Shared Pleasures (1992) on the interplays between the Hollywood film industry and business policies. These studies in media culture and media economy are still relevant today. However, in the light of the recent dynamics of media developments – especially in the digital field – the processes of media culture and media economy have to be reconsidered and reconceptualized. In the context of economics, there exists a branch of so-called “cross-media” research, and, in fact, there seem to be some good reasons that the notion of “crossmedia” is very attractive for economic approaches. Since the 2000s, for example, some researchers of media economy focus on economic factors of media convergences in sports media (Nickolson 2007). In our so-called digital era, new activities of producers, consumers, and – especially – prosumers ask for a replacement or a network-elaboration of former “value chains,” or of linear models of media economy. However, these studies are reduced to two arenas of forces of our media landscape: (a) the arena of technological change, including the channels of transmission and (b) the arena of increasing cross-media properties. This fact shows that the relevance of cross-media perspectives is at least partly present in the research horizon of economists who would like to establish an economy of cross-media processes; on the other hand, notwithstanding, it also proves the urgent necessity of a transdisciplinary revision and foundation of an economy of intermediality in historical and theoretical fields. The development and applying of an economy of intermediality in historical and theoretical fields would, as already mentioned above, broaden an economic perspective of cross-media research by the integration of categories such as mediality, genre, media interaction, ludification, economy of attention, social function (Müller 2010b), and cultural and social capitals (Bourdieu 1966). In the following, we will not stress traditional (financial or monetary) vectors of economic power plays of intermedia but vectors of economies of attention, which, naturally, always go along with capitalizable aspects.

Intermedial Power Plays and Economies of Attention, First-Person Shooters, War Games, and a Plane Crash in TV News Digital games prove to be one of the fastest growing phenomena in our twenty-first century media landscape, and without any doubt, they are also of crucial relevance for TV in the digital age. It is therefore useful to take a short look at the TV formats of news and documentaries, where, at a first glance, we would not expect intense ludifications. In the twenty-first century, it is a common practice of audiovisual mass media to broadcast documentary clips taken from a helmet camera of a shooter or soldier. That

112

J. E. Mueller

kind of “documentary” representation is very often spread on internet platforms, but also a central perspective and device of first-person shooter games and as such a quite instructive example of intermedia power plays of attention on the basis of ludification- or gamification-processes. The pictures produced by the head camera are therefore not only to be linked to generic patterns and modes of reception of “direct news” but also to socialized points of view of first-person shooting games. In these games, the player/user has to solve a series of problems, such as eliminating hostile fighters, exterminating aliens, or superhuman monsters (Bouwknegt 2011). The individual actions or “moves” in the game are predetermined by a set of variables and rules that enable the players to choose from a bandwidth of various options of action; this allows them to experience and finally conclude the game in their own manner. We cannot underestimate the economic power of such kinds of ludifications, which manifests itself in concrete value chains, but also in economies of attention (Franck 2018). An internet or TV live version of this genre of firstperson shooting games (or as well of other games) transforms their configurative principles into hybrid configurations and forms of narrative and narrated realities. It is evident that economies of attention have become a central vector and factor of current media production and reception (or – nowadays – also a “motivating” power for employees of big companies). In our times, news is not only produced as infotainment, but its generic patterns underlie the influences of intermedial power plays of attention and are remediated by various procedures of gamification and ludification. Currently, in times of the Ukraine war, we are confronted almost daily with news clips of exploding tanks, helicopters, and artillery projectiles that – due to their digital bases of recording and representation – have become very close to war games and their economic power of attraction. As another case, we would like to refer here briefly to the report of flight Nr. 522 of Helios Airways (on August 14, 2005) from Larnaka to Athens in the news of the German private TV station RTL2. This news is characterized by an interplay and a switch between short documentary sequences of reactions of affected persons and family members, or officials at the airport of Larnaka at Cyprus and game-like sequences representing passengers and pilots having lost consciousness on board (due to a failure of the air-conditioning system of the plane filled with “fog”), which end with several digitally and electronically produced “shots” of its final crash into a group of hills in Greece. The construction of this unit of a TV news program shows that, currently, news is not only produced as infotainment but also remediates many procedures of gamification and ludification. The intended sensationalist and horrifying effects of this news, which literally brings us down to the final crash of the plane, are evident; that is why we will only mention that this news oscillates between ludified, docudramatic, and pseudo-informative elements. We would like to conclude this section of possible future paths for intermedial research with a remark on a central research perspective: remediating the manifold digital game patterns in TV news obviously embraces highly complex intermedia processes that use and reduce ludic game patterns in order to constitute a coherent narrative form – in this case, of an airplane crash – while using the powers of an

6

Intermedialities, Societies, and Power Histories

113

economy of attention in order to give way to the fascination and thrill of the topos of a disaster. In this perspective, the crash of flight Nr. 522 of Helios Airways can be seen as a complex interplay of intermedial processes with patterns of news formats (or genres), games, of profits of TV stations, and with – not to forget – the powers of an economy of attention in order to attract as many viewers as possible. And this especially in times of a great monetary value of numbers of viewers or clicks on specific websites (not only of so-called influencers). Let us now briefly take a closer look at another relevant paradigmatic case, at Michael Jackson and his song and clip Thriller.

Michael Jackson’s Thriller as Another Test Case of Intermedial Economic Power Plays Without any doubt, Michal Jackson’s Thriller is one of the cornerstones of twentieth and twenty-first century audiovisual popular culture. The production of the video clip of Thriller (1982) opened a new dimension of media interactions between popular music and visual culture, and it also gave way to a second and much bigger success story of this song. It is believed to have triggered a new form of TV station, MTV, which, in times of YouTube, has already become history. In the light of our research axis of intermedial power plays of an economy of attention, it is evident that Michael Jackson is not only an intermedia phenomenon including thousands of remediations of his “personal paedophile (?) philosophy,” or of millions of remediations of his clip Thriller, but also a producer of various intertwined forms of monetary, social, and cultural capitals (Müller 2012, 2015b). From our perspective (which shows some parallels to Jens Schröter’s proposal of an intermedia politics; Schröter 2008), we should focus on the historical development of interferences, intersections, and networkings between economic usages and typical patterns of action of Jackson’s users/prosumers/fans, whereby we have to take into consideration the power plays and social functions of these processes. We are at present confronted with a machinery of commercialization of up to around 20.000000 of Thriller recyclings with “personal” material, symbolic-ideological inputs of the users, and the correlating materials or ideational gratifications, ranging from books, songbooks, biographies, fan journals, documentaries, films, dances to T-shirts, Jackson or Thriller clothes, video clips, etc. The intermedia networks and types of action, which come along with the options of the Web 2.0 ff., generate new historical forms and power plays of intermedia economies and bring about – among other aspects – new types of fan cultures as well as media critiques. These fan cultures take place as of superpositions, interactions, (historically variable) demarcations, paragones. (Here we refer to the term Paragone delle Arti, which has been coined in fifteenth-century Italy and since that time has stirred generations of artists and/theorists/poetologists with the question which art could claim primacy regarding its artistic skills, for example sculpture versus painting or versus literature. See for example Baxandall (1987)). The abovedescribed competitions and strivings for dominance etc., between several champs (Müller 2015b) must be considered as (more or less serious) power plays of an economy of symbolic and material goods, of economies of intermedia profiles of

114

J. E. Mueller

media usage and correlating formats or genres, of raw materials of vision, of “starpersonalities,” of particles of “(star-) realities,” of media recylings and habitus. The powerplays include the struggle for control of “right” and “adequate” pictures of the “star Michael Jackson.” In our perspective of economies of attention, intermedia recyclings are to be seen as the main factors in the ensuing and ongoing success of Thriller. “Werewolf-dancer Jackson” is a central element and an icon in these intermedia processes; we do still meet him “bodily”/“physically” or medially represented in our social or augmented realities, for example, as a TV advertisement for a scooter, a clip in Second Life, as a remake of young singers in talent shows, as dance performances, as a prototype for avant-garde or Pop artists, or as a mass-restaging in the Cebu Provincial Detention and Rehabilitation Centre, a Philippine maximum-security prison for dangerous criminals (murderers, rapists, and drug addicts). All these remediations must be seen before the background of economic power plays of attention. Thriller has become one of the most popular videos which ever have been put on the internet. As the example of Thrill the World (2015, an internet-live celebration of dance and community, inspired by Michael Jackson’ clip) shows, public reenactings of Jackson’s Thriller can be organized as a worldwide event with some thousands of participants. This event inspired by Michael Jackson’s video has a strong economic and materialist dimension (it functions as a platform for charity activities), but also a personal and psychic one. It gives numerous Jackson-fans the opportunity to perform and dance “their admired two face personality” and to show themselves playing with the plenitude of (contradictory and overlapping) aspects of the Jackson-star-images. The transformation of the Jackson audiovisual raw material into worldwide dance performances gives way to an intermedial integration of this material into the selfconcepts and everyday life of the performers. In a material, symbolic, and metaphorical sense, Thrill the World is augmenting the social and ideological realities of the performers and the performing communities and is based on the power plays of an economy of attention. This list of power plays of an economy of intermedia processes could be extended to hundreds or thousands of further examples of the case “Michael Jackson” and his “Thriller.” Here we limited ourselves to only few examples in order to convey a first idea of the power spectrum of an economy of attention, being at work in our current landscape of an intermedia plenitude.

(Medieval) Mystery Plays and the Powers of an Economy of Attention At the end of this section, let us realize another zooming out into intermedial power processes and conflicts of attention, which took place some time ago, i.e., during medieval times. Mystery plays know a long tradition in antique and medieval societies. It would go much too far to elaborate in detail their philosophical and religious bases and backgrounds (cf. Warning 1971), but regarding our intermedial research axis of ludifications, social functions, and powers of an economy of attention, they constitute relevant cases for a study of interactions between symbolic and material values and goods. This research axis brings to light not only an intermingling of clerical,

6

Intermedialities, Societies, and Power Histories

115

popular, and folkloric narrations as well of theatrical and intermedial traditions, but also a clash of economic interests and maxims of clerical rituals and ludifications by laymen and medieval guilts. Mystery plays are hence characterized by an intermedial and “ludificated” takeover of central Christian narrations as the Easter-Passion of Christ by traveling showmen or actors who “told,” “sang,” and made music together in order to amuse numerous audiences and spectators. In categories of an economy of attention, they possessed a very high or even the highest value of popular entertainment. In late medieval times, in some cases, these mystery or passion plays went on for several days and constituted big attractions for the citizens of these communities. They are to be considered as outstanding historical examples of power plays of multi- and intermedial spectacles and of effects of economies of attention in specific sociohistorical circumstances. There are many historical sources (also in the years after the great plague) which give insight also into the material value (in the sense of monetary profits or losses) of such kind of events for the cities, where the passion plays took place. In many of these passion plays, Christ’s Tomb played an important role; tombs were, for example, carried along as artificially assembled constructions in the processions or as annually built-in-caves in the churches. Understandably, the tradition of passion plays implied some sort of nuisance for clerical, noble, and bourgeois leaders who tried to get control over these multi- and intermedial practices, whose social function drifted more and more away from religious and spiritual recollection and reflection to a sort of “carnival procession” and shared pleasure (see also Faulstich 1996, 182 ff.). Towards the end of this development, even parades organized by bishops and priests run into danger to end up in anarchic conditions and states, where – due to fanaticism, alcohol, and drugs – the borderlines between the realities of the play and social realities diffused in such a way that in some cases the Christ-actor was crucified “in reality,” which produced a sort of macabre entertainment, which had to be stopped by clerical and noble authorities. Offshoot-versions of these intermedial plays continued to generate enormous profits in terms of attention, but also in terms of material and monetary enrichment and – not to forget – by performing a central social function of gathering crowds and focusing them on central biblical narrations. Mystery plays offered thrill and entertainment also in the form of Christmas plays or, in modern times, for example, in the form of parables, such as Hofmannsthal’s Jedermann (1911), a play about a rich sinner, his guilt, and God’s punishment, which is performed yearly at the Salzburg Festival.

Economies of Attention, Antique, and Prehistoric Ludifications and Recyclings of Material and Social Capital – a Short Conclusion The tradition and social function of such kind of intermedial power plays of attention can, at least, be traced back to antique times, for example, the Egyptian OsirisMysteries, where the passing by and procession of the statue of Osiris interfered with manyfold cultural activities of Egyptian priests and citizens in the public segment of the procession, which moved on towards the desert, where divine rites were performed. Even if it is rather difficult to reconstruct these antique activities, we

116

J. E. Mueller

can nevertheless find strong links and interactions between material and cultural goods, as we have described them above for social and economic power plays of attraction of intermedial processes. We should also not forget that almost all societies know the remediation and transformation of social acts of fighting or hunting into dances. That kind of intermedial performances, which usually take place at outstanding events, for example, at religious festivals, at funerals, at social meetings, but also as combat or street dances (for example, old Egyptian stick fighting), imply complex intermedial interplays between specific types of action and social values and powers. They presuppose intermedial power plays of attention. Not to forget that, until today, we can meet on the African continent numerous intricate forms of trans- and intermedial plays between human bodies, gestures, verbal, audiovisual, and performative arts (Fendler 2021). There are also many indications that prehistoric caves, their spatial and sonic qualities constitute – together with their paintings – dispositifs for ludifications and aestheticizations of central prehistoric lifeforms (and activities). In this sense, they can be seen as incubators for effects of life (Münch 2004) or deep experiences. It is more than likely that homo erectus might have been inspired by the light of his (straw) fire in the Wonderwerk Cave to start intermedial plays between his body, light, shadows, and stone walls of the cave which constituted around one million years ago first power plays of prehistoric economies of human attention. However, if we make assumptions about that kind of first intermedial events, we should remember the fact that in prehistoric times, many societies would not have been in need of (and did not know) personalized ownership or property, as we understand it nowadays. In this sense, it is evident that (not only prehistoric) social concepts of sharing or personal profit have strong implications on historic power plays of economies of attention and on social functions of intermedial processes.

Intermedialities and Political Power Plays A Brief Introduction During the past decades, it has become a commonplace of academic discourses that media and intermedia interact closely with political powers. Luke’s critical theory and historiology of Screens of Power (1989) and Curran’s book on Media and Power (2002) can be seen as examples of politological, sociological, and sociohistorical approaches to historical interactions between media and political powers. Yet, until today, inter-medial aspects did not really play an essential role in these studies, and only recently, factors as “hybridity,” or even “intermediality, have become research issues of a few projects (for example, Baym 2014). Before this background of current studies in media and media politics, it appears indispensable to widen the theoretical frame of “politics in the age of hybrid media” (Chadwick et al. 2016) to” politics in the age of intermedia” and to apply an intermedia network history to the reconstruction of that kind of processes taking place in the field of politics and media. Before the background of this requirement,

6

Intermedialities, Societies, and Power Histories

117

we will, in the following subsection, put an accent on politics and intermedial power plays. The exertion of political and economic power – in all societies – implies not only the diffusion of desired but also the suppression of specific “disturbing” or “inconvenient” stories or news. Especially wars have changed into wars of “suited” images for the conflict parties. Propaganda machineries and gatekeepers of media networks are eager to broadcast “right” pictures and sounds, such as “friendly and helpful soldiers”; at the same time, they hinder the publication of politically inopportune pictures of injured, killed, or torturing members of their army. Usually, in cases of severe international conflicts or “wars,” such as the events of 9/11 or the Russian war in Ukraine, a strong tendency of press and media for “patriotic” and “unified” points of view and opinions is to be found. However, media, cannot and must not be blamed for not showing all pictures in their news, for example extremely cruel images of killed or injured children as they are “produced” in the (forgotten) wars in Syria, Yemen, Ethiopia, and in several other countries. “Traditional” audiovisual media producers in modern societies are permanently confronted with the question of what to show and what not to show. In the digital age, billions of faked or not-faked pictures are circulating in “news pipelines” and on the internet, which brings about serious problems of selection and authenticity for the news-makers. The gatekeepers of the “old” “Leitmedium” (“key medium”) television, for example, must be aware of their ethical responsibility with regard to the specific “realities” they reconstruct. It is a commonplace that in times of the global village, the world we see (and can imagine of) is medially constructed and the result of intricate processes of mediations and remediations, of intermedial processes of representations, of facts and fictions, and of our own desires for authenticity. All that is not in this world of represented images and sounds usually does not exist for us. For that reason, political and economic powers have an enormous interest in the exclusion of elements which might irritate our (ideologically shaped) views of the world and of the “worlds of wars.” Wars have become wars of cameras and of microphones, of hysterizations, of quick effects, and of digital constructions and manipulations.

Abu Ghraib (2004): Intermedial Wars and Wars of Images In this light, let us first briefly comment on the case of Abu Ghraib, i.e., the accidental proliferation of mobile phone pictures taken in the US-American prison in Iraq in the year 2004. The pictures taken in the prison of Abu Ghraib have become emblems of the Iraq War of images. They give proof of the power of media in realizing manifold counterattacks regarding political powers, especially the US-American and British governments. The famous “hooded man,” connected to electric wires, entered into collective memories of Western and Eastern societies and fed the collective imaginary – this with its powerful denotations and connotations, for example, of humiliation, suffering, crucifixion, of Nazi concentration camps, of sadomasochistic practices, of abuses of power by military and private male and female guards, etc. The international publication of this and other Abu Ghraib photos led to a

118

J. E. Mueller

penalization of some subaltern staff and the director of the prison, as well as later to the closure of the prison. In this respect, the power play of intermedia, the war of images, the publishing of hidden and secret actions together with the representation of pictures which point to Western and Eastern canons of images, brought about a noticeable reaction in the sociopolitical field. The denotations and connotations of these pictures probably have been perceived quite differently in the eyes of Westerners and Easterners, yet, their iconic potential “worked” with great intensity in the cultural spheres of the West and East. As to the interdependencies of media and economic and political power, the “hooded man” and his many “companions” also must be considered as a paradigm of the power processes of intermedia recyclings. Let us remember: these images had been taken as mobile phone “private pictures” by members of the prison guards in the power context of Abu Ghraib. By “leakages,” they ended up in the circuits of the Web 2.0, which could not be sufficiently controlled by the political powers of the United States and Britain. Obviously, the options of the digital media did undermine the opinion sovereignty of the leading Western states. The entrance of the “private” pictures onto internet platforms gave way to a whole series of further remediations and comments in audiovisual mass media, television, the press, etc. They could not be suppressed anymore and led to some of the political and social consequences mentioned above. By the way, some weeks and months later, the “hooded man,” who could be identified due to his crippled hand, was given back a human face/his face and his image found various intermedia recyclings, for example in advertisement. In the case of Abu Ghraib, the power of remediated images seems – at least to a certain extent – to have reestablished the dignity of some of the prisoners. Apart from that, the intermedial transformations contributed to the formation of numerous stories and histories.

Another Intermedial Power Play in the Digital Age: Sina Weibo and Amok in Beijing During the past 20 years, the power and social functions of “traditional” print and audiovisual mass media have undergone a remarkable change, and in several respects, they are confronted with an erosion of influence. Parallel to these developments, the market value of print media has seen a dramatic decline, which is primarily caused by the rapid expansion of online media (with cost free news and broad offers of free entertainment). All over the world, local, regional, and national newspapers are closing, or are – more or less desperately – seeking for new crossmedia business models which could motivate readers/users/customers to pay for online content. A similar situation is found in the book market, where e-books and e-commerce are threatening “traditional” forms of book culture and commerce, and similar, even more revolutionary processes can be seen in the cultural field of music. But let us get back to our question of intermedial power plays in the digital age. Before taking a more differentiated look at some central future trends, we should keep in mind that all the options of digital media and media networks reconstructed below presuppose a very simple, but crucial fact: the accessibility and availability of

6

Intermedialities, Societies, and Power Histories

119

digital network platforms and gadgets. Millions of people on the African continent still do only have very and restricted opportunities to engage in the digital and virtual worlds. So, the following remarks must be read under this premise. Remembering our concepts of power described in section “Some Brief Comments on Power(s) and Intermedialities,” some of their central factors such as the desire to make happen what one wants to happen, or the correlation of power with the production and control of knowledge also appear to be true for media power plays in the digital era. Webs 2.0–5.0 have proved to be a rather useful tool for the organization of protests and demonstrations. In many cases (for example, in Arabian countries, but also in South Korea), the political and economic powers tried to control the traditional print and audiovisual mass media, i.e., press and television, but failed to suppress internet communications. The “new” internet platforms do in fact open further and previously unknown options of free speech, of representations of counter-realities (as, for example, the website Anonymous) and of political organization. Blogs or microblogs such as the Chinese Sina Weibo (see He 2018, 71 ff.) offer digital platforms for alternative discussions of “truths,” “official news,” or of views on local and international events. In times of glocalization, they can undermine dominant (Western or Eastern) ideologies. Precisely these possibilities are the reason why many “democratic” or dictatorial governments all over the world decide to deny access to critical international or global websites, as it happens in these days in Russia, China, and Iran. Concluding, we should take a short glance at trans- and intermedial recyclings of medial representations of a revolt in China in the year 1994 (He 2018, 71 ff.). In this year, a desperate action and shooting of the Chinese elite soldier (Tian Mingjian) took place in Janguomen Street close to one of the central city gates of Peking. The well-trained lieutenant Mingjian ran amok after his wife had lost her first child and had died during a forced abortion of her second child. This shooting was initially almost unnoticed in China (the evening of this day, there was only one very short report in the newspaper Beijing Evening News), but, by chance a Canadian TV crew shot this scene, which then had been broadcast on Canadian TV. This event was subsequently also reported and commented in many newspapers and TV stations in the United States and in Europe. Regarding our current issue of intermedial processes, recyclings, and political power plays, we have to note that the news spread in various foreign countries and a few months later (in 1994) was picked up in Chinese internet and discussed in greater detail, but found a another very powerful remediation in the year 2009 on the then started microblogging platform Sina Weibo, a fact which, nowadays, is considered to be one of the main catalysts for the ending of the One Child Policy in China.

A Short Aside on the Speed of Intermedial Political Power Plays The above described intermedial processes motivate another short aside in this contribution: we must keep in mind that remediations from paper media to TV, to digital media and vice versa, as well as grassroot or bottom-up journalism, offer many possibilities for the “news-greed” of recipients or users and for the constitution

120

J. E. Mueller

of intermedial “hot spots” which then spread images, sounds, and short texts in “real time” across our globe. This acceleration of information flows in the age of digital media plenitude (Bolter 2019) has severe influence on serious and professional journalism and on the quality and authenticity of news and (hi)stories: we are in danger of “twittering and tweeting journalism to death,” Gabriele GoderbauerMarchner in a lecture at the Media Department of the University of Bayreuth. The described speeding-up also leads to new sorts of power conflicts between traditional and digital media about opinion leaderships and, to some extent, it triggers off an increasingly faster “media-rat race” between all parties involved in media-powerconflicts. As we have seen from our examples of the many war images in our news, central criteria for the success of war propaganda lie in the instant availability of shocking images and sounds, which can get a new dimension by using head cameras to produce pictures, which would then not only be linked to “direct news” but also to “intermedially standardized” points of view of first-person shooters, of tank explosions, of helicopter flights, or of killing actions in shooting games. Multilayered network systems leave traditional dichotomic and hierarchic concepts of media interactions behind, and lead to a liberation of predefined power structures, but, on the contrary, can also enhance existing power structures in political tightly controlled media. Under these medial and political circumstances, the only and last chance of survival for press and print media might be the deliberate choice of slowing down to a temporal and reflective distance in relation to that sort of “instant-journalism,” with the latter, by the way, being not in all cases as “democratic and open for everyone” as it pretended and still pretends to be. Political campaigns, campaigns of warfare and of ideas, depend very much on the austerity and the possibilities for uncontrolled intermedial reflections of the images, sounds, and ideas.

Intermedial Power Nodes and Market Places Intermedia Networks, Nodes, and Power Plays: A Brief Introduction Complex discursive interactions between nature, technology, society, and media lead us to the paradigm of intermedia networks. The paradigm of intermedia networks contains elements as: threads, lines, holes, edges, knots, entanglements and nets, ideas and imaginations of nets, nodes, junctions, and junction lines, but also “holes,” social functions of networks in different social fields (in the sense of Bourdieu 1966, 1993), spatial and temporal marking of the “in-between” of systems, diachronic and synchronic (to use again two terms which have become out of fashion) research axes which are generated by network approaches, only to mention the most prominent vectors of this concept (Castells 1996, 470 f.; see also Koubek 2008, 41; Hepp 2004, 92; Hepp et al. 2008). The dynamics of media transformations and power plays in media networks and the processes of permanent changes can be best grasped by the axes de pertinence of intermediality and remediations (Bolter and Grusin 1999). In our approach to intermedial power plays, interactions between genres and genre patterns in networks and nodes of networks will be included.

6

Intermedialities, Societies, and Power Histories

121

A Short Zoom Out on Networks, Market Places, and Power Plays of Colporteurs and Peddlers in Early Modern Times If we focus on intermedial networks in Early Modern Times, we must keep in mind that urban and rural spaces constitute nodes for spectacles and representations of theatrical, musical, and pictorial genres, in which “real” spatial network threats are interwoven with “virtual” networked threads, which by this generate multilevel and three-dimensional webs. Our historiological intermedia network perspective leads us to the conclusion that from sixteenth to nineteenth century, colporteurs and peddlers took over a relevant historical function. They were in a literal and metaphorical sense: media porters, media providers, media vehicles, and media mediators who span and connect with their tours in time and space new, (also) de-territorialized networks. The spinning of these webs always implied the forming of and the acting in specific nodes. In this historical context, urban and village assembly spaces and markets are especially significant; they function as interfaces between social, political, economic, medial, informative, and ludic networks, and are intermedial power nodes. The “real” space of these places was the power basis for the assembly and the global intermedial networking of numerous imaginary plays and spectacles of which the inter- and multimedia pictorial (and acoustic) stories of the Laterna Magica have been among the most attractive for more than three centuries. Colporteurs and peddlers functioned as “broadcasters” of information and imago (i)nations for the masses, and they were the crucial media vehicles at the urban and village nodes. These historical media and the Laterna Magica (“a great box in which the Savoyards carry objects they offer to show for curiosity,” Larousse 1872, own translation) were imbedded on the market places into global multi- and intermedial networks, which ranged from music, multi- and intermedia performances to theatre plays. In Early Modern Times, these urban and rural spaces constituted nodes and sites of power for spectacles and representations of theatrical, musical, and pictorial genres, in which “real” spatial network-threads are interwoven with “virtual” networked threads, which by this are essential preconditions for the options and generations of multilevel and three-dimensional webs. Our search for traces of theoretical fragments of intermedial networks could be extended to include numerous elements, such as those found in Baxandall’s (1987) now classic work on painting and experience in fifteenth-century Italy. There, Baxandall reconstructs the interconnections between pictorial artworks, economy, natural sciences (especially optics and mathematics), as well as travel and work networks of the masters and their assistants.

Intermedialities and the Struggle for Dominance of Symbolic Universes (“Sinnwelten”) Symbolic Universes: A Brief Introduction In our introductory sections “ “Everything Is Intermedial”: Some Preliminary Notes” and “Some Brief Comments on Power(s) and Intermedialities,” we pointed to the

122

J. E. Mueller

fact that the approach of Sociology of Knowledge and, especially its comments on processes or “bodies” of symbolic systems of knowledge, is of high relevance for an intermedial network history and also for a historiology of intermedial power plays. Sociology of knowledge shares several basic assumptions and common fields of interest with cultural, semiological, and literary studies, such as, for example, the study of the historical functions of beliefs, values, and traditions, which are expressed in language, art, (inter-)media, religion, and myth for the constitution of symbolic universes. These worlds of meaning construct and legitimate social realities and can be seen as a central prerequisite of our being in the world. In such a perspective, a central function of our description and tentative explanation of prehistoric intermedial power plays in caves (see section “Intermedialities and the Power Plays of Materialities: Prehistorical Caves, Paintings, Human Movements, and Sounds”) would have to be seen in a more or less “theoretical” reassurance and legitimation of (historical) human existence and of human being in the world. Symbolic systems of knowledge order and structure human visions of the world, and by placing personal and social identities in a symbolic world of meaning, they constitute frames of and for our human existence. The legitimation of things implies the “telling of stories,” why things are what they are (Berger and Luckmann 1966, 94), and this intermedial storytelling takes place at different levels: (a) a pre-theoretical, (b) a theoretical, (c) a differentiated theoretical body of knowledge, and, finally, (d) at the level of symbolic universes (for example, religions). With regard to our critical examination of historical intermedial power plays with and within symbolic worlds of meaning, we must point to the fact that these power plays oscillate between the levels mentioned above.

“Parole per gli occhi”: “Eye Languages” or “Pictural” Intermedia Networks from the Sixteenth to the Nineteenth Century In Early Modern Times, not only the “scriptural” media of letters and (first) pamphlets or of newspapers played an important role for economic, academic, and social exchanges over very great distances (also over oceans), but from the second half of the sixteenth century onwards, also the “pictorial” medium of copper engravings, graphics, and drawings were spread by colporteurs in a radius of several hundreds or even thousands of kilometers. These pictures, which had very often been produced by Savoyards in a sort of industrial way, offered sacral and popular motives to a very broad audience, i.e., to a great part of the European population, and in eighteenth century also in great part of Asian and Southern American populations. Pictorial presentations, which developed into a “language of the eye,” dealt in the eighteenth century, for example, with topics such as romances, political caricatures, medical advices, far away and exotic country sides, farming advices, recipes, beauty consults, etc., and very often were strongly interconnected with songs, instrumental music, recited poems, and so on. They led to an explosion of the presence and selling of pictorial material and of shared elements of symbolic worlds of meanings, i.e., of myths, beliefs, values, and traditions. The pictures spread by the colporteurs imply power plays of remediations of the canon of “classical” models and pictorial pre-configurations of Western societies,

6

Intermedialities, Societies, and Power Histories

123

which interact with economic, social, medial, and technological networks and facilitate the dissemination, “broadcasting,” and intrusion of that kind of pictures into the collective imaginary of vast territories and of generations of their inhabitants. In this sense, the colporteurs functioned as “bodily” broadcasters who continued medieval traditions of traveling people and minstrels (Faulstich 1996, 227 ff.), and who initiated complex media processes which deployed their power and (cognitive and emotional) functions in specific intermedia, technological and socioeconomic networks (Müller 2015a; Brunetta 1997).

Laterna Magica and Networks of Collective Imaginations and Power Plays of Common Worlds of Meaning Estimatingly, until the end of the nineteenth century, several million people must have accessed Laterna Magica spectacles on the European and on other continents. These spectacles proved to be far more than simple representations of pictures: they constituted a global network of encounters with unknown worlds, that stretched from microscopic proximity to the exotic distance of alien fauna, flora, and cultures. The Laterna Magica, which first had been described in Athanasius Kircher’s (1671) book Ars Magna Lucis (1645/1646), developed into a very successful dispositif (we prefer this term to Comolli’s translation as “apparatus”), an economic and social vehicle for a global network of magic light-plays and (scientific) instruction. If we bracket (in a phenomenological sense) the apparative configurations of the Laterna Magica and focus instead on processes of and within intermedia networks as well as on historical power plays of this paradigmatic machine of Early Modern Times, we would, for example, find “standardized” options of an iconic, but also acoustic and musical lingua franca of almost all European (and many other) countries. These pictures, which – by the way – in many cases also were (partly) moving pictures, opened kaleidoscopic views on common worlds of meaning, symbolic universes, historical events, myths, social manners, technological wonders, etc. (Mannoni 1999). The colporteurs provided and arranged popular picture-spectacle-networks, which with their intermedia recyclings of “stereotypes” and pre-figurations and traditional (hi-)stories, entered the collective imaginary and coined the pictorial codes and imaginary spaces of great parts of European and foreign populations. In eighteenth-century France and Paris, for example, Laterna Magica – presentations and events – reached as a “fléau des aristocrats” (a “plague of aristocrats”), the illiterate Parisian population, and instructed the citoyens about the ongoing events of the French Revolution. These presentations or shows of the Savoyard colporteurs often started with songs. Multi- and intermedial picture shows and songs of the “colporteurs” clearly interacted with central pictorial, but also central sociopolitical power networks and historical systems of social knowledge, and of social interests. These shows were based on economies of attention and attraction and did not only constitute networks of imaginations but also worlds of meaning and news networks for great parts of the population.

124

J. E. Mueller

A Final and Brief Zoom In and Flashback to the Video Art Production “God’s Greatest Gift Is . . .” The video production God’s Greatest Gift Is . . . (Thibault et al., 1985) is the last of 10 parts of the “video-opera” OUT and deals with IN- and OUT- realities in an imagined year 2084, where a simulacrum of the former US-President Reagan is stimulating an (also imagined twenty-first century) audience to move between these two levels of reality. It is a production with 58 units/“shots,” lasting 2 minutes and 47 seconds, where Reagan as talking head, “documentary” clips, music, sounds, written texts and signs, are inextricably woven into an intermedial representation. In our perspective of intermedial power plays with symbolic worlds of meaning, this video still is particularly relevant for a short and paradigmatic flashback and review; it focuses on the symbolic and ideological powers of one of the central pillars of contemporary American society: the symbolic system and universe of Christian faith, which is confronted with a flood of distorted pictures and sounds of everyday – and war – violence, evoking a wide range of moods as well as meanings on the side of the viewers/recipients. Without any doubt, a flashback (almost 40 years after the production of God’s Greatest Gift . . . Is Human Life and after our first approach to it, Müller 1996) is relevant in terms of a historical localization of early video art in specific social, technological, and artistic contexts. It leads us to the conclusion that the production of this video was only possible due to the foundation of a cooperative network of independent artists in Montreal (P.R.I.M., Productions, Réalisations Indépendantes de Montréal), which, at that time, offered the opportunity to assemble electronic equipment and tools, which then were extremely precious and hard to get, and to form a very productive platform for avant-garde artists, outside of the power centers of big media companies. The P.R.I.M. artists elaborated their projects in this studio from the early 1980s onwards. In our brief summary of the historical phases of “pictural” intermedia networks from the sixteenth to nineteenth century, we pointed to several indications for a building up of a (more or less) global system of understanding and of meaning, of a historical universe, or of central parts of it. Colporteurs constructed, in resonance with the voices/songs of the presenters, for centuries iconic models and imag(o) inations for the collective imaginary and for the collective memory (Assmann 2011) of societies, but also “scientific” ideas of our world, its geography, its cities and villages, as well as of its inhabitants and of the universe. These intermedial power plays of the early modern period contribute to new symbolic areas or universes of meaning, partly replacing traditional religious models. In contrast to some main functions of intermedial networks in the early modern period we had reconstructed above, the intermedial power play of God’s Greatest Gift ... Is Human Life, undermines global systems of meaning by the deconstruction of Reagan’s political usurpation of a cornerstone of the Christian explanatory model of our Western world. While demonstrating and exercising and making use of the power of one of the central ideologemes, i.e., of the decisive legitimating instance of the consciousness of the American society, God, Reagan is debunked as politician exercising the power of destruction of human life. The aesthetic procedures of this

6

Intermedialities, Societies, and Power Histories

125

clip make the recipient experience an everyday kaleidoscope of horror in the contrasting interplay of words, sounds, and images. In this sense, the clip delivers an intermedial deconstruction, which, due to its polysemic structures, processes, and power plays, leaves its recipient with a plenitude of volatile denotations and connotations, and which, make her/him, via these mise en abyme, turn to symbolic and ideological abuses of political and religious powers.

Intermedialities, Social Institutions, and Power Plays of Social Memories Intermedia Networks, Storage, and Functional Memories: A Brief Introduction Nora (1984–1992) has demonstrated that places of remembrance (“lieux de mémoire”), also symbolic and intermedial sign clusters, myths, or configurations of paintings (as, for example, we also find them in caves), or works of art, are of central relevance for a culture of social memories. Before this background, we must, in the sense of Semiohistory (Schmid 2000), keep in mind that histories and social memories are constituted by various sign systems and intermedia; they are reactivated by specific social groups or individuals in specific historical situations. These processes always imply complex interplays between often deliberately generated oblivion or forgetfulness, transformation in time, as well as “spotlight” illuminations of a selection of intermedial imag(o)inations/historical pictures (“Geschichtsbilder,” Schmid 2000) – and this in a way which “open[s] up to a future horizon” (Assmann 2011, 396). Following this line, memories are to be conceived of as activities which imply various (inter)-medial processes and which revitalize and reshape sign clusters in order to initiate power plays of remembrance. These activities go along with procedures of legitimation of social, political, and religious powers and interests (as we can currently see in numerous intermedial legitimations of the Ukraine war), and processes of forming of identities or of strengthening feelings of belonging to social groups and classes, but also with de-legitimations of these memories. Remembrance or/and oblivion always imply semiological operations and activities between several modes of memory, especially between our storage memory (i.e., as an unstructured assembly of sedimented and autonomous elements of signs and social elements of knowledge) and our functional memory (i.e., inhabited memories, deliberately used by social groups, memories which are strongly linked to the present) (Assmann 2011; Hartnack 2018). These operations are accompanied by power plays on many social, institutional, ideological, and medial levels, as we could already see in our paradigmatic studies of Abu Ghraib and Sina Weibo; they take place in intermedial networks. The Tapestry of Bayeux and Intermedial Power Plays of Memories We conclude our encounters of intermedial power plays with a visit of the unique Tapestry of Bayeux (which was finished after the year 1066). Over a length of almost 70 meters, and with a 58 “scenes,” constituted by pictural and textual elements, this

126

J. E. Mueller

impressing artifact tells an intermedial story of the Norman Conquest of England by the Norman, William the Conqueror, in the eleventh century. In our postmodern times, it can only be recommended to walk along this tapestry and to generate a quasi-cinematographic perception and experience by replacing the movement of the (nowadays usually 24 frames) of a film role by the movement of our bodies and eyes in front of this masterpiece and by this constituting a narration. This masterpiece, which is characterized by a mixture of iconographic networks and a “visual polyphony” (Bernstein 1986, 124 ff.), deploys its functions in different social and historical contexts over nine and a half centuries. If, during the first years after its production, it might have primarily served as a core narration, “lieu de mémoire” (Nora 1984–1992), and central element of functional memory in a political and clerical-narrative space, in order to legitimize the success and political power of William the Conqueror and the defeat of Harold and its English troops, it later changed its status and shifted into the vast area of storage memory. Even if from fifteenth to eighteenth century the tapestry was being hung annually in Bayeux Cathedral during the week of the Feast of St. John the Baptist, it was widely “forgotten” and did not really seem to have a general or nationwide value. This situation changed, when, in the first half of the eighteenth century, some drawings of “pictures” of the tapestry were discovered in a Parisian Museum and when, in 1724, it was referred to by a member of the French Academy, which after that was followed by a rediscovery and a discursive series of further research. Since then, the Tapisserie de Bayeux has served again as a central element of French functional memory, as a sort of intermedial reminder of historical greatness of the Norman ancestors and of the Catholic church. We would go too far into details if we listed all the adventures and travels of this tapestry during the nine and a half centuries, for example, during the period of the French Revolution, during Napoleon’s times, or during the occupation of France by the Nazis, which it had to overcome. Yet, some further remarks might be allowed regarding its functional, i.e., social, and political power during its existence in the twentieth century. In the 1940s, German anthropologist, researcher, and SS-man Herbert Jankuhn (who had discovered the Viking city Haithabu in the 1930s) carried out an intensive study of the tapestry in order to get deeper insights into the common roots of Vikings, Normans, and Norman culture which should manifest themselves in the tapestry, which then was misinterpreted and misused as a highlighting of a masterpiece of Norman and Arian culture, as well as of Arian supremacy. The goal of this action was a deliberate insertion of this intermedial masterpiece into the ideology and functional memory of the National Socialists. In 1944, an ironical remediation of the tapestry found itself on the front page of the journal New Yorker where it transformed and changed the direction of the Norman Conquest into a comic scenery with a conquest of the Allied Forces in World War II, almost on the same historical beeches very close to Bayeux. This historical play with a central element of French and Norman cultural memory into a memory of the D-Day and the invasion of the Allied Troops activates and integrates the tapestry into sociopolitical networks,

6

Intermedialities, Societies, and Power Histories

127

networks of memories, ideological and military power plays, as well as sceneries of Second World War. During the 950 years of its intermedial existence, the Bayeux Tapestry has moved between different layers and social functions of memories. The memorizing power of the Tapestry has been reused with many different loadings of meaning in many different sociohistorical, ideological, and political contexts, whereby its mysterious and polysemic status still gives reason for various new forms of remembrance. The preliminary last step of the historical recyclings and remediations of the Tapestry of Bayeux is to be seen in its new function as “meme generator” (2022) on interactive websites.

Conclusion A Word at the End or Towards a History of Social Archetypes of Intermedial Functions and Power Plays Having reached the end of our historiological approaches to power plays between intermedialities and societies, we recall the starting point of this journey, Elleström’s statement that “everything is intermedial.” His succinct sentence directs us to the central challenge of all intermedial research and historiography, the need to make reasoned choices of research fields of the “everything” and of theoreticalmethodological tools to be employed. Hence, histories of intermediality must focus on clearly delineated fields at a time, while shifting others into the horizon of the respective research axis, and write (or reconstruct through audiovisual representations, etc.) one of numerous possible histories. Our results of a search into intermedialities, societies, and power histories give first evidence for the relevance of an application and implementation of an intermedial network research axis. Given the abundance and complexity of intermedial phenomena and processes, only a few paradigmatic cases in this broad field of our (still) epistemic search concept could come into focus. We made choices, which, on one hand, should give an idea of the large historical and phenomenological range of intermedial power plays and on the other hand, should also test the applicability/viability of our approach of an intermedial network research axis. We aimed at the description and explanation of some central historical connections between intermedial processes, social functions, and power plays, which take place on several levels, stretching from individual, to social, institutional, political, medial, or generic realms. Before the background of our seven types of historical intermedial power plays, we must stay aware of the fact that in many cases, these – for pragmatic reasons of research – isolated types of power plays go along with further types or kinds and also with less dynamic “mono media.” However, with our approach, we have put a deliberately and comprehensibly chosen accent on intermedia. Finally, in the spirit of Roland Barthes (2002), these historical and historiological choices have led us to theoretical assumptions.

128

J. E. Mueller

Before the background of Barthes’ dictum, Lars Elleström’s research démarche could be described as inversely directed. Elleström’s deep reflections on the theoretical foundation of the concept of intermediality in turn lead to histories of intermedialities, as we can see from some of his remarks, for instance, on “the socially determined existence” of media types (or “genres,” I would like to add, Müller 2009, 2014), of technical media (Elleström 2021, 65), or on the “crossspatiotemporal” representations of “cross-modalities” (Elleström 2021, 69). Our intermedial histories have acquired some qualities of a “historicized” phenomenology. Such a “historicized phenomenology” might imply a contradiction in itself, given the fact that there is a theoretical and methodological gap between phenomenological and historiological thinking, but there exist procedures, which allow to juxtapose or to relate phenomenology and history, for example, in some sort of “parallel actions.” That kind of “hybrid” approaches seem to be rather helpful for historiological studies of intermedial power plays. (See our remarks in section ““Everything Is Intermedial”: Some Preliminary Notes.”) The categories and perspectives we might use for this enterprise oscillate in between semiological, sociohistorical, phenomenological, economic, and scientific states or statuses, whereby the results of their application might be questioned with regard to their plausibility and also by perspectives and assumptions of counterfactual or alternative histories (Demandt 2001; Wenzelhuemer 2009). A conjectural historiography (Schmid 2009; Weber 2000; Hartnack 2018) deals with undone historical developments or possibilities. Since historical events and processes – also intermedial phenomena – can never be completely grasped, the task is to “always think critically of the possibility of alternative causes and courses” (Tellenbach 1994, 298; Hartnack 2018, 47). Against this background, it becomes clear that alternative histories, which even today are sometimes still considered marginal or even “esoteric,” can indeed add relevant aspects to “fixed and written” histories. They serve not least as a corrective of reductive and one-sided historical findings. “We have to hold on to counterfactuals at all costs – and we have to provide them with the status of untouchable epistemological instruments – because, in fact, there are ‘many pasts’, too. All but one of them may be deficient, outright false, often intentionally, criminally misleading – as there are dishonorable, disfiguring ‘conspiratorial readings’ of history – but as hardly anyone knows anything about the real history, all those crude and often outright insane readings coexist with ‘the real thing’ – they do have more concrete influence than sane, rational, reasoned and correct accounts” (Schmid 2009, 80). While following our research axes of a history of intermedial networks and social functions, we should stay aware that our histories are only a selection of many possible alternative histories and that there are many switching points in history, where social or technological developments could have taken a completely different direction and development. It would also seem promising to undertake a confrontation and rapprochement in the theoretical-historiological interstices between Elleström’s communication model of intermedial modalities and modes with a concept of social functions and of sociohistorical archetypes of action, of archetypes which are triggered by and go along with different intermedial (power-) processes. The archetypes of that “what

6

Intermedialities, Societies, and Power Histories

129

intermedialities can do to us” should be located on an intermedial level between theory and history and might stretch from networking activities, ludifications, sympraxis, power plays, to an “effet de vie” an “effect of life” (Münch 2004). Let us finally not forget that such kind of an enterprise always confronts us with the evasive tendencies of intermedia phenomena which are only accessible through the traces they have left. If, in the reader’s eyes, our text functioned not only as a summarizing and concluding overview but also as an impulse for the continuation and expansion of historiological research, it will have fulfilled its intended function.

References Allen, Robert, and Douglas Gomery. 1985. Film history. Theory and practice. New York: Knopf. Altman, Rick. 1999. Film/genre. London: BFI. Assmann, Aleida. 2011. Cultural memory and western civilization. Arts of memory. Cambridge: Cambridge University Press. Barthes, Roland. 2002. OEuvres Complètes. (Nouvelle édition revue, corrigée et présentée par Éric Mart.) Tome I-V. Paris: Seuil. Baudry, Jean-Louis. 1978. L’effet cinéma. Paris: Éditions Albatros. ———. 2003. Ideologische Effekte erzeugt vom Basisapparat. In Der kinematographische Apparat, ed. Robert F. Riesinger, 27–39. Münster: Nodus. Bauer, Ingrid, Helga Embacher, Ernst Hanisch, Albert Lichtblau, and Gerald Sprengnagel, eds. 2004. >kunst >kommunikation >macht. Sechster Österreichischer Zeitgeschichtetag 2003. Innsbruck/Wien/München/Bozen: StudienVerlag. Baxandall, Michael. 1987. Die Wirklichkeit der Bilder. Malerei und Erfahrung im Italien des 15. Jahrhunderts. Frankfurt a. Main: Athenäum. Bayeux. 2022. Meme generator. https://www.eteknix.com/bayeux-tapestry-meme-maker/; https:// www.cnet.com/culture/internet/bayeux-tapestry-medieval-meme-generator-historic-tale-con struction-kit/ Baym, Geoffry. 2014. Stewart, O’Reilly, and the rumble 2012: Alternative political debate in the age of hybridity. Popular Communication: The International Journal of Media and Culture 12 (2): 75–88. Berger, Peter L., and Thomas Luckmann. 1966. The social construction of reality. A treatise in the sociology of knowledge. New York: Doubleday. Bernstein, David L. 1986. The mystery of the Bayeux tapestry. London: Guild Publishing. Blurton-Jones, N.G. 1987. Tolerated theft, suggestions about the ecology of sharing, hoarding and scrounging. Information (International Science Council) 26 (1): 31–54. https://doi.org/10.1177/ 053901887026001002. Bolter, Jay David, and Richard Grusin. 1999. Remediation. Understanding new media. Cambridge, MA/London: MIT. Bolter, Jay David. 2019. The digital plenitude. The decline of elite culture and the rise of new media. Cambridge, MA/London: MIT. Bourdieu, Pierre. 1966. Champ intellectuel et projet créateur. Les Temps Modernes 246: 865–906. ———. 1993. The market of symbolic goods. In The field of cultural production: Essays on art and literature, ed. Tom Huhn et al., 112–141. New York: Columbia University Press. Bouwknegt, Hans. 2011. Beyond the simulacrum: A conceptual semiotic approach for the analysis and design of digital media. Münster: Nodus. Bruhn, Jørgen, and Anne Gjelsvik. 2018. Cinema between media. An intermediality approach. Edinburgh: Edinburgh University Press. Bruhn, Jørgen, and Henriette Thune. 2018. In between life and death: Sophie Calle’s Rachael, Monique. In The power of the in-between: Intermediality as a tool for aesthetic analysis and

130

J. E. Mueller

critical reflection, ed. Sonya Petersson, Christer Johansson, Magdalena Holdar, and Sara Callahan, 25–48. Stockholm: Stockholm University Press. Brunetta, Gian Piero. 1997. Il viaggio dell’Icononauta. Venezia: Marsilio. Castells, Manuel. 1996. The information age: Economy, society and culture. Vol. 1. Malden, MA: Blackwell. Chadwick, Andrew, James Dennis, and Amy P. Smith. 2016. Politics in the age of hybrid media: Power, systems, and media logics. In The Routledge companion to social media and politics, ed. Axel Bruns, Gunn Enli, Eli Skogerbo, Anders Larsson, and Christian Christensen, 7–22. New York: Routledge. Cisneros, James. 2007. The remains to be seen. Intermediality and institution. In Intermédialité et socialité [Intermediality and sociality], ed. Marion Froger and Jürgen E. Müller, 15–28. Münster: Nodus. Clüver, Claus. 2019. From the “mutual illumination of the arts” to “studies of intermediality”. IJSVR 3, 2: 63–74. Last accessed 07 May 2022. Curran, James. 2002. Media and power. London/New York: Routledge. Daniel, Sharon. 2007. Public secrets. https://digitalartarchive.siggraph.org/artwork/sharon-danielpublic-secrets/ Demandt, Alexander. 2001. Ungeschehene Geschichte. Ein Traktat über die Frage: Was wäre geschehen, wen . . .? Göttingen: Vandenhoek und Ruprecht. Elias, Norbert. 2012. In On the process of civilisation, ed. Stephen Mennell, Eric Dunning, Johan Goudsblom, and Richard Kilminster. Dublin: University College Dublin Press. Elleström, Lars, ed. 2010. Media borders, multimodality and intermediality. Houndmills: Macmillan. Elleström, Lars, Massimo Fusillo, and Mattia Petricola. 2020. Everything is intermedial. A conversation with Lars Elleström. Between 10/20: 27–45. https://doi.org/10.13125/2039-6597/ 4443. Last accessed 07 May 2022. Elleström, Lars, ed. 2021. Beyond media borders, volume 1: Intermedial relations among multimodal media. London: Palgrave Macmillan. Faulstich, Werner. 1996. Medien und Öffentlichkeiten im Mittelalter. 800–1400. Göttingen: Vandenhoek & Ruprecht. Fendler, Ute. 2021. Polyrhythmic gestures: Relational perspectives on/from verbal, audio/visual and performative arts across the African continent. https://www.youtube.com/watch? v¼SN2XGtEGHrI (New year lecture of Africa multiple cluster of excellence, University of Bayreuth). Fiske, John. 1993. Power plays, power works. New York: Verso. ———. 2010. Understanding popular culture. 2nd ed. London: Routledge. Foucault, Michel. 2008. The history of sexuality. Munich: Penguin. Franck, Georg. 2018. The economy of attention. New York: Sage. Gomery, Douglas. 1992. Shared pleasures. Wisconsin: University of Wisconsin Press. Gumbrecht, Hans Ulrich. 1992. Making sense in life and literature. Minnesolta: Minnesota University Press. Guynn, William. 2014. The art of national projection: Basil Wright’s Song of Ceylon. In Documenting the documentary: Close readings of documentary film and video, ed. Barry Keith Grant and Jeannette Sloniowski, 64–80. Detroit: Wayne State University Press. Guiyoba, François, Robert Fopa Kuete, and Albert Jiatsa Jokeng, eds. 2022. Le discours scientifique comme pratique intermédiale en Arts, Lettres et Sciences humaines. Paris: Éditions L’Harmattan. Hall, Stuart. 1973. Encoding and decoding in the television discourse. Birmingham: Centre for Contemporary Cultural Studies. ———. 1974. Media power: The double bind. Journal of Communication 24 (4): 19–26. Hartnack, Marina. 2018. “Was wäre geschehen, wenn . . .”. Eine Analyse der filmischen Konstruktion alternativhistorischer Geschichtsbilder des Nationalsozialismus. Münster: Nodus. He, Yuan. 2018. Krisen in chinesischen Medien und deren mediale Konstruktion – Insbesondere auf der Mikroblogging-Plattform Sina Weibo. Münster: Nodus.

6

Intermedialities, Societies, and Power Histories

131

Helbo André. 2022. Lire le spectacle vivant. Bruxelles: Académie royale des Sciences, des Lettres et des Beaux-Arts de Belgique. Hepp, Andreas. 2004. Netzwerke der Medien. Medienkulturen und Globalisierung. Wiesbaden: Verl. für Sozialwissenschaft. Hepp, Andreas, Friedrich Krotz, and Carsten Winter, eds. 2008. Connectivity, networks and flows. Conceptualizing contemporary communications. Cresskill, N.J: Hampton Press: The Hampton Press communication series. Hofmansthal, Hugo von. 1911. Jedermann. Das Spiel vom Sterben des reichen Mannes. Berlin: S. Fischer Verlag. Huizinga, Johan. 1949. Homo Ludens. A study of the play-element of culture. London: Routledge & Kegan Paul. Jacobs, Emma. 2017. With archaeoacoustics, researchers listen for clues to the prehistoric past. Atlas Obscura. https://www.atlasobscura.com/articles/archaeoacoustics. Last accessed 07 May 2022. Jenkins, Henry. 2006. Convergence culture: Where old and new media collide. New York: New York University Press. ———. 2009. Revenge of the origami unicorn: Seven core concepts of transmedia. Available online at The Revenge of the Origami Unicorn: Seven Principles of Transmedia Storytelling. http://henryjenkins.org/blog/2009/12/the_revenge_of_the_origami_uni.html. Last accessed 07 May 2022. ———. 2014. Convergence – I diverge!. Technology Review. Online, last visited 24 September 2014: http://pps-rcastiglione.googlecode.com/svn-history/r123/trunk/MKT/mail_6/converge_ cultureJJ2.pdf Johansson, Christer, and Sonya Petersson. 2018. Introduction. In The power of the in-between: Intermediality as a tool for aesthetic analysis and critical reflection, ed. Sonya Petersson, Christer Johansson, Magdalena Holdar, and Sara Callahan, 1–21. Stockholm: Stockholm University Press. Johansson, Christer. 2018. Old and new media: On the construction of media history. In The power of the in-between: Intermediality as a tool for aesthetic analysis and critical reflection, ed. Sonya Petersson, Christer Johansson, Magdalena Holdar, and Sara Callahan, 375–406. Stockholm: Stockholm University Press. Jost, François. 2009. La Télé-réalité: Grandeur et misère de la téle-réalité. Paris: Éditions Le cavalier bleu. Kircher, Athanasius. 1671 (2021). Athanasii Kircheri Ars magna lucis et umbrae, in X. libros digesta; quibus admirandae lucis & umbrae in mundo, atque adeo universa natura, vires effectusque uti nova, ita varia novorum reconditiorumque speciminum exhibtione, ad varios mortalium usua, panduntur ; editio altera priori multa auctior. Delhi: True World of Books. Kloepfer, Rolf. 1999. Sympraxis. Dresden: Dresden University Press. Koubek, Jochen. 2008. Vernetzung als kulturelles Paradigma. Eine Kulturtheorie des Internet. Saarbrücken: Verlag Dr. Müller. Larousse. 1872. Grand Dictionnaire Universel Larousse. Paris: Larousse. Luckmann, Thomas. 1989. Prolegomena to a social theory of communicative genres. Slovene Studies 1/2: 159–167. Luke, Timothy W. 1989. Screens of power. Ideology, domination and resistance in informational society. Urbana\Chicago: University of Illinois Press. Mannoni, Laurent. 1999. Le grand art de la lumière et de l’ombre. Archéologie du cinéma. Paris: Nathan. Manovich, Lev. 2001. The language of new media. Cambridge: MIT Press. Mitchell, W.J.T. 1994. Picture theory. Chicago: University of Chicago Press. Moser, Walter. 2007. L'interartialité: pour une archéologie de l’intermédialité. In Intermédialité et socialité, ed. Marion Froger and Jürgen E. Müller, 69–92. Münster: Nodus. Müller, Jürgen E., ed. 1987. Texte et Médialité. Mana (Mannheim Analytiques): Mannheim.

132

J. E. Mueller

———. 1995. Dokumentation und Imagination: Zur Ästhetik des Übergangs im Dokumentarfilm. Transit Levantkade. In Perspektiven des Dokumentarfilms, ed. Manfred Hattendorf, 127–148. München, diskurs film. ———. 1996. Intermedialität. Formen moderner kultureller Kommunikation. Münster: Nodus. ———. 1997. Literaturwissenschaftliche Rezeptions- und Handlungstheorien. In Neue Literaturtheorien – Eine Einführung, ed. Klaus-Michael Bogdal, 176–200. Göttingen, Vandenhoeck & Ruprecht. ———. 1998. Intermedialität als poetologisches und medientheoretisches Konzept. Einige Reflexionen zu dessen Geschichte. In Intermedialität. Theorie und Praxis eines interdisziplinären Forschungsprogramms, ed. Jörg Helbig, 31–40. Berlin: Erich Schmidt Verlag. ———. 2000. L’intermédialité, une nouvelle approche interdisciplinaire: perspectives théoriques et pratiques à l’exemple de la vision de la télévision. Cinémas 10.2–3: 105–134. ———. 2001. Tele-Vision als Vision: Einige Thesen zur intermedialen Vor- und Frühgeschichte des Fernsehens (Charles François Tiphaigne de la Roche und Albert Robida). In Autoren, Automaten, Audiovisionen. Neue Ansätze zur Medienästhetik und Telesemiotik, ed. W.B. Hess-Lüttich Ernest, 187–208. Wiesbaden: Westdeutscher Verlag. ———. 2003. Dispositiv – intermedial? Einige Gedanken zu den dispositiven Zwischen-Spielen des Films. In Der kinematographische Apparat, ed. Robert F. Riesinger, 247–260. Münster: Nodus. ———., ed. 2008a. Media encounters and media theory. Nodus: Münster. ———. 2008b. Intermedialität und Medienhistoriographie. In Intermedialität analog/digital. Theorien, methoden, analysen, ed. Joachim Paech and Jens Schröter, 31–46. München: Wilhelm Fink Verlag. Müller, Juergen E. 2009. Mediale Recyclings und Re-Mediationen im digitalen Zeitalter – zur Auflösung des `Werk´-Begriffs. In Konvergenz der Medien – Konvergenz des Rechts? ed. Jörg Gundel, Peter W. Heermann, and Stefan Leible, 19–30. Jena: Wissenschaftliche Verlagsgesellschaft. Müller, Jürgen E. 2010a. Intermedialität digital: Konfigurationen, Konzepte, Konflikte. In Intermediale Inszenierungen im Zeitalter der Digitalisierung. Medientheoretische Analysen und ästhetische Konzept, ed. Andy Blättler, Doris Gassert, Susanna Parikka-Hug, and Miriam Ronsdorf, 17–40. Bielefeld: transcript. ———. 2010b. Intermediality revisited: Some reflexions about basic principles of this Axe de pertinence. In Media borders, multimodality, and intermediality, ed. Lars Elleström, 237–252. Basingstoke: Palgrave Macmillan. ———. 2012. Populärkultur, mediale Recyclings, soziale Räume und ökonomische Prozesse – oder Anmerkungen zu einer intermedialen Ökonomie der Popikone Michael Jackson. In Transformationen des literarischen Feldes in der Gegenwart, ed. Klaus Michael Bogdal, Achim Geisenhanslüke, and Heribert Tommek, 169–185. Berlin: Synchron. ———. 2014. Genres analog, digital, intermedial – zur Relevanz des Gattungskonzepts für die Medien- und Kulturwissenschaft. In Intertextualität, Intermedialität, Transmedialität. Zur Beziehung zwischen Literatur und anderen Medien, ed. Volker Dürr and Tobias Kurwinkel, 70–102. Würzburg: Königshausen & Neumann. Mueller, Juergen E. 2015a. Intermediality in the age of global media networks – Including eleven theses on its provocative power for the concepts of "convergence," "transmedia storytelling" and "actor network theory". SubStance, Intermedialities 44 3 (138): 19–52. Müller, Jürgen E. 2015b. And where do we go next? Some Current and Future Fields of Research in Media Culture and Media Economy – Illustrated by the Example of Michael Jackson’s Thriller. In University of Otago, Dunedin, Media, Film and Communication. Working Paper Series 2, https:// www.otago.ac.nz/mfco/research/otago040229.html. Last accessed 09 May 2022. ———. 2020a. L’intermédialité hier et aujourd’hui: retour sur un concept encore en devenir – avec quelques pas archéologiques dans les enjeux intermédiatiques de grottes. In FORMES ET (EN) JEUX DE L’INTERMÉDIALITÉ DANS L’ESPACE EUROPÉEN d’hier à aujourd’hui, ed. Patricia Viallet, 23–54. Würzburg: Königshausen & Neumann.

6

Intermedialities, Societies, and Power Histories

133

———. 2020b. “Intermediality”: Some Comments on the Current State of Affairs of a Search Concept. Scripta Uniandrade. Revista da Pós-Graduação em Letras 18, 1: Parts 1 and 2, 1–30. (Revised and extended version of “L’intermédialité hier et aujourd’hui: retour sur un concept encore en devenir. . .”). ———. 2022. A re-visiting of some (pre-) historical intermedial playgrounds of cinema. In Intermedial encounters, studies in honour of Ágnes Pethő, ed. Melinda Blos-Jáni, Hajnal Király, Mihály Lakatos, Judit Pieldner, and Katalin Sándor, 65–76. Cluj-Napoca: Sciencia Kiadó. Münch, Marc-Mathieu. 2004. L’effet de vie ou le singulier de l’art littéraire. Paris: Honoré Champion Éditeur. Nickolson, Mathew. 2007. Sport and the media: Managing the nexus. Amsterdam: Elsevier. Noguez, Dominique. 1979. Éloge du cinÉma expÉrimental: dÉfinitions, jalons, perspectives. Paris: Centre Georges Pompidou. Nora, Pierre. 1984–1992. Les lieux de mémoire. 7 volumes. Paris: Gallimard. Odin, Roger. 1994. Sémio-pragmatique du cinéma et de l’audiovisuel. Modes et institutions [Semio-pragmatics of the cinema and of the audiovisual. Modes and institutions]. In Towards a pragmatics of the audiovisual. Theory and history, ed. Jürgen E. Müller, 33–46. Münster: Nodus. Pethő, Ágnes. 2020. Cinema and intermediality: The passion for the in-between. Newcastle upon Tyne: Cambridge Scholars Publishing. Rajewsky, Irina. 2002. Intermedialität. Tübingen/Basel: A. Francke Verlag. ———. 2014. Intermedialität, remediation, multimedia. In Handbuch Medienwissenschaft, ed. Jens Schröter, Simon Ruschmeyer, and Elisabeth Walke, 197–206. Stuttgart/Weimar: J.B. Metzler. Reznikoff, Iegor. 2000. Prehistoric paintings, sound and rocks. In Studien zur Musikarchäologie III (Colloque International, Michaelstein), ed. Ellen Hickmann, Anne D. Kilmer, and Ricardo Eichmann, 39–56. Rahden (Westfalien): Verlag Marie Leidorf Gmbh. ———. 2002. Soundscapes in the past: Adding a new dimension to our archaeological picture of ancient cultures. The conversation. http://theconversation.com/soundscapes-in-the-past-adding-anew-dimension-to-our-archaeological-picture-of-ancient-cultures-80861. Last accessed 07 May 2022. ———. 2010. La dimension sonore des grottes paléolithiques et des rochers à peintures. In. L’art pléistocène dans le monde / Pleistone art of the world /Arte pleistoceno en el mundo, ed. Jean Clottes, Actes du Congrès IFRAO, Tarascon-sur-Ariège, septembre 2010, CD-45-56. Robida, Albert. 1883. Le Vingtième Siècle. Genève\Paris: Editions Slatkine (1981). Schmid, Georg. 2000. Die Geschichtsfalle. Über Bilder, Einbildungen und Geschichtsbilder. Wien: böhlau. ———. 2009. Counterfactuals and future histories. Retrospective imagining as an auxiliary for the scenarios of expectance. Historical Social Research 34 2: 74–87. Schröter, Jens. 2008. Das ur-intermediale Netzwerk und die (Neu)Erfindung des Mediums im (digitalen) Modernismus. In Intermedialität analog/digital. Theorien, methoden, analysen, ed. Joachim Paech and Jens Schröter, 579–601. München: Wilhelm Fink Verlag. Schüttpelz, Erhard. 2007. Ein absoluter Begriff. Zur Genealogie und Karriere des Netzwerkkonzepts. In Vernetzte Steuerung. Soziale Prozesse im Zeitalter technischer Netzwerke, ed. Stefan Kaufmann, 25–46. Zürich: Chronos. Schutz, Alfred, and Thomas Luckmann. 1973. The structures of the life-world. Vol. 1 and Vol. 2. Evanston: Northwestern University Press. Steinmaurer, Thomas. 2015. Mobile Individuen im Netz. Zur Theorie und Geschichte mediatisierter Kommunikation. Wiesbaden: Springer VS. Suzman, James. 2020. Work. A history of how we spend our time. London/Oxford/New York/New Delhi/Sidney: Bloomsbury Circus. Tellenbach, Gerd. 1994. “Ungeschehene Geschichte” und ihre heuristische Funktion. Historische Zeitschrift 258: 297–316.

134

J. E. Mueller

Thibault, Alain, Raymond, Miguel, Collin, Jacques and Panet-Raymond. 1985. OUT, Chapter: GOD’S GREATEST GIFT IS . . .). (P.R.I.M., Productions, Réalisations Indépendantes de Montréal) Thrill the world. 2015 ff. https://thrilltheworld.com/. Last accessed 07 May 2022. Till, Rupert. 2014. Sound archaeology: Terminology, Paleolithic cave art and the sound scape. World Archaeology 46 (3): 292–304. de la Roche, Tiphaigne. 1760. Giphantie. Paris: Hachette Livre. Warning, Rainer. 1971. Ritus, Mythos und geistliches Spiel. In Terror und Spiel. Probleme der Mythenrezeption, ed. Manfred Fuhrmann, 211–239. München: Wilhelm Fink. Weber, Gregor. 2000. Vom Sinn kontrafaktischer Geschichte. In Virtuelle Antike. Wendepunkt der Alten Geschichte, ed. Kai Bordersen, 11–23. Darmstadt: Wissenschaftliche Buchgesellschaft. Weber, Max. 1922. Wirtschaft und Gesellschaft. Tübingen: Winckelmann (1980). Wenzelhuemer, Roland. 2009. Counterfactual thinking as a scientific method. Historical Social Research 34 (2): 27–54. Wiehl, Anna. 2019. The ‘new’ documentary nexus. Networked/networking in interactive assemblages. Newcastle: Cambridge Scholars Publishing. Wolf, Werner. 2002. Intermedialität – ein weites Feld und eine Herausforderung für die Literaturwissenschaft. In Literaturwissenschaft – intermedial, interdisziplinär, ed. Herbert Foltinek and Christoph Leitgeb, 163–192. Wien: Verlag der Österreichischen Akademie der Wissenschaften. ———. 2018. Selected essays on intermediality by Werner Wolf (1992–2014): Theory and typology, literature – music relations, transmedial narratology, miscellaneous transmedial phenomena. Leiden/Boston: Brill-Rodopi. Woodburn, James. 1982. Egalitarian societies. Man, the Journal of the Royal Anthropological Institute 17 (3): 431–451.

7

Montreal School of Intermediality: Beyond Media Studies Re´my Besson

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 1: Reflexive Definition of the Contours of the Montreal School . . . . . . . . . . . . . . . . . . . . . . . . . Part 2: What the Montreal School Is Not . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 3: The Montreal School and “Living Together” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 4: Affinities with Other Intermedial Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 5: Internal Variation of the Montreal School . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

136 136 138 142 150 153 156 156

Abstract

This chapter presents the intermedial perspective developed by the Montreal School. In order to avoid yet another debate on the definition of the term intermediality, the adopted approach is rather to situate this work at the articulation of other intermedial approaches (mainly centered on co-presence, transfer, and the emergence of new media) and media archaeology, to insist on either the differences (Part 2) or the complementarities between these viewpoints (Part 4). It is also a question of showing the heuristic effects of the work of the main members of the aforementioned school (Bem, Despoix, Froger, Mariniello, Méchoulan, Villeneuve) from a corpus of texts mainly taken from a double issue of the journal Communication et langages, entitled Études intermédiales. À la rencontre de l’École de Montréal (2021). What emerges is the importance of the notions of milieu and mediality, but also of “living together.” It is a way of insisting on the fact that the singularity of this school rests on the purpose of the research carried out by its members, the latter aiming less at the study of media for media’s sake, than at a reflection of a political nature (Part 3). The concluding section insists on the perspective put on the above-mentioned notions by R. Besson (*) University of Montreal, Montreal, QC, Canada e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_7

135

136

R. Besson

members of the school itself (Larrue, Vitali-Rosati). This avoids giving the impression that the school of thought in question corresponds to a unified perspective (Part 5). Keywords

Montreal school · Living together · Milieu · Mediality

Introduction This chapter presents the intermedial perspective developed by the Montreal School. In order to avoid yet another debate on the definition of the term intermediality, the adopted approach is rather to situate this work at the articulation of other intermedial approaches (mainly centered on co-presence, transfer, and the emergence of new media) and media archaeology, to insist on either the differences (Part 2) or the complementarities between these viewpoints (Part 4). It is also a question of showing the heuristic effects of the work of the main members of the aforementioned school (Bem, Despoix, Froger, Mariniello, Méchoulan, Villeneuve) from a corpus of texts mainly taken from a double issue of the journal Communication et langages, entitled Études intermédiales. À la rencontre de l’École de Montréal (2021). What emerges is the importance of the notions of milieu and mediality, but also of “living together.” It is a way of insisting on the fact that the singularity of this school rests on the purpose of the research carried out by its members, the latter aiming less at the study of media for media’s sake, than at a reflection of a political nature (Part 3). The concluding section insists on the perspective put on the above-mentioned notions by members of the school itself (Larrue, Vitali-Rosati). This avoids giving the impression that the school of thought in question corresponds to a unified perspective (Part 5). But first, it is useful to look at the name itself, as there is nothing obvious about it; on the contrary, it should be considered as a fact of discourse.

Part 1: Reflexive Definition of the Contours of the Montreal School Starting from the observation that the name “Montreal School” is a fact of discourse means that it is not only a matter of questioning the choice of terms according to lexical or stylistic variables, but also of placing them in the context in which they appear and of grasping the discursive and strategic issues associated with them (Guerin 2002). In this respect, the use of the term “school” should be questioned. Indeed, whether in the field of art history or in the humanities and social sciences, it has been strongly criticized throughout the twentieth century. The terms “focus,” “center,” “movement,” “tradition,” “networks,” “approach,” and “collective” are usually preferred (Fabiani 2005; Peltre 2007). Similarly, the assignment to a place,

7

Montreal School of Intermediality: Beyond Media Studies

137

in this case a city, is not self-evident. Sometimes, it is a researcher or a concept that is put forward. In order to understand the formation of this syntagm, it is relevant to examine its context of emergence (to identify the first researchers to use it and the period in which this usage becomes recurrent). It should be noted that the term is not present in the first Montreal publications on intermediality. It does not appear in communications (posters, programs, etc.) nor in the communications of the researchers at the first Montreal conference in 1997. Nor does it appear in the founding texts of the Centre de Recherche sur l’intermédialité (CRI, later renamed CRIalt) or in the founding of the journal Intermédialités in 2003. At the time, researchers in Comparative Studies (Philippe Despoix, Silvestra Mariniello, Éric Méchoulan, Suzanne Paquet, Will Straw, Johanne Villeneuve) and in Film Studies (André Gaudreault in particular), mainly at the University of Montreal, who accounted for many of those who were the first to adopt this approach, did not use this expression. Most of the occurrences were formulated some 20 years after the founding colloquium. They are found in the writings of CRIalt researchers such as Philippe Despoix (2014), Éric Méchoulan (2017), Caroline Bem, Rémy Besson, Suzanne Beth, and Claudia Polledri (2017), and Jean-François Vallée (2017). The founding of the Montreal School is thus a retrospective naming act. It qualifies a group that had already organized numerous research projects, collective publications, and international conferences. It should also be noted that it is a case of self-naming. The identification of this temporality and of these first speakers leads us to question once again the reasons that led to the choice of the term “school.” It is possible to hypothesize that it is less a reference to nineteenth-century historiography, which was fond of such divisions of the academic field, than to antiquity, during which it was not uncommon for philosophers and their students to meet in a place that was both physical and symbolic. However, this is not enough to explain the emergence of the expression. It leads us to question the strategic dimension of such a naming act. In short: what leads these researchers to feel the need for such an expression 20 years after starting their research? The historiography of the founding of schools in the humanities and social sciences developed the idea that the creation of a school is less often the result of a renewal of the problems of a research field, than to mark widespread dissent. The historiographer Christopher Lloyd explains: Schools are possible, even necessary, when beliefs about the chief causal characteristics and possible understandings of the nature of the world are indeterminate, lacking any undisputed foundation of warranted true belief. (2009: 375)

To create a school is to claim a singular perspective in a field of study with uncertain or contested contours. This designation can, therefore, be seen as a way of both acknowledging that other ways of doing intermedia studies exist, while at the same time distinguishing itself from them. It is therefore essential to present the approaches from which the school differentiates itself (Part 2). For the moment, it is important to question the existence of a definition of the contours or issues of this

138

R. Besson

school. This will lead us to question the links between institutional membership of the Center (CRIalt) and identification with the school. Although several researchers have used the expression since 2014, the formulation of a precise definition of its stakes and contours does not emerge at this exact moment. Thus, while Despoix uses the expression “Montreal School” in the title of a paper presented at the launch of the twentieth issue of the journal Intermédialités, he prefers in the introduction to the latter to talk about “the Montreal crucible” and “the international collective” (2012–13: 8–9). Finally, in 2017, when Méchoulan refers to the Montreal School, he puts the expression in inverted commas, preceded by “in what some have called.” The expression seems to be used more as a way of identifying a group of researchers associated with the CRIalt than as a clearly defined syntagm. A more complete formulation is however proposed by Éric Méchoulan in a double issue of the journal Communication et langages (Méchoulan, Éric, and Elsa Tadier 2021). We will come back to it throughout this text. For the moment, let us note that he clearly states that: The term ‘intermediality’ announces a project that is certainly interested in media, but its core purpose is mediality. Which is not the same. Of course, all mediality passes through media, and therefore we analyze media. But we do not analyze media for media’s sake, we analyze media for mediality effects. (35)1

While it is important to remember that the Montreal School is based on a tradition of comparative studies and a strong commitment to decompartmentalize disciplines (Despoix 2010), it would be possible to formulate the somewhat provocative hypothesis that the Montreal School is distinguished from other approaches to intermediality by the fact that it focuses less on media than on mediality. The latter term, as well as that of milieu, will be explored in the third part of this chapter. This insistence on the notion of mediality leads us to underline the centrality of politics, since, as Elsa Tadier clearly identified in her introduction to the issue of Communication et langages, “the relationship to politics inevitably arises when we take human mediations as our object” (2021: 19). Beforehand, it is important to present the intermedial perspectives that place the notions of co-presence, transference, and media emergence at their center.

Part 2: What the Montreal School Is Not Méchoulan’s refusal to be interested in media for media’s sake is, in fact, a relevant entry point for understanding the types of intermedia approaches from which the Montreal School distinguishes itself. This approach moves away from perspectives

1 Le terme d’«intermédialité» annonce un projet qui certes s’intéresse aux médias, mais le cœur de son propos est la médialité. Ce qui est différent. Bien sûr toute médialité passe par des médias, et de ce fait, nous analysons les médias. Mais nous n’analysons pas les médias pour les médias, nous analysons les médias pour leurs effets de médialité.

7

Montreal School of Intermediality: Beyond Media Studies

139

whose objective is, strictly speaking, a better understanding of the functioning or a keener appreciation of the form taken by a medium which is understood as a singular cultural production. This approach amounts to considering, as Irina Rajewski does in a historiographical text on the German tradition, a definition of intermediality: as a category for the concrete analysis of medial configurations, which includes artistic and cultural practices of all kinds (texts, films, performances, paintings, installations, comics, video games, internet blogs, logos, etc.). (2015: 34)2

Such an approach leads to the identification of forms or processes that fall within the domain of intermediality. This is an important distinction, since most members of the Montreal School insist that intermediality is not so much about the object under consideration than about the approach, the gaze, the method adopted by the researcher. Let us take the time to outline the main achievements of this perspective by following Rajewski’s text. Four types of phenomena are identified. One can speak of “intermedial transposition” [Medienwechsel], which corresponds to the passage of a text from one type of media to another. One can think of the adaptation of a book into a play or a film into a television series. It is also possible to study phenomena of plurimediality [Medienkombination] such as the co-presence of several types of media in the same cultural production. There are, for example, comic books in which not only drawings but also photographs are integrated, as well as plays in which film excerpts are projected onto a screen. Some researchers also work on “intermediary references” [intermediale Bezüge], which corresponds to the fact that one medium is quoted in a different media production. For example, a contemporary art exhibition can be mentioned in a novel or a video game in a blog post. Finally, Rajewski cites the category of transmediality, which corresponds to the variation of content in different media. If such an approach is adopted, intermediality may be contrasted with intramateriality. The latter refers to relationships between cultural productions that belong to the same medium. It can be a question of the remake of a film or the broadcasting of an extract from a television series in an episode of another television series. There is no shortage of examples. What emerges is that intermediality is about identifying thresholds/boundaries, which are located at the junction between media (both mass media and cinema, literature, theatre, etc.) and between cultural productions (a TV show, a film, a play, etc.). It is a way of highlighting similarities (allowing for crosscutting studies of common properties of certain media) and differences (avoiding a form of generalized relativism) between media and between cultural productions within these media (Elleström 2010). While acknowledging the importance of such approaches, Méchoulan explicitly distances himself from them when he writes in his 2021 text:

2

Comme catégorie pour l’analyse concrète de configurations médiales, ce qui inclut des pratiques artistiques et culturelles de toute sorte (textes, films, performances, peintures, installations, bandes dessinées, jeux vidéo, blogs internet, logos, etc.)

140

R. Besson

For some, this can be reduced only to transformation issues, displacement, adaptation, to the link between one medium and another, for example, or between one recognized form, mediated as such, and another. Let’s take the example of opera. What happens to it when it becomes a film? (30)3

The use of the terms “for some” and “only” reflects the fact that, to him, these effects of transfer and co-presence do not seem to be placed at the center of the investigation. In the same issue of Communication et langages, Froger and Bem say the same thing when they highlight the desire of a researcher from the Montreal School of intermediality, James Cisneros (current director of the journal Intermédialités) to: Refuse to take the usual path of studying filmic adaptations that presupposes a narrative that transcends its mediation and that identifies narrative differences that ultimately serve to distinguish the technical media of film and literature. (2001: 258)4

Following the hypothesis which was formulated previously, the Montreal School of intermediality is defined by the fact that it is not interested in media for their own sake when the latter are considered as quasi-synonyms of singular cultural productions (a film, a play, a performance, etc.). It should be noted that the school in question does not consider the emergence of “new media” as its main object either. This is a point worth exploring further, as some important Montreal researchers are adopting this perspective. This approach in terms of emergence leads to placing the media as such (and not this or that particular form) at the center of the work of the intermedialists. More precisely, intermedial studies are interested in the formation of new media. As early as 2003, Johanne Villeneuve, who is a member of the CRIalt, explains on this score: Intermediality is a concept that goes beyond the idea that there can be interaction between given media. The concept can refer to periods where the primary characteristic is emergence or appearance. Sometimes it refers to the plurality of recognized institutional media, sometimes to the very movement of recognition at the heart of which the heterogeneity of a new medium appears. (2003: 13)5

3

Pour certains, cela peut être ramené seulement à des enjeux de transformation, de déplacement, d’adaptation, de lien entre un médium et un autre médium par exemple, ou entre une forme reconnue, médiatisée comme telle, et une autre. Prenons l’exemple de l’opéra. Que lui arrive-t-il lorsqu’il devient un film? 4 Refuse[r] d’emprunter le chemin habituel de l’étude de l’étude des adaptations filmiques qui présuppose un récit transcendant sa médiation et qui repère des différences narratives servant in fine à distinguer les médiums du cinéma et de la littérature. 5 L’intermédialité est un concept qui déborde l’idée selon laquelle il puisse y avoir interaction entre des médias donnés. Le concept peut désigner des époques dont la caractéristique première est l’émergence ou l’apparition. Il vise tantôt la pluralité des médias institutionnels reconnus, tantôt le mouvement même de la reconnaissance au cœur de laquelle vient à paraître l’hétérogénéité d’un nouveau medium.

7

Montreal School of Intermediality: Beyond Media Studies

141

This approach is based on the clear distinction between what is a technological innovation and the moment when a medium is constituted as such (Gaudreault and Marion 2000). Studying so-called early cinema, André Gaudreault explains that “the institutionalization of a medium is a slow process.” (2008: 145). It thus spans some 20 years (period 1890–1915). The researcher specifies: what we are obviously referring to when we speak of cinema today – even if it may only be implicit – is this institution that cinema has become with its rules, constraints, exclusions and procedures. (2008: 120)6

To understand this type of phenomenon, Gaudreault has developed the notion of the cultural series. An emergent medium is thus a heterogeneous form that gradually acquires a sufficient degree of stability, specificity, and legitimacy to be designated as a medium. In this case, the intermediary approach is mobilized in order to maintain – for as long as possible – a certain degree of heterogeneity between the elements that make up an emerging medium. This amounts to saying that “intermediality appears before the medium” (CRIalt text presentation available on its website, quoted by Vallée 2017). In the case of cinema, the latter emerges notably from photography, chronophotography, the magic lantern, and light projection. The presupposition underlying this interpretation is that a new medium always emerges from an earlier medium, modifying certain of its aspects and uses (Bolter and Grusin 1999). Intermediality is then conceived as an approach to renew the history of the media themselves. Despoix writes, in this regard, that Gaudreault’s research on cinema conducted in Montreal was a source of inspiration for the school’s work (2012–13: 9). This reflects an interest in similar, but distinct subjects. This approach has more affinities with media archaeology, which compares different phenomena of media emergence in order to show how the emergence of present-day “new media” is clarified when we know the history of past “new media.” Media archaeology also looks at technological innovations which have become a cultural series, but which have never (or only for a short period) undergone the process of institutionalization necessary for it to be called media (Parikka 2013). The comparison with media archaeology reflects the fact that this type of approach has as its goal the understanding of media identity. So there is something different from the Montreal School. What is interesting in this case is that researchers such as André Gaudreault, who is one of the founding members of CRIalt, are therefore considered to be on the fringe of the Montreal School. The same is true of the work currently being carried out within the international research partnership TECHNÈS (2015–2022) led by Gaudreault, which aims to renew the history of cinema by placing the investigation of technical issues at the center.

Ce à quoi nous faisons de toute évidence référence, lorsque nous parlons aujourd’hui de cinéma – même si la chose peut n’être qu’implicite – c’est à cette institution qu’est devenu le cinéma avec ses règles, ses contraintes, ses exclusions et ses procédures. 6

142

R. Besson

Part 3: The Montreal School and “Living Together” While we have defined the approaches to intermediality that set the Montreal School apart, it now remains for us to develop the axis that distinguishes it (without aiming at a precise theoretical definition of the issues addressed by each of its members). This axis is characterized less by a theme or a type of object than by the fact that it shifts, quite radically, the purpose of the research. This is where the essential gesture proposed by the researchers who congregate under this name lies. Let us note, to begin with, that according to Méchoulan: Intermediality is interested in everything that allows beings, things, techniques, institutions to be attached to situations; it is, in my opinion, all these ways of creating links that constitute the object of intermedial investigations. (2021: 29). (L’intermédialité s’intéresse à tout ce qui permet d’attacher des êtres, des choses, des techniques, des institutions, dans des situations; ce sont à mon sens toutes ces façons de créer des liens qui font l’objet des investigations intermédiales.)

Such a definition aims to move away from any form of technocentrism, which is, in turn, a critique of the intermedial perspective centered on the notion of emergence. As Caroline Bem writes in her introduction to an issue aimed at mapping the practices of intermediality, “from the outset the review [Intermédialités and the CRIalt more generally] anchors the media fact on the side of the human element, in a relational field that goes far beyond that of a simple technicality of transmission” (2017). The aim is not, either, to understand the phenomena of co-presence or transfer within the same cultural production or from one cultural production to another; rather, the aim is to study relationships, regardless of the environment in which they take shape. This is what Méchoulan expressed in the quotation already discussed at the beginning of this chapter, “we do not analyze media for media’s sake (. . .) [the] issues are, I would say, a bit broader” (2021: 35). (Nous n’analysons pas les médias pour les médias (. . .) [les] enjeux sont, je dirais, un peu plus large.) This broadening is linked to the fact that the role of individuals and sociocultural groups is placed at the center of the investigation. The considered shift thus leads to the political question being placed at the heart of the subject. The notion of “politics” is not to be understood as a quasi-synonym for the defense of a political line, as is sometimes the case in social science schools and journals (the notion of school then comes dangerously close to that of political party). Rather, it is a question of focusing on the political, that is, on the organization of life in the city and conditions for living together. As early as the second issue of Intermédialités, Villeneuve asked himself, somewhat provocatively: Should we not then take the problem head-on, by linking the materiality of human mediations (the media, but also the technical devices) to the ideal of mediation, which consists in ‘living together’? (12). (Ne faut-il pas alors prendre le problème frontalement, en liant la matérialité des médiations humaines (les supports, mais aussi les dispositifs techniques) à l’idéal de la médiation qui consiste à «vivre ensemble»?)

7

Montreal School of Intermediality: Beyond Media Studies

143

This shift is very important, because objects considered until now as being at the center of the investigation (singular cultural productions and emerging media) become part of an environment that encompasses them. They are “supports, but also technical devices” mobilized by social actors for “living together.” We will return to the way in which this observation allows us to articulate the different intermediary perspectives in the following section (Part 4: Affinities with Other Intermedial Perspectives). For the moment, let us note that this amounts to positing that attention is focused mainly on the conditions of possibility of the unfolding of a shared experience. (Here we find a calling into question similar to that of Rancière (2000).) This necessitates the use of two concepts that are essential to the Montreal School: milieu and mediality. It is useful to define these two terms. In the simplest possible way, a milieu is that environment which encompasses both human actions and technical devices. It can be defined as the combination of a time, a space, an individual or sociocultural group and one or more media supports (Besson 2015). (To be absolutely precise, for each occurrence we would have to distinguish between the combination of media support (stone, paper, screen) and a specific, noticeable inscription (a letter, a book, a text message, a film, a comic strip, a website). We use the single formula media support by convention.) Such a definition corresponds to environments that are very limited in time (an exchange of glances) as well as to phenomena that are common over several centuries (e.g., the evolution of the way we look at each other). It also makes it possible to apprehend milieus at both the micro level (the meeting between two people) and the macro level (the evolution of a society or of life on planet earth). The medium can be as simple as a sheet of paper exchanged between two people and as complex as the set of screens needed for a virtual reality experience. It is then close to the notion of “technical media” as defined by Elleström, “technical media are the very tangible devices needed to materialize instances of media types” (2010: 12) or medium as “technical apparatus” (Méchoulan 2021: 38). However, the notion of medium is understood in a broader sense, since it includes even bodies and orality. There is no longer any question of conceiving it primarily as being detached from the human element (instrumental vision), but on the contrary, as inscribed in the flesh and in the language. In this respect, culture (the collective level) and memory (the individual level) can be considered as supports. In any case, what should not be lost sight of is the role of the individual or of the sociocultural group. Thus, the focus is on phenomena of transmission of knowledge and experience. Transmissions are then considered as the fabric through which human communities take shape (Besson and Polledri 2020). For the Montreal School, the presence of human beings is therefore consubstantial to the existence of a milieu (this centrality will be questioned in the concluding part of this text). To this very simple definition, we must, at the very least, add the notion of institution, that of “a normative power mutually subjecting individuals to certain practices on pain of punishment” (Alain Berrendonner cited by Gaudreault 2008: 118). Indeed, a milieu never takes shape outside a set of social, cultural, political, and economic norms that preexist it and survive it. This amounts to saying that the milieu is always “situated” (Haraway 1988).

144

R. Besson

Let us now take an example of a milieu as we understand it, which we will then try to follow throughout this section, namely a situation well known to researchers in the humanities and social sciences: a paper delivered at a conference. As Méchoulan explains in relation to this type of intervention, it is a matter of questioning “the modes of transmissibility of this knowledge, in other words, the ways in which knowledge appears audible and comprehensible” (2021b: 167). (Les modes de transmissibilité de ces connaissances, autrement dit, les façons pour des savoirs d’apparaitre comme audibles et compréhensibles.) It is understood that the paper in question lasts between 20 and 30 min. This duration is purely a convention, which can be questioned. More and more often, a paper, including in the Human and Social Sciences, lasts barely 5 min, sometimes even 180 seconds. The space is also standardized. The room where the presentation takes place is most often organized in such a way that the speaker faces the audience, is sometimes placed a little higher than the audience. He or she may be seated or standing behind a desk or lectern. Many exceptions can be envisaged (multivoice communication, questioning the frontality, etc.), but they are rarely implemented. (Such an observation was developed in the framework of the Labcom RiMeC in Toulouse, France (ANR, 2013–16), which aims to rethink the conference medium.) There are many media supports. They can be sheets of paper read out by the speaker, or a computer that allows him or her to share a slide show that is shown on a central screen via a projector. Moreover, the number of media supports increases considerably if, instead of focusing our attention solely on the speaker, we consider the members of the audience as equally important. This example seems relevant to us because it shows that the study of a milieu can sometimes seem to be limited to the enumeration of its elements. Such work corresponds to the identification of what makes human mediation possible; it is by no means the end of the research. Indeed, for mediation to occur, it is relevant to consider the elements that set the milieu in motion. These elements are designated by the notion of mediality. Will Straw specifies that it: refers to the modes of objectification, transmission and circulation of cultural expression in all its forms. It can therefore refer as much to objects and machines as to discursive formations or forms of sociability (2015). (Such an observation was developed in the framework of the Labcom RiMeC in Toulouse, France (ANR, 2013–16), which aims to rethink the conference medium.)

It is therefore less a question of objects – in the instrumental sense of the term, although this is sometimes the case – than of operations, approaches, processes, dynamics, dynamic factors. (These terms have been borrowed from the article by Bem and Froger (2021: 253–54 and 259).) If we take the example of the beginning of a conference paper, what animates the milieu is, for example, the fact that the speaker starts to speak or that someone asks him a question. On a more theoretical level, it is then a question of understanding what is at stake in the transition from silence to discourse, from the absence of noise to its presence, from a physical, technical, anthropological, and even philosophical point of view. It is also possible to study

7

Montreal School of Intermediality: Beyond Media Studies

145

nonverbal signs (opening the slide show, turning off the lights, closing the door of the room, etc.). These ritualized gestures mark the entry into scientific communication. Mediality is therefore constitutive of the observed phenomenon. Like milieu, it is integrated into the institution (as part of a set of norms). It is possible to go so far as to consider that mediality participates in distinguishing one milieu from another, insofar as what is set in motion constitutes a distinction. Let us now return to the question of terminology, as there is no consensus on this issue within the Montreal School. Indeed, in order to account for both the notion of milieu and mediality, Marion Froger prefers to use the expression “media configuration” which she defines as follows: [Media configuration] is made up of dynamic networks of gestures, images, acts of language or discourse, anchored in practices that mobilize human and non-human and natural and artificial entities, in communication processes based on symbolic crossings. Rather than insisting on the media object, we will therefore insist, in fact, on the mediation process enabled by a specific device. (2001: 320). ([La configuration médiatique] est constituée de réseaux dynamiques de gestes, d’images, d’actes de langage ou de discours, ancrés dans des pratiques qui mobilisent des entités humaines et non humaines, naturelles et artificielles, dans des processus de communication élaborés sur des croisements symboliques. Plutôt que d’insister sur l’objet média, on insistera donc, en fait, sur le processus de médiation habilité par un dispositif spécifique.)

It also happens that the expression of media environment, notably used by Vallée to create links between intermediality and media ecology (2017), or even of mediating circumstances (we will come back to this in Part 5) is preferred to the term “milieu.” In any case, the objective is to question the idea that we have accepted until now (by convention and to make our point more clearly), that milieu preexists mediality. It is more relevant to consider that milieu and mediality are co-constructed in a form of dialectical relationship that is never suspended. In this respect, let us recall that, as early as 2003, Méchoulan wrote that: The prefix inter aims to highlight an unnoticed or hidden relationship, or, more so, to support the idea that the relationship is in principle primary: where classical thought generally sees isolated objects that are then linked to one another, contemporary thought insists on the fact that objects are above all nodes of relationships, movements of relationship slowed down enough to appear immobile. (11). (Le préfixe inter vise à mettre en évidence un rapport inaperçu ou occulté, ou, plus encore, à soutenir l’idée que la relation est par principe première : là où la pensée classique voit généralement des objets isolés qu’elle met ensuite en relation, la pensée contemporaine insiste sur le fait que les objets sont avant tout des nœuds de relations, des mouvements de relation assez ralentis pour paraître immobiles.)

This amounts to saying that in order for a medium to exist, that is, in order for us to be able to apprehend it as such, it is necessary for a relationship to be established between its various constituent elements. Put differently, there is no environment without mediation. There is no “living together” without the creation of something that is situated between the constituent elements of the environment. From this observation, Méchoulan draws a founding principle for the Montreal School, “the

146

R. Besson

relationship is in principle primary.” It would be possible to add that, from then on, it is the study of these relationships that is placed at the center of this intermedial perspective. Intermediality is conceived as a way of studying what lies between the medium and the supports, between the milieu and the medialities, between the medialities. Every physical and symbolic movement, every word spoken, every noise heard, every sound emitted, every variation of light, etc. can be interpreted as the creation of a relation. Their material inscription on a physical support is obviously to be taken into account. The questions of links between materiality and supports are therefore essential. And, if we study longer-term phenomena, it is also possible to consider more profound cultural changes that are no longer of the order of an occurrence (a sound, a noise, etc.) but of the transformation of a group’s relationship to the body, to movement, to speech, to sound, etc. Once again, developments in the field of technology must also be considered. It is essential to take into account the evolution of the media for recording body movements, sound waves, light, etc. Thus, it is not necessary to link several media for intermediality to occur (which would be useful, for example, in a media archaeology approach). This may however be relevant, creating between-milieus (more on this later). In any case, the use of these competing terms – milieu, configuration, circumstances, environment – underlines the fact that researchers in Montreal intermedia studies are more interested in media dynamics than in the elements that constitute the milieu. It should also be noted that Méchoulan maintains that the term “milieu” is the best one to describe this dynamic by referring to the polysemy of the word. He explains: milieu as that which is between objects or subjects, and milieu as that which envelops objects or subjects. (. . .) [milieu] implies both ‘in the middle of’ and ‘in the milieu of’. (2021: 35). (Le milieu comme ce qui est entre des objets ou des sujets, et le milieu comme ce qui enveloppe des objets ou des sujets. (. . .) [milieu] suppose à la fois «au milieu de» et «dans le milieu de».)

He adds that “mediality is everything that constructs milieu and milieus,” in the sense that mediality participates in setting in motion space, duration, individuals or social groups and the media supports that constitute the milieu, as well as allowing us to characterize what is played out between these different elements (space, duration, individuals/groups, and supports). As the demonstration risks becoming too abstract, it is useful to go back to an example, namely that of a paper delivered during a conference. In order to increase the heuristic value of the subject, we propose considering the article published by Despoix in Communication et langages. What interests us within the framework of this chapter is above all to give an account of Despoix’s approach. His article aims to understand the singularity of the method of the German art historian Aby Warburg (1866–1929) by studying a lecture-projection the latter gave in Hamburg in 1905. In this scientific paper, Warburg presented (among other things) the concept of the “pathos formula” (Pathosformel). Despoix did not choose to study only the written sources related to the text of this lecture. On the contrary, he

7

Montreal School of Intermediality: Beyond Media Studies

147

worked from the texts and images and moreover chose to articulate this intervention with Warburg’s conception of an exhibition and a booklet. Warburg used three complementary modes of visualization: exhibition, projection, printed image, all based on reproduction techniques - photography extending the engravings at the heart of the artistic transmission process itself via projection and distributed plates. (2021, 186). (Exposition, projection, image imprimée, tous basés sur des techniques de reproduction – la photographie venant prolonger via la projection et les planches distribuées, les gravures au cœur du processus de transmission artistique lui-même.)

Despoix considers each of the three modes of vizualization as “intermediary operators” (171) whose aim is to set the medium under study in motion. The aim is to understand what is at stake in the gestures of exhibiting and projecting, as well as to study the techniques that make these gestures possible. Despoix then praises Warburg’s great intellectual finesse, which enabled him to mobilize distinct “operators” according to his objectives. He explains that each of the moments articulated in the display of the triple monstration device had its own function: material contextualization for the exhibition, demonstrative commentary as an overture to the ‘pathos formula’ denomination for the conference-projection; finally, visual condensation supporting notion in the portfolio, extended by the Ovidian illustration of the published summary. (187) (Contextualisation matérielle pour l’exposition, commentaire démonstratif ouvrant à la dénomination «formule de pathos» pour la conférence-projection ; enfin condensation visuelle soutenant la notion dans le portfolio, prolongé de l’illustration ovidienne du résumé publié.)

It is therefore a question of gesture (exhibiting, projecting), of techniques (engraving, photography), but also of functions (contextualization, commentary, condensation), all three of which are used to transmit knowledge. This allows Despoix to better understand how, for Warburg, the elaboration of a new concept – here, that of Pathosformel – was not communicated only through writing, but also through the creation of different devices involving the creation of links between texts and images. He concludes, “in this sense, this complex of interrelationships constitutes the very mediation of the thinking pertaining to the differentiated relations of image and word” (188). (Ce complexe d’interrelations constitue en ce sens la médiation même d’une pensée des rapports différenciés de l’image et du mot.) Despoix thus envisages the proto-intermedial character of Warburg’s approach. Indeed, it is by unearthing the proceedings of the conference-projection from the archives of the Kulturwissenschaftliche Bibliothek Warburg, then by linking it to other devices (exhibition, booklet) that the researcher worked on an intermedialist level. He took as his object of study the fact of creating links between media supports, and then interpreted them by mobilizing the notions of gestures, techniques, and functions. In this case, thinking about the format of scientific communication from the point of view of intermediality means questioning both the conditions of possibility and the way in which knowledge is transmitted.

148

R. Besson

It would now be possible to adopt a comparative approach that would amount to connecting milieu, presented in the first part of our demonstration – a present-day academic communication – to the one presented by Despoix. This would be a way of creating an in-between temporality of a similar type of milieu (though different, as the singularity of Warburg’s device must be taken into account). The result would be simultaneous effects of continuity and rupture linked to the state of society, to the evolution of its relationship to knowledge, time, space, and media supports. The technical dimensions should also be taken into account. Thus, the advent of the digital age, which produces an impression of ubiquity, should also be taken into account (Besson 2021b). As Mariniello notes, “the consequences [of this advent] are immense and affect all aspects of living together, as well as the perception we have of ourselves as individuals” (2011: 14). (Les conséquences [de cet avènement] sont immenses et touchent à tous les aspects du vivre-ensemble ainsi qu’à la perception que nous avons de nous-mêmes en tant qu’individus.) It would also be possible to multiply the examples. Let us note, in this respect, that in the same issue of Communication et langages, André Habib (2021) adopts an approach which is comparable to that of Despoix, in order to reconstruct the first projection of Zen for Film (1962–1964. . .) by the contemporary artist Nam Jun Paik in New York, in the context of the Fluxus movement. This type of approach can therefore be successfully applied to the field of contemporary art, whether it be experimental film, performance, or installation art. This point is important, because it should not be assumed that the Montreal School is mainly concerned with phenomena in the field of knowledge transmission. On the contrary, the members of the school are often interested in phenomena pertaining rather to the realm of experience and feeling. It is now relevant to focus on the demonstration mode, by example. Indeed, the examples used in this chapter have an undeniable familiarity with the fact that the members of the Montreal School of intermediality work, most often, by formalizing case studies. Indeed, apart from the recurrent debate since the 1990s about the very definition of the term intermediality, few purely theoretical texts have been published. It seems that theorization is mainly done through the presentation of a case. This is not anecdotal, as this approach has been the subject of research by Bem and Froger for several years. They explain: A case study can be seen as a snapshot that stops time. The case would be of use to see, i.e. to slow down, a dynamic phenomenon. The case is thus a mediator for the analyst, a phenomenal objectness where what processes sweep along - in the form of remains – is deposited. (2001: 255). (Une étude de cas peut s’envisager comme un instantané qui arrête le temps. Le cas servirait à voir, c’est-à-dire à ralentir un phénomène dynamique. Le cas est donc un médiateur pour l’analyste, une objectité phénoménale où se dépose – sous forme de restes – ce que les processus emportent.)

This is to say that the study of a case is certainly the most suitable approach for researchers of the Montreal School of intermediality. First of all, it should be noted that the construction of a case enables them not to be overwhelmed by the infinity of possible relationships between the different elements being studied. Moreover, this affinity with case studies is also explained by the possibility of considering the notions of case and milieu as quasi synonyms. This means that the choice of a case corresponds to the

7

Montreal School of Intermediality: Beyond Media Studies

149

identification of an object that allows for the study of relationships between a space, time, sociocultural individuals/groups, and media supports. A problem statement is then defined, insisting on one or several types of media relations, transforming the object of study into a case. Finally, there is a tendency for researchers in the Montreal School to choose the most limited (and “simple”) cases possible in order to study the links between a limited number of elements in as much depth as possible (it seems more relevant to survey a large number of relationships). This can be a conference-projection, an exhibition, a booklet (or all three, as in the case of Despoix). This means that the object being studied does not need to be composed of various media (multimedia) or to link various media (co-presence/transfer) to be considered from the point of view of intermediality. Bem and Froger insist on this point when they explain: [it] is, on the contrary, a question of seizing each object as a witness to the mediations from which it proceeds or in which it participates. In short, to consider objects within the horizon of the constitutive mediality of cultural phenomena. (2001: 259). ([il] s’agit au contraire de se saisir de chaque objet comme d’un témoin des médiations dont il procède ou auxquelles il participe. Bref, d’envisager les objets à l’horizon de la médialité constitutive des phénomènes culturels.)

The partiality of the Montreal School’s researchers for the triad of milieu, mediality, and case study awards us a better understanding of why the works that fall under this school of thought regularly focus on cultural productions that represent situations related to “living together,” most often literature, photography, film, and theatre. Indeed, they regularly select cases that did not take place in the city, but which are the result of photographic, theatrical, or cinematographic productions or literary narratives. Let us return one last time to the study of scientific communications. If we want to highlight the political aspect of such a subject, it is sometimes possible to study a case from a film. Let us try this exercise with a short sequence from Alice and the Mayor (Nicolas Pariser 2019). In this French drama about the exercise of power, Anaïs Demoustier and Fabrice Luchini play, respectively, a young philosopher and the ageing mayor of the city of Lyon in France. At the beginning of the film, the elected official is in crisis, which he believes is due to the fact that he has run out of new ideas. Instead of calling on a media training coach, he recruits a young philosopher with whom he chooses to have regular discussions. The sequence that interests us has, as its object, the expression of a dispute between the two of them. The filmed space is the back of a car driven by a chauffeur, the exchange lasts barely a few seconds and no other media support other than the voices and bodies of the two protagonists are mobilized. Under pressure, the mayor gets angry and regrets that the results of the work they are doing together are not bearing fruit quickly enough. He then condescendingly (and this is conveyed by his body language) expresses the idea that academics are only good at organizing conferences. According to him, in the darkest hours of French history, in June 1940, intellectuals were idle. Stung to the core, the philosopher replies by asking him if he knows Strange Defeat by the historian Marc Bloch. In the face of the mayor’s cautious silence, she explains that this book was written during the Second World War and that the researcher paints a striking picture of the situation in France at that time.

150

R. Besson

This reference is altogether relevant, the philosopher is thus putting the politician in his place by proving, by example, that history makes it possible to interpret the present and that a researcher is able to act in the city when current events refer to the tumults experienced by past generations. It could be studied from the point of view of intermediality studies, since there is an “intermedial reference” [intermediale Bezüge], the title of a historian’s work being quoted in a fiction film (without the book medium being presented in the scene). But for the Montreal School, this response is, in fact, disappointing. Indeed, Alice, the young philosopher, could have responded to the mayor by taking the symposium question, namely, the transmission of knowledge, seriously. This would have meant considering that media issues, particularly those relating to the format of scientific communications, are just as important as those relating strictly to political ideas. This observation leads us to stress the importance of a reflection on the conditions of possibility of speaking in the city. It could have addressed the question of the relationship between image and word, projection and exhibition, publishing and declaration, as developed by Despoix regarding Warburg. The study of this milieu is, in itself, political, in the sense that it has to do with “living together.” In this case, the cultural production is not seen as a representation of public space (it does not show a symposium), but rather leads to a reflection on how fiction grabs hold of an intermedial issue (or fails to do so, as in Alice and the Mayor).

Part 4: Affinities with Other Intermedial Perspectives The question of the identification of media milieus in singular cultural productions deserves to be developed beyond the single example mentioned above. This is a relevant avenue for bringing the Montreal School closer to other intermedia perspectives. The question is how researchers who take part in this school of thought use these concepts in research that ostensibly belongs to other perspectives (studies of processes of co-presence or the transfer of forms from one cultural production to another, history of the emergence of new media, etc.). The aim is to show that the specificity of the school lies mainly in the (political) issues it raises and not in the objects (cultural productions) being mobilized. Indeed, if a milieu can be located in a film (or a particular sequence as we have just shown), a photograph, a play, a novel, then the notions of co-presence and transference are certainly relevant to this type of study. Let us take the example of the article published by Silvestra Mariniello in the issue of Communication et langages. The article in question focuses on two films by Marie-Annick Bellon (1924–2019), a documentary entitled The Evasion (1989) and a fiction film, Children of Chaos (1989). Both productions deal with the way a theatre company (Théâtre du Fil) works with marginalized youth in the Parisian suburbs. More to the point, the documentary – shot first – is conceived as a logbook leading up to the production of the fiction. It seems quite clear that the concepts of co-presence and transference can be mobilized in a way that is not unique to the Montreal School. It is about the link between theatre and film, and between documentary and fiction. This could lead to a reflection in terms of mediativity

7

Montreal School of Intermediality: Beyond Media Studies

151

(Gaudreault and Marion 1998), that is to say, to think about the most suitable format for transmitting this or that aspect of the project in question. It would then be a matter of asking what belongs more to the realm of fiction or documentary, or even what specifically theatrical aspects are difficult for cinema to grasp. It would also be interesting to reflect on what is transformed, what is lost, what is developed, that is to say, what is transferred during the self-adaptation of documentary into fiction. Such questions underlie Mariniello’s work, but the purpose of her article is not by any means aimed at that level. Rather, her focus is on the staging of “living together.” What interests her is “the impact of theatre on the lives of ‘certain human beings’,” in this case marginalized youth. The aim is therefore less on the media side, considered for its own sake, than on the side of the effects of media processes on the life of the considered group. It is possible to go even further by hypothesizing that it is not so much the staging of “living together” in these films that interests Mariniello, but rather these productions (the two films) as traces of a process that took place between the director, the members of the company, and the young people during the creation of both the play (on the side of the young people and the troupe) and the film (on the part of the director). It is this milieu of durations, spaces, the ensemble of social actors and media supports that is the focus of the intermedial investigation. What is particularly interesting in this case is that the “living together,” which is the object of study, is achieved through the creation of both theatrical and cinematographic cultural productions. It is a way of putting the question of creation back into the city by asking how this type of process affects those involved (in this case young people in difficulty). (We want to make it clear here that we are talking about young people in difficulty, but in another text Mariniello was also interested in prisoners detained in Rome, caught up in a process of theatrical creation that was the subject of a documentary film by Paolo and Vittorio Taviani, Caesar Must Die (2012). The challenge was to understand how the encounter with the theatrical text and the involvement in its cinematic remediation had transformed them (2018).) And it is from this shift, from a study of forms of creation as representation, to a consideration of their production in the public space, that intermediality is conceived. It is this shift that allows us to fully understand what Mariniello writes about two sequences of the films studied: We are confronted with relations, relations of relations, nothing but relations. The relation between cinema and theatre involves the relation between theatre and life, as well as that between cinema and experience (. . .); the present relates to the past; the abuses of the past staged by the play must be related to the violence of the modern State on these troubled youths . . . These are just some of the folds that the sequence unfolds, writing the complex network of relations between the two media. (2021: 59). (Nous sommes confrontés à des relations, des relations de relations, rien que des relations. La relation entre le cinéma et le théâtre implique la relation entre le théâtre et la vie, ainsi que celle entre le cinéma et l’expérience (. . .); le présent se rapporte au passé; les abus du passé mis en scène par la pièce théâtrale sont à mettre en relation avec la violence de l’État moderne sur ces jeunes en difficulté. . . Ce ne sont là que quelques-uns des plis que la séquence déploie, écrivant le réseau complexe des relations entre les deux médias.)

152

R. Besson

The subject is thus both explicitly political (it is a question of the violence of the modern state), but also political, in the sense that the media under consideration participate in the life of the city. Moreover, the use of the verb “involve” that we underlined in the quotation reflects this movement from media (theatre and cinema) to “living together” (life and experience in the quotation). This means that the study of media is put at the service of a political reflection. This reflection goes beyond, way beyond, a mere reflection on the media. Let us now consider another example, which allows us to understand a different type of link between the Montreal School’s approach and other intermedia perspectives (Besson, 2021a). The article I published in the issue of Communication et langages deals with the presence of filming devices (mainly cameras and amateur cameras) in the entire filmography of the American director Frederick Wiseman (some 50 documentaries shot between 1967 and 2020). It is not a question of identifying effects of co- presence or transfers between several singular cultural productions (a play and a film) or between different media (theatre and cinema), but of noting that technical equipment, linked to cinematographic and photographic creation, is present in a significant number of Wiseman’s documentaries. It is thus a question of identifying another type of co-presence, which is situated at the articulation between an aesthetic approach and an approach centered on technique. Such a reflection leads to a re-evaluation of the issues at stake in Wiseman’s cinema, whose political dimension is often limited to the fact that he criticizes the functioning of American institutions (school, prison, police station, hospital, asylum, etc.). To put it briefly, the recurrent presence of these devices contradicts the widely held interpretation that Wiseman is merely recording long interpersonal exchanges taking place in the institutions he films. The identification of these media supports, that went unnoticed for a long time, account for a more reflexive dimension of his cinema. However, this research is not limited to the reversal in question. Rather, the demonstration aims to develop a political reflection. It is a question of understanding what the presence of these devices tells us about the imaginative world of technology. Here, the notion of an imaginative world is charged with a Utopian aspect, which means that by showing the devices in question, Wiseman makes their role (surveillance, self-staging, etc.) apparent and thus offers the spectators the seeds of critical thought. As with Mariniello, it is not a question of considering films for their capacity to represent the city, but rather for their capacity to transform it. The critical dimension of Wiseman’s cinema is thus displaced from its explicitly political subject (criticism of institutions) to a reflection that is no less political, but which is more of a media issue. A final example of these links between the work of the Montreal School and other approaches is examined. This example shows the possibility of bringing together research on “living together” and questions relating to the emergence of new media. Indeed, in an article devoted to the appearance of the television set in the homes of North American families, Villeneuve takes care to return to the articulation between this technical equipment and the radio set that was already there (2015). She analyses the change from transmitting sound waves to associating sound with images. She considers the strictly technical dimensions of such a change (size of the device, quality of transmission, number of channels broadcast, etc.), but also what it implies in terms of the layout of the living room and the cost to users. She thus questions a cultural

7

Montreal School of Intermediality: Beyond Media Studies

153

series in the process of institutionalization – in the sense that Gaudreault has given to this term – which takes place somewhere in the 1940s. However, as with Mariniello and Besson, she is not interested in these issues, related to the emergence of a new medium, for their own sake. Rather, it is the changes it brings about in the lives of American families that she seeks to understand. It is always the same shift, not from aesthetics, but from an historical approach to media, towards something more political. In the precise case of the parameter of the TV-box, the apparatus’s three-dimensionality, but especially the constitutive place it occupies in the house, at the very site of the fireplace, redoubles yet again the function of the ‘milieu’ (. . .) it does indeed occupy the site of the fireplace, both physically and symbolically, a source of heat and comfort. (. . .) television imposes itself by taking the place usually reserved for the fire place in the house. (88).

Thus, by mobilizing the notion of milieu, Villeneuve aims to understand not only the function of the television set for its own sake, but what the presence of this device changes in the organization of the living room of North American families. In the quote above, she then develops the hypothesis of a substitution of the fireplace by the television, that is, the replacement of a physical heat source by a symbolic heat source. The point here is not to discuss this hypothesis, but to show how, in each of the three above-mentioned cases, the approaches developed by other intermedial perspectives are useful to the Montreal School. Beyond the differences, this brings out affinities that would not otherwise have been thought of. By highlighting these connections, the main difference between these approaches becomes even clearer: it lies in the research issues.

Part 5: Internal Variation of the Montreal School It must be emphasized that some researchers, such as Larrue and Vitali-Rosati, members of the Montreal School, notably in the sense that they published an article in the issue of Communication et langages, express different sensibilities. They have been working with CRIalt for years. This means that they do not pose the question of politics through the prism of “living together” as it has been defined so far. However, their reflections remain compatible with the school’s approach. They enable us to give an idea of the distinct debates and perspectives that coexist within the school. This allows us to emphasize that if, to us, “living together” seemed to be a relevant interpretation key with regard to the collection of texts in the issue of Communication et langages, other notions will perhaps take on more importance in the years to come. Similarly, positions that are currently marginal may become more central. For the purposes of this text, we will present three of their main arguments, both to criticize and render more complex the model that has been developed so far. Firstly, we note that the theatre studies researcher Jean-Marc Larrue is interested in the new materialism and in object-oriented ontology (Guay et al. 2022). An important player in the intermedial turn taken by French-speaking theatre studies since the early 2000s, Larrue helped to deconstruct the idea that the mediativity of theatre is linked to

154

R. Besson

the presence of actors on stage (2011). He has developed the idea of another co-presence (other than that of the audience and the actors), that of the human element and of technical devices (microphone, amplifier, gramophone, etc.). He has now made the expression of Nick J. Fox and Pam Alldred, who want to “de-privilege human agency” (2015: 399), his own. This amounts to considering the importance of nonhuman agents. In the context of this text, this work can be interpreted as adding complexity to the model of the environment as considered by the Montreal School. This approach leads to a rethinking of the distinction between individuals and social groups on the one hand and media supports on the other. On the contrary, it proposes to create links between the two and to reflect on the ways in which media supports act and, in fact, that their performativity is not always linked to human actions. It is therefore a question of rethinking the parameters of milieu. This may result in a return to a greater centrality of technique in intermedial reflections, on the model of an autonomy of media supports with respect to the human element, but also to reflect on environmental issues (which has rarely been the case in the studies of the Montreal School) by taking into account the capacity of nature to act. Furthermore, Larrue has conducted a critique of the very notion of intermediality, which he considers marked by history (2015: 31). In particular, he considers the term “medium” to be a dubious legacy of a structuralist worldview. This observation has led him to identify a post-media turn in intermedial studies, which he situates around the year 2005. Drawing on the most recent work of Gaudreault and Marion on the reconfigurations of the cinema medium in the digital age (2013), as well as that of Jenkins on the transmedia character of contemporary cultural productions (2006), he considers that, “henceforth, the question of the identity of the medium becomes secondary to its action and its situation in transmedia configurations” (2015: 47). This amounts to saying that cultural series never really become institutionalized and that they always continue to share common characteristics with other media configurations. Such a proposition is entirely compatible with an interpretation in terms of milieu. Moreover, he deduces that it is necessary to move from media thinking to research centered on the notion of mediation. Here we find the idea presented by Mariniello according to which, “we are confronted with relations, relations of relations, nothing but relations” (2021: 59). Building on the work of Alexander R. Galloway (2013), Larrue posits in Communications et langages that: The media are nothing but discursive constructions, harmful because they stand in the way of or distort reality. It is therefore important to free ourselves from them and focus on what really exists, mediations. (2021: 121). (Les médias ne sont que des constructions discursives, nuisibles parce qu’elles font écran à la réalité ou la déforment. Il importe donc de se libérer d’eux et de se concentrer sur ce qui existe vraiment, les médiations.)

Together with the digital humanities researcher (Larrue and Vitali-Rosati 2019), this led him to propose a different formulation from that of milieu, developed by Méchoulan, or of media configuration, proposed by Froger, or that of mediating circumstances. In Communication et langages, Vitali-Rosati summarizes the issue as follows:

7

Montreal School of Intermediality: Beyond Media Studies

155

How can we grasp the specificity of a media situation and its particularities if we completely and radically renounce any form of essentialism? (. . .) The idea at the root of the mediating configurations is to try to allow for an understanding in the essentialist aftermath of language without losing a radically anti-essentialist thought. (2021: 157). (Comment peut-on saisir la spécificité d’une situation médiatique et ses particularités si l’on renonce complètement et radicalement à toute forme d’essentialisme ? (. . .) L’idée à la base des configurations médiatrices est de tenter de permettre la compréhension dans l’après-coup essentialistes du langage sans perdre une pensée radicalement anti-essentialiste.)

Such a perspective can be seen as calling into question of the centrality of the notion of milieu to insist almost exclusively on mediality. It is also a way of insisting on the fact that these are all discursive configurations created in the aftermath of case studies by the researchers. According to them, there is therefore no milieu as such, but rather mediating circumstances that are necessarily posterior to the observed phenomenon. The two researchers thus aim to avoid any form of structuralism (Larrue’s rejection of the term media) and any form of essentialism (Vitali-Rosati’s rejection of the term “milieu”) by adopting a constructivist posture pushed to its extreme. This leads Larrue to develop what we propose to present as a third means of making the Montreal School model more complex. Indeed, the researcher is currently working less on the conditions of “living together” than on what escapes the paradigm of communication and representation. While the cultural productions (films, plays, photographs, etc.) studied by the Montreal School often correspond to the staging of public space, he is interested, following the example of Richard Grusin (2015), in situations of radical mediation, that is to say, singular configurations where mediation refers to nothing other than itself. He explains that this approach “applies, for example, to the modern magic show (where one sees that one does not see), but it also applies, in the world of the performing arts, to performance – as a stage practice – and to most circus acts” (2021: 131). (S’applique, par exemple, au spectacle de magie moderne (où on voit qu’on ne voit pas), mais elle s’applique aussi, dans le monde des arts vivants, à la performance – en tant que pratique scénique – et à la plupart des numéros circassiens.) He adds, again relying on Gallaway and Thacker’s concept of “dark media,” that we must go beyond this departure from the representational model and look at mediation phenomena that are not necessarily communication phenomena (in the sense of the transmission of a message or the manifestation of a content). It is then a question of looking at what malfunctions, what remains below and beyond communication for human beings, that is to say, to take an interest in “inaccessibility in itself” (Thacker quoted by Larrue 2021: 132). If the importance of “living together” for researchers of the Montreal School has been emphasized, it remains to be identified here that it is important to look at what escapes it and remains unattainable to the perception of the agents in the studied milieu. Larrue concludes that this “amounts, to mention the magic show again, to perceiving that there truly is something imperceptible, which remains imperceptible even though we perceive it” (2021: 132). (Revient, pour reprendre le spectacle de magie, à percevoir qu’il y a bien quelque chose d’imperceptible, qui le demeure bien qu’on le perçoive.)

156

R. Besson

Conclusion This chapter shows that, although the Montreal School has affinities with other approaches to intermediality that are more focused on the notions of co-presence, transfer between cultural productions, and the emergence of “new media,” it is no less singular in its issues. Its members claim the importance of the notion of milieu, which implies relations between time, space, human elements, and media supports, all of which are embedded in institutions. The objects of other approaches (singular cultural productions and emerging media) are thus fully considered by the members of the Montreal School, but they are articulated with more political issues that have to do with “living together.” It is this epistemological shift that is the decisive element in identifying membership in this school. This means that being a member of the CRIalt or practicing intermediality in Montreal is not enough. It also means that the same researcher can publish texts that fall within the framework of this school of thought, as well as texts that do not. The interpretation of form, of functioning, of the advent of relations between media or even the study of the place of media in the city do not correspond to the research aims of this school. Rather, it is a question of reflecting on the conditions of possibility to transmit knowledge and experience, and on the way in which the studied environments are set in motion. What is being considered is the relationship between media and milieu, between milieu and medialities, between milieus, and between medialities. It is thus not so much a question of studying objects such as media or a cultural production, than operations, approaches, processes, dynamics. It is the relations created by these phenomena of mediality that are at the center of intermedial investigation. We thus developed the idea, presented by Méchoulan, according to which “we do not analyze media for media’s sake, we analyze the media for mediality effects” (2021: 35). (Nous n’analysons pas les médias pour les médias, nous analysons les médias pour leurs effets de médialité.) This approach leads us to think in terms of milieu, environment, configuration and the media situation, inhabited by human and nonhuman agents, and in which they act. This leads us to conclude, with Mariniello, that intermediality for the Montreal School: “marks the shift from a theory of society that contains the media - a generally established conception nowadays - to one in which society, socialities and media continuously co-construct and destroy each other.” (2011: 13) (Marque le passage d’une théorie de la société qui contient les médias – conception généralement établie de nos jours – à une théorie où société, socialités et médias se coconstruisent et se détruisent en permanence.)

References Bem, Caroline. 2017. Introduction. L’intermédialité est la carte autant que le territoire. Intermédialités: 30–31. Bem, Caroline, and Marion Froger. 2021. Construction collective d’une théorie en chantier. Les études de cas dans la revue Intermédialités. Communication et langages 2: 247–267.

7

Montreal School of Intermediality: Beyond Media Studies

157

Bem, Caroline, et al. 2017. La nouvelle sphère intermédiatique (colloque du CRI–1999) à l’épreuve de la remédiation: supports, approches et discours. Intermédialités: 30–31. Besson, Rémy. 2015. Intermediality: Axis of relevance. SubStance 44 (3): 139–154. Besson, R. and Polledri, C. 2020. Introduction. Intermédialités / Intermediality, (36), 1–24. https:// doi.org/10.7202/1080949ar ———. 2021a. Regard intermédial sur l’imaginaire de la technique dans le cinéma de Frederick Wiseman. Communication et langages 2: 135–153. ———. 2021b. Alice et le congrès: une expérimentation d’imagination intermédiale. In Quel congrès voulons-nous? ed. Monique Martinez and Nina Jambrina, 171–185. Paris: L’Harmattan. Bolter, Jay David, and Richard Grusin. 1999. Remediation: Understanding new media. Vol. 4, 208. MIT Press. Despoix, Philippe. 2012. Traverser dix ans d’Intermédialités: introduction. Intermédialités 20: 7–10. ———. mars–mai 2014. L’école montréalaise de l’intermédialité à travers sa revue. Conférence de la Bibliothèque Gaston Miron. ———. 2021. Re-produire, sérier, nommer les empreintes de pathos: geste intermédial et ‘conceptdispositif’ chez Aby Warburg. Communication et langages 2: 171–193. Despoix, Philippe, and Sylvano Santini. 2010. Intermédium et porosité disciplinaire: parcours intermédial de Philippe Despoix. Spirale 233: 41–44. Elleström, L. 2010. The Modalities of Media: A Model for Understanding Intermedial Relations. In Media Borders, Multimodality and Intermediality ed. Lars Elleström. Palgrave Macmillan, London. https://doi.org/10.1057/9780230275201_2 Fabiani, Jean-Louis. 2005. Faire école en Sciences Sociales. Un point de vue sociologique. Les Cahiers du Centre de Recherches Historiques. Archives 36. https://doi.org/10.4000/ccrh.3060 Fox, Nick J., and Pam Alldred. 2015. New materialist social inquiry: Designs, methods and the research-assemblage. International Journal of Social Research Methodology 18 (4): 399–414. Froger, Marion. 2001. Le centre de recherche sur l’intermédialité (CRI) de l’Université de Montréal. Visio. Revue internationale de sémiotique visuelle 6 (2–3): 317–326. Galloway, Alexander R., Eugene Thacker, and McKenzie Wark. 2013. Excommunication: Three inquiries in media and mediation. University of Chicago Press. Gaudreault, André. 2008. Cinéma et attraction: pour une nouvelle histoire du cinématographe. Paris: CNRS. Gaudreault, André, and Philippe Marion. 1998. Transécriture et médiatique narrative : l’enjeu de l’intermédialité . . . In La transécriture. In Pour une théorie de l’adaptation, ed. André Gaudreault and Thierry Groensteen, 31–52. London: Nota bene. ———. 2000. Un média naît toujours deux fois. Sociétés & Représentations 9 (2): 21–36. ———. 2013. La fin du cinéma?: un média en crise à l’ère du numérique. Paris: Armand Colin. Grusin, Richard. 2015. Radical mediation. Critical Inquiry 42.1: 124–148. Guay, Hervé, Jean-Marc Larrue, and Nicole Nolette. (2022, Forthcoming). Introduction. Théâtre et les Nouveaux matérialismes. Presses de l’Université de Montréal. Montréal Guerin, Olivia. 2002. Les processus de nomination des référents non répertoriés dans les récits de voyage: faits de langue ou de discours? L’Information grammaticale 92 (1): 8–12. Habib, André. 2021. . . . et des poussières. Autour de Zen for Film (1962–1964. . .) de Nam June Paik. Communication et langages 2: 95–113. Haraway, Donna. 1988. Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist Studies 14 (3): 575–599. Jenkins, Henry. 2006. Convergence culture. New York: New York University Press. Larrue, Jean-Marc. 2011. Théâtralite, médialité et sociomédialité: Fondements et enjeux de l’intermédialité théâtrale. Theatre Research in Canada 32 (2): 174–206. ———. 2015. Du média à la médiation: les trente ans de la pensée intermédiale et la résistance théâtrale. Théâtre et intermédialité: 27–56.

158

R. Besson

———. 2021. Où en sommes-nous et où sommes-nous? L’intermédialité, le théâtre et le reste. Communication et langages 2: 115–134. Larrue, Jean-Marc, and Marcello Vitali-Rosati. 2019. Media do not exist: Performativity and mediating conjunctures. Amsterdam: Institute of Network Cultures. Lloyd, Christopher. 2009. Historiographic schools. In A companion to the philosophy of history and historiography, 371–380. Chichester: Wiley-Blackwell. Mariniello, Silvestra. 2011. L’intermédialité: un concept polymorphe. Inter Media. Littérature, cinéma et intermédialité: 11–30. ———. 2018. Du Sens de l’intimité dans la polis médiatique globale: César doit mourir, Paolo et Vittorio Taviani (2012). In Le Partage de l’intime: Histoire, Esthétique, Politique: Cinéma, ed. Frédérique Berthet and Marion Froger, 93–112. Montréal: Presses de l’Université de Montréal. ———. 2021. À travers l’intermédialité, le travail de l’analogie. Réflexions à partir de deux films de Yannick Bellon. Communication et langages 2: 51–67. Méchoulan, Éric. 2003. Intermédialités: le temps des illusions perdues. Intermédialités 1: 9–27. ———. 2017. Intermédialité, ou comment penser les transmissions. Fabula. La recherche en Littérature. ———. 2021. Des congrès comme atmosphères. In Quel congrès voulons-nous? ed. Monique Martinez and Nina Jambrina, 163–170. Paris: L’Harmattan. Méchoulan, Éric, and Elsa Tadier. 2021. Tentative d’épuisement de l’intermédialité: entretien avec Éric Méchoulan. Communication et langages 2: 27–49. Méchoulan, Éric. 2021b. Des congrès comme atmosphères. In Quel congrès voulons-nous? ed. Monique Martinez Thomas and Nina Jambrina, 163–169. Paris: L’Harmattan Nicolas, Pariser. 2019. Alice and the Mayor [Alice et le maire] by Nicolas Pariser. https://www. imdb.com/title/tt8903840/ Parikka, Jussi. 2013. What is media archaeology? Oxford: Wiley. Peltre, Christine. 2007. Introduction des usages d’un outil. In La notion d’« école », ed. Christine Peltre and Philippe Lorentz, 31–52. Strasbourg: Presses Universitaires de Strasbourg. Rajewsky, Irina. 2015. Le terme d’intermédialité en ébullition: 25 ans de débat. Intermédialités, 19–54. Paris: SFLGC. Rancière, Jacques. 2000. La partage du sensible: esthétique et politique. Paris: La Fabrique Éditions. Straw, Will. 2015. Introduction. Intermédialités 26: Habiter (la nuit)/inhabiting (the night). Tadier, Elsa. 2021. L’intermédialité montréalaise, ou l’art de penser la relation. Communication et Langages 2: 9–25. Translated from the French by Naòmi Morgan Vallée, Jean-François. 2017. Intermédialité et écologie des médias: essai de cartographie comparative. Intermédialités: 30–31. Villeneuve, Johanne. 2003. La symphonie-histoire d’Alfred Schnittke: intermédialité, cinéma, musique. Intermédialités 2: 11–29. ———. 2015. The TV-box: Reconsidering a lost television set, Santa Claus and the ants. SubStance 44 (3): 73–97. Vitali-Rosati, Marcello. 2021. Le fait numérique comme conjonctures médiatrices. Communication et Langages 2: 155–170.

8

Case Studies as a Heuristic of Intermediality Marion Froger and Caroline Bem

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Towards a Heuristic of Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Epistemic Options Within Inter- and Transdisciplinarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Role of “Research Media” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality Within Disciplinary Academic Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality Within Interdisciplinary Academic Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Epistemic Consequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Case Studies in the Journal Intermédialités/Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Role of Case Studies in the Journal Intermédialités/Intermediality . . . . . . . . . . . . . . . . . . . Serializing the Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

160 162 162 165 165 167 168 170 171 171 174 181 181

Abstract

While it recognizes that there does not currently exist a generally accepted, unifying theory of intermediality, this chapter aims to shed light on the ways in which at least one area of intermedial research has come to constitute itself through a process of simultaneously collective and cumulative inquiry. Taking as a starting point the Montreal-based, international, and peer-reviewed journal Intermédialités/Intermediality – the sole journal dedicated entirely to intermedial inquiry – this contribution explores how case studies published between the journal’s inception in 2003 and the time of writing (2022) have given rise to a heuristic of intermediality. In particular, the chapter builds on the M. Froger (*) Université de Montréal, QC, Canada e-mail: [email protected] C. Bem Saint Paul University, Ottawa, ON, Canada e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_20

159

160

M. Froger and C. Bem

recommendations formulated by Jean-Claude Passeron and Jacques Revel in their 2005 book Penser par cas to “highlight [...] the reciprocal implication that takes place between the articulation of theory and the unfolding of inquiry.” By seeking to identify the “descriptive concepts” that have allowed “previously unobservable phenomena” to be deciphered, it aims to reflect on the multiple ways in which the collective process of knowledge gathering promoted by the journal’s investment in case studies gives rise to a “theoretical reconfiguration” whose specific empirical, provisional, and circumstantial character remains rooted in particularity. By delving into the archive of articles of a singular journal – one that was born from a unique geographical, linguistic, and disciplinary context – the chapter also performs itself as a “case study” acting, in turn, as an open invitation for the (hopefully) ongoing emergence and theorization of more intermedial case studies well into the future. Keywords

Intermedial case studies · Intermédialités/Intermediality · Theories of the case · Humanistic inquiry · Intermedial methods · History of intermediality · Montreal School of Intermediality

Introduction This chapter takes as its central object of inquiry the archive of articles published in the Montreal-based, international, and peer-reviewed journal Intermédialités/ Intermediality (henceforth Intermédialités). Its first aim, then, is to catalogue the epistemic options that present themselves to intermedial scholars when they engage in case-study-driven research. The chapter conceptualizes these options from an intermedial perspective, which means that it considers them through the lens of peerreviewed journals considered as medium. The premise of this chapter is that the role attributed to case studies changes over the course of a given journal’s existence, shifting in importance from mere sample to paradigmatic case. This approach builds on Bent Flyvbjerg who summarizes various forms of sampling by taking as a starting point a variety of modes of case selection, as well as the myriad ways in which these might garner disruptive power. These include: “extreme/deviant cases,” “maximum variation cases,” “critical cases,” and finally, “paradigmatic cases” (2006: 230). Thus, established journals tend to be particularly welcoming of case studies which challenge methods or models of inquiry at times when they turn to look back upon their own history (at the moment of anniversary issues for instance) or as they negotiate major editorial shifts (the arrival of a new editor-in-chief or editorial board, or the switch from paper-based to online-only publishing, to name a few possibilities). By contrast, emerging journals often grant more visibility to case studies as a way to stake a claim to a particular field or subfield or to the reconfiguration of a particular field or subfield. Founded in Montreal (Quebec, Canada) in 2003, Intermédialités remains the only journal dedicated entirely to the

8

Case Studies as a Heuristic of Intermediality

161

publication of intermedial research while showcasing scholarship originating within a wide variety of disciplinary frameworks and which takes as its focus a virtually unlimited array of objects of study. As such, the archive of articles (and of intermedial case studies) naturally consolidated by the journal over its almost 20 years of existence functions as the chapter’s simultaneously irreducible (by virtue of its richness) and highly specific (due to its clear delimitation) case study. Through the theoretical back-and-forth it establishes between the singular (the journal, but also the high proportion of case study-based contributions within its pages) and the general (an accumulation of intermedial case studies giving rise to a general, if still fragmentary, understanding of at least one version of intermedial inquiry whose focus rests on medial operations), the chapter ultimately both makes a case for the specificity of the intermedial case study and performs itself as an intermedial case study acting, in turn, as an open invitation for the (hopefully) ongoing emergence and theorization of intermedial case studies well into the future. The findings presented in this chapter take their origin in research conducted by Froger, who was editor-in-chief of Intermédialités, and Bem, who was her postdoctoral student in the Département d’histoire de l’art et d’études cinématographiques at Université de Montréal from 2016 to 2018. Since 2016, Froger and Bem have been collecting case studies from the journal and ordering them in a variety of ways. Thus, we focus, in turn, on their central objects of study which we have grouped by categories that include visual and time-based artworks, human features such as the voice or face, musical pieces, literary texts, performative or research-based events, as well as more incongruous objects such as an ethnographic expedition, an art history lecture, or the motif of the slaughterhouse in filmic and literary works. In addition, we have also catalogued them according to their methodological and conceptual choices (close reading, historization, theorization, as well as approaches ranging from apparatus theory to ethnography) or to the medial relations they aim to identify, label, and theorize. We have presented this work at conferences and in the context of graduate seminars on intermedial theory. The present chapter constitutes the most advanced version of this research. Finally, it should be noted that Froger and Bem presented an early version of this research at the convention of the International Comparative Literature Association in Vienna in the summer of 2016, as well as in Philippe Despoix’s graduate seminar “Champs des études intermédiales” held at Université de Montréal in the spring of 2018, and in the two summer schools “Témoigner en images” organized by Rémy Besson and Claudia Polledri for the Centre d’études et de recherches internationales at Université de Montréal in the summers of 2017 and 2021. Froger also presented parts of this research at the conference “Penser les revues de cinéma et audiovisuel aujourd’hui,” which took place in Paris at the Université Sorbonne-Nouvelle on December 16, 2019. A first article based on aspects of this research was published in French in the journal Communication & Languages in 2021 (Bem & Froger 2021). The present chapter expands substantially upon this earlier work; to date, it offers the most advanced findings of this ongoing research project on the role of “casethinking” (Penser par cas, Passeron and Revel 2005) within intermedial research.

162

M. Froger and C. Bem

Towards a Heuristic of Intermediality Epistemic Options Within Inter- and Transdisciplinarity Under the term “epistemic options,” this chapter understands the collection of possibilities that presented themselves to researchers once these decided to approach the conditions, processes, and meanings of symbolic productions from the perspective of complex mediatic interplays. First and foremost, the development of this perspective required researchers to cross those disciplinary boundaries that had historically contributed to the separation of media (communications) from mediums (art) and, consequently, of their respective studies (media and communication studies on the one hand, art history, film studies, and visual culture on the other). This new perspective also required that researchers reconsider the approaches and theoretical frameworks that had previously been carefully elaborated within the boundaries of disciplinarism (i.e., hermeneutic, semiotic, and formal analysis as well as their central notions of text, sign, and form). Thus, interdisciplinarity became a foundational principle of intermediality by allowing for the consideration of evermore complex case studies where the production and the effects of texts, signs, and forms were correlated to specific medial properties. It also required intermedial researchers to consider elements arising from different corpuses and, at times, to mobilize a variety of methods (micro/meso/macro analysis, close/contextual reading, etc.). Ultimately, the encounter between interdisciplinarity and intermediality also promoted the creation of collectives federated around a shared issue. As a result, two further epistemic options were adopted.

Conceptual Modeling of Intermedial Relations The first epistemic option took the shape of a modeling of intermedial relations with Lars Elleström offering one of the most convincing contributions within this area. For instance, Elleström noted: “I have offered a model for understanding media and intermedial relations, and the point of models is precisely to put aside specific details to make possible a view that is more generally valid. Therefore, I hope that the model may also offer a starting point for methodical analyses in the service of various research questions attaching to mediality at large and more specifically media interrelations” (2021: 84). Elsewhere, he also pointed out: “I really think that the world must be understood as systematically as possible, while at the same time acknowledging the incredible complexity of the mind, of culture, of knowledge, of communication” (2020: 31). Thus, modeling relies on systemic coherence but also reinforces itself from one case study to the next (that is to say that individual case studies feed into a larger model of inquiry and contribute to defining its parameters). In turn, case studies are developed on the basis of adequate models and become canonical examples that feed back into the coherence of modeling. Yet, there are instances when a singular “case” can make visible medial relations in an unforeseen way or even bring to the fore new relations between media. What is more, a single case study can even go so far as to defy, through its very complexity, a preexisting

8

Case Studies as a Heuristic of Intermediality

163

chain of previously modeled interactions. In such instances, the case moves away from the series of facts or of predictable phenomena that had been modeled; a singular case successfully gives rise to corrections of a previously established model and of its theoretical bases. However, since cases which trouble general laws cannot fundamentally be granted generic value, this process of re-evaluation of a serialized model through the singular does not occur without resistance. Rather, according to Jean-Claude Passeron and Jacques Revel, the singularity of the case introduces a form of perplexed judgment, and, as such, it harks back to a provisional yet intolerable situation of “undecidability” (2005: 16). Later, they speak of an “experience of mental disadaptation which is associated with the identification of a case as such.” According to them still, “the case finds its first subjective, entirely negative definition in the interruption it imposes as much on the habitual movement of perceptive experience as on the expected itinerary of descriptive, argumentative, or prescriptive discourse. [...] When singularity makes a case it installs a perplexity of judgment. [...] Most often, the case is born from conflict between rules and applications, as well from the provisional yet intolerable situation of undecidability that results from this tension” (2005: 16; own translation). As they write a little further, “by making immediately visible the reciprocal implications that exist between the articulation of theory and the unfolding of inquiry, case-thinking shines a light on a property shared by all scientific knowledge; this is as true within the history of the hard sciences as it is within that of the historical sciences” (2005: 44; own translation). Thus, the case constitutes an enigma that needs to be solved precisely as it resists modeling or, at the very least, as it calls for more nuanced delimitations of its sphere of applicability to emerge.

Critical Perspectives on Conditions of Intelligibility and the Reconfiguration of Fields of Study The second epistemic option falls under the general heading of mapping, a methodological impulse that tends to accompany the reconfiguration of disciplinary areas (in this case, communications, arts, and literature studies primarily) in response to the study of objects that prove to require interdisciplinary attention. Mapping endeavors makes it possible to identify intermedial moments or turns within certain areas of “X studies.” This is reflected, within Intermédialités itself, by a growing number of issues dedicated precisely to the intersection between intermedial scholarship and various “studies” areas including urban studies (Cisneros and Straw 2009), memory studies (Besson and Polledri 2020b), night studies (Gwiazdzinski and Straw 2015), and border studies (Darroch et al. 2019b). More generally, mapping as a methodological gesture makes it possible to group together studies that inquire into the meaning production of intermedial relations as they occur at both a local and a global level, all the while taking into account the notion of milieu. In the inaugural issue of Intermédialités which is titled “Naître” (being born), journal founder Éric Méchoulan writes that “since mediality is the milieu wherein, thanks to institutions, transformative forces morph into signs and ideas into powers, the prefix [inter] attracts the attention of constant double games with the phenomena of

164

M. Froger and C. Bem

History.” Still according to Méchoulan, however, a third quality, this time of the suffix “-mediality” is that it refers to the “quality of being this or that.” “In the same way that inanity designates the fact of being vain,” writes Méchoulan, “so intermediality is a throwback to the very character of what finds itself between milieux. The suffix [–mediality] adds a further element of reflexivity which makes even more visible the effects of immediacy that constitute milieux, as well as the articulations that make them up or haunt them” (2003: 22; own translation and emphases). Within this context, meaning production refers to the assemblage of shared practices, imaginaries, and discourses and, more generally, to the collections of mediations that, together, come to signify specific cultural contexts. In such instances, case studies abound and become valorized precisely due to their heuristic function as “type cases.” Here, the singularity of the case becomes a means of opening up new areas of investigation. Yet, it is possible to push a case study’s function even further by turning the case into a critical tool of epistemic investigation. In both her introduction to a special double issue of Critical Inquiry on the case (2007b), as well as in the introduction to its follow-up or companion issue “Missing Persons” (2007a), Lauren Berlant catalogues a number of ways of “making a case” including, perhaps first and foremost, the fact of going against those very norms that regulate how objects come to be regarded as meaningful and exemplary. According to Berlant, as long as a case study is not able to change the conditions of exemplarity or of intelligibility of an already installed system, it barely qualifies as a case study and continues, rather, to be relegated to the domain of particularity. However, in those instances when a case study is able to alter these conditions, then that change has a profoundly transformative effect upon an entire personal or collective “sensorium” that had heretofore been perceived as impermeable to change: “I described the case as an event that takes place in a variety of institutional, disciplinary, and ordinary-life contexts that are shaped by the practice or expression of expertise” (Berlant 2007a: 1). “Sometimes, though, an event more than perturbs; it disturbs, creates a louder noise that opens up the field of debate about expertise, modes of description, narration, evaluation, argument, and judgment” (Berlant 2007b: 671). To “make a case,” then, according to Berlant, is to stray from a given model’s realm of applicability with the intent of drawing attention to the case study itself and, in the process, of revealing interpretative possibilities that had heretofore been obscured or neglected. Within intermedial studies, making a case thus becomes synonymous with a form of resistance against the very process of object selection (the set of rules that dictate what makes an object worthy of inquiry). In addition, intermedial case studies also allow scholars to stray productively and to take certain risks while asserting the specificity of their methodological gestures. This is echoed in Berlant’s reflections about being on the editorial collective of Critical Inquiry where she observed that: “Cultural studies and new forms of interdisciplinarity continue to foment new forms of exemplarity and singularity in the humanities and the social sciences in ways that constantly force us at Critical Inquiry to reconsider whether a said knowledge object can support the general claim being made about it” (2007b: 670).

8

Case Studies as a Heuristic of Intermediality

165

The Role of “Research Media” The term “research media” designates any means by which a research activity is effectively turned into academic production, and academic production into knowledge that can be disseminated. As a disciplinary phenomenon – defined both as an organizational principle and a framework of production – this concept has already garnered substantial amounts of attention from intermedial studies quarters. Indeed, from the outset, the rise of the field has been accompanied by intermedial analyses of intermedial studies. Thus, James Cisneros (2007) asks whether the university’s crisis facilitated the emergence of intermedial studies, while Jürgen Müller (2000) argued in favor of an academic re-organization of disciplines. By framing academic journals as objects of study, this chapter builds on this scholarship, especially when it adopts the notion that “journals, taken as a whole, have the function of delineating a field. They work to define disciplinary and sub-disciplinary distinctions, as well as pertinent methodologies, lend coherence to this or that School, construct this or that object of study [...]. While one journal is essential to a given discipline, another is more specialized, another, still, trans-, inter-, or multidisciplinary, etc. Through their very existence, journals contribute to making the academic field legible, navigable, and, thus, questionable” (Damerdji et al. 2018: 12; own translation). Within the social sciences, questions of intra- and interdisciplinary gatekeeping have morphed into power struggles that extend to the academic journals of these fields. A similar phenomenon exists within the humanities: here, journals tend to stake disciplinary claims at the same time as they open themselves up to interdisciplinarity as a way of renewing the approaches and objects of study of a given discipline; other publications, in turn, attempt to free themselves from the shackles of disciplinarity by building on new cultural phenomena which they have also contributed to identifying in the first place. To summarize, then, academic journals occupy a key role in the dynamics that reconfigure the academic playing field, acting simultaneously as actors within and as witnesses to its evolution. This chapter takes as its starting point a targeted search conducted in the ProQuest database and whose primary aim was to focus on the so-called “framing” or gatekeeping aspects of academic journals in order to trace major trends in the penetration rate of intermedial studies across humanities disciplines. The ProQuest search engine makes it particularly easy to identify those journals that publish primary source articles of intermedial theory. While this search does not, by any means, pretend to be exhaustive, it does hopefully provide enough elements to invite further reflection and promote future studies, possibly focusing on platforms other than ProQuest and also conducted in languages other than English.

Intermediality Within Disciplinary Academic Journals So-called disciplinary journals responded to the emergence of the intermedial field with the publication of dedicated special issues where varied epistemic options were developed by means of “exemplary” case studies (i.e., the study of objects or

166

M. Froger and C. Bem

phenomena considered to be pertinent to the disciplinary field in question). Thus, in 2000, the Montreal-based journal Cinémas published a special issue titled “Cinema and intermédialité” (Mariniello 2000), while in 2001 Theatre Research International published a special issue on interdisciplinarity within theatre studies (Shevtsova 2001) where Patrice Pavis predicted the intermedial future of that field (Pavis 2001). Indeed, an article published even more recently in that journal further confirms Pavis’s prediction (see Defrantz 2021). To add a non-European, non-North American-centric perspective, in 2014, the official journal of the Society for Cinema and Media Studies (then titled Cinema Journal) published a special issue dedicated to Latin American cinema which featured a contribution titled “Calling for Intermediality: Latin American Mediascapes,” which was penned by one of the issue’s co-editors, Ana M. López (2014). More recently, in 2019, the Romanian journal Ekphrasis published a special issue titled “The Anthropocene and Intermedial Ecocriticism” (Bruhn and Lutas 2020), and a special issue of Trends in Classics outlined the possibilities opened up by an intermedial turn within those disciplines (Dinter and Reitz-Joosse 2019), while recent articles in semiotics journals prove especially attentive to intersemiotics (Helbo 2016; Melnikova 2020). The earlier special issues, in particular, conceptualized intermediality as a work in progress, and, consequently, they invested substantially in modeling, but also in the case study as a practice through which new hypotheses might emerge. While Convergence allowed André Gaudreault and Philippe Marion to showcase their article “The Cinema as a Model for the Genealogy of Media” as early as 2002, CLCWeb: Comparative Literature and Culture published the models put forward by Wolf and Schröter in a 2011 special issue dedicated entirely to intermediality (Schröter 2011; Wolf 2011). In 2015, SubStance gave Éric Méchoulan a platform to present his “hermeneutics of platforms,” (Méchoulan 2015) and, most recently, the journal Between offered Lars Elleström the possibility to gaze back on his “modalities of media” (2020). Finally, a recent issue of Communication & Langages introduced a greater French-reading audience to the history and methods of the Montreal School of Intermediality (Tadier and Méchoulan 2021), and, in particular, it highlighted the important place occupied by case studies within the output of this group of intermedial scholars (Tadier 2021). A second group of thematic issues that evidences a connection with intermedial research focuses either on familiar questions, such as cultural transfers (Huglo and Villeneuve 2005), adaptation (Hutcheon et al. 2012), and reenactment (Artières 2019), or on new technological phenomena whose full impact remains yet to be measured. Interestingly, a Screen article that tracked the evolution of the field of film and television studies through a systematic review of 10 years’ worth of publications within one of the discipline’s key journals noted a significant shift toward “digital media” marking, at the same time, a move away from capital-T theory within the publication’s pages (Doherty 2014). It would seem, then, that such journals feature intermedial research as one approach among several that holds the power to rejuvenate the study of specific objects (Papagiannouli et al. 2020). Many of these contributions rely on the modeling of intermedial relations such as theorized most famously in texts by Jay David Bolter and Richard Grusin (1999), André Gaudreault

8

Case Studies as a Heuristic of Intermediality

167

and Philippe Marion (2002), Irina Rajewsky (2005), Chiel Kattenbelt (2008), Lars Elleström (2010), and Liliane Louvel (2010), which rank among the all-time most cited. We found 237 articles that mention at least one of these authors in ProQuest, while articles found through the largely French-language database Isidore mention them at least 186 times (research verified on February 21, 2022). Within the last decade, the circulation of these models has given rise to a flurry of articles pertaining to intermediality within a wide range of disciplinary journals.

Intermediality Within Interdisciplinary Academic Journals While most search results returned when entering the lexeme “intermedial” in ProQuest point to articles that are found in disciplinary journals belonging to cinema and media studies, theatre and performance studies, and comparative literature, those hits that refer back to articles published in interdisciplinary journals display a particular set of characteristics worth highlighting. Thus, in spite of their interdisciplinarity, those journals whose claim to specialization rests on their study of the relations that exist between different materials or semiotics such as Word and Image and Interfaces: Image, text, language, occupy a relatively small area of intermedial inquiry where they tend to focus, primarily, on the analysis of remarkable works of artistic and cultural creation. In fact, many of those articles that focus on intermediality in Word and Image do so through case studies of works of art and literature. Yet, it is also noteworthy that transversal groupings of studies, such as those offered rather systematically by the journal Interfaces over recent years, consistently invite renewed understandings of concepts such as that of the “gesture,” which comes to be reconsidered through the notion that certain works leave behind traces, or even through an understanding of works themselves as traces of transmission (Aymes and Bernez 2018a, b). At any rate, the ProQuest search aptly demonstrates how, from one case study to the next, journals are patiently building up scholarly confidence in the heuristic pertinence of “paying close attention” to specific artworks. What is more, works of art are not the only objects that experience a revitalized sense of attention once they are approached through an intermedial lens. Indeed, technical objects can benefit from a similarly renewed level of attention ushered in by the adoption of new methods of inquiry. Among those interdisciplinary journals that intently followed the rise of intermedial studies, Early Popular Visual Culture is especially deserving of attention. By embracing an historical approach, this journal provides an account of its specific period of interest by homing in on the major shift in literacy that was induced by the rise of techniques of mechanical reproduction and media of mass circulation (the press, photography, and the cinema) and which led to the emergence of the societal phenomenon of visual culture. It is probably unsurprising, then, that Early Popular Visual Culture took a particular interest in Gaudreault and Marion’s work on early cinema (2005, 2013), as well as, more recently, in the writings of scholars focusing on media archaeology. Interestingly however, a special issue dedicated by Early Popular Visual Culture to media

168

M. Froger and C. Bem

archaeology (Huhtamo and Galili 2020) features relatively few occurrences of the terms “intermedial” and “intermediality.” This likely has to do with the fact that these terms are used on a broader scale than is “media archaeology.” Indeed, media archaeology presents itself, in turn, as either a branch of intermediality or as its center. Thus, another interdisciplinary journal that focuses on the same period as Early Popular Culture, the journal Modernism, mentions the term “intermediality” more frequently (26 articles), while the term “media archaeology” appears in only 1 out of more than 400 articles that deal with media-related topics. Based on these findings, it becomes evident that considering media archaeology as the center of intermedial studies risks significantly reducing what Jürgen Müller calls intermediality’s “axes of pertinence” (2015). One final group of interdisciplinary journals focuses on social, political, philosophical, and historical questions which are approached through the lens of cultural production and, at times, through that of medial aspects as well. As a search through the ProQuest database makes clear, the interdisciplinary journals most permeable to intermediality are, in declining order: Acardia (59 articles); Humanities (20 articles); Partial Answer (18 articles); Discourse (4 articles). By way of example, a quick search through the archive of Representations reveals that even though only one article addresses directly the question of intermediality, multiple contributions do in fact deal with the medial aspects of their objects of study such as, for example, the visual as a medial aspect of historical writing in the special issue of Representations titled “Visual History: The Past in Pictures” (Bleichmar and Schwartz 2019). More generally, however, mediality remains only one among several aspects that receive regular consideration within this type of interdisciplinary publication where, generally speaking, contributions tend to evidence a rather weak degree of intermedial integration. In other words, if the case studies published in such journals have become increasingly nuanced and complexified due to their authors’ growing awareness of their objects’ intermedial aspects, mediality overall remains only one among several components of the analyses they put forward. At the same time, there has been a global rise in the recognition of intermedial approaches beyond questions limited to media and their relation to the production of objects and medial processes. To summarize, then, due to interdisciplinary journals’ calls for an increase in dialogue across the humanities and, also, between the humanities and the social sciences, intermediality contributes to larger cross-disciplinary conversations.

Epistemic Consequences The exercise of browsing through more than 1200 academic articles published within the last 10 years and which are considered primary references (search term “intermedial” in ProQuest), as well as of scouring the archives of the most reputable journals of arts, literature, and media scholarship, inevitably suggests that intermedial research is a fundamentally “scattered” affair. Indeed, notwithstanding attempts to model semiotic interactions provoked by different media of symbolic production, and despite the integration of medial components within analyses of

8

Case Studies as a Heuristic of Intermediality

169

cultural objects, as well as the fact that more systematic attention is growingly being paid to phenomena of mediation, the intermedial field remains without precise delimitations, perhaps exactly by virtue of the fact that intermedial scholarship is indeed “scattered” across a variety of academic journals, both disciplinary and interdisciplinary. In such conditions, it is difficult to delimitate precisely what a coherent corpus of intermedial scholarship might look like; the “eclectic” nature of intermedial scholarship might itself be conceptualized as intermedial. This, in turn, raises the question of the appropriate choice of method for its study as an object, especially since intermedial scholarship itself tends, by definition, to mix and match methods. No academic journal in existence – not even Intermédialités which, it is worth recalling, remains the sole journal to date to dedicate itself entirely to the study of intermediality – has made it a mission to tackle the epistemic fuzziness that characterizes intermedial scholarship. Founded in Montreal in 2003, Intermédialités was born at the crossroads of North American and European scholarship. Rather than seeking to offer a singular definition of intermediality, the journal proclaims its investment in open-ended theorization couched in a plurality of intermedial approaches whose multiplicity of application is only rivaled by the plethora of objects of study to which they are applied, as well as by the manifold definitions of intermedial phenomena that appear within the journal’s pages and, finally, by the variety of extensions of the field to which they give rise. This plurality takes the journal out of its traditional role: instead of lending increased readability to the academic field, Intermédialités entertains a willful relationship with epistemic fuzziness. This intentional fuzziness is further reinforced by the fact that, through its deliberate showcasing of articles that focus on all manner of cultural production, across all temporal eras and geographic areas combined, the journal squarely situates itself beyond interdisciplinarity. Thus, each issue sheds light on a specific operation or experience of mediation, signified by the action verbs that act as titles for each issue. (the list of issue titles can be accessed here: http://intermedialites.com/en/ category/issues/). From this perspective, Intermédialités refuses to privilege one particular model for the analysis of intermedial relations over another; instead, it opens itself up to philosophical reflections on the very nature of mediation itself, as well as on the strengths of given media rather than, say, on their determinism. In the same vein, the journal also refuses to privilege any one definition of media or mediality; instead, each article seeks to formulate a new theory of mediality rooted in the singularity of its particular object of study and of the medial relations it entertains with itself and its milieu (a set of relations which each article will have taken great care to unfold as carefully as possible). Intermédialités promotes an a-paradigmatic research practice that oscillates between three poles: (1) the valorization of the singularity of given objects of study including events, situations, as well as their individual medial effects; (2) the taking into account of particular knowledge gaps and marginal facts, such as traces, indices, and symptoms, which come to be considered as so many revelatory instances of more general historical and social phenomena; and (3) the exploration of the intermedial melting pot where forms of art and life (e)merge together.

170

M. Froger and C. Bem

It should be added that Intermédialités’s publication process is particularly rigorous for authors and members of the editorial board alike. Indeed, each published article gives rise to in-depth epistemological reflections on intermedial objects of study and on what might constitute such objects in the first place. While each article is evaluated by two to four blind peer reviewers chosen primarily for their expertise in a given article’s subfield(s) (i.e. seventeenth-century theater, silent film, etc.), the articles are also closely evaluated internally, by at least two members of the journal’s editorial board. Where external evaluations focus on more traditional tenets of the peer review process (scholarly soundness, level of expertise in relation to the article’s topic, depth of research, etc.), internal evaluations also take into account an article’s fit, not only with the thematic issue at hand, but with the greater project of the journal. Thus, each article gives rise to a collective inquiry among members of the journal’s editorial board as they consider an object’s contribution to the greater body of work developed within the journal’s pages. Finally, the editorial office sends authors a detailed letter accompanied by condensed versions of anonymous feedback that is drawn from both internal and external reviews. Often, an overarching theme to the letters and review feedback will be to invite authors to enter more willfully into dialogue with preexisting definitions of intermediality and, also, with the journal’s longer history as they, in turn, contribute their piece to it. Ultimately, researchers who publish in the journal are able to adopt a form of indisciplinarity since even the most cursory consultation of the journal’s tables of contents will reveal the wide variety of disciplines covered by its contributors, from literature and art fields to the social sciences, where scholars rarely consider themselves as intermedialists. Considering these highly specific conditions, what forms of knowledge can be seen to emerge within the ground that Intermédialités has claimed for itself?

Case Studies in the Journal Interme´dialite´s/Intermediality This chapter looks back on almost 20 years’ worth of publications in the journal Intermédialités as a way to evaluate the heuristic virtues of case studies for intermedial scholarship. Intermedial case studies unfold symbolic objects in order to show clearly the multiple strata of mediations from which they derive. Similarly, entire medial milieux come to be seen as veritable laboratories of transformation once they are placed under the intermedial microscope. In this way, the role of each intermedial case study in arresting time and allowing itself to sediment within an object becomes, in itself, an object of study. While the singular object of study at the center of this chapter – paths forged through Intermédialités’ archive of articles, most of which are case-study-driven – might appear, at first glance, to originate from a practice of aimless wandering and erratic accumulation within the journal’s archive, it is precisely the fragmented nature of the case studies archived as intermedial object of study that mirrors intermediality’s potential to make sense out of cultural dynamics that are themselves fragmented. After establishing what it is about the case study as method that is inherently intermedial, this section ultimately

8

Case Studies as a Heuristic of Intermediality

171

proposes that the epistemological act of considering sets of case studies within a singular archive as case-study-worthy objects in themselves turns the intermedial method itself – case-study building as a heuristic – back onto intermedial meaningmaking. Consequently, from a collection of studies that apprehend objects as events that are both surprising and non-reproducible emerges the image of a generalized method which may well destabilize the foundations of knowledge-making, yet which simultaneously acts to both preserve and foreground the very basis of epistemic endeavor (understanding as a self-reflexive and action-driven process wherein the researcher glimpses themselves caught in the very act of meaningmaking).

The Role of Case Studies in the Journal Interme´dialite´s/Intermediality The wide array of objects that are studied within the pages of Intermédialités ranges from “expected” objects of study, such as works of art, specific artistic practices, and theoretical texts, to objects that are not typically associated with intermedial scholarship, such as singular activities and events, paradigms, and apparatuses, as well as all manner of formal, material, and ideal phenomena. Among “expected” objects, one might count art works, ranging from the Tapisserie de Bayeux to contemporary intermedial art (Schmitt 2010), as well as the reenactments of Marina Abramović (Benichou 2011), and the circulation of early modern theoretical discourses on poetry and theatre (Blocker 2005), the less “expected” object of a UNESCO convention on the protection and promotion of diversity (Rousseau 2017/2018). While the individual intermedial case studies published in the journal were initially intended to shed light on mediation processes, it is only once they are considered alongside one another that their collective contribution to intermedial theory emerges. In particular, this contribution lies in their collective articulation of the complex relationships that exist between materialities, techniques, institutions, imaginaries, and discourses, along with an illumination of the ties that connect a singular object of analysis to its larger context or milieu. Thus, the methodological gesture of serializing or working with an accumulation of case studies culled from the journal’s archive highlights the manifold ways case studies work to open up the intermedial field to self-investigation and, ultimately, to a re-evaluation of the emergence of intermedial method itself.

Serializing the Case What can be gained from considering sets of published case studies of heterogeneous objects in a serialized way? Most case studies published in Intermédialités focus on identifying, parsing, and ultimately pushing the definition of “medial operations” within objects themselves. Further, they might also focus on the relations between objects or between an object and its given milieu. However, rather than end at shedding light on objects in and for themselves, the journal’s case studies reveal

172

M. Froger and C. Bem

how intermediality, taken as a method, broadens preexisting definitions of media and mediality. Ultimately, individual case studies promote the idea that, in order for an object of study to be medial, it has to act as fundamentally transformative within a concrete medial operation. This sense is only further reinforced once case studies published within the journal are considered alongside one another, through a cumulative process. The accumulation of case studies within the journal Intermédialités allows for both the observation and the study of intermediality’s coming-into-being by grounding it within a series of time periods. Reading the journal through its case studies, so to speak, gives rise to a new narrative about the emergence of intermediality. From this perspective, intermediality is no longer framed exclusively in terms of the “hard limits” of a discipline, but, by outlining its fuzzier contours, intermedial methods are revealed to be concerned primarily with mediation. In an essay that received significant attention upon its publication, The Interface Effect, Alexander Galloway emphasizes the important shift in perspective brought about by the work of Friedrich Kittler (2012). Through his insistence on specific medial operations (storing, transmitting, processing), Kittler is more invested in mediation than in media for themselves. Consequently, any type of artifact might come to be viewed as a medium, so long as it fulfills such operations of mediation. By leaving behind the paradigm of communication, it becomes possible to identify a wider variety of mediations. These might include, from an anthropological perspective, contact operations between different orders of reality (e.g., temporal, spatial, and biological orders); from a philosophical standpoint, modalities of appearing and happening; or from a social science perspective, operations geared at minimizing disputes or at the articulation of differences which, in the realm of the arts, take the shape of creating shared experiences brought forth by the sheer power of media. These broad definitions of mediation in turn lend intermedial inquiry a greater breadth of application. From here, we might attempt a more synthetic definition: within our current field of intervention (the analysis of cultural production), mediations are a set of operations that constitute our experience of an object, give rise to encounters with the Other, and, ultimately, shape the effects of relations. To exploit the heuristic potential of serialization is to consider from the perspective of seriality the entire sets of articles that were not originally intended to complement each other but whose publication within a singular thematic issue has worked to connect them to one another. In the case of Intermédialités, while each of the journal’s thematic issues aims to feature a wide interpretative variety of responses to a given theme, certain more recent issues have radically expanded the limits of what might constitute an intermedial object of study in the very formulation of their themes and call for papers (CFPs) by, in particular, steering contributors towards more overtly social themes (see, for instance: “Inclure (le tiers)”/“Including (the third term)” in 2013 (Froger and Maazouzi), “Cacher”/“Concealing” in 2018, “Ressentir (les frontières)”/“Sensing (borders)” in 2019, and “Témoigner”/ “Witnessing” in 2020). In these issues, the breadth of case studies proposed constitutes in itself a potential intermedial case study, one that is often at least partially theorized within the issues’ robust introductions (Froger 2013; Casemajor and Toupin 2018; Darroch, Engle, and Rodney 2019a; Besson and Polledri 2020a). A

8

Case Studies as a Heuristic of Intermediality

173

further example is found in the introduction to the special issue “Mapping (Intermediality)” (2017/2018) which superimposes its account of the growingly global span of intermedial research over the figure of the map taken as an intermedial object in a definition that reaches from Deleuzian rhizomatic theory to Borgesian fantasies of life-sized maps in contemporary literature (Bem 2017/2018). More specifically, two contributions to issue 8 of Intermédialités (“Envisager”/ “Facing,” Villeneuve 2006) reflect on the medial resonances that connect modern concerns with visageity to a reflection on the contrast between older (painting) and more recent (photography) media of image reproduction. In her article on the face in the seventeenth century for that issue, Lucie Desjardins brings to light a number of fault lines when she intermedially interrogates literary descriptions of portraits on the one hand and literature and literary discourse on visageity on the other (Desjardins 2006). In so doing, Desjardins serializes various texts that evoke painted portraits or that describe their characters primarily in terms of poses and features, thereby allowing her to identify a new form of consciousness whose emergence she locates in the seventeenth century. According to Desjardins, this development hinges on a subject’s newfound consciousness of their own projected image, and as such it gives rise to a new set of concerns: is the face a deceptive surface that serves the vanities of worldliness? Is interiority invisible, hidden, or even fully inaccessible? For a long time, the painted portrait held well-defined functions. It made it possible to ensure the presence of an absent person and to identify a person on the basis of their resemblance to their own likeness, and it turned visageity into an interface that was essential to the revelation of interiority. The transparency of the medium (here painting) served the symbolic and social effectiveness of the portrait. But intermedial writing (whether it takes the shape of ekphrasis or of rhetorical borrowing) initially cast confusion on the face by missing the relevance of the medium to its perception (in this case, that of literature that relies on the processes of painting). Following Guido Goerlitz’s article published in the same issue, it was not until the twentieth century that the question of the medium was first raised in relation to the face, with the advent of photography (Goerlitz 2006). As Goerlitz notes, it is on the basis of his reflection on the photographic portrait which imitates the painted likeness that Siegfried Kracauer makes of painting the medium par excellence that allows unity to emerge where previously there was none. Indeed, the comparison between the two media allowed Kracauer to highlight the link between, on the one hand, the unification of diverse elements within representation and, on the other hand, the domination of the object or subject represented. This, in turn, encouraged Kracauer to value the fragmentary, the detail, and the incomplete, as well as precisely those media that make it possible to apprehend objects in such ways (photography, but even more so cinema with its deployment of montage) and in direct opposition to cultural industries’ or artists’ use of media, suspected of wanting to establish the medium’s domination. From this perspective, Kracauer’s reflection constitutes an original act of sorts, the point of origin for a reflection on the powers of those media brought forth by modernity and whose serialization allows him to reveal their historical depth in unprecedented ways. The studies by Desjardins and Goerlitz evidence similar concerns for the identification of factors conspiring to create

174

M. Froger and C. Bem

significant changes and to identify the effects of such changes upon a medium’s environment or milieu. The serialization of cases makes it possible to identify the constitutive dynamics of those very singularities that cases both unpack and embody. The adoption of a simultaneously qualitative and comparative approach makes it uniquely possible to navigate the particularities of this dynamic. However, a significant issue often encountered within the social sciences is that the number of cases that a singular researcher or research team can take up is limited; this is where case selection becomes central to the constitution of legible series. Indeed, it is useful to remember that the very justification of the possibility of making out of a single occurrence a case and out of a set of cases a series is, in itself, essential to the constitution of case studies. This is even more true when the series at hand are not givens but, rather, come to be constituted as their analysis progresses. Thus, the heuristic potential of serialization is fully revealed once the entire sets of articles that were not originally intended to complement each other come to be considered as series. While this can be the result of publication within a singular thematic issue, as seen above, at other times, connections can be drawn (as in the writing of this chapter) between in appearance unrelated articles that coexist within the greater archive of a journal. In the latter instance, the heuristic gesture of serialization takes into account a journal’s entire output which becomes an intermedial object of study in its own right. The serialization of case studies grants us access to the study of “mediality” rather than to the study of media in and for themselves. Within Intermédialités, the accumulation of cases has allowed those scholars who study them to create heretofore unattempted rapprochements between them (unattempted because it is seemingly impossible, according to traditional methods of inquiry). In other words, serialization creates the conditions of possibility for serendipity to take hold. For a number of years now, the concept of serendipity has become widely used to point to the crucial (or “aha”) moment at which a new scientific discovery is made. Thus, we might imagine a series of case studies constituted, not on any previously defined theoretical or methodological basis, but whose connection, on the contrary, makes possible theoretical and methodological discoveries. Within this context, serialization allows for the study of mediality as a polymorphous and ever-changing phenomenon whose conditions (re)appear singular with each new observation. Serialization, then, invites scholars to read their peers’ work with a completely new aim: i.e., reading, not in order to seek out those who might be working on the same objects or adopting the same approaches as oneself, but rather, reading writings which successfully describe complex sets of mediations with the goal of putting forward unattempted rapprochements. From there, accumulated case studies become a common ground of observation for medial phenomenality.

Diagram This section presents the results of a diagrammatic analysis which is inspired by Fernand Deligny’s maps of customary and wandering lines that were influential, in

8

Case Studies as a Heuristic of Intermediality

175

particular, to the late thinking of Gilles Deleuze when he wrote that “maps [. . .] are superimposed in such a way that each map finds itself modified in the following map, rather than finding its origin in the preceding one: from one map to the next, it is not a matter of searching for an origin, but of evaluating displacements” (Deleuze 1998: 63). The original subjects of Deligny’s research were the physical wanderings of autistic children: not only did the superimposition of the patterns they traced reveal children’s ritualistic repetition of specific itineraries, but it also made visible how certain landmarks became concrete nodes of transient encounter between children and others who did not otherwise communicate directly with each other. Once abstracted from Deligny’s model of application, diagrammatic analysis consists in superimposing the wanderings traced by different case studies in order to identify similar points of contact between them. Thus, nodes of mutual and recurrent fixation emerge between case studies. Deligny would have referred to these nodes as entanglements (chevêtres), and they appear even between case studies whose objects of analysis are seemingly worlds apart. But what is it that gives rise to such insistence, to use another Deleuzian term, from one case study to the next? To analyze case studies, then, necessarily means to partake of a work in progress with no endpoint in sight since nodes of entanglement are necessarily bound to morph and evolve over time, in accordance with the accumulation of case studies, but also with the researcher’s own attention span. Ultimately, within this chapter’s case study (itself made up of a collection of case studies culled from the Intermédialités archive and superimposed in the above-described way), it is possible to isolate at least four such nodes of entanglement which are of primary epistemic importance. In what follows, special attention will be paid (1) to the “medial properties” of objects, as opposed to distinct media per se, (2) to a milieu which can be apprehended by means of projection within a specific object of study, as when Deleuze writes that “[...] the variations of an object are projections that envelop a relation of movement and rest as their invariant (involution). And since each relation involves all the others to infinity, following an order that varies with each case, this order is the profile or projection that in each case envelops the face of Nature in its entirety, or the relation of all relations” (Deleuze 1998: 142–143); (3) to the ways historic narration acts as a mediation of the past which gives way to an analysis that turns the case study into both a narrative and a mediation, and, finally, (4) to the affective charge that specific objects direct towards form and to the individual ways in which researchers respond to this charge.

Medial Properties Certain intermedial case studies are defined by the fact that they are less rooted in an analysis of relations between media than they pay attention to the medial properties of their objects of study. These properties might include the ways objects establish a sense of connection, such as when the recorded voice becomes synonymous of embodiment, but they could equally suggest separateness, such as when the written or printed word acts to install distance from the body. In other words, case studies which focus on medial properties are preoccupied, first and foremost, with the ways in which objects respond to a desire for presence or, on the contrary, with how they

176

M. Froger and C. Bem

respond to absence. When Francis Gingras (2004) studies various translations and adaptations into prose of Ovid’s Metamorphoses, he pays a special kind of attention to the medial properties displayed by the genres of both opera and the novel; while the first ties love to song, the voice, and music’s ability to render present an absent loved one, the second connects love to the realm of death where bodies are reunited and where writing is the dominant medium. Similarly, when Hélène Jacques (2005) studies Denis Marleau’s mise-en-scène of Les Aveugles (Maurice Maeterlinck 1890), she cares specifically about those medial properties that are particular to the use of video projection within the context of theatrical production such as, most saliently, video’s effect of transforming actors’ bodies into specters. Or when Spyros Papapetros (2013) probes into the medial properties of built structures, he opens himself up, analytically, to putting in question the very notion of “dwelling.” In doing so, he begins by evoking how nineteenth-century architectural theory’s conception of the tree reconnects with the very tenets of animism: “The tree,” he writes, “represents the structural and symbolic axis that supports the building and establishes a connection with the natural origins of the high rise” (2013: 35; own translation). Ultimately, Papapetros concludes that even if modern cities no longer seek to directly evoke forests and trees, they continue nonetheless to evidence the presence of an “architectonic soul” which is “pure light, pure vision, and pure extension” (2013: 47; own translation). Even though each of these authors discusses cultural objects originating within widely different eras and societies, the tensions each identifies to underlie their medialities inevitably express a longing for presence and connection. Simultaneously, these tensions also highlight desires brought forth by the very medial processes that come under scrutiny within each case study – instances where a sense of absence and separation from the object becomes itself objectified.

The Case as a Snapshot that Captures a Given Milieu’s Dynamic A case study can be thought of as a snapshot that freezes time. The case makes it uniquely possible to see or grasp, which is to say that it radically slows down a dynamic phenomenon. Therefore, the case acts as a mediator for the researcher writing; it takes the shape of a phenomenological objectivity into which it is deposited – in the form of remains or ruins – what medial processes themselves risk taking away. Within this context, it is useful to distinguish between a centripetal approach (here, the constitutive dynamics underlying objects of study are brought towards the midpoint or milieu) and a centrifugal approach, which establishes the study of objects as the very means by which that midpoint or milieu can be rejoined (indeed, in French, one of the meanings of milieu is “midpoint”). The latter approach is most commonly encountered within the case studies published in Intermédialités; indeed, the centrifugal approach is uniquely equipped to shine a light on the manifold ways a given milieu is being transformed. By way of example, in his contribution to the journal’s issue “Projeter”/“Projecting” (2014/2015), Olivier Lugon focuses on the plethora of dispositifs of image projection that were deployed during Expo ‘64 in Lausanne. His contention is that these dispositifs acted as aesthetic vectors for the promotion of the automatization of everyday life that was

8

Case Studies as a Heuristic of Intermediality

177

just beginning to take hold in Europe and, thus, that they effectively functioned as invisible actors of penetration (Lugon 2014/2015). Each case becomes responsible for the unfolding of a specific phenomenon by following as many ramifications and maps (technical, cultural, economic, and political) as possible. The case, then, becomes the node of intersection or entanglement between these ramifications. This definition of the intermedial case harks back to a social science-centric defense of the case study as a means of producing singular “images” or snapshots of the entire environments of interest that are rendered accessible through them. For instance, when Michael Cowan studies the visual effects of silent sci-fi films, he is able to show how the political imaginary of an entire era comes to be transformed by the arrival of radio (Cowan 2014). Johanne Lamoureux, in turn, sees in the recurrence of a singular motif – the slaughterhouse – the anxiety that underlies a particular form of between-the-wars critical discourse (Lamoureux 2008). According to Lamoureux, images of slaughterhouses acted, in turn, as literary, cinematographic, and painterly metaphors which condensed the effects of mechanization that were concomitant with the rise of both the assembly and the slaughterhouse line. As such, these images contributed to the spread of a conception of capitalism as not only exploitative of its proletarian workforce, but also as literally slaughtering the bodies of workers and feeding off their sacrifice. Once again, eras and objects differ significantly across the above-cited examples, yet in each of these cases the medial processes that affect various milieux are identified through the application of an unusual lens of analysis. These range from a certain projection dispositif to special effects within silent sci-fi films, to the insistence on a particular visual motif. Ultimately, these elements become “media” in themselves, defined in the primary sense which Aristotle attributed to the term, as a metaxu whose main function was to “make things visible” (for an account of mediality in Aristotle’s thinking, see Alloa 2009). Here, “media” designates the possibility of grasping or envisioning in a way that is no longer merely “sensible,” as it was for Aristotle, but intellectual as well.

Historical Narrative as an Act of Mediation: The Researcher as Actor of Mediation Currently, there exists within intermedial studies no routinization of the case, at least not in the sense discussed by Passeron and Revel in relation to the social sciences. Consequently, intermedial research has no regime of logico-hypothetical deduction that would conceive of the case study as a necessary prelude to paradigmatic change. Nor does intermedial research create a theoretical hegemony to be overthrown, as Lauren Berlant writes of the case in relation to literary studies, where the case study becomes a means of changing the conditions of exemplarity or intelligibility of a given theory (Berlant 2007a). According to Berlant, something deep within the object inevitably escapes the researcher’s gaze failing, as a consequence, to be reflected as an operator of mediation (i.e., something of the medial properties of the object always escapes analysis); conversely, at the level of the researcher, “the case-study method always assumes the sociality of knowledge, the circulation of discourse as its condition, and the clarifying obligation of analytic narrative”

178

M. Froger and C. Bem

(Berlant 2007b: 668). For Berlant, the case sheds light on normative judgment (on the real) from which it proceeds but which it can then also contest. According to her, case studies make it possible to denounce the claim to transparency that is carried by any theoretical approach with a generalist aim (Berlant 2007b). The case thus invites the researcher to wonder about their own relation to the object, that is to say about each theoretical, methodological, and affective tool they mobilized in order to approach it. Even after the intermedial field opens itself up to, and is in turn opened up by, the case study, the outcome of the above-outlined anti-hegemonic struggle remains undecided. As a result, researchers writing in Intermédialités adopt an attitude similar to Berlant’s in that they turn their attention to works which include a reflexive dimension pertaining to their own mediality. What is more, Intermédialités contributors are typically interested in developing intermedial ways of producing reflexivity: indeed, most contributions to the journal seek to understand the researcher’s own position as mediator and to identify what, exactly, within their own methodological gesture, but also within the general conditions of research production, might qualify as medial operation. In other words, contributors to Intermédialités typically have little interest in normative judgments, and, in fact, they would rather free themselves of these once and for all – instead, they tend to pay close attention to that which sets their approach apart and which depends on a given researcher’s (often idiosyncratic) way of mediating their objects of study. “To tell,” “to restore,” “to redo,” to translate,” etc. are so many of Intermédialités’s issue titles that refer to acts of narration and to the studies published within these individual journal issues that tell specific stories. Thus, storytelling is of primary concern to the writing intermedialist who might easily conceive of themselves as the last “narrator” in the chain of (theoretical) storytelling. As discussed above, contributors to the journal tend to focus on the medialities that are at work within discourses, including the contributor’s own. For instance, Nia Perivolaropoulou demonstrates how sensorial experiences of urban and cinematographic space provide a starting point for the attention that Kracauer paid to architectural and social forms in view of cataloguing the major cultural shifts of his time (Perivolaropoulou 2009). Similarly, Elitza Dulguerova takes as her central object of study historical exhibits which are being re-mounted by museums for a contemporary audience. Focusing on how museal institutions become the narrators of their own history, her analysis ultimately reveals how the failure to fully reproduce a historical exhibit for a present audience paradoxically acts to reinforce that original exhibit’s historical impact (Dulguerova 2010). Finally, Frédérique Berthet historicizes the many forms of witness accounts delivered on film by Marceline Loridan, a concentration camp survivor. With great precision, Berthet details the importance of modes of attention to archival documents and the ways in which it comes to be reflected within her own mode of writing. In so doing, Berthet develops a new level of connection with Marceline, which relies on a close reading of her apparitions and on the forms of testimonial Marceline adopts as well as on the details which constitute their delicate memorial substance. Through a welcoming and empathetic attitude, Berthet acts as a mediator, providing an additional space in her writing where Loridan’s historical narrative can be deposited and kept safe (Berthet 2020).

8

Case Studies as a Heuristic of Intermediality

179

In these ways, and from case to case, the importance of very different kinds of mediality (and mediation) within historical narratives – that is, within a given researcher’s unique gaze as it translates into historicizing discourse – emerges as a central aspect of the intricate tapestry of mediations that Intermédialités contributors have been busy weaving, both individually and collectively, over the past 20 years.

The Force of Form and Its Effects The present chapter takes as its central object of study the transversal reading of an entire archive of case studies to parse, in particular, one form of methodological mediation researchers might be led to perform in their own case-study-driven output. Among the overt intermedial strategies adopted by researchers, readers might be invited to see and hear the medial processes that are tied to a specific object by paying direct attention to media forms and no longer simply to discursivity. An apt example comes from a contribution titled “Conférence-projection et performance orale: Warburg et le mythe de Kreuzlingen” where Philippe Despoix takes as his object of study art historian Aby Warburg’s 1923 Kreuzlingen lecture on serpent ritual (Despoix 2015). Despoix’s primary aim is to shine a light on a heretofore undertheorized Warburgian method, namely, to highlight the protocinematographical character of the art historian’s utilization of a great number of projection slides (many of which have unfortunately since gotten lost). In his analysis, Despoix shows how, through the development of a unique image presentation technique, Warburg was able to elaborate a dispositif which Despoix terms “a double theatre of memory projections” where the “visual reminiscence of heterogenous cultural strata” encounters the “rememoration of Warburg’s singular trajectory” (2014/2015: 27; own translation). Ultimately, then, Despoix’s aim is not simply to outline that Warburg’s technique is proto-cinematographic but, rather, he ends up highlighting the proto-intermedial particularities of Warburg’s mode of working. Indeed, if Warburg himself tended to accumulate case studies, it was precisely with the aim of distilling, if not a universal form of understanding they might have contained, then at least a collective form of knowledge born from both images and medialities. A method like Despoix’s returns the act of writing to the heart of the researcher’s work, not only as a technique for transposing rational operations onto a data set, but as a facilitating agent for grasping the particularities of sensorial and participatory experience. In this way, the approaches collected in Intermédialités veer closer to aesthetics than to the logical and dialectical adjustments of discourse that are constitutive of the social sciences. Such methods, in other words, center on the act of intelligibility as it connects to those sensorial-aesthetic experiences (of forms, rhythms, vibrations, and textures) that new media allow. As a result, a community of gestures is being formed, on the one hand, by the writer-researcher and, on the other, by the object of study’s maker. This imagined community becomes a transient space where each party reflects on the power of their respective medium. In their own way, each actor reveals the central role occupied by techniques as well as by a given narrator’s particular sensitivity. In his analysis of Michelangelo Antonioni’s Blow Up (1967), which was adapted from a novella by Julio Cortàzar, James Cisneros shows that both the writer and the filmmaker elected

180

M. Froger and C. Bem

to lend a central role to technology in the narrator’s structuring of the story. Indeed, the latter is less a narration of a central event than it takes the form of an incomplete and fragmented snapshot made from the perspective of a technological eye. As Cisneros notes, the two objects (the novella and the film) each produce a reflection resulting from formal choices that destabilize the narrative as it is being told. In the process of producing this analysis, Cisneros reflects on his own sensitivity to form and on the role played by his own receptor-body in this theoretical endeavor (Cisneros 2003). In yet another example, Jasmine Pisapia focuses on the ways in which the anthropologist Ernesto de Martino resorted to photography and film to construct an “archive of intensities” wherein he recorded not only the specific dances he was studying, but also his own reactivity to “intense” gestures (much like what Warburg had begun to imagine before him) (Pisapia 2011). Finally, in a contribution titled “Interactivity and Affect in Intermedial Art: Theorizing Introverted and Extraverted Intermediality,” Jasper Sluijs and Anneke Smelik take as their objects of analysis two works of art: the sound installation Son-O-House (2004) and the crowdsourced animation Elephants Dream (2006). As the authors observe, these works invite the spectator into a state of cognitive and affective fuzziness. Subsequently, Sluijs and Smelik undertake to theorize the affective charge of intermedial art, even if they do not assume that this particular charge can necessarily be attributed to other, similar works. As they make clear, this particular affective intensity results from the particular encounter between themselves, as researcher-narrators, and these two works. Moving beyond the initial, ready-traced interpretative path they had adopted (by building on Bolter and Grusin’s theory of remediation), Sluijs and Smelik end up devising their own theoretical response to this encounter by developing a pair of concepts (intra- and extraverted intermediality) which they seek to apply to their experience (Sluijs and Smelik 2009). To seek out the origin of formal force (and not the causes of an effect) implies paying particular attention to the subject doing the reacting (in this case, the researcher themselves). What traces might such reactions to case studies leave behind? At the level of Intermédialités as a whole, the perceived effect is one of ever-renewing conceptual inventiveness since a large portion of articles published in the journal put forward new concepts, either implicitly or explicitly, in detail or in passing. These concepts might be poetic or playful, but they are always original. Over the years, these have included the notion of the “in-between-ness” (être-entre and inter-esse) of intermediality, theorized by Éric Méchoulan in his introduction to the journal’s inaugural issue by the title of “Naître” (being born, 2003), where Méchoulan also develops the original concepts of “production of presence” (production de présence), “sensorial dispositif” (dispositif sensible), and “fold of time” (pli du temps); the concepts of “minimal medium (deictic and infra-thin crystal)” (médium minimal [crystal déictique et inframince]), “trace product” (produit trace), “inter-vision” (entre-vision), and “plicature” that appear in the article “La transparence: obsession et métamorphose” by Mireille Buydens (2004); and, when in reference to the decomposition of photograms, André Habib speaks of “images in arrested movement that present as compressed stratified durations which are deployed into as many figures of matter” (Habib 2004: 160; own translation).

8

Case Studies as a Heuristic of Intermediality

181

Conclusion As attested by their exponential presence across journals indexed in ProQuest over the last 10 years, case studies constitute an especially efficient tool for promoting intermedial studies’ penetration rate across the humanities. A critical mass of these studies has built on exemplarity to harness cases’ heuristic role and thus challenge existing models of intermedial relations while renewing traditional questions that take into account key changes introduced by the rise of new media technologies. Meanwhile, in addition to focusing on objects that have been traditionally considered to be intermedial, such as works of art, the large and very diverse cohort of international scholars who have contributed to Intermédialités over the past 20 years have this in common that they have often elected to deploy case studies with the intention of exploring a field of study that reaches beyond the humanities. In particular, they have done this by tackling objects that have traditionally been the focus of the social sciences, such as practices, gestures, and collective phenomena. By building on the work of these scholars, this chapter has argued that the archive of case studies of a given journal (here, Intermédialités) can in itself become an intermedial object. The intermedial gesture of serializing case studies published in the journal has given rise to new historical perspectives on medial phenomena that had heretofore been ignored by more generalized macroanalytical approaches. Diagrammatic analysis in particular reveals that the journal’s case studies pay attention to (1) those phenomena of presence and absence that are at the heart of mediation, (2) the role that operators and agents of mediation come to play in what might be called a “production of forms,” and (3) cognitive-affective effects on these forms and their reflection within academic texts that both theorize and carry their traces within them. As a result, a new heuristic role emerges for case studies, which is to familiarize their readership with the notion that every object has the potential to be intermedial. In addition, case studies also promote the careful “unfolding” of given objects in order to reveal their milieu all the while introducing anxiety and attention into the very core of knowledge production by making visible or tangible that which cannot be generalized. Ultimately, however, these case studies manifest as so many points or nodes which, together, give rise to a profoundly rhizomatic conception of intermediality.

References Alloa, Emmanuel. 2009. Metaxu. Figures de la médialité chez Aristote. Revue de métaphysique et de morale 62 (2): 247–262. Artières, Philippe. 2019. Introduction. Reconstituer le passé, les enjeux pluriels de pratiques multiples. Sociétés & Représentations 47: 9–13. Aymes, Sophie, and Marie-Odile Bernez. 2018a. Gestures and their traces. Special issue. Interfaces 39. ———. 2018b. Gestures and transmission. Special issue. Interfaces 40. Bem, Caroline, ed. 2017/2018. Cartographier (l’intermédialité)/Mapping (Intermediality). Special issue, Intermédialités/Intermediality 30–31.

182

M. Froger and C. Bem

Bem, Caroline, and Marion Froger. 2021. D’un cas à l’autre: Construction collective d’une théorie en chantier. Les études de cas dans la revue Intermédialités. In Études intermédiales: A la rencontre de l’École de Montréal, ed. Elsa Tadier and Éric Méchoulan. Special issue, Communication & Langages 208–209: 247–267. Bénichou, Anne. 2011. Marina Abramović: The Artist Is [Tele]Present. Les nouveaux horizons photographiques de la (re)performance. Intermédialités/Intermediality 17: 147–167. Berlant, Lauren. 2007a. Introduction: What does it matter who one is? In Missing persons, ed. Lauren Berlant. Special issue, Critical inquiry 34 (1): 1–4. ———. 2007b. On the case. In On the Case, ed. Lauren Berlant. Special issue, Critical inquiry 33 (4): 663–672. Berthet, Frédérique. 2020. je suis vivante ! – Marceline Loridan-Ivens (1928–2018). Intermédialités / Intermediality (36). https://doi.org/10.7202/1080957ar Besson, Rémy, and Claudia Polledri. 2020a. Introduction. In Témoigner/Witnessing, ed. Rémy Besson and Claudia Polledri. Special issue, Intermédialités/Intermediality 36. https://doi.org/10. 7202/1080949ar. ———, ed. 2020b. Témoigner/Witnessing. Special issue, Intermédialités/Intermediality 36. Bleichmar, Daniela, and Vanessa R. Schwartz, eds. 2019. Visual history: The past in pictures. Special issue. Representations 145: 1. Blocker, Deborah. 2005. The hermeneutics of transmission: Deciphering discourses on poetry and the arts in early modern Europe (1500–1800). Intermédialités/Intermediality 5: 37–60. Bolter, Jay David, and Richard Grusin. 1999. Remediation: Understanding new media. Vol. 4, 208. Cambridge, MA: MIT Press. Bruhn, Jørgen and Liviu Lutas, ed. 2020. The anthropocene and intermedial ecocriticism. Special issue, Ekphrasis. Images, Cinema, Theory, Media 24: 2. Buydens, Mireille. 2004. La transparence: obsession et métamorphose. Intermédialités/ Intermediality 3: 51–77. Casemajor, Nathalie, and Sophie Toupin. 2018. Introduction. The Mediality of concealment: Material practices and symbolic operativity. In Cacher/concealing, ed. Nathalie Casemajor and Sophie Toupin. Special issue, Intermédialités/Intermediality 32. https://doi.org/10.7202/1058467ar. Cisneros, James. 2003. How to watch the story of film adaptation. Cortázar, Antonioni, blow-up. Intermédialités/Intermediality 2: 115–131. ———. 2007. Remains to be seen. Intermediality, Ekphrasis, and institution. In Intermédialité et socialité. Histoire et géographie d’un concept, ed. Marion Froger and Jürgen Muller, 15–28. Münster: Nodus Publikationen. Cisneros, James, and Will Straw, ed. 2009. Bâtir/Build. Special issue, Intermédialités/ Intermediality 14. Cowan, Michael. 2014. The realm of the earth: Simultaneous broadcasting and world politics in interwar cinema. Intermédialités/Intermediality 23. https://doi.org/10.7202/1033343ar. Damerdji, Amina, Samuel Hayat, Natalia La Valle, Anthony Pecqueux, and Christelle Rabier. 2018. Éditorial. Le savoir-faire des revues. Tracés: Revue de Sciences humaines 18: 11–24. Darroch, Michael, Karen Engle, and Lee Rodney. 2019a. Introduction: Sensing (Borders). In Ressentir (les frontières)/Sensing (Borders), ed. Michael Darroch, Karen Engle, and Lee Rodney. Special issue, Intermédialités/Intermediality 34. https://doi.org/10.7202/1070869ar. ———, ed. 2019b. Ressentir (les frontières)/Sensing (Borders). Special issue, Intermédialités/ Intermediality 34. Defrantz, Thomas F. 2021. Intermediality and queer African American improvisation: Dianne Mcintyre, sounds in motion. Theatre Research International 46.2 (7): 115–127. Deleuze, Gilles. 1998 [1993]. Essays clinical and critical. Trans. Daniel W. Smith and Michael A. Greco. London: Verso. Desjardins, Lucie. 2006. De la “surface trompeuse” à l’agréable imposture. Le visage au XVIIe siècle. Intermédialités/Intermediality 8: 53–66. Despoix, Philippe. 2015. Conférence-projection et performance orale: Warburg et le mythe de Kreuzlingen. Intermédialités/Intermediality 24–25. https://doi.org/10.7202/1034167ar.

8

Case Studies as a Heuristic of Intermediality

183

Dinter, Martin T., and Bettina Reitz-Joosse. 2019. Intermediality and Roman literature. Special issue. Trends in Classics 11: 1. Doherty, Helene. 2014. Tracking change in screen: The empirical route. In Screen on screen, ed. Finn Daniels-Yeomans. Special issue, Screen 55 (3): 421–431. Dulguerova, Elitza. 2010. L’expérience et son double: notes sur la reconstruction d’expositions et la photographie. Intermédialités/Intermediality 15: 53–71. Elleström, Lars. 2010. The modalities of media: A model for understanding Intermedial relations. In Media borders, multimodality and intermediality, ed. Lars Elleström, 11–48. London: Palgrave Macmillan. ———. 2021. The modalities of media II: An expanded model for understanding intermedial relations. In Beyond media borders, ed. Lars Elleström, vol. 1, 3–91. London: Palgrave Macmillan. Elleström, Lars, Massimo Fusillo, and Mattia Petricola. 2020. “Everything is intermedial:” a conversation with Lars Elleström. Between X (20): 26–45. Engberg, Maria, and Bolter Jay David. 2020. The aesthetics of reality media. Journal of Visual Culture 19 (1): 81–95. Flyvbjerg, Bent. 2006. Five misunderstandings about case-study research. Qualitative Inquiry 12(2): 219–245. Froger, Marion. 2013. Introduction. Socialité et médialité: inclure du tiers. In Inclure (le tiers), ed. Marion Froger and Djemaa Maazouzi. Special issue, Intermédialités/Intermediality 21. https://doi.org/10.7202/1020617ar. Froger, Marion, and Djemaa Maazouzi, ed. 2013. Include (le tiers). Special issue, Intermédialités/ Intermediality 21. Galloway, Alexander R. 2012. The interface effect. Oxford: Polity press. Gaudreault, André, and Philippe Marion. 2002. The cinema as a model for the genealogy of media. Convergence 8 (4): 12–18. ———. 2005. A medium is always born twice. Early Popular Visual Culture III (1): 3–15. ———. 2013. Measuring the “double birth” model against the digital age. Early Popular Visual Culture XII (2): 158–177. Gingras, Francis. 2004. Aimer hors chant: réinvention de l’amour et invention du “roman”. Intermédialités/Intermediality 4: 18–43. Goerlitz, Guido. 2006. Visage et ornement. Remarques sur une préhistoire de la visagéité photographique dans la modernité allemande chez Simmel et George. Intermédialités/Intermediality 8: 103–119. Gwiazdzinski, Luc, and Will Straw, ed. 2015. Habiter (la nuit)/Inhabiting (The Night). Special issue, Intermédialités/Intermediality 26. Habib, André. 2004. L’amour, en ruines: notes sur quelques photogrammes du film de Bill Morrison Light is Calling. Intermédialités/Intermediality 4: 157–162. Helbo, André. 2016. Semiotics and performing arts: Contemporary issues. Social Semiotics 26 (4): 341–350. Huglo, Marie-Pascale, and Johanne Villeneuve. 2005. Introduction: Memory, media, art. SubStance 34 (1): 78–80. Huhtamo, Erkki, and Doron Galili, eds. 2020. Media archeology. Special Issue. Early Popular Visual Culture 18: 4. Hutcheon, Linda, Massimo Fusillo, and Marina Guglielmi, eds. 2012. Adaptation: The transformations of stories through code-switching. Special issue, Between 04: 2012. Jacques, Hélène. 2005. Un théâtre d’acteurs vidéographiques: Les aveugles de Denis Marleau. Intermédialités/Intermediality 6: 79–94. Kattenbelt, Chiel. 2008. Intermediality in theatre and performance: Definitions, perceptions and medial relationships. Cultura, lenguaje y representación 6: 19–29. Lamoureux, Johanne. 2008. Le travail de la viande. Intermédialités/Intermediality 11: 13–34. López, Ana M. 2014. Calling for Intermediality: Latin American Mediascapes. Cinema Journal 54 (1): 135–141.

184

M. Froger and C. Bem

Louvel, Liliane. 2010. Le tiers pictural: Pour une critique intermédiale. Presses universitaires de Rennes. Rennes. [Edited and translated by Angeliki Tseti in 2018 as The pictorial third. An essay into intermedial criticism. London: Routledge.] Lugon, Olivier. 2014/2015. The automatic exhibition: Slide shows and electronics at the Swiss National Exhibition 1964. Intermédialités/Intermediality 24–25. https://doi.org/10.7202/1034161ar. Maeterlinck, Maurice. 1901–1902 [1890] . Théâtre illustré de 10 compositions originales. [Vol. 1]. Bruxelles : E. Deman. Mariniello, Silvestra, ed. 2000. Cinéma et intermédialité. Special issue, Cinémas 10: 2–3. Méchoulan, Éric. 2003. Intermédialités: le temps des illusions perdues. In Naître, ed. Éric Méchoulan. Special issue, Intermédialités/Intermediality 1: 9–27. ———. 2015. Intermediality: An introduction to the arts of transmission. SubStance 44 (3): 3–18. Melnikova, Irina. 2020. Intermedial references and signification: Perception versus conception. Semiotica 2020 (236–237): 231–250. Müller, Jürgen, E. 2000. L’intermédialité, une nouvelle approche interdisciplinaire: Perspectives théoriques et pratiques à l’exemple de la vision de la télévision. In Cinema et intermédialité, ed. Silvestra Mariniello. Special Issue, Cinémas 10 (2–3): 105–134. Müller, Jürgen E. 2015. Intermediality in the age of global media networks—Including eleven theses on its provocative power for the concepts of “Convergence,” “Transmedia Storytelling,” and “Actor Network Theory”. SubStance 44 (3): 19–52. Papagiannouli, Christina, Joanne Scott, and Piotr Woycicki, eds. 2020. Inclusive intermedialities. Special issue, International Journal of Performance Arts and Digital Media 16: 2. Papapetros, Spyros. 2013. L’âme est l’architecte de son propre corps: Animisme et tectonique entre la forêt ancienne et la ville moderne. Intermédialités/Intermediality 22. https://doi.org/10.7202/ 1024115ar. Passeron, Jean-Claude, and Jacques Revel. 2005. Penser par cas. Raisonner à partir de singularités. In Penser par cas, ed. Jean-Claude Passeron and Jacques Revel, 9–44. Paris: Éditions de l’École des hautes études en sciences sociales. Pavis, Patrice. 2001. Theatre studies and interdisciplinarity. Theatre Research International 26 (2): 153–163. Perivolaropoulou, Nia. 2009. Entre textes urbains et critique cinématographique: Kracauer scénariste de la ville. Intermédialités/Intermediality 14: 19–35. Pisapia, Jasmine. 2011. Archives du pathos: Ernesto De Martino et la survivance. Intermédialités/ Intermediality 18: 45–69. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités/Intermediality 6: 43–64. Rousseau, Philip. 2018. Sur les ondes transculturelles. La diversité (des expressions) culturelle (s) selon l’UNESCO. Intermédialités/Intermediality 30–31. https://doi.org/10.7202/1049949ar. Schmitt, Jean-Claude. 2010. Broder les rythmes: A propos de la Tapisserie de Bayeux. Intermédialités/Intermediality 16: 23–34. Schröter, Jens. 2011. Discourses and models of Intermediality. CLCWeb: Comparative Literature and Culture 13: 3. https://doi.org/10.7771/1481-4374.1790. Shevtsova, Maria, ed. 2001. Special focus: Theatre and interdisciplinarity. Special issue, Theatre Research International 26: 2. Sluijs, Jasper, and Anneke Smelik. 2009. Interactivity and affect in intermedial art: Theorizing introverted and extraverted intermediality. Intermédialités/Intermediality 13: 177–196. Tadier, Elsa. 2021. L’intermédialité montréalaise, ou l’art de penser la relation. In Études intermédiales: A la rencontre de l’École de Montréal, ed. Elsa Tadier and Éric Méchoulan. Special issue, Communication & Langages 208–209: 9–26. Tadier, Elsa et Éric Méchoulan, ed. 2021. Études intermédiales: A la rencontre de l’École de Montréal. Special issue, Communication & Langages 208–209. Villeneuve, Johanne, ed. 2006. Envisager/Facing. Special issue, Intermédialités/Intermediality 8. Wolf, Werner. 2011. (Inter)mediality and the study of literature. CLCWeb: Comparative Literature and Culture 13: 3. https://doi.org/10.7771/1481-4374.1789.

9

Linnaeus University Center for Intermedial and Multimodal Studies and the Legacy of Lars Ellestro¨m Jørgen Bruhn

and Beate Schirrmacher

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Understanding the Similarities Between Dissimilar Media – The Work of Lars Elleström . . . A Media-Centered Approach to Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Developing, Adapting, and Connecting the Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion: Current and Future Developments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

186 187 190 194 197 200

Abstract

This chapter introduces The Linnaeus University Centre for Intermedial and Multimodal Studies (IMS) and presents the theoretical and institutional legacy of its founder Lars Elleström to the field of intermedial studies. We present central aspects of his theoretical framework and we discuss how concepts such as the media product, the media modalities, and media transformation respond to central challenges in intermedial theory. Also, we demonstrate how they have been applied in intermedial analysis at IMS and internationally as a flexible framework that can be connected with approaches to media. Finally, we put forth some of the challenges to Elleström’s framework as a media-centered model of communication and hint at possible ways to meet these. Keywords

Intermediality · Multimodality · Modalities · Lars Elleström · Communication

J. Bruhn (*) · B. Schirrmacher Linnaeus University, Växjö, Sweden e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_65

185

186

J. Bruhn and B. Schirrmacher

Introduction The Linnaeus University Centre for Intermedial and Multimodal Studies (IMS) is an interdisciplinary research center situated in Växjö, Sweden, which takes a broad approach to media and communication. Having received its status as a center of Excellence in 2015, IMS now houses an interdisciplinary research environment in which scholars of literature, film, linguistics, design, musicology, art history, religion, and media and journalism explore the intermedial and multimodal complexities of communication. Sometimes referred to as the Växjö school of intermediality, the group has worked for more than a decade to establish a theory and terminology for the analysis of intermedial relationships that draw on insights from multimodality and semiotics. In the center’s activities, intermedial, multimodal, and media and communication approaches highlight specific aspects of the complexity of media and communication. This broad approach to the mixedness of human communication in many different media types is one of the main characteristics of IMS. IMS aims to develop and implement theoretical concepts that can describe, analyze and compare a broad array of different media phenomena ranging from feature films and music in animation films to mathematical equations and the truthfulness of news items, to mention just some of the material that has been analyzed. IMS explores these aspects as they play out in current social challenges, such as the effects of the digital transformation and climate change communication. IMS further aims to develop a complementary understanding of multimodal simultaneity and intermedial interrelations that leads to media analyses in which detailed examinations of media and modes can be complemented with a view of the role of media as social infrastructure and institutions in the same way as this is explored in media and communication studies. The work of Lars Elleström has been and is central to the activities and aims of IMS: both his theoretical framework for media, intermedial relationships, and communication and his continual work that aimed to bring intermedial, communication, and multimodal scholars together. Lars Elleström (1960–2021) established IMS as a research center and was head of the center until his untimely death in 2021. His academic career reflects the development of intermedial studies, that is, the transition from comparative literature via interart studies to what is now referred to as intermedial studies. In what follows we present a sketch of his academic career and his major works and then describe his theories and terminology in some detail. After that, we explain some of the methodological and analytical implications of Elleström’s general medial framework and how they have been adapted, developed, and applied to understand specific media contexts and relations within IMS and internationally. Finally, we discuss what we believe are the challenges to Elleström’s systematization and hint at possible ways to confront these. We then examine the potential uses for his systematization in the future and suggest how they can be developed.

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

187

Understanding the Similarities Between Dissimilar Media – The Work of Lars Ellestro¨m Lars Elleström studied comparative literature (litteraturvetenskap) at Lund University and his Ph.D., in 1992, was a dissertation on the Swedish writer, critic, and translator Karl Vennberg. Looking back on his early academic output, it is clear that he did not begin as an intermedial scholar. This is partly because intermedial studies as we would refer to it at the time of writing, and which is influenced by Elleström’s own contributions, did not really exist when he was a student of literature in the 1980s. It is more precise to say that he became an intermedial scholar and that he was inspired by early influential figures in the field such as Ulla-Britt Lagerroth and Hans Lund, both Swedish, who at conferences and in publications brought together national and international researchers interested in interart relationships. In the Swedish context, the anthologies I musernas tjänst [In the Service of the Muses] (Lagerroth 1993) and Interart Poetics (Hedling et al. 1997) take a broad perspective on comparative art questions, indicating the move from an interart perspective to an intermedial one, and this was continued in the writings of scholars such as W. J. T. Mitchell, Werner Wolf, Irina Rajewsky, Claus Clüver, and others. Also in the Swedish context, the anthology Intermedialitet. Ord, bild och ton i samspel [Intermediality. The interrelations of words, images and sound] (Lund 2002) provides Swedish students with an introduction to anything from stamps to musicalized fiction, concrete poetry, and liturgy as intermedial phenomena. In the introduction to the volume, Hans Lund clearly stresses how the relationship between what are perceived as different media types and art forms is based on the overlapping as “words, images and tone/sound can rarely be seen in total isolation from each other” (Lund 2002, 10, our translation). When we look at Elleström’s early publications, it becomes evident that although his academic studies was focused on comparative literature, he was trained in an academic environment that was very open to questions that were not exclusively “literary.” This can be seen in his early work, for instance, in the short and pragmatic textbook written in Swedish on how to read and analyze poetry (Elleström 1999). It is still in use today and it has several important comparative media and comparative art aspects to it. Here, Elleström was approaching poetry “proto”-intermedially by seeing poetry as being beyond any literary genre conventions as language that is perceived as sound and image, as “poetry for the ear and the eye” (1999, 12, our translation). Another important early work is Divine Madness, with the significant subtitle On Interpreting Literature, Music, and the Visual Arts Ironically (Elleström 2002). Divine Madness discusses and compares different literary genres and art forms from different epochs, but it does not refer to the early interart and intermedial discussions that would later be central to Elleström’s work because they acted as stepping stones toward his own theorizations. In this book, though, there is the important element of comparative investigations, which are, from a certain point of view, the raison d’être of his subsequent intermedial investigations.

188

J. Bruhn and B. Schirrmacher

After his interdisciplinary study on irony, he began his theoretical engagement with intermedial questions. According to Elleström himself (in a short autobiographical Swedish text that reviews some of his work (Elleström 2021a), this began when he started working with what usually is referred to as concrete poetry. The literary genre of concrete poetry taught him an important lesson regarding investigating in depth the more general approaches to interart comparisons; instead, he approached iconicity as the underlying principle of this particular form of literature. At that point I did not know it, but in retrospect it was the publication of “Iconicity. Visual, auditive and cognitive signs in literature and other media” [Elleström, 2008], that introduced a focus in my research that later on has become totally dominant, and which is likely to occupy me as long as I will have the energy and passion to continue. (Elleström 2021a, 137 our translation)

This study of iconicity in visual poetry, he continues, led him to “construct a more and more general conceptual framework,” and he gradually “realized that I had a huge work waiting for me, where the project on poetry and intermediality was a small (if also very dear) side-track” (Elleström 2021a, 137 our translation). Elleström’s approach to intermedial studies ran parallel to what he had demonstrated was the case regarding the iconicity aspects of visual poetry, namely, that the underlying framework should be the semiotic and cognitive theory and philosophy of Charles Sanders Peirce. This suggestion implied a fruitful move away from the relative dominance of Saussurean semiology. Saussure’s linguistic theory had a major impact on the development of linguistics and semiotics during large parts of the twentieth century, but this language-based model creates difficulties when it comes to comparing language to the way images, sounds, and other forms of basic media types communicate. Elleström’s critique of theories that are solely based on language as the structuring principle was to become a leitmotif in his work. Elleström’s second major insight was of a more research strategic kind: when he reviewed the existing research, he found that many of the issues that interested him, for instance the interrelation between words and images, which he began untangling as semiotic relations and not just medium-specific affordances, were investigated either in interart/intermedial research or in multimodal studies – that is, not in one, combined effort. These fields, interart and intermediality on the one hand and multimodality on the other, were often inspired directly or indirectly by semiotics. Elleström’s major move in his conception of intermediality was to adopt insights from multimodality to develop an intermedial framework based on a “multimodal conception of mediality” (Johansson 2021a, 256) and on a Peircean semiotic framework that conceives signs not as static objects but as parts of relational processes. In Elleström’s article “The modalities of media: A model for understanding intermedial relations” (included in Elleström 2010a), these insights provide the grounds for the theoretical concerns he identifies as the main problems of intermedial studies around 2010, in particular concerning a definition of medium and the lack of a framework that addressed the multilayered complexity of mediation.

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

189

“The modalities of media” was followed by a steady flow of texts in which ideas from this article were developed, refined, revised, and brought to bear on new areas. Among the most important of these articles and books is a (translated and reworked) article on iconicity (“Iconicity as meaning miming meaning and meaning miming form” (Elleström 2010b) a treatise on media transformation (Elleström 2014), an investigation of truthfulness as a transmedial phenomenon that has its foundations in indexicality (Elleström 2018b), an investigation of narration in different media (Elleström 2019) and the work on a model of communication that is centered on media as “the intermediate stage in communication” (Elleström 2018a). The refinements developed in these works subsequently led to “The modalities of media II: An expanded model for understanding intermedial relations,” a substantially revised version of the 2010 article. This article will probably become the reference text when it comes to Elleström’s theoretical framework, not least because it was almost immediately translated into Portuguese and Chinese.1 In the extended and modified version of this article, the four modalities are the central parts of a “media-centred model of communication”: the balance has changed, and Elleström seems to have moved away from being mostly interested in defining media and focuses on intermedial relations and communication instead. Finally there are two very significant articles on symbolicity (Elleström 2022) and on translation (Elleström 2023a) that were published posthumously. In addition, Elleström edited and co-edited important volumes on intermedial questions: after Media Borders (Elleström 2010), he co-edited Iconic Investigations (Elleström et al. 2013), Transmediations (Salmose and Elleström 2020), and an ambitious two-volume work covering a range of intermedial and multimodal questions and interrelations, Beyond Media Borders: Intermedial Relations Among Multimodal Media, Volume 1 and 2 (Elleström 2021).

1 Among important Elleström translations is Chinese is 拉斯
埃斯特洛姆:《媒介的模态:跨媒介研 究理论与实践》, 陈军、欧荣译, 杭州:浙江大学出版社, 2023年。(Elleström 2023b), which includes four important articles: (1) “The modalities of media: a model for understanding intermedial relations,” (2) “The Modalities of media II: an expanded model for understanding intermedial relations,” (3) “Representing the anthropocene: transmediation of narratives and truthfulness from science to feature film,” and (4) “Bridging the gap between image and metaphor through cross-modal iconicity: an interdisciplinary model.” Elleström has also been translated into a Brazilian context, mainly with the following: Elleström, Lars. As modalidades das mídias II: um modelo expandido para compreender as relações intermidiais. Trans. supervised and revised by Elaine Barros Indrusiak. Technical revision by Ana Claudia M. Domingos and Camila A. P. de Figueiredo. Porto Alegre: Edipucrs, 2021, and the collected volume Elleström, Lars. Midialidade: ensaios sobre comunicação, semiótica e intermidialidade. Org. by Ana Munari, et al. Porto Alegre: EdiPUCRS, 2017, which include Elleström, Lars. Coeréncia e veracidade na comunicação: Indicialidade intracomunicacional e extracomunicacional. Trans: Marcelo Pires de Oliveira. Revista Famecos, 2019. Elleström, Lars. Identificando, construindo e transpondo as fronteiras das mídias. Trans. Camila Figueiredo. Proceedings of the X Seminário de Pesquisa II Encontro Internacional VII Jornada Intermídia, 2019. Jørgen Bruhn’s work, too has been translated to both Brazilian and Chinese, most recently, 乔根
布鲁恩:《叙事文学的跨媒介性:媒介至关重要》, 徐长生、欧荣译, 杭州:浙江大学出版社, 2022 年。(Bruhn 2022).

190

J. Bruhn and B. Schirrmacher

Elleström had been approached by Palgrave to be the main editor of this Palgrave Handbook of Intermediality, and he was responsible for planning it and for its structure, but after his death the editing was taken over by Jørgen Bruhn, Asun López-Varela, and Miriam Vieira. Apart from Elleström’s development of theoretical and philosophical concepts, he was also instrumental in producing solid and dynamic institutional infrastructures and in disseminating knowledge about intermedial research. Internationally, Elleström developed the Nordic Society for Intermedial Studies that had been started by Hans Lund and Ulla-Britt Lagerroth, to change it into an even more international network, The International Society for Intermedial Studies, which had an inaugural conference in Cluj, Romania, in 2013. The aim of the society is to promote intermedial research and postgraduate education by means of conferences, seminars, and projects, and anyone involved in intermedial research can become a member.2 Since 2013, international conferences under the auspices of the International Society for Intermedial Studies have been organized all over the world; the latest one was in Dublin in 2022. In a more local effort at Linnaeus University, Elleström, with a team of colleagues, managed to secure stable long-term funding for what was named in 2015 the Linnaeus University Centre for Intermedial and Multimodal Studies (IMS). The center receives its funding in 5-year cycles directly from Linnaeus University’s Vice Chancellor, and Elleström, with colleagues, has established the center as an international node that brings together intermedial and multimodal research and teaching.

A Media-Centered Approach to Communication A central aspect of Elleström’s intermedial approach to media theory lies in acknowledging and relating similarities and dissimilarities between media types and his theoretical framework must be seen as a response to several challenges of intermedial theory. Given the fact that the field of intermediality explores the relationship between different kinds of media, intermedial theories need to develop concepts that define how the differences and similarities between media can be approached. One way to tackle this challenge is to use a typology to categorize media interrelations into different types such as media combinations, transformations, and representations and to organize a hierarchy of different groups and subgroups. This typological differentiation, however, also involves the challenge of finding viable ways to map the innumerable relationships between different media and how to conceive of the boundaries between different media types and intermedial phenomena. A fundamental question that intermedial theory has to address concerns the boundaries of media and whether a concept of media borders is needed for describing intermedial relationships: Irina Rajewsky (2010) discusses and defends the concept of media 2

More information on https://intermedial-studies.com/ (accessed 25 Jan 2023).

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

191

borders as fundamental for the conception of intermedial relationships and phenomena in the important article “The problematic status of media borders in the current debate about intermediality.” She asks: if no borders are crossed, how can there be intermedial relationships? At the same time, the study of intermedial phenomena repeatedly draws attention to the fact that these lines and borders between different media types are difficult to draw, that the boundaries are not in any way clear cut, and that media borders tend to dissolve the closer you look at them. These observations relate to the overlapping of words, images, and sounds being different and yet intertwined forms of communication, which Hans Lund drew attention to in 2001. Words, images, and organized nonverbal sounds are clearly different forms of communication, yet at the same time, it can be difficult to draw a clear line between text and image in a comic panel, or between the prosody of spoken words and the melody of a song. Thus, intermedial phenomena tend to draw attention to conventional media borders and at the same time transgress and dissolve the notion that such borders exist at all. As a consequence, W. J. T. Mitchell has stated provocatively that “there are no visual media” because we interact with all media with all our senses (2005). But if we accept, like Mitchell, that all media are mixed media, how can we describe, analyze, and compare the different ways in which different media mix with each other? Thus, while conceptual distinctions are necessary in intermedial analysis, they are difficult to apply in a way that can describe the complexity of intermedial communication. This is the problem that Elleström addresses in “The modalities of media” (2010), but from a new perspective. Seeing intermediality as a “bridge between medial differences that is founded on medial similarities” (2010, 12), he develops a bottom-up approach that addresses the variety and complexity of intermedial relations by defining the fundamental characteristics that all media types and products share. He initially points out that “intermediality has tended to be discussed without clarification of what a medium actually is” (2010, 11). This was a bit of an exaggeration, of course, given that influential definitions of medium as distinct means of communication specified by the use of specific channels and semiotic systems could be found, for instance, in the work of Werner Wolf (1999, 35–36), or the differentiation of media as technologies, discourses, and institutions that is applied in the field of media and communication studies (see, for instance, Jensen 2010, 41). However, what Elleström wanted to point out was the difficulty in providing an encompassing definition that could grasp the confusing complexity of the communicative process that includes objects, sign systems, and institutions and that creates both social infrastructures and discourses. Therefore, all of the following, in some situations, could be called media: devices such as computers and smartphones, cameras, and projectors; interfaces such as screens; products such as a specific film; genres and art forms such as cinema; and institutional production systems such as those used in Hollywood. In addition, when we talk about, for instance, literature, cinema, or journalism, we often do so by referring to certain materials (books, films, newspapers) but we actually think of them as specific forms of communication that are shaped by cultural conventions in a specific sociohistorical

192

J. Bruhn and B. Schirrmacher

context. This is not a problem in everyday speech or within a particular media context, but it becomes problematic in approaches to intermedial relationships and when it comes to complex contemporary digital media convergences. The difficulties of grasping the complexity and multilayeredness of mediation, Elleström points out, often lead to skewed comparisons and definitions, for instance, between the verbal and the visual, and when “conceptual units” such as literature and visual art are compared, the “materiality of media is generally not distinguished from the perception of media” (2010, 15). Elleström stresses that “media are both similar and different” (Ibid.), and therefore he proposes a model that can highlight both “crucial divergences and fundamental parallels between all sorts and variants of media forms” (16). Elleström chooses to center his discussion on what he calls “media products,” which refer to specific communicative objects. Instead of distinguishing between different categories, which could provide an overview of the almost endless variety of countless objects, Elleström starts by taking a closer look at the concrete objects, novels and news articles, performances and social media postings, games and films, photographs and exhibitions that we use when communicating information and experiences, asking how we approach them in communicative situations, what they all have in common, although in different ways. Our understanding of media, Elleström stresses, derives from our interaction with “an endless row of forms of physical objects, phenomena and actions” that we use or perceive as media products, ranging from forms such as “written texts, songs, scientific diagrams” to “meals, ceremonies, hairstyles” (Elleström 2021, 14). A media product, according to Elleström, is not an object as such but is a “function rather than an essential property” of an object when we see an object as a potentially meaningful and communicative entity. Our understanding of different conventional media types, such as literature, theater, news, or conversations, is based on our interaction with and experience of almost countless individual media products, such as specific printed editions (or audio files, or e-books) of specific novels, specific theater performances, newspapers we have read, TV or radio news bulletins, and countless instances of taking turns to produce utterances in face-to-face communication. Individual media products all involve the material, perceptual, semiotic, and conventional levels of mediation. We interact with (a) material objects that provide a physical interface and (b) specific spatial and temporal qualities that (c) we perceive with our senses, and that we (d) interpret as a mixture of iconic, indexical, or symbolical signs. This is what Elleström identifies as media modalities, which he defines as the “necessary categories of media traits” (2021, 47). The material modality focuses on the material objects and physical phenomena and how they function as interfaces of communication. The spatiotemporal modality concerns the spatial and temporal dimensions of physical objects and phenomena, although we tend to engage with some media types as primarily spatial objects and with others as temporal events. The sensorial modality focuses on the sensual perception of the material interfaces and their spatial and temporal dimensions. These material, spatiotemporal and sensorial qualities of media products provide what Elleström calls the “presemiotic” (2021, 47) data that we perceive and interpret

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

193

in the semiotic modality as signs. Using C. S. Peirce’s well-known terms, Elleström differentiates between three sign relationships that are based on similarity (icons), contiguity (indices), or convention and habits (symbols). All three relationships are present in all kinds of signs, even if one of them is more prominent and the other two are less so, depending on the specific context. With the help of the media modalities, specific media products can be described, analyzed, and compared with reference to the specific set of modes within these modalities that qualify a media product in a specific social and historical context. This distinction also allows a conception of the relationship between individual media products and media types that can confirm, challenge, and expand our conventional understanding of their media type. This is what happens when audiobooks expand received notions of literature. Literature is now a media type that is no longer primarily disseminated via printed books. Furthermore, the four modalities provide the possibility of conceiving the different conceptual aspects of media products. As we engage with objects, sign systems, and institutional conventions at the same time, the material objects such as folded sheets of paper, books, or smartphones provide the material surfaces for texts and images, and these basic media types can be organized according to different contextual and operational qualifying aspects such as news, graphic novels, memes, or family albums. Elleström refers to the interfaces of material objects as technical devices of display that provide access to basic media types that are defined by their modality modes. In different media contexts these basic media types are organized according to different contextual and operational qualifying aspects. These categories make it possible to describe how technical devices of display such as folded sheets of paper, books, or smartphones provide surfaces for texts and images, basic media types that can be organized according to different contextual and operational qualifying aspects such as news, graphic novels, memes, or family albums. Thus, the modalities and the media types enable a fine-grained description of any specific media product in any specific media type, as exemplified in much work inspired by Elleström. They also enable comparisons between the media types and modes of, for instance, film, literature, music, computer games, and news. The framework can, for example, be used to describe how the modes of an artistic performance can deliberately push and challenge conventional expectations of a qualified media type such as theater. It also allows more detailed descriptions of how digital technology transforms our understanding of all kinds of news, literature, film, photography, and games, or one can tease out the similarities and differences between analogue and digital basic media types, to name just some examples. In sum, via the framework of the media modalities it becomes possible not only to agree with Mitchell about the mixedness of all media but to conduct an analysis demonstrating how all media are mixed and how the material, perceptional, and semiotic characteristics interact with conventions to shape the very communication they facilitate. Regarding the intermedial relationships between combinations, transmediations, and representations, Elleström’s media modalities provide the ground for a more flexible understanding of intermedial relationships. For instance, previous

194

J. Bruhn and B. Schirrmacher

theoretical approaches conceptually differentiated between combination of independent media types and integrations of media where the individual parts cannot be understood separately (see, for instance, Clüver 2007, 26; Lund 2002, 21; Rajewsky 2002, 19). Elleström avoids these strict divisions and emphasizes “that there is a floating scale between combination and integration” (2021, 76). And approaching media products via the four modalities highlights that combination and integration can even be seen as two different perspectives on the same media product, whether you focus on combination in the production process or on the deep integration of the perception process (Bruhn and Schirrmacher 2022, 103). For a long time, transmediation (such as adaptations of a narrative) of media representation (for instance, in film or literature) and formal imitation of media types (for instance, in cinematic writing) have been studied separately. Elleström’s concept of media transformation (Elleström 2014) stresses instead the interdependency between transmediation and representation of media characteristics; neither can take place without the other. For instance, an ekphrastic poem verbally transmediates the depicted scene and represents aspects of the basic and qualified media types of visual art. Of course, novel-to-film adaptations, ekphrastic poems, and cinematic writing are very different forms of media transformations, but in all cases transmediation includes the representation of the modes and materials of previous mediations, and the extensive representation of media types transmediates a certain constellation of content and form.

Developing, Adapting, and Connecting the Framework One of the strengths of Elleström’s work has been the very general character of his theory and terminology, and he aimed to make the ideas generalizable and “to offer a broadly applicable, well-developed and distinct but flexible theoretical framework” (2021, vol 2, 214). His goal was to cover all media types, or all areas of human communication. However, while the model is meant to “encourage methodical analysis of media and their interrelations,” it does not imply a specific analytical methodology of its own. Instead, these methods “must ultimately be formed based on one’s specific aims and goal” (2021, vol 2, 214). Thus, intermedial researchers at IMS as well as researchers around the world have developed different ways to employ these categories in intermedial analysis in different contexts. The media modalities and media types have been used to analyze many different things, for instance, the relationship between the text and the image(s) in photo poetry (Almgren White 2011), ekphrastic poetry (Tornborg 2014), the transmedial elements in musicalized fiction (Schirrmacher 2012), in sound art (Vandsø 2014), drama education (Beswick 2016), visual art problematics (Petersson et al. 2018), theatrical performance (for instance, Lavender 2014; Crossley 2019), headphone theatre (Klich 2017) the modality modes of architecture (de Vieira 2021), audio and audiovisual podcasts (Johansson 2021a), and the relationship between a musical score and a musical performance (Caers 2022). In the two-volume anthology Beyond Media Borders (2021), a wide range of scholars from different disciplines apply the

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

195

model and combine it with different theoretical and disciplinary approaches to explore, for instance, the intermediality of digital texts, multimodal aspects of performance, the concept of reading and how it relates to audiobooks, verbal and graphical representations of landscapes in maps, and the representation of climate change in different media, among others things. Based on the media types and modalities, Bruhn (2016) and Bruhn and Gjelsvik (2018) have developed an intermedial methodology to analyze the important role of media representation in narrative literature and in film, and Christer Johansson integrated Elleström’s media modalities with media archaeology and communication theory to analyze and compare audio and video podcasts (2001a) and to conduct an in-depth analysis of the work of the Swedish modernist writer Eyvind Johnson (Johansson 2021b). The concept of media transformation has been used to examine how journalism is depicted in graphic narration (Domingos and Rodrigues Cardoso 2021) and how trials are transmediated in journalistic court reports (Löwe and Schirrmacher 2021). The applicability of intermedial concepts and Elleström’s framework in particular to a broad variety of media types and contexts has been developed further in publications that followed IMS conferences, such as Transmediations (Salmose and Elleström 2020), Multispecies Storytelling in Intermedial Practices (Bencke and Bruhn 2022), and Truth Claims Across Media (Schirrmacher and Mousavi, Forthcoming). The most comprehensive effort to date to develop analytical methods and to apply the Elleström model to different media types is Intermedial Studies. An Introduction to Meaning across Media (Bruhn and Schirrmacher 2022). The volume presents Elleström’s media modalities and media types as a set of tools that can be used to critically analyze the different layers of mediation in specific media products. The volume is a group effort of about 20 writers, most of them closely connected to IMS, and has three aims. There are chapters that map the media and modes of a range of qualified media types, and demonstrate how the modalities can be used to discover and describe similarities and dissimilarities between, say film and literature. Other chapters present multistep analytical models for intermedial analysis of combinations and media transformations (with a focus on transmediation or media representation, respectively). Finally, a range of chapters explores how the intermedial perspective can expand to explore vlogs and GIFs in social media, different forms of intermedial performance, highlight the spatiotemporal aspects of performance, and discuss truthfulness as an intermedial phenomenon. As Elleström’s intermedial framework is based on a multimodal conception of communication, it plays an important role in bringing together intermediality and multimodality as two theoretical perspectives that have approached the mixedness of communication from different directions. It is important to stress that Elleström’s media modalities are not compatible with what is called a mode, or a semiotic resource in multimodality (see, for instance, Jewitt et al. 2016, 5). Yet Elleström’s framework can be understood as providing the nodes that are needed to connect the different perspectives and facilitate collaborations between intermedial and multimodal research: ideally, this would facilitate new ways of conceptualizing the integration of modes and the transformation across media types as complementary

196

J. Bruhn and B. Schirrmacher

approaches to the mixedness of human communication (Jensen and Schirrmacher 2022). Another important aspect in Elleström’s recent texts is the notion of truthfulness in communication (2018b): this is examined by exploring how media affect communicative strategies for conveying reliable information and creating credibility, for instance, in artistic representations of climate change and or mis- and disinformation in journalism (as discussed in Bruhn et al. 2022a and in Bruhn et al. 2022b). The IMS-conference “Trust Me! Truthfulness and Truth Claims Across Media” addressed current social challenges such as fake news and disinformation and the troubled communication forms in times of a climate- or health-related crises. A forthcoming volume explores truth claims of media further and highlights how the perception of truthfulness is based on an interplay between facts, patterns of coherence, and different modes of engagement that connects truth claims and truthfulness with concepts such as testimony and authenticity (Schirrmacher and Mousavi, Forthcoming). During the first few years after the center was established, a handful of relatively general research topics guided the work of the Växjö school, including themes like meaning-making, interactivity, narrativity, and learning. Elleström’s approach to truthfulness in communication (2018b) inspired the development of truthfulness as a research topic at IMS, and researchers looked into how integration and transformations of media relate to reliability, credibility, and authenticity in communication (Bruhn et al. 2022b, Schirrmacher and Mousavi, Forthcoming). In recent years, there has been a tendency for research clusters to orientate themselves toward more specific topics that explore the relation of modes and media in the context of contemporary social challenges, and at the time of writing there are four such active clusters in Växjö. The first cluster to materialize was IMS Green: Mediations of Climate and Ecological Emergency. The research cluster asks what is meant by terms such as “climate,” “ecology,” and “emergency” and what happens when they are connected with intermedial and multimodal theories and methods. The ecological emergency is seen as a mediated phenomenon occurring in many different media types, inside and outside the arts, in fiction and non-fiction forms, which needs to be critically investigated, analyzed, and compared. The climate and ecological crisis is rendered sensorial, intelligible, and imaginable through complex processes of mediation that operate in and between diverse media types. IMS Memory: Mediating Cultural Memory studies the mediation of cultural memory and the circulation of mediated memories from a broad intermedial and multimodal perspective. The cluster examines how memory is remediated, embodied, and performed in a wide context of media relations involving the production, distribution, exhibition, and reception of mediated memories in, for instance, spoken word poetry, graphic novels, memoirs, and other autobiographical narratives, archives, databases, metadata, interfaces, and documentary filmmaking. IMS News: Intermediality, Multimodality, and News explores the role of professional journalism and news in a digitized society, asking how modes and communication in a hybrid media environment affects the production, perception, and

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

197

dissemination of news. The cluster approaches the mix of media and modes in news by using intermedial, multimodal, and communication theory. Projects explore, for instance, how young citizens perceive news, how transmedial workflows define newsroom strategies, the narrative patterns of journalism, and the importance of body language and music in audiovisual journalism. IMS Literacy: Intermediality, Multimodality, and Learning – the most recent cluster in IMS – develops intermedial and multimodal responses to current discussions of media and digital literacy as a new form of media literacy. The group is investigating how intermedial and multimodal knowledge can be used for creative, inclusive, and flexible teaching and for information-related and creative activities. Intermediality and multimodality are explored as didactic tools for teaching language and literature in a mediatized society.

Conclusion: Current and Future Developments Generally speaking, one important question facing IMS is how the intermedial approach to media can be connected with other theoretical understandings of media to contribute to a more holistic analysis of media and communication. Elleström’s model differentiates between conceptual aspects, technical devices, and basic and qualified media types, which facilitates consistent intermedial comparisons. While Elleström’s framework has proven useful in specifying and sometimes suggesting solutions to theoretical challenges in intermedial studies – and has been a dynamic catalyst for research in and around IMS – the multimodal conception of media and a media-centered model of communication offer potential, which involves new challenges or calls for renewed work on unsolved questions. As indicated above, an intermedial perspective on media combinations and integrations could be fleshed out with a more fine-grained analysis demonstrating how multimodal semiotic networks spread across basic media types, for instance, to pinpoint how image-based modes of gestures are supported by musical modes such as rhythm or pitch (Jensen and Schirrmacher 2022). The newly founded graduate school, MIDWorld (Multimodality and Intermediality: Humanist Research in a Digital World), constitute one step toward adopting this more holistic media approach. The modalities of media and the media-centered model of communication can be used to zoom in on different aspects that are in focus in other approaches to media, including media archaeology, media and communication studies, media production, and journalism studies. The relational character of Elleström’s framework provides the nodes that are needed to connect with other forms of medial analysis, as Christer Johansson’s approach to media analysis demonstrates (Johansson 2021a, b). The usefulness of the media modalities may also improve interartistic collaboration (Caers 2022). In a similar way, the framework of the media modalities can be used to improve journalist practice in general, and can help practitioners and producers working in all forms of media production to assess the consequences of technological digital development for their productions. The media modalities is a

198

J. Bruhn and B. Schirrmacher

flexible tool to address the quickly changing, converging, and transmedial characteristics of the digital infosphere (Schirrmacher and Holt 2022). One major challenge is how to connect the specific definitions of a media product with the understanding of media as integrated parts of institutions and infrastructures, that is, how to connect the media-centered model of communication with the role of media in society. Using the recent interest in medial infrastructures as inspiration, it would be interesting to ask in which ways the global trading networks, or the electrical grid or indeed the World Wide Web should be understood as the necessary precondition for the circulation of media and information. Infrastructural studies clearly are inspired by Marshall McLuhan’s media theory, yet remain unconnected with the semiotic and terminological logic of Elleström’s position. Future studies could discuss the interrelations between the material and technical preconditions of communication – the communicative forms themselves – and the social conditions of production, dissemination and reception. Elleström’s model differentiates between technical devices and basic media types that are used and formed according to the qualifying aspects. Although the three dimensions may be as clear as they can possibly be in theory, such notions almost inevitably create borderline troubles. For instance, saying that a technical device has a material interface is a useful distinction, but exactly how does it connect to the rest of the material characteristics of a media product? The screen, loudspeakers, and keyboard may constitute the interfaces and thus the technical media of display of a computer, but what about the hardware and software? What is the position of the binary codes, or the algorithms? How can Elleström’s concept of the media product, designed to solve problems in intermedial analysis, be connected with more general understandings of media as interfaces and infrastructures and to the whole network of media production, storage, and display? The term “interface” has been theorized as a consequence of the occurrence of digital media (Andersen and Pold 2011; Galloway 2012). An example of an interface could be the small screen through which we interact with and navigate apps and other services on our mobile phone. Computers and smartphones provide access to different media products, news, films, photographs, social media posts, art, etc. As a technical device of display, the screen (together with the loudspeakers and headphones) is the interface for several basic media types such as digital texts, images, gestures, auditory text, speech, and sounds that integrate differently depending on the qualified media type that is being used, such as news, blogs, podcasts, memes, or emojis. But how can we approach the interactivity of the digital screen, which not only gives us access to media products but also allows us to manipulate them? Considered as a technical device of display, the interface of a screen shares some characteristics with the pages of books, but they are dynamic rather than stable and thus share some of their qualities with performative spaces. And can the malleability of digital media types be found in the increased possibility of interactivity being a feature of the technical media of display (created by engineers or programmers), or is it a cultural feature that relates to some of the aspects of qualified and contextual media?

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

199

From the perspective of cultural studies, the focus on analyzing a media product in relation to the modalities is questionable, since it does not take into sufficient consideration the social and historical situatedness of all forms of communication. The media-centered model of communication and a media analysis based on the modalities alone appears to neglect how all media are produced and used in social interaction as well as the important question of the unequal local and global access to media (Mousavi 2021). A specific space is reserved in Elleström’s systematization for social and cultural context, for instance under the heading of qualifying and contextual aspects, but these are quite summarily described. As Elleström mentioned concerning his communication model: In addition to its innate basic capacity to perceive and interpret mediated qualities, the mind is inclined to form cognitive import on the basis of acquired knowledge, experiences, beliefs, expectations, preferences, and values – preconceptions that are largely shaped by culture, society, geography, and history. It is clear that this concept is immensely important for the outcome of communication. [. . .] As this is a widely recognized fact that has been extensively theorized in various ways, I have bracketed it in my outline of the new model. (Elleström 2018a, 290)

Related to the question of the somewhat undertheorized social contexts, Heidrun Führer has argued that a fifth, “social” modality should be added in order to integrate sociohistorical elements into Elleström’s model (in Bäckström et al. 2022, 214–221). Alternatively, but in the same vein, the situatedness of communication could be spelled out by stressing historical, aesthetic, sociopolitical, and cultural qualifying aspects. A description of how the basic media types in a specific media product are formed by the qualifying aspects makes it possible to highlight the social, political, and economical aspects that until now often remain undeveloped. Another topic that could be developed further is the relation between the four modalities. The material, the sensorial, and the spatiotemporal modalities are defined by Elleström as “presemiotic” modalities that are the foundation upon which the semiotic modality can operate. The term presemiotic seems to suggest a temporal or hierarchical sequence in perception. The choice of prefix indicates an implicit hierarchy, or at least a temporal sequence (roughly speaking, we perceive three modalities first and then we make meaning out of them), which does not cover the complex interrelation between perception and meaning-making in human communication. A more refined description of the relation between the modalities could describe the four modalities as nonsequential, simultaneous. The Elleström model could not only be used in analyses to relate similarities between dissimilar media types, but also to flesh out how different processes take place simultaneously. When using the model in analysis it becomes clear that the relations and possible intersections between the modalities can be developed. It could be argued that the category of the spatiotemporal modality needs to be developed. In Elleström’s writings it is not clear whether the spatiotemporal modality is based on Kant’s notion of the a priori nature of time and space for human perception or whether the inspiration is less philosophical and is based more on the natural sciences. It

200

J. Bruhn and B. Schirrmacher

seems relevant to ask whether, and if so how, time and space should be considered universal categories, or whether conceptions and perceptions of time and space are historical, as suggested by the Neo-Kantian philosopher Ernst Cassirer and the cultural historian M. M. Bakhtin. A related question is whether the sensorial modality is a universal category or not: it is well known, as pointed out by Elleström himself, that not all human beings necessarily have identical setups for their senses (for example, some people are color blind and some have a hearing impairment), but does this affect the idea of the sensorial modality? And what about medieval ideas regarding a “sense of the heart?” At the moment of Lars Elleström’s untimely death in December 2021, the international influence of his work and the research environment he had inspired had been steadily increasing. Elleström had quite detailed plans to write an ambitious work that aimed to collect and summarize his key ideas and insights into a kind of systematic Summa. Unfortunately, he never had the chance to do so, but his work will probably continue to stimulate ideas and critical responses in the global intermedial research environments, not least because some of his central texts have been translated from English into other languages and are used in academic environments where his ideas are discussed and are influencing a new generation of scholars. It is left to others, inside and outside IMS, to continue his work by discussing it, criticizing it, developing it, and supplementing it, which is exactly what Lars Elleström would have wanted.

References Almgren White, Anette. 2011. Intermedial narration i den fotolyriska bilderboken: Jean Claude Arnault, Katarina Frostenson och Rut Hillarp, 2011. Växjö: Linnaeus University Press. Andersen, Christian U., and Søren Pold. 2011. Interface criticism: Aesthetics beyond buttons. Aarhus: Aarhus University Press. Bäckström, Per, Heidrun Führer, and Beate Schirrmacher. 2022. The intermediality of performance. In Intermedial studies. An introduction to meaning across media, ed. Jørgen Bruhn and Beate Schirrmacher, 198–225. London: Palgrave. Bencke, Ida, and Jørgen Bruhn. 2022. Multispecies storytelling in intermedial practices. Brooklyn: Punctum Books. Beswick, Katie. 2016. Ten in a bed: Literacy, intermediality and the potentials of low-tech. Research in Drama Education 21(3): 337–348. Bruhn, Jørgen. 2016. The intermediality of narrative literature: Medialities matter. London: Palgrave Macmillan. ———. 2022. The intermediality of narrative literature: Medialities matter. Trans. Changsheng Xu, and Rong Ou. Hangzhou: Zhejiang University Press. Bruhn, Jørgen, and Anne Gjelsvik. 2018. Cinema between media: An intermediality approach. Edinburgh: Edinburgh University Press. Bruhn, Jørgen, and Beate Schirrmacher. 2022. Intermedial studies: An introduction to meaning across media. London: Routledge. Bruhn, Jørgen, Niklas Salmose, Beate Schirrmacher, and Emma Tornborg. 2022a. Truthfulness as transmedial phenomenon. In Intermedial studies: Communicating meaning across media, ed. Jørgen Bruhn and Beate Schirrmacher, 225–255. London: Routledge.

9

Linnaeus University Center for Intermedial and Multimodal Studies and. . .

201

Bruhn, Jørgen, Niklas Salmose, Beate Schirrmacher, and Emma Tornborg. 2022b. Truthfulness as transmedial phenomenon. In Intermedial studies: Communicating meaning across media, ed. Jørgen Bruhn and Beate Schirrmacher, 225–254. London: Routledge. Caers, Vincent. 2022. Umbraphonics. An electro-acoustic performance format for mediating musical scores. Leuwen: KU Leuwen, Luca School of Arts. Clüver, Claus. 2007. Intermediality and interarts studies. In Changing borders: Contemporary positions in intermediality, ed. Jens Arvidson, Mikael Askande, Jørgen Bruhn, and Heidrun Führer, 19–38. Lund: Intermedia Studies Press. Crossley, Mark. 2019. Intermedial theatre: Principles and practice. London: Macmillan Education. Domingos, Ana M., and J. Rodrigues Cardoso. 2021. Media representation and transmediation: Indexicality in journalism comics and biography comics. In Beyond media borders, ed. Lars Elleström, vol. 2, 79–117. London: Palgrave Macmillan. Elleström, Lars. 1999. Lyrikanalys: En introduktion. Lund: Studentlitteratur. ———. 2002. Divine madness: On interpreting literature, music, and the visual arts ironically. Lewisburg: Bucknell University Press. ———. 2010a. Media borders, multimodality and intermediality. Basingstoke: Palgrave Macmillan. ———. 2010b. Iconicity as meaning miming meaning and meaning miming form. In Signergy, ed. C.J. Conradie. Amsterdam: John Benjamins. ———. 2014. Media transformation: The transfer of media characteristics among media. Houndsmill/Basingstoke/Hampshire/New York: Palgrave Macmillan. ———. 2018a. A medium-centered model of communication. Semiotica 224: 269–293. ———. 2018b. Coherence and truthfulness in communication: Intracommunicational and extracommunicational indexicality. Semiotica 225: 423–446. ———. 2019. Transmedial narration: Narratives and stories in different media. Cham: Springer. ———. 2021. Beyond media borders. Intermedial relations among multimodal media. Vol. 1–2. Cham: Springer. ———. 2021a. Ikonicitet [TfL 2008:1]. Med författarens kommentar, tretton år senare, och inledning av Jørgen Bruhn i Tidsskrift för litteraturvetenskap 51: 118–140. ———. 2022. Symbolicity, language, and mediality. Semiotica 247: 1–32. ———. 2023a. Intermedial approaches. In The Routledge handbook of translation theory and concepts, ed. Reine Meylaerts and Kobus Marais, 389–409. New York: Routledge. ———. 2023b. The modalities of media: Theory and criticism of intermediality. Trans. Jun Chen, and Rong Ou. Hangzhou: Zhejiang University Press. Elleström, Lars, Olga Fischer, and Christina Ljungberg. 2013. Iconic investigations. Amsterdam: John Benjamins. Galloway, Alexander R. 2012. The interface effect. Cambridge, UK: Polity Press. Hedling, Erik, Hans Lund, and Ulla-Britta Lagerroth. 1997. Interart poetics: Essays on the interrelations of the arts and media. Amsterdam: Rodopi. Jensen, Klaus Bruhn. 2010. Media convergence: The three degrees of network, mass, and interpersonal communication. London/New York: Routledge. Jensen, Signe Kjær, and Beate Schirrmacher, 2022. Stronger together: The added value in combining intermedial and multimodal approaches. In Presentation at the 6th International Society for Intermedial Studies Conference. In between and across: New directions, mappings and contact zones, Dublin, 1–3 Sept 2022. Jewitt, Carey, Josephus J. Bezemer, and Kay L. O’Halloran. 2016. Introducing multimodality. London/New York: Routledge. Johansson, Christer. 2021a. YouTube podcasting, the new orality, and diversity of thought: Intermediality, media history, and communication theory as methodological approaches. In Digital human sciences: New objects – New approaches, ed. Sonya Petersson, 253–284. Stockholm: Stockholm University Press. https://doi.org/10.16993/bbk.k. ———. 2021b. Det materialiserade ordet: Medium och mening i Eyvind Johnsons författarskap. Stockholm: Makadam. http://su.diva-portal.org/smash/get/diva2:1607442/FULLTEXT01.pdf

202

J. Bruhn and B. Schirrmacher

Klich, Rosemary. 2017. Amplifying sensory spaces: The in- and out-puts of headphone theatre. Contemporary Theatre Review 27(3): 366–378. Lagerroth, Ulla-Britta. 1993. I musernas tjänst: Studier i konstarternas interrelationer. Stockholm: Symposion. Lavender, Andy. 2014. Modal transpositions toward theatres of encounter, or, in praise of ‘Media Intermultimodality.’ Theatre Journal 66.4: 499–518. Löwe, Corina, and Beate Schirrmacher. 2021. How to tell the story of a trial. Media transformation in court reporting. In Narratives in the criminal process, ed. Frode Hel-mich Pedersen, Espen Ingebrigtsen, and Werner Gephart, 277–297. Frankfurt am Main: Vittorio Klostermann. Lund, Hans (ed.). 2002. Intermedialitet: ord, bild och ton i samspel. Lund: Studentlitteratur. Mitchell, W.J.T. 2005. There are no visual media. Journal of Visual Culture 4: 257–266. Mousavi, Nafiseh. 2021. Your language escapes me! In Translation and the global city: Bridges and gateways, ed. Judith Woodsworth, 206–221. New York: Routledge. Petersson, Sonya, Christer Johansson, Magdalena Holdar, and Sara Callahan. 2018. The power of the in-between: Intermediality as a tool for aesthetic analysis and critical reflection. Stockholm: Stockholm University Press. Rajewsky, Irina O. 2002. Intermedialität. Francke: Tübingen. ———. 2010. Border talks. The problematic status of media Borders in the current debate about intermediality. In Media borders, multimodality and intermediality, ed. Lars Elleström, 51–68. Basingstoke: Palgrave Macmillan. Salmose, Niklas, and Lars Elleström. 2020. Transmediations: Communication across media borders. New York: Routledge. Schirrmacher, Beate. 2012. Musik in der Prosa von Günter Grass: Intermediale Bezüge – Transmediale Perspektiven. Stockholm: Acta Universitatis Stockholmiensis. Schirrmacher, Beate, and Kristoffer Holt. 2022. Alternative media, news and intermediality. Distinguishing mainstream from alternative news – Towards an intermedial approach. In Presentation at the 6th International Society for Intermedial Studies Conference. In between and across: New directions, mappings and contact zones, Dublin, 1–3 Sept 2022. Schirrmacher, Beate, and Nafiseh Mousavi. Forthcoming. Introduction: The dynamics of truthfulness and media. In Truth claims across media, eds. Beate Schirrmacher, and Nafiseh Mousavi. London: Palgrave. Tornborg, Emma. 2014. What literature can make us see: Poetry, intermediality, mental imagery. Malmö: Bokbox. Vandsø, Anette. 2014. Rheo: Japanese sound art interrogating digital mediality. Acta Universitatis Sapientiae. Film and Media Studies 9: 141–154. Vieira, Miriam. 2021. Building bridges: The modes of architecture Beyond media borders. Intermedial relations among multimodal media. Vol. 2., ed. Lars Elleström, 59–79. London: Palgrave. Wolf, Werner. 1999. The musicalization of fiction: A study in the theory and history of intermediality. Amsterdam: Rodopi.

Intermediality in Brazil: A Diachronic Survey

10

Camila Augusta Pires de Figueiredo, Miriam de Paiva Vieira , Ana Cla´udia Munari Domingos, and E´rika Viviane Costa Vieira

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Brief History of Intermediality Research in Brazil . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Brief Selection of Publications by the Cited Research Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

204 205 215 216 216 221 224

Abstract

Intermídia, a term coined in the 1960s by Dick Higgins, can be found in an article by Claus Clüver in Brazilian publications as early as 1997 (Clüver C, Literatura e Sociedade: Revista de teoria e literatura comparada (DTLLC/FFLCH, Universidade de São Paulo) 2: 37–55, 1997). The notion of intermidialidade was widely discussed in the early 2000s in courses also taught by Clüver, who has often been invited as guest professor at Brazilian Universities (Universidade Federal de Minas Gerais (2002, 2004, 2007, 2008, 2017) and Universidade Federal de Alagoas (2002)). Since then, a great deal has been published in Brazil C. A. P. de Figueiredo (*) Universidade Federal de Minas Gerais, Belo Horizonte, Brazil e-mail: camilafi[email protected] M. de Paiva Vieira (*) Universidade Federal de São João del Rei, São João del-Rei, Brazil e-mail: [email protected] A. C. Munari Domingos Universidade de Santa Cruz do Sul, Santa Cruz do Sul, Brazil e-mail: [email protected] É. V. C. Vieira Universidade Federal dos Vales do Jequitinhonha e Mucuri, Diamantina, Brazil e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_8

203

204

C. A. P. de Figueiredo et al.

based on the translated works of intermedial specialists, namely, Clüver himself, Bolter, Bruhn, Grusin, Elleström, Kattenbelt, Lund, Moser, Müller, Wolf, Rajewsky, Schröter, to name a few. A lot more has also been published within the production of research groups officially registered in the national directory (Diretório dos Grupos de Pesquisa no Brasil, Conselho Nacional de Desenvolvimento Científico e Tecnológico, CNPq: http://lattes.cnpq.br/web/ dgp/home), such as Núcleo de Pesquisa em Conexões Intermidiáticas (NucCon), Grupo de Pesquisa Hipermídia e Linguagem (UFSC), Intermídia: estudos sobre a intermidialidade (UFMG), among others. The aim of this Margolin’s is to historicize intermedial studies diachronically in Brazil. In order to do so, we will rely on the production registered with the national official curriculum online platform Lattes (Plataforma Lattes do Conselho Nacional de Desenvolvimento Científico e Tecnológico, CNPq: http://lattes.cnpq.br/) by members of research groups which have been officially registered in the national directory entitled Diretório dos Grupos de Pesquisa no Brasil, by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). Keywords

Intermediality · Brazil · Research groups · Publications · Intermedial studies

Introduction Intermídia, a term coined in the 1960s by Dick Higgins, can be found in Brazilian publications as early as 1997, in the article “Estudos interartes: conceitos, termos, objetivos” by Claus Clüver (1997). His contribution to spreading intermedial studies in Brazil is unquestionable. The notion of intermidialidade was widely discussed in the early 2000s in courses in which Clüver was a guest professor at Universidade Federal de Minas Gerais (2002, 2004, 2007, 2008, 2017) and Universidade Federal de Alagoas (2002).1 The concept of intermediality is here understood as both a phenomenon that occurs across media and a category of critical analysis. Thus, Brazilian scholars – in the areas of Comparative Literature, Interart Studies, Journalism, Visual Arts, Communication, Dance, Design, among other related fields – study contemporary media products, including adaptations, ekphrases, illustrations, performances, installations, among other intermedial phenomena. This chapter aims to historicize intermedial studies diachronically in Brazil. Hence, we will rely on the production registered with the national official curriculum online platform Lattes2 by members of research groups which have been officially registered in the national

1

It should be noted that he had taught courses on semiotics and concrete poetry at Universidade de São Paulo in the 1980s and 1990s. 2 Plataforma Lattes do Conselho Nacional de Desenvolvimento Científico e Tecnológico, CNPq: http://lattes.cnpq.br/

10

Intermediality in Brazil: A Diachronic Survey

205

directory entitled Diretório dos Grupos de Pesquisa no Brasil, by the Conselho Nacional de Desenvolvimento Científico e Tecnológico3 (CNPq). Therefore, based on this proposal, we chronologically cataloged the research groups whose central object of investigation is intermediality. Among those cataloged, the following will receive attention in this chapter: Núcleo de Estudos e Produção Hipermídia Aplicados ao Jornalismo, Caligrafias e escrituras: Diálogo e intertexto, Intermídia, GPMLA – Grupo de Pesquisas em Mídia, Literatura e Outras Artes, Núcleo de Pesquisa em Conexões Intermidiáticas, ENTELAS: Grupo de Pesquisa em Conteúdos Transmídia, Convergência de Cultura de Telas, SEMIC – Semióticas Contemporâneas, Dança e Intermidialidade e Design e Intermidialidade.4 For reasons elicited in the following section, we included two groups that were not registered in CNPq: the Iconicity Research Group and Intermidialidade: Literatura, Artes e Mídias (Anpoll). In addition to a small history of their genesis, we present these groups’ research objectives, their main actions and publications, and main references, as well as the researchers who conceived and currently lead the groups. Besides pointing to the variability and versatility of research in intermediality in Brazil, we hope that this survey will be able to account for the dimension of interest this field of research has reached in the country and in its statement and continuity. For a selection of the research group members’ production, please see Appendix I.

A Brief History of Intermediality Research in Brazil In 2000, during her doctoral mobility in Singapore, researcher Raquel Ritter Longhi5 had access to Higgins’s concept and reflections on how literature conformed to digital media and the convergence of languages. At a time when electronic art was on the cutting edge, but digital art was still hardly discussed, Longhi worked with Julio Plaza’s ideas of intersemiotic translation and transduction6 in order to deal with the issue of creation. In her turn of the century research, Longhi also relied on seminal texts by Lucia Santaella. To address media reshaping, she counted on Jay Bolter and Richard Grusin’s Remediation, whereas to deal with intermedia as an 3

Available at: http://lattes.cnpq.br/web/dgp/home For the other groups, please see Appendix II, p. 27. 5 Available at: http://lattes.cnpq.br/4649814099154237 6 In the volume Tradução intersemiotica (1987), Plaza develops Jakobson’s well-known concept that reaches beyond the “interpretation of verbal signs through systems of non-verbal signs” (2005, p. 65), incorporating both the semiotic categories of C.S. Peirce and the notions of multimedia and intermedia. In “The third-generation, techno-poetic images,” Plaza observes that in the transposition of images from the analog to the digital model, a process of transduction takes place, in which a signal – a sound event, a gesture, a photograph, for example – is translated by an image. In this act of recreation, we have a “passage from one structured code to another,” from which “a relationship between the natural, analogical physical world and the conceptual and digital universe of technology” (1993 apud James Zortea Gomes, “Rastros do desenho e seus desdobramentos no vídeo digital” (Master’s thesis, UFRGS, 2010), 24) emerges. 4

206

C. A. P. de Figueiredo et al.

intersection of codes, the relationship between textual creation in the technological environment and poetry in computer culture, she relied on Antonio Risério’s Ensaio sobre o texto poético em contexto digital.7 As a professor at the Department of Journalism at the Universidade Federal de Santa Catarina, Longhi teaches courses in the areas of online journalism, graphic journalism, cyberjournalistic narratives, and platformization, as well as different formats and genres of journalism. She currently coordinates the study group Núcleo de Estudos e Produção Hipermídia Aplicados ao Jornalismo [Center for Hypermedia Studies and Production Applied to Journalism]8 (Nephi-Jor),9 linked to the research group Hipermídia e Linguagem [Hypermedia and Language].10 With several research projects and academic activities in the field of multimedia production, Nephi-Jor brings together professors, independent researchers, graduate students, undergraduates, as well as professionals from fields related to journalism and multimedia academic activities. The group is interested in new formats and the production of journalistic content for the web, that is, how journalism creates and remodels narratives in digital media, including new content production technologies such as augmented reality, virtual reality, spherical images, and platforms as a production environment and content distribution and sharing. It is also interested in the analysis of social networks with disinformation in journalism, longform journalism, and algorithmic mediations and innovation in journalism. Although the group’s production relies mainly on the imbrications of hypermedia, among their main bibliographic references are the works of Dick Higgins, who is precisely the creator of the concept of intermedia, which the group’s researchers adapt to their studies on cyberjournalistic narratives with an emphasis on the concept of hypermedia. Other references are Júlio Plaza, Josep Maria Català, Lúcia Santaella, Arlindo Machado, Janet Kolodzy, John Pavlik, Marie-Laure Ryan, Jay David Bolter and Richard Grusin, and Edgar Morin, with emphasis on the first three authors. In 2002, starting from the intermedial inherent condition of contemporary art, the research group Caligrafias e escrituras: Diálogo e intertexto no processo escritural na arte contemporânea [Calligraphies and Scriptures: Dialogue and intertext in the writing process in contemporary art]11 was created by the artist and Fine Arts professor Maria do Carmo de Freitas Veneroso.12 With experience in the field of fine arts, theory, criticism, and art history and working mainly on the subjects of contemporary art, engraving, artist’s book, photography, relations between word and image, memory, archive, and appropriation in art, Veneroso pursues the (inter) relations between the arts and media, focusing on printed art and the multiple

7

Risério (1998), Bolter and Grusin (2000), Santaella and Noth (2001), and Santaella (2001) We are responsible for all the translations, unless mentioned otherwise. 9 Group website: https://jij.ufsc.br/nephi-jor/ 10 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/9116076487823664 11 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/0528209044068156 12 Available at: http://lattes.cnpq.br/1612365823765731 8

10

Intermediality in Brazil: A Diachronic Survey

207

images, especially the “livro de artista” [artist book] and the expanded field of printmaking and its intersections and counterpoints with word and image in the context of contemporary art. One of the aims of the research group Caligrafias e escrituras is to ground its members’ artistic productions in the theoretical discussion on word/image relations. In 2007/2008, Veneroso hosted professor Clüver during his stay as visiting professor at Escola de Belas Artes, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil. In 2009, she taught the courses “Word & Image in 20th Century Art” and “The Book: printing, text, image” at Indiana University, Bloomington, United States, as a Visiting Artist within PIVA – Program for International Visiting Artist-in-Residence. In 2014/2015, she coordinated a joint project with Indiana University on engraving and intermediality, which hosted Professor Edward Bernstein. Besides taking part in national and international symposia, congresses, and exhibitions, the group, which is mostly composed of visual artists and designers, has organized national and international events such as Seminário Internacional Perspectivas do Livro de Artista (2009), Seminário Internacional O Livro de Artista na Universidade (2013), and II Colóquio Internacional Escrita, som, imagem (2019), this last one in partnership with Intermedia Group. Veneroso has also curated several exhibitions, among them Cross Pollenization: emerging artists in contemporary Brazilian printmaking (USA and Australia), in 2011; and Circuito Polímatas, in 2019. Also in 2002, Professor Thaïs Flores Nogueira Diniz13 invited Clüver as a visiting professor at the School of Letters, Universidade Federal de Minas Gerais. Diniz, whose master’s thesis focused on the theater of Peter Shaffer and Antonin Artaud and doctoral research centered on the notion of adaptation, emphasizing cultural aspects translated in the process of transposition between different media, included film in her research after meeting Clüver. In her postdoctoral fellowship, she concluded that in addition to the plot, characters, and culture, adaptations also take into account evocations, implicit or explicit ideas, the spirit of the work, and its ideology. Her current personal research deals with film adaptations made from Shakespeare’s adapted plays. Meantime, in a broader view, she coordinates the umbrella project titled “Studies on Intermediality,” which aims at (a) discussing the concept of intermediality and the relationships between the various forms of discourse/art/text/media, (b) investigating the relevance of the concept of intermediality for approaching media products that increasingly take advantage of their medial complexity. The group gathers several subprojects coordinated and developed by members, who, in their turn, analyze literary/visual/artistic media products through the combination of different media, grounded on the concept of intermediality. Among Diniz’s theoretical production on film studies, Literatura e Cinema: da semiótica à tradução cultural (1999/2003) and Literatura e Cinema: tradução, hipertextualidade, reciclagem (2005) should be mentioned.

13

Available at: http://lattes.cnpq.br/2891259327805135

208

C. A. P. de Figueiredo et al.

At the end of his first professorship mission14 at UFMG, in 2002, Clüver had a meeting with several professors who all realized that despite carrying their research out within different departments – namely, Literary Studies, Visual Arts, Design, Music, Communication, Theater and Film Studies –, their work had one thing in common: the interrelationship among the arts. From that meeting, Grupo de Pesquisa Intermídia: Estudos sobre a Intermidialidade [Intermedia: Studies on Intermediality] was created.15 Since 2005, the group, chaired by Professor Thaïs Flores Nogueira Diniz and co-chaired by Professor Claus Clüver, has been registered with CNPq. Linked to the research group, the Intermedia study center/group Núcleo de Estudos sobre a Intermidialidade [Intermedia: Center for Studies on Intermediality]16 was also created, also coordinated by Thaïs Flores Nogueira Diniz along with Márcia Maria Valle Arbex-Enrico.17 Arbex-Enrico is full professor at the School of Letters at the Universidade Federal de Minas Gerais, where she teaches French Literature. Her research interests rely on Comparative Literature, especially the relations between word and image. Among her publications, we can highlight the authored books Alain Robbe-Grillet e a pintura: jogos especulares (2013) and Sobrevivências da imagem na escrita: Michel Butor e as artes (2020). Not long before the establishment of Intermedia Group in 2002, Professor Solange Ribeiro de Oliveira18 stood out for her enthusiasm for Interart Studies by teaching this subject at undergraduate and graduate levels. Professor emeritus at Universidade Federal de Minas Gerais and adjunct professor at Universidade Federal de Ouro Preto, Professor Oliveira taught courses in the areas of English Language and Literature, Linguistics Applied to the Teaching of English, Theory of Literature, and Comparative Literature. She obtained her doctorate in Letters and became full professor at the School of Philosophy at Universidade Federal de Minas Gerais, in addition to taking a postdoctoral residence in Chapel Hill, North Carolina, USA. The publication of her book Literatura e artes plásticas: o Künstlerroman na ficção contemporânea in 1993 attests to her involvement with interdisciplinary approaches. She also subsequently published Literatura e música: modulações pós-coloniais (2002) before the official establishment of the Intermedia research group. Her research career has evolved around Interart Studies and cultural studies. Among her most important theoretical-critical publications in the area, we can highlight De mendigos e malandros: Chico Buarque, Bertold Brecht, John Gay em uma leitura transcultural (1999); Perdida entre signos: Literatura, Artes e Mídias, hoje (2012); and Alvoroço da criação: a arte na ficção de Clarice Lispector (2019). Undoubtedly, her dedication to Interart Studies has contributed enormously to the constitution of Intermedia Group as it is today.

14

Clüver had previously taught courses at other universities in Brazil, but not aiming at intermedial studies. For more on Clüver’s relation with Brazil, see the chapter by Diniz & Oliveira. 15 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/0817494927918913 16 Group website: http://www.letras.ufmg.br/nucleos/intermidia/ 17 Available at: http://lattes.cnpq.br/0642418673482397 18 Available at: http://lattes.cnpq.br/6870324305782229

10

Intermediality in Brazil: A Diachronic Survey

209

One of Intermedia Group’s greatest contributions is the collection of seminal articles in the field of intermediality translated into Brazilian Portuguese: Poéticas do visível: ensaios sobre a escrita e a imagem (2006); Intermidialidade e estudos interartes: desafios da arte contemporânea, v.1 and v.2 (2012); A intermidialidade e os estudos interartes na arte contemporânea (2020); as well as the volumes resulting from the biannual international colloquium held in 2017, 2019, and 2021: Escrita, som, imagem: perspectivas contemporâneas (2019); Escrita, som, imagem: leituras ampliadas (2020); Escrita, som, imagem: novas travessias (2020); and Escrita, som, imagem: natureza em foco (2022), respectively. In 2017, Ana Cláudia Munari Domingos19 and Glória Maria Guiné de Mello,20 of Intermedia Group, together with Ana Paula Klauck,21 as well as other group members who worked as translators or who had other functions, such as Erika Vieira22 and Miriam Vieira,23 edited the translation into Portuguese of seven articles by Lars Elleström in Midialidade: ensaios sobre Comunicação, Semiótica e Intermidialidade. Later, in 2021, members of the group came together again for a new translation into Portuguese, namely, the first part of Beyond Media Borders, Volume 1: Intermedial Relations among Multimodal Media (2020). With the financial support of Capes and Universidade Federal do Rio Grande do Sul, Elaine Indrusiak24 coordinated the translation, while Ana Cláudia Munari Domingos and Camila A. P de Figueiredo25 edited the volume As modalidades das mídias II (2021). Intermedia Group promotes conferences based in several universities all over Brazil and is also highly concerned with the applicability of intermedial notions in teaching. This interest has resulted in graduate and undergraduate courses on intermedial studies in several of the member institutions/universities. Nowadays, the group counts with 38 members from universities all over Brazil, such as Centro Universitário Campos de Andrade, Centro Universitário Una, Universidade do Estado de Minas Gerais, Universidade do Estado do Rio de Janeiro, Universidade Federal da Bahia, Universidade Federal de Juiz de Fora, Universidade Federal de Minas Gerais, Universidade Federal do Paraná, Universidade Federal do Rio Grande do Sul, Universidade Federal de São João del-Rei, Universidade Federal de São Paulo, Universidade Federal dos Vales do Jequitinhonha e Mucuri, Universidade Presbiteriana Mackenzie, Universidade de Santa Cruz do Sul, Unisantacruz, as well as foreign universities: Florida State University; University of Indiana, Bloomington; University of Lund, Sweden; Université de Lyon 3, France; and University of Macau, China. The main theoretical references used by group members are Claus Clüver, Jay David Bolter, Jørgen Bruhn, Heidrun Führer, Irina Rajewsky, Lars

19

Available at: http://lattes.cnpq.br/8636942838445477 Available at: http://lattes.cnpq.br/4457144436870096 21 Available at: http://lattes.cnpq.br/2598750363094867 22 Available at: http://lattes.cnpq.br/3013583440099933 23 Available at: http://lattes.cnpq.br/1117606028406532 24 Available at: http://lattes.cnpq.br/6726224076135944 25 Available at: http://lattes.cnpq.br/0750019205710872 20

210

C. A. P. de Figueiredo et al.

Elleström, Lauren Weingarden, Richard Grusin, Werner Wolf, Veronique Plesch, Walter Bernhard, and Walter Moser. Most of these scholars have already come to Brazil, invited by the group, either to teach graduate and undergraduate courses, as visiting professors at the Institute for Advanced Transdisciplinary Studies (IEATUFMG), or to give lectures at conferences. Universidade Federal de Uberlândia (UFU) accommodates the GPMLA (Grupo de Pesquisas em Mídia, Literatura e Outras Artes) [Research Group in Media, Literature and Other Arts],26 also registered with CNPq and coordinated by Professor Leonardo Francisco Soares.27 Created in 2008 within the premises of the Institute of Letters and Linguistics, the aim of the group is to provide an investigative space within the area of literary studies in dialogue with other semiotic systems. The group proposes to study the comparative limits between literature and other languages as an active research field within the institution, aiming at transdisciplinary dialogues based on varied themes. Thus, it is organized around three lines of research: (i) intermediality, (ii) homoculture and languages, and (iii) literature and cinema: narrative forms and processes. Since its creation, the group has held research meetings, organized and/or participated in events inside and outside UFU, and published works in different periodicals in Brazil and abroad. Also interested in the connections between journalism and new media, the research center Núcleo de Pesquisa em Conexões Intermidiáticas [Research Center on Intermedial Connections]28 (NucCon), linked to the research group Centro de Convergência de Novas Mídias [New Media Convergence Center], registered with CNPq,29 was composed of Communication professors and graduate and undergraduate students from Universidade Federal de Minas Gerais. Geane Alzamora,30 one of the founders, is Associate Professor at the Department of Social Communication at Universidade Federal de Minas Gerais. She has experience in the field of Communication, with an emphasis on theories of communication, semiotics, and theories of journalism, working mainly on the topics of transmedia dynamics, digital communication, and multiplatform journalism. Alzamora often collaborates with Brazilian scholar Renira Gambarato, who has long been working abroad and is currently associate professor of media and communication studies at the School of Education and Communication, Jönköping University, Sweden. Under the light of the medial textualities research line, NucCon is interested in discussing concepts and carrying out empirical and/or applied research on the various types of contemporary intermedia relations, with a special interest in the development of research methodologies, as well as the notion of convergence. Founded in 2012 by professors Carlos D’Andréa,31 Geane Alzamora, and Joana

26

Available at: http://dgp.cnpq.br/dgp/espelhogrupo/4203330025093910 Available at: http://lattes.cnpq.br/8507310300864306 28 Group website: http://nuccon.fafich.ufmg.br/ 29 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/5115999990049365 30 Available at: http://lattes.cnpq.br/8572132339129140 31 Available at: http://lattes.cnpq.br/0283817427921969 27

10

Intermediality in Brazil: A Diachronic Survey

211

Ziller,32 the center also works with concepts such as hypermediation, transmediality, transmedia, and transmedia narrative associated with Peircean semiotics, under different approaches. For instance, we can find the study of networks, algorithms, mediation technology of informational flows within circulation networks (Carlos D’Andrea); the investigation of lesbian behaviors on digital media platforms (Joana Ziller); or the convergence of communication and literature through intermedial studies (Vanessa Brandão).33 Besides known names from intermedial studies, such as Claus Clüver and Irina O. Rajewsky, their conceptual support relies on theoreticians such as Henry Jenkins, Carlos Scolari, John Law, Bruno Latour, and Michel Foucault. Members have published articles in relevant Brazilian and foreign journals, and between 2014 and 2015, several participated in the survey “Sport mega-events in digital media connections,” carried out in an international agreement with the National Research University Higher School of Economics, based in Moscow (Russia). Created in 2012, the research developed by ENTELAS: Grupo de Pesquisa em Conteúdos Transmídia, Convergência de Cultura de Telas [Research Group on Transmedia Content, Screen Culture Convergence]34 focuses on the possibilities of multiple screens of transmedia content and its convergence in plurimedial space. Their lines of action approach television narratives and sliding transmedial subject matters, in a variety of (technical) media such as television, cinema, mobile devices, computers, games, and comic books. Additionally, diversity is also manifested in discussions about sexuality and gender and how they are represented in different media. The group is coordinated by Felipe de Castro Muanis35 and Christian Hugo Pelegrini,36 professors at the Film and Audiovisual undergraduate course and the Arts, Culture, and Languages graduate courses of the Institute of Arts and Design, Universidade Federal de Juiz de Fora. Muanis is a journalist who undertook his doctoral mobility at the Bauhaus Universität-Weimar, Germany, financed by DAAD/ Capes, and his postdoctoral fellowship at the Institut für Medienwissenschaft RuhrUniversität Bochum, Germany, with research in documentary comics. Muanis was also a visiting professor at Universität Paderborn. ENTELAS’s main interests are image, audiovisual, new media, illustration, comics, language, spectatorship, and reception. Although we stated our survey would be based on the research groups registered with CNPq’s national directory, we believe that two groups should be included here due to their protagonism in the growth of intermedial studies in Brazil. The first is the

32

Available at: http://lattes.cnpq.br/5352059274589464 Available at: http://lattes.cnpq.br/1784236086723270 34 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/583150915278579. Group page: https://www. facebook.com/entelasufjf/?ref¼page_internal 35 Available at: http://lattes.cnpq.br/3754220449291813 36 Available at: http://lattes.cnpq.br/1220551391795710 33

212

C. A. P. de Figueiredo et al.

Iconicity Research Group,37 led by João Queiroz,38 professor at Institute of Arts and the School of Communication Studies, Universidade Federal de Juiz de Fora. Member of the International Association for Cognitive Semiotics (IACS) and of the Group for Research in Artificial Cognition (UEFS, Brazil), Queiroz is associate researcher of the Linguistics and Language Practice Department at the University of the Free State (South Africa). His research interests include cognitive semiotics, Peirce’s semiotics and pragmatism, and Brazilian and South American arts and literature. Creativity, creative translation, intersemiotic translation, diagrammatic thinking, semiosis, and metassemiosis are among the main (medial) phenomena investigated by Queiroz. IRC dedicates its investigation to iconic processes in art (such as music, dance, photography, literature), science (computer modeling, network analyses, graph theories), and philosophy (logics, aesthetics, mind philosophy). In 2014, IRC organized the 1 Congresso Internacional de Intermidialidade [1st International Conference in Intermediality]. In his first visit to Brazil, Lars Elleström was the keynote speaker and gave the lecture “A Medium-Centered Model of Communication.” Among several conferences on intermediality, IRC organized the Colóquio Internacional: Intermidialidade, Literatura e outras artes [International Colloquium: Intermediality, Literature and other arts] at the Pontifícia Universidade Católica do Rio de Janeiro (PUC-RJ) in 2018, which also counted with the presence of Elleström. The group has widely published internationally, in books and journals39 (please see a selection in Appendix I). The other group not listed nor registered with CNPq is Intermidialidade: Literatura, Artes e Mídia [Intermediality: Literature, Arts and Media],40 linked to Associação Nacional de Pós-Graduação e Pesquisa em Letras e Linguística (ANPOLL) [the Brazilian Letters and Linguistics National Association]. As previously mentioned, Intermedia Group is composed of scholars from varied fields, such as music, fine arts, communication, design, literary studies, among others. Most Intermedia Group members who happen to be literary studies professors are also likely to be a part of the ANPOLL work group, which also gathers researchers from other universities in Brazil such as Universidade Federal da Paraíba, Universidade Tecnológica Federal do Paraná, and Universidade Estadual de Londrina. With a transdisciplinary approach, the working group was founded in 2014 by Brunilda Reichmann41 and Eliana Lourenço Lima Reis.42 Reichmann’s expertise is in the area of English language literatures, comparative literature, and intermediality, with an emphasis on adapting novels for films and television series. Full professor at Universidade Federal do Paraná, she is currently coordinator and professor of the graduate program in Literary Theory at Centro Universitário Campos de

37

Group website: https://iconicity-group.org/ Available at: http://lattes.cnpq.br/4794107935753176 39 Available at: https://iconicity-group.org/publications/ 40 Group website: https://anpoll.org.br/gt/intermidialidade-literaturas-artes-e-midias/ 41 Available at: http://lattes.cnpq.br/1169236163815772 42 Available at: http://lattes.cnpq.br/3352889046827069 38

10

Intermediality in Brazil: A Diachronic Survey

213

Andrade-Uniandrade, Curitiba, and editor of the journal Scripta Uniandrade, which has published several dossiers on intermediality. Reichmann has invited Claus Clüver, Lars Elleström, Jørgen Bruhn, Hans Ulrich Gumbrecht, and Jürgen Müller to give lectures at international conferences held at her institution. Since 2018, the group has been coordinated by Ana Cláudia Munari Domingos (UNISC) and Miriam de Paiva Vieira (UFSJ). In 2021, along with the Intermedia Group, the work group Intermidialidade: Literatura, Artes e Mídia [Intermediality: Literature, Arts and Media] (Anpoll) organized the 1st International Meeting of Researchers in Intermediality, a remote event that counted with the participation43 of coordinators and members of research groups from different countries, such as the International Society for Intermedial Studies; Grupo de Investigação em identidade(s) e intermedialidade(s) [Identity(ies) and Intermediality(ies) Investigation Group] Portugal; Centre for Literary and Intermedial Crossings Research group, Belgium; Arts and Humanities Research Council (AHRC), United Kingdom; Center of Cultural Studies, University of Bern, Switzerland; Hipermídia e Linguagem [Hypermedia and Language Group], Universidade Federal de Santa Catarina, Brazil; Intermídia Group Universidade Federal de Minas Gerais, Brazil; Grupo Design e Intermidialidade [Design and Intermediality Group], Uniacademia, Brazil; Linnaeus University Centre for Intermedial and Multimodal Studies (IMS), Sweden; and Núcleo de Pesquisa em Conexões Intermidiáticas (NucCon) [Research Center on Intermedial Connections], Brazil. This meeting was a unique experience and an opportunity for contact with many specialists and international research groups and will very much likely have a follow-up to create an annual tradition. The group has been very productive in collaborative publications and journal dossiers organization. In 2015, Vinícius Carvalho Pereira44 created the group Semióticas Contemporâneas [Contemporary Semiotics] (SEMIC) at Universidade Federal do Mato Grosso.45 As part of the Department of Letters and the graduate program in Language Studies, Pereira pursued a postdoctoral fellowship at the University of

43

Agnes Petho (Sapientia Hungarian University of Transylvania, Romenia), Alex Martoni (Universidade Federal de Juiz de Fora, Brazil), Beate Shirrmacher (Linnéuniversitet, Sweden), Brunilda Reichmann (Centro Universitário Campos de Andrade-Uniandrade, Brazil), Clodagh Brook (University of Dublin, Ireland), Eunice Maria Silva Ribeiro (Universidade do Minho, Portugal), Gabriela Rippl (Universität Bern, Switzerland), Geane Carvalho Alzamora (Universidade Federal de Minas Gerais, Brazil), Hernán Ulm (Universidaded Nacional de Salta, Argentina), Irina Rajewsky (Freie Universität Berlin, Germany), James Cisneros (CRIAlt, Université de Montreal, Canada), Janine Hauthal (Vrije Universiteit Brussel, Belgium), Lars Elleström (Linnéuniversitet, Sweden), María Andrea Giovine (Universidad Nacional Autónoma de México), Maria Cristina Ribas (Projeto Prociência, Universidade do Estado do Rio de Janeiro, Brazil), Marina Grishakova (University of Tartu, Estonia), Marion Froger (Université de Montreal, Canada), Matia Petricola (University of L’Aquila, Italy), Raquel Ritter Longhi (Universidade Federal de Santa Catarina, Brazil), Vanessa Brandão (Universidade Federal de Minas Gerais, Brazil), Thaïs Flores Nogueira Diniz (Universidade Federal de Minas Gerais, Brazil), Thiago Berzoini (Uniacademia, Brazil), Xaquín Núñes Sabaris (Universidade do Minho, Portugal). 44 Available at: http://lattes.cnpq.br/5304593788129950 45 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/4228875465821246

214

C. A. P. de Figueiredo et al.

Nottingham, UK. He is also the coordinator of the Extended Nucleus of Semiotic Engineering and Digital Art and a member of several international organizations, such as the Electronic Literature Organization (ELO), Rede de Literatura Eletrônica Latino-Americana [Latin American Electronic Literature Network] (LitELat), Multilingual African Electronic Literature Database & African Diasporic Electronic Literature Database (MAELD & ADELD), and the Anpoll work group Teoria do Texto Poético [Poetic Text Theory], which is associated to the Brazilian Association of Comparative Literature (ABRALIC). SEMIC members mainly focus on contemporary semiotic processes. Their main interests are intermedial studies, literature and digital technologies, semiotics, aesthetics and politics, and semiotics and humancomputer interaction. For SEMIC, research within the scope of intermedial studies brings together research on aesthetic projects that mobilize different materialities, and media and languages, favoring a comparatist dialogue likely to highlight the processes by which cultural productions become increasingly complex and diversified. Created in 2017 by the Dance and Performing Arts professor Daniella de Aguiar,46 the research group Dança e Intermidialidade [Dance and Intermediality],47 Universidade Federal de Uberlândia, focuses on the relations between dance, other arts, and media. With a doctorate degree in Comparative Literature, Aguiar undertook doctoral mobility at Indiana University – Bloomington, under the supervision of Claus Clüver. Her main research interest is related to dramaturgy in dance, through theoretical, artistic, and pedagogical perspectives, as well as the creative relationships between different arts and the development of creation procedures and tools. The concepts of intersemiotic translation, multimediality (or multimodality), and hybrid forms are employed by Dança e Intermidialidade group both theoretically and artistically, similarly to those of the group Caligrafias e escrituras. The members are interested in the topics of creativity, artifacts, and their cognitive niches: dance and semiosis, new dance theories, and dance and cognitive technologies. The main methods and models used by the research group are intermedial studies, semiotics and pragmatism (Peirce), distributed cognition, and philosophy of art. Created in 2018, the research group registered as Design e Intermidialidade [Design and Intermediality]48 deals with design, visual cultures, and narratives. Coordinated by Thiago Berzoini,49 professor of Art History and Aesthetics at the Architecture and Urbanism and Interior Design undergraduate programs at Uniacademia, in Juiz de Fora, the group is interested in the resignification of space and interior design through anthropological and cultural studies. Its members

46

Available at: http://lattes.cnpq.br/3089944755374031 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/4491759765414589. Group website: https:// dancaintermidialidade.wordpress.com/ 48 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/1440498688523156. Group website: https:// www.cesjf.br/campus-arnaldo-janssen-localizacao/177-ensino-extensao-e-pesquisa/grupos-deestudo/4149-estudos-sobre-design-e-intermidialidade-design-cultura-visual-e-narrativas.html 49 Available at: http://lattes.cnpq.br/1866914701962923 47

10

Intermediality in Brazil: A Diachronic Survey

215

visually analyze interiors and their narratives by means of medial representations and also examine the cultural and sensory implications of interior design. They claim that reflections on design interfaces with cultural, sociological, and aesthetic aspects may be promoted by means of software experimentation. Berzoini’s view on intermediality involves challenging perceptions and testing the borders of cultural studies, philosophy, and pop culture, involving aspects such as corporeity, immobility, otherness, and the notions of translation and treason. The group’s main references include notions on intermediality seen in Irina Rajewsky, Dick Higgins, Thaïs Diniz, and Claus Clüver and on transmedia storytelling with Henry Jenkins, Stephen Dinehart, Carolyn Handler Miller, and Janet Murray. They also rely on the work of Gilles Deleuze, Eva Heller, Israel Pedrosa, Jacques Derrida, Jean-Luc Nancy, and Zygmunt Bauman. Several other research groups in our database use the words “intermedial” and/or “intermediality” more as a supplement to their main inquiries than as a phenomenon or a category of critical analysis, as in the groups mentioned above. A vast majority of these groups investigate adjacent areas, and some of the coordinators are also part of one of the previously mentioned groups. For instance, Literatura, cinema e cultura visual [Literature, cinema and visual culture],50 was created at Universidade Federal da Paraíba in 2009 by Genilda Azerêdo,51 who also belongs to Intermidialidade: Literatura, Artes e Mídia (Anpoll). Besides the relations between narratology and intermediality, Azerêdo is interested in film adaptation, theories of literary and filmic narrative, and poetic narratives. A member of both Intermidialidade: Literatura, Artes e Mídia (Anpoll) and Intermedia Group, Ana Luiza Ramazzina Ghirardi52 coordinates with Renata Philippov,53 the research group Língua e literatura: interdisciplinaridade e docência [Language and literature: interdisciplinarity and teaching]54 which focuses on language, literature, and teaching, at Universidade Federal de São Paulo since 2012. As for the other listed groups, you may find them in Appendix II.

Conclusion It became clear during this survey that intermedial studies have been embraced with enthusiasm by Brazilian scholars, especially in the second decade of the twenty-first century. Academia has neither implemented a Program nor a full course, as seen in some countries, but several Brazilian universities have incorporated courses on intermediality into their undergraduate and graduate curriculum. Although theater and music are inherently plurimedial, our database did not accuse any register of 50

Available at: http://dgp.cnpq.br/dgp/espelhogrupo/3449664025149340 Available at: http://lattes.cnpq.br/2336287299613364 52 Available at: http://lattes.cnpq.br/7481586427673190 53 Available at: http://lattes.cnpq.br/9950264048329182 54 Available at: http://dgp.cnpq.br/dgp/espelhogrupo/7345965930648618 51

216

C. A. P. de Figueiredo et al.

research groups based on programs within these two fields, in spite of some of those mentioned, such as the Intermedia Group, sheltering the research of musicians and theater scholars. There seem to be two main trends followed by the research groups interested in intermedial studies: those belonging to the visual art, design, and dance programs tend to benefit from the notions of intermediality to create their own authorial/creative media products, whereas those based on Literary and Language Studies, as well as the Communication and Journalism programs, tend to pursue a critical analytical standpoint; that is, their members either discuss intermedial phenomena present in the investigated media products or the processes in their medial interrelation. In the case of the latter trend, although theoretical discussions are present in the selected bibliography, a majority of the publications present study cases, which indicates that Brazilian academic production should undoubtedly be on the map of intermedial research as secondary sources, but Brazilian scholars interested in intermedial studies still seem somehow apprehensive of plunging into the production of their own theories. Acknowledgments The present work was supported by the Brazilian Council for Scientific and Technological Development, CNPq—Conselho Nacional de Desenvolvimento Científico e Tecnológico (processes numbers: 309678/2021-8 and 304566/2021-7).

Appendix I Brief Selection of Publications by the Cited Research Groups Grupo de pesquisa Hipermídia e Linguagem (Registered with CNPq) • Núcleo de Estudos e Produção Hipermídia Aplicados ao Jornalismo (Nephi-Jor) Lenzi, A. “Evolución en la práctica del reportaje multimedia: tres experiencias del diario brasileño Zero Hora” in Periodismo en nuevos formatos: estado del arte del ciberperiodismo, narrativas y tecnologías emergentes, org. by J. M. F Vivar, 1. ed. 347–354. Madrid: Editorial Fragua, 2017. Longhi, Raquel Ritter. “Immersive narratives in web journalism. Between Interfaces and Virtual Reality.” Estudos em Comunicação, v. 1, 145. 2017. Marciano, C. N., K. Winques, M. C. Rigo. “Di soli a soli: os caminhos da reportagem multimídia” in Gêneros e Formatos no Ciberjornalismo – Estudos e Práticas, 1. ed. Organized by R. R. Longhi, R.C.R Paulino, 17–37. Florianópolis: Insular, 2016. Paulino, R. C. R., M. L. Empinotti. “Sucesso novo em formato “antigo”: periódicos matutinos para tablet e o caso do La Presse+” in Jornalismo Móvel: Linguagem, géneros e modelos de negócio. Organized by J. Canavilhas, C. Rodrigues, 289–318. Covilhã: Editora LabCom, 2017. Silveira, S. C. “Jornalismo ubíquo e dispositivos móveis: uma análise do produto do jornal The Guardian” in Jornalismo Móvel: Linguagem, géneros e modelos

10

Intermediality in Brazil: A Diachronic Survey

217

de negócio, organized by J. Canavilhas, and C. Rodrigues, 411–434. Covilhã: Editora LabCom, 2017. Winques, K., R. J. Torres. “Qual o papel das novas ferramentas na transformação do jornalismo?” in Questões para um jornalismo em crise, organized by R. Christofoletti, 49–66. Florianópolis: Insular, 2015.

Grupo de pesquisa Caligrafias e escrituras: Dia´logo e intertexto no processo escritural na arte contemporaˆnea (Registered with CNPq) Veneroso, M. C. F. Caligrafias e Escrituras: diálogo e intertexto no processo escritural nas artes no século XX. 1 ed. Belo Horizonte: C/Arte, 2012. v. 1. Veneroso, M. C. F., L. G. Pimentel. “Intermidialidade: cruzando fronteiras”, Revista Pós: Revista do Programa de Pós-graduação em Artes da Escola de Belas Artes da UFMG, v. 1, n. 2 (nov. 2011-abril 2012). Veneroso, M. C. F, M. A Melendi (Org.). Diálogos entre linguagens. Belo Horizonte: Editora C/Arte, 2009. v. 1. Grupo de Pesquisa Intermídia: Estudos sobre a Intermidialidade (Registered with CNPq) • Intermídia: Núcleo de Estudos sobre a Intermidialidade • GT Intermidialidade: Literatura, Artes e Mídia (ANPOLL) Books of Translated Articles Arbex, Márcia (Org.) Poéticas do Visível: ensaios sobre a escrita e a imagem. Belo Horizonte: FALE-UFMG, 2006. Diniz, Thaïs Flores Nogueira, Camila A. P. de Figueiredo, Solange Ribeiro de Oliveira (Org). A intermidialidade e os estudos interartes na arte contemporânea. Santa Maria: Editora UFSM, 2020. Diniz, Thaïs Flores Nogueira, André Soares Vieira (Org.). Intermidialidade e estudos interartes: desafios da arte contemporânea. v. 2. Belo Horizonte: FALE-UFMG, 2012. Diniz, Thaïs Flores Nogueira (Org.). Intermidialidade e estudos interartes: desafios da arte contemporânea. v. 1. Belo Horizonte: Editora UFMG, 2013. Elleström, Lars. As modalidades das mídias II: um modelo expandido para compreender as relações intermidiais. Trans. supervised and revised by Elaine Barros Indrusiak. Technical revision by Ana Claudia M. Domingos and Camila A. P. de Figueiredo. Porto Alegre: Edipucrs, 2021. ___. Midialidade: ensaios sobre comunicação, semiótica e intermidialidade. Org. by Ana MUNARI, et al. Porto Alegre: EdiPUCRS, 2017. Book Organization Arbex, Márcia, Thaïs Diniz, Miriam Vieira (Org.). Escrita, som, imagem: perspectivas contemporâneas. Belo Horizonte: Editora Fino Traço, 2019.

218

C. A. P. de Figueiredo et al.

Arbex, Márcia, Camila A. P. de Figueiredo, Cecília Lima, Miriam Vieira (Org.). Escrita, som, imagem: leituras ampliadas. Belo Horizonte: Editora Fino Traço, 2020. Diniz, Thaïs Flores Nogueira, Alex Martoni, Maria Cristina Ribas (Org.). As tranças da arte: questões sobre Literatura e Intermidialidade hoje. 2022 (upcoming). Diniz, Thaïs Flores Nogueira, Camila A. P. de Figueiredo, Cecília Lima, Eliana Lourenço Reis (Org.). Escrita, som, imagem: natureza em foco. Belo Horizonte: Editora Fino Traço, 2022. Reichmann, Brunilda (Org.). Assim transitam os textos: ensaios sobre intermidialidade. Curitiba: Appris Editora, 2016. Veneroso, M. C. F., T. F. N. Diniz, A. M. Mendes (Org.). Escrita, som, imagem: novas travessias. Belo Horizonte: Programa de Pós-graduação em Artes/EBA/ UFMG, 2020. Authored Books Arbex, Márcia. Sobrevivências da imagem na escrita: Michel Butor e as artes. Belo Horizonte: Editora Relicário, 2020. Oliveira, S. R. Alvoroço da criação: a arte na ficção de Clarice Lispector. Belo Horizonte: Editora UFMG, 2019. ___. Perdida entre signos: literatura, artes e mídias, hoje. Belo Horizonte: Programa de pós-graduação em Letras da UFMG, 2012. ___. Literatura e Música: Modulações Pós-Coloniais. São Paulo: Perspectiva, 2002. Journal Dossiers Alvarez, Aurora, Cristine Mattos. “Intermidialidade”. Cadernos de Pós-Graduação em Letras. (Online). São Paulo: Editora da Universidade Presbiteriana Mackenzie, v. 21, n. 3, 2021. DOI: 10.5935/cadernosletras.v21n3p11-21. Alvarez, Aurora, Thaïs Diniz. “Aspectos atuais da intermidialidade”. Revista Todas as Letras, Revista do Programa de Pós-Graduação em Letras da Universidade Presbiteriana Mackenzie, v. 19, n. 1, 2017. Arbex, Márcia, Miriam Vieira. “Imagens legíveis, textos visíveis”. Revista Vis: Revista do programa de pós-graduação em arte, Universidade de Brasília, v. 19, 2020. Bruhn, Jørgen; Márcia Arbex, Thaïs Diniz. “Literatura e Arte: as Fronteiras em Discussão”, Aletria, Revista do Programa de Pós-Graduação em Letras: Estudos Literários da Faculdade de Letras da UFMG, v. 27, n. 2, 2017. Camati, Anna, Brunilda Reichmann. “Questões de intermidialidade: relações, transações e fronteiras”, Scripta Uniandrade – Revista do Programa de PósGraduação em Letras UNIANDRADE, v. 16, n. 3, 2018. ___. “Shakespeare intermidiático, intertextual e intercultural”, Scripta Uniandrade Revista do Programa de Pós-Graduação em Letras UNIANDRADE, v. 14, n. 2, 2016.

10

Intermediality in Brazil: A Diachronic Survey

219

Clüver, Claus, Eliana Reis, Thaïs Diniz. “Intermidialidade”, Aletria, Revista do Programa de Pós-Graduação em Letras: Estudos Literários da Faculdade de Letras da UFMG, v. 14, n. 2, 2006. Domingos, Ana Cláudia Munari, Lars Elleström (In memoriam). “Faces e ângulos da intermedialidade: terminologia e discurso intersemiótico”, Revista 2i, n. 5, 2022. Ramazzina, Ana Luiza, Irina Rajewsky, Thaïs Diniz (Orgs). “Intermidialidade e referências intermidiáticas”, Revista Letras Raras, v. 9, n. 3, 2020. Reichmann, Brunilda. “Intermidialidade”, Scripta Uniandrade Revista do Programa de Pós-Graduação em Letras UNIANDRADE, v. 19, n. 3, 2021. Reichmann, Brunilda, Eliana Reis, Thaïs Diniz. “Intermidialidade”. Aletria, Revista do Programa de Pós-Graduação em Letras: Estudos Literários da Faculdade de Letras da UFMG, v. 23, n. 3, 2013. Vieira, Erika, Heidrun Führer, Miriam Vieira. “Literatura, intermidialidade e ensino”, Revista Letras & Letras, v. 37, n. 1, 2021.

Grupo de Pesquisas em Mídia, Literatura e Outras Artes (Registered with CNPq) Ribeiro, Ivan Marcos, Leonardo Francisco Soares (Org.). Revista Letras e Letras: Estudos de Intermidialidade, Uberlândia: EDUFU, 2011. v. 27. Ribeiro, Ivan Marcos. “Interlocuções entre Literatura e Pintura na Obra.” Circulação, tramas e sentidos na literatura, org. by Betina Ribeiro Rodrigues da Cunha, Rogério da Silva Lima, Rio de Janeiro: Editora Bonecker, v. 1 (2019): 229–238. Soares, Leonardo. F. “Das relações perigosas entre literatura e cinema: para além da fidelidade”. Aletria: Revista de Estudos de Literatura, v. 23, (2013): 87–97. ___. “Narrar por imagens: fricções entre cinema e literatura”. Literatura: Espaço Fronteiriço. Colatina/ Chicago: Clock-Book, v. 1 (2017): 59–74. ___. “Literatura, Cinema e Televisão: o caso Jorge Amado.” Amado Jorge: um retrato de muitas faces. Org. by Betina Rodrigues Ribeiro da Cunha, Carlos Reis. Rio de Janeiro: Bonecker; UFU, v. 1 (2018): 77–104. Soares, Leonardo Francisco, Ivan Marcos Ribeiro, Kênia M. A Pereira, Gilson José dos Santos (Org.). Interfaces: literatura, artes e mídias (Série: A escrita literária: teorias, histórias e poéticas – n 7. Rio de Janeiro/ Uberlândia: Dialogarts/ Editora da Universidade Federal de Uberlândia, v. 1 (2021). Grupo de pesquisa Centro de Convergeˆncia de Novas Mídias (Registered with CNPq) • Núcleo de Pesquisa em Conexões Intermidiáticas (NucCon) Alzamora, G. C, R. R. Gambarato. “Peircean Semiotics and Transmedia Dynamics. Communicational Potentiality of the Model of Semiosis”. Ocula – Occhio semiotico sui medi, v. 15, 2014. Alzamora, G. C, C. Braga, C. “Las redes sociales, armas de protesta. Twitter y Facebook en las protestas de movimientos sociales en España y Brasil.” El uso de las redes sociales: ciudadanía, política y comunicación La investigación en

220

C. A. P. de Figueiredo et al.

España y Brasil. Carme Ferré Pavia (Org.). Bellaterra: Institut de la Comunicació, Universitat Autònoma de Barcelona, (2014): 16–29. Alzamora, G. C, V. Barros, J. Malta, J. “IReport for CNN Transmedia Storytelling On The Brazilian Protests in 2013. Brazilian Journalism Research (Online), v. 11, (2015): 188–2013. Brandão, V. C. “Do Facebook para o livro: redes sociais digitais como espaço para a escrita (poética) do cotidiano”. Escrita, som, imagem: perspectivas contemporâneas. Org. by Márcia Arbex; Miriam de Paiva Vieira; Thaïs Flores Nogueira Diniz. Belo Horizonte: Fino Traço, v. 1 (2019): 63–81. D’Andréa, Carlos. “Conexões intermidiáticas entre transmissões audiovisuais e redes sociais online: possibilidades e tensionamentos.” Comunicação Midiática (Unesp), v.10, n. 2, (maio/ago. 2015): 61–75. Gambarato, R. R, G. C. Alzamora, L. Tárcia. “Russian News Coverage of the 2014 Sochi Winter Olympic Games: A Transmedia Analysis”. International Journal of Communication 10 (2016): 1446–1469.

ENTELAS: Grupo de Pesquisa em Conteu´dos Transmídia, Convergeˆncia de Cultura de Telas (Registered with CNPq) Muanis, F. “Between photography and drawing: the documentary comics as translation of the city”. International Journal of Comic Art, v. 13 (2011): 599–613. ___. “Hipermediação e interatividade: por uma crítica do documentário como um espaço plurimidiático”. Rumores (USP), v. 13 (2019): 66–81. ___. “Comics e cinema-vérité: estratégias do cinema nas bandas desenhadas documentais.” Cinema e Outras Artes, Volume II: Arquitetura, Artes Plásticas e Sonoridades. Org. by Anabela Branco de Oliveira, Ana Catarina Pereira, Liliana Rosa, Manuela Penafria, Nelson Araújo. Covilhã: Editora LabCom.IFP, v. 2, (2019): 85–105. Iconicity Research Group Fernandes, A. L, J. Queiroz. ‘Os Sertões’ de Euclides da Cunha — a intervenção de uma ‘tecnografia’ intermidiática e multimodal”, MATLIT (Materialidades da Literatura) 9(1) (2021): 79–102. Queiroz, J, A. Fernandes, M. Castello-Branco, P. Ata. “From Webern’s serialism to concrete poetry — Intersemiotic translation as a generative, anticipative, and metasemiotic tool”. Perspectives: Studies in Translation Theory and Practice, (2021) 1–25. Queiroz, J, P. Ata. “Intersemiotic translation as a thinking-tool – scaffolding creativity in dance”. Transmediations! Communication across Media Borders, edited by Niklas Salmose and Lars Elleström. Routledge. 2020. Queiroz, J., ed. “Intermediality, Transmediality, Multimodality and Intersemiotic Translation”. International Journal of Semiotics and Visual Rhetoric (IJSVR). Volume 3, Issue 2 (special issue). 2019. Vitral, L., D. Aguiar, J. Queiroz. “Intersemiotic translation as an intermedial strategy for artistic production: the photographic translation of a mobile art project”. Media networks now & then: netzwerke der intermedialität: Festschrift für Jürgen E. Müller. Hamburg, Verlag Dr. Kovac. (2018): 281–296.

10

Intermediality in Brazil: A Diachronic Survey

221

Grupo de Pesquisa Semio´ticas Contemporaˆneas (SEMIC) Moreira, Maria Elisa Rodrigues, org. Cruzando fronteiras: literatura, interartes, intermídia. Cuiabá: EdUFMT, 2020. Moreira, Maria Elisa Rodrigues, Juan Ferreira Fiorini, org. Palavra e imagem na literatura contemporânea. Belo Horizonte: Tradição Planalto, 2020. Moreira, Maria Elisa Rodrigues, Vinícius Carvalho Pereira, Juan Ferreira Fiorini, org. Palavra e imagem em deslocamento. Belo Horizonte: Tradição Planalto, 2020. Moreira, Maria Elisa Rodrigues, Rosângela Fachel, Juan Ferreira Fiorini. org. Intermídias, transmídias e estudos culturais. Foz do Iguaçu: Editora CLAEC, 2021. Pereira, V. C. “Textos Artísticos que Geram Textos Artísticos: Uma Análise Semiótica de Motores Textuais de Rui Torres”. MatLit: Materialidades da Literatura, v. 7, (2019): 153–174. Grupo de Pesquisa Danc¸a e Intermidialidade (Registered with CNPq) Aguiar, D, J. Queiroz. “Semiosis and intersemiotic translation.” Semiotica 196 (2013): 283–292. ___. “Intermediality: phenomena and some research problems”, Lumina (UFJF. Online), v. 9 (2015): 1–4. Aguiar, D, P. Ata, J. Queiroz. “Intersemiotic translation and transformational creativity”. Punctum. International Journal of Semiotics, 1 (2) (2015):11–21. Vitral, L. A, Daniella A, João Queiroz. “An Intersemiotic Translation of a Mobile Art Project to a Photographic Essay”, Photographies, v. 9 (2016): 91–107. Grupo de Pesquisa Design e Intermidialidade: Design, Cultura Visual e Narrativas (Registered with CNPq) Berzoini, Thiago. “A ribalta transmídia: experimentações em narrativa transmídia pela Cia. Teatral Caravela das Artes”, 4 Congresso Nacional – 1 Internacional, v. I (2013): 1152–1157. Berzoini, Thiago, A. L Magalhães, P. Correa, et al. “Considerações iniciais sobre a virtualização dos espaços museológicos”, ANALECTA, v. 6 (2021): 56–68. Berzoini, Thiago. “A Realidade como mídia”, ANALECTA, v. 7 (2022): 1–10.

Appendix II Catalog of research groups with intermediality as their main object of investigation according to CNPq Directory • Observatório de Crítica literária, Ensino e Criação, Universidade Estadual da Paraíba [Literary Criticism, Teaching and Creation Observatory], Universidade Estadual da Paraíba (since 2006) Sebastien Joachim

222

C. A. P. de Figueiredo et al.

• Literatura, cinema e cultura visual [Literature, Cinema and Visual Culture], Universidade Federal da Paraíba (since 2009) Genilda Azerêdo • Poéticas da Imagem [Poetics of Imagery], Universidade Federal de Uberlândia (since 2009) Nikoleta Tzvetanova • Núcleo de Estudos em Literatura e Intersemiose [Center for Literary and Intersemiotic Studies], Universidade Federal de Pernambuco (since 2010) Ermelinda Maria Araujo Ferreira and Maria do Carmo de Siqueira Nino • Criação e Ciberarte [Creation and Cyberart], Universidade Federal de Goiás (since 2010) Gazy Andraus • Grupo de Estudos e Pesquisas em Educação e Linguagens – GEPEL – [Education and Language Studies Research Group], Instituto Federal de Educação, Ciência e Tecnologia do Rio Grande do Norte (since 2012) Luisa de Marilac de Castro Silva • Língua e literatura: interdisciplinaridade e docência [Language and Literature: interdisciplinarity and teaching], Universidade Federal de São Paulo (since 2012) Renata Philippov and Ana Luiza Ramazzina Ghirardi • Literaturas, Mobilidades e Identidades [Literatures. Mobilities and Identities], Universidade Federal de Sergipe (since 2013) Celia Navarro Flores and Fabiano Carlos Zanin • (Des)caminhos da modernidade ao contemporâneo: estudos em literatura [(By) ways from modernity to the contemporary: studies in literature], Universidade Tecnológica Federal do Paraná (since 2014) Rogério Caetano de Almeida and Naira de Almeida Nascimento • Estudos de tradução e intermidialidade em língua espanhola – Axolotl [Studies on Translation and Intermediality in Spanish], Universidade Federal do Maranhão (since 2014) Edimilson Moreira Rodrigues and Josenildo Campos Brussio • Literatura e Linguagens: fronteira, espaço, performance, memória [Literatures and Languages: frontiers, space, performance, memory], Universidade Federal do Estado do Rio de Janeiro (since 2014) Maria José Cardoso Lemos and Luciana Paiva de Vilhena Leit • Grupo de Estudos de Literatura e Crítica Contemporâneas – GELCCO [Contemporary Literature and Criticism Study Group], Universidade Estadual da Paraíba (since 2015) Wanderlan da Silva Alves and Isis Milreu • Ética, Estética e Filosofia da Literatura [Ethics, Aesthetics and Philosophy in Literature], Universidade Federal de Rondônia (since 2016) Vitor Cei Santos and Christian Otto Muniz Nienov • Modernismo periférico: poéticas do século XX [Modernism on the Periphery: twentieth Century Poetics], Universidade Estadual de Mato Grosso do Sul (since 2017) Marcos Vinícius Teixeira and Dirlenvalder do Nascimento Loyolla

10

Intermediality in Brazil: A Diachronic Survey

223

• LiterArtes [LiterArts], Universidade de Brasília (since 2017) Sidney Barbosa and Alessandra Matias Querido • Machado de Assis: novas perspectivas e abordagens [Machado de Assis: new perspectives and approaches], Centro Universitário Campos de Andrade (since 2019) Greicy Pinto Bellin and James Remington Krause • Tradução, Literatura e Cultura [Translation, Literature and Culture], Universidade Estadual de Mato Grosso do Sul (since 2019) Lucilia Teodora Villela de Leitgeb • Grupo de Formação Intermidialidade e estilo – GFIE [Development Group on Intermediality and Style], Universidade do Estado do Pará (since 2019) Sandra Mina Takakura • Grupo de Pesquisa em Poéticas Moderna e Contemporânea – GPPMC [Modern and Contemporary Poetics Research Group], Universidade Federal de Rondônia (since 2019) Paulo Eduardo Benites de Moraes • Audiovisual sem destino [Audiovisuals with no destination] – Visual Arts Research Group, Universidade Federal do Rio Grande do Sul (since 2019) Elaine Athayde Alves Tedesco • Anglia- Grupo de Estudos de Língua e Literaturas Anglófonas [Studies on English Language and Literature], Universidade do Estado de Minas Gerais (since 2020) Gabriela da Cunha Barbosa Saldanha • Literatura em Língua Inglesa [English Language Literature], Universidade Tecnológica Federal do Paraná (since 2020) Marcia Regina Becker and Regina Helena Urias Cabreira • VISU – Grupo de Pesquisa e Extensão em Arte, Imagem e Visualidades da Cena [Research and Extension programs in Art, Image and Visualities of the Scene], Universidade Federal do Recôncavo da Bahia (since 2020) Dorotea Souza Bastos • DRAMATIC – Grupo de Pesquisa em Dramaturgia: Teorias, Intermídias e Cena Cultural [Research Group in Dramturgy: Theories, Intermedias and Cultural Scene], Universidade Federal de Ouro Preto (since 2020) Alex Beigui de Paiva Cavalcante • Interartes e Intermídia: diálogos entre cultura e arte [Interarts and Intermedia: dialogs between culture and art], Universidade São Judas Tadeu (since 2021) Dafne Di Sevo Rosa • GPETILS – Grupo de Pesquisa em Estudos da Tradução e Interpretação [Translation and Interpretation Studies Research Group], Pontifícia Universidade Católica do Rio de Janeiro (since 2021) Teresa Dias Carneiro • Shakespeare e as modernidades [Shakespeare and the modernities], Universidade do Estado do Rio de Janeiro (since 2021) Fernanda Teixeira de Medeiros and Liana de Camargo Leão

224

C. A. P. de Figueiredo et al.

References Bolter, Jay David, and Richard Grusin. 2000. Remediation: Understanding New Media. Cambridge: MIT Press. Clüver, Claus. 1997. Estudos Interartes: conceitos, termos, objetivos. Literatura e Sociedade: Revista de teoria e literatura comparada (DTLLC/FFLCH, Universidade de São Paulo) 2: 37–55. Jakobson, Roman. 2005. Linguística e comunicação. São Paulo: Cultrix. Plaza, Júlio. 1987. Tradução intersemiótica. São Paulo: Perspectiva. Plaza, Júlio. 1993. As imagens de terceira geração, tecno-poéticas. In: Parente, André (org.). Imagem-máquina: a era das tecnologias do virtual. Rio de Janeiro: Editora 34. Risério, Antonio. 1998. Ensaio sobre o texto poético em contexto digital. Fundação Casa de Jorge Amado. Santaella, Lucia. 2001. Matrizes da linguagem e pensamento: sonora, visual, verbal: aplicações na hipermídia. São Paulo: Iluminuras. Santaella, Lucia, and Winifrid Noth. 2001. Imagem: cognição, semiótica, mídia. São Paulo: Iluminuras.

An Overview of Intermedial Studies in China

11

Rong Ou

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Criticism of Sister Arts in the Twentieth Century . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Studies in Comparative Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Poetics of Ekphrasis in the New Century . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Studies in Arts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Text-Image Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality and Narratives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

226 226 230 233 237 239 240 248 248

Abstract

This chapter makes an overview of the contemporary interart/intermedial studies in China and unfolds six stories: (1) Criticism of sister arts in the twentieth century, (2) interart/intermedial studies in comparative literature, (3) poetics of ekphrasis, (4) intermedial studies in arts, (5) text-image studies, and (6) crossmedia and transmedia narrative. As English publications by Chinese scholars on intermedial studies are far from sufficient, the chapter can serve as a window for international readers to have a glance of China’s intermedial studies. Keywords

Intermedial studies · Interart poetics · Comparative literature · Ekphrasis · Crossmedia narrative

R. Ou (*) Hangzhou Normal University, Hangzhou, China e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_11

225

226

R. Ou

Introduction With a holistic view of nature and culture in mind, Chinese education used to be a holistic education: art and literature, poetry, painting, calligraphy, and music were required of the Chinese Scholar-official class; and as more and more talented writers and artists with three or four perfections turned up and practiced the intermedial play of sister arts in their works, the tension between word and image and paragone of painting and poetry characteristic of the West exist less prominent in China, particularly before the modern China, just as Zong Baihua 宗白华 (1897–1986), a prolific philosophy and art scholar, known as “the father of modern Chinese aesthetics,” argues that Various traditional arts in China ... not only have their own unique systems, but often influence each other and even contain each other... Therefore, in terms of aesthetic particularity and total aesthetics, there is commonality and convergence among them. (Zong 1981:31)

In terms of academic studies, literature, history, and philosophy were required of a scholar as well. Social and natural sciences were not developed in old China. Nevertheless, in the late nineteenth century and the early twentieth century, China was forced to open to the West, as more and more Chinese went to Japan, Europe, and America, and thus, the Western education system was introduced to China. Modern schools and universities were founded, and education became more disciplined and divided into literature, fine arts, social science, business, natural science, etc. After the People’s Republic of China was founded, for a long time, China learned from the Soviet Union in many aspects, including education system. The boundaries of disciplines and majors were even stricter, and artistic creation and academic studies were more divided as well. During the Cultural Revolution from 1960s to 1970s, China witnessed an intellectual and cultural “waste land.” Since the 1980s, China has begun to recover its cultural tradition and communicate with the world along with its opening up to the West. Concerning intermedial studies in China, there are five stories to be told in modern and contemporary China based on the intellectual and cultural renaissance and international exchanges.

Criticism of Sister Arts in the Twentieth Century The practice of tihuashi (poems about painting or inscribed on painting) and a general interest in the sister arts of poetry and painting have a long tradition in China. It can be traced back to “Heavenly Questions” by Qu Yuan 屈原 (c. 340 BC–278 BC) and Su Shi’s 苏轼 (1037–1101) famous compliments on Wang Wei’s 王维 (701–761) poetic and visual creation “there is poetry in

11

An Overview of Intermedial Studies in China

227

painting, painting in poetry.”1 The traditional Chinese literati had a more integrated view of art and did not make strict distinctions between literary and artistic categories. The sense of competition between sister arts in the West since the Renaissance is rare in Chinese poetics. Even though modern literati like Wang Guowei 王国维 (1877–1927) and others were influenced by Western aesthetics and had a sense of artistic categorization, they were still mostly concerned with the interaction and convergence of sister arts. As Chinese men of letters used to prefer intuitional and empirical appreciation of literature and art, there were mostly fragmentary and poetic commentaries on the interart criticism like Su’s aphorism (as compared to, for instance, the rather logical and analytical expositions in Lessing’s Laocoon). The senior scholars in the twentieth century who had overseas experience with bilingual and transcultural competence began to do serious and theoretical work on interart criticism, featuring scholars as Zhu Guangqian 朱光潜 (1897–1986), Zong Baihua, and Qian Zhongshu 钱锺书 (1910–1998). Zhu’s On Poetry (1943) was the first insightful monograph on interart poetics, in which, after an exquisite elaboration on poetry’s complex relation to painting and music, Zhu concludes To imitate painting in poetry, the poet tends to represent the reality, and enters into realism; to imitate music in poetry, he tends to emphasize self-expression, and enters into romanticism. This distinction, though old-fashioned, is fundamental. Coleridge phrased it well: one is either a Platonist or an Aristotelian. We can extend this sentence to say: “A poet is born to go towards either picture or music and focus on either objective representation or subjective expression.” When we say “focus”, in fact these two tendencies are in harmony and in many cases there are compromises . . . Most of the truly great poets can reconcile these two conflicts, so that their poems are both pictorial and musical, and the representation of the external image can communicate the inner self at the same time. (Zhu 2013:131)

In addition, Zhu argues for the convergence of different arts with “the beauty of silence”: “It is true that the verbal art cannot fully convey one’s emotional implication, and even if it could, it would not be what literature should aspire to. The same is true of all the fine arts, where the complete representation is not only impossible but unnecessary,” as in poetry, where “the words are not enough,” in music, where “silence is better than sound at some moment,” on the stage, often at the most lively and dramatic moments when “suddenly all is calm, showing a kind of silent and mysterious scene,” and in the statue’s “concealment,” that is, “less is more and silence is beauty” (Zhu 1980:347–358).

The “Heavenly Questions” or “Questions to Heaven” is a poem attributed to Qu Yuan. It contains a series of questions. According to a legend, Qu Yuan wrote this series of questions in verse after viewing various scenes depicted on temple murals; specifically, it is said that following his exile from the royal court of Chu, Qu Yuan looked upon the depictions of the ancestors and the gods painted upon the walls of the ancestral temple of Chu and, then, in response, wrote his questions to Heaven, upon these same walls. Victor H. Mair, “Heavenly Questions,” in The Columbia Anthology of Traditional Chinese Literature, ed. Victor H. Mair, Columbia University Press, pp. 371–386. See Chapter on “Traditional Chinese Painting.”

1

228

R. Ou

Concerning painting and poetry, Zong believes that the artistic conception of Chinese poetry and Chinese painting converges in the representation of the landscape. The perfect integration of poetry and painting is the perfect integration of emotion and scenery, which is the artistic conception: Landscape has become a medium for poets and painters to communicate their sentiment and thought, so they all love to resort to the realm of landscape as the source of inspiration and expression. This is very different from the Western approach to the human body as the main object of artistic creation since ancient Greece. To Chinese artists, the mountains and the earth are a reflection of the poetic mind of the universe; the active mind of the painter-poet is itself a creation of the universe. (Zong 1981:62)

Zong further explores the origin of all Chinese art, namely, the rhythm of music and the spirit of dance. He quotes Chuang Tzu’s words and explains that the perfection of artistic conception lies in the integration of Tao and Art, which is in harmony with the emptiness; “the rhythm of music is their essence. The rhythm of the life is the final source of Chinese art” (Ibid.66). He asserts that the dance is “a symbol of the ultimate state of all artistic expression, a symbol of the creative process of the universe,” because it has “the highest degree of rhythm, rhyme, order, rationality, and at the same time the highest degree of life, movement, energy, and passion”; it is “the fundamental law and the most ardent motion that materializes and incarnates the unfathomable realm of the mysteries” (Ibid.67). He finds in dance the archetype of all Chinese art: Chinese calligraphy and painting tend to move and dance; the flying eaves of traditional architecture manifest the dance posture, and the poets’ ambition “to write with the brush of chasing light and shadow, to write the heart of human and nature” is, according to Zong, the ultimate ideal and the highest attainment of Chinese art (Ibid.71). As an extension and supplement to Zong’s theory that poetry and painting are one and the same, Qian Zhongshu, in his essay “Chinese Painting and Chinese Poetry” (1935), questions and analyses in depth the view that “Chinese poetry and painting are integrated and consistent,” thus suggesting that classical Chinese poetry and painting criticism are not consistent and that the Southern School of painting is of the highest quality, but the Southern School of poetry is not the mainstream (Qian 2002:5). The essay affirms the revaluation of the commonality of Chinese poetry and painting in the framework of Western artistic paradigms and explains the drive for the transformation of poetry and painting in terms of andesstreben. The focus of the essay, however, is on the difference between traditional poetry and painting in terms of the criteria of appreciation; that is, paintings are of the highest quality in the Southern School of Chinese painting with artistic beliefs in minimalism and impressionism, represented by Wang Wei, while poetry is of the highest quality in the realistic school represented by Du Fu 杜甫 (712–770), a conclusion reached on the basis of an overview of the history of Chinese classical poetics, a valuable contribution to the study of the relationship between poetry and painting at the time. Coincidentally, all three scholars have responded to Lessing’s Laocoon at different times.

11

An Overview of Intermedial Studies in China

229

Zhu is the first Chinese scholar to do critique on Lessing’s theory of the heterogeneity of poetry and painting in On Poetry. He affirms the aesthetic value of Lessing’s theory but also points out its shortcomings: there is no denying that each art form is limited by its medium, but the art’s greatest success often lies in conquering the obstacles of the medium. It is common for painters to use shapes and colors to produce the effect of language and sound and for poets to use language and sound to produce effects of shape and color. Zhu cites tihuashi of Du Fu and Su Shi to illustrate that the painter seems to be telling a story, while the poem contains more elaborate visual images than the painting, and Lessing’s demand for the painting to represent “a pregnant moment in time” and stillness is inappropriate for the appreciation of Chinese traditional painting. Furthermore, the listing of things in poetry, which Lessing rejects, is common in Chinese Fu writing, lüshi (rhythm poetry), and lyrics, and therefore, Lessing’s doctrine is not suitable for analyzing Chinese poetry and painting (Zhu 2013:141–2). Zhu finished his Chinese translation of Lessing’s Laocoon in 1965 and had it published in 1979, appended with an extensive translator’s note, providing valuable introduction to the historical background for Lessing’s writing and making insightful analysis on its strength and weakness. In Aesthetic Walks (1959), Zong also responds to Lessing’s Laocoon. Likewise, when relating it to Chinese poetry, Zong finds that Lessing’s theory is not entirely valid: “Many ancient Chinese lyric poems are purely depicting scenery, depicting an objective realm, like what Wang Guowei called ‘The realm of no self’ in his Renjian Cihua, and do not represent the action of the subject, nor even express the poet’s subjective emotions directly, but they are full of poetic ambience and sentiment.” Taking Wang Changling’s poem of “The First Day” and Adolph Von Menzel’s painting as examples, Zong proposes that great works are unity of poetry and painting: poetry and painting each has its own specific materiality that limits its communicative power and scope and cannot nor need substitute each other (Zong 1981:8–9). Each of them can draw the other into its own art form as much as possible. The perfect integration of poetry and painting (mutual communication and immersion) is the perfect union of sentiment and scenery, which is the so-called “artistic conception” (Ibid., 13). What interests Qian is the space-time category proposed by Lessing in Laocoon. In his essay “Reading Laocoon” (1962), he quotes a number of Chinese classical poems to illustrate the limitations of painting in terms of temporal representation and argues, “What is picturesque in poetry cannot always be represented in painting.” Qian further points out that even “the painterly poem” that represents static scenes may not necessarily be transformed into “the physical painting.” For instance, it is difficult to represent the temporal continuity in painting; other senses such as smell, touch, and hearing and the inner state of mind rather than sorrow, joy, anger, and sadness, as well as the atmosphere and ambience, are also “difficult” and “impossible” to portray in painting, and it is not only an issue of time and space (Qian 2002:38). Qian favors the notion of “Augenblick,” which is German for “a moment,” put forward by Lessing. Qian takes “Augenblick” as “a pregnant moment,” like in

230

R. Ou

history painting, the artist should select the most intriguing and imaginative moment in the story, and this moment is like a woman’s “pregnancy”: it contains all that has gone before and all that is to come. This concept, Qian argues, applies not only to history painting but also to figure painting, and there is similarly “a pregnant moment” in Chinese painting: a moment that ends shortly, leaving much room for the audience’s imagination (Ibid. 48–50). In addition, Qian adopts the concept of “a pregnant moment” in literary criticism and analyzes meaningful and functional “pregnant moments” in Chinese classical novels, European novels and plays, as well as Chinese popular entertainment such as “storytelling” and Pingtan.2

Intermedial Studies in Comparative Literature Similar to what happens in the West, interart and intermedial studies also develop in the field of comparative literature in China. In 1981, Comparative Literature Society was founded in Peking University, and a comparative literature research center was set up as well which began to compile and publish a series of Comparative Literature Studies. In 1983, the Tenth Conference of International Comparative Literature was held in New York, and Yang Zhouhan 杨周翰, as the first delegate from Mainland China, attended the conference. In the same year, the first Sino-US Bilateral Comparative Literature Symposium was held in Beijing and the journal of Comparative Literature in China was founded. In 1985, China Comparative Literature Association (CCLA) was founded, and the first conference was held in Shenzhen. Since then, a conference has been held every two or three years regularly and the 13th conference was held in 2021. Introduction to Comparative Literature (1984) by Lu Kanghua 卢康华 and Sun Jingyao 孙景尧 was the first Chinese comparative literature monograph (which also serves as a textbook for early teachers and students of comparative literature). The two scholars include “the interrelations of literature and other art forms” as a part of comparative literature study in their defining the characteristics of the discipline (Lu and Sun 1984:76); nevertheless, later on when they elaborate on the methodology of comparative literature, they exclude the interdisciplinary studies like literature and other arts from the scope of the discipline (Ibid. 187). Yue Daiyun 乐黛云 in Principles of Comparative Literature (1988) offers a chapter on interdisciplinary studies of literature and other arts. At the outset of the chapter, Yue proposes that Each art, like literature, painting, music, sculpture, has its own unique history of development, content and structure, and method of communication, but it is undeniable that they are all expressions and representations of human thinking and feelings. Therefore, whether in

Pingtan 评弹, also known as Suzhou pingtan, is a musical/oral performance art popular in southeastern region of China. Its contents are enriched with the mixed performing techniques of storytelling, joke cracking, music playing, and aria singing. 2

11

An Overview of Intermedial Studies in China

231

the West or the East, the phenomenon of various art forms interpreting each other is always common and diverse. (Yue 1988:210)

Furthermore, the Transdisciplinary Studies of Comparative Literature (1989) edited by Yue and Wang Ning 王宁 includes essays of interart and intermedial studies like Yue’s “Literature and Other Arts,” Wu Xiaoming’s “Literature and Music,” and Wang Changjun’s “Poetry and Painting.” Yang Zhouhan champions the interart studies and remarks in the preface: According to the general definition of comparative literature, it includes the comparative study of two or more literatures of different countries, different nations or different languages; it also includes the study of the relationship between literature and other disciplines, other arts or other media of expression. For the former, our scholars have done a lot of work, and made striking achievements, especially in the comparative study of Chinese and Western literature, which shows the characteristics of Chinese comparative literature research; for the latter, what we have done is obviously insufficient, and many scholars are still quite unfamiliar with this research method. (Yang 1989:1–2)

Like Yue, Chen Bei 陈悖, and Liu Xiangyu 刘象愚 in the Introduction to Comparative Literature (1988) includes “literature and other arts” in the contents of the book and defines the discipline as follows: Comparative literature is an open literary study; from a macro and an international perspective, it takes various literary relations as its object of research that transcends the national, lingual, cultural, and disciplinary boundaries and it is characteristic of comparative awareness and inclusiveness in terms of theory and methodology. (Chen and Liu 1988:21)

Nevertheless, some scholars have disapproved the cross-disciplinary study. Wang Xiangyuan 王向远, another renowned comparative literature scholar in China, has been critical of the interdisciplinary studies, for “it blurs the boundary of comparative literature as a discipline,” which leads to comparative literature being engulfed by “comparative culture”; he argues that Interdisciplinary study is broader than single-disciplinary study, and “interdisciplinary studies” of literature and other disciplines is naturally broader than “literary study”. Comparative literature is the study of literature, but the “interdisciplinary studies” of comparative literature often ceases to be “literary studies.” (Wang 2002:11)

Despite the disputes, interdisciplinary studies of comparative literature have boomed, including literature and religion (Tan 2002), literature and psychology (Fen 2003), literature and philosophy (Luo 1999), literature and science (Jiang 2021), and literature and anthropology (Ye 2018a) from which develops the branch of literary anthropology (Ye 2018b). Comparative study of literature and other arts, however, has not flourished in the discipline of comparative literature. Among the 13 triennial conferences of CCLA, only one has “literature and other arts” as the topic for a panel. In 1999, on the 6th conference hosted by the College of Literature and Journalism of Sichuan University, there was a panel with the topic “the

232

R. Ou

interaction between media culture and comparative literature,” composed of presentations of Huang Mingfen’s 黄鸣奋 “Revolution of Literature and Art in Computer Age,” Li Siqu’s 李思屈 “Media Culture, Commercials and Verbal Power,” and Huang Lin’s 黄琳 “Comparative Literature and Mass Media.” Other than CCLA, a few other institutions have organized conferences related to literature and art. In 2001, the International Symposium of Comparative Literature organized by the Association of Chinese and Comparative Literature (ACCL) called for papers on “comparative study of literature, film and art as cultural communication.” In 2016, Literature and Culture Research Center of South China Normal University organized the Colloquium of Music and Literature, on the topics of music and literature, old music and modern music, and poetry and local music. In 2017, the Trans-boundary Workshop: Dialogue between Literature and Fine Arts, Music, Theater, and Film was held by the Department of Chinese Language and Literature of Fudan University. Scholars from the different fields got together and discussed five topics: Calligraphy and poetics; literature and theater; musical culture from linguistic and philosophical dimensions; from classicism to modernism: visual art and social significance of its culture; and from literature to cinema: history and cultural criticism of Chinese film. In 2020, the proceedings entitled Chinese and Western Literature and Art Trends and Trans-Boundary Thinking: A Dialogue between Literature and Music, Fine Arts, Music, Theater, and Film edited by Yang Naiqiao 杨乃乔 was published. In the preface, Yang claims that in present global age, the knowledge explosion inevitably promotes the outward expansion of disciplinary boundaries, and Reflecting on the source of human history, literature and art inherently have an indispensable inter-aesthetic relationship in their original state; since modern times, the division of disciplines for human knowledge has been more or less logical, but it entails a kind of academic tribalism that builds walls and dams between different disciplines. (Yang 2020:1)

He declares “There is no need for us to discuss the legitimacy of cross-border research any more. The unboundary era of literary and art studies has arrived” (Ibid. 4). Nevertheless, even if the book consists of five sections with headings of literature, music, fine arts, theater, and film, only few essays really involve interart or intermedial studies. It is rather “a tentative start” of transboundary research, as Yang acknowledges at the end of the preface. Nevertheless, the school of literary anthropology, an outstanding representative of interdisciplinary research in Chinese comparative literature, has developed very fast since 1990s. The pioneer of the school, Ye Shuxian 叶舒宪 has perfected the quadruple-evidence method 四重证据法 to conduct cultural hermeneutics, which involves intermedial studies. Ye proposes that many Chinese classical literature and mythology can be approached from four aspects: Chinese philology and exegesis of extant literature as the primary evidence; the unearthed ancient documents including oracle bones, bronze inscriptions, or bamboo slips as the secondary evidence; the field work of anthropology and ethnology including oral legends, folk songs, and ritual performance as the third evidence; and the archaeological excavated objects

11

An Overview of Intermedial Studies in China

233

and images as the fourth evidence (Yang and Ye 2019; Ye 2020). In Jade Myth Belief and Huaxia Spirit (2019), Ye applies the quadruple-evidence method to argue for the thesis that the belief in jade myths constructs the core value of the Chinese civilization. In A Mythological Approach to Exploring the Origins of Chinese Civilization (2022), Ye makes intriguing interpretations of the Chinese ancient totems such as the bear and the owl and explores the origin of Chinese civilization using the quadrupleevidence method, which integrates ancient and unearthed literature, oral transmission, and archeological relics and graphs.

Poetics of Ekphrasis in the New Century Around the turn of the twenty-first century, along with more and more frequent international academic exchanges, the Western interart poetics and intermedial theories have been introduced into China where it integrated with Chinese practice and spurred a flowering of scholarship on interart/intermedial studies. Joyce C.H. Liu 刘纪蕙 and Zhaoming Qian 钱兆明 are the two pioneering scholars of interart studies. Liu’s Eight Treatises on Literature and Art (1994) manifests the author’s transcultural and interart competence of scholarship, which covers the cultural interpretation of intertextuality and counterpoint in Benjamin Britten’s War Requiem; the dialectical relationship between music and narrative in Jean-Luc Godard’s Préenom Carmen; the confrontation, juxtaposition, and satire of words and music in romantic opera; and the heterogeneous voices and multiple subjects in Richard Strauss’ Die ägyptische Helena; Liu also attaches importance to the cultural criticism of mass communication and the film industry, as well as teaching Western culture via literature and art. Appling the poetics of ekphrasis of Hagstrum, Mitchell, Krieger, and Heffernan, Liu’s insightful essay “Palace Museum vs. Surrealist Collage: Two Modes of Cultural Identity Construction in Modern Taiwan Ekphrasis Poetry” (1996) investigates the interaction between literature and visual art, exploring how visual culture and imagination shape the perspective and representation of literary text and how literary text appropriates and transforms the visual image to achieve its own political agenda. In Modernist Response to Chinese Art (2003), Qian explores the impact of Chinese art on three American modernist poets (Ezra Pound, Wallace Stevens, and Marianne Moore), in terms of both its specific importance to their development and its larger importance to modernism generally. Making use of the pioneering efforts of Mitchell, Steiner, and others in the study of verbal and visual representation, Qian conducts in-depth and fascinating readings of American modernist ekphrasis, to uncover an intricate interchange between vision and verse and across age and culture. Li Hong 李宏 (2003) published the first essay on ekphrasis in the field of art history, in which he examines G. Vasari’s ekphrasis of many works of art in the Lives of the Most Eminent Painters, Sculptors and Architects (originally 1550). Li observes that for Vasari, the description of a work of art can be understood as a response to a particular image. The strong illusion in the work arouses his imagination and makes him sometimes mistake the image for reality itself. On the other

234

R. Ou

Fig. 1 Publication Year Tendency 2003–2021 on ekphrasis

hand, the images strongly stimulate his fantasy, causing him to project a certain meaning into the theme and form of the images he describes. In this essay, Li adopts Fan Jingzhong’s 范景中 Chinese translation of ekphrasis as yige fuci 艺格敷词. Since then, art critics and historians in China have begun to get familiar with the concept of ekphrasis. With the criticism of ekphrasis in poetry by Tan Qionglin 谭琼 琳 (2010), Zhaoming Qian (2012), Ou Rong 欧荣 (2012) and that in fiction by Huang Beibei 黄蓓蓓 (2009) and Wang An 王安 (2010), literary critics in China begin to be familiar with the poetics of ekphrasis. From 2003 to December 2021, searching with the subject of “ekphrasis” from the cross-database of China National Knowledge Infrastructure (CNKI), the largest academic publication database in China,3 one can find 120 publications, including 96 journal articles and 23 thesis (4 PhD thesis, 19 Master’s theses) and two conference papers. Judging from Fig. 1 indicating the publication year tendency, from 2003 to 2014 the criticism of ekphrasis in China remained steady, with less than five publications each year. From 2014 to 2021, there was considerable advancement of the research, peaking with 20 publications in 2019 and 2020, with a drop to 14 in 2020. Figure 2 shows that criticism of ekphrasis distributes in eight disciplines, with world literature on the top (55.15%), followed by literary theory (18.38%), fine arts (8.8%), foreign language (5.15%), and Chinese literature (5.15%). There is also the criticism of ekphrasis in music and dancing, theater, film, and video art, as well as Chinese literature. 3

CNKI is a cross-database consisting of a variety of publications including journals, books, proceedings, monographs, these & dissertations, etc. where one can search for the target source with terms of “subject,” “title,” “key word,” “author,” etc. The website in Chinese is https://www. cnki.net/, and the website in English is https://oversea.cnki.net/index/ (accessed 2021/8/26).

11

An Overview of Intermedial Studies in China

235

Fig. 2 Discipline distribution of ekphratic criticism

Fig. 3 China authors’ distribution related to ekphrastic criticism

Figure 3, indicating China authors’ distribution related to ekphrastic criticism, shows that Ou Rong produces the most of publications on ekphrasis. Of 14 journal articles (one not included in CNKI), 11 involve close readings of poetry and painting, poetry and music, poetry and dance, poetry and architecture, poetry and theater, one interview with Lars Elleström, and two reviews of Western interart/ intermedial studies. In the interview, which took place in 2020 when Ou was a visiting researcher at Linnaeus University, Elleström discussed the necessary expansion of interart studies to intermedial studies, some key concepts of intermedial studies, major approaches to intermedial studies, academic evaluations on

236

R. Ou

intermedial scholars, the beneficial interaction between intermedial studies and literary studies, and the history of ISIS. At the end of the interview, Elleström wishes to see more Chinese and Asian scholars to participate in the international exchanges of intermedial studies (Ou 2021). The second review (2020) offers an overview of the development and changes of contemporary interart poetics in the West and examines the evolution of the term “ekphrasis” in connotation and extension from a rhetoric term to a poetic genre and to a critical key concept, as well as the localization of the notion in China. Due to different research fields, Chinese scholars have different understandings and translations of ekphrasis. In art history and art theory, Fan’s yige fuci is the most popular Chinese equivalent with the emphasis on its descriptive function; in the semiotics and iconology, Hu Yirong 胡易容 translates it as fuxianghua 符象化, Shen Yadan 沈亚丹 puts it as zhaoxing miaoshu 造型描述, and Wang Dong 王东 as tushuo 图 说, underlining its plastic and visual quality; in the novel criticism, Wang An and Cheng Xilin 程锡麟 put it as yuxiang xushi 语象叙事, stressing on the narrative function of verbal painting; and in the criticism of poetry, C.H. Liu takes ekphrasis as duhuashi 读画诗, Tan Qionglin as huihuashi 绘画诗, and Zhaoming Qian as kuayishushi 跨艺术诗, regarding it as a subgenre of poetry (Ou 2020:131–2). Ou, however, observes that as the current state of interart and intermedial research shows, the scope of ekphrasis has expanded beyond “the verbal description of artworks” or “the verbal representation of visual representation,” and she suggests that In the modern sense, ekphrasis can be used to refer to the transformation or rewriting between different artistic media and different artistic texts, thus translating ekphrasis to yigefuhuan 艺格符换 in Chinese can refer to the dynamic metamorphoses between different artistic texts and different sign systems, whereas the above translations fail to cover the interactive transformation between different artistic texts and the continuous, dynamic mutual/multi-directional influences. (Ou 2013:244)

Qiu Hemin 裘禾敏, a translation scholar, notes that the translation of academic concepts has multiple dimensions, and through comparative analyses, he states that the Chinese translation of yigefuhuan for ekphrasis is more appropriate and “more in line with the current era of pictorial-turn for word and image relations. It not only takes into account the fields of literature, art history, and art criticism, but also accommodates other art categories such as film and television, advertising design, etc. In this way, the scope of the concept can be considerably broadened (2017:91).” Besides, Ou has published two monographs related to intermedial studies. In Les Fleurs du Mal: Urban Writing in Anglo-American Modern Poetry (2018), focusing on the response of modernist poets to the “urbanization” of Western society, the author explores the three ideological sources of Anglo-American modernist urban poetics, namely, Baudelaire’s pioneering modernity poetics, the British cultural and artistic tradition at the turn of the twentieth century, and the aesthetics of life from the East Asian culture. With regard to three hotspots of contemporary modernism studies: modernism and oriental culture, modernism and interart criticism, and modernism in urban writing, Ou’s study takes the process of urbanization and

11

An Overview of Intermedial Studies in China

237

urban lifestyles as a lens for reexamining Anglo-American modernist poetry, exploring the transformation of poetics from an idyllic to an urban model in modern cultural space, reflecting an aesthetic taste closely related to everyday life and illustrating the poetic strategies of “the aestheticisation of everyday life” in terms of flâneur (the artist-poet of the modern metropolis) andesstreben4 (the aspiration of one art form toward another) and ekphrasis. Museum of Words: European and American Interart Poetics (2022), deriving its title from James Heffernan’s Museum of Words: The Poetics of Ekphrasis from Homer to Ashbery (1993), offers a comprehensive study of European and American interart poetics. Apart from the introduction, the study is divided into two parts: “historical and theoretical picture” and “critical practice of case studies.” The theoretical part includes an overview of the academic history from Aristotle, Horace, and Lessing to Irving Babbitt and Clement Greenberg, a survey of current scholarship, a scrutiny of representative interart theories on the word, image, and music and poetics of New Media and a comparative reflection on Chinese interart poetics from the antiquity to the twentieth century. The critical part consists of interart/intermedial readings of a number of English, American, German, and Chinese poems, including a transmedial and transcultural interpretation of W. B. Yeats’ “When You Are Old” in the contemporary Chinese context and a track of Mulan’s journey from a folk song of the Northern Wei period (386–534) to the diverse media products at present, from China to the world via Maxine Hong Kingston’s fiction and Disney cartoons and movies.

Intermedial Studies in Arts The School of Arts of Nanjing University has a strong team of intermedial studies in arts led by Zhou Xian 周宪, He Chengzhou 何成洲, and Zhou Jiwu 周计武. They have organized three symposia on “Intermedial Studies in Arts” (2019, 2021, 2022), covering the topics such as “Present agendas of intermedial studies,” “Intermedial studies of art and science,” “Dialogue of Intermedial studies: China and West,” “Fine arts and intermediality,” “Adaption and intermediality,” “Intermediality and performativity,” etc. They also have published a series of articles to shape the discourse of “intermedial studies in arts.” According to Zhou Xian, in the history of art theory, there are four paradigms to understand the interrelations of different arts: the paradigm of sister arts, historical paradigm, art typology, and comparative arts. Zhou champions the latest intermedial studies, for “it overcomes the linguistic-centrism and literature-centrism of 4 Flâneur, a French word, means “a city roamer” and is adopted by Charles Baudelaire to describe Constantin Guyes as a genius of observer, artist, and philosopher in Le peinture de la vie moderne (1863), and later on, Walter Benjamin picks the term to describe Bauderlaire in Charles Baudelaire: A Lyric Poet in the Era of High Capitalism (1937–1938) (Andesstreben is a German aesthetic concept, introduced into English literary world by Walter Pater. See H. Walter Pater, The Renaissance: Studies in Art and Poetry, Dover: Dover Publications, 130–131).

238

R. Ou

Fig. 4 Zhou Jiwu’s Model: Five elements of art (2020a: 211)

comparative arts, and is most productive for the discipline construction and knowledge production of art theory” (Zhou 2019:7). There is always the tension of the diversity and unity of various arts in the knowledge system of art theory, with the diversity leading to various art theories and with the unity guiding the general theory of art. Intermediality is an effective solution to the tension and provides more solid theoretical and methodological basis for the knowledge production of art theory (Zhou 2020b:202). Moving beyond M. H. Abrams’ four elements of art in a convenient triangular pattern of literary criticism, Zhou Jiwu (2020a) proposes a theoretical model of five elements of art (Fig. 4). In his model, the medium is the intermediary link between the artist and the audience, the production of art and the reception of art, the work, and the universe. As the core element of art, the medium, together with the artist, the audience, the universe, and the work, constructs the coordinates of artistic creation and art study. In the dual cultural contexts of the pictorial turn and the intermedial turn, the intermedial studies of art break the language-based and literature-centered paradigm, have not only expanded the research field of art theory but have also effectively explained the commonalities between various art forms, achieved interdisciplinary integration of methodologies, and provided a legitimate basis for the cross-media construction of art theory. He Chengzhou suggests Chinese scholars should be active and innovative in the intermedial studies and attentive to the intermedial interplay of theater as a composite art, intermediality between literature and art, digital humanities, and intermediality in transculatural studies; he encourages Chinese scholars to draw the inspiration from the traditional Chinese art for the construction of media theory, make use of existing intermedial theories to examine Chinese traditional and current cross-media phenomena, and advance the development of intermedial studies in international dialogue (He 2019:12). To illustrate his suggestion, he does a case study of “theatre-fiction” from the perspective of intermediality (2020). He observes that as a literary subgenre, “theatre-fiction” is an important phenomenon in both Chinese and world novels. In light of theory of intermediality, its intermediality features two major categories: the intermedial hybridity and the intermedial

11

An Overview of Intermedial Studies in China

239

reference. Borrowing from contemporary theories on theatre and performance, his study focuses on the modes, the process, and the effects of intermedial interactions in such “theatre-fiction” as Virginia Woolf’s Between Acts and Mo Yan’s Sandalwood Death so as to carry out detailed analysis on the innovations in the novelistic forms, the aesthetic experiences of the readers and the role of fictional works in the reimaging of both history and reality. Li Jun 李军 and his team at the School of Humanities of Central Academy of Arts are strong at the intermedial studies of art history. Li’s A Visible History of Art: From Church to Museum (2016) is based on the author’s fieldwork and collection of large amounts of firsthand materials during his research tour in Europe. Through the minute iconographic scrutiny and architectural exploration of nineteen case studies including the birth of Louvre, Giorgio Vasari’s The Last Judgment, Saint Francis Altarpieces by Bonaventura Berlinghieri, and the Church of San Francesco of Pescia, etc., an intermedial history of art is constructed revealing the intriguing links between the modern museum institution, the narrative of modern art history, and the pre-modern institution of church building and its spatial layout. In A Transcultural History of Art: On Images and Their Doubles (2020), Li explores the intermedial play of painting and poetry, illuminating the relations between Su Shi’s poetry on the Red Cliff and Qiao Zhongchang’s 乔仲常 Illustration of the Second Prose Poem on the Red Cliff, the interaction between Lin Huiyin 林徽因 as a poetess and Lin as an architect, and the “Pictorial Turn” of Shen Congwen 沈从文 as a writer to Shen as an art historian; and with case studies of a transcultural journey of four maps of China, images of silk worm related to the Silk Road, a narrative exhibition with visual design and so on, the author pictures a “visual transcultural history of art.” Besides, in “Texture” and “Text” (2022), Li presents another case study of intermedial history of art. This paper focuses on the complex connection between the Buddhist plastic arts in northern China and the classical poetics in southern China through the fourth to the sixth century, that is, the intermedial parallel between the “texture” of plastic arts and the “text” of the contemporary poetry. It sheds new light on the unique artistic practice of “inverse image” and “inverse script” in this period. With this case study, the author hopes to explore a methodology of an intermedial art history.

Text-Image Studies Zhao Xianzhang 赵宪章 and his team at Nanjing University and Chen Ming 陈明 and his colleagues at Peking University are two main forces of text-image studies in China. As a professor of Chinese literature, Zhao started his text-image studies in 2010s. He and his colleague has edited six volumes of Literature and Image (2012–2017); his papers on text-image studies has been translated into English and published in Italy as Text-Image Theory (2021). In the book, Zhao engages in comparative semiotic studies on Chinese traditional literature and arts; dwelling on the relationship between literary text, painting, and calligraphy, this book aims to inquire into

240

R. Ou

the iconic links between the verbal and the visual. Based on the author’s expertise of classical Chinese literature and arts, it draws inspiration from Western semiotics and Chinese poetics, using the latter to activate the former and constructs an intellectual “Esperanto” for Sino-Western exchanges. In addition, he and his colleagues’ eightvolume History of Chinese Text-Image Relationship (2020), covering the text-image interaction for thousands of years from Pre-Qin Era (the Paleolithic Period–221 BC) to Qing Dynasty (1636–1912), forms a combination of “history” and “criticism” with Text-Image Theory, exerting strong impacts on the study of the text-image relationship to make new breakthrough in China. Since 2017, the Research Center of Eastern Literature of Peking University headed by Chen Ming has organized five symposia on “Literary Text and Image Studies.” The center has been working at the project of the Collection and Study of the Illustrated Books of Ancient Eastern Literature. The aims of the project are as follows: First, doing the research on the basis of first-hand illustrated manuscripts. Second, fully understanding the role of written texts related to illustrations, and clearly sorting out the corresponding relationship between the visual and the verbal. Third, making full use of multilingual versions of illustrated books for comparative study. Fourth, revealing the unique discourse and connotation of images, and illuminating the mechanism and value of images in cultural communication and dissemination.5

The research team has produced a series of related papers, represented by Chen Ming’s “Study on the Image Tradition of Ancient Eastern Literature” (2016), Mu Hongxian’s 穆宏燕 “Significance of Illustrated Manichaean Canons as History of Art” (2017), and Zhao Jinchao’s 赵晋超 “Wonders of Book of Surprises: An Examination of the Depictions of the Architecture in a Jalayirid Manuscript and its Two Ottoman Copies” (2018).

Intermediality and Narratives The most thriving branch of intermedial studies in China develops in the field of narratology, known as transmedial narrative or cross-media narrative studies. In 2007, the First International Conference on Narratology and the Third China Symposium on Narratology was held in Nanchang. Quite a few prominent narrative scholars attended the conference, like James Phelan, Robyn Warhol-Down, Peter J. Rabinowitz, Brian McHale, Shen Dan 申丹, and Fu Xiuyan 傅修延. The symposium consisted of four topics: (1). Frontier Theories of Narratology; (2). Narrative Theory and Practice: East and West; (3). Expansion of Narrative Theory: Interdisciplinary and Intermedia Narratives; and (4). Reinterpreting Narrative Works from 5

See the interim report on the project, http://www.nopss.gov.cn/n1/2018/1203/c410171-30439332. html (accessed 2022/8/25). The project is funded by the National Office for Philosophy and Social Sciences (NOPSS) of China.

11

An Overview of Intermedial Studies in China

241

New Perspectives. In 2008, Vol. 1 of the Narrative Series edited by Fu Xiuyan published the conference proceedings, with six articles related to intermediality, namely, Li Li’s 李立 “Spatial Narrative of Fu6 and Painting in Han Dynasty,” Rabinowitz’s “Western Music and Narrative Theory,” Ye Qing’s 叶青 “Shaping of the Tradition of Chinese Pictorial Narrative,” Yu Yue’s 余悦 “Narrative of Chinese Tea Art and Its Academic Significance,” and Wang i’s 王琦 “Narrative of Film and Video Commercials.” In 2008, Jiangxi Academy of Social Sciences organized the first symposium on the theme of transmedial narrative, which consisted of five topics: (1). Chinese narrative tradition in intermedial perspective; (2). transmedial narrative and a general narratology; (3). andesstreben in narrative; (4). narrative in transmedial communication and transformation; and (5). intertextual narrative. The concise and insightful summary of the symposium underscores the significance of intermedial studies in the conclusion, “Many scholars believe that a transmedia trend has emerged in current narrative studies, which reflects the broadening horizon and the deepening exploration of narratology” (Long 2008:66). In 2009, Vol. 2 of the Narrative Series edited by Fu published the selected conference papers, with 18 articles under six headings related to intermediality: (1) Chinese Narrative Tradition in Intermedial Perspective; (2) Transmedial Narrative and Development of Narratology; (3) Transmedial Narrative and Spatial Narrative; (4) Intertextual Narrative; (5) Film and Television Narrative; and (6) Transmedial Narrative: Characteristics and Others. The highlights of the volume include Fu Xiuyan’s research on the narrative and aesthetic power of Chinese ancient Bronze inscriptions, Wang Yafei’s 王亚菲 scrutiny on the performative narrative of Chinese opera, Ye Qing’s paper on “Paifang 牌坊7 narrative,” Hu Yingjing’s 胡颖 峰 paper on the intertextual rhetoric of narration in Chinese classical music poetry, and Ling Yu’s 凌逾 fantastic study of Lust, Caution exemplifying the intermedial interplay of novel and film. When we search for publications on transmedial narrative from CNKI, we can find that the publications can be classified into several categories: general studies on transmedial narrative, more specific studies on pictorial narrative, musical narrative, sonic narrative, and others. Long Diyong 龙迪勇 and Ling Yu are two scholars who produce more scholarship than others on the general studies of transmedia narrative in China. Long has been long engaged in the study of “spatial narrative” and has taken the lead in constructing the spatial narratology in China. As previous narrative studies

Fu 赋 often translated “rhapsody,” “prose poem” or “poetic exposition,” is a form of Chinese rhymed prose that was the dominant literary form during the Han dynasty (206 BC–AD 220). Fu is intermediary piece between poetry and prose in which a place, object, feeling, or other subjects are described and rhapsodized in elaborate details and from as many angles as possible. 7 A paifang, also known as a pailou, is a traditional style of Chinese architectural arch or gateway structure, often beautifully decorated, with the pillars usually painted in red, the beams decorated with intricate designs and Chinese calligraphy, and the roof covered with colored tiles, complete with mythical beasts. 6

242

R. Ou

have focused on the temporal dimension of narration, when the focus of research is placed on the spatial dimension, it actually involves cross-media narrative issues if examined from the medial perspective. Long’s article “The Spatial Representation of Temporal Narrative Media” (2007) examines the intermedial issue of how temporal medium of “words” can effectively represent “space.” The paper is a pioneering work in the study of transmedial narrative in China and produces the motivation for him to organize the First National Symposium on Transmedia Narratives in 2008. Since then, Long has presided over a number of related projects, including the NOPSS funded project “Transmedia Narrative Studies,” and published Spatial Narrative Studies (2014), Spatial Narratology (2015), Long Diyong’s Selected Essays (2019), and more than twenty papers, focusing on spatial and transmedia narrative studies. Long’s analysis of classical and modern literary and artistic phenomena, such as the image and text in Chinese history painting, Chinese traditional architecture and Zhanghui xiaoshuo8 of Ming and Qing Dynasties (2015), the pictorial representation of theatrical scenes in Han pictures (2018), the music narratives of Western novels (2018), the intermedia narratives of James Joyce’s fictions (2020), and the medial transformation in modern Chinese literature (2021), has led to the construction of his paradigm of cross-media narrative studies, and his main ideas are as follows. Long argues that although different media have different “narrative properties,” in the actual artistic creation exists the phenomenon of the so-called andersstreben, that is, a medium type seeks to imitate the features of another media type while keeping its own characteristics. According to Long, andersstreben constitutes one of the two theoretical pivots for cross-media narrative. The other one is what the French sociologist Gabriel Tarde proposes as “the laws of imitation,” especially the law of imitation of superiors by inferiors. Likewise, the superior medium of an era tends to become the object of imitation by other media as a “model” in emotional expression, narration, and argumentation, and this is precisely the raison d’etre for the existence of cross-media narratives (Long 2017:17–26). Long divides the cross-media narratives into three basic types. He argues that the most creative andersstreben occurs in the mutual imitation of temporal and spatial art, as temporal narratives like poetry and fiction intend to acquire the “spatial form” of plastic art or spatial art like painting and sculpture want to create literary and narrative “pictorial poetry.” The second type of cross-media narratives occur between spatial art and spatial art (i.e., painting sometimes strives for a sculptural three-dimensional effect when telling a story through a two-dimensional plane), and the third type occurs between temporal art and temporal art (i.e., some modern novels strive for a musical narrative) (Long 2019:190–1). To sum up, according to Long, andersstreben and its corresponding cross-media narratives constitute the “common factor” and theoretical basis for comparative studies of literature and other arts, often giving rise to the effects that cannot be

8 Zhanghui xiaoshuo 章回小说, novel in chapters, main format for long novels from the Ming onward, with each chapter headed by a summary couplet.

11

An Overview of Intermedial Studies in China

243

achieved by literature and art that merely adhere to the original single medium. Although the cross-media creation has existed since the ancient time, the conscious cross-media narratives are mainly a “modern” phenomenon: modern writers and artists who are truly original have always consciously or unconsciously engaged in andersstreben; that is, while maintaining their own specific media, they also use or release the “power of another medium” and engage in cross-media narratives. This is in fact the fundamental reason why modern literature and art are different from traditional ones and have their unique charms (Long 2019:194). Ling Yu is also a leading figure of transmedial narrative studies. Besides journal articles, Ling has published five books related to intermedial studies in three stages. Stage 1: Intermedial studies on one writer. Transmedial Narrative: New Ecology of Xi Xi’s Novels (2009) illuminates the stylistic innovations of Xi Xi 西西 (also known as Sai Sai), one of Hong Kong’s most beloved and prolific authors, focusing on the cinematic narration in East Side Story, the cross-reference of word and image in My City, narrating the unnarratable in Marvels of a Floating City, and readerinvolved interactive narration in Flying Carpet, noting how Xixi’s narrative breakthrough reflects the cutting-edge trend of literary transformation and sets creative examples for challenging the crisis of literature. The book also explores the limitations of her experiments. Stage 2: Intermedial studies on a collection of writers. The Cross-media Culture in Hong Kong (2015) covers Hong Kong literature and art in the 1980s and 1990s, exploring the intermedial connection between literature and radio, film, television, digital network, cyber symbols, cultural geography, and performing arts in the new media era. The book makes a thorough study of the history, the characteristics, the styles, and the significance of Hong Kong’s cross-media culture and highlights its uniqueness. Intermediality: Selected Reading of Hong Kong and Taiwan Narrative Works (2012) edited by Ling serves as a textbook for the courses of Hong Kong and Taiwan literature, with interesting selection and parallel reading of novel and film, like Liu Yichang’s novel Tête-bêche vs Wong Kar-wai’s film In the Mood for Love and Eileen Chang’s novella vs Ang Lee’s film Lust, Caution; food in literature, like Ye Si’s Postcolonial Food and Love; literature and dance, like Huang Biyun’s Bloody Carmen and flamenco; literature and opera, like Kenneth Hsien-yung Pai’s Dreaming in the Garden and the Kunqu opera The Peony Pavilion; and drama and film, like Stan Lai’s Secret Love for the Peach Blossom Spring. Stage 3: Intermedial studies of creation, circulation, and production. With the publication of Crossover Creativity (2018), Media Convergence: Literary CrossMedia Dissemination in the Cyber Era (2021), and Cross-Border Creativity in Conversations (2021), Ling explores the principles of intermedia creation and successful communication cases with an attempt to construct the intermedia cultural production system. With her keen vision and unique insights, the author intends to make it possible an interactive communication and collaboration between art, literature, and technology, between media and audience. In this stage, Ling expands her intermedial studies beyond the criticism of literature and art to the transmedia dissemination of literary works, the pedagogy, and the practice of the crossover cultural production.

244

R. Ou

Apart from publications, Ling has initiated and managed the WeChat public account kuajie jingwei 跨界经纬 (Cross-Boundary Web) for more than 5 years. It tweets an article every day. Up to December 12, 2021, there have been 2361 tweets. The platform consists of three sections: “Cross-border scholarship,” “Cross-border creation,” and “Chinese cross-border masters.” So far, this account has more than 10,000 fans and has gained a certain amount of attention and reputation. More and more teachers and students have benefit from the platform and jointly promoted the vigorous development of intermedial studies. Overlapped with cross-media and transmedial narrative studies, there are pictorial narrative, musical narrative, and sonic narrative studies. Pictorial narrative scholars are interested in exploring the image’s propensity to tell stories in the visual arts and religious rites. As Long develops his research paradigm from space narrative and pictorial narrative to cross-media narrative, there are overlaps of the general transmedial narrative studies and the specific pictorial narrative studies in his scholarship; the latter includes his study of narrativity in Chinese history painting, Chinese traditional architecture (2015), and the pictorial representation of theatrical scene in Han pictures (2018) as mentioned above. Apart from Long, Wang Kexiang 王克祥 is another leading scholar on pictorial narrative in China. Wang’s scholarship involves the study of the narrativity of woodcut New Year pictures (2016), the pictorial narrative in digital display design (2017), the aesthetic analysis of pictorial narrative in Kunqu opera (2018c), and his narrative reading of the pictorials of the Republican period of China (1912–1949) (2018b) and Illustrated opera books of Ming Dynasty (2018a). Wang perfects his pictorial reading of Chinese traditional cultural heritage in the book Tushuo Huaying 图说画映 (2018) that he co-authors with Guo Jianan 郭佳楠 (Wang & Guo 2018). Other scholars’ wonderful essays include the narrative approach to the paper cut (Xia 2021), the film poster (Hu 2021), the comics (Zhu 2021), the documentary (Wang and Chen 2021), and Naxi ethnic’s religious worship (Yang 2021), to name a few. The musical narrative scholarship can be classified into three categories: narrative in music, musical narrative in fiction, and musical narrative in film, theater, and TV dramas. Wang Xuqing 王旭青 is the leading musical narrative scholar in China in the field of music studies. With four monographs – Richard Strauss’s Symphonic Poems: Context, Program, Musical Narrative (2010), Saint-Saëns’ Symphonic Poems (2011), Art of Words: Introduction to Theory of Musical Narratives (2013), and Analysis, Narration, Rhetoric: Essays on Music Theory (2018) – Wang’s scholarship involves modern theories of musical narrative and addresses whether and how to use narrative as a means to musical understanding, whether as musicians or nonmusicians. Wang takes into account the music-theoretical and music-historical sides of narrative and analysis and criticism, along with the tradition of semiotics, giving Chinese musicians and scholars a thorough and balanced view of musical narrative theory in the West for the past few decades and developing her own independent ideas, which serve as basis of her strong interpretation of works from Chopin to John Corigliano. Her journal article “Anti-narrative, Non-narrative, Neo-narrative: Popular Terms in Contemporary Musical Narrative Research” (2014) clarifies three

11

An Overview of Intermedial Studies in China

245

concepts and exemplifies their viability in musical analysis. Apart from narrative approach to instrumental music, some scholars are concerned with the musical narrative in opera (Dong 2015; Li 2015) and in dance (Xu 2014); Chen Dezhi notes the “anxiety” of musical narrative from the “influence” of literary narratology (2014). In terms of musical narratives in fiction, some scholars conduct comprehensive studies on the musicality of various aspects of the work, such as Chen Danling’s 陈 丹玲 analysis of musical narrative in Eileen Chang’s 1940s fiction (2016) and Liu Xiaorong’s 刘晓榕 research on musical narrative in Kazuo Ishiguro’s fiction (2019). Some focus on how musical representation helps with characterization and the unfolding of the plot, such as Wang Xizhong’s 王希翀 study of Hardy’s fiction (2020) and Chen Yan’s 陈妍 interest in the musical narrative in The Dream of Red Chamber (2018). Some link the structure of the fiction to music, with an emphasis on the construction of the novel’s framework (Liu 2006), and some note the influence of blues on African American fiction (Lin 2015). In terms of musical narrative in film, Zhao Jiang 赵江 makes a book-length study of the interwoven relationship between film music and film narrative (2020). Du Juan 杜娟 in her essay (2019) explores the correlation between the development of film art and the promotion of the status of music and analyzes the diverse functions of musical narrative in film. Other scholars conduct case studies of musical narrative in specific films, noting the effect of music on unfolding the story, intensifying the atmosphere, and enriching the characterization. The Musical Narrative of TV Dramas (2009) by Wu Aifang 吴爱芳 approaches the narrative of TV drama music, considering that TV drama is a “narrative” art and the purpose of music in TV drama is to serve the narrative of TV drama. This book focuses on the narrative characteristics, narrative function, and narrative techniques of TV drama music and the structural layout of TV drama music narratives. Overlapping with cross-media narrative and musical narrative are sound narrative/sonic narrative studies, which focus on the narrative potential in audial representation in literature. Fu Xiuyan is one of the pioneering and leading scholars in China to have committed to the auditory narrative/acoustic narrative. He proposes in “A Preliminary Study of Acoustic Narrative” (2013) that the introduction of the concept of acoustic narrative into the area of narratology is closely linked with the corrective efforts in dealing with the crisis of sensory culture in modern life. In an era of “picture reading” with heavy reliance on visual perception, other sensory channels are largely neglected. His study of acoustic narrative revisits the significance of senses in literary narrative with a special emphasis on the implication of acoustic space and feelings (as the underlying structure of literature). The analysis of acoustic narrative employs the sonic vocabulary proposed by the author: lincha (observation by hearing) instead of guancha (observation by seeing) and yinjing (soundscape) in place of tujing (landscape). According to Fu, imitation and imagination are two forms of acoustic narration by means of and synesthesia. Onomatopoeia refers to the naming of a thing (or action) by a vocal imitation of the sound, the use of which suggests the importance of sound event in narrative, while synesthesia in the

246

R. Ou

narrative of visual and auditory experiences refers to perceptions that cross over the senses of hearing and sight (the ear for the eye and hearing shapes). The experience of hearing shapes, though developing from onomatopoeia (the analogy between sounds), is more effective in the evocation of imagination due to the interplay of senses. One of the major tasks of acoustic narrative studies is to revisit the classics through the perspective of acoustic experience. The emphasis on the multiplicity and uniformity of the senses will, according to Fu, return us to the primary function of language as a technological form of senses. Fu (2017) notes three types of uncertain auditory perception in narration: auditory hallucination, psychic hearing, and overhearing in narrative works. He argues that these three types of uncertain “hearing” stand opposite to authenticity, possibility, and integrity, respectively. As far as the degree of uncertainty is concerned, auditory hallucination exceeds psychic hearing while psychic hearing surpasses overhearing. Uncertain perception will lead to uncertain expression, but the blurred and dreamy auditory events can often enrich the connotations of a text so as to stretch the readers’ imagination and provide more food for their thought. Moreover, uncertain “hearing” of this kind can also provide motivation for the beginning, development, and turn of a story as well as add finishing touches to highlight the characters and the theme of a story. Together with his scrutiny on “soundscape” (2015) as well as the narrative production of auditory space (2020), Fu formulates his Auditory Narratology (2021). He defines “auditory narration” as “the expression and writing of narrative works in relation to auditory perception” and claims that, however, many new means of storytelling have been made available by media changes, they are not essentially free from the imitation of oral narrative. The blunting of human hearing is linked to the increasing noise of our surroundings, and the use of auditory events of all kinds is rarely seen in today’s narratives, nor do critics seem to sense a shrinkage or regression in this respect. The lack of perceptual experience leads to the lack of expression, which calls for the innovation of discursive tools and the introduction into the field of narratology of the concepts and terms closely related to the sense of hearing such as lingcha (auscultation) and soundscape. The study of “the uniqueness of voice” requires a departure from the metaphysical research paradigm to track the phonological remnants of written texts by means of close reading and restore the “on-the-spot life” of the text from an auditory perspective. Fu attaches great importance to the close reading of texts and criticizes the tendency “to talk about theory in isolation from texts and literature”: The study of aural narratives has, of course, also emerged from the “aural turn” in the humanities, but this “turn” is a shift from a “listening focus” to an “audio-visual focus”, not a shift away from literature itself. It is important to note that some studies are indeed leaving literature and focusing on the study of sound and hearing, a technical, acoustic rather than literary approach, which is not desirable. (Fu 2021:96)

In addition to original researches on transmedia/cross-media narratives by Chinese scholars, the Chinese translation and publication of the Western scholarship

11

An Overview of Intermedial Studies in China

247

since 2010s has helped to advance Chinese scholarship, such as Jacob Lothe’s Narrative in Fiction and Film (trans. Xu Qiang, 2011), Henry Jenkins’ Convergence Culture: Where Old and New Media Collide (trans. Du Yongming, 2012), MarieLaure Ryan’s Avatars of Story (trans. Zhang Xinjun, 2014), Narrative Across the Media edited by Marie-Laure Ryan (trans. Zhang Xinjun et al. 2019), Elleström’s The Modalites of Media (Trans. Chen Jun & Ou Rong, 2023), and Jørgen Bruhn’s The Intermediality in Narrative Literature (trans. Xu Changsheng & Ou Rong, 2022). Last but not the least, Zhao Yiheng’s 赵毅衡 pioneering and extended research on what he terms as “a general narratology” has provided theoretical framework and methodology for transmedial narrative studies for some scholars in China. Zhao put forward his proposition of a general narratology on 2008 Transmedia Narrative Symposium and published a series of articles (2008, 2010, 2013) and a monograph (2013) to illuminate his theory. At the beginning of his book entitled A General Narratology, Zhao defines the term as “General narratology interprets the common rules of all narrative genres” (Zhao 2013:1) and makes an elaborate classification of narrative texts in Table 1. In Zhao’s mind, the text communicates between the narrator and the narratee, and there is an intentional relationship between the two sides. Zhao points out that narratives could be classified into documenting narrative, performing narrative, and conative narrative in terms of textual intentionality and temporal directionality: What distinguishes these three narrative texts lies in temporal directionalities of textual intentionality: the past directionality that emphasizes on the documenting is declarative; the present directionality that focuses on the performing is interrogative; the future directionality that directs to an admonition is imperative. (Zhao 2013:36) Table 1 Classification of narrative texts (Zhao 2013:1) Temporal directionality Past

Past/present

Present

Present-like

Future

Appropriate medium Documenting narrative: writing, speech, image, sculpture Documenting and performing: film and digital recording Performing: body, video, material, speech, Performing-like: mental image, perception, and message Conative narrative: any medium

Factual narrative History, biography, news report, diary, confession, legal defense, history mural

Fictional narrative Novel, narrative poetry, narrative song lyrics

Documentary, TV interview

Drama, audio, and video recording of the performance Theater, contest, games, video games

Live broadcasting, live speech

Mental transmission

Advertisement, promise, fortune-telling, prediction, swear

Dream, hallucination

248

R. Ou

Zhao’s discussion of textual intentionality incorporates performing narrative and conative narrative into the narratological study, thus bridging narrative study with intermedial study: Performing narrative refers to the semiotic text that makes use of a ‘non-specific’ medium to tell the story. Media used for the performing include the body, speech, music, sound, image, shadow, etc. There are many types of performing narrative, among which the largest is ‘present continuous’ performance, including drama, dance, singing, concert, magic, exhibitions, speeches, oral dictation, sandbox deductions, ceremonies, etc. Performing narrative also comprises ‘competitive’ narrative, such as game, gambling, duel, etc. that are held for gaining victory, and ‘game-related’ narrative, i.e., games or video games with no purpose or only a virtual purpose (such as pleasure and the score). (Ibid., 38)

Zhao’s innovative exploration and broad thinking make narratology go beyond the literary studies and open to all genres and bring it new vitality, as Rao Guangxiang 饶广祥 puts in his review of Zhao’s book: The book is an attempt to embrace the challenges brought by the age of “post-truth,” when narratives have become our life, built by various media, including mass media, digital media, interactive media, virtual media, etc. Our selves are constructed by the multi-media narratives, so are our worlds of meaning. In this sense, A General Narratology reveals to us how we live semiotically in this world, with the most important ability of human beings, that is, the ability to narrate, to tell stories. (Rao 2018:563)

Conclusion The author has reviewed six areas of intermedial studies in China; nevertheless, as the boundaries are always constructed, the six areas are not completely divided but overlapped to some extent. As China is a big country, this chapter mainly covers the interart/intermedial scholarship of Chinese mainland. Due to the various limitations, the author is unable to cover all aspects of the topic and can only provide a general picture of Chinese scholarship of intermedial studies. Compared with the international scholarship, the English publications by Chinese scholars are much less, and there are more Chinese translations of English studies rather than the other way around. As English publications by Chinese scholars on intermedial studies are far from sufficient, the author intends for this chapter to serve as a window for international readers to have a survey of China intermedial landscape.

References 陈悖 Chen, Bei, 刘象愚 Liu, Xiangyu, 1988. 比较文学概论 (Introduction to Comparative Literature). 北京: 北京师范大学出版社. 陈德志 Chen, Dezhi. 2014. “影响与焦虑:文学叙事学与音乐叙事学的关系” (Influence and Anxiety:The Relationship between Literary Narratology and Musical Narratology). 南昌航空 大学学报(社会科学版) 1: 106–112.

11

An Overview of Intermedial Studies in China

249

陈明 Chen, Ming. 2016. “古代东方文学的图像传统初探” (Study on the Image Tradition of Ancient Eastern Literature). 国外文学 1: 36–46. 董蓉 Dong, Rong. 2015 “毒剂效力:威尔第歌剧《奥赛罗》中的音乐叙事” (The Power of Poison”: Musical Story Telling in Verdi’s Otello). 南京艺术学院学报(音乐与表演), 2015(03): 89–95. 冯川 Fen, Chuan. 2003. 文学与心理学 (Literature and Psychology). 成都: 四川人民出版社. 傅修延 Fu, Xiuyan. 2013. “听觉叙事初探” (A Preliminary Study of Acoustic Narrative). 江西社 会科学 2: 220–231. ———. 2015. “论音景” (On Soundscape). 外国文学研究 5: 59–69. ———. 2021. “听觉叙事研究的缘起、话语创新与范式转换” (The Origin, Discourse Innovation and Paradigm Shift of the Study of Auditory Narratology). 中国文学批评 4: 89–97. ———. 2020. “叙事与听觉空间的生产” (Narrative and the Production of Auditory Space). 北京 师范大学学报 4: 89–98. ———. 2017. “幻听、灵听与偶听——试论叙事中三类不确定的听觉感” (Auditory Hallucination, Psychic Hearing and Overhearing: On Three Types of Uncertain Auditory Perception in Narration). 思想战线 3: 99–110. 蒋承勇 Jiang, Chengyong. 2021. “科学与文学理念之现代性转型——现实主义‘写实’特质成 因考论” (Science and the Modern Transition of Idea of Literature: On the Causes of “Realistic” Traits of Realism). 社会科学 11: 171–178. 何成洲 He, Chengzhou. 2019. “跨媒介研究的问题意识” (Problem Awareness in Intermedial Studies). 艺术理论与艺术史学刊 2: 9–12. ———. 2020. “跨媒介视野下的’戏剧-小说’研究” (“Theatre-Fiction” from the Perspective of Intermediality). 南京师范大学文学院学报 4: 33–42. 胡华芳 Hu, Huafang. 2021. “电影海报的图像叙事及其美学意蕴” (Pictorial Narration and Aesthetic Implications of Film Posters). 江西社会科学 7: 240–246. 李宏 Li, Hong. 2003. “瓦萨里《名人传》中的艺格敷词及其传统渊源” (Ekphrasis and Its Origins and Tradition in Vasari’s Lives). 新美术 3: 34–45. 李晓畅 Li, Xiaochang. 2015. “民族歌剧《木兰诗篇》的音乐叙事及其社会价值研究” (Musical narration and Social Value of National Opera Mulan) [Diss.]. 西南大学, 2015. 李军 Li, Jun. 2016. 可视的艺术史:从教堂到博物馆 (A Visible History of Art: From Church to Museum). 北京: 北京大学出版社. ———. 2020. 跨文化的艺术史:图像及其重影 (A Transcultural History of Art: On Images and Their Doubles). 北京: 北京大学出版社. ———. 2022. “‘纹’与‘文’——一个跨媒介艺术史研究的案例” (“Texture”and“Text”: A Case Study of Intermedial Art History). 艺术管理 1: 35–45. 林晓彬 Lin, Xiaobin. 2015. “非裔文学独特的音乐叙事风格研究——以布鲁斯音乐风格为例” (A Study on the Unique Musical Narrative of African-American Literature: Taking the Blues Style as an Example). 福州大学学报(哲学社会科学版) 3: 90–93. 凌逾 Ling, Yu. 2009. 跨媒介叙事:论西西小说新生态 (Transmedial Narrative: New Ecology of Xi Xi’s Novels). 北京: 人民出版社. ———. 2015. 跨媒介香港 (The Cross-media Culture in Hong Kong). 北京: 社会科学文献 出版社. ———. 2018. 跨界网 (Crossover Creativity). 北京: 中国社会科学出版社. 刘纪蕙 Joyce C.H. Liu. 1994. 文学与艺术八论:互文•对位•文化诠释 (Eight Treatises on Literature and Art: Intertextuality, Counterpoint and Cultural Interpretation). 台北: 三民书局. ———. 1996. “故宫博物院 vs 超现实拼贴:台湾现代读画诗中两种文化认同之建构模式” (Palace Museum vs. Surrealist Collage: Two Modes of Cultural Identity Construction in Modern Taiwan Ekphrasis Poetry). 中外文学 7: 66–96. 刘春勇 Liu, Chunyong. 2006. “《许三观卖血记》的音乐叙事” (Musical Narration of The Legend of Xu Sanguan’s Blood-selling). 广播电视大学学报(哲学社会科学版) 4: 13–16. 卢康华 Lu, Kanghua, 孙景尧 Sun, Jingyao. 1984. 比较文学导论 (Introduction to Comparative Literature). 哈尔滨: 黑龙江人民出版社.

250

R. Ou

龙迪勇 Long, Diyong. 2008. “叙事学研究的跨媒介趋势——‘跨媒介叙事’学术研讨会综述” (The Cross-media Tendency in Narratological Studies: A Summary of the Seminar on “CrossMedia Narrative”). 江西社会科学 8: 57–66. ———. 2017. “模仿律与跨媒介叙事–试论图像叙事对语词叙事的模仿” (The Imitation to the Words of Narrative by the Image Narrative). 学术论坛 2: 13–27. ———. 2019. “ ‘出位之思’与跨媒介叙事” (“Andersstreben” and Cross-media Narrative). 文艺 理论研究 3: 184196. 罗立刚 Luo, Ligang. 1999. 宋元之际的哲学与文学 (Philosophy and Literature of Song & Yuan Period). 上海: 复旦大学出版社. 穆宏燕 Mu, Hongxian. 2017. “摩尼教经书插图的艺术史意义” (Significance of Illustrated Manichaean Canons as History of Art). 美术研究 4: 62–67. 欧荣 Ou, Rong. 2013. “说不尽的《七湖诗章》和‘艺格符换’” (The “Seven Lakes Canto”, Ekphrasis, and So on). 英美文学研究论丛 1:229–249. ———. 2020. “语词博物馆:当代欧美跨艺术诗学概述” (Museum of Words: A Survey of Contemporary European and American Interart Poetics). 上海交通大学学报(哲学社会科学版) 4: 126–137. ———. 2021. Intermedial Studies: An Interview with Lars Elleström. 外国文学研究 2: 15–30. ———. 2018. 恶之花:英美现代派诗歌中的城市书写 (Les Fleurs du Mal: Urban Writing in Anglo-American Modern Poetry). 北京 :北京大学出版社. ———. 2022. 语词博物馆:欧美跨艺术诗学研究 (Museum of Words: European and American Interart Poetics). 北京: 北京大学出版社. 钱锺书 Qian, Zhongshu. 2002. 七缀集 (Patchwork: Seven Essays on Art and Literature). 北京: 三 联书店. 钱兆明 Qian, Zhaoming. 2003. The Modernist Response to Chinese Art: Pound, More, Stevens. Charlottesville: University of Virginia Press. 裘禾敏 Qiu Hemin. 2017. “《图像理论》 核心术语 ekphrasis 汉译探究” (Chinese Translation of the Key Concept of Ekphrasis in Picture Theory). 中国翻译 2: 87–92. 饶广祥 Rao, Guangxiang. 2018. A general narratology, by Henry Yiheng Zhao. Social Semiotics 4: 559–554. 谭桂林 Tan, Guilin. 2002. 百年文学与宗教 (Religion and Chinese Modern Literature in the 20th Century). 长沙: 湖南教育出版社. 王克祥 Wang, Kexiang. 2018a. “明代戏曲插图本的叙事系统——基于文字叙事和图像叙事的 研究” (Narrative System of the Drama Illustrated Edition in Ming Dynasty). 美术教育研究 23: 1 ———. 2018b. “图像叙事——民国画报中的‘抗日战争’” (Pictorial Narrative: Anti-Japanese War in the Pictorials of the Republican Period). 大众文艺 23: 78–79. ———. 2018c. “昆曲艺术中的图像叙事美学分析——以《西厢记》 为例” (Pictorial Narrative Aesthetics in Kunqu Opera: A Case Study of “Romance of the Western Chamber”). 戏剧之家 32: 910. ———. 2017. “数字展示设计中的图像叙事研究” (Pictorial Narrative in Digital Display Design). 艺术科技 8: 48. 王克祥 Wang, Kexiang, 郭佳楠 Guo Jianan. 2018. 图说画映 (Tales of Images). 哈尔滨: 东北林 业大学出版社. 王向远 Wang, Xiangyuan. 2002. 比较文学学科新论 (New Theory of Comparative Literature). 南 昌: 江西教育出版社. 王旭青 Wang, Xuqing. 2014. “反叙事
无叙事
新叙事:当代西方音乐叙事研究中的热门术语” (Anti-narrative, Non-narrative, Neo-narrative: Popular Terms in Contemporary Musical Narrative Research). 美育学刊 4: 90–94. ———. 2013. 言说的艺术: 音乐叙事理论导论 (Art of Words: Introduction to Theory of Musical Narratives). 北京: 人民音乐出版社. ———. 2018. 分析
叙事
修辞音乐理论研究论稿 (Analysis, Narration, Rhetoric: Essays on Music Theory). 上海: 上海三联书店. 汪振军 Wang, Zhenjun & 陈梦 Chen, Meng. 2021. “传统美学“赋比兴”与图像叙事中的语言构 境——以《匠人
匠心》系列纪录片为例” (Image Narrative of Documentary Series Craftsman
Ingenuity). 新闻爱好者 8: 70–72.

11

An Overview of Intermedial Studies in China

251

夏金凤 Xia, Jinfeng. 2021.“徐州剪纸的图像叙事功能初探” (A Preliminary Study on the Pictorial Narrative of Xuzhou Paper-cut). 戏剧之家 2: 187–188. 徐慧琳. Xu, Huilin. 2014. “叙事舞蹈作品中的叙事话语研究 (Narrative Discourse Study of Narrative Dance Works) [Diss.]. 东北师范大学. 杨骊 Yang, Li, 叶舒宪 Ye, Shuxian. 2019. 四重证据法研究 (Quadruple Evidence Method). 上海: 复旦大学出版社. 杨乃乔 Yang, Naiqiong, ed. 2020. 中西文学艺术思潮及跨界思考 (Chinese and Western Literature and Art Trends and Trans-Boundary Thinking: A Dialogue between Literature and Music, Fine Arts, Music, Theater and Film), 上海: 复旦大学出版社. 杨杰宏 Yang, Jiehong. 2021. 纳西族日月崇拜的图像叙事研究 (Research on Image Narration of Sun and Moon Worship of Naxi Nationality). 民族艺术研究 4: 152–160. 杨周翰 Yang, Zhouhan. 1989. “Preface”, in 乐戴云, 王宁 (Yue, Daiyun & Wang, Ning) eds. 超学 科比较文学研究 (Transdisciplinary Studies of Comparative Literature). 北京: 中国社会科学 出版社, 12. 叶舒宪 Ye, Shuxian. 2018a. 文学与人类学 (Literature and Anthropology). 西安:陕西师范大学 出版总社. ———. 2018b. 文学人类学探索 (Exploration of Literary Anthropology). 西安:陕西师范大学出 版总社. ———. 2019. 玉石神话信仰与华夏精神 (Jade Myth Belief and Huaxia Spirit). 上海: 复旦大学 出版社. ———. 2020. “物证优先:四重证据法与’玉成中国三部曲’” (Material Evidence Priority: Quadruple Evidence Method and the “Jade-Making China Trilogy”). 国际比较文学 3:415-437. ———. 2022. A Mythological Approach to Exploring the Origins of Chinese Civilization, trans. Hui Jia and Jing Hua. Singapore: Springer Nature. 乐黛云 Yue, Daiyu. 1988. 比较文学原理 (Principles of Comparative Literature). 长沙:湖南文艺 出版社. 乐戴云 Yue, Daiyun, 王宁Wang, Ning. eds. 1989. 超学科比较文学研究 (Transdisciplinary Studies of Comparative Literature). 北京: 中国社会科学出版社. 赵晋超 Zhao, Jinchao. 2018. Wonders of Book of Surprises: An Examination of the Depictions of the Architecture in a Jalayirid Manuscript and its Two Ottoman Copies. 欧亚学刊 7: 211–226. 赵宪章 Zhao, Xianzhang. 2021. Text – Image Theory: Comparative Semiotic Studies on Chinese Traditional Literature and Arts. Genzano di Roma: Aracne. 赵毅衡 Zhao, Yiheng. 2013. 广义叙述学 (A General Narratology). 成都: 四川大学出版社. 周计武 Zhou, Jiwu. 2020a. “艺术的跨媒介性与艺术学理论的跨媒介建构” (The Intermediality of Arts and the Intermedial Construction of Art Theory). 江海学刊 2: 210–218. 周宪 Zhou, Xian. 2019. “跨媒介性与艺术理论学科” (Intermediality and the Discipline of Art Theory). 艺术理论与艺术史学刊 2: 4–8. ———. 2020b. “作为艺术理论方法论的跨媒介性” (Intermediality as a Methodology of Art Theory). 江海学刊 2: 202–209. 朱光潜 Zhu, Guangqian. 2013. 诗论 (On Poetry), 北京:中华书局. ———. 1980. “无言之美” (Beauty of Silence), in 朱光潜美学文学论文选集 (Selected Aesthetic and Literary Essays of Zhu Guangqian), 长沙:湖南人民出版社, 347–358. 朱逸宁 Zhu, Yining. 2021. “从插图到连环画—《三国演义》两种图像艺术形态的比较” (From Illustrations to Comic Strips: A Comparison of Two Visual Art Forms in The Romance of the Three Kingdoms). 江苏第二师范学院学报 6: 15–22. 宗白华 Zong, Baihua. 1981. 美学散步 (Aesthetic Walks). 上海: 上海人民文学出版社.

Intermediality, Semiotics, and Media Theory

12

Toma´š Chudy´ and Richard Mu¨ller

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Technological Subconscious of Semiotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Asynchronous Confluences in the Formation of Saussure’s Semiology . . . . . . . . . . . . . . . . . . . Semiosis, Interpretation, and Machine: The Peircean Way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jakobson’s Mixed Heritage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eco’s Advances and Sidesteps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Emergence of Media Difference in Film and Cultural Semiotics (Metz, Lotman) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Semiotic Subconscious of the Technical Media Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Three Aspects of “Sign” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Human and the Machinist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Kittlerian Turn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Symbolic, Sign, and Signal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Narrative Plea for Engineers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wooing and Repudiating Semiotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How Should We Really Regard This Narrative? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toward a Media-Semiotic Convergence (with a Detour) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality as a Meeting Ground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Canonical Phase: A Reversal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Process of Media Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Impulse of Technical Functionalities (and Its/Their Limits) . . . . . . . . . . . . . . . . . . . . . . . . . . . Media Qualification and “Laterality” of Technics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

254 256 256 258 260 261 262 264 264 265 266 269 270 271 273 274 276 276 277 278 279 280 282

T. Chudý (*) Czech National Bank, Prague, Czech Republic R. Müller (*) Institute of Czech Literature of the CAS, Prague, Czech Republic e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_18

253

254

T. Chudy´ and R. Mu¨ller

Abstract

The chapter explores two complementary perspectives: the lack of an explicit focus within semiotics on media technology and a converse lack of semiotic awareness in the classic media theory. The slow genesis of the medium concept clearly affected the formation of modern semiotics with Peirce and Saussure. It is shown that within this process, the technological aspect constituted the “place of strongest resistance” and to what extent the subsequent developments in semiotics have been marked by this initial condition. Technological media science, best represented by Kittler, is then shown to develop out of a blindness to the semiotic implications of its own premises. Even if it allows us to see how new ways of signification come to exist as a result of technical inventions, this insight is preconditioned by a gesture of erasure of semiotic difference. It appears truly ironic that Kittler’s writing is still so semiotically rich. In the field of intermediality, we can observe an initial reversal in weighing the importance of the cognitive vs the technical dimension of the medium. The consequence of this is that intermedial research has tended to lose sight of the internal logic of media evolution. What has not yet been resolved is how the technical condition should be conceptualized if it contributes to the “qualification” of the medium. It is argued that both semiotics and technics should be seen as converging on a common theoretical terrain, in a complicated, chiasmatic relationship. Technics and semiotics need to be studied together. Keywords

Semiotics · Technical media · de Saussure · Peirce · Kittler · Sign

Introduction How should we assess the semiotic approach to intermediality? The answer is less obvious than we might think. One way is to regard it as a backdrop against which much of intermedial theory and practice makes sense: Individual arts can be seen as semiotic systems, and the translation between them is essentially configured by the character of the signs in question. Is such perspective complete, however? After all, one may ask: signs as opposed to what? Here, the concept of data can be used contrastingly. Intermediality would soon run dry if it derived its interest just from translating data streams between different signalling systems instead of signs.1 It 1

Even the data-transformative branches of art, called mapping (Simanowski 2006: 69–76), clearly use signifying sets of data, and so qualify by all means as signs in the precise semiotic sense, e.g., human dance steps translated into music, the distribution of Jewish dwellings before WWII on museum grounds, the stock price of a corporation, and pixel-representation of bits on the CNN website – all of which work only by focusing on their semantic context. From the semiotic standpoint, even the postmodern mapping that aestheticizes trivial data appears symptomatic both of the growing tendency toward datafication and data analytics and theiconic turn.

12

Intermediality, Semiotics, and Media Theory

255

then seems safe to say that data should and must be able to fall under the category of the sign in order to come within the remit of suitable objects of art. This also suggests, however, a need for a reconceptualization of the relationship between semiotics and those fields and respective methods which evolved into today’s media, data, and information science. On the following pages, two complementary perspectives are explored: One concerns the lack of an explicit focus within semiotics on media technology, and the other, a converse lack of semiotic awareness in the classic media theory. However, as will be shown, both semiotics and technics should be seen today as contributing to and converging on a common theoretical terrain. The inquiry into the genesis of the media concept by John Guillory (2010) reveals that “the concept of a medium of communication was absent but wanted for the several centuries prior to its appearance, a lacuna in the philosophical tradition that exerted a distinctive pressure, as if from the future, on early efforts to theorize communication. [. . .] The emergence of the media concept in the later nineteenth century was a response to the proliferation of new technical media – such as the telegraph and phonograph – that could not be assimilated to the older system of the arts. [The] development of new technical media perplexed thereafter the relation between the traditional arts and media of any kind” (Guillory 2010: 321–322). When we look at the place of technics and technology in the modern semiotic discourse beginning with Charles S. Peirce and Ferdinand de Saussure, we find a corollary silence – a latency of technological motifs, which appear in their works in passing and without systematic scrutiny. As we will show, this latency not only marks the emergence of modern semiotics from the realms of philosophical, logical, and linguistic thinking but also the subsequent development of the discipline. Even more interestingly, the modern discourse on technics (beginning with the Hegelian Ernst Kapp and perhaps best exemplified by Friedrich Kittler) remains largely uninterested in the semiotic side of the problem – namely, that different technologies are entangled with different syntactic and semantic regimes of signification as well as different signifying practices. Thus, the latency of the media concept in the humanities up to the point when media studies is formed with Marshall McLuhan is related to the heritage of the mutual blindness between semiotic and technological thought and objects. The questions that ensue from our double perspective are the following: If the foundations of semiotic thinking lie in nontechnical systems, or systems where the presence of technics seems incidental, what consequences does this have for the junction between semiotics and theories of media and intermediality? What should be the place of the concept of technics in semiotics, and conversely, the concept of semiotic systems in the theory of technics and media theory? To what extent does the technical setup of the sign affect its meaning and reception? Has semiotic use been latent at the core of reflections on technical media? Is there a genealogy of the technical condition hidden within the history of modern semiotics?

256

T. Chudy´ and R. Mu¨ller

The Technological Subconscious of Semiotics Asynchronous Confluences in the Formation of Saussure’s Semiology As Guillory writes, “[i]t has not escaped anyone’s notice that linguistics turned increasingly in the twentieth century to the scene of communication and to the task of modeling this scene. Saussure’s inaugural Course in General Linguistics depicts communication in its starkest form, as two talking heads whose mouths, ears, and brains are linked together by lines composed of dots and dashes. However firmly this picture insists on the speech scenario, its slackly suspended lines hint at the telegraph or the telephone, a visual pun that Saussure probably did not intend” (Guillory 2010: 349). The character of the communication sign, the relationships between the signifier and the signified as well as the gap between the phonic substance and “the nebula of thought” (Saussure), is explained by a whole “troop of medial metaphors” (ibid.: 350). “Saussure invokes the conventional, Lockean thesis – ‘The value of a word is mainly or primarily thought of in terms of its capacity for representing a certain idea’ (CGL, p. 112) – in order to refute it. His theory of the signifier and signified as a composite articulation asserts to the contrary an indissoluble or constitutive link between these two elements of the sign. The articulating function is different from representation and is expressed in Saussure’s analysis by a series of figures: the action of wind producing waves on water, scissors cutting through the recto and verso of paper, and the coin as medium of exchange. [. . .] [W]e can register the extent to which an unstated concept of medium governs the figures, a small troop of medial metaphors conscripted to fend off the model of representation” (Guillory 2010: 350). In modelling communication, however, Saussurean theory includes some cybernetic principles avant la lettre as well: the connection between communication sides acquires a circular form, where the receiving and sending positions change places. The emphasis is not on multidirectional interaction (as in the post-WWII communication theory of Gregory Bateson) but on systemic reciprocity (as shown by the diagram on p. 11), where hearing and phonation constitute the process of the sign’s formation between the acoustic image (signifier) and concept (signified). As Gerold Ungeheurer (2004) observed, motifs reminiscent of cybernetics can also be found in Karl Bühler’s work (as early as his Krise der Psychologie). Bühler’s “Organon model” – a model of communication which Roman Jakobson, Jan Mukařovský, and the Prague School reworked in the 1930s to include art2 – featured three communication functions defined by a focus on (1) sender, (2) receiver, and (3) objects and states of affairs (Bühler 2011/1934). Every point on this “triangle” is as capable as the other two of initiating signification. Language sound signals become signs via their triple relationship, the three communication functions.3 We

Jakobson’s and Mukařovský’s crucial modification was to add the “fourth” function: the aesthetic function, which pertained to the sign itself (Mukařovský 1976/1936). 3 The diagram displaces writing (again, see below on Saussure) by assuming that the sign paradigmatically acquires the sound form. 2

12

Intermediality, Semiotics, and Media Theory

257

might translate this idea into a more distinctly cybernetics metalanguage by saying that the semiotic situation connects the transmitter, receiver, and world in a mutual communication circuit. Thus, we can see a prefiguring of circular communication processes as well as a contradiction (in Saussure) between the assumptions of the face-to-face model, on the one hand, and (displaced) motifs of technical transmission and distance – which any medium requires – on the other. How far, however, does the recognition that signs are constituted of signals in communication lead in the direction of media awareness? To what extent is this insight preconditioned by pushing the presence of the technical and media issues to the theory’s subconscious? How does this affect media thinking? As stated, among Saussure’s points of departure is the rejection of the Lockean representational paradigm: words are not representations of ideas. The signifier and signified form differential planes whose interconnection in the course of signification cannot erase their fundamental nonidentity. The principle of differentiation says: the identity of the written form (such as t) is formed in all variations by its differentiation with respect to f, l, d, etc.; the phonological value is formed in opposition to its distinctive opposites (in Czech, for example, to ť, etc.). This is the very guiding principle of speech, and of the linguistic study of speech, where everything random and individual must be cast aside in favor of the necessary and social (Saussure 2011/1916: 11–17). Thus, the sign becomes independent of substance (ibid.: 11–14). The gap of nonidentity between phonic matter and the matter of ideas is too great to be erased, as the notion of arbitrariness explains. Conversely, the coupling of the signifier and the signified appears so natural the moment a linguistic sign is used that it takes considerable effort to realize their nonidentity, as expressed by the metaphors of paper and coin. What has escaped the attention of Saussure and his followers is this: It is precisely other media, using different substances and ways of fabricating signs, that make it possible to tear apart the signifier and the signified again, so to speak and give them the form of a different but comparable sign, in which a new differential unity takes shape. That Saussure’s linguistics brings with it this kind of latent medial implications can also be demonstrated by the fact that the different extent of the unity of signifier and signified in different languages – that analogous signs have different positions within different systems – is illustrated by his use of simple images (a tree, a dog). This evokes the comparability of another media semiotic system, whose distinctiveness nevertheless remains obscured. In brief, the lacuna in semiotic thought has lied for decades in its suppression of the media difference as a semiotic difference. An early exception can be seen in the art theory and aesthetics of the Prague School and later part of its classical period (1930s to early 1940s). It was not a particular media theory that informed its (still atrophied) media awareness but its interest in the artistic avantgarde and its tendency to expand the realm of materials. The representatives of the Prague School (Jan Mukařovský, Roman Jakobson, Jindřich Honzl) recognized the semiotic precondition of various art forms (film, architecture, photography, literature, theatre) to reside in their material differences and their common ground (as arts) to reside in the aesthetic function (cf. Veltruský 1976; Matejka and Titunik 1976; Chvatík 1970; Müller and Šidák 2020).

258

T. Chudy´ and R. Mu¨ller

The suppression of the media difference, nonetheless, played into the formation of semiotics in more ways than one. Saussure’s semiological project challenged the primacy of writing, overturning its “tyranny” (Saussure 2011/1916: 31) and asserting its derivative nature vis-à-vis spoken language: “Language and writing are two distinct systems of signs; the second exists for the sole purpose of representing the first” (ibid.: 23). Since the study of language cannot rely solely on the study of written records, one must insist that “[t]he linguistic object is not both the written and the spoken forms of words; the spoken forms alone constitute the object” (ibid.: 23–24). This is apparently also true of any collection of phonographic recordings (in any language), of which Saussure makes a few seemingly insignificant mentions. From our perspective, this raises the question (one which has not yet been resolved): To what extent does the existing media situation – the crisis of traditional symbolic systems,4 caused by the emergence of technical apparatuses such as the phonograph (ibid.: 23), film, and typewriter – shape Saussure’s semiology and related approaches? The emergence of these apparatuses, which also made it possible to record and reproduce with unprecedented accuracy the physical effects of the “primary media” directly associated with the human body (voice, movement, gesture), may have placed an increased demand on the systematization of the facts of speech, on the explanation of the existence of both the sound and the conceptual aspects of speech as mutually correlated plans. Contrary to what Kittler suggests in his Gramophone, Film, Typewriter (1999), this is not a case of straightforward determination but rather an unrecognized media crisis that motivates Ockham’s razor, with which de Saussure separates from the true object of linguistics the potentially infinite flood of phonographic samples of all languages, as well as all written records and which operates within the immanent development of the discipline.5 Needless to say, there is a strong leaning toward its idealistic line.

Semiosis, Interpretation, and Machine: The Peircean Way If we turn our attention to the seminal work of Charles S. Peirce, we will be struck by different aspects of the problem. The difference is not only between linguistics and philosophy but also between different understandings of semiotics (semiology). If As expressed emphatically by Kittler: “Once the technological differentiation of optics, acoustics, and writing exploded Gutenberg’s writing monopoly around 1880, the fabrication of so-called Man became possible. His essence escapes into apparatuses. Machines take over functions of the central nervous system, and no longer, as in times past, merely those of muscles. And with this differentiation – and not with steam engines and railroads – a clear division occurs between matter and information, the real and the symbolic. [. . .] The physiology of eyes, ears, and brains have to become objects of scientific research. For mechanized writing to be optimized, one can no longer dream of writing as the expression of individuals or the trace of bodies. The very forms, differences, and frequencies of its letters have to be reduced to formulas. So-called Man is split up into physiology and information technology” (Kittler 1999: 16). 5 On the question of predecessors and influences, see, for instance, de Mauro 1972: 380–389 et passim. 4

12

Intermediality, Semiotics, and Media Theory

259

Saussure built his theory on the systematic desubstantivization of language and linguistic inquiry, Peirce is focused on relationships, processes, logic, and genesis. As Miroslav Petříček points out, Peirce’s is an ontological semiotics, aiming at the universal explication of being and the origins of communication, whereas de Saussure strives at constructing a model of how communication works (Petříček 1989; cf. Eco 1968) and how language functions as “the most important” of all semiotic systems and set of social institutions (Saussure 2011/1916: 15–16). We will notice that Peirce’s examples of different types of signs as well as his explanations of the sign and its general characteristics happen to include recording techniques and technical objects such as writing, photographs, clocks, barometers, thermometers, hygrometers, and weathercocks (all these can be found in his expositions on the index and on the sign, for example.6 Instead of paying too much attention to their technical character, Peirce understands them as part of an all-pervasive semiotic process of interpretation and mediation; in this respect, they are not different from individuals, physical phenomena, social facts, ideas, mathematical symbols, or conventions of geometrical representation. Similarly, the difference between the spoken and written word is of little importance. Peirce’s general claim says as much: “As a medium, the Sign is essentially in a triadic relation, to its Object which determines it, and to its Interpretant which it determines” (Peirce 1998: 544). However, the question of the relationship between the functioning of a machine and the process of semiosis is not absent in Peirce’s thinking. Perhaps the best example is the distinction between semiosis and quasisemiosis and Peirce’s thoughts on the character of the relationship between interpretation and consciousness (see also Nöth 2003). Simply put, if quasisemiosis involves a purely dyadic relationship – i.e., a cause and effect type of relation, as exemplified by the thermostat – genuine semiosis must involve an interpretation of the dyadic (causal) process. After all, “every cognition is of the nature of a sign” (MS [R] 914:5). According to Peirce, it can be assumed with sufficient reason that interpretation is tied to consciousness or its analogue: “Although the definition [of semiosis] does not require the logical interpretant (or, for that matter, either of the other two interpretants) to be a modification of consciousness, yet our lack of experience of any semiosis in which this is not the case, leaves us no alternative to beginning our inquiry into its general nature with a provisional assumption that the interpretant is, at least, in all cases, a sufficiently close analogue of a modification of consciousness to keep our conclusion pretty near

6

See Peirce 1960–1966 [CP4:447]; Peirce 1998: 274; here (as elsewhere), Peirce makes an interesting distinction between genuine and degenerate indices; in the case of the latter, the index conveys not only information but also an iconic representation (“Firstness”) associated with the object that has given rise to the indexical relation (a “weather-cock calls up an image of a quarter of a horizon” is an example given in MS [R] 491:3–4; a similar idea can be found in Peirce 1998: 306–7). “A photograph, for example, not only excites an image, has an appearance, but, owing to its optical connexion with the object, is evidence that that appearance corresponds to a reality” (Peirce 1960–1966 [CP 4:447]); MS [R] 491:2–3; MS [R] 939:45–6; MS [R] L67:37–38. See also Peirce 1986: 66–8; Peirce 1986: 76; Peirce 1960–1966 [CP 7: 355–6].

260

T. Chudy´ and R. Mu¨ller

to the general truth” (Peirce 1960–1966 [CP 5:485]). Thus, concerning the nature of this analogue (that semiosis requires something like consciousness), there is no necessary “human” element, and it is open to analogies in the realms of the biological, inorganic, and mechanical: “Thought is not necessarily connected with a brain. It appears in the work of bees, of crystals, and throughout the purely physical world” (Peirce 1960–1966 [CP 4:551]) and “a piece of apparatus for performing a physical or chemical experiment is also a reasoning machine, with this difference that it does not depend on the laws of the human mind, but on the objective reason embodied in the laws of nature” (Peirce 2000: 70). Complementarily, there is a mechanistic analogue in the human mind (acts of multiplication, for instance), so that the action of the mind might depend on various kinds of interaction, including that between man and machine, body and mind, as a “breeding ground” or locus of semiotic process – “localization in a sense in which a thing may be in two places at once” (Peirce 1960–1966 [CP 7:366]) – as Peirce somewhat enigmatically puts it. This line of thinking opens up a persuasive interpretation of today’s sophisticated AI systems. Since it is not established whether these systems possess something like consciousness and – which is especially doubtful – whether they can be said to understand (cf. Chalmers 2020), it is also not clear – at least in Peircean terms – whether AI deals in and of itself with the possible meanings of the signs it is operating with (i.e., semiosis). With Peirce, importantly, the question of the relationship between machine and consciousness can be posed, at least theoretically, at the level of the interrelation between mechanism and meaning-making. As for the position of substance, Peirce distinguishes three aspects of the vehicle (representamen) of the sign:7 qualisign is a pure quality, a possibility as such; sinsign is an actual fact, the embodiment of the vehicle, an actually existing thing or event; and legisign is the rule according to which the vehicle can be used, a replica, convention created by repetition, or identity of the sign in different uses. From this, the particular semiotic knowledge of technics has to be reconstructed. If Gilbert Simondon (2016), for instance, says technics is a “mediator” between nature and culture, the Peircean way of bringing the same problem to our awareness would be that technics opens up still new loops between Firstness, Secondness, and Thirdness. That would also support the thesis that a (new) technics only becomes conceivable and usable after a kind of signifying process sets it in motion. This is exactly the idea put forward by media theorist Hartmut Winkler, to which we shall return later.

Jakobson’s Mixed Heritage The next point requiring closer examination is Jakobson’s famous semiotic communication model (Jakobson 1981b/1958), where the set of coexisting communication

7

In accord with the omnipresent triad of Firstness, Secondness, and Thirdness.

12

Intermediality, Semiotics, and Media Theory

261

components (sender, message, code, context, contact, and addressee) corresponds to a variable set of focuses (Einstellungen) – i.e., communication functions (emotive, poetic, metalingual, semantic, phatic, and conative). Jakobson’s thinking corresponded well to the emerging conceptual foundations of the information age (an early example can be found in Jakobson 1981a/1949; cf. Geoghegan 2011); such cybernetic mechanisms as the feedback loop, the tendency toward equilibrium in information levels, and the principles of computer technology conceptualized by Turing as the Universal Discrete Machine all play an important, if rather latent, role. In this way, Jakobson’s model was readily accepted in the new postwar information age context, even while it obviated the material mode of signal transmission that alters the character of signification. As we shall see, however, this circumstance was not an obstacle but a condition for its incorporation into the foundations of intermedial research. With Jakobson’s distinction between interlinguistic, intralinguistic, and intersemiotic translation (Jakobson 1971/1958), the recognition of semiotic difference as media difference is, thus, merely adumbrated. Still, it was Jakobson’s concept of intersemiotic translation that inspired Claus Clüver’s idea of intersemiotic transposition to produce one of the first distinctively intermedial theoretical frameworks (Clüver 1989). What remains beyond the concept of intersemiotic translation, equally affecting the roots of the intermedial paradigm, however, is the simultaneous existence of the different levels of coding, including the technical. As we shall see, this omission will haunt intermediality up to recent times. In this acutely latent stage of media thinking, we can also witness how the principles of one medium work as a latent metaphor informing the features of a general model. The word “translation,” for instance, already suggests the modelling power of language as the most complex and only universal of all semiotic communication systems (Jakobson 1971: 673). Even in Eco’s later semiotics such tendency is repeated.

Eco’s Advances and Sidesteps The technical foundation appears in Eco’s seminal Theory of Semiotics, which presents a distinction between s-code, a monistic system of differences and responses (syntactic, semantic, or behavioral), and code, a complex form of rule correlating two such systems together. Again, however, this is expressed through a latent metaphor for the general condition of transmission rather than as a special layer that mediates on the basis of its own principles and, thus, also shapes the form of sign and communication. The entanglement of transmission and information volume, as well as capacity problems, with communication – no communication happens without a transmission of signals (Warren Weaver’s “level A”) – leads Eco to distinguish four different approaches, of which only two are properly semiotic, while the other two constitute “the lower threshold of semiotics” (Eco 1976: 41): 1a. structural theory of the statistical properties of an information source, 1b. structural theory of the generative properties of an s-code, 2a. engineering study of the transmission of signals, and 2b. transmission of significant pieces of information

262

T. Chudy´ and R. Mu¨ller

used for communication purposes. Thus, a mathematical theory of information (1a) and studies in information engineering (2a) overlap with semiotics only at those points where they contribute to the development of more economic categories to deal with semiotic problem proper. The “Watergate Model” – a warning system making use of the approaches characterized above as 1a and 2a – serves as a “lower threshold” for semiotics involving such motifs as regulation, refilling, draining, insufficiency, overflow, and alarm and related regulation and communication mechanisms and channels. The metaphor constitutes the latent standard model of technological reality, but it is limited accordingly, letting the quantitative approach to communication processes return “through the back door.”8 The Watergate Model cannot explain how the technical fabrication of signs affects the nature of representation and the receiving situation. We might take the general difference between analogue and digital (and digitized) photography as an example. One might object that in terms of perception, there is almost (or no) difference at all, but the matter is more complicated. The determining factors can be identified by the following questions: Do we perceive the photography in print or on the screen? What is the scope of its digital manipulation, as opposed to the professional’s intervention in the darkroom? Is photography received in the contexts and quantities allowed by social media or in a physical album? Such circumstances completely alter the very epistemology and ontology of the image.9 In semiotic terms, all dimensions of semiosis – syntactic, semantic, and pragmatic – may involve technical aspects (and vice versa). It would be hardly possible for Roland Barthes to base his theory of photography in Camera Lucida (Barthes 2010/1980) on the principle of ça-a-été if he was thinking about its digital shadow or counterpart (cf. Attridge 1997). Similarly, the television image acts upon the viewership very differently in the contexts of flow vs post-flow TV, where the whole organization of production and reception, its economic and ideological implications (state vs corporate), is different and changes our approach. In short, the technical production of the sign crucially alters its creation, the possibilities of its distribution, its reception, and its meaning, reciprocally and within the same “loop.”

The Emergence of Media Difference in Film and Cultural Semiotics (Metz, Lotman) An important step in the development toward technological awareness in semiotics is Christian Metz’s film semiotics, especially his Psychoanalysis and Cinema: The 8

Eco himself corrected this quantitative approach by qualitative semiotic reinterpretation already in the 1965 edition of his Open Work (Eco 1989/1965). 9 Just think of the Russian propaganda and its attempts to blur any difference between truth and lie in the face of the photographies of the Russian war crimes: The fact that a vast majority of these photographs are firmly embedded in the digital culture is revealing. The satellite and the drone perspective might represent the new provisional horizon of factuality – the symbolic anchoring of the imaginary – in this context.

12

Intermediality, Semiotics, and Media Theory

263

Imaginary Signifier (Metz 1983). Medially progressive in spite of – not because of – his indebtedness to Saussurean semiology, Freudean psychoanalysis, and its Lacanian psychoanalytic rephrasing, Metz’s work draws on 1970s French film theory (a survey of which can be found in Aumont 1997: 130–147). The key point is not that cinema is the technique of the Lacanian imaginary,10 but the idea that the subject (mis)identifies with the viewing source – what Metz calls identification with the camera – so that the true allure of cinema lies in the disappearance of the subject in a technical mirror, where the signifier is mixed and characterized by absence, since everything is a moving photographic recording. Equally mixed is the system of codes, some of which belong to film as such (montage, visual and auditory analogy, etc.) and others to a more general set of “languages,” such as narrative codes, for instance (Metz 1983: 27–28). Film comes to be seen as a specific media variant of the process of the formation of the subject developing under the conditions of capitalist society. And this is another key point: the existence of the cinematographic institution, which functions within and simultaneously creates certain structure of desire and limits, i.e., the cinematographic space, intimately tied to the capitalist structure of society and its mode of production. The purpose of this institution is to create and reflect “the good object,” a spontaneous desire to “go to the movies,” working within a parallel structure ( juxtastructure) where the symbolic and the imaginary intermingle and support one another. The apparatus – and the conditions of its functioning – is of crucial importance to the cinematographic effect: The machinery of the apparatus has to appear as missing in order for the film to function as fetish, and only within the double play of concealment and awareness can the apparatus acquire its constitutive role in the cinema function. Yuri Lotman and his cultural semiotics must not be overlooked in this context. Lotman begins to use the word channel interchangeably with the coding system (Lotman 2000; cf. Müller et al. 2020: 461); the “channel,” thus, covers not only the physical and mental aspect of the connection between sender and receiver but is equally generative of the structure of the message. Taking into account the “channel” aspect of communication (transcending its role in establishing connection) allows Lotman to distinguish three functions of sign systems (and equally, of intelligence and “self-conscious” semiotic objects such as culture and “texts”) which had always flown under the radar of semiotics but are indispensable to media thinking: the transmissive, creative, and memory functions (Lotman 2000: 11–19). Nevertheless, Lotman does not specify how the “channel” aspect molds the functioning of the sign system as opposed to its “coding” aspect. That’s why “noise” and “uncertainty” are, more often than not, explained as creative, which theoretically obscures the negentropic aspect of information circuits: To contain entropy within controllable limits is a necessity for all communication systems to remain stable. The distinction between continuous and discrete coding systems (i.e., also channels) is a crucial step in recognizing the various media tensions within the media product as well as particular media systems (for instance, the “discrete” aspect of the film image as a 10

The same idea appears later in Kittler (1999) but in semiotically less developed terms.

264

T. Chudy´ and R. Mu¨ller

result of transformative processes). Cultural semiotics also explains how the same message can fulfil several functions at once, how its interpretation changes within a different set of semiotic objects, and the larger cultural implications of the tendency to use continuous vs discrete semiotic objects in order to fulfil the memory function. For Lotman, media difference is key to cultural dynamics, even if it stays merged with linguistic difference, and the theory generally remains susceptible to a residual tendency to linguo- and textocentrism. Every medium is treated as a “text”; text, however, is understood as (semiotically and medially) heterogenous (Lotman 2009: 77). Finally, Lotman’s cultural semiotics does not allow for a full-fledged theorization of the actual channel – the media dispositif – and its role in cultural dynamics. If one uses the theory in order to characterize our digital presence, for instance, the implications have to be “teased out.”11 This is because the technical foundations do not come under particular analysis. In other words, the channel is fully “semioticized,” which obscures the factors of the evolutionary logic of technics sui generis, the autonomous aspects of its evolutionary pressure, and the mutual entanglement of causes and effects in the processes of the technical modification of the human.

The Semiotic Subconscious of the Technical Media Theory Three Aspects of “Sign” It is a paradox that from all variants of the word sign, the one that is etymologically most fruitful is the German Zeichen which is derived (just as with the English word “token”) from PIE *deyḱ- (“to show,” Köbler 2014), with two other main candidates being the ancient Greek σῆμα, which relates to PIE *dʰyeh2- (“to notice,” “to think,” Brugman 1889: 348), and “sign” or, in Latin, “signum,” which most likely stems from PIE *sek- (“to cut”) or *sekʷ- (“to follow,” Pokorny 1959: 897).12 At the same time, the concept of a universe composed of signs, of Peircean origin, seems not to require a human to discern them yet. From all three possibilities, the one which captures this sense best would indeed be the sign, i.e., a well-segmented unit which makes other facts follow, with the “token” aspect meaning that something is shown, or as Heidegger would put it, revealed and uncovered, and, as the third equivalent suggests, ultimately is noticed. This triad – follow, show, notice – well characterizes what is at stake. Now in Kittler and much of the technologically oriented media science, there is no lack in exploring the “following” (implications, causality) nor the “showing” (the reading of symptoms of the Freudian, Lacanian, or other sort). What the theory has 11

Cf. International Journal of Cultural Studies 2015, volume 18, and the Special Issue No. 1: The Uses of Yuri Lotman (and the articles there by Indrek Ibrus, Peeter Torop, Maarja Ojamaa, and Nicola M. Dusi). 12 Cf. also Bühler 2011: 40 ff who etymologizes the words for “sign.”

12

Intermediality, Semiotics, and Media Theory

265

the greatest problem to identify is the “noticing” aspect. In other words, the triad may be translated loosely as an analogue to production, representation, and reception, which are all familiar categories of media theory. However, it is not entirely precise: The noticing aspect is not the same as reception, which can be also mechanical or machinist; it involves something more, which is related to Peirce’s notion of the “interpretant.” In this context, it is worth inquiring what is meant by the semiotic aspect of “noticing” – which is the main blind spot in the classic technological media theory – in terms of data quantity. It is commonly accepted that the number of synapses among neurons within the brain, and thus, the amount of the “inner” activity of the brain, exceeds by several digital orders the number of contact points with the “outer” world (of which the visual data set is the dominant one).

The Human and the Machinist “However, the messages produced by a computer in the interface of humans and machines are either messages conveyed by a human sender and mediated by the computer or they are quasi-signs resulting from an automatic and deterministic extension of human semiosis” (Nöth 2003: 86). If so, is the technological semiosis in machines governed by the paradigm of semiosis in the human brain? The discussion of arguments for and against machines having a mind is telling in this respect. Let us mention it: Nöth (2003: 87 ff.) refers to Searle’s “Chinese Room Argument,”13 saying that because machines save mental work, they must be mind machines, since the Americans in the Chinese room do have minds and intentions. But that implication does not hold true – the metaphor may or may not be valid on this point; there is no necessary conclusion or compelling argument. Moreover, drawing on the very metaphor that the author wants to criticize does not seem to be a fair step to make. The right conclusion would be either to dismiss the metaphor as a whole or to draw a negative conclusion: not that the machines must have minds but that it has not been shown that they cannot have one. Further, the argument that machines that save mental work must be mental machines (Nöth 2003: 88) suffers from the flaw of yet another premature conclusion: What has been shown is that they save mental work, but that in no way means that those machines must possess a mind themselves. Perhaps what is taking place in the

“Searle explains his argument by means of his famous parable of the Chinese room in which messages are processed by people who do not even understand the meaning of the individual words. The servants in this room are monolingual Americans who receive the messages in Chinese, but are nevertheless able to process them on the basis of numerical instructions that tell them how to combine and correlate the elements of the incoming messages. Consequently, these Americans (alias the computer) do not understand (and hence are not affected by semiosis), ‘because the formal symbol manipulations by themselves don’t have any intentionality; they are quite meaningless; they aren’t even symbol manipulations, since the symbols don’t symbolize anything’” (Nöth 2003: 86–87).

13

266

T. Chudy´ and R. Mu¨ller

process of “saving the mental work” is a transformation of the “mental” character of the work into something else which need not be called “mental.” Thus, nothing has been shown but an unwarranted transfer of the “mental” from a human to a machine via the aspect of “saving.” What is that, ultimately, if not begging the question? We must leave this question of machine minds undecided, and with it also the issue of their genuine semiosis, as does Nöth. What this impasse reveals, however, is that we are tempted to compare the function of machines/technological media with the “natural” function as experienced in (healthy) humans. But let us for a moment take a look at them not from the point of view of human capabilities (as does Marshall McLuhan, most famously, in his concept of “the extension of man”) but from that of human disabilities. If the media tools according to Kittler are all derived from physical handicaps (of the body, the senses), as he successfully shows (in Thinking Colours and/or Machines (2006)14 against the Heideggerian utopia that their origin can be attributed to the lifeworld), and not from the natural use of human abilities, including language and speech, then it is true in a very ontologically profound sense that new ways of signification also come into the world as a result of ruptures, of rendering the natural ways unapparent and perhaps even inconceivable (A similar line of reasoning is developed by Yuri Lotman, already mentioned, with an emphasis however on divergences within semiotic objects and codes.).

The Kittlerian Turn What lies at heart of the Kittlerian turn is thus not a pan-medialism but a true description of the key moment relating to the end of symbolic monopoly, or, more precisely, to the distinction between textual media (printing press) and technological media (starting more or less with the phonograph): “Textual media transform the linguistic-symbolic into an operable code; technological media, by contrast, transform the contingency-based, material, real itself into a code that can be manipulated” (Krämer 2006: 100). Characteristically, however, Kittler himself soon switches his attention from the origins in human disability to the variable functions that compensate for it. And these functions relate precisely to what may be done with the signs from a material perspective: storage, erasure, transfer, etc. This turn may be summarized in many ways. One is achieved with a concrete example by the canonical Kittlerian interpreter, Sybille Krämer: “The Fourier transformation [here broadly speaking transforming a wave signal into the graphic form of a sinusoid based on analysis of the wave’s values, TCh] accomplishes for the material realm of signals what the Greek alphabet achieved for the symbolic realm of language” (Krämer 2006: 101). The difference between time axis manipulation in textual and technological media should not detract from the fact that they are the

14 Another main work where disability is prominent is, of course Gramophone, Film, Typewriter (Kittler 1999).

12

Intermediality, Semiotics, and Media Theory

267

same kin. What is different is the absence of the initial (symbolic) grid through which the real must fall. A typical example is music as notated by textual media and as captured in its original auditory form. The unintended repercussions produced by manipulative medial shifts that are of universal concern and affecting the lives of the planet’s population stem of course from their sign character: Consider an example based on pure visual illusion, Gerardus Mercator’s 1569 invention of cylindrical map projection whereby countries in the near-polar regions appear to be much larger than they are in reality. Whereas the case of Greenland is not much of an issue from the geopolitical standpoint, the giant appearance of Russia on the map allows its representatives to draw readily discursive consequences of their “vast land” warranting the status of its “sphere of influence” (while Canada, whose area as depicted on the map is subject to the same distortion, does not seem to take discursive advantage of it). What should be granted to Kittler, however, is the persistent focus on the key importance of the technology of communication (as opposed to other concepts of technological media). And what is this but the transmission – production, manipulation, reproduction – of signs? Signs must be considered then as having no existence of their own but only that which is bound within the constraints of the particular medium used to convey them. One of the important implications of this conclusion is that theorizing about signs is outweighed by the practice of using them; furthermore, this practice always entails – or rather, is composed of – gestures that are its very essence. This brings us close to the point made by the distinguished semiotician Roland Barthes, who accurately describes such a practice in his essay on the practice of music (he himself mastered the piano) as quite a different experience from the one of a listener who has never had the perspective – or should we say inner stance? – of a potential performer with his or her body, be it as a player or conductor (or even a composer) (Barthes 1977). In similar fashion, any discussion of the nature of signs must be based on practice (both production and reception), on pragmatics in a broad sense, and never abstract from it. But it’s more complicated than this. An illustrative example of the essential and spare parts of semiotic gesture can be found in Edward Bellamy’s novel Looking Backward: 2000–1887 (1888). It is not by accident that the main character’s absolute enchantment with the futuristic media world is provoked by the system of music distribution via “telephone” (of course, the wireless mode had not yet been conceived). Perhaps the main driving force behind this invention is not to be overlooked: “We have simply carried the idea of labor saving by cooperation into our musical service as into everything else” – it is “labor saving” not in the sense of sparing humans the practical semiotic gesture itself but the unnecessary, forced activity while preserving singing and playing as such. Even more telling is the reaction of a nineteenth century visitor to this imagined environment: “It appears to me [. . .] that if we could have devised an arrangement for providing everybody with music in their homes, perfect in quality, unlimited in quantity, suited to every mood, and beginning and ceasing at will, we should have considered the limit of human felicity already attained, and ceased to strive for further improvements.” So much for the prophecy – not of merely mass music distribution but even more universally of the internet as such. Taking into account

268

T. Chudy´ and R. Mu¨ller

that the unforeseen extension into all sign categories and all ways of communication has nearly been achieved, is the main claim that we should “cease to strive for further improvements” still valid? Clearly no. Let us stay with the novel for a moment. Another unexpected feature of music distribution corresponds with today’s instant availability of signs: listening to the part of music that is best suited to one’s mood without having to “sit for hours listening to what you did not care for.” Explicitly, the satisfaction of taste is accentuated as the prominent benefit brought by technology. Perhaps there is not one but two key driving forces: saving labor and instant satisfaction by cutting the “unwanted remainder.” But is it so simple? Most media theory (from McLuhan to Manovich) written after the fact of the media boom, roughly marked by plurality of audiovisual media emerging at the turn of the twentieth century, bears witness to what we should call a fundamental change in the semiotic process (introduced by the media technological dispositif and new cultural practices). Moreover, music in this futuristic layout is not recorded. Rather, it is performed live in the “music rooms in the city, perfectly adapted acoustically to the different sorts of music.” None of the Kittlerian “Aufnahme” or “time axis manipulation” but pure, almost inner, interconnection between the performers in the selected music room and the listener.15 ll in all, the technics change but not the signs used. This is only confirmed when noticing the style of the novel: it keeps the late nineteenth century language and ways of expression to convey the nineteenth century imagination, which, of course, has a somewhat disruptive effect to the (media-theory) message that we tend to extract from it. In this, however, we are already subconsciously succumbing to a post-Kittlerian bias. On the other hand, Bellamy’s novel illustrates well the dialectic between the intentions (even dreams) and unintended aspects of technical progress that are only revealed ex post facto. Surprisingly, this applies also to Kittler’s practice.16 Even if the turn to machines, materiality, and hands-on media practice with all its exuberant possibilities opens new vast spaces liberated from the constraints of “classical” social and motivational conventions of communication theory, reserving more attention to the proliferation of new styles of communicating, including language pragmatics development, there still remains a certain resistance on the part of signs that is attested to by this very development. True, ways of communicating signs have been severely impacted by the technology. But then again, all that the new uses of technology are doing from a certain point of view is conquering new worlds for signs to occupy. They feed into the sign systems. Let us recall that Peirce did not confine the domain of signs to those produced by humans, let alone those that humans produce intentionally. Nonetheless, the field of semiotics with all its wealth of approaches should play a key role in

15

Let us signal here that Bellamy’s 1887 setting precedes the environment of most of the stories cited by Kittler which suggests that Kittler’s examples are very much already under the influence of the real technological achievements. That arguably contributes to Kittler’s forgetting the semiotic gesture, being already fascinated by the machinist functioning. 16 Here we refer to his famous preoccupation with ‘soldering the circuits.’

12

Intermediality, Semiotics, and Media Theory

269

the assessment of these media technologies. This will not happen when the text is seen as a “word salad” (Wortsalaat in original Kittlerian), but only if it is taken into the equation as both dependent to some degree yet also a key variable.

The Symbolic, Sign, and Signal However, the semiotic layer is very often confused with the symbolic layer. Again, it is Sybille Krämer who makes a simple observation in this respect: “Associating media with the symbolic is not exactly a unique approach. After all, isn’t the interpretation of media as the material carriers of events with signs commonplace in media studies? And yet, Kittler does not tread on this common ground, for media are not inevitably linked to the symbolic but only exclusively when pre-technological, literary media are involved” (Krämer 2006: 98). Even though Kittler has no doubt endeavored exceptionally to mark the dividing line between the pre-technological media and those of the technological era, the obfuscation of what the other kinds of signs are, and, more importantly, how they function outside the domain of recording (Speicherung), still affects his analysis. We now turn to a more detailed discussion of Kittler’s (lack of proper) relation to semiotics. Indeed, he “does not tread on this common ground” wherein media is directly associated with the symbolic. Yet, when detecting what lies beyond the symbolic realm, he falls into the trap (partially due to having subscribed to the Lacanian triadic master scheme) of not recognizing fully the semiotically relevant communicative function in the imaginaire (film or generally new visual media) nor the réel (gramophone, or generally acoustic media), since his chief concern is how they first embody psychophysical automatisms and then how they go about the technical task of recording (and later reproducing and manipulating, of course) the subject matter, which by itself is just a data flow or even a time flow (“[t]exts and scores – Europe had no other means of storing time,” Kittler 1999: 4). The semiotic frameworks, like military reconnaissance, taking “shots” of animal locomotion, or even Henry James dictating his novels to an assistant typist, merely set the stage for analysis, but semiosis is not its main focus. In other words, as much as Lacan’s theory is indebted to Saussure’s concept of signification and the sign, in Kittler, this concept falls into oblivion and is neglected. As one of the most important anomalies of this theory, Krämer mentions the fact that Kittler does not consider “oral language and. . .the (unrecorded) voice as media” (Krämer 2006: 99). And it may indeed be a certain underlying Hegelianism (“When Hegel summed up the perfect alphabetism of his age, he called it Spirit,” Kittler 1999: 16) that prevents him from treating anything unrecorded as but a fleeting or absurdly insignificant phenomenon – let us recall the farcical labor of professor Pschorr who tries to recover Goethe’s voice, using his corpse to construct a reverse phonograph but who ends up capturing Goethe’s “live” erroneous argumentation known from the Theory of Colours (against Newton), and eventually his snoring, in the short story by Salomo Friedlaender quoted by Kittler (1999: 59–68). It is farcical precisely because it ignores the inevitable fact that vibrations of the voice must fade and eventually disappear. The comedy here does not, therefore, lie in trying to

270

T. Chudy´ and R. Mu¨ller

attribute importance to oral speech but in trying to revive it again, not as a sign this time (indeed, that happens earlier in the story but is dismissed as parody and deception, Kittler 1999: 62)17 but as a signal. The obsession of dividing the time axis into discrete units by technological means and how to go about it rests on the unreflected assumption that symbols are the only means for filtering a signal out of noise. This in turn stems from the other deceptive identification of symbols (not signs) with the storing of time. As may be easily seen, while the storing function is to a certain extent the main prerogative of symbols, it is not the main purpose or typical characteristic of other signs (index, icon) which serve their purpose only in the very (often gestural) context in which they are used and cannot be set apart from the time flux within which they occur (this irreducibility could then be tracked back to several philosophical schools, especially to the Bergsonian way of taking durée as a living – and meaningful – stream of consciousness). In short, yes, interpreting media as a material vehicle for sign operations could be seen as standard (cf. Winkler 2008), while reducing signs to symbols is – at least from a Peircean perspective – not permissible. A musical sound also contains a timbre, whereas the score cannot. Yet this timbre too is part of signification, even part of signs as events which are captured by the analogue media. Lars Elleström (2010) called this the material modality which is part of the meaning. In other words, what is thus captured is not “pure signal” or “the real” but again signs, albeit a different kind of signs suited to the new technology. Moreover, the technical dispositif (e.g., Fourier analysis) does make a certain semiotic aspect repeatable, but it cannot create their meaning (other than that related to the reproducibility or reproducing as such). Thus, the coding which is called a “second transition,” i.e., calculability, is truly second to the first coding – and potentially decoding – of the semiotic. A playful rearrangement of signs such as may be heard in every skilled interpretation of a Rachmaninov prelude or Beethoven symphony is obviously not equal to contingency processing in Shannon’s terms (taken over by Kittler). In other words, manipulation – in its stochastic connotation – does not occur except in the realm of chaos (or noise), while rearrangement occurs in the semiotic domain.

A Narrative Plea for Engineers It is truly ironic that Kittler’s writing is so semiotically rich – not just in the symbolic order but also in the rhythm and artistic interplay of fictional illustrations – and his theory seems almost to have derived its main ideas from nontheoretical uses of language. This is telling in two ways. First, it shows how hands-on media practice precedes media theory. Second, it proves that even neglected semiotic aspects (here

And to be noted, another example of acoustic sign (not signal) in the story “Death and the Shell” by Maurice Menard is the ominous melody which cannot be captured by musical score, but its only effect is the death to its listener – yet another case of failed communication (Kittler 1999: 51–55).

17

12

Intermediality, Semiotics, and Media Theory

271

fiction, poetry, and historic proclamations) can hold sway over the intended content. In other words (to paraphrase Marshall McLuhan), media use is the message. However, even when voice becomes the object of media theory (Kittler et al. 2014), the list of its culturally relevant aspects does not include any mention of its semiotic aspect (in the Peircean sense), as if making concentric movements around a gravity center it fails to recognize. A concert of connotations (themselves semiotic!), thus, occurs in Kittler’s breakthrough work Gramophone, Film, Typewriter as well as his late Musik und Mathematik. What is at stake in the interim Dracula’s Vermächtnis (Kittler 1993) is a shift to the heart of the matter. This shift is marked, of course, by Shannon’s mathematical theory of communication. For Kittler, Shannon is the inventor of the vocoder (together with Turing) and so is one of the ex-post-facto technological contributors to the WWII outcome. In Dracula, Shannon’s approach is treated for what it is – a true potential for manipulation of the sign(al) without regard to the resulting signification (or nonsignification). This metamorphosis of sign into signal allows the thinker to draw on the opposition with noise (which would not be possible in the case of a sign): They become two sides of the same equation, or at least of the same balance. But again, this is not what we do with a sign – signs are a given, and the theory of their functioning can do without opposing them to nonsigns (modern artists aside – but even there, the nonsigns eventually become signs). The interpretation of this shift should relate to origins. Shannon in his Mathematical Theory of Communication states: “The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem” (Shannon 1948: 379). And Kittler, by converting his engineering problem into the media theory problem, thus, “engineerizes” the theory. It’s not just that, however. In contrast to the Barthesean “practica” (which remains indebted to the semiotic approach), Kittler converts, at least in significant part, even the practice into the same engineering problem. Using his own words, it remains to be deciphered why steps originally taken for “testing” or “experimental” purposes (Kittler 1993: 150, 166), like the wiring together of signal and noise, gain universal importance.

Wooing and Repudiating Semiotics All attempts to reveal at least some semiotics in Kittler must face the fact that it is only the Lacanian theory of signs that is considered (not even the Saussurean one).18

Nothing changes with the accidental mention of “indexical media” in the context of Kittlerian studies (Salisbury and Sale 2015: xx). Put into context of a very peculiar theoretical framework, its true meaning is referring again to Jacques Lacan, which then leaves them to mean “media that register the imprint of the real.”

18

272

T. Chudy´ and R. Mu¨ller

Perhaps the cardinal problem lies in the assumption that “order of the written culture, be it literary of philosophic, could construct meaning only of elements which were themselves meaningful” (1993:166–7), with the consequence that the materiality of significant letters taken for themselves destroys this illusion. This may certainly hold true with Markov chains of letters, but the situation changes if we listen to a chain of sounds, for example. The same premise is not valid, given that compositions of sounds, to the extent that they are articulated sounds (as letters are), may already convey meaning or – at least – constitute signs (indices, icons). The line between chaos/noise and order/sign(al) is not clear-cut in nonalphabetical media. But if so, the “frontal attack” on authors of literary texts, who are likened to the “devil” (1993: 168) is an incorrect metaphor, to say the least.19 Yet Kittler’s interest in the development of mathematic operators (and technical operators in general, cf. Kittler 1993: 149–160) also brought him to the limits of his approach – Leibniz’s epochal shift to the use of algebraic signs is not just a testimony to the overall differentiation of the functional system (in the sense of Niklas Luhmann) of mathematics and “mathematicized” technology-oriented science but also of Kittler’s concealed and inadvertent semiotics. The difference of use and quotation is a reflection that belongs to semiotic pragmatics. However, this aspect of his technical semiotics is short-lived – the argument about logical operators becoming self-sustaining (which could be a contribution to the theory of signification and thus still remain within the realm of semiotics) spills over into arguments made about nonsignification and noise. This walking on the edge of humanities, as has been observed, is then leveraging precisely on the oblivion of their pillars (the role of the “subject,” for example), in order to not be carried away by human movers as masters of the development. Let us not forget, however, that Kittler’s primary point of view is that of a historian, not a systematist. A semiotics complementary to his technical radicalism, from this point of view, is not prevented. What happens in reality is that semiosis in its most explicit form is traded for a theorem that allows for distinguishing only between necessity and chance, later translated as strategy vs chaos (Kittler 1993: 164, 166), and that narrows the focus from the continuation of sending and answering information (which is essential for a communications theorist such as Niklas Luhmann, among others) to just sending the information, as the smallest single parcel of communication. By now, it should be evident that this represents, to a certain extent, a misinterpretation of communication. It seems here as if the early Kittler, a theorist of linguistic operators who is mindful of the pragmatics of signs, has ceded his place to the Kittler of later works, who is fascinated by the relegation of discourse to a secondary rank (1993: 166). Nevertheless, semiotic implications are drawn from the most materialist premises: “Of all the instruments, wood and hammers, metals and

Not only is audiovisual/iconic indexical semiotic expression not affected by the “spell” of the symbolic; the symbolic readjusts to the new environment: the media theoretical importance of experiments like lettrism (visual poetry) reveals that nonsignifying elements themselves can be brought to signify within the realm of literature – or by pushing its boundaries.

19

12

Intermediality, Semiotics, and Media Theory

273

bells, have the highest quotient of noise. They function phatically – as a call to church or to a conflagration – and not poetically” (1993: 171). Yet, the short circuit snaps in again: “where the media alliance between music and poetry fails, there begins the mathematical return of the old chaos” (Kittler 1993: 171). And moreover, it is presented – in unmistakably Kittlerian narrative fashion – almost as a revenge on the poetry which has “excommunicated” clangor (and, by extension, also noise) from the domain of its autopoetic system (a paraphrase of Goethe, who serves, as always, as a good pretense for Kittler to convey his own stance, Kittler 1993: 170–1). To avoid all possible doubt, Kittler adds that it is with Jakobson’s idea of the poetic function as a focus on the immediate experience of signs that “the signal-noise ratio is brought ‘to its maximum’” (1993: 170), thus representing the direct antipode and counterforce to the desirable reexamination of the Greek alphabet, which introduced what should be seen as only a limited analysis of voice, “wiring” it to writing. Here, the negligence of semiotics reveals – albeit again for a fleeting moment – its negative connotation. But is not Kittler himself the one who tries to make most of noise as a sign, thus far outreaching Jakobson’s “guilt”? In a way, Kittler’s manner of handling noise is so poetic, at times it appears almost as a personalization within his own narrative, presented mainly in his Gramophone, Film, Typewriter, of the technical media’s coming of age.

How Should We Really Regard This Narrative? In a most abstract way, Hans Ulrich Gumbrecht has described it as a “consistent intellectual gesture of making what is real ‘ready-to-hand’ and then concentrating on the ‘self-unconcealment’ of what is ready-to-hand” (Gumbrecht 2013: 323), which then translates in concrete terms, for example, as the oblivion of the Greeks vis-à-vis the origin of Greek letters imported from Phoenician icons (Kittler 2009:101; Derrida spoke of this as early as 1981, but it is Kittler who developed its most crucial aspect) and of course as the reawakening of the materiality of the medium in the nineteenth century. In order to better understand this, let us return to Sibylle Krämer’s insightful text. She relates the gesture to the historical (WWII) narrative: The relation between signals and noise can also be interpreted as that between a coded signal and its deciphering by enemy intelligence (Krämer 2006: 102). In what world? One may ask. Hardly universally, and not even when you serve on a military submarine in the post-WWII global military political context. Even there, the noise of the ocean is for most part just the noise of the ocean. Clearly, what is meant by this, again, is a shift of focus: Discourse analysis turns into cryptanalysis (Krämer 2006: 102), or rather can turn into it. Cryptanalysis presupposes interception of the signal as its key moment. But then, are we not encountering here the third key aspect of the sign, i.e., the “noticing” aspect? Perhaps. Kittler’s swing to cryptography would then amount to a living record of the omnipresence of signs within the technological communicational context. And inevitably so, what is possible for the mathematician Shannon in his mathematical

274

T. Chudy´ and R. Mu¨ller

theory, namely, disregard for the communicated content, is not entirely possible for the historian Kittler. In his vision of the media complex (Medienverbund) as it takes over the steerage of history, he searches for an almost pan-semiotic approach to noise as the encoding of a hidden signal, something that would figure, at its most extreme, as the texture of nature. It is almost as if an unspoken quest for sign hunting can be deciphered behind the (nearly paranoid) suspicion that every noise can be interpreted as a message waiting to be decoded. In a way, this claim seems to be supported by Kittler’s very words when describing Alan Turing’s revolutionary description of a computer – it reads and writes signs (Kittler 1999: 18). However, his whole point in this regard is to emphasize the reduction of signs, both in terms of quality (from the array of characters on a typewriter to binary 1 s and 0 s) and in that of processing steps (storing, erasing, replacing). In other words, discrete machines are made for easing and simplifying the task of semiosis, not for giving it its due. In phrases such as “the interfacing of a data processing system represents an image, voice, or text only as a ‘surface effect’” (Krämer 2006: 103), the oblivion to which the indexical nature of data processing has been relegated becomes visible: “representation” and “effect” accurately bring forward its aspects while failing to draw on the semiotic aspect. By avoiding McLuhan’s derivation of the media from the human body, Kittler disconnects media from the primordial sensory experience but not from the intellectual semiotic capacity. In other words, to treat the newly arising capability of technological media to capture noise as a moment of its liberation from signs means to disavow the specific contextual reminiscences of such a discovery. More than anything else, in fact, it testifies once more to the inevitability of the communication purpose in communication media. Any such finding cannot but be overdetermined by such a purpose.20

Toward a Media-Semiotic Convergence (with a Detour) Having discussed Kittler’s semiotic blindness, let us now return to the general argument. While in the first part of our chapter, we discussed the technological

20

The true function of communicated signs themselves lies no doubt in their context. Media are never free of the social environment and their content, though they fall into the different function categories of social systems (art, economy, law) and are pulled into the intricate network of society as a whole. A telling example is the arrest of people holding up blank “protest” signs by the Russian authorities (Van Brugen 2022). This blank space is never free of meaning, and this is all the more true in a situation where the aim of the authorities, as is well-known, is to counterbalance, camouflage, or cover up things that are not called by their proper names (“special military operation” in the place of “war”). In this environment, where one might otherwise consider a blank sign to convey no meaning, we see that signification does in fact take place (this, of course, is precisely the intention), and it is because of this that those holding the signs are punished. The authorities cannot help but notice. Thus, what is at stake is not just the overload of context imposed by public institutions but also the contrary: the almost involuntary subduing of public authority to the mechanism of contextual determination of blank space. In this way, they resemble Kittler’s unwanted testimony to semiosis that is never avowed.

12

Intermediality, Semiotics, and Media Theory

275

subconscious of semiotics, the larger genealogical perspective reveals a complementary problem in the sphere of media theory, its semiotic naivety, which leads to the question whether this mutual anopia between technology and semiotics might be key to what is actually at stake in the evolving relation between (new) technics and the semiosphere. Bolter and Grusin (2000), and their influential concept of remediation, is another case in point. The governing principle of this concept of media evolution is certainly not the difference between semiotic systems. The double logic of remediation is that every new medium strives to oust its predecessor by acquiring better access to immediacy, which it does by strengthening its mediating character (hypermediacy). This logic is structured along the lines of the dynamics of closeness and distance, reflecting the different modes of sensorial experience; among them, the visual mode dominates. Thus, the characteristics of the evolution of visual media are projected onto the global processes of media development as remediation. A key difference in the audiovisual media era would rather lie between the former ready-made character of virtually all signs within the common range (e.g., bound to common materials as in Aristotelian ἐν oἷς, Poetics 1448a) and the new on-demand signs which are principally determined by their technological purpose (e.g., assemblers) but can nevertheless emerge as signs for practical use, in the case of new media artists, for example. And with this enlargement and spread of the improvisatory character of signs, we see a gradual convergence between semioticians and media theorists. Thus, there seems to be a point of intersection with the semiotically (half-)blind media theory after all. Among media theorists, one may see the German author Winfried Nöth as a pioneer, with regard to his concept of media as semiotic machines. While Marshall McLuhan elaborates at length on media influence on the cultural production of signs, and on the capacity to read them, without naming it, thus (typically in his distinction between hot and cold media, his focus on content over form in phonetic writing, on the narcotization effect of media, on the effect of framing, etc., McLuhan 1964), it is Hartmut Winkler who identifies the new semiosphere as the direct result of the new media sphere (Winkler 1997, 2008). An important shift, however, occurs: The media complex does not imitate the processes within the neural structure but creates a fullyfledged new layer of collective memory, or shared signs. Winkler claims more specifically that the symbolic/semiotic dimension is not just one among others but the irreversibly central aspect of media (Winkler 2008: 211) which establishes patterns for reshaping and reducing the complexity of the world. In order to do this, they must be based on repetition (which throws quite a different light on the quest to reduce redundancy in the mathematical communication theory, as will be presented below). Winkler’s “sign machines” are preceded by Winfried Nöth’s “semiotic machines,” the latter working not just with symbols but also icons and indices. Still, in Nöth’s conception, these icons and indices are identified with computer language commands (not with “expression,” in the sense typically given to this term by intermedial theory). The broad semiotic scope and horizons widened by the materiality of new media, and that are dear to intermedial theory, thus, escape Nöth’s perspective. However, Winkler gets somewhat closer to the different socialsemiotic functions, although he focuses on their economy (Winkler 2004). Yet, the

276

T. Chudy´ and R. Mu¨ller

circulation of signs in media stems from the interplay of their arrangement and use within this arrangement. The arrangement is, of course, given by the material articulation possibilities of the individual media. Here, we arrive at last from the other end to the concept of media difference as a semiotic difference. In order to better understand this concept, let us recall that according to Winkler, there are two “leaps” (Winkler 2004: 138) – one from the expression to the code and the other from the expression to media technology. And the (pre-)articulation of the material for the latter (a copper wire, for example) determines in advance what can be expressed (a reiteration of McLuhan’s media is the message that would then be: technics is always articulation and articulation is always technical). The progress compared to McLuhan lies exactly in the focus on semiotics: The body of signifiers (Signifikantenmaterial) is selected (articulated) according to functional criteria (Winkler 2004: 139). This selection then follows a specific logic of systemic deficit (Systemspannung); where a lacuna arises in the discourse (in the layer of signifiers) as produced by the current media, the new media technique steps in to produce what should have been produced but was not (here, there is an obvious link to Lotman’s concept of explosion in the semiosphere). Although this thesis is partly formulated in Saussurean terms, it may apply in a (more systematically technologized) Peircean world as well. In the US context, further alterations to the signification process have also been famously brought to the fore by Lev Manovich in his Language of New Media (Manovich 2001) – in media operation, signs are designed to represent in new ways but also to enable essential teleaction. This in a way reflects the development in Winkler, but Manovich claims that the nature of signs itself has changed – something that correlates with their fundamental determination by material articulation in Winkler.

Intermediality as a Meeting Ground Canonical Phase: A Reversal We are now in the position to understand how the field of intermediality has been influenced by the difficult relationship between semiotics (signification, communication) and technics/technology, and their study. The “canonical” definitions of the medium as a communication dispositif by Werner Wolf and Irina Rajewsky are illustrative of a certain tendency in the theory of intermediality to underestimate technics, a tendency it inherited from semiotic thinking. The guiding theoretical principle is found in the difference between semiotic systems, but the role of technics is only incidental. The very definition of the medium as a distinctive communication dispositif employing one or more semiotic systems is distinguished from an alternative (albeit original) definition of the medium (this one is seen as reductive) as a material-technical channel for the transmission of information. Rajewsky uses the same distinction in her Intermedialität (Rajewsky 2002: 7). Wolf writes: “In contrast to the way the concept is sometimes used in media theory, ‘medium’ here – as well as in the following discussion – does not, first and foremost, mean the mere

12

Intermediality, Semiotics, and Media Theory

277

technically and materially defined channel used to transmit information (such as writing, printing, broadcasting, CD, etc.), but a communication dispositif conventionally regarded as distinct in the sense of a cognitive frame of reference” (Wolf 2017: 65, translation our own.21 The cognitive approach to the communication dispositif means that it is characterized, first, by the usage of a certain sign system (or a combination of systems), and, only then, by the usage of a certain technical medium or communication channel. In other words, we first perceive the frame such as “this is a film message,” and within that frame of reference, we can begin to distinguish relevant communication channels. In actuality, the relationship between the technical and the semiotic dimension of the medium goes through a reversal in this canonical phase of intermedial research.22 The question is: Does not such a reversal mean we lose sight of the activity of the medium in the “rudimentary” sense – its “mere” technical-material side – either in the sense of the conditions and initiations of media development (even evolution), or in the sense of the extension and expansion of a sensorium, which is evidently also at play in the formation of meaning? Wolf, to be sure, does recognize the initiating role of technology, answering the question of what is empowered by the technical aspect when a new technical disposition is introduced into an established media constellation (e.g., hyperfiction; the transition from oral lyricism to the written form) but not in the actual definition and not in the foundations of his approach.

The Process of Media Differentiation Then, there is the question of senses (what Lars Elleström would call sensorial modality). Can McLuhan’s idea of sensorial extension be applied to the study of intermedial forms at all? McLuhan ties the extension of the senses to what he calls “amputation,” the lessening of a perceptual (and epistemic) capacity, as a simultaneous effect of the same process. Nevertheless, the idea can be applied to cognitive and semantic dimensions. In other words, extension entails and has a counterpart in blinding, not only at the level of cultural evolution (as McLuhan has pointed out in his own way) but also at the level of cultural “content” (meaning) and the semantics of an individual work. For example, it can be pointed out that visualization in calligram and experimental poetry suppresses traditional sound (verse) forms. Film can be said to make the fictional mind “opaque,” as opposed to the “transparenticization” that Dorrit Cohn discusses in her Transparent Minds as a sophisticated effect of the verbal narrative mediation and text construction. These examples correspond to a canonical In a different essay, the reductive meaning of the concept of the medium includes “institutionelle Übertragungskanäle” (Wolf 2017: 154) and is in need of being complemented by semiotic systems; it is the typical “nicht nur . . . sondern auch” (not only . . . but also) phrase against which there is no objection except that in the subsequent train of thought, the work of pure semiotic difference (from a cognitive perspective) typically prevails over a more inclusive approach 22 The genealogy of intermedial thinking is traced by Jedličková and Fedrová, 2020. 21

278

T. Chudy´ and R. Mu¨ller

intermedial maxim: Effects are always mutual, since the media always interact in the mediasphere and every medium is a mixed medium. A computer game typically weakens the need for semantic interpretation, while the opposite tendency introduces a process of differentiation into the media form (the interactive action adventure Heavy Rain attempting to bring in a new form of immersive fictional narration is a case in point). Such line of thinking, recognizing the semantic implications of understanding media as bodily extensions, finally concerns the question of virtual reality. Is virtual reality an ever eluding horizon of total media convergence, a promise of their erasure, the media degree zero? Is it a radical limit beyond which there is no extension (except by “rebooting”), but which is also a radical blindness? Be it as it is, the particulars of the technological support to sign production are important not only because of the evolutionary logic, where technics “inputs” its specific limits and possibilities. After all, these might become less central in the interplay between actual media usage and expectations. Technological support is important, because every new technics enabling new media usage also uncovers what previously seemed beyond communication and beyond representation. The resulting process, however, cannot be characterized merely by such motifs as acceleration, expansion, extension, more “reality,” etc. – motifs familiar to anyone reading the works of McLuhan, Bolter and Grusin, Manovich, Kittler, and many others. If in new media we are indeed able to grasp something which used to be uncommunicable (unshowable, etc.), we are “losing” something at the same time. This “something” is not a mere evolutionary residue, but a reminder of the deeper uncommunicability of the human mind (Luhmann), or from a different angle, a reminder of the fact that speed, range, and clarity of signal are insufficient conditions for both communication and representation, even if the number of situations where their heightened parameters become necessary are seen to increase. At the same time, and necessarily with the same frequency, the opposing communication phenomena can also be used and are indeed used often in the field of art media: slowness, limited range, impeded signal (noise), etc. Let us add: Noise, thus, becomes “cultural noise” (Meyer 1957 in the context of modernistic music), coded noise, or (what Lotman might have called) “cultural interference.”

The Impulse of Technical Functionalities (and Its/Their Limits) Roberto Simanowski is one of the intermedialists who makes up for the neglect of technical-media aspects of the classical intermedial theory. Significantly, the playful experiments that are presented as intermedially fruitful draw on the technical functionalities of new media and their aesthetic and gestural creativity. Their innovation is often described as “semantic overwriting” and “enhancing a symbolic value” (of public architecture), or even as the transposition of one semiotic layer into the medium itself (e.g., theatralization of the internet which thanks to its anonymity enables role swapping, assuming fictive identities, and – in the reverse direction – Chattheater, a staging of internet chats) (Simanowski 2006: 15–19). This seems to adumbrate his later emphasis that a close (i.e., semiotic) reading is

12

Intermediality, Semiotics, and Media Theory

279

unavoidable and that “eventually, it is necessary to move from phenomenology to semiotics, from description to interpretation” (Simanowski 2011: x–xi). Another way to theorize the relationship between the technical and semiotic is Uwe Wirth’s distinction of degrees of intermediality. They run from degree zero (still within the confines of expressive media only: thematization of one medium in another – literary reflection of painting) through medial modulation of the sign system (oral speech to writing to printing press to digital writing) and the ensuing reconfigurations and associations over to conceptual grafting (e.g., film-like cuts in contemporary writing) (Wirth 2006: 32 ff.). The reconfigurations of the sign system entail an important sub-distinction between its technical aspect and its ex-post staging, or, between its mere use and its conscious knowledge, which lead to a change in the conditions of the signs system’s performative embodiment and settingup – like when a literary text is adapted for theater. And this is then called “hard intermediality,” paradigmatically enacted through the computer’s integrative functions with its outer framing made by the hardware and inner framing by the software (Wirth 2006: 33–34). There, the intermedial operations are naturally based on indices and icons, with the hyperlink as a cardinal index serving also as a herald of change of the media contexts (Wirth 2006: 37–38). There are a number of objections to raise against such an outlook. Computer “integration” might erase crucial differences, including the “material modality” of media products (the digitized work of art as a support for unsuccessful interpretations, digitized music effacing acoustics, etc.), and suppress more complex hermeneutic codes and modes of understanding; certain media practices might be pushed forward without a corresponding need on the side of the audience (the book form remains preferable to readers), etc. Hyperlink too might be seen today as not much more than an emblem of our “scattered attention,” or critiqued for its role in dividing society owing to its power of “redirecting,” used in advertising and social media.

Media Qualification and “Laterality” of Technics Finally, Lars Elleström (2010) introduced a complex system of media relations, where various modalities of the medium are taken into account (material, sensorial, spatial, semiotic) and the ways fundamental media aspects group together are distinguished (technical, basic, qualified media). In Elleström’s distinction, technical media acquire an interesting “lateral” position.23 How does the model, indebted to Peircean semiotics, address the role of the technical (and material) disposition on the creation and interpretation of the sign? We will answer this question by taking the example of Elleström’s Media Transformation (2014) and his analysis of three short films by Jan Švankmajer.

The singling out of “technical media of display” in Elleström’s expanded model for understanding media relations (2021) seems unhappily to set the problem of technics/technology in larger sense aside. 23

280

T. Chudy´ and R. Mu¨ller

Elleström distinguishes sensorial configurations and spatiotemporal modalities associated with a particular qualified medium (such as fine art, animated film), a group of media products (such as Arcimboldo’s paintings), specific media products, and compound media characteristics (particular to a certain qualified medium, or a certain media product). Technicity is not absent, but it is “dissolved” both in analysis and in perception (which is subject to analysis). There is a statement, for instance, of a simple fact that moving formations on the screen can be stopped, while static formations on the canvas are immovable. Let us imagine, however, that a technical possibility arises to make a painted representation move. The moment this happened and the difference was complicated, the model would subsume the initiative of technics back again into a synchronic system of effects as cognitive phenomena. Technical constraints and possibilities, thus, never appear in the plane of their interdependence, governed by their inherent, underived logic or condition of possibility. A small step in the direction to make static pictorial representation move, albeit in purely digital form, is made by generative-adversarial-network-based systems, developed within the deep learning area of AI. The application called Deep Nostalgia, for instance, allows a particular photograph to be set in motion (independent of any spatiotemporally related series of photographs) by correlating it with an underlying video of a different but shape-related object, here, a human face. Without recognizing the technological principle whereby a moving image is grafted onto a static visual configuration through a series of approximations – much like the infamous “deep fakes” are created – the perception and interpretation of these animated digital images is bound to remain misguided. The faces of Thomas Alva Edison, Abraham Lincoln, or Queen Victoria, familiar from old photographs, awkwardly look around, smile slightly, and frown. If one were to understand these representations through metaphor, it is as if (the still very low) artificial intelligence itself came to life through these faces. The same technology is used to animate digitized paintings (e.g., the Mona Lisa or the Armada portrait of Queen Elizabeth I). The grotesque and disturbing effect of these animated photographs and paintings is not a question of the qualified medium, genre, or submedium, but the result of the limitations of the technical medium, which characterize the architecture of the AI system in question (GAN) in the current state of development. It’s a different story with Švankmajer’s stop-motion animation technique. The grotesque effect is created with the “anachronistic” intention of exposing an irreducible materiality that cannot be erased by the analogue film medium, and which is working its way to the surface in the same way that the natural and unconscious forces of decomposition and expansion are working their way to the surface and beneath the surface of things. In short, the qualification of the medium must also be qualified technically.

Conclusion The slowness which marked the emergence of the concept of the medium from the realms of artistic, linguistic, and philosophical discourse and practice equally affected the formation of modern semiotics beginning with Ferdinand de Saussure

12

Intermediality, Semiotics, and Media Theory

281

and Charles S. Peirce. It is shown not only that within this process the technological aspect constituted the “place of strongest resistance,” its “hard core,” but that we find a complementary problem in the newer field of the theory and philosophy of technics: an incipient suppression of the semiotic character of technological processes. If Saussure founded his linguistic approach on the systemic desubstantivization of language and linguistic inquiry, the media and especially technical side of communication acquired the form of symptoms, signs of suppressed forces: marginal notes on the phonograph, drawings illustrating the logic of the linguistic system while obscuring media difference, the sidelining of writing. If Peirce included technical objects in his thoughts on semiosis, it was within a universalist paradigm of processual logic where the technical logic sui generis remained concealed. Both “strategies” of concealment have affected the subsequent development of semiotics, even in the information age and after media discourse proper was formed. The characteristics of one media form (such as oral language, writing, text) were projected onto the larger part of the media field; media systems were (mis)understood as purely semiotic entities; the material modality of media remained marginalized; and the technological character of the medium was seen as a sole factor, rather than a complex transforming the conditions of signification. In the course of the formation of media discourse, all these tendencies in semiotically oriented communication and media theories had to be overcome, which applies to the theory of intermediality and intermedial research as well. From a complementary perspective, technological media science (best represented by the works of Friedrich Kittler) has been shown to develop out of a blindness to the semiotic repercussions of its own epistemological operations. As a consequence of its initial radicalism, technological media theory has long misperceived semiosis as a mere epiphenomenon of the technical functioning of media, and in particular of the practice of signal transmission and manipulation. Mesmerized by the immense opportunities of processing and storage in various materials, by their growing structural complexity (up to the current variety of digital memory media and their progressive miniaturization), and by the increasing speed of data processing, the theory, in search of new mathematical and engineering problems and solutions, has left behind the semantic, social, and cultural aspects of sign production. Even if the “extensions of man” model developed by McLuhan and his followers, which was itself based on a metaphoric or even iconic treatment of bodily and technical functions, has brought important insights into the cultural dynamic, no link with the field of semiotics had been established. Explicit appropriation of both technical and semiotic aspects has not occurred until the recent offshoots of media and intermediality theory in the USA (Manovich and others) and Germany (besides the authors already mentioned, the systems theory of Niklas Luhmann and his successors occupies a special place in this context). Even in the field of intermediality, we can see an initial disbalance in the shifting of attention from technics to semiotics, a reversal in weighing the importance of the technical and the semiotic (cognitive) dimension of the communication dispositif. The consequence of this reversal is that intermedial research has tended to lose sight of the rudimentary plane of the medium, either in the sense of the conditions and

282

T. Chudy´ and R. Mu¨ller

initiatives of media development (even evolution) or in the sense of the extension and expansion of a sensorium, which is evidently also at play in the formation of meaning. Importantly, even today, it is still not clear how to define the different arrangements of media coding (its levels or modes) nor what the significance of the technical qualification of qualified media is, let alone how to grasp it (the intrusion of AI into artistic practice – and vice versa – offers a good insight into what is at stake). The critical aspects of understanding media (not absent in McLuhan, for instance) have also not fully come to the fore in intermedial research. The relationship between technics and semiotics, thus, appears to be a complicated and chiasmatic one, marked by a continuous impetus of development. Technics and semiotics need to be studied together. Acknowledgments This text was proofread within the project Development of Research and Popularisation Resources of the Institute of Czech Literature of the CAS, CZ.02.2.69/0.0/0.0/ 18_054/0014701, co-funded by the EU’s European Structural and Investment Funds within the operational programme Research, Development and Education.

References Attridge, Derek. 1997. Roland Barthes’s Obtuse, Sharp Meaning and the Responsibilities of Commentary. In Writing the Image After Roland Barthes, ed. J.-M. Rabaté, 77–89. Philadelphia: University of Pennsylvania Press. Aumont, Jacques. 1997. The Image. Trans. C. Pajackowska. London: British Film Institute. Barthes, Roland. 1977. Musica Practica. In Image, Music, Text, Essays Selected and Translated by Stephen Heath, 149–154. London: Fontana Press. ———. 2010. Camera Lucida: Reflections on Photography. Trans. R. Howard. New York: Hill and Wang. [1980]. Bolter, David J., and Richard Grusin. 2000. Remediation: Understanding New Media. Cambridge: MIT Press. Brugman, Karl. 1889. Grundriss der vergleichenden Grammatik der indogermanischen Sprachen, Volume 2. Strassburg: Karl J.Trübner. Bühler, Karl. 2011. Theory of Language: The Representational Function of Language. Trans. D. F. Goodwin. Amsterdam: John Benjamins. [1934]. Chalmers, David. 2020. GPT-3 and General Intelligence. Daily Nous. https://dailynous.com/2020/ 07/30/philosophers-gpt-3/#chalmers. Accessed 10 June 2022. Chvatík, Květoslav. 1970. Strukturalismus und Avantgarde. München: C. Hanser Verlag. Clüver, Claus. 1989. On Intersemiotic Transposition. Poetics Today 10: 55–90. de Mauro, Tullio. 1972. Notes biographiques et critiques sur F. de Saussure. In Ferdinand de Saussure, Cours de linguistique générale, 319–389. Paris: Éditions Payot & Rivages. de Saussure, Fedinand. 2011. Course in General Linguistics, ed. Perry Meisel and Haun Saussy. Trans. W. Baskin. New York: Columbia University Press. [1916]. Derrida, Jacques. 1981. Plato’s Pharmacy. In Dissemination, 61–172. Trans. Barbara Johnson. Paris: Editions du Seuil. Eco, Umberto. 1968. La struttura assente. Introduzione alla ricerca semiologica. Milano: Bompiani. ———. 1976. A Theory of Semiotics. Bloomington/London: Indiana University Press. ———. 1989. The Open Work. Cambridge: Harvard University Press. [1965].

12

Intermediality, Semiotics, and Media Theory

283

Elleström, Lars. 2010. The Modalities of Media: A Model for Understanding Intermedial Relations. In Media Borders, Multimodality and Intermediality, ed. idem, 11–48. New York: Palgrave Macmillan. ———. 2014. Media Transformation: The Transfer of Media Characteristics Among Media. New York: Palgrave Macmillan. ———. 2021. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In Beyond Media Borders, ed. Lars Elleström, vol. 1, 3–91. Cham: Palgrave Macmillan. Geoghegan, Bernard Dionysius. 2011. From Information Theory to French Theory: Jakobson, LéviStrauss, and the Cybernetic Apparatus. Critical Inquiry 38: 96–126. Guillory, John. 2010. Genesis of the Media Concept. Critical Inquiry 36: 321–362. https://doi.org/ 10.1086/648528. Gumbrecht, Hans Ulrich. 2013. Afterword in Friedrich Kittler, Truth of the Technological World, Essays on the Genealogy of Presence. Stanford, California: Stanford University Press. Jakobson, Roman. 1971. On Linguistic Aspects of Translation. In Selected Writings 2: Word and Language, ed. Stephen Rudy, 260–266. The Hague: Mouton. [1958]. ———. 1981a. Language in Operation. In Selected Writings 3: Poetry of Grammar and Grammar of Poetry, ed. Stephen Rudy, 7–17. The Hague: Mouton. [1949]. ———. 1981b. Linguistics and Poetics. In Selected Writings 3: Poetry of Grammar and Grammar of Poetry, ed. Stephen Rudy, 18–51. The Hague: Mouton. [1958]. Jedličková Alice, and Stanislava Fedrová. 2020. Teorie intermediality. Zjevnost vztahů, unikavost média. In Richard Müller, Tomá Chudý, et al., Za obrysy média: Literatura a medialita, 501–562. Praha: Karolinum/Ústav pro českou literaturu AV ČR. Kittler, Friedrich. 1993. Dracula’s Vermächtnis. Leipzig: Reclam Verlag. ———. 1999. Gramophone, Film, Typewriter. Transl. Geoffrey Winthrop Young and Michael Wutz. Stanford, CA: Stanford University Press. ———. 2006. Thinking Colours and/or Machines. Theory, Culture & Society 23 (7-8): 39–50. ———. 2009. Musik und Mathematik, I.II. München: Wilhelm Fink. Kittler, Friedrich, T. Macho, and S. Weigel. 2014. Zwischen Rauschen und Offenbarung: Zur Kultur- und Mediengeschichte der Stimme. Berlin, Boston: Akademie Verlag. Köbler, Gerhard. 2014. Indogermanisches Wörterbuch, 5th edition. http://www.koeblergerhard.de/ idg/idg_d.html. Krämer, Sibylle. 2006. The Cultural Techniques of Time Axis Manipulation. Theory, Culture & Society 23 (7–8): 93–109. Lotman, Yuri. 2000. Universe of the Mind: A Semiotic Theory of Culture. Trans. A. Shukman. Bloomington: Indiana University Press. ———. 2009. Culture and Explosion. Trans. W. Clark. New York: Mouton de Gruyter. Manovich, Lev. 2001. Language of New Media. Cambridge: MIT Press. McLuhan, Marshall. 1964. Understanding Media: The Extensions of Man. London: Routledge. Matejka, Ladislav, and Irwin R. Titunik, ed. 1976. Semiotics of Art: Prague School contributions. Cambridge: MIT Press. Metz, Christian. 1983. Psychoanalysis and Cinema: The Imaginary Signifier. Trans. C. Britton et al. London: Palgrave Macmillan. Meyer, Leonard. 1957. Meaning in Music and Information Theory. The Journal of Aesthetics and Art Criticism 15: 412–424. Mukařovský, Jan. 1976 [1936]. Art as Semiotic Fact. In Semiotics of Art: Prague School contributions, ed. Ladislav Matejka and Irwin R. Titunik, 3–9. Cambridge: MIT Press. Müller, Richard, and Pavel Šidák. 2020. Reflexe mediality v Pražské škole. In Za obrysy média: Literatura a medialita, ed. Richard Müller, Tomáš Chudý, et al., 91–116. Praha: Karolinum/ Ústav pro českou literaturu AV ČR. Müller, Richard, Tomáš Chudý, et al. 2020. Za obrysy média: Literatura a medialita. Praha: Karolinum/Ústav pro českou literaturu AV ČR.

284

T. Chudy´ and R. Mu¨ller

Nöth, Winfried. 2003. Semiotic Machines. S.E.E.D. Journal (Semiotics, Evolution, Energy, and Development) 3: 81–99. Peirce, Charles S. 1960–1966. In The Collected Papers of Charles Sanders Peirce, ed. Charles Hartshorne and Paul Weiss, vol. 8 vols. Cambridge: Harvard University Press. [CP]. ———. 1986. In Writings of Charles S. Peirce: A Chronological Edition, 1872–1878, ed. Christian J.W. Kloesel, vol. 3. Bloomington: Indiana University Press. [W 3]. Peirce, Charles S. 2000. Writings of Charles S. Peirce: A Chronological Edition, 1889–1890, vol. 6, ed. Nathan Houser et al. Bloomington: Indiana University Press. [W 6] ———. 1998. In The Essential Peirce: Selected Philosophical Writings (1893–1913), ed. Nathan Houser et al., vol. 2. Bloomington: Indiana University Press. [EP 2]. Petříček, Miroslav. 1989. Sémiologie – od Saussura k Peircovi? Kritický sborník 9: 14–27. Pokorny, Julius. 1959. Indogermanisches etymologisches Wörterbuch. Bern: Francke. Rajewsky, Irina O. 2002. Intermedialität. Tübingen: Francke. Salisbury, Laura, and Stephen Sale. 2015. Editors’ Introduction to Kittler Now. Current perspectives in Kittler Studies. Cambridge (UK)/Malden (USA): Polity Press. Shannon, Claude Elwood. 1948. A Mathematical Theory of Communication. The Bell System Technical Journal 27: 379–423. 623–656, July, October. https://doi.org/10.1002/j.1538-7305. 1948.tb01338.x. Simanowski, Roberto. 2006. Transmedialität als Kennzeichen moderner Kunst. In Transmedialität. Studien zu paraliterarischen Verfahren, ed. Urs Meyer, Roberto Simanowski, and Christoph Zeller, 39–81. Göttingen: Wallstein-Verlag. ———. 2011. Digital Art and Meaning. Reading Kinetic Poetry, Text Machines, Mapping Art, and Interactive Installations. Minneapolis/London: University of Minnesota Press. Simondon, Gilbert. 2016. On the mode of existence of technical objects. Minneapolis: Univocal Publishing. Ungeheurer, Gerold. 2004 [1967]. Die kybernetische Grundlage der Sprachtheorie von Karl Bühler. In Sprache und Kommunikation, ed. H. Karin Kolb and Walter Schmitz, 128–146. Münster: Nodus. Van Brugen, Isabel. 2022. Russia Arrests Multiple People for Holding Up Blank Signs. Newsweek Online. https://www.newsweek.com/russia-ukraine-war-invasion-protests-police-arrest-activ ists-holding-blank-signs-paper-1687603. Accessed 26 May 2022. Veltruský, Jiří. 1976. Some Aspects of the Pictorial Sign. In Semiotics of Art: Prague School Contributions, ed. Ladislav Matejka and Irwin R. Titunik, 245–261. Cambridge: MIT Press. Winkler, Hartmut. 1997. Docuverse. Zur medientheorie der Computer. München: Klaus Boer. ———. 2004. Diskursökonomie. Versuch über die innere Ökonomie der Medien. Frankfurt am Main: Suhrkamp. ———. 2008. Zeichenmaschinen, oder warum die semiotische Dimension für eine Definition der Medien unerlässlich ist. In Was ist ein Medium? ed. Stefan Münker, 211–221. Frankfurt am Main: Suhrkamp. Wirth, Uwe. 2006. Transmedialität. Zur Ästhetik paraliterarischer Verfahren. In Transmedialität. Studien zu paraliterarischen Verfahren, ed. Urs Meyer, Roberto Simanowski, and Christoph Zeller, 19–38. Göttingen: Wallstein-Verlag. Wolf, Werner. 2017. In Selected Essays on Intermediality by Werner Wolf (1992–2014), ed. Walter Bernhart. Boston: Brill Rodopi.

Intermediality and/in Translation

13

Marta Kaźmierczak

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality as (a Product of) Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intersemiotic Translation: Naming the Phenomenon and Delimiting Its Scope . . . . . . . . . . . Beyond Intersemiotic Translation: Variations of the Term and Holistic Concepts . . . . . . . . . What’s in a Name? Adaptations, Re-creations, Transmediations, and More . . . . . . . . . . . . . . . Intermediality and Its Translational Vicinities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intersemiotic Translation and Translation Proper: Similarities and Research Insights . . . . . Translation Studies Addresses Intermediality in Translated Texts . . . . . . . . . . . . . . . . . . . . . . . . . Interlingual Translation Faces Intermediality: Examples and Concepts in Action . . . . . . . . . Interlingual Translation Behind Intermedial Transactions: Unacknowledged Presences . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

286 287 287 291 293 298 298 300 305 310 314 316

Abstract

The chapter describes interrelations between intermedial phenomena and translation from several angles. First, intermediality is discussed as a product of translation (transformation into a different medium), with a range of examples and an account of the development of the theoretical concept of intersemiotic translation and of its uses to capture and probe intermedia relations. It is shown that intermedial transformations are investigated under various names, and it is probed whether terminology makes a difference to the scope and ways of scholarly interrogations. Secondly, the interface of intermediality with translation proper is outlined. Similarities between intersemiotic translation and interlingual translation are indicated. Intermediality in polycode texts or in the form of intermedial references embedded in verbal texts constitutes a potential challenge for translators and becomes a translation studies issue. Consequently, methodological propositions which account for that are described and problems are M. Kaźmierczak (*) Institute of Applied Linguistics, University of Warsaw, Warsaw, Poland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_15

285

286

M. Kaźmierczak

highlighted by empirical material. Importantly, the co-presence of various codes is currently recognised as both a constraint and help in the translation process. The final section demonstrates that interlingual translation is often a middle step between a source text and its remediation and indicates implications of this. Selected studies are cited to exemplify potential lines of analysis, and the necessity for interdisciplinary approaches to the intermedial-translational entanglements is underscored. Keywords

Intermediality · Intersemiotic translation · Interlingual translation · Semiotic codes · Terminology

Introduction Semiotic complexity of certain artistic phenomena as well as interrelations between arts both have a long history in culture and have been subject to comparative studies long before the rise of the notion intermediality (see, e.g., Rajewsky 2005: 44). This, however, has not necessarily been matched by equal interest in their translational embedding and contexts. The perspectives of intermediality and of translation were explicitly first brought together in Roman Jakobson’s 1959 proposition of intersemiotic translation. The considerations adopted a linguistic perspective, but almost in passing Jakobson outlined a seminal differentiation of translation into intralingual (rewording), interlingual (also called “proper,” in line with the prototypical construal of translation as mediating between two natural languages), and intersemiotic – “interpretation of verbal signs by means of signs of nonverbal sign systems” (1959/2000: 114), which he also called transmutation. In the discussion below, the notion of “intersemioticity” will be used largely synonymously to “intermediality,” as a consequence of the existence of a set of terms including the adjective, a terminological tradition which takes Jakobson’s proposition as a point of reference. A second terminological caveat to be made is that translation will be construed in the strict sense, that is, as a process involving a transformation of signs or a product of such a process (while static states on the one hand, and processes which do not relate to signs on the other do not constitute translations). Loose or metaphorical applications of the term may be referenced for explanatory purposes. As Irina Rajewsky (2005: 51–52) explains, intermediality can usefully be construed as three broad categories of phenomena: medial transpositions (where intermediality manifests itself in how a media product emerges), media combination (where it is an inherent quality of a certain genre of media, like theater or film), and intermedial references (evoking or imitating entities of a medium in a different medium, e.g., painting or cinema in literature). All three types can come into the orbit of translation as practice and become an issue in translation studies, albeit in different ways. The first part of the present chapter focuses on medial transpositions,

13

Intermediality and/in Translation

287

i.e., processes and phenomena which may be viewed as translational in character. The second part features all the three categories, given that all of them can be subject to interlingual translation, as well as function in various interlocks with it. Analogies between translation proper and intermedial transformations will also be considered, as well as possible methodological affinities in their respective study. An effort will be made to demonstrate that a reliable enquiry into the phenomena in question must be multidirectional and interdisciplinary.

Intermediality as (a Product of) Translation Intersemiotic Translation: Naming the Phenomenon and Delimiting Its Scope An account of the development of the theoretical concept of intersemiotic translation and its uses to capture and probe intermedia relations should naturally begin with establishing what phenomena and processes classify under this category. Jakobson’s concise definition cited above is only complemented by very general examples of such a transformation: “intersemiotic transposition – from one system of signs into another, e.g., from verbal art into music, dance, cinema, or painting” (1959/ 2000: 118). Thus, the transformations explicitly named by the scholar account for the following types of transmutation (unless stated otherwise, examples throughout the chapter are mine – M.K.): – From word to music – program music (instrumental), e.g., Benjamin Britten’s Six Metamorphoses after Ovid, op. 49. – From word to dance – ballets whose librettos derive from verbal arts; examples range from established classics like Ludwig Minkus’ Don Quixote to new creations like Alice in Wonderland with Joby Talbot’s music, first directed by Christopher Wheeldon in 2011 for The Royal Ballet. – From word to cinema – countless instances, given that cinema from its inception relied on literary resources, e.g., István Szabó’s Mephisto (1981) from Klaus Mann’s 1936 novel or Joe Wright’s 2007 drama Atonement from Ian McEwan’s novel (2001). – From word to visual arts – with Shakespeare being the source of innumerable paintings capturing episodes from dramas, to cite only Titania and Bottom by Johann Heinrich Füssli (1793–1794, Kunsthaus Zürich) or Claudio and Isabella by William Holman Hunt (1850, Tate Gallery). The implicit flexibility of Jakobson’s concept manifests itself in including in its scope the transformation into a cinematic product. The scholar thus concedes that intersemiotic translation may result not only in a change of semiotic system but also in the emergence of a multicode, multimodal work. The change in the number of codes came to be addressed theoretically later on (as discussed in section “Beyond Intersemiotic Translation: Variations of the Term and Holistic Concepts”),

288

M. Kaźmierczak

but subsuming film under the scarce definition seems to rely on a fact acknowledged early in the history of cinema, that it has its own “language,” however syncretic. Film adaptations of literature have been called intersemiotic translations and approached as such (among others: Hopfinger 1974; Cattrysse 1992). They even came to be treated as prototypical forms of intersemiotic translation and profited from numerous academic investigations. A similar reasoning is behind treating as intersemiotic translations also theater adaptations of sources originally not meant for the stage and musical-verbal works whose librettos are re-workings of verbal texts (e.g., Hector Berlioz’s 1846 dramatic legend The Damnation of Faust, op. 24, operas like Giuseppe Verdi’s La Traviata, Charles Gounod’s Faust, or Nico Muhly’s 2017 Marnie, based on Winston Graham’s novel). This extends to more recent genres, like computer games based on literary material (say, on The Lord of the Rings). Jakobson’s enumeration of the domains to which a product of transmutation can belong is by no means intended as a closed catalogue. However, his proposition is unidirectional: it only envisages translating from a human language into a nonverbal code (or not purely verbal one). Hence, Jorge Díaz Cintas suggested extending the definition of intersemiotic translation onto the interpretation of nonverbal signs by verbal ones, so that it covered phenomena such as audio description (cf. Díaz Cintas 2005: 4), perceived both popularly and professionally as a form of translation. Actually, a theoretical framework that assumed unrestricted directionality had already existed. In 1986 Gideon Toury postulated a general binary division into intersemiotic translation (which involves a change of code) and intrasemiotic translation – within the scope of one system of signs. Jakobson’s intra- and interlingual translation would both then be branches of the intrasemiotic (Toury 1986: 1114). A number of scholars in various countries apparently independently came up with the same solution (Balcerzan 2011: 303; Eco 2004: 236). Even before, reinterpreting Jakobson’s concept as a transaction between different semiotic systems explicitly underwrote empirical analyses (e.g., Hopfinger 1974: 82). Toury’s framework allows us to classify as transmutation further phenomena: – A change from music into text – e.g., Kornel Ujejski’s “Translations from Chopin” (Tłumaczenia Szopena, a cycle of poems in Polish, published in 1866) – From film into (narrative) text, i.e., novelisations of films – e.g., Willow by the Canadian writer Wayland Drew from the American fantasy drama directed by Ron Howard (both 1988) – Reworking of the visual code into the verbal one – audio description (intra- and interlingual) Moreover, by criticising Jakobson’s division as being “readily applicable only to texts” (1986: 1113), Toury implicitly admits cases of “translation” in which (pure) language code is not involved on either side. This would account for extending the notion of transmutation over still more types of transformations:

13

Intermediality and/in Translation

289

– From a graphic novel into a film – e.g., Persepolis (2000–2003) by Marjane Satrapi ! Persepolis written and directed by the author in 2007. – From a computer game into a film – e.g., Assassin’s Creed (2007) ! Assassin’s Creed directed by Justin Kurzel (2016). – From a film into a theatrical performance – like The Lion King, the 1997 Broadway musical based on the 1994 animated Disney film of the same title. – From a film into an opera – a relatively recent type, the earliest example that the present writer was able to locate is Orphée, a 1991 chamber opera by Philip Glass, based on Jean Cocteau’s film Orphée from 1950 (followed by the composer’s further homages to Cocteau: La Belle et la Bête, 1994, and Les Enfants Terribles 1996 – in these cases the films under transformation themselves have literary precedents); the film-to-opera practice has been on the rise in the twentyfirst century, with works like The Exterminating Angel composed by Thomas Adés in 2016 on the basis of Luis Buñuel’s 1962 film El ángel exterminador. – From opera to ballet – e.g., George Balanchine’s La Sonnambula selectively based on Vincenzo Bellini’s 1831 opera, with a changed plot (more about the nonobvious in this type of transformation: Anderson 2002). – From visual arts to music – e.g., Goyescas: Los majos enamorados, piano suite op. 11, composed by Enrique Granados in 1911, or Ottorino Respighi’s Botticelli Triptych, whose three movements reference three masterpieces housed in the Uffizi Gallery: “Spring,” “The Adoration of the Magi,” and “The Birth of Venus.” – From music to visual arts – e.g., American composer Robert Jager’s 1996 Suite from ‘Edvard Munch’; Danuta Kwapiszewska’s sculpture Dancer: FantaisieImpromptu by Frédéric Chopin (orig. Tancerka. Fantazja-Impromptu Fryderyka Chopina). – From the visual to the musical-verbal – William Hogarth’s cycle of paintings, and then engravings, as foundation for Igor Stravinsky’s 1951 opera The Rake’s Progress: A Fable (with libretto by W.H. Auden and Chester Kallman). – Animation based on music – e.g., Wiesław Bober’s realisations of Antonio Vivaldi’s Le quattro stagioni (The Four Seasons) for the Poznań Animation Studio, 1990–1995. The broadening of the scope of the term since 1959 has naturally acknowledged the emergence of new modes of cultural expression and new translation practices which they have triggered. Expanding the definition should, however, be restricted on three accounts: (1) very faint homology between the languages of interacting media/art forms, (2) a transaction not being of a translational character (but, say, transcoding or complementation), and (3) a translation not being intersemiotic one (not involving a change of code). As for problematically distant homologies, already some of the final examples from the above list may be found controversial and may trigger demands that they be reclassified as inspirations, not translations, inasmuch as the mutual translatability of music and visual arts has been contested rather than satisfactorily substantiated. Admittedly, there is a semiotic classification of translation types which includes inspirational translation, as opposed to its conventionalized forms (Gottlieb 2007,

290

M. Kaźmierczak

esp. 2007: 36–37, 39); nonetheless, the demarcation and criteria of allocating to categories are not sufficiently clear and convincing. Meanwhile, for instance, in the oeuvre of the Lithuanian composer and painter Konstantinas Čiurlionis (1875–1911), his mostly abstract canvases titled “sonatas” and “symphonies” strive to convey musical impressions but are not declared translations of his own or anyone else’s musical compositions. This is much less problematic in the case of a cycle of images, as the existence of narrative that can be adopted in musical or musicalverbal transformations (like Stravinsky’s opera) provides a palpable, identifiable common denominator for the source and the target artwork. The requirement of translational character prevents subsuming under the name “intersemiotic translation” phenomena which amount to transcoding (while a mere re-expression of data is not necessarily a translation) or which do not involve a transformation of semiotic components, but their accumulation – therefore constituting combination, not transposition of media (see Rajewsky’s differentiation cited in the Introduction). The third criterion may appear patently obvious, yet the entanglement with other semiotic resources occasionally causes some specialised types of interlingual mediation to be mistaken for intersemiotic translation. However, if a process does not entail transposing from one system of signs to another, there is no change of code. As such, it does not fit Jakobson’s definition of transmutation (above and 1959/2000: 114–118). Table 1 demonstrates phenomena for which the label “intersemiotic translation” is appropriate in contrast with ostensibly similar ones for which it is not. The issue will yet return, but with reference to complementations, it bears stressing that the above does not amount to claiming that no illustration is ever a translation – the caveat is only made that illustrating is not by nature or by definition a translational activity per se, and illustrators are not bound by obligations of translational kind (the argumentation is fully developed in Kaźmierczak 2018a: 16–26). Similarly, in Claus Clüver’s view, only “certain” illustrations classify as intersemiotic transpositions (2007: 24, 27), and only some ekphrases1 do (1989: 68, 82–83), although his criteria are not explicit. In the same vein but with reference to genre, Clüver gives the status of word-to-music transpositions to tone poems but Table 1 Intersemiotic translation vs media combination or interlingual translation Intersemiotic translation Graphic novel based on literature Program music Audio description Film adaptation Computer game based on literature or film

Not intersemiotic translation (but. . .) Illustrating (complementation, i.e., media combination) Setting a text to music (complementation) Translating a song, an opera for singing (“melic” or “vocal” translation) Translating a film script (audiovisual translation) Translating a computer game (localizing)

Table adapted from (Kaźmierczak 2018a: 27)

1

A verbal ekphrasis is essentially a representation.

13

Intermediality and/in Translation

291

not to songs (2007: 24) – assumedly, the latter represent media combination. Translating multimodal items, typified by film, is not intersemiotic translation either. Audiovisual translation constitutes a mediation between two languages, not between two different semiotic systems (cf., e.g., Remael 2001: 13–14), despite the verbal text being embedded in a polycode medium.

Beyond Intersemiotic Translation: Variations of the Term and Holistic Concepts Many of the terminological and methodological considerations concerning transactions between media are indeed underwritten by concerns for comparability of messages formulated in different semiotic systems and for disparate semiotic resources available (e.g., Rajewsky 2005: 50). In 1989 Claus Clüver embarked on proving analytical usefulness of what he preferred to term intersemiotic transposition. (It should be noted that the noun “transposition” appears in Jakobson’s explanation of translation; see quote in section “Intersemiotic Translation: Naming the Phenomenon and Delimiting Its Scope”). In his paper, he sets out to validate the idea of such transformations in the first place, claiming that “a successful intersemiotic transposition should not be considered less possible than a successful interlingual translation of a poem” (1989: 62). He proceeds to demonstrate that approaching a work as an intersemiotic transposition may significantly influence the reading or interpretation of it (1989: 62ff). However, the rationale like Clüver’s, that the feasibility of interlingual translation justifies in principle the possibility of conveying comparable messages across different codes and media, is by no means universally adopted, in view of the essential lack of homology between particular systems.2 In particular, Umberto Eco was, as a semiotician, keenly aware of the problem and very articulate on how various media have different means at their disposal and therefore of necessity communicate different things. Consequently, he declined to use the term “translation” whenever the physical matter was changed – while sustaining the use of the word transmutation (2004: 315–344). Moreover, Eco formulates his concept of translation with reference to that of reversibility of the process, allowing to recover essential meanings of the source (2004: 67). This premise is not taken to extremes (only transcoding is fully reversible), but its adoption invalidates the concept of intersemiotic transactions as translations. To try this out on examples given earlier in this chapter (section “Intersemiotic Translation: Naming the Phenomenon and Delimiting Its Scope”), it becomes immediately obvious that not only is Chopin’s composition not recoverable from a sculpture but also that reconstructing a novel on basis of even a most faithful cinematic adaptation would hardly produce a

2

One to radically negate it has been Émile Benveniste, who is explicit that signs pertaining to various semiotic systems are not mutually convertible and that between the systems only a certain homology may obtain, and an arbitrary one at that (1969: 9).

292

M. Kaźmierczak

resemblance to the source. Besides, in his considerations Eco subjects the notion of translation to that of interpretation and therefore talks about intersemiotic interpretation (2004: 236). He actually postulates a tripartite division into intra- and intersystemic interpretation and transcription (2004: 236), which entails grouping together intersemiotic and interlingual interpretation (translation proper) as branches of the intersystemic one. Henrik Gottlieb (2007) proposed a complex semiotic typology with reference to the conceptualisation of translation as multidimensional and modelled it for a number of parameters. One of them being semiotic identity or nonidentity between the source and the target text, he defines intersemiotic translation as one in which “the one or more channels of communication used in the translated text differ(s) from the channel(s) used in the original text” (Gottlieb 2007: 35). He further applies (across both inter- and intrasemiotic types) a division that helps describe the source and the target work in terms of comparable or altered semiotic complexity, i.e., the number of semiotic codes. Diasemiotic translation consists in changing the semiotic channel(s), while the categories of supersemiotic and hyposemiotic translation describe processes as a result of which the target text displays respectively more and fewer semiotic channels than the original (2007: 36). Worth noting are also the concepts which integrate the totality of culture as translational-semiotic-intermedial processes. George Steiner in his seminal book After Babel (Steiner 1975) considers a wide variety of intersemiotic relations and discusses them in connection with, or separately from, interlingual translation. Particularly important from our point of view is Chap. 6, “Topologies of Culture” in which the scholar recasts the notions of culture and translation in terms of kinesthetic concepts. Since for Steiner culture is to a large extent “the translation” of previous meaning(s) (Steiner 1975: 415), he examines topics, iconology, motifs, archetypes, as well as genres. He indicates certain works “serv[ing] as focus for a ‘topological space’ of mutual readings and challenges” (Steiner 1975: 460). The discussion of the image of Hippolytus’ death in three languages – as presented by Euripides, Seneca, and Racine – is situated on the borderline between considerations of inspiration and translation (Steiner 1975: 430–436). Among the sequences of translations analysed between languages and art media, there is an intermedial one, built around a song from Goethe’s Faust, together with musicalisations of the original and of the French translation (Steiner 1975: 418–423). The chapter highlights the thesis that “a culture advances, spiralwise, via translations of its own canonic past” (Steiner 1975: 436), inspiring for any researcher of the relations between intermediality and translation. Peeter Torop’s work Totalnyi perevod (Тороп 1995) is an interesting proposition of a synthesising character. Starting from translation studies as an interdisciplinary field and from Mikhail Bakhtin’s concept of “another’s word,” the Estonian scholar formulates his theory of “total translation” as the passage of texts (in a very broad sense) into a foreign culture. Within the framework of the “total,” Torop distinguishes the following types of translation: textual (translation of a whole text into a whole text, an example of which is interlingual literary translation), metatextual

13

Intermediality and/in Translation

293

(translation of a whole text into a culture), in[tra]- and intertextual (a system of “intexts” as part of an author’s poetics and introducing another’s word into one’s utterance), and extratextual (translation of a text from a natural language into another type of code) (see Тороп 1995: 13–14). As can be seen, Torop defines as intertextual translation what is called intertextuality by most researchers, while Jakobson’s intersemiotic translation becomes extratextual for him. He provides an elaborate classification of the manifestations of the phenomena which he has outlined, but his research is only marginally related to interlingual translation; in fact it amounts to transferring the notion of translation into semiotics. Since the scholar treats the entire human textual activity at large as a kind of translation (the dialectics of the self and the other and the mediating role of culture are instrumental in this construal), his model has a theoretical character and does not purport to offer a research methodology for comparing originals and translations of whatever semiotic composition.

What’s in a Name? Adaptations, Re-creations, Transmediations, and More “Intersemiotic translation” is not the only designation for processes of transformation between media, and possibly not the one most frequent in current discourses. There has been a proliferation of terms, to which individual scholars, schools, or disciplines give preference in this interdisciplinary field. Recent terminological propositions include re-creation (Testa 2002), intersemiotic transposition (Clüver 1989; Dusi 2015), media substitution (Plett 1991, as a form among intertextual transformations), intermedia transposition (Wolf 2002), remediation (Bolter and Grusin 1999/2000, primarily in the context of new media), media transformation, and transmediation (Elleström 2017/2020). Before commenting in the context of translation and exigencies of research on selected new ones, let us note that adaptation is a label of a long tradition in discourses in most European languages, with the advantage of being part of general language as well, fixed in dictionaries, with a potential of being intersubjectively used by numerous writers and readers. Despite the emergence of new terms, it is still popular with film scholars (McFarlane 1996; Stam et al. 2005: 39, 60, 209, 214) and for broadly conceived collective multidisciplinary endeavors (like HajdukGawron and Madeja 2013); extending the application of the notion and of implied research paradigms to phenomena less studied, like illustrations or “sourced” pop-up books, has also been advocated (Newell 2017). Given that the term “adaptation” can have a number of distinct meanings with reference to interlingual translation, it has retained its currency at the interface of translation studies and multimedia research (e.g., Raw 2012; Krebs 2014). Carlo Testa devised his conceptual framework for cinematic transpositions of literature, with a starting point in discarding the label “adaptation,” underpinned, as he sees it, by patronizing attitudes (2002: 5). His subscribing to the notion re-creation (2002: 4–13 and passim) is motivated i.a. by the wish to underscore

294

M. Kaźmierczak

both the artistic autonomy of the target work and the importance of cultural contexts for its emergence and shape (2002: 8). His typology, if not absolutely consistent, features categories which may facilitate describing certain phenomena of medial transposition in general. Among other types, Testa distinguishes mediated re-creations, i.e., ones filtered through “an intermediate epiphany” of other works (2002: 18, 123–158), e.g., a cinematic version of an opera based on a literary text (2002: 19, for the analysis of the film, Carmen, see 2002: 125–142). This can be extrapolated whenever a studied work may also result from an intermedial operation consisting of several stages. It is an important asset, given that chains of subsequent transformations, accumulating and furcating, are frequent in the realm of intermedia, as can be illustrated by developing an example from the first section of the chapter. A chain with Granados’ Goyescas at the center is presented in Fig. 1. The nature of the operetta film, which won the Biennale Award at the 1942 Venice Film Festival, could be succinctly described as a mediated re-creation. The same label could perhaps be extended to interpret situations when intersemiotic translation relies on interlingual mediation. However, despite the fact that he is studying the relations of Italian cinema with European literature written in various languages, Testa (2002) does not refer to interlingual translations at all. Nicola Dusi (2015), in turn, propounds the already mentioned term transposition as one to replace others (like intersemiotic translation, transmutation, adaptation) and to subsume all transformations, whether they be “audiovisual, musical, theatrical, performative” (2015: 202). She indicates the prefix trans- as bringing in the semantics of going beyond the original, which in her view entails “multiplying its semantic potential” (Dusi 2015: 203). Nonetheless, the same affix also forms a part of the word. . .translation, while emphasising reinterpretation and relocation is a trait shared by many definitions or explanations of other terms, including adaptation. It is also worth noting that the words “transposition” and “to transpose” often appear in scholarship (including the present chapter) in a non-terminological sense, but in accordance with their lexicographical meanings, with regard to all kinds of transformations and transactions involving media. Fig. 1 A chain of intermedia transformations

Goya’s pictures

Granados’ piano suite Goyescas

Granados’ opera Goyescas operetta film Goyescas adapting the opera, dir. Benito Perojo

Consuelo Velázquez’s song “Bésame Mucho” on a motive from the suite/the opera

13

Intermediality and/in Translation

295

To continue the overview, the much cited concept of remediation is not clearly delineated, but it allows, among other things, for the notion of media remediating other media, in the sense of, e.g., transposing genres and their attributes rather than individual works between media. This goes beyond the notion of translation in scope. To illustrate, Nora Alter’s (2007) discussing audiovisual essay as a genre parallel to the literary essay is controversial under the signpost of translation (presumably intersemiotic), while it would have been more legitimately examined as remediation. For Lars Elleström (2017), media transformation is an overriding concept, denoting “transfer of transmedial characteristics” (2017: 667), the two kinds of which are transmediation and media representation. Elleström distances himself (2017: 670) from the loose understanding of remediation by Jay David Bolter and Richard Grusin (1999/2000: 55: “all mediation is remediation”), to further work toward his definition of transmediation. Remediation is a repeated mediation, with the latter meaning “a physical realization of entities” available for perception (2017: 668). Transmediation, in turn, involves “repeated mediation of equivalent sensory configurations by another kind of technical medium,” which makes it a transmedial remediation, strictly speaking, but Elleström opted for transmediation as a handier shorthand (2017: 670, emphasis in the source). Transmediation is an analytical perspective (2017: 668) and is, eminently, a broader notion than many others in the discussed group. As noted by Elleström himself, the change of medium “from libretti, scores, scripts” – then presumably into their intended stage, aural, screen, and other realisations – is rarely subsumed under the notion of adaptation (2017: 673–674). This is perhaps because transmediation seems a designation more apt for the enumerated types than any pre-existing one – the transformation does not resemble what one may readily call a translation or anything akin to it. The way from a novel to a film as a complete process is traditionally perceived as intersemiotic translation/adaptation, but giving flesh to a script is not universally so. Elleström’s further examples of situations not referred to as adaptations, like switching between the oral and the written mode (2017: 674), show a different angle. Such switches constitute a change of modality – a change of channel of perception in the recipients’ perspective – but not a translation.3 The concept of transmediation may help name – and thus position appropriately within the field of research – semiotic phenomena and processes that would be difficult to subsume under the term “translation” in a rigorously accurate translation studies paradigm. (A) translation (prototypically) re-creates a single textual basis, while multimodal works created from prior material often encompass more than one such base – and this means here material of the same semiotic kind, not integrating diverse semiotic codes like word and music. To exemplify, the Ukrainian composer Admittedly, Gottlieb (2007: 39) includes in his “total taxonomy” of translation even mere changes of modality (audiobook recording and musical notation) as “conventionalized” types, but this is as controversial as his inclusion of Morse code decryption and challengeable on similar grounds. For mere transcoding not being a translational activity, see Eco’s reservations with regard to operations on Morse code (2004: 236–237).

3

296

M. Kaźmierczak

Yevhen Stankovych’s opera When the Fern Blooms (orig. Коли цвіте папороть, 1978, first complete performance 2017; cf. Stankovych 2017) draws on EastSlavonic folklore (from folk epic material to ritual) as well as on Nikolai Gogol’s tales: Taras Bulba (1835/1842) and “St. John’s Eve” from the story collection Evenings on a Farm Near Dikanka (1829–1832, both in Russian). It would have been awkward to label it an intersemiotic translation of the heterogeneous sources, and such a classification could be called into question on the grounds of the emerging work not fitting the definition of translation as such. Conversely, it can well be designated a transmediation of all of the materials of which the librettist availed himself, and the designation will hardly be disputed. It could also be described by another term from Testa’s repertory, hypertextual re-creation, i.e., one which incorporates more than one text by a given author (2002: 19; he applies this framework to, e.g., Visconti’s Death in Venice (2002: 183–201)), provided that the definition be extended to allow for mixed authorship. Concepts which encompass but also clearly delineate such situations offer a methodological advantage inasmuch as – it bears repeating – many products of intermedial transactions integrate a number of sources even when purporting to rely on a single one. This is well recognised with reference to literary film adaptations, which often draw on canonical illustrations, on earlier filmed versions, etc (see, e.g., Newell 2017: 4 on The Wizard of Oz 1939 feature, or comments in section “Interlingual Translation Behind Intermedial Transactions: Unacknowledged Presences” on Zeman’s films). It can be seen that a preference for specific terminology may be matched by an analytical interest in a broader spectrum of intermedial relations or a shift in the kinds of media that primarily come into focus, with a lesser emphasis on the verbal. However, using or not using the word “translation” as the component of the terms does not mean a qualitative change in approach. This manifests itself in the attitude to broadly understood fidelity. The concept of fidelity is frequently dispraised as a bane of or menace to film adaption – with reference to criticism and popular reception (e.g., McFarlane 1996: 8–11). It should, however, be mentioned, that criticism does not always rely on a narrow understanding of adequacy or on restrictive notions about the adaptors’ creative prerogatives. When “Pinter fails Fowles” in Susan Lorsch’s view (1988), it is not because as a scriptwriter he took liberties with the substance of The French Lieutenant’s Woman. Quite the contrary, his creating “an objective correlative [. . .] of the novel’s double viewpoint” (1988: 145) through means as ingenious as adding a new twentieth-century plot about the filming of Fowles’ plot or his capturing the relationship between reality and fiction (1988: 149) is fully appreciated. The perceived failure lies in not fulfilling the purpose of the narrative experiments: not making the viewer experience (as readers are brought to) the absence of conventional narrative (as well as spiritual) authority (1988: 151–152), an unsettling freedom which distinguishes the modern human condition from the Victorian one. In fairness, reviewers do make allowances for differences in semiotic resources available to media and for media specificity, and demurrers are not typically motivated by essential misconceptions about the nature of intermedial transactions.

13

Intermediality and/in Translation

297

Whatever the ostensible demands of adequacy to the source, the scholars are unanimous that “fidelity,” a concept from which they usually distance themselves by using inverted commas, is infeasible in transactions that lead to creating a work in a different medium. It is not only the adaptation scholars who have been gladly announcing that, e.g., “the reel Shakespeare looks blatantly unauthorized” (Zhang 2008: 89), “a rigorously exact transposition from literature to film is not possible (or indeed, desirable)” (Testa 2002: 4), or “[f]idelity in operatic adaptation is literally impossible” (Stenström 2007: 131). Those who operate on translational categories assert the same (Kmieciak 2015: 213). Also, for instance, [Yelena] Brazgovskaya, who discusses verbalisation of music, i.e., evocations of music in literature, within the paradigm of intersemiotic translation (Бразговская 2014), comes to the conclusion that the transposition between the two media can only be “probabilistic” (2014: 43–44). Nonetheless, a definitional separation of intersemiotic translation and transmutation has been proposed as well (Kaźmierczak 2018b: 155), with the suggestion that the latter be reserved for very free intermedial transpositions. That variation in terminology does not determine a qualitative difference in approach is further illustrated by the analytical procedures applied, similar whether or not intermedial transformations are investigated as translations. Marta Kmieciak (2015) analyses Richard Strauss’ opera Salome as an intersemiotic translation of Oscar Wilde’s drama. First, she examines how the musical code re-creates or transforms meanings of its literary basis. That one may subvert the other is exemplified in Iokanaan’s dialogue with Salome: verbally, the Prophet declares utter indifference to Salome’s charms, yet music is at times amorous (2015: 225). Interestingly, the composer’s own libretto is based on the successful German stage translation (received much better than Wilde’s French-language original). Kmieciak does not probe the interlingual mediation – as it did not matter for the composer-librettist – but she looks for meaningful cuts and compressions applied to the playtext and any resulting shifts in meaning. She observes that references to love are excluded from the final monologue, thus making the operatic Salome more unambiguously perverse and steeped in desire, with the shade of the spiritual not coloring the carnal (2015: 225–226). Kmieciak extends her analysis to a specific stage realisation, the 2008 Metropolitan Opera production, to reveal how the director, Jürgen Flimm, relies partly on the opera’s text, but partly on the play itself, for elements of the sets or visual motives that re-shift stresses as compared with Strauss’s vision. For instance, the motive of the mirror, she claims, has been excluded from the libretto, so alluding to it visually must have been a response to Wilde’s source text (2015: 229). Kmieciak’s protocol is not substantially different from what it would have been for a reliable analysis under any other term than intersemiotic translation. Instead, what the example highlights is that research on intermedial relations entails a multidirectional enquiry into various components of works as well as requires triangulating between various mediators who exercise agency on various stages of the emergence of an actual multimodal product (for an opera, this is indeed a staging rather than a printed edition of the score).

298

M. Kaźmierczak

Intermediality and Its Translational Vicinities This part of the chapter will be devoted to outlining interrelations between intermediality and translation proper on various levels. They range from similarities between the two spheres, through intermedial qualities in works under translation, to significant, if often overlooked, background presences of interlingual mediation in producing transpositions or media combinations. The interlocks and interfaces generate the need for methodological reflection which helps to embrace the phenomena conceptually as well as to deal with them in translation practice. They also dictate some research questions, e.g., concerning the role of interlingual translation in shaping a final intermedial product.

Intersemiotic Translation and Translation Proper: Similarities and Research Insights Scholars have indicated certain parallels between intersemiotic translation and translation proper. George Steiner, in the context of words transformed into music, enumerates similarities on several levels. Firstly, he extends his concepts of trust, appropriation, incorporation, and restitution as phases of interlingual translation to hold for intersemiotic translation as well (Steiner 1975: 416). Moreover, the means at a composer’s disposal (key, rhythm, instrumentation) correspond in his view to the means available to a translator; a series of compositions to the same text is analogous to a translation series of the same source text. There is also the necessity to choose the tradition according to which one interprets/composes and the risk of misreading the “original”; the text may possibly be manipulated; there are cases of “improving” on the original. Finally, a great setting means an added value, as an outstanding translation does (Steiner 1975: 416–418). Steiner’s considerations feature the notion of a series of translations (in Anglophone scholarship usually referred to as retranslation(s)). Indeed, the potential to generate successive transformations into a certain medium looks akin to the seriality of translations into a given natural language, whereas renderings of the same source into various target media compare with a verbal text being translated into diverse target languages. For instance, Shakespeare’s Hamlet has received at least four operatic versions with librettos in four languages – Italian, French, Russian, and English – composed by Franco Faccio, 1865, Ambroise Thomas, 1868, Vladimir Kobekin, 2008 (a comic take), and Brett Dean, 2017. Across media, it has been turned into numerous musical compositions (e.g., Ferenc Liszt’s symphonic poem, S.104, and Ophelia-Lieder by Johannes Brahms or Henri Tomasi’s trombone quartet Être ou ne pas être), paintings (e.g., J.E. Millais’ Ophelia, 1851, Tate Gallery; Władysław Czachórski’s Actors before Hamlet, c. 1872, National Museum in Warsaw), feature film adaptations, animated shorts, ballets, radio plays, manga comic books, and video games – even the list of target domains is not complete here.

13

Intermediality and/in Translation

299

In a recent study, Izabela Szymańska (2020) surveys three English-language films (from 1981, 1995, and 2008) adapting Jane Austen’s Sense and Sensibility, to locate significant patterns as one would in a sequence of literary retranslations. She focuses on the following parameters: compressing and modernising the dialogue, the system of characters and characterisation, the idea of marriage, and the visual contrast between the manor which the heroines have to leave and the cottage to which they move. She discovers progressiveness in the changes applied to the source, with the successive intermedial versions catering more and more strongly to contemporary audiences on both the emotional level (partnership model of a relationship) and the cognitive level (strengthening the visual message about the change in financial status). Such sequences of works reveal another analogy. Interlingual retranslations of the same original remain in some or other relation to each other – see, for instance, the theorisation of the structure of the series by Anna Legeżyńska (1986/1999) or a recent detailed study by Sharon Deane-Cox (2014). The same may well hold for successive medial transpositions. The interrelations (rivalry, polemic character, homage, derivativeness, following an established tradition, etc – again, characteristics known to obtain between items representing translation proper) have been probed most frequently with regard to movie adaptations. Indeed, it seems to be an established truth in film studies that “diverse prior adaptations can come to form part of the hypotext available to a film-maker coming relatively ‘late’ in the series” (Stam et al. 2005: 214). Examples in other types of transformations have been discussed less frequently. Still, Clüver describes a German poetic transposition of Breughel’s painting that appears reminiscent of an earlier English ekphrasis (Clüver 1989: 75) – a relation made problematic by the difference in language and the fact that Clüver does not consider the earlier verbal take, W.C. Williams’, a transposition/ translation (1989: 68, 82–83). Cecilia Alvstad (2008: 101) only mentions – as a field requiring investigation – the illustrators’ familiarity with previous illustrations to the same (translated!) tale, and their possibly drawing on each other in their drawings. The category of (un)translatability, a central one in reflection (both scholarly and nonacademic) on translation, resurfaces in reflections on various types of medial transposition. Meanwhile in translation studies, concern about translatability has triggered warnings that the pictorial language is by no means universal – for the sake of translating multimodal humor (Kaindl 2004: 183) or with emphasis on the common cultural knowledge presupposed by a multimodal message (Remael 2001: 18). Untranslatability has also been attributed to paintings steeped in historically and culturally motivated symbolism, even when accompanied by verbal text trying to transpose and explicate their meaning (Bałuk-Ulewiczowa 2004). Furthermore, attempts at applying specific approaches worked out in translation studies to intersemiotic transformations have been made, especially with regard to film adaptations. In this field, polysystem theory has been proposed as a promising paradigm. The advantages specified by Patrick Cattrysse (1992: 54) can be formulated as follows: (1) it helps shift from source- to target-orientation; (2) it does not rely on strong equivalence or fidelity concepts and facilitates concentrating on what

300

M. Kaźmierczak

a given translation actually is instead of on what it should be; (3) it invites looking for more general mechanisms operating on bigger sets of translated text as opposed to what seems to be a prevailing tendency: comparing isolated instances of source text – target text pairs with limited further implications. Cattrysse convincingly shows on his corpus of films noir that norms, a key polysystem notion, can be identified as governing the production of adaptations (1992: 56–59). Nonetheless, as the scholar admits, this methodology gives one few tools for conducting a textual comparison (1992: 60). Alicja Helman and Wacław Osadnik (1996) complement his observations, i.a. stressing the constant interactions of any film system with other literary/film/cultural systems and making it explicit that popular culture phenomena, not only high artistic ones, are embraced by the paradigm and that importance is assigned to the study thereof (1996: 652). Szymańska shows (2020: 252–253) pertinence for the study of film adaptations of concepts adopted (cultural capital) or coined (patronage) within the paradigm of the cultural turn in translation studies (cf. Bassnett and Lefevere 1998), that is, one focused on acknowledging and probing cultural and social contingencies of translation practices. Feminist translation theory has been used for analysing instances of choreography as intersemiotic translation (Tsiakalou 2018). As a general guidance, Aline Remael’s remark may be useful that when confronted with intermediality (in her discourse: multimodality), translation scholars need not relinquish their trusted tools, but rather should contribute their expertise to constructing interdisciplinary approaches (2001: 21). On a different, not methodological note, but with a long historical tradition, a parallel between translating and artistic media is expressed in sets of metaphors that liken the translator to a painter (seventeenth–eighteenth century, e.g., John Dryden, Jean d’Alembert), engraver (France), lithographer (Vyazemsky), castmaker sculptor (Gnedich), etc. To be stressed, a comparison with arts by no means serves elevating the status of translators, given that the core idea of the metaphorical set is that of translator as a copyist – e.g., the commonality with painting is seen in the expectation to represent nature/the model. The category of mimesis serves here ascribing to translation a secondary status as to a derivative activity. Translation has traditionally been construed in mimetic terms, yet some of the stereotyping conceptualisations were early re-evaluated (cf. Tytler 1790/ 1992: 132 on translator and painter).

Translation Studies Addresses Intermediality in Translated Texts Works that are products of medial transposition or media combination as well as text containing intermedial references can be – and frequently are – subject to intralingual translation. Intermediality then becomes an issue with which translators are confronted and often proves a translational challenge. Therefore efforts to name and conceptualise the interrelations and their implications have also been undertaken from the perspective of translation studies. Methodological propositions meant to account for intermediality in interlingual translation include polysemiotic translation (Szczerbowski 2005), intersemiotic dominant (Bednarczyk 2010) and intersemiotic

13

Intermediality and/in Translation

301

aspects of translation (Kaźmierczak 2018a, b). The latter two also partially grow from concern about loose uses and misapplication of the term “translation” and of Jakobson’s notion, and the entailed confusion in the research field. Intersemiotic dominant of translation, a term proposed by Anna Bednarczyk, concerns identifying the translational dominant with regard to a polysemiotic text (Bednarczyk 2010). Dominant (a term adopted from structuralist literary studies) refers to a translation invariant, or the element(s) or feature(s) of the original whose preservation is necessary if the target text is to retain its essential characteristics and meanings (this has been variously theorised as objective, subjective, or intersubjective). Bednarczyk draws attention to the fact that in translating multimodal genres and works which include verbal elements, like opera, poster, concrete poetry, or pop music video, a translation dominant may well be located in the/a nonverbal layer (Bednarczyk 2010: 404). The concept of “intersemiotic translation dominant” envisages that when it is one of the semiotic codes that plays a key role in a polycode work, a translation would prioritise this layer as well (2010: 404 and passim). This layer will constitute an invariant, while the other or others may be sacrificed by the translator(s) or may be subject to modification in the translation process. That said, a given translator’s assumption of the dominant is subjective and may not be shared by translation critics (Bednarczyk 2010: 403, 404) or recipients. Admittedly, for many multicode products all semiotic components are equally important, as illustrated by concrete poetry (2010: 404–405). It should therefore be added that Bednarczyk’s proposal complements, in her intention, Tadeusz Szczerbowski’s concept of polysemiotic translation, which is, as Bednarczyk construes it, applicable when the semiotic codes of a translated work weigh equally (Szczerbowski 2005; cf. Bednarczyk 2010: 406, 411). Then an optimal translation requires an adequate and complete rendition of all layers.4 A third concept is that of intersemiotic aspects of translation (proper) by Marta Kaźmierczak (2018a, b). The term refers to such a situation of mediating between languages (or comparing language versions) in which taking into account other semiotic codes/layers apart from the verbal one is characteristic or even obligatory (Kaźmierczak 2018b: 159, italics in the source; for a broader contextualisation of the concept, see Kaźmierczak 2018a). The essence of this concept is differentiating between intersemiotic translation and certain related phenomena that tend to be mistakenly subsumed under this label. If intersemioticity (or intermediality) is Polysemiotic translation, as defined by Tadeusz Szczerbowski, “in contrast to intersemiotic translation, consist in that messages in at least two different semiotic codes do not replace each other, do not compete with each other, but coexist in one semiotic text or work” (Szczerbowski 2005: 123, trans. mine – M.K.). The author presents this phenomenon primarily on the example of advertising but also mentions such a polysemiotic genre as computer games (derived from literary works) (Szczerbowski 2005: 129). Szczerbowski draws attention to reception issues – the aim of polysemiotic translation is to make perception more attractive and the work more accessible (Szczerbowski 2005: 126, 128). It is not stated explicitly, but the conclusion seems to essentially amount to transforming from a single-semiotic medium to a multimodal (and polysensory) one. It would thus be an extension of, and a clarification on, Jakobson’s typology, similar to that of Gottlieb’s, albeit not that specific.

4

302

M. Kaźmierczak

understood as all kinds of references of a verbal text to nondiscursive media and arts, then the translation of works containing such references should not be called , since no change of code occurs (e.g., an ekphrastic poem rendered into a different language remains a poem). Neither does such a change take place in audiovisual translation or localisation: a film translated into a film and a game into a game remain within the same medium. Consequently, “translating between natural languages a polysemiotic work, or a verbal text which refers to non-discursive media, constitutes an interlingual transfer accommodating intersemiotic aspects and should be studied from precisely such a perspective” (Kaźmierczak 2018a: 31, emphasis in the source). The difference between intersemiotic translation and a translation of a multicode text is outlined in Fig. 2. When a literary text (in the example: Joseph Conrad’s novella) is transformed into a graphic novel (like the one by Catherine Anyango and David Z. Mairowitz), this constitutes intersemiotic translation (or Gottlieb’s suprasemiotic one). When the graphic novel is translated into another language (e.g., from English into Dutch), still the same codes form it, while the alteration within the verbal channel constitutes interlingual translation. (Interestingly, such an operation may pertain to intralingual translation, if an existing target-language rendition of the given literary source is used: in the cited case, the Dutch translator previously translated Conrad’s novella himself.) Intersemiotic aspects of translation also obtain between an interlingual translation and an intersemiotic translation that derive from the same source work, as illustrated in Fig. 3. The relationship may often remain latent; however, it becomes relevant for reception and scholarly interrogation when the two target works are brought together in an intentional and/or artistic way. For instance, a poetic translation may be employed to “bear witness” to a poetic film’s being a transmutation of the same original. Such situations may reveal that while both transactions have generated products that are satisfying in their own medium and preserving an unequivocal connection with the original (to a degree relative to the medium and the kind of

Fig. 2 Heart(s) of Darkness: intersemiotic translation vs interlingual translation of a bisemiotic text

13

Intermediality and/in Translation

303

Source text

ion lat ns ra

int er lin gu al

t ic

tra ns lat

ot mi rse

ion

e int

target text

intersemiotic aspects of translation

target work

Fig. 3 Intersemiotic translation, interlingual translation, intersemiotic aspects of translation

Fig. 4 Relations among a source and derived artworks in terms of adequacy (from Kaźmierczak 2018b: 164, caption changed)

transformation), their respective departures from their common source may create a significant gap between the derivate works. This has been demonstrated (Kaźmierczak 2018b) on the relationships between “Panna Anna,” a Polish poem by Bolesław Leśmian, the German short feature Lilith originating from it, and “Missy Trissy,” a verse translation of the poem, outlined in Fig. 4. The homology between the German film and the English version of Leśmian’s poem proves significantly weakened. The two emerged completely independently of each other, and the reason for their non-coordination is that intersemiotic aspects of translation have not been (and could not have been) taken into account in preparing the verbal translation. The very relation between the derivatives would elude definition and disappear from the field of research without the concept which brings it out in relief. While both Bednarczyk (2010: 413) and Kaźmierczak (2018a: passim) intend to define with greater precision the proper subject of translation research (excluding from it what does not bear any relation to the verbal or to the interlingual), the

304

M. Kaźmierczak

concepts by no means question the legitimacy of investigating intermediality or interrelation between semiotic resources in the context of translation. On the contrary, the intersemiotic aspects of translation are an extremely promising and rich research field (for a map see Kaźmierczak 2018a: 29–30). Indeed, the consciousness of the plurisemiotic character of translated items, of the materiality and medial character of cultural products, has been growing within translation studies and influencing the discipline. The postulate of taking carefully into account the illustrations when translating bimedial texts has repeatedly been made with reference to children’s literature (e.g., Oittinen 2000: 100–114, a significantly earlier plea outside English in Adamczyk-Garbowska 1988: 159). Emer O’Sullivan aptly captures the relational character of the challenge that the visualverbal complexes pose for translators: “The more intricate the interplay between words and pictures, the more complex the task of translating. Difficulties arise when pictures and words tell different stories or when the text consistently does not refer to what can be seen in the pictures” (O’Sullivan5 2006: 114). Klaus Kaindl argued in 2005 that in examining the translations of songs, interest in the nonverbal dimension should not be limited to structural constraints posed by the music (2005: 238). With reference to pop songs, he advocates going beyond the traditional verbal orientation toward acknowledging socio-semiotic contexts, i.a. realising that mediation is crucial for popular music (2005: 240). He insists that a translation analysis should account for the fact that a song is not a bi- but a polysemiotic product and indicates that the nonverbal level may also be transformed in translation, even radically so. Accordingly, he engages with aspects like arrangement, vocal presentation, the (public) image of the singer, and the visual component of musical video or performance. The concept of bricolage (2005: 241–242) helps him analyse the meaning-making and the mediated character of chosen pop songs performed outside their original linguistic and cultural environment. The awareness that translation should embrace “more than just words” (Gottlieb 2007: 33) has been growing, not only in the sense of admitting cultural and social contexts but also in terms of intermedial frameworks. It needs clarifying, however, that in contemporary Western translation research, intermediality features primarily under the term multimodality (Kaindl 2013; Pérez-González 2020 – “intermediality” does not appear in the index of the leading translation-studies encyclopedia to which the article belongs), understood as simultaneous use of several semiotic modes. Notably, in audiovisual translation it is nowadays recognised not only as a constraint but also as a potential “blessing in disguise” (Díaz Cintas and Remael 2007: 54–55). Jorge Díaz Cintas and Aline Remael explain in the context of subtitling, which is supposed to be possibly least obtrusive for the viewer, that “subtitlers are actually under an obligation to make the most of the images’ narrative function. They can and must rely on them to abbreviate text, leaving out redundant information and thereby allowing the film to tell its own story” (2007: 54). The interaction between the

Note that despite the title, O’Sullivan’s paper is not about “translating pictures” into anything else, but about interlingually mediating the visual together with the verbal.

5

13

Intermediality and/in Translation

305

dialogue and the information variously encoded in the visual layer or other channels of a screen product can be accounted for by means of the so-called multimodal analysis (Taylor 2004).

Interlingual Translation Faces Intermediality: Examples and Concepts in Action The concepts described above are operative ones meant to facilitate empirical investigations of all the possible interfaces of interlingual translation and intermedial (intersemiotic/multimedia/polycode) texts. Phenomena that typify the empirical problems and illustrate possible lines of enquiry will now be presented. First, an intermedial reference will be considered, and the dilemma which it poses will help demonstrate how the intersemiotic dominant may be chosen for translation and/or for analysis. Further discussion will showcase translational issues connected to media combinations. A very frequent connection, whose pragmatic importance has early been recognised, is that between a verbal text and the accompanying visual. The example below demonstrates a shifting status of the image, implied or present as a paratext. Wisława Szymborska’s poem Ludzie na moście (lit. ‘People on a bridge’) brings an ekphrasis of a wood engraving by Hiroshige Utagawa (1797–1858) titled Downpour at Ohashi Bridge (Ōhashi atake no yūdachi). The intermedial reference (cf. Rajewsky 2005: 52) encompasses a sensitive point in the following segment, where the Polish text describes what ‘can be seen’ (the anaphoric “widać”) in the print: Nic szczególnego na pierwszy rzut oka. Widać wodę. Widać jeden z jej brzegów. Widać czółno mozolnie płynące pod prąd. Widać nad wodą most i widać ludzi na moście. Ludzie wyraźnie przyspieszają kroku, bo właśnie z ciemnej chmury zaczął deszcz ostro zacinać. (Szymborska 1990/1996: 2, emphasis added; for English translation see below)

The crux is that the Polish noun czółno denotes a boat, especially a dugout, while Hiroshige’s engraving arguably presents a raft (Fig. 5). A translator into any language faces the alternative of re-creating either the poet’s lexical choice or the implied visual pre-text. The divergence becomes conspicuous (rather than just implicit) and the dilemma urgent if the visual counterpart is physically present. This happens when the print evoked by the title poem of the collection is reproduced on the cover of a foreign edition, as is the case with People on a Bridge, an Englishlanguage volume of Adam Czerniawski’s renditions (Szymborska 1990/1996) (Fig. 6).

306

M. Kaźmierczak

Fig. 5 Hiroshige Utagawa, Downpour at Ohashi Bridge (image from Europeana, where it is marked as ‘Public domain’; https://www.europeana.eu/pl/item/90402/RP_P_1956_753)

With the relationship between the text and the image thus actualised, the coherence between the actually co-present layers is at stake. Nonetheless, Adam Czerniawski follows the verbal original rather than the visual pre-text in his translation: At first glance, nothing special. You see water. You see a shore. You see a boat sailing laboriously upstream. You see a bridge over the water and people on the bridge. The people are visibly quickening their step, because a downpour has just started lashing sharply from a dark cloud. (Szymborska 1990/1996: 3)

13

Intermediality and/in Translation

307

Fig. 6 Cover of Szymborska’s volume; from https://www.worldcat.org/ title/people-on-a-bridgepoems/oclc/24549879

The same lexical choice is repeated in the next stanza, “The boat sails on motionless” (cf. Czółno płynie bez ruchu, Szymborska (1990/1996: 3), which confirms that Czerniawski chose his intersemiotic dominant in the verbal layer. The discrepancy between the text and the cover illustration may well be blamed by readers on the translator. Perhaps to fend off this charge, the poem is presented with the original on the facing page (1996: 2) – an exception (along with the final poem) in the generally monolingual English edition. The translation’s intermedial relation is thus open to a verification, with the original as a tertium comparationis. Whether to reproduce a relation of contradiction entailed by an original semiotic complex is an individual and case-based decision, and translators’ choices are not to be valued a priori (in this case the prestige of the Nobel Prize winner may have underlain prioritising the verbal). Nonetheless, an intermedially informed translation analysis should give account of such situations and dilemmas. The actual co-presence of nonverbal material increases the translation challenge, as the materialisation of Szymborska’s visual pre-text in the English-language edition of her poems has demonstrated. Let us move from intermedial reference to media combination, of which such co-presence is a defining feature. The subfield of illustrations for/in translations typifies the issue. Dmitry Yermolovich (2016) emphatically shows that illustrators, not necessarily familiar with the original or with the original language, “[w]ithout even knowing it, [. . .] not infrequently find themselves trapped by translators” (2016: 11). An insightful overview of various Russian illustrations to eight different Russian renditions of Lewis Carroll’s Alice

308

M. Kaźmierczak

tales supports his titular commandment: “As you translate, so shall you draw.” He does not limit himself to tracking instances of text and image in a translated edition being at odds (or being congruent despite a potential pitfall), but insists on a foreign illustration’s alignment with the verbal original (2016: passim). In fact, his own double practice as translator and illustrator is so rigorous that it invites the reversal of his creed: as you draw – and you would draw in line with the original – so shall you translate. His plea to translate “in such a way that no faithful illustrator of the [. . .] translation could depict anything contradictory to the original text” (2016: 16, emphasis in the source) expresses an unusual level of concern for the imaging potential of the original verbal layer. Translators, in turn, may find themselves trapped by the visual. What a number of scholars underscore is that various media have unlike spheres of indeterminacy (esp. Eco 2004: 327–331). Unspecified in a written text and left for readers to visualise, a woman said to be beautiful (Eco 2004: 328) or a bogey (Alvstad 2008: 93–94, 97–99) necessarily receives a physical appearance in a film or an illustration. This corresponds with Jakobson’s observation that what matters translation-wise is that languages differ essentially in what they must, not in what they may convey (1959/2000: 116). While Jakobson’s linguistic examples featured the category of gender (1959/2000: 117), the issue also returns repeatedly in observations on intersemiotic contexts of translation – what might have been more easily obviated in a translation between the verbal and verbal only, becomes an increased difficulty when a problematic attribute is expressed (explicitated) in the visual tie-in. Grammatical gender of the noun usually predetermines the sex of animal characters in fairy tales and fables but also in personifying conceptualisations of natural phenomena like Sun and Moon, or, notoriously, personifications such as Death (for intersemiotically informed examples, see Yermolovich 2016: 11–13; Eco 2004: 323–324; Bałuk-Ulewiczowa 2004: 103–104). Let us move to another type of media combination for the next case. The correlation between co-present codes is also an issue in music-linked translation. This extends beyond the obviously constrained type of rendition where the target text is intended to be sung to the original melody (in English usually called “singable,” also “vocal”). For instance, there can be a perceived dissociation between the musical layer and the written verbal text as well. This is illustrated by a case of a Polish rendition of the texts of Richard Strauss’ Vier letzte Lieder (Four Last Songs, settings of Hermann Hesse and Joseph von Eichendorff) provided for the performance in 2015 in the Lodz Philharmonic, Poland, in the form of a printed leaflet for the audience (Strauss [2015]). The original German verbal text of the last song, “Im Abendrot” (the only one by Eichendorff), contains an image of two larks, zwei Lerchen, flying up into the sky: Zwei Lerchen nur noch steigen nachträumend in den Duft. (Strauss 2015)

Imitation of their voices is also a program motif in the piccolos’ part in the score (cf. recordings, e.g., Strauss 1984: no. 4, 30 5000 and Fig. 7). Musically, this is a very important motif, since it returns at the end of the work as a counterpoint

13

Intermediality and/in Translation

309

Fig. 7 Strauss, Eichendorff, “Im Abendrot” score excerpt: piccolos (Strauss 1950: 49; copyright 1950 by Boosey & Co. Ltd.; from IMSLP score library, where marked as ‘Public Domain’)

(see Krause 1964: 462) to the closing line, in which the weary wanderers ask whether death is approaching: “Ist die etwa der Todt?” (Strauss 2015). In the Polish translation, ‘Only two little birds yet/soar in the fragrant air’ (gloss transl. mine; Pol.: “Jeszcze tylko dwie ptaszyny/wznoszą się w wonnym powietrzu,” Strauss 2015). The unfortunate generalisation is rhyme-driven: ptaszyny, “little birds,” appear for the sake of rhyming with doliny (“valleys”). Strangely enough, the

310

M. Kaźmierczak

translator transferred the lark motif from the last song to the first one: in “Frühling,” the original mention of Vogelsang (“birdsong”) is replaced by the phrase ‘the voice of the lark’ (Pol.: “głos skowronka”). In a translation of a lyric cycle authored by one poet and free from any intermedial entanglement, such a move could have counted as a successful compensation. In this case, however, the lark “has flown” from Eichendorff’s poem into Hesse’s and from a place in the score of a programmatic character to one where the composer does not employ similar illustrative measures in music. Because of that the translation failed its audience in terms of informing them about the alignment of verbal and musical meanings, even though being sung was not its intended use.

Interlingual Translation Behind Intermedial Transactions: Unacknowledged Presences Interlingual translation is frequently a middle step between a source text and its transformation into a different medium. However, the awareness of interlingual mediation tends to be suppressed: in very many instances it is obscured from view or even impossible to reconstruct from the state of total erasure what interlingual translation was taken as basis by makers of various transmutations. For example, films based on literary material acknowledge the author(s) of that material (although according to Cattrysse [1992: 38], they often enough present themselves as autonomous, unless it is the source that actually lends them prestige), but when the source was originally cast in a different language than the film, the translator(s) are not credited. The scale of suppression becomes evident the moment one realises how many, or how significant, film adaptations are produced cross-culturally. American films based on non-English-language literature are perhaps not that numerous, but still there are War and Peace (dir. King Vidor, 1956), Madame Bovary (dir. Sophie Barthes, 2014), and Verne and Dumas adaptations to cite.6 Examples that give weight to the claim on account of importance rather than number include Michelangelo Antonioni’s Blow-Up (1966) from Julio Cortázar’s story “Las babas del diablo” (1959), Luchino Visconti’s Death in Venice (Morte a Venezia, 1971) from Thomas Mann’s German novella, or Kubrick’s Eyes Wide Shut (1999) from Arthur Schnitzler’s Traumnovelle, 1926 (Eng.: Dream Story). Polish films based on non-Polish writing include renderings of E.A. Poe, Mark Twain, W.M. Thackeray, Prosper Mérimée, Anatole France, Alexandre Dumas fils, Aleksandr Pushkin, Mikhail Lermontov, Ivan Turgenev, and Jan Potocki (who was Polish but wrote in French). The Czech director Karel Zeman based most of his spectacular animation +

With Hollywood, a different, if adjacent, category is more abundant: remakes of foreign films. Scent of a Woman can be cited, the 1992 remake of Profumo di donna, 1974, both of them based on Il buio e il miele (‘Darkness and honey’) by Giovanni Arpino. In such remakes the enigma of interlingual mediation shifts to the stage of the intrasemiotic film-to-film transformation. 6

13

Intermediality and/in Translation

311

live-action footage productions on Jules Verne’s novels (also drawing on Verne’s French illustrators). His Baron Prášil, 1961, in turn, takes up Gottfried August Bürger’s Baron Münchhausen (Feldzüge und Abenteuer des Freiherrn von Münchhausen, 1786/1789, itself a re-writing of an English source), with the intercultural complexity heightened by the visual layer intentionally evoking Gustave Doré’s engravings from the tale’s French edition. Russian cinema extensively draws on foreign texts, but whose translations underlie Russian film adaptations of J.K. Jerome’s Three Men in a Boat (dir. Naum Birman, 1979) or Robert Thomas’ Cherchez la femme (dir. Alla Surikova, 1982), among many others? Notably, a work may be transmuted more than once in the same foreign(-language) context, for instance, Choderlos de Laclos’ Les Liaisons dangereuses has two English-language cinematic versions: Stephen Frears’ Dangerous Liaisons (1988) and Miloš Forman’s Valmont (1989). A symptomatic case of passing the intralingual aspect over in silence can be found in Jean-Luc Godard’s 1963 film Le Mépris, based on Il disprezzo, a novel by Alberto Moravia,7 whose translation into French became available within a year from the original publication (transl. Claude Poncet, 1955). Whether Godard, responsible for script as well as direction, worked from the Italian original is not to be known (at least not from the spoken credits which open the motion picture, whatever information metatexts may divulge). The plot turns this particular example into its own mise en abyme with regard to translation issues: an American producer employs first an Austrian director and then a French playwright to make a film out of Homer’s Odyssey. Naturally, what matters most are the increasingly tense relations between the characters, and Godard’s cinematic homage to cinema, not the language and version in which Odyssey is read for the sake of the script-within-the-script. Nonetheless, the silence on the topic becomes “audible” in the context of the credit elision with regard to the presumed mediation from Italian. By contrast, a theater production would normally credit the translator, and in the Polish multimedia context, this also includes theater on screen – the 1975 television stage adaptation of Moravia (dir. Andrzej Łapicki) unequivocally credits Zofia Ernstowa, the author of the 1961 Polish rendition of his novel. Musical and musical-verbal examples given in the first section of the chapter pose the same puzzle. One is left to wonder whether Britten could read Ovid in the original or rather got his inspiration for Six Metamorphoses from a translation, and then from which of the numerous ones. Gounod’s librettists for Faust, Jules Barbier and Michel Carré, may have worked with the German text of Goethe’s drama or with a French translation. While the general audience may not be keenly interested in knowing what (if anything) La Traviata’s text set by Verdi owes to a translator of Dumas’ La Dame aux camélias, a scholar will be stricken by how difficult it proves to obtain facts or even clues in this respect. One is often restricted to speculation,

The film’s title is Contempt in English distribution, while the novel’s translation (by Angus Davidson) has been published as A Ghost at Noon. The diversification of the characters in terms of nationality referenced further in this paragraph characterises the film, not the book. 7

312

M. Kaźmierczak

considering that even establishing whether a composer could speak a given language needs a fairly thorough biographical research. Lack of clues about interlingual mediation similarly affects other areas of research into intermedia. Agnieszka Majcher indirectly admits that she was constrained to make assumptions as to the translated versions underlying the foreign film adaptations of the Polish classic of science fiction Stanisław Lem. A translator’s signature may be preserved in neologisms traceable to a particular print edition (Majcher 2015: 149), a clue specific to the genre of the source text. A characteristic treatment of proper names may also provide such evidence. In the case of Zeman’s 1961 animation, the eponymous baron appearing as Prášil, not Münchhausen, testifies to a connection with translation, or at least with the national Czech reception of the narrative. What we could term translational-proper aspects of intermediality is thus as a rule implicitated and overlooked. Despite what may be said about unduly privileging the verbal in studies on intermedial transpositions (e.g., Stam et al. 2005: 63), researchers can also be charged with increasing the invisibility of translation and translators. For instance, when comparing verbalisations of the same musical piece written by a Polish writer and by a Russian musicologist, Brazgovskaya glosses over the fact that the two transmutations are realised in two different languages, proffering probably her own translation as if it were transparent (Бразговская 2014: 40–42). Johan Stenström’s (2007) exploration of Giacomo Puccini’s La Fanciulla del West is a very rare example of a study on adaptation where the language mediation is mentioned at all: he recounts that the composer saw a Broadway staging of David Belasco’s The Girl of the Golden West, but did not start co-drafting the libretto of the opera until after having read the play’s Italian translation (2007: 129). The realisation that interlingual mediation looms in or behind many intersemiotic transformations logically leads (or should lead) to querying whether, to what extent, and in what ways the intermediary step influences the final other-media or intermedial product. The issue may be formulated in terms of quality and so it is done by Majcher (2015) who, however, does not resolve the question convincingly. In her paper the translations of Lem’s two novels corresponding to a shallow and to a critically appraised movie are assumed rather than shown to be bad and good, respectively, with textual evidence limited to single isolated phrases. Stenström (2007: 131), like most researchers, concentrates on the transformation on the narrative level, but in analysing Puccini’s take on the American West, he exceptionally also indicates that despite the effort to render the coarseness of the characters’ speech, the original linguistic varieties of the miners, Mexicans and Indians could not be captured in the libretto (2007: 131). However, whether this happens already in the Italian translation of the play remains unsaid. Although Stenström provides (2007: 133–134) glimpses of Belasco’s playtext, he does not compare it with the Italian translation nor even with the libretto itself (nor does he name the translator). Yingjin Zhang (2008) contributes an interesting insight when analysing a 1931 modernised and re-localised Chinese version of Shakespeare’s The Two Gentlemen of Verona. He briefly engages (2008: 94–96) with intertitles to this silent film, which are bilingual and remain in a remarkable two-directional interlock. Some passages

13

Intermediality and/in Translation

313

derive from the comedy’s original text and required rendering into Chinese, while some of the lines, especially covering the “Sinicised” layer of the plot, constitute the adaptors’ input, with the source being in Chinese and the translation in English. Zhang concentrates on instances of mutual untranslatability in this traffic, as contrasted with verbal-visual compatibility. Apparently what draws this researcher to translational matters is the fact that the fluid character of the intertitles contributes to ways in which the film overall challenges the notions of authority and authorship in adapting practices (2008: passim). Some truly in-depth multilayered studies of the interrelation between intermedial and interlingual transformations of the same work remain unpublished (Zuzanna Kołodziejczyk, Interlingual and intersemiotic translation: Venedikt Yerofeev’s poem “Moscow Stations” («М осква–Петушки») on stage in Russian, English and Polish, MA thesis, University of Warsaw 2018). A study exemplary in accounting for the entanglements of intermediality and translation proper was carried out by Jessica Yeung (2008). Her interdisciplinary examination embraces interlingual and intersemiotic translations “surrounding” Gustav Mahler’s Das Lied von der Erde – the composition both relies on earlier interlingual mediations and has been subject to further translational actions. For the libretto, Mahler employed texts of poems that were free versions by Hans Bethge of Chinese classic poetry, based on previous German and French translations, some of which were themselves indirect. Yeung pursues the Chinese originals, the mediating intra- and interlingual sources used by Bethge, and the composer’s alterations to Bethge’s texts. Then she gives an account of the meaning-making in music (Mahler’s sequencing and tailoring the poems to his purposes and the reflexive relation between the verbal and the musical layers bear out the scholar’s decision to treat this process as an instance of intersemiotic translation, not just as complementation), with both semiotic components being shown as harnessed to express the composer’s life philosophy (2008: 285). Yeung argues that the very structure of the composition pivots on the idea of translation. Metonymically representing “another world,” translation authorises the composer to experimentation, to going beyond the Western canons of music (2008: 289). Founded on consciously “inauthentic” Chineseness (2008: 288), Mahler’s masterpiece was at the turn of the twenty-first century transformed into a dance production in Hong Kong, with the German texts of the poems translated into Chinese to be read between the movements. Despite the popular expectation of such an enterprise to be a “homecoming,” the makers of the performance – the choreographer, the painter responsible for design, and the poet-translator – all chose to underscore the artificial, constructed, or “external” quality of Chineseness inscribed in the European musical-verbal work (2008: 289–292). Thus, the translational character emphatically permeated the resulting theatrical piece (2008: 292). Yeung concludes that such an “intertextual network” involving intersemiotic and interlingual transactions prompts one to perceiving translations “as original rather than derivative” (2008: 293). Mahler’s dealing with translated texts to be set is shown by Yeung as exercising artistic freedom and authority over the verbal as well. Nevertheless, it should not be overlooked that translation significantly shapes the complementations or

314

M. Kaźmierczak

transmutations which it underlies. This impact may be illustrated by musicalisations created for various language versions of a shared original poem. Take the Scottish ballad “Edward, Edward”: Franz Schubert (D 923) and Carl Loewe (op. 1, no. 1) musicalised its German translation by Johann Gottfried Herder, while Pyotr Tchaikovsky (op. 46, no. 2) set the Russian rendition by Aleksei Tolstoy. Although in most respects the works indubitably express the composers’ individual artistic choices, certain similarities between the two German compositions and their difference from the Russian one are accounted for by their respective verbal layers. Basic rhythmical properties of the songs depend on the meters chosen by the two translators, and masculine or feminine line endings predefine the kinds of musical cadences. Without going into a detailed translation analysis, let us further note that Herder preserved the interjection “O!” ending every fourth line in the anonymous original, and the German composers setting his text included these elements. Aleksei Tolstoy (Толстой 2016 vol. 1: 704–707)8 omitted the exclamations and they do not feature in Tchaikovsky’s duet. This presence or absence of interjections appearing at regular interval will be a feature perceptible to any recipients who have the opportunity to compare performances or recordings (cf., e.g., Loewe 1989; Schubert 2015 vs Rubinstein et al. 2001), regardless of their linguistic competence. Such an experience, dispelling the illusion of sameness of the underlying text, may, in turn, generate a greater awareness of the significance of translation for intermedial transactions and products.

Conclusion The chapter has presented intermediality and/in translation from varied angles. The interrelation of the two encompassed at least the following aspects: – Intermediality not infrequently results from processes of translational nature. – Intersemiotic translation and interlingual translation generically share certain properties and tendencies. – Verbal texts carrying intermedial references as well as polysemiotic works (whether resulting from medial transposition or media combination or primary ones) often – if they contain verbal components – undergo interlingual translation; this gives rise to intersemiotic aspects of translation proper. – Interlingual mediation may be a part of (or be prior to) a complex process of transformation into a different medium. Intermediality as a product of translation has been discussed in section “Intersemiotic Translation: Naming the Phenomenon and Delimiting Its Scope.” Roman Jakobson’s founding concept of intersemiotic translation, which first 8 Working from a German version (though not from Herder’s, as some sources suggest) only later collated with the original (cf. Толстой 2016: vol. 2: 707–710).

13

Intermediality and/in Translation

315

highlighted this fact, has served as a starting point for mapping a broad range of contemporary cultural phenomena generated by transformations into a different medium. As explained, however, what can legitimately qualify as intersemiotic translation needs to be distinguished on the one hand from media combination (which is not a transposition) and, on the other, from certain types of interlingual translation (which are not intermedial). Jakobson’s concept has since 1959 been subject to reinterpretations and redefinitions, in response to dynamic developments in cultural products and translation practices, including entirely new types of translation. Nowadays, intermedial transactions are also described by a number of terms, some of which help address the specificity of contemporary intermedial practices, e.g., inasmuch as a transposition often entails merging various texts into one. Sections “Beyond Intersemiotic Translation: Variations of the Term and Holistic Concepts” and “What’s in a Name? Adaptations, Re-creations, Transmediations, and More” present a selection of theoretical propositions as well as substantiations and reservations voiced by scholars with reference to the applicability or scope of particular terms, including, i.a., transposition, transmutation, adaptation, and transmediation. Terminological preferences may correlate with certain differences in analytical interests of the scholars who use them, but ultimately do not seem to determine the scope of research in radically different ways. Similarities between intersemiotic translation and translation proper (section “Intersemiotic Translation and Translation Proper: Similarities and Research Insights”) can be indicated with regard to the process, as suggested by Steiner, as well as to how they function as products. They share seriality, i.e., reoccurrence of successive renditions based on the same source text; moreover, interrelations within a sequence of intermedial transpositions resemble interrelations among retranslations. The categories of (un)translatability and adequacy have been applied to both types of transformations (the resistance against the fidelity paradigm does not subvert this parallel), and there have been attempts to extrapolate certain methodologies from translation studies, notably polysystem theory, to the field of intermedial research. The parallels notwithstanding, intersemiotic translation and an interlingual translation of a multicode text are two distinct things. Indeed, myriads of multimodal products undergo interlingual translation and translations function in multimodal contexts (section “Translation Studies Addresses Intermediality in Translated Texts”). As a result, intermediality becomes part of a translator’s task, and frequently a challenge, as illustrated in section “Interlingual Translation Faces Intermediality: Examples and Concepts in Action,” with examples of verbal-visual and verbalmusical works. Besides, re-creating an intermedial reference within a verbal text in the absence of the evoked visual element (or aural one, for that matter) differs from translating a bisemiotic or polysemiotic complex, with the nonverbal component(s) co-present. Translation studies has been addressing these issues by proposing operative concepts which can help choosing translation priorities and serve analysing translations (intersemiotic dominant, multimodal analysis for translation purposes).

316

M. Kaźmierczak

As interrelations between intermediality and translation form a cluster of phenomena, some of them deserve careful distinguishing from others and precise terminological distinctions. It bears repeating that it is crucial to distinguish intersemiotic translation from situations when mediating between languages requires taking into account semiotic codes other than the verbal. The latter is the case of interlingual transfer accommodating intersemiotic aspects. Finally (section “Interlingual Translation Behind Intermedial Transactions: Unacknowledged Presences”), translation proper is often involved at a certain stage in the emergence of complementations or transmutations but usually remains unacknowledged. Still, its silent presence is a shaping one, which makes relevant the task of interrogating to what extent and in what ways the interlingual mediation influences the final other-media or intermedial product. These “translational aspects of intermediality” have received little attention so far. Issues related to intermediality have a practical meaning for many translation types and genres. They are encountered not only by translators of artistic products, on which the chapter has focused, but also by localisers and copywriters who carry out transcreation across cultural context, and even by technical translators. Awareness of this has been growing in translation studies. In intermedia studies, in turn, interlingual mediation awaits a fuller acknowledgment of its supportive role in medial transformations. Research on any interlocks of intermediality and translation needs to be multifaceted, to comprehend all semiotic layers of the works under scrutiny and to account for interventions by the various agents who co-create a multimodal product. While so far translation scholars have shown more interest in intermediality than intermedia studies (in its various incarnations) has shown in translational matters, there are attempts (note Raw 2012 or Krebs 2014 referenced in this chapter) and opportunities to bring the representatives of the disciplines together and thus to create synergies from mutual exchange of insights.

References Adamczyk-Garbowska, Monika. 1988. Polskie tłumaczenia angielskiej literatury dziecięcej: Problemy krytyki przekładu [Polish translations of English children’s literature: Issues in translation criticism]. Wrocław: Ossolineum. Alter, Nora M. 2007. Translating the Essay into Film and Installation. Journal of Visual Culture 6 (1): 44–57. Alvstad, Cecilia. 2008. Illustrations and Ambiguity in Eighteen Illustrated Translations of Hans Christian Andersen’s “The Steadfast Tin Soldier”. Meta 53 (1): 90–103. https://doi.org/10.7202/ 017976a. Anderson, Jack. 2002. Opera Into Dance? Not as Simple as It Sounds, The New York Times, March 3, 2002, section 2, page 10. https://www.nytimes.com/2002/03/03/arts/dance-opera-into-dancenot-as-simple-as-it-sounds.html. Accessed 1 Mar 2022. Balcerzan, Edward. 2011. Tłumaczenie jako “wojna światów”: W kręgu translatologii i komparatystyki. [Translation as a 'war of worlds': In the domains of translatology and comparative studies] 3rd ed. Poznań: Wydawnictwo Naukowe UAM.

13

Intermediality and/in Translation

317

Bałuk-Ulewiczowa, Teresa. 2004. Values in the Visual Arts and Untranslatability: Powerless Pastors-Faithful and Wolves in Sheep’s Clothing. In Przekładając nieprzekładalne [Translating the untranslatable], vol. 2, ed. Olga Kubińska and Wojciech Kubiński, 101–108. Gdańsk: Wydawnictwo Uniwersytetu Gdańskiego [Gdańsk University Press]. Bassnett, Susan, and André Lefevere. 1998. Constructing Cultures: Essays on Literary Translation. Clevedon: Multilingual Matters. Bednarczyk, Anna. 2010. Intersemiotic Dominant of Translation. In Meaning and Translation, ed. Barbara Lewandowska-Tomaszczyk and Marcel Thelen, 403–414. (“Lodz Studies in Language” vol. 19), Frankfurt am Main, Berlin, Bern, Bruxelles, New York, Oxford, Wien: Peter Lang. Benveniste, Émile. 1969. Sémiologie de la langue (1). Semiotica 1 (1): 1–12. Bolter, Jay David, and Richard Grusin. 1999/2000. Remediation. Understanding New Media. Cambridge, MA/London: MIT Press. Cattrysse, Patrick. 1992. Film (Adaptation) as Translation: Some Methodological Proposals. Target 4 (1): 53–70. Clüver, Claus. 1989. On Intersemiotic Transposition. Poetics Today 1 (10): 55–90. ———. 2007. Intermediality and Interarts Studies. In Changing Borders: Contemporary Positions in Intermediality, ed. Jens Arvidson, Mikael Askander, Jørgen Bruhn, and Heidrun Führer, 19–37. Lund: Intermedia Studies Press. Deane-Cox, Sharon. 2014. Retranslation: Translation, Literature and Reinterpretation. London: Bloomsbury. Díaz Cintas, Jorge. 2005. Audiovisual Translation Today: A Question of Accessibility for All. Translating Today 4: 3–5. Díaz Cintas, Jorge, and Aline Remael. 2007. Audiovisual Translation: Subtitling. London/New York: Routledge. Dusi, Nicola. 2015. Intersemiotic Translation: Theories, Problems, Analysis. Semiotica 206: 181–205. https://doi.org/10.1515/sem-2015-0018. Eco, Umberto. 2004. Dire quasi la stessa cosa: Esperienze di traduzione. 5th ed. Milano: Bompiani. Elleström, Lars. 2017. Transfer of Media Characteristics among Dissimilar Media. Palabra Clave 20 (3): 663–685. https://doi.org/10.5294/pacla.2017.20.3.4. Gottlieb, Henrik. 2007. Multidimensional Translation: Semantics Turned Semiotics. In Challenges of Multidimensional Translation. Proceedings of the Marie Curie Euroconferences MuTra 2005, ed. Sandra Nauert, Heidrun Gerzymisch-Arbogast, 33–61. Saarbrücken: Saarland University. Hajduk-Gawron, Wioletta, and Agnieszka Madeja, eds. 2013. Adaptacje I: Język – literatura – sztuka [Adaptations I: Language – literature – art]. Katowice: Uniwersytet Śląski [University of Silesia]. Helman, Alicja, and Wacław Osadnik. 1996. Film and Literature: Historical Models of Film Adaptation and a Proposal for a (Poly)System Approach. Canadian Review of Comparative Literature/Revue Canadienne de Littérature Comparée, Special Issue: Literature and Film, ed. Roumiana Deltcheva, Wacław M. Osadnik and Eduard Vlasov, 23(3): 645–658. Hopfinger, Maryla. 1974. Adaptacje filmowe utworów literackich: Problemy teorii i interpretacji [Film adaptations of literary works: Issues in theory and interpretation]. Wrocław: Ossolineum. Jakobson, Roman. 1959/2000. On Linguistic Aspects of Translation. In The Translation Studies Reader, ed. Lawrence Venuti, 113–118. London: Routledge. Kaindl, Klaus. 2004. Multimodality in the translation of humour in comics. In Perspectives on Multimodality, ed. Eija Ventola, Cassily Charles, Martin Kaltenbacher, 173–192. Amsterdam: John Benjamins. Kaindl, Klaus. 2005. The Plurisemiotics of Pop Song Translation: Words, Music, Voice and Image. In Song and Significance: Virtues and Vices of Vocal Translation, ed. Dinda L. Gorlée, 235–264. Amsterdam: Rodopi. ———. 2013. Multimodality and Translation. In The Routledge Handbook of Translation Studies, ed. Carmen Millán and Francesca Bartrina, 257–269. London/New York: Routledge.

318

M. Kaźmierczak

Kaźmierczak, Marta. 2018a. From Intersemiotic Translation to Intersemiotic Aspects of Translation. Przekładaniec. A Journal of Translation Studies 34–35 (English Issue: Word and Image in Translation): 7–35. https://doi.org/10.4467/16891864ePC.18.009.9831. ———. 2018b. From Poetry into Film – and Back (through Translation). In Poésie-TraductionCinéma / Poetry-Translation-Film, ed. Vincent Dussol and Adriana Şerban, 145–168. Limoges: Lambert-Lucas. Kmieciak, Marta. 2015. O przekładzie intersemiotycznym – refleksja nad tekstem, muzyką i scenicznością Salome (Oscar Wilde – Richard Strauss – Jürgen Flimm) [On intersemiotic translation: reflections on the text, music, and stage properties of Salome]. In Libretto i przekład [Libretto and translation], ed. Elżbieta Nowicka, and Alina Borkowska-Rychlewska, 213–232. Poznań: Poznańskie Towarzystwo Przyjaciół Nauk [The Poznań Society for the Advancement of Arts and Sciences]. Krause, Ernst. 1964. Richard Strauss: The Man and His Work [orig.: Richard Strauss: Gestalt und Werk], trans. John Coombs. London: Collet’s. Krebs, Katja, ed. 2014. Translation and Adaptation in Theatre and Film. London/New York: Routledge. Legeżyńska, Anna. 1986/1999. Tłumacz i jego kompetencje autorskie [Translator and his authorial competences], 2nd ed. Warszawa: Wydawnictwo Naukowe PWN. Loewe, Carl. 1989. Balladen. Perf. Thomas Quasthoff, bass-baritone, and Norman Shelter, piano. Parlophone Records. CD. Lorsch, Susan E. 1988. Pinter Fails Fowles: Narration in The French Lieutenant’s Woman. Literature Film Quarterly 16 (3): 144–154. Majcher, Agnieszka. 2015. Does the quality of interlingual translation influence the quality of the intersemiotic translation? On the English language film adaptations of S. Lem’s The Futurological Congress and Solaris in the light of their translations into English. Journal of Language and Cultural Education 3 (3): 144–151. McFarlane, Brian. 1996. Novel to Film: An Introduction to the Theory of Adaptation. Oxford: Clarendon Press. Newell, Kate. 2017. Expanding Adaptation Networks: From Illustration to Novelization. London: Springer. O’Sullivan, Emer. 2006. Translating Pictures. In The Translation of Children’s Literature: A Reader, ed. Gillian Lathey, 113–121. Clevedon/Buffalo/Toronto: Multilingual Matters. Oittinen, Riita. 2000. Translating for Children. New York/London: Garland Publishing. Pérez-González, Luis. 2020. Multimodality. In Routledge Encyclopedia of Translation Studies. Third Edition, ed. Mona Baker and Gabriela Saldanha, 346–351. London/New York: Routledge. Plett, H.E. 1991. Intertextualities. In Intertextuality, ed. H.E. Plett, 3–29. Berlin/New York: de Gruyter. Rajewsky, Irina. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. Intermédialités 6: 43–64. Raw, Laurence, ed. 2012. Translation, Adaptation and Transformation. London: Bloomsbury. Remael, Aline. 2001. Some Thoughts on the Study of Multimodal and Multimedia Translation. In (Multi)Media Translation, ed. Yves Gambier and Henrik Gottlieb, 13–22. Amsterdam: John Benjamins. Rubinstein, Anton, Piotr Tchaikovsky, and Nicolai Rimsky-Korsakov. 2001. Russian Duets ¼ Duety Rosyjskie. Perf. Urszula Kryger, Jadwiga Rappé, Jerzy Knetig, Tomasz Herbut. DUX. CD. Schubert, Franz. 2015. “Das Ewig-Weibliche”: Franz Schubert’s Women in Music. Perf. Martha Guth, soprano, Penelope Crawford, fortepiano. Musica Omnia. CD. Stam, Robert, Robert Burgoyne, and Sandy Flitterman-Lewis. 2005. New Vocabularies in Film Semiotics. London/New York: Routledge. Stankovych, Yevhen. 2017. When the Fern Blooms (orig. Коли цвіте папороть, 1978), operaballet, text Oleksandr Stelmashenko, dir. Vasyl Vovkun, Lviv National Opera, recorded 15 December 2017, film director Serhii Krutsenko, prod. The Little Films. [available on

13

Intermediality and/in Translation

319

Opera Vision 20.08.2021 – 20.02.2022, then extended; flashback available at: https:// operavision.eu/performance/when-fern-blooms, accessed 25 May 2023]. Steiner, George. 1975. After Babel: Aspects of Language and Translation. Oxford: Oxford UP. Stenström, Johan. 2007. Bel Canto at the Californian Frontier: The Adaptation of Puccini’s Opera La Fanciulla del West from Belasco’s Play The Girl of the Golden West. In Changing Borders: Contemporary Positions in Intermediality, ed. Jens Arvidson, Mikael Askander, Jørgen Bruhn, and Heidrun Führer, 129–136. Lund: Intermedia Studies Press. Strauss, Richard. 1950. Vier letzte Lieder. London: Boosey & Hawkes. Score. ———. 1984. Vier letzte Lieder. In Richard Strauss, Lieder. Perf. Jessye Norman, soprano, Gerhard Bosse, violin, Gewandhausorchester Leipzig; conductor Kurt Masur. Berlin: Eterna. LP record. ———. 2015. Vier letzte Lieder op. posth. (1948) / Cztery ostatnie pieśni na sopran i orkiestrę op. posth., texts by Hermann Hesse and Joseph von Eichendorff, trans. M. Bartnikowska. Leaflet (bilingual). Szczerbowski, Tadeusz. 2005. Reklama jako przekład polisemiotyczny [A commercial as polysemiotic translation]. In: Między oryginałem a przekładem [Between original and translation] X: Między tekstem a obrazem: Przekład a telewizja, reklama, teatr, film, komiks, Internet, ed. Urszula Kropiwiec, Maria Filipowicz-Rudek, and Jadwiga Konieczna-Twardzikowa, 123–131. Kraków: Księgarnia Akademicka. Szymańska, Izabela. 2020. On the Analogies between Translation and Film Adaptations of Literary Classics. In Moving Between Modes, ed. Mikołaj Deckert, Monika Kocot, and Aleksandra Majdzińska-Koczorowicz, 251–263. Łódź: Wydawnictwo Uniwersytetu Łódzkiego [Lodz University Press]. https://doi.org/10.18778/8220-191-8.17. Szymborska, Wisława. 1990/1996. People on a Bridge: Poems, introd. and trans. Adam Czerniawski. London/Boston: Forest Books. Taylor, Christopher. 2004. Multimodal Text Analysis and Subtitling. In Perspectives on Multimodality, ed. Martin Kaltenbacher, Cassily Charles, and Eija Ventola, 153–172. Amsterdam: John Benjamins. Testa, Carlo. 2002. Masters of Two Arts: Re-creation of European Literatures in Italian Cinema. Toronto: University of Toronto Press. Toury, Gideon. 1986. Translation: A Cultural-Semiotic Perspective. In Encyclopedic Dictionary of Semiotics, ed. Thomas A. Sebeok, vol. 2, 1111–1124. Berlin: Mouton de Gruyter. Tsiakalou, Ourania. 2018. Dancing Through the Waves of Feminism: Martha Graham and Marie Chouinard as Intersemiotic Translators. In Key Cultural Texts in Translation, ed. Kirsten Malmkjaer, Adriana Şerban, and Fransiska Louwagie, 25–34. Amsterdam: John Benjamins. Tytler, Alexander Fraser. 1790/1992. From the Essay on the Principles of Translation. In Translation – History – Culture: A Sourcebook, ed. André Lefevere, 128–135. London/New York: Routledge. Wolf, Werner. 2002. Towards a Functional Analysis of Intermediality: The Case of TwentiethCentury Musicalized Fiction. In Cultural Functions of Intermedial Exploration, ed. Erik Hedling and Ulla-Britta Lagerroth, 15–34. Amsterdam/New York: Rodopi. Yeung, Jessica. 2008. The Song of the Earth: An Analysis of Two Interlingual and Intersemiotic Translations. The Translator 14 (2), spec. issue Translation and Music: 273–294. Yermolovich, Dmitry. 2016. As You Translate so Shall You Draw. Knight Letter 97: 11–22. Zhang, Yingjin. 2008. From Shakespeare’s Drama to Early Chinese Cinema: Authority and Authorship in Literary Translation and Film Adaptation. Yearbook of Comparative and General Literature 54: 83–102. Бразговская, Е.Е. 2014. Вербализация музыки как межсемиотический перевод [Verbalisation of music as intersemiotic translation], Критика и семиотика [Criticism and semiotics] 1: 30–47. Толстой, Алексей К. 2016. Полное собрание стихотворений: в 2-х т. [Collected poems, in two vols.], ed. В.А. Котельников, А.П. Дмитриев. С-Петербург: Наука / РАН. Тороп, Пеэтер. 1995. Т отальный перевод [Total translation]. Тарту: Издательство Тартусского ун-та / Tartu Ülikooli Kirjastus.

Visual Citation in Intermedial Relations

14

Ana Luiza Ramazzina-Ghirardi

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Compagnon’s Perspective on Citations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Rise of Visual Citation as a Theoretical Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Visual Citation in Various Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Visual Citation and Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Visual Citation in Practice: Mundano’s Appropriation of Portinari’s Coffee Farm Worker . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

322 323 324 326 332 334 337 338

Abstract

In “La seconde main ou le travail de la citation” (1979, 2016), Antoine Compagnon examines in depth the phenomenon of verbal citations and offers a theoretical framework to understand this concept, which has become referential for linguists. According to the author, verbal citations have the status of “a validity criterion, a means for controlling the enunciation, and a regulatory device” (Compagnon, La seconde main, ou, Le travail de la citation. Points, Paris, 2016: 12). The omnipresence of images and their centrality to the different media created from the late twentieth century onwards have prompted a renewal and enlargement of this concept by means of the notion of visual citation. Clüver (Aletria: Revista De Estudos De Literatura 14:10–41, 2006), Groensteen (Bédé, ciné, pub et art: d’un média à l’autre. Infolio, Gollion, 2007), and Elleström (Adaptation studies: new challenges, new directions. Bloomsbury, London, 2013), among others, adopt this broader notion of citation in their studies on intermedial relations. (Chambat-Houillon, Marie-France, De l’audiovisuel vers le télévisuel: deux modèles de citation pour les émissions de télévision. Revue Ci-Dit, Nice, 2009) further enlarges the concept in her studies on TV programs. A. L. Ramazzina-Ghirardi (*) UNIFESP, São Paulo, Brazil e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_17

321

322

A. L. Ramazzina-Ghirardi

Visual citations are recurrent in plurisemiotic environments such as television, photography, film, drama, comics, etc. They function in connection with intermedial process such as intermedial references, intramedial references, ekphrasis, remediation, transmedial remediation, transmediation, and others. This chapter offers a critical overview of the different uses of the term “visual citation” in intermedial studies and examines its working when applied to different media and intermedial processes. Keywords

Visual citation · Intermediality · Plurisemiotic environments

Introduction In “La seconde main ou le travail de la citation” (1979), Antoine Compagnon suggests that the use of textual markers to indicate a citation first appears in the sixteenth century and is linked to the editorial dynamics that emerged with the creation of the printing press. Compagnon posits that the technological revolution spurred by Gutenberg’s invention is at the root of the modern understanding of what a citation is, of how it works, and how it becomes “a specific category within the text” (Compagnon 2016: 305). In the twenty-first century, a new technological revolution and the plethora of virtual media it has produced have been reshaping all aspects of communication – the production, reception, circulation of texts – in a process which gives prime place to images. In this new context, images are unceasingly re-appropriated and reproduced in different media, establishing a new dynamics of citation: the visual citation. Because of its insights on the deep connections between technology, media, and citation, Compagnon’s work constitutes an apt starting point for the study of visual citation as a novel form of replicating and resignifying images in different media. The concept of visual citation emerges, thus, as a theoretical tool to help understand, in the context of the new media, the process of quoting another work, a textual strategy that has long been a topic in literary studies. This process of exporting images produced in one media to another helps understand the broader, complex dialogue that takes place between different media. Visual citation has, therefore, become a prime object of analysis in intermedial studies and an array of scholars of intermediality have been paying increasing attention to it, exploring its similarities and dissimilarities to its verbal counterpart. This effort to map the specific contours and the basic traits of visual citation has resulted in the rise of a new conceptual repertoire centering on the diverse aspects of inserting, in a new media product, images created to function elsewhere. Concepts relating to the process of visual citation include “intermedial reference” and “intramedial reference” (Rajewsky), “ekphrasis” (Webb; Clüver; Elleström; Vieira), “remediation” (Bolter & Grusin; Baetens; Rajewsky; Elleström), “transmediation,” and “representation” (Elleström), among others.

14

Visual Citation in Intermedial Relations

323

In order to present the concept of visual citation and the dynamics it implies, this chapter examines, first, the interweaving of cited and citing text in verbal products. It presents Antoine Compagnon’s key contention that a new meaning emerges in the interplay between cited-citing texts. The presentation of Compagnon’s contribution is followed by an analysis of Goodman’s discussion of pictorial citation and its specificities. The third section presents an overview of the main theoretical proposals dealing with the question of visual citation understood as the process of transporting an image into a different medial context. It then discusses pivotal moments in search of a concept of visual citation. An example of visual citation – Mundano’s appropriation of Portinari’s Coffee Farm Worker – is discussed in the fifth section. The chapter ends by suggesting that, by shedding light on key aspects of the interaction between different media, the concept of visual citation allows for a more nuanced understanding of broader processes in intermediality.

Compagnon’s Perspective on Citations Compagnon defines a verbal citation as the act of repeating an utterance or a series of utterances: “a citation is a repeated statement, that is to say, a clause or a string of clauses” (Compagnon, 2016: 68). Apart from repeating or reiterating, the act of citing resignifies the quoted text by the very act of inserting it within the discursive structure of the quoting text. This process creates a space within the text in which meaning is “launched and re-launched” leading to a dislocation and renewing of the quoted text (Compagnon 2016: 53). This moving back and forth from quoting to quoted text creates a dialogue between cited/cited works likely to reinforce the argumentative flow of the text in which it is integrated. Compagnon stresses this reiteration as the distinctive trait of quoting or citing. This repetition constitutes a complex strategy – once the very fact that an utterance is replicated in a different text, it displaces its original meaning by making it dialogue with the text in which it is inserted (the citing text). That is why, according to Compagnon, attempts to define “citation” exclusively from the standpoint of a discursive theory centered on the internal dynamics of the text tend to be unfruitful. He holds that citations “just exist as power.” Thus, understanding them demands perceiving citations as phenomena intrinsically linked to the singularity of each individual work as it is only in this context that citations reveal the specific forces that produce them, i.e., that produce an actual instance of text dislocation. The tension between the cited and the citing text is, thus, intrinsic to the functioning of verbal citations. Compagnon observes that the use of direct quotes may work as a device to validate or enhance the authority of the citing text and as a means to buttress its message. If ill used, however, this strategy may have the opposite effect and enfeeble the citing text by deferring the discursive authority to the cited text (Compagnon 2016: 12–13). In any case, Compagnon argues that in order to fully understand a citation one has to try to identify the goal the author has in using the cited text. According to him, the act of quoting “supposes, in fact, that

324

A. L. Ramazzina-Ghirardi

somebody seizes a portion of text, applies it to something else because s/he wants to say something different” (Compagnon 2016: 46). The interplay between sameness/ difference is key to the citation process. Compagnon assigns two values to citations: the use [or signifying], value, which is linked to the meaning proper of the quote, and the repetition value, which is linked to the act of citing. According to the author, the meaning attached to the quoted text is denser than that of the citing text and it is this dialogue between meanings that helps weave the plurality of meanings of the latter. The act of quoting authoritative authors functions as a way to lend the citing text an aura of previous approval and to support the assertions of the citing text (Compagnon 2016: 512). Compagnon considers the citation a “curious linguistic object” once it is “uttered by two voices, that of the first author and that of the author replicating it” (Compagnon 2016: 512). Compagnon’s influential perspective on the dynamics of verbal citations offers a valuable conceptual framework for the study of non-verbal, visual citations. The process of replicating and resignifying, in a new media, images produced by a different author has become omnipresent in the image culture of the late twentieth and twenty-first centuries, at the same time that the technical possibilities for performing such replication have multiplied. Different versions of concept of visual citation represent the effort by theorists to describe and understand this novel communicative device and to understand the manners by which it impacts meaning in the citing and the cited media.

The Rise of Visual Citation as a Theoretical Problem Nelson Goodman is considered a pioneer in the study of the dynamics of non-verbal citation due to his analysis of this practice in painting and in music. In “On Some Questions Concerning Quotation” (1974), later expanded into his celebrated Ways of Worldmaking (1978), Goodman examines what he terms “pictorial quotation,” which he defines as the reproduction of a painting through copying or other technique. Goodman realizes that the media revolution starting to take place at the end of the twentieth century is giving rise to fresh forms of producing and reproducing images. Working almost at the same time that Antoine Compagnon develops his seminal theory, Goodman argues that these new forms of image exchange and borrowing bear important similarities with the process of verbal citation as described by the French author. Acknowledging that there are similarities, however, does not mean denying crucial differences. While Compagnon examines citations within a wellestablished media – the written text – Goodman has to deal with them as they occur by novel processes in new media. He is therefore careful to stress that the phenomena that occur between images may be understood “as if” they are a citation. In verbal texts, Goodman points out, quotes can be either direct, when they are usually identified by the use of inverted commas or indentation, or indirect, by means of paraphrase, in which case they are usually identified by the use of references. These clear-cut possibilities of indicating the borrowing from another text, traditionally available for verbal citations, are

14

Visual Citation in Intermedial Relations

325

absent in the process of image replication. The unique features of visual language make his attempt to adapt a concept developed in connection with verbal texts rather challenging. He is aware that his investigation belongs to a different realm than that of Compagnon. Goodman’s research aims specifically at understanding the insertion of a pre-existing picture into a new pictorial product. He believes that our “ways of combining and constructing symbols” are crucial to furnishing us with tools to create “worlds” (Goodman 1978: 56). Images are a key part of this process and they may be subject to a process of citation akin to that occurring in written texts. He suggests the possibility that visual media cite images the same way that “a string of words can quote another string of words” (Goodman 1978: 41). He thus fittingly starts his investigation by undertaking a comparative study of quotation and its nearest analogues so as to determine the conditions under which one could properly speak of quotations in non-verbal systems. Goodman structures his project of designing a relevant investigative path around a very clear central question: “when do we have quotation in non-verbal systems?” (Goodman 1978: 47). He acknowledges, from the beginning, that the simple fact that one image contains another is not sufficient to establish a quotation: “[a] double portrait does not quote the contained portraits; a seascape does not quote the picture of a ship in it” (Idem, ibid). His contention is that quoting in the visual realm requires more than the mere reproduction of an image: “nor does the reference to a picture by another constitute quotation” (Goodman 1978: 47). This cautious approach is characteristic of Goodman’s study on pictorial citation. He appears more focused on posing new questions than on finding definite answers. This open approach will prove valuable for the integration of his theory into contemporary intermedial studies. Goodman argues that there are two sine qua non conditions for a citation to take place: “(a) containment of what is quoted or some other replica or paraphrase of it” and “(b) reference to – by naming or predication – of what is quoted.” To these, he adds a third condition, which is necessary for direct citations: “(c) the replacement of the denoted and contained expression by any other of the language results in an expression that denotes the replacing expression”(Goodman 1978: 44). Given these conditions, for Goodman, “a picture directly quotes another if it both refers to and contains it” (Goodman 1978: 47). For the quoting to actually occur, however, it is necessary that one is able to identify the presence of these two key elements (containment, reference) in a pictorial work. And this requirement, in turn, leads the author to look for devices that, in the realm of painting, might function as the equivalents of the quotation marks in verbal texts: “what are the means by which a picture refers to another that it contains? In other words, what is the pictorial analogue of quotation marks?” (Goodman 1978: 47). In his characteristically cautious diction, he offers only tentative answers to these questions: “as quotation marks are put around an expression to quote it directly, a picture of a frame may be put around a picture to quote it directly; and there are other devices, such as painting it as on an easel, or as hanging on a wall, that work in the same way” (Idem, ibid). Focusing on pictorial citations, Goodman argues that pictures belong to a distinct symbolic system and that each picture is unique. That is why, according to him, it would be difficult for an imagetic system to reproduce the quoting strategies of written

326

A. L. Ramazzina-Ghirardi

texts: “what we consider to be direct pictorial quotation will depend upon what we are willing to take as an adequate analogue of replication in direct verbal quotation” (Goodman 1978, 49). His remarks seem to bespeak his concern with distinguishing between quoting and plagiarism, that is to say, between the act of resignifying an image by placing it in a new work and the mere copying of a whole painting. In what regards indirect citation, Goodman understands that it can only be determined through an analysis of the context and the traits of the concrete work. This type of citation occurs, according to him, when “a painted [. . .] function[s] as an analogue either of quotation marks or of “that”” (Goodman 1978: 49). Goodman stresses that the cited image may belong to a different system than that of the citing work. This observation anticipates the problem of visual citation between semiotic systems – for example, the citing of a painting or a photography in comics – which has been at the heart of important contemporary research. Goodman’s work is contemporary to Compagnon’s and, as pointed out, bears many points of resemblance with it, notably the attention to the discursive implications of the act of inserting an existing product into a new one. Goodman’s choice of verbal citation as the default category to think about image transposition may help explain why his texts present a more nuanced diction, less assertive than those by Compagnon. Nonetheless, Goodman’s work may be seen as the starting point for the discussions involving visual citation. He lays the theoretical groundwork for later analyses of the complex process of the transfer and replication of images across different media and helps establish visual citation as a discrete topic of research.

Visual Citation in Various Media Marie-France Chambat-Houillon explores the dynamics of visual citation in her research on TV programs. Though focused specifically on television, her theoretical insights provide useful guidance for the study of this practice in other media. Chambat-Houillon argues that certain discourses are incapable of citing other discursive practices due to their lack of semiotic competences. She illustrates her argument by pointing to the limits of the literary text to fully quote all the aspects of images (paintings, drawings, photography) (Chambat-Houillon 2009: 2 – own translation). According to her, in order to overcome the obstacle posed by the nature of the written text vis-à-vis images, literature “makes use of the trans-semiotizing of iconic signs into linguistic signs, the only type of sign acceptable and compatible with the final locus of integration: the written text” (idem, Ibid). According to Chambat-Houillon, literature puts in motion a “practice of intrasemiotic citation, even homosemiotic citation, where the cited fragment and the citing discourse share the same signifying nature” (Idem, Ibid). Her argument refers to various strategies that verbal language offers to translate visual objects into words, such as the use of ekphrasis in literary works. As Goodman had done before her, Chambat-Houillon takes citations in verbal texts as the starting point for her discussion of citations in TV programs. The plurisemiotic nature of the media television, suggests Chambat-Houillon, as well as its technical resources, allows it to perform visual citations in ways not available

14

Visual Citation in Intermedial Relations

327

to literary texts. This proficiency in referring to images makes the citation less evident to the spectator once they appear involved in a broader set of signs “whose organizing principle is significantly less explicit and coercive” than that of the written text (Chambat-Houillon 2009: 2–4 – own translation). The mediatic characteristics of television allow it both to present reality directly (such as in direct coverage) and indirectly, by replicating excerpts of other imagetic discourses. Discussing visual citation on television, Chambat-Houillon revisits the values attributed to verbal quoting by Compagnon and Goodman in their effort to distinguish between direct and indirect citation of a given work. In the TV realm, these two main strategies translate into different enunciation strategies that can be organized into “two main families of citation”: replica citations and performance citations. Replica citations, as the name clearly indicates, reproduce the image as it is, preserving its integrity. An example of this practice occurs when one TV show uses footage taken from other shows or films. According to Chambat-Houillon, this is the most common practice in TV emissions because it is in line with the literary dynamics of quoting which “proposes the literal and specific inscription of one dimension of the cited work” (Chambat-Houillon 2009: 5 – own translation). Chambat-Houillon points to American TV series Dream On (1990–1996) as a good example of replica citations. Throughout its episodes, the series makes use of old black-and-white movies and TV shows to explore and illustrate the feelings and thoughts of the leading character. In this case, the spectator immediately recognizes that these are pre-existing sequences and images that are being cited by the new program. Replica citations, thus, merely insert unchanged fragments of a former work into the new media product. Performance citations, on the other hand, do not directly replicate a previous existing sequence or image but use it to develop a metatextual commentary or argument and to position the author in relation to the sequence or image cited. In this case, “repetition does not take place, thus, by the conservation of the initial materiality of the cited work, but by its re-creation, which is triggered by the productive context of the [TV] show” (Chambat-Houillon 2009: 6 – own translation). In this manner, performance citations do not replicate an unadulterated version of the cited work, but represent it via imitation. In performance citations, similarities with the cited work are desirable, once it is paramount that the audience perceives that a reference process is taking place: “Fidelity is the pragmatic condition of this category once, for it to exist, it is necessary that the spectator recognizes the work or discourse being reproduced” (Idem, ibid – own translation). The American TV series Modern Family (2000–2009) offers a prime example of performance citation. Episode 13 of 4th season (“Fulgêncio”) shows the christening of Jay and Gloria’s son. The scene of the baptism, in which Phil appears as godfather to the child, is interspersed with scenes of violence perpetrated against other characters, in clear reference to one of the most memorable sequences of Coppola’s The Godfather (1972). Unlike what happens in replica citations, performance citations resignify the fragment cited by incorporating it to the narrative flow and the technical characteristics of the new media product. The aim of this discursive strategy is to make the fragment immediately recognizable as part of another

328

A. L. Ramazzina-Ghirardi

work; thus performance citation will only function if the receptor is proficient to identify the cited work and to incorporate its meaning to the new narrative. Chambat-Houillon analysis of visual citation in television provides material apt for productive dialogue with studies on other media and their strategies for citing non-verbal products. Her work is a testimony to the critical potential of applying the conceptual framework of intermediality to analyze the reconfiguration of images within different semiotic systems. In line with the research concerns of Goodman and Chambat-Houillon, Thierry Groensteen’s work also focuses on the analysis of the processes of imagetic citation. Groensteen approaches this phenomenon by discussing the role that visual citations play in the shaping of the narrative structure in comics. His “La bande déssinée mode d’emploi” (2015) debates the range of possibilities for citing images in comics and examines “the interplay of explicit or implicit citations which serve as a source for both text and images” (Groensteen 2015: 140 – own translation). To illustrate his point, Groensteen refers to the episode “Histoire désopilante” in Gotlib’s Rubrique-à-brac. In this story, Gotlib evokes the iconic scene in Chaplin’s Modern Times, in which the main character frantically tries to keep tightening screws at the rhythm imposed by the assembly line. Groensteen suggests that this translation of a movie scene to the language of comics is emblematic of the possibilities of dialogue between different media systems. When discussing the role of citation in the structuring of comics narrative, Groensteen suggests that comics, due to their characteristics as media, are particularly suitable for the use of citations. He argues that they “offer a prime locus for an aesthetics of borrowing and pasting” (Groensteen 2007: 77 – own translation). This suitability to receive and absorb different types of citations springs, on the one hand, from the fact that comics combine two different semiotic systems, namely, text and image, which allows for verbal, visual, or mixed citations. On the other, it derives from the discontinuous nature of the narrative in comics, which makes it possible for a citation to find a specific, limited insertion point within the plot. Groensteen, in line with Goodman and Chambat-Houillon, also emphasizes that receptors are key for this process of collage and borrowing to fully function. Groensteen highlights the importance of the reader as a “witness” to the citation process and as an “accomplice” who helps the author put in motion the dynamic interplay of multiple citations (Groensteen 2015: 144 – own translation). For Groensteen, as visual citations lack the explicit quotation marks characteristic of verbal texts, they can only materialize and come to a full sense if the receptor, by deploying her previous media repertoire and encyclopedic knowledge, is able to identify specific images and signs. Groensteen suggests that authors of comics often explore the multiple possibilities of visual referencing to build a web of citations. Even the most implicit of these becomes likely to be recognized because authors create a narrative environment that keeps receptors constantly alert to the possibility of a citation coming up. At the same time, they guide the reader’s attention in the desired direction. By constantly overlapping visual and verbal citations, authors of comics transform readers into detectives looking for clues to decipher the references scattered along the story.

14

Visual Citation in Intermedial Relations

329

Authors have however to be careful, warns Groensteen, not to create so dense a web of citations as to mislead the readers and push them to over-interpret. He illustrates this danger by making reference, once more, to Gotlib’s “Histoire désopilante” (1973). Groensteen criticizes Gotlib’s “piling up of citations” whose ensemble fails to coherently integrate into the main narrative and muddles the plot (Groensteen 2007: 90 – own translation). These examples should help understand, according to Groensteen, the functioning of citations in comics. The flexibility of their use makes them fit to enhance the new comics narrative, serving as valuable accessories to add depth to the story being told. Nevertheless, as already pointed out, there are dangers attached to this openness of comics to receiving external images: too many citations in a single work may divert the attention of the receptor and paradoxically diminish the possibility of effective dialogue between texts. Groensteen’s scrutiny of the use of citations within the semiotic system of comics adds another layer to the discussion of visual citations in different media. The emphasis he places on the role of the receptor and her encyclopedic knowledge (as defined by Eco) to detect specific references suggests that the proper understanding of the process of visual citation requires that due and constant attention be given to the author-text-reader triad. Groensteen’s attention to dynamics of visual citations in comics bears important points of contact with Claus Clüver’s work on the use of citations of signs, texts, and images taken from different semiotic systems and their impact on the internal functioning of specific media. Goodman, Chambat-Houillon, and Groensteen converge on the idea that the imagetic nature of visual citations often makes them less discernible than their verbal counterpart, a trait that adds importance to the interpretive skills of the receptor. Clüver, for whom “text and images are intrinsically fused in plurisemiotic environments,” examines the links between the processes of citation and the transposition of intermedial texts. Clüver gives special attention to what he names “intermedial text” and defines it as a system which “puts together two or more sign systems and/or media in such a way that the visual and/or musical, verbal, kinetic and performative aspects of their signs become inseparable and indissociable” (Clüver 2006: 18 – own translation). Clüver suggests that this type of text can be transposed to another semiotic environment; in this case “the new text can be understood only as a citation [...] in order to obtain an intermedial effect by means of intermediality” (Clüver 2006: 18– own translation – emphasis added). In Inter textos/Inter artes/Inter media, Clüver illustrates this idea by analyzing two celebrated sculptures: Robert Indiana’s Love – which, in turn, was inspired in a Christmas card the artist had been commissioned to create by the Museum of Modern Art in New York City – and General Idea’s AIDS to exemplify the complex relations between intertexts and citations. Clüver is interested in the ways the citation process functions to multiply the possible meanings to the citing work. AIDS can only be fully understood, suggests Clüver, if spectators recognize it as a citation of Indiana’s work, that is to say, if they become aware of the visual reference produced by the artists of General Idea. This recognition, in turn, depends on the

330

A. L. Ramazzina-Ghirardi

identification of the specific manner of combination of the two semiotic systems – images and verbal text – which characterizes both works. According to Clüver “the intertextuality of this intermedia text is based on its visual signs; however, without the meaning expressed by its verbal signs, the whole would make no sense at all” (Clüver 2006: 29 – own translation). Clüver’s analysis of AIDS explores the connection between mediatic re-appropriation of images and intermedial texts and greatly contributes to the current debate on visual citation. By discussing the process of media transposition, from the standpoint of intermedial texts, Clüver offers important clues for understanding citation between different media and/or sign systems and opens a path for making visual citation a central topic in intermediality research. Marie-Laure Ryan is another researcher who adopts an intermedial perspective in her approach to the dynamics of visual citation. She takes Lubomír Dolezel’s work to develop her analysis of the connections between citation and fictional narratives. She proposes that Dolezel’s categories to discuss the possible ways by which fictional universes relate – namely, via expansion (new stories are added to a fictional world in a manner that events in the original narrative are respected and replicated), modification (changes are made to the original plot), and transposition (the plot is transferred to a different time or spatial setting) – be complemented by a fourth one: citation. Ryan argues that a citation occurs in a fictional universe when a new element is added to a given narrative universe without being totally absorbed by. This happens when the new object cannot be harmonized nor made to blend with the environment in which it is inserted. According to Ryan, both Dadaism and Surrealism make ample use of this strategy. An example of this type of citation would occur, she argues, if a character in The Lord of the Rings used a lightsaber from Star Wars. In this case, “the imported element is not integrated in the storyworld, and the effect is one of dissonance and incongruity” (Ryan 2013: 367). Due to the strangeness it produces, this type of imagetic citation is immediately recognizable by receptors, who promptly identify that the new object is a stranger to the citing media. A more recent example of such practice is offered by Sofia Coppola’s Marie Antoinette (2006). In one of the scenes in the movie, it is possible to see, for a brief moment, a pair of twentieth-century Converse All Star sneakers among the queen’s impressive collection of eighteenth-century shoes. The anachronism of the citation is immediately recognizable by the spectators who are instantly aware that this element does not belong to the universe depicted by the fictional work. This oddity is likely to trigger attempts at producing some interpretive hypothesis capable of coherently integrating this element into the general meaning of the narrative. It thus potentially expands the expressive force of the movie (asked about it, Coppola said that the presence of the sneakers was intended only as a joke). The type of citation proposed by Ryan has at its heart the ideas of the intentional incongruity of the new element and the interpretive efforts it triggers. It constitutes a citation strategy that deliberately breaks with the receptor’s expectations on the internal functioning of the media. Her work helps enlarge the possibilities for conceptualizing visual citation between complex sign and media systems. Lars Elleström offers important theoretical tools to understand visual citation from the standpoint of intermediality. He suggests that studies in this area pay

14

Visual Citation in Intermedial Relations

331

attention to three complementary types of media: basic, qualified, and technical. Basic and qualified media are considered “abstract” in his theory, while technical media are concrete. Basic media represent “intermediate entities that enable transfer of cognitive import between at least two minds” (Elleström 2021: 54). Qualified media depend on the historical, cultural, and social circumstances as well as on their aesthetic and communicative contexts. Technical media represent the means by which both basic and qualified media materialize. In connection with these three complementary types of media, Elleström establishes four modalities of media: material, space-temporal, sensorial (all of which he considers pre-semiotic), and semiotic. Material media are the corporeal interfaces which make the media accessible to the perceptor’s mind; space-temporal media are the spatial and historical elements which structure the perceptor’s apprehension of the material media; sensorial media are the human senses (eyesight, hearing, taste, touch, and smell) needed to perceive the media; semiotic media refers to the representational elements (icon, index, and symbol as defined by C. S. Peirce) which help elaborate the perceptor’s representation of the media. Due to their function in the process of representation, semiotic media are key to conveying meaning through a specific media product. Elleström’s categories shed light on the dynamics that takes place when one image is transported between media, i.e., when there is an instance of visual citation. They allow for an analytical perspective capable of simultaneously relating visual and verbal citation and of differentiating between them. In what regards verbal citation, Elleström observes that letters and words with their “symbolic sign functions” will guide the process of interpretation (Elleström 2021: 51). Lars Elleström also discusses the process of citation within his studies on media transformation. When presenting his concept of representation, Elleström observes that “[a] media product may hint at, allude to or refer to another medium, it may mention or name another medium, and it may quote, cite or comment on another medium” (Elleström 2013: 121). Representation thus occurs, according to Elleström, when the receptor [cognitively] responds to a given media product. He suggests that the sign system which springs from representation is structured around three main elements: illustration (representation based on similarity), indication (representation based on proximity), and description (representation based on habits and conventions). According to this author, when one says that a media product represents something, what is meant is that such product triggers a process of interpretation (Elleström 2021: 38–40). He suggests that representation may be symbolic, iconic, or indexical and proposes “the term description to denote the process of symbolic representation, depiction for iconic representation and indication to refer to indexical representation” (Elleström 2013: 121). For Elleström, the internal functioning of description involves the processes of citing and commenting. Such processes, he adds, have recently become associated with the phenomenon of illustrating once it is possible to say that images in a given media product “quote” images in another product. Elleström’s observations help enlarge the debates on visual citation by expanding its universe of inquiry. His insights indicate that the study of visual citation should involve more than the analysis of image quoting and include the quoting of one type of media by another.

332

A. L. Ramazzina-Ghirardi

Visual Citation and Intermediality As pointed out at the start of this chapter, the key characteristic of citations is the insertion of a fragment of a previously existing text (be it verbal, visual, or others) into a new one. This movement between two texts (the Latin meaning for “citare” is “to put [something] in movement”) establishes an interplay of former/new senses, of repetition/creation, which acts to amplify and resignify both the citing and the cited text. Citations can only successfully occur, however, if receptors identify the insertion as a discursive device, that is to say, if they can understand that a borrowing is taking place, that an external element is being integrated into the body of the new text. Citations are, therefore, an intrinsically “inter-”phenomenon, in the sense that it always involves at least two texts. Arguably because of this dialogic characteristic of citations, in general, and of the necessary interplay they produce between texts, visual citations have become a recurrent topic within the studies of intermediality. The process of replicating or recreating an image within a different work has been analyzed in connection with the concepts of intermedial reference, intramedial reference, remediation, and transmediation, among others. Within intermediality studies, analyses of quoting strategies must take into account that a media product does not have a monomodal semiotic configuration but articulate a variety of systems – verbal, sound, image – whose interplay is intrinsic to the constitution and understanding of the product. This plurality of systems and the variety of citation strategy schemes it allows have generated a wealth of theoretical proposals to describe different aspects of visual citations. Irina Rajewsky offers the concept of intermedial reference, which occurs when a main (primary) media refers to another (secondary) media by thematizing, reproducing, or referring to elements, techniques, or structures of the latter media. The main media uses its own codes, tools, and methods to emulate the secondary media. This means that it refers to the other media rather than, explicitly or implicitly, citing it. “Intermedial references” she argues “are thus to be understood as meaning-constitutional strategies that contribute to the media product’s overall signification: the media product uses its own media-specific means, either to refer to a specific, individual work produced in another medium” (Rajewsky 2005: 52–53). Analyzing visual citations from the standpoint of intermediality also requires that one bears in mind that media are composed of a series of relations arising from the different semiotic elements they are composed of. This complexity allows for a plurality of ways in which a primary media can re-appropriate and signify a secondary media. In what regards specifically the citing of images, there are two main strategies by which the main media can cite an image from a secondary media: 1. Directly (the replication of an image as originally conceived) with the resources of the different mediatic environment. This type of citation makes the act of citing more explicit.

14

Visual Citation in Intermedial Relations

333

2. Indirectly (use of semiotic elements pertaining to the citing media) with the resources of its own environment. In this case, it is not always possible to identify the cited image and the intention of the producer in making the citation. Because both processes are key to visual citation, they deserve further explanation. (i) Direct citation: When the citing media duplicates or replicates the cited media product preserving the traits of the semiotic system in which it was produced, the citation becomes explicit. In this case, even if the receptor does not identify the work being cited, she is, nevertheless, aware that there is a citation being made. I refer to this modality of citation as “decal visual citation,” as it is an explicit reference to the cited image. Such reference becomes immediately identifiable due to the difference between the means of expression in the cited and in the citing media. It is as if the citing media, within its semiotic interior, made a decal of the cited media, thus making it evident that it is quoting another work. In this kind of citation, receptors readily perceive that there is an insertion of an image coming from another source even if the cited image is not part of their encyclopedic knowledge. This occurs because there are elements in the new image that are strange to the mediatic characteristics of the citing work. Didier Lefèvre’s “The Photographer” serves as an apt illustration of decal visual citations. In these comics, the French photographer narrates his journey alongside Doctors Without Borders in Afghanistan during the Soviet-Afghan war. In “The Photographer,” drawings characteristic of comics alternate with photos or photograms of real people and landscapes to expand the narrative possibilities of Lefèvre’s work. Images of a caravan crossing the rocky Afghan sceneries and photos showing a little child’s frightened stare at men with machine guns or of the unexpected use of a goat in a class aimed at teaching the nurses how to proceed during surgeries all intermingle with the drawings to give immediacy and depth to the story being told. By inserting photographs in the semiotic interior of the comics, the author makes a decal visual citation that sheds a different kind of light on the narrative, as the receptor is aware that there are two semiotic systems functioning simultaneously. (ii) Indirect citation: It occurs when the citing media emulates another media product and incorporates it by using its own semiotic resources. In this case, the visual citation is less likely to be recognized, once there are no evident marks indicating that an external object has been inserted. The cited image undergoes a metamorphosis to acquire the characteristics of the citing media. It can only be perceived as a citation if receptors have the encyclopedic knowledge needed to identify the quote. This type of visual citation makes the cited image mediatically homogeneous to the environment of the citing media, thus making less conspicuous the dialogue between the cited image and citing text. I refer to this modality of citation as “inter-

334

A. L. Ramazzina-Ghirardi

glossing visual citation.” This expression was chosen based on a notable observation by the sixteenth-century philosopher Michel de Montaigne: “there are more books about books than about any other topic: we do nothing but quote each other [entregloser]” (Montaigne, Essays, III – own translation). One could perhaps paraphrase Montaigne and say that “there are more images about images than about anything else and they are unceasingly citing – interglossing – one another.” In this sense, visual citation establishes a dialectic relation between two media, which refer back and forth to each other. It is as if the citing and the cited media commented each other, establishing a citation dynamic that will be effective only if receptors have the requisite media model repertoire needed to understand that a dialogue is at play. Ari Folman and David Polonsky’s comics version of “The Diary of Anne Frank” (2017) provides a good example of inter-glossing visual citations. To portray the ambivalent feelings of the main character regarding her mother – as registered in the verbal language of the diary – the authors transport into the comics semiotic environment two works produced in a different media: the paintings The Scream (1893) by Edvard Munch and Portrait of Adele Bloch-Bauer (1907) by Gustav Klimt. The success of citing these paintings as a narrative strategy depends entirely on the cultural proficiency of the receptor. If she does recognize the two masterpieces, then this insertion will add substantial layers of meaning to the story line: the distorted face in Munch’s painting suggests Anne’s emotional turmoil, while the luxury and ambiguity of Klimt’s image suggests Anne’s ambivalent feelings towards her family. The paintings enhance the working of the citing media and strengthen its narrative power. Both the inter-glossing and the decal visual citations become possible because of the multitextual system engendered by media references and point to the complexity of the idea of visual citation in studies of intermediality.

Visual Citation in Practice: Mundano’s Appropriation of Portinari’s Coffee Farm Worker The work “A member of the fire brigade” (2021) by Brazilian artist Mundano is an apt illustration of the functioning of inter-glossing [visual] citation. Mundano is considered an “artivist,” a mix of artist and activist, and uses his art to fight for a variety of social justice issues. To create his massive graffiti, painted on the wall of a tall building in downtown São Paulo, Mundano used ashes and coal from four forest reserve areas which are routinely the object of illegal “queimadas” (criminal burning of the forest): the Amazon, the Atlantic rainforest, the Cerrado, and the Pantanal, both in central Brazil. The ashes and the coal were mixed with water and paint to create new colors which the artist named “Decimated amazon,” “Endangered Pantanal,” “Greedy Cerrado,” and “Devastated Atlantic rainforest.” The member of the volunteer fire brigade (“brigadista”) represented in the painting is Vinicius, a young man who lives in the Cerrado region. Vinicius, alongside an army of volunteers, tries to stop the

14

Visual Citation in Intermedial Relations

335

environmental destruction caused by illegal fires. Mundano uses his work to denounce to the world the atrocities of environmental crimes and to honor the humble, anonymous people who valiantly work to save the fauna and the flora of the region.

Mundano’s Brigadista da Floresta, 2021. (Photo by André D’Elia)

Mundano’s work stages a visual citation of Candido Portinari’s masterpiece Coffee Farm Worker (1934 – w81
h100 cm, image available at https://masp.org. br/acervo/obra/o-lavrador-de-cafe). Portinari’s painting, a landmark in Brazilian art, also intends to offer a critical look at the dire social consequences of the coffee export industry at the heart of the Brazilian economy in the early twentieth century. Portinari makes extensive use of vivid colors to convey his political message: the deep red earth suggests the fires and the devastation of the land, while the similar tones used for the worker and the coffee beans seem to imply capitalism’s brutal exploitation of the workforce in search for profit. The interplay of similarities/dissimilarities between the two works makes for the representation process (as defined by Elleström – see above) at work in visual citations. Similarities are plenty: Mundano’s graffiti replicates, in minute detail, a

336

A. L. Ramazzina-Ghirardi

number of elements in Portinari’s painting – in both works, the main characters are African-Brazilian young men; both are portrayed looking to the right, holding a hoe, legs and arms in identical position; in the expression of both, one can detect a pensive mood as they scrutinize the horizon, far away. The shadow projected by the arm of the worker, the oversized hands and feet, and the felled tree, all key traits of Portinari’s work, are also present in Mundano’s. This plethora of similarities makes the differences all the more striking and composes the functioning of the visual citation. In Portinari’s piece, the main character is leaning on the hoe, as if momentarily resting, in a passive attitude, the hoe seemingly a continuation of his own body. Mundano’s “brigadista” holds the hoe as if it were a weapon and displays an active, defiant stance against the danger. While the young man in Portinari’s work appears shirtless and barefoot, suggesting his poverty, Mundano’s subject wears all the gear – boots included – needed to accomplish his task. The choice of colors may be, however, the major difference in both works: Portinari’s rich palette is replaced by the omnipresent gray in Mundano’s denunciatory graffiti. Receptors with knowledge of Portinari’s work are immediately struck by the disappearance of virtually every shade except the ones of a burned-to-the ground, devastated forest. Portinari’s blue gives place to Mundano’s leaden tones to depict the sky as seen through the thick smoke rising from the scorched forest. Mundano also adds new objects to his work, which are elemental to the message his graffiti is meant to convey: an hourglass serves as a memento of the urgency of taking actions to protect the environment; a loudspeaker implies the need to make the problem of criminal deforestation known to the world; the carcass of a reptile is a reminder of the destruction, by the “queimadas,” of every form of wildlife. Mundano’s visual citation thus updates, reinforces, and resignifies Portinari’s social criticism. Portinari (1903–1962) was deeply involved in social and political causes; his work was structured around key themes of Brazilian life and culture – religion, folklore, labor – and stages a complex, unrelenting denunciation of social inequality in the country. Mundano’s work makes a representation (Elleström, above) of Portinari’s work of social criticism. Portinari’s indictment of a deeply unfair model of economic exploitation is renewed and strengthened by Mundano’s reminder that the centuryold social problem is now compounded by unprecedented ecological devastation. Though certainly very powerful in itself, Mundano’s aesthetic solutions and social criticism gain great depth if receptors are capable of identifying his inter-glossing visual citation and establish, in their minds, a dialogue between Mundano’s work, the cited media, and their own cultural repertoire. The citing and cited media establish a sophisticated structure of references that allows for dislocation, replication, and reinforcement of themes and that expand the possibilities of meaning of both works. Mundano’s visual citation serves, therefore, as a good example of the means by which an image can cite another and helps answer some of the questions posed by Goodman in “On Some Questions Concerning Quotation” (1974). Mundano’s “brigadista” is an analogue of Portinari’s Coffee Farm Worker and serves as an

14

Visual Citation in Intermedial Relations

337

organizing element to the complex dynamics of emphases and resignifying put in motion by “A member of the fire brigade.” Visual citations become therefore possible when the multitextual system of the citing media is capable of receiving forms of sign coming from different media. In this process, the full effect intended by the producer will depend on receptors’ ability to identify the cited media and on their proficiency in the different mediatic systems put in dialogue by the visual citation. Visual citations demand, hence, that receptors adopt an active stance and that they subject the new work to a constant scrutiny to look for possible references to previous works. That is why visual citations, both in their inter-glossing and in the decal varieties, depend on the mind of the receptor for the representation and the actual communication of the meaning generated in the producer’s mind.

Conclusion The contemporary revolution in communication technology and the remarkable expansion of media products it has entailed have exponentially complicated the practice of quoting. Though remaining an unsurpassable starting point, Compagnon’s pioneering efforts to conceptualize verbal citations are not sufficient to make sense of the novel types of referencing that are omnipresent in the contemporary media landscape. In the image culture, visual citations occupy a major place in the new strategies of quoting. The insertion of an image in different medial milieus and different semiotic systems, far from being a mere replication, implies the reconfiguration of their internal functioning and of the web of meanings attached to them. Transporting images into the new media products, be it explicitly (by processes such as decal citation) or implicitly (by processes such as inter-glossing citation), and inserting them into new semiotic systems transform both the citing and the cited product. Visual citations establish a dialectic interaction between the cited image and the new media that translates into a complex reconfiguration of their semiotic structure. In order to be properly understood, visual citations require that receptors deploy their encyclopedic knowledge, their media repertoire, and their hermeneutic abilities to identify the elements put in tension. Visual citations function, thus, much like a painting placed behind a glass frame: receptors simultaneously see both the image and their own faces reflected on the surface protecting it. This co-presence utterly transforms the perception of the object being observed as it makes viewers constantly aware not only of their own position as interpreters but also of the materiality of the painting, of the technical apparatus its production involves, and of the media system within which it is set. Visual citations constitute a key topic in studies of intermediality once it encapsulates some of the major theoretical challenges posed by the emergence of the image culture and the contemporary deep overhaul of communication media and strategies.

338

A. L. Ramazzina-Ghirardi

References Chambat-Houillon, Marie-France. 2009. De l’audiovisuel vers le télévisuel: deux modèles de citation pour les émissions de télévision. Nice: Revue Ci-Dit. http://revel.unice.fr/symposia/ cidit/index.html?id¼408. Clüver, Claus. 2006. Inter textus / inter artes / inter media. Aletria: Revista De Estudos De Literatura 14 (2): 10–41. https://doi.org/10.17851/2317-2096.14.2.10-41. Compagnon, Antoine. 2016. La seconde main, ou, Le travail de la citation. Paris: Points. Elleström, Lars. 2013. Adaptation within the field of media transformations. In Adaptation Studies: New Challenges, New Directions, ed. Jørgen Bruhn, Anne Gjelsvik, and Eirik Frisvold Hanssen. London: Bloomsbury. ———. 2021. The Modalities of Media II: An Expand Model for Understanding Intermedial Relations. In Beyond Medial Borders, Volume 1. Intermedial Relations among Multimodal Media, ed. Lars Elleström. London: Palgrave Macmillan. Goodman, Nelson. 1978. Ways of world making. London: Hackett Publishing Company. Groensteen, Thierry. 2007. Gotlib ou l’euphorie citationnelle. In Bédé, ciné, pub et art: d’un média à l’autre, ed. Colloque international "Bédécinépubart" and Philippe Kaenel. Gollion: Infolio. Groeensteen, Thierry. 2015. Le citationnel. In La bande dessinée: mode d’emploi, ed. Thierry Groensteen. Bruxelles: Les Impressions Nouvelles. Rajewsky, I.O. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. Intermédialités/Intermediality 6: 43–64. https://doi.org/10.7202/1005505ar. Ryan, Marie-Laure. 2013. Transmedial storytelling and transfictionality. Poetics Today 34 (3): 361–388. https://doi.org/10.1215/03335372-2325250

Reformulating the Theory of Literary Intermediality: A Genealogy from Ut Pictura Poesis to Poststructuralist In-Betweenness

15

Bowen Wang

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Genealogy of Ut Pictura Poesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Literary Intermediality out of Interdisciplinarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Poststructuralist Perspective: The Philosophy of In-Betweenness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

340 342 348 353 360 361

Abstract

The Horatian dogma of ut pictura poesis positions itself at the interdisciplinary confluence not only of poetry and painting, but also, to a large extent, of rhetoric and aesthetics, epistemology and philosophy, cognition and other related social sciences. The theoretical discussion on the relationship of different media first emerged as a structural parallel that stressed kinship and harmony in Aristotelian poetics, and was then transformed into paragonal debates that emphasized difference and exceptionality during the Renaissance. Later, it arrived at an extreme delimitation of temporal and spatial arts, which deepened the historically empirical concern regarding philosophical and aesthetic fundamentals in the implications of binary opposition. Therefore, to sidestep the formal comparativity or time/space dichotomy as delineated from a genealogical approach, the notion of intermediality should be reformulated as a conceptual and ideological extension of ut pictura poesis. Its intermedial core shall be better understood on a theoretical basis of in-betweenness underpinned by a poststructuralist tendency. These both provide alternatives to the universality, radicalism, grand narrative, and fixed identity championed in the traditional literary and other art forms. Coming from the position of the in-between, intermediality has offered a series of new perspectives that transcend a limited point of view, cross the boundaries of B. Wang (*) Trinity College Dublin, Dublin, Ireland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_5

339

340

B. Wang

stereotyped sections, decenter long-established authorities, link disconnected fragments, and search for revolutionary potentials. At the ultimate end, it creates a dynamic, fluid, and highly interactive communication that was scarce in the preexisting literary and art historiography. Keywords

Intermediality · Ut pictura poesis · Theory · Poststructuralism · In-between

Introduction In “The Interart Movement,” a short foreword to Interart Poetics: Essays on the Interrelations of the Arts and Media from the “Interart Studies: New Perspectives” conference held in May 1995 at Lund University, Stephen Greenblatt, inspired by the realignment and new departments created in natural sciences, questioned the monolithic and stable departmental structures in the contemporary study of literature (Greenblatt 1997: 13). He proposed to reestablish this academic subject by shifting the intellectual alliance and transforming the institutional configuration. On this occasion, the conventional way of reading and analyzing literature seems to be somehow less vigorous within the old disciplinary and fixed medial boundaries. It needs to be reread and reconsidered in a transdisciplinary, intermedial regime by allying literature itself with visual art, music, sculpture, cinema, dance, theatrical performance, and other nonverbal forms of media. To achieve this, the new collaboration of intermediality has to be preconditioned to face many obstacles: the habitual mode of knowledge, the old operation of power dynamics, and the established hierarchy of literal word as the dominating medium throughout the intellectual history. The collapse of this borderline and the shaping of a conjecture cannot be a simple task. For instance, Stanley Fish and his supporters might be one of the strongest and most persistent opposition parties against the idea of intermediality – out of interdisciplinarity. As a defender of the possession of knowledge, the protocol of communities, and the historical achievements of an existing discipline, Fish radically suspects this new pedagogy of antiknowledge and the fragile identity of any interdisciplined subject. He thus declares the inescapability of institutionalization via the hidden political nature and ideological agenda of any medium or intermedia: “The blurring of existing authoritative disciplinary lines and boundaries will only create new lines and new authorities; the interdisciplinary [or intermedial] impulse finally does not liberate us from the narrow confines of academic ghettos to something more capacious; it merely redomiciles us in enclosures that do not advertise themselves as such” (Fish 1989: 18).1 He is unquestionably anti-formalist,

1 This article is later collected in Fish, There’s No Such Thing as Free Speech, and It’s a Good Thing (Oxford: Oxford University Press, 1994) 231–242. About his idea of English studies as a political phenomenon, see Professional Correctness: Literary Studies and Political Change (Cambridge, MA: Harvard University Press, 1995).

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

341

neo-pragmatist, and obsessed with a community of ideal readers – meaning “ideal” in his own view – equipped with an agreed set of special knowledge and analytical methods concentrating on the fixed meaning. However, Fish’s defense of both his self-organized interpretive community and literature as an independent, selfsufficient institution is highly likely to fail in its task. This is due to the contradictory fact that shared interpretive strategies are inconsistent with the subjectivist way of reading that he repeatedly emphasizes as a generative activity. In this vein, his paradox between the stability and variety of interpretations renders the consensus of reading impossible to achieve. Unlike Fish, Greenblatt at the start actively attempts to resist any medial or “disciplinary hegemony” and to discover in the media-oriented “interdisciplinarity an important means of generating new knowledge” (Greenblatt 2005: 3). From the determined skepticism toward the unified description of cultures, histories, or texts, he and new historicists have shifted their interest from inwardly literary and aesthetic values, such as Formalism and New Criticism, which treat literature as an ahistorical unity organic on its own, to a collection of practices synthesizing close reading and extrinsic analysis. His idea of cultural poetics aims to erode the firm ground or working distinction found by Fredric Jameson, which “reconfirms that structural, experiential, and conceptual gap between the public and the private, between the social and the psychological, or the political and the poetic, between history or society and the ‘individual’” (Jameson 1981: 20). This liberation of reading literature from an absolute medium also echoes Vincent Leitch, who appeals to “uproot the frozen text; break down stereotypes and opinions; suspend or baffle the violence and authority of language; pacify or lighten oppressive paternal powers; disorient the Law; let [criticism] discourse float, fragment, digress” (Leitch 1986: 51) from a deconstructive stance. By definition, the study of intermediality tends to replace limit, structure, solidity, order of things with decentralization, fluidity, discontinuity, relation, and process. Perhaps more importunately pressed now than at any previous time, its intermedial or transmedial concerns are some different and even heterodox appreciations of the contemplation and accumulation of new artistic experience in our generation. Overall, what Greenblatt, Jameson, and Leitch are trying to identify here is a common desire of collaborative and cooperative force that associates literature with other arts and nonverbal subjects, a transfer that views things as an intricate, dynamic pattern of confluences and divergences instead of purely single form or bare parallelism. This chapter, therefore, aims to reformulate the theory of intermediality (the popular subcategory of interdisciplinarity) as an overarching platform to scrutinize the multiple, interactive, and transformative relationships among different media and artforms. It intends to answer some preliminary but significant questions: au fond, what is intermediality? How does it function between distinctive modes of media and arts? How does intermediality act as an actual agency devised to facilitate communications among different domains, whether technological, aesthetic, or ideological? What is the historical development of ut pictura poesis, and in its lifelong antagonism toward paragone, how has its evolutionary version of intermediality substituted for the predominant binary system and false dichotomy

342

B. Wang

since Gotthold Ephraim Lessing? From a poststructuralist perspective, how shall we understand and reconsider the theoretical evolution of intermediality from ut pictura poesis? Based on the “political implications” (Hayles and Pressman 2013: xiii) of this media-oriented interdisciplinarity, the intermedial framework goes beyond the transcendentally paragonal nature of comparativity to a poststructuralist sense of in-betweenness. Intermediality is not just concerned with one particular side; it mediates the binaries and interrogates the commingled relationship between the text-based and the audiovisual, form and content, the symbolic/intelligible and the technical/sensual, subject and object, center and periphery, self and other, the discursive and the pictorial, poets and artists, from a democratic equidistance.

The Genealogy of Ut Pictura Poesis Intermediality and its closely related concern about word-image relations are in fact not a product of modernity. As early as Roman antiquity, Horace in his Ars Poetica, taking a lead from the Greek poet Simonides of Ceos’ influential formulation that “painting is mute poetry and poetry a speaking picture” in Auctor ad Herennium (also inscribed in Plutarch’s Moralia), composed his reputed idea of ut pictura poesis, namely “as is painting so is poetry.” Introducing this specific analogy reveals an embryo sense of intermediality. This Horatian simile brought forth the fundamental premise of the kinship of descriptive and iconic modes of art. However, it does not focus overly on the essential dissimilarities of different media but instead emphasizes the structural and functional similitudes between these two artforms. Ut pictura poesis, in the beginning, signifies a casual comparison and a general notion of parallels. Serving as different representational forces, poetry and painting have been considered since Aristotle’s Poetics to be the arts of mimesis, the ideal imitation of physical appearances and the exterior world, thus uniting the poetic and the pictorial as instrumental tools of mimetic representation. According to Aristotle, in spite of their use of distinct categories of medium, poets and painters employ similar elements in their formal compositions: on function, to represent or mimic objects by rhythm and language or color and shape; on subject matter, to imitate humans in action and their moral nature; on standard, to be judged by the vividness or resemblance of reality; and on structure, to treat plot in tragedy and outline/design in picture as equally principal constituents of representation (Aristotle 1997: 45–73). Furthermore, based on the classical idea of imitation and pedagogical theory, and no longer dismissed from the Platonic commonwealth for purveying an inferior and distorted form of the ideal Truth, poets and painters have shared the same purpose: their artistic practices are aimed not only to please but also to instruct. Sir Philip Sidney, apparently influenced by Horace and Aristotle, also conceived of “poesy” as an art of mimesis, “a representing, counterfeiting, or figuring forth – to speak metaphorically, a speaking picture; with this end, to teach and delight” (Sidney 1989: 86). Over and above this, he even took advantage of the metaphorical power of ut pictura poesis to defend poetry against philosophy and history. By providing the “perfect picture” as a painter, the poet can yield “the powers of the mind an image of

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

343

that whereof the philosopher bestoweth but a wordish description” and the historian “the particular truth of things [without] general reason of things” (Sidney 1989: 90). The visible effect and pictorial naturalism of visual art equip a poetic line with painterly techniques to represent the scale, depth, and shape of human experience; they both work reciprocally as an inseparable twin. However, this medial inseparability does not imply the inherent connection of intermediality. Later, John Dryden, in his preface “A Parallel of Poetry and Painting,” exemplified from Sophocles’ Oedipus to Nicolas Poussin’s The Institution of the Eucharist, further elucidated on this essential parallelism between poet and painter in respect of characterization, action, properties, chief end, decorum, and manners of fiction for representing the perfection and deficiency of figures as natural objects in both portrait and comedy or tragedy. In particular, he placed additional emphasis on the artistic invention alongside the Aristotelian theory of imitation as a significant part of their parallel media relations: “Without Invention a Painter is but a Copier, and a Poet but a Plagiary of others. Both are allow’d sometimes to copy and translate” (Dryden 1695: xxxiv). Horatian credo, as the earliest model of intermediality, not only sets a common theme for rhetorical and oratorical discussions, but also, more profoundly, provides a meeting ground for an intermedial exploration of poetry-painting in a series of comparison and contrast. The aphorism of ut pictura poesis became a bequest and the embryo of humanistic theory of arts grew in greatest popularity during the Italian Renaissance, when poetry and painting were first referred to as “sister arts” – albeit not as a harmonious couple but as rivals (Markiewicz and Gabara 1987: 537). The rivalry for precedence between different arts and media became increasingly fierce, especially with the tendency toward medial purification and artistic distinction, clearly demonstrating an attempt to establish ascendancy of one art over the other. At that time, Leonardo da Vinci first escalated the war known as paragone, a medially monotheist war against our notion of intermediality starting from centuries ago. He lifted the painting from its status as craftmanship to an independent art form over poetry; this is the point at which the ranking or classing competition corresponding with the tradition of comparison among media is reckoned to have officially begun, and it soon became a perennial debate throughout Western intellectual and cultural history.2 More clearly, his defense of painting extended the classical parallel in terms of measure and function into the artistic competition struggling for a revised hierarchy. In his treatise on pictorial art, da Vinci proclaimed this sisterly emulation as a Renaissance enterprise and elevated painting to the class of liberal arts – he even placed painting as the highest standard of human knowledge, countering Aristotle’s primacy of tragedy and Michelangelo’s sculpture. Through his statement that the painter was the “lord of all types of people and of all things” (Da Vinci 2008: 185), da Vinci 2

This paragonal competition, in the twentieth century, includes not only highbrow classical art forms but also lowbrow mass media and popular culture, such as television programs, advertisements, broadcasts, comics, and computer games, reinvigorated by a sociological analysis of Pierre Bourdieu’s distinction, a moral and political struggle of classification. See more in Bourdieu, Distinction: A Social Critique of the Judgement of Taste, translated by Richard Nice (Harvard University Press, 1996).

344

B. Wang

explained how the superiority of painting meant that it surpassed all other forms of art: The eye which is called the window of the soul is the chief means whereby the understanding can most fully and abundantly appreciate the infinite works of nature; and the ear is the second, which acquires dignity by hearing of the things the eye has seen [. . .] And if you, O poet, tell a story with your pen, the painter with his brush can tell it more easily, with simpler completeness, and less tedious to follow. If you call painting dumb poetry, the painter may call poetry blind painting. Consider then which is the more grievous defect, to be blind or dumb? Though the poet is as free as the painter in the invention of his fictions his creations do not give so great a satisfaction to men as paintings do; for though poetry attempts to describe forms, actions, and places in words, the painter employs the actual similitude of the forms, in order to reproduce them. (Da Vinci 2008: 190)

Focusing on the prominent faculty of vision, he ranked the eye/seeing first, then the ear/hearing, and thus prioritized painting, dependent on visual quality, over poetry, dependent on auditory. His anti-intermedial theory of aesthetics treated blindness as a more pathetic defect than dumbness; notwithstanding this, it rattled the foundational dictum proclaiming poetry a “speaking picture” but painting a “silent poetry,” as was the presuppositional colonialization of various arts. Relating to new technical problems and the attempt to develop a scientific theory of aesthetics, to a large degree, this subversion broke up the state of muteness in painting and endowed it with a paramount sense of sight. In the same vein, following John Locke’s empiricist notion that sight is “the most comprehensive of all our senses, conveying to our minds the ideas of light and colours, which are peculiar only to that sense; and also the far different ideas of space, figure, and motion” (Locke 1829: 83), Joseph Addison appears to have been close to rephrasing Lockian words into his own style of periodical essay, linking aesthetic tradition to modern psychological science: “Our sight is the most perfect and most delightful of all our senses”; it “fills the mind with the largest variety of ideas” and “furnishes the imagination” (Addison 1970: 397). For both, the eyes, sight, and culture of connoisseurship are the most natural direct sources of human ideas, from which complex abstraction is originated as a result of an association with basic sensual perceptions. The early intermedial discussion gradually expanded its realm from formal categories to a more philosophical concern about how word/ image could affect our ways of feeling and thinking. Based on the epistemological transformation from the Cartesian orientation toward mind, painting and plastic arts were then considered superior to metrical or rhetoric compositions, for the former were assigned the power of visuality and the “primary pleasures of the imagination” (Addison 1970: 398), whilst the latter, censured from Plato’s Socrates, were full of figurative words and literary tropes that could do nothing but “insinuate wrong ideas, move the passions, and thereby mislead the judgment” (Locke 1829: 372).3 Their 3

Socially, it is in Renaissance that painters and sculptors culminated the exodus and institutionally became part of the academies of fine art, which complemented the aesthetic emancipation of visual art in general.

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

345

alleged power of optical perception and the prioritized sense of sight, from Thomas Hobbes to David Hume, asked poetry to produce a scene of images and mental vision that organically integrated the experience of both readers and beholders, which furthermore intensified the paragone and its impossibilities of an intermedial conversation. The paragonal hierarchy of arts has by no means merely been a hierarchy of different media or ways of classification, but rather a contest of physical senses that took place in the Renaissance and even a struggle for dominance between the perceptible and the imaginative, body and mind, the natural and the artificial, and time and space. In the mid-eighteenth century, Lessing, with a fundamentally anti-intermedial work entitled Laocoon: An Essay on the Limits of Painting and Poetry, opposed the Horatian pattern of ut pictura poesis resulting in the confusion of arts. Without any intermedial potential, he argued for medial dualism instead, assuming that “succession in time is the province of the poet, co-existence in space that of the artist” (Lessing 1984: 100), in the manner of compartmentalization of different media. He intended to disassociate them as two radically independent, incompatible modes of representation that operate in each dimension with incommensurable measures and subjects: poetry obeys the temporal rule of sequentiality and arbitrariness, whereas painting sticks to the spatial law of simultaneity and likeliness. To achieve this, Lessing demanded a critical reassessment of the structural border and instituted strict limits for the irreducible differences of bed and board between the traditionally perceived sister arts on his “first principles”: If it be true that painting employs wholly different signs or means of imitation from poetry, – the one using forms and colours in space, the other articulate sounds in time, – and if signs must unquestionably stand in convenient relation with the thing signified, then signs arranged side by side can represent only objects existing side by side, or whose parts so exist, while consecutive signs can express only objects which succeed each other, or whose parts succeed each other, in time. (Lessing 1984: 91)

Laocoon aims toward a deviation of the parallel line from the commonplace direction of neoclassical convention in order to redefine the relationship between verbal and visual representations by assigning generic boundaries to both. Retrospectively, Lessing might be the opposition leader to vote against the establishment of intermedial studies. Rather than call pictura-poesis sisterhood, he prefers to see poetry and painting as “two just and friendly neighbors, neither of whom indeed is allowed to take unseemly liberties in the heart of the other domain, but who exercise mutual forbearance on the borders, and effect a peaceful settlement for all the petty encroachments which circumstance may compel either to make in haste on the rights of the other” (Lessing 1984: 110). Lessing was not the first to build up the border in between, but unapologetically the most important critic to dualize pictorial and poetical arts into mutually exclusive, polemical constructs. His oppositional binarism did not manage to distinguish

346

B. Wang

one from another only within the inside domains of genre or sense, but developed into a broad binary system of historical patriarchy:4 Painting Space Natural signs Narrow sphere Imitation Body External Silent Beauty Eye Feminine

Poetry Time Arbitrary (man-made) signs Infinite range Expression Mind Internal Eloquent Sublimity Ear Masculine (Mitchell 1987: 110)

Like female nudity on canvas, the painting as a motionless body is presupposed to lack eloquent power and be femininely mute under the all-pervasive masculine gaze, confined within a narrow sphere in contrast with the poetic infiniteness and its domination of discourse, expression, and history. In this way, Lessing and his contemporary Edmund Burke, who rendered the word sublime and the image beautiful – for the most “lively and spirited verbal description” can “raise a stronger emotion” than the best picture (Burke 1998: 55) – ranked poetry above painting as a new aesthetic cornerstone in the Enlightenment, with cognition above perception, imagination above memory, and fiction above verisimilitude. In this anti-pictorial stain and the sanctification of word, poetry was regarded as disconnected with the representational potency to form sensible images but empowered to affect through a spiritual power of sympathy, imagination, and sublimity.5 The intermedial relationship between poetry and painting, at this time, was no longer a purely formal or theoretical theme but instead a political, psychological, and ideological issue, which, like that of “countries, of clans, of neighbors, of members of the same family,” was not only “subject to versions of [their] laws, taboos, and rituals” (Mitchell 1987: 112) but also capable of changing and regulating the sociocultural forms of our life. Neither neoclassical paragone, debating the relative value of sister arts, nor Lessing’s dichotomy between time-based poetry and space-based painting would prove persuasive any longer. These monomedial viewpoints gradually lost their fascination as the borderline between each medium was dissolved by the interart influences and exchanges active during the nineteenth and early twentieth centuries. Instead of a laocoonized underpinning, artists noticed an interrelatedness and 4

Mitchell’s another list of dualistic conceptions between word and image comes alongside his analysis of Blake’s composite artistry of book (poetry) and scroll (painting) emblematically as an allegory of good and evil: book serves as mechanical/reason/judgement/law/modern/science, while scroll as handcrafted/imagination/forgiveness/prophecy/ancient/art. See his Picture Theory: Essays on Verbal and Visual Representation (Chicago: University of Chicago Press, 1995). 5 Denis Diderot furthers this differentiation between poetry and painting, see more in Markiewicz and Gabara (1987: 541).

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

347

productive interchange between different media or artforms in the late romantic and modernist periods. Friedrich Schiller, working in varying roles as a lyric poet, philosopher, historian, and playwright, contended against the purified existence of art within an aligned generic category, which, in his view, had even become a historical hindrance to achieving the fullest nature of one’s aesthetic experience. The “perfect style in each and every art” should “remove the specific limitations of the art in question without thereby destroying its specific qualities, and through a wise use of its individual peculiarities, is able to confer upon it a more general character” (Schiller 2004: 470). In a strong sense of multimediality, one of Schiller’s Letters on the Aesthetic Education of Man sets out his utopian blueprint of a unified system of various arts to replace the traditional separateness that had been predominated by mimetic theory; the aim was to overcome the objective frontiers between distinct medial specificities and realize true aesthetic perfection as the ultimate mission of art – not as an instrumental skill but as an autonomous expression of psyche and morality. Later on, in a proto-intermedial study on Renaissance literature and art, Walter Pater blurred Lessing’s neat line of demarcation between word and image by referring to “The School of Giorgione”: [A]lthough each art has thus its own specific order of impressions, and an untranslatable charm, while a just apprehension of the ultimate differences of the arts is the beginning of aesthetic criticism; yet it is noticeable that, in its special mode of handling its given material, each art may be observed to pass into the condition of some other art, by what German critics term as Anders-streben [or Richard Wagner’s multimedia term Gesamtkunstwerk] – a partial alienation from its own limitations, through which the arts are able, not indeed to supply the place to each other, but reciprocally to lend each other new forces. (Pater 1980: 105)

Both Pater and Wagner, as late Romantics, underlined the concept of a total work of art – anders-streben or Gesamtkunstwerk – as a pre-modernized version of intermediality for synthesizing various modes of forms into a universal, all-embracing artwork.6 The relational lines between poetry and painting here ceased to be paralleled or diverging; they began to lend reciprocal support to each other. However, the joining force that reunited them did not rely upon the common duty of representing external reality but instead on the romanticist theory of expressiveness. In the mid-nineteenth century, John Ruskin reformulated the idea of ut pictura poesis by claiming that painting “is properly to be opposed to speaking or writing but not to poetry [for both] painting and speaking are methods of expression” (Ruskin 1900: 12–13). Since the trends of Kant and Samuel Coleridge, the purpose of art and literature has undergone a crucial change in social and cultural aspects – no longer to copy or imitate but instead to make and convey. Poetry and painting should not be considered as facsimile representations of human action or natural substance; they shall belong to their own sphere of invention and expression, where poets and

See Wagner, “The Art-Work of the Future” and Other Works, translated by William Ashton Ellis (Lincoln: University of Nebraska Press, 1994).

6

348

B. Wang

painters are not copiers or transcribers but a group of original geniuses brimming with emotions, visionary conceptions, and spontaneous overflow of feelings. From mimesis to romantic expressionism, an updated version of ut pictura poesis shifts the cultural debates with new paradigms in order to reverse the mimetic mirror and thus reflect the expressive lamp of one’s inner mind. The multimedia notion of Gesamtkunstwerk and its function as a force of unification have boosted a growing investigation into conceptualizing the interface between the linguistic and the pictorial. It thereby demonstrates the revolutionary potential for the introduction to intermediality, a philosophically and epistemologically systematic extension of ut pictura poesis, in greater depth.

Literary Intermediality out of Interdisciplinarity As archaeologically investigated, intermediality shall be regarded as a historical experience learning from the paragonal conflicts between pictura and poesis, and works as a fair place of meeting for individual artforms with no rank or class, where one might travel across from one defining category smoothly to another. In Death of a Discipline, Gayatri Chakravorty Spivak has put forward the idea of “transgressive readings” (Spivak 2003: 56), which build upon the weight of the alterity, the communication with the other, in an opening-up process, for the reason that To be human is to be intended toward the other. We provide for ourselves transcendental figurations of what we think is the origin of this animating gift: mother, nation, god, nature. These are names of alterity [. . .] And thus to think of [alterity] is already to transgress, for, in spite of our forays into what we metaphorise, differently, as outer and inner space, what is above and beyond our reach is not continuous with us as it is not, indeed, specifically discontinuous. We must persistently educate ourselves into this peculiar mindset. (Spivak 2003: 73)

This “peculiar mindset” of transgression helps to restore the disconnected as a redemption of continuity. Intermediality, as one of the transgressive readings of interdisciplinarity, shifts its focus from expanding to bridging, from the polar to the multilateral, from the centralization of the monopoly to a relational link on each side. Encouraging a more subtle and sensitive reading of new compositions and identities, the intermedial method pays more conscious attention to the in-between relationship, the movement across regions, and the processes of cultural circulation and hybridization. Capturing the transcendental relation is the axis of its principle. From interdisciplinarity to intermediality, we must be wise to both the importance and the trouble of integrating different views and sources, and constantly alert to the blurring of genres that shall never purely be an absorption, a rearrangement of property, or a redrawing of dividing lines. Intermediality can be regarded as a branch or subdivision of interdisciplinarity, which – in its narrow sense – occurs only in the fields of literature, history of art, architecture, cinematic studies, musicology, dramatic and theatrical research, aesthetics, and media studies. The medial relationship is perceived as different and unhappily

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

349

divided by traditional approaches to the sisterhood of text and image, for example, the study of their translatability, correspondence, ekphrasis, thematology, adaptation, and influence and reception from one side to the other (Weisstein 1973: vii). This brings the comparative tradition into focus in order to examine the two sides’ relationship as competitors and rivals, or as the borrower and lender fixed in their own spaces. Spivak cautions us against these stereotypes: such a view is “antihybridist, culturally conservative, ‘ontopologist’ [Derrida’s term]” and “parochial” (Spivak 2000: 21, 2003: 9). In contrast, the new and vast field of intermediality replaces old ideas with a more wideranging interchange of boundaries between different disciplines and cultures. Emphasizing the notions of crossing, mutualizing, the kindred, and the hybrid, intermediality is not only a newborn artistic phenomenon but also an innovative methodology and radical philosophy or way of (re)thinking. As a vital means of removing the long-standing barriers between media and asking for a revaluation of their productive exchange, intermedial studies has been defined in a plethora of different ways: a “bridge between medial differences” (Elleström 2010: 12); “media border-crossings and hybridization” with a “heightened awareness of the materiality and mediality” of cultural and artistic productions (Rajewsky 2005: 44, 2010: 51–52); “links” and “cross-breeds” between various forms of art (Herzogenrath 2012: 2); a desire to fuse or a “conceptual fusion” of scenario and audiovisuality (Higgins 1984: 30); a “relationship between (inter) texts or medialities” with a huge range of cultural practices (Bruhn 2006: 14); a “complex and highly dynamic set” of relations among different media (Emden and Rippl 2010: 10); a “transgression of boundaries between media” whose objects shall be “semiotic complexes or entities” (Wolf 2005: 252); “artistic interrelatedness,” interrelations, or the interconnectedness within each art happening in the contemporary medial age (Eilittä 2012: vii; Clüver 2007: 20; Hallet 2015: 605); and the acquisition of “a plurality of identities” through continual translation from one medium to another (Punzi 2007: 10). Though these have varying terminologies and registers, they unanimously stress the concepts of association – “bridge,” “links,” “relationship,” “interrelatedness” – of hybrid – “fusion,” “hybridization,” “plurality,” “dynamic” – of destruction – “border-crossings,” “transgression,” “cross-breeds,” “transformative[ity]” – all signifying an intrinsically revolutionary change for literary and artistic interpretation. Accordingly, the subjects of intermediality are less concerned with new types or phenomena than with altered ways of seeing, spaces for possibilities, diversification of genres, and eventually a new perspective that escapes the confines of limited dimensionality. This shall be a vantage point out of thoughts and behaviors assigned by a single dimension; it will instead be set in motion and negotiation at all times. Hence, this new concept of intermediality appears more open-minded and variable, substituting the narrowed vision historically and ideologically trapped in its own borders with dynamic energy and critical potentialities for problem-solving. Aside from intermediality, a broad spectrum of terminologies is also used to describe the frequent contact between literature, art, and other nonverbal aspects, for

350

B. Wang

example, interart poetics, textual/visual studies, intertextuality, sister arts paradigm, comparative literature, inter-aesthetic research, multimodality, and so forth.7 Although it might be hard to tell which phrases and definitions are authentic or solely legitimate, the common nature of their naming and the conceptualization of intermediality refers to an emphasis on the process, the linkage, and the very position sitting in between, characterized by a simultaneously collaborating and framebreaking force. It benefits from uniting these two contradictory yet interactive features. Learning from Greenblatt’s cultural poetics, the intermedial approach synthesizes aesthetic-oriented studies with media-historical ones (Johansson and Petersson 2018: 2) to bridge these traditionally disconnected areas. As a “travelling concept” (Bal 2002: 22) that could travel among different disciplines, media, and cultures, intermediality seeks to combine formal, semiotic, iconographical analysis with historical, social, ideological interpretation in order to fuse literary and artistic experience with the specificity of their practices across time, place, and sociocultural environments. Etymologically, the prefix “inter-” is derived from the word “intermediate”; its Latin origin, intermedius, means being situated in the middle or between two things. Moreover, “inter-” is an action that bridges and connects two separate domains. It does not restrict the object to scientifically categorized or specialized disciplines in its conventional code, but rather defies pure structure and explores contingencies and indeterminacies when moving between and crossing over any border that is in place. Transposition, combination, reference, and remediation are typical and preferable ways to describe the intermedial practices of two or more modes of articulation. They respectively but comparably elicit a transformative, evolutionary, referential, and refashioning process from the substratum of the sourced medium to the targeted as a new constitution (Rajewsky 2005: 52–53). Never thematically or structurally in isolation, all intermedial forms occur in a constant dialectic with each other in a meticulous production of the tactile, visual, olfactory, and acoustic copresence. In this way, intermediality is responsible for three major roles: collaboration involving interrelation between two art forms; frame-breaking in a cross-disciplinary, crossmedia field; and epistemologically, on top of the previous two roles, creation of a new way of reading/seeing so as to sidestep binary oppositions, open up bright prospects, and foster the expansion of perspectives within a richly multidimensional tapestry.

7

On a more elaborate and clearer differentiation of these terms: interart studies and intermediality studies, see Bruhn, “Heteromediality” in Elleström 2010: 225–226; interart studies, studies of intermediality or intermedial studies, see Clüver 2007: 20–22, and his new article “From the ‘Mutual Illumination of the Arts’ to ‘Studies of Intermediality’,” International Journal of Semiotics and Visual Rhetoric, 3.2 (2019): 63–74; intermediality and mixed media (multimediality), see Schröter 2012: 19–20; intermediality, intertextuality, and interartiality, see Müller, “Intermediality Revisited: Some Reflections about Basic Principles of the Axe de pertinence” in Elleström 2010: 244–245; intermediality, inter-aesthetic research and interart studies, see Bruhn 2006: 13–14; intermediality, multimodality, and visual cultural studies, see Rippl 2015: 16–21; intermediality and multimodality, see Hallet 2015: 605–607.

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

351

First, instead of existing in a relationship of mutual plagiarism, it captures one medium by means of locating it with another, in order to analyze and interpret the text or image through the dynamics of the intermedial process. In contrast to multimediality, transmediality, and plurimediality, which underline mixedness, juxtaposition, and monomedial purity accumulated in a linear or cyclic arrangement, intermediality is strongly underpinned by the genetic heterogeneity of media and the mediation between them – an always-mediating relation between “subject and object, technical and symbolic, sensual and intelligible” (Herzogenrath 2012: 4). On the premise that “all media are mixed media” (Mitchell 2005: 260) in terms of sensory and symbolic elements, the intermedial approach ontologically considers every medium, a romantic lyric or a portraiture, not to exist in isolation but rather in a Kantian sense, transcendentally in relation to other media before the existence of its pure pattern. Similar to Immanuel Kant’s perplexed notion of things-in-themselves that delineates a subjectively universal foundation of forms or categories named by pure reason preexisting in humans’ mental process, the idea of media is viewed by intermediality as “relations-in-between,” having been mixed from the outset. Media are thus deemed “relational constellations” and “situational incidences” (Rippl 2015: 17) rather than fixed or purified. No detached observation is possible when the total experience is instantaneously involved across the frame. As such, it gives us the opportunity to range across the triangulated divide of society/technology/aesthetics (the historical/constitutive/empirical) and integrate different objects and theoretical frames that were previously thought incompatible. In the formalist manner of defamiliarization and estrangement, it allows for an interactive reading and a plurality of meaning-making from the intermediate standpoint between distinct means of communication. The second motivation of intermediality, based on the essential impurity and hybridity of different artistic formations, is to dissolve and transgress the borders between the established scientific landscape and move beyond the underlying institutional restrictions. Its critical awareness of fluidity and interchangeability between different representations could promote an interdisciplinary strategy, contesting the clearly separated monomediality or disciplinarity that should be the result of “purposeful[ly] and institutionally caused blockades, incisions, and mechanisms of exclusion” (Schröter 2012: 30). Despite the generic function of framings as scripts or schemata of certain expectations (Wolf 2014: 127), frames or clear-cut borderlines in current disciplines and media act more like stereotypes, a constructed system of hierarchy that needs to be abolished.8 Intermediality, following Marshall McLuhan’s dictum that all media appear as “extensions of man” (McLuhan 1965: 3) within a much wider mixed cultural life, works to solve the problem of words and images. The problem does not solely refer to “a matter of definitions of essences and separation of practices, but of how people communicate: with one another, with the past, with others” (Bal 1999: 169). The function of frame-breaking, especially in its postmodernist sense, is to find the alternative frame – Wolf’s dynamic “framing” 8

See Wolf (2006: 21–27), for the differentiation between framing and framing border.

352

B. Wang

(Wolf 2006: 10), or the metaleptic violation of the cognitive storage system where the conventional order of narrative becomes empowered. As an indispensable part of interdisciplinary enterprise, the study of intermediality, provides a dynamic and cooperative alternative to the constructed disciplinary nature of subjects and art forms alike, thereby forcing us to reconsider concepts such as knowledge, authority, truth, value, superiority, and legitimacy of disciplines, which we have so far taken for granted. It not only denies the infrastructural departmentalization of the academy, but also cuts down the difference between present and past, other and self, object and subject, the spatial and the temporal, formal analysis and historical discussion. It starts to intersect the comparative approach, which views things as disparately attached in their pure forms, with the intermedial approach, which concerns media particularities and the productive relations between them. From the perspective of intermediality, re-sourced and revivified, we see borders as links. In addition to the power of interconnecting and destabilizing, intermedial studies encodes a perceptual and semiotic shift from an inflexible, enclosed mindset to what Spivak praises as transgressive reading: an interwoven, all-inclusive network wherein lies the potential for a deeper and more comprehensive understanding. The expansion of understanding here is grounded on a similar sense to Francis Bacon’s new torch, but in contrast to his rationalist quest for reason and scientific positivism in the dark days of the Middle Ages, it now requires a renewal and regeneration of the vantage, a constantly negotiating viewpoint, and a stereoscopic vision composed of distinct sources in an “unflattening” kingdom. It is interesting to note this utopia of the “unflattening,” which means a “simultaneous engagement of multiple vantage points from which to engender new ways of seeing” (Sousanis 2015: 32). It does not issue step-by-step instructions but serves “as an attitude – a means of orientation – a multidimensional compass, to help us find our way beyond the confines of ‘how it is,’ and seek out new ways of being in directions not only northwards and upwards, but outwards, inwards, and in dimensions not yet within our imagination” (Sousanis 2015: 46).9 Thus, the last but principal role of intermediality can be defined as the poetics of unflattening that synthesizes reading with seeing in a dynamic process: the verbal is not just a linearly discrete arrangement of words, and the visual does not function only as a static plane; they overlap, interpenetrate, and generate an open-up ecosystem. Intermediality is not concerned with the traditional duality of time and space but instead with the double orientation and interactive cycle of the sequential and the simultaneous. They are different but inextricable media, modes of expression, and tools of thought, relaying an in-between meaning across the disciplinary structure that stubbornly seeks a solid “pattern of one-dimensional thought” (Marcuse 2007: 14). 9

Nick Sousanis’ Unflattening in 2015, his doctoral dissertation in Education at Columbia University, is the first comics-styled dissertation inextricably linking academic discourse with graphic presentation to explore how the way that people structure knowledge has constructed our ways of thinking. Acting against the flatness of linear viewpoint, his “unflattening” incorporates word and image as new forms of expression to generate a multiplicity of meaning from different vantage points.

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

353

New requests are made on reality at this pictorial or visual turn of our age. The long-lasting singularity of direction and the flatness of horizon may be unsubstantial and invalided out of the highly efficient transmission between the sender and receiver of clusters of images and visual information, which hence need to be reformed or substituted completely. They are not habitual categories within their original frame, but webs of a rhizomatic tissue of connections, exposed to us as an “ongoing process, a flowing interaction [with] their structures dissolving” (Hughes 2013: 81–82). Intermediality, out of interdisciplinarity, is an alternative “perspective, a Weltanschauung” of rewriting modernity (Lyotard 1991: 29) that offers us a new engagement with a sonnet or a still life, experience between readership and spectatorship, and distribution between perception and cognition. If media, now and everywhere, are the “environment” for living and infrastructures of life (Mitchell and Hansen 2010: xiii), intermedial studies shall by no means be an overhaul of technical entities but rather a new lifestyle and ecologically healthier attitude toward the human soul and the outside world.

Poststructuralist Perspective: The Philosophy of In-Betweenness Intermediality has become a ubiquitous buzzword since the 1980s and 1990s; it was rooted in the German academic environment, soon prevailed in Amsterdam, Scandinavia, and other European countries, and eventually, in this century, developed a mature global reputation throughout contemporary English-speaking research communities.10 Its initial development has been seen as a methodological expansion of the idea of intertextuality in the late 1960s and the traditional interart studies from antiquity. Precisely because this new paradigm conceives of “both text and image are media” as a precondition, the segregation between literature and other forms of art has come to an end, and the scholarship has even moved on beyond conventional interart (high art) in collaboration with mass media and popular culture (low art). We should be alert to the monomedial imperialism and hegemonic impulses of language or nonverbal forms of art, echoing Lessing’s aesthetic politics of the contest for domination over different media and subjects, aiming to colonize and reduce the native of the other. Instead of marginalizing writing/printing or picture/visuality, or placing a single emphasis on one side, intermediality treats them not as two disjointed axes but as a constantly ongoing bilateral negotiation, from an exceptional point of view that stands in between. It was Ferdinand de Saussure’s structuralist linguistics that for the first time read language as a sign with two surfaces and thus literature as a signifying system in which meanings are produced. Like Gérard Genette’s narratology, the Saussurean system of language does not recognize the literary text as a “message” containing The first officially documented usage of this term “intermediality” was from German scholar Aage A. Hansen-Löve when demonstrating the relationship between literature and visual art in Russian symbolism.

10

354

B. Wang

some amount of distorted information about authorial intention or social context, but rather as a “code without a message” (Genette 1988: 137). In a similar manner, treating language as sign, code, or at any rate a medium, the intermedial constitution also emphasizes that the realization of one given medium depends on “what the others are not” (De Saussure 2011: 117) and on an interrelationship with each other where they take up different positions in the common system. However differently, intermediality extends the scope from text alone to image and other nonverbal expressions within its discussion; moreover, unsatisfied with the colonialization of semiotics and post-Saussurean logocentrism, it allows us to start thinking about the feasibility of applying the linguistic model to all artistic and cultural constellations. At this point, intermedial theorists critique Saussure and his followers for exclusively focusing on “the content, the signifié [signified] or cognitive side while neglecting the material signifiant [signifier]-side” (Rippl 2015: 8), and, moving in a new direction, appeal to the important place of the signifier in terms of not only the rhetoric or aesthetic autonomy but also the moment of addressing between the sender and the receiver, consciously or unconsciously, transgressing the boundary of varied media. As N. Katherine Hayles has called for in her book Writing Machines, a “media-specific analysis” of print and digital writing shall take the “materiality of the medium” into close consideration (Hayles 2002: 28). This is why the form or structure, the problematics of representation, the texture of media (both word and image), and the way this “material significant-side” of the semiotic system becomes involved in the generation of narrative and symbolic meaning can nowadays catch critical attention from intermedial research. As such, structuralist presuppositions that the world is ontologically grounded on a structural system of language can no longer be insurmountable. Nelson Goodman’s language or grammar of art and Ernest Gombrich’s pictorial psychology both challenged Susanne Langer’s differentiation between discursiveness and visuality based on her assertion that “the laws that govern this sort of articulation [visual forms] are altogether different from the laws of syntax that govern language” (Langer 1942: 81). Concerned with the text-image difference, Langer’s gender politics reveal a consistently imbalanced relationship between word and image by viewing their joining as a “successful rape” (Langer 1957: 86), pillage, tragic marriage, or even bastardization of any one kind of art. On the other hand, compared to Langer’s essentialism, Goodman and Gombrich argued for the “conventionalist view” (Wang 2021: 81) that nonlinguistic symbols can be also seen as codes operating in a conventional system, whether grounded on a preexisting sociocultural entity or the transcendental knowledge of human existence, whether a system of symbols or a system of schemata. The intent was to dismantle the fundamental boundary between the discursive and the visible in the Western philosophical tradition that had been in place since Plato’s “Allegory of the Cave.” Alluding to Goodman’s Language of Art, W. J. T. Mitchell, in his collection of essays The Language of Images, treated the iconographic study of image equally to that of language, namely linguistics, in order to facilitate “the semantic, syntactic, communicative power of images to encode messages, tell stories, express ideas and emotions, raise questions, and ‘speak’ to us” (Mitchell 1980: 3). The paradoxically

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

355

dialectical relationship between language and art, which appears distinctive yet comparable, is therefore in urgent need of an unfixed, flexible explanation. As defined, intermediality per se concentrates not on substances but on the process or activities that take place in between; these are not only collaborative and productive, but also transgressive and deconstructive. The core of intermediality is the philosophical concept of in-betweenness, or in its other names such as “betwixt-and-between” (Greenblatt 1997: 15) and “between the between” (Herzogenrath 2012: 2). From a semantic perspective, “inter-” as a prefix exhibits an uncanny sense in its own meaning: “inter-” differentiates, or more specifically, acknowledges the difference and preexisting borderline in between the different parts; however, it also joins and offers a means of exchange and connection. With no established or reinforced frame, it overcomes the difference by “negating the negation” (Bennington 1999: 104) and finds the truth in a self-reflexively dialectical manner. Because it stresses the “inter-” of the existence as being in between the pre-separating media through an incessant process, never with an ultimately fixed shape, intermediality distinguishes itself from multimediality, plurimediality, or transmediality. Although they share the same system of interrelation and a plural identity, intermediality, in a manner distinct from any other contemporary critical jargons, sees a work as always in transit, “continually translated from one medium into another, thus acquiring a plurality of identities, generated as a trace of the movement itself” (Punzi 2007: 10). This reshaping of an in-between identity without a shared community and the blurring between verbal and visual signs will find greater theoretical alignment in the realm of poststructuralism, where theorists give primacy to the signifier over the signified and the interpretative multiplicity of language and meaning. Giving birth to intermediality, intertextuality at the outset was coined by Julia Kristeva to read a text as constructed from a mosaic of relations in a synthesis of Saussurean semiology and Mikhail Bahktin’s dialogism. The inclusiveness of the signifying process has been marked by a necessary “indebtedness to both” modalities (Kristeva 1984: 24), in order to interpret the poetic language both symbolically/literally and semiotically/pictorially. Bakhtin was one of the pioneers critiquing the preconceived schemes of historicist literary criticism, which highlighted the dominant message of the exterior sociocultural context or the psychologically authorial intention behind the work. Discontented with this representational homogeneity of the realistic novel, Bakhtin appealed to a plurality of independent connected voices, namely the “polyphony” of a whole formed by the “interaction of several consciousnesses [and a variety of media], none of which entirely becomes an object for the other” (Bakhtin 1999: 18), thereby constructing a multileveled world and destroying the ordinary narrative in a generic monotone. From the fundamentally monologic/ homophonic to the polyphonic/intermedial, an ultimate dialogicality within a carnivalistic effect unfolds a dialogic relationship between different characters, styles, genres, and even media; this dialectic relationship is an almost “universal phenomenon, permeating all human speech and all relationships and manifestations of human life – in general, everything that has meaning and significance” (Bakhtin 1999: 40), both verbally and visually. Based on Bakhtin’s theory, Kristeva’s official

356

B. Wang

introduction to her notion intertextualité (intertextuality) in 1966, further broke through the censorship of a structured mind and freed the open-ended play of dialogism between in-text and extra-text, between the textual and the non-textual into a cross-border networking of intermediality, to replace the static hewing of texts with a model where literary structure does not simply exist but is generated in relation to another structure. What allows a dynamic dimension to structuralism is his conception of the “literary word” as an intersection of textual surfaces rather than a point (a fixed meaning), as a dialogue among several writings: that of the writer, the addressee (or the character) and the contemporary or earlier cultural context. (Kristeva 1986: 35–36)

Neither a motionless point nor a fixed identity, Kristeva’s intertextuality – or its updated version of intermediality – turns out to be a relation with another, an in-between process, and an intersection of different genres through a reordering. The interplay of the symbolic and the semiotic, through a dynamic transfer from the semiotic chora to the genotext as a pre-symbolic space, negates the subject’s unity and immediate certainty, and articulates the blurring, fluidity, and subversiveness of poetic and pictorial representations.11 Roland Barthes never used the term intermediality in public, but his evidently intermedial title of Image-Music-Text does reveal a strong sense of expanding the scope of intertextuality from the literary arena to the imaginal and musical, as well as deciphering literary works and paintings along with photographs, musical compositions, posters, advertisements, television programs, comics, films, etc. Under consideration within a general study of culture, they can all be seen as an off-centered “intertext” circulating in between or a capitalized “Text” without closure (Barthes 1975: 36, 1991: 153).12 This media-oriented interdisciplinarity, in Barthes’ view, happens when “the solidarity of the old disciplines breaks down – perhaps even violently, via the jolts of fashion – in the interests of a new object and a new language neither of which has a place in the field of the sciences that were brought peacefully together, this unease in classification being precisely the point from which it is possible to diagnose a certain mutation” (Barthes 1977: 155). He pointed out that these disciplines or media cannot be isolated in their phenomenal existence; they can only be connected in a multiple, dialectical way without any determination of one class. His deliberate neutrality toward different branches of knowledge was beneficial for grasping his new informational totality of word and image, which was either historically separated (reaching a climax with Lessing’s binarism) or (from the 11

To understand more about Kristeva’s psychoanalytic terms, e.g., the semiotic/symbolic, chora, genotext, and phenotext, see several sections in her Revolution in Poetic Language, “The Phenomenological Subject of Enunciation” 21–24, “The Semiotic Chora Ordering the Drives” 25–30, “Genotext and Phenotext” 86–89. 12 More about Barthes’ theories of text, intertext, and the capitalized “Text,” see chapters in his Image-Music-Text, “The Death of the Author” 142–48 and “From Work to Text” 155–64; “Intertext” in The Pleasure of the Text 35–36; “Masson’s Semiography” in The Responsibility of Forms 153–56).

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

357

standpoint of structuralist linguistics) tended to homogenize the visible into a totalitarian semiotic system of language.13 Through discussing the case of a photograph and its photographic message in the twentieth century, Barthes worked out a significant reversal between denotation/representation and connotation/figuration: the image no longer illustrated the text, and instead, words had come to realize the picture. “Formerly, there was reduction from text to image; today, there is amplification from the one to the other” (Barthes 1977: 26). Exemplified in André Masson’s mid-1950s composite artistry of his surrealist painting and Chinese ideogram, especially with regard to the philosophy of Zen Buddhism, Barthes made a sample of how to do an intermedial analysis. He spoke highly of the hybrid form that he termed “semiography,” intertextually circulating between both Texts of word and image. This circulation of varied media, indeed, constitutes this distinctive characteristic: [I]t is the circulation of the “arts” (or elsewhere: of the sciences) which produced movement: “painting” here opens the way to “literature,” for its seems to have postulated a new object ahead of itself, the Text, which decisively invalidates the separation of the “arts.” (Barthes 1991: 153)

Under this circumstance, Masson’s calligraphical forms of art provide a good example to help us realize the intermedial relatedness and cross-fertilization of writing and painting: Masson’s work during this period demonstrates that the identity of written and painted features is not contingent, marginal, baroque (obvious only in calligraphy – a practice, moreover, virtually unknow to our civilisation), but somehow persisted in, obsessive, including both the origin and the perpetual present of any drawn line: there is a unique practice, extensive to any functionalisation, which is that of an undifferentiated “graphism.” (Barthes 1991: 154)

The visual and the verbal represent two distinct instruments of culture, technology, and aesthetics, enabling us to understand our history and social infrastructures as fundamental to human existence. Their reciprocal relation explores a utopian union of image-text as undermining forces against the existing departmentalization and the hegemony of logocentrism. It is via the communication of intermediality between these two epistemological modes, disparate yet cooperative, that a multiplicity of reading/seeing is achieved and an inclusively cultural space, which is open, unlimited, without hierarchy or partition, comes into being. From Barthes’ Text and textuality to media and mediation, hence, Masson’s typical “true intertextuality” (Barthes 1991: 153) would be better renamed as intermediality in an increasingly pictorial-driven epoch.

13

On the struggle between the linguistic and the pictorial for a unified system of semiology as a science of signs, see Barthes’ ethical and ideological concern in the essay “Is Painting a Language?” collected in The Responsibility of Forms 149–52.

358

B. Wang

The intermedial relationship of language to painting, as Michel Foucault stated, is “an infinite relation” (Foucault 1989: 10) with its latent illumination of the clash between discursivity and visibility. “It is not that words are imperfect, or that, when confronted by the visible, they prove insuperably inadequate. Neither can be reduced to the other’s terms: it is in vain that we say what we see; what we see never resides in what we say” (Foucault 1989: 10). Contesting the reducibility of the visual to the textual or linguistic, Foucault recognized the heterogeneous nature of seeing and saying as points of difference. On a basis of these different productive spaces between the visible (seeable) and the discursive (thinkable), the in-between distance or process of differentiation transcends the old power dynamics of discourse and vision and searches for a new formulation of representation. This intermedial distance could be “read, appreciated, deciphered” (Foucault 2011: 42) in the redoubling of pictorial and discursive spaces. Hence, the generation of meaning and interpretative contingency derives from the heterotopian “space where one speaks to the space where one looks; in other words, to fold one over the other as though they were equivalents” (Foucault 1989: 10), and namely from the exchange between the spaces of the seeable and the sayable, of thought and speech. Analogous to Diego Velázquez’s Las Meninas and Édouard Manet’s Bar at the Folies-Bergère, infinite relations or dynamics of conscientious change illustrate a new episteme for viewing the complex relationships among the artist (Manet, Velázquez and his artistfigure in painting), the subject-model (the waitress and King Philip IV’s family), the spectator (us), and the work of art (the mirrored image and the meta-panel therein).14 Foucault’s archaeological study on the rift between text and vision, content and expression, the invisible and the represented, escapes from both narrow separatism and false universalism and asks for new emancipatory possibilities for the scopic regime of modernity. Instead of a fixed gaze on one side, the flexible, ever-changing point of view within a sense of Nietzschean perspectivism constitutes an alternative to the established position and foundation of resemblance for realistic or empirical representations. Its reciprocal visibility from every side embraces a “whole complex network of uncertainties, exchanges, and feints” (Foucault 1989: 5), where there appears to be an intersection between the plastic and the linguistic. Thus, the intermedial episteme replaces the word/image subordination in the tradition of Western art with the relational infinity in the “midst of this dispersion” (Foucault 1989: 18) – spreading out the frame of canvas and crossing the law of genre. The idea of dispersion echoes Jacques Derrida’s différance, which subverted the paragonal hierarchy between poetry (the written) and painting (the graphic) through his disbelief in the predominant phonologism and logocentric stance prioritizing the trace of the signifier, the visible sign over the verbal discourse, in a deconstructive effect of dissemination. The temporospatial concept of différance designates the ontotheological production of both “differing/deferring” during the development of

On Foucault’s elaborate analysis of these two unorthodox, innovational paintings, see “The Place of the Viewer” in Manet and the Object of Painting 73–79 and “Las Meninas” in Order of Things 3– 18.

14

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

359

space/time (Derrida 1997: 23); its attention shifts from the certain identity of the Saussurean signified, the authentic presence of speech praised in Plato’s Phaedrus, to the materiality of the signifier or visible trace of writing.15 In an intermedial manner, the production of meaning and signification through this trace constitutes a refusal to accept any imposed structural constraints and, similarly to Foucault’s power/knowledge relation, foregrounds an open and never-ending process of infinite play of signifiers. Additionally, in his deconstructive reading of Phaedrus, Derrida resorted to the ambivalence and transmutation of pharmakon from a drug to a remedy, a poison to a cure, with an essence that lies in “having no stable essence, no ‘proper’ characteristics, it is not, in any sense (metaphysical, physical, chemical, alchemical) of the word, a substance [but] rather the prior medium in which differentiation in general is produced” (Derrida 1981: 125–126). The complicity of its ambivalent virtues as a movement makes crossed connections between the historically oppositional concepts that prescribe and supply the separated regions. Derrida’s pharmakon is comparable to the notion of intermediality because it also constitutes the medium in which opposites are opposed, the movement and the play that links them among themselves, reverses them or makes one side cross over into the other (soul/body, good/evil, inside/outside, memory/forgetfulness, speech/writing, [and time/ space, poetry/painting, textual/visual in our case]). It is on the basis of this play or movement that the opposites or differences are stopped by Plato. The pharmakon is the movement, the locus, and the play: (the production of) difference. It is the differance of difference. It holds in reserve, in its undecided shadow and vigil, the opposites and the differends that the process of discrimination will come to carve out. Contradictions and pairs of opposites are lifted from the bottom of the diacritical, differing, deferring, reserve. (Derrida 1981: 127)

Decentralizing the phonocentric model of language and its binary oppositions, pharmakon or différance as a pure form of trace is a productive play of any sensible plenitude, allowing the articulation of different modes of signs or media – phonic, visual, or symbolic – “between two orders of expression” (Derrida 1997: 62–63) and deconstructing the underlying metaphysical and conceptual oppositions behind: [I]t covers the motif of the law in general, of generation in the natural and symbolic senses, of birth in the natural and symbolic senses, of the generation difference, sexual difference between the feminine and masculine genre/gender, of the hymen [of inter-] between the two, of a relationless relation between the two, of an identity and difference between the feminine and masculine. (Derrida 1992: 243)

Without the dichotomy of nature and culture, all other binaries in the preexisting phallogocentric system would fall apart and then reach to a negotiation of intermediality. What Derrida emphasized in his reading of deconstruction was indebted to the intermedial philosophy of in-betweenness, an incommensurable and disseminating relation between the two polarized options. It focuses on the movement, locus, playfulness, process of differences, and mixed mediation of

15

There is a deconstructive reading by Derrida, “Plato’s Pharmacy” in Dissemination 61–172.

360

B. Wang

traditional polarity. In this vein, the absence of authority and primacy of the Platonic spoken subject is effectively able to liberate the presence of multiple meaning in a supplementary chain of infinite différance, reserve, hymen, and pharmakon. Influencing and influenced by not only media theories and artistic practices but also continental philosophy, postcolonial culture, feminism, and queer studies, the study of intermediality, in alliance with a poststructuralist tendency, addresses itself to questioning and retheorizing the pivotal claims of a postwar society in the new century, for example, hybridity, relation, identity, alterity, interval, multiplicity, difference, authenticity, displacement, and the temporospatial. Kristeva’s semiosis, Barthes’ inter-Text, Foucault’s episteme, and Derrida’s dissemination all conceptualized a dynamic of relations and a productive process of in-betweenness. The epistemological and political character, by investigating specific media and their mediation, brings fresh blood to the project of intermediality and helps to offer new ways of “contesting the diremptions of modernity – that is, the institution of the autonomous domains of arts, politics and science” and “by rejecting the totalising mediation as well as the lures of immediacy and particularism [. . .] to provide an alternative both to the unification promised in different ways by dialectical systems (the legacy of Hegelianism and orthodox Marxism) or the grand narratives of modernity (of which the Habermasian theory of communicative action is only one instance), and to the dispersal bemoaned or celebrated as the hallmark of the postmodern condition” (Oosterling and Ziarek 2011: 1). To summarize in short, intermediality provides an alternative to universalism, radicalism, grand narrative, isolation, and fixed identity by enabling us to stand from the position of the in-between, transcend a limited point of view, cross the boundaries of stereotyped sections, decenter long-established authorities, link disconnected fragmentation, search for revolutionary potential, and, at its ultimate end, create a dynamic, fluid, and highly interactive community.

Conclusion The sustained and fertile conception of ut pictura poesis represents not so much a formalist group of thoughts or practices on verbal and visual representation, but rather a talking point for aesthetic and philosophical preoccupations about word and image as basic modes of humanistic intellectual thinking. It positions itself at the interdisciplinary confluence of not only literature and art, but also, to a large extent, rhetoric and aesthetics, epistemology and philosophy, and psychology and other related social sciences. The broad historical span of different critics and their ideas allow for a careful chronological examination on the evolution of the text-image relationship, and is particularly attentive to its social, political, and ideological contexts. As shown in the first archaeological part of this chapter, the sister arts tradition started with a discussion about relation or relatedness between different types of media being like pointing out the physical similarities in a family photograph. The division between arts that arose in the Renaissance witnessed the development of paragone as a cultural phase, where the enterprise of differentiation

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

361

into categories or classes fostered the purification and distinction of media. The transformation from a structural parallel that stressed kinship and harmony to a paragonal debate that emphasized difference and exceptionality, soon featuring the extreme delimitation of temporal and spatial arts, deepened the formal or empirical concern for philosophical and aesthetic fundamentals in the implications of a chain of binary oppositions. To sidestep the dualism and dichotomous thinking directed by Lessing’s exceptionalism of each form, pre-modernist theorists continued to focus on these concerns by examining the new nexus of technology and society and turned toward a union of arts and media based on their creative partnership and crossfertilized affinity, creating a straighter engagement with Horace’s dogma. The twentieth-century tendency toward intermedial conditions constituted an interaction or reunification given that the era of specializing and classification had lost the huge influence it had previously held. In the Hegelian spirit of dialectics, intermediality could be understood as a synthesis unifying classical parallelism and its oppositional binarism, the positive concept of comparison as thesis and its negation of contrast as antithesis. The essence of intermediality, again, does not lie in a simple combination of preexisting formulations but works as a more flexible, dynamic, and dialectical process in-between historically paralleled or separated domains. Its sublation of division and boundaries, as a utopian proposition, concretizes the Marxist ideal in order to overcome the distribution of labor force and its specialization into rigid statuses in a classless communist society. In this light, intermedial reconstructions critique the convention of treating an artwork derogatively, as a pure single good in the circulation of global capitalism, and spur it to break free from the authority of alienation and homogenization caused by the rituals of the bourgeois society.16 The subversive relationship between language and imagery is the negotiation or reconciliation of the other, the opposite self. Intermediality, on a dialectical and dialogical basis, does not assume an actual juxtaposition of literary pictorialism or pictorial literariness, charging its partner with unspeakable or invisible deficiencies of dissonance; it instead brings mediating and communicating into specific focus, which encodes the collaboration between poets and painters within their lines on the page and the canvas with reciprocities, tensions, and iconoclastic powers of aesthetic modernity.

References Addison, Joseph. 1970. Addison and Steele: Selections from the Tatler and the Spectator. New York: Holt, Rinehart & Winston. Aristotle. 1997. Poetics. Trans. George Whalley, ed. John Baxter and Patrick Atherton. Montreal: McGill-Queen’s University Press.

16

To know at more length the political implications of intermediality, peculiarly from the perspective of Marxism, see Jen Schröter, “The Politics of Intermediality,” Film and Media Studies 2 (2010) 107–24; Higgins, “Intermedia” 1984: 21–31.

362

B. Wang

Bakhtin, Mikhail. 1999. Problems of Dostoevsky’s poetics. Trans. and ed. Caryl Emerson. Minneapolis: University of Minnesota Press. Bal, Mieke. 1999. Art, language, thought, and culture: Cultural analysis today. In The contemporary study of culture, ed. Paul Gilroy et al., 169–192. Vienna: Turia + Kant. ———. 2002. Travelling concepts in the humanities: A rough guide. Toronto: University of Toronto Press. Barthes, Roland. 1975. The pleasure of the text. Trans. Richard Miller. New York: Hill & Wang. ———. 1977. Image-music-text. Trans. Stephen Heath. London: Fontana Press. ———. 1991. The responsibility of forms: Critical essays on music, art, and representation. Trans. Richard Howard. Berkeley: University of California Press. Bennington, Geoffrey. 1999. Inter. In Post-theory: New directions in criticism, ed. Martin McQuillan et al., 103–119. Edinburgh: Edinburgh University Press. Bruhn, Jørgen. 2006. The intermediality of narrative literature: Medialities matter. New York: Palgrave Macmillan. Burke, Edmund. 1998. In A philosophical enquiry into the origin of our ideas of the sublime and the beautiful, ed. Adam Phillips. Oxford: Oxford University Press. Clüver, Claus. 2007. Intermediality and interarts studies. In Changing borders: Contemporary positions in intermediality, ed. Jens Arvidson et al., 19–37. Lund: Intermedia Studies Press. Da Vinci, Leonardo. 2008. In Notebooks, ed. Thereza Wells. New York: Oxford University Press. De Saussure, Ferdinand. 2011. Course in general linguistics. Trans. Wade Baskin. New York: Columbia University Press. Derrida, Jacques. 1981. Dissemination. Trans. Barbara Johnson. London: Athlone Press. ———. 1992. Acts of literature. Ed. Derek Attridge. New York: Routledge. ———. 1997. Of grammatology. Trans. Gayatri Chakravorty Spivak. Baltimore: Johns Hopkins University Press. Dryden, John. 1695. Preface of the translator, with a parallel of poetry and painting. In The art of painting, trans. C. A. du Fresnoy, Dryden, i–lvi. London: J. Heptinstall for W. Rogers. Eilittä, Leena. 2012. Introduction: From interdisciplinarity to intermediality. In Intermedial arts: Disrupting, remembering and transforming media, ed. Eilittä et al., viii–xiii. Newcastle upon Tyne: Cambridge Scholars Publishing. Elleström, Lars. 2010. The modalities of media: A model for understanding intermedial relations. In Media borders, multimodality and intermediality, ed. Elleström, 11–48. New York: Palgrave Macmillan. Emden, Christian J., and Gabriele Rippl. 2010. Introduction. Image, text and simulation. In ImageScapes: Studies in intermediality, ed. Emden and Rippl, 1–20. New York: Peter Lang. Fish, Stanley. 1989. Being interdisciplinary is so very hard to do. Profession 89: 15–22. Foucault, Michel. 1989. The order of things: An archaeology of the human sciences. New York: Routledge. ———. 2011. Manet and the object of painting. Trans. Matthew Barr. London: Tate Publishing. Genette, Gérard. 1988. Structuralism and literary criticism. In Twentieth-century literary theory: A reader, ed. K.M. Newton, 135–140. New York: Palgrave Macmillan. Greenblatt, Stephen. 1997. The interart movement. In Interart poetics: Essays on the interrelations of the arts and media, ed. Ulla-Britta Lagerroth et al., 13–15. Amsterdam: Rodopi. ———. 2005. In The Greenblatt reader, ed. Michael Payne. Oxford: Blackwell Publishing. Hallet, Wolfgang. 2015. A methodology of intermediality in literary studies. In Handbook of intermediality: Literature-image-sound-music, ed. Rippl, 605–618. Berlin: Walter de Gruyter. Hayles, N. Katherine. 2002. Writing machines. Cambridge: MIT Press. Hayles, N. Katherine, and Jessica Pressman. 2013. Introduction. Making, critique: A media framework. In Comparative textual media: Transforming the humanities in the postprint era, ed. Hayles and Pressman, vii–xxxiii. Minneapolis: University of Minnesota Press. Herzogenrath, Bernd. 2012. Travels in intermedia[lity]: An introduction. In Travels in intermedia [lity]: Re-blurring the boundaries, ed. Herzogenrath, 1–14. Dartmouth: Dartmouth College Press.

15

Reformulating the Theory of Literary Intermediality: A Genealogy. . .

363

Higgins, Dick. 1984. Horizons: The poetics and theory of the intermedia. Carbondale: Southern Illinois University Press. Hughes, Mary Joe. 2013. The move beyond form: Creative undoing in literature and the arts since 1960. New York: Palgrave Macmillan. Jameson, Fredric. 1981. The political unconscious: Narrative as a socially symbolic act. Ithaca: Cornell University Press. Johansson, Christer, and Sonya Petersson. 2018. Introduction. In The power of the in-between: Intermediality as a tool for aesthetic analysis and critical reflection, ed. Petersson et al., 1–21. Stockholm: Stockholm University Press. Kristeva, Julia. 1984. Revolution on poetic language. Trans. Margaret Waller. New York: Columbia University Press. ———. 1986. Word, dialogue and novel. In The Kristeva reader, ed. Toril Moi, 34–61. New York: Columbia University Press. Langer, Susanne K. 1942. Philosophy in a new key. Cambridge: Harvard University Press. ———. 1957. Problems of art: Ten philosophical lectures. New York: Scribner. Leitch, Vincent B. 1986. Deconstruction and pedagogy. In Theory in the classroom, ed. Cary Nelson. Champaign: University of Illinois Press. Lessing, Gotthold Ephraim. 1984. Laocoon: An essay on the limits of poetry and painting. Trans. Edward Allen McCormick. Baltimore: Johns Hopkins University Press. Locke, John. 1829. An essay concerning human understanding. London: Thomas Tegg. Lyotard, Jean-François. 1991. The inhuman: Reflections on time. Trans. Geoffrey Bennington and Rachel Bowlby, Stanford: Stanford University Press. Marcuse, Herbert. 2007. One-dimensional man: Studies in the ideology of advanced industrial society. New York: Routledge. Markiewicz, Henryk, and Uliana Gabara. 1987. Ut pictura poesis. . . A history of the topos and the problem. New Literary History 18 (3): 535–558. McLuhan, Marshall. 1965. Understanding media: The extensions of man. New York: McGraw-Hill. Mitchell, W.J.T. 1980. Introduction: The language of images. In The language of images, ed. Mitchell, 1–8. Chicago: University of Chicago Press. ———. 1987. Iconology: Image, text, ideology. Chicago: University of Chicago Press. ———. 2005. There are no visual media. Journal of Visual Culture 4 (2): 257–266. Mitchell, W.J.T., and Mark B.N. Hansen. 2010. Introduction. In Critical terms for media Studies, ed. Mitchell and Hansen, vii–xxii. Chicago: University of Chicago Press. Oosterling, Henk, and Ewa Plonowska Ziarek. 2011. General introduction. In Intermedialities: Philosophy, arts, politics, ed. Oosterling and Ziarek, 1–7. New York: Lexington Books. Pater, Walter. 1980. In The renaissance: Studies in art and poetry, ed. Donald L. Hill. Berkeley: University of California Press. Punzi, Maddalena Pennacchia. 2007. Literary intermediality: An introduction. In Literary intermediality: The transit of literature through the media circuit, ed. Punzi, 9–14. New York: Peter Lang. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation. Intermediality: History and Theory of the Arts, Literature and Technologies 6: 43–64. ———. 2010. Border talks: The problematic status of media borders in the current debate about intermediality. In Media borders, multimodality and intermediality, ed. Elleström, 51–68. New York: Palgrave Macmillan. Rippl, Gabriele. 2015. Introduction. In Handbook of intermediality: Literature-image-sound-music, ed. Rippl, 1–31. Berlin: Walter de Gruyter. Ruskin, John. 1900. The complete works of John Ruskin: Modern painters. Vol. II. New York: Kelmscott Society. Schiller, Friedrich. 2004. Letters on the esthetic education of man. Trans. E. M. Wilkinson and L. A. Willoughby. In Critical theory since Plato, ed. Hazard Adams and Leroy Searle, 461–472. Belmont: Wadsworth Publishing. Schröter, Jens. 2010. The politics of intermediality. Film and Media Studies 2: 107–124.

364

B. Wang

Schröter, Jens. 2012. Four models of intermediality. In Travels in intermedia[lity]: Re-blurring the boundaries, ed. Herzogenrath, 15–36. Dartmouth: Dartmouth College Press. Sidney, Philip. 1989. In An apology for poetry (or the defence of poesy), ed. Geoffrey Shepherd. Manchester: Manchester University Press. Sousanis, Nick. 2015. Unflattening. Cambridge: Harvard University Press. Spivak, Gayatri Chakravorty. 2000. Translation as culture. Parallax 6 (1): 13–24. ———. 2003. Death of a discipline. New York: Columbia University Press. Wang, Bowen. 2021. Demystification of the “innocent eye”: Nelson Goodman, Ernest H. Gombrich, and the limitation of conventionalism. Journal of Comparative Literature and Aesthetics 44 (1): 81–88. Weisstein, Ulrich. 1973. Comparative literature and literary theory: Survey and introduction. Trans. William Riggan. Bloomington: Indiana University Press. Wolf, Werner. 2005. Intermediality. In Routledge encyclopedia of narrative theory, ed. David Herman et al., 252–256. New York: Routledge. ———. 2006. Introduction: Frames, framings and framing borders in literature and other media. In Framing borders in literature and other media, ed. Wolf and Walter Bernhart. Amsterdam: Rodopi. ———. 2014. Framings of narrative in literature and the pictorial arts. In Storyworlds across media: Toward a media-conscious narratology, ed. Marie-Laure Ryan and Jan-Noël Thon, 126–147. Lincoln: University of Nebraska Press.

Transmedial Narratology and Transmedia Storytelling

16

Raphae¨l Baroni, Anaı¨s Goudmand, and Marie-Laure Ryan

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toward a Media-Conscious Narratology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Transmedial History of Narratology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toward a Reconceptualization of Narrativity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedia Storytelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedia Storytelling, Adaptation, and Transfictionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedia Storytelling and Transmedia Marketing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedia Storytelling, Transmedia Worlds, and Storyworld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Media Convergence/Hierarchy Between Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedia Storytelling and Participation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

366 367 367 372 375 377 379 380 382 384 385 386

Abstract

The notion of transmediality, when used in conjunction with narrative, can be understood in two ways. Transmedial narratology is the expansion of the discipline beyond the medium of language-based narrative for which it was originally developed, to narrative media such as film, drama, comics, and video games. Its purpose is the study of the expressive devices of narrativity in different media. Transmedia storytelling refers to the migration of narrative content across various media. This phenomenon is as old as the existence of media of communication, but it has received a boost through the development of digital media. The first part R. Baroni (*) University of Lausanne, Lausanne, Switzerland e-mail: [email protected] A. Goudmand Sorbonne University, Paris, France e-mail: [email protected] M.-L. Ryan Independent Scholar, Bellvue, CO, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_16

365

366

R. Baroni et al.

of this entry describes the evolution of narratology from a language-centered to a transmedial field of study, and the second examines the various manifestations of transmedia storytelling, as well as the theoretical implications of the phenomenon. Keywords

Transmedial narratology · Transmedia storytelling · Storyworld · Narrative theory

Introduction The last decades have seen a proliferation of narratological research devoted to the notion of transmediality (see Ryan 2013; Ryan and Thon 2014; Thon 2016; Baroni 2017; Elleström 2019). However, the use of the adjectives transmedia or transmedial varies among scholars. In the broad sense – which is the most commonly used but also the vaguest – transmedial narratology refers to the study of “narrative practices in different media” (Herman 2009: 194). This meaning is retained by Werner Wolf, who regards transmediality as a set of features common to different media: Transmediality concerns phenomena which are non-specific to individual media and/or are under scrutiny in a comparative analysis of media in which the focus is not on one particular source medium. Being non-media specific these phenomena appear in more than one medium. (Wolf 2011: 5)

Wolf distinguishes transmediality from intermediality, a term he uses to refer to the combination or mixing of multiple media within a single work, such as image and language in children’s books, or music and language in operas (Wolf 2011) (The term multimodality is also used to describe this practice.). Here the term transmedial narratology will be used to describe the extension of narratology beyond languagebased narrative, the medium for which it was originally developed. In a second, narrower sense, transmediality refers to the migration of narrative content across various media (Thon 2015: 440). We will use the term transmedia storytelling to designate this phenomenon. Pioneered by the work of Jenkins (2006), the study of transmedia storytelling focuses primarily on the large franchises developed by the entertainment industry, such as Star Wars or Harry Potter, but it can also concern any type of cross-media relation, such as adaptation. The emergence of digital technologies has not only complexified and enriched narrative forms, but it has also been a decisive factor in the creation of transmedial storyworlds. The first section of this entry will be devoted to the development of narratology from a discipline narrowly concerned with language-based texts to a mediumconscious project, while the second section will discuss the cultural phenomenon of transmedia storytelling.

16

Transmedial Narratology and Transmedia Storytelling

367

Toward a Media-Conscious Narratology In what follows we will present the various historical stages that have marked the development of transmedial narratology, as well as the reconfiguration that the notion of narrativity has undergone through this evolution from something defined by modes of expression to autonomous cognitive representation (Some elements of this section have been partially taken from an article published in the journal Poétique, with the editor’s permission. See Baroni (2017).).

A Transmedial History of Narratology From the very beginning of narratology, story was regarded by the founding fathers of the discipline as a medium-transcending concept. As Claude Bremond observed in 1964, the seed of the transmedial ambition of the structuralist conceptualizations of narrativity was already contained in the formalist work of Vladimir Propp, who played a major role in revealing, behind the particular contents of a hundred Russian fairytales, a limited number of spheres of action as well as an invariant sequence of 31 functions. Bremond claims that through this process of abstraction, Propp discovered “a layer of autonomous significance endowed with a structure which can be isolated from the whole of the message: the story [le récit]” (We use here the translation by Prince (2014).) (1964: 4). He further explains: The structure of the latter is independent of the techniques that support it. It can be transposed from one to another without losing anything of its essential properties: the subject of a tale can serve as argument for a ballet, that of a novel can be brought to stage or screen, one can recount a movie to those who have not seen it. These are words we read, images we see, gestures we decipher, but through them it is a story that we follow, and it can be the same story. The narrated [le raconté] has its distinctive significant elements, its racontants: these are not words, images, or gestures but the events, situations, and behaviors signified by words, by images, by gestures.1 (Bremond 1964: 4–5)

In another seminal article published in 1966, Roland Barthes asserted that narratives are found in a “prodigious variety of genres, each of which branches out into a variety of media, as if all substances could be relied upon to accommodate man’s stories”: Among the vehicles of narrative are articulated language, whether oral or written, pictures, still or moving, gestures, and an ordered mixture of all these substances; narrative is present in myth, legend, fables, tales, short stories, epics, history, tragedy, drame [suspense drama], comedy, pantomime, paintings (in Santa Ursula by Carpaccio, for instance), stained-glass windows, movies, local news, conversation. (Barthes 1975: 237)

1

Although, to our knowledge, Bremond’s article has not been translated in its entirety, we use here an extract translated by Gerald Prince (2014).

368

R. Baroni et al.

The neologism narratology was actually coined three years later when Tzvetan Todorov realized that a “science of narrative” had to cut across disciplines, because narratives can be found in “fairytales, myth, film, and dreams” (1969: 10). In its early days, narratology was thus conceived as a way to deal with the most diverse forms of narrative. Due to their cultural prominence in the twentieth century, movies, comics, and stories conveyed by pictures were going to play a major role in this project. This first stage of narratology centered on the structures of the story (the narrated), and the effects of the medium (or substance) were often neglected. Symptomatic of this neglect is Bremond’s claim that the structure of a story can be transposed from one medium to another “without losing anything of its essential properties.” Yet it seems obvious that alterations to the structure of the story are inevitable in the process of adaptation. To take a trivial example, the length of a film or a play is limited by cultural and technical constraints, and the adaptation of a novel to the screen or to the stage almost inevitably involves cuts in the storyline. Another factor in the reduction of the original scope of narratology to verbo-centric conceptions of narrativity was the dominance in the seventies of a structuralist paradigm imported from linguistics. Gérard Genette, the most prominent figure in narratology, built on the opposition between the diegetic and the mimetic modes in Plato’s poetics to extradiegetic (external), though film is a mimetic medium according to the preceding conception.) to defend a narrow conception of narratology, insisting that the verbal act of a narrator should be considered a necessary condition of narrativity. For Genette, narrative is not based of the dyad story/discourse but on a triad story/narrative/narrating (histoire/ récit/narration) which stresses the importance of the productive act of a narrator. I propose, without insisting on the obvious reasons for my choice of terms, to use the word story for the signified or narrative content (even if this content turns out, in a given case, to be low in dramatic intensity or fullness of incident), to use the word narrative for the signifier, statement, discourse or narrative text itself, and to use the term narrating for the producing narrative action and, by extension, the whole of the real or fictional situation in which that action takes place. (Genette 1980: 27)

For Genette, this inclusion of the narrating act leads to a distinction between two narratologies, one thematic, confined to the analysis of the structures of the story, and the other modal, dealing with the structure of the narrative discourse conceived as a specific “verbal mode” of “representation” of the story, in other words, as the production of a narrator: there is room for two narratologies, one thematic in the broad sense (analysis of the story or the narrative content), the other formal or, rather, modal (analysis of narrative as a mode of “representation” of stories, in contrast to the nonnarrative modes like the dramatic and, no doubt, some other outside literature). But it turns out that analyses of narrative contents, grammars, logics, and semiotics have hardly, so far, laid claim to the term narratology, which thus remains (provisionally?) the property solely of the analysis of narrative mode. This restriction seems to me on the whole legitimate, since the sole specificity of narrative lies in its mode and not its content, which can equally well accommodate itself to a “representation” that is dramatic, graphic, or other. (Genette 1988: 16)

16

Transmedial Narratology and Transmedia Storytelling

369

By asserting that the “sole specificity of narrative lies in its mode and not its content” and by defining this mode as the discourse of a narrator, Genette turns his back on previous conceptions of narrativity and excludes a vast territory from his analysis of narrative structures such as analepsis, focalization, speed of narration, etc. – the territory of visual mimetic media, for which, as we shall see, these concepts are fully relevant. It is indeed easy to show that many of the categories developed by Genette apply just as well to graphic or audiovisual narratives as to language-based stories. For instance, a movie represents a story in a certain order, with a certain speed, from a certain perspective, and sometimes it is even told by a narrator, as illustrated by the techniques of flashbacks, cuts, subjective camera, and the occasional use of voice-over. The narratological analysis of works representing the mimetic mode could not therefore remain long confined to what Genette regards as a thematic approach, and the first medial expansion of modal narratology was notably achieved under the impulse of film studies in the late 1970s and the 1980s. Seymour Chatman pioneered this expansion by applying the opposition between story and discourse to the analysis of film. He suggested that nonverbal narratives share with literary fictions not only their capacity to construct a reference to existents or events – which concerns the “story” or “content” of the narrative – but also their capacity to configure the presentation of this content. This configuration represents a form of “discourse” or “expression.” Chatman writes: Narrative discourse, the “how,”. .. divides into two subcomponents, the narrative form itself – the structure of narrative transmission – and its manifestation, its appearance in a specific materializing medium, verbal, cinematic, balletic, musical, pantomimic, or whatever. Narrative transmission concerns the relation of time of the story to time of the recounting of story, the source of authority for the story: narrative voice, “point of view, “and the like. Naturally, the medium influences the transmission, but it is important for the theory to distinguish the two. (1978: 22)

Insofar as Chatman acknowledges that “the medium influences the transmission,” a number of conceptual rearrangements need to be made in order to adapt to mimetic representation the modal theory for which Genette has laid the foundation. Chatman suggests, among other aspects, that flashbacks are not a simple equivalent of “verbal analepsis,” because in cinema, “‘flashback’ means a narrative passage that ‘goes back’ but specifically visually, as a scene, in its own autonomy, that is, introduced through some overt mark of transition like a cut or a dissolve” (1978: 64). Despite this early work in cinema studies, narratology as a discipline faltered in the 1980s and was slow to broaden the scope of modal approaches, because formalist and structuralist paradigms entered a period of contestation. In a pivotal article, Shlomith Rimmon-Kenan described this period of “crisis” (1989: 157) as a consequence, among other factors, of the cultural turn of the humanities, and she advocated for a renewal based on a better inclusion of the medium in narratological research. Another important point for future research is her pledge for intermedial comparison as a methodological tool for expanding narratology without neglecting the constraints that each medium imposes on the narrative “representation”:

370

R. Baroni et al.

The inclusion of the medium within narratology has to be at least partly comparative, exploring the commonalities as well as the differences between media and the effect both have on the story, the text, and its narration. It seems to me that 1) most of the distinctions between media will turn out to be matters of degree rather than of absolute presence or absence of qualities; and 2) what is a constraint in one medium may be only a possibility in another. (1989: 161)

Long before the development of a global (i.e., English-speaking) narratology, French film theorists André Gaudreault (1988) and François Jost (1989) proposed a framework adopting this intermedial method in order to deal with what Alain Boillat (2007) calls “filmic enunciation.” Gaudreault based his reflection on a reassessment of the apparent opposition between the mimetic and the diegetic modes in Plato’s Republic. According to Gaudreault, mimesis “is and remains undeniably a diegesis, a narrative, diegesis dia mimeseos says Plato, that is to say a narrative by the means of imitation” (Gaudreault 1999: 61). Inspired by the concept of “Grand Image Maker” (Grand Imagier) proposed by Albert Laffay (1947) to describe filmic enunciation, Gaudreault (199: 91) proposes a notion of “monstrator,” which can be considered as an analog to a “narrator,” but whose domain is the building of a narrative “by the means of imitation.” Jost also made an important contribution to the early extension of modal narratology by showing that focalization, in order to be applied to filmic representation, needed to be reconceptualized, leading to a clearer distinction between the knowledge provided by the representation – the instances of focalization outlined by Genette – and the perception of the scene – the analysis of subjectivity, when a narrative seems to be filtered by a character-focalizer. Jost shows that audiovisual representations use specific techniques for representing subjectivity, which can be signified by internal ocularization (a visual angle associated to the point of view of a character) or by internal auricularization (a modulation of the soundtrack indicating a subjective audition). Both Gaudreault’s and Jost’s contribution to narrative theory are examples of how a transmedial approach helps to renew the concepts and methods of a general “science of narrative.” They did not simply extend the application of narratological concepts to the study of mimetic narratives by bracketing off the effects inherent to the “substance,” like Bremond, and Chatman to a certain extent; they also used an intermedial comparison of the specific ways verbal and audiovisual narratives deal with narrative transmission, leading to the introduction of new narratological concepts such as monstration, ocularization, and auricularization. On the scale of a general conceptualization of narrativity, their works also showed that some concepts needed to be rediscussed when adopting a non-verbo-centric conception of narrativity: for instance we should rethink the opposition between mimetic and diegetic modes, or the distinction between perception and knowledge in the study of focalization. In the late 1990s, in an effort of synthesis and generalization of those principles, Philippe Marion set the foundation of a media-conscious narratology (narratologie médiatique), which includes a reflection on media-specific affordances – labeled “mediativity” (médiativité) – and on narrative contents considered

16

Transmedial Narratology and Transmedia Storytelling

371

“mediagenic” because they contribute particularly well to the “expressive and communicative potential developed by the medium” (1997a: 80). The turn of the century has witnessed a rise of interest for nonverbal narrative and an explosion of studies on media such as comics, drama, and video games. The study of “graphic narratives” has been particularly prominent, due to the change of artistic reputation of this medium from lowbrow to highbrow (Chute 2008). After the wide success of works such as Art Spiegelman’s Maus and Alison Bechdel’s Fun Home, many narratologists – the majority of whom still belong to literature departments – recognized that graphic novels have literary qualities, but that the first task should be to explore the specific ways this medium tells stories or represents itself (Stein and Thon 2013; Kukkonen 2013). Narratological tools developed for other media proved to be useful to some extent – with the importation of concepts such as “monstration” (Groensteen 2010) or “ocularization” (Mikkonen 2012) – but new concepts needed to be developed for each medium. Several studies explore the effects of the spatialization of the narrative progression, leading to the exploration of “braiding effects” (Groensteen 2016) and of nonlinear form of narrative progression (Baroni 2016b). There is also great interest for the study of subjectivity in graphic narratives (Horstkotte and Pedri 2011; Mikkonen 2012) and for the exploration of graphic forms of enunciation – or “graphiation,” a term coined by Marion (1993). This term describes indexical reference to the gesture at the origin of the image, an effect of major interest in autobiographical forms of storytelling (Baroni 2021; Horstkotte and Pedri 2011). Two particular narrative forms were also the subject of intense discussions, insofar as they appeared to be borderline cases for a narratological model: isolated still image and instrumental music. The former raises issues because of its lack of temporal extension in the representation of the story, while the latter, when not dependent on text as in the opera or songs, draws heavily on the hearer’s imagination, due to the lack of intrinsic signification of its sound substance. Thus, both forms challenge the basic definition of narrativity as an interplay between the temporalities of discourse and story, as expressed, for instance, by Meir Sternberg: I define narrativity as the play of suspense/curiosity/surprise between represented and communicative time (in whatever combination, whatever medium, whatever manifest or latent form). Along the same functional lines, I define narrative as a discourse where such a play dominates: narrativity then ascends from a possibly marginal or secondary role. .. to the status of regulating principle, first among the priorities of telling/reading. (1992: 529)

Yet, in mentioning the “manifest or latent” nature of these two temporal dimensions, Sternberg opened the door to media that are only capable of representing a single moment of the narrated time – like photographs or paintings – or that convey stories only in a latent form, like instrumental music. Therefore, studies of pictorial and musical narrativity have focused on elements that elicit the active participation of the audience, leading to the elaboration of more or less variably imagined storyworld (Baroni and Cobellari 2011). Wolf calls these element “narremes” (2017: 90) and Marion “incitants narratifs” (Marion 1997b: 134).

372

R. Baroni et al.

As we can see, the evolution of narratology toward a transmedial exploration of the “structure of narrative transmission” has raised an awareness of the constraints that each medium imposes on the message, thus leading to what Marie-Laure Ryan and Jan-Noël Thon have described as a “media-conscious narratology” (2014). This new framework, under the label of “transmedial narratology,” has led to the redesign of established concepts to make them flexible enough to be adapted to multiple or even to all media, while developing a reflection on the specific ways in which each medium deals with aspects such as character-building, spatial representation, anachronisms, the representation of subjectivity, etc. A narratology attentive to the effects of media must be aware of the limits of media determinism and avoid essentialization. Wolf suggests that a comparative media approach can “help to avoid one-sided generalizations which could be observed in previous mono-medial research, be it focused exclusively on verbal texts (as has been the case in literary studies) or on the visual arts (as has been the practice of most scholars of art)” (2003: 193).

Toward a Reconceptualization of Narrativity The existence of narratives that combine multiple media, as well as the possibility of expressing narrative meaning through different media, suggests that this meaning is the product of a fundamental cognitive ability rather than depending on particular formal devices or specific modalities of expression. As Elleström explains: Media obviously have their communicative capacities because of our cognitive faculties, and it is almost absurd to suggest the notion of a cognitive system working in such a way that representations of events through one kind of medium could not in any way be matched by representations of events through other media forms. A brain that harbors a cognitive system composed of secluded, media-specific strata of information would be dysfunctional. However, we do have the capacity to communicate about things through different forms of media in such a way that narratives in various media types connect to each other in highly meaningful ways. (2019: 4)

If we admit that narrativity is a media-transcending cognitive construct and that, contrary to Genette’s position, comics, films, images, or plays are incarnations of narrativity in different media, it becomes obvious that an artefact is a narrative because it tells a story, and not because it uses a specific mode of expression. The presence of a narrator remains therefore only optional for narrative artefacts. The basic conditions of narrativity reside instead in the representation of characters or objects, the presence of change, and the embedding of these events in a network of explanations involving causality, intentionality, planning, or other such factors (Ryan 2004: 8–9). According to Monika Fludernik (1996: 12), this network of interrelated semantic elements expresses the fundamental function of narrative artefacts, which she defines as the representation experientiality. By this term she means “the quasi-mimetic evocation of ‘real-life experience’” (1996: 12). In this model, “narrativity is a

16

Transmedial Narratology and Transmedia Storytelling

373

function of narrative texts and centers on experientiality of an anthropomorphic nature” (1996: 26). Fludernik insists on the transmedial applicability of her definition and on the importance of the process of “narrativization,” which describes “a reading strategy that naturalizes texts by recourse to narrative schemas” (1996: 34). This constructivist conception of narrativity regards no media object as narrative per se. It is well-suited to account for the very diverse forms of medium-specific experiences, which involve varying degrees of effort to narrativize an artifact. Obviously, the narrativity of a painting or of a piece of instrumental music may involve greater efforts to construct than that of a Hollywood movie or a detective novel, and some audiences can even refuse or be unable to recognize it, especially since the narrative ability of these media is quite limited, compared to that of language and film. But as Fludernik shows, even modernist novels, such as James Joyce’s Ulysses, can challenge this process of narrative naturalization by problematizing the application of familiar narrative schemas. Interactive or serialized forms of narration, as well as the implicit stories conveyed by music, photographs, or paintings, also invite us to apprehend these basic elements of narrativity as dynamic experiences correlated to cognitive schemas rather than as structures inscribed in the artefacts. The content of a narrative should be seen as an “ontologically unstable matrix of possibilities” (Dannenberg 2008: 13) rather than as a fixed structure inherent to story content. This cognitive and dynamic definition proves to be extremely important for the study of transmedia storytelling, which is serialized, more or less improvised, often collectively managed, and systematically “dispersed” over several media. In this case, the configuration of a coherent and stable plot represents a cognitive challenge rather than as a stable horizon easily accessible to the audience. Among other aspects of narrativity recently discussed in the light of transmedial narratology, we have seen a growing number of scholars contesting the importance of the figure of the narrator, not only for narratives belonging to the mimetic mode, but also for verbal representations that do not refer explicitly to the act of narrating (Patron 2021). Fludernik insists on the fact that, unlike Chatman, she “emphatically refuse[s] to locate narrativity in the existence of a narrator (even if implicit, implied or covert)” (1996: 26). This raises the question of the status of the avatars of the verbal narrator created in theories of filmic (or graphic) enunciation. While it seems logical to talk about a narrator when a character in a movie (or in a comics) is shown telling a story within the story, we can question the status of a “covert narrator” (or “monstrator,” or “Grand Imagier”) when it is the implicit origin of a first-degree narrative representation. In such cases, it seems that there is no need for differentiating this narrative function from the real creator of the narrative artefact, be it single or collective, as in a movie production. If so, the “monstrator” or the “covert narrator” would just be avatars of the author, whether real or “implied.” Three medial conceptions of narratorhood can thus be distinguished: (1) a narratorial instance is an obligatory feature of narrativity (Genette, Chatman); (2) a narratorial instance is obligatory in verbal media, because language needs to be uttered by an agent, whether anthropomorphic or not, but optional in visual media – it is present in voice-over narration, or absent otherwise (Ryan 2022) – (3) a narrator is optional in

374

R. Baroni et al.

all media. In language-based narratives, it occurs only in first-person narration (Patron 2021). Another important field of research concerns the study of focalization, which proves to be of particular interest because the representation of subjectivity is a highly media-specific aspect of narratives (Thon 2016). While literary texts use the same stylistic techniques for representing thoughts, feelings, or perceptions (Rabatel 1998), narratives belonging to the mimetic mode offer their own forms of codification for these different aspects of subjectivity. As already seen, Jost has raised attention to the way visual perception can be rendered by a process of ocularization, while the rendering of the sense of audition uses techniques of auricularization. Filmic and graphic narrative have also developed a variety of technique for the evocation of nonvisual or nonauditive mental activity (Horstkotte and Pedri 2011; Mikkonen 2012; Alber 2017; Reinerth and Thon 2017). For instance, graphic narratives use medium-specific forms of codification for emotions such as “emanata,” a kind of ideogram, or stylistic modulations of the drawing, such as expressing anger, surprise, or fear through an expressive color in the background. Moreover, thoughts in movies (in comics as well) can take the form of a voice-over or that of scenic depictions of a memory, of a projected future, of a dream, or of a desired but impossible action. Other fields of investigation include the building of characters, the representation of the temporal and spatial dimensions of the storyworld and the exploration of the specific ways each medium deals with the parameters of fictionality and factuality. Transmedial approach to characters shows that mimetic media offer more complete visual and auditive representation than verbal narratives, while the opposite prevails with the representation of mental life, but mimetic media also involve possible conflict between the physical embodiment of a character by an actor and the way the actor or the audience imagines this character (Reis and Grünhagen 2021). The appearance of a character in multiple storyworlds also raises the question of the identity of this figure, which can vary tremendously from one incarnation to another yet without losing its identity, as we will see in the next section. Another element that is codified very differently in different media is time. Its involvement in the receptive process ranges from the “heterochronic” experience of textual and graphic narratives, where the user determines the pace of reading or viewing, to the “homochronic” experience of theatrical performance or film screening, where the audience has no control over the temporal unfolding of the narrative. This distinction is subverted by digital technologies, which, by offering new ways to navigate or pause the narrative, makes possible an increasing number of “polychronic” artefacts (Gaudreault and Marion 2013). Each medium has its own way of arranging the order, determining speed, and modulating the rhythm of narrative representation, and a transmedial exploration of narrative time includes paying attention to specific elements that segment, configure, accelerate, or decelerate time. Crossmedial comparisons can also shed new lights on previously neglected parameters of a general theory of narrative time. For instance, as we saw with Chatman’s discussion of flashbacks, film studies have drawn attention to a

16

Transmedial Narratology and Transmedia Storytelling

375

distinction in the functioning of analepsis between recounted past (with the event being simply evoked verbally by a narrator or a character) and enacted past (where the event is shown in the form of an autonomous scene, as if it happened in the present). We can extend such a distinction to the study of novels, since a verbal representation can be just a rapid, retrospective evocation of an event, or alternatively can lead to a shift of its deictic center to the past in the case of “dramatized” forms of analepsis (Baroni 2016a). A final question concerns whether some media are intrinsically more fictional than others, or whether the representation of factual events simply requires different strategies in different media. While some scholars may regard a drawn picture as less truthful and therefore less factual than a photograph, because it is obviously created by the hand of an author instead of being a mechanically produced indexical sign (Ryan 2020), others are more interested in exploring the strategies inherent to graphic narratives for giving an honest account of a lived experience, for example, by highlighting the drawn status of the image (or its level of “graphiation”) in order to remind the public that what they see is only a reconstruction of the past (Baroni 2021).

Transmedia Storytelling In 2003, Henry Jenkins coined the concept of “transmedia storytelling,” which he later developed in his influential book Convergence Culture: Where Old and New Media Collide (2006). In this book, he identifies and analyzes a new aesthetics that relies on the active participation of audiences and on their ability to navigate between different media channels. According to Jenkins, “a transmedia story unfolds across multiple media platforms, with each new text making a distinctive and valuable contribution to the whole” (2006: 95–96). He then clarified its definition in a blog entry: “Transmedia storytelling represents a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience. Ideally, each medium makes its own unique contribution to the unfolding of the story” (2007: n. pag., original emphasis). Transmedia storytelling is an essential aspect of a larger cultural paradigm that Jenkins referred to as “convergence,” an umbrella term which encompasses the technological, industrial, and sociocultural changes that have impacted entertainment in recent decades, following the rise of digital media. Convergence includes “the flow of content across multiple media platforms, the cooperation between multiple media industries, and the migratory behavior of media audiences” (Jenkins 2006: 2–3) alike. At the time of the publication of Convergence Culture, transmedia storytelling was epitomized by The Matrix. It has since developed spectacularly in the entertainment industry and has become a dominant cultural practice. American transmedia franchises such as Star Wars, the Marvel Cinematographic Universe (both of which belong to the Disney company), or Harry Potter,

376

R. Baroni et al.

whose narratives unfold across films, television series, novels, comic books, and video games, are raking in huge profits. Japanese media franchises have also gained international popularity using multiple media. Narratologists have shown a keen interest in transmedia storytelling in the last few years, probably because it resonates with a speculative model that they believe applies to narrative comprehension in general: the mental reconstruction of storyworlds using textual cues (Herman 2009). Insofar as they are based on sophisticated world-building strategies, transmedia stories offer a concrete and top-down coordinated expression of what remains implicit in “classical” narratives, namely, the projection of storyworlds by audiences who fill in the gaps left by what is actually represented by the narratives. Transmedia storytelling is a process that focuses on “the art of world making” (Jenkins 2006: 22; Wolf 2012) or “worldbuilding” (Jenkins 2006: 114). Creators are encouraged to design “compelling environments that cannot be fully explored or exhausted within a single work or even a single medium” (2006: 114). Transmedia world-building can therefore be regarded as the epitome of what Matt Hills refers to as “hyperdiegesis” (2002: 104): the virtually infinite and perpetually expanding storyworld is no longer coextensive with a single narrative. As Jenkins puts it, “The world is bigger than the film, bigger even than the franchise – since fan speculations and elaborations also expand the world in a variety of directions” (2006: 114). Marie-Laure Ryan suggests that the success of transmedia storytelling is due to the rewarding experience it provides: [T]here is also a reason for popularity that operates not only in the case of transmedial storytelling but also in regard to serials and monomedial transfictionality: once we have invested sufficient mental energy to construct a storyworld, we want to collect the dividends of our efforts by being able to return to this world as often as we want. Immersion takes some time to develop, but with transmedial storytelling, serials, and transfictionality, we are already immersed when new events are told, because our imaginations have built themselves a long-lasting home in the storyworld. (Ryan 2013: 385)

Hence the need for a transmedial narratology: in this context, it has become essential to develop “media-conscious” tools of analysis (Ryan and Thon 2014), in order to grasp the narrative issues not at the level of a single medium, but within an ecology encompassing a number of different media. The purpose of this section is to examine how the theoretical framework of transmedial narratology has been used to dialogue with, enrich, and/or nuance Jenkins’s notion of transmedia storytelling. In its original formulation, Jenkins’ model is based on an ideal conception of what transmedia storytelling should be: a top-down coordinated project based on collaborative authority and developing coherent storytelling, free of logical inconsistencies and redundancies. This model, which applied only partially the defining principles Jenkins identified, failed to consider how the entertainment industry actually uses transmedia. Narratologists generally seek to distance themselves from the promotional dimension of Jenkins’ writings and to develop approaches accounting for existing practices of production and reception.

16

Transmedial Narratology and Transmedia Storytelling

377

Transmedia Storytelling, Adaptation, and Transfictionality Whereas transmedia storytelling is often associated with the new possibilities offered by digital media, Jenkins acknowledges that it is not an entirely new phenomenon (2011). He states that its history goes as far as the Bible or the Homeric epics (2009a, n. pag.). Transmedia storytelling now has its archeologists (see Scolari et al. 2014): many scholars have diachronically examined early examples for transmedial storytelling from nineteenth-century and twentieth-century media culture, even though they were not labeled as such. Citing the many apocryphal adventures of Frankenstein or Sherlock Holmes and the transmedia strategies used by early television, Jason Mittell stated that “the strategy of expanding a narrative into other media is as old as media themselves” (2014: 253). In an attempt at historicizing the concept, Matthew Freeman (2016) studied early examples from the twentieth century (The Wizard of Oz, Superman). Following such perspectives, transmedia storytelling should not be conceived as a break with previous modes of narration, but rather as their historical continuation. According to Jenkins, digital technologies are only one of the factors explaining its contemporary success: The current push for transmedia has emerged from shifts in production practices (shaped by media concentration, in some cases) or reception practices (the emergence of Web 2.0 and social media), but it has also come from the emergence of new aesthetic understandings of how popular texts work (shaped in part by the rise of geeks and fans to positions of power within the entertainment industries). (Jenkins 2011, n. pag.)

However, Ryan objects that if one accepts Jenkins’ normative definition of transmedia storytelling as a top-down coordinated process, biblical stories, Greek myth, or retellings of Sherlock Holmes cannot be regarded as such since they “are not the result of a deliberate decision by an authority to distribute narrative content across different media; rather, they are the result of a bottom-up, grassroots phenomenon” (Ryan 2015: 2). Furthermore, those historical antecedents do not satisfy another core principle of transmedia storytelling: the fact that it should offer a “unified” experience, relying on “additive comprehension” (Jenkins 2006: 127), meaning that every installment has to bring new information to/about the story. Transmedia stories are, in this definition, supposed to create cohesive and coalescent storyworlds, which de facto excludes most early instances of transmedia storytelling that emerged from TV or comics, since they essentially consisted of retelling preexisting stories. This apparent contradiction requires a differentiation with regards to Jenkins’ definition of transmedia storytelling. He initially stated that there is a key difference between transmedia storytelling and adaptation: “for many of us, a simple adaptation may be ‘transmedia’, but it is not ‘transmedia storytelling’ because it is simply re-presenting an existing story rather than expanding and annotating the fictional world” (2009a, n. pag.). In contrasts to transmedia storytelling, which relies on narrative continuity across various media channels, intermedial adaptations such as the Harry Potter or The Lord of The Rings movies would thus be characterized by a logic of redundancy and would have more

378

R. Baroni et al.

to do with crossmedia storytelling, which consists of telling the same story using a different medium. Jenkins’ clear-cut distinction between transmedia, crossmedia, and adaptation has led to much discussion by scholars who have taken more nuanced positions. Christy Dena was the first to argue that “adaptation is not always (or ever) redundant” (2009: 160) and that it “resonates with the spirit of transmedia” (2009: 156). Ryan also sought to bring flexibility to the rather rigid model proposed by Jenkins. Rather than excluding adaptation altogether, she defines transmedia storytelling as a hybrid of adaptation and “transfictionality” (2013, 2016), using Richard Saint-Gelais’ proposed term to refer to the sharing of fictional elements (characters, locations, events, etc.) by two or more texts (2011). Ryan relativizes the notion of transmedia storytelling as a process that is unified and coordinated from the very beginning. In addition to the transmedia franchises planned top-down such as The Matrix, she identifies a principle she calls the “snowball effect”: in those cases, the initial story is adapted in a different medium after hitting commercial success, generating in a secondary way a franchise involving media different from the medium in which the story was first created (2013: 363). Novel-based franchises such as Harry Potter or The Lord of the Rings are common examples of the snowball effect. Transmedia franchises thus often include an adaptative component, since they “typically present lots of overlap between documents” (2016: 5). She adds that “because every medium has different expressive power, no two retellings will convey the exact same information” (2016: 5). On the other hand, transfictionality, meaning the migration of fictional entities across different texts, is not necessarily transmedial. In fact, most of the works that Saint-Gelais studies in his book are monomedial literary narratives. Ryan regards transmedia storytelling as “a special case of transfictionality – a transfictionality that operates across many different media” (2013: 366). According to her, transfictionality involves three semantic operations with respect to the initial text, which she borrows from Lubomír Doležel (1998): expansion adds new stories that take place before (prequel) or after (sequel) the original story, or even in between (midquel), modification involves changes of the original story’s plot, and transposition keeps the same plot, but changes the temporal and/or spatial setting. She claims that expansion is the main operation used in transmedia storytelling because it “is the only [one] that preserves the integrity of the storyworld” (2015: 3), but modifications and transpositions can also be observed, especially in “snowball” transmedia projects (Thon 2015: 33). Jenkins has himself challenged his initial definition by stating that transmedia storytelling is not limited to transmedia franchises that construct a strong sense of continuity. It can rely on a logic of “multiplicity” using alternative retellings or parallel universes (2009b: n. pag.). “Multiplicity” is described by Jenkins as a “liberating” concept that allows to extend transmedia storytelling to operations other than the mere expansion and to grassroots expression such as fan fiction. Following Dena’s reflections, his views have also evolved toward a less binary and more scalar conception of the distinction between adaptation and extension: “It might be better to think of adaptation and extension as

16

Transmedial Narratology and Transmedia Storytelling

379

part of a continuum in which both poles are only theoretical possibilities and most of the action takes place somewhere in the middle” (2011: n. pag.).

Transmedia Storytelling and Transmedia Marketing Since Jenkins’ notion has been largely popularized in the entertainment industry, narratologists have often adopted a cautious – even suspicious – attitude toward it. For instance, Ryan wonders if it is an “industry buzzword” rather than a “new narrative experience” (2016). Indeed, even though transmedia storytelling is not limited to big commercial franchises – since according to Jenkins, it can be developed within a “high end transmedia system,” like a media franchise, as well within a “low end transmedia system,” like a low-budget or independent project (2011: n. pag.) –, it is often used by media conglomerates that own the intellectual property of the characters and works produced. Setting up coordinated fictional universes requires heavy commercial and legal operations. According to Colin B. Harvey, transmedia storytelling is the result of “legally framed interactions,” i.e., “transactions between owners of intellectual property (IP) rights, in-house operatives, licensees, and, of course, consumers” (2014: 280). In such a context, the line between transmedia storytelling and licensing or branding is somehow fuzzy: is tie-in content transmedia storytelling? What about by-products such as toys and other merchandise, or promotional materials? Jenkins insists on the creative aspects of transmedia storytelling and regards it therefore as distinct from marketing strategies and brand extension (2011): it serves a narrative purpose and relies on co-creation and collaboration, whereas licensing is essentially used to promote the brand. He thus values the vision of synergy that shaped The Matrix, which he contrasts with the licensing system that typically generates products “governed to much by economic logic and not enough by artistic vision,” “redundant,” “watered down,” or “riddled with sloppy contradiction” (2006: 105). In the same way, Jason Mittell states that “we need to avoid confusing general transmedia extensions with the more particular mode of transmedia storytelling” (2014: 254), drawing on Jonathan Gray’s study of “media paratexts” (2010): promos, trailers, posters, etc. are created to produce “hype” and have mainly a teasing function and not a narrative one. Matthieu Letourneux, in turn, proposes a reverse perspective: he links the current use of transmedia storytelling in the entertainment industry to the shift toward a postFordist economy and regards it as a displacement of value toward the brand. He suggests that transmedia universes should therefore be regarded as a branding strategy giving an overall coherence to the range of products offered by the brand coordinating them (Letourneux, Matthieu. 2020: §13). Transmedia storytelling is therefore first and foremost a commercial trend and should be understood not as a phenomenon different from branding, but as one of its subcategories. Some argue that those “paratexts” should not be excluded from a narratological approach of transmedia storytelling. This is because, on the one hand, paratexts often have a proto-narrative function: trailers, even posters or book covers, create suspense or

380

R. Baroni et al.

curiosity and contribute to the construction of the storyworld (see Goudmand 2018; Baroni 2020: 201–203) and, on the other hand, because excluding paratexts would amount to artificially and normatively isolating the more “noble” narrative contents from the other products even though they serve the same commercial purpose. Some characters are indeed explicitly designed not only to expand the storyworld, but also to make cute toys in order to multiply profits (see BB-8 or Grogu “Baby Yoda” in Star Wars).

Transmedia Storytelling, Transmedia Worlds, and Storyworld Even though Jenkins stresses the importance of “world-building” (2006: 114), he still considers narrativity as a fundamental part of transmedia experience: transmedia storytelling aims at creating both a unified world and a unified story, “a narrative so large, it cannot be covered in a single medium” (2006: 95). His analysis of The Matrix, for instance, is based on a very classical conception of narrative. He shows that the content of the different media (films, anime, video games) form a cohesive story arc with an initial state, complications, and a resolution (2006: 95). He stated elsewhere that transmedia storytelling creates in fact a serial in which the plot is distributed across various media channels: “We can think of transmedia storytelling then as a hyperbolic version of the serial, where the chunks of meaningful and engaging story information have been dispersed not simply across multiple segments within the same medium, but rather across multiple media systems” (Jenkins 2009c: n. pag.). However, as Florent Favard has shown (2020), such a conception is not unproblematic from a narratological perspective, since transmedia projects never focus on a single sequence, nor even on the different sequences of an overarching story. This is not even the case for The Matrix: the video game Enter the Matrix, for example, is a sequel to the second film of the trilogy, but some episodes of the computer-animated series The Animatrix break away from the main story and offer autonomous developments. This is why Lisbeth Klastrup and Susana Tosca prefer to focus on “transmedia worlds,” defined as “abstract content systems from which a repertoire of fictional stories and characters can be actualized or derived across a variety of media forms” (2004: n. pag.) rather than on transmedia storytelling. They thus question the centrality of narrative, which is qualified as a secondary or even optional feature. For the same reason, Ryan rejects the connection between serial and transmedia storytelling in favor of the unifying concept of the storyworld: “Transmedia storytelling is not a serial; it does not tell a single story, but a variety of autonomous stories, or episodes, contained in various documents. What holds these stories together is that they take place in the same storyworld” (Ryan 2015: 4). She argues that the transmedia experience is more exploratory than narrative, and she goes as far as calling the expression “transmedia storytelling” a “misnomer”: [T]ransmedia storytelling is not a game of putting a story together like a jigsaw puzzle, but rather a return trip to a favorite world. It satisfies the encyclopedist’s passion for acquiring

16

Transmedial Narratology and Transmedia Storytelling

381

more and more knowledge about a world, or the collector’s passion for acquiring more and more souvenirs, but not the detective’s passion for reconstructing a story out of disseminated facts. The term “transmedia storytelling” is therefore a misnomer: the phenomenon should rather be called transmedia world-building. (Ryan 2015: 4–5)

Ryan contrasts “plot-dominated” genres, such as tragedy or jokes, with “worlddominated” genres (2015: 5), such as fantasy and science fiction: transmedia projects generally favor the latter, which are better suited for to the creation of multiple stories. The storyworld, rather than the plot, is “what holds together the various texts of the system” (2013: 363). According to Ryan, transmedia storytelling differs from other narrative forms because of the specific way the storyworld relates to texts. Storyworlds can indeed bear three types of relations to texts (2013: 365): • A one-text/one-world relation: the text is the only mode of access to the determinate storyworld. • A one-text/many-worlds relation: the text is indeterminate and can inspire many interpretations. • A many-texts/one-world relation: this relation is typical of the multiple retellings of the same story in oral tradition, but is also the one developed in transmedia storytelling. In order for a “one-world/many-texts” system to work, audiences have to assume that the storyworld is ontologically unified despite the semiotic plasticity of its setting and characters. For example, the young padawan Ahsoka Tano was created for the computer-animated series Star Wars: Clone Wars, where she is voiced by actress Ashley Eckstein. She later appeared, among others, in a novel and a video game and, as an adult, as a supporting character in the computer-animated series Star Wars: Rebels. She returned again in the live-action series The Mandalorian where she was interpreted by actress Rosario Dawson, who not only provided her voice, but also her face. Recipients are encouraged to assume that live-action Ahsoka is the same character as animated Ahsoka, and not a different version of the character or a “counterpart,” to use the logicians’s terminology. Thon (2015: 29, 2016: 61) applies Kendall Walton’s “principle of charity” (1990: 183) as a hypothesis to explain why these differences of representation do not affect the experience of transmedia storyworlds: audiences will rather ignore the evident semiotic disparities of the storyworld than disregard it as paradoxical or illogical. Thon points out, however, that the concept of the “single (story)world” (2015: 24) is insufficient to address the complexity of transmedia franchises and their numerous logical inconsistencies. In his views, these franchises are better understood using a gradual model distinguishing between: • The local medium-specific storyworlds of single narrative works • The “glocal” and noncontradictory transmedia storyworlds constructed out of multiple narrative works

382

R. Baroni et al.

• The global and often contradictory transmedia storyworlds, which he calls transmedia “universes” He also draws a distinction between the “local work-specific characters” (specific instantiations of characters in a single text), the “global transmedia character networks consisting of work-specific characters, some of which may, under certain conditions, coalesce into a single transtextual or even transmedia character” and, on the even larger scale, the “transmedia figures” they are connected to (2022: 142, original emphasis). This model has the advantage of being more in line with the majority practices of the transmedia franchises, which are regularly reshaped by various operations such as reboots or “retroactive continuity” – especially the “snowball” franchises, which rely a lot on redundancy and modification. But it is also useful to describe more precisely the functioning of global and noncontradictory (or at least generally noncontradictory) storyworlds. For instance, the term “Mandoverse” is used by the Star Wars creative teams to refer both to The Mandalorian and its spinoffs (The Book of Boba Fett, Ahsoka, etc.). The “Mandoverse” is therefore designed as a “glocal” subworld within the global overarching Star Wars universe. Similarly, Raphaël Baroni uses the term “local coagulation” (2020: 213) when he talks about the Marvel series building their own coherence on their broadcasting platform, Netflix (Jessica Jones, Luke Cage, Iron Fist, etc.). Marvel Studios, inspired by the strategy previously implemented in the comics, has taken this logic a step further by assigning numbers to the different subworlds that contradict each other (Earth-616, Earth-838, etc.). These worlds coexist within a global universe in which the existence of parallel realities and timelines – as well as the possibility to move from one to another – is established as a scientific truth. For example, in Spider-Man: No Way Home, Peter Parker from the Marvel Cinematic Universe (MCU), interpreted by Tom Holland, meets Peter Parker from Sam Raimi’s trilogy (Tobey Maguire) and Peter Parker from Marc Webb’s films (Andrew Garfield). This narrative stratagem, labeled “Multiverse” by Marvel – which is only possible due to agreements between the studios that produced the different films (Columbia Pictures, Sony Pictures and Marvel Studios) – allows to retroactively reinstate films that did not initially share the same universe within the “single, multidimensional storyworld” (Thon 2022: 146) of the MCU.

Media Convergence/Hierarchy Between Media Another aspect of Jenkins’ model that has been widely commented on is the normative idea of a lack of hierarchy between the different media channels of the transmedia system. According to Jenkins, all elements should have the same importance, regardless of the medium. But this goal is in fact unattainable: the distinction between core works and peripheral works persists insofar as the actual recipients are rarely able or willing to navigate seamlessly between all media. For the transmedia

16

Transmedial Narratology and Transmedia Storytelling

383

projects to move beyond a niche audience, the core medium must retain some autonomy from the others. In his analyses of transmedia strategies in television, Mittell has shown that the hierarchy between texts and paratexts cannot be eliminated: he contrasts the balanced transmedia prevalent in Jenkins’ conception with the unbalanced transmedia used in television in order to attract a large audience (Mittell 2014: 294, original emphasis). This remark applies not only to television franchises such as Lost, but also, more generally, to all transmedia franchises (see Goudmand 2018: 569). In order to differentiate between core texts and secondary ones, many scholars have adopted a naval metaphor that has become conventional in the entertainment industry: the core text is referred to as the “mother ship.” Mittell indicates that transmedia strategies aim “both to protect the mother ship for television-only viewers and to reward participation for transmedia-savvy fans” (Mittell 2014: 256. See also Jenkins 2011, Ryan 2016: 7). In other words, the experience of transmedia franchises remains monomedial for a significant part of the recipients, who only watch the core works (movies, TV series) and ignore the transmedia tie-ins. Even The Matrix, though Jenkins’s prime example, takes into account the varying investment of audiences: the films can be watched on their own and are understandable to viewers who do not play the video games or watch the anime. Most transmedia expansions thus have an ancillary function. The term “canon” also refers to the hierarchy between multiple works: it designates the institutional distinction, made by the authors, the owners, or their representatives within the franchise, between sanctioned texts and apocryphal texts. When a transmedia franchise has reached an advanced stage of development, the canon can be subdivided into different hierarchical levels. The best-known example of this process is Star Wars: in 2000, facing increasing fan demand for coherence and continuity, Lucas Licensing commissioned longtime fan Leland Chan to create the “Holocron continuity database” in order to organize the rather chaotic “Expanded Universe.” The different works were classified into six levels of canonicity: the higher levels were the audiovisual works that stood closest to George Lucas’s “vision” (G-Canon for George Lucas Canon, T-Canon for Television Canon), while the lower levels were represented by books, comics, video games, and other media. Following the acquisition of LucasFilm by Disney in 2012, however, the whole “Expanded Universe” was excluded from the canonical domain and rebranded as “Star Wars Legends,” mainly because numerous works contradicted the new movie trilogy which was then under development. In this perspective, the canonical texts represent the factual domain of the fictional universe, while the legends represent what could have been – non-actualized possibilities. Yet, as Thon explains using the example of Star Wars (2022: 39), one must be careful not to overestimate the importance or the operative use of the term or concept of “canon,” since the institutional hierarchy between works is bound to evolve as expansions occur. It is not in itself sufficient to guarantee the coherence

384

R. Baroni et al.

of the global storyworld (see also Letourneux 2017: 497). This is the reason why Baroni (2020) considers the concept of plot, which has often been disregarded in transmedia storytelling studies as the most relevant tool for understanding the different degrees of integration of storylines. Contrary to Ryan’s insistence on the predominance of the storyworld, he asserts that plot continuity is an indispensable criterion for discerning core narratives from peripheral narratives (2020: 206). For instance, in Star Wars, video game characters can be affected by events that take place in the movies (such as the explosion of the Death Star in Star Wars: Battlefront II), but their actions cannot impact the plot of the movies, which forms the core of the narrative system. Similarly, for Goudmand (2015: 16–17), the exclusion of the “Expanded Universe” from the Star Wars canon proves that the storyworld remains dependent on plot continuity. The goal of this operation was precisely to be able to tell new stories in the sequels of the initial trilogy without risking redundancy or inconsistency.

Transmedia Storytelling and Participation Within the highly hierarchical transmedia systems, fan creation is typically perceived as marginalized, despite Jenkins’ emphasis on participation, since it does not enjoy the same legitimacy as licensed works: “fan fiction by definition is not canonical” (Ryan 2016: 5). For Ryan, equating transmedia storytelling with fan participation, as if they were equally defining characteristics of the same phenomenon, is inaccurate, since not all transmedia works generate fan fiction (she takes the example of a German project much less popularized than the big commercial franchises, Alpha 7.0) and since, conversely, single-media narratives can generate fan fiction (2015: 11). She argues that “fan fiction is a by-product rather than a core constituent of transmedia storytelling” (2015: 11). She nevertheless insists that it is an interesting field of narratological study: the operations of expansion, modification, or transposition can also be used to analyze how the works of fans relate to storyworlds. However, while most narratologists use the notions of storyworld or plot to theorize the differentiated integration of elements in transmedia projects, Maria Lindgren Leavenworth takes the opposite approach in her study of fan fictions around The Vampire Diaries: “the concept of storyworld holds great potential for allowing a leveling of hierarchies between sanctioned and unsanctioned products” (2014: 315). She claims that “fanon” – meaning the unlicensed fan productions that have gained legitimacy within the fan community, even if they contradict the canon – should not be considered inferior to authorized works. In her perspective, fan interpretation participates in the construction of the storyworld in the same way as the sanctioned instantiations. She concludes that “in a contemporary media climate that encourages audience participation in meaningmaking processes, it seems worthwhile to stress the importance of seeing texts on par with each other rather than in hierarchical terms of original and copy, derivation and influence” (2014: 329).

16

Transmedial Narratology and Transmedia Storytelling

385

Conclusion An important part of a media-conscious narratology is the distinction between medium-free concepts (i.e., semantic concepts that apply to narratives in all media because they define narrativity, such as character, action, setting, causality, motivation), transmedial concept (i.e., concepts applicable to several, but not necessarily all, media, such as narrator, focalization, and interactivity or concepts inspired by another medium, such as camera-eye narration), and medium-specific concepts, such as frame, gutter, and speech bubble for comics. Another part of the project is the comparison of the expressive power of media with respect to narrative content. Different media consist of different types of signs. Transmedial narratology therefore dovetails with semiotics: it asks what language can do that images cannot and vice versa (Chatman 1978); what moving images can do that still ones cannot; what kinds of stories are best fitted for interactive media, compared to noninteractive ones; how this and that aspect of narrative can be rendered with sound alone, as in radio narrative or with moving images without language, as in mute film and mime; and how space can be represented in purely temporal media such as language and music and time in purely spatial ones such as paintings (Lessing 1984 [1767]) and single-frame cartoons. Transmedial narratology studies how constraints are overcome in semiotically poor media (McLuhan’s [1996] cool media), how multiple modalities are coordinated in the service of narrativity in semiotically rich media (McLuhan’s hot media), and how media try to imitate each other to expand their narrative resources. Despite the large amount of work that has been devoted to transmedia storytelling in recent years, it remains a relatively new field of study that presents many challenges for narratology. First, it requires researchers to develop skills in multimedia analysis, whereas they are often trained to study only one or two media. Second, the important size of most media franchises in the entertainment industry makes them difficult to understand in their entirety (Ryan 2016: 7) and adds the challenge of integrating operations pertaining to the domain of marketing and licensing into the theoretical framework of narratology. In a context where transmedia stories are constantly evolving and expanding, a narratological approach can no longer be limited to the study of the narrative representations enclosed within a given text: it must also take into account the marketing strategies of the entertainment industry as well as the participation of audiences. Narratologists dealing with transmedia face a Herculean task: ideally, they should be well versed in law and economics as well as in media studies, fiction theory, fan studies, and cognitive science in order to gain a holistic view of transmedia storytelling. Such a task can only be achieved through transdisciplinarity and collaboration between scholars, as Jared Gardner suggests by rephrasing Jenkins’ definition: “Transdisciplinarity represents a process where integral elements of research get dispersed systematically across multiple scholarly channels for the purpose of creating a unified and

386

R. Baroni et al.

coordinated understanding of transmedial stories. Ideally each scholar makes her own unique contribution to the unfolding of the story about the story” (Gardner 2017: 87, original emphasis).

References Alber, Jan. 2017. The Representation of Character Interiority in Film: Cinematic Versions of Psychonarration, Free Indirect Discourse and Direct Thought. In Emerging Vectors of Narratology, Ed. Per Krogh Hansen and al., 265–283. Berlin & Boston: De Gruyter. Baroni, Raphaël. 2016a. Dramatized Analepsis and Fadings in Verbal Narratives. Narrative 24 (3): 311–329. ———. 2016b. (Un)natural temporalities in Comics. European Comic Art 9 (1): 5–23. ———. 2017. Pour une narratologie transmédiale. Poétique 182: 155–175. ———. 2020. Hiérarchies et intrigues des récits transmédiatiques. In Introduction à l’étude des cultures numériques, ed. Raphaël Baroni and Claus Gunti, 193–216. Malakoff: Armand Colin. ———. 2021. Of Mice as men: A transmedial perspective on Fictionality. Narrative 29 (1): 91–113. Baroni, Raphaël, and Alain Corbellari, eds. 2011. Rencontres de narrativités: perspectives sur l’intrigue musicale. Cahiers de narratologie 21. https://doi.org/10.4000/narratologie.6390. Accessed August 12, 2022. Barthes, Roland. 1975 (1966). An introduction to the structural analysis of narrative. New Literary History 6(2):237–272. Translated by Lionel Duisit. Boillat, Alain. 2007. Du Bonimenteur à la voix-over. Voix-attraction et voix-narration au cinéma. Lausanne: Antipodes. Bremond, Claude. 1964. Le message narratif. Communications 4 (4): 4–32. Chatman, Seymour. 1978. Story and discourse: Narrative structure in fiction and film. Ithaca: Cornell UP. Chute, Hillary. 2008. Comics as Literature? Reading graphic narrative. PMLA 123 (2): 452–465. Dannenberg, Hilary. 2008. Coincidence and counterfactuality. Plotting time and space in narrative fiction. Lincoln/London: University of Nebraska Press. Dena, Christy. 2009. Transmedia practice: Theorising the practice of expressing a fictional world across distinct media and environments. Ph.D. dissertation. University of Sydney. Unpublished. https://ciret-transdisciplinarity.org/biblio/biblio_pdf/Christy_DeanTransm.pdf. Accessed 14 July 2022. Doležel, Lubomír. 1998. Heterocosmica: Fiction and possible worlds. Baltimore: Johns Hopkins UP. Elleström, Lars. 2019. Transmedial narration. Narrative and stories in different media. Cham: Palgrave Macmillan. Favard, Florent. 2020. Le transmedia storytelling est-il soluble dans la narratologie ? Les Cahiers de la SFSIC, Collection, 12-Varia, Questions de recherche. https://cahiers.sfsic.org/sfsic/index. php?id¼404. Accessed 14 July 2022. Fludernik, Monika. 1996. Towards a “Natural” Narratology. London: Routledge. Freeman, Matthew. 2016. Historicising transmedia storytelling. Early twentieth-century transmedia story worlds. New York: Routledge. Gardner, Jared. 2017. Transmedial narratives in the age of mixed media. Narrative Culture 4 (1): 76–88. Gaudreault, André. 1999. Du littéraire au filmique. Système du récit. Paris: Armand Colin. First publication in 1988. Gaudreault, André, and Philippe Marion. 2013. La fin du cinéma. Paris: Armand Colin. Genette, Gérard. 1980 (1972). Narrative discourse: An essay in method. Ithaca: Cornell University Press. Translated by Jane E. Lewin.

16

Transmedial Narratology and Transmedia Storytelling

387

———. 1988 (1983). Narrative discourse revisited. Ithaca: Cornell University Press. Translated by Jane E. Lewin. Goudmand, Anaïs. 2015. Les séries transmédiatiques. Des univers sans fin? PRO 9: 8–18. ———. 2018. Récits en partage. Expériences de la sérialité narrative en culture médiatique. Ph.D. dissertation. EHESS, Université de Lausanne. Unpublished. http://www.theses.fr/ 2018PSLEH115. Accessed 14 July 2022. Gray, Jonathan. 2010. Show sold separately: Promos, spoilers, and other media paratexts. New York: New York UP. Groensteen, Thierry. 2010. The Monstrator, the Recitant and the Shadow of the Narrator. European Comic Art 3 (1): 1–22. ———. 2016. The art of braiding: A clarification. European Comic Art 9 (1): 88–98. Harvey, Colin B. 2015. A taxonomy of transmedia storytelling. In Storyworlds across Media: Toward a media-conscious Narratology, ed. Marie-Laure Ryan and Jan-Noël Thon, 278–294. University of Nebraska Press: Lincoln. Henderson, Brian. 1983. Tense, mood, and voice in film. Film Quarterly 36 (4): 4–17. Herman, David. 2009. Basic elements of Narrative. Chichester: Wiley-Blackwell. Hills, Matt. 2002. Fan cultures. London/New York: Routledge. Horstkotte, Silke, and Nancy Pedri. 2011. Focalization in graphic Narrative. Narrative 19 (3): 330–357. Ingarden, Roman. 1961. Aesthetic experience and aesthetic object. Philosophy and Phenomenological Research 2 (3): 289–313. Jenkins, Henry. 2006. Convergence culture: Where old and new media Collide. New York: New York UP. ———. 2007. Transmedia storytelling 101. Confessions of an Aca-Fan. 22 Mar 2007. http:// henryjenkins.org/2007/03/transmedia_storytelling_101.html. Accessed 14 July 2022. ———. 2009a. The aesthetics of transmedia: In Response to David Bordwell (Part One). Confessions of an Aca-Fan. 10 Sept 2009. http://henryjenkins.org/blog/2009/09/the_aesthetics_of_ transmedia_i.html. Accessed 14 July 2022. ———. 2009b. The revenge of the origami unicorn: seven principles of transmedia storytelling (well two actually five more on friday). Confessions of an Aca-Fan. 12 Dec 2009. http:// henryjenkins.org/blog/2009/12/the_revenge_of_the_origami_uni.html. Accessed 14 July 2022. ———. 2009c. The revenge of the Origami Unicorn: The remaining four principles of transmedia storytelling. Confessions of an Aca-Fan. 12 Dec 2009. http://henryjenkins.org/blog/2009/12/ revenge_of_the_origami_unicorn.html. Accessed 14 July 2022. ———. 2011. Transmedia 202: Further reflections. Confessions of an Aca-Fan. 1 Aug 2011. http:// henryjenkins.org/2011/08/defining_transmedia_further_re.html. Accessed 14 July 2022. Jost, François. 1989. L’Œil-caméra. Entre film et roman. Lyon: Presses universitaires de Lyon. Klastrup, Lisbeth, and Susana P. Tosca. 2004. Transmedial worlds – Rethinking cyberworld design. in Proceedings of the international conference on Cyber- worlds 2004. http://www.itu.dk/ people/klastrup/klastruptosca_transworlds.pdf. Accessed 14 July 2022. Kukkonen, Karin. 2013. Studying comics and graphic novels. New York: Wiley Blackwell. Laffay, Albert. 1947. Le récit, le monde et le cinéma. Les Temps Modernes. 21: 1578–1600. Lessing, Gotthold Ephraim. 1984 [1767]. Laocoön: An essay on the limits of painting and poetry. Translated win an introduction by Eward Allen. Baltimore: Johns Hopkins UP. Letourneux, Matthieu. 2017. Fictions à la chaîne. Littératures sérielles et culture médiatique. Paris: Éditions du Seuil. ———. 2020. Penser les fictions sérielles en régime postfordiste. Cahiers de narratologie 37. https://doi.org/10.4000/narratologie.10488. Lindberg Leavenworth, Maria. 2014. In Storyworlds across media: Toward a media-conscious Narratology, eds. Marie-Laure Ryan and Jan-Noël Thon, 315–31. Lincoln: University of Nebraska Press.

388

R. Baroni et al.

Marion, Philippe. 1993. Traces en cases. Louvain-la-Neuve: Academia. ———. 1997a. Narratologie médiatique et médiagénie des récits. Recherches en communication 7: 61–88. ———. 1997b. Les images racontent-elles? Variations conclusives sur la narrativité iconique. Recherches en communication. 8: 129–148. McLuhan, Marshall. 1996. Essential McLuhan, ed. Eric McLuhan and Frank Zingrone. New York: Basic Books. Mikkonen, Kai. 2012. Focalization in Comics. From the specificities of the medium to conceptual reformulation. Scandinavian Journal of Comic Art 1 (1): 71–95. Mittell, Jason. 2014. Strategies of storytelling on transmedia television. In Storyworlds across Media: Toward a media-conscious Narratology, ed. Marie-Laure Ryan and Jan-Noël Thon, 252–277. University of Nebraska Press: Lincoln. ———. 2015. Complex TV: The poetics of contemporary television storytelling. New York: New York UP. Patron, Sylvie. 2021. Optional-Narrator theory. Principles, perspectives, proposals. Lincoln: University of Nebraska Press. Prince, Gerald. 2014. Narratology and translation. Language and Literature 23: 23–31. Rabatel, Alain. 1998. La Construction textuelle du point de vue. Lausanne/Paris: Delachaux et Niestlé. Reinerth, Maike Sarah, and Jan-Noël Thon, eds. 2017. Subjectivity across media: Interdisciplinary and transmedial perspectives. Londres: Routledge. Reis, Carlos, and Sara Grünhagen. 2021. Characters and figures. Conceptual and critical approaches, (dir.). Coimbra: Almedina. Rimmon-Kenan, Shlomith. 1989. How the model neglects the medium: Linguistics, language, and the crisis of Narratology. The Journal of Narrative Technique 19 (3): 157–166. Ryan, Marie-Laure, ed. 2004. Narrative across media. The languages of storytelling. Lincoln/London: University of Nebraska Press. ———. 2013. Transmedial storytelling and transfictionality. Poetics Today 34 (3): 362–388. ———. 2015. Transmedia storytelling: Industry Buzzword or new narrative experience? Storyworlds: A Journal of Narrative Studies 7 (2): 1–19. ———. 2016. Transmedia narratology and transmedia stroytelling. Artnodes. E-journal on Art, Science and Technology 18: 1–10. ———. 2020. Fact, fiction and media. In Narrative factuality: A handbook, ed. Monika Fludernik and Marie-Laure Ryan, 75–94. Berlin: De Gruyter. ———. 2022. A new anatomy of storyworlds: What is, What If, As If. Columbus: Ohio State UP. Ryan, Marie-Laure, and Jan-Noël Thon. 2014. Storyworlds across media: Toward a mediaconscious Narratology. University of Nebraska Press: Lincoln. Saint-Gelais, Richard. 2011. Fictions transfuges: La transfictionnalité et ses enjeux. Paris: Éditions du Seuil. Scolari, Carlos A., Paolo Bertetti, and Matthew Freeman. 2014. Transmedia archaeology: Storytelling in the borderlines of science fiction, comics and pulp magazines. New York: Palgrave Macmillan. Stein, Daniel, and Jan-Noël Thon. 2013. From comic strips to graphic novels. Contributions to the theory and history of graphic narrative. Berlin/Boston: De Gruyter. Sternberg, Meir. 1992. Telling in time (II): Chronology, Teleology, Narrativity. Poetics Today 13 (3): 463–541. Thon, Jan-Noël. 2015. Converging worlds: From transmedial storyworlds to transmedial universes. Storyworlds: A Journal of Narrative Studies 7 (2): 21–53. ———. 2016. Transmedial Narratology and contemporary media culture. University of Nebraska Press: Lincoln. ———. 2022. Transmedia characters/transmedia figures: Drawing distinctions and staging re-entries. Narrative 30 (2): 139–147. Todorov, Tzvetan. 1969. Grammaire du Décaméron. The Hague/Paris: Mouton.

16

Transmedial Narratology and Transmedia Storytelling

389

Walton, Kendall. 1990. Mimesis as make-believe: On the foundations of the representational arts. Cambridge, MA/London: Harvard UP. Wolf, Werner. 2003. Narrative and Narrativity: A Narratological reconceptualization and its applicability to the visual arts. Word and Image 19 (3): 180–197. ———. 2011. (Inter)mediality and the study of literature. Comparative Literature and Culture 13(3). https://docs.lib.purdue.edu/clcweb/vol13/iss3/2/. Wolf, Mark J.P. 2012. Building imaginary worlds: The theory and history of subcreation. New York: Routledge. Wolf, Werner. 2017. Transmedial Narratology: Theoretical foundations and some applications (fiction, single pictures, instrumental music). Narrative 25 (3): 256–285.

The Narrator: A Transmedial Device

17

Liviu Lutas

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Transmediality of the Narrator and of Narration: A Theoretical Overview . . . . . . . . . . . . . . . Oral Literature and News Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Autobiography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Poetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Film . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Painting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

392 393 400 402 402 403 406 407 409 410 411

Abstract

In this chapter, the concept of the “narrator” is presented and put to a test, especially in its necessary connection to a linguistic sphere. On the basis of a theoretical discussion of narration as a transmedial phenomenon, that is a phenomenon existing in various media, the chapter will try to show that the narrator too is a transmedial device, whose existence is not necessarily related to language it its strictest linguistic meaning. The narrator is rather a mental construction in the process of communication at the basis of certain media types. Admittedly, its origin as a theoretical concept in the field of modern narratology is mainly the novel as epitomized by the eighteenth and nineteenth centuries. Indeed, in no other specific media product has the narrator had such a prominent role as in these novels, which have been used by narratologists for the development of most of the existing concepts in the field. However, the fact that a device is prominent in a certain media type should not restrict its use to that media type alone. On the L. Lutas (*) Linnaeus University, Växjö, Sweden e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_30

391

392

L. Lutas

contrary, its applicability to other media is especially interesting to study. This is what will be done in this chapter, starting with a general theoretical discussion of the concept of “narrator” and continuing with an overview of the possibility of the existence of the narrator in both literature and other media types, both languagebased and not. Keywords

Narrator · Narratology · Transmedial narration · Transmediality · Genette · Narrative

Introduction The aim of this chapter is to make an overview of the concept of narrator and its transmedial potential. Transmedial potential means the narrator’s possible existence in other media types than narrative literature and even media types that are not language-based ones. The use of the term “transmedial” or rather “transmediality” in this chapter is based on its definition by Lars Elleström as “referring to the general concept that different media types share many basic traits that can be described in terms of material properties and abilities for activating mental capacities” (2019: 5). The narrator is one such possible shared media trait, and its existence in different media types is worth a close analysis, especially since narration is a shared goal of probably most media types, and of course of all narrative media types. But the narrator is presumably not a prerequisite of all narration, some media types, even narrative ones, using other means in order to narrate. The chapter starts with an overview of the theoretical discussions around the possible transmediality of the narrator and of narrativity in general. The theoretical references in this part are mainly made to the field of narratology, and especially its most recent developments toward transmedial narratology, which allows an opening of narratology toward other media types than written literature. The debates about the definition of the concept of “story” in general are particularly important from this point of view. Another field which informs this study is intermediality, as represented by German (cf. Wolf 2005) and Swedish (cf. Elleström 2010) scholars, for whom the communicative goal is at the core. Both transmedial narratology and intermediality in general are beneficial thanks to the flexibility of their approach, which allows for written literature to be seen as less central, and for the scholar achievements in the field of literary studies to be applied to other media types in a natural way. On the basis of this overview, the chapter then investigates the possible existence of the narrator in a number of media types. It starts with the media type where the presence of a narrator is the most obvious, that is narrative literature, and continues with media types where it is less obvious that a narrator is needed, but where language still has an important role, such as oral literature, news reporting, autobiography, poetry, comics, and films. Finally, the chapter addresses the possibility of a narrator in two media types, painting and music, where language is absent, but where

17

The Narrator: A Transmedial Device

393

narrativity is arguably present in different forms and at different degrees. The conclusion which ends the chapter states that an abstract entity that can be called “the narrator” can be used transmedially by receivers of media objects, with the goal of bringing together the details into a meaningful coherence.

The Transmediality of the Narrator and of Narration: A Theoretical Overview Seeing the narrator as a transmedial trait seems like a problematic endeavor, considering the traditional connection of the concept of the narrator to a spoken or written discourse, or in other words to language. According to a recent definition, by Uri Margolin, “a narrator is a linguistically indicated, textually projected, and readerly constructed function, slot, or category whose occupant need not be thought of in any terms but those of a communicative role” (2012). Clearly, terms like “linguistically,” “textually,” and “readerly” used in the definition above all belong to the semantic field of language, denoting clearly the verbal-based characteristic of the concept. However, even though Margolin does not extend the possible existence of a narrator to other media in his article, his highlighting of the “communicative role” at the core of the concept highlights exactly the potential of an opening toward such an extension. However, it must be acknowledged from the very beginning that the concept of the “narrator” has been practically exclusively studied in the field of written, or sometimes oral, literature. Already for the ancient Greeks, more precisely for Plato, the narrator was “the mediating instance between the characters’ speech and the audience” (cf Margolin 2012) which was at the basis of the distinction between narrative and drama. In the Socratic dialogues from his 375 BC work The Republic, Plato distinguished three modes of expression: plain diegesis, mimetic diegesis, and the mixed mode (cf Collobert 2013). These modes are mainly distinguished by Plato on the basis of the presence or absence of a narrator. The first mode, plain diegesis, is where the narrator is all important and “tells” the events to the audience, while the second one, the mimetic diegesis, is characterized by the absence of a narrator, the events being played by actors in front of an audience. The dichotomy “mimesis vs diegesis” is reflected in the better-known dichotomy “showing vs telling,” used extensively from the end of the nineteenth century, among others by Henry James, through theorists such as Percy Lubbock, Franz K Stanzel, and Wayne C Booth, until the birth of the more systematic theory of narratology represented by the French theorist and father of modern narratology, Gérard Genette. It must be emphasized, however, that the dichotomy “showing vs telling” was essentially used inside the framework of the media type of written literature, and not as a ground for distinguishing between different media types, as the case is with the dichotomy “mimesis vs diegesis.” A literary text could be closer either to the mode of showing or to the mode of telling depending on a number of factors, among which the narrator had undoubtedly an important place. According to an ideal based on the realist novels of the second half of the nineteenth century, these theorists

394

L. Lutas

found that the literary texts where the narrator seems to be absent, giving thus an impression of “showing” the events and the characters, are qualitatively superior to the texts where the presence of the narrator is salient and where the mode of “telling,” thus, is dominating. Lubbock states, for instance, that “the art of fiction does not begin until the novelist thinks of his story as a matter to be shown, to be so exhibited that it will tell itself” (Lubbock [1922] 1954: 62). Interestingly enough, this idea of letting the story tell itself is reflected in the works of one of the precursors of modern narratology, by French linguist Émile Benveniste, from 1955, in his opposition of the concepts of story (histoire or récit) and discourse (discours). For Benveniste, these concepts are to be understood as types of enunciations, to be distinguished from one another on the basis of a clear presence of a speaker (locuteur) in the case of discourse in contrast to the lack of a speaker (or in other words of a narrator) in the case of the story (pp. 238–241). Indeed, claims Benveniste, the enunciations of the story type (histoire or récit), characterized in French by the use of the third person pronoun and of the passé simple tense, give the impression that they are not uttered by a narrator, but rather that the events tell themselves (“les événements semblent se raconter eux-mêmes,” p. 241). The discourse (discours), on the other hand, contains the traces of a speaker who uttered it, such as the first-person pronoun “I,” the present tense and the passé compose tense, and the deictic adverbs, that is adverbs indicating the place and the time of the utterance, such as “now,” “today,” “here,” etc. As already mentioned, Gérard Genette is the theorist who succeeded in bringing more order in these rather problematic approaches to the concept of “narrator,” in his systematic narratological model. Genette’s most important contribution when it comes to the study of the narrator is his concentrating more pragmatically on the constituents that affect the narrative aspects of a literary text. Basically, Genette based his model on the questions “who talks?” (qui parle?) and “who sees?” (qui voit?) in a literary work. This gave him the possibility to study the concepts of voice and focalization as separate entities, instead of only studying the narrator. The concept of voice, especially, helped getting beyond the rather unconvincing claim that stories of certain kinds, such as those in the third person, give the readers the impression of telling themselves. What happens really, claims Genette convincingly, is that the narrator could be “homodiegetic,” that is a character in the “diegesis,” the narrated world, or “heterodiegetic,” that is not a character in the “diegesis.” A heterodiegetic narrator does not use the first-person pronoun when narrating about a world where he does not belong. Or if he or she does, it would be on another narrative level, the “extradiegetic” level, for instance, by addressing the reader, or rather the narratee. Nineteenth-century literature abounds in such examples, traditionally called cases of authorial intrusion or romantic irony. One often studied example in narratology is the following extract from Honoré de Balzac’s Les illusions perdues: “While the venerable churchman climbs the ramps of the Angouleme, it is not useless to explain the network of interests into which he was going to set foot” (quoted by Fludernik 2003a. In the French original: “Pendant que le vénérable ecclésiastique monte les rampes d’Angoulême, il n’est pas inutile d’expliquer le lacis d’intérêts dans lequel il allait mettre le pied”).

17

The Narrator: A Transmedial Device

395

Genette’s contention is, consequently, that all utterances in a text are uttered by somebody, and even stories told in the third person (“he”) are told by an “I,” even when this “I” is not explicit, at least not at the diegetic level: “all narrative is, explicitly or not ‘in the first person’, since its narrator can at any moment identify himself by the pronoun I” (“tout récit est, explicitement ou non, à la première personne puisque son narrateur peut à tout moment se désigner lui-même” (Genette 1983: 65)). “Showing,” in the context of literature, would thus be nothing more than an illusion, built among other things on conventions on how the first-person pronoun should or should not be used. In other words, there is always a voice behind every utterance, a voice which is always attributable to a person, an “I,” the narrator. This is also the point made by another well-known literary theorist, Mieke Bal, who considers the existence of a speaker as the logical consequence of the use of language: As soon as there is language, there is a speaker who utters it; as soon as those linguistic utterances constitute a narrative text, there is a narrator, a narrating subject. From a grammatical point of view, this is always a ‘first person’. In fact the term ‘third-person narrator’ is absurd: a narrator is not a ‘he’ or a ‘she’. At best the narrator can narrate about someone else, a ‘he’ or ‘she’ – who might, incidentally, happen to be a narrator, as well. (Bal 1997: 22)

On these grounds, Genette built his narratological model, published in Narrative Discourse (Discours du récit) in (1980 [1972]), and still largely used as the basis of narrative studies around the world. Without going too deeply in the details of this model, it should be mentioned that the concept of the narrator is at its core. Indeed, as Genette claimed in Narrative Discourse Revisited (1988 [1983]), 11 years after the publication of Narrative Discourse, there could be no such thing as a story without a narrator, as there could be no utterance without an enunciator (“Le récit sans narrateur, l’énoncé sans énonciation me semblent de pures chimères, et, comme telles, ‘infalsifiables’” (Genette 1983: 68)). Genette’s claim from 1983 was made as an answer to the reactions of a number of theorists who criticized the importance of the narrator in Genette’s model, on the basis of a noncommunicational or poetic theory of narrative which gained a certain success in the 1980s and 1990s. Japanese theorist Sige-Yuki Kuroda and American literary scholar Ann Banfield were the first to question the need of a narrator in literature, since, they argued, literature cannot be reduced simplistically to a communicational situation between an author and a reader or between their abstract counterparts, the narrator and the narratee. Literature, for them, is rather a poetic object, a creation, where playing with words is more important than the communication of events. Basing herself on these theories, Sylvie Patron (2009) has criticized Genette’s insistence on the necessity of a narrator in every work of literature and formulated a theory of an optional narrator. Such theoretical discussions seem to complicate even more the attempt of seeing the narrator as a transmedial concept. Indeed, how could this be done when the narrator can be an optional feature even in the media type from which it generated as

396

L. Lutas

a concept and where it has its most natural place? The answer is that even when subscribing to the noncommunicational theories of literature, one cannot disregard that the communicational grounds are still dominant in literary theory, and the cases of literary text without a narrator are both arguable and rather exceptional. Besides, it is especially via the communicational model that the narrator is most easily studied outside of the media type of its birth, which is literature. Indeed, the media model conceived by Swedish theorist Lars Elleström is also built, as most media models, on a communicational model. In his 2018 article “A medium-centered model of communication,” Elleström argues for communication between two minds as the basis of understanding of the concept of medium in general. In his attempt to formulate a communicational model in its most irreducible shape, Elleström sees communication as a transfer of cognitive import, which can be seen as a more pragmatic term for something like content, ideas, meaning, or message, between a producer’s mind and a receiver’s mind, via a media product (2018: 282). Elleström’s media-centered communicational model is applicable to all media types, so literature is easily studied with its help. In such a communicational frame, an author can be seen as the creator’s mind communicating with the reader’s mind via the media product which is, for instance, a specific novel. Importantly, for Elleström, the media product is an extension of the creator’s mind (2018: 281), which means that the receiver does not recreate, so to speak, a message, or a meaning, sent out by the creator but rather creates a meaning on the basis of the traits perceived and thereafter interpreted by him or her in the media product. This opens up the possibility of applying this model even to cases of such media products that the poetic theory of narrative would not consider relevant from a communicational point of view. But where is the narrator’s place in such a model? The answer that seems to come most directly to our mind is that the narrator is the alter ego of the author, belonging thus to the sphere of the creator’s mind. According to such a view, the real communicative situation between the author and the reader has an analogical counterpart in the communicative situation between the narrator and the narratee. The difference is that in the real situation, the media product is the novel, whereas in the abstract situation, the media product is the narrator’s utterance. And this utterance is generally of an abstract type too, as is the narrator, the reader having to imagine its existence beyond its incarnation in the novel’s materiality. The utterance can, for instance, be spoken, but it can be written too, and actually, as in the case of epistolary novels or diaries, it can give the illusion that it corresponds exactly to the written text of the novel. However, it is also possible to place the narrator at the other end of the communicational model. This is, for instance, what is done, indirectly, in Margolin’s definition, quoted at the beginning of this article: “A narrator is a linguistically indicated, textually projected and readerly constructed function, slot or category.” In other words, the narrator is rather the result of a creation of the reader, according to the indications of the text. This is in line with recent development in narratology, especially cognitively oriented narratology (Nünning 2001; Fludernik 2003a; Herman 2010). According to such views, the narrator is not inherent to the text,

17

The Narrator: A Transmedial Device

397

but rather constructed by the readers in their minds, on the basis of textual indications, using cognitive capabilities, or schemata. The cognitive approach is one of the ways which opens up the possibility to see a narrator even in the more complex cases of media types where there should not exist narrator according to the traditional definition of a person telling a story to somebody. Indeed, if the narrator ultimately is a construct in the receiver’s mind, this allows for cases where the receiver interprets rather weak indications of the presence of a narrator in other media types, for instance, through analogies to literature. However, there is another important aspect to consider when studying the possible existence of a narrator in other media types than language-based ones: the narrative capacities of these media types. Indeed, it seems counterintuitive to study the existence of a narrator in media types which are not narrative at all. The transmediality of narration has been the object of many more studies than the transmediality of the narrator. Most of these studies are about narration in one media type other than written narrative literature. Examples are studies about narration in film (Bordwell 1985; Chatman 1978; Jost 2004; Thompson 2003), comics (Groensteen 2011; Kukkonen 2011), graphic narratives (Mikkonen 2011), painting (Kafalenos 1996), photography (Kafalenos 1996; Ranta 2013), visual arts in general (Wolf 2003), music (Meelberg 2006; Seaton 2005; Wolf 2002; Elleström 2018), hypertext (Hayles 2001), computer games (Neitzel 2007), mathematics (Elleström 2019), or architecture (Psarra 2009). However, there have been important attempts to study narration as a transmedial feature from general perspectives too. The earliest of such studies were published actually before the publication of Genette’s narratological model, by two of the precursors of narratology: Claude Bremond (1964) and Roland Barthes (1977 [1966]). One unfortunate consequence of the publication of Genette’s Narrative Discourse was that because of his insistence on basing the model on literary texts, the transmedial aspect of narration was to be more or less ignored by scholars until the beginning of this century, when it suddenly became all the more frequent (Herman 2004; Ryan 2006, 2013; Thon 2016; Elleström 2019). What scholars study when it comes to the general transmediality of narration is to what degree narration can exist in all media. Indeed, basing themselves on the importance of stories for humans through history, some consider that all media have, to a certain extent, the capacity to convey a story. In such view, stories would be, to certain extents, completely transmedial, meaning that they can appear in all media. Seymour Chatman claims, for instance, that the “transposability of the story is the strongest reason for arguing that narratives are indeed structures independent of any medium” (Chatman 1978: 20). However, more recent studies conclude that the narrative capacity, or narrativity, should rather be seen as a matter of degree (Ranta 2013: 3), some media types, as those based on language, having a very strong such capacity, while other, such as music, having to rely on a weaker definition of narrativity and of narrative in general, such as James Phelan’s definition: “a rhetorical act: somebody telling somebody else on some occasion and for some purpose(s) that something happened” (Phelan 2005: 18). One of the scholars who consider narrative as a more complex phenomenon is Marie-Laure Ryan. On the basis of that complexity, she insists on the primacy of

398

L. Lutas

language as a narrative medium. Language-based media types, she argues, are the most appropriate ones to represent narratives because a narrative is, according to her, and thus mirroring Benveniste and Genette, a discourse conveying a story. Basing herself on cognitive narratology, Ryan (2007) defines a story as a mental image consisting of four elements: 1. A spatial element, or the setting, populated by characters and objects 2. A temporal element, thanks to which different events can be related in different ways to one another 3. A mental element, according to which the events involve intelligent agents with a mental life and emotions 4. A formal and pragmatic element, advocating closure and a meaningful message Different media types, claims Ryan convincingly, are differently apt at conveying different elements of a story, the language-based ones being undoubtedly the best fitted for all the four cases. However, language-based media types are not the best fitted for each specific element taken separately. Films, photographs, or paintings are, for instance, more appropriate than language to convey the spatial element of a story, since they represent the storyworld directly, iconically, or indexically, not through words, which function symbolically. The image of objects from the storyworld created in the heads of the readers using words is most certainly less precise than if those objects were represented indexically, by filming or photographing them, or iconically, by drawing, painting, or sculpting them. Music, even of the instrumental type, is very apt at conveying the formal element, maybe even better than narrative literature, since, as Ryan claims, there is in music “a forward movement, a desire-for-something-to-come, a tension calling for a resolution. In music as in narrative, the appreciator may have a powerful sense that a dénouement is imminent (perhaps more so in music, for in literature the coming end is often signaled not by narrative devices, but by the number of pages left to be read)” (Ryan 2012). Music can also be, in some ways, better apt at representing the temporal element, since music itself is built in many ways on the succession of sounds in time. It is especially the second and the third features, Ryan argues, that are highly language-dependent. Indeed, language has a temporality that seems to suit the succession of events in time. As we saw above, so has music too, but, as Ryan claims, “music does not suggest the passing of time by showing its effects on concrete existents: it captures time in its pure form” (2012). Likewise, film has also a temporality that suits the one of the events happening in a story, but the causal relation between events is more easily represented through language, claims Ryan, this time more unconvincingly. Indeed, it cannot be disputed that in a static image, the viewer must interpret the causal relation between different events that appear simultaneously. But it is hard to see why words are needed to make the viewer of a film understand, for instance, that a man died because a gun was shot at him first. Actually, even in the case of static pictures, the media product has different possibilities to suggest temporal changes, as Elleström convincingly argues in his article “The modalities of media: A model for understanding intermedial relations” (2010:

17

The Narrator: A Transmedial Device

399

20–21). One such possibility is what Elleström calls “virtual time,” meaning “the capacity of individual pictures to depict not only one static moment but a series of occurrences. Interpretations of still images of what we, on iconic grounds, take to be moving objects or creatures always include an interpretation of where the object or creature was ‘before’ and ‘after’ the frozen time in the image” (2010: 21). Ways to suggest such movements are, for instance, “objects represented with blurred contours or stretched and transparent objects” (Elleström 2010: 21) that give the receiver the illusion that time has passed in a frozen image. One could add the placing of different events in a picture as to give the impression of a succession in time, sometimes because of the viewer’s habit to watch from the left to the right, or the putting together of several stop-motion photographs in one and the same image, in order to give the impression of motion, as in photographer Eadweard Muybridge nineteenth-century photographic experiment used in order to show that a horse has all four feet in the air during some parts of its stride. The concept of “virtuality,” mentioned above in connection to the temporal aspect, is very interesting to apply to other transmedial traits. Or rather, it is through “virtuality” that certain traits can be represented or at least suggested in media types where they do not belong naturally. Elleström studies, for instance, “virtual space,” which he defines as a notion that “covers the effects of media that are not three dimensional spatial on the level of the material interface but that nevertheless receive a spatial character of depth in the perception and interpretation” (2010: 20). What Elleström means, of course, is the use of linear perspective in painting, which has reached a level of very high quality during Renaissance and has become one of the most distinctive traits of Renaissance painting in general. Virtuality can, of course, be used in all media types as a way of suggesting certain traits that are otherwise difficult, or impossible, to realize because of the limitations of the media type in question. Elleström himself mentions this possibility indirectly, when he mentions that “verbal narratives also create various sorts of virtual spatiality in the mind of the listener or reader – not only abstract, conceptual spatiality, but virtual worlds within which the reader can navigate” (2010: 20). What Elleström does not mention is that virtuality is actually at the core of the concept of the narrator in general or rather at the core of two of the main components of the concept of the narrator according to Genette’s narratological model: voice and focalization. Indeed, these are actually the most important features of the narrator in written literature, which is the main object of Genette’s study, and they answer to the two questions already mentioned above: “who speaks?” (¼ voice) and “who sees?” (¼ focalization). However, the material interface of the media type of written literature, which is at the basis of Genette model, does not allow for the existence neither of voice nor of sight, or at least not beyond the pure printed letters of the text. The written text does not actually “speak,” as Susan Lanser also puts it. “Are terms like ‘voice’ even applicable to written discourse?” she asks. “Is it not naively anthropomorphic to say that there is an author, or any person, speaking in or through the text?” (Lanser 1981: 112). What happens is rather that the written text creates the illusion of somebody speaking, sometimes so well that readers more or less hear a voice in their heads. One of the early narratologists, Franz Karl Stanzel, is convinced of this:

400

L. Lutas

“Whenever a piece of news is conveyed, whenever something is reported, there is a mediator – the voice of a narrator is audible” (1984: 4). But what really exists there is not a voice, but a written text. The voice is “virtual,” in the same way as the “virtual time” or the “virtual space” stipulated by Elleström for media types that cannot incorporate time and space. Written literature cannot incorporate sound, so the concept of voice can only have a virtual existence. Likewise, focalization, which is what is seen by the narrator, can only have a virtual existence too in written literature, since language does not have the possibility of actually showing but only of describing. By the same token, the narrator can be such a trait that can have an existence in other media types than literature, even such media types that do not have a verbal interface. And the fact that it can be difficult to find ways of suggesting a narrator could even lead to innovative and unexpected improvisations, which sometimes become conventions. The result of these reflections is that narration can be present, at different degrees and in a weaker sense, such as the sense formulated by Phelan, in many more media types than it first appears. Therefore, Elleström’s who suggestion that even media types which are not mainly narrative in their nature have narrative capacities seems valid: The term ‘transmedial narration’ should be understood to refer to all varieties of transmediality and transmediation where narration is a media characteristic that is significant enough to be observed. In the most general terms, then, the concept of transmedial narration includes the notion that an abundance of different media types share traits that give them narrative capacities. (Elleström 2019: 6)

The question is whether the concept of a narrator, which is arguable one of the most language-dependent features in written narrative literature, can have an existence outside of written literature. Below follows a short overview of some media types from the point of view of their ability to host a narrator. For the sake of clarity, the overview starts with media types that are close to written narrative literature, moving toward media types that are more remote, in order to question the transmediality of the concept of narrator.

Oral Literature and News Reporting One of the most obvious media types from the perspective of the proximity to written narrative literature is oral storytelling, sometimes called oral literature. According to the Canadian Encyclopedia: The term oral literature is sometimes used interchangeably with folklore, but it usually has a broader focus. The expression is self-contradictory: literature, strictly speaking, is that which is written down; but the term is used here to emphasize the imaginative creativity and conventional structures that mark oral discourse too. Oral literature shares with written literature the use of heightened language in various genres (narrative, lyric, epic, etc.), but

17

The Narrator: A Transmedial Device

401

it is set apart by being actualized only in performance and by the fact that the performer can (and sometimes is obliged to) improvise so that oral text constitutes an event. (https://www. thecanadianencyclopedia.ca/en/article/oral-literature-in-english)

As it appears in this definition, there are obvious connections between written and oral literature. One connection which is not mentioned above concerns the traditionally alleged historical origins of written narrative literature in the ancient art of telling stories (see, for instance, Fludernik 1996). However, there are important differences between written and oral literature, not only of the level of the material interface but from the point of view of the narrator. Is seems quite obvious that the storyteller as a person, in contrast to the author as a person, cannot avoid being the narrator of the story, since it is his or her voice that the audience hears literally. Admittedly, a storyteller can impersonate another person and can especially change his or her voice as to give the impression that he or she is a character from the story being told. But the audience cannot completely fall for the illusion, since they usually are aware of the fact that it is the same person telling the story. Certainly, in the case of oral storytelling broadcast through a medium like the radio, where the voice is heard without the person of the storyteller being seen, the illusion that there are different voices, among which the voice of a narrator who can be extradiegetic, that is from a world outside of the story being told, can become more intense and credible. The relationship between oral storytelling and written literature can be used as an asset for creating aesthetic effects in both media types. In the case of oral storytelling, an example would be the imitation of written literary features, either by simply quoting from written works or, even more interestingly, by imitating a narrator’s conventional formulas, such as “dear reader” or high literary language. More interestingly, written literature can play with its origins in oral storytelling in similar ways, by imitating formulas used traditionally by oral storytellers, such as addressing the audience and incorporating the audience’s response. Innovative examples of such practices cand be found in written literature from parts of the world where oral storytelling still has a certain importance, such as Africa or the West Indies. In the case of writers such as Patrick Chamoiseau or Raphaël Confiant, founders of the “Creoleness” movement (créolité) in the French West Indies in the 1980s, the incorporation of parts imitating oral storytelling has an ideological implication too, being seen as a tribute to the oral storytelling of the Creole slaves of the French Antilles (the conteurs) in order to explore and promote the cultural and literary notion of “creoleness” (cf Wells 1994). Chamoiseau’s play, in several of his novels, with how the narrator incorporates the conteurs’ speech is interesting from a narratological point of view, since it highlights the importance of the concept of “voice” of the narrator in a concrete way. Certainly, in the case of oral storytelling broadcast through a medium like the radio, where the voice is heard without the person of the storyteller being seen, the illusion that there are different voices, among which the voice of a narrator who can be extradiegetic, that is from a world outside of the story being told, can become more intense and credible.

402

L. Lutas

Autobiography Another interesting term of comparison, even this one from the field of written media types, is autobiography. The most important scholar who studied this field, Philippe Lejeune, has taken on the concept of the narrator as defined by Genette, certainly as a result of the fact that his study, Le pacte autobiographique (1975), was published only 3 years after Genette’s groundbreaking Narrative Discourse. For Lejeune, things are quite simple as to the persona of the narrator in an autobiography: The narrator is the same person as the author and the protagonist. This is a formula that has survived ever since, even for more complex cases of hybrid genres as autofiction (for the genre of autofiction, see, for instance, Effe and Lawlor 2022). However, the study of the narrator in autobiographical works can be interesting from other points of view, as, for example, the author’s personality. Indeed, a thorough study of the narrator in autobiographies, with the help of the same narratological tools as for the narrator of fictional works, can unearth differences as to the real-life author and the way he or she appears in other media, such as newspapers or TV recordings. Thus, as in the case of news reporting, mentioned above, the relationship to reality in the case of autobiography is a factor that generally complicates the study of a narrator. More recent studies in the field of autobiography have often open the field toward other media in interesting ways. Argentinian scholar Leonor Arfuch (2002) has, for instance, coined the concept of “biographic space” (espacio biográfico) in order to open up the field of biography toward the textual hybridization that characterizes contemporary culture, where the private individual experience has a major role and is more easily displayed. Thus, Arfuch includes in this space traditional genres such as biography, autobiography, memoirs, life stories, diaries, but also the more recent genres such as interviews, conversations, portraits, testimonies, talk-shows, realityshows, etc. This proliferation of genres and media, Arfuch argues, contributes to a more general questioning of the notion of individual “subject,” which in its turn, I argue, has major effects on the concept of the narrator and its relation to the real author. Brazilian scholar Diana Klinger (2007) comes to a similar conclusion in her studies of contemporary Latin-American autobiographical works, where the fictionalization of the author leads to a complex relationship between the author and the narrator, that Lejeune had taken for granted.

Poetry Another written media type in which the concept of the narrator is disputable is poetry. At a first sight, stipulating the existence of a narrator in a genre seemingly defined by its absence of narrativity can be erroneous. However, the existence of so-called narrative poetry is an argument against such a simple conclusion. Besides, even lyric poetry has been approached narratologically. In their article “Narration in Poetry and Drama” (2012), Peter Hühn and Roy Sommer mentioned the concept of “transgeneric narratology” and its possible virtues in the study of all kinds of poetry:

17

The Narrator: A Transmedial Device

403

“Transgeneric narratology proceeds from the assumption that narratology’s highly differentiated system of categories can be applied to the analysis of both poems and plays, possibly opening the way to a more precise definition of their respective generic specificity.” From the perspective of this article, it is not the case of the narrator in narrative poetry that is interesting, since that is a rather similar case to narrative literature. It is clearly more arguable whether lyric poetry can be studied in the same way, since it seems to lack one of the most important features requested of a narrative: narrativity. But this is a statement that should be reconsidered. Indeed, lyric poetry could be seen as incompatible with the view of narrative as the retelling of events and their succession in time. Firstly, a lyric poem is usually considered to express personal emotions or feelings rather than events (Scott 1990). Secondly, time in poetry has been seen as reduced to a single moment (Kafalenos 2006: 157–78), the moment of a revelation, for instance, hindering thus the representation of the succession of events. Meanwhile, as in the case of virtual time or space, mentioned above, time succession can be suggested even in poetry, in subtle ways. The limitation inherent in the time aspect is thus, ultimately, a challenge from the narratological point of view. Thirdly, but even more importantly from the point of view of the need of a narrator, there is the traditional view, still widespread even today, that lyric poetry is unmediated. An author, identified as the speaker of the poem, conveys an experience directly to the reader, without using the filter of a narrator, as the case is in narrative literature or arguably in narrative poetry (cf Hühn and Sommer 2012). Scholars have though put this immediacy into question, showing that it is rather an illusion (Bernhart 1993: 366–68) and that the study of a narrative voice in poetry can even put the concept of the narrator in general to an interesting test. Thus, Kraan (1991: 222–23) attempts to go beyond the question of immediacy, without stipulating a narrator though, but by using three other concepts: the empirical author, the implied author, and the “lyric subject” in poetry. According to him, these three instances could merge or diverge during different time periods (222–23). A decade later, Hühn (2004: 147–51) makes a quadripartite model of possible subjects: the biographical author, the abstract (or implied) author, the speaker/narrator, and the protagonist or character. However, even if the concept of a narrator appears explicitly in this model, it is in a weaker sense than in narrative texts, since what the narrator speaks of is not a succession of events that have occurred in a storyworld but rather personal experiences.

Comics A media type which lies a step further away from written literature, without totally departing from it, is comics. Indeed, comics shares with literature a language-based element, namely, the speech balloons that contain dialogues between characters, narration, sound effects, or other information. Conventionally, the balloons used for narrative information are different in their form compared to the balloons used for dialogues between characters. At the same time, comics contains another element, which is visual: the drawings in the panels. Typically, comics consists of a sequence

404

L. Lutas

of such panels of images and text balloons, spread over the page in different ways, but generally following the traditional reading direction, from left to the right and downward. Images and text narrate together, in combination, but there can be parts which lack words altogether. Such cases are especially interesting from a narratological point of view, since the narration is still performed, only by images, but images which are related to one another in ways that can give the illusion of time passing from one panel to the other. However, as Kai Mikkonen convincingly argues, it is hard to imagine a whole narrative in comics made up by images without words, since “a causal connection between the events is usually required, related, for instance, through a narrator or an experiencing character” (2017: 16). The question of a narrator in the case of a media type containing both text and image is especially interesting to study. One might be tempted to consider a narrator as the only entity producing the speech balloons that are narrative at an extradiegetic level, that is speech that cannot be heard by the characters in the comics, but only read by us, readers. As mentioned above, usually these speech balloons are formally different from the speech balloons containing dialogues at the diegetic level. In other words, in the case of comics, a visual element, namely, the speech balloons, helps in distinguishing the narrative levels, which are to be constructed by the readers in their heads in the case of narrative literature. The fact that a narrator can be identified so easily by the use of special speech balloons does, however, not mean that the question of the narrator in comics in general is settled. This is rather only a special case, corresponding in some ways to the voice-over narrator in films. On the contrary, the problem of a narrator in comics is very complex and has been the subject of a number of narratological studies which have led to several models. The majority of these models are shaped on a combination of literature narratology and film narratology, with aspects stemming from visual studies, indermediality, and multimodality in general (cf Mikkonen 2017: 9). As such, the studies of the narrator in comics are a good example of the problematic expansion of the concept of “narrator” to other media types. One of the main names in the field, Belgian scholar Thierry Groensteen (2011: 86–87), considers, for instance, that the application of narratology, and especially the theory of the narrator, to the field of comics studies, should not be done without certain changes and revisions of the concepts. The result are a number of separate theories of narrators in comics, based partly on the narrator in films, neither of which having imposed itself as Genette’s model had succeeded in the field of written narrative literature. As I already mentioned, film narratology has an important influence on comics narratology. It is especially French-Canadian scholar André Gaudreault’s seminal narratological work from the 1987, Du littéraire au filmique. Système du récit, translated into English in 2009 as From Plato to Lumière: Narration and Monstration in Literature and Cinema, that has laid the ground for comics narratology, especially his redefinition of Plato’s concept of “mimesis” as “monstration.” A film narrative, according to him, is the result of two basic modes of narrative communication: “narration” and “monstration” (Gaudreault 2009: 7). Narration in this context is not as much the voice-over, used in a small number of films, and

17

The Narrator: A Transmedial Device

405

therefore quite a marginal phenomenon, but the way in which individual shots can be related to one another in such ways as to form a story (Gaudreault 1987, 31). The verbal ingredient is one important part of the narration, since language is an effective way to link shots in a causal way. “Monstration,” on the other hand, is what is communicated by the image in itself, not in relation to other shots and, thus, not in the sequencing of the images. It has to do with a sense of the present in the evolving events (1987: 31). These two modes of verbal and visual narration are assigned to two distinct entities in Gaudreault’s model: the narrator and the monstrator. These two entities are placed in their turn under an encompassing narrator at a higher level of cinematic narration, called the “mega-narrator,” or “grand image-maker,” by Gaudreault. It is this narrator’s global responsibility to shape the mise-en-scène, the set design, the lighting, and the acting (Gaudreault 2009: 72) and to coordinate the different sources of expression, such as images, sounds, words, and music. This narrator corresponds quite well to Seymour Chatman’s version of the cinematic narrator, defined in a depersonalized way as “the composite of a large and complex variety of communicating devices” (1990: 134). Coming back to comics, it is not hard to see why these theories of the narrator in film have been so influential even for comics narratology. The two main modes of narrative communication, “narration” and “monstration,” are easy to apply to a media type which also consists of verbal and visual narration. Therefore, it was not surprising that only a couple of years after Gaudreault’s work from the 1980s, a scholar applied the model to comics. The scholar who did this was Belgian Philippe Marion, in his reworked doctoral dissertation, Trace en cases (1993). On the basis of Gaudreault’s cinematic model, Marion claimed that there is a tension in comics between the narrator and the monstrator or rather between the narrative function and the painterly function, or even simpler, between the text and the image (1993: 212). The painterly function makes the reader concentrate on the image as isolated from the narrative continuum, while the narrative function encourages the reader to look beyond the individual picture and to try to grasp the narrative movement. In Marion’s view, drawings in comics are harder to grasp than pictures in cinema. The narrative part, on the other hand, is similar to the one in cinema, since pictures follow one after the other sequentially, as in cinema, but in a way that allows the readers to stop at their own will, in order to be able to concentrate on the more artistic feature of the images. What is important to understand is that in comics, these two functions, the “monstration” and the “narration,” are not completely separated. As in film, they are both a part of a higher narrative agency, or a mega-narrator, which integrates the two functions in a global entity. That entity would correspond to the narrator of a novel but is more complex because of the need to integrate the narration at the abstract level with the more concrete aspect of the images and their layout on the page. That kind of narrator goes beyond, in some ways, the mere abstract narrator of a written narrative, moving toward a plural narratorial consciousness, which is responsible for the whole narrative organization. This plurality reminds of the cinematic narrator, who is difficult to connect to one single creator, when thinking of the complexity of a filmic creation which involves many different persons. In

406

L. Lutas

comics, the plurality is rather a duality, a writer and an illustrator, but their collaboration leads to a creative form of tension inherent in the concept of narrator.

Film Having highlighted the complexity of the concept of narrator in comics, it could now be appropriate to take on a media type which, narratologically, as I mentioned above, is close to comics: film. In comparison with written literature, on the other hand, film is further away than the other media types studied until now: oral literature, poetry, and comics. However, film is a complex media type, able to incorporate other media types, among which all the ones studied above. It is true that the temporal media affordance of a film, being formed of moving images, does not seem to fit the atemporal feature of photographs or paintings. However, film is versatile and possesses, for instance, the ability to freeze the flow of the images and to show a photograph or a painting as frozen in time. Likewise, film can incorporate written literature, for instance, by simply showing the pages of a book or by letting somebody read out the book, which would correspond to a transmediation, according to Elleström’s terminology (2014). Film can incorporate also theatre, by simply filming a theatre scene, or poetry, in the same way as for written literature. This versatility is based on the fact that film consists of both images, written text, and sound. However, what is interesting from the perspective of an examination of the narrator in film is not the cases where a narrator from other media type is imitated, such as the voice-over of a narrator, or a simple representation of a literary work. It is rather the complex concept of the film narrator that I already started to approach in the overview of comics above. And from this perspective, André Gaudreault’s already mentioned study is illustrative. I will not insist more, however, on the two entities already discussed above: the monstrator and the narrator. Rather, I will discuss the problematic aspect, already hinted at in the other media types studied, whether there exists a narrator in film at all, except for the cases of voice-over narration. Indeed, given the traditional connection of the concept of the narrator to a spoken or written discourse, mentioned at the beginning of this article, a narrator who does not narrate in words is not a proper narrator, which would put into question the possibility of a cinematic narrator. The way out of this problem is by acknowledging that narration can be performed even with other means than simply verbal ones, such as visual means, editing, and sound. Therefore, as we have seen in the case of comics, the possibility of a more complex narrator has been introduced, a narrator who takes the responsibility for the use of all the narrative means, a mega-narrator. Or rather an “audiovisual narrative instance” (Kuhn 2009), which brings together cinematographic devices (camera, editing, sound) and mise en scène (the arranging and the composition of the scene filmed by the camera). What film, thus, contributes to in relation to literature is to underscore the fact that the narrator does not need to be a single anthropomorphic instance. Genette’s

17

The Narrator: A Transmedial Device

407

approach to the narrator in narrative literature as two separate constituents, voice and focalization, is mirrored in film by the role of the camera as the mediator who both sees and hears. And indeed, some, such as the French director Alexandre Astruc, who coined the term “caméra stylo,” equate the camera with the narrator (Astruc 1948). But such a view would mean that cinema and literature are very similar media types and that cinema possesses a language which is similar to the language of literature. However, even if the notion of “language” has been used in relation to cinema, it has been a rather weak analogy to usual language, as a semiotic system or a code, of its own (see for instance Metz 1966 or Chatman 1990). In such a perspective, the concept of “narrator” in film would only be used metaphorically, in the same way as “language” is used in relation to film. Meanwhile, if the concept of the “narrator” is to be taken as a construction in the receivers’ minds, the analogy to literature would not be so far-fetched after all. Indeed, even in literature, as we saw in Margolin’s definition at the beginning of this article, the narrator can be seen as an instance of a mediator who is not only telling a story but rather occupies an abstract position which the reader can use too in his reading process. The same can be said for film, which is also built on a similar communicational model, according to which the viewer has to construct a story on the basis of the clues from the film. In such a view, the narrator is a virtuality, as in literature too, which helps the viewer construct the story and all its elements. The camera would, thus, not only be the instance which tells the story and sees the events but would rather represent the virtual narrator, and, thus, the virtual position to be occupied by the viewer in relation to the story.

Painting This final point can be applied to another media type where the narrator cannot have an existence other than virtual: painting. First of all, painting, like static pictures in general, lacks the possibility to incorporate the temporal element, one of the four elements which define a story according to Ryan’s contention. In a way that reminds of poetry, but even more extremely, a painting captures a frozen moment in time, instead of a sequence of moments. Narrativity is a problematic feature in a media type which cannot represent a temporal sequence. This is a point made already in 1766 by German literary theorist Gotthold Ephraim Lessing, in his essay “Laocoon: On the Limits of Painting and Poetry,” where he emphasized, against earlier views, that painting could not achieve the same things as poetry, because of its material limitations regarding temporality. However, this lack has been turned quite successfully in an aesthetic asset, painters being forced to find ways to suggest the passing of time. This is already mentioned above, in the discussion of Elleström’s concept of “virtual time” in relation to static pictures in general, that is “the capacity of individual pictures to depict not only one static moment but a series of occurrences” (2010: 21). Werner Wolf (2005) follows a similar line of thought, but more specifically adapted to paintings, in his concept of “pictorial narrative,” meaning paintings which convey stories, despite their medial

408

L. Lutas

incapacity of representing narrative elements such as time. Wolf distinguishes three such kinds of pictorial narratives: monophase works that represent one frozen moment in a story through a single image, polyphase works that represent several moments within the same image, and finally a series of pictures that capture a sequence of events. The last kind of pictorial narratives, the series of pictures, is very similar to the use of panels in comics, and thus to film, and can be studied with the same tools even from the point of view of the existence of a narrator. Polyphase and especially monophase pictorial works are harder to study from a narrativity perspective and, thus, also from the point of view of the possible use of a narrator. However, as Wolf puts it (2005), this narrativity deficit can generate a curiosity in the viewer, who can be encouraged to put the frozen moment from the painting in a continuum, by filling gaps about the past and the future. The viewer’s narrative capacity is, thus, put to a harder test by a pictorial narrative than by a verbal narrative. Even if these interesting ideas about the narrativity of paintings have not generally been applied to the concept of a possible narrator, it can be argued that by the mere mention of narrativity, and thus, by a communicational context, the question of the narrator cannot be avoided. However, the material differences between the two media types, and especially the lack of a verbal element in painting, exclude the existence of a narrative voice, even in the form of a voice-over, as in film. Admittedly, a narrator could be represented visually in a painting, in ways that could even suggest a difference between the narrative levels, but this could only be done either by indicating it in the title of the painting or by following certain conventions, or schemata. Indeed, painting does not in itself have the capacities of differentiating between narrative levels. Interestingly enough, this narrative deficit is highlighted in an animated film about a painting: Jean-François Laguionie’s Le tableau (The painting) from 2011. Both temporality and narrative levels are added in innovative ways to the painting represented in the film, by making it come to life after letting the camera break the frame and enter the world of the painting. However, beyond the innovative aspects of this film, what is thus shown are the virtualities inherent in a painting, in a concrete way which could mirror what happens in the viewers’ heads. One of the most spectacular additions is the character of a narrator, who addresses the viewer at an extradiegetic level, for instance, at the very moment when the frame is broken and the camera enters the world of the painting. “Voilà ! Vous venez de pénétrer dans le tableau” (“Here you are! You just entered the painting”), the narrator says, before jumping through the frame and becoming one of the characters inside the painting. What is especially highlighted, thus, is that a narrator seems to be a virtual possibility in such cases, in similar ways to narrators that are not represented in literature or those narrators whose location is suggested by the location of the camera in film. In paintings too, the technique of perspective has been perfected in such a way as to create a virtual point which would correspond to the place of an observer watching the scene in the painting. This virtual point would correspond to the point where focalization is placed in literature, which is the point from which the narrator perceives the events of the story. What this shows, in conclusion, is that in painting,

17

The Narrator: A Transmedial Device

409

not less than in literature and in film, the point of view is above all a virtuality that is to be concretized by the listeners/viewers/readers on the basis of the clues from the media product. And even if the narrator in painting has a clearly less prominent place, especially when it comes to the possibility of being perceived as a human being conveying a story, it can be a tool that helps (re)constructing the meaning of the media product.

Music This leads to a final media type to study, arguably the most remote narratively from a language-based narrative media type: instrumental music. I choose to concentrate on instrumental music rather than music where words are involved because what is interesting in the perspective of this article is to analyze the capacity to incorporate a narrator in a media type where verbal language is absent, or almost absent, except for the title. Before discussing any form of narrator in instrumental music, the question is whether narrativity has any relevance at all for this media type. Indeed, even if music consists of sound, it is not the same kind of sound as in spoken language. As MarieLaure Ryan puts it (2012), instrumental music consists rather of “pure sounds,” which are sounds without either symbolic (conventional) or iconic meaning in themselves. Music, Ryan argues, cannot, for instance, imitate speech nor can it represent thoughts, narrate events, or express causal relations, at least not in an indisputable way for all listeners. Indeed, there is always a large element of individual interpretation in music, even when it comes to the similarity of the sounds with sounds in the real world. That is even more applicable for narrativity in music, which, as Werner Wolf argues (2005), cannot be literal or determinate. It is rather indeterminate, exactly because it is always disputable. Indeed, as Nattiez (1990) claims, some listeners do not even pay attention at all to a presumed story, something that is rarely the case in a language-based narrative. However, music is actually very well apt to convey other elements of a narrative. Going back to Ryan’s four elements constituting a narrative, it is rather clear that music is especially good at conveying the formal element, meaning the building of a tension and a closure, but also the temporal elements, as Elleström convincingly argues (2019: 120). The spatial element is more problematic. Indeed, even if a sense of spatiality can be suggested, simply by the way the instruments are placed (Elleström 2019:120), this is quite far from the complex criterion demanded by a storyworld populated by characters and objects. Besides, as Ryan points out, even if the temporality of music is suited to imitate the temporality of a story, the causality between the events that music might succeed to imitate cannot be established in an undisputed way, or rather not at all. What can be suggested, though, is the existence of events or characters, a musical theme or an instrument, or even a note or a pitch, corresponding for instance to an event or a character, and a relationship between these, for instance by building up tensions at a formal level between them (cf Tarasti 1994).

410

L. Lutas

It is through this last point, the tension in music between what can be presumed to be represented events or characters that the existence of a narrator can be tentatively formulated. Indeed, as Elleström suggests, in order for a listener to interpret the different music themes or instruments as events or characters and to relate them to one another, “it might be necessary to construe an overarching virtual narrator being responsible for, say, the representation of harsh clashes between existents and events” (2019: 123). Some theorists, like Robert Hatten (1991), have even proposed the existence of a narrator’s discourse, making an analogy between musical themes and a narrative voice. Hatten based his analogy on the extreme contrasts in music that could be seen as shifts in “level of discourse.” A sudden change in mood, argues Hatten, may place “all of the previous musical discourse in a new perspective” (1991: 88). This would correspond to narrators in literature introducing new and contrasting events or representing earlier events in new perspectives. However, appealing such theories as Hatten’s of a narrator in instrumental music might seem, it would be a mistake not to take the difference between the two media types, music and narrative literature, into consideration. Indeed, most of the conclusions about a narrator are based on analogies to literature, and few listeners would even go into that direction if there weren’t an indication in the title. Besides, there are quite few examples of adaptations of literary work into pure instrumental musical works. Usually, such adaptations are made rather into musicals, operas, or other musical genres where words are involved too. A sonata like Beethoven’s The Tempest is not even undoubtedly an adaptation of Shakespeare’s play, showing how arbitrary such connections really are. Consequently, if a narrator were to exist in instrumental music, it is rather of the overarching kind, not completely different from the mega-narrator from film and comics. Such a narrator is an abstract construction in a listener’s head, on the basis of either indications in the title or interpretations of contrasts between musical themes.

Conclusion This overview could continue with many other media types, which could all contribute to nuancing the concept of the narrator and its transmediality. It suffices to mention sculpture and the dramatic tension in the ancient Greek group statue of Laocoon and his sons, now at the Vatican Museums, which inspired Lessing’s theory on the differences between plastic arts and literature, to realize how narration can be sublimated in a frozen moment and how a mega-narrator can be construed in the beholder’s head on the basis of the interpretation of what comes before and especially after this moment. It is, admittedly, an abstract narrator, an imaginary construction, and a virtuality, but as in the media types of painting, comics, or music studied above, it helps the receiver construct or reconstruct a narrative meaning. However, even in literature, the narrator is an abstract construction in the reader’s heads, as we saw in Margolin’s definition at the beginning of the chapter, and it is only in cases when the narrator actually speaks in the first person or uses deictic adverbs such as “here” and “now” that the virtuality of his or her existence becomes

17

The Narrator: A Transmedial Device

411

concrete. Admittedly, as narratologists as Genette and Bal have argued, a narrator in literature can always refer to himself or herself as an “I,” but the reader is actually not aware of this until it really happens and construes a figure of a narrator in his or her head on the basis of earlier experiences, or conventions. Since all media, especially aesthetic media, are to be considered in a communicational context similar to the one in literature, it comes as no surprise that the receivers always look for an entity that brings together the details into a meaningful coherence. That entity can easily be called “the narrator.” However, such a concept of “narrator” is more flexible than its initial definition and obligatory connection to written narrative literature.

References Arfuch, Leonor. 2002. El espacio biográfico: dilemas de la subjetividad contemporánea. Buenos Aires: Fondo de Cultura Económica. Astruc, Alexandre. 1948. Naissance d'une nouvelle avant-garde: la caméra-stylo. L'Écran Français 144: 5–6. Bal, Mieke. 1997. Narratology: Introduction to the theory of narrative. Second edition. Toronto: Toronto Press Incorporated. Barthes, Roland. 1977 [1966]. Introduction to the structural analysis of narratives. In Image— Music—Text, trans. Stephen Heath, 79–124. New York: Hill and Wang. Bernhart, Wolfgang. 1993. Überlegungen zur Lyriktheorie aus erzähltheoretischer Sicht. In Tales and ‘their telling difference’: Festschrift für Franz K. Stanzel, ed. H. Foltinek et al., 359–375. Heidelberg: Winter. Bordwell, David. 1985. Narration in the fiction film. London\New York: Routledge. Bremond, Claude. 1964. Le message narrative. Communications 4: 4–32. Chatman, Seymour. 1990. Coming to terms: The rhetoric of narrative in fiction and film. New York \London: Cornell University Press. ———. 1978. Story and discourse: Narrative structure in fiction and film. Ithaca\New York \London: Cornell University Press. Collobert, Catherine. 2013. Literariness in Plato: Narrators and narrative modes in the dialogues. Revue de Métaphysique et de Morale 80 (4): 463–476. Effe, Alexandra, and Hannie Lawlor, eds. 2022. The autofictional: Approaches, affordances, forms. Cham: Palgrave Macmillan. Elleström, Lars. 2019. Transmedial narration: Narratives and stories in different media. Cham: Palgrave Macmillan. ———. 2018. A medium-centered model of communication. Semiotica 224: 269–293. ———. 2014. Media transformation: The transfer of media characteristics among media. Houndmills: Palgrave Macmillan. ———. 2010. The modalities of media: A model for understanding intermedial relations. In Media borders, multimodality and intermediality, ed. Lars Elleström, 11–48. Basingstoke: Palgrave Macmillan. Fludernik, Monika. 2003a. Scene shift, metalepsis, and the metaleptic mode. Style 37 (4): 382–400. ———. 2003b. Commentary: Narrative voices—Ephemera or bodied beings. New Literary History 32: 707–710. ———. 1996. Towards a ‘natural’ narratology. Vol. 25, 97. London\New York: Routledge. Gaudreault, André. 2009. From Plato to Lumière: Narration and Monstration in literature and cinema, trans. Timothy Barnard. Toronto: University of Toronto Press. ———. 1987. Narration and Monstration in the cinema. Journal of Film and Video 39 (2): 29–36. Genette, Gérard. 1988 [1983]. Narrative discourse revisited, trans. Jane E. Levin. Ithaca\New York: 892 Cornell University Press.

412

L. Lutas

———. 1983. Nouveau discours du récit. Paris: Seuil. Groensteen, Thierry. 2013 [2011]. Comics and narration, trans. Ann Miller. Jackson: University Press of Mississippi. Hatten, Robert. 1991. On narrativity in music: Expressive genres and levels of discourse in Beethoven. Indiana Theory Review 12: 75–98. Hayles, N. Katherine. 2001. The transformation of narrative and the materiality of hypertext. Narrative 9 (1): 21–39. Ohio State University Press. Herman, David. 2010. Narrative theory after the second cognitive revolution. In Introduction to cognitive cultural studies, ed. L. Zunshine, 155–175. Baltimore: Johns Hopkins UP. ———. 2004. Toward a transmedial narratology. In Narrative across media: The languages of storytelling, ed. Marie-Laure Ryan, 47–75. Lincoln\London: University of Nebraska Press. Hühn, Peter. 2004. Transgeneric narratology: Applications to lyric poetry. In The dynamics of narrative form, ed. J. Pier, Berlin: de Gruyter, 139–58. Hühn, Peter, and Roy Sommer. 2012. Narration in poetry and drama. In The living handbook of narratology. https://www.lhn.uni-hamburg.de/node/40.html Jost, François. 2004. The look. From film to novel: An essay in comparative narratology. In A companion to literature and film, ed. Robert Stam and Alessandra Raengo, trans. Robert Stam, 71–80. Malden, MA: Blackwell. Kafalenos, Emma. 1996. Implications of narrative in painting and photography. New Novel Review 3: 53–64. ———. 2006. Narrative Causalities. Columbus: Ohio State UP. Klinger, Dian. 2007. Escritas de si, escritas do outro. Rio de Janeiro: 7letras. Kraan, Menno. 1991. Towards a model of lyric communication: Some historical and theoretical remarks. In Russian Literature 30: 199–230. Kuhn, Markus. 2009. Film narratology: Who tells? Who shows? Who focalizes? Narrative mediation in self-reflexive fiction films. In Point of view, perspective, and focalization: Modeling mediacy in narrative, ed. P. Hühn et al., 259–278. Berlin: de Gruyter. Kukkonen, Karin. 2011. Comics as a test case for transmedial narratology. SubStance 40: 34–52. Lanser, Susan Sniader. 1981. The narrative act: Point of view in prose fiction. Princeton: Princeton Univ. Press. Lejeune, Philippe. 1975. Le Pacte autobiographique. Paris: Seuil. Lubbock, Percy. 1954 (1922). The craft of fiction. London. J. Cape. Margolin, Uri. 2012. Narrator. In The living handbook of narratology. https://www.lhn.unihamburg.de/node/44.html Marion, Philippe. 1993. Traces en cases. Travail graphique, figuration narrative et participation du lecteur. Essai sur la bande dessinée. Louvain-la-Neuve: Université Catholique de Louvain: Academia. Meelberg, Vincent. 2006. New sounds, new stories: Narrativity in contemporary music. Leiden: Leiden University Press. Metz, Christian. 1966. La grande syntagmatique du film narratif. Communications 8: 120–124. Mikkonen, Kai. 2017. The narratology of comic art. New York\London: Routledge. ———. 2011. Graphic narratives as a challenge to transmedial narratology: The question of focalization. Amerikastudien/American Studies 56: 637–652. Nattiez, Jean-Jacques. 1990. Can one speak of Narrativity in music? Journal of the Royal Musical Association 115: 240–257. Neitzel, Britta. 2007. Metacommunication in (computer)games and play. In Self-reference in the media, ed. W. Nöth and N. Bishara, 237–252. Berlin/New York: de Gruyter. Nünning, Ansgar. 2001. Mimesis des Erzählens. In Erzählen und Erzähltheorie im 20. Jahrhundert, ed. J. Helbig, 13–47. Heidelberg: Winter. Patron, Sylvie. 2009. Le Narrateur. Introduction à la théorie narrative. Paris: Armand Collin. Phelan, James. 2005. Living to tell about it: A rhetoric and ethics of character narration. New York: Cornell University Press.

17

The Narrator: A Transmedial Device

413

Psarra, Sophia. 2009. Architecture and narrative. The formation of space and cultural meaning. London\New York: Routledge. Ranta, Michael. 2013. (Re-)creating order: Narrativity and implied world views in pictures. Storyworlds 5: 1–30. Ryan, Marie-Laure. 2013. Transmedial storytelling and transfictionality. Poetics Today 34: 361–388. ———. 2012. Narration in various media. In The living handbook of narratology. https://www.lhn. uni-hamburg.de/node/53.html ———. 2007. Toward a definition of narrative. In The Cambridge companion to narrative, ed. D. Herman, 22–35. Cambridge: Cambridge UP. ———. 2006. Avatars of story. Minneapolis\London: University of Minnesota Press. Scott, Clive. 1990. Vers libre: The emergence of free verse in France, 1886–1914. Oxford, UK: Clarendon Press. Seaton, Douglas. 2005. Narrative in music: The case of Beethoven’s ‘tempest’ sonata. In Narratology beyond literary criticism: Mediality, Disciplinarity, ed. Jan Christoph Meister, Tom Kindt, and Wilhelm Schernus, 65–81. Berlin\New York: Walter De Gruyter. Stanzel, Franz. 1984. Theory of narrative. Translated by Charlotte Goedsche, Cambridge, Cambridge University Press. Tarasti, Eero. 1994. A theory of musical semiotics. Bloomington: Indiana University Press. Thompson, Kristin. 2003. Storytelling in film and television. Cambridge, MA: Harvard University Press. Thon, Jan-Noël. 2016. Transmedial narratology and contemporary media culture. Lincoln\London: University of Nebraska Press. Wells, Catherine. 1994. L’oraliture dans Solibo Magnifique de Patrick Chamoiseau. Québec: Grelca. Wolf, Werner. 2005. Pictorial Narrativity. In The Routledge encyclopedia of narrative theory, ed. D. Herman et al., 431–435. London: Routledge. ———. 2003. Narrative and narrativity: A narratological reconceptualization and its applicability to the visual arts. Word & Image 19: 180–197. ———. 2002. Intermediality revisited: Reflections on word and music relations in the context of a general typology of Intermediality. In Word and music studies, ed. Suzanne M. Lodato, Suzanne Aspden, and Walter Bernhart, 13–34. Amsterdam: Rodopi.

Intermediality, Teaching, and Literacy

18

Ana Cla´udia Munari Domingos, E´rika Viviane Costa Vieira, Miriam de Paiva Vieira , and Camila Augusta Pires de Figueiredo

Contents The Same First Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (Inter)mediality and Literacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Systematizations for Educational Purposes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality and Teaching: Reading Media or Simply “Reading” . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

416 420 422 427 431

Abstract

Literacy and learning to read media forms have been somewhat underestimated, even if the wide-ranging debates about the conceptualization of the notion of intermediality involving the interrelationships between arts and media have been fruitful and achieved the status of an academic field. The concern with reading the media started to be part of the curricula around the 1980s, with the introduction of television in classrooms, in the context that Santaella calls media culture; however, these actions were often thought of from an intertextual perspective, and the systematic introduction of intermediality studies in pedagogical practices still tends to be subordinated to broader fields, such as comparative literature, film and media studies, and even art history. Among the initiatives to systematize the A. C. Munari Domingos (*) Universidade de Santa Cruz do Sul, Santa Cruz do Sul, Brazil e-mail: [email protected] É. V. C. Vieira (*) Universidade Federal dos Vales do Jequitinhonha e Mucuri, Diamantina, Brazil e-mail: [email protected] M. de Paiva Vieira Universidade Federal de São João del-Rei, São João del-Rei, Brazil e-mail: [email protected] C. A. P. de Figueiredo Universidade Federal de Minas Gerais, Belo Horizonte, Brazil e-mail: camilafi[email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_21

415

416

A. C. Munari Domingos et al.

teaching of intermediality, Semali and Pailliotet’s stands out, which, at the turn of the century, points to the need to mobilize critical reading and writing skills with and through different systems of signs (which then come to be considered forms of media). Rajewsky, Wolf, and Elleström propose to integrate (inter)mediality into literary studies. To Bruhn, Bruhn and Gjelsvik, and Bruhn and Schirrmacher, intermediality is treated as an object and method of potential investigation, as well as a theory. Meanwhile, both Cutchins and Hallet provide attempts at systematization, the first seeking to take advantage of adaptations to teach literature and the latter aiming at an intermedial analysis of literary objects. To understand the entry of intermediality studies in pedagogical practices, and departing from the proposal of Semali and Pailliotet, the objective of this chapter is to assemble and comment published works written in the English language on literacy and the systematic teaching of intermediality. Keywords

Mediality · Intermediality · Education · Intermedial literacy · Critical media literacy

The Same First Questions A wide-ranging number of debates about the conceptualization of the notion of intermediality involving the interrelationships between arts and media have been fruitful and achieved the status of an academic field, whereas literacy and learning to read media forms have been somewhat underestimated. Irina Rajewsky (2005), Werner Wolf (2008), and Lars Elleström (2021) propose to integrate mediality and intermediality into comparative literature, since all media share similarities and differences that, when observed in a comparative way, can collaborate to understand the meaning of media products. To Jørgen Bruhn (2016), Bruhn and Anne Gjelsvik (2018), and Bruhn and Beate Schirrmacher (2021), intermediality is treated as an object of study, a method of potential investigation, as well as a theory. Meanwhile, both Dennis Cutchins (2010) and Wolfgang Hallet (2015) provide attempts at systematization, the first seeking to take advantage of adaptations to teach literature and the latter aiming at an intermedial analysis of literary objects. However, the entry of intermediality studies in pedagogical practices is only inaugurated by Ladislau Semali and Ann Pailliotet (1999) who, at the turn of the century, point out to the need to mobilize critical reading and writing skills with and through different systems of signs (which then come to be considered forms of media) due to the emergence of digital media. “What is intermediality and why study it in US classrooms?”, the title of the first chapter of Semali and Pailliotet’s book – Intermediality: The Teachers’ Handbook of Critical Media Literacy (1999) – raises a question that could be extended to several other countries. The interrogation brings up two questions to which we should have had concrete answers a long time ago. The first of them, on the definition of

18

Intermediality, Teaching, and Literacy

417

intermediality, is still repeated by researchers who are unaware of the field of studies but, above all, distrust the practice of understanding texts and genres as media, since the word “media” remains associated with the field of communication and is frequently used to refer to mass media, journalism, or a technological apparatus. The second part of the question is directly related to the first one, because, while the concept of intermediality is unknown in its complexity, it was, in the same way, understood to be a name given to interactions between gadgets, or that relates only to issues that are intrinsic to technical mediality. Another question raised by the authors in that first chapter, and which matters here for obvious reasons, has to do with the relationship between changes in the media context in recent decades. With technological evolution, should intermediality be taught and integrated to school curriculum? Is it possible for intermediality to be understood as a methodology for reading media in the sense of an intermedial pedagogy (Semali and Pailliotet 1999)? In the direction of pedagogical proposals suitable for learning to read media in contemporary times, we understand literacy as a condition that can be evaluated in view of different reading skills, including the awareness of mediality and intermediality based on the already-given notion that all media are interrelated. In this sense, intermediality becomes a methodological operator. In this chapter, departing from the proposal of Semali and Pailliotet, and assembling published works on literacy and the systematic teaching of intermediality written in the English language, we seek to answer these questions based on the connection between teaching, literacy, and the relations between media. For this purpose, we mean to assemble published research on the systematic teaching of intermediality and tackle these initial questions by addressing the concepts of media, reading, literacy (and digital literacy), and intermediality.1 In the first section, the elements that are the basis of Semali and Pailliotet’s discussion are presented, starting from the same contextualization of the problem raised by the authors: the importance of a definition of intermediality in order to associate it with literacy. It is at this moment that the many questions that condition the inclusion of intermediality in a school pedagogy or even its understanding as a methodology for reading the media are listed. In this first section, a synthesis of Semali and Pailliotet’s ideas is presented, placing it as an axis for the discussion of binomial intermediality and teaching. In the second section, based on Werner Wolf, in a comparison with Irina Rajewsky and Lars Elleström, some definitions2 of media, mediality, intermediality, and literacy, through the perspective of literary studies and semiotics, are aligned. In this section, we attempt to show how these concepts can provide the analysis of media products from different forms of culture.

1

Since the terminology used by authors herein studied is alike, yet presents some subtle differences, we compiled a synthesizing chart by the end of the chapter. 2 Since the terminology used by these authors is alike, yet presents some subtle differences, we compiled a synthesizing chart by the end of the chapter.

418

A. C. Munari Domingos et al.

In the third section, there is a brief history of some of the main attempts at the inclusion of media and their relationships in education, by Dennis Cutchins, Wolfgang Hallet, Jørgen Bruhn, Anne Gjelsvik, Beate Schirrmacher, and Lars Elleström. In this section, it is clear after all that, even if from different perspectives or with different bases, the same question is imposed: thinking about intermediality in a pedagogical bias in school reading involves, above all, distinguishing an adequate terminology that begins with the definition of the term “media.” Whether from specific areas such as cinema, or understanding intermediality from the broad field of communication, the relationship between intermediality and literacy is based on this concept of media and its features. In the last section, we close the chapter by returning to the initial questions posed by Semali and Pailliotet; the basic question is consolidated: understanding media is a sine qua non condition for critical reading. Finally, as a way of contributing to the theoretical delimitations that are precisely at the basis of the problem, we present a small comparative glossary of the main concepts discussed throughout the chapter. Before proceeding in the direction of these concepts – and already contributing to their definition – it is necessary to show that in the work of Semali and Pailliotet, one can perceive certain limitations to the term intermediality that we still consider problematic and sometimes reductive, such as those observed in the foreword of the book by series editors Kincheloe and Steinberg (1999, vii): They understand that the mediascape and its omnipresence in the lives of students demands a reconceptualization of literacy that expands the concept to include facility with the communicative arts. (. . .) The facility includes the ability to read and make meaning of language arts as well as the visual representations of drama, art, film, video, and television.

On the one hand, the ubiquity of media, which has become routine in contemporary times, requires, for the authors, that the definition of literacy encompass “artistic” communication practices; on the other hand, when enumerating what the authors understand as the aggregation of these new artistic genres, we realize that first, there is an emphasis on the visual issue that ends up confusing the edges of these genres: can’t the visual representation of drama be art? How are television and video different? What about cinema? Finally, do these changes in the definition of literacy contribute to a reading of mass media including journalism? As the authors make us see, some terms used in education are broad but vague, while other narrower ones are linked to mindsets that can be exclusive, prescriptive, and even dogmatic or irrelevant (Semali and Pailliotet 1999, 312). Coming from different fields, they do not contribute to an interdisciplinary perspective of the communication sphere. Despite what we consider a terminological mismatch (rather than a conceptual one), the meaning given to intermediality is broad – as we understand it should be when we deconstruct the word inter/media. For the authors, intermediality contributes to understanding the media as a sociopolitical tool. In this sense, for them, intermediality makes it possible to observe how the media are used to manage society, reinforcing power structures or making them visible (for a social change). Although sometimes there seems to be a direction toward digital media when we think about the daily practices of young students, intermediality demarcates the

18

Intermediality, Teaching, and Literacy

419

space of social interactions. In addition, it involves not only facilitating the dynamics of learning in the classroom, considering texts and genres known by students, but also building citizenship beyond school walls. In this sense, when we talk about media, we talk about all human forms of communication; when we talk about reading, we talk about human life; when we talk about literacy, we talk about learning to live. Hence, intermediality becomes not only a pedagogy for literacy, as proposed by the authors, but also a tool for reading media in all communication practices. Semali and Pailliotet (1999) provide the conceptual framework for their volume in the introduction. It must be pointed out that the initiative to publish the book arose from the National Reading Conference (NRC): Media Literacy Group when facing the rise of mass media and technology in educational contexts in the USA. According to the editors, the term “intermediality” ensued from the authors’ concerns with the changes in literacy conceptualizations, theories, and educational contexts, including conditions for instruction and learning. For them, “intermediality” can stimulate thinking and learning of multiple texts in students and provide scholars with a methodology to read printed and visual representations; “intermediality,” in this perspective, requires expertise in understanding and generating printed and visual media. Their conceptualization of “critical media literacy” is constructivist and departs from Paulo Freire’s3 notions of emancipatory education and the critical theory of the Frankfurt school (17–18) in their willingness to propose pedagogical actions with the aim of transforming lives: “our goal is to show how critical scholars and educators construct dynamic middle grounds, embracing relations among theory and practice it” (3). Thus, the critical pedagogy background that unfolds throughout the book’s chapters reasserts that a critical education leads to democratic public life. The approach follows the critical thinking tenets proposed by Jacques Derrida (1986), Paul Althusser (1986), and John Fiske (1989) in that their textual approach is broad and inclusive and reflects political issues, since “texts are not value-neutral, unchanging, ‘objective’ artifacts; they convey shifting meanings and reflect cultural ideologies” (4). Their views on media and schooling are consonant to Marshall McLuhan’s warning that “schools may be irrelevant and boring for students if they do not open up their curriculum and instruction to include today’s technologies and media with the accompanying learning styles individuals bring to classrooms” (9). With this perspective in mind, the authors mean to bring up a model of critical media literacy instruction. Regarding the historical moment when the volume was launched, the internet was a novelty and electronic devices meant technological revolution at its very beginning. Educators did not know how to deal, define, or teach electronic texts because of their ghost-like materiality: texts existed, but they couldn’t be touched on a paper surface. It is not surprising thus that their notion of intermediality is presented rather incipiently and optimistically. Additionally, we cannot forget that the idea of text was expanding and changing

3

A reference to Paulo Freire’s notion of critical pedagogy. Paulo Freire (1921–1997) was a Brazilian educator and philosopher who wrote Pedagogy of the Oppressed (1968), a foundational text of critical pedagogy. The tenets of this field of knowledge claim for an education free of a colonized culture and mind.

420

A. C. Munari Domingos et al.

because of the spread of the internet around the world: now there were visual texts, filmic texts, and electronic texts, among other categories, coexisting simultaneously. Therefore, intermediality at the time was a term under construction, and this is echoed in Semali and Pailliotet’s whole work, especially as they leave the task of reflecting on teaching and, from that, creating their own intermedial definitions to their readers.

(Inter)mediality and Literacy In “The Relevance of Mediality and Intermediality to Academic Studies of English Literature” (2008), Werner Wolf shows that Wolfgang Weiss introduced the concept of mediality to English literature studies at the end of the 1970s, remaining limited to editorial issues. We may note that even today most editing studies remain separate from research on intermediality and ignore the notions of mediality and, even more, that of intermediality. Actually, in line with the time understood by Lucia Santaella (1992) as the formation of media culture, it was only in the 1990s, precisely in view of the new media, especially the internet, that some introductory works on literary theory began to mention the terms media, mediality, and multimedia, among others, showing the beginning of a debate on how the new digital forms were related to the sedimented space of printed literature. Following the introduction of this terminology into English-language literary studies, Wolf saw a paradigm shift that he called the intermedial turn, to which he would add the medial turn, “in the face of so many fashionable ‘turns’” (2008, 16). For Wolf (2008), the inclusion of the notions of mediality and intermediality in school curricula involves two issues: either maintaining the centrality of literature and its relations with other media or including the study of other media in the curriculum. Both lead to the issue of comparative literature itself, the role of educators and their ability and skills to work with interactions between media and even for interdisciplinary activities, based on collaboration with professionals from other fields. However, Wolf’s (2008) main concern lies in the disintegration of the central axis of literary studies, already so overloaded with perpendicular themes, which is mainly to read the great literary works. For him, this interest in including digital media at any cost in school curricula may already reflect a discomfort with the very fact that literature is an academic discipline, or perhaps, we may add, one that monopolizes studies on culture, arts, and, finally, human communication, to the detriment of others that can be as important to human education as literature is. Three years later, in “(Inter)mediality and the Study of Literature” (2011), Werner Wolf returns to this issue, also under the prism of the intermedial turn and still problematizing the terms mediality and intermediality in school education for literature. Making the importance of these concepts more evident for the integral formation of students, he proposes a typology for intermedial phenomena, especially for their integration into the field of narratology. In fact, Wolf addressed the issue of a taxonomy for intermediality in several of his texts, especially in the relationship between literature and music. What interests us when thinking about intermediality in the mainstay of literacy, however, are the more general categories, which allow the understanding of the field of media and their interactions in a broad way.

18

Intermediality, Teaching, and Literacy

421

From his field of knowledge, mainly literary studies, Wolf (2011, 4) takes into account the centrality of literature to present five ways of understanding its relationship with other media: literature is a medium that shares transmedial characteristics with other media; both provides and borrows material from other media; combines with other media into a single cultural artifact; and refers to other media in various ways; and literature is part of a historical process of remediation. Considering the question of literature being a written medium, he proposes to systematize these relationships around the category of basic semiotic macromodes or macro-frames. As abstract categories, macromodes – for example, narrativity – need “materiality” for their realization, that is, textual genres, a type of discourse, or rather, media. A macroframe is a category that unites the idea of genre, subgenre, and media, thinking of it as independent media which can be realized from different “materialities,” meaning media. Macro-frames are thus defining characteristics of a medium. For example, the macro-frame classical ballet, a historical genre, assumes the body-in-motion media as dominant. The argument, a type of discourse, assumes the verbal macro-frame as dominant. It is also in this sense that Wolf understands transmediality, as a macroframe that can be realized in different media, such as narratives, which are mediated in dramas, films, and comics. This integration of different perspectives – textual genres, discourse genres, and media – into a single category is an interdisciplinary way of analyzing texts, becoming a terminology that serves different areas. Therefore, it is proper to intermediality, as the media is placed as a significant object of analysis, as a means of realizing different macro-frames in different ways. Another study that is worth mentioning and revolves around literature, though not literacy, is Irina Rajewsky’s seminal article “Intermediality, Intertextuality and Remediation: A Literary Perspective on Intermediality” (2005). Her studies proved to be fundamental to the teaching of media and intermediality because of its wellpresented categorization. Her attempt to define the term is also clear-cut: intermedia would be a generalization for the phenomena that occur at the borders of conventional media, and that is not, in this perspective, intramedial nor transmedial. Her approach is established, from the place of literature as a fundamental and critical condition or category for the analysis of individual or specific media configurations. Rajewsky’s categorization thus becomes fundamental for the structural analysis of intertextual and interdisciplinary relationships in the field of literature, and she divides the relations between media into media combination, intermedial transposition, and intermedial reference. Even if not dealing directly with literacy, Lars Elleström’s categorization allows texts to be understood as media products, while genres and subgenres are analyzed through the deconstruction of their elements. Thus, forms of communication are always understood from their interactions, that is, from intermediality. In “The Modalities of Media” (2010, 2021), Elleström addresses the issues of mediality and intermediality from the broad field of human communication. For him, understanding aspects of mediation and interactions between media is essential to reach the meaning of media products. Moreover, although grounded on comparative literature, Elleström did not focus his studies on literature, but on the elements inherent to all media, on what brings them together, and, mainly, on what distinguishes them. Despite not bringing literacy issues to the debate, his proposal of

422

A. C. Munari Domingos et al.

media categories is a relevant contribution to pedagogical issues. Therefore, in the same way as Semali and Pailliotet and Werner Wolf, Elleström begins by asking about a definition for the term media, showing that it is from this delimitation that one can both start to understand what intermediality is and take it as a methodology for analyzing media products. It should be noted that Elleström’s perspective is that of human communication and, thus, his typology is understood as irreducible, since he deconstructs the media into their intrinsic elements. However, we can say that he separates what Werner Wolf understands as being one, the macro-frames. For Elleström, there are two categories: the basic media types and the qualified media types. In the first case, media can be analyzed from the four modalities inherent to all of them: material, spatiotemporal, sensorial, and semiotic. They are, as in Wolf’s case, abstract categories that deconstruct media in order to understand their characteristics, in what distinguishes them and in what they share with each other. Thus, a novel, a narrative whose architextuality is historical, is, in a printed book, inorganic and two-dimensional, visual, temporal, and verbal. A novel-report published as a book presents exactly the same modes of the four modalities. This is how the second type of media, qualified media, becomes a category that distinguishes media from one another, when the basic modalities are not enough. The qualified media type takes into account operational and contextual issues, for example, the historical architextual perspective, which understands the novel as a long fictional narrative that presents a hero in conflict, while the novel-report brings the conflicts of characters that exist outside the story’s diegesis. Here, Elleström relies on semiotics to show that fiction has an iconic relationship with reality, while reportage has an indexical relationship. Again, even though it does not deal directly with literacy, Elleström’s categorization allows texts to be understood as media products, while genres and subgenres are analyzed through the deconstruction of their elements. Thus, forms of communication are always understood from their interactions, that is, from intermediality.

Systematizations for Educational Purposes The first attempts at systematizing the teachings of intermediality can be attributed to the pedagogical phenomenon of using filmic adaptations in classrooms. Initial discussions regarding the teaching of intermediality can be accredited to this practice and its subsequent questioning because the exhibition of the film was followed by explanations related to adaptations and how the film does not directly correspond to the book. A good number of literature educators applied a filmic adaptation to illustrate or fill in their literature classes at least once in their careers. Consequently, debates related to fidelity, primacy of the literary, film industry, and mass media culture were inevitably tackled. By accessing so much nonliterary knowledge in literature classes, little by little the teaching of intermediality gained force and space. However, how could teachers effectively use film adaptations and other media in their literature classes?

18

Intermediality, Teaching, and Literacy

423

Dennis Cutchins (2010), in his article “Why adaptation matters to your literature students,” stresses the importance of relational, contextual, dialogical readings of literary texts when educators use film adaptations effectively in the classroom. Verbal texts and their filmic adaptations are closely related, one informing the other, complementing themselves. The author points out the positive outcomes of this practice, such as double mindedness in reading both film and narratives and the ability to identify literary qualities in literary works. Informed by Bakhtinian theories and Julia Kristeva’s notion of intertextuality, Cutchins (2010, 92–93) reintegrates relativism to meaning – “meaning in films, in literature, or in other texts is rarely absolute. . .” – and context to literacy: “. . .to become literate, in the sense of learning to read in a less absolute, more contextual manner.” In educational contexts, this dialogical perspective reinforces the relevance of establishing connections among texts because students become apt to read texts in the light of others. Mieke Bal, in her turn, while discussing film reading in some of her works, for example, in Looking In: The Art of Viewing (2001), mentions “seeing in detail” as a metaphor for the close reading of media products. Although Semali and Pailliotet (1999) draw upon critical pedagogy issues more incisively, Wolfgang Hallet (2015) seeks to systematize a methodology of analysis of intermedial reference in literary texts. His work contributes to intermedial teaching and research, as it provides terminological tools and categorizations to better approach intermedial references in literary texts. At first, it does not seem to contribute as a form of pedagogy, but offers a didactic perspective to novices in intermedial practices in a way that allows one to identify the media product either through a close reading or a wide reading of the literary text. To begin with, aware that a single methodological approach cannot comprise all forms of media interrelations, his study revolves around four types of medial interconnections, depending on the transparency of intermediality (whether the intermedial reference is transparent or concealed to the reader): genre-specific intermediality, types of representation in a literary text, imitation of media by a literary text, and types of media represented or referenced. The genre-specificity of intermedial relations turns back to the literary text and the way a narrative or a poem (the literary genre) shapes the occurrence of another medium. In terms of representation of media in a literary text, it is necessary to identify the forms in which media can be conveyed because it must be described in detail. As for the imitation of media by a literary text, it refers to the way the narrative discourse transforms itself into the art form4, because the medium can be re-presented in the mode of a literary genre. Finally, concerning the media represented or referenced, this analytical dimension is involved in the identification of the types of media that the literary text addresses, their communicative or cultural value, and why they arouse interest in literature (Hallet 2015, 608). In the second part, Hallet (2015, 610) proposes to reflect upon the systemic levels of intermediality at which other media are addressed in the literary text. For him, it is

4

Hallet (2015) refers to media as art forms in his article to name the different types or art and media.

424

A. C. Munari Domingos et al.

important to identify whether it is a single artifact, a genre, or a semiotic system that is referenced in the text. When literature addresses a single media, it is also referring to its whole semiotic system, and it is implied that this would be the first systematic level. For instance, the narrative may employ an ekphrasis or mention a painting and use these painting techniques in the structure of the novel. The secondary level regards the entire medial or aesthetic genre that is involved in the literary text, and this involvement reaches a level of translation of the art to the literary work, such as a music style and compositional structure when identifiable in literary narrative techniques, for example. Finally, a text may refer to thematize or comment upon a whole aesthetic or medial semiotic system at a third level of intermediality. This level recollects media in a complete systemic dimension, such as the film industry, studios, film companies, and cinemas being referred to in a novel. In a third moment, Hallet (2015) calls attention to how intermedial references and representation affect the meaning of a literary text in terms of intrinsic and extrinsic literary qualities, which are named “intratextual functions of intermediality” and “extratextual intermedial functions.” The first set of functions refers to the basic constituent genres of a narrative, for instance, and how they are impacted by the referred works of art, such as plot, character, themes, narrative discourse, and metafictional elements. On the other hand, departing from Werner Wolf’s proposal (1999), Hallet claims that the extratextual functions of intermediality are related to the meta-level of aesthetic, medial and cultural reflection, or self-reflection (Hallet 2015, 48–50). These functions can be named metafictional functions of literature, as in the self-reflexive discourses of a novel; meta-aesthetic and meta-medial functions, as in the features, qualities, or affordances of the medium that is referenced or explicitly addressed, often in comparison to or across other media; and, finally, the meta-cultural functions, which are a more general reflection and critique of general cultural developments, processes, and practices. In short, Hallet (2015) states that his methodological contribution to the literary field is only a starting point and there are other aspects of relevance to be analyzed in a literary text besides its intermedial qualities. This systematization points out to essential aspects for a comparative analysis that contributes to the initiation of literature and intermedial studies. Influenced by Bakhtin and grounded on Peircean semiotics, Jørgen Bruhn (2016) developed an analytical three-step methodological procedure: cataloging, structuring, and contextualizing. He argues that literary narratives very often incorporate, in form and content, extraliterary material which is constantly ignored by readers and researchers. There would, however, be a considerable gain in understanding the central points of these texts if an approach focused on media were employed – that is, tools of communicative action used within and outside the arts. Aiming at delving into the possibilities generated by qualified media as vital components of novels, in accordance with Bal’s suggestion of “seeing in details,” Bruhn’s method takes up the principles of close reading so popularly spread by the American New Criticism: attentive reading, followed by the observation of patterns of repetition, similarities and contradictions, and, finally, by the interpretation of the text. His approach goes through three main aspects: the conceptualization of mediality and intermediality,

18

Intermediality, Teaching, and Literacy

425

the expansion of the scope of intermedial and literary studies, and the suggestion of an intermedial methodology to be applied to literary narratives. Meanwhile, intermediality is treated as an object of study, as a method of investigation, and as a theory. It is worth noting that the term mediality is used to designate part of a communication process, and the term media the final product. The proposed methodology combines the theoretical field of intermediality with the specific field of literary analysis and enables seeing literature with fresh eyes, from a new perspective. However, as the author emphasizes, one must keep in mind that the three steps can only be applied if there is respect for the unity of the text along with its specificities, which demands, in each reading, a creative reconfiguration of the method. The volume titled Cinema Between Media: An Intermediality Approach (2018), co-edited by Jørgen Bruhn and Anne Gjelsvik, aims at rethinking the practice of conventional film analysis from concepts and analytical tools derived from the studies of intermediality in an innovative vision by combining these two theoretical fields in an analytical model designed to be used both in pedagogical contexts and in theoretical discussions about the form and functions of cinema. A differential of Cinema Between Media in relation to The Intermediality of Narrative Literature is the understanding of medialities as a (leit)motif. For Bruhn and Gjelsvik (2018), the recurrent presence of medialities produces meaning on several levels simultaneously, as they become part of the interpretation of the film in question, when properly analyzed. This proposal to analyze the presence of media aspects in cinematographic or literary works constitutes, according to the authors, a maieutic method, since a certain dimension of the investigated object is emphasized in order to allow access to aspects that would probably not be detected in another form of analysis. In order to put into practice the proposed compilation of media within the boundaries of cinema and, thus, enable an interpretation and contextualization beyond the theme of the investigated media products, Bruhn and Gjelsvik assume that readers have certain skills in cultural analysis and claim to be aware that the first step is easy to carry out but the second and third require a certain degree of creativity, analytical training, and practice. By introducing central terms and methodologies of intermedial studies, the aim of Intermedial Studies: An Introduction to Meaning Across Media (2021), co-edited by Bruhn and Beate Schirrmacher, is to enhance intermedial literacy. Their volume does not actually define literacy, but rather elucidates ways to describe, analyze, and discuss numerous communicative processes across the conventional media borders by introducing tools to understand media built upon Lars Elleström’s definitions of media (2021). Bruhn and Schirrmacher make it clear that in order to understand intermedial relations, intermedial research must provide analytical tools and terminology. Hence, it may be assumed that intermedial literacy means to provide terminology that may serve as theoretical tools to analyze how the material qualities of media of display and basic media types convey certain truth claims (9). For them, intermedial studies are rather distinctive nowadays due to responses to societal challenges that we have been going through, such as global warming and the fake news phenomena. In a certain manner, this version of intermedial literacy is in

426

A. C. Munari Domingos et al.

consonance with Semali and Pailliotet’s (1999) critically informed view yet somehow influenced by McLuhan. Without great pretensions of representing contemporary media products, in terms of historical or geographical representation or even genres, the combination of the premises developed by Bruhn and Gjelsvik and by Bruhn and Schirrmacher fulfills its pedagogical purpose of exemplifying the proposed analytical method in a clear and didactic way, and it has the potential to benefit its target audience: graduate students, academics in varied fields of studies, and other interested parties. The case studies aiming at a wide variety of media demonstrate how form and content have a very close relationship and offer new perspectives to the investigated objects, also contributing to the development of a theory that encompasses possibilities of carrying out an intermedial approach in the analyses of other fields as well. It should be also mentioned that the case studies from the three volumes mentioned clearly illustrate the theoretical proposal for the study of the transfer of media characteristics in a media transformation, elaborated by Lars Elleström (2010, 2014, 2021). In turn, Lars Elleström (2010, 2021), although not including a teaching perspective, shows how intermedial reading collaborates to reach the complex meaning of media products by observing them in their interactions. It is by comparing the media with each other that we perceive what is or is not intrinsic to each one of them. On the other hand, Elleström (2010, 2021) also points to the important issue of mediality when we talk about media. Mediality, or the way in which media transfer cognitive value between the mind of a producer and a receiver, is based on the configuration of the four modalities inherent to all media. Understanding the processes of mediation is the first step toward understanding the processes of representation, that is, the way in which perceivers construct meaning. Thus, for Elleström (2010, 2021), understanding intermedial relationships in a broad sense evokes the consciousness of the role of media and the mediation processes in the construction of sense. Likewise, he understands mediality as inherent to intermediality and, thus, a process that must be analyzed in the reading of all types of media. Therefore, he proposes the inclusion of these two perspectives in literary studies, or, more broadly, in cultural analysis. Regarding teaching, all the proposals presented end up leading us to the same questions: What is a medium? How to define media and mediality? What is intermediality about? These seem to be questions that are repeated in works that relate intermediality and literacy. As we know, studies in intermediality often bring up epistemological questions, we would even say metatheoretical, about the limits of this area of research, and this issue often crosses education, although without deepening these relationships. The interrogation is often whether intermediality can also be considered a methodology – or what Semali and Pailliotet understand as a pedagogy. This is how, even transversally, some scholars have brought up the theme of intermediality as a process of understanding media in a broad sense, such as Lars Elleström (2010, 2017, 2021), categorizing intermedial phenomena, such as in Rajewsky (2005), or reaching the meaning of media products, such as the proposals of Werner Wolf (2008) and Lars Elleström (2010, 2014, 2021).

18

Intermediality, Teaching, and Literacy

427

With the abovementioned authors, we have meant to bring up their concepts of media, mediality, intermediality, and literacy. We understand that discerning these concepts between different areas that deal with reading is the first step toward the inclusion of (inter)mediality in school curricula, perceiving it as intrinsic to literacy. In this sense, we meant to compile these definitions, as well as some typologies, to show that intermedial literacy, after all, is about developing the ability to critically make connections between media.

Intermediality and Teaching: Reading Media or Simply “Reading” So far, we have brought into discussion diverse views on intermedial and medial concepts that may be integrated into educational contexts as methodological tools for textual analysis. The practice of textual analysis involves, consequently, the act of reading. To Semali and Pailliotet (1999, 6), the notion of reading is permeated by their enlarged understanding of “text.” Text is a broad category, a construction that involves “active, varied transactions of meaning making.” Thus, their notion of reading implies active involvement in processes of understanding and construction of meaning because this activity is the basis of a modern type of literacy which, for them, is intermedial. They state: “By reading, we don’t just mean passively receiving information or decoding print” (Semali and Pailliotet 1999, 6). As for the case of filmic adaptations in specific, Cutchins (2010) defines reading as an engagement between the reader and the literary or filmic texts. It involves a relational, dialogic understanding of the verbal and filmic works. There is a Bakhtinian notion of reading here implying that all texts are in constant dialog, as also suggested by Bruhn’s three-step model (2016) further developed in the works of Bruhn and Gjelsvik (2018) and Bruhn and Schirrmacher (2021). In short, reading media is relational, dialogic, and critical because it is informed by contextual clues. Critical literacy in intermediality may contribute by enhancing the reader’s abilities not only to understand information but to evaluate it in diverse and multiple formats, be it an art form, mass media, or digital content. No medium exists in a vacuum, and a critical perspective on media transforms the whole idea of how we produce, evaluate, and consume media products today. Knowledge relies on deep historical and cultural connections that is shaped by the media available at the time they are produced. Reading requires the acknowledgment of material, modal, critical, and aesthetic instances, one complementing the other; at the same time, all and each of them interfere in the process of making and processing knowledge. Therefore, an intermedial approach to education brings critical concerns in terms of representation of genre, class, and ethnicity; the evaluation of aesthetic qualities of a work of art; the historical context in which the media product was produced and their material conditions; and the senses these media arouse. Thus far, this chapter assembled and commented on the relations between intermediality and literacy departing from the questions on the definition of intermediality and the motives to use this concept as a methodological tool. Firstly,

428

A. C. Munari Domingos et al.

we presented the proponents Semali and Pailliotet (1999) who highlight the terminological mismatch in education that does not contribute to an interdisciplinary perspective of the communication sphere. Their view on intermediality comes with a critical literacy twist, in order to unfold shifting meanings and cultural ideologies that underlie contemporary discourses. Nevertheless, by emphasizing the visual media and thus blurring the edges among them, the authors lead us to understand that the boundaries between media are fundamental for a literacy proposal. It is in this sense that intermediality, as we defend, becomes such a methodological operator. In the second part of this chapter, we paralleled the notions of mediality, intermediality, and literacy, mainly through the studies of Werner Wolf (2008, 2011) and his formulations on the intermedial turn and mediality. Not only is Wolf’s work paramount to the study of intermedia, but he also pinpoints the centrality of literature to the categorization of media products. This last topic, the categorization of media products, is briefly tackled through Rajewsky (2005). Finally, Elleström’s (2021) categorization allows us to see texts as media products and intermediality as part of the broad field of human communication. These authors’ supplementing proposals enable transplanting intermedial theory into pedagogical practices. In an attempt to trace the history of initiatives to teach intermediality, in the third part, we assembled some theoretical work that aimed to discuss the subject of media, intermedia, and mediality with educational purposes. In our trajectory, we could identify articles advocating in favor of adaptations to teach literature (Cutchins 2010); the analysis of intermedial reference in literary texts (Hallet 2015); the establishment of a methodological procedure of cataloging, structuring, and contextualizing literary narratives that incorporate art in form and context (Bruhn 2016); the reflection on the practice of film analysis using intermediality (Bruhn & Gjelsvik 2018); the use of analytical tools and terminology derived from intermediality to enhance intermedial literacy (Bruhn & Schirrmacher 2021); and the understanding of the notion of mediality and the role of media in the process of meaning construction (Elleström 2010, 2021). In the last topic, we questioned whether currently we were reading media or just reading. No conclusion is possible, even though our discussion points to the complexity of interactions among artistic media, mass media, and digital media. Research on the area can thrive in its attempts to untangle these complex relations. It seems feasible at this point to resume the opening question: “What is intermediality and why study it in classrooms?”. Thus, the answers about media, mediality, and digital literacy have turned out to be rather relevant within school pedagogy. This issue still seems a bit problematic insofar as the limits for these concepts come from different fields of study and often from different perspectives by scholars who deny closed approaches, evoking an openness to diversity. On the other hand, we see no disagreement when it comes to affirming intermediality as inherent in culture – and even in human thought.

18

Intermediality, Teaching, and Literacy

429

Synthesizing Terminology Chart As a way of contributing to the understanding of the concepts presented in the chapter, below we bring a synthesis of the definitions of the authors herein studied. Media Semali and Pailliotet (1999) The notion of media is the result of a compilation of dictionary possibilities: “something intermediate (…) middle state or degree (…) an intervening state through which a force acts or an effect is produced (…) any means, agency or instrumentality (…) environment (…) any material used for expression or delineation” (3). All in all, this definition of media reveals itself as broad and undefined, ranging from mass communication media to electronic devices. This lack of delimitation suggests how incipient the research on the theme was in the late 1990s. However, it also suggests that the notion of media was limited to mass communication media and digital media, due to its emergence and apparent inability to deal with its knowledge and practical use at the time Wolf (2008) Media is defined from Marie-Laure Ryan’s concept of “the hollow pipe interpretation” (Media and Narrative 2005), seeking to resolve the issue between two extremes: a very broad definition, around McLuhan’s idea that media is the extension of man, and a second one, which is restricted to material or technical means. Thus, a medium encompasses both the conception of a means of communication and also a specific instance of a semiotic system, thus showing itself to be culturally distinct Elleström (2021) A medium is an extension of the human mind – and not of man, as proposed by Marshall McLuhan. To reach the concept of media, it is necessary to consider the complexity of its structure and its different types. For studies on intermediality, he prefers not to understand production and storage devices as media. The focus is on the media product, which works as a media in the broadest sense, as it is the element that realizes cognitive import between two minds, that of the producer and that of the receiver. Thus, media is at the same time the material and virtual space that realizes communication. A sheet of paper is a medium; the writing on the sheet is a medium; a story written on the sheet of paper is a medium. A specific short story, for example, Anton Chekhov’s “Anguish,” is a media product. The written narrative is a basic media type. Russian short story is a qualified media type Bruhn and Schirrmacher (2021) Media is understood as an invisible but crucial basis of human communication as well as the materiality that allows the media shape of what is communicated. Influenced by McLuhan’s idea that “the medium is the message,” it is important for these authors to stress that communication is dependent on material objects and, as a consequence, “the choice of media defines what and how we communicate” (10)

MEDIALITY / MEDIALITIES Elleström (2021) Mediality is the way all modes of media modalities act in order to mediate cognitive import between a producer’s mind and a perceiver’s mind. Mediality, the act of transfer, occurs almost simultaneously with representation, the act of making sense of the media product Bruhn (2016) Medialities is a term used by Bruhn to discuss media as “specified clusters of communicative forms” (Bruhn 2016, 17) that “may be briefly defined as tools of communicative action inside or outside the arts” (1)

430

A. C. Munari Domingos et al.

(Digital) Literacy Semali and Pailliotet (1999) Quoting Flood and Lapp (1995), literacy is seen by Semali and Pailliotet as “the ability to function competently in the ‘communicative arts’, which includes the language arts as well as the visual arts of drama, art, film, video, and television” (5). Although this perspective sounds more aligned with intermedial literacy, it assumes a broad sense of reading and, therefore, of media. The authors claim that media literacy is a key concept in their work, as far as media are understood as constructions that build our reality, while at the same time our social contexts are constructed and changed by media (14–15). The main concern of the content of the book chapters revolves around how best to approach the emerging technologies, especially those of the mass media and the internet. For them, the need to examine aesthetic and social characteristics of mass media as significant areas for education of viewers is “what has come to be known as critical media literacy” (15). However, it is the third chapter that brings one of the best elaborate definitions of digital literacy of the volume as “the ability to understand information – and more important – to evaluate and integrate information in multiple formats that the computer can deliver” (Macaul et al. 1999, 56) The lack of an elaborate view on digital literacy in Semali and Pailliotet (1999) may be explained by the small number of research on the topic at the time. However, their concern relies heavily on media literacy. For them, media literacy expands the notion of critical literacy, as it includes a critical instance to all media texts (18). Critical media literacy enables students and teachers to bridge existing learning contexts and build new ones (7). Still, according to the authors, critical literacy projects should work towards the promotion of social justice and the critique of hegemonic ideologies (18)

Intermediality Macaul et al. (1999) To approach intermediality in the classroom, Macaul et al. (1999, 57) conceive the phenomenon according to Weiner (1996, 308) who says that it is “a combination of art forms such as audiotape and print material or film and live performance, also called mixed media or multimedia.” In this way, Macaul, Giles and Rodenberg (1999) refer to intermediality in educational contexts as “connections across convergence of various media,” in which meaning is constructed by putting together a variety of media messages, social experiences and conventions Rippl (2015) Questioning the reason for the interest in intermediality, Gabriele Rippl (2015, 1) proposes: One of the reasons why it is impossible to develop one definition of intermediality is that it has become a central theoretical concept in many disciplines such as literary, cultural and theater studies as well as art history, musicology, philosophy, sociology, film, media and comics studies – and these disciplines all deal with different intermedial constellations which ask for specific approaches and definitions. In this way, observing the way intermediality spreads in different fields of knowledge, the question of why it should be understood as a methodology is explained from its interdisciplinary possibility, when observing the media and their relationships without being tied to closed disciplinary concepts Semali and Pailliotet (1999) Intermediality is a methodological tool: “intermediality provides a methodology to read printed and visual representations of meaningful ideas and the influence of multimedia on learning, pedagogy, and social practices in educational communities” (2). At the same time, intermediality implies connectedness among texts; textual processes; and, thus, also contexts, because media systems permeate our lives (Kellner, 1995 apud Semali & Pailliotet 1999, 6). The authors claim that “intermediality extends the range of the term literacy to encompass contemporary multiliteracies in their various modes” (22)

18

Intermediality, Teaching, and Literacy

431

Wolf (2008) Intermediality is defined, in a narrow sense, as every interaction of one more medium in a human artifact and, in a broad sense, following Irina Rajewsky (2002), as any phenomenon in which the limits of different media are crossed with each other. Certainly, this definition has little chance of being challenged, as it affirms the basic idea of the term – inter/media. Intermediality, however, can be many-sided when we begin to question the meanings of media, since it can encompass not only many kinds of forms of communication as well as their different (or similar) affordances and languages. Furthermore, it is necessary to take into account that media, per se, are always inter Elleström (2021) He shows us that the intermedial view can be a way of reaching the meaning of media products when looking at the characteristics of media: In brief, one might say that the crucial “inter” part of intermediality is a bridge, but what does it bridge over? If all media were fundamentally different, it would be hard to find any interrelations at all; if they were fundamentally similar, it would be equally hard to find something that is not already interrelated. However, media are both different and similar, and intermediality must be understood as a bridge between media differences that is founded on media similarities Nagib and Jerslev (2014) They understand André Bazin as a pioneer of intermediality in film studies precisely because he focuses on understanding cinema elements from an “intertextual” perspective. At this point, they show how Bazin’s analysis ends up questioning what cinema is. It is in this sense that intermediality can be understood, in a broad sense, as a way of reading (media), as it, after all, asks the same question of all media

Acknowledgments The present work was supported by the Brazilian Council for Scientific and Technological Development, CNPq—Conselho Nacional de Desenvolvimento Científico e Tecnológico (processes numbers: 309678/2021-8 and 304566/2021-7).

References Bal, Mieke. 2001. Looking. In The Art of Viewing. London: Routledge. Bruhn, Jørgen. 2016. Intermediality and Narrative Literature: Medialities Matter. London: Palgrave Macmillan. Bruhn, Jørgen, and Anne Gjelsvik. 2018. Cinema between Media: An Intermediality Approach. Edinburgh: Edinburgh University Press. Bruhn, Jørgen, and Beate Schirrmacher, eds. 2021. Intermedial Studies: An Introduction to Meaning Across Media. London: Routledge. https://doi.org/10.4324/9781003174288. Cutchins, Dennis, Laurence Raw, and James Welsh, eds. 2010. The Pedagogy of Adaptation. Plymouth: The Scarecrow Press. Elleström, Lars. 2010. Media, Modality and Modes. London: Palgrave, Macmillan. ———. 2014. Media Transformation: The Transfer of Media Characteristics among Media. Houndmills: Palgrave Macmillan. ———. 2017. Midialidade: ensaios sobre comunicação, semiótica e intermidialidade. Domingos, A. C. M.; Klauck, A. P.; Mello, G. M. G. de. (Org.). Porto Alegre: EDIPUCRS. ———. 2021. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In Beyond Media Borders, Vol. 1: Intermedial Relations among Multimodal Media, ed. L. Elleström, 3–91. London: Palgrave Macmillan.

432

A. C. Munari Domingos et al.

Hallet, Wolfgang. 2015. A Methodology of Intermediality in Literary Studies. In Handbook of Intermediality: Literature-Image-Sound-Music, ed. Gabriele Rippl, 605–618. Berlin: De Gruyter Mouton. Kincheloe, Joe L., and Shirley Steinberg. 1999. Series Editors’ Foreword. In Intermediality: The Teachers’ Handbook of Critical Media Literacy, ed. Ladislau Semali and Ann W. Pailliotet, vii–ix. Boulder: Westview Press. Macaul, Sherry, Jackie K. Giles, and Rita K. Rodenberg. 1999. Intermediality in the Classroom: Learners Constructing Meaning Through Deep Viewing. In Intermediality: The Teachers’ Handbook of Critical Media Literacy, ed. Ladislau Semali and Ann W. Pailliotet, 53–74. Boulder: Westview Press. Nagib, L., and A. Jerslev. 2014. Introduction. In Impure Cinema: Intermedial and Intercultural Approaches to Film, ed. L. Nagib and A. Jerslev, xviii–xxxi. London, New York: I. B. Tauris. Rajewsky, Irina O. 2005. “Intermediality, Intertextuality and ‘Remediation’”: A Literary Perspective on Intermediality. Intermédialités / Intermediality 6: 43–64. https://doi.org/10.7202/ 1005505ar. Rippl, Gabriele. 2015. Handbook of Intermediality: Literature Image - Sound - Music. Berlin, München, Boston: De Gruyter. Santaella, Lucia. 1992. Cultura das mídias. São Paulo: Razão Social. Semali, Ladislau, and Ann W. Pailliotet, eds. 1999. Intermediality: The Teachers’ Handbook of Critical Media Literacy. Boulder: Westview Press. Weiner, R. 1996. Webster’s New World Dictionary of Media and Communication. New York: Macmillan. Wolf, Werner. 2008. The Relevance of Mediality and Intermediality to Academic Studies of English Literature. In Mediality / Intermediality, ed. Martin Heusser, Andreas Fischer, and Andreas H. Jucker, 15–44. Tübingen: Gunter Narr. (SPELL - Swiss Papers in Language and Literature, Vol. 21). ———. 2011. “(Inter)mediality and the Study of Literature.” CLCWeb: Comparative Literature and Culture 13.3. Available at: https://docs.lib.purdue.edu/clcweb/vol13/iss3/2/. https://doi.org/ 10.7771/1481-4374.1789.

Intermedia, Multimedia, and Media

19

Ken Friedman and Lily Díaz

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction: Recovering a History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Four Histories of Intermedia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The First History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Second History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Third History of Intermedia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Fourth History of Intermedia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Three Directions for Intermedia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedia, Multimedia, and New Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multimedia Then and Now . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reconsidering Multimedia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion: Toward an Archeology of Intermedia, Multimedia, and Media . . . . . . . . . . . . . . . . . . Bibliography Essay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

434 434 436 437 439 440 441 442 443 444 447 449 452 452

Abstract

This chapter recovers the lost history of intermedia. This history relates to the earliest use of the term intermedia, a word coined in 1965 by Fluxus co-founder Dick Higgins. Intermedia traces its development from earlier periods into the modern context of work by Fluxus artists who blended conceptual art, technical media, social concerns, events, music, and what would come to be performance art. The chapter also seeks to clarify the notion of intermedia in relation to other contemporary terms such as multimedia and new media. We argue that the concept of intermedia has many pasts and histories. We can view these from different frames of reference, and each frame points to different directions with K. Friedman (*) College of Design and Innovation, Tongji University, Shanghai, China L. Díaz Art and Media Department, Aalto University, Espoo, Finland e-mail: lily.diaz@aalto.fi © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_43

433

434

K. Friedman and L. Díaz

different and interesting futures. This broad framework enables us to consider the intersections of old technologies and new. Such a framework permits us to develop robust research programs that deliver information, education, entertainment, and art in useful and useable forms. Keywords

Conceptual art · Fluxus · History · Hybrid · Intermedia · Multimedia · New media

Introduction Introduction: Recovering a History Five decades ago, artist Dick Higgins (1966b, 1969a: 22) published an artwork titled Intermedial Object #1 (Fig. 1) in the form of a performable score. It resembled the event scores and instruction pieces of Higgins’s colleagues in the international laboratory for experimental art, design, and music known as Fluxus. Something Else Press invited those who received the proposal to send photographs and movies of resulting objects. In this playful, poetic, and partially impossible way, Dick Higgins exemplified and published one of the first works of art to bear an explicit designation as “intermedia.” Higgins coined the term intermedia at the end of 1965 to describe art forms that draw on several media, growing into new hybrids. Intermedia works cross the boundaries of recognized media, often fusing the boundaries of art with media that have not previously been considered art forms. Higgins (1966a, 1969a: 11–29; 2001) published a now-legendary essay describing an art form appropriate to artists who feel that there are no boundaries between art and life. Along with many artists and composers, Higgins felt the time had come to erase the boundaries between art and life. From this, it followed that there could be few boundaries between art forms, perhaps none, and that new forms of art could enter the previously distinct media from the larger life world. For several years after Higgins published his essay, the term intermedia was primarily visible in the influential circle of artists, architects, and composers in and around Fluxus. Many artists active in intermedia art forms in the 1960s took part in Fluxus, including the Korean artist Nam June Paik, and the Japanese Ay-O, Takehisa Kosugi, Shigeko Kubota, Yoko Ono, and Mieko Shiomi. Germans Wolf Vostell and Joseph Beuys were active in the field along with the Swede Bengt af Klintberg, Danes Addi Køpcke, and Henning Christiansen, and Icelander Dieter Roth. French artists Jean Dupuy, Robert Filliou, and Ben Vautier were key intermedia artists. So were Lithuanian-born Americans George Maciunas and Jonas Mekas, as well as Higgins himself and such Americans as Alison Knowles, Jackson Mac Low, Ken Friedman, Al Hansen, Geoffrey Hendricks, Davi et Hompsom, Ben Patterson, and Emmett Williams.

19

Intermedia, Multimedia, and Media

435

Fig. 1 Original conceptual formulation of Intermedia Object by Dick Higgins, 1965. (© Archives Ecart, Genève)

The interpretations these artists gave to intermedia ran from the simple and primitive to the technically sophisticated. At one end of the spectrum, there were the folklore-based projects of Sweden’s Bengt af Klintberg, the actions of Milan Knizak in Czechoslovakia, and the poetry performances of American Emmett Williams. At the other, there were Nam June Paik’s dazzling video proposals, the sophisticated book-print-installation works of American Alison Knowles, or Higgins’s innovative radio plays and computer-generated art works. The 1960s and 1970s saw little writing on intermedia. A small number of perceptive books and articles used the term (see, for example, Becker 1977; Berger, Dorfles, Glusberg, Moles, and Udo Kulterman 1980; Frank 1967; Friedman, 1972, 1973, Glusberg 1969; Higgins 1967, 1968, 1969a, b; Klintberg 1977; Lester 1966,

436

K. Friedman and L. Díaz

1968; Renate Matthaei 1972; Gene Youngblood 1970). Nevertheless, the term did not take hold. Today, several thousand colleges and universities offer intermedia courses. Many of these institutions offer full intermedia programs in departments of intermedia studies. Nevertheless, the term multimedia is far better known. The first recorded citation of multimedia was noted over 35 years ago, in 1962. The original meaning of the term was “using, involving, or encompassing several media” (Merriam-Webster). It came to encompass many various kinds of technologies used conjointly in differing combinations. The Oxford English Dictionary defines multimedia as “designating or pertaining to a form of artistic, educational, or commercial communication in which more than one medium is used” (OED 2021: unpaged). In contrast, the OED defines intermedia as the plural of intermedium, and it offers none of the applicable artistic definitions of the term intermedium. The two words have had distinctly different levels of reception. In 2018, a library search tool located 44,740 peer-reviewed articles with the term “intermedia” in the major search fields, but most of these are scientific articles using the term in the technical sense of a chemical or biological medium. In contrast, the search tool located 240,399 articles with the term “multimedia” in the major search fields, and most use the term as it is used here. While there has been a massive growth of usage for the term “intermedia” over the past few years, “multimedia” still predominates. A 2023 Google search yields 150,000,000 hits for the term “intermedia.” A Google search yielded 1,990,000,000 hits for the term “multimedia,” a massive difference. There has been significant growth in scholarly publications as well – a Google Scholar search for the term “intermedia” yields 1,110,000 hits, but a significant number involve the natural sciences and technology. The comparable figures for “multimedia” are again larger, at 6,100,000. Although the word has yet to be recognized in a general dictionary, the term’s influence in creative endeavors can be gauged by its inclusion in the Getty Art & Architecture Thesaurus Online. In this controlled vocabulary used by experts to describe different knowledge domains related to art and architecture, the term intermedia is defined as: “The concept that certain contemporary works merge already known art forms to inaugurate a new type. If the resulting art form gains currency and acquires a name, it becomes a new medium and is no longer intermedia.” While a few authors compare and distinguish between the two terms (see, for example, Walker 1977: 167–168), intermedia and multimedia are often conflated and confused. This chapter seeks to retrieve important distinctions embodied in the term intermedia. Equally important, it will point to a history older than the word itself, a history that might open new conceptual territory for media development today.

Four Histories of Intermedia Intermedia has many pasts, several definitions, and at least two futures. In this chapter, we seek to address four histories, three directions, and some interesting futures. While the histories of intermedia are multilayered and complex, the history of the word “intermedia” is relatively simple. Higgins used the word to describe the

19

Intermedia, Multimedia, and Media

437

tendency of an increasing number of interesting artists to cross the boundaries of recognized media or to fuse the boundaries of art with media that had not previously been considered art forms. When asked how he happened to create the term “intermedia,” Higgins once noted that Samuel Taylor Coleridge had used the term over a century and a half before he rediscovered it. Higgins was too modest. Coleridge used the term “intermedium” once – apparently once only – to refer to a specific issue in the work of Edmund Spenser. Coleridge used the word “intermedium” in “Lecture Three: ‘On Spenser’” in a way that resembled Higgins’s use of the term “intermedia.” Nevertheless, Coleridge used a word different in meaning and in form, referring to a specific point lodged between two kinds of meaning in the use of an art medium. Coleridge’s word “intermedium” was a singular term, an adjectival noun (Friedman 1999: 158; 2018: 13–14). In contrast, Higgins’s word “intermedia” refers to a tendency in the arts that became a range of art forms and a way to approach the arts. Higgins said that he might have read the Coleridge essay in his years at Yale or Columbia, taking it in subconsciously. This may be true. Even so, Higgins coined a new word in the term “intermedia,” giving it the current form and contemporary meaning it holds to this day (Friedman 1998b).

The First History This history of intermedia is about the artists who began to develop intermedia projects in the late 1950s and early 1960s. It is a history that is linked with the history of concept art. The early version of concept art was first defined by Henry Flynt in his essays on Concept Art from the late 1950s through 1961 and published in a seminal essay in An Anthology (Flynt 1963: unpaged). When using the term concept art, we are not referring to the better known conceptual art developed in the mid-1960s by Joseph Kosuth. Conceptual art, in the way Kosuth has sought to develop it, is a form of art about art in which ideas about art, as a form of philosophizing, become the art itself. An example of such art can be appreciated in Kosuth’s work, One and Three Chairs, exhibited at the Museum of Modern Art in New York. The work includes a wood folding chair. To the left is a photograph of the chair and to the right a dictionary definition of the term chair. As the artist himself explains: The art I call conceptual is such because it is based on an inquiry into the nature of art. . .. Thus, it is . . . a working out, a thinking out, of all the implications of all aspects of the concept ‘art.’ . . . Fundamental to this idea of art is the understanding of the linguistic nature of all art propositions, be they past or present, and regardless of the elements used in their construction.

The earlier form of concept art was different from the later conceptual art exemplified in Kosuth’s work in a significant dimension. Concept art, in Flynt’s definition, was an art form of which the primary element is ideas, as the primary element of music is sound. In later years, it became clear that Flynt did not intend his definition quite as he stated it. Flynt’s idea of concept art was something specific and

438

K. Friedman and L. Díaz

arcane, an argument with pre-Socratic Greek mathematics. In the meantime, George Maciunas, a co-founder of Fluxus along with Higgins, had seized on the term, applying it to the work of a diverse and interesting group of artists. (Flynt was horrified when Maciunas applied his coinage to these artists and their work. Nevertheless, he did not clarify what he meant by what he wrote until the 1990s.) Higgins, in turn, built on the ideas and practices of these artists to shape his concept of intermedia. Higgins’s (1966a) essay served as a focal point for artists and thinkers developing the ideas and issues he described. Higgins was not alone for around the same time John Brockman began using the term for the Expanded Cinema festivals he organized at Filmmakers’ Cinematheque in New York. While Higgins’s essays promulgated the term among artists and scholars, Brockman’s work helped bring the term to wider use among filmmakers, surfacing to address the larger public with in a New York Times article that appeared shortly after Higgins’s essay was published (Lester 1966). From this perspective, intermedia is an art that lies on the edge of boundaries between forms and media. Intermedia also exist between art forms and nonart forms. It is sometimes difficult to imagine an intermedia form before it is created, but many can be imagined in theory. In 1967, Ken Friedman built on the idea of Higgins’s Intermedial Object #1 to create a matrix of possible intermedia. This matrix was an elaborated inventory listing media forms. It permitted one to create different kinds of matrices, combinations, or permutations, suggesting groupings or configurations of media. In different configurations, Friedman (1967) used it to generate conceptual possibilities for new intermedia forms. These were often expressed as percentage possibilities, describing, for example, an intermedia form comprised of 10% music, 25% architecture, 12% drawing, 18% shoemaking, 30% painting, and 5% smell. The combinatorial approach developed using matrices made it possible to imagine many kinds of intermedia forms. One artist might combine aspects of typesetting, cooking, pyrotechnics, and farming. Another might embrace baking, sculpture, sewing, and perfumery. Some intermedia were thought experiments that were never realized. Others led to concrete results. A study of the past shows just how astonishing the possibilities might be. For example, the great culinary pageants of the medieval times can be regarded as intermedia that later became an art form in their own right. Thus, the court pageant – with its staged theatrical representations of huge sea battles celebrated in the flooded hall of a palace featuring music and dance in accompaniment, or the masked balls that were also allegorical pilgrim journeys – are now appreciated as historical examples of performance art (Di Felice 1980: 25–27). Many contemporary art forms arguably emerged from intermedia to become new and distinct media in their own right. These include such media as artist’s books, stamp art, mail art, and video art. And so did social sculpture, concept art, and conceptual art, along with concrete poetry and visual poems. When different forms merge, we see an intermedia form. The success of intermedia is seen in the coherence of mergers (or the convergence) that gives rise to new forms. The Art and Architecture Thesaurus points out how the most

19

Intermedia, Multimedia, and Media

439

successful intermedia forms eventually cease to be intermedia, developing characteristics of their own to become established media with names, histories, and contexts of their own. By the late 1960s, the intermedia tradition had begun to develop a literature. Higgins’s own Something Else Press published many books and pamphlets in this literature (see Frank 1983). John Cage (1961, 1969) was also a major contributor to the intermedia literature. The concept of intermedia took on life in Europe with intermedia festivals and books (Goetze et al. 1969). Around the world, George Maciunas and others produced a major stream of artifacts, multiples, books, and projects under the Fluxus imprint (see Hendricks 1982, 1983a, b, 1989; Milman 1992; Phillpot and Hendricks 1988).

The Second History The second history of intermedia began soon after the publication of Dick Higgins’s “Intermedia” essay. This history began to move intermedia from a small community of experimental artists into a larger frame of art and social life. The first vehicle for this involved the American university system and the first two intermedia courses. The first flowered and vanished. The second went on to become a central forum of diffusion for the intermedia idea. The first intermedia class began in 1967 when Ken Friedman organized and taught an intermedia course at the San Francisco State University Experimental College. In the fall of 1967, the SFSU Department of Radio, Television, and Film offered the course for credit as the first course specifically titled intermedia ever to be offered in a college or university. While the experimental course system at San Francisco State University offered a marvelous opportunity to work with new ideas and themes, these courses were not part of the regular curriculum. There were few resources and no salary for teaching. Despite its success, Friedman’s intermedia course soon vanished. The real history of intermedia in higher education began at the University of Iowa when the School of Art and Art History brought Hans Breder to its faculty. Breder launched the Iowa intermedia program in 1968, where it flourished under his direction until he retired in 2000. This was the first intermedia program to offer a complete curriculum up to and including the MFA. Breder’s intermedia program was an arena in which to explore what Breder termed “the liminal spaces” between the arts (Breder 1995). The arts, for Breder and his students, included visual art, music, film, dance, theater, and poetry. Later, this concept of a liminal space was expanded into a larger collaboration with the liberal arts, including comparative literature, anthropology, psychology, and communication studies. Breder and his students explored spaces between media approaches that crossed the boundaries between artistic and scholarly practices, hybridizing and exploiting media, genres, and the social and political universes they represented. Over the years since, intermedia has become a rich field of artistic practice and scholarly inquiry, and there are now many degree programs in

440

K. Friedman and L. Díaz

intermedia and intermedia studies. There are several departments dedicated to intermedia in universities and art schools as well as thousands of intermedia classes in traditional programs. There is also an International Society for Intermedial Studies. Some of these programs are not located in the arts but in such fields as information science, computer studies, and even the social sciences.

The Third History of Intermedia This leads to the third history of intermedia. This history locates intermedia in a social, technological, and historical context, and it reveals some possible reasons to explain why and how intermedia emerged at this time in human history. In this sense, intermedia is a consequence of postindustrial society and the knowledge economy to which it gave rise. In discussing the morphology of economic growth, Australian economist Colin Clark (1940: 337–373) classified economies as primary, secondary, and tertiary. Primary economies extract wealth from nature, secondary economies transform extracted material through manufacturing, and tertiary economies engage in services. Two more economists undertook pioneering work in the economic and political economics of information, Canadian Harold Innis (1950, 1951) and American Fritz Machlup (1962, 1979). Others – such as Marshall McLuhan (1962, 1964, 1967a, b, also McLuhan and Watson 1970) – contributed to the ideas and theories that would set the information society in focus. Sociologist Daniel Bell (1976: lxxxv) built on the work of these thinkers to describe three kinds of society. Preindustrial society extracts resources, industrial society fabricates goods, and postindustrial society processes information. Artist Nam June Paik (1976) contributed to the theory of the media in the postindustrial society. Today, such thinkers such as Manuel Castells (1996a, b, c) and Saskia Sassen (1991, 1994, 1996, 1998) have developed this inquiry further. Ken Friedman examined the relationship between these issues and intermedia in a lengthy series of articles (1998a, c, d, e, f, 2002, 2003a, b, c, 2004, 2005a, b). Technological change – and sociocultural reactions to that change – brought about what is now labeled the information society. This term is shorthand for a rich and complex stream of issues that affect working life, culture, and most facets of human interaction and behavior in the developed (or postindustrial) nations. These influences have had many effects. One effect has been the blurring of the boundaries between different kinds of media. In the telecommunication, information technology, and entertainment industries, the phenomenon of digital convergence is driving this blurring of borders. In the arts, this involves both digital convergence and the human response to a world being shaped by convergent media. In a philosophical sense, the translation of media to digital form shifts the balance between media. This paradoxically leads to a new emphasis on media that cannot take digital form such as a live chamber concert or a stage performance. We can record these and digitize the recordings. But, though it is an area of current research, we cannot yet fully digitize the live experience. These multivalent trends become

19

Intermedia, Multimedia, and Media

441

visible in the way that new media – and intermedia – affect old media. They also affect the world in which all mediated human interaction takes place.

The Fourth History of Intermedia This is a history that taking a communications perspective seeks to extend back in time and forward into the future. A medium, simply put, is a tool for delivering information. The first media were vehicles for communication. The oldest natural media are voice and language. Written media developed reasonably soon afterward in evolutionary terms. They emerged first as painted symbols on walls, much later as abstract marks on rocks or sticks, later still as alphabets or ideograms on clay tablets and papyrus. The fact that there are many media has always meant the possibility of intermedia. Intermedia has been with us since the dawn of time, with or without the explicit designation. The history of intermedia began with the birth of human communication. In 1984, A. J. N. Judge undertook a large-scale survey of all possible media for a study on information and understanding (Judge 1984). In 1998, for a European Union project undertaken in collaboration with Judge, Friedman (1998c, f) built on the earlier media matrices to extend the survey with a research request sent to over 20,000 scholars, artists, critics, and theorists around the world. The final inventory was a list of roughly 1600 possible communications media from the abacus and abbreviations to the zarzuela and “zines.” The inventory involves an approach exploring the kinds of possibilities that Herbert Blumer labels sensitizing concepts, first steps that open an idea for exploration (Blumer 1969: 2–21, 140–150; see also Baugh 1990; van den Hoonard 1997). This is a heuristic procedure, and the inventory was gathered without regard to possible challenges to the ontological and epistemological status of any specific proposed medium. Rather, all proposals were gathered for later development and consideration. This method represents a quasi-systematic application of Lenat’s (1983: 352–354) approach to heuristics, in which heuristic methods enable the development of new domains of knowledge. Within these domains, new heuristics are needed together with new representations. While Lenat’s work addresses the problem of knowledge acquisition in information systems, Friedman’s approach was a generative process. It is admittedly naive, and the inventory raised problems that could not be readily resolved other than by positing definitions. Nevertheless, the inventory and the matrices that flow from it in a generative process have raised useful and challenging issues. These concern the nature, status, and possibilities of each medium – and, in some cases, the question of how it may function as a medium. These issues involve the past, present, and future of intermedia. The history of the past has yet to be imagined. It is a history can be explored by examining three directions that intermedia may take.

442

K. Friedman and L. Díaz

Three Directions for Intermedia The intermedia concept was visible in three artistic directions of the late 1950s and early 1960s. The first was a technical direction typified by multichannel, multimodal presentations that rarely seemed to take coherent shape. The second was a simple philosophical direction typified by event scores and concept art. The third was a fuzzy, boisterous direction typified by happenings and later traditions in performance art. The first direction emphasized engagement with technology. This was an era when artists working with media – including multimedia – often presented separate and disparate art forms at the same time. This was often a fruitless approach. In contrast, some artists explored the boundaries of technology and art to examine the larger social meaning of information technology in a postindustrial society. These artists often made good use of intermedia theory. In a powerful sense, these artists began to explore the generally unrealized dimensions of digital computer code and information flows. These would begin to render all media fluid when digital control began to break down boundaries between separate forms of input, transmission, and output. This was visible in the electronic music of John Cage and Richard Maxfield or the early television experiments of Nam June Paik and Wolf Vostell. It blossomed in the art and technology programs of the 1960s, elegantly exemplified by artists such as Jean Dupuy or Newton and Helen Harrison, and in the video art of the 1970s. The second direction emphasized simplicity. This was a tradition of conceptual exploration. Often anchored in Zen Buddhism or philosophy, this stream was typified by the event structures of George Brecht, the early concept art of Henry Flynt, and the neo-haiku theater of Mieko Shiomi and Yoko Ono. In the 1960s, it entered a second phase with George Maciunas’s publishing program for Fluxus, the radical reductive films of Paul Sharits and the expanded use of events and scores for objects, installations, and performances by Dick Higgins, Ben Vautier, Robert Watts, Milan Knizak, Robert Filliou, Ken Friedman, and others. The third direction emerged from the ambiguous and often-boisterous tradition of happenings pioneered by Allan Kaprow, Al Hansen, Claes Oldenburg, Milan Knizak, and others, including Dick Higgins and Wolf Vostell among them. These streams were never as separate as some maintained, and they never constituted the single forum that others described. Rather, in overlapping and forming each other, they led to the flowering of new art forms that typified the 1960s and 1970s. Conceptual art, artists books, performance art, installation, video, and many other artistic media and traditions emerged from them. While it remained possible to separate art forms for scholarship, historical reflection, or theoretical distinction, Higgins’s vision of intermedia was that our time often calls for art forms that draw on the roots of several media, growing into new hybrids. The conceptual importance of intermedia is its profound yet often paradoxical relationship to new media. Intermedia is important because it emphasizes conceptual clarity and categorical ambiguity. The intermedia concept is powerful because it stretches across the boundaries of all media, many of them old. Intermedia provide

19

Intermedia, Multimedia, and Media

443

impetus to new media while offering a balance to the overly technological bent that new media can sometimes engender. Intermedia links many forms of media conceptually and require us to consider them in terms of human effects. This creates a sympathetic yet challenging position from which to interrogate and conceptualize new media. It strengthens the development of new media by encouraging us to think in large cultural terms. Intermedia are not an art of technical applications but an art of subtle ideas. From simple conceptual ventures that enter the liminal space between ancient, classical media to the most sophisticated techno-social hybrids, the concept of intermedia opens significant territories. Even so, nearly any intermedia space that we can enter lies in one of the three directions that such artists as Dick Higgins, John Cage, George Brecht, or Jean Dupuy first explored in the 1960s.

Intermedia, Multimedia, and New Media An important characteristic that distinguishes intermedia from multimedia is the melding of aspects of different media into one form. In addition to recovering the concept of intermedia, it may also be useful to examine the earliest concepts of multimedia. These were often closer to Higgins’s idea of intermedia than many of the multimedia concepts that have held sway in intervening years. Examining these issues will bring us forward in time to the new, emergent possibilities of multimedia in a world of digital convergence. Much multimedia work has focused on using computer technology to deliver combinations of images and sound. The earlier concept was more useful, pointing to multimedia as a variety of tools for delivering information, education, and entertainment in useful and useable forms. The early idea of multimedia were flexible and involved many kinds of technology from the simplest to the most advanced (for examples, see: Albright 1973; Block 1974; Bory 1968; Bush 1945; Cage 1969; Crane and Stofflett 1984; Di Felice 1980; Friedman and Gugelberger 1976; Glusberg 1971; Judge 1984; Kahn 1999; R.E.L. Masters and Houston 1968; Packer and Jordan 2001; Paik 1964; Paik and Moffett 1995; Porter 1971; Ravicz 1974; Theall 1992; Tofts and McKeich 1998, 1999; Tofts et al. 2003; Zurbrugg, 1993, 2004). One striking feature of early multimedia experiments was a rich variety of hardware and software. The central issue was engaging a variety of media to address any number of senses. Multimedia ventures included: sound technology with phonograph, telephone, tape recorder, radio, public announcement systems, specially created sound devices, experimental instruments for contemporary music and standard musical instruments; visual technology with drawing, printing, printmaking, lithography, silk-screen, photography, silent motion pictures, and video along with archaic techniques ranging from printing blocks and stones to modern equivalents such as rubber stamps and postage stamps; language, poetry, and narrative approaches enlivened the early, robust art and communication experiments of the late 1950s and the early 1960s.

444

K. Friedman and L. Díaz

Multimedia Then and Now Multimedia has existed for centuries. The concept of multimedia is an intellectual construct distinct from the specific focus on technology characterizing discussions of multimedia today. There are many more kinds of multimedia technology than DVD entertainment programs, game simulations, or infotainment sites on the World Wide Web. Some of these technologies have existed for centuries. Consider, for example, drama and pageant. Drama as we know it today goes back to ancient Greece. Pageantry has an even more unusual history. In the Middle Ages and the Renaissance, pageantry moved beyond court ceremonial and church ritual to become an event that surrounded, engaged, and embraced the audience with appeals to every sense imaginable (Di Felice 1980, 1984). The emblemata of the Middle Ages were another example of a multimedium. Emblemata were printed images reproduced with wise sayings or religious quotes. They were generally made with woodcuts. Emblemata brought images and text to an illiterate population. They permitted viewers to see a visual representation of symbolic content along with instructive or religious text presented in memorable verse form. This helped the illiterate audience to grasp and remember a text that few could read (Alciati 2004 [1531]; Friedman and Gugelberger 1976). Multimedia have always involved enriched and multiplied uses of single media. A medium, simply put, is a tool for delivering information. Merriam-Webster’s dictionary defines a medium as, among other things: . . .2: a means of effecting or conveying something: as a (l): a substance regarded as the means of transmission of a force or effect (2): a surrounding or enveloping substance . . .. b plural usually media (l): a channel or system of communication, information or entertainment—compare—mass medium (2): a publication or broadcast that carries advertising (3): a mode of artistic expression or communication (4): something (as a magnetic disk) on which information may be stored.

As vehicles of communication, written media developed and emerged first as painted symbols on walls, much later as abstract marks on rocks or sticks, later still as alphabets or ideograms on clay tablets and papyrus. Pictorial media are the oldest media in common use. Paintings, drawings, lithographs, maps, and the like survive still. Pictorial media today are primarily used for entertainment, along with communication and information. They often add entertainment value to educational or informative products. Text media such as letters or manuscripts remain the oldest standard medium in active use for communication, education, and information. Books and newspapers are the next oldest in active use for communication and education while entertainment and infotainment have, for most people, been taken over by pictorial and pictorial-sonic media such as film, television, and video. Information from news to political campaigns is now delivered as infotainment for vast audiences who read little and watch much. It also serves illiterate audiences who do not read at all. The most widely used media of the twentieth century are vocal and visual. The telegraph is now a highly specialized but little used medium. The telephone is

19

Intermedia, Multimedia, and Media

445

perhaps the most used public medium. Both were developed in the nineteenth century. The far more recent telefax grew out of these. Photography was also born in the nineteenth century. Motion pictures came only a few decades later when mechanical ingenuity applied to photography made film possible. Television and motion pictures are the contemporary mass media of infotainment, along with their descendants, video, DVD, and Internet streaming. Recent technological developments have made it so that all these media are converging into ubiquitous mobile telephony units, such as the iPhone capable of displaying a range of effects that most people think of as multimedia. While this view of multimedia is correct as far as it goes, it is mistaken in broad principle. Multimedia remains what they have always been. Multimedia comprises any of the dozens of possible media that can be combined in hundreds of ways to communicate, to teach, to inform, and to entertain. The idea of multimedia as a relatively standardized combination of special effects used primarily by producers of games and entertainment constrains the larger possibilities of the field to a flat, one-dimensional understanding. This negates the potential power of the multimedia concept. In practical terms, thinking of multimedia in this one-dimensional way limits us to the kinds of multimedia defined by the designers of games and infotainment. This misunderstanding hampers the production of multimedia for a host of other possibilities. The demand for special effects occasioned by this misunderstanding drives the budget costs up and use value down on many multimedia projects that can be realized more effectively with simple techniques than with complicated technologies. The failure to understand the genuine character and potential of the multimedia era is a failure to realize the fullest and best potential of an exciting tool. Understanding multimedia requires looking beyond special effects enabled by new technology. Marshall McLuhan’s famous probes into the meaning of media explored dozens of different media and their meaning (McLuhan 1962, 1964, 1967a; McLuhan and Watson 1970). For McLuhan, a “medium—while it may be a new technology—is any extension of our bodies, minds or beings” (Gordon 1997: 43). This emphasis was made clear in the subtitle of McLuhan’s (1964) most famous book, Understanding Media: The Extensions of Man. This distinguishes the useful concept of multimedia in a unified or holistic program from the notion of multimedia as multiple media performed simultaneously. This concept was summed up in Higgins’s term, intermedia. In Higgins’s concept, the term intermedia referred to art forms that draw on the roots of several media, growing into new hybrids. Like film or opera, intermedia can be seen whenever several individual media grow into forms that are effective and convincing media in their own right. This is also a helpful definition for multimedia. The qualities that link the intermedia concept to one conception of multimedia separate the technological approach to multimedia from a deeper and more philosophical approach. By the middle of the 1960s, the term “multimedia” was being applied to any form of artistic experiment that involved several media used at the same time. The stereotypical conception of multimedia was the kind of activity labeled a “multimedia happening” that one could see in art schools and museums around the world. It is

446

K. Friedman and L. Díaz

represented by motion picture clichés of the 1960s as a kind of post-beatnik séance with someone reading poetry while musicians play drums and saxophones over the rumbling of audiovisual equipment as dancers move around sculptured and random objects against a background of film projections or a light show. This image was reinforced in the public imagination by the light shows that accompanied rock concerts at such venues as the Fillmore Auditorium or the Avalon Ballroom in San Francisco or Timothy Leary’s psychedelic evenings in New York. The comparison is unfair, since these light shows were, in fact, sophisticated multimedia presentations created by teams of artists who worked carefully to achieve artistic effects far superior to the incomprehensible collation of sounds and images seen in the stereotyped representations of multimedia presentations sometimes depicted in television and film. The era was typified by both kinds of multimedia events. Nevertheless, there is a distinction between multimedia and intermedia. The generation and birth of new art forms does not always happen with multimedia. In multimedia, by definition, many things happen at once. George Maciunas’s expanded arts charts and diagrams explored these issues in visual form (Hendricks 1982: 270–271; 1989: 329–332, 350; Schmidt-Burkhardt 2003). In Maciunas’s terms, most multimedia presentations were “neo-Baroque.” They remain separated. In contrast, intermedia tended toward the unified sensibility that Maciunas labeled “neo-haiku.” Even when they emerge from several forms, they flow into one stream. Many kinds of action and many things taking place at the same time typify multimedia pieces. This often involved scattered and confused collations that pull the spectator in different directions. Intermedia tend to involve focus and clarity of thought. This distinction is captured in the difference between happenings and events. Happenings were largely multimedia forms, even though many happenings were more sophisticated than most multimedia events. Happenings generally involved lots of action on an explosive field. Low in coherence and intentional value, the happening did not last long as a medium for art, performance, or theater. Nevertheless, the influence of the happening continues to echo in these fields, and in such related fields as music and television. While Fluxus and happenings were often linked in histories, exhibitions, and some theories, they related to each other in dialectic of opposing tendencies. They were twinned polarities, two faces of the same coin. It is no coincidence that half the happening artists became painters and sculptors, Claes Oldenburg, for example, or Jim Dine, and Red Grooms. The other half was active in Fluxus. These were such artists as Al Hansen, Dick Higgins, Wolf Vostell, and Milan Knizak. Allan Kaprow’s work offers an interesting point of reflection. Kaprow moved from art history and painting into happenings. Unlike the others, he neither retreated to painting nor worked in the Fluxus context. Instead, his work became increasingly intimate in its focus, taking on the psychological tone and event-oriented edge characteristic of much Fluxus work. Al Hansen was another unique figure (Hansen 1965; Hoffmann 1996). Of all the early “happeners,” only Hansen maintained the original happenings ethos. Although he was closely allied to the Fluxus artists, his rollicking, chaotic work typified the neo-Baroque tone of the earliest happenings.

19

Intermedia, Multimedia, and Media

447

Reconsidering Multimedia From our perspective, considering multimedia involves three separate and related challenges. First is the challenge of using media – any media and all kinds of media – in combination to deliver information, communication, education, or entertainment. Second is the challenge of using appropriate techniques and technology to translate, store, transmit, and deliver those media. Third is the challenge of transcending the stereotyped usage of the term “multimedia,” with all that it has come to imply. The recent usage of the term multimedia has become common for an understandable reason. That reason is the location of the multimedia phenomenon in the hands of any number of high technology companies and organizations who use their ability to combine computers, DVD, Internet sites, or other related technologies in the second sense of the word to deliver multimedia content as we have defined it in the first sense of the word. Before considering the future of multimedia, it is worth examining a few historical examples of convincing multimedia. Multimedia began in the earliest stages of prehistory. The difference between describing today’s technology-driven forms of multimedia and early forms is that they generally have not been described under the rubric of multimedia. The symbolic value of pageantry, the use of symbolism and drama to heighten the effect of a message has always had a role in information management. Because of this well-known principle, the ability of today’s technology to dress a message in multiple forms and enhance delivery through several channels has affected the growth of multimedia. Shannon and Weaver’s (1949) mathematical formula established a well understood theory of information and the role of media in communicating information from one point to another. Earlier models of multimedia were less precise, but the very ambiguity of earlier multimedia examples gave them a tactile richness that enhanced communication through multiple channels. In many cases, this has been helpful. Dramatizing action, embellishing words with images, rendering images interactive all have their purpose. Opera is an example of an early multimedium. Stage drama is another. Today, a good example of a common multimedia application is the realization of drama in film. A perfect example of this is Kenneth Branagh’s (1989) rendition of Shakespeare’s [1599, 1623] (1961, 1991) classic play (theatre) Henry V. This also demonstrates the evolution of one multimedia artwork through four centuries. The first performance of Henry V took place in Shakespeare’s London four centuries ago. There was no way in those days to convincingly render the vast scenes that the playwright set before his audience. Shakespeare’s prologue to Henry V describes the problem of rendering the action of the play on stage: . . . but pardon, gentles all, The flat unraised spirits that hath dar’d On this unworthy scaffold to bring forth So great an object: can this cockpit hold The vasty fields of France? Or may we cram Within this wooden O the very casques That did affright the air at Agincourt? (Henry V, Prologue: 8–14)

448

K. Friedman and L. Díaz

Shakespeare’s technology did not permit the proper rendition of battle scenes. Stage battles were limited to duels among a few players or the “brawl ridiculous” (Henry V, IV: Chorus, 51) that failed to capture the reality of the battlefield. Plays rely on the imagination for envisioning action much as books or poetry do. Shakespeare appealed to his audience, inviting them to fill in missing scenes and embellish sketched action with their own “imaginary forces” (Henry V, Prologue: 18). “Work, work your thoughts,” the chorus admonishes viewers, asking them to “eke out our performance with your mind” (Henry V, III: Chorus, 25, 35). Theater audiences from Shakespeare’s day to our own have been obliged to grip each “story; in little room confining mighty men, mangling by starts the full course of their glory” (Henry V, V, II: Chorus: 2–4). In realizing the grand design of Shakespeare’s historical drama, Kenneth Branagh’s film was closer to Shakespeare’s vision than a stage-bound realization. The actor-director-screen writer of the film captured the reality of late medieval battle. In doing so, he elevated a magnificent play into a grand multimedia performance using drama, action, choreography, geography, and music to portray an historical event in fruitful rendition (Fig. 2). Fig. 2 Three roles for one Henry V. Kenneth Branagh directs and acts in his screenplay’s film adaptation of the Battle of Agincourt in Shakespeare’s drama. (© Getty Images)

19

Intermedia, Multimedia, and Media

449

Audiences are now so used to film as an established medium that they do not conceive it as a multimedia form. This is as it should be. When multimedia is truly effective, the viewer should not so much be aware of the media as be aware of the spectacle revealed. At some point, the elements in a successful film may flow together so smoothly that what begins on the technical scale as multimedia, merges with the spectator’s horizons into a form of intermedia. Pioneer French film director and producer Abel Gance’s use of multiple panorama formats (including polyvision) in his biopic Napoleon is early an example of this. McLuhan’s observation that “the medium is the message” helps to describe the unconscious perception of media and the failure to consciously perceive them. An effective medium is so much an extension of the witness that the medium recedes into the background while content is the foreground. McLuhan argued that the way we perceive, the way we communicate, and the way we interact with our communication media affects the way we think. This mirrors similar views in the social psychology of George Herbert Mead and in Wilhelm Dilthey’s hermeneutics of the life world. This vision of media proposes that the way human beings give voice to the world through media shape perception, paralleling the well-known Sapir-Whorf hypothesis that language affects the way human beings perceive the world (Sapir 1973; Whorf 1969). When perception is embedded in the flow of daily experience, the ways in which we perceive are not subject to conscious inspection. To see how we see, we must step back from the subject and consider the act of seeing. To hear how we speak, we must step back from what we say and attend to the act of speaking. In other words, we need to become more cognizant of how embodiment colors our knowledge of being in the world. Our use of multimedia is so prevalent that most of us fail to realize that we are involved in producing and perceiving multimedia. Only the latest and bestadvertised multimedia projects capture our attention. As a result, the other forms of multimedia, including the simplest and most effective, disappear as multimedia forms precisely because they are simple, effective, and so widely used that their nature as multimedia becomes invisible.

Conclusion: Toward an Archeology of Intermedia, Multimedia, and Media Recovering the histories and multiple forms of intermedia and multimedia serves an important purpose in developing richer approaches to new media in the future. This work is also being undertaken in fields such as economic history, studies in science and technology, communication theory, and other fields. This is a history that has yet to be written is from an interdisciplinary history of intermedia and multimedia crossing the boundaries of art, technology, and communication, linking them for deeper understanding. The past half-century has seen several important studies on media written by scholars in these fields. Of relevance are media archaeological

450

K. Friedman and L. Díaz

approaches combine excavation metaphors with a Foucauldian nonlinear viewpoint of history (see Zielinski (2006) regarding the exploration of how unique practices illuminate discourse; Huhtamo and Parikka applications of an archeological approach to the history of media technologies (2011); and Parikka (2012) for the symbiosis between the natural and the artificial). Stuedahl (2001: 1) has also captured the concept nicely where she discusses “constructing new communication forms with resources from the past.” Continued research and exhibitions of works by already mentioned Fluxus and Experiments in Art and Technology (E.A.T.) participants are excellent opportunities to deepen our knowledge that can not only yield new understanding regarding shifts in epistemic frontiers but also illuminate truly revolutionary art and science collaborations. For example, already in 1968, Jean Dupuy’s artwork Heart Beats Dust (Cone Pyramid) registers concern regarding self-quantification and embodiment. The work was originally shown at The Machine as Seen at the End of the Mechanical Age, Museum of Modern Art, New York, NY, November 27, 1968–August 24, 1969 (see Dupuy, Jean “Heart Beats Dust” video in the “References” section). Variations of the installation depict red dust matter vibrating to the sound of human heartbeats. The number of observers interacting with the work through a stethoscope varies from one to six. As we argue in this chapter, such a comprehensive study of intermedia and multimedia would require four specific research streams. The first is a history of intermedia and multimedia in the arts and communication from prehistory to modern times. The second is a history of the theories and conceptual development of intermedia and multimedia in the twentieth century. The third is a taxonomy and description of communication and art media, including dead and obsolete media. The fourth is a conceptual research program examining the possible future of new media and new uses of intermedia and multimedia. Fragmented and partial efforts in all these fields exist already. A systematic review and summary of current literature would provide a solid foundation for future work. This would involve the first three streams, and it would make a serious contribution toward realizing them. The fourth program is more complex, linking applied research and development to theory. This program will require the involvement of new emerging disciplines and techniques as well as scholars and practitioners working in parallel with the other three programs. Here Díaz (1997: 287) has proposed the Digital Archaeology as a new form of creative practice that uses virtual environments and technology as instruments that can afford us “new ways of understanding research data” in context. Managing knowledge, transmitting information, and creating art depend in great part on the tools and media we use. This program begins with the simple but challenging premise that the concepts of intermedia and multimedia have been lost or vastly diminished since their inception in the early 1960s. It follows from this premise that a reconsidering intermedia and multimedia together will lead to new concepts, developments, and practical outcomes. The concept of intermedia has never been fully understood. The concept of multimedia has been reduced from a broad, rich framework for integrated

19

Intermedia, Multimedia, and Media

451

communication to a series of sometimes-fruitless technical innovations. By reclaiming the multiple methods of communication and demonstrating the ways in which well-understood and even primitive communication tools can be allied with contemporary technologies, we can open avenues toward a dense, multi-channeled communication practice. Internet, World Wide Web, multimedia, and applications such as intranet technology affect the way we live and work. Most of these conceptions involve a limited variety of tools associated with high technology. A broader framework must consider the hundreds of intersections of old technologies and new. A robust research program involves considering intermedia and multimedia in terms of the widest possible variety of tools for delivering information, education, and entertainment in useful and useable forms. Dick Higgins’s 1966 Intermedial Object #1 has continued to surface in different incarnations over the years. It was last seen in Geneva in a 1997 exhibition (Bovier and Cherix 1997: 65–66). The picture of Higgins’s original score somehow seems more solid and stable than the humble and somewhat ambiguous object pictured on the subsequent page. The object was built in the early 1970s. The Geneva realization of Intermedial Object #1 marries the solid craftsmanship of home workshop carpentry with the good-natured zeal of early installation art. The image has the delightful appearance of a child’s toy lantern, 7 points up on the x-axis of Higgins’s humor scale (Fig. 3). Managing knowledge, transmitting information, and creating art depend in great part on the tools and media we use. This program begins with the simple but challenging premise that the concepts of intermedia and multimedia have been lost or vastly diminished since their inception in the early 1960s. It follows from this premise that a reconsidering intermedia and multimedia together will lead to new concepts, developments, and practical outcomes.

Fig. 3 Intermedial Object #1 by Dick Higgins. (© Archives Ecart, Genève)

452

K. Friedman and L. Díaz

The media archeology of this object offers an opening to an unexplored continent. The history of intermedia and multimedia are not simply valuable because they disclose an intriguing and little-known past. Retrieving this history also opens a renewed vision of the open future.

Bibliography Essay A bibliographic essay developing these themes would begin with these books: Acland and Buxton 1999; Bailey 1996; Bell 1973, 1999; Beniger 1986; Block 1990; Boorstin 1985, 1998; Boyer 1996; Braudel 1979, 1980, 1992a, 1992b, 1992c; Burke and Ornstein 1997; Cairncross 1998; Castells 1983, 1989, 1996a, 1996b, 1996c, 1999, 2001; Castells and Hall 1994; Chandler 1977, 1994; Chandler and Cortada 2000; Droege 1998; Drucker 1973, 1990, 1993, 1996[1959]; Eisenstein 1979; Febvre and Martin 1997; Flichy 1995; Gimpel 1992; Hobart and Schiffman 1998; Innis 1950, 1951, 1980, 1995; Jensen 1987; Jones 1995, 1997, 1998, 2002; Kahn 1999; Landes 1983; Machlup 1962, 1970, 1978,1979, 1980, 1982a, 1982b, 1984; Machlup and Mansfield 1983; McLuhan 1962, 1964, 1967a, 1967b; McLuhan and Watson 1970; McNeill 1984; Mokyr 1992; Needham 1965; Needham, Ling, and de Solla Price 1960; Norman 1993; Ochoa and Corey 1995; O’Donnell 1998; Peters 1999; Postman 1993; Rifkin 1987; Robins and Webster 1999; Rosenberg and Birdzell 1986; Sassen 1991, 1994, 1996, 1998; Schumpeter 1981.

References Acland, Charles R., and William J. Buxton, eds. 1999. Harold Innis in the new century. Montreal/ Kingston: McGill-Queen’s University Press. Albright, Thomas. 1973. A dialogue with ken Friedman. Omaha: Joslyn Art Museum, Xerox Corporation and Omaha Flow Systems. Alciati, Andrea. 1531/2004. A book of emblems. The Emblematum liber in Latin and English. Trans. and ed. John F. Moffitt. Jefferson: McFarland Books. Bailey, James. 1996. After thought: The computer challenge to human intelligence. New York: Basic Books. Baugh, Kenneth, Jr. 1990. The methodology of Herbert Blumer. Critical interpretation and repair. Cambridge: CUP. Becker, Wolfgang. 1977. The pictorial process and intermedial activity. In Der Ausgestellte Kunstler: Museumskunst seit 45. Aachen: Neue Galerie, Sammlung Ludwig. Becker, Jürgen, and Wolf Vostell, eds. 1968. Happenings – Fluxus, Pop Art, Nouveau Réalisme. Hamburg: Rowohlt. Bell, Daniel. 1973/1999. The coming of post-industrial society. A venture in social forecasting. New York: Basic Books. ———. 1999. The coming of post-industrial society. A venture in social forecasting. New York: Basic Books. Beniger, James R. 1986. The control revolution. Technological and economic origins of the information society. Cambridge, MA: Harvard University Press.

19

Intermedia, Multimedia, and Media

453

Berger, Rene, Gillo Dorfles, Jorge Glusberg, Abraham Moles, and Udo Kulterman. 1980. Theoretical analysis of the intermedia art form. New York/Buenos Aires: Experimental Intermedia Foundation and Centro de Arte y Communicación. Block, René, ed. 1974. Multiples: Ein Versuch die Entwicklung des Auflagenobjektes darzustellen/ An attempt to present the development of the Object Edition. Berlin: Neuer Berliner Kunstverein. Block, Fred. 1990. Post-industrial possibilities. A critique of economic discourse. Berkeley: University of California Press. Blumer, Herbert. 1969. Symbolic interactionism. Perspective and method. Englewood Cliffs: Prentice-Hall. Boorstin, Daniel J. 1985. The discoverers. New York: Random House. ———. 1998. The seekers. The story of Man’s continuing quest to understand his world. New York: Random House. Bory, Jean-Francois, ed. 1968. Once again. New York: New Directions. Bovier, Lionel, and Christophe Cherix. 1997. L’irresolution commune d’un engagement equivoque. Ecart, Geneve, 1969–1982. Geneve: Mamco et Cabinet des estampes. Boyer, M. Christine. 1996. CyberCities. Visual perception in the age of electronic communication. New York: Princeton Architecture Press. Branagh, Kenneth. 1989. William Shakespeare’s Henry V. Braudel, Fernand. 1979. Afterthoughts on material civilization and capitalism. Baltimore: Johns Hopkins University Press. ———. 1980. On history. Chicago: University of Chicago Press. ———. 1992a. The structures of everyday life: The limits of the possible, Civilization and capitalism: 15th–18th century. Vol. 1. Berkeley: University of California Press. ———. 1992b. The wheels of commerce, Civilization and capitalism: 15th–18th century. Vol. 2. Berkeley: University of California Press. ———. 1992c. The perspective of the world, Civilization and capitalism: 15th–18th century. Vol. 3. Berkeley: University of California Press. Brecht, George, and Robert Filliou. 1967. Games at the Cedille, or the Cedille takes off. New York: Something Else Press. Breder, Hans. 1995. Intermedia: Enacting the liminal. Performance Arts Journal XVII, 50/51. Brockhampton Press. 1994. Dictionary of ideas. London: Brockhampton Press. Burke, James, and Robert Ornstein. 1997. The Axemaker’s gift: Technology’s capture and control of our minds and culture. New York: Tarcher Putnam. Bush, Vannevar. 1945. As we may think. The Atlantic Monthly 96 (1): 101–108. Cage, John. 1961. Silence. Middletown: Wesleyan University Press. ———, ed. 1969. Notations. New York: Something Else Press. Cairncross, Frances. 1998. The death of distance. How the communications revolution will change our lives. London: Orion Business Books. Castells, Manuel. 1983. The city and the grassroots. A cross-cultural theory of urban social movements. Berkeley: University of California Press. ———. 1989. The informational city: Information technology, economic restructuring, and the urban-regional process. Oxford: Blackwell. ———. 1996a. The information age. Economy, society, and culture. Vol. I. The rise of the network society. Oxford: Blackwell Publishers. ———. 1996b. The information age. Economy, society, and culture. Vol. II. The power of identity. Oxford: Blackwell Publishers. ———. 1996c. The information age. Economy, society, and culture. Vol. III. End of millennium. Oxford: Blackwell Publishers. ———, ed. 1999. Critical education in the new information age. Manuel Castells, Ramon Flecha, Paulo Freire, Henry A. Giroux, and Peter McLaren. New York: Rowman and Littlefield Publishing.

454

K. Friedman and L. Díaz

———. 2001. The internet galaxy. Reflections on internet, business, and society. Oxford: Oxford University Press. Castells, Manuel, and Peter Hall. 1994. Technopoles of the world. The making of 21st century industrial complexes. London: Routledge. Chandler, Alfred D. 1977. The visible hand. The managerial revolution in American business. Cambridge, MA: The Belknap Press/Harvard University Press. ———. 1994. Scale and scope: The dynamics of industrial capitalism. Cambridge, MA: Belknap Press. Chandler, Alfred D., and James W. Cortada, eds. 2000. A nation transformed by information. New York: Oxford University Press. Clark, Colin. 1940. Conditions of economic progress. London: Macmillan. Corner, Philip, Alison Knowles, Benjamin Patterson, and Tomas Schmit. 1965. The four suits. New York: Something Else Press. Crane, Michael, and Mary Stofflett, eds. 1984. Correspondence art: Source book for the network of international postal art activity. San Francisco: Contemporary Arts Press. Di Felice, Attanasio. 1980. From Bernini to Beuys: Historical sources of performance and timespace art. National Arts Guide 2 (4): 24–32. (Partially reprinted in Gregory Battcock, Robert Nickas, eds. The Art of Performance. New York: E. P. Dutton, 1984). ———. 1984. Renaissance performance: Notes on prototypical artistic actions in the age of the platonic princes. In The art of performance. A critical anthology, eds. Gregory Battcock and Robert Nickas, pp 3–23. New York: E. P. Dutton, Inc. Díaz, Lily. 1997. Digital archeology: Design research and education–Connecting historical narratives and digital environments, 283–287. Leonardo: The MIT Press. Droege, Peter, ed. 1998. Intelligent environments. Spatial aspects of the information revolution. Amsterdam: Elsevier. Drucker, Peter F. 1973. The age of discontinuity. Guidelines to our changing society. New York: Harper Torchbooks. ———. 1990. The new realities. London: Mandarin. ———. 1993. Post-capitalist society. Oxford: Butterworth-Heinemann. ———. 1996/1959. Landmarks of tomorrow: A report on the new “post modern” world. With a new introduction by the author. Rutgers: Transaction Publishers. Dupuy, Jean. 2016. Hearts beats dust (Cone Pyramid) in operation Color film 40 900 , Production: Musée du quai Branly – Jacques Chirac, for the exhibition “Persona, étrangement humain” – January 26 – November 13, 2016. https://vimeo.com/425178489. Accessed 24 Apr 2023. Eisenstein, Elizabeth L. 1979. The printing press as an agent of change. Cambridge: CUP. Erkki, Huhtamo, and Parikka Jussi, eds. 2011. Media archaeology. Berkeley/Los Angeles: University of California Press. Febvre, Lucien, Henri-Jean Martin, and David Gerard. 1997. The coming of the book: The impact of printing 1450–1800, Verso classics, 10. London: Verso Books. Flichy, Patrice. 1995. Dynamics of modern communication. The shaping and impact of new communication technologies. London: Sage. Flynt, Henry. 1963. Essay: Concept art (Provisional version). In An anthology, ed. La Monte Young. New York: Jackson Mac Low, no page numbers. (Reprinted by Heiner Friedrich, 1973). Frank, Peter. 1967. Intermedia and happenings: When the arts and life meet. Unicorn 17/2 (Spring). ———. 1983. Something else press. An annotated bibliography. Kingston, New York: McPherson and Company. Friedman, Ken. 1967. Intermedia and expanded art. San Francisco: Department of RadioTelevision-Film and Experimental College, San Francisco State College, San Francisco, California. ———. 1972. The aesthetics. Regina: University of Saskatchewan. ———. 1973. The aesthetics. Devon: Beau Geste Press. ———. 1998a. Cities in the information age: A Scandinavian perspective. In The virtual workplace, ed. Magid Igbaria and Margaret Tan, 144–176. Hershey: Idea Group Publishing. ———. 1998b. Dick Higgins, 1938–1998. Umbrella 21 (3/4).

19

Intermedia, Multimedia, and Media

455

———. 1998c. Digital convergence and multimedia: A cultural and technological inventory. In Beyond convergence. Proceedings of the twelfth biennial international telecommunications society conference. Stockholm: ITS, Center for Information and Communications Research, Stockholm School of Economics, and the Swedish Transport and Communications Research Board. ———, ed. 1998d. The Fluxus reader. Chichester: Academy Editions, Wiley. ———. 1998e. Information, place and policy. Built Environment 24 (2–3): 83–103. ———. 1998f. Multiple views on multimedia and the world wide web: A European perspective. In Supplement to INFO2000 project proposal. Contextual information for biological conservation. Brussels: Union of International Associations, World Conservation Monitoring Centre, Institute for European Environmental Policy, and the Nordic Centre for Innovation. ———. 1999. Dick Higgins, 1938–1998. In Umbrella. The anthology, ed. Judith A. Hoffberg, 157–163. Santa Monica: Umbrella Editions. ———. 2002. Intermedia. In The encyclopedia of new media, ed. Steve Jones, 245–247. Thousand Oaks: Sage. ———. 2003a. Intermedia: four histories, three directions, two futures. In Good, bad, irrelevant. The user and the future of information and communication technologies. A transdisciplinary, proactive and collaborative knowledge-building conference. COST Action 269. 3–5 September 2003, 460–464. Helsinki: Media Lab, University of Art and Design Helsinki UIAH. ———. 2003b. The wealth and poverty of networks. Part one: Networks in a space of flows. Issue 1. July 2003. New Media Poetics. ———. 2003c. The wealth and poverty of networks. Part two: Intermedia and media convergence before the internet. Issue 2. December 2003. New Media Poetics. ———. 2004. Jack Ox and intermedia. In Jack Ox, ed. Miroslaw Borusiewicz, 20–39. Lodz: Muzeum Sztuki in Lodz. ———. 2005a. Intermedia: Four histories, three directions, two futures. In Intermedia: Enacting the liminal, ed. Hans Breder, and Klaus-Peter Busse, publisher. Schriftenreihe des Instituts fuer Kunst an der Universitaet Dortmund. Abteilung 3: Intermedia Studien. BOD i.V. Dortmund. Dortmund: University of Dortmund Press. ———. 2005b. The wealth and poverty of networks. In At a distance: Precursors to internet art and activism, ed. Annemarie Chandler and Norie Neumark, 408–422. Cambridge, MA: MIT Press. ———. 2018. Thinking about Dick Higgins. In Intermedia, Fluxus and the Something Else Press. Selected writings by Dick Higgins, ed. Steve Clay and Ken Friedman, 12–21. Catskill/New York: Siglio Press. Friedman, Ken, and Georg M. Gugelberger. 1976. The stamp and stamp art. Front 1 (4): 6–10. Getty Research Institute. 2018. Art & architecture thesaurus ® online. http://www.getty.edu/ research/tools/vocabularies/aat/. Accessed 14 Feb 2022. Gimpel, Jean. 1992. The medieval machine. The industrial revolution of the middle ages. London: Pimlico. Glusberg, Jorge. 1969. Argentina Inter-Medios. Argentina inter media. Buenos Aires: Centro de Arte y Communicacion. ———. 1971. Arte de Sistemas. Buenos Aires: Museo de Arte Moderno. Goetze, Jochen, Klaus Staeck, and Friedrich Gerling, eds. 1969. Intermedia 1969. Heidelberg: Edition Tangente. Gordon, W. Terrence. 1997. McLuhan for beginners. London: Writers and Readers. Hansen, Al. 1965. A primer of happenings and time/space art. New York: Something Else Press. Hendricks, Jon, ed. 1982. Fluxus, etc.: The Gilbert and Lila Silverman collection. Southfield Hills: Cranbrook Museum of Art. ———. 1983a. Fluxus, etc.: Gilbert and Lila Silverman collection. Addenda I. New York: Ink. ———. 1983b. Fluxus, etc.: Gilbert and Lila Silverman collection. Addenda TWO. Pasadena: California Institute of Technology. Baxter Art Gallery. ———. 1989. Fluxus codex. New York: Harry N. Abrams, Inc.

456

K. Friedman and L. Díaz

Hersey, Paul, and Kenneth H. Blanchard. 1972. The management of organizational behavior. Utilizing human resources. Englewood Cliffs: Prentice-Hall. Higgins, Dick. 1966a. Intermedia. Something Else Newsletter 1 (1): 1–6. ———. 1966b. Intermedial object #1. New York: Something Else Press. ———. 1967. Intermedia. Wormwood Review 7 (1): 15–22. ———. 1968. Intermedia. Library Journal (June 1), 2213–2214. ———. 1969a. Foew & ombwhnw: A grammar of the mind and a phenomenology of love and a science of the arts as seen by a stalker of the wild mushroom. New York: Something Else Press. ———. 1969b. Toward the 1970s. Somerville: Abyss Publications. ———. 1984. Horizons. The poetics and theory of the intermedia. Carbondale: Southern Illinois University Press. ———. 1997. Modernism since postmodernism: Essays on intermedia. San Diego: San Diego State University Press. ———. 2001. Intermedia. In Multimedia. From Wagner to virtual reality, ed. Randal Packer and Ken Jordan, 27–32. New York: W.W. Norton. Hobart, Michael E., and Zachary S. Schiffman. 1998. Information ages. Literacy, numeracy, and the computer revolution. Baltimore: The Johns Hopkins University Press. Hoffmann, Heike, ed. 1996. Al Hansen: An introspective. Köln: Kölnisches Stadtsmuseum. Hoonard, Will C. van den. 1997. Working with sensitizing concepts. Analytical field research. Qualitative research methods series 41. A Sage University paper. Thousand Oaks: Sage. Innis, Harold. 1950. Empire and communications. Oxford: OUP. ———. 1951. The bias of communication. Toronto: University of Toronto Press. (Reprinted with an introduction by Paul Heyer and David Crowley. Toronto: University of Toronto Press, 1995.) ———. 1980. In The idea file of Harold Adams Innis, ed. William Christian. Toronto: University of Toronto Press. ———. 1995. In Staples, markets, and cultural change, ed. D. Drache. Montreal/Kingston: McGill-Queen’s University Press. Jensen, Oliver. 1987. The American heritage history of railroads in America. New York: Bonanza Books. Jones, Steven G., ed. 1995. Cybersociety. Computer-mediated communication and community. London: Sage. ———., ed. 1997. Virtual culture. Identity and communication in cybersociety. London: Sage. ———., ed. 1998. Cybersociety 2.0. Revisiting computer-mediated communication and community. London: Sage. ———., ed. 2002. Encyclopedia of new media. London: Sage. Judge, A.J.N. 1984. Forms of presentation and the future of comprehension. Brussels: Union of International Associations. Kahn, Douglas. 1999. Noise, water, meat. A history of sound in the arts. Cambridge, MA: MIT Press. Klintberg, Bengt af. 1966. Stockholmsspelet. Stockholm: Norstedt och Soener. ——— af. 1977. Dick Higgins, intermediakonstnar. Kalejdoskop 3–4/51–52. Klüver, Billy. 1980. E.A.T. bibliography, August 12, 1965–January 18, 1980. Revised edition. New York: Experiments in Art and Technology. ———. 1983. What are you working on now? A pictorial memoir of the 60’s. New York: Experiments in Art and Technology. Kosuth, Joseph. 1965. One and three chairs. Museum of Modern Art online exhibition. https:// www.moma.org/collection/works/81435. Accessed 14 Feb 2022. Landes, David S. 1983. Revolution in time. Clocks and the making of the modern world. Cambridge, MA: The Belknap Press of Harvard University Press. Lenat, Douglas B. 1983. Toward a theory of heuristics. In Methods of heuristics, ed. Rudolf Groner, Marina Groner, and Walter Bischof, 351–404. Hillsdale: Lawrence Erlbaum Associates, Publishers. Lester, Elenore. 1966. So what happens after happenings? The New York Times, September 4, 1966. http://www.brockman.com/press/1966.09.04.NYT.html. Accessed 14 Feb 2022.

19

Intermedia, Multimedia, and Media

457

———. 1968. Intermedia: Tune in, turn on – And walk out? New York Times Magazine, May 12, 30–34. Machlup, Fritz. 1962. The production and distribution of knowledge in the United States. Princeton: Princeton University Press. ———. 1970. Education and economic growth. Lincoln: University of Nebraska Press. ———. 1978. Information through the printed word: The dissemination of scholarly, scientific, and intellectual knowledge. New York: Praeger Publishers. ———. 1979. Knowledge and information. Princeton: Princeton University Press. ———. 1980. Knowledge and knowledge production. Princeton: Princeton University Press. ———. 1982a. Knowledge, its creation, distribution, and economic significance. Vol. 1. Knowledge and knowledge production. Princeton: Princeton University Press. ———. 1982b. Knowledge, its creation, distribution, and economic significance. Vol. 2. The branches of learning. Princeton: Princeton University Press. ———. 1984. Knowledge, its creation, distribution, and economic significance. Vol. 3. The economics of information and human capital. Princeton: Princeton University Press. Machlup, Fritz, and Una Mansfield, eds. 1983. The study of information. New York: Wiley. Masters, R.E.L., and Jean Houston, eds. 1968. Psychedelic art. New York: Grove Press. Matthaei, Renate. 1972. Situation und Theorie der intermedia. Merkur 293 (September): 884–899. McLuhan, Marshall. 1962. The Gutenberg galaxy. Toronto: University of Toronto Press. ———. 1964. Understanding media. New York: Signet Books. ———. 1967a. The medium is the massage. An inventory of effects. With Quentin Fiore. New York: Bantam Books. ———. 1967b. Verbo-voco-visual explorations. New York: Something Else Press. McLuhan, Marshall, and Wilfred Watson. 1970. From cliché to archetype. New York: Pocket Books. McNeill, William H. 1984. The pursuit of power: Technology, armed force and society since A.D. 1000. Chicago: University of Chicago Press. Merriam-Webster, Inc. 2020. Merriam-Webster’s collegiate dictionary, 11th edition. Springfield. https://www.merriam-webster.com/dictionary/medium?src¼search-dict-box. Accessed 17 Feb 2022. Milman, Estera, guest editor. 1992. Fluxus: A conceptual country. Visible language 26/1-2. Providence: Rhode Island School of Design. Mokyr, Joel. 1992. Lever of riches: Technological creativity and economic progress. Oxford: Oxford University Press. Mollerup, Per. 1997. Marks of excellence. The history and taxonomy of trademarks. London: Phaidon. Needham, J. 1965. Science and civilization in China, Vol. 4, Pt. 2: Mechanical engineering. Cambridge: CUP. Needham, Joseph, W. Ling, and D.J. de Solla Price. 1960. Heavenly clockwork: The great astronomical clocks of medieval China. Cambridge: CUP. Norman, Donald A. 1993. Things that make us smart. Reading: Addison-Wesley. O’Donnell, James J. 1998. Avatars of the word. From papyrus to cyberspace. Cambridge, MA: Harvard University Press. Ochoa, George, and Melinda Corey. 1995. The timeline book of science. New York: Ballantine. OED. 2021. OED online. Oxford English Dictionary. Ed. J. A. Simpson and E. S. C. Weiner, 2nd edition. Oxford: Clarendon Press, Oxford University Press. https://www.oed.com. Accessed 14 Feb 2022. Pacey, Arnold. 1992. The maze of ingenuity. Ideas and idealism in the development of technology. Cambridge, MA: The MIT Press. Packer, Randall, and Ken Jordan, eds. 2001. Multimedia: From Wagner to virtual reality. New York: Wiley. Paik, Nam June. 1964. Utopian laser television. In Manifestoes. New York: Something Else Press. ———. 1976. Media planning for the post-industrial society. (Reprinted in The Electronic Superhighway. Nam June Paik and Kenworth W. Moffett, eds. New York, Seoul, and Fort Lauderdale:

458

K. Friedman and L. Díaz

Holly Solomon Gallery, Hyundai Gallery, and the Fort Lauderdale Museum of Art, 1995, pp. 39–47.) Paik, Nam June, and Kenworth W. Moffett, eds. 1995. The electronic superhighway. New York/ Seoul/Fort Lauderdale: Holly Solomon Gallery/Hyundai Gallery/Fort Lauderdale Museum of Art. Parikka, Jussi. 2012. What is media archaeology? Cambridge, UK: Polity Press. Peters, John Durham. 1999. Speaking into the air. A history of the idea of communication. Chicago: University of Chicago Press. Phillpot, Clive, and Jon Hendricks, eds. 1988. Fluxus: Selections from the Gilbert and Lila Silverman collection. New York: Museum of Modern Art. Porter, Bern. 1971. I’ve Left. Newhall: Something Else Press. Postman, Neil. 1993. Technopoly. The surrender of culture to technology. New York: Vintage Books. Ravicz, Marilyn Ekdahl. 1974. Aesthetic anthropology: Theory and analysis of pop and conceptual art in America. Doctoral dissertation. Los Angeles: Department of Anthropology, University of California. Rifkin, Jeremy. 1987. Time wars. The primary conflict in human history. New York: Touchstone – Simon and Schuster. Robins, Kevin, and Frank Webster. 1999. Times of the technoculture. From the information society to the virtual life. London: Routledge. Rosenberg, Nathan, and L.E. Birdzell Jr. 1986. How the west grew rich. The economic transformation of the industrial world. New York: Basic Books. Sapir, Edward. 1973. Selected writings of Edward Sapir in language, culture and personality. Berkeley: University of California Press. Sassen, Saskia. 1991. The global city: New York, London, and Tokyo. Princeton: Princeton University Press. ———. 1994. Cities in the world economy. Thousand Oaks: Pine Forge/Sage. ———. 1996. Losing control? Sovereignty in an age of globalization. New York: Columbia University Press. ———. 1998. Globalization and its discontents. New York: The New Press. Schmidt-Burkhardt, Astrit. 2003. Maciunas’ learning machines: From art history to a chronology of fluxus. Detroit/Berlin: The Gilbert and Lila Silverman Fluxus Collection and Vice Versa Verlag. Schumpeter, Joseph. 1981. Capitalism, socialism, and democracy. London: George Allen and Unwin. Shakespeare, William. 1961. The complete works of William Shakespeare. Ed. Hardin Craig. Chicago: Scott, Foresman and Company. ———. 1991/1623. King Henry V. Ed. J. H. Walter. The Arden Edition of the Works of William Shakespeare. London: Routledge. Shannon, Claude E., and Warren Weaver. 1949/1964. The mathematical theory of communication. Urbana: University of Illinois Press. SOED. 1993. The new shorter Oxford English dictionary. Ed. Lesley Brown. Oxford: Clarendon Press, Oxford University Press. Spoerri, Daniel. 1966. An Anecdoted topography of chance. Ed. with annotations by Emmett Williams. New York: Something Else Press. Staeck, Klaus, ed. 1969. Intermedia 1969. Düsseldorf: Editions Tangente. Stuedahl, Dagny. 2001. Design as multidisciplinary negotiation and learning – Constructing new communicational forms with resources from the past. Project proposal for doctoral dissertation work in the NEMLIG project. Intermedia Department. University of Oslo. The Research Council of Norway Project number: 132550. Theall, Donald F. 1992. Beyond the orality/literacy dichotomy: James Joyce and the pre-history of cyberspace. Postmodern Culture 2/3 (May 1992). http://www.iath.virginia.edu/pmc/text-only/ issue.592/theall.592. Accessed 14 Feb 2022.

19

Intermedia, Multimedia, and Media

459

Tofts, Darren, and Murray McKeich. 1998. Memory trade: Prehistory of cyberculture. Sydney: Interface Books. ———. 1999. Parallax: Essays on art, culture, and technology. Sydney: Interface Books. Tofts, Darren, Annemarie Jonson, and Alessio Cavallaro. 2003. Prefiguring cyberculture: An intellectual history. Cambridge, MA: MIT Press. Tuchman, Maurice. 1971. A report on the Art and Technology Program of the Los Angeles County Museum of Art, 1967–1971. Los Angeles: Los Angeles County Museum of Art. Union of International Associations. 1998. INFO2000 project proposal. Contextual information for biological conservation. Brussels: Union of International Associations, World Conservation Monitoring Centre, Institute for European Environmental Policy, and the Nordic Centre for Innovation. Walker, John A. 1977. Glossary of art, architecture, and design since 1945. 2nd ed. Hamden: Linnet Books. Whorf, Benjamin Lee. 1969. Language, thought and reality. Selected writings of Benjamin Lee Whorf. Cambridge, MA: MIT Press. Youngblood, Gene. 1970. Intemedia theater. In Expanded cinema, 365–386. London/New York: Studio Vista/Dutton. Zielinski, Siegfried. 2006. Deep time of the media. Towards an archaeology of Hering and seeing by technical means. Cambridge, MA: The MIT Press. Zurbrugg, Nicholas. 1993. The parameters of postmodernism. London: Routledge. ———, ed. 2004. Art, performance, media: 31 interviews. Minneapolis: University of Minnesota Press.

Citational Aesthetics: For Intermediality as Interrelation

20

Mieke Bal

Contents Introduction: “Inter-” for Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermezzo 1: Recycling Names (to Make Art Socially Relevant) . . . . . . . . . . . . . . . . . . . . . . . . . . . . Quotation Through Mutual Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermezzo 2: Recycling Myths (to Bring Science and Religion into Discussion) . . . . . . . . . . . . Semiosphere: Wavering and Hovering Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermezzo 3: Characters as (Inter-)Media Products (to Bring Theory in for Sense-Making) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality in and for the World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermezzo 4: The Artistic-Political Power of Imperfection (to Shame the Bad Guys) . . . . . . . Conclusion: Analyzing and Teaching Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

462 464 466 475 477 483 485 487 489 490

Abstract

In this chapter the concept of intermediality is brought in contact with a specific branch of that approach, namely, quotation. In line with a concept of intertemporality called “pre-posterous history,” quotation is a specific way of integrating older and newer media products and their connections to different media. The chapter develops a variety of forms of quotation through a close look at cases from recent media history. These quoting media products present theoretical issues as they are implied in the representational practice of the past yet can only be perceived through the detour of the present, which transforms both these theoretical issues and the preceding works that are quoted. The insistence on the preposition “inter” implies a focus on relationality, rather than the border-crossing that “trans-” signifies. Meaning-making appeals to semiotic, a theory of signs and sign-use not bound to any specific medium. The integration of theory and the

M. Bal (*) Amsterdam School of Cultural Analysis (ASCA), Amsterdam, The Netherlands e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_49

461

462

M. Bal

practice of analysis of concrete cases is also an element of the chapter’s lesson: intermediality’s teachability. Keywords

Semiotics · Semiosphere · Meaning-making · Medial “impurity,” Analysis · Quotation · Pre-text · Pre-posterous history

Introduction: “Inter-” for Integration Intermediality is seen as a transgression of media borders and an integration of elements from different media types (according to Lars Elleström’s terminology as laid out in his 2019 and 2021 books). In order to make this less focused on transgression and foregrounding more strongly the integration, I consider quotation: the adoption into one work, or “media product” – to use a term that is not mediumspecific – of elements from another. These adoptions can be noticed, which makes the cases more easily “teachable”: the insights to be gained can be transferred to others with clarity. Therefore, this chapter of the handbook is organized around theoretical issues representing aspects of intermediality seen as quotation: as a recasting of past words and images in new, later, or contemporary media products. Each case shows specific ways in which quotation is vital to the new artwork as well as to the source from which it is derived and for which, due to the mutuality of time (as opposed to chronological linearity) it thereby becomes, in turn, a source. Such quoting media products thus present theoretical issues as they are implied in the representational practice of the past yet can only be perceived through the detour of the present, which transforms both these theoretical issues and the preceding works that are quoted. The concept of quotation, which serves as the central theoretical focus or “hub” of this chapter, will lead us beyond the common understanding of quotation. This has aptly been summarized by McEvilley (1993: 168–169). This author rightly points out that quotation is not a unified practice with unified goals. But, going beyond McEvilley’s differentiation of the art practice called “quotationalism,” I will explore how this practice redefines and complicates the notion of quotation itself, as a crucial aspect of media products as well as transfers, and transformations, from one media product to another, through inflections of their respective media. In short, quotation is indispensable to understand intermedial practice and the media products that practice produces.1

I use the term “media products,” following Lars Elleström’s choice to avoid medium specificity as well as undefinable vague common words. If I use “text” or “image,” it is to denote a media product within a particular medium. Classical works at the background of this chapter are the two recent volumes edited by Elleström (2021), with a long introduction that is almost a handbook in itself, which he published after his earlier book Transmedial Narration from 2019 (with the same publisher).

1

20

Citational Aesthetics: For Intermediality as Interrelation

463

When the term “intermedial(ity)” appears, I cannot help but briefly mention the key role Lars Elleström has played in the development of that field. With the combination of rigor and creativity, he and the participants of the institute IMS at Linnaeus University in Växjö, Sweden, which he co-founded and shaped, have made a decisive step forward, to which I luckily have become a close witness and sometimes participant. As an academic who is also a filmmaker I am very much aware of intermediality. The inevitable integration of words and images, color, sound, narrativity, and technological effects and more clearly demonstrates that no single disciplinary framework will do to understand, analyze, and teach the significant and pervasive participation of intermediality in culture. As W.J.T. Mitchell has rightly argued, there are no “essential” differences between media, even if they differ in institutional and formal appearances (1987: 2–3). What catches my eye in the title of the two edited volumes Elleström published in 2021 is primarily that word “relations,” in combination with the preposition “inter-.” This relationality is particularly important to me. In the title of those two earlier volumes, Beyond Media Borders: Intermedial Relations among Multimodal Media, every word counts and is programmatic. Briefly, “inter-” stands for, or is, relation, rather than accumulation, as in “multi-.” It is also to be distinguished in crucial ways from the (currently overused) preposition “trans-,” which denotes a passage through, without impact from, another domain. I realize many colleagues use “trans-” without implying such indifferent passages, but given my commitment to relationality I will stick with “inter-.” There is another aspect to this interest in “inter-,” which is part of what I have come to call “inter-ship”: its frequent use in different contexts. This makes the relationality appear in different framings. An obvious case is interdisciplinarity. There are many other forms of inter-ship in all of which the focus on relation is important. Just think of intertextual, international, intermedial, intercultural, and interdiscursive. Inter-ship as a focus encourages an awareness and closer reflection on relationality. And the closeness of my neologism to the concept of internship, which denotes learning through practice, yields a very welcome association. The phrase “beyond media borders” in the main title suggests a commitment to transgressing those borders that academic traditions have so insistently drawn up around their fields, mostly through specific methodologies and definitions, whereas their key terms – think of “text” and “image” – remain vague. With his consistent interest in media as intermedial per se, his many edited volumes, and as director of the Växjö-based Linnaeus University’s Centre for Intermedial and Multimodal Studies (IMS), Lars Elleström has become a primary authority in that domain that is best characterized as one that does not fit any of the traditional disciplinary concepts, yet is probably the largest, most frequently practiced mode of communication among humans, indispensable for human life. His recent untimely passing compels us all to work in the wake of his intellectual dynamism, where meticulous accuracy goes hand in hand with creative thinking. Elleström’s ongoing focus on the idea of the semiotic, a concept and field that on its own already indicates the need for the “beyond” in the books’ main title, demonstrates a consistency of thought without dogmatism. The semiotic is the fourth

464

M. Bal

of the modalities necessary to communicate (2021: 46). He uses the concept of “meaning-making” in a precise sense. Both the producer and the perceiver process the media product for meaning. I consider this as characteristic of the semiotic perspective. Choosing one concept from among the many that pertain to intermediality, I bind my chapter to the specific, a mode of thinking characteristic of Elleström’s intellectual posture. I seek to continue his intellectual attitude of a mode of conceptual reasoning where nothing is taken for granted. I too keep intermediality and semiotics intricately connected, as he always did. This connection is necessary because an always dynamic “meaning-making,” the area of analysis for semiotics, is constantly relevant in intermedial processes. One issue in theorizing is to balance generalization with detailed analysis. This is why, in addition to considering it key for intermediality, my special focus on quotation as the integration of elements from preceding media products into a following one is a means to limit myself and thus preclude unduly generalizing claims that might entail vagueness. In this handbook, we continue Elleström’s thinking: its modes and its contents, even if not systematically adopting his concepts, to avoid a high degree of difficulty for the readers.2 In what follows I will present quotation in different forms of intermediality, which are also bound up with interdisciplinarity, and some problems of traditional disciplinary habits I seek to move “beyond.” First, ironically following (with a wink) chronology, I consider the concept of quotation in its most traditional disciplinary framings. These framings struggle with historicity and the issue of time. I wish to alert the users of this handbook to the way a single concept inevitably branches out to others and other times. Then I will turn to a key concept from semiotic theory, in order to clarify the necessary bond between intermediality and semiotics: “semiosphere.” As intermezzos I will briefly evoke a few examples of artworks that deploy intermediality, as mini-samples of intermediality analysis that does not stop at the comparison of two media, fatally put in binary opposition. Instead, these short pieces suggest how the intermediality within each work does its work by moving outwards into and on behalf of the world. These intermezzos are meant to demonstrate how easy it is to analyze, and hence, also, to teach, intermediality in practice.

Intermezzo 1: Recycling Names (to Make Art Socially Relevant) Movement, of the smallest, subtlest kind, trembles through an immense plaza consisting of large slabs, each four and a half meters long, 1.28 cm wide, in sand color, with a grainy surface of extremely fine pebbles, designed to resist the absorption of water. Nearly effaced names are written on them, in a dark hue. These are written with black sand. Overwriting these are other names, in the same 2

I wrote a preface to Elleström’s two-volume edited publication, in which I further comment on his intellectual attitude, which I consider eminently productive, creative, and stimulating. I try not to repeat that preface here.

20

Citational Aesthetics: For Intermediality as Interrelation

465

size and font, in shallow relief engraved in the slabs. Suddenly, a shiny drop of water appears, rolling towards the relief, and then more, until the letters of the name are filled, and the water becomes a slightly convex shiny surface, surmounting the flatness of the slabs. After a few minutes, the water letters start to tremble; then they disappear. Appearance and disappearance: the quoted names keep moving. Movement, as physical instability and as affective effect, produces turmoil. Sand to write with, relief carved in stone, and moving water drops: the media borders are made redundant. The flat ground on which the visitor must walk, the humble material, and the constant unsteadiness: these are the basic tenets of the Colombian artist Doris Salcedo’s work Palimpsesto (2017). Is it an installation? a sculpture? No genre or media name is satisfactory. The material, spatiotemporally disposed, sensorially perceptible elements of the media product already call on the idea of intermediality. The water names nearly overwrite the sand names, which, while being put in the shadow of the water names, remain as a palimpsest, a trace of forgotten people. This is the semiotic modality: these quoted names make heartbreaking meaning. Meticulous in her research, Salcedo only inscribes names that were really those of refugees who were abandoned to drown in the Mediterranean Sea. This is a quoting that is both an homage to the anonymous drowned and an indictment of European unresponsiveness. And where the Europeans who condone the political indifference do not stop to mourn the unnecessarily lost lives, the glistering water drops suggest that, instead of its inhabitants, the earth is crying.3 This too-brief description of Salcedo’s masterpiece demonstrates how the simple act of quoting (here: names) in an integration of different media, makes art extraordinarily relevant for the social domain. It helps us to grasp how such intermediality between simple, sociopolitical names (most often used orally) and their written form artistically inflected is essential for the functioning of art in and for the world. This requires a brief note on the concept of the political. In a clear and concise book, Belgian political theorist Chantal Mouffe defines the two key terms that she distinguishes, “politics” and “the political,” as follows: by “the political” I mean the dimension of antagonism which I take to be constitutive of human societies, while by “politics” I mean the set of practices and institutions through which an order is created, organizing human coexistence in the context of conflictuality provided by the political. (2005: 9)

In this distinction, politics is the institution, the organization that settles or suppresses conflict; the political is the open world where conflict “happens.” Thanks to the political, social life is possible. Politics, however, constantly attempts to

I use the term “refugees” as the overarching one, including “migrants,” out of respect and acceptance for their absolute need to escape a deadly situation, whether it is war or poverty that compels them to risk and, frequently, lose their lives. I published a book on Salcedo’s work in 2010, hence, before she made Palimpsesto. For an analysis of Palimpsesto, see Huyssen (2018). For a short documentary about this work (which needs a moving image to be really visible), see https:// vimeo.com/239840671

3

466

M. Bal

dampen the political, turning tension into consensus, with its own exclusions. According to Mouffe’s view, everyday life, including the intimacy that inhabits it, pertains to the political. This is the sphere where conflict, tension, and disagreement can flourish, as can reconciliation and other forms of change in relations. It is there that such intermedial, quoting artworks as Salcedo’s must be understood.

Doris Salcedo, Palimpsesto, 2017

Quotation Through Mutual Time “Quotation,” the recycling of words, figures, styles, or motifs taken from a preceding “source” and inserted, modified or not, into a newer work, is both central and problematic in intermedial thinking. It stands at the intersection, or meeting point, of the art-historical method of iconography and the literary concept of intertextuality and, hence, of the two disciplines, art history and literary studies, both keen to spot instances of quotation. Both these disciplines have been developed already in the early twentieth century as what they have become and still are. And although I am sometimes taken aback by dogmatic attitudes they appear to encourage, I am not opposed to these or other disciplinary specializations. These remain indispensable, even if insufficiently moving “beyond borders” for the study of intermediality at stake in this handbook. The term intertextuality was introduced by the Soviet philosopher of language Mikhail Bakhtin (1895–1975). Significantly, the preposition “inter-” is part of the term. The concept refers to the ready-made quality of signs, in his case, linguistic ones, which a writer or image-maker finds available in the preceding media products – Bakhtin, studying language products, speaks of “texts” – that a culture has produced. Iconography seems to be the examination of precisely this reuse of earlier

20

Citational Aesthetics: For Intermediality as Interrelation

467

forms, patterns, and figures within a new media product. But here, it cannot be called “text,” a term that would seem to pertain to “linguistic imperialism,” even though German linguist Martin Fuchs has decisively undermined that concern (2001). And whereas all the terms that follow are imprecise, the visual context in which this concept and the method of analysis it denotes have been developed called for the word “image,” translated into an age-old term as “icon.” Here, the founder of the approach is Erwin Panofsky (1892–1968). He developed the twofold concepts of iconology and iconography, the latter of which has become the central methodological tool of art history. Hence, this dual concept of iconography and/as intertextuality might be a good place to begin integrating visual and linguistic traditions of interpretation in the area where they meet, thanks to the fundamental intermediality of all media products as I have argued above in the wake of Mitchell and Ellestrøm’s publications.4 Three features, all of which are as crucial as they are problematic, characterize both iconography and intertextuality. In the first place, iconographic analyses and literary source or antecedent studies tend to see the historical precedent as the birthplace which more or less dictated to the later artist what forms could be used. By adopting forms from the work of an earlier artist, the later artists seem to be under the spell of his predecessors’ influence; they implicitly or explicitly declare their allegiance and debt to them. The British art historian Michael Baxandall convincingly proposed reversing the passivity implied in that perspective, a move that I find greatly important for a revision of chronology into “inter-temporality.” He proposed to consider instead the work of the later artists as active interventions in the material handed down to them (1985: 58–62). He did this in a book that nonetheless remains firmly historical, in a most subtle and detailed manner. This reversal, which also affects the relation between cause and effect and even the meaning of these two concepts, complicates the idea of precedent as origin and thereby makes the claim of historical reconstruction very tricky. This is why I use the term “pre-text,” to merge the temporal sequence with the intellectual posture that so easily becomes dogmatic, as well as implying an ironic wink, with the meaning of “pretext” as deceptive pretention or excuse lurking around the corner.5

4 For the most accessible publications, see Panofsky (1995) and Bakhtin (1968 and 1981). A brilliant analysis of Panofsky’s theory was written by Michael Ann Holly (1984). For a useful succinct presentation of the relevance of Bakhtin’s ideas for contemporary cultural analysis, see Hirschkop and Shepherd (1989). In an illuminating book Esther Peeren deploys Bakhtin’s theoretical views for an analysis of “popular culture,” an area where intermediality is very present and quotation frequently used (2008). For an in-depth analysis of “icon” from a philosophical and historical perspective, see the indispensable study by Marie-José Mondzain (2005). I have written many times on the need for interdisciplinarity and the nevertheless ongoing relevance of disciplines, most specifically in a book from 2002. 5 Baxandall later published an important book with Svetlana Alpers (1994) where the authors make a convincing case for the relevance of the reversal of the chronological perspective. Being partial to ambiguity, I tend to come up with terms that are ambiguous, implying, with a wink, an opposition to meanings others may have attached to them or to the concepts they replace. Where possible, a hyphen indicates the ambiguity, such as here. These neologisms contribute the importance of staying “on speaking terms” with that to which one objects.

468

M. Bal

A second feature that is both different and similar between the theory of intertextuality and the practice of iconography as source studies is the place and functioning of meaning. This is key to semiotic thinking and a bit taboo in iconographic study. Iconographic analysis frequently avoids interpreting the meaning of the borrowed motifs in their new contexts – the later artwork or media product. This makes some sense; to borrow a motif is not a priori also to endorse a meaning. In contrast, the concept of intertextuality as deployed more recently than iconography implies precisely that the sign borrowed, because it is a sign, inevitably comes with a meaning. This is the consequence of the indispensable semiotic perspective. Not that the later artists necessarily endorse that meaning, but they will have to deal with it: to reject or reverse it, ironize it, or simply, often unawares, insert it into the new text – media product. This transfer of meaning is not alien to art-historical practice; for example, it is also how art historian Mary Garrard (1989) uses precedents in her – basically iconographic – analysis of Artemisia Gentileschi’s Susanna from ca 1610. But interpretation is not the central goal here, at least not in the sense of classical hermeneutics, of constructing a unifying logos. The undecidability of the visual especially, but not only there, is paradigmatic of the production of meaning in general.6 Instead of classifying and closing meaning as if to triumphantly solve an enigma, an intertextual, quotational study of aftereffects or what Freud would call Nachträglichkeit would attempt to trace the process of meaning production over time (in both directions: present/past and past/present) as an open, dynamic process, rather than map the results of that process. Instead of establishing a fixed one-to-one relationship between sign or motif and meaning in line with Saussure, I emphasize here the active participation of visual images in cultural dialogue, the discussion of ideas in media products of all kinds. This alone makes these intermedial, which inflects the “cognitive import” or interpretation. It is in this sense that I claim that art, literature, and cinema “think.” This is not a simplistic personification but brings the intellectual reflection it triggers to the fore. What I mean comes close to French philosopher and art historian Hubert Damisch’s famous concept of “theoretical object” – sometimes overused, hence threatened with vagueness. As Damisch explains it in an interview with Yve-Alain Bois, a theoretical object ... obliges you to do theory but also furnishes you with the means of doing it. Thus, if you agree to accept it on theoretical terms, it will produce effects around itself ... [and] forces us to ask ourselves what theory is. It is posed in theoretical terms; it produces theory; and it necessitates a reflection on theory. (Bois et al., 1998: 8)

6

Hermeneutics of seeing is problematic but worth considering before rejecting it. It is effectively countered in a short text by French psychoanalyst Jean Laplanche. See Davey (1999). For me, an example of a nuanced deployment of hermeneutics is the work of Austrian literary critic Leo Spitzer (1887–1960), especially his collected essays on French classics (1983). The inevitability but also productivity of undecidability, brought to the fore by Derrida in all his works cited here, is the central topic of a recent collective volume edited by Mette Høeg (2021).

20

Citational Aesthetics: For Intermediality as Interrelation

469

This makes theory, and more broadly, thinking, a crucial part of intermediality. In the dynamic between the works as objects, their perceivers, and the time in which these come together, accompanied by the social buzz that surround both, a compelling collective thought process emerges. Damisch’ concept of the theoretical object sometimes seems to suggest these are objects around which theories have been produced. At other times, as in the interview quoted here, he attributes to the artwork the capacity to motivate, entice, and even compel thought. This is the meaning of the concept I endorse here.7 A third difference-in-similarity between the theory and practice of iconography and intertextuality resides in the textual character of intertextual allusion. The difference is not as stark as it might appear to be. For iconography tends to refer visual motifs back to written texts, such as the classical texts from antiquity and of mythology. This makes the method in itself intermedial – to avoid using the term “intertextual,” which would be confusing in the context of the distinction between the two “schools” I am making. To get closer to the intermediality concept as developed by Elleström and his colleagues, I would like to try to take the textual nature of precedents seriously, but as a visual textuality, and therefore use the more general term mediality. By recycling elements, motifs, or forms taken from earlier works, an artist takes along the media product from which the borrowed element has been broken away, while constructing a new work with the debris. The new image-as-text – say, a mythography – is “contaminated” by the discourse of the precedent and thereby fractured so to speak, ready to fall apart again at any time. The fragility of the objectifying, distancing device of mythography and history, even contemporary reality, is displayed by the taint of “firstperson” subjectivity in, for example, Rembrandt’s painting The Night Watch from 1642 (Amsterdam, Rijksmuseum), where he subtly inserts the visual discourse of selfportraiture in the middle in the far background. In the terms of the influential French linguist Émile Benveniste (1902–1976), objectifying narrative is then partly transformed into subjective discourse.8 Intertextuality – the specific practice of quotation which is also the object of iconography – is, in this sense, a particular instance of the more general practice of interdiscursivity: the mixture of various different visual and discursive modes that Bakhtin called heteroglossia. Thus, this “textualizing” iconography will consider visual principles of form – such as chiaroscuro, color, folds, surface texture, and different conceptions of perspective – as “discursive positions” that entertain interdiscursive relations with other works. This form-making or shaping occurs not only in figurative but also in abstract art. “Quotation,” then, is a concept that by itself already stands at the intersection of art history and literary analysis.

7 For a nuanced explanation that takes Damisch’s view along, with convincing analyses that demonstrate the validity and productivity of the idea that art and thought “work together,” see Alphen (2005). 8 Narrative and discursive as used here are derived from Benveniste’s distinction between “histoire” and “discours” (1966).

470

M. Bal

But it moves beyond those “media borders.” In what follows, quotation will be seen in a number of distinct ways, each of which illuminates – through their theoretical consequences – one aspect of the art of the present and the art of the past in their mutual “contamination,” which opens up the field of intermediality most forcefully. A first instance concerns direct, literal quotation. According to classical narrative theory, direct discourse, or the “literal” quotation of the words of characters, is a form that reinforces mimesis – the classical term for realism. As fragments of “real speech,” they authenticate the fiction. In narrative, the quotation of character speech is embedded in the primary discourse of the narrator. In visual art, such embedding structures are less conspicuous and rarely studied, despite the frequent use of quotation. In artforms that are in and of themselves intermedial, such as most clearly, cinema, unpacking the dense mixture of source or pre-text and quotation is even more complicated. Second, in their new (quoting) framework, these fragments of things having been said or shown before are the product of a manipulation by the producers and their context, or semiosphere. Rather than serving reality, they serve a reality effect (Barthes 1986), which is in fact the opposite of realism – a fiction of it. Thus, they function like shifters, allowing the presence of multiple realities within a single image. This conception of quotation recurs throughout film and (its) history. One instance where this happens perhaps most emphatically is the way one of the most deceptively illusionistic works from the past, such as Caravaggio’s Head of Medusa, from 1598 (Uffizi Gallery, Florence), resurface in Belgian sculptor Ann Veronica Janssens’ sculpture Le corps noir.9 Third, according to Bakhtinian dialogism, quotations stand for the utter fragmentation of language itself. They point in the directions from which the words have come, thus thickening, rather than undermining, the work of mimesis. This conception of quotation turns the precise quotation of utterances into the borrowing of discursive habits, and, as a result, intertextuality merges into interdiscursivity. And this interdiscursivity accounts for pluralized meanings – typically, ambiguities – and stipulates that meaning cannot be reduced to the artist’s intention. Finally, deconstructionism paradoxically harks back to what this same view might repress when it presents the polyphony of discursive mixtures a little too jubilantly. Stipulating the impossibility of reaching the alleged, underlying earlier speech, the “pre-text,” this view emphasizes what the quoting subject does to its object. Whereas for Bakhtin the word never forgets where it has been before it was quoted, for Derrida it never returns there without the burden of the excursion through the quotation.10

For a detailed analysis of this case of quotation, see chapter five of my 1999 book. Caravaggio’s Medusa, painted on a convex shield, is a prominent instance of the intermediality between two- and three-dimensionality. Janssens takes this up in her sculpture and includes the first–second-person discourse as well. 10 Derrida’s claim is most clearly and persuasively phrased in Limited Inc., e.g., 155, where he points out that any sign may be cited. This principle of citationality makes it impossible to determine “literal” meaning. On this more general claim, see also his Speech and Phenomena (1973: 130). 9

20

Citational Aesthetics: For Intermediality as Interrelation

471

The meanings of the concept of quotation as exemplified by intertextuality and iconography engage the relation between text and/or image and reality beyond the question of reference. Their orientation leads from the image to the outside world in which it operates: from the close environment of the work’s own frames to the world outside those frames. In contrast, other meanings and deployments of the concept focus on meaning coming from the outside in. Hence, their simultaneous mobilization also entails a questioning of the very limit that separates outside from inside: a limit, or border, beyond which we must move if we are to grasp intermediality. This questioning in turn challenges the notion of intention that is so pervasively predominant in the cultural disciplines. In more recent media art, this question has faded into the background. One reason for this is the collective nature of the making. The predominance of intentionalism in art history and other disciplines in the humanities has been discussed by others for its ideological import. David Carrier, for example, imputes it to a generalized humanism and counters this humanist intentionalism as follows: The humanist will think that my account leaves out one further essential point–that narrative must be a truthful representation of the artist's intention. A central argument of this book is that this appeal to the artist's intentions adds nothing. (1991: 7)11

To substantiate this claim, he alleges the example of Caravaggio. This example demonstrates for him the projection that goes on in biographical criticism, in which the available knowledge informs the kind of criticism and programs the discourse: Something is known of Caravaggio's life, and there are a number of near-contemporary responses to his work. It is therefore possible, as is not really the case with Piero, to interpret his painting as an art of self-expression. Since the evolution of Caravaggio's art is unusually complex, it is tempting to narrate the history of his artistic evolution as a story of his personal development. (1991: 7–8)

Carrier’s chapter “Caravaggio: The Construction of an Artistic Personality” (1991: 49–79) substantiates this contention that the more there is known about an artist’s life, the more seductive the trap of intentionalism becomes, even when the “intention” can only be, contradictorily, fantasized as unconscious.12 By opposing precisely, the unrestrained projection that takes place in the kind of psychoanalytic criticism that turns effect into intention, others stay rigorously on the side of historical evidence. Some of this work is meticulous and, for that reason, convincing. From there, the quoted work becomes easily idealized as a work of genius. But it is not the iconographical quotations that cause the paintings being works of genius, as is so often alleged for great art from the past (“old master art”). To be sure, “genius,” and its romantic connotation, is a word I recommend we avoid. But what

11

For this argument, see also Carrier (1982). See the chapter “Intention” in my 2002 book on concepts for a critical (but also nuanced) analysis with detailed examples.

12

472

M. Bal

gets lost in an overanxious attempt to totally dismiss genius is the media products’ persistent appeal, or their address – which I call their “second-personhood.” It is the artworks’ appeal that informs and warrants the effort, time, skill, and money that are involved in the scholarship concerning them. The images’ historical reconstruction does not answer the question how such reconstruction helps to understand the paintings better. For this reconstruction leaves one aspect unaccounted for, the very aspect I feel compelled to attend to, because this pertains to the meaning and importance of their intermedial travel: what do these older media products contribute and mean to today’s culture, including its political domain (in Mouffe’s sense)? This question helps to stay more rigorously away from intention than even the most meticulous examination of theological, literary, or (audio-)visual precedents. That is the condition for avoiding “pre-text” thinking. To restate my case slightly differently, the historical iconographic approach pushes us away from the paintings in two ways: by reducing them to iconographical particularities only and by referring them back to (often textual) sources.13 Unfortunately for my purpose here, the alternative that Carrier offers has a vagueness not unlike the humanist caricature that he opposes: The test of these interpretations [...] is not, as the humanist thinks, whether they match some hypothetical reconstruction of the artist's intentions. That test is both useless in practice, as we can have only indirect knowledge of these intentions, and methodologically flawed. The right test is simply whether these interpretations help us understand the paintings. (1991: 197)

I suppose Elleström would have liked the scholar to be more specific about what “understanding” means; he calls it “cognitive import.” Is it understanding “what they have to say” or “what they do,” and how does this emerge from quotation? The reason for this vagueness, I submit, can be found in another of Carrier’s statements, which appeared in the text “Derrida as Philosopher” and which explicitly and ambivalently discusses Derrida’s conception of quotation (1994: 149–164). Having quoted Derrida’s statement that “[T]hese differences. . . are neither inscribed in the heavens, nor in the brain, which does not mean that they are produced by the activity of some speaking subject” (1981: 9), Carrier admits to being “baffled” by it (1994: 160). But he need not be. In an irremediably binary mindset, he sees Derrida’s position as inconsistent because, while welcoming “the death of the author,” he wrongly assumes that a simple “being there” prior to construction is the only alternative: “I am puzzled by the notion that these contexts are somehow there prior to being constructed” (1994: 161). Clearly, he misunderstands Derrida here. Too bad he was not enabled by the later discussion of the mind in Elleström’s introductory essay (2021: 4–91). For the latter, the mind – not at all separate from the body but on the contrary, “profoundly embodied” – is “(human) consciousness that originates in the brain and is particularly manifested in perception, emotion, thought, reasoning, will, judgment, memory and imagination” (13). 13

On the problematic notion of genius, see the feminist angle proposed by British philosopher Christine Battersby (1989).

20

Citational Aesthetics: For Intermediality as Interrelation

473

Carrier’s misinterpretation of Derrida’s words is predicated on an implied binary opposition, according to which subjectivism is the only alternative to an untenable objectivism. What Derrida denies is not, however, the constructedness of context but its (intentional) construction by an original, autonomous, authentic, speaking subject. The displacement of the recycled item from one medium to another, therefore, is no less based on a construction, even if this construction is historically plausible. Instead of Carrier’s alternative, I contend that the subject’s agency – which matters in a way that his or her intention or psychic makeup does not – consists not of inventing but of intervening, of a “supplementation” that does not replace the media product it explains but adds to it. It is in this sense that I would like to propose the specific, verifiable, and, I submit, socially relevant idea of quotation in contemporary media practice as a valid ground for an interpretation that accounts for a different sense of “understanding.”14 Such an interpretation that shifts from intention to the media product’s agency neither contradicts historical evidence, which it may accept but does not make decisive, nor does it project present concerns upon it. Quotation does not construct a fictitious intention or unconscious psychic makeup, nor is it a totally relativistic subjectivism in which anything goes. Instead, it supposes that the media products are rigorously contemporary in their effect; there lies their agency. Therefore, it makes the historical art more important because it keeps it alive and does not isolate it in a remote past, buried under concerns that we do not share. This is a good place to wind all this up by recalling a famous example of that contemporary sociocultural effect, which is what makes quotation, and hence the intermediality it produces, important. In a widely known book, Judith Butler judiciously deploys Derrida’s conception of quotation in an argument on constructivism and performativity, to rearticulate her theory of sexuality as a result, not a cause, of particular performative behavior (1993). She quotes Derrida’s statement that performative utterances cannot succeed unless they repeat – hence, quote – an already coded, repeatable (“iterable”) utterance. As a consequence, Derrida argues, “the category of intention will not disappear; it will have its place, but from that place it will no longer be able to govern the entire scene and system of utterance” (1988: 18). The subject whose “intention” is involved in the making or the processing of the media product, in other words in the uttering of the speech or image act, steps into a citational practice that is already whirling around; the speech act is larger than the subject of the utterance – the producer of the media product – can possibly foresee or control.15 Butler uses this argument to articulate an alternative to the misguided constructivism that only replaces the intentional subject with a personified “construction,” which, as she puts it, “belongs at the grammatical site of the subject.” Instead, she

14

The notion of the supplement, of supplementing is best explained in Derrida’s Of Grammatology (e.g., 1976: 208). 15 Butler is answering critics of her earlier book (1990), who took her notion of performance in a literalizing, theatrical sense. This misunderstanding could only occur within a voluntarist conception of intention, against which Butler’s response argues emphatically.

474

M. Bal

proposes a materialism which I find wonderfully suited to the materially engaged art that constitutes the subject matter of this section on quotation. She writes: What I would propose in place of these conceptions of construction is a return to the notion of matter, not as site or surface, but as a process of materialization that stabilizes over time to produce the effect of boundary, fixity, and surface we call matter. (1993: 9; emphasis in text)

The materialism Butler articulates here in the context of her theory of sexuality resonates with the one I consider a crucial element in contemporary intermediality as bound up with quotation: hence, with the past, in mutuality, relationality (inter-ship). In all four conceptions of quotation, the relation with what is quoted is established from the vantage point of the quoting media product that it situated and functions in the present. Whether the fragment from the quoted older media product is enshrined or abducted, dispersed or unreflectively absorbed, the resulting (complex) one is both a material object and an effect. All of the different meanings attached to the idea of quotation carry with them an epistemological view, a concept of representation, and an aesthetic. Indeed, their inseparability is perhaps the most important contribution of the theory of quotation to that of intermediality. The juxtaposition of these conceptions of quotation therefore enables us to grasp issues of past art and present vision pertaining to the understanding, the history, and the activity of looking and other forms of communicating. Translating the meanings of quotation as developed in language-centered theories into a visual context further clarifies how contemporary quotation really changes older media products. This transformation is possible because, as Elleström insists, media products are imbrications of four indispensable modalities: material, spatiotemporal, sensorial, and semiotic modalities (2021: 44).16 To sum up the meeting between quoted and quoting practices, in the most traditionalist form of quotation, recycling iconographic figures, motives, or styles from past art: engaging the art of the past in its theoretical potential is a way of quoting it in all the meanings mentioned above. First, the old master art is endorsed as the historical “real” – including the iconographic precedent – which determines the belatedness inherent in being situated in history. Second, contemporary artists, acting as narrators who quote and thus appropriate the cultural inheritance, embed the appropriated old master art in their own work, thus endowing it with the glamour of historical reference, the historicizing “reality effect.” Third, such visual quotations fragment and pluralize visual representation into a “polyphonic multitude,” whose aspects are neither arbitrarily collated nor “democratically” distributed. Instead, they demonstrate the difference between the illusion of wholeness and mastery pertaining to the artist caught in art history and the somewhat messy, yet much richer, visual culture of live images. Finally, the images of today present us with old master art that is entirely ours, one of which the old master could have had no knowledge of, or agency upon, what we see “him” to be now, an irreversibly new-old master, who changes the one we thought we knew as well as the historical illusion that we knew him.

16

This indispensable collaboration explains the connection in the name of the program in Linnaeus University of I (intermedial) and M (multimodal) Studies (IMS).

20

Citational Aesthetics: For Intermediality as Interrelation

475

Intermezzo 2: Recycling Myths (to Bring Science and Religion into Discussion) The reality of the people who drowned due to social-political indifference has shown, in the first intermezzo, how art can encroach on that indifference through the deployment of the media – sand, water, stone, and the Western alphabet, the latter being deployed to write Arabic names, so that the Western perceiver who belongs to the indicted culture can read the names. This is an instance of “interlanguage.” Here I will demonstrate the binding of art and society through the intermedial connection between two conceptions of life, the one based on science, the other on religious myth. The Norwegian artist Edvard Munch (1863–1944) actively quoted from a variety of precedents. I allege here a painting titled Metabolism, which refers to the organic process of ongoing life, death, and regeneration, hence a scientific term used for nonreligious political and cultural persuasions. That title is part of the work and in that sense establishes a discussion with the visual image.

Edvard Munch, Metabolism (1898–1899)

476

M. Bal

For there, it is impossible to avoid seeing an “Adam and Eve” scene. Even the famous tree where, according to the story in Genesis, the serpent was hiding, is present between the naked woman and man. Quoting the biblical story is, however, far from recycling it wholesale, including its meaning. On the contrary; it “discusses” it. This discussion, even disagreement, shows in the details. The fact that neither of the two figures has open eyes suggests that both avoid looking at each other. Such failure to engage can be explained as alluding to the shame about their nakedness, the first biblical punishment for the transgression. But the bodies and their poses are more telling than the traditional story would suggest; this is how the artist produces intermediality. There is a funny discoloration around the man’s genitals. He seems to be wearing a kind of cover over it in transparent brown. The penis is still clearly visible, so the point of the garment is not shame remedied by the proverbial fig leaf. The scientific idea of metabolism as digestion, death and regeneration, can substitute for the myth. The painting becomes a “thought image” due to the combination of these two interpretations. This compels us to think; Damisch would call it a “theoretical object.”17 But how does this media product become intermedial? The pre-text is textual, linguistically written down, but primarily living on in social thought. Here, the visual mode of color actively participates in the production of meaning. The woman has long hair, reddish brown (“chestnut”). The outline of this color around her head might hint at a halo. The man’s hair is green, with a dark blue outline, but no hint of a halo. The background is made up of thick vertical lines which must be tree trunks, so thin because of the typical Munchian presentation of depth, a forest of trees set off against a bright turquoise-blue sky. Some dark blue, swaying (cinematic), horizontal-oblique strips could be tree branches. Just above the center a band of orange makes the blue seem like a coastline or a simple color accent breaking potential monotony. From the figures’ hips down the background is dark green. The man’s knees are red, a discoloration foregrounded by outlines veering towards orange. This could be a visual quotation of the part of the curse in the textual media product of Genesis that says that he will have to toil for a living. That would be predictive. In a typical integration of iconicity and indexicality, red knees would then signify labor, working the earth. However, with Munch’s use of color for accents in abstract patches, we can never be sure whether the meanings we attribute to such details are part of the vision of the producer, projected on the protagonist-focalizer, or our own projections. This is the point of this intermedial discussion between the textual and the visual media products: to involve the artist, producer of the figuration of the protagonist, and the perceiver, the viewer, in the production of meaning (the transfer of cognitive import), each with their own responsibility. We all draw upon our education and memories of the moods it produced, but the outcomes differ. The most exuberant colors are on the tree trunk in the middle. While its top is pink, the middle part has cobalt blue, bright green, and orange-red lines, which help make the trunk more clearly visible. They also seem like a bouquet of flowers in 17

On the concept of the thought image, see Richter (2007).

20

Citational Aesthetics: For Intermediality as Interrelation

477

veneration of the scarcely visible fetus inside the trunk. This signifies the regeneration that results from metabolism. The feet of both figures are outlined in red, but differently. Her foot looks as if she is wearing a shoe, as light and transparent as the man’s loincloth. The woman seems to want to walk towards the man. But his left foot points more clearly outwards, as if about to walk away. Her arm is held towards the man, but he does not reciprocate. His one arm protects his stomach, while his right hand is behind his head; showing his body and withholding it. But what will come of the metabolism if he persists in that refusal? This question allows us to reconcile the two interpretations through revising the interpretation of the myth as it has lived on for so long. The story of the tree, the prohibition, and the transgression in the Bible concerns nothing like greed or seduction, as the misogynistic cliché interpretation has it, but the endorsement of the mortal condition for which procreation compensates: the individual dies, but the species does not. According to this interpretation of the famous story, the woman did the wise thing, and refusing intercourse is not a good idea. In this sense, the myth and the scientific theory can shake hands, thanks to the intermediality of the media product, where color and line meet the linguistic pre-text.18

Semiosphere: Wavering and Hovering Media From a semiotic point of view, to take the presence in the present of what historically precedes into account through focusing on quotation as an active element of intermediality makes the analysis more, rather than less, historically responsible. It also makes the works from the past as well as their ongoing presence continue to matter. This is where quotation intermediates on behalf of contemporary social relevance. In this context it is important to keep in mind what Ernst van Alphen wrote about this in a different context: It is thanks to this dependency [on discursive frameworks and acts of framing] that testimony as well as history writing are relevant beyond the production of knowledge of the past. It is because discursive frameworks belong to the present, and framing acts take place in the present that memory of the past–knowledge of history–can have consequences for our contemporary and future world.19

One can push this reflection on the implications of the production of meaning through time, by means of quotation and intertextuality, further in the direction of self-reflection, because art historians or literary scholars, like any perceiver, be they 18

The fetus is more clearly visible in X-ray photographs. It is in the tree trunk, painted with a heavy pigment, i.e., lead white, which appears white on the X-ray image. Most oil paint becomes more transparent with time, and thus, this feature becomes more visible as well. I curated an exhibition in the Munch Museum in 2017, in which this painting also figured. See my book Emma & Edvard (Bal 2017) for a more extensive analysis of this painting. On the Genesis story, see my analysis in the final chapter of my 1987 book. 19 Alphen (1997: 64). In this respect, this point of quotation is also connected to Michael Holly’s book on the predictive import of old art, her version of “preposterous history” (1996).

478

M. Bal

viewer of images or reader of texts, cannot help but bring to the media products their own legacy of discursive precedents. This attention to self-reflection matters for the emancipation of the perceivers from the passivity wrongly attributed to them. The formerly called “recipient” is not a passive mailbox but as a perceiver participates in the construction of meaning – Elleström calls it “cognitive import.” Reading images – or processing visual media products – entails the inevitable mixing of these previous signs with those perceived in the media product now. This input from the present is emphatically not to be taken as a flaw in our historical awareness, or as a failure to distance ourselves from our own time as is the case in naive “presentism,” also named “paronthocentrism.” Rather, it is to be taken as an absolutely inevitable proof of the presence of the cultural position of the perceiver, hence also the analysts and their students, and their “cultural community” or “semiosphere” within the analysis. The latter term brings the theory of quotation into the realm where intermediality theory and semiotics join forces.20 In a very useful volume occasioned by the 100th birthday of Russian-born literary scholar Juri Lotman (1922–1993), who was to become one of the foremost semioticians, Peeter Torop published a chapter on the concept of “semiosphere” (2022: 296–307). He opens his chapter with the statement that this concept “marks his [Lotman’s] move towards dynamic cultural analysis.” He continues that “the concept has travelled from one terminological field to another.” And he ends this introductory paragraph recalling an earlier publication, “‘semiosphere’ marks the complementarity of disciplines studying culture, the movement towards the creation of general theory of culture and flexible methodology” (all emphases added). In addition to everything else I can learn from that rich book, in these opening sentences, three phrases speak to me in particular: “cultural analysis,” “the concept has travelled,” and “flexible methodology.” The first of these, “cultural analysis” as distinct from “cultural studies” formed the grounding of a research institute I co-founded at the University of Amsterdam in 1995, the Amsterdam School for Cultural Analysis (ASCA). The insistence on “analysis” is crucial and congenial to Lotman’s commitment to close reading. The second underlies my book on “travelling concepts” in the Humanities. And the third statement has been a longtime directive for my work, both research and teaching. Boundaries between disciplines have never satisfied me and always hindered the depth of thinking and analysis and the communicative nature of teaching. And as Elleström writes: “Borders should always be disputed” (54).21 From early on my academic work has been semiotic-inspired, and the three angles Torop mentions in that first paragraph explain why. Lotman, along with Charles Sanders Peirce (1839–1914), from whose work Elleström’s derives his primary I mean the ordinary-sounding term “cultural community” a bit specifically. In analogy with “textual community” (Brian Stock) or “interpretive community” (Stanley Fish), I define it as a social realm wherein cultural communication is possible on the basis of more or less common semiosis. The term “semiosphere” fits more emphatically with the semiotic framing that intermediality theory needs. That term also introduces the great semiotician Yuri Lotman. 21 Torop is an Estonian scholar of the semiotics of culture, who initiated and edited this volume with Marek Tamm. 20

20

Citational Aesthetics: For Intermediality as Interrelation

479

vision and concepts, has been my semiotic sources of inspiration. Lotman insisted on close analysis, as a more detailed engagement than what cultural studies propagated. Hence, the phrase “cultural analysis” in the title of our research institute. The fact that concepts, and conceptual thinking, “travel,” adapting to and transforming through disciplinary, geographical and historical shifts in different semiospheres, was particularly relevant for Lotman in his Russian-European and politically transforming context. And although he remained primarily focused on literature, that art form was never isolated from the wider context in which he worked. Nor is it monomedial.22 My interest in semiotics concerns the integration of philosophy (thinking) and (close) analysis (doing), as well as the resistance against medium essentialism and disciplinary constraints, with their methodological dogmas and the resulting blindness to the dynamism of intermediality. Semiotics offers the possibility and tools to facilitate that integration, so that my commitment to both, with teaching as an important third, could ease in as an activity that made sense – a phrase to be taken literally as well as figuratively. On the side of philosophy, in addition to, especially, Spinoza, Bergson, and Deleuze, Theodor Adorno always accompanied my thinking. That attachment is due to his integration of sociopolitical wisdom with philosophical rigor. And recently, just one week before the worldwide lockdown of Spring 2020, I encountered that double integration once more. To my delight, I was invited by the famous film school in Łódź, Poland, to make an experimental “essay film.” I had one week to make the film, in a semi-foreign semiosphere, working with actors, cinematographers, sound engineers, and editors of whose language I did not understand at all. Fortunately, English was, as usual, a helpful tool. The word “essay” in its meaning of “trying,” in turn in its Anglo-Saxon two meanings of “attempting” and “challenging,” was more than appropriate. From beginning to end, ambiguity and its productive side effects and affects accompanied the process and the resulting film. Ambiguity, with the subsequent instability of meaning, which necessitates recognition of the participatory work of the perceiver, offered a great contribution. I wish to put ambiguity at the heart of the concept of semiosphere, which denotes the greater sociocultural environment within which intermediality functions. This is what makes it, as well as semiotic practice, both stable in the sense of theoretically (but not practically) delimited, and unstable, since no meaning functions alone.23 As I have argued through the examples discussed above, ambiguity is crucial to meaning-making, as is intermediality, since the ante- or pre-text always resonates along. Both the interacting media products and their multiple meanings are wavering: undecidability roars its head again. And specific semiospheres can become prominent as the meanings shift and multiply. To understand, negotiate, and deploy

22

In the order of the Torop citations, see my books related to these: 1996, 2002, 2022. For my views of teaching, specifically in relation to visuality, see the interviews in Lutters (2018). The invitation came from Dr. Jakub (Kuba) Mikurda. The genre of the essay film has recently been discussed widely (esp. Rascaroli 2008). Mostly, however, these essays are films on films, so they are highly self-reflexive and relatively difficult to bring to bear on other issues.

23

480

M. Bal

meanings in any semiosphere, an alertness to ambiguity as productive is of crucial importance. And here, specifically, it sustains intermediality. To make the case for the fruitful effects of ambiguity, I first consider the effect of one sign – a tiny one. This is in fact a simple one much used in Europe, which changed everything in the film I made: the exclamation mark “!”. It is not a word, nor a letter; it is not part of an alphabet, nor a signifier carrying a signified, in the line of Saussure. It is neither an icon nor an index, in Peirce’s semiotics, renamed by Elleström as depiction and deiction, respectively. Nor can it be considered part of Lotman’s “secondary modelling systems,” because it does not translate language into an artistic text. Yet, it is undeniably something like a sign, semiospherically specific as it is, and as such it is quite powerful. From a written sign it changes meanings, intonation, and interaction with perceivers. This makes it intermedial. Thanks to the exclamation mark, there are simultaneously two titles to the 2020 film, each carrying their own meaning, semiosituation, effects, and intermediality status. The first is IT’S ABOUT TIME. This ends on a full stop, denoting the subject or theme of the film – what the film is “about,” in the ordinary sense of that preposition. This is a kind of “third-person discourse,” the impersonal language use where the object (the “about”) is absent from the scene. This thematic center concerns my ongoing interest in and argument for revising our sense of history by turning the linearity of chronology into a mutual movement or directionality between present and past, as was at issue in the previous section. I have argued for that temporal mutuality by proposing the term “pre-posterous” (1999). This term is ambiguous, with an ironic wink, alluding to the way I had been scolded for writing about art in a (wrongly) allegedly ahistorical way, considered “pre-posterous” in the sense of “absurd,” whereas I simply (“literally”) sought to make the prepositions “pre-” and “post-” interchangeable, dialogic, mutual. So, the parts and aspects of the film that relate to that theoretical discussion of time and history are intermedial between theoretical, “third-person” discourse and audiovisual figuration. But that tiny sign, a shifter between two intermedialities, changes everything in the film’s title: the exclamation mark “!”. The title is, in the final version, IT’S ABOUT TIME! with the subtitle: REFLECTIONS ON URGENCY. Thus, while the “about” meaning with its “third-person discourse” continues, it appeals to a typical phrase current in the Anglo-Saxon semiosphere: it’s about time we do something; something must happen! This is the rallying call for climate activism; as such, it is a first–secondperson discourse, in a personal, interactive language situation. That is a different “pre-text” quoted for the film’s narrative. The small sign completely transforms the meaning and the communicative situation, turning it more strongly to the political. There is an intermedial consequence: that written exclamation mark changes the way we pronounce the title, becoming a sound figure. It makes us raise our voices and even imaginatively raise a warning finger, with a transformation of meaning and of address as a consequence. It is what turns a film on (about) an intellectual issue into a political one – thanks to the ambiguity of the English – with the result of integrating the two. This integrative transformation through intermediality and ambiguity is the

20

Citational Aesthetics: For Intermediality as Interrelation

481

most important aspect of intermediality theory and semiotics together over disciplinary fields – their relevance.24 But then, the more common and immediately recognizable issue of intermediality emerges. For developing the script, designing an audiovisual media product through quoting out of written pre-texts, in other words, for “doing,” practicing intermediality, I took on the mythical figure of Cassandra. She could foresee the future but, in an antique #MeToo case – a pre-posterous quotation of a still-current sociopolitical issue – when she declined to sleep with him, she was cursed by her employer, the god Apollo, to never be believed. Cassandra’s story was retold – quoted – by East-German writer Christa Wolf in an updated, “pre-posterous” version from 1983. With this story I tried to “figure” the rallying call implied in the English phrase “It’s about time!” The verb “to figure,” on which more below, stands for the effort to make a figural shape for the thoughts on the indifference of people towards the imminent ecological disaster of the world, so that they would see (literally) the urgency, in other words, to “intermediate” this for contemporary sociopolitical relevance. The exclamation mark of the title indicates that side of the film’s title and makes the “about” meaning of the title’s words into a multimodal media product urgently necessary for the political. For, chrono-logic entails a false belief in progress. Hence, the critique of it through mutual temporality is already a discursive political appeal, not only an “about” third-person discourse. In addition to the intermediality between the antique sources quoted, the modern novel, and the recent film, the Cassandra story came to my attention again through the brilliant artwork by Indian intermedia and multimodal artist Nalini Malani, who mobilized it in several works of painting and drawing, video-shadow-plays, and animations, in other words, in different media, producing a number of very different media products, almost impossible to distinguish media-wise. This brings up the question, not of what is quoted and how the quoted and the quoting media products are positioned in history, but of how this quoting is done. This is the question that brings us closer to an intermediality of a wavering, flexible kind. Searching for an in-between concept that could accommodate both audiovisuality and semiotic thinking within the framework of intermediality, but reluctant to jump too fast to conceptual conclusions, I invoke the French philosopher Jean-François Lyotard. He came up with the concept of “the figural” in his attempt to overcome the tenacious word-image opposition and the divide between disciplines that it produced. This concept is crucial for a quotation-oriented view of intermediality. Elleström argues that “[B]ecause being visual is a sensorial trait and being verbal is a semiotic trait, it is pointless to oppose the two” (2021: 7). I would nuance the difference between these traits – both visual and verbal pertain to both the sensorial

The film can be watched on my website, at http://www.miekebal.org/artworks/films/its-abouttime/. My 1999 book Quoting Caravaggio lies at the heart of the intellectual reflections (“about”). In the film, Cassandra as teacher quotes from that book.

24

482

M. Bal

and the semiotic, albeit in different proportions. Nevertheless, this argument helps in changing the habit of binary thinking.25 In his 1971 PhD thesis Lyotard argued for language as more dynamic than it is usually seen, turning it into a force, a movement. He made it more “sensorial.” As such, he argued, language is closer to the Freudian unconscious as laid out in The Interpretation of Dreams (1900) than to any Saussure-derived structuralist conception of it: dynamism as opposed to structural stability. This view creates waves that might work for transferring not just meanings but thoughts and their forms, including but not limited to images, from one medium to another. And due to a blindness for intermediality in theoretical writing, the importance of visuality in Freud’s theory of interpretation, with its semiotic implications, has been more or less neglected, whereas it is crucial for his thinking. Including, especially, force in his concept of language, Lyotard describes meaning as sense – including sensuality – in terms that include affect, sensation and intuition, movement, and also spatiality. Although he speaks only about language, his view of that medium is already intermedial in itself. For him, language and the meanings it produces are primarily dynamic. This corresponds with Lotman’s flexible methodology as described in Torop’s opening paragraph. Force, for Lyotard, is inherent in language, and it is ... nothing other than the energy that folds and wrinkles the text and makes of it an aesthetic work, a difference, that is, a form... And if it expresses, it is because movement resides within it as a force that overturns the table of significations with a seism that makes sense....26

Notice the attempt at intermediality even in the choice of words. I find the word “seism,” which invokes movement, particularly powerful in this revision of what language is and does. “Folds” and “wrinkles” applied to “text” import visual and tactile aspects to the linguistic media product. These words affiliate language with, specifically, cinematic language, based on the etymological sense of “movement” (kinetic) rather than any technical specificity. Both language and the cinematic as an informational tool, a mode of communicating, and an art form are prominent in, among many others, the European semiosphere.27 In my search (attempt) for semiotic forms that could make the thoughts I wanted to propose and convey in a different medium from the pretexts, the ambiguity had to 25 The first chapter of my study of Malani’s video-shadow-plays (2016) is devoted to the Cassandrarelated video-shadow-play titled IN SEARCH OF VANISHED BLOOD she made for the Kassel Documenta in 2012. 26 I quote from film and philosophy scholar D.N. Rodowick’s rendering of Lyotard’s concept (2001: 9–10). To grasp the concept more fully, it is rewarding to read Rodowick’s first chapter, “Presenting the Figural,” 1–44. 27 In addition to Rodowick’s important book, see Vlad Ionescu (2018) for another excellent explanation of the figural in relation to and distinction from “figure” and “figurative,” within the context of art history in its relation to psychoanalysis and philosophy. This author discusses ideas of influential theorists of images. I discuss the conception of the cinematic as kinetic apropos of the paintings by Edvard Munch and Flaubert’s prose in Madame Bovary, making an implicit case for the figural and intermediality (2017: 24–43).

20

Citational Aesthetics: For Intermediality as Interrelation

483

remain intact. Hence, the “about” and the rallying cry had to stay paired, even intertwined. That intertwinement is the cognitive import I wanted to transfer, or quote. The characters had to figure meanings as signs, no matter of what (Peircean) category exactly – icon, index, symbol, and probably a mixture, as happens more often than not. The primary issue is the interaction between the figures, which is where narrativity comes in. I figured – a verb I now use in Lyotard’s sense – the “about” idea through an intermedial contact between one medium and another, mobilizing an old master media product cinematically animated: contact, rather than translation.

Intermezzo 3: Characters as (Inter-)Media Products (to Bring Theory in for Sense-Making) For the purpose of turning intermediality into contact I staged the enactment of a tableau vivant of Cassandra’s lover Aeneas as Caravaggio’s John the Baptist in the Wilderness (1604). The living body of the actor playing Aeneas and the photocopy of the seventeenth-century painting – a still of a still – intermedially produced a media product: the tableau vivant. Then, during a history lesson in which Cassandra (now acting as a teacher) explains pre-posterous history to her lover-student, dressed up as a Walter Benjamin look-alike, the media whirl around. In a discussion with his teacher, Aeneas quotes a passage from Benjamin’s fifth thesis on the philosophy of history, which has been deeply influential for my thinking on history: “[E]very image of the past that is not recognized by the present as one of its own concerns threatens to disappear irretrievably” (emphasis added). Theory – here, philosophy – participates as a medium in itself.28 Then, the staged copy of the Caravaggio painting, with the tableau vivant just seen still in the perceiver’s mind, is literally and concretely turned into an intermedial product when an allegedly abstract but, in fact, highly sensuous contemporary painting by American artist David Reed shifts over it, for a moment almost – but not quite! – hiding it. In the same vein, I also staged interactions of Cassandra with two contemporary paintings, by the brilliant South-African-Dutch painter Ina van Zyl. These paintings are made in a hyperrealistic mode and thereby precariously balance on the sharp and impossible distinction between reality and fiction, as well as semiotic utterances and “real life.” In both instances, the figures, actors, and characters, in interaction with the media products of the other medium, become figural and, as such, figurations of ideas. These two figures are thus themselves figural instances of preposterous history, as what rhetorical analysis would call its personifications. The same can be said about their figuration of intermediality, through quotation – literal quoting

Needless to say, this Benjamin quote supports the idea of “pre-posterous history.” To preserve the past, it must be made actual in the present and have relevance there.

28

484

M. Bal

of the texts they read and figural quoting of the postures.29

The enactment

The history lesson

The live actors, Adrian Budakow as Aeneas and Magdalena Żak as Cassandra, were fabulous. To make a somewhat banal point, which does, however, concern the 29

Cassandra reads the key passage from the introduction of my 1999 book on pre-posterous history, with its Benjaminian overtones, and the enactment of the Caravaggio, with the Reed painting shifting over it, is an example, or embodiment of it.

20

Citational Aesthetics: For Intermediality as Interrelation

485

semiosphere with its political domain, as well as intermedial dialogue: a preliminary question I had for Adrian was if he would mind appearing half-naked, in the impersonation just mentioned. This was necessary to make the concept of tableau vivant literal, as an intermediation between a historical, two-dimensional figuring of male sex appeal in the Caravaggio into a live dramaturgical three-dimensional act, which seemed relevant in the contemporary identity craze. This also figures my conception of Baroque as both philosophical and artistic, on behalf of social issues. Such mundane-seeming issues are all part of designing a film intermedially quoting a novel. No strictly delimited semiotic theory can obscure it: such interventions also have an ethical side. This is obvious but also theoretically plausible, since the semiosphere is also a sociopolitical sphere.

Intermediality in and for the World The process of intermediality characteristic of filmmaking gets more banal and more intermedial, as well as “worldlier” or geared to the bond between art and society. For Cassandra I had to develop ideas about how to visually figure stubbornness and despair – from literature to film. In that context there is something that, in its materiality, is also firmly anchored in the semiosphere, as well as partaking of the intermediality: costumes. As a teacher she looks proper and serious, in a black suit, her hair in a bun, as in the video still. For the scenes in her home, her parents’ palace, I brought a shapeless and colorless (off-white) silk dress, underneath which she wore her own contemporary punkish half-boots. This combination alludes, of course, to preposterous history. I also brought a necklace consisting of large links, a chain that, coming close to merging iconicity with indexicality, would bring in the idea of captivity, spoken about in Wolf’s novel. This brought the serious historical (alas, nonfictional) topic of slavery into the temporary European semiosphere. Is that historically justified, even in a traditional sense of history? Unfortunately, it is. In that sense, history cannot be ignored. I had been intensely focusing on slavery in an intermedial and intertemporal video project I have made in 2019, DON QUIJOTE: SAD COUNTENANCES. And, although Cervantes created Don Quijote after five and a half years of suffering slavery in Algiers, after he was captured within the Mediterranean, we know only too well that slavery also continues to occur, including within the European semiosphere. This has been amply studied in a historical perspective, but also in the contemporary one, which is shocking in its actuality. And in Wolf’s novel, from (then) East Germany, Cassandra reflects on her captivity, even if it is in the rich palace of her parents.30

30

The sixteen screens of the Don Quijote project can be found on my website, at http://www. miekebal.org/artworks/installations/don-quijote-sad-countenances/. An extensive catalogue of the Don Quijote project has been published in English and Spanish (Bal 2020). On slavery in Cervantes’ case, see the incisive, well-documented award-winning study by Colombian literary scholar María Antonia Garcés (2005). On contemporary slavery, see Bunting and Quirk (eds.) 2017.

486

M. Bal

Semiosis does not easily signify time. Especially the Peircean category of iconicity, of which the ground (as Peirce calls it) is similarity, does not lend itself to produce a sign of time. Yet, for an effective intermediality I was determined to bring together, not in harmony but as a “discussion,” my many concerns about time. As mentioned above, I have developed and put to work a notion of time that acknowledges that not only the past influences the present, hence also the future, but also the other way around. This concerns chrono-logic. But there is more to temporality that relates intermedially different media to one another. An important figural aspect of time is rhythm. This has a bodily side to it, which is vital if we want to recognize the importance of the body as not separate but at one with the mind. This was a strong issue in the film on Descartes, REASONABLE DOUBT, which I made in 2016. In any film, rhythm is always important, but especially in one primarily concerning time thematically and politically. In view of the second meaning of the title, the rhythm is almost hectic, becomes stronger as the film progresses, and closes on a frantic dance by Cassandra. Dancing becomes an icon for overwrought passionate bodily behavior. When she stops dancing, or rather, shaking her body in protest, she ends up saying: “the future is now,” perverting chronology even more strongly, but with a real-sounding urgency.31 Indeed, the backbone of the recent film as well as the (literally) quoted novel is Cassandra’s temporal awareness. Her repeated call for urgency is key, both to the ancient myth and Wolf’s subjectivation of it and to my attempt to make a socially relevant film on this issue. The exclamation mark in the title as a shifter pushes the two issues together, intermedially bringing sound and rhythm together. Aeneas’ interest in participating in political power, his rationality, and his resistance against Cassandra’s wisdom figure the other side. This is the impossibility of Cassandra’s wisdom winning the upper hand in a semiosphere where men have more influence than women, and (official) politics rules over, and overrules, the (social) political. And in addition to these three aspects of temporality, one of which, rhythm, is figured in Cassandra’s frantic disco dance towards the end, the most personal, intimate moment in the film, I thought, should also be a strong instance of an intermedial figuration of time, one when the near-future infringes on the figures’ personal lives and the power relations are put on hold. I also sought to intermedially quote contemporary love stories and their sentimentality, setting that off against political relevance. This became the moment when, walking in the rainy city streets, holding hands, Cassandra dumps Aeneas as her lover, because he remains too close to the powers-that-be, resulting in a near future in which he would become stultified. Whereas he asks her to come with

For more information on the film REASONABLE DOUBT in which I challenge the cliché that Descartes separated body and mind so rigorously and the links to the installation pieces, see http://www.miekebal.org/artworks/installations/reasonable-doubt/. The feature film can be accessed through this link: https://player.vimeo.com/video/149385127. Password: CinemaSuitcase.

31

20

Citational Aesthetics: For Intermediality as Interrelation

487

him to escape from the dangerous place wherein they are caught, she simply, and still, affectionately, refuses. This, in her wording, concerns the future – one she rejects. She abandons him with the poignant words: “I cannot love a hero. I do not want to see you being transformed into a statue.” This wording can be comparable to the exclamation mark: a gearshift of meanings. The metaphor of “a statue” suggests rigidity, stultifying, death. This is the ending of Wolf’s novel. Cassandra’s words cited above, “the future is now,” spoken after her frantic disco dance, marks the end of the film. I have brought in the essay film to argue that the intermediality of film is in and of itself socially politically relevant. Only a media product that is by definition both multimodal and intermedial can bring the issues I have mentioned so close that they get under the perceiver’s skin. That effect, which concerns affect, needs art that is not so much, or not only, activist, but activating. It makes perceivers think, but what they think is up to them. This is why the question of fiction and reality is not a binary opposition, no more than words and images. Fiction is part of reality and is impossible without bringing it up constantly, framing the elements of the fiction within it. The concept of activating art needs yet another philosophical inspiration. According to the ground-breaking philosophy of language developed by John Austin, mentioned above, it is better to change gears and consider media products not informative but enhance their performativity. This can result in the shift from activist art, which persuasively focuses on specific political issues, trying to tell perceivers what to think, to activating art that seeks to strengthen the performativity of the signs and the media products but leaving the specific interpretations and thoughts, and perhaps subsequent actions, up to the perceivers. That performative force – mind Lyotard’s view of language – makes the transfer between producer and perceiver more effectively affective.

Intermezzo 4: The Artistic-Political Power of Imperfection (to Shame the Bad Guys) On Christmas eve 2020, Nalini Malani drew this media product as number 41 of 89 works on paper she made during the first COVID-19 lockdown. She started this series when she got stuck in Amsterdam and could not return to Bombay, nor have access to her studio and equipment there. On the kitchen table in Amsterdam, she started to draw and did not stop. The collection of sheets was immediately exhibited in the Art Museum of The Hague. The media product I have selected for this intermezzo ended up on the back cover of a beautiful 2021 book in which the 89 drawings are reproduced in actual size and on the same paper on which they were made. A perceiver can see the figure here as a child, a fantasy figure, half-animal, half-human, or something else, clearly fictional, but the lettering points us inevitably to a political view.

488

M. Bal

Nalini Malani, Exile – Dreams - Longing, 2020, nr. 41.

Of course, the presence of words turns the drawing immediately into an intermedial media product. But that is by far not all there is to its intermediality. On the image the figure looks like a monster. Of course, the written words make it hard to see it otherwise. “All tyrannies rule through fraud and force” is a clear political statement. But I want to seek intermediality in other details than this loud and clear indictment of the currently widespread dictatorships. Behind the statement in large print are some small figures, one of which is a girl drawn in terracotta – also the background color – and looks like the young girl Malani frequently calls up in her worldwide autobiographical tenor, her version of herself and/as Alice. This is her way of subtly inserting first–second-person discourse, with her own presence being a participant. The thought bubble the girl figure lets fly upwards like a gas-filled balloon, or alternatively, tries to catch, cries out for the liberty that the monster is destroying through fraud and force. Her light-colored shape cannot overrule the black-outlined monster, who is looking the other way anyway. The force and weakness of the lines participate in the meaning-making. The monster, looking away, totally ignoring the small girl, is callously walking on a small, headless human figure. Underneath it, two hands, one drawn in red and one in black, point

20

Citational Aesthetics: For Intermediality as Interrelation

489

to it: the red one to the neck-without-head and the black one to the monster stepping on it. These hands work as indexical signs. This is all very subtle and requires time to take in. A full description of the sheet is impossible. But the longer we look, the more comes up. On the left-hand corner at the bottom is a head with spread arms, an open shouting mouth, and over it, the word “freedom,” this time written in blue. That figure might evoke a drowning child. Between the monster, drawn in black, and the letters that indict him and his peers, some other words are written, barely readable: magic, lies, terror, cheat. The vaguely, almost abstract figuration above the monster’s large bold head, of a pair of black dots, perhaps eyes, held together by two fingers pointing in opposite directions, might be seen as a member of society asking the monster/tyrant to account for his unjust rule. What happens between media, here, is a numerous, endless play. The intermediality is not primarily the inclusion of words in the image, but more significantly between the words mechanically printed in black and large, which state the situation of the world as in newspaper headings, and the other words, hand-drawn in color and small. As a perceiver of this image I see in this difference the power and the powerlessness – politics and the political. I submit that the fundamental intermediality that makes this sheet a strong intervention in the social-political world is primarily located in the different lettering, rather than in the inclusion of text in a visual drawing. For this, Malani’s riveting and demanding image, which will hold the viewer’s attention for quite a long time, is compelling. It is precisely in the irregularities, the hand-written letters that through their technical imperfection draw attention to the fact that the “official”-seeming letters are not perfectly printed, in spite of their pretention. The top line has letters that are not opaque black. And the words are obliquely disposed on the sheet. This compels me to look again at the sheet as a whole. The somewhat strange composition in which the tyrant, the ruler, neither has a full body nor is he central, the technique of layering staining instead of a perfect opacity of colors, and the red dots evoking blood spatter spilling over the lines of the drawing: all these features produce the time-consuming effect the media product generates. The intermediality resides primarily in the imperfection.32

Conclusion: Analyzing and Teaching Intermediality A short ending on the how question: with intermediality both pervasive and complex, how can we grasp its working, through analysis and teaching? With “analysis” I mean the work of interpretation in detail, what used to be called “close reading.” I do not care much for generalizing, nor for characterizing media product as “intermedial,” since practically all of them are that. Such generalizations, therefore, do not have much specific theoretical or analytical power. They do not yield specific 32

On imperfection as an artistic and political ruse, see the collective volume from 2022, edited by Kelly et al.

490

M. Bal

information about how the works work. Instead, I have tried to demonstrate, in the four intermezzo’s, how special attention for the way the intermediality in the media product works illuminates the special effects it emanates. This is a skill that can be taught, through the dialogic teaching of an attitude, rather than a storm of terminological and conceptual specifications. This is why I have not used all the technical terms that pertain to intermediality.33 What we must practice and help students to learn to practice is the attitude of close, detailed attention to those features that connect the media product to its own primary medium as well as the others that it integrates. This is a goal of my focus on quotation, a device that is specific enough to require precise, detailed analysis. In contrast to the preconception that assumes close reading narrows the work down, such attention to the specific workings of the intermedial relations, not simply diagnosing the fact of intermediality, opens it up. The difference and connection between black sand and glistering water drops in Salcedo’s Palimpsesto, and the formal differences between the large and the small letters in Malani’s number 41, are two examples of this opening up. Instead, it is the assumption of mono-mediality – which ignores the other intervening medialities – that closes the media product for the way it deploys intermediality to address political and social issues, as well as aesthetic effects. It is in view of the need to open the artworks up and let them affect their perceivers that is so urgently needed. This requires resisting the illusion that, for example, literary texts are just linguistic works, mostly appealing to semiotic interpretation, and visual artworks only consist of colors, lines, and composition, appealing to the sensorial experience but having no meaning to worry about.

References Alpers, Svetlana, and Michael Baxandall. 1994. Tiepolo and the pictorial intelligence. New Haven: Yale University Press. Austin, J.L. 1975. [1962] how to do things with words. Cambridge: Harvard University Press. Bakhtin, Mikhail. 1981 The Dialogic Imagination: Four Essays by M.M. Bakhtin, ed. Michael Holquist, trans. Caryl Emerson and Michael Holquist. Austin, Texas: University of Texas Press. Bal, Mieke. 1987. Lethal love: Literary-feminist readings of biblical love stories. Bloomington and Indianapolis: Indiana University Press. ———. 1996. Double exposures: The subject of cultural analysis. London and New York: Routledge. ———. 1999. Quoting Caravaggio: Contemporary art, preposterous history. Chicago, IL: University of Chicago Press. ———. 2002. Travelling concepts in the humanities: A rough guide. Toronto: University of Toronto Press. ———. 2010. Of what one cannot speak: Doris Salcedo’s political art. Chicago, IL: The University of Chicago Press.

33

In Lutters (2018) I explain at length why and how teaching must be dialogic in order to work at all. My position there can be summed up in the slogan: “if you don’t learn from your students, you are a bad teacher.”

20

Citational Aesthetics: For Intermediality as Interrelation

491

———. 2016. In medias res: Inside Nalini Malani’s shadow plays. Ostfildern, Germany: Hatje Cantz. ———. 2017. Emma & Edvard Looking Sideways: Loneliness and the cinematic. Oslo: Munch Museum / Brussels: Mercatorfonds; Yale University Press. ———. 2020 Don Quijote: Tristes figuras; Don Quijote: Sad Countenances. Murcia, Cendeac (Ad Litteram). ———. 2021. “Foreword: Mediations of method.” v-ix. In Beyond media Borders: Intermedial relations among multimodal media, volumes 1/2, ed. Lars Elleström. Basingstoke: Palgrave Macmillan. ———. 2022. Image-thinking: Artmaking as cultural analysis. Edinburgh: Edinburgh University Press. Barthes, Roland. 1986. In [1968] “The Reality Effect.” 141–54 in The Rustle of Language, ed. Roland Barthes . New York: Hill and Wang.trans. Richard Howard. Battersby, Christine. 1989. Gender and genius: Towards a feminist aesthetics. London: The Women’s Press. Baxandall, Michael. 1985. Patterns of intention: On the historical explanation of pictures. New Haven, CT: Yale University Press. Benjamin, Walter. 1969. Illuminations. Schocken: Trans. Harry Zohn. Edited and with an introduction by Hannah Arendt. New York. Benveniste, Emile. 1971. [1966] problems in general linguistics. Trans. Mary Elizabeth Meek. Coral Gables, FL: University of Miami Press. Bois, Yve-Alain et al. 1998. A Conversation with Hubert Damisch. 3–17 in October 85 (Summer). Bunting, Annie, and Joel Quirk, eds. 2017. Contemporary slavery: Popular rhetoric and political practice. Vancouver: UBC Press. Butler, Judith. 1990. Gender trouble: Feminism and the subversion of identity. New York: Routledge. ———. 1993. Bodies that matter: On the discursive limits of "sex". New York: Routledge. Carrier, David. 1982. Art without its artists? British Journal of Aesthetics 22: 233–244. ———. 1991. Principles of art history writing. University Park, PA: Pennsylvania State University Press. ———. 1994. The aesthete in the City: The philosophy and practice of American abstract painting in the 1980s. University Park, Pennsylvania: Pennsylvania State University Press. Davey, N. 1999. The hermeneutics of seeing. In Interpreting visual culture: Explorations in the hermeneutics of the visual, ed. Ian Heywood and Barry Sandwell, 3–29. London and New York: Routledge. Derrida, Jacques. 1973. [1967] speech and phenomena and other essays on Husserl's theory of signs (studies in phenomenology and existential philosophy), trans. David B. Allison and Leonard Lawlor. Evanston, IL: Northwestern University Press. ———. 1976. Of grammatology. Trans. Gayatri Chakravorty Spivak. Baltimore: Johns Hopkins University Press. ———. 1981. Positions. Translated by Allan Bass. Chicago: University of Chicago Press. ———. 1988. Limited Inc. Trans. Samuel Weber. Evanston, IL: Northwestern University Press. Elleström, Lars, ed. 2019. Transmedial narration: Narratives and stories in different media. Basingstoke: Palgrave Macmillan. ———, ed. 2021. Beyond media Borders: Intermedial relations among multimodal media, volumes 1 and 2. Basingstoke: Palgrave Macmillan. Fish, Stanley. 1980. Is there a text in this class? Cambridge, MA: Harvard University Press. Fuchs, Martin. 2001. Textualizing culture: Hermeneutics of Distanciation. In Travelling concepts, ed. Joyce Goggin and Sonja Neef, 55–66. Amsterdam: ASCA Press. Garcés, María Antonia. 2005. Cervantes in Algiers: A Captive’s tale. Nashville, TE: Vanderbilt University Press. Garrard, Mary. 1989. Artemesia Gentileschi: The image of the female hero in Italian baroque art. Princeton: Princeton University Press.

492

M. Bal

Hirschkop, Ken, and David Shepherd, eds. 1989. Bakhtin and cultural theory. Manchester: Manchester University Press. Høeg, Mette. 2021. Literary theories of uncertainty. London: Bloomsbury. Holly, Michael Ann. 1984. Panofsky and the Foundations of Art History. Ithaca, and London: Cornell University Press. ———. 1996. Past looking: Historical imagination and the rhetoric of the image. Ithaca and London: Cornell University Press. Huyssen, Andreas. (2018) A Palimpsest of Grief: Writing in Water and Light, 4–11 in Alsaho: Doris Salcedo, White Cube catalogue for Doris Salcedo show (Musumeci: Italy). Ionescu, Vlad. 2018. “The figure in time: On the temporality of the figural”, 11–61 in Laura Marin and Anca Diaconu, working through the figure: Theory, practice, method. Bucureşti: Editura Universităţii din Bucureşti. Kelly, Caleb, Jakko Kemper, and Ellen Rutten, eds. 2022. Imperfection: Studies in mistakes, flaws and failures. London: Bloomsbury. Laplanche, Jean. 1996. Psychoanalysis as Anti-Hermeneutics. Trans. Luke Thurston. 7–12 In Radical Philosophy 79 (Sept./Oct). Lutters, Jern. 2018. The trade of the teacher: Visual thinking with Mieke Bal. (Interviews). Amsterdam: Valiz. Lyotard, Jean-François. 2020. [1971] Discourse, figure, Antony Hudek and Mary Lydon (trans.). Minneapolis, MS: The University of Minnesota Press. Malani, Nalini. 2021. In Exile – Dreams - longing, ed. Johan Pijnappel. Den Haag: Kunstmuseum. McEvilley, Thomas. (1993). Ceci n'est pas un Bidlo? Rethinking Quotational theory. In The Exile's return: Toward a redefinition of painting in the post-modern era (pp. 167–173). Cambridge University Press. Mitchell, W.J.T. 1987. Going too far with the sister arts 1–10. In Space, time, image, sign: Essays in literature and the visual arts, ed. James A.W. Hefferman. New York: Peter Lang. Mondzain, Marie-José. 2005. Image, icon, economy: The byzantine origins of the contemporary imaginary, trans. Rico Franses. Stanford, CA: Stanford University Press. Mouffe, Chantal. 2005. On the political. New York and London: Routledge. Panofsky, Erwin. 1995. [1955] meaning in the visual arts. Harmondsworth: Penguin. Peeren, Esther. 2008. Intersubjectivities and popular culture: Bakhtin and beyond. Stanford: Stanford University Press. Rascaroli, Laura. 2008. The essay film: Problems, definitions, textual commitments. Framework: The Journal of Cinema and Media 49 (2): 24–47. Richter, Gerhard. 2007. Thought-images: Frankfurt school writers’ reflections on damaged life. Stanford: Stanford University Press. Rodowick, D.N. 2001 Reading the figural, or, philosophy after the new media, Durham and London: Duke University Press. Spitzer, Leo. 1983. Essays on seventeenth-century French literature. Trans., edited, and with an introduction, by David Bellos. Cambridge/New York: Cambridge University Press. Stock, Brian. 1983. The implications of literacy: Written language and models of interpretation in the eleventh and twelfth centuries. Princeton: Princeton University Press. Torop, Peeter. 2022. Semiosphere. In The companion to Juri Lotman: A semiotic theory of culture, ed. Marek Tamm and Peeter Torop, 296–307. London: Bloomsbury. van Alphen, Ernst. 1997. Caught by history: Holocaust effects in contemporary art, literature and theory. Stanford, CA: Stanford University Press. ———. 2005. Art in mind: How contemporary images shape thought. Chicago: University of Chicago Press. Wolf, Christa. 1988. [1983] Cassandra: A novel and four essays, translated by Jan Van Heurck. New York: Farrar, Straus and Giroux.

Part II Intermedial Perspectives on Media Until the Nineteenth Century: A Living Legacy

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since the Eleventh Century

21

Rong Ou

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Traditional Chinese Painting: A Work of Sister Arts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Genius of Three or Four Perfections: Integration of Sister Arts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Dynamic Interaction and Fruitful Rivalry of Sister Arts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Artistic Creation as Social and Cultural Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

496 497 502 507 510 517 518

Abstract

Traditional Chinese painting is rarely a pure painting, but a composite artwork with the dynamic intermedial play among painting, poetry, seal, and calligraphy in one scroll/album. The practice of inscribing a poem in painting arguably can be dated back to the eleventh century, pioneered by Su Shi, a prolific poet/scholar/ painter of Northern Song Dynasty. With a reference and challenge to a few western scholars’ theoretical frameworks of intermediality, this chapter explores the complex interaction of different media in the traditional Chinese painting. On the one hand, there is strong collaboration among different agents. Very often, after a painter finishes his/her share of work, a poet who may be his/her friend, patron, customer or later collector, composes a tihuashi (comparable to an ekphrasis) for the painting, then a calligrapher writes the poem on the painting, and the artwork is not completed until seal(s) stamped on it. Sometimes, the painter, the poet, the calligrapher, and the seal engraver are identical of one versatile talent; in this case, the intermedial collaboration is most integrated. Nevertheless, when they are different operators, there is subtle resonance and rivalry among them: the painter dominates the scroll, sparing limited space for other actors, while the poet intends to go further through the verbal depiction of R. Ou (*) Hangzhou Normal University, Hangzhou, China e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_24

495

496

R. Ou

what is visual and the nuance of what is not visual, the calligrapher competes with the ink brush, the same medium as the painter’s, and the seal engraver strives to be impressive with a strikingly red pattern on the ink scroll, all of which provide clues what roles the different media play in the performance and participation of the art. Keywords

Traditional Chinese painting · Tihuashi · Ekphrasis · Calligraphy · Seal

Introduction Concerning the relation of text to picture, Hans Lund, the Scandinavian pioneer interart/intermedia scholar, provides a scheme to establish three main categories: combination, integration, and transformation (1982). By combination, he means “a coexistence, at best a cooperation between words and pictures,” such as the old emblematic writings; by integration, he refers to that “a pictorial element is a part of the visual shape of a literary work,” like the concrete poetry of Modernism; by transformation, he refers to the text in which “the information to the reader about the picture is given exclusively by the verbal language,” like an ekphrastic poem (Lund 1992: 8–9). Later on, Lund expands this triple framework to cover the division of all possible intermedial relations in his introductory chapter in Intermedialitet, with the category of “combination” subdivided into “interreference” and “co-existence” (Lund 2002: 21). Lund’s efficient intermedial schematization has been elaborated by Claus Clüver (1993) and Leo Hoek (1995) with semiotic orientation, whose observations have been synthesized by Eric Vos (1997) in a table that is slightly modified by Clüver to reformulate the word-image relations in a schema of four categories: transmedial relation, multimedia discourse, mixed-media discourse, and intermedial discourse (Clüver 2007: 26). Iria Rajewsky also proposes a heuristic tripartite division of intermedial practices into medial transposition, media combination, and intermedial references (Rajewsky 2005: 53). These categorization of intermedial phenomenon can work as points of reference for us to analyze the traditional Chinese painting, which is a more complex and comprehensive form of intermedial practice, as Rajewsky notes that “a single medial configuration may certainly fulfill the criteria of two or even of all three of the intermedial categories outlined above.” (Ibid.) Traditional Chinese painting has been rarely a monomedial product, but a composite artwork with a dynamic intermedial play among painting, poetry, calligraphy, and seal in one scroll/album/screen/fan since Northern Song Dynasty (960–1127). Nevertheless, the word-image relationship of Chinese paintings cannot be fully illustrated by any of the above single category. With a historical research and a case study of a few representative works, the chapter explores the complex interaction of different media in the traditional Chinese painting.

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

497

On the one hand, there is strong collaboration among different agents of artistic practice. Very often, after a painter finishes his/her share of work, a poet who may be his/her friend, patron, customer, or a later collector/connoisseur, composes an ekphrastic poem for the painting (known as tihuashi 题画诗), then a calligrapher writes the poem on the painting, and the artwork is not completed until seal (s) stamped on it. The ekphrastic poem helps to provide the background information of the painting, to highlight what is depicted, to reveal the intention of the painter, to make a comment on the painting, and/or to extend the poet’s emotion inspired or epitomized by the imagery in the painting. The ekphrastic poem survives on the scroll with the effort of the calligrapher, and the seals help to identify the painter and the calligrapher. Sometimes, the painter, the poet, the calligrapher, the seal engraver are identical of one versatile talent; in this case, the intermedial collaboration is most integrated, illustrated in Tang Yin’s figure paintings. On the other hand, when they are different operators, there is subtle and fruitful rivalry among them: the painter dominates the scroll, sparing limited space for other actors, while the poet intends to go further through the verbal depiction of what is visual and the nuance of what is not visual, the calligrapher competes with the ink brush, the same medium as the painter’s, and the seal engraver strives to be independent with a strikingly red pattern on the ink scroll, all of which provide clues on what roles the different media play in the performance and participation of the art, best illustrated in Wen Zhengming’s flower painting and Shu Shi’s three pieces of tihuashi for Wen Tong’s bamboo painting. Nevertheless, in Chinese traditional culture, the poetic and artistic creation, besides being a means of making a living or the service for royal family, is an important part of social interaction. For traditional Chinese literati, they do literary and artistic creation not for profit, but as gifts to their friends or colleagues, as tokens of friendship; therefore, there is more affinity and resonance rather than “power struggle” or “gendered opposition” between word and image as argued by W. J. T. Mitchell in Iconology (1986) and James Heffernan in Museum of Words (1993), best illustrated in a cycle of works inspired by Wang Shen’s landscape painting and the ekphrastic network deriving from the celebrated Western Garden Gathering.

Traditional Chinese Painting: A Work of Sister Arts The word-image relationship in Chinese art developed from one of complementary illustration to one of complex integration. Qi Gong 启功 (1912–2005), a distinguished contemporary painter, calligraphy, and scholar, in an illuminating article (1991), examines the relationships between poetry, calligraphy, and painting from a number of different perspectives. Concerning the relationship between poetry and calligraphy, Qi finds that, as a vehicle of literature, calligraphic technique and style cannot exist independently of the poetic content. First, as the literary content to be written can influence the calligrapher’s emotion, his particular emotion at the time of writing will in turn exert influence

498

R. Ou

on his calligraphy. Second, the calligraphic style should be appropriate to the content of a particular poem. For example, it would be considered in poor taste to use Yan Zhenqing’s 颜真卿 (709–85) round, full characters and substantial strokes to transcribe a subtle line of poetry, and it is proper to use Zhu Suiling’s 诸遂良 (596–658) graceful characters and delicate strokes instead. Qi concludes, however, “While poetry and calligraphy are linked in some way, the relationship between them is neither as close nor as complex as the relationship between poetry and painting” (Qi 1991: 12). Concerning the relationship between calligraphy and painting, the pictorial nature of the Chinese written characters, possibly its most outstanding feature when compared with phonetic transcriptions of western languages, finds the inherent affinities between writing and painting. The saying “calligraphy and painting have the same source” (shu hua tong yuan 书画同源) is often cited as evidence that the two arts are closely related as indicated in Fig. 1. Traditional painting involves essentially the same techniques as calligraphy and is practiced with a brush dipped in black ink or colored pigments; oils are not used. The brush is the instrument of both the calligrapher and the painter, and the results produced by the use of dots and strokes are similar. Even in the works of some artists, the bamboo leaves and tree branches and the outlines and texture-strokes of their mountains and rocks are virtually identical to the brushstrokes used in calligraphy. Writing brush, ink stick, ink slab, and paper are known as the Scholar’s Four Jewels or the Four Treasures of Study (wen fang si bao文房四宝); together with seal, there are common daily companions to traditional educated Chinese. A typical scene for a traditional Chinese painter to do his work is depicted thus: On days when he was going to paint, he would seat himself at a clean table, by a bright window, burning incense to right and left. He would choose the finest brushes, the most exquisite ink; wash his hands, and clean the ink-stone, as though he were expecting a visitor of rank. He waited until his mind was calm and undisturbed, and then began (Jenyns 1966: 134)

People appreciate good calligraphy just much as they appreciate good painting. Still, according to Qi Gong, “this does not imply that calligraphy and painting are

Fig. 1 Different scripts of 馬/马 (horse) along the Chinese history

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

499

fundamentally the same.” He proposes the saying “Calligraphy and painting have the same core, or the central idea” (shu hua tong he 书画同核) should be more accurate a characterization of the relationship of the painting and calligraphy (Qi 1991: 13). In Qi’s opinion, the relationship between the painting and poetry is the most intimate: Poetry and painting are siblings from the same womb; both emerge from the emotions and the environment of daily life, and both must have beauty and the power to move people. . . .. Of course, not every painter is a poet, nor every poet a painter. Still, every good poem or painting must possess a certain quality that calls forth admiration. The elements of a good poem may not be those of a good painting, and vice versa, but the qualities which constitute excellence are fundamentally the same for both. Thus, there are dissimilarities as well as underlying similarities between the two arts (Qi 1991: 14–15)

There are two types of “poetry in painting and painting in poetry.” In the first type, there are “painterly” phrases in poem to fuse scene and mood or vivid descriptions of scenery, and there are “poetic” literati paintings that are more artistic and literary than artisans’ mechanic visual representation. The second type refers to a painting with its extended poem, known as tihuashi, in which painting and poetry are linked in a thematic or tangible manner. The Chinese character-word ti (题), when used in the term tihuashi, means either “to write about” or “to inscribe on,” hua (画) means “painting” and shi (诗) means “poem(s)” or “poetry.” Therefore, tihuashi, in the narrow sense, refers to poems written on the picture, and in the broad sense, it refers to all poems written about paintings, inscribed on the painting or independent of the painting. The origin of tihuashi remains a subject of debate. Some scholars trace it back to “The Riddles” (or “Heavenly Questions” 天问), a remarkable poem composed by Qu Yuan 屈原 (340?–278 BC), a great Chinese poet and politician in the State of Chu during the Warring States period (475–221 BC). Slandered by corrupt ministers and banished to the remote region of the state by King Huai of Chu (reigned 328–299 BC), Qu Yuan visited the shrines of the former kings and the ancestral halls of the nobles, painted with magnificent and fantastic pictures of heaven and earth, mountains and rivers, gods and spirits, jewels and monsters, and the wonders and the deeds of ancient sages. Inspired by the wall paintings, the poet asked over 170 questions about the universe, the creation of the heavenly bodies, the earth, myths and legends, and historical events in his poem. These “raving questions” were said to have been written on the walls with the pictures as well, which has not been confirmed with the extant archeological findings (Liu 2010: 21–24). Another prevailing argument on the origination of tihuashi is regarding tihuashi as the convergence of huazan (画赞 eulogy complementing painting) which existed as early as the Han period (206 BC–AD 220) and yongwushi (咏物诗 literally, poetry written about material objects or things), which was quite a poetic vogue during the Six Dynasties period (220–589) (Shen 1993: 8). Daan Pan also relates tihuashi to nonpoetic literary genres of huaming (画铭 epitaph complementing painting), huaba (画跋 postscript or colophon about painting), and huaji (画记 notes on painting) (Pan 2011: xi). Before the Northern Song period, tihuashi and its related

500

R. Ou

painting are mostly separate, or there are few extant paintings inscribed with tihuashi before the Song and Yuan Dynasties. Generally speaking, writing of tihuashi as an independent subgenre of poetry rises in the Tang period (618–907), booms in Song (960–1279) and Yuan period (1279–1368), comes to its full fruition in Ming (1368–1644) and Qing period (1644–1911) and declines in modern China. In theory, tihuashi relates to almost all genres of painting, while in practice, there are more tihuashi upon/about the landscape paintings than flower and birds paintings and figure paintings. The most comprehensive official anthology of tihuashi, entitled Yuding lidai tihuashi lei (御定历代题画诗类 The Imperial Authorized Compilation of Classified Tihuashi of the Past Dynasties) and compiled by the royal academician Chen Bangyan 陈邦彦 (1678–1752) under the auspices of Emperor Kangxi (1654–1722) in the year 1707, collects about 9000 poems, and the classification of tihuashi is indicated in Table 1. As to the specific styles of tihuashi inscribed on the painting, Li Xunting李旭婷 studies the collection of the Palace Museum as a sample, where the largest number of traditional Chinese paintings in China is collected. Li finds that, according to Gugong bowuyuan lidai huihua tishi cun (故宫博物院历代题画诗存 Palace Museum Anthology of Tihuashi of Past Dynasties), wujue (five-character quatrain) and qijue (seven-character quatrain) are the most popular poems of inscription, followed by wulv (five-character octave) and qilv (seven-character octave), the poems of more than ten lines (wugu/gupai and qigu/qipai) are much less popular, owing to the limited space of the pictorial compositions, as indicated in Table 2. Both tables also manifest the historical growth of tihuashi on paintings since Song period. Before the Northern Song Dynasty, there were few paintings with inscriptions on them except for the painter’s signature at an inconspicuous corner. When certain qualities of the ideographic writing system were conjoined with larger intellectual and aesthetic developments in China during the late Song and the early

Table 1 Classification of Tihuashi in Emperor Kangxi Anthology (Li 2021: 121) Tang Song Jin and Yuan Ming

Poems on landscape 58 321 1336 1514

Poems on birds and flowers 24 249 1032 1029

Poems on figure 32 254 866 688

Table 2 Subgenres of Tihuashi in Palace Museum Anthology (Li 2021: 120) Tang Song Jin and Yuan Ming Total

Wujue 0 2 8 62 72

Qijue 0 12 27 140 179

Wulv 0 2 0 19 21

Qilv 0 8 11 21 40

Wugu/gupai 0 0 2 8 10

Qigu/qipai 0 0 2 9 11

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

501

Yuan dynasties, that is, in the late thirteenth and early fourteenth centuries, the pictorial nature of Chinese writing did evolve into an influential force and helped to shape the aesthetics of calligraphy and painting, and characters were not used simply as the unavoidable media but were also made use of as symbolic systems. Likewise, after the development of realistic representation in painting reached its full flowering during the late Song Dynasty, early Yuan painters turned increasingly to symbolism, known as xieyi (写意) or freehand style. When eventually the image became overladen with metaphorical implication, it could no longer be expressed without the help of language. Painting, by its nature, has limitations. An unidentified portrait does not tell the viewer who is depicted. An unidentified landscape does not tell the viewer where the scene is located. In these cases, information to supplement the painting is needed. One way of supplying such information might be to attach a note (huaji) to the painting, but a note could identify only the person, place, or type of subject that is depicted; it could not convey the artists’ feelings about the subject. Inscribing a few words or a poem on the painting could serve to clarify what the painter/viewer/poet has in mind. The large space in traditional Chinese painting since Song Dynasty physically leaves the room for proper ekphrastic inscriptions and red seals function much aesthetically in ink paintings as well. Many visual artists were deeply influenced by the aesthetics of literature and drew upon China’s rich tradition of verbal imagery in creating their paintings. Through masterful calligraphy, vivid depictions of historical narrative, and subtle images containing personal allusions, the classic Chinese painting illustrates the affinities that painting shares with calligraphy and the poetry, most evident visually in the supple brush line that gives formal expression to all. And the stamping of a proper seal or seals gives a final touch to the works of art. With the rise of the landscape painting and literati painting as siyi (“four arts” 四 艺), namely, the mastery of music, Go, calligraphy, and painting, the four main academic and artistic talents were required of the scholar-official class, and more and more talented artists turned up and practiced the intermedial play in their works of three perfections of poetry, painting, and calligraphy or four perfections of poetry, painting, calligraphy, and seal. Poetry and painting, known as the sister arts in the West, both served the same master – the man of letters. Poets, immersed in the beauty of scenery, transformed visual observations into pure and polished phrases, while painters transformed inspiring poems into pictures. Thus the aspects of poetry and painting are inextricably connected (Xie 1991: 3). It was Su Shi 苏轼 (1037–1101), the leading scholar-artist of the late Northern Song Dynasty, first advocated that there is “poetry in painting and painting in poetry” (Su 2016: 166) in praising Tang poetpainter Wang Wei 王维 (701–61)and one of the earliest to put down the poem and painting on a same scroll. The seal became more functional aesthetically also in Song Dynasty, not simply as a supplement to the artist’s signature, but also as a part of the painting with its unique combination of white and red when ink painting became the norm of the art. By brushing in a poem and stamping seal(s) on the painting and thus using both word and image, the artist(s) created a verbal discourse and a broader context in which to express himself/themselves.

502

R. Ou

Genius of Three or Four Perfections: Integration of Sister Arts In Iconology (1986), W. J. T. Mitchell addresses the relation between literature and the visual arts as essentially paragonal: a struggle for dominance between the image and the word. Building on Mitchell’s ideological treatment of the word-image relation, James Heffernan, in Museum of Words (1993) investigates ekphrasis, the literary representation of visual art, as “the expression of a duel between male and female gazes, the voice of male speech striving to control a female image that is both alluring and threatening, of male narrative striving to overcome the fixating impact of beauty poised in space.” (1) Nevertheless, neither of the two theories is adequate to interpret the word-image relations in a traditional Chinese painting. Clüver’s stress on viewer’s interpretive encounter with visual art (2019) and Heidrun Führer’s emphasis on the dynamic relationship the recipient is cooperatively engaged in as spectator in an ekphrasis (2014) and Zhao Xianzhang’s 赵宪章 theory of wordimage mutual emulation (2021) can shed some light on our understanding of the composite art of Chinese painting. The term “the three perfections” (san-jue 三绝) was coined during the eighth century in praise of the Tang poet-painter Zheng Qian (郑虔) (d. 764), who excelled in poetry, calligraphy, and painting. The term is similar to authors traditionally called “Doppelbegabungen” by German critics, that is, authors who combine and to a certain degree master the literary as well as the pictorial medium. The term of “the three perfections,” however, simply denotes an artist’s ability in the three art forms, but does not clarify the relationships between them and the term overlooks the importance of seal in the Chinese painting. As is illustrated above, classic Chinese painting has been rarely a pure painting; poetry, calligraphy, painting, and seal, or “the four perfections” (si-jue 四绝), have been practiced together in single works of art since the eleventh century. In a work of four perfections, the ekphrastic poem/inscription helps to provide the background information of the painting, to highlight what is depicted, to reveal the intention of the painter, to make a comment on the painting, and/or to extend the poet’s emotion inspired or epitomized by the painting. The ekphrastic poem survives on the scroll with the effort of the calligrapher, and the seals help to identify the painter and the calligrapher and balance the painting in terms of color and structure. Equal accomplishment in four perfections is highly demanding and beyond the capacity of most literati. One of the rare geniuses who was gifted at poetry, calligraphy, painting, and seal engraving was Tang Yin 唐寅 (1470–1524), an artist in mid-Ming Dynasty. He was a prodigy and ambitious for a successful political career at an early age, but the notorious examination scandal of 1499 ruined his prospects for an official career as he was accused of cheating in the Metropolitan Examination. Disillusioned with politics, Tang Yin travelled a lot, devoted himself to the art and made a living as an artist, excelling at painting of figures, flowers, birds, and landscape. Meanwhile he cultivated himself as a cynical and haughty figure, and immersed himself in drinking and sensual comforts. He once reflected on his life as “fifty years of dancing and singing, sleeping in midst of beauties and drinking,” thus coloring his work with bitterness and playfulness.

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

503

Lady with Fan in Autumn Breeze (秋风纨扇图, Fig. 2) is Tang Yin’s masterpiece of figure painting in which a graceful lady stands to one side in a court, holding a fan in her hands; there is melancholy between her eyebrows to the eyes of Chinese viewers who understand the inscription or who are familiar with the story of Lady Ban. The long tassels of the fan flow down and flutter away with the edge of the lady’s gown in the autumn wind. The big fan with long tassels looks very heavy against the emaciated frame of the lady, highlighting her pathetic delicacy. Rocks and bamboos in contrast white and black beside her heighten the ambiance of gloom and loneliness. The whole ink painting is not monotonous owing to Tang’s excellent variation of large and thin lines as well as heavy and light strokes. Red seals in different parts, with the painter’s different stylistic signatures, not only add a few bright touches to the black-white ink painting, but also make the structure of the painting well balanced – the two tail seals, one smaller name seal (名章) functioning as the confirmation of the painter’s signature read up-downright-left (唐寅私印) and the other bigger leisurely seal (闲章), end the inscription and keep two lines of words balanced and symmetrical with similar length; the head seal is used to counterbalance the second heavy tail seal; the middle seal, alongside the head seal vertically and alongside the head of the lady horizontally, with a red image between black words and black hair of the lady; two red seals on the lower right counterbalance the dark of the rocks. (Leisurely seal, a seal as an object of artistic value, not used for practical purposes, usually inscribed with a motto or a line of poetry.) The seals on the lower left are left by the successive collectors/connoisseurs of the painting. On the left top of the painting is the painter’s self-inscription, a seven-character quatrain: 秋来纨扇合收藏, 何事佳人重感伤, 请把世情详细看, 大都谁不逐炎凉。

Silk fans are to be put aside in Autumn, For what the lady is so grieving? On the human world meditate, Who can escape the fickleness of the fate?

Daan Pan, in his perceptive study of tihuashi (2011), argues that As an interartistic genre, tihuashi relates to almost all genres of painting, and individual poems from this genre can be classified in accordance with the genres of the source paintings or their own thematic concerns – such as the representation (or re-creation) of painterly scenes (or things associated with source paintings), the appraisal of painterly artistry or personality, or the expression of poetic sentiment inspired by paintings. (Pan 2011: xi)

In fact, the interrelation of tihuashi and the painting is more complex than Pan’s summary. He overlooks the difference between the painter’s self-inscription and others’ inscription. In Tang’s case, the poem does not represent the painterly scene as the poem and painting are in the same scroll, nor does it appraise the painterly artistry or personality, as it is a self-inscription; nor is it the expression of poetic sentiment inspired by the painting; rather, it parallels to the painting and voices the painter/ poet’s sympathy for and identification with the lady in the painting.

504

R. Ou

Fig. 2 Tang Yin, Lady Ban Holding a Fan, Paper, c. 1499, Hanging scroll, Shanghai Museum, Shanghai

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

505

The calligraphy of the inscription is not taken at random, not in the style of the Regular Script (楷书, also called Standard Script) which is too formal and conventional, nor the Cursive Script (草书) which is too wild and individualistic, but in the style of Running Script (行书, also called Semicursive Script) which is delicate and graceful, in agreement with the gentleness of ladyship represented in the painting. There is an allusion in the painting and the poem, referring to Lady Ban. Lady Ban (班婕妤 48–2 BC) was one of the concubines of Cheng Emporor of Han Dynasty, a very talented woman. She is known from the illustrations of Gu Kaizhi 顾恺之 (d. ca. 406 A.D), now in the British Museum, of Admonitions to the Palace Ladies (女史箴图). In Panel Two, she was depicted as a role model for court ladies, refusing the emperor’s invitation to ride with him in his palanquin for fear of distracting him from state affairs. After she lost favor of the emperor, she composed a five-character poem “Song of a Round Silk Fan”(also known as “Song of Regret”). Likening her situation to that of the fan, she signs with a forlorn plaint: so appreciated in the warmth of summer, only to be cast aside in the chill of the autumn, as follows. 新裂齐纨素, O fair white silk, fresh from the weaver’s loom, 鲜洁如霜雪。 Clear as the frost, bright as the winter snow— 裁为合欢扇, See! Courtship fashions out of thee a fan, 团团似明月。 Round as the round moon shines in heaven above, 出入君怀袖, At home, abroad, a close companion thou, 动摇微风发。 Stirring at every move the grateful gale. 常恐秋节至, And yet I fear, ah me! That autumn chills, 凉飚夺炎热。 Cooling the dying summer’s torrid rage, 弃捐箧笥中, Will see thee laid neglected on the shelf, 恩情中道绝。 All thought of bygone days, like them bygone. (Translation of Herbert Giles 1884: 101)

Lady Ban’s “Song of Regret” initiated the poetic composition of “Palace Lament” (宫怨诗), and since then, the phrase of “the autumn fan” has often alluded to a lovelorn beauty. In Lady Ban’s poem, the poetess compares herself to the fan; in Tang Yin’s poem, the poet compares himself to the abandoned lady and consoles himself to see through the vicissitudes of earthly life. As the central figure, the lady in the painting performs her life story in front of us; the painter/poet participates in the performance through the inscription by playing himself as the lady; the message embodied in the inscription is highlighted by the painter’s three leisurely seals: the head seal “far from the madding crowd” (游方之外), the middle seal “spot of epiphany” (会心处), combined with the tail seal “No. 1 on the Dragon/Tiger lists, 1000 times getting drunk amidst the company of girls 龙虎榜中名第一, 烟花队里 醉千场)” to suggest the painter’s Taoist and cynical attitude towards “the fickleness of the fate,” thus constituting a dynamic interaction and integration of painting, poetry, calligraphy, and seals. (“The Dragon/Tiger lists 龙虎榜, ” alludes to the fierce imperial examinations. In 1498, Tang placed first in the Provincial Examination in Nanjing and then travelled to Beijing to take part in the Metropolitan Examination to be held the next year, which brought about a twist to the course of his life.)

506

R. Ou

Fig. 3 Tang Yin, Gift of Tao Gu, Silk, Hanging Scroll, Palace Musuem, Taibei

Another illustration of Tang Yin’s four perfection is his Gift of Tao Gu (陶榖赠词 图, Fig. 3) which alludes to an anecdote of Song Dynasty. In the early Northern Song Dynasty, Tao Gu (903–970) went to the Southern Tang, a small kingdom as an envoy. Tao Gu was very arrogant and rude to the king of the Southern Tang. Han Xi-zai, a resourceful official of the Southern Tang, made a beautiful maidservant to

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

507

seduce Tao. Tao surrendered himself to the beautiful girl for a one-night stand and flattered her with a poem entitled “Having a Wild Time” (风光好). Several days later, the king of the Southern Tang invited Tao to a banquet and Tao posed his arrogance and pretentiousness again. The king asked the maid to come out and sing a song for them. What she sang was the poem composed by Tao, who was very embarrassed and became a laughing stock. In Tang Yin’s painting, Tao Gu sits on a couch, and there is paper, brush with black ink in a red ink slab aside, a candlestick with a red candle between Tao and the maid. The graceful maid is playing Pipa 琵琶, a traditional Chinese musical instrument. Tao is beating time to the music with his hands and feet, rather pleased and relaxed. The trees, the rocks, the screen as the background are well-structured and depicted with fine strokes. On the left bottom of the painting, there is a stove half hiding a boy servant crouching on the ground and eavesdropping, along the diagonal line of the upper right bottom is Tang Yin’s inscription poem in running script: 一宿姻缘逆旅中, 短词聊以识泥鸿。 当时我作陶承旨, 何必尊前面发红。

One-night encounter at a courier station, Short lyrics composed as a gift of the love. If I were the envoy Tao I wouldn’t be blushed upon the banquet.

The poem not only identifies the characters in the painting, relating the gist of the anecdote, facilitating viewers to associate the painting with the story, but also involving the painter-poet and the viewers into the situation in a playful tone. The red seals correspond to the red ink slab and the red candle, counterbalancing the dominant heavy color of black and brown.

Dynamic Interaction and Fruitful Rivalry of Sister Arts Qi Gong, in his paper, “Relationships between Poetry, Calligraphy, and Painting,” argues that only those works that combine the artist’s own poetry, calligraphy, and painting attain the highest form of artistic expression (Qi 1991: 18–19). It is not always the case. A work of art with different operators can tell us more intricate and dynamic interaction between sister arts and contribute more to the interart poetics and intermedial studies. In Chinese culture, the plum blossom, the orchid, the bamboo, and the chrysanthemum are known as the Four Figures of Moral Integrity (四君子). The plum blossom relates to the strength in adversity, the orchid to the noble purity, the bamboo to the uprightness, and the chrysanthemum to faithfulness. Chinese artists tend to draw on the Four Figures to inspire themselves or project their ideals onto these plants. In Orchids and Bamboo (兰竹图, Fig. 4) by Wen Zhengming (1470–1559), the painter presents us a thicket of orchids mixed with a branch of bamboo growing out of rocks, which indicates the poet’s perseverance of moral purity and uprightness in an adverse environment. On the right upper hand is the painter’s self-inscription in Running Script:

508

R. Ou

Fig. 4 Wen Zhengming, Orchids and Bamboo (1543), Hanging Scroll, Paper, Palace Musuem, Taibei

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

风裾月佩紫霞绅, 秀质亭亭似玉人; 要使春风常在目, 自和残墨与传神。

509

A fairy god immersed in wind and moonlight, A fairy lady in delicacy and grace; To keep the spring time always in sight, Is to draw the essence of the view with the residual ink.

The inscription reveals the poet’s identification with the noble characters of the bamboo and the orchid and his belief in the immortality of the art. After the signature of Zhengming are the artist’s name seal (Seal of Wen Zhengming 文徵明印) and his residence seal (Seal of Symposium Hall 悟言室印). In the middle is an inscription in Regular Script by Cai Yu 蔡羽 (? ~ 1541), Wen Zhengming’s contemporary artist and scholar: 兰幽在深谷, 竹操雅相宜, 对此良无恨, 清氛拂研池。

The orchid is in the secluded dale, And matches with the upright bamboo, One feels carefree at the view, A refreshing breeze blows the ink slab.

The first two lines of the poem depicts and comments on the prominent images of the orchid and the bamboo (“the upright bamboo” is a pun, both a visual depiction and a moral judgment), with imagination of the setting – “ in the secluded dale” that is not represented in the painting. The last two lines extend the poet’s emotion inspired at the sight of the painting: the third line indicates the viewer’s sentimental response by finding solace from the peaceful and graceful images, and the last line is resonant with the last line of the painter’s own poem. After the signature of the poetcalligrapher is the seal of his stylistic name:a Toist (左虚子). Wen could have produced the painting for Cai, and Cai responds with the poem above, as the orchid and the bamboo allude to their friendship and their common pursuit of moral integrity and Taoist way of life. On the right upper hand is the inscription in Running Script of the Emperor Qianlong (1711–1799) of Qing Dynasty written about two centuries later. 洒洒芳蕤个个叶, 都於尘外挺仙姿; 闲中设复相求友, 谁不云然契性宜。

Voluminous blossoms and sparse leaves All stand up beyond the earthly concerns. Seeking friends in my leisure who can agree with me in pleasure?

The poem ends with two red seals of the emperor’s. The first one goes: A Man of Leisure (几暇怡情) and the second one goes: A Man of Taste (得佳趣). The poem reveals the emperor’s envy of the noble plants far from the madding crowd and his loneliness and desolation of no matching company to share his leisure owing to his high position. With regard to the spatial design of the painting, the painter-poet’s own inscription takes up the largest space and ends with two big seals. Cai’s inscription ends with a smaller seal, to keep the length of his lines balanced. Though distinguished as an emperor, Emperor Qianlong cannot trespass the rules of aesthetics and ends his

510

R. Ou

poem with two smallest seals. By comparison, different poems reveal different responses of the viewers to the same images, and they also constitute dynamic interactions and dialogues among different artists across the time and space. Another fascinating case of the dynamic interaction between painting and poetry is Su Shi’s a cycle of poems for one painting, involving the painting, the painter, and the collector. The cycle of poems is entitled “Shu caopuzhi zhi suocang yuke huazhu sanshou” (书晁补之之所藏与可画竹三首 “Three Poems of Yuke’s Bamboo Painting in Cao Puzhi’s Collection,” 1087). Yuke 与可 refers to Wen Tong 文同 (1018–79), a prominent bamboo painter and Su’s cousin. Cao Puzhi 晁补之 (1053–1110), a great writer and Su’s good friend. The first poem praises the painter’s artistry and exemplifies Su’s aesthetics of painting: 与可画竹时, 见竹不见人。 岂独不见人, 嗒然遗其身。 其身与竹化, 无穷出清新。 庄周世无有, 谁知此凝神。

When Yuke was painting bamboo, He saw bamboo without seeing himself. Not only did he see none of himself, But trance-like, he forgot his own body. His body changed with the bamboo; Inexhaustible freshness emerging. Zhuang Zhou is no longer in this world; So who knows such absorption! (Pan 2011: 248)

According to Pan, this poem “portrays a true artist at creation in his best spiritual form, in which he becomes oblivious of his self and enters a mutual metamorphosis with the painterly subject” (Pan 2011: 249). The second poem laments on the death of the painter: at the sight of the painting, the poet misses the talented artist all the more. The third poem praises the nobility of Cao’s personality: poor as he is, he would rather starve to keep the painting. The painter is gone, but the painting remains, with related poems to convey the artistic ideals embodied in the picture and tell intricate stories beyond the picture.

Artistic Creation as Social and Cultural Interaction In Chinese traditional culture, the poetic and artistic creation, besides a means of making a living for professional artists or the royal service for court painters, is an important means of social and cultural interaction for men of letters. For traditional Chinese literati elite, they do literary and artistic creation not for profit, but as gifts to their friends or colleagues, as tokens of friendship; therefore, there is more affinity and resonance rather than “power struggle” or “gendered opposition” between word and image, best illustrated in a cycle of works inspired by Wang Shen’s landscape painting and the ekphrastic network deriving from the historical Western Garden Gathering. To understand the landscape painting of the Northern Song Dynasty, the best way is to read a brilliant collection of short essays by Guo Xi (郭熙 c. 1020–c. 1090), introduced to the West by Lawrence Binyon in Painting in the Far East as “one of

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

511

the greatest of all Chinese landscape painters” (Binyon 1908: 128). In his seminal critical text The Noble Features of Forest and Streams (林泉高致), written around 1080, the celebrated painter illuminates in lucid and graceful diction the purposes and techniques of painting. Guo “insists on [the ‘far-off effect’] as necessary to unit” and “The painter must have varied experience, must build on incessant observation. . .. but above all things he must seize essentials and discard the trivial.” Due to his influence, according to Binyon, the “Sung landscape is built up of tones rather than of lines . . .. The artists worked almost entirely in monochrome; and they chose for subject all that is most elemental and august in nature”; Guo expresses once and for all the guiding sentiment of Chinese landscape painting, the aim of which, according to Guo, is to enable those who wish to “enjoy a life amidst the luxuries of nature” but “are debarred from indulging in such pleasure” to “behold the grandeur of nature without stepping out of their houses,” for the landscapes in the masters’ paintings allow viewers to travel, to dream of gazing, dwell and wander, suggested the celebrated painter (Binyon 1908: 128–9). Guo’s treaty is a valuable aid to the artists’ practice and viewers’ understanding of the Chinese landscape painting. Du Fu (杜甫, 712–770), the greatest Tang poet, in poetic diction, foretells the similar function of the landscape painting. His “Ten Rhymes Presented on Viewing the Painting Mount Min and River Tuo in Master Yan’s Courtroom” (奉观严郑公厅 岷山沲江画图十韵) celebrates the lofty landscape mural that allows the viewers “enjoying the grandeur of nature without stepping out of their houses.” 沱水临中座, 岷山赴北堂。 白波吹粉壁, 青嶂插雕梁。

River Tuo flowing to the central seat, Mount Min reaching to the northern hall. White waves crashing on the plastered wall, Green peaks sticking in the carved beams.

Misty River and Layered Peaks (烟江叠嶂图) by Wang Shen (王诜, ca. 1048after 1104) now collected in Shanghai Museum illustrates another fascinating intermedial play of poetry, calligraphy, and painting. Wang Shen was a calligrapher, painter, poet, and politician of the Northern Song Dynasty. He is best known for his surviving paintings, poetry, and calligraphy, and for his associations with prominent statesmen and early amateur literati artists such as Su Shi, Huang Tingjian (黄庭坚, 1045–1105), and Mi Fu (米芾, 1051–1101). At first Wang Shen produced a blue and green landscape painting of Misty River and Layered Peaks (Fig. 5) for Wang Gong (王巩 ca. 1048 to ca. 1117, courtesy name Dingguo). The hand scroll opens with a panorama of an ample stretch of water. Across the river, the layered peaks open into a vista that curves to the left, as if drifting between sky and water. There is misty cloud among the peaks, tall and straight trees stand on the mountains, waterfalls flow down the rocks, and a pavilion is half visible in the forest. The artist’s pictorial vision presents the scene as peaceful and secluded. The imagery is concrete, yet in subtle ways it evokes the transcendent realm (Fig. 6). After appreciating the work at Wang Gong’s house, Su Shi composed the renowned tihuashi “Written on Misty River and Layered Peaks in the Collection of

512

R. Ou

Fig. 5 Wang Shen, Misty River and Layered Hills (1088), hand scroll, silk, Shanghai Museum, Shanghai

Fig. 6 Details of Misty River and Layered Peaks by Wang Shen (1088), hand scroll, silk, Shanghai Museum, Shanghai

Wang Dingguo” (书王定国所藏烟江叠嶂图) as his emotions were aroused at the sight of the tranquil and harmonious scenery. His poem consists of a verbal representation of the misty river and layered hills and his Toist ideals inspired by the landscape. The poem starts with 12 lines of the presentation of natural scenery, like a depiction of real mountains and water: 江上愁心千叠山, 浮空积翠如云烟。 山耶云耶远莫知, 烟空云散山依然。 但见两崖苍苍暗绝谷, 中有百道飞来泉。 萦林络石隐复见, 下赴谷口为奔川。 川平山开林麓断, 小桥野店依山前。 行人稍度乔木外, 渔舟一叶江吞天。(Figs. 7 and 8) On the river, a heart of sorrow, a thousand tiered hills; A pile of floating verdure, the hills look like clouds and mist. Are they hills or clouds? Too far away to tell; Once the clouds and mist disperse, the hills remain. I only see the dark green twin cliffs shading an abyssal ravine, Wherein a hundred springs fly from afar. Winding through woods and rocks, they disappear and reappear; Falling into the ravine mouth, they turn into a rushing river.

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

513

Fig. 7 Details of Su Shi’s Inscription on the Painting (1088). Shanghai Museum, Shanghai

Fig. 8 Details of Su Shi’s Inscription on the Painting (1088). Shanghai Museum, Shanghai The river levels out where the hills open up and woody foothills end; A small bridge and a rustic inn sit against the hills. Travelers walk not far behind the tall trees; A fishing skiff drifts where the river engulfs the sky. (Pan 2011: 240)

These lines by Su Shi expresses elation at the merging of objects and emotion in a painting, the unconscious feeling that the viewer is actually in a landscape. The painting gives the poet a sense that he is losing himself among mountains and woods, as though he is facing a real landscape. The painting is a paramount example of Want’s artistic accomplishment, which balances pictorial description with

514

R. Ou

expressive brushstroke to provide, as Guo wrote, landscapes in which one may physically and mentally ramble. Su’s lucid phrases and subtle languages are a measure of the beauty of landscape; they are virtually the painted scenery in Wang’s Misty River and Layered Peaks, thus reaching the attainment of “painting in poetry and poetry in painting.” Line 13–14 exposes the ekphrastic nature of the poem – “the verbal representation of visual representation” as James Heffernan defines “ekphrasis” (Heffernan 1993: 3): the previous poetic lines are not presentation on the real nature but re-presentation of a painting, a visual representation. 使君何从得此本, Where did you get this painting, Sir? 点缀毫末分清妍。 Its minute details look fresh and charming. Then, the poet imagines his living a pastoral life represented in the painting and associates it with his past 5 years of exile life. Su ends his poem with repeated signs, reluctant to return the painting to its owner, and indicating his longing for a retreat into the country life: 还君此画三叹息, Returning the painting to you, Sir, I sigh thrice; 山中故人应有招我归来篇。 My friends from the hills should send me poems, calling for my return. (Fig. 7) Learning about Su’s love of the painting and Su’s poetic response to his work, Wang Shen also composed a poem as an echo to Su’s poem, recalling his own exile life and purpose of his artistic creation as a means of escape and retreat from the political struggle. He ends his poem with a promise of another misty painting for Su and expectation for more of Su’s poem. Wang Shen did produce an inking painting of Misty River and Layered Peaks for Su Shi which is quite different from the first one in terms of the color, the structure, and the mood (Fig. 6). Su in turn responded to Wang’s ink painting and poem with a second poem to comfort Wang and compliment him on his artistic attainments to which Wang replied with another. There are four poems altogether on the scroll of the ink painting of Misty River and Layered Peaks (Liu 2010: 166, 172). As the painting is executed in the form of hand scroll, and the regular format of mounting and framing the hand scroll leave infinite space for colophons, as indicated in Fig. 9, the poems can be inscribed one after another. In this way, the two artists converged in a historic masterpiece of poetry, calligraphy, and painting.

Fig. 9 Framing of hand scroll

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

515

Since then, “Misty River and Layered Peaks” have become an appealing motif for later poets and artists to make countless reproductions. In a large running-script calligraphic hand scroll by Zhao Mengfu 赵孟頫 (1254–1322) of Yuan Dynasty, now collected in the Liaoning Provincial Museum, Zhao wrote out the Su’s poem (Fig. 10). Zhao’s bold transcription of Su’s immensely popular poem in turn inspired the fifteenth-century Ming artists Shen Zhou 沈周 (1427–1509) and Wen Zhengming to create the landscapes in painting that are now mounted together with Zhao’s calligraphy in the same scroll (Figs. 11 and 12).

Fig. 10 Zhao Mengfu, Su’s Misty Poem, detail of hand scroll, ink on paper, Liaoning Provincial Museum, Shenyang, undated

Fig. 11 Shen Zhou, Landscape Inspired by Su’s Misty Poem and Zhao’s Calligraphy, dated 1507

Fig. 12 Wen Zhengming, Landscape Inspired by Su’s Misty Poem and Zhao’s Calligraphy, dated 1508

516

R. Ou

This sequence of artworks not only reveals a particularly intricate set of interrelationships between poetry, calligraphy, and painting, but also documents a centuries-long discourse among poets, painters, calligraphers and the viewers, revealing a dynamic relationship of “a community of artists,” illustrating what Barbetti claims with the examples from the ekphrastic medieval vision that “Ekphrasis is a verb, not a noun.” (Barbetti 2011: 9). Chinese art practice defines ekphrasis as a dynamic process of recreation and interarts exchange, instead of “a still moment of the poetry” as Murray Krieger argues for (Krieger 1992: 267), or rather the wordimage exchange “is reciprocal, as the text in turn finds new reality through visualization by the painter” as David Rosand proposes in the Renaissance art: “image begetting image”(Rosand 1990: 61). The more complex case lies in the collective execution of artworks when artistic creation used to be a lofty and noble means of social interaction for Chinese literati gathering. On such occasions, each of the poets writes a poem for the same painting, or each of the painters produces a painting for the same poem, or a group of poets and painters work together on one poem or a painting. The most renowned event like this is Gathering in the Western Garden (西园雅集) in the Northern Song Dynasty. The original figure painting Picture of the Elegant Gathering in the Western Garden (西园雅集图, Fig. 13), attributed to Li Gonglin (李公麟, 1040–1106), depicts the congregation of 16 literati—including Su Shi and his younger brother, Su Zhe 苏辙 (1039–1112), Huang Tingjian 黄庭坚, and Mi Fu 米芾 at the official residence of Wang Shen. During the gathering, there were endless art activities such as painting and calligraphy creation, composing poems, reading, playing music, and singing. This historic occasion epitomizes the intimate conviviality between poets and painters. The painting was followed up with Mi Fu’s essay “An Account of Picture of the Elegant Gathering in the Western Garden” (西园雅集图记) in which Mi celebrated the spiritual significance of the gathering, claiming, “The pleasure of pure aloofness was no greater than this occasion.” According to Pan, “Mi’s essay itself can be read as a piece of tihuashi by virtue of its poetic language. Gathering in the Western Garden, along with Li’s pictorial rendition of it, made garden gathering a regular feature of literati pastime, which provided an ideal venue for interartistic activities” (Pan 2011: 108). The picture and the text join forces to promote the event to the position of a model for later generations. It is deeply loved and respected by literati and scholars, and it is also the object of literati competition. This was especially the case during the late Ming and the Qing dynasties, when private scholarly gardens became an architectural fad, partly spurred by a booming urban commercial culture in areas south of the Yangtze River. This classical model of scholarly gathering not only became recurrent subjects for poetry and painting but also was popularly reenacted among the literati. As a result, a large number of excellent works on the theme of the artistic Gathering have emerged in the following dynasties, including paintings, calligraphy, utensils, etc., constituting an ekphrastic network originating from that prominent Western Garden.

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

517

Fig. 13 Li Gonglin, Picture of the Elegant Gathering in the Western Garden, Paper, Palace Museum, Taibei, dated 1087

Conclusion Although it is common knowledge that Chinese poetry, calligraphy, painting, and seal known as the four perfections have been practiced together in single works of art, the precise relationship between them has yet to be fully explored. In terms of intermediality, a traditional Chinese painting can be a transmedial text, a multimedia text, a mixed-media text and an intermedial text all by itself, regarded from different angles and it can involve media transposition, media combination, intermedial references all at once. The broad interarts approaches in the chapter suggest further directions for the study of Chinese art. In China, “where the conceptual unity between poetry, calligraphy, painting and seal derived from a holistic view of culture, the tension between word and image characteristic of the West exist less prominent” (Murck and Fong 1991: xxii). The author should add that the conceptual unity between poetry, calligraphy, painting, and seal derives not only from a holistic

518

R. Ou

view of culture, but from a holistic view of nature, which is regarded as the primal source of artistic creation by traditional Chinese men of letters. With a holistic view of nature and culture in mind, they aspire to embody the rhythmic vitality in their creation, the highest transmedial quality according to classic Chinese aesthetics. Chinese artworks, as products of different value and thought systems, invite westerners to think about them in new ways. The richness and complexity of the four perfections in traditional Chinese painting will enrich the contemporary interart/ intermedial studies. As acutely pointed out by Clüver, his schema of intermedial relations is “inevitably static and treats the texts as objects” (Clüver 2007: 26) and Jürgen E. Müller has suggested that we should “understand the indissoluble connection of diverse media as a fusion and interaction of different medial processes” – without suggesting, however, where such processes take place (Qtd. in Clüver 2007: 27). The appreciation of the traditional Chinese painting, which is a subtler form of intermedial practice, can help to manifest the fusion and interaction of different medial processes.

References Barbetti, Claire. 2011. Ekphrastic medieval visions: A new discussion in interarts theory. New York: Palgrave Macmillan. Binyon, Lawrence. 1908. Painting in the Far East: An introduction to the history of pictorial art in Asia especially China and Japan. London: Arnold. Clüver, Claus. 1993. Interartiella studier: en inledning, trans. Stefan Sandelin. In I musernas tjänst. Studier i konstarternas interrelationer, ed. Ulla- Britta Lagerroth, el. 17–47. Stockholm: Stehag. ———. 2007, 1993. Intermediality and interarts studies. In Changing borders: Contemporary positions in intermediality, ed. Jens Arvidson, Mikael Askander, Jørgen Bruhn and Heidrun Führer, 19–37. Lund: Intermedia Studies Press. ———. 2019, 1993. On gazers’ encounters with visual art: Ekphrasis, readers, ‘iconotexts’. In Ekphrastic encounters: New interdisciplinary essays on literature and the visual arts, ed. Richard Meek and David Kennedy, 238–257. Manchester: Manchester University Press. Führer, Heidrun, and Bernadette Banaszkiewicz. 2014. The trajectory of ancient Ekphrasis. In On description, ed. Alice Jedlickova, 45–75. Prague: Akropolis. Giles, Herbert A. 1884. Gem of Chinese literature. Shanghai: Kelly & Walsh. Heffernan, James. 1993. Museum of words: The poetics of Ekphrasis from Homer to Ashbery. Chicago/London: The University of Chicago Press. Hoek, Leo. 1995. La trasnposition interseémiotique. Pour une classification pragmatique. In Rhétorique et image. Textes en hommage à Kibédi Varga, ed. Leo Hoek and Kees Meerhoff, 65–80. Amsterdam: Rodopi. Jenyns, Soame. 1966. A background to Chinese painting. New York: Schocken Books. Krieger, Murry. 1992. Ekphrasis: The illusion of the natural sign. Baltimore/London: The Johns Hopkins University Press. 李旭婷 (Li, Xuting). 2021. 语图合体对题画诗发展的影响 (Impact of word-image combination on the development of Tihuashi). 文艺评论 05: 118–128. 刘继才 (Liu, Jicai). 2010.中国题画诗发展史 (History of Chinese Tihuashi), 沈阳:辽宁人民出版 社。 . Lund, Hans. 1992. Text as picture: Studies in the literary transformation of pictures. Trans. Kacke Gotrick. Lewiston: Edwin Mellen. ———. 2002, 1992. Medier i samspel. In Intermedialitet. Ord, bild och ton i samspel, ed. Hans Lund, 7–23. Lund. Studentlitteratur

21

Traditional Chinese Painting: An Intermedial Play of Sister Arts Since. . .

519

Mitchell, W. J. T. 1986. Iconology: Image, text, ideology. Chicago: University of Chicago Press. Murck, Alfreda, and Wen C. Fong, eds. 1991. Words and images: Chinese poetry, calligraphy, and painting. Princeton: Princeton University Press. Pan, Daan. 2011. The lyrical resonance between Chinese poets and painters: The tradition and poetics of Tihuashi. Amherst: Cambria Press. Qi, Gong. 1991. The relationship between poetry, calligraphy, and painting. In Words and images: Chinese poetry, calligraphy, and painting, ed. Alfreda Murck and Wen C. Fong, 11–20. Princeton: Princeton University Press. Rajewsky, Irina. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités 6 (automne): 43–64. Rosand, David. 1990. Ekphrasis and the generation of images. Arion 1: 61–105. 沈树华(Shen, Shuhua), ed. 1993. 中国画提款艺术(Art of Inscriptions of Chinese Painting). 北京: 人民美术出版社, 1993. 苏轼 (Su, Shi). 2016. “书摩诘蓝田烟雨图” (Notes to Wang Wei’s Painting Mist and Rain over Langtian),” in东坡题跋 (Inscriptions and Postscripts by Su Shi). 杭州:浙江人民美术出版社, 166. Vos, Eric. 1997. The eternal network: Mail art, intermedia semiotics, interarts studies. In Interart poetics. Essays on the interrelations of the arts and media, ed. Ulla-Britta Lagerroth, Hans Lund, and Erik Hedling, 325–337. Atlanta: Rodopi. Xie, Zhiliu. 1991. Reflections on the poetic quality and artistic origins of Ch’u Ting’s summer mountains. In Words and images: Chinese poetry, calligraphy, and painting, ed. Alfreda Murck and Wen C. Fong, 3–10. Princeton: Princeton University Press. Zhao, Xianzhang. 2021. Text – Image theory: Comparative semiotic studies on Chinese traditional literature and arts. Genzano di Roma: Aracne.

The Anchor and the Dolphin: A History of Emblems

22

Heidrun Fu¨hrer, Cecilia Victoria Muszta, and Viktor Ferdinand Kova´cs

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alciato: The Inventor of Emblems? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Speaking to the Reader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ekphrasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmediation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comparing Word and Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Similarities and Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alciato as a Renaissance Humanist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Authorized “Fathers” of Their Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Translation: Preserving the Truth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An Early Media Discourse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Emblems and Imagination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alciato’s Book Title Emblema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Editio Princeps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Printed Emblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Title: The Anchor and the Dolphin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Role of Print Shops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Applied Emblems, Art, and Iconic Emblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

522 524 526 527 527 528 529 531 531 533 534 535 537 538 539 540 541 542 543 546

Abstract

This chapter explores the history of emblems, which appear on a broad spectrum between printed literature, visual art, and material culture. Starting from Andrea Alciato’s Emblematum liber of 1531, the chapter explores multiple approaches on how emblems can be understood in the context of other multimodal media types such as insignia, badges, epigrams, hieroglyphs, and visual artifacts. Differentiating between historical and modern semiotic models, it explains the emergence H. Führer (*) · C. V. Muszta · V. F. Kovács Lund University, Lund, Sweden e-mail: [email protected]; [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_25

521

522

H. Fu¨hrer et al.

of the notion of “emblem” in relation to tropes such as symbol, ekphrasis, or allegory. It also deconstructs the common narrative of emblems being characterized by a standardized tripartite pattern consisting of a picture, title, and motto, and their historical confinement to the epochs of the Renaissance and Baroque, specifically between the years 1531 and 1750. By analyzing early modern and modern examples, the chapter demonstrates the historical, technological, and sociocultural influences directing the fluid expectations of this genre. Against the various normative attempts to define the emblem, the chapter strives to contribute to a more open-ended approach to emblems, enabling the identification of them even in modern cultural products. Keywords

Emblem · Epigram · Ekphrasis · Allegory · Symbol · Transmediation

Introduction For some, emblems might seem a remote media product. Others, when searching for emblems on the internet, will find multiple visual symbols, signs, or logos. But do these belong to the same media type? In this chapter, emblems are explored as a historical and modern multimodal media product. Thereby, also the early modern and modern emblem discourse will critically be touched upon with its theoretical preconditions. Most of these binaries concern the basic media word and image (Lund 2002) that are often regarded as fundamental for “the emblem.” In general, emblems are defined by their tripartite structure. This chapter emphasizes the various material and technical grounds for displaying emblematic media products, the historicity of sensorial perception and the process of name-giving, and the transmedial heritage of the historical notion of emblems and their cognate media products. All these elements influence the semiosis process of the perceiver (Elleström 2021; Jakobson 2002). Since the nineteenth century, the modern history of the emblem has been intertwined with studies of Renaissance literature (Green 1870). Often, emblem books were denounced as “curiosities, oddities and aberrations” (Praz 1964: 8) and as a “degenerated form of allegory” (Fabiny 2005: 192). Their hybridity, meaning their intermedial composition of word and image, had been criticized as much as their “outmoded” rhetoric. However, early emblem researchers strived to reestablish the value of emblem books. Fascinated by historical collections of illustrated books, some Renaissance collectors produced bibliographies to establish emblem books as a hitherto neglected symbolic media type (an overview in Dees 1986). Multiple bibliographies established a discourse on emblems by collecting books with the word “emblem” in their title. Do emblems have no more in common than the “basic pattern of figure plus verse” (Saunders 1989: 117)? The upcoming scholarly discourse operationally qualified these printed editions. The amazing value of the Emblematum liber, Book of Emblems, of Andrea Alciato (1492–1550) is not

22

The Anchor and the Dolphin: A History of Emblems

523

only due to the more than 170 editions of this book in multiple languages and countries (Enenkel and Smith Paul 2017: 2). Moreover, emblems became a field for research since the many references to emblems in canonized literary collections of epigrams and baroque dramas qualified epigrams as intertextual and intermedial source media (Freeman 1948; Schöne 1964; Daly 1979; Peil 1986, 1992; Höltgen 1986, 1996). Up to the 1970s, epigram research was grounded on theoretical reflections on the “original” of Alciato’s Emblematum liber, 1531. Based on his book, the canonical epigram was defined as a three-part structure consisting of an inscriptio (motto), pictura (picture), and subscriptio (explanatory text in verses or prose). In general, two types of emblem books were outlined: the illustrated epigrams or the “tripartite emblem proper” (Enenkel 2019: 4, 55f), the “naked” word epigrams consisting of words without a materialized picture, or emblema nuda. Outside of the canonized disciplinary fields of literature and “high art,” a third group of emblems, self-contained iconographic media on multiple material grounds (Praz 1964 [1939]), had been established. Emblems in the “applied arts,” such as the coats of arms, badges, devices, etc., are notoriously hard to differentiate from objects consisting of words and images with a similar structure. These media products have clearly medieval roots and a pragmatic function. The same goes for religious emblem books and emblematic devices that were disseminated on the basis of an ideal collective religious identity in the name of propaganda dei, various material grounds, and in public performance (Harms 1973; Peil 1979; Diehl 1986: 50; Hämmerle 2019). One of these applied epigrams the printer’s mark be studied in section “Printed Emblems.” As emblem research strived to develop an emblem theory, the so-called protopoetological debates about emblems, allegories, hieroglyphs, rhetoric, and theatre of the sixteenth and seventeenth centuries served as a terminological ground for modern scholarship (Scholz 2002). However, early modern scholarship worked with polysemantic notions and another understanding of signs, unlike modern Peircean semiotics. The historical debate about the priority of language and the Christian worldview will be brought up in the debate about res et verba, body and soul, and as word and image. In general, modern epigrams’ researchers fall into two positions: those who prioritize the materialized image, pictura (Heckscher 1954; Panofsky 1955; Ong 1959; Schöne 1964; Henkel and Schöne 1967), and those who prioritize the power, complexity, and witty rhetoric of poetic words as directing the reading of either a materialized pictura or a memorized mental image (Freeman 1941; Vickers 1988, 2003; Manning 1990, 2002). Arguably, the area of emblems spans between 1531 and 1750, from Alciato’s Emblematum liber to the time when illustrated books with allegorical iconography became less popular (Burke 1997). Still, from the 1970s onward, emblem research deconstructed the early normative conception of emblems as starting with a fixed “birth date” of 1531 with Alciato’s writings. Emblems became abstract containers of – typically an early modern – meaning, often characterized as an “allegorical construction of word and image” (Heckscher and Wirth 1959: 85). The supposed death of emblems came with the allegedly new empirical sciences of the

524

H. Fu¨hrer et al.

Enlightenment and “new” aesthetic norms of the Romantic era. Thus, emblems were considered an art form that was typical for the zeitgeist of the sixteenth to eighteenth centuries, instead of media products that simultaneously provide dissonant meanings and formats (Greenblatt 1980; Jung 2014). Outside of the established definitions in the emblem books, this media type is also materialized as symbolic combination of word and image. This emblematic media type comprises both historical and contemporary material products such as playing cards, vases, tapestries, printer’s marks, advertising, tattoos, etc. just to name some examples (Russell 1995; Daly 1999, 2014; Graham 2001; Harper et al. 2005; Mödersheim 1994, 1999, 2010; Probes and Mödersheim 2014). The chapter is structured as follows. After this overview, a short poem from Alciato’s Book of Emblems introduces the core ideas about what has been considered as an emblem and how they can be read in the context of his time. The second section foregrounds the genesis of emblems from dialogical epigrams in the sense of “speaking image” and from ekphrases, allegories, and other rhetorical figures. These epigrams are often composed in three parts – a title or motto, a verbal image, and commonplaces. The third section situates Alciato in the rhetorical tradition of the Renaissance and emphasizes that the “inventor of emblems” had neither the power over the material production nor the reception of “epigrams.” In the fourth section, the historical discourse and the humanists’ search for knowledge will be intertwined with the newly developed print manufacture, contextualizing Alciato’s book within the book production of the Aldine Press to which the title of this chapter refers. The chapter concludes by raising questions about how to handle the notion of the emblem, which had been constructed and deconstructed numerous times in the ever-changing academic discourse.

Alciato: The Inventor of Emblems? Andrea Alciato (1492–1550) was the author of the Emblematum liber, published in 1531 (Alciato 1531). Andrea Alciato was a lawyer and a self-educated humanist scholar from Milan. He was successful in reviving the ancient Greek tradition by collecting, translating, and writing epigrams which he coined as emblems. He is known even today for these poems and celebrated as the “father” (pater et princeps) of a new literary genre, the emblem. Pitched from Alciato’s oeuvre, we consider the following epigram to introduce the derivative nature of epigrams from ancient inscriptions, Roman epigrams, and the popular contemporary intermedial sources of Alciato: Paupertatem summis ingeniis obesse ne provehantur Dextra tenet lapidem, manus altera sustinet alas: Ut me pluma levat, sic grave mergit onus. Ingenio poteram superas volitare per arces, Me nisi paupertas invida deprimeret.

22

The Anchor and the Dolphin: A History of Emblems

525

This epigram, written in Latin, is mostly read in a translation to English that strives to choose the “right” words that shall correspond to the “content” of the poem. The performative semiotic of language will be discussed later. We present here one of the various possibilities of translation: Poverty prevents the advancement of the best of the best talents. My right hand holds a rock, the other bears wings. As the feathers lift me, so the heavy weight drags me down. By my mental gifts I could have flown through the heights of heaven, if malign poverty did not hold me back. (Alciato, emb.121, transl. by Knott 1996)

The epigram presents a person speaking in the first-person perspective, bearing a rock in one hand and the other hand provided with feathered wings. Formally, the poem is introduced by a title or a motto, “Poverty prevents the advancement of the best of the best talents,” that matches with the two concluding verses of the poem, called sententia: “By my mental gifts I could have flown through the heights of heaven, if malign poverty did not hold me back.” Together with the proverbial truth, they suggest a moral interpretation of the verbal image. The speaking figure is identified as a talented person burdened by poverty. This tripartite composition consisting of a motto, an image, and a concluding sentence had been normatively regarded as the emblematic structure (Heckman 1969: 877). As an emblematic principle, it has been attributed to non-illustrated books of epigrams as well as to illustrated emblem books – the two main types of the literary genre of emblems. Semantically, the poem exemplifies the didactic allegorical mode of the highly appreciated media type of epigrams. They are linked with the rhetoric of allegories, or extended metaphors, and are either appreciated for their “enigmatic thoughts” (Fletcher 2010: 11) or repudiated as incoherent and arbitrary combinations of several objects. In any case, allegories are used to present abstract ideas through concrete characters and situations. They are found in literary texts and in graphic images that depict human figures often identified by specific attributes or by verbal phrases, called tituli. This practice is partly anchored in iconographic conventions and explanatory texts. Still, allegories command the activity of the perceivers to make their own interpretations. This openness and inherent instability of the allegorical epigram demands that the reader combine inconsistent elements, such as the stone and wings, of the verbal image with the abstract concepts of talent and poverty. Due to their study of classical rhetoric, involving a specific memory technique, ars memorandi, the erudite humanists followed this ancient tradition when constructing a story out of attributing single elements, such as the wings and the stone, instantiating the speaking man with a persuasive, moral sense as a meaningful whole. Still, the rhetorical effect depends on the participatory involvement of the perceiver. In this case, the opposition between flying and being burdened is easily connected with the abstract and proverbial statement of the introductory title. The opposition between mental gifts and malign poverty is metaphorically visualized; wings represent metonymically the potentiality to fly as a metaphor for the desired effect of “higher” knowledge, while the heavy stone hyperbolically exemplifies the

526

H. Fu¨hrer et al.

burdens of poverty. In rhetorical terms, the allegorical poem presents the figure of a speaking man, a personification, with a struggle in life, in a general way that “should expound some moral or ethical truth” (Goeglein 2007: 47). Such commonplaces seem to promise eternal knowledge.

Speaking to the Reader Alciato’s Book of Emblems is a collection of epigrams, small poems that often provide surprising moral commonplaces. Vitalizing the ancient tradition of epigrams, Alciato’s poverty poem lets a first-person voice “speak” to the reader. Initiating a dialogue is the best method to evoke the effect of immediacy. Still, Alciato’s “speaking” person is a personification that gives an abstract idea a voice and thus a persona who animates and presents something inanimate. This figure, called prosopopoeia, represents both the subject and the object. Like a dead person, speaking from an epitaph, the speaker is unreal, always there and not there. This allegorical performance blurs the distinction between time and space, materiality and immateriality, and word and image. Historically, several names and tropes, such as emblems, epigrams, memory images, prosopopoeia, hypotyposis, and ekphrasis, are performatively applied to describe ritualized or habitual experiences of immediacy in cultural media types, inside and outside of literary genres. These rhetorical categories presuppose an experienced cognitive effect of immediacy as they command full attention. The experience of immediacy, presentness, is an effect that matches the modern term of liveness, which “is not an ontologically defined condition but a historically variable effect of mediatization” (Auslander 2012:3, italics our). In a “ritualized rhetoric of presence” (Nelson 2010), a multimodal rhetoric can create a presence in “zones of intensity” (Führer and Kraus 2019), meaning that liveness is stipulated from shared experiences, memories, habits, values, conventions, and transmediated media products. This virtual communicative process challenges the common binaries between word and image, real and unreal, past and present, seeing and hearing, or any media platforms presumed to channel that interaction in varying degrees of intensity. On a theatrical stage, such ambivalent allegorical performance is a common practice, for example, not only in the Everyman plays of late medieval times or in Shakespeare’s Hamlet but also in films such as Ingmar Bergman’s The Seventh Seal, 1957.

Speaking Images The historical roots of the literary genre of epigrams are inscriptions on stones, walls, buildings, and epitaphs. A familiar term is graffiti. Consisting of words and/or images, or something in-between writing and drawing, the epigrammatic inscriptions encompass everything from formal inscriptions, written inscriptions on walls, or scribbled images on houses. Most of them speak to any passer-by to make them commemorate people, gods, or moral ideas (Milnor 2011). These inscriptions that perform a dialogue with an anonymous passer-by are called “speaking images”

22

The Anchor and the Dolphin: A History of Emblems

527

(Hagstrum 1958: 23). Like the verses put into the mouth of the “speaking person,” they often provide a moral statement to contemplate for passer-by who reads them. What is important is that epigrams and/or emblems are media products of ambivalent hybridity. As inscriptions or graffiti, they either visualize words or are materialized picture-images. Graffiti or epigrammatic inscriptions that speak to the passer-by create a sense of direct physical presence so that both the matter and the meaning of the described actions are carried into the reader-spectator’s very soul. Thinking is considered an act of mental images, whereby words can elicit thinking and thus provide an imagined sensorial experience that can give life to any abstract commonplace that has value for the addressed community. Modern advertising works on the same principles.

Ekphrasis In the communicative process of direct perception, the borderline between subject and object and word and image becomes blurred. Engaged in communication, the perceiver experiences the fusion of media modalities in a way that makes it sometimes impossible to separate and pinpoint exactly its components. Sound poetry, even when written down as verses, is a familiar literary media type that performatively intertwines the sensorial modality – sound, rhythm, and sight – with the semiotic modality in a dialogue with the meaning-creating perceiver (Clüver 2002). In the same way, dialogical epigrams intertwine the sensorial modality of vision and sound with the semiotic modality following the same performative habit. Their potentiality to evoke an overwhelming experience of liveness had been given a Greek name, enargeia. It is regarded as a force that is most often linked to the trope of ekphrasis as a vivid description of something that evokes imagination (Webb 2009; Führer and Banaszkiewiczy 2014). It evokes the same immersive reaction that reduces the distinction between subject and object, or the inner and outer spheres, as it draws the reader into a virtually evoked scene to which she becomes a witness. In Renaissance poetry, ekphrasis is a widely practiced technique that embraces not only a range of objects that are vividly presented but can be intertwined with personifications, symbols, allegories, and emblems (Gombrich 1971; Eisendrath 2018). This rhetoric can be explored in the poetic and dramatic works of Petrarch and Spenser and Marlowe and Shakespeare. The “speaking image” of the person in the first part of Alciato’s epigram is such an ekphrastic image that imbues a described object with animation and stipulates the mental images of thinking. Thus, several rhetorical tropes are intertwined to evoke thinking.

Transmediation Already, Greek scholars started to collect epigrammatic inscriptions to immortalize a historical effect of liveness in their inscriptive voice. They transformed the inscriptive voice into an “authored” literary tradition on other material and institutionalized

528

H. Fu¨hrer et al.

grounds, that we now call transmediations (Elleström 2019). Even though these transcriptions of the inscription changed their material and spatiotemporal modality, they did not lose the suggestive audio-visible effect. Both the collector and the reader need to reconstruct and imagine the vision of these inscriptions (Gutzwiller 1998: 7). The humanist habit of systematically collecting, translating, and transmediating ancient texts and inscriptions continues to imitate the performative practice of ancient scholars. As part of his practice of collecting and translating ancient sources, Alciato also transmediated inscriptions from Milanese buildings. In his unpublished manuscript, the commonplace exists in a tripartite combination of a text, a drawing, and an allegorical explanation (Laurens and Vuilleumier 1993: 89). Furthermore, he collects from multiple sources to construct his poems: his moral conclusion, or sententia, imitates the satirical verses by Juvenal, a Roman author writing in the late first and early second century CE. In Juvenal’s collection of epigrams (Satires 3.164–5), we find a similar commonplace: “It is hard for people to rise when straitened circumstances stand in the way of their natural abilities.” Alciato’s humanist contemporaries were familiar with the genre of epigrams and the dialogical tradition that continued to speak across time and space in an ongoing dialogue with the passers-by (Norton 1999: 2). In short, epigrams or emblems are practiced in a dynamical relationship between the object’s effect on the subject, with no clear separation between the inner mental space, storing and producing imageries as thought-things, and the outer physical and static pictorial signs in a spatial field, such as a picture. This ambivalence becomes lost in an emblem discourse that claims a principal distinction between word and image. The motif of the speaking person and the gesture of the outstretched arms are often repeated composites and patterns. Still, they can be considered a singular phenomenon in a poem, like Alciato’s epigram, as an autonomous artifact. Alternatively, they can be traced as references to transmediations of other sources. The concept of transmediality assumes “that different media types share basic traits that can be described in terms of material, or more broadly physical, properties and abilities to activate mental capacities” (Elleström 2019: 2). In the humanistic context of reviving ancient culture, epigrams are an ideal source for commonplaces, and emblems often integrate and transform other intermedial media products to provide a variation of a contemporarily accepted wisdom.

Comparing Word and Image In the first illustrated edition of Alciato’s Emblematum liber, the material existence of the picture, the added material pictura, evokes the effect of comparing words and images. For instance, the pictorial bearded man is depicted in a scholarly coat and has outstretched arms, one raised, while the other points downward. These non-verbally mentioned pictorial elements can identify the personification of the “speaking voice” as a contemporary doctoral scholar. Still, the image on its own would have been hard to understand without familiarity with the identifying attributes of a scholar. The illustration of poverty by a stone and of knowledge by wings

22

The Anchor and the Dolphin: A History of Emblems

529

assumes a metaphoric combination between words and things, verba et res. In Alciato’s poem, the man, presented as a still image with his outstretched arms, illustrates the timeless struggle between the burdens of poverty and the universal desire for higher knowledge. This verbal or visual image can be understood literally and metaphorically, as depicted/described and as imagined. The great number of emblem books helped to popularize and standardize this process of meaningmaking. However, emblems are not an entirely new invention. The description and depiction of abstract concepts have older roots. Apart from the ancient sources, medieval allegorical illustrations in churches, buildings, and books are important for cultural conventions. In this aspect, the allegorical Christian Physiologus, a collection of moralizing beast tales dated to the second century AD, is a prototypical handbook explaining the dogmatic Christian religious and allegorical meaning from the so-called Book of Nature, arranged by God to instruct humanity. The book described animals and fantastic creatures to teach the dogmatic truth that was thought to be embedded in nature. Circulating in many illuminated manuscript copies, these bestiaries can be considered emblem-like modes of expression (Saunders 2016) that prioritize the derivative nature of emblems over the “sudden birth” of a new genre.

Similarities and Differences Having illuminated some characteristics of Alciato’s poem with respect to medieval allegories, another contemporary source will be shown. The similarities are so tempting that they have been declared to be normative characteristics for emblems in a way that gives Alciato his central role in emblematics. The example is taken from Francesco Colonna’s enigmatic book Hypnerotomachia Poliphili that was first published in 1499 by Aldus Manutius in Venice. The book is notable for its printing designs, its innovative illustrations, its figured capitals, and its impact on emblems. In this book, Colonna recounts the allegorical struggle of Poliphilo, a lovesick protagonist and scholar of pagan antiquity who wants to unite with his love Polia. Through the eyes of the dreaming protagonist, the reader envisions the engraving of a female figure on a bridge, in a landscape filled with marvelous ruins. This engraving depicts a female figure sitting on a stool holding a pair of wings in her right hand and a tortoise in her left hand. She lifts her left leg as if she would like to rise. Furthermore, in his dream, Poliphilo sees a titulus added to this image: “Control speed by sitting, and slowness by rising” (Velocitatem sedendo, tarditatem tempera surgendo). This example shows clear similarities to Alciato’s tripartite epigram with a titlelike motto, the verbal image of an allegorical figure and a short explanation in the form of a sententia. In both cases, the motto helps to explain the enigmatic gesture of these allegorical figures. Both are personifications that visualize an announced enigmatic paradox. Alciato’s gender-neutral speaking voice becomes a woman in Colona’s verbal image that is imagined as a material image. The identificatory

530

H. Fu¨hrer et al.

attributes are also similar: one of their hands is equipped with wings. This is a standardized heraldic symbol for flight that is still in use in the French armed air forces (AAE). However, instead of Alciato’s stone, the sitting woman holds a tortoise, a symbol that stands in for multiple abstractions such as longevity, patience, and invulnerability. The tortoise is an attribute to signify Venus, the Roman goddess of love, but it also belongs to Hermes, the messenger between the divine and the human realm. This visual knowledge belongs to standardized iconography and the heraldic lexicon, the “database” built from ancient and medieval sources. As we see here, the semantics of both word and image depend on their context. In Alciato’s ekphrastic image, wings are both an indexical and a symbolic sign to signify the flight to a higher knowledge, whereas the same wings in Colonna’s ekphrasis stand in for the abstract concept of speed. Thus, wings – as both a verbal and a visual sign – have a natural referential similarity to things, but as they are used differently in the context of this described scene, their symbolic meaning changes. The stone symbolizes poverty according to Alciato’s motto, whereas the tortoise stands in for slowness and a direct contrast to speed. This opposition is also indicated visually in the sitting figure raising her leg to indicate controlled movement. Word and image are bound together even further by another personification, a nymph called Logista. While following Poliphilo, she performs as the speaking figure who addresses and explains to him the picture language of the hieroglyphic image of the sitting woman. The name of the nymph, Logista, announces her function. Derived from Greek logos, meaning word, order, and rationality, her verbal explanations represent the explanatory conclusion, the sententia, of the enigmatic image. Moreover, Colonna’s connection between allegorism and the enigmatic real of divinity is anchored in the extant Greco-Roman dream books (Struck 2010: 63). While we related Alciato’s epigram to ancient dialogic inscriptions and ekphrases, Colonna’s example draws from another ancient source: Egyptian hieroglyphs. Colonna’s book consists of influential explanations of this picture language. Stressed by Poliphilo’s dreams and the fairytale-like figure of a nymph, he foregrounds those enigmatic symbols that conflate the human and divine sphere in the ambiguity between materialized and envisioned images in a neo-Platonic tradition. Thus, according to the conventional habit of reading, this female figure represents an abstract morality independent of its materialization in word or image. The semiotic modality of a media product always intertwines the material and mental signs to reveal possible meanings to this figure. Ambiguously, the figural personification strives away from her personal self toward a general idea. On the other hand, as she indicates not someone else but just herself, she ambivalently combines a certain referential and a self-referential structure. The same can be said about the wings, the stone, and the tortoise. As attributes in a larger allegorical scene, they gain the sign character of symbols. They are always grounded in contemporary cultural convention (Burke 1997). These allegories express a struggle in the generalized terms of commonplaces. Note that this differs from modern media products where we often find the same expressed as an individual psychological struggle. Applying such modern reading conventions, the unstable position of the figure in the epigram can pragmatically refer to the difficult socioeconomic conditions of humanist scholars

22

The Anchor and the Dolphin: A History of Emblems

531

such as Erasmus of Rotterdam (1466–1536) or the printer Aldus Manutius (1450–1515) who will be introduced below. In short, epigrams are “speaking images” that evoke a performative dialogue with another world of the past and the interactivity between subject and object. They promise the gift of truth from the past in the form of a commonplace. Their truth is independent from a material pictura. Even the materiality of words can elicit thought images if they belong to a common reservoir of memorized images. However, these changes depend on the perceivers’ cultural contexts. Indeed, we observed a striking similarity between an ekphrastic description in picture language in Colonna’s Hypnerotomachia and the tripartite format of epigrams or emblems. Both feature a title and an allegorical or symbolic image combined with an explicative commonplace. Interestingly enough, Colonna’s book is not considered as an emblem book, but merely as a forerunner and source as it was published before Alciato’s emblem book.

Alciato as a Renaissance Humanist To fully understand Alciato’s emblematic, we need to view it in the sociocultural context of Renaissance humanism. Developed from the genre of epigrams, emblems are regarded as a “truly new Renaissance genre” (Russell 1995: 279; Manning 2002: 14 f). Usually, the Renaissance is regarded as a period of its own, different from the “Dark Middle Ages” (Nelson 2007), that spans to the Baroque up to the eighteenth century. The Renaissance period was formed by the self-proclaimed role of humanists, “the midwives of the ‘rebirth’ of a classical culture” (Waswo 1999: 25) who emphasized change and progress. Newly discovered ancient texts became a normative model for a group of erudite humanists, The Respublica litterarum, proclaiming a standard of accuracy when translating and exploring ancient sources. In general, Renaissance scholars wanted to publish new knowledge gained by the study of ancient languages directly from rediscovered sources such as Cicero and Quintilian in the case of prose and Horace and Vergil in the case of poetry (Eisendrath 2018). Thus, knowledge was normatively drawn from ancient texts rather than from empirically exploring the order of natural things. As they revealed the inaccuracy of the Vulgate compared to the Greek and Hebrew Bibles (Taylor 2014: 330), the clerical Latin, the scholastic logic, and the medieval anagogical biblical hermeneutic (Ohly 1959) were called into question. However, even when announcing ancient cultural achievements as incontestably superior to that of their own time and place, they distrusted the scholastic doctrines and use of language and not the Christian truth.

Authorized “Fathers” of Their Time Francesco Petrarca, also called Petrarch (1304–1374), is the key figure of this movement in the fourteenth century. Commonly, he is considered the initiator of

532

H. Fu¨hrer et al.

the Italian Renaissance and entitled the “father of humanism.” He became famous for combining Christian philosophy with the ethical norms of classical culture. His poems were mostly written in Cicero’s Latin, as he found in Cicero’s letters “abstract formulations of the truth” (Wilcox 1987: 154). Petrarca’s ekphrastic description of the ruins of the Eternal City, Rome, vividly describes the multiple pragmatic material objects of a past culture in a way that evokes an immediate image in the readers’ minds by foregrounding his individual views (Eisendrath 2018). In 1341, Petrarca was crowned poet laureate, poeta laurus, on the Capitol in Rome. Corresponding to the title of Alciato’s Emblematum liber in 1531, Petrarca’s title as poeta laurus and erudite poet, poeta doctus, served mainly the contemporary authorization. The same goes for Alciato’s title as pater et princeps of emblems. This title authorizes the new literary genre of emblems similar to the Aristotelian canon (Scholz 2002). The humanist habit of collecting, translating, re-ordering, quoting, and transmediating Greek and Latin media sources can be explained by the rhetorical rules of amplifying and presenting multiplicity, copia. This favored humanist practice of “encyclopedizing” multiple topics in a pedagogic way in these small poems (Daly 2016: 357) has its roots in ancient and medieval rhetoric. Collections of bestiaries, herbaria, and lapidaries provided a mirror of imagines mundi, images of the world, in order to assign a symbolic meaning to all “things” in the world. Alciato’s Emblematum liber exemplifies this erudite tradition of collecting, translating, and combining wisdom elements from other media sources. He intertwines the performative rhetoric of epigrams and ekphrasis to shape the specific symbolic and allegorical character of emblems. The poeta doctus, as both translator and collector of commonplaces, assumes the role of an interpreter to unconceal the Truth, reminiscent of the role of Logista in Colonna’s book.

Knowledge, Res, and Verba The one-to-one correspondence between things and name-giving words, res and verba, known as the scholastic correspondence theory of truth, presupposes a referential link to realia, things of reality. It is grounded in the divinity of the words (of the Bible) and earthly things, the “Book of Nature.” A distinction is made between content, res, and form, verba, and between the corruptibility of the body (res extensa) and the incorruptibility of the soul, conceiving of flesh and spirit as separate entities. A “Christian humanist,” Desiderius Erasmus (1466–1536), emulated this scholastic view in De ratione studii (On the Method of Study), 1512, with the following words: “In principle, knowledge as a whole seems to be of two kinds: of things and of words” (Erasmus 1978: 666). Accordingly, things are only learned by words, understood both as the “Words of God” and as the “sounds attached to them” on the grounds of John 1:1: “In the beginning was the Word, and the Word was with God, and the Word was God.” Here, Erasmus proposed a performative intertwinement of words with the “reality” of things instead of separating them into two realms. Word, logos, in the Christian understanding refers to the entirety of the cosmos ordered by God. Thus, in this context, words as well as images, in the modern sense of basic media, are anchored in a meaningful whole. On these grounds, he established a multimodal

22

The Anchor and the Dolphin: A History of Emblems

533

rhetoric grounded on the Ciceronian ethics of ethos and virtus, both leading to the renewal of social and political missions as renewed Christian ideals. For Erasmus, only an exegesis from the original language can disclose the hidden sense of the Book (Brown 1984). Thus, he translated the New Testament from Greek to Latin and published a Greek/Latin parallel version with separate erudite annotations. By doing so, correcting the corrupted Latin Vulgate in 1516, he both challenged the notion of a correspondence between words, mind, and reality and made it the declared aim of his poetry and education to restore this correspondence (Waswo 1999: 33). Still, source texts have the potential to initiate new ways of thinking about material things. This gives power to an erudite person since, according to Erasmus, the one “who is not skilled in the force of language is, of necessity, short-sighted, deluded, and unbalanced in his judgment of things as well” (Erasmus 1978: 666).

Translation: Preserving the Truth As a byproduct of the performative activity of collecting, transforming, and remodeling ancient ideas in contemporary cultural products, translation became both a legitimate poetic domain and a debated field of its own (Taylor 2014). The numerous translations to and from Greek and Latin aimed at enhancing this interconnected reservoir of words and things and thereby contributing to a refined way of thinking. Thus, the act of translation is consciously embedded in “the basic problem of how our uses of language produce the human world” (Hauser 1995: xv). Alciato was active as a translator and poet. He translated 160 epigrams from this Anthologia Graeca, a collection of Roman and Christian epigrams, for a new edition, entitled Epigramma Graeca selecta. Famous humanists such as Erasmus and Thomas Moore (1478–1535) also produced translations for this publication in 1529. The Anthologia Graeca was first printed in Florence and became the famous model for Alciato’s Emblematum liber in 1531. Alciato transferred 50 epigrams from a Greek epigram collection to his book (Praz 1961: 25–26). These epigrams are now referred to as emblems. However, contemporaries would have understood these epigrams rather “as a miscellany, without being coordinated” (Miedema 1968: 246). This argumentation links the title emblema to the humanist habit of collecting as much “knowledge” as possible. In general, the performative act of translation intertwines the theoretical distinction between form and content while operating in the tension between maintaining as much of the “original” source as possible and adapting the sociocultural past to a different modern code system (Jakobson 2002: 127). On the one hand, a new awareness of the complexity of language was developed. Rather than providing a singular interpretation, the translator becomes aware of the multiple levels of possible meanings in source texts. Words are more than a transparent carrier of a preexisting (divine) knowledge, and rhetoric is more than a style decorating the “content”; both are inseparably entangled in shaping knowledge, and neither can be separated as an autonomous component. The same goes for the abstract basic media of word and image. On the other hand, in the early Renaissance,

534

H. Fu¨hrer et al.

translation became an autonomous and “transparent” practice detached from commentaries, paraphrases, etc. (Basile 2021). To summarize so far, Alciato’s translations of epigrams are more than a mere philological exercise and compilation of ancient texts (Sgarbi 2014), in particular when regarding the change of the notion of epigram to emblem.

An Early Media Discourse Based on a performative understanding of rhetoric as constructing truth, an early media discourse reflects on the unstable referential relationship of words and images in the process of translation and meaning-making. To the historical background of emblems belongs the fundamental question of how the basic media, word and image, are interrelated in this hermeneutical process. Moreover, as the dissemination of rediscovered ancient cultural knowledge, rhetoric, and philosophy were didactic tasks for Renaissance humanists, they sought support in the new print technology. Like Erasmus, Alciato reflects on the significance of words and things in De verborum significatione (On the Meaning of Words), 1530, one year before the emblem book finally appeared in print: Words indicate, things are indicated. However, things, too, can sometimes indicate, as for example the hieroglyphics in Horus and Chaeremon; and proceeding from this idea, we have also written a book in verse with the title Emblemata. (quoted after Miedema 1968: 242)

This quote explains two important things: Firstly, Alciato speaks of his emblems as a “book in verse with the title Emblemata.” He did not mention illustrations. Secondly, while he seems to give a signifying priority to words, verba, he also assumes that some things, res, can speak for themselves. Obviously, the “things” he has in mind are divine things which he exemplifies by hieroglyphs, meaning in Greek “sacred writing” in a picture language as explored by the Stoic philosopher Chaeremon. Different from the variety of human languages, a divine punishment after Babel, hieroglyphs promise to be the “pure,” transparent signs that signify the divine primordial essence of the universe (Dieckmann 1955). In the previous section, we already saw a connection between Alciato’s emblems and hieroglyphs. As only priests under divine inspiration can read the allegorical structure of these signs, Colonna’s book about hieroglyphs introduced a nymph as interpreter. Two things are important here: Firstly, words and things are not distinguished as separate entities of word and image. Rather, both are grounded on an analogy. Secondly, this thinking is akin to the typological hermeneutic that distinguishes between the literal and the symbolical sense of the Bible, whereby the latter transforms the medieval anagogical hermeneutic into an allegorical. This hermeneutic presupposes two opposing ideas of interpretation: one assumes that a precise deeper meaning can be found under a surface, the neo-Platonic symbolon. The other

22

The Anchor and the Dolphin: A History of Emblems

535

medieval and Patristic tradition proposes symbolism as a way of speaking of something unknowable, “a silence that teaches secretly” according to Pseudo-Dionysius (quoted after Eco 1985: 386). According to this view, symbols are not translatable symbols. In medieval theology, all contradictoriness and ambiguity are merely semiotic. Ambiguity is thus a question of interpretation, but not an ontological one since the world has been made as a whole by God. Or, as Umberto Eco would have it, already in medieval metaphysics and universal symbolism, “there is the quest for a code” (Eco 1985: 387) and an ordered way of limiting the potentiality of possible meanings according to the four senses (literal, allegorical, moral, and anagogical). This hermeneutic model should guarantee the right of the Church to be the warrant of the “one” interpretation. Later, the way to the modern sense of symbolism comes in with the decision to distinguish between a trivial surface and an essential deeper meaning according to what the Bible teaches. The encyclopedic collections of the didactic bestiaries such as the Physiologus give meaning to the “things” of the real world. As a result, the same creature, such as the lion, can take the contrasting meanings of being both the figure of Christ and the figure of the Devil (Eco 1985: 393). The correct solution is provided by the context in which the figure is positioned. In the same way, the hieroglyphic (alleged) picture language reveals the Truth in an allegorical fashion as it employs both verbal and pictorial generalizations in an inorganic or dissonant combination of autonomous motifs. On these grounds, a “hieroglyphical mode of thought” has been postulated (Wittkower 1968; Dieckmann 1970). All these modes of thinking, the hieroglyphical, typological, symbolic, and emblematic modes, give priority for ideas creating a coherent representation of reality.

Emblems and Imagination In the previous section, in order to explain the historical emblem discourse, the early semiotic relationship of res et verba had been foregrounded, but in fact, the early modern discourse already combined the debate directly with emblems. Instead of the transcendent dimension of emblems in the context of hieroglyphs and biblical allegory, the concept of memory is emphasized by Francis Bacon (1561–1616), one of the leading scholars in natural philosophy and scientific methodology. Like Erasmus and Alciato, he took part in the debate about res and verba. He argues that “words are images of things or matter” (Howell 1946: 136). When he links emblems in a purposeful interconnection to the mnemonic function, he justifies the ambivalence between the material and virtual modalities of words. In The Advancement of Learning (1605) Chap. 5, he writes: Embleme deduceth conceptions intellectually to images sensible, and that which is sensible more forcibly strikes the memory, and is more easily imprinted than that which is intellectual. (quoted after Green 1870: 2)

536

H. Fu¨hrer et al.

According to Bacon, emblems foster the memorizing of abstract concepts. Moreover, they refer to a general, perhaps transhistorical practice of sense-making. Sensemaking includes a process of memorizing by means of images and of imagining words-as-if-images. According to the rhetorical theory of memory, the value of emblems is related to spatial memory (Carruthers 1998). This rather recent trend in emblem research has its roots in Bacon’s home which was decorated with emblems as local mnemonic topics (Engel 1999: 125). Like many of his contemporaries, Bacon makes no difference between emblem, symbol, and a hieroglyphic or allegorical Sinn-bild as a sense-making image. These are synonymous because, as Bacon assumes, “an affinity with the things signified” (quoted after Shrieves 2013: 4) exists. An affinity is not the same as an identity. The ambiguity of emblems depends on the perceiver’s prioritization of either the materialized pictura or the verbal image and will vary between different perceivers. This ambiguity corresponds to the entitling – and thus categorization – of unillustrated books as emblem books (Praz 1964: 139), and this argumentation, typical of the Renaissance, is anchored in the Aristotelian concept of imagination. Imagination is regarded here as an intermediary between perception and thought, since “the soul never thinks without a mental picture” (Aristotle, De Anima 432a). Those modern emblem scholars who follow this hermeneutic and semiotic perspective propagate an analogy on the cognitive level rather than the material distinction between the basic media of word and image in the media artifact (emblem). Nevertheless, this does not imply the Renaissance idea of a transparent medium grounded on the hieroglyphic picture language, which is linked to the Divine, whereas the Romantic symbol consists of the harmonious form and content unity opposed to the allegory (Berefeldt 1969: 201). Both concepts of allegory and symbol resist human reason and systematization, as favored in the mathematical logic of Peircean representation. They refuse a linear cause-and-effect relation and dualistic dichotomies. Rather, they “respect the whole of the traditional wisdom because even where there is contradiction between assumptions, each assumption can bear a part of truth, truth being the whole of a field of contrasting ideas” (Eco 1985: 399f). When transferring this to emblems, multiple and even dissonant interpretations are possible. In the semiotics of Saussure, the category of symbols is related to myth and therefore encompasses signs such as the cross, the hammer, heraldic signs, and emblems. This acknowledges the infinite polysemy of symbols that are paradigmatically open to infinite meanings but bound by their context. By contrast, a Peircean “symbol” is a habitual conventional sign referring to a general rather than an individual object. Thus, symbols can represent their objects, independent of any resemblance or any real connection. An understanding is guaranteed due to habit or convention (Belluci 2021). According to him, the signifier is linked to its meaning by mathematical laws, that is, precise conventions, and interpretable by other signifiers. In this view, there is no self-contradiction, called the coincidentia oppositorum, but a linear sequence of cause and effect. This guarantees the identity principle. Most often, these differences are not considered, and the term symbol is used in a blurry

22

The Anchor and the Dolphin: A History of Emblems

537

way, referring to an integrated media representation in a general sense (Elleström 2019: 5).

Alciato’s Book Title Emblema The scholarly definition of an emblem started pragmatically from the title Emblematum liber in 1531. It was assumed that this title indexically refers to what follows: the combination of word and image on the same material space. This conclusion could be proven by the notion’s etymological roots. While the Greek notion signifies “any mounted or inserted part, from an insole in a shoe to a cultivated branch grafted onto a wild tree,” the Latin emblema signifies “a technical term for objects d’art and is used of inlaid work” (Miedema 1968: 239). Applied in rhetoric, it is pejoratively used for mere decorative expressions. In the editorial history of the multiple editions, translations, and commentaries of Alciato’s book, the understanding of emblema as mosaic-like embellishment on multiple material grounds became the one that had been mostly repeated (Russel 1999), albeit retrospectively. The first time it appears is in the translated French version of Alciato’s dedication to Conrad Peutinger (1465–1547) in 1536. From now on, the metaphor of a mosaic became accepted as it allows one to think of emblems as composite of their constituent parts, i.e., sententia, eikon, and versus (Miedema 1968: 248). Even today, the notion of “emblem” or its adjective “emblematic” is often linked with “embellishment” and synonymously used with “symbol.” In the previous section, we approached the term from cultural sources and its ancient heritage. Another way is to find sources, where and when Alciato expresses his intentions. One already quoted text was from 1539. However, in 1523, he wrote a letter to his college Peutinger to whom he sent and dedicated his emblema: I have, on these Saturnalia, to do a favour to Ambrosius Vicecomes, put together a small volume of epigrams, which I have given the title Emblemata: for I describe in the single epigrams something that is, taken from history or nature, somehow appealing. Starting from there, painters, goldsmiths, and metal casters can produce what we call‚ Shields’ (scuta), and pin on hats and that which we list as signs (insigne) such as the Anchor of Aldus, the Dove of Froben. (quoted after Wolkenhauer 2018: 6)

Alciato’s quote relates the title emblemata again to his collection of epigrams. They are specified as “appealing” things from history or nature. This explanation might allude to the collection of natural things in the tradition of the Physiologus in order to assign these things a symbolic meaning. Alciato also proposed a pragmatic function for epigrams or emblems when he suggested them as templates such as badges, scuta, and printer’s marks, insignia. Indeed, there is a long tradition of craftsmen using books such as the Physiologus as handbooks for decorating churches or walls in other buildings. The letter quoted above will be further discussed in the next section for developing an appropriate approach to emblems as “applied arts,” in order to elucidate upon their pragmatic function.

538

H. Fu¨hrer et al.

Editio Princeps The puzzling title Emblematum liber of the editio princeps by Heinrich Steiner (Steyner) (ca. 1500–1548) in Augsburg, Germany, had been used as an anchor to consider the collection of 104 short epigrammatic poems with 97 illustrations by the painter Jörg Breu (c. 1475–1537) and the woodcuts by Heinz Schäufelein (1480–1540). This intended and authentic emblem format (Carlisle 2018: 165) became the successful version. Thus, one might conclude that the solution did not seem repugnant to the eyes of the contemporaries. Although Alciato explained his intention to call his epigrams emblems in the context of a hieroglyphic and allegorical interpretation of things, his disagreement with Steiner’s edition refers to the many printing errors, the quality of the wood cuts, and the disorder. Still, Steiner’s successful version was reprinted three times, twice in 1531 and once in 1534. Despite Alciato’s protests about the printing errors, he reworked, expanded, and reprinted his epigrams in cooperation with the befriended Parisian printmaker Christian Wechel (1522–1554). The following French editions changed the understanding of emblems in a way that seemed to justify the assumed difference between emblems and epigrams on material grounds. Beside the new illustrations, Wechel constructed a coherent and authorized design already in the first edition of 1534. On one page, the title or motto pointed indexically to the more elaborate woodcut image, pictura, and the subsequent explanatory poem. After 1534, Wechel’s design was regarded as the standard of the tripartite construction (Kusler 2017). Indeed, the recognizable tripartite emblematic pattern on one page made it easier to establish a coherent meaning and to recognize them as emblems rather than ekphrastic epigrams according to the ancient genre conventions. However, this tripartite design existed already in Alciato’s ekphrastic poem: [I]n the 1540’s the form of the emblem was determined by tradition; that people were still aware that an emblem was really an epigram, but that the trio of caption-figure-epigram was the conventional form in which the emblem was presented. (Miedema 1968: 247)

Still, neither does an “original” of the emblem exist nor a finite Emblematum liber by Alciato. Rather, the multiple translations of Alciato’s work, the additions and subtractions of poems and illustrations, allow the modern scholar to link many versions to a network. The same puzzle had to be done when exploring the reasons for the delayed printing process of Alciato’s emblems. In Alciato’s case, this fact establishes the book as a teamwork, a cooperative media product, like a film or theatre play. Many people, technologies, and socio-political factors are intertwined in the process of production beside the intention of the author. The socio-political factors causing the delay of the printed book were the unstable economic situation of the printer houses and the political situation in Augsburg, where Conrad Peutinger worked within the patronage of the Emperor Maximilian (1459–1519) and his successor Charles (1500–1558). The illustrations clearly expanded the reading audience beyond the small circle of erudite humanists. This served not only the economy of the printer but also the upcoming protestant movement that challenged

22

The Anchor and the Dolphin: A History of Emblems

539

the authority of the Catholic Church by demanding vernacular bibles and mediative tests for the broad public, not just for the initiated ones. The dissonant religious currents are mirrored in the emblem books, such as in the variety of religious emblem books, emblemata sacra. In short, already by the time of Alciato’s death in 1550, emblems had evolved on their own in different directions blending the two printed versions of emblema nuda and illustrated emblems with the applied emblems that decorated the interior settings of walls, tapestries, etc. To sum up, when embedding the production of emblems as historical media products or media types into the context of the Renaissance, the definition of emblems cannot be limited to the first edition of Alciato’s Book of Emblems of 1531. Rather, by adding the potentiality of memory and imagination, emblems intertwine external space with the internal space of meaning-making, as well as different material grounds where the book constitutes just one technical medium of display. This notion is connected to both the genre of epigrams and ekphrasis.

Printed Emblems In this section, we widen the view on emblems by considering the influence of printing technology and the emerging printing houses on the development of this genre, taking Aldus Manutius’ printing house as both a humanistic trend setter and trend follower. By doing so, this section expands the topic of printed literary emblems to the broader field of “applied emblems.” The famous Venetian printer Aldus Manutius (1450–1515) was a grammarian and humanist teacher, but he also created a printing house that became known as the Aldine Press, run by a family dynasty over many decades. Aldus Manutius had a vast influence on the history of the literary genre of emblems and on its expansion to applied art. He published famous Renaissance poets, such as Dante and Petrarch, and contributed significantly to the increased popularity of epigrams and picture-images of hieroglyphs during the Renaissance. In 1503, 1521, and 1551, Aldus reprinted different versions of the same Anthologia Graeca that Alciato had previously translated. Like other anthological collections, the epigrams provide various commonplaces composed in short sequences rather than as a coherent narrative. These collections presuppose active perceivers interested in composing multiple entities into a new whole. In his reprints of the Anthologia Graeca, Aldus added the alternative title Anthologia florilegium; rather than merely announcing the content of the book in a descriptive title, he offered an appealing metaphor in his title: a bouquet of flowers, florilegium. This metaphor visualizes the result of collecting separate sources, referring to knowledge drawn from inscriptions and epigrams. Like the Physiologus, these compiled arrangements serve to classify and order the world according to Logos, the Divine Beauty, and Truth. In the same vein, Alciato’s title Emblematum liber had been suggested as a metaphor for the highly appreciated format of an encyclopedia similar to Erasmus’ Adagia, his collection of commonplaces (Scholz 2002). Several editions and translations of the Adagia and the Emblematum liber organized their collections after headings or keywords, and in

540

H. Fu¨hrer et al.

the same manner, Albrecht Schöne and Arthur Henkel provide the richly annotated anthology Emblemata: Handbuch zur Sinnbildkunst des XVI. und XVII. Jahrhunderts, 1967. Their bibliographical Supplement (1976) contains 2338 entries assembled from the major European emblem books in Latin and in multiple vernaculars and classifies emblems by motto, themes, and motifs (e.g., animals, Gods, mythological persons, etc.). Like an imaginary text-image museum, this manual guides the searching eyes of modern readers through allegorical and emblematic culture. In 1505, Aldus published the rediscovered Hieroglyphica of Horapollo. Explaining 189 hieroglyphic symbols, this text from late antiquity was thus considered an ancient Greek dictionary and became famous through its many translations. Already in 1499, Aldus published Colonna’s Hypnerotomachia with 172 illustrations. While illustrated books had increased in popularity by the end of the fifteenth century, the Aldine Press focused mostly on scholarly (non-illustrated) works; however, the illustrated Hypnerotomachia Poliphili was an exception. Designing both words and images to an elaborate whole of a picture language, this became one of the most celebrated books of the Renaissance. This book establishes the notion of emblema (Miedema 1968) and contains the image of the sitting female figure with the wings and the tortoise which we introduced in the previous section. Another illustration is that of an anchor and a dolphin, quoted by Alciato as a well-known symbol adapted by Aldus Manutius in a slightly modified version, together with the motto festina lente, as his personal emblematic printer’s mark, insigne. Aldus’ early humanistic book production thus influenced Alciato’s concept of epigrams and emblems, perhaps on the basis of the books about hieroglyphs (Giehlow 2015 [1915]; Volkmann 1923; Daly 1979). These books are also important when tracing transmediated motives applied in literature and in the iconography of contemporary art up to the nineteenth century; the assumed hieroglyphic or emblematic mode of thinking disturbs the limited frame of 1531–1750 (Wade 2010–2015) and the generic narrative of emblems as originating from Alciato’s Book of Emblems. The famous Aldus printing house published a new edition of Alciato’s Emblematum liber in 1546, replicating the structured layout of Wechel’s editions but increasing the number of emblems by 86 new illustrations. Thus, not the “content” of the epigrams but the “form,” the bi-modality and combination of word and image, became a conventionalized marker of emblems.

The Title: The Anchor and the Dolphin We saw already in Alciato’s dedication to Peutinger a reference to the practical use of epigrams as a source of inspiration for designing identificatory attributes in a world of guilds, such as insignia, shields, or pin-hats. All these small, self-contained media products are already earlier products that might be classified as “applied art,” often characterized by a motto and a symbolic or allegorical image. Even when this suggestion of a pragmatic utilization of epigrams or emblems might be an addition of the editor (Miedema 1968), the anchor explicitly identifies Aldus, and the dove

22

The Anchor and the Dolphin: A History of Emblems

541

Johann Froben (1460–1527), and it would require cognitive activity on the perceiver’s part to add the missing attribute and to see a printer’s mark for what it is: a dolphin entwisted around an anchor, in the case of Aldus. Just as the same words can be combined to form different sentences, the anchor and the dolphin can be used in other visual contexts with different meanings as well. Often, the dolphin might stand for speed, and an anchor can mean anything from trust in God, hope, deliberation, self-discipline, or merely a halt (Daly 1979). Printer’s marks often provide a tenuous composition of various pictorial symbols to which verbal signs are often added. Since 1502, Aldus used as his pertinent maxim or motto festina lente, “to hasten slowly,” in combination with his printer’s mark. As we have seen in the previous section, the notion of motto or titulus has the indexical function of bridging its meaning with the following text or image, reducing interpretational ambiguity (Barthes [1964] 1977). Even when the motto is not directly visible, it belongs to Aldus’ famous emblematic printer’s mark as an “inlaid” media representation. Thus, the motto is bound to the image even when it is not inscribed. The perceiver can construct a plausible symbolic or emblematic relation when memorizing the text.

The Role of Print Shops Certainly, the emerging printing technology economically governed, enabled, or at the very least participated in what has been generalized as the “visual culture” (Ong 1959) and mass culture of the “Gutenberg Galaxy” (McLuhan 1962). Rather than treating books as individual creations of single authors, the circulated medieval and Renaissance manuscripts are embedded in a cooperating community of humanists elaborating on general concepts in order to offer orientation and new knowledge. In this humanist context, the printing shop of Aldus Manutius provided more than technology. His Philhellenic Academy contributed in a decisive manner to the study and cultivation of Greek literature by combining the old with the new, the oral with the visual. His newly developed typography for Greek types both imitated Greek manuscript writings and facilitated the process of reading with standardized punctuation and paragraphs. The sensorial and semiotic aspects of printed books overarch the distinctions between word and image. His portable octavo edition, often regarded as the prototype of the paperback format, is both cheaper and easier to transport and to read. The weight and the material features (or modalities) of a book have multiple effects. Aldus’ print shop combined Renaissance aesthetics, humanism, and economy. Aldus cooperated with many humanist scholars in the production of new, meticulously edited schoolbooks in Greek and Latin for the recently founded universities. Erasmus was one of these scholars, staying with Aldus for 7 months in 1508 to work on the second and expanded edition of his collected commonplaces, Thousands of Adages (Barker 2001). Erasmus was the first author to explain and categorize a variety of proverbs or adages at such a length, and their enigmatic character was highly appreciated as ancient wisdom that foreshadowed Christian doctrines (Wesseling 2002: 82). In this book’s dedication to Aldus, Erasmus also

542

H. Fu¨hrer et al.

elaborated on the commonplace festina lente, and by doing so, he enhanced the impact – or the brand value – of Aldus’ motto already in use. Thus, rather than the role of a singular author, we emphasize the network of interacting scholars in producing and disseminating their works. In short, the motto and the picture-image of the printer’s mark are a well-known sign combination that identifies the Aldus Press in the erudite community, even when the motto is not directly visible. Both can and shall identify the owner of a printing house. However, at that time, all these traits were widely copied by rivaling printing houses, even though local authorities tried to protect artisans such as Aldus with specific privileges against such unauthorized imitations, anticipating modern copyright regulations. By contrast, authors such as Erasmus or Alciato were scarcely protected and had to fight for economic support. Thus, the epigram quoted in the second section might allude to this economic situation. Emblematic printer’s marks from the sixteenth century are similar to modern company logos. Let us briefly consider the logo of the sports fashion company Nike. Both the printer’s mark and the logo are performatively established self-referential signs memorized by the perceiver. Thus, it becomes a habit to add the memorized slogans or mottos – just do it or festina lente – to the logo. Indeed, even the swoosh has an erudite iconographic background that leads back to ancient sources: the wing of the Greek goddess of victory, Nike, is the model for the swoosh. The unspoken reference of Aldus’ motto festina lente became the outspoken reference in the case of Nike, which became the name of the company, although not all customers might associate the name Nike with victory and the logo with the wings of a goddess. The “depth” of an interpretation, when actively composing parts to a bigger whole, depends on the perceiver’s ability to dive into the emblematic magnetic field and notice more than the dolphin and the anchor, so to speak. Still, Aldus’ emblematic trademark, consisting most often of word and image, combines identificatory, educational, moral, and economic functions. It has similarities to emblems, badges, and the Italian impresa (Scholz & Wolkenhauer 2018: 6. 11). While being distinct and separate from other media products, they are related to each other and belong to the categories of “applied arts” or “applied emblems” (Scholz 2018: 270). However, these early modern printer’s marks are assumed to be more than “just a figuratively formulated statement of personal identity”; rather, they were also “a conscious attempt at finding a place for that identity in the natural order of things” (Scholz 2018: 292). In this regard, they might differ from modern logos.

Applied Emblems, Art, and Iconic Emblems Unlike printed literary emblems, these applied emblems are mostly iconographic artifacts. In a similar fashion to the mediaeval iconographic tradition of marking iconic figures or personifications by tituli and fixed attributes, for most applied emblems, a text, an inscriptio, motto, or titulus, is added to facilitate their “reading” and identification – a practice seen in the two examples above. The stronger the link between the attributes and conventional knowledge, the lesser the need for

22

The Anchor and the Dolphin: A History of Emblems

543

identificatory titling. Thus, neither mottos nor iconographic signs were an invention of the print industry. From the thirteenth century onward, mottos appeared on seals, crests, shields, badges, or imprese, often shared by several families (Fowler 2021: 147). So far, we have encountered purely verbal emblems and emblems with word and image such as the aforementioned printer’s marks and logos, but we can also find examples of emblems where the subscriptio is inscribed in a manner that merges inscriptiones with the picturae, whereby the pictorial assumes the dominating role. “The Triumphal Arch” (1515) by Albrecht Dürer (1471–1528) is a 3-meter-high printed woodcut and a chiefly pictorial emblem, the text being difficult to read even up close. Drawing from the auctoritas of Horapollo’s Hieroglyphica, this iconographic masterpiece constructs a pictorial narrative in a symbolic picture language full of “cryptic emblematic allusions” (Panofsky 1948: 176–177), glorifying the divine rulership of Maximilian. This hieroglyphic woodcut seems to confirm the emblematic mode of thinking, using the universal picture language, albeit without following the formal criteria for emblems. Still, Erwin Panofsky equates the iconography of this pragmatic artifact, produced with a political and divine mission, with the symbolic layers used in high art. Thus, he dissolves the borderline between autonomous l’art pour l’art artworks and pragmatic, applied artifacts produced both with manufactural printing technology and without (such as imprese, badges, devices, etc.). Outside of the field of literary production, this emblematic principle can be extended to imprese and devices (Hekscher and Wirth 1959), as well as architectural decorations (Bath 2016) or festive rituals (Nelson 2010). This expansion and multiplicity of emblems on multiple material grounds destabilizes the notion of the year 1531 as their year of birth. In summary, when considering emblems in the light of the printing technology and socioeconomic conventions of the fifteenth to seventeenth centuries, it is clear that multiple initial conditions or agencies – technological, economical, and political – have “influenced” the emblem formats. In the new printing shops, many new editions appeared without being authorized by the author and even beyond the influence of the printer. Due to the economic instability of the small printing houses – resulting from their lack of a guild structure, as well as from the lack of supportive copyright regulations – the printing process did not result in a stable mass production of unified or uniform works during the fifteenth to seventeenth centuries.

Conclusion Finally, we formulate the questions of “what is an emblem?”, “what an is epigram?”, and “what is not an emblem?”. Are there specific media requirements, styles, and topics? In the previous sections, we used the words “epigram” and “emblem” interchangeably. We called a piece of text without any illustrations an emblem, because the author called it an emblem. Others called a 3-meter-large printed woodcut by Albrecht Dürer an emblem, even though the text is clearly subordinated to the visual and fulfills primarily a pictorial function. Emblems are often considered

544

H. Fu¨hrer et al.

to be an offspring of the Renaissance, a period that aimed to bring order in a world full of obscurity, yet strangely enough, there seems to be no order and clarity in the definition of this genre. Assuming there would be a generally accepted definition, the question arises as to whether or not emblems are bound to a specific epoch. In case we do not conceive of them as bound, the search for emblematic elements in modern graphical design, poems, and commercials could be a fruitful endeavor to critically challenge these hitherto standardized interpretations. Before we start, we must cite a standard characterization of emblems: These Emblem Books all have the same characteristics. Each emblem was made up of a symbolical picture, a brief motto or sententia, and an explanatory poem. The purpose of the motto was to complete and interpret the picture, while the picture gave meaning to the motto: neither could stand without the other. The purpose of the poem, the third essential feature, was to explain the whole and point to moral. (Freeman 1941: 151)

Let us then turn to the question of whether all emblems follow the tripartite structure. Freeman, like many other emblem scholars, defined emblems by three parts: a symbolic picture, a brief motto, and an explanatory poem. Yet, the early editions of Alciato’s Book of Emblems already presented many emblems without any pictures. Newer editions of Alciato’s book contained more pictures, which implies that the editor added these without permission of the author. We called the logo of the sports fashion company Nike an emblem or at least an applied emblem. The Nike swoosh clearly fulfils the requirement of a symbolic picture. But where is the motto or even the explanatory poem? The motto is not necessarily visible, nor is the “poem.” We argued that the company name Nike functions as the subscriptio or inscriptio, promising victory. The goddess was not, originally, associated with sport shoes, but sport shoes can easily be related to the goddess Nike, patroness of victory. The company’s slogan Just do it is the wellknown motto that it is even implied without being rendered visible by print. From this perspective, we can thus find in the logo an emblematic tripartite structure in which the image dominates the material modality of the media product and the perceiver adds the implied basic medium of word. Conversely, the case of Dürer’s woodcut “The Triumphal Arch” can be unfolded in its tripartite structure with a title, a pictura constructed of many allegorical images, and the subscriptio which as inscriptio reduces the lexicographical function of the basic medium of word to foreground its visual potential. This huge emblematic media product acts as an advertisement of Emperor Maximilian’s divine-like power and victory, and likewise, Alciato’s poverty poem can be considered as an advertisement for the Humanist Movement, always searching for knowledge despite the oppressive economic conditions of scholars. Emblem books are full of examples of illustrated poems explaining the symbolon, or the whole, and pointing to moral commonplaces, but is this enough to make these points into an all-encompassing definition of emblems? What do we lose, if we exclude illustrations without a motto or a poem or non-illustrated poems? Could these examples not be the basis for concepts, such as the ekphrasis, and point to

22

The Anchor and the Dolphin: A History of Emblems

545

further multimodal complexities of the medium? What does it mean to explain the whole? It might originally be related to the eternal Truth, the divine wisdom, and the symbolon. Insisting on this point would exclude applied emblems, where “the whole” might be the success of a print shop in the sixteenth century or a modern sports fashion company. With this in mind, we might ask whether emblems are autonomous media types or rather mosaic-like inlays in other media products. This question could, as we saw it earlier, be answered directly by going back to the roots, ad fontes, and making an etymological study of the word “emblem.” That would relate emblems to the manufacturing of inlaid products. Books with a compactified summary in the form of an emblematic frontispiece would fall into this assumed category. Printer marks or pins on hats do not fall into this category, but Alciato, the “father” of emblems, listed them in a quote we presented earlier. One is tempted to add a prescribing adjective to the word emblem, if we have an object that requires a compromise regarding classification. One example is the applied emblem, and so are the humanistic, moralist, political, alchemistic, and religious, including “spiritual,” “devotional,” or “meditative,” emblems. Again, it is not clear what the “whole” is or how we reach it – not even in the religious context. Other questions should be raised: for instance, are emblems relics from past epochs or artifacts unbound by time? Has there ever been a starting point? Was Alciato the author and the inventor of the genre, or was there more? If Alciato’s idea of emblems differed from the printed result, how much did print technology and the printing business condition or avalanche the development of our understanding of emblems as a media type? The assumption that the genre of emblems died is linked to the understanding of history in terms of epochs. Emblems were originally thought to be products of the Renaissance. Still, we also pointed out similarities to medieval media products. Moreover, can we exclude that emblematic structures exist in modernity? Dividing time into epochs like Middle Ages, Renaissance, and Baroque has been questioned by countless scholars, this model having given way to a new understanding of the long chronological and cultural continua (Enenkel and Ottenheym 2018). The performative and oral culture of the medieval times can be found in the rituals of the Baroque, or in modern festivals, just like the collection of epigrams and emblems continued the tradition of encyclopedic anthologies, such as the Physiologies and the Bestiaries (Blair 2013). The scholarly narrative about the rise, maturity, and decay of the emblem as a media type is influenced by this idea of dividing the flow of time into strictly separated periods or epochs. Unfortunately, this approach prevents modern research from linking the structures from Alciato’s first emblematic poems to modern icons, logos, and perhaps even tattoos. What is called for is a way of thinking about emblems that does not start from a fixed point or predefined form, one that is willing to consider them as multimodal transmediations without clear and distinct beginnings, ends, or defining characteristics, instead conceiving of the emblem as an interpretive approach on the part of the perceiver.

546

H. Fu¨hrer et al.

References Alciato, Andrea. 1531. Emblematum liber. Augsburg: Steyner. ———. 1550.(1996) Emblemata, Lyons, Trans. Betty I. Knott with Introduction by John Manning. Aldershot: Scolar Press. Auslander, Philip. 2012. Digital liveness: A historic-philosophical perspective. PAJ: A Journal of Performance and Art 34 (3): 3–11. Barker, William. 2001. The Adages of Erasmus. Toronto: University of Toronto Press. Barthes, Roland. [1964] 1977. The Rhetoric of the image. In Image-music-text. Trans. S. Heath, 32–51. rpt. London: Wm. Collins Sons and Co. Basile, Gastón J. 2021. Res and Verba: The early humanist translations of natural science and medical texts. Arts et Savoirs [En ligne]. https://doi.org/10.4000/aes.3704., http://journals. openedition.org/aes/3704 Bath, Michael. 2016. Books and buildings: Recursive Emblems in the applied arts. In EMBLEMATICA. An Interdisciplinary Journal for Emblem Studies, ed. Mara R. Wade, 167–193. New York: AMS Press, Inc. Berefeldt, Gunnar. 1969. On symbol and Allegory. The Journal of Aesthetics and Art Criticism 28: 201–212. Blair, Ann. 2013. Revisiting Renaissance Encyclopaedism. In Encyclopaedism from antiquity to the Renaissance, ed. Jason König and Greg Woolf, 377–397. Cambridge: Cambridge University Press. Brown, Andrew J. 1984. The Date of Erasmus’ Latin translation of the new testament. Transactions of the Cambridge Bibliographical Society 8 (4): 351–380. http://www.jstor.org/stable/41154623 Burke, Peter. 1997. History of Allegory. INTI 45: 337–351. Carlisle, Rachel Masters. 2018. From "Art of Memory" to Naturalism: Andrea Alciato and the development of the early modern Emblem Comitatus. A Journal of Medieval and Renaissance Studies 49: 165–187. Carruthers, Mary J. 1998. The Craft of thought: Meditation, rhetoric, and the making of images, 400–1200. Cambridge: Cambridge University Press. Clüver, Claus. 2002. Concrete sound poetry: Between poetry and music. In Cultural functions of intermedial explorations, ed. Erik Hedling and Ulla-Britta Lagerroth, 163–178. New York: Amsterdam. Daly, Peter M. 1979. Literature in the Light of the Emblem: Structural parallels between the emblem and literature in the sixteenth and seventeenth centuries. Toronto/Buffalo/London: University of Toronto Press. Daly, Peter. 1999. Literature in the Light of the Emblem. Toronto: University of Toronto Press. https://doi.org/10.3138/9781442676732. Daly, Peter M. 2016 (2014). The Emblem in early modern Europe: Contributions to the theory of the Emblem. Farnham/Surrey/Burlington, VT: Ashgate. Dees, Jerome. 1986. Recent studies in English Emblem. English Literary Renaissance. 16: 391–420. Dieckmann, Liselotte. 1955. The metaphor of Hieroglyphics in German Romanticism. Comparative Literature 7: 306–312. Dieckmann, Liselotte. 1970. Hieroglyphics;: The history of a literary symbol. Editorial: Washington University Press Diehl, Huston. 1986. Graven images: Protestant Emblem books in England. Renaissance Quarterly 39 (1): 49–66. https://doi.org/10.2307/2861583. Eco, Umberto. 1985. At the roots of the modern concept of symbol. Social Research 52 (2): 383–402. http://www.jstor.org/stable/40970376. Eisendrath, Rachel. 2018. Poetry in a world of things: Aesthetics and Empiricism in Renaissance Ekphrasis. Chicago: University of Chicago Press.

22

The Anchor and the Dolphin: A History of Emblems

547

Elleström, Lars. 2019. Transmediation: Some theoretical considerations. In Transmediations: Communication across media borders, ed. Niklas Salmose and Lars Elleström, 1–13. New York/London: Routledge Routledge Studies in Multimodality. Elleström L. (2021). The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In L. Elleström (Ed.) Beyond Media Borders, Volume 1. Palgrave Macmillan. https://doi.org/10.1007/978-3-030-49679-1_1 Enenkel, Karl A. E. 2019. The invention of the Emblem Book and the transmission of knowledge, ca. 1510–1610. Brill’s studies in art, art history, and intellectual history, ed. Walter Melion. Leiden: Brill. Enenkel, Karl A.E., and Konrad Ottenheym. 2018. Introduction. In The Quest for an appropriate past in literature, art and architecture, ed. Karl A.E. Enenkel and Konrad A. Ottenheym, 1–16. Leiden: Brill. Enenkel, Karl A.E., and J. Smith Paul, eds. 2017. Emblems and the natural world, Intersections: Interdisciplinary studies in early modern culture 50. Leiden: Brill. Engel, William E. 1999. Mnemonic Emblems in the Humanist discourses of knowledge. In Aspects of Renaissance and Baroque symbol theory, 125–142. New York: AMS Press. Erasmus, Desiderius. 1978. On the method of study, de ratione studii ac legendi interpretandique auctores. In Collected works of Erasmus: Literary and educational writings 2: de copia/de ratione studii, ed. C. R. Tompson, trans. and annot. by B. McGregor. Toronto/Buffalo/London: University of Toronto Press. Fabiny, Tibor. 2005. The emblem: Document of bad taste or medium of artistic value? Neohelicon 14 (2): 191–202. https://doi.org/10.1007/bf02094684. Fletcher, Angus J.S. 2010. Allegory without ideas. In Thinking Allegory otherwise, ed. Brenda Machosky, 1st ed., 9–34. Redwood City: Stanford University Press. Fowler, Alastair. 2021. Remembered words. Essays on genre, realism, and Emblems. Oxford University Press. Freeman, Rosemary. 1941. George Herbert and the Emblem Books. The Review of English Studies 17: 150–165. ———. 1948. English Emblem Books. London: Chatto & Windus. Führer, Heidrun and Banaszkiewicz, Bernadette. 2014. The trajectory of ancient ekphrasis. In Akropolis, ed. Alice Jedličková, 45–75. Führer, Heidrun, and Anna Kraus. 2019. Leaving the white cube of Ekphrasis: Gordon MattaClark’s conical intersection. In Transmediations: Communication across media borders, ed. Niklas Salmose and Lars Elleström, 97–11. New York/London: Routledge. Giehlow, Karl. 2015 [1915]. The Humanist Interpretation of Hieroglyphs in der Allegorie der Renaissance. Jahrbuch der Kunsthistorische Sammlungen des Allerhöchsten Kaiserhauses. In the Allegorical Studies of the Renaissance. ed. and transl. by Robin Raybould. Leiden: Brill, 32(1), 1–132. Goeglein, Tamara A. 2007. Early modern Emblems books as memorial sites. The Princeton University Library Chronicle, 69(1), 43–70. https://doi.org/10.25290/prinunivlibrchro.69. 1.0043. Gombrich, E.H. 1971. Personification. In Classical influences in European Culture AD 500–1500, ed. R.R. Bolgar, 247–257. Cambridge: University Press. Graham, David. 2001. Introduction. In An interregnum of the Sign: The Emblematic age in France. Essays in Honour of Daniel S. Russell, ed. David Graham. VII–XIV, Glasgow Emblem Studies, 6. University of Glasgow, I–XIV. Green, Henry. 1870. Shakespeare and the emblem writers: An exposition of their similarity of thought and expression. Preceded by a view of emblem-literature down to A.D. 1616. London: Trübner. Greenblatt, Stephen. 1980. Renaissance self-fashioning: From more to Shakespeare. Chicago: Chicago UP. Gutzwiller, Kathryn J. 1998. Poetic Garlands: Hellenistic Epigrams in context. Berkeley: University of California Press.

548

H. Fu¨hrer et al.

Hagstrum, Jean. 1958. The sister arts: The tradition of literary Pictorialism and English poetry from Dryden to Gray. Chicago: University of Chicago Press. Hämmerle, Tobias E. 2019. Flugblatt-Propaganda zu Gustav Adolf von Schweden. Eine Auswertung zeitgenössischer Flugblätter der Königlichen Bibliothek zu Stockholm. Marburg: Büchner-Verlag. Harms, Wolfgang. 1973. Der Fragmentcharakter emblematischer Auslegungen und die Rolle des Lesers. Gabriel Rollenhagens Epigramme. In Deutsche Barocklyrik: Gedichtinterpretationen von Spee bis Haller, ed. Martin Bircher and Alois M. Haas, 49–64. Bern/München: Francke Verlag. Harper, Anthony J., Ingrid Höpel, and Susan Sirc. 2005. Emblematic tendencies in the art and literature of the twentieth century. Glasgow: Centre for Emblem Studies. Hauser, Gerard. 1995. Between philosophy and rhetoric: Interpositions within traditions. Philosophy and Rhetoric 28 (3): iii–xvii. Heckman, John. 1969. Emblematic structures in Simplicissimus Teutsch. MLN 84: 876–890. Heckscher, William S. 1954. Renaissance Emblems: Observations suggested by some emblembooks in the Princeton University Library. Princeton University Library Chronicle 15: 55–68. Heckscher, William S., and Karl-August Wirth. 1959. Emblem, Emblembuch. Reallexikon zur deutschen Kunstgeschichte, v, fasc. 49, 50, Stuttgart col. 85–228. Henkel, Arthur, and Albrecht Schöne. 1967. Emblemata: Handbuch zur Sinnbildkunst des XVI. Und XVII. Jahrhunderts. Stuttgart: J. B. Metzler. Höltgen, Karl Josef. 1986. Aspects of the Emblem: Studies in the English Emblem tradition and the european context. Kassel: Reichenberger. ———. 1996. Francis Quarles’s Emblemes and Hieroglyphikes: Some historical and critical perspectives. In Emblem books and the telling image, ed. Ayers Bagley et al., 1–28. New York: AMS Press. Howell, A.C. 1946. Res et Verba: Words and things. ELH 13 (2): 131–142. https://doi.org/10.2307/ 2871594. Jakobson, Roman. 2002. Fundamentals of language / by Roman Jakobson ; Morris Halle. - Reprint of the 2., rev. ed. 1971. - Berlin ; New York : Mouton de Gruyter, 2002 ISBN 3-11-017283-6 Jung, Theo. 2014. Zeitgeist in early nineteenth-century political discourse. Contributions to the History of Concepts 9 (1): 24–49. https://doi.org/10.3167/choc.2014.090102. Kobson, Roman. 2002. Fundamentals of language / by Roman Jakobson; Morris Halle. - Reprint of the 2., rev. ed. 1971. - Berlin ; New York : Mouton de Gruyter. ISBN 3-11-017283-6 Kusler, Agnes. 2017. Marginalia towards the reconstruction of Alciato’s concept of the Emblem. In Emblems and Impact. Vom Zentrum und Peripherie der Emblematik, ed. Ingrid Hoepel and Simon McKeown, 321–376. Cambridge: Cambridge Scholars Publishing. Lars, Elleström. 2021. The Modalities of Media II: An expanded model for understanding intermedial relations. In Beyond media borders, ed. Lars Elleström, vol. 1, 4–92. Palgrave Macmillan. https://doi.org/10.1007/978-3-030-49679-1_1. Laurens, Pierre, and Florence Vuilleumier. 1993. De l’archéologie à l’emblème: la genèse du Liber Alciati. Revue de l’Art 101: 86–95. Lund, Hans. 2002. Medier in samspel. In Intermedialitet: ord, bild och ton i samspel, ed. Hans Lund, 9–22. Lund: Studentlitteratur. Manning, John. 1990. Whitney’s “choice of Emblems”: A reassessment. Renaissance Studies 4: 155–200. ———. 2002. The Emblem. London: Reaktion Books. McLuhan, Marshall. 1962. The Gutenberg Galaxy: The making of typographic man. Toronto: University of Toronto Press. Miedema, Hessel. 1968. The Term Emblema in Alciati. Journal of the Warburg and Courtauld Institutes 31: 234–250. Milnor, K. 2011. Between Epigraph and Epigram: Pompeian Wall writing and the Latin Literary tradition. Ramus 40 (2): 198–222. https://doi.org/10.1017/S0048671X00000400. Mödersheim, Sabine. 1994. “Domini doctrina coronat”: Die geistliche Emblematik Daniel Cramers (1568–1637). Frankfurt am Main: Lang.

22

The Anchor and the Dolphin: A History of Emblems

549

———. 1999. Skin deep – Mind deep. Emblematics and modern Tattoos. In Emblems from Alciato to the Tattoo. Selected papers of the Leuven International Emblem Conference 18–23 August, 1996, ed. Peter M. Daly, John Manning, and Marc van Vaeck, 309–333. Turnhout: Brepols. ———. 2010. Visual culture and the German middle ages. Monatshefte 99 (2): 223–225. Nelson, Janet L. 2007. The dark ages. History Workshop Journal: 191–201. Nelson, Bradley J. 2010. The persistence of presence: Emblem and ritual in Baroque Spain. Toronto: University of Toronto Press. Norton, Glyn P. 1999. Introduction. In The Cambridge history of literary criticism, ed. G.P. Norton, 1 – 2 2 . C a m b r i d g e : C a m b r i d g e U n i v e r s i t y P r e s s . h t t p s : / / d o i . o rg / 1 0 . 1 0 1 7 / CHOL9780521300087.002. Ohly, Friedrich. 1959. Vom geistigen Sinn des Wortes im Mittelalter. Zeitschrift für deutsches Altertum und deutsche Litteratur 83: 1–23. Ong, Father Walter J. 1959. From Allegory to Diagram in the Renaissance Mind: A Study in the Significance of the Allegorical Tableau. JAAC 17: 423–440. Panofsky, Erwin. 1948. Albrecht Dürer. Princeton: Princeton University Press. ———. 1955. Meaning in the visual arts. Garden City: Doubleday Anchor Books. Peil, Dietmar. 1979. Zur Diskussion über “angewandte Emblematik”. Germanisch-Romanische Monatszeitschrift. 29: 200–207. ———. 1986. Emblematische Fürstenspiegel im 17. und 18. Jahrhundert: Saavedra — Le Moyne — Wilhelm. Frühmittelalterliche Studien, Gryter: Berlin/New York, 20: 54–92. ———. 1992. Emblem types in Gabriel Rollenhagen’s Nucleus emblematum. Emblematica 6: 255–283. Praz, Mario. 1961. Piranesi: magnificenza di Roma: [nelle vedute di Giovanni Battista Piranesi] Praz, Mario. 1964 [1939]. Studies in seventeenth-century Imagery. Rome: Edizioni di Storia e Letteratura, ed., London: Warburg Institute. Probes, Christine M., and Mödersheim, Sabine, eds. 2014. The art of persuasion. Emblems and Propaganda. Glasgow Emblem Studies. Russell, Daniel. 1995. Emblematic structures in Renaissance French culture. Toronto: University of Toronto Press. Saunders, Alison. 1989. The sixteenth-century French Emblem: Decoration, diversion, or didacticism. Renaissance Studies 3 (2): 115–133. JSTOR, http://www.jstor.org/stable/24409371. Accessed 16 Apr 2023. ———. 2016. More French emblematic predecessors, godly and amorous. EMBLEMATICA. An Interdisciplinary Journal for Emblem Studies 22: 69–106. ———. 2002. Emblem und Emblempoetik. Historische und Systematische Studien. Berlin: Erich Schmidt Verlag. Scholz, Bernhard, and Anja Wolkenhauer, eds. 2018. Typographorum emblemata. The Printer’s Mark in the context of early modern culture. Berlin: De Gryter. Schöne, Albrecht. 1964. Emblematik und Drama im Zeitalter des Barock. München: C. H. Beck. Sgarbi, Marco. 2014. The Italian mind: Vernacular logic in Renaissance Italy (1540–1551). Medieval and Renaissance Authors and Texts. Leiden: Brill. Shrieves, Katherine Irene. 2013. Animate texts. Hieroglyphical reading practices in early modern England, 1564–1658. Diss Chapel Hill. Struck, Peter T. 2010. Allegory and ascent in neoplatonism. In The Cambridge companion to Allegory, ed. R. Copeland and P.T. Struck, 57–70. Cambridge University Press. Taylor, Andrew. 2014. Introduction: The translations of Renaissance Latin. Canadian Review of Comparative Literature 41 (4): 329–353. Vickers, Brian. 1988. Rhetoric and poetics. In The cambridge history of renaissance philosophy, ed. Charles B. Schmitt, Quentin Skinner, and Eckhard Kessler, 715–745. Cambridge: Cambridge University Press. ———. 2003. English Renaissance Literary Criticism. Oxford: Oxford University Press.

550

H. Fu¨hrer et al.

Volkmann, Ludwig. 1923. Bilderschriften der Renaissance. Hieroglyphik und Emblematik in ihren Beziehungen und Fortwirkungen, 1923 (Hieroglyphic, Emblem, and Renaissance Pictography, 2018). Wade, Mara R. 2010–2015. What is an Emblem? University of Illinois at Urbana-Champaign, 2010–15. http://emblematica.grainger.illinois.edu/help/what-emblem. Waswo, Richard. 1999. Theories of Language. In The Cambridge History of Literary Criticism, 3 The Renaissance, ed. Glyn P. Norton, 25–35. Cambridge, New York and Melbourne: Cambridge UP. Webb, R. 2009. Ekphrasis, imagination and persuasion in ancient rhetorical theory and practice. Farnham. Wesseling, Ari. 2002. Dutch proverbs and expressions in Erasmus’ Adages, colloquies, and letters. Renaissance Quarterly, 55: 1, 81–147. JSTOR, 10.2307/1512533. Wilcox, Donald J. 1987. The measure of times past. Pre-Newtonian Chronologies and the Rhetoric of relative time. Chicago and London: University of Chicago Press. Wittkower, Rudolf. 1968 (1972). Hieroglyphics in the early renaissance. In Developments in the early renaissance: Papers of the second annual conference of the center for medieval and early renaissance studies. State University of New York at Binghamton, 4–5 May 1968, ed. Bernard S. Levy. 58–97. Wolkenhauer, Anja. 2018. Sisters, or mother and daughter? The relationship between Printer’s Marks and Emblems during the first hundred years. In Typographorum emblemata. The Printer’s Mark in the context of early modern culture, ed. Anja Wolkenhauer and Bernhard Scholz, 3–28. Berlin, NY: De Gryter.

The Age of Wonder and Entertainment: An Introduction to Intermedial Networks in Baroque Culture

23

Massimo Fusillo and Mattia Petricola

Contents Introduction: What Does “Baroque” Mean? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Framing Baroque Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Counter-Reformation and Monarchical Absolutism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A New Knowledge System in a New Cosmos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An Age of Conflict and Contradictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Understanding Baroque Aesthetics across Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Excess and Wonder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The “Ornamental Impulse” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Self-Reflexivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . “Works of Artifice” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performativity and Theatricality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Painting and Architecture: Two Roman Ceilings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pietro da Cortona’s Allegory of Divine Providence and Barberini Power . . . . . . . . . . . . . . . . . Giovan Battista Gaulli’s Triumph of the Name of Jesus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Courtly Feast and the Birth of Modern Entertainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Double Marriage in Mantua . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Pleasures of the Enchanted Island . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Theater and the Rise of the Opera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The First Operas and Divas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tragicomedy and the Disruption of Traditional Genres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rethinking Genres Beyond the Tragicomedy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Meta-Theater and Don Juan as a Symbol of Baroque Intermediality . . . . . . . . . . . . . . . . . . . . . . Conclusion: Baroque Media Products After the Baroque . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

552 553 554 555 556 557 557 557 558 559 560 561 562 563 564 565 566 568 568 569 571 572 573 574

The authors discussed, planned, and structured this chapter together. Massimo Fusillo then focused on writing sections “A Double Marriage in Mantua” and “Theater and the Rise of the Opera” and Mattia Petricola focused on the other sections. M. Fusillo (*) · M. Petricola Department of Human Sciences, University of L’Aquila, L’Aquila, Italy e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_27

551

552

M. Fusillo and M. Petricola

Abstract

In its broadest sense, the notion of “baroque” is used today to define the period in the history of European culture going from the end of the Renaissance to the dawn of the Enlightenment. This chapter attempts to trace some paths through the baroque mediascape in order to give the reader an idea of how it can be investigated from the perspective of intermedial studies. More specifically, this chapter’s aim is twofold. On the one hand, it interrogates artistic phenomena, media products, performances, and intermedial relations that are specific to the baroque mediascape; on the other hand, it shows how these products contributed to a significant extent to shaping the mediascape in which we ourselves are immersed. Section “Framing Baroque Culture” investigates how baroque culture emerged in Europe and what its defining elements are. Section “Understanding Baroque Aesthetics across Media” explores the main tenets of baroque aesthetics (excess, wonder, ornamentation, self-reflexivity, artifice, and theatricality) and the emotional responses that baroque media products strived to elicit. Section “Painting and Architecture: Two Roman Ceilings” focuses on the relation between painting and architecture in two painted ceilings – Pietro da Cortona’s Allegory of Divine Providence and Barberini Power and Giovan Battista Gaulli’s Triumph of the Name of Jesus. Section “The Courtly Feast and the Birth of Modern Entertainment” analyzes courtly feasts and festivals as the quintessential form of entertainment of the baroque age. Section “Theatre and the Rise of the Opera” is centered on intermediality in baroque theater, with a specific focus on opera, tragicomedy, and metadrama. The conclusion asks the question: how does one investigate the survival, reemergence, citations, and echoes of the Baroque from the half of the eighteenth century onwards? Keywords

Intermediality · Baroque · Scopic regime · Entertainment · Performativity

Introduction: What Does “Baroque” Mean? In its broadest sense, the notion of “baroque” is used today to define the period in the history of European culture going from the end of the Renaissance to the dawn of the Enlightenment – that is, from the late sixteenth to the early eighteenth century. This definition, however, is the result of a complex chain of intermedial transfers, and its application in the humanities has been debated since at least the nineteenth century (Johnson 2012). According to the classical etymology of the term, “baroque” comes from the Portuguese adjective barroco, which was used to describe “the bizarre irregular natural pearls so highly prized by collectors [. . .] both for their weird beauty and great rarity” (Calloway 2000: 7). Eighteenth-century art critics later used it as a derogatory term to describe many seventeenth-century works of art, which they saw as “extravagant, willfully bizarre, strained, or given to emotional excess” (Hoxby 2019: 517). The word “baroque” still partially retains this negative connotation, as it

23

The Age of Wonder and Entertainment: An Introduction to. . .

553

is often used to describe something as excessively ornamented or grotesquely convoluted. In the mid-nineteenth century, historians like Jacob Burckhardt and Wilhelm Lübke began to use it in the context of academia to describe the degeneration of post-Renaissance art. It lost its derogatory tone and entered the lexicon of art history – more specifically, of the history of architecture – only 30 years later, thanks to the work of Heinrich Wölfflin (1994). He saw the Baroque as a style “characterized by painterliness, grandness, massiveness, and movement” (Lyons 2019: 1). Over the course of the twentieth century, the notion of baroque has been progressively adopted ever more widely in the visual and performing arts, literary studies, and cultural history (Maravall 1986 represents a milestone in this last respect). It has often been employed as a complement to – or in competition with – other historiographical concepts such as those of âge classique (France), Späthumanismus (Germany), and English Renaissance. Furthermore, it has been investigated by philosophers like Walter Benjamin (2009) and Gilles Deleuze (1993). Given these premises, it would be impossible to give a detailed account of the complex intermedial networks that made up the baroque mediascape within the limits of this chapter. The following pages will then trace some paths through this mediascape to give the reader an idea of how it can be investigated from the perspective of intermedial studies. More specifically, this chapter’s aim is twofold. On the one hand, it interrogates artistic phenomena, media products, performances, and intermedial relations specific to the baroque mediascape; on the other hand, it attempts to identify their contemporary “descendants,” thus showing how these products contributed significantly to shaping the mediascape in which we ourselves are immersed. Section “Framing Baroque Culture” investigates how baroque culture emerged in Europe and what its defining elements are. Section “Understanding Baroque Aesthetics across Media” explores the main tenets of baroque aesthetics (excess, wonder, ornamentation, self-reflexivity, artifice, and theatricality) and the emotional responses that baroque media products strived to elicit. Section “Painting and Architecture: Two Roman Ceilings” focuses on the relation between painting and architecture in two painted ceilings – Pietro da Cortona’s Allegory of Divine Providence and Barberini Power and Giovan Battista Gaulli’s Triumph of the Name of Jesus. Section “The Courtly Feast and the Birth of Modern Entertainment” analyzes courtly feasts and festivals as the quintessential form of entertainment in the baroque age. Section “Theatre and the Rise of the Opera” is centered on intermediality in baroque theater, with a specific focus on opera, tragicomedy, and metadrama. The conclusion asks the question: how does one investigate the survival, reemergence, citations, and echoes of the baroque from the half of the eighteenth century onwards?

Framing Baroque Culture As we have seen, the definition of the word “baroque” is open to an impressively wide range of interpretations. In the context of this chapter, the Baroque (with a capital B) will be considered not simply as a literary and artistic style but rather as a

554

M. Fusillo and M. Petricola

worldview or, to use Michel Foucault’s (2012) terminology, as an episteme. Thus, in order to investigate the intermedial networks that shaped the Baroque, it is imperative to understand how this worldview and its defining elements emerged. The progressive shift from the Renaissance age to the Baroque was catalyzed by four major sociohistorical transformations: the Catholic Counter-Reformation, the rise of political absolutism, a new understanding of cosmology in both astronomy and philosophy, and a general reorganization of the very concept of “knowledge.” Each of these transformations had a palpable impact on virtually every aspect of the European mediascape, as well as on numerous media products and practices in European colonies, particularly in Latin America.

Counter-Reformation and Monarchical Absolutism In 1517, with the publication of Martin Luther’s Ninety-five Theses and the emergence of the Protestant Reformation, the Catholic Church’s hegemony over Europe was dramatically shattered. As a consequence, the Catholic Church reaffirmed its religious, social, and political authority through a process known as the CounterReformation, which found its material expression in the Council of Trent (1545–1563), during which a number of Catholicism’s fundamental tenets were redefined. As we will see in section “Understanding Baroque Aesthetics across Media,” the monumental task of making this new order visible to the world fell on sacred art and architecture. The close association between Catholicism and the style that we call baroque was already well established in seventeenth-century culture, especially among anti-Catholic believers. In his Milton and the Baroque, Roston (1980) provides a fascinating insight on this issue from an intermedial perspective: the author of Paradise Lost – who was a Puritan – represents Hell’s Pandemonium as a “devilish [. . .] baroque structure, such as he had seen on his visit to Italy in 1638” (32): Built like a Temple, where Pilasters round Were set, and Doric pillars overlaid With Golden Architrave; nor did there want Cornice or Frieze, with bossy Sculptures grav’n; The Roof was fretted Gold. Not Babylon Nor great Alcairo such magnificence Equalled in all thir glories, to inshrine Belus or Serapis thir Gods, or seat Thir Kings, when Egypt with Assyria strove In wealth and luxury. (Milton 2003, book I, v. 713–22)

The Catholic church was not the only institution of the time concerned with exercising power and displaying it through the arts. European monarchies progressively developed into rigidly centralized structures with control over vast colonial

23

The Age of Wonder and Entertainment: An Introduction to. . .

555

empires, thus amassing enormous amounts of wealth, resources, and luxury goods. As a result, royal courts became incredibly large, rich, and lavish. The Baroque is an age of massive princely palaces, from Versailles in France to El Escorial in Spain through the Winter Palace in Austria, among many others. As we will see in section “Painting and Architecture: Two Roman Ceilings,” royal courts and palaces represented complex media ecosystems in which writers, artists, performers, architects, and engineers were tasked with celebrating their sovereign’s power and entertaining the court.

A New Knowledge System in a New Cosmos The rise of Counter-Reformation Catholicism and absolutist monarchies ran parallel with a profound cultural crisis brought about by a radical reconsideration of how the physical universe works. The Ptolemaic understanding of the universe – in which the Earth stands, fixed and immobile, at the center of the universe, with all the other stars and planets revolving around it – had already begun to collapse after the publication of Copernicus’s On the Revolutions of the Heavenly Spheres (De Revolutionibus Orbium Coelestium) in 1543. At the beginning of the seventeenth century, the Copernican revolution permeated European culture in general. The invention of the telescope brought a whole new level of depth to the study of celestial bodies, and, in 1632, Galileo published his famous Dialogue on the Ptolemaic and Copernican Systems. This revolution in the field of astronomy, in turn, triggered important transformations in philosophical thought, as it became increasingly difficult to make sense of the very notions of space and time in anthropocentric and theological terms. In other words, the consequences of embracing Copernicanism “involved not merely a new astronomical theory but a basic reappraisal of the nature of the universe and man’s place within it” (Roston 1980: 11). French scientist and philosopher Blaise Pascal perfectly encapsulated the anxieties that such a reappraisal raised among European intellectuals in his famous meditations on the “eternal silence” of the “infinite spaces” of the cosmos (Pascal 2008: 73): When I consider the short span of my life absorbed into the preceding and subsequent eternity, the small space which I fill and even can see, swallowed up in the infinite immensity of spaces of which I know nothing and which knows nothing of me, I am terrified, and surprised to find myself here rather than there, for there is no reason why it should be here rather than there, why now rather than then. Who put me here? On whose orders and on whose decision have this place and this time been allotted to me? (Pascal 2008: 26).

To complicate matters even further, the Copernican revolution represented only one, albeit momentous, process within a more general metamorphosis of the European episteme. Until the Renaissance, human knowledge resided

556

M. Fusillo and M. Petricola

within books and was primarily founded on the authority of ancient authors. With the elaboration of empiricism and the scientific method, the fundamental locus of knowledge began to shift from books and the past to one’s direct experience of the world in the present and the analysis of actual objects, artifacts, and phenomena. The origins of natural history, art collecting, and museography as we know them today can be found precisely in this epistemological shift.

An Age of Conflict and Contradictions Nevertheless, it is important to keep in mind that the older foundations of the European episteme did not simply disappear to make room for these new paradigms of knowledge. On the contrary, the baroque age was shaped by violent and frequent collisions between mutually irreconcilable strategies to make sense of the world. As John D. Lyons argues in the very first page of the Oxford Handbook of the Baroque, “[o]n one hand, there was a commitment to rule, reason, classification, correct perception, and rigid structure. On the other hand, there was a fascination with disorder, with the irrational, with all that does not fit into existing classifications, with illusion, and with rebellion against order” (2019: 1). According to Michel Foucault, the character of Don Quixote perfectly embodies the contradictions of seventeenth-century culture, in which the relations between “words” and “things,” once founded on resemblances, similitudes, and allegories, are collapsing irreversibly: Don Quixote is a negative of the Renaissance world; writing has ceased to be the prose of the world; resemblances and signs have dissolved their former alliance; similitudes have become deceptive and verge upon the visionary or madness; things still remain stubbornly within their ironic identity: they are no longer anything but what they are (Foucault 2012: 53).

The Baroque was the age of René Descartes, Francis Bacon, Galileo Galilei, Johannes Kepler, and the founding of the first academies of sciences – The Royal Society of London and the French Académie des sciences, respectively, in 1660 and 1666. Yet it was also the age of exuberant geniuses like Athanasius Kircher and Giuseppe Arcimboldo; the age in which the novel was a newly born genre of low cultural value, and poems were often indecipherable without a profound knowledge of Europe’s best-known emblem book, Cesare Ripa’s Iconologia; and the age of the vertiginous allegories of Giambattista Marino, Luis de Góngora, and the Mexican poet-nun Juana Inés de la Cruz. But how did all these cultural transformations, revolutions, and contradictions concretely shape the intermedial networks of the baroque mediascape? To answer this question, one must explore the central tenets of baroque aesthetics and the emotional responses baroque media products strived to elicit.

23

The Age of Wonder and Entertainment: An Introduction to. . .

557

Understanding Baroque Aesthetics across Media Excess and Wonder The Baroque is usually seen as a “culture of excess” – or rather, to borrow from the title of Stephen Calloway’s monograph (2000), “the culture of excess” (emphasis ours). In the North Atlantic world, baroque art and literature are generally perceived as overwhelmingly rich, grand, convoluted, and exaggerated. While these effects are undeniably central to baroque aesthetics, it is essential not to confuse excess and exaggeration with unnecessary overabundance or gratuitous obscurity. In fact, in baroque culture, all those elements that today’s readers and viewers might perceive as needlessly decorative played a crucial role in constructing the aesthetic experience of a media product. Anthony Cascardi describes the “standard repertoire of features” that are commonly associated with the Baroque as including “exaggeration, exuberance, [. . .]; contrast, opposition, and tension”; all these characteristics “typically sit beside ornamentation, and sometimes characterize it as well (since not just any kind of ornamentation would be described as baroque)” (2019: 453). More specifically, Cascardi distinguishes between “figures of force (intensity, exaggerated tensions, exuberance)” and “figures of form (doubling, opposition, repetition)” (2019: 453) as the two main sets of strategies that define baroque aesthetics. All these “figures” are primarily mobilized to elicit a very specific aesthetic and emotional response: wonder. The equivalent of the Spanish notion of maravilla, the Italian meraviglia, the French merveilleux, and the German Wunder, baroque wonder indicated a class of emotions – astonishment, surprise, confusion, admiration, amazement, and rapture, among many others – elicited by unexpected metamorphoses, frantic accumulations, spectacular effects, ingenious hyperboles (Johnson 2010), and the encounter with something new, unfamiliar, and bizarre. In literature, wonder was triggered through the use of “wit” (Italian: ingegno; Spanish: ingenio), that is, the ability to create unexpected associations between words, mental images, and concepts. The application of such an ability – rigorously theorized in such essays as Baltasar Gracián’s Wit and the Art of Inventiveness (Agudeza y arte de ingenio, 1648) and Emanuele Tesauro’s Aristotle’s Lens (Il cannocchiale aristotelico, 1654) – characterized a number of seventeenth-century literary movements and styles across Europe, from marinism to gongorism (initiated, respectively, by Italian poet Giambattista Marino and Spanish poet Luis de Góngora), and from John Lily’s euphuism to the southern European concettismo.

The “Ornamental Impulse” In more general terms, baroque media products strived to provoke wonder by following what Michael Yonan (2019) calls an “anti-classical, anti-rational

558

M. Fusillo and M. Petricola

ornamental impulse” (415): “The sumptuously decorated churches of Bernini and Borromini bedazzle the eye with complex manipulations of space, gilded surfaces, inlaid marble, and artworks of every kind. The gorgeously verbose writings of Cervantes and Milton take their readers on journeys of vivid lexical abundance, while the soundscapes of Bach, Vivaldi, and Handel thrill the ear with sounds of beautiful intricacy. These and other well-known examples of baroque decorative generosity have made the term virtually synonymous with the ornamental” (2019: 409). This “ornamental impulse” is probably the element that has contributed the most to the discrediting of baroque media products from the mid-eighteenth to the early twentieth century. In 1929, Italian philosopher and critic Benedetto Croce famously wrote that “art is never baroque, and the baroque is never art” (Croce 1993, quoted in Hoxby 2019: 517). Despite this, baroque ornamentation is understood today not as something “added to” a specific media product to make it more “decorative,” but as an essential vehicle of meaning and an integral part of the aesthetic experience. In the words of Cascardi, “the intricate syntax of Góngora’s Soledades (Solitudes), which are composed in the strophic form of the silva (a term related to selva, “forest”), is not a decoration added to a poem about a wanderer but shapes the essence of what it means to be lost and searching for direction. In baroque music, the contrapuntal organization of the fugue (as in Bach’s Ricercar a 6) is not just a mechanism for the extensive ornamentation of a theme. Still, it constitutes the very essence of the composition” (2019: 450).

Self-Reflexivity In the visual and performing arts, the search for superb compositions and the presence of an “ornamental impulse” were undoubtedly two of the main tendencies that animated baroque aesthetics. There is, however, at least one other tendency that deserves to be mentioned here. As a result of the cultural crises mentioned above and the significant technological advancements in the production of mirrors, artists and dramatists came to question the very nature of the processes governing perception, representation, and media. Baroque images and plays are thus often intensely selfreflexive – nowhere more so than in Diego Velázquez’s painting Las Meninas, as Michel Foucault brilliantly demonstrates in the first chapter of his book The Order of Things (2012). The intersection of baroque artists’s interest in wonder and selfreflexivity generated a pervasive fascination for optical illusions, anamorphoses, and trompe-l’œil techniques. Essays like Jean-François Niceron’s Curious Perspective (La Perspective curieuse, 1638) were devoted to the study of such techniques. According to Lyons, “[t]o produce an image that seems to be the thing itself, and then to let viewers realize that they have been duped is to astound them and make them admire the work of art” (2019: 8). Christine Buci-Glucksmann goes as far as to argue that “one could [. . .] define the baroque eye as an anamorphic gaze. In its somewhat instinctual appetite for the marvelous, for artifice, the unexpected, and the

23

The Age of Wonder and Entertainment: An Introduction to. . .

559

distortion of appearance but also principally in its attention to constructing artistic artifacts, works of artifice, displaying a law and its variations, its transgressions, its points of view” (2013: 9).

“Works of Artifice” Creating awe-inspiring “works of artifice” (to borrow Buci-Glucksmann’s expression) often required a close interrelationship between visual arts, performative skills, and outstanding engineering abilities, as shown by the enormous success of automata and machine theater plays all across Europe. According to Jessica Keating, “in the automaton, the mechanical arts and the liberal arts, or to put it another way, techne and theoria, are literally fused together” (2019: 201). Produced mostly by German artist-engineers, automata were considered entertaining artifacts as well as highly prized collector’s objects. From an intermedial perspective, one of the most fascinating aspects of automata is that their aspect was often based on other artworks. By examining three automata of the flagellation of Christ “modelled after a renowned [. . .] small bronze flagellation group attributed to the Bolognese sculptor Alessandro Algardi” (Keating 2019: 186), Keating argues that automata’s “particular brand of animation fostered new spectator practices,” since they “supplemented the artwork from which they were derived” and “elaborated upon an action or a narrative that was only implied in their static models” (187). Furthermore, these art objects could also have played a significant role as catalyzers of philosophical (self-) reflection, as their success coincided with the affirmation of the mechanical philosophies elaborated by Robert Boyle, Thomas Hobbes, Renée Descartes, and Julien de La Mettrie (Kang 2011; Keating 2019). At the other end of the spectrum from the automaton’s portable size and intimate performance were the gigantic contrivances of machine theater, an expression which indicates “transnational multimedia performances” centered on “scene changes and special effects – such as flights, metamorphoses, magic and supernatural appearances/disappearances from above and below the stage, and lighting and sound effects – all achieved through technology” (Visentin 2019: 388). Machine theater plays were performed in royal courts as well as in public theaters. Productions such as the opera Orpheus and the ballet The Marriage of Thetis (see Fig. 1) for example, were staged by Giacomo Torelli at the Palais-Royal in Paris, respectively in 1647 and 1654. On the other hand, François de Chapoton’s Orpheus’s Descent to the Underworld (La descente d’Orphée aux enfers) was performed at the Theatre du Marais in Paris in 1654 (Visentin 2019: 395–396). As complex, intermedial works of mechanical ingenuity whose performances astonished and entertained at the same time, automata and machine theater plays could be considered the baroque representants of a class of media products still very present in today’s mediascape, from the astounding special effects of Broadway theater productions to a symbol of 1980s Americana like ShowBiz Pizza Place’s animatronic show.

560

M. Fusillo and M. Petricola

Fig. 1 One of the theater sets designed by Giacomo Torelli for the ballet The Marriage of Thetis (Les Noces de Thétis), engraved by Israël Silvestre. (From Décorations et machines aprestées aux nopces de Tetis, Paris, 1654)

Performativity and Theatricality It may be worth remarking, at this point, that matters related to performativity and theatricality permeated the baroque mediascape well beyond the confines of traditional stages and theatres. From Villa Aldobrandini in Frascati and Villa d’Este in Tivoli to Louis XIV’s Versailles, baroque villas and gardens have often exploited the performative nature of water features like fountains, jets, and cascades. In the words of Stephanie Hanke, water “activates different senses simultaneously and is particularly suited for the synesthetic involvement of a visitor, who is much more than a mere ‘beholder’” (2019: 88). Such involvement reached its maximum intensity in the theatres d’eau (water theaters), quintessentially baroque media products at the intersection of theater, visual arts, water engineering, and garden design. They were “water showplaces, in which this element is presented in its various visual and acoustic forms, often in a dialectical relation with an elaborate sculptural ensemble” (Hanke 2019: 100). Through the theatricalization of water, gardens were thus transformed into places of “dynamic entertainment” (Spielmann 2019: 817). In the cultural history of landscape architecture, water theatres are the obvious ancestors of the Bellagio Fountain Show in Las Vegas.

23

The Age of Wonder and Entertainment: An Introduction to. . .

561

According to Roston, the baroque “tendency to theatricality which broke down the traditional divisions between the arts” (1980: 3) is epitomized by the Cornaro Chapel in Santa Maria della Vittoria in Rome, “where the ecstasy of St. Teresa was presented at the height of her sensuous vision as a stage piece witnessed by noblemen seated in boxes at either side” (Roston 1980: 3). One of the most complex examples of baroque intermediality, the chapel combines sculpture, metalwork, lighting effects – the Ecstasy is illuminated from above by a hidden window – fresco, and stucco to create the sense of a sacred event happening in real time. By admiring the Ecstasy, onlookers join the Cornaro family’s noblest members, sculpted on the sidewalls of the chapel, as audience members at a collective show that brings together the sacred and the profane, the living and the dead.

Painting and Architecture: Two Roman Ceilings In a 1988 essay entitled Scopic Regimes of Modernity, historian and critical theorist Martin Jay coined the expression “Cartesian perspectivalism” to define “what is normally claimed to be the dominant, even totally hegemonic, visual model of the modern era, that which we can identify with Renaissance notions of perspective in the visual arts and Cartesian ideas of subjective rationality in philosophy” (Jay 2009: 4). According to Jay, space is conceptualized in Cartesian perspectivalism – a modality of vision which still influences a number of aspects of today’s mediascape – as “geometrically isotropic, rectilinear, abstract, and uniform” (4). In other words, it is as though the Renaissance-Cartesian mind perceived space through a single, disembodied, static godlike eye. However, despite its hegemony, the adoption of such a scopic regime has not been the only option available in the North Atlantic world. Indeed, Jay spends the second half of his essay defining alternative scopic regimes that run counter to Cartesian perspectivalism. Among them, we find what is defined as a “baroque form” or “style”: “[i]n opposition to the lucid, linear, solid, fixed, planimetric, closed form of the Renaissance, or as Wölffiin later called it, the classical style, the baroque was painterly, recessional, soft-focused, multiple, and open” (16). By drawing on Buci-Glucksmann’s definition of the anamorphic gaze (mentioned in section “An Age of Conflict and Contradictions”), Jay affirms that “the baroque self-consciously revels in the contradictions between surface and depth, disparaging as a result any attempt to reduce the multiplicity of visual spaces into any coherent essence” (17). Furthermore, he highlights the importance of an intermedial approach to the “baroque visual experience,” whose “strongly tactile or haptic quality [. . .] prevents it from turning into the absolute ocularcentrism of its Cartesian perspectivalist rival” (17). This section will put Jay’s theory into practice by analyzing two works that exemplify the deeply embodied and performative qualities of baroque visuality through their use of perspective and optical illusions in relation to the vast architectural spaces of which they are part. They are among seventeenth-century Rome’s most famous ceiling frescos: Pietro da Cortona’s Allegory of Divine Providence and Barberini Power (Il Trionfo della Divina Provvidenza e il compiersi dei suoi fini

562

M. Fusillo and M. Petricola

sotto il pontificato di Urbano VIII, 1632–39), and Giovan Battista Gaulli’s Triumph of the Name of Jesus (Trionfo del nome di Gesù, 1676–79).

Pietro da Cortona’s Allegory of Divine Providence and Barberini Power In 1623, Maffeo Vincenzo Barberini was elected pope and took the name of Urban VIII. In order to display their newly acquired prestige as one of Rome’s most important families, the Barberinis needed a new residence, one more appropriate to their social status. They thus commissioned the construction of Palazzo Barberini, a palace on the side of the Quirinal Hill that today houses the National Gallery of Ancient Art (Galleria Nazionale d’Arte Antica). Urban VIII hired some of the foremost artists of the time to decorate the palace’s interiors. The most important and prestigious work was assigned to Pietro da Cortona, commissioned to paint the gigantic ceiling of the main hall (the salone) on the piano nobile (main floor). The work began in 1632 and took 7 years to complete. The fresco represents the triumph of Divine Providence under Urban VIII’s pontificate, a complex allegorical glorification of the Barberini pope himself. The fresco’s incredibly rich iconographic program was based on The Election of Pope Urban VIII (L’elettione di Urbano Papa VIII, 1628), an allegorical and encomiastic poem by Francesco Bracciolini dell’Api. In an extravagant blend of contemporary history, pastoral fable, epic narration, and classical myth, the poem describes an allegorical battle between vices and virtues in Rome between Pope Gregory XV’s death and the election of his successor Urban VIII, who finally brings peace to the city. Pietro da Cortona gives visual life to these allegories by creating the illusion of a vast, open space that exists beyond the ceiling, populated with an incredible number of figures and scenes. With their imposing presence and lavish decoration, the painted impost blocks and lintels divide the ceiling into five panels while creating the illusion of real architectural elements. In the central panel, the personifications of Divine Providence and Immortality present the Barberini family with a crown made of stars. However, Cortona does not portray any member of the family. Through an intermedial feat of artistic genius that has fascinated art historians for centuries, we find in their place three large bees that represent the family’s emblem. Four mythical episodes surround the central panel, each painted from a different point of view. The act of viewing the fresco is thus transformed into a “dynamic process” in which, as Angela Ndalianis argues, “[t]he Renaissance ideal of a perspectivally guided representation [. . .] is replaced by a baroque concern with complex, dynamic motion and multiple perspectives that are dependent on the position of the viewer in relation to the work” (2003: 360). One of the scenes, for example, depicts the goddess Minerva chasing the Giants – an allegory of wisdom’s triumph over ignorance – adopting an accentuated foreshortened perspective (scorcio) to give the illusion of the Giants actually falling from the ceiling. Ndalianis also notes in this same area of the fresco that “[i]n the cornice that intersects with Minerva and the Giants [. . .] numerous figures and swirling clouds tumble and float in front of and behind the painted stucco frames

23

The Age of Wonder and Entertainment: An Introduction to. . .

563

with the result that the narrative from one panel literally spills into the narrative of another” (2003: 361). In a typically baroque gesture, Cortona blurs the boundaries between the representation and the frame that “contains” it, thus engaging in a metareflexive dialogue between painting and (illusory) architecture. Cortona’s ceiling does not only epitomize how intermedial networks shaped the art of painting; it is also a fascinating example of how such overwhelmingly lavish encomiastic works were remediated to function as tools of both symbolic and political power. In 1642, 3 years after the fresco’s completion, an engraving of Cortona’s work was included in an encomiastic volume by Girolamo Tezi entitled Description of the Barberini Palace on Quirinal Hill (Aedes Barberinae ad Quirinalem Descriptae). Considered to be one of the first art books ever created (Tetius 2005: 119–158), the Description was a luxury good meant to display the splendor and authority of the Barberini court and to be acquired by wealthy members of courts all across Europe. However, Palazzo Barberini’s salone was open not just to nobles and high-level delegations but “to any person who could make a presentable appearance at the palace gate during appropriate hours” (Scott 1991: 19). In order for the fresco’s allegories to be understood by visitors, a specially written guidebook – not dissimilar from the ones we commonly find in today’s museums – was created: Rosichino’s Description of the Paintings in the Salon of the Barberini Lords (Dichiaratione Delle Pitture Della Sala De’ Signori Barberini, 1640). These two examples aptly show how, in baroque culture, patrons of the arts were already employing a number of intermedial strategies for presenting the same media product to different “target groups” (to put it in today’s terms), from that of “highbrow” connoisseurs to the one usually defined as “the general public.”

Giovan Battista Gaulli’s Triumph of the Name of Jesus Given the importance that the Counter-Reformation played in shaping the religious mediascape of the Baroque, it should come as no surprise that two of the greatest masterpieces in the history of the Catholic Baroque are the Church of the Gesù and the Church of Saint Ignatius of Loyola. They were built in Rome, the heart of the Catholic world, on a commission from the Society of Jesus, one of the most potent religious orders at that time, founded in 1534, just 11 years before the Council of Trent. Both churches feature extraordinary ceiling decorations designed to elicit an intense emotional and aesthetic response. The Triumph of the Name of Jesus, painted by Giovan Battista Gaulli (also known as Baciccio) in the Church of the Gesù between 1676 and 1679, attempts to render that sense of ecstatic beatitude that, according to Catholic doctrine, virtuous believers will experience after death in the presence of Jesus. In order to construct such a blissfully overwhelming experience, Baciccio adopted two main strategies. Firstly, Jesus was represented (or, rather, nonrepresented) as a circle emanating a blinding light on which the Christogram IHS is superimposed in a combination of painting and writing. To give the sense of the unbearable brightness of Jesus’s light, the contours of the painted characters

564

M. Fusillo and M. Petricola

become increasingly blurred the closer they are to Jesus. Secondly, this religious experience was constructed as being so overwhelming that it could not be contained within the frame’s limits. The representation of the blessed ones who admire Jesus’s light while sitting on clouds spills over the frame on both the left and right sides of the fresco, thus reconfiguring the traditional relations between painting and architecture. It has been argued that this technique was used to “draw the spectator in by suggesting that the heavenly scene is happening here and now” (Snodin and Llewellyn 2009: 91). The same technique was employed on the lower side of the frame. However, this time we see a “squirming mass of naked figures tumbling downwards over the frame, some of them characterized as the vices of vanity, avarice, and heresy” (Zucker 2013: 367). They are the sinners and the damned, banished beyond the reach of the sacred light, who “seem to fall towards us into the real space of the church” (Snodin and Llewellyn 2009: 91). Furthermore, as noted by Mark Zucker, the scene was also related to the decorative elements surrounding the fresco: “as if this iconographic extravaganza were not enough, just below the ceiling are sixteen over-life-sized statues symbolizing foreign lands proselytized by the Jesuits – China, Ethiopia, Mexico, Brazil, etc. – gazing upward in awe, just as visitors to Il Gesù must have done and, for that matter, still do” (2013: 367). As demonstrated by examinations of Cortona’s and Baciccio’s ceilings, the baroque scopic regime is determined by an intricate, ever-evolving relation between the artwork, the space in which it is situated, and the viewer. Contrary to Cartesian perspectivalism, baroque vision is intrinsically embodied, as the aesthetic experience of these artworks unfolds in both time and space according to the movements of the viewer’s eyes and body. In other words, baroque culture was structured by a supremely performative scopic regime.

The Courtly Feast and the Birth of Modern Entertainment As mentioned in section “Counter-Reformation and Monarchical Absolutism,” with the rise of monarchical absolutism, courts became one of the central elements of baroque political and social life. The vast majority of courtiers were hosted in massive palaces with little to contribute to the political life of their kingdom, especially in times of peace. Therefore, they needed to be entertained. This was made possible through continuous cycles of activities and performances –something structurally analogous happens today on cruises worldwide. The most popular entertainments were “hunting, leisure walks (promenade), dramatic shows, concerts, gambling, dancing (balls), and pyrotechnics” (Spielmann 2019: 805). New entertainment forms like the French court ballet and the English masque were also specifically designed to be enjoyed in a courtly palace. By combining all these activities on an astonishingly opulent scale in an ephemeral setting, the feast represented the pinnacle of court entertainment, and many a baroque artist, from Bernini to Rubens, used their skills in service of creating such events. According to Guy Spielmann, the feast “constituted a deliberate rupture in the flow of ordinary life” (2019: 806) which “had to remain an exceptional event by nature but also by

23

The Age of Wonder and Entertainment: An Introduction to. . .

565

necessity since it could only be achieved through considerable expense” (2019: 805). Like the other media products analyzed in this chapter, feasts aimed at causing an intense sensory overload coupled with a sense of overwhelming wonder. In this regard, Snodin and Llewellyn note that “André Félibien, chronicler of Louis XIV’s elaborate fêtes, coined a rather felicitous expression when he described an entertainment held at Versailles in 1668 as ‘agréable désordre’ (pleasurable disorder).” In sum, in the seventeenth century, the court feast became an exceptional intermedial assemblage of media products created through the synergy of architecture, dance, music, poetry, drama, tourneys, jousts, fights, and other spectacular events. The purely ephemeral and performative character of this intermedial experience might also be considered one of the features that would allow a fascinating comparison between a baroque feast and certain large-scale forms of contemporary entertainment like, for example, extravagant celebrity parties or the opening ceremonies at the Olympics.

A Double Marriage in Mantua A significant example of a courtly feast is the one organized in Mantua in 1608 to celebrate the double wedding of the Duke of Savoy Charles Emanuel I’s daughters, Margherita and Isabella, with Francesco IV Gonzaga and Alfonso II d’Este. This manifold event combined serious, “high” performances focused on drama, opera, and ballet, with a more exuberant and creative schedule, including carousels, naval battles on the lake, spectacular banquets, and various chivalric performances. This last was directly inspired by the Duke of Nemours, a famous man of the world and amateur who wrote and directed several theatrical events (Metlica 2020). This extraordinary variety of performances and aesthetic experiences derived from the interaction between the French tradition of court ballets (McGowan 2008; HarrisWarrick 2016) and the Italian performative praxis, aiming at producing the most typical of baroque aesthetic experiences: astonishment. The variety of spectacular events also implied an impressive mixture of social abilities: prestigious artists with technicians and artisans, sublime poets and musicians together with the first professional actors (the Fedeli by Giovan Battista Andreini) who could sing, play instruments, and act at the same time. Moreover, the feast was distributed over several days and in totally different spaces (natural; artificial; social; historical; such as streets, squares, palaces, estates, new theatres). Spectacular events were omnipervasive phenomena in the baroque age. Finally, the Mantua feast leads us to another crucial intermedial practice: ekphrasis, a media transformation practice dating back to Homer’s description of Achilles’s shield (Heffernan 2004; Romagnino 2019). The spectacular event was loosely remediated by the previously mentioned Italian poet Giovan Battista Marino, active in France and Italy and involved in the feast’s planning. Description dominates over narration in his abnormal, gigantic, and labyrinthine poem Adonis (L’Adone). In Chant 5, a long ekphrasis of a feast shows several parallels to the account of the double wedding in Mantua by the scenographer Federico Follino. At the same time,

566

M. Fusillo and M. Petricola

it sounds more like a broad poetic description of a court spectacle, full of references to other legendary later events and especially to astonishing theatrical machines. Marino describes an intermedial hybrid form made up of dance, poetry, acting, and astounding scenographic inventions (Priest 1982; Metlica 2020). This ekphrasis of a performative event can be considered a highly complex form of media representation, which Lars Elleström (2019) defines as “the notion of one medium representing another medium” (4). In other words, “[m]edia representation is at hand whenever a medium presents another medium to the mind of a communicatee” (4), like in the case of “an autobiography describing a photograph, or a film depicting a person communicating through sign language” (5). On the other hand, the pictorial ekphrasis was also highly developed, with an impressive richness of detail and visual pregnancy in the baroque novel, especially in the “novel of novels,” Astrea (L’Astrée, 1607–1627), an immense pastoral work by Honoré d’Urfè, who also produced several tragicomedies and other dramatical works.

The Pleasures of the Enchanted Island One of the most famous court festivals of the seventeenth century was the one organized, over 3 days in May 1664, at Versailles by the young Louis XIV (who ruled France from 1643 to 1715) under the title of The Pleasures of the Enchanted Island (Les Plaisirs de l’Île Enchantée). Versailles is still quite far from being the grandiose palace we know today, as the court would move there permanently only in 1682. However, the festival was an occasion to inaugurate the magnificent gardens designed by André Le Notre. Even more importantly, the Plaisirs was an opportunity to immerse its 600 participants in a sort of parallel world. Inspired by literary romance and separated from the reality of mundane events, this is a world on which the king reigns supreme as an omnipotent demiurge. The Enchanted Island of the festival’s title is the fictional island inhabited by the sorcerer Alcina in Ludovico Ariosto’s 1532 epic poem Orlando furioso. In Ariosto’s fiction, the Saracen knight Ruggiero lands on the island after flying on a hippogryph. He is seduced by Alcina’s sexual charm and ultimately manages to escape once the sorceress’s spell has worn off. The Plaisirs reenact this narrative with Louis XIV himself playing the role of Ruggiero. In playing the main protagonist in a courtly performance, Louis XIV adhered to a broad European tradition followed by, among others, his father Louis XIII and by Holy Roman Emperor Leopold I – one of the latter’s portraits “shows him as a gallant shepherd in an exuberant red and gold costume for the pastoral play La Galatea in 1667” (Spielmann 2019: 811). The festival began with a solemn procession led by the Sun King as Ruggiero, followed by 13 nobles who, chosen to be part of this fictional “security detail,” assumed “the role of solar satellites” (Wine 2001: 82) in the eyes of the other courtiers. The procession closed with a chariot on which the god Apollo and the personifications of the four ages stood. The second day featured a course de bague, a

23

The Age of Wonder and Entertainment: An Introduction to. . .

567

Fig. 2 Israël Silvestre, The destruction of Alcina’s palace and the cessation of her enchantments, represented by a firework display (Rupture du Palais et des enchantements de l’Isle d’Alcine, representée par un feu d’Artifice). (From Les plaisirs de l’ile enchantée, Paris, Imprimerie Royale, 1673)

competition in which the contestants “driv[e] a lance through a suspended ring” (Wine 2001: 83) while riding on horseback; it was followed by a performance of Molière’s play The Princess of Elis (La Princesse d’Élide). On the third day, the festival ended with Louis-Ruggiero’s escape from the enchanted island and the destruction of Alcina’s palace, represented through a sumptuous machine theater play, a ballet, and a spectacular fireworks display (see Fig. 2). Snodin and Llewellyn provide a compelling analysis of these events worth quoting at length: Held captive by [. . .] Alcine, the knights were released only on the third and final day of the festivities, by means of a magic ring placed on the finger of Louis/Ruggiero. Simultaneously, the palace of Alcine was demolished by fireworks detonated within its walls. As the smoke cleared on the scene’s charred remains, courtly spectators could not have doubted the occasion’s real agenda. [. . .] The destruction of the fictional palace of Alcine had restored to sight the château of Versailles and the material basis of Louis’s power in the real world. With their unique combination of sound and coloured light effects, fireworks were used thus to overwhelm spectators and articulate the relationships between ruling powers and society at large. (2009: 167)

As this passage makes clear, the festival creates a multilayered intermedial network involving Ariosto’s poem, the character of Louis-Ruggiero, and the

568

M. Fusillo and M. Petricola

courtiers-viewers. According to Marie Claude Canova-Green (2001), the Plaisirs was a fundamental step in shaping Louis XIV’s identity in the eyes of the court as based on a sense of contemplation filled with wonder (130). By moving the court away from Paris and keeping it isolated for days in the self-contained world of Versailles with its gardens, fountains, lakes, and forests, Louis XIV cast himself as a king capable of turning the pastoral utopia of d’Urfé’s L’Astrée into a reality (Apostolidès 1981: 94; Canova-Green 2001: 121). Like Alcina, the king possesses authentic world-making powers; but contrary to the sorceress, he uses his powers for the good of his court (Canova-Green 2001: 127). From another perspective, by playing the role of Ruggiero, Louis embodied a character who renounced the satisfaction of carnal desires in a world of illusion to perform his duties as a warrior in the “real” world. In this sense, the Plaisirs was a rite of passage where the king, tempted by the “low” pleasures of the flesh, emerges as a new man ready to devote himself to the responsibilities of governing a nation. In Jean-Marie Apostolidès’s (1981) view, Louis-Ruggiero embodies “the antithesis of Don Juan,” since he “has to go beyond his fascination for absolute and instant pleasure [ jouissance] in order to realize his essence as a courtier-hero” (100, own translation). In conclusion, the Plaisirs created an intermedial space-time bubble within the ephemeral limits of the fête, thus allowing the collective construction of a world in which the political issues of Louis XIV’s France could be read in the light of myth, epic, and fantasy (Apostolidès 1981: 66–92).

Theater and the Rise of the Opera The First Operas and Divas The 1608 Mantua feast analyzed in section “A Double Marriage in Mantua” was not only a noteworthy event in the history of the courtly feast, but also a fundamental moment in the history of musical theatre, with the premiere of Claudio Monteverdi’s opera Arianna (libretto by Ottavio Rinuccini) and the resumption of Battista Guarini’s comedy L’idropica, enriched with some musical interludes. Opera is undoubtedly the most relevant intermedial art form of the baroque age. It was founded in the late Renaissance in Florence by a poetic circle, the Camerata de’ Bardi, which revived the synergy of the arts (poetry, music, dance, and theater) at the core of ancient Greek tragedy through heightened speech (recitar cantando). Claudio Monteverdi further developed this new artistic genre, creating a more flexible, dissonant, and effective declamatory melody and producing a profound interaction between words and music (Rosand 2007). The score has been lost, except for the famous and successful Lamento, the first great operatic aria in history, which dramatizes a significant literary topos, especially from the point of view of gender visions: the abandoned woman. Arianna was played by a famous actress, Virginia Ramponi, wife of Giovan Battista Andreini, playwright and director of the company. Actresses played a crucial role in the theatrical performances of the baroque age. Their audacious experimentation exploited the paradoxical contamination between

23

The Age of Wonder and Entertainment: An Introduction to. . .

569

the tragic and the comic, the sublime and the ridiculous, bringing an innovative focus on erotic themes. This important trend was begun by Isabella Andreini, Giovan Battista’s mother, a poet and the first diva in history celebrated all over Europe. Her legacy was further developed by actresses from the commedia dell’arte (a form of theater rooted in popular culture and based on improvisation, very successful in France and England) and several other figures in the first half of the seventeenth century. Their performances were characterized by some effective tropes, such as madness or cross-dressing. They gave special energy and vitality to the traditional theatrical praxis, which enhanced the fascination for theater characterizing the entire seventeenth century and faced various forms of censorship (Taviani 1991). During the Baroque, operas became more and more anticlassicist and experimental. They were full of whimsical, bizarre, or paradoxical situations, with multiple plots unfolding in parallel and bold stylistic contaminations.

Tragicomedy and the Disruption of Traditional Genres Tragicomedy is the genre that best condenses not only the key features of hybridity, mobility, and metamorphosis at the core of the baroque universe (Rousset 1989) but also the experimental anxieties of late Renaissance and baroque literature and their search for new expressive strategies. The term “tragicomedy” was used for the first time by Plautus in Amphitryon’s prologue, in a very effective passage where it appears to be a result of Mercurius’s improvisation. In this archetypal case, the main criterion is the content: it is a tragicomedy because it has two gods as the main characters. Social division is fundamental in ancient aesthetics, with gods and heroes for tragedy and the common people for comedy. Nevertheless, we can find some traces (an embryo) of the modern meaning of tragicomedy as the overlap between two opposite stylistic patterns since Amphitryon contains uncanny tones coming from the theme of the double in a comic structure. As a matter of fact, the term tragicomedy in the modern period expresses, even more than the overlap between tragedy and comedy, the difficulty of univocal classifications, the pleasure of fusion and hybridization, the desire to exceed simple classical forms and rigid distinctions (Foster 2004). Its textual configurations are multifarious: the reprisal of ancient myths of metamorphosis, implying death and transfiguration, as in John Lyly’s pastoral drama Endymion (1579); the dissonant contiguity between the sublime and the ridiculous, as in sacred dramas (very striking in Sant’Irene by Loreto Vittori, an extremely popular virtuoso); the profound ambiguity between the tragic and the comic, which will characterize late modernity (Hammond 1994; Gigliucci 2012). On one hand, tragicomic authors seek to recover a primordial, ritual unity: an undifferentiated archaic theatre, which has fascinated intellectuals and artists for centuries, obsessed by the problem of origins. On the other hand, a tragicomic mixture intends to reflect the unpredictable complexity of human life, which does have nuanced intersections rather than strong dichotomies. This theoretical position was held by one of the foremost European dramatists, Lope de Vega, in a short treatise delivered to the Madrid Academy and considered the first

570

M. Fusillo and M. Petricola

manifesto of modern theater, The New Art of Writing Plays in this Age (El arte Nuevo de hacer comedias en este tiempo, 1609). Lope de Vega declared that the mix between comic and tragic produces variety and pleasure and, above all, gives a good example of “truthfulness” (naturaleza). Even clearer is the formulation by a French playwright, François Ogier, who wrote, in the preface to the drama Tyr at Sydon (1628), that to refuse to alternate between comic and tragic scenes means to ignore the true human condition, which constantly mixes laughing and crying. Along with this increasing success, tragicomedy aroused intense controversy at the same time, especially in countries and contexts with a more robust classical tradition. The Aristotelian system of literary genres was based on a clear (stylistic and ideological) distinction between tragedy and comedy: Horace’s Poetics (vv. 91–92) admitted only comedy with a slightly serious tone, whereas Cicero absolutely refused any overlapping of the two theatrical genres (On the Best Kind of Orators 1,1). Plautus’s case seemed to be more of an exception regarding this premise. In a 1601 essay entitled A Compendium of Tragicomic Poetry (Compendio della poesia tragicomica), Italian playwright Giovan Battista Guarini defined his own pastoral play The Faithful Shepherd (Il Pastor fido, 1590), as well as Torquato Tasso’s Aminta, as tragicomedies. This generated a fierce debate between him and Giason de Nores, an Aristotelian professor of moral philosophy, who labelled The Faithful Shepherd as a monstrosity. Since it melds polar entities and socially opposite universes, like a hermaphrodite, tragicomedy often evokes definitions based on a logic of separation and purity: for example, Philip Sydney calls it a mongrel in An Apology for Poetry (1595). In 1637, the enormous success of Pierre Corneille’s tragicomedy Le Cid generates another fierce debate, this time in France – a country characterized by a strong neoclassical and rationalist tradition. Le Cid was heavily criticized by, among others, George Scudéry as a play lacking verisimilitude and structural homogeneity. In 1660, Corneille systematically defended his aesthetic program in Three Discourses on Dramatic Poetry (Trois Discours sur le poème dramatique), acknowledging some errors at the same time. Already in 1648, however, he had classified Le Cid just as tragedy: the season of tragicomedy was clearly declining (Guichemerre 1981). Tragicomedy’s most flourishing period was the beginning of the seventeenth century, immediately before Corneille’s masterpiece. In France, Alexander Hardy was the most popular representative of this genre. With Hardy, tragicomedy became a truly modern genre. He considered it not just as a minor theatrical form inherited from the Middle Ages or a simple tragedy with a happy ending, but as a new genre based on love, adventure, stylistic contamination, violent passions, heroism, coup de théâtre, simulation, deceptive appearances, and cross-dressing. His vast production (more than 600 dramatic poems, with only 34 surviving for inclusion in his complete works published in 5 volumes) was conceived for the stage. Hardy was strictly linked with the troupe of actors Le Comédiens du Roi, headed by Valleran Le Conte and active at the Hôtel de Bourgogne and in the provinces. This is why his theater was not concerned with Aristotelian and classicist theory. There was no respect for

23

The Age of Wonder and Entertainment: An Introduction to. . .

571

dramatic unities and decorum (the so-called bienséance); on the contrary, his plays were animated by continuous changes of setting (Deierkauf-Holsboer 1972). From an intermedial point of view, it is interesting to mention an unusual and abnormal work: a dramatization of the longest, most complex Hellenistic novel, Heliodorus’s Aethiopica (Aithiopika, third century AD). Beloved by baroque poets and artists, the novel is imbued with theatrical metaphors and media representations. Instead of selecting some significant episodes, as other dramatists did in Europe, Alexander Hardy composed a tragicomedy of eight dramatic poems, to be performed on eight consecutive evenings, published for the first time in 1623, followed by a second edition in 1638 (Hardy 2014; Fusillo 2018). This interesting and quite unique operation can be described through a key category of popular culture, particularly vital in our contemporary intermedial imagination: seriality. We cannot say much about the pragmatic context of Hardy’s théâtre consécutif. A remnant of the medieval theatrical tradition, a serial representation was quite rare but not unusual, especially in Elizabethan England. However, it usually ran for 2 or 3 days, never eight. Apart from the historical context, it reminds us of some modern experimentations that question textual borders and challenge the limits of the public’s expectations, from Wagner’s Tetralogy to many contemporary stage productions or videoart creations (Matthew Barney’s Cremaster; Jan Fabre’s Mount Olympus). The following generation of poets in France (Du Ryar, Auvray, Rassyguier, Pichou) harshly criticized Hardy’s theater and his vision based on continuity and realism, preferring a more refined stye, full of visual and musical effects. Their tragicomedies gave a new prominence to erotic themes and the representation of eccentric passions, especially La filis de Scire by Pichou, a writer usually depicted as a libertine, atheist, and homosexual (Lombardi 1995).

Rethinking Genres Beyond the Tragicomedy In Elizabethan England, where the classicist tradition was weaker, the vitality of theater preferred mixed forms. When Polonius introduces the company of actors in Hamlet (act 2, scene 2), he lists a considerable amount of pure and overlapping genres: “The best actors in the world, either for/tragedy, comedy, history, pastoral, pastoral-comical,/ historical-pastoral, tragical-historical,/tragical-historical-comicalpastoral, scene individable/or poem unlimited.” Shakespeare’s output itself shows a large number of subgenres and nuances, and defies a rigid classification, especially in his last phase and in the romances (Pericles, Cymbeline, The Winter’s Tale), which share, with Hellenistic and baroque novels, a poetics of excessive theatricality and a series of tragicomic topoi (especially the apparent death). Not by chance, two centuries later, excess will become the key element of the melodramatic imagination, showing a certain continuity between two anti-classicist aesthetics. In his most canonical tragedies as well, we find comical insertions that produce comic relief. The now-iconic Romeo and Juliet, for example, has been performed alternatively with a comic or a tragic ending by James Howard, as reported by John Downes in his

572

M. Fusillo and M. Petricola

theatrical history Roscius Anglicanus (see René Weis’s introduction to Shakespeare 2012: 57–58). Shakespeare’s last masterpiece, The Tempest, overlaps the revenge tragedy with low comicality, romance with philosophy, offering at the same time a profound metalinguistic reflection on theatre. This play would be adapted several times in various forms in many cultures, giving the basic structure for a summa of contemporary intermediality, the movie Prospero’s Books by Peter Greenaway. At the beginning of the late modern era, drama and the novel would ultimately dissolve the binary “tragedy vs. comedy,” which, according to Auerbach (2013), had defined Western literatures for centuries. This dissolution was made possible by the emergence of a new aesthetic category defined by the adjective “serious.” A profound cross-fertilization of tragedy and comedy would be at the core of the Romantic revolution. Not by chance do we again find the tragicomedy in the twentieth century, in an extreme, radical, and tragic writer like Samuel Beckett, who used this term to describe his masterpiece Waiting for Godot, indicating a grotesque, metaphysical void.

Meta-Theater and Don Juan as a Symbol of Baroque Intermediality The great theater of the world (Theatrum Mundi) is a widespread metaphor that dates back to ancient philosophy and early Christian thinking. It became the most characterizing feature of baroque poetics, expressing the lability and fluidity of the notion of reality (Christian 1987). It is paradoxical that theatre assumed such a metaphysical value in baroque culture; on the other hand, its social legitimation was still struggling to be fully recognized. However, in the seventeenth century, theatrical companies finally began to become professional and gain a secular, autonomous space. For these reasons, the baroque age presented a truly exceptional number of metadramas, both in a concrete (as in Shakespeare’s Hamlet or in Molière’s L’illusion comique) and a metaphorical meaning (as in Shakespeare’s The Tempest). The various versions of the life of Saint Genest, actor and martyr (by Lope de Vega, Desfontaine, and Rotrou) could fall into the same category. Metadramas celebrated the seductive power of theatrical representation, exploring the fluctuating border between fiction and reality, attaching philosophical values to the actor’s role. Actors were able to catch the ambiguity and contradictions of human psychology, living in a marginal space, in a liminal and extra-quotidian zone. Tragicomedy and metatheatre are also the main reasons why Don Juan became the characterizing myth of the baroque age. His adventures imply the tragic sufferance of abandoned lovers, comical misunderstanding and disguise, feasts, and divine punishments. Moreover, seduction is a rhetorical strategy, a performance of intense acting that oscillates between rules, scripts, and free improvisation. Before the famous, materialistic version by Molière, Giovan Battista Andreini (the manager of the company I fedeli) devoted a gigantic, hyperbaroque work to the figure of Don Juan (Andreini 2003), seen as a demonic atheist. It followed the rules of the French spectacular drama (pièce a grand spectacle), exploiting music, singing, continuous change of astonishing settings (seascapes, palaces, even Hell itself), mythical references, and complex machines. In those same years, Andreini worked at an unusual version of Saint Magdalen’s life, characterized by a pictorial style with direct media

23

The Age of Wonder and Entertainment: An Introduction to. . .

573

representations of contemporary paintings (Domenico Fetti’s Melancholy and Guido Cagnacci’s St. Mary Magdalen), and at a particularly ambiguous play, Love in the Mirror, full of triangular desire and female homoeroticism: a homage to his wife Virginia Ramponi and her lover Virginia Rotari- Florinda and Lidia on stage (Andreini 1997). His figure thus could synthesize the disruptive, anti-normative, and polymorphic character of baroque intermediality.

Conclusion: Baroque Media Products After the Baroque This chapter has explored some of the most historically and aesthetically relevant intermedial networks developed in the context of baroque culture between the late sixteenth and the early eighteenth century. We have seen that one may explore baroque visual culture as a set of phenomena closely related to Catholicism, pinpoint the “ornamental impulse” across the most disparate media, analyze how baroque media products informed political thought and vice versa, or how they contributed to the theories of realism. But how does one investigate the survival, reemergence, citations, and echoes of the baroque from the mid-eighteenth century onwards? In other words, how can we study the baroque after the Baroque? In this chapter, we have tried to pinpoint relations of filiation or analogy between certain baroque media products and their contemporary “descendants.” This same attitude has been adopted by Guy Spielmann, who notes that “[w]hile the Italian and French ‘grand opera’ [. . .] has remained a province of elites, it has also spawned the musical, through the British ballad opera of the early eighteenth century; by the end of that century, high baroque gardens, Arcadian playgrounds for a small coterie of aristocrats had evolved into ‘Vauxhalls’ and ‘Tivolis,’ providing entertainment to a growing populace: the forerunners to our amusement parks” (2019: 818). A different set of hermeneutical tools for studying the “baroque after the Baroque” has centered around the notion of “neobaroque” (or “neo-baroque”), which has been debated in academic circles from several perspectives since at least the 1980s. Italian semiotician and cultural theorist Omar Calabrese (2017) saw the neobaroque as the hegemonic worldview of our time – that is, as an alternative definition of the postmodern condition. Film and media scholar Angela Ndalianis applies the term neo-baroque to the current entertainment industry’s tendency to disrupt the limits imposed by frames, boundaries, and taxonomies, thus catalyzing the emergence of a mediascape in which “[c]losed forms are replaced by open structures that favor a dynamic and expanding polycentrism” (2004: 25). At the same time, some scholars never use the word “neobaroque” while embracing positions that are in many ways analogous to those elaborated by theorists of the neobaroque. For example, in examining today’s mediascape with regard to the Baroque, Christine Buci-Glucksmann has argued that “[t]he virtual pushes the baroque to its extreme [. . .] and develops a culture of flux, of artifacts and a new kind of image, the flux-image” (2013: xv). Many more authors, works, and ideas could be discussed; unfortunately, this goes beyond the scope of this chapter. What we can do, in conclusion, is to remark that this intense

574

M. Fusillo and M. Petricola

interdisciplinary debate demonstrates the incredible vitality enjoyed by the Baroque in our age and the immense potential it still possesses to produce wonder.

References Andreini, Giovanni Battista. 1997. Amor nello specchio. Ed. Salvatore Maira and Anna Michela Borracci. Rome: Bulzoni. ———. 2003. Don Giovanni, o, L’estrema avventura del teatro: il nuovo risarcito convitato di pietra di Giovan Battista Andreini: studi e edizione critica. Ed. Silvia Carandini and Luciano Mariti. Rome: Bulzoni. Apostolidès, Jean-Marie. 1981. Le roi-machine: spectacle et politique au temps de Louis XIV. Paris: Editions de Minuit. Auerbach, Erich. 2013. Mimesis: The representation of reality in Western literature. Trans. Willard R. Trask. Princeton: Princeton University Press. Benjamin, Walter. 2009. The origin of German tragic drama. Trans. John Osborne. London: Verso. Buci-Glucksmann, Christine. 2013. The madness of vision: On baroque aesthetics. Trans. Dorothy Zayatz Baker. Athens: Ohio University Press. Calabrese, Omar. 2017. Neo-baroque: A sign of the times. Trans. Charles Lambert. Princeton: Princeton University Press. Calloway, Stephen. 2000. Baroque baroque: The culture of excess. London: Phaidon. Canova-Green, Marie Claude. 2001. Espace et pouvoir dans Les plaisirs de l’île enchantée (1664). Seventeenth-Century French Studies 23: 121–138. https://doi.org/10.1179/c17.2001.23.1.121. Cascardi, Anthony J. 2019. Experience and knowledge in the baroque. In The Oxford handbook of the baroque, ed. John D. Lyons, 448–470. Oxford: Oxford University Press. https://doi.org/10. 1093/oxfordhb/9780190678449.013.4. Christian, Lynda G. 1987. Theatrum mundi: The history of an idea. New York: Garland. Croce, Benedetto. 1993. Storia dell’età barocca in Italia. Milan: Adelphi. Deierkauf-Holsboer, Sophie W. 1972. Vie d’Alexander Hardy, poète du roi, 1572–1632. Paris: Nizet. Deleuze, Gilles. 1993. The fold: Leibniz and the baroque. Minneapolis: University of Minnesota Press. Elleström, Lars. 2019. Transmediation: Some theoretical considerations. In Transmediations: Communication across media borders, ed. Niklas Salmose and Lars Elleström, 1–13. London: Routledge. Foster, Verna A. 2004. The name and nature of tragicomedy. London: Routledge. Foucault, Michel. 2012. The order of things. London: Routledge. Fusillo, Massimo. 2018. The serial dramatization. Alexander Hardy’s tragicomedy Théagène et Cariclée. In Re-wiring the ancient novels, ed. E. Cueva, S. Harrison, H. Mason, W. Owens, and S. Schwartz, vol. 1.: The Greek novels, 279–288. Groningen: Barkhuis. Gigliucci, Roberto. 2012. Tragicomico e melodramma: studi secenteschi. Milan: Mimesis. Guichemerre, Roger. 1981. La tragi-comédie. Paris: PUF. Hammond, Frederick. 1994. Music and spectacle in baroque Rome: Barberini patronage under urban VIII. Vol. 24, 29. New Haven: Yale University Press. Hanke, Stephanie. 2019. Water in the baroque garden. In The Oxford handbook of the baroque, ed. John D. Lyons, 87–118. Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/ 9780190678449.013.14. Hardy, Alexandre. 2014. Les chastes et loyales amours de Théagène et Cariclée. Ed. Antonella Amatuzzi and Daniela Dalla Valle. Paris: Garnier. Harris-Warrick, Rebecca. 2016. Dance and drama in French baroque opera: A history. Cambridge: Cambridge University Press.

23

The Age of Wonder and Entertainment: An Introduction to. . .

575

Heffernan, James A.W. 2004. Museum of words: The poetics of ekphrasis from Homer to Ashbery. Chicago: University of Chicago Press. Hoxby, Blair. 2019. Baroque tragedy. In The Oxford handbook of the baroque, ed. John D. Lyons, 515–539. Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190678449. 013.17. Jay, Martin. 2009. Scopic regimes of modernity. In Vision and visuality, ed. Hal Foster, 3–23. Seattle: Bay Press. Johnson, Christopher D. 2010. Hyperboles: The rhetoric of excess in baroque literature and thought. Cambridge: Harvard University Press. ———. 2012. Baroque. Ed. Roland Greene, Stephen Cushman, Clare Cavanagh, Jahan Ramazani, and Paul F. Rouzer. The Princeton encyclopedia of poetry and poetics. Princeton: Princeton University Press. Kang, Minsoo. 2011. Sublime dreams of living machines: The automaton in the European imagination. Cambridge: Harvard University Press. Keating, Jessica. 2019. The automaton. In The Oxford handbook of the baroque, ed. John D. Lyons, 184–210. Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190678449. 013.40. Lombardi, Marco. 1995. Processo al teatro. La tragicommedia barocca e i suoi mostri. Pisa: Pacini. Lyons, John D. 2019. Introduction: The crisis of the baroque. In The Oxford handbook of the baroque, ed. John D. Lyons, 1–23. Oxford: Oxford University Press. https://doi.org/10.1093/ oxfordhb/9780190678449.013.21. Maravall, José A. 1986. Culture of the baroque: Analysis of a historical structure. Trans. Terry Cochran. Minneapolis: University of Minnesota Press. McGowan, Margaret M. 2008. Dance in the Renaissance: European fashion, French obsession. New Haven: Yale University Press. Metlica, Alessandro. 2020. Le seduzioni della pace: Giovan Battista Marino, le feste di corte e la Francia barocca. Bologna: Il Mulino. Milton, John. 2003. Paradise lost. Ed. John Leonard. London: Penguin Books. Ndalianis, Angela. 2003. Architectures of the senses: Neo-baroque entertainment spectacles. In Rethinking media change: The aesthetics of transition, ed. David Thorburn and Henry Jenkins, 355–374. Cambridge: The MIT Press. ———. 2004. Neo-baroque aesthetics and contemporary entertainment. Cambridge: The MIT Press. Pascal, Blaise. 2008. Pensées and other writings. Ed. Anthony Levi. Trans. Honor Levi. Oxford: Oxford University Press. Priest, Harold M. 1982. Marino, Leonardo, Francini, and the revolving stage. Renaissance Quarterly 35: 36–60. https://doi.org/10.2307/2861452. Romagnino, Roberto. 2019. Théorie(s) de l’ecphrasis entre antiquité et première modernité. Paris: Classiques Garnier. Rosand, Ellen. 2007. Opera in seventeenth-century Venice: The creation of a genre. Berkeley: University of California Press. Roston, Murray. 1980. Milton and the baroque. London: Palgrave Macmillan. Rousset, Jean. 1989. La littérature de l’âge baroque en France: Circé et le paon. Paris: Corti. Scott, John B. 1991. Images of nepotism: The painted ceilings of Palazzo Barberini. Princeton: Princeton University Press. Shakespeare, William. 2012. Romeo and Juliet. Ed. René Weis. London: Bloomsbury. Snodin, Michael, and Nigel Llewellyn, eds. 2009. Baroque, 1620–1800: Style in the age of magnificence. London: Victoria & Albert Museum. Spielmann, Guy. 2019. Court spectacle and entertainment. In The Oxford handbook of the baroque, ed. John D. Lyons, 801–824. Oxford: Oxford University Press. https://doi.org/10.1093/ oxfordhb/9780190678449.013.34. Taviani, Ferdinando. 1991. La commedia dell’arte e la società barocca: la fascinazione del teatro. Rome: Bulzoni.

576

M. Fusillo and M. Petricola

Tetius, Hieronymus. 2005. Aedes Barberinae ad Quirinalem descriptae. Ed. Lucia Faedo and Thomas Frangenberg. Pisa: Edizioni della Normale. Visentin, Hélène. 2019. Machine plays. In The Oxford handbook of the baroque, ed. John D. Lyons, 385–408. Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190678449. 013.36. Wine, Kathleen. 2001. Honored guests: Wife and mistress in Les plaisirs de l’île enchantée. Dalhousie French Studies 56: 78–90. Dalhousie University. Wölfflin, Heinrich. 1994. Renaissance and baroque. Trans. Kathrin Simon. Ithaca: Cornell University Press. Yonan, Michael. 2019. Ornamentation. In The Oxford handbook of the baroque, ed. John D. Lyons, 408–426. Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190678449. 013.38. Zucker, Mark. 2013. Iconography in renaissance and baroque art. In A companion to renaissance and baroque art, ed. Babette Bohn and James M. Saslow, 361–380. Chichester: WileyBlackwell.

Intermediality in Seventeenth-Century Baroque Celebrations in Hispanic America: Commissions, Poetry, and Ephemeral Architecture

24

Mauricio Va´squez Arias and Andre´s Burbano Valde´s

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Context of Medial Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Allegorization and Ekphrastic Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Story of a Misreading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Symbolic Programs and Commissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An Intermedial Way of Thinking and Creating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions: Intermedial Encounters Between the Baroque Machines of Sigüenza and Sor Juana Inés . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

578 581 583 584 587 590 593 599

Abstract

This chapter is concerned chiefly with a set of early forms of media convergence and intermediality in the Hispanic American context, associated with what is now defined as curation, poetry, and ephemeral architecture. The foregoing is based on the exploration of two cases: Sor Juana Inés de la Cruz and her work Neptuno Alegórico and Carlos de Sigüenza y Góngora with Teatro de virtudes políticas. For this, the concepts and perspectives offered by media archeology and intermedial studies are used. As general conclusions of the research, the encounter with early forms of media convergence for the deployment of an articulating symbolic program can be highlighted, as well as the existence of a set of knowledge and formations that prepared the intellectuals of the time to compose various media through the practices of ingenuity and sharpness, for the purpose of celebration or commiseration regarding the central events of the Spanish empire. M. Vásquez Arias (*) School of Arts and Humanities/Creation Area, Universidad EAFIT, Medellín, Colombia e-mail: mvasqu23@eafit.edu.co A. Burbano Valdés Faculty of Arts and Humanities, Open University of Catalonia, Barcelona, Spain e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_29

577

578

M. Va´squez Arias and A. Burbano Valde´s

As a central hypothesis, it is proposed that the intertwined history of America and Spain offers an interesting matrix to understand different modes of encounter between media, thanks to which contemporary ideas of convergence, intermediality, transmediality, and transmediation can be rethought. Keywords

Baroque · Intermediality · Transmediation · Convergence · Ephemeral architecture · Allegorical poetry · Media archeology

Introduction The Hispanic American baroque, like the baroque universe as a whole, has been the object of a wide range of focused inquiries that explore its myriad aspects in detail. The colonial dynamics of what once was one of the largest empires in the world have been consistently examined through disciplines like historical studies, aesthetics, art history, and literary studies in academic institutions from Spain, Latin America, and North America. The broad spectrum of scholarly work on the subject includes noteworthy contributions by Víctor Mínguez, Inmaculada Rodríguez, Pablo González Tornel, and Juan Chivas Beltrán from the Universidad Jaume I and the Universidad de Las Palmas de Gran Canaria in Spain. This cluster of scholars has recently published a series of art-historical studies that span a broad historical context and focus on iconographic analyses and interpretations of printed works, a preeminent form of expression and aesthetics during the baroque period within the confines of the former Spanish empire.1 The group has also undertaken a 20-year project titled “Baroque Triumphs” (Rodriguez and Mínguez 2020), which has led to the publication of remarkable studies, including: La fiesta barroca. El reino de Valencia (1599–1802) (Mínguez et al. 2010); La fiesta barroca. Los virreinatos americanos (1560–1808) (Mínguez et al. 2012); La fiesta barroca. Los reinos de Nápoles y Sicilia (1535–1713) (Mínguez et al. 2014); La fiesta barroca. La corte del Rey (1555–1808) (Mínguez et al. 2016); La fiesta barroca. Portugal hispánico y el imperio oceánico (Mínguez et al. 2018); and Un planeta engalanado. La fiesta en los reinos hispánicos (Mínguez et al. 2019). It seems fair to say that this team is the most consolidated center of baroque studies in Spain, and they have focused their research on the role of propaganda, imperial cohesion, and the political control of artistic activities during the period. Following a parallel research initiative outlined by Mínguez in his earlier work Los reyes distantes: Imágenes del poder en el México virreinal (1995) the group has also examined the reaffirmation of colonial logics during the period.

1 See Mínguez (2009, 2017), Mínguez and Chiva (2014), Mínguez and Rodríguez (2015), and González et al. (2019).

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

579

Two scholars from the University of Silesia, Oskar Rojewski and Mirosława Sobczyńska-Szczepańska, have collaborated with these Spanish peers in yet another important publication, supported by the Universidad Jaume I of Castellón de la Plana and the Universidad de Valencia. In this book, Courts, Nobles, and Festivals: Studies on the Early Modern Visual Culture (2019), they expand their range of analysis to include other European nations. There are also scholars in America who collaborate regularly with this group. Notable centers for these lines of inquiry include the Colegio de Michoacán and the Universidad Autónoma de México, with important contributions by scholars like Herón Pérez Martínez (2009), Bárbara Skinfill Nogal (2002), and Eloy Gómez Bravo (2002). The Center for the Study of Traditions at the Colegio de Michoacán has embarked on the quixotic task of translating Filippo Picinelli’s Mondo Simbolico (1653), a compilation of emblems that exerted an influential role in the baroque world. Alongside this translation they have developed a collection entitled Emblemata: Estudios de literatura emblemática, thus creating a comprehensive interdisciplinary space enlivened by international seminars and colloquia. They have also published numerous studies of the baroque tradition, including Esplendor y ocaso de la cultura simbólica (Pérez and Skinfill 2002); Las dimensiones del arte emblemático (Skinfill and Gómez 2002), and Creación, función y recepción de la emblemática (Pérez and Skinfill 2012). These works, which predominantly focus on literary analysis, explore, among other topics, the symbolic aspects of baroque celebrations in New Spain and the role of lexical-visual connections in emblematic creation, through the study of sources from the hieroglyphic tradition in different locations and the work of important figures like Horapollus (2011), Alciatus (1995), and the aforementioned Picinelli (2012). Other Mexican scholars have also pursued a specific direction of inquiry that is partly inspired by the extensive and comprehensive work by Nobel prize winner Octavio Paz (1982), whose study Sor Juana Inés de La Cruz o las trampas de la fe (1982) examines the significance of efforts by the criollo populations of New Spain to construct an identity of their own, and the emergence of unique intellectual configurations among these criollos. Paz keenly tracks the negotiations, mixtures, and reworkings of the European tradition that went into developing properly American aesthetic and political formations, and relies on Sor Juana Inés as a narrative thread through which to examine the work of figures like the scholar Carlos de Sigüenza y Góngora, to be discussed below. Outstanding works that follow in this direction include “Saldos del criollismo: el Teatro de virtudes políticas de Carlos de Sigüenza y Góngora a la luz de la historiografía de Fernando de Alva Ixtlilxóchitl” by Pablo García (2009). To continue with this brief overview of academic accounts of the Hispanic American baroque, mention should be made of the book Inventing Lima: Baroque Modernity in Peru’s South Sea Metropolis by Alejandra Osorio (2008), which examines the social and political life of the viceroyalty of Peru and analyses the role of baroque celebrations in that context. Osorio notes that, at least by the time at which she conducted her research, access to and preservation of archives and documentary sources were scarce in Peru in comparison to Mexico. She aims to

580

M. Va´squez Arias and A. Burbano Valde´s

show how the viceroyalty was traversed by dynamics of hybridization and social and cultural mixture that drove baroque cultural life, which leads her to challenge marxist accounts that have promoted a narrow perception of the viceroyalty as an enclave of segregation, domination, and colonial exploitation. We should also mention the work of US scholar More (2012), from the University of Pennsylvania, whose work advances a combination of two theoretical matrices, respectively, derived from Walter Benjamin’s work on the German Trauerspiel and Derrida’s (1997) thoughts on the archive. More explores the process by which a Novohispanic tradition was invented in order to initiate an American project grounded on what she calls a criollo hermeneutics, carefully considering the contradictions and racial tensions between criollos and indigenous, mestizo, and mulato communities in New Spain and in the American continent as a whole. A few noteworthy contributions by Colombian scholars include Fiestas, fastos y duelos. Orden y conformación en Santa Marta, siglo XVIII by Édgar Rey Sinning (2018), and the collective volume Imaginación mediática en Hispanoamérica: Variantología de lo transmedial entre los siglos XVI y XIX (Vásquez et al. 2020). Rey Sinning’s book is an interesting contextualization of baroque celebrations in Santa Marta, Colombia, and it scrutinizes the extended development of baroque logics in that context as well as their close connection to the religious dynamics of the time. The collective volume stems from an effort led by professors Mauricio Vásquez and Diego Fernando Montoya, from the Masters Program in Transmedia Communications at the Universidad Eafit, with the support of Andrés Burbano from the Universidad de los Andes and Sigfried Zielinski from Berlin’s University of the Arts. The first part of the book assembles contributions by 21 authors, all graduates of the program, who articulate media-archaeological and intermedial perspectives in order to reconstruct the transmedia logics proper to Hispanic American baroque celebrations (Vásquez et al. 2020). Individual essays delve into interesting topics including commissions, ephemeral architecture, allegorical poetry, allegorical chariots, and theater. Lastly, it is crucial to mention the work of Humberto Borja (2019, 2021). Borja’s project Los ingenios del papel is an extensive compilation of materials from the Latin American baroque analyzed from the point of view of visual culture, and it exists both as a printed book and as a digital platform. The most important aspect of this project is the effort to analyze the material using data visualization techniques in line with recent work in the digital humanities, relying on a previously assembled database of files and images known as ARCA. As these references clearly suggest, research on the baroque is a current concern, and there is great potential for the use of new techniques and theoretical perspectives in discussions of these outstandingly wide, abundant, and diverse bodies of work. An additional point to be made is that these different studies all agree on the idea that the baroque machine,2 understood as an intermedial rhetorical machine and as

In the scholarly literature the notion of “baroque machine” has become an established way of referring to the combination and convergences to be found in baroque celebratory assemblages of the kind to be analyzed in the following pages.

2

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

581

an emblematic and symbolic construct, functioned as a crucial axis running across ritual, mortuary, and ostentatious celebrations, and negotiations around the distribution of power, which required the articulation of a variety of artistic, symbolic, and expressive forms within the Atlantic system, a territorial complex of intense cultural, technological, political, and commercial exchange between Spain and what was known at the time as the New World (Mínguez et al. 2012, 11). These symbolic machines encompassed triumphal arches, mounds, catafalques, ornaments, and statues, among other architectural, emblematic, allegorical, and artistic constructs, and played a central role in the design of complex intermedial programs produced in connection with celebrations and events across much of the Hispanic American colonial world for over three centuries. We have access to the content of these celebrations and funeral rituals through chronicles and written reconstructions that depict various kinds of composition ranging from the festive to the mournful. This text also include detailed descriptions that yield a vision of the multiplicity of media elements assembled therein and of the attending performances and collective rituals through illustrations, blueprints, and engravings that traveled through transcontinental routes of dissemination and exchange and which clearly attest to the intellectual and aesthetic impact of the criollo and mestizo worlds. These reconstructions document unique ways of orchestrating various trades and intellectual operations in order to attain purposes that were clearly delimited by the social and political life of the age. In this way, artistic, literary, and architectural models were promoted, disseminated, and also put into crisis through a praxis that called for the participation of agents from extremely diverse backgrounds acting in organized but occasionally discordant fashion: the guilds, clerical and aristocratic powers, universities and academics, poets, chroniclers, painters, playwrights, craftsmen, and general trade practitioners, among others. An intense mode of cultural activity thus took form around extensive creative programs and projects that transformed spaces and times in Hispanic American cities.

The Context of Medial Relations America – from North to South and from the sixteenth to the early nineteenth century – was home to complex media articulations that assembled architecture, painting, sculpture, poetry, and pyrotechnics; to this we may add sensorial explorations of smells, flavors, and dances orchestrated through highly charged experiential devices that were installed as vehicles for collective encounter during the baroque period. Thus, “[t]he baroque feast, in Europe and America, was an agglutination of all the arts, ephemeral by definition, which transformed daytime and nighttime urban spaces. The fact that its memory has endured up to the present time is mainly due to published narrative accounts of the festivities – often accompanied by illustrations” (Mínguez et al. 2012, 21). For our current hermeneutic purpose, we will situate ourselves in the seventeenth century and in the viceroyalty of New Spain, whose capital was Mexico City. The

582

M. Va´squez Arias and A. Burbano Valde´s

viceroyalty covered a territory that is much larger than the current nation state of Mexico, including vast expanses of North America, the northern part of South America, several Caribbean islands, and the Philippine islands on the Pacific. The creation of educational institutions like colegios mayores and universities in the Americas gave rise to a lively and extremely stimulating intellectual milieu, encouraged, among others, by the holistic spirit of the Jesuits, studious missionaries who traveled across the Eastern and Western worlds and later the Americas. The effervescence of the Counter-Reformation gained strength through the global expansion of Jesuit knowledge; based on Loyola’s understanding of spiritual exercises as connected to the senses the Jesuits created a range of media practices informed by the devices for creating and projecting images studied by Athanasius Kircher (1601–1680), a German Jesuit based in Rome, which came to be known all around the world through the books that he compiled and published (Kittler 2010, 85). In this world a great diversity of ideas and practices was able to coexist within the same discursive and cultural space: hermeticism and kabbalah, partly disseminated through the popularization of the philosophy of Raimundo Lulio (1232–1316); natural magic, inspired by Kircher’s work and his interest in the revelation and manipulation of the secrets of nature through all manner of artifacts; egyptology and the study of indigenous cultures, whose exotic associations were naturalized on the basis of presumed similarities between forms of representation used in such distant locations, leading to the construction of contrived theories that drew historical links between the two regions (García 2009, 225). The staging of royal celebrations related to various aspects of courtly life in Spain and to events like births, crownings, weddings, and the death of members of the royalty (Mínguez 1995, 23) was one of the most important political and social activities of the period. With the exception of viceroyal entry processions and a few specific events, these celebrations were held in the absence of the person being honored. In that sense, they were commemorative exercises intended to evoke the presence of a member of the royalty through a kind of symbolic intercontinental teleportation. From the point of view of reception, this type of baroque celebration was likewise a prominent public event – in the habermasian sense of representative publicity (Habermas 1981) – in colonial society. Its potential for transforming collective spaces was extremely high and its capacity for sensory impact was probably unlike that of any other event (Mínguez 2009, 2012, 2017; Lorente 1994, 337–338). In one way or another all social strata and sectors were present at these events or took part directly in the processions, whether by building allegorical chariots on behalf of their guilds or by marching on foot – of course, under a hierarchical order based on their origin and social standing. The baroque celebration was a great collective performance whose participants could see, read, hear, think, and perform a complex symbolic weave, intentionally orchestrated and prepared, and funded partly by the crown and partly through contributions and donations by other sectors (the silversmiths, for example). Although it functioned at first as a propaganda apparatus intended to secure political allegiance to the empire by highlighting attributes like grandiosity, spectacularity, and colorfulness (Mínguez 1995, 23),

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

583

Fig. 1 Entrance of viceroy Archbishop Morcillo in Potosí, Museo de América (Madrid). Melchor Pérez Holguín, 1718. This image is one of the most complete extant pictorial representations of a baroque celebration

through a gradual process of transformation it would eventually become a space where the criollo intellectual class was able to consolidate and present its thought and symbolic identity (García 2009) (Fig. 1).

Allegorization and Ekphrastic Relations We understand the baroque as a game of mirrors and a theater of mirages, as a labyrinth of references and intersecting symbolic paths, and as the performance of a simulacrum (Baudrillard 1978) spreading out into colonial urban spaces (Buxó 2009). The creation of visions, sensations, and beliefs was critically achieved by allegory and emblematic production, the latter being a generic designation for a wide variety of lexical-visual forms that includes empresas and hieroglyphs, among others. The baroque age developed a particular interpretation of the horatian phrase ut pictura poesis, placing both the word and the image within a single creative continuum orchestrated by the cohesive power of the symbolic and by the strength of the allegorical. This vision was materialized in emblematic compositions and creations whose influence spread to artistic forms like architecture and painting, and it also set the structural groundwork for the development of ludic and festive forms. Moreover, baroque celebrations and the explanatory accounts of their symbolic programs constitute a unique approach to the relations between visual and architectural and textual production through the creation of interesting ekphrastic linkages, understood as verbal elaborations of productions, whether visual or threedimensional forms (Bilman 2013). To understand how such relations were woven it is necessary to work with concepts that are capable of capturing these unique forms of articulation, which in the present time can arguably be interpreted as forms of inter- and transmediality (Schröter 2011; Elleström 2017) rooted in the concept of

584

M. Va´squez Arias and A. Burbano Valde´s

wit. The range of intellectual skills captured by this concept took on a particular significance in the baroque through academic training in colegios and universities, thanks to two crucial sources: Gracián (2010) and Tesauro (1741). Among the most important contributions to the modern theorization of allegory and the language of images we may mention The Origin of the German Trauerspiel (Benjamin 2012); the section that Panofsky (2016) devotes to “Titian’s Allegory of Prudence;” Fletcher’s (2002) work, entirely devoted to the work of allegoresis within and beyond the baroque context; and Eco’s “The Perfect Language of Images” (1994). Benjamin (2012, 159–192) is clearly committed to a dialectical account of allegorical procedures instantiated by the specific forms of the German drama or Trauerspiel. According to this reading, allegory would be dialectical by virtue of its capacity to designate, articulate, and ultimately synthesize contradictory dimensions. These contradictions take on the form of specifically allegorical tensions between the theological and the artistic, the mythological and the political. Allegory would then be a way of systematically exhibiting these tensions by hiding them, and it would thus be expressive of the conflict between sacred value and profane comprehensibility. Lastly, Benjamin notes that allegorical hieroglyphs synthetically represent extensive encyclopedic elaborations and exemplify an incursion of the figurative arts into the framework of discursive representation (Horts quoted by Benjamin 2012, 179).

The Story of a Misreading In Benjamin’s account, allegorical forms of expression developed out of an erroneous understanding of Egyptian hieroglyphs by fifteenth-century humanists. The misunderstanding was instigated by Alberti (Praz 1989, 24–25) through his reading of the pseudo-epigraphic work Hieroglyphica by Horapollus (2011). Humanist scholars cultivated the notion that hieroglyphic constructions encoded a lost form of wisdom (Eco 1994, 127–128) and set out to write their own by using images of things, and to configure enigmatic inscriptions that were then embedded in medals, columns, triumphal arches, and other artistic products of the Renaissance (Benjamin 2012, 170–171). In that sense, the consolidation of emblematics as an art form during the baroque could be understood as an example of what Bloom (1992) calls misreading, or as the involuntary invention of a tradition. Consequently, the definition of symbolic programs could be understood as an extension of that practice, although it is one that gave rise to an imaginative discipline and a compelling and widespread form of expression. Kircher’s studies of hieroglyphs may also be understood as a contribution to this invention of a tradition, bearing in mind that his work was incredibly influential for the Novohispanic baroque. Kircher’s inquiries converged with polygraphic efforts to consolidate a universal language and to design an artifact, a steganographic machine or ark, that would be capable of encrypting and decrypting messages through a kind of universal translator (Zielinski 2011, 198–219; Gómez 2019, 30–31) (Fig. 2).

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

Fig. 2 Athanasius Kircher, Oedipus Aegyptiacus, Tomus I. “Elogium XXVII”

585

586

M. Va´squez Arias and A. Burbano Valde´s

To some extent, the same intention remained alive in allegorical expression, and baroque machines can be understood as an effort to materialize it. This configuration would also absorb elements from the work of authors like Comenius, who attempted to consolidate a graphic repertoire to be used in the promotion of literacy and an emblematic compendium for the teaching of Latin. Comenius’s The World in Images (Comenius 2017) thus aimed to compile, for educational purposes, “the images and names of all the fundamental things in the world and the activities of life.” The connections that we are drawing here are by no means gratuitous; instead, they provide us with the cognitive context that allowed for the emergence of a form of expression proper to Hispanic American baroque celebrations, and draw our attention to important works that, through their dissemination by the Jesuits, came to function as a creative repertoire in the Novohispanic context. Thanks to Kircher, in this context “hieroglyphic configurations thus became a sort of hallucinatory device in which all possible interpretations could merge” (Eco 1994, 140). Eco argues that hieroglyphic writing fulfilled a twofold function: on the one hand it worked as an ideogram, that is, as a representation that partially relies on iconic procedures (appealing to the resemblance between an image and what it represents) or rhetorical substitutions; on the other, when certain concepts proved impossible to represent through these two mechanisms, the images could also function as phonograms (Eco 1994, 128). This field of signification wherein abstract concepts are replaced by concrete forms and thereby hidden, encrypted, and rendered enigmatic, sheds light on the kind of artistic playfulness that was most proper to the baroque. Known generally as emblematics, although it relied on other lexical-visual devices like empresas or divisas, this mode of expression instantiates a historical way of relating word and image, and of composing structures endowed with higher levels of intermedia complexity by integrating them in systems that might encompass architecture, painting, and other components brought to play in the staging of a celebration. As noted above, these assemblages were by no means a mere trend; rather, they were rooted in a complex system of thought and adhered to a set of compositional rules that even reached into the domains of the private and the individual. According to Mario Praz: “Emblems are [. . .] things (representations of objects) that illustrate a concept; epigrams are words (a concept) that illustrate objects (such as a work of art, a votive offering, a grave). The two terms are thus complementary [. . .]” (Praz 1989, 24). Emblematic and epigrammatic forms were both part of an ekphrastic system that allowed for a high degree of bidirectionality, mutual reference, and codependency between these two kinds of expressive resource and whose enigmatic products imposed considerable hermeneutical challenges, which partly explains why this art form can be so captivating as an object of study. Enigmaticity as a principle met a variety of motivations and aims within those medial assemblages. Firstly, to hide hermetic knowledge from the profane was understood as an intellectual amusement that could attest to the sagacity of their authors, and it entailed an invitation to engage in the game of deciphering concepts and moral notions. Those who interacted with these constructions were also compelled to walk through the assemblages, examine the inscriptions, and try to make

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

587

sense of the references, sources, and odd figures. Enigmatic creativity can thus be understood as an interesting motor that could provoke addressees to explore symbolic programs embodied in a wide range of emblematic forms and direct them to search beyond the explanations provided by their authors and a superficial layer of meaning.3

Symbolic Programs and Commissions Only such a polymorphous world could have produced a figure like Carlos de Sigüenza y Góngora (1645–1700), a criollo (that is, the son of Spaniards born in American lands) whose sensibility, erudition, and curiosity led him to take a simultaneous interest in several disciplines, including mathematics, rhetoric, archaeology, and astrology (at the time not yet differentiated from astronomy in its contemporary sense). Drawn by the general spirit and intellectual renown of the Jesuit order, Sigüenza set out to train for priesthood at the age of 15 and spent over 7 years studying theology and what we would now describe as humanist studies (Leonard 1984). After one or several nightly escapades – the exact number is not certain – to the city of Puebla, he was eventually expelled from the Company of Jesus on August 3, 1667, an event that set a bitter point of inflection in his biography and which sheds light on a few of his character traits: he was studious but adventurous and rebellious, somewhat irascible and resentful (Leonard 1984). After this episode Sigüenza continued his studies at the Real y Pontificia Universidad de México, where he devoted himself to canonical studies and history. He would later become a professor at the same institution, apparently not too committed to his teaching duties on account of his various occupations and inquiries. One of his many passions was the study of indigenous languages, and it seems that he may have been proficient in náhuatl. His thirst for knowledge also led him to become a fervent collector of codices, archives, and artifacts from the indigenous world, along with a variety of optical and measuring devices. In a lucky break he was entrusted with the documents and archives of the castizo Fernando de Alva Ixtlilxóchitl (1578–1650) by the latter’s son, Juan de Alva Cortés, which held pictographic documents, commentaries, transcriptions, and original manuscripts from Mexico’s ancient inhabitants (García 2009, 223). These biographical and circumstantial twists and turns prepared the ground for Sigüenza’s Teatro de virtudes políticas, one of the most interesting works of the Novohispanic baroque. Sigüenza’s Teatro has been analyzed from literary, historical, and political perspectives, but we would like to argue here that only an intermedial approach 3

Here it might be interesting to draw a parallel with the central role played by practices of deciphering in the contemporary theory and practice of inter- and transmediality. Henry Jenkins (2017) has argued that one of the logics of transmedia production and orchestration is inspired by alternative reality games based on solving mysteries, deciphering enigmas, and solving puzzles. A similar principle could be said to provide the mechanism that enlivens the baroque assemblages that we are discussing, and it was a common practice among their authors.

588

M. Va´squez Arias and A. Burbano Valde´s

(Elleström 2010) will capture the full significance of this work and the intellectual and creative character of its author. According to Lorente (1994, 363) the work is an effort “to transfer the written language to the artistic-symbolic disposition of the arch. The written text is thus clearly directed at fulfilling the horatian phrase ut pictura poesis, which set the challenge of simultaneously harmonising different aesthetic codes.” To describe it synthetically and narrowly, Sigüenza’s Teatro is, to begin with, an ekphrastic text, that is: a textual description of a visual and spatial assemblage comprising refined symbolic materials (Bilman 2013), designed and composed by Sigüenza himself to be installed in an architectural structure. Although the text is a depiction of the assemblage, the latter was also materially realized. The written account displays a broad knowledge of both history and mythology, and deploys a wide range of scholarly resources to provide a justificatory framework loaded with dense moral and political intentions. In addition to that, it is also a record of a set of aesthetic and conceptual guidelines, accompanied by some instructions, that can be used during a process of artisanal and artistic construction, something analogous to a technical script for a performance and an exhibition installation. The process thus described calls for the work of several artists and can accordingly be understood, following Martin (2009), as something like a multimedia installation, to describe it in contemporary terms. The context for this commission was a typically baroque form of celebration to mark the arrival of the Conde de Paredes y Marqués de la Laguna, and his wife María Luisa Manrique de Lara, in November of 1680. The ceremony, known as a recibimiento, was a complex political ritual that would establish the couple as the ruling viceroys of New Spain. Following the Roman tradition of the triumphus,4 two ephemeral architectural structures were prepared; Sigüenza was commissioned to design one of them, and his Teatro de virtudes políticas (Sigüenza 1984) was the result of that commission. Sor Juana Inés de la Cruz was the recipient of the second commission, and the results are to be found in her work Neptuno alegórico (Cruz 2009). These structures were intended to serve as the symbolic center for the viceroy’s complex entrance ritual, and it was customary for the local cabildo to assign the task of their design to well-known scholars from the clerical sphere, like Sor Juana, or from academic institutions, like Sigüenza. These scholars were expected to demonstrate attributes such as erudition and wit in their design (Cuadriello 1994). First and foremost, they were selected because they were regarded as recipients of knowledge of the rhetorical, literary, and artistic skills required for the composition of allegorical constructs that would often include a lema or mote, an image or symbol, and an epigraph composed in different poetic meters, each of which would have diverse implications in light of the theories of mimesis, semiotics, syntax, and rhetoric that were in vogue at the time (Graham 2012, 27). While Sor Juana can be described as self-taught, Sigüenza’s skills were obtained through academic training.

4

A “triumph” was a military reception ceremony celebrating a victory by the emperor.

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

589

The task called for the creation of a symbolic or iconographic program that could be perceived as transversal to the entire assemblage and could thus function as an articulating motif for the celebration. As we have mentioned, the triumphal arch was the centerpiece of the spatial arrangement. The arch was an ephemeral structure in which ordinary materials like paper maché, cardboard, and stucco were used to create a sense of volume and simulate long-lasting materials like marble and bronze (Vincent 2009, 27; Mínguez 2012). The structure would be up for the duration of the celebration – somewhere between 3 days and a week – and after that it would be disassembled so that the materials and structure could be used again in other celebrations and events. To compose the symbolic program authors like Sor Juana and Sigüenza relied on an assortment of lexical-visual forms that could be combined and interpreted according to guidelines encoded in canonical works used in rhetorical training in the universities. One such work was Catalejo artistotélico by Emanuele Tesauro (1741), where readers could find an assortment of empresas (heroic symbols), emblems (moral symbols), and hieroglyphs (doctrinal symbols), alongside a few fantastic, allegorical, and metaphorical ornaments. As Mínguez has noted, “the use of emblematics in Mexican ephemeral art is such that all the different elements [. . .] – paintings, poems, etc. – appear articulated around an emblematic nucleus that then ramifies widely in a variety of artworks” (Mínguez 1995, 27). An author who was commissioned to work on such a project was also expected to be acquainted with the tradition of classical mythology, history, the biographies of those being homaged, the architectural forms and styles typically used in such events, and those elements of pictorial and sculptural theory that would allow them to oversee the material realization of the work alongside local craftsmen and artists. According to Cuadriello (1994, 85–86) the work “required scholars to come into contact with master painters (and to a lesser degree with woodworkers and architects) so that the motes and epigrams could be endowed with a ‘body’ or a ‘plastic depiction.’” Sigüenza’s work is unique in this respect, since he chose to disregard the customary repertoire of mythological motifs and decided instead to develop his own, dense hermeneutical framework in order to create connections between the world of ancient Mexico, Egypt, and the Olympic pantheon (Sigüenza 1984, 182–183; Mínguez 1995, 39). Sigüenza’s aim was to validate the use of indigenous motifs and to acknowledge the history of American peoples who were decimated by the colonizers, and he accordingly conceived a triumphal arch with 12 boards devoted to the same number of Aztec kings. These local historical figures were to function as a mirror and as a guide to the political virtues for the new viceroy, and each board elaborated on the meaning of the name, the virtues, and the deeds of the selected Aztec rulers: Huitzilopochtli, Acamapich, Huitzilihuitl, Chimalpopocatzin, Itzcohuatl, Motecohzuma Ilhuicaminan, Axayacatzin, Tizoctzin, Ahuitzotl, Motecohzuma Xocoyotzin, Cuitlahuatzin, and Cuauhtémoc. Sigüenza thus came up with an intelligent and ingenious way of recovering indigenous history within the framework of a ritual that originated in the Roman world and was later adopted by the Spanish monarchy and replicated in colonial territories. This ritual was now deconstructed through the cunning of a criollo

590

M. Va´squez Arias and A. Burbano Valde´s

scholar who was interested in the history of Mexico’s indigenous peoples and who intended to acknowledge their relevance in the colonial world as a factor in the emergence of a specifically American historical consciousness. Sigüenza’s project displays the kind of dissonant mixtures that are proper to the baroque and a syncretism that was typical of the criollo intellectual classes during this formative stage. Moreover, his work clearly relies on the notion that the indigenous world needs to be translated by someone else, and on the assumption that indigenous peoples cannot themselves be trusted with that task. Let us read Sigüenza’s own description of his work which, as he put it, “was one of the most exquisite and unique things seen in these times, according to the opinion of those who are versed in the arts” (Sigüenza 1984, 185): This most wonderful machine of colors was enlivened [. . .] with the passionate spirit of the Mexican emperors, from Acamapich to Cuauhtémoc. They were joined by Huiczilopotchli, not simply in order to fill out the number of boards, but because he was worthily deserving of the praise, for he was the one who led them from their homeland, whose location remains unknown, to these provinces, known in ancient times as Anáhuac. [. . .] They were portrayed adorned with feathers in many hues and wearing the costume that they most prized. (Sigüenza 1984, 187)

Arguably, and duly maintaining proportions and historical distance, Sigüenza’s work can be described as an early instance of intermedia design. His project envisioned a convergent mode of composition for installing and exhibiting ideas, nourished by a variety of semiotic and aesthetic resources and offering content to be seen, read, heard, thought about, and made. If we examine the expressive capacity of this baroque construct it is clear that the project allowed for a plurality of simultaneous pathways for interpretation, reading, and decipherment: although it can be read chronologically as a historical account of the 12 Aztec rulers, the way in which the images were arranged in the constructed arch made room for other interpretations (Lorente 1994, 364). The readings produced by Sigüenza himself and by many of his interpreters indicate that this plurality was consciously cultivated in a manner reminiscent of contemporary notions of hypertextuality (Landow 1997). Sigüenza himself refers to this strategy when he notes that “the topic that I will be deciphering requires me to explain it, not by following the order of the boards that all could see, but according to the chronology of the Mexican empire, with which I am acquainted” (Sigüenza 1984, 194), which clearly shows that chronology was not the structuring guideline for the arrangement. As Graham has pointed out, “we often note that emblem makers want to impose on us, the readers, an order of reading that is not sequential but almost ‘hypertextual’” (Graham 2012, 28).

An Intermedial Way of Thinking and Creating This creative regime thus enabled a way of organizing knowledge expressively, through the depiction of a structure that goes beyond the specific content of each object in the assemblage. It likewise called for the adaptive use of diverse materials,

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

591

techniques, and arts while promoting wit as a criterion for skill. Authors could demonstrate this wit by following the guidelines of a scholarly system of thought (Tesauro 1741) and employing different types of construction: oratorial, or fixed in discourse; lapidary, or fixed in lasting materials and spatial forms; and symbolic, or fixed through metaphorical forms in empresas, emblems, hieroglyphs, and ornaments, among others. Moreover, authors like Sigüenza were expected to be able to produce and coordinate actions in the short term. As Buxó points out: If we measure the amount of time that passed between the arrival of the Marqués and Marquesa de la Laguna in Veracruz and the official ceremonies during which they were sworn in and took possession of the cabildos, we realize that the scholars who conceived these ceremonies, as well as the architects, painters, and woodworkers who were in charge of building and ornamenting the symbolic constructions, had less than three months to carry out their enormous task. (Buxó 2007, 43)

It is worth pointing out that this humanist creative regime, provocatively intermedial, did not promote disputes between the different arts, techniques, and disciplines, but rather brought them all together in order to meet a specific aim in response to a particular occasion. Likewise, the intellectual skills of wit, fantasy, and erudition (Cuadriello 1994, 85; Farré 2007, 9) were cultivated as a way to process a range of available knowledge and as a source of intellectual pleasure to be derived from solving the riddles and enigmas. Even the use of mathematical structures inspired by pythagorean and platonic schemas, and invested with hermetic significance, was framed as a compositional resource, as Sigüenza coyly allows his readers to see when he explains the numerical implications5 of the underlying structure of his assemblage: It would have been unwise to combine these twelve emperors with the twelve patriarchs or celestial signs (an achievement that would have required a craftier pen that my own), since the arithmetics of Pythagoras, the philosophy of Plato, the theology of Orpheus, and the warnings of Pedro Bungo de Míster, Numeror, page 386, contain plenty of extraordinary delights that could be used to adorn this number. Since my intent is not to procure useless curiosities to be looked at, but rather to represent heroic virtues as examples, I was forced to forsake those exterior ornaments which virtue does not fancy. (Sigüenza 1984, 187)

Here Sigüenza feigns to apologize for busying himself with useless curiosities in order to call attention to what is in fact a fairly complex, erudite, and creative intellectual operation. He acknowledges outright that the choice of 12 historical characters, each of which was mapped onto a virtue, is not necessarily original, since there was an established use of sets like the 12 patriarchs or celestial signs to create

5

As mentioned above, Sigüenza had engaged in the study of mathematics from early on. Numerical relations play different roles in the compositional strategy of his Teatro. Lorente (1994, 367–368) has examined the poetic symmetry between the 12 epigrams, each of which contained 12 verses, with 4 in redondillas, 4 in octavas reales, and 4 in décimas; each of these structures had a material equivalent in the triumphal arch.

592

M. Va´squez Arias and A. Burbano Valde´s

symbolic correspondences between sets having the same number of elements. Sigüenza also composed epigrammatic descriptions of each of the 12 emperors, and each of these short poems was itself a 12-verse structure, with 4 octosílabos, 4 octavas reales, and 4 décimas. Thus, the metric of the poem resonates in the narrative structure, which in turn resonates in the spatial construction, so that the mysticism of numbers could be used to connect the abstract world with the historical world and with the spiritual world of the virtues. We see then that Sigüenza envisioned his assemblage as an invitation to discover relationships that were laboriously woven and paradoxically highlighted in a written text where he denies having drawn them. In this way, Sigüenza creates a space for the conjunction of different dimensions of culture without neglecting to present the topics in a manner accessible to an ordinary understanding, by conceiving symbolic constructs that were capable of offering multiple levels of interpretation. As Sagrario López explains, the iconography used in these constructions referred back to three basic elements established by the classical rhetoric of Cicero and Quintilian: to teach (docere), to give joy (delectare), and to convince (movere), and they required the mandatory rhetorical steps: inventio, dispositio, elocutio, actio. In order to prepare the program authors applied the rules, codes, and systems used to configure rhetorical discourse. (López 2003, 245–246)

Authors like Sigüenza had to make sure that their creations would satisfy the needs of both the highest social spheres and the popular illiterate sectors, as well as those of people who were semiliterate or who engaged in other forms of symbolic practice that may not have been taken into consideration in their intellectual milieu (Gruzinski 1994, 2016). To achieve this, they had to devise strategies of creative collaboration, relying on the support of artists and craftsmen whose names were often left out of the accounts of the celebrations; occasionally they were also required to engage in complex negotiations with members of different guilds, as noted by Acosta et al. (2020). In light of this we may say that the production of symbolic programs was a complex process of organizing intellectual activities and creative teamwork to meet strict temporal and representational conditions using local resources. The authors who were assigned these commissions put their prestige, and their visibility in the viceroyal context, on the line. If we thus consider the skill required to deploy this range of expressive resources and to find the means most suited to the circumstances, we can regard this practice as a unique precedent through which to understand contemporary processes of intermedial convergence and weaving. As López argues, “all festive programs follow a guiding thread derived from a unitary idea. Their authors must apply their erudition and rhetorical technique to show their capacity to establish relationships and correspondences, in other words: to show off their wit” (López 2003, 246). According to Mínguez (2017) such guiding threads could be variously drawn from the discourses of mythology, meteorology, botany, astronomy, eschatology, or from sources such as bestiaries and accounts of the mores of indigenous peoples.

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

593

This unusual effort to achieve a coherence between a whole and a set of parts that, although different, remain within a shared thematic-allegorical axis, has been described as “conceptism,” understood as a form of expression based on the exercise of wit. As Blanco explains, “[t]he style of an author is described as conceptist when it abounds in concepts. And what is a concept? The same as a witticism. And what is a witticism? Well, many things, [. . .] an obscure metaphor, or one that has been stretched beyond the range of convenience, a forced antithesis, a play of words, a false subtlety, etc.” (Blanco 1988, 15).

Conclusions: Intermedial Encounters Between the Baroque Machines of Sigu¨enza and Sor Juana Ine´s This brief overview of the main components of the Novohispanic baroque system of thought and expression allows us to reconstruct the development and unique features of the triumphal arch designed by Sor Juana Inés de la Cruz, as described in her work Neptuno alegórico (2009). As noted above, Sor Juana’s project was closely linked to Sigüenza’s, and although much has been written about these two works (Saucedo 2007; Torres 2019) little has been said from the point of view of intermediality. According to Buxó: the mise en scène of Neptuno alegórico was conceived as a total spectacle or representation wherein a knowing fusion was achieved between the various resources of poetry, painting, set design, and, in the case of the entrance ceremonies as they were performed [. . .], music, dance, lighting, and fireworks, all of it in conjunction [. . .] with a revered corpus of scholarly knowledge, both sacred and profane, ancient and modern, that was put to use by Sor Juana’s fertile imagination. (Buxó 2007, 53)

We have previously analyzed the use of such mise en scène procedures in Sigüenza’s Teatro de virtudes, and a similar line of analysis can be applied to the work of Sor Juana Inés. In order to do so, we will now list a few key features of Neptuno alegórico that will allow us to weave a relationship between the meanings of the two works (Fig. 3). As we noted earlier, both Sigüenza and Sor Juana Inés were commissioned to design triumphal arches for the same celebration. Sor Juana Inés de la Cruz was initially offered 150 pesos reales in exchange for her work, although the cabildo later decided to raise the sum to 200 pesos reales to acknowledge her efforts (Saucedo 1999). Neptuno alegórico is often described in the literature as a work of prose, but in fact it is an example of mixed writing, since an entire section was written in verse. Overall, the text is to be read as an ekphrastic exercise providing a programmatic subtext – as the software, one might say – for a piece of ephemeral architecture, a triumphal arch celebrating the arrival of the new viceroy in New Spain, a hardware that is no longer extant and to which we have access today only through written accounts. The text is divided in four parts: the first is a dedication on behalf of the

594

M. Va´squez Arias and A. Burbano Valde´s

Fig. 3 Cover page of the princeps edition of Neptuno alegórico (Mexico 1680)

Iglesia Metropolitana de México; the second outlines “the reasoning behind the allegorical factory and the way in which the fable is applied,” where Sor Juana provides a scholarly explanation of her use of Neptune as the metaphorical embodiment of the new viceroy; the third is a “detailed account of the inscriptions and canvases,” where each component of the ephemeral architecture is explained with an extremely high degree of literary, visual, and spatial detail; the final section, which is also the most complex from a literary point of view, is an explanation of the arch in verse.

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

595

Sor Juana Inés conceived a symbolic program aimed to establish a parallel between the Marqués de la Laguna and the God Neptune. The program was embodied by an allegorical assemblage that included eight large “canvases” on the top part of the arch and six relatively small “hieroglyphs” on the bottom part. The canvases were narrative and emblematic paintings. The hieroglyphs were more abstract and could be more adequately described as emblems or empresas. Each of the fourteen paintings had its own lema painted on a wooden strip; and at the foot of each there was an “epigram,” a poetic text written in calligraphic script. These [. . .] enormous emblems were inserted into the structure of the arch – which was readjusted following the author’s indications – to be seen by an audience of hundreds – if not thousands – of people on the day of the entrance celebration. Sor Juana relishes the multiplicity of meanings to be potentially drawn out of things and words. (Martin 2009, 31–32)

Buxó has attempted a reconstruction of the Neptuno alegórico that stands out for several reasons. On the one hand, his analysis is an early and convincing attempt to interpret the work from an intermedial perspective, consistently drawing attention to the implicit and explicit relations between diverse art forms and establishing a historical background for the baroque understanding of these relations. According to Buxó, Simónides de Ceos and the “last baroque treatise writers” determine the tradition that set the guidelines for producing forms of reciprocal interaction between the visual arts and poetry (Buxó 2007, 51). This claim partly supports the thesis that the existence of powerful and suggestive intermedial practices in the baroque, as evidenced by specific types of encounters between the arts and media, was grounded on a body of reflection about that convergence as such, which was known to and skillfully applied by Novohispanic intellectuals. As Buxó argues: The main political function assigned to these symbolic machines was certainly that of transforming the everyday city into an artificious and autonomous space that could set the stage for the colorful courtly rites of apotheosis of the earthly monarchies or, in any case, of their respective vicars. But this does not exhaust their significance. If we take their artistic singularity into account, we can describe them as one of the most ambitious achievements of baroque poetics: the production of symbolic apparatuses wherein poetry and painting work together and complement each other, whether on a canvas, a triumphal chariot, a printed work, a dramatic performance, or an architectural monument. (Buxó 2007, 49–50)

In these lines Buxó distances his account from customary explanations of these art forms that read them within the framework of imperial propaganda and colonial subordination, and emphasizes the existence of a system of media that enabled the presentation of symbolic programs through a creative regime wherein the sister arts of painting and poetry provided a matrix for articulating different applications and uses of the same thematic motifs or concepts in allegorical chariots, images, theatrical performances, printed works, and architectural constructions. Buxó also stresses the simultaneously political, artistic, and philosophical role of these “machines,” arguing that they were much more than fanciful or artificious endeavors. In his view, these machines established a unique site for the expression of thought by allowing for an ambiguous form of play between fact and spectacle. From that point of view, baroque symbolic programs can be described as a

596

M. Va´squez Arias and A. Burbano Valde´s

compelling and early example of the use of intermedial convergence for purposes that go beyond entertainment, and as a critical landmark for understanding processes of transmediation (Elleström 2017) that exceed the fields of fiction or narrative, without denying or disregarding their importance. Buxó argues that the title of Sor Juana’s assemblage offers important suggestions for understanding her unique approach to this conversation between the spectacular, the pragmatic, and the artistic. In his reading, Neptuno alegórico fulfills two central purposes: The first was to [. . .] represent, through diverse and concurrent semiotic means, the political and moral virtues that an ideal ruler should have and which, of course, the new viceroy shall have [. . .], under the assumption that [. . .] the new Mexican ruler will exceedingly satisfy his vassals’s expectations of justice and happiness. The second was to render that representation or political simulacrum into an extremely complex work of art that would depict the confluence, analogous to that of rivers flowing into the ocean – Neptune’s domain – of all the “colors” or chosen rhetorical resources. But that is not all, since all of the arts, at least those that address the senses of hearing and sight (in particular poetry, painting, and sculpture) will contribute their own means of mimesis and thereby construct a “lucid” allegorical spectacle: luminous by the coloring and brightness of its canvases and images, but also by the metaphorical and intellectual dazzle of its “reasons.” (Buxó 2007, 50)

Although it may seem like a space of creative freedom, the production of a triumphal arch by commission was embedded in an understanding of poetic creation as an activity compelled to adhere to strict constraints and placed under the regulatory gaze of the cabildos and ecclesiastical authorities that functioned as the patrons. Similar conditions determined the development of justas and poetic contests where poets crafted complex interpretations and connections with the aim of obtaining awards and public recognition through texts whose mandate was to rescue the truths of classical knowledge by aligning them with christian dogma. Thus, it makes no sense to apply the modern understanding of creative freedom to this framework in which artistic skills were strictly codified by rhetorical knowledge and the authoritarian weight of government institutions and censorship. In spite of this, the intertextual, as a primary form of the intermedial (Bruhn 2016), weave established in Sor Juana’s Neptuno alegórico and Sigüenza’s Teatro de virtudes is interesting in that it exemplifies the capacity of their authors to overstep these conditions for literary and artistic production. A reason for this is that the two authors were bound by feelings of complicity and friendship that led them to share their symbolic programs during the writing process and thus allowed them to explicitly refer to each other’s work. This also allowed the two authors to play with the contrast between their individual strategies: while Sor Juana covered her emblematic assemblage with a high dose of flattery in order to diplomatically disguise a request for improvements to be made in the city of Mexico, Sigüenza used his work to convey the criollo appropriation of a forgotten indigenous past, which he managed to display quite openly through an ingenious recourse to widely accepted sources of validation. The fact that, in this case, the two works established an additional layer of meaningful interaction between them, entails that this intermedial assemblage was endowed with an additional level of complexity produced by the two interwoven

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

597

Graph 1 Explanation of the intermedial complexity levels of the projects by Carlos de Sigüenza and Sor Juana Inés de La Cruz

symbolic programs and the ephemeral architectures and performances that embodied them in the context of the celebration. The following schema explains the different levels that make up this complex weave (see Graph 1): A comprehensive analysis of Sigüenza’s and Sor Juana’s projects must take stock of at least three levels that correspond to three different kinds of intermedial relations to be found in each of the two works. In the first level we find the written accounts (Sigüenza 1984; Cruz 2009) in which the authors present an overview, a justification, and an explicit set of theoretical, mythological, and historical references. Here the two works can be read as enigmas that enter into conversation with each other through a typically intertextual encounter, accompanied by explanations that allow for their decipherment. Paz offers a very compelling account of this level of interaction between the monumental enigmatic construct and the keys provided by the explanatory texts: Triumphal arches underwent the same evolution as the rest of the arts during the baroque period. As if they were not only a physical object but a materialized concetto, they progressively took on the form of monumental enigmas. The canvases and all available surfaces in walls and columns were covered in reliefs, medals, emblems, and inscriptions. The monument became a text and the text became an erudite charade. To decipher the meaning of the monument one was forced to resort to wise explanations and meditations. Like Duchamp’s Great Glass, unintelligible without the notes compiled in the Boîte verte, the triumphal arches of the baroque age had to be accompanied by a book in which the meaning of the paintings, emblems, and inscriptions was ingeniously explained with the help of the most extravagant erudition. (Paz 1982, 201–202)

In addition to this intertextual conversation between the Teatro and the Neptuno the two authors recorded their positive appreciation of each other’s work; in fact,

598

M. Va´squez Arias and A. Burbano Valde´s

Sigüenza seems to have made some critical remarks about Sor Juana’s creative process, to which the nun in turn replied (Paz 1982; García 2009). In the second level we find descriptions to be used as instructions for the production of visual and architectural works (structure, color patterns, and iconographic references). Here, the texts prefigure a structure that remains entirely virtual in its written form, but that will later be effectively realized by architects, painters, and craftsmen, and placed at the center of the ceremonial celebration of the viceroy’s triumphal entrance. The text here sets guidelines to be followed by these future makers, but it also functions retrospectively as the sole existing record of the existence of a specific cultural object whose limits grow blurry in our imagination even as we engage in a relatively well-informed hermeneutical exercise. On this level we encounter a set of relations between text, construction, and exhibition, as well as the definition of a mechanism by which it can be said to endure after its natural dissolution, since it is understood as an ephemeral construct. This, then, is a kind of textuality that functions as an imprecise trace in the absence of an accompanying image that can determine certain limits, as in the books of emblems and festive accounts illustrated with copious engravings. Both texts are thus able to create a kind of temporal distension between the composition, the realization, and the afterlife of their respective assemblages. Their nature can accordingly be described as a sort of performative ekphrasis, that is, as a textual account of an object (or a set of objects) encompassing its conception, construction, and recollection after having disappeared. In the third level we find the boards, which enable: (1) a space for relational play between image and text through the combination of lemas, images, and epigrams; and (2) an expansion of the motif chosen by the authors for the celebration, which each board elaborates in a unique way, although in articulation with the symbolic program and its expressive, ideological, and moral intent. The boards can also be read as a platform for weaving new connections to the rest of the baroque intermedial system: poetic jousts, masquerades, processions, allegorical chariots, and other constructs that were used to call attention to the play of erudition and wit during the celebration. If we were to restrict our analysis to the emblematic boards alone, with their allegorical strategies and their decipherment, they would not be sufficient to account for processes by which dissimilar characteristics were relayed through miscellaneous media. Although individual emblematic boards complemented and expanded on each other, they remain part of the same intermedium, while transmediation, in the sense proposed by Schröter (2011) and Elleström (2017), only occurs through the game of total spectacularization and participation in the celebration by a variety of social players (hierarchically organized, of course). By way of synthesis we may say that the first two levels in our analytical schema allow us to draw exclusively intertextual and ekphrastic relations,6 while in the third

6

In this context, the use of the concept of intertextuality includes the semiotic modality as a starting point that does not exclude the other modalities of intermedial description: material, sensory, and spatiotemporal proposed by Elleström (2010).

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

599

we also encounter processes of transmediation, understood as a formal feature that is displayed and shared in the set of constructions and expanded in other media beyond the emblematic boards. There are several components in the Neptuno that can be said to refer to computational processes and indicate that the paths through which such concepts have been historicized are as deep as they are complex – and that they predate the development of digital computers as such. On the one hand, as we have noted, the assemblage is structured by an instance of software, that is, a symbolic program accounted for by the literary text in exceptional detail. This program operates at the conceptual level in a very precise way, using metaphor in order to communicate a content that is filled with references to the present and the past. Other components of Sor Juana’s work can also be said to shed light on contemporary processes. On the one hand, the assemblage relies on encrypting, specifically in its use of hieroglyphs to establish a parallel between the viceroy and the mythical Roman deity. The hieroglyphs encode this parallel through constructions whose meanings remain latent and are only open to those who have the key by virtue of their acquaintance with classical literary references. Indeed, hieroglyphics were understood as an Egyptian technique that could be used to render the invisible visible. In addition to this, the Neptuno was also a work of political theater, since its author ventured to propose a system of recommendations for the present time, calling on the new ruler to focus his attention on specific local issues (like the canalization of the lake over which Mexico City was built, or the unfinished construction of the city’s cathedral) and to use his power to resolve them wisely. For these and other reasons, an intermedial understanding of baroque expressive assemblages in Hispanic America may offer us new ways of looking not only at this age, but at the intersections that contemporary technologies are now beginning to weave among themselves.

References Acosta, José, María Camila Borrás, et al. 2020. Comisariado, organización e invención del programa simbólico de las fiestas barrocas entre los siglos xvi y xviii en Hispanoamérica. In Imaginación mediática en Hispanoamérica: Variantología de lo transmedial entre los siglos XVI y XIX, ed. Mauricio Vásquez, Diego Montoya, Andrés Burbano, Siegfried Zielinski, et al. Medellín: Editorial EAFIT. Alciatus [Alciato]. 1995. Emblemas. Madrid: Akal. Baudrillard, Jean. 1978. Cultura y simulacro. Barcelona: Kairós. Benjamin, Walter. 2012. El origen del Trauerspiel alemán. Madrid: Abada. Bilman, Emily. 2013. Modern ekphrasis. Bern: Peter Lang. Blanco, Mercedes. 1988. El mecanismo de la ocultación. Análisis de un ejemplo de agudeza. Criticón 43: 13–36. Bloom, Harold. 1992. La cábala y la crítica. Caracas: Monte Ávila Editores. Borja, Jaime. 2019. Los ingenios del pincel: Geografía de la pintura y cultura visual en América colonial. Caracas: Editorial Universidad de Los Andes. and a hypermedia that complements the physical book available on the following website https://losingeniosdelpincel.uniandes.edu.co/ #!/intro/1

600

M. Va´squez Arias and A. Burbano Valde´s

———. 2021. Los ingenios del pincel: Geografía de la pintura y cultura visual en América Colonial. Caracas: Editorial Universidad de Los Andes. Bruhn, Klaus. 2016. Intermediality. In The international encyclopedia of communication theory and philosophy, 1–12. Chichester: Wiley. Buxó, José. 2007. Poética del espectáculo barroco: El Neptuno alegórico de Sor Juana. In Teatro y poder en la época de Carlos II: Fiestas en torno a reyes y virreyes, ed. Judith Farré. Barcelona: Universidad de Navarra-Iberoamericana Editorial. ———. 2009. Arco y certamen de la poesía mexicana colonial (siglo XVII). Xalapa: Universidad Veracruzana. Comenius, Johanes. 2017. El mundo en imágenes. Esto es: Imágenes y nombres fundamentales en el mundo y de las actividades de la vida. Barcelona: Libros del Zorro Rojo. Cruz, Juana Inés. 2009. Neptuno alegórico. Madrid: Cátedra. Cuadriello, Jaime. 1994. Los jeroglíficos de la Nueva España. In Patronato del Museo Nacional de Arte. Juegos de ingenio y agudeza: La pintura emblemática de la Nueva España. Madrid: Banamex-Accival. Derrida, Jacques. 1997. Mal de archivo. Una Impresión Freudiana. Madrid: Trotta. Eco, Umberto. 1994. La búsqueda de la lengua perfecta en la cultura europea. Barcelona: GrijalboMondadori. Elleström, Lars, ed. 2010. Media borders, multimodality and intermediality. New York: Springer. ———. 2017. Transfer of media characteristics among dissimilar media. Palabra Clave 20 (3): 663–685. https://doi.org/10.5294/pacla.2017.20.3.4. Farré, Judith. 2007. Palabras preliminares. In Teatro y poder en la época de Carlos II: Fiestas en torno a reyes y virreyes, ed. Judith Farré. Barcelona: Universidad de Navarra-Iberoamericana Editorial. Fletcher, Angus. 2002. Alegoría. Madrid: Ediciones Akal. García, Pablo. 2009. Saldos del criollismo: El Teatro de virtudes políticas de Carlos de Sigüenza y Góngora a la luz de la historiografía de Fernando de Alva Ixtlilxóchitl. Colonial Latin American Review 18 (2): 219–235. Gómez, Ignacio. 2019. Athanasius Kircher. Itinerario del éxtasis o las imágenes de un saber universal. Madrid: Siruela. González, Pablo, Inmaculada Rodríguez, Juan Chiva, and Víctor Mínguez. 2019. Un planeta engalanado: La fiesta en los reinos hispánicos. Castelló de la Plana: Publicacions de la Universitat Jaume I. Gracián, Baltasar. 2010. Arte de ingenio. Tratado de la Agudeza. Madrid: Cátedra. Graham, David. 2012. Fuentes, formas y funciones emblemáticas. Historia, morfología y lectura. In Creación, función y recepción de la emblemática, ed. Herón Pérez and Bárbara Skinfill. Zamora de Hidalgo: El Colegio de Michoacán A.C. Gruzinski, Serge. 1994. La guerra de las imágenes. Guatemala City: Fondo de Cultura Económica. ———. 2016. La colonización de lo imaginario: Sociedades indígenas y occidentalización en el México español. Siglos XVI–XVIII. Guatemala City: Fondo de Cultura Económica. Habermas, Jürgen. 1981. Historia y crítica de la opinión pública: La transformación estructural de la vida pública. Barcelona: Gustavo Gili. Horapollus [Horapolo]. 2011. Hieroglyphica. Madrid: Akal. Jenkins, Henry. 2017. Transmedia Logics and Locations. In B. W. L. Derhy Kurtz & M. Bourdaa (Eds.), The Rise of Transtexts: Challenges and Opportunities (pp. 192–220). Routledge. Kittler, Friedrich. 2010. Optical media. Oxford, UK: Polity. Landow, George. 1997. Hypertext 2.0. The convergence of contemporary critical theory and technology (rev ed). Baltimore: Johns Hopkins University Press. Leonard, Irving. 1984. Prólogo. In Seis obras, ed. Carlos Sigüenza. Caracas: Biblioteca Ayacucho. López, Sagrario. 2003. La erudición de Sor Juana Inés de la Cruz en su Neptuno alegórico. La Perinola 7: 241–270. Lorente, Antonio. 1994. Don Carlos de Sigüenza y Góngora, educador de príncipes: El Theatro de virtudes políticas. Literatura Mexicana 5 (2): 335–371.

24

Intermediality in Seventeenth-Century Baroque Celebrations in. . .

601

Martin, Vicent. 2009. Introducción. In Neptuno alegórico, ed. Juana Inés Cruz. Madrid: Cátedra. Mínguez, Víctor. 1995. Los reyes distantes: Imágenes del poder en el México virreinal. Castelló de la Plana: Publicacions de la Universitat Jaume I. ———. 2009. Imágenes jeroglíficas para un imperio en fiesta. Relaciones. Estudios de Historia y Sociedad 30 (119): 81–112. ———. 2017. Jeroglíficos para un Imperio. La cultura emblemática en el virreinato de la Nueva España. Quiroga: Revista de Patrimonio Iberoamericano 11: 56–68. Mínguez, Víctor, and Juan Chiva. 2014. La fiesta barroca. Castelló de la Plana: Universitat Jaume I. Servei de Comunicació i Publicacions. Mínguez, Víctor; González Tornel, Pablo y Rodríguez Moya, Inmaculada, 2010. La Fiesta Barroca. El Reino de Valencia (1599–1802). Triunfos Barrocos. Vol. I. Castellón: Universitat Jaume I, 405 págs, 383 ils. b/n. ISBN: 978-84-8021-783-5. Mínguez, V., Chiva, J., Gonz.lez Tornel, P. y Rodr.guez Moya, I. 2012. La fiesta barroca. Los virreinatos americanos (1560–1808). Triunfos barrocos: Volumen segundo. Universidad de las Palmas de Gran Canaria. Mínguez, Víctor, and Inmaculada Rodríguez. 2015. Las enseñanzas de Andrea Alciato en la corte de Venecia: Una tabla de la Colección Barbosa-Stern atribuida a Massys. Anales del Instituto de Investigaciones Estéticas 37 (106): 53–74. Mínguez, Víctor, Juan Chiva, Pablo González, and Inmaculada Rodríguez. 2012. La fiesta barroca. Los virreinatos americanos (1560–1808). Castelló de la Plana: Publicacions de la Universitat Jaume I. Mínguez, Víctor, Pablo Gonzalez, Juan Chiva, and Inmaculada Rodríguez. 2014. La fiesta barroca. Los reinos de Nápoles y Sicilia (1535–1713). Vol. 3. Castelló de la Plana: Publicacions de la Universitat Jaume I. Mínguez, Víctor, Pablo González, Juan Chiva, and Inmaculada Rodríguez. 2016. La fiesta barroca. La corte del Rey (1555–1808). Castelló de la Plana: Publicacions de la Universitat Jaume I. Mínguez, Víctor, Juan Chiva, Pablo González, and Inmaculada Rodríguez. 2018. La fiesta barroca: Portugal hispánico y el imperio oceánico. Castelló de la Plana: Publicacions de la Universitat Jaume I. ———. 2019. Un planeta engalanado. La fiesta barroca en la monarquía hispánica. Castelló de la Plana: Publicacions de la Universitat Jaume I. More, Anna. 2012. Baroque Sovereignty: Carlos de Siguenza y Gongora and the Creole Archive of Colonial Mexico. Philadelphia: University of Pennsylvania Press. Osorio, Alejandra. 2008. Inventing Lima: Baroque modernity in Peru’s South Sea metropolis. New York: Springer. Panofsky, Erwin. 2016. El significado en las artes visuales. Madrid: Alianza. Paz, Octavio. 1982. Sor Juana Inés de La Cruz o las trampas de la fe. Barcelona: Seix Barral. Pérez, Herón. 2009. Literatura emblemática: Balances, perspectivas y continuidades. Relaciones. Estudios de Historia y Sociedad 30 (119): 11–18. Pérez, Herón, and Bárbara Skinfill, eds. 2002. Esplendor y ocaso de la cultura simbólica. Zamora de Hidalgo: El Colegio de Michoacán A.C. ———, eds. 2012. Creación, función y recepción de la emblemática. Zamora de Hidalgo: El Colegio de Michoacán A.C. Picinelli, Filippo. 2012. El mundo simbólico. Los instrumentos mecánicos, los instrumentos de juego. Zamora de Hidalgo: El Colegio de Michoacán A.C. Praz, Mario. 1989. Imágenes del Barroco: Estudios de emblemática. Madrid: Siruela. Rey, Edgar. 2018. Fiestas, fastos y duelos: Orden y conformación en Santa Marta, siglo XVIII. Miami: La Carreta. Rodríguez, Inmaculada, and Víctor Mínguez. 2016. Visiones de un imperio en fiesta. Madrid: Fundación Carlos de Amberes. Rodriguez, Inmaculada, and Víctor Mínguez. 2020. El proyecto Triunfos Barrocos: El estudio de la fiesta renacentista y barroca en el grupo IHA (Universitat Jaume I). NORBA. Revista de Arte XL 23: 185–202.

602

M. Va´squez Arias and A. Burbano Valde´s

Rojewski, Oskar, and Mirosława Sobczyńska-Szczepańska, eds. 2019. Court, nobles and festivals: Studies on early modern visual culture. Katowice: University of Silesia Press. Saucedo, Carmen. 1999. Decreto del Cabildo Catedral de México para que a Sor Juana Inés de la Cruz se le paguen 200 pesos por el “Neptuno alegórico”. Relaciones. Estudios de Historia y Sociedad 20 (77): 183–191. ———. 2007. Sor Juana y Carlos de Sigüenza: Una amistad entre genios. Buenos Aires: Lumen. Schröter, Jens. 2011. Discourses and models of intermediality. CLCWeb: Comparative Literature and Culture 13 (3). https://doi.org/10.7771/1481-4374.1790. Sigüenza, Carlos. 1984. Seis obras. Caracas: Biblioteca Ayacucho. Skinfill, Bárbara. 2002. Los caminos de la emblemática novohispana: Una aproximación bibliográfica. Zamora de Hidalgo: El Colegio de Michoacán A.C. Skinfill, Bárbara, and Eloy Gómez, eds. 2002. Las dimensiones del arte emblemático. Zamora de Hidalgo: El Colegio de Michoacán A.C. Tesauro, Emanuele. 1741. Cannocchiale aristotelico, esto es, Anteojo de larga vista o Idea de la agudeza e ingeniosa locucion que sirve a toda arte oratoria, lapidaria y symbolica examinada con los principios del divino Aristoteles. Medellín: Antonio Marín. Torres, Daniel. 2019. Finezas barrocas entre Sor Juana Inés de la Cruz y Don Carlos de Sigüenza y Góngora. Delaware Review of Latin American Studies 53: 13–44. Vásquez, Mauricio, Diego Montoya, Andrés Burbano, Siegfried Zielinski, et al. 2020. Imaginación mediática en Hispanoamérica: Variantología de lo transmedial entre los siglos XVI y XIX. Medellín: Editorial EAFIT. Vincent, Martin. 2009. Introducción. En S. J. I. de la Cruz, Neptuno alegórico (pp. 9–60). Cátedra. Zielinski, Siegfried. 2011. Arqueología de los medios. Hacia el tiempo profundo de la visión y la audición técnica. Bogotá: Universidad de Los Andes.

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

25

Dominika Bugno-Narecka

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Two Major Forms of Collecting: Museum and the Cabinet of Curiosities . . . . . . . . . . . . . . . . . . . . Medium-Centered Model of Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Cabinet of Curiosities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Modalities of the Cabinet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Ambiguity of the Cabinet and Curiosity/Wonder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Historical Cabinet as a Qualified Medium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transhistorical Transmedial Cabinets in the Twentieth and Twenty-First Centuries . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

604 606 607 609 612 614 616 619 623 624

Abstract

Cabinets of curiosities are considered to be a projection of one’s thinking about oneself, one’s origin, and one’s place in the world. The variety of gathered objects emphasized the significance, power, and intellectual abilities of the collector, and highlighted the smooth transition from nature to culture as well as the gradual evolution of modern science from the second half of the sixteenth century onward. The attempts to organize the cabinet and systematize the collected items according to their distinctive features correspond with Lars Elleström’s model for understanding intermedial relations. Drawing from this analogy, the chapter presents the intermedial cabinet of curiosities as a qualified media type, distinguishing its modalities and discussing its qualifying aspects. It approaches the historical cabinet of curiosities as a complex media product which combines and unifies the elements of chemical laboratories, zoos, art galleries, and museums. The chapter also provides examples of more contemporary cabinets in order to problematize further their intermedial character. Contemporary D. Bugno-Narecka (*) Institute of Literary Studies, John Paul II Catholic University of Lublin, Lublin, Poland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_26

603

604

D. Bugno-Narecka

cabinets of curiosities – individual artworks, buildings and gardens housing entire collections, books, film settings or themes, online tools, and social media – can be considered as equally elaborate media products as their historical predecessors. The chapter highlights interrelations and interactions between the original cabinet of curiosities and its various contemporary realizations. Despite the change in media type, all cabinets still convey the image of the world and communicate the state of knowledge of a particular era and a specific “collector.” Keywords

Cabinet of curiosities · Premodern collection · Intermediality

Introduction Cabinets of curiosities are an example of qualified media that thrived in Europe between 1540 and 1740 (Bredekamp 1995: 28). A common stereotypical view presents the cabinet of curiosities as a chaotic, irrational, and bizarre predecessor of the contemporary museum (cf. Preston 2006). The cabinet is frequently perceived as “a disordered jumble of unconnected objects, many of which were fraudulent in character” (Hooper-Greenhill 1992: 79). Despite this rather negative general perception, Stephanie Bowry (2015: 66) observes that “early collections have often been treated as bygone and irrelevant means of representing the world, when, in fact, the cabinet has demonstrably continued to haunt the artistic and intellectual imagination on a grand scale throughout the nineteenth, twentieth and early twenty-first centuries.” Hence, the cabinet of curiosities can be regarded as a distinctive transhistorical phenomenon, the numerous instances of which can be found not only in the late Renaissance and the Baroque periods, but also at present. What distinguishes the cabinet from other forms of collecting is its drive to encompass and include the entire state of knowledge of particular era or collector. As a result, the collections were often enormous and entailed nonobvious items that cluttered not only the purpose-built furniture but also the entire rooms. Owners of vast cabinet collections attempted to categorize and systematize these objects by investigating their properties and indicating essential features the objects had in common and by which they differed from one another. Frequently, the objects were similar in some, if not all, aspects, like being a natural or man-made object. Consequently, they were easy to categorize, for instance, as plants, animals, paintings, or coins. However, it also happened that a given object belonged to several categories as its features and properties exceeded the established criteria. To provide an example, a chameleon or a crocodile was an animal species but also an exotic specimen for a North European collector. The arduous attempt at studying and classifying objects in the cabinet undertaken by the collectors in the sixteenth and seventeenth centuries in order to understand the mechanisms of the universe is analogical to Lars Elleström’s work on the irreducible model for understanding intermedial relations (2010, 2014, 2021). By distinguishing four media modalities

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

605

and then modes within those modalities, Elleström provides a similar, more contemporary framework for understanding media which constitute a significant part of our reality. What is more, he equips us with a universal tool which helps us understand the specificity and complexity of the cabinet. Hence, the chapter reads the cabinet of curiosities through Elleström’s model so as to emphasize the intermedial character of the historical cabinet and explain the cabinet’s function as a media product in the process of communication. Starting with distinguishing the two forms of collecting, namely the cabinet collection and the museum collection, and placing them as media products in the medium-centered model of communication (sections “Two Major Forms of Collecting: Museum and the Cabinet of Curiosities” and “Medium-Centered Model of Communication”), the following chapter focuses on the intermedial cabinet back in the sixteenth and seventeenth centuries (“Intermedial Cabinet of Curiosities”) as well as in the twentieth and twenty-first century (“Transhistorical Transmedial Cabinets in the Twentieth and Twenty-First Centuries”). Intermediality is broadly understood here as the study of complex interrelations and interactions among all forms of dissimilar communicative media types that result from crossing media borders (Elleström 2010, 2014, 2021) and will be manifested in the section “Intermedial Cabinet of Curiosities” in the discussion of different forms the historical cabinet may acquire, that is, the collection itself, its inventory, catalogue, and illustration – engraving or drawing. The chapter approaches the original, historical cabinet of curiosities as a complex media product which combined and unified the elements of what is nowadays known as chemical laboratories, zoos, art galleries, and thematic museums. Consequently, to describe the historical cabinet of curiosities, the modes of the four modalities of the cabinet will be distinguished in the fourth section “The Modalities of the Cabinet,” followed by the reflection on the ambiguity of the concepts of the cabinet, curiosity, and wonder in section five. Subsequently, the cabinet will be presented as a qualified media type (section “The Historical Cabinet as Qualified Medium”). The contextual and operational qualifying aspects of the cabinet will be explained here. The last part of the chapter (“Transhistorical Transmedial Cabinets in the Twentieth and Twenty-First Centuries”) provides a short overview of modern cabinets of curiosities in order to determine the scope and problematize further the intermedial character of the notion. Despite the scientific distribution of objects and the specialization of knowledge, contemporary cabinets of curiosities can take the form of equally elaborate media products as their historical counterparts. The contemporary realizations of the cabinet include but are not limited to: artworks by Damien Hirst, Mark Dion, or Joseph Cornell; entire exhibitions, e.g., “From Wunderkammer to Museum, 1599–1850,” organized in 2012 at the Grolier Club in New York; buildings and gardens housing vast collections, e.g., John Soane’s Museum in London or gardens of Little Sparta; books by Guillermo del Toro, Orhan Pamuk, or Patrick Gale; film settings or themes, e.g., in American Horror Story (2014–2015), Nightmare Alley (2021), or Guillermo del Toro’s Cabinet of Curiosities (2022); or online tools and social media, e.g., Google or Facebook. This section thus investigates interrelations and interactions between the original cabinet of curiosities and its

606

D. Bugno-Narecka

various contemporary realizations. Despite changing the media type, all cabinets still convey the image of the world and communicate the state of knowledge of the specific era and, in particular, of an individual “collector,” i.e., the owner, curator, or author of the cabinet. This qualifying aspect integrates the various media products in question but at the same time the technical medium of display transforms the qualified medium and each media product accordingly.

Two Major Forms of Collecting: Museum and the Cabinet of Curiosities Although, in the words of Susan Pearce (1995: vii), “‘[c]ollecting’ is difficult to define,” it can be intuitively described as “the gathering together of chosen objects for purposes regarded as special” (Pearce 1995: vii). Collecting can be further characterized as a typical human activity dating back to the prehistoric times when our ancestors learned to gather food. Hence, at first collecting and gathering activities were associated with the pragmatic purpose of providing sustenance. They gained a new status of a distinctive media product which enables communication in the sixteenth and seventeenth centuries, when the Europeans started to explore the world and brought with them a wide range of exotic objects and stories from distant lands that accompanied those objects (Das 2015). Consequently, impressive enormous collections were “an attempt to manage the empirical explosion of materials that wider dissemination of ancient texts, increased travel, voyages of discovery, and more systematic forms of communication and exchange had produced” (Findeln 1994: 3). Collecting and collections soon became a distinct qualified medium known in English as the cabinet of curiosities, which communicated status and power, as mostly wealthy royals and aristocrats could afford to acquire rare and wondrous objects. The other group of collectors included scholars, physicians, apothecaries, and members of the clergy (MacGregor 2007). While royals and aristocrats obtained “curiosities” not only to gain and communicate the acquired knowledge, but also to impress and entertain their peers, scholars, physicians, and apothecaries did so not only to study and broaden their intellectual horizons (Das 2015), and thus claim their position among the intellectual elite, but also to make medicines (Findeln 1994; MacGregor 2007). As Paula Findeln (1994: 3) puts it, “[t]hrough the possession of objects, one physically acquired knowledge, and through their display, one symbolically acquired the honor and reputation that all men of learning cultivated.” With time, collections began to include objects of sentimental, aesthetic, or economic value, broadening and problematizing the scope of the collection. Still, some philosophers and historians consider collecting to be a kind of narcissistic and frivolous pastime (Pomian 1990), while for others it is a “man’s attempt to control nature” (Bredekamp 1995: 27) and the surrounding world. The specificity and the scope of the collections changed with the Enlightenment and the establishment of the modern museum: from communicating a broad and unified vision of the world in a cabinet of curiosities to a topical diversification and

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

607

thematic specialization exemplified, for instance, by a natural history museum or a gallery of modern art. Hence, modern museums offer insight into only a selected fragment of reality: a distinguished figure, an important event, history of a place, community, or some other phenomenon. Even though the institution of the museum itself has changed over the years, dragging the practical realization of the idea(s) along (cf. Witcomb 2003), the main purpose behind any modern museum or recent collection is “to tell a specific story, and objects are gathered as they relate to the story” (Hooper-Greenhill 1992: 206). The objects are devoid of their original historical context as they are displayed in sterile, neatly organized spaces and exhibition showcases that limit the visitor’s interaction with the objects. Due to the historical value and fragility, the exhibited objects need to be protected from possible damage. As it will be shown in the next section, that is not the case with the cabinet of curiosities.

Medium-Centered Model of Communication Regardless of the differences signaled above, both the cabinet of curiosity and modern museum are parts of communication process. According to a Medium-centred model provided by Lars Elleström (2021: 10), there are “three indispensable and interconnected entities[. . .]: 1. Something being transferred; 2. Two separate places between which the transfer occurs; 3. An intermediate stage that makes the transfer possible.” As far as “something being transferred” is concerned, Elleström proposes “the term ‘cognitive import’ to refer to those mental configurations that are input and output of communication” (2021: 12), like meaning, significance, or ideas. The “two separate places” are “the producer’s mind” and “the perceiver’s mind” understood as “the mental places in which cognitive import appears” (Elleström 2021: 12). The third entity, “the intermediate stage that enables the transfer of cognitive import from a producer’s to a perceiver’s mind” (Elleström 2021: 13), is referred to as “media product.” Both cabinets of curiosities and museums fit Lars Elleström’s (2018, 2021) Medium-centred model of communication, though in slightly different ways. In the case of a museum collection, knowledge of a particular theme is transferred between the institution of museum, and, more specifically, exhibition curator (the producer’s mind) on the one hand, and museum visitor (the receiver’s mind) on the other. The space of museum exhibition with all its contents performs the role of media product. As suggested in the previous section, the function of any museum collection is to communicate the story which precedes the means to tell it. Consequently, first there is a theme or an idea for a particular collection (cognitive import on the producer’s part, after Elleström 2021), and then objects (media products) are gathered to reflect and reconstruct the given theme or idea: event, phenomenon, or life of a prominent figure. In other words, technical media of display, and, consequently, the displayed media products (i.e., individual museum exhibits) are determined by an a priori cognitive import (i.e., the story) as well as by specific qualifying aspects (i.e., social, cultural, and historical context of the

608

D. Bugno-Narecka

exhibition, the general purpose, use, and function of the collection, as well as the aesthetic convention followed; cf. Elleström 2010, 2014, 2021). In contrast, predecessors of modern museum curators, i.e., Renaissance and Baroque collectors, first gathered diverse objects and then examined them in order to come up with a story explaining the item’s meaning, origin, and place within the larger scheme of the universe (Das 2015; Oldenburg 1978). These collectors studied particular media products and their modalities to establish their cognitive import, i.e., “mental configurations that are the input and output of communication” or “a very broad notion of meaning” (Elleström 2018: 279–280). For them, “specific story” (Hooper-Greenhill 1992: 206) was the story of the world as they knew it, i.e., as was accessible to them and communicated by the object. The producer’s mind, and hence the cognitive import on the producer’s part, as well as the act of production represented by Elleström (2018: 282, 2021: 16) in the visual diagram of the communication model on the left side of media product seem to be neglected, omitted, or simply inaccessible to the collector (perceiver) in the process of communication, mainly because cabinets of curiosities were attempts to reconstruct the lost Edenic knowledge. In the words of Horst Bredekamp (1995: 40), they were “a metaphor for the human brain gradually reacquiring Edenic wisdom.” Filling the space of a room or a cabinet with an ever-expanding collection could be compared to the formation of human intellect from the initial void state (Bredekamp 1995; Zytaruk 2011). It was believed that the more complex things were gathered, the closer to the Edenic state of knowledge one could get by studying them and making connections. In other words, the more complex media product, the closer to reconstructing the original cognitive import the perceiver became. It is interesting to observe how, by advancing and going ahead, collectors constructing cabinets of curiosities desired to go back, i.e., to return to Eden, in order to restore the knowledge lost with the fall of the first couple. From this perspective, the collectors’ desire and endeavor can be perceived as a melancholic progress toward the reinstatement of what is irreversibly gone (cf. Lepenies 1992; Radden 2000; Schwenger 2006; Zytaruk 2011). Another possible application of Elleström’s (2018, 2021) Medium-centred model of communication to the cabinet of curiosities and its owner or visitor shows that the producer’s mind, the cognitive import on the producer’s part, and the act of production converge with those of the perceiver’s and the act of perception to the point where the producer and the perceiver are considered to be the same person: the collector or a visitor to the collection. The latter, like the former, was invited to individually study the objects and come to the conclusions concerning the objects on one’s own. As it has been indicated above and will be shown later in the chapter, the former might have also wished to transfer his/her own cognitive import, in particular one related to communicating power and social status, by means of collected items, and hence become the producer in the complex process of communication. As indicated in the previous section, the development of modern museums and the systematization of knowledge narrowed down the “specific story” of any collection, i.e., the cognitive import in the process of communication: instead of communicating a history of the entire world which was contextually unified,

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

609

universal, and harmonious, yet diversified and rich, it became a more individualized and localized story of a person, object, place, or event that has constituted a significant part of the greater world. Consequently, individual media products on display in a museum (which is a separate media product in itself) are more carefully selected and less random than the media products gathered by Renaissance and Baroque collectors. What is more, the producer’s side of the communication model is better marked in the museum and media products are protected from the curiosity of the visitors’ senses other than sight. With the cabinet of curiosities, “deference [was] paid to the visitors/guests, particularly if they were of high rank, rather than to objects on display” (Classen 2020: 275); hence, the objects frequently got broken, damaged, or even stolen (Classen 2012, 2020). However, that did not discourage collectors from assembling, studying, and even trading countless objects.

Intermedial Cabinet of Curiosities Horst Bredekamp (1995) observes that there have been numerous attempts to organize the cabinet collection in space. Various principles were used to categorize the gathered objects and arrange them in some kind of order. But due to the infinite variety of objects that could be collected and the frequent crossing of the borders between media types the gathered objects belonged to as well as between modes of material, sensorial, spatiotemporal, and semiotic modalities, there was no single universal form or formula that could comprise everything (Bredekamp 1995), make clear distinctions between different categories of items, and lead to a relatively precise establishment of qualified media types. That is one of the reasons why elements of Lars Elleström’s model for understanding intermedial relations will be applied in the subsequent sections, but first the historical attempts to order the collected items will be considered. One noteworthy theoretical attempt to systematize the cabinet back in the sixteenth century was Samuel Quiccheberg’s work Inscriptions or Titles of Most Complete Theatre from 1565. It was “the earliest known treatise on how to establish, order and arrange a collection in the early modern period” (Bowry 2015: 73; cf. Bredekamp 1995; Hooper-Greenhill 1992). As a rule, nothing was hidden from the visitor’s sight, and consequently the “focus of the collection was squarely on the collector himself ” (Bowry 2015: 78). Quiccheberg’s vision of the ordering system was similar to the physical and conceptual theatre and amphiteatre, i.e. ,“a place in which all forms of knowledge were to be performed and enacted from the particular vantage point of the collector” (Bowry 2015: 85). Similarly to a theatre the cabinet of curiosities combines and integrates individual media products (collected items), and does so within the designated storage space. Yet, it is the collector or visitor who classifies objects, arranges and rearranges them, and makes connections between them. Among the numerous collecting practices and collection arrangements, i.e., the strategies to combine and integrate media products, Horst Bredekamp (1995: 34–35) examines the Hapsburg practice of distributing and dividing the collected items into

610

D. Bugno-Narecka

four main categories: naturalia (comprising three kingdoms: mineral, vegetable, and animal), artificialia (arts and crafts, e.g., ancient sculptures and works of art), scientifica (machines and devices used to measure time and space), and exotica (“arts and crafts of the peoples of East Asia and the Americas,” Bredekamp 1995: 35). These categories were “organised roughly according to the hierarchy of the materials” (Bredekamp 1995: 31): from natural through “meshing nature and art, to its elevation to a ‘pure’ work of art” (Bredekamp 1995: 31). In many instances, however, artificialia included objects that were both organic and inorganic, natural and man-made (nature reworked by human hand), while items belonging to exotica could be “woven into all three kingdoms of nature as well as the sphere of artificialia” (Bredekamp 1995: 35). Consequently, the clear-cut distinction between the works of nature and the works of human art was impossible to be maintained (cf. Carrier 2006), as many items, like sculptures or automata, were thought to have hybrid characteristics of nature reworked by man. Despite the countless attempts to systematize the arrangement of accumulated media products, the most frequent strategy of combining and integrating these media products in the cabinet of curiosities was based on “deliberate disorder” and aimed at emphasizing the deconstruction of the borders between kingdoms the objects represented and, consequently, between the items themselves (Bredekamp 1995: 73). That led to a constant unresolvable tension between nature and culture which formed a continuum with the imaginary, the supernatural, and the monstrous: The alleged remains of legendary creatures – giants, unicorns, satyrs, basilisks – took their place next to real but puzzling phenomena such as fossils, loadstones, and zoophytes; previously unknown creatures such as the armadillo and the bird of paradise, and a plethora of ordinary artifacts that filled the gaps between one paradox and the next. From the imaginary to the exotic to the ordinary, the museum was designed as a continuum. (Findeln 1994: 3)1.

Due to such features, cabinets of curiosities can be interpreted as a fold combining nature and culture into a continuum of kingdoms and manifesting a smooth transition from one realm to another, from one category of items through transient forms to another. The fold is a model in which difference and identity, or different and the same, distance themselves and move away from each other and then approach or come closer (Deleuze 1993). By analogy, nature and culture in a cabinet of curiosities do not exist separately, but account for diverse unity and unified differences. The presence of natural objects in the cabinet of curiosities indicated the inevitable existence of human being and man-made artworks or machines, and the other way round: man and his works pointed to the existence of nature.

1 In her study, Paula Findeln (1994) uses the term museum while referring to and describing the practice of collecting typical of the cabinet of curiosities. Also, Renaissance and Baroque collectors thought of their collections as museums, as can be observed in the names included in the catalogues and inventories, e.g., Musei Wormiani, Musaeum Calceolari (cf. Bredekamp 1995).

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

611

Florence Fearrington (2012) and Stephanie Bowry (2015) indicate the sad fate of many cabinets of curiosities, which underlines the very personal, private aspect of the collections. Unless secured by the collector’s last will and turned into a museum, collections were disassembled and individual items of the collection were sold after the owner’s death (Bowry 2015). For instance, Mary Cavendish’s “notable collections at Bulstrode Hall, Buckinghamshire, were auctioned off a year after her death in 1786” (Fearrington 2012: 11). Some valuable items were donated to or purchased by public institutions. The collectors who wished to leave a mark and protect their collection from disintegration institutionalized their collections. “Many private collections were institutionalized in the 18th century, among them the Wunderkammer of Kilian Stobaeus, who gave his collection to the University of Lund so that it could be used by his students, one of whom was Carl Linnaeus” (Fearrington 2012: 11), a man that contributed significantly to the development of modern taxonomy and indirectly to the disintegration of the cabinet. Consequently, the historical cabinet of curiosities has not survived in the original material form. There is no preserved physical realization of the original cabinet, but as Stephanie Bowry (2015: 22) has observed, “the curiosity cabinet revealed itself to be diverse and idiosyncratic in its manifestations – even within the same cabinet, which arguably produced endless different versions of itself in various formats – through collections, images, catalogues and inventories.” What has been preserved till our times are “inventories and catalogues, university and academy archives, correspondence between naturalists, travel journals, and the publications resulting from the collection of nature” (Findeln 1994: 8). Many collectors had all items in their collections catalogued and listed in inventories, and then described in collection catalogues, which today provide us with a rough idea as to the scope and enormity of the cabinet and the wealth of each collector. For instance, MacGregor (2007) mentions an engraving, a printed catalogue and a series of illustrated manuscript inventories, all of which provide information about the cabinet of Manfredo Settala in Milan. As a result, the information concerning the historical cabinet of curiosities comes from media products that transformed and represented it, namely the catalogues, inventories, and images of various sorts (engravings, paintings, watercolors, and drawings). In terms of semiotic modality and type of signs involved, illustrations in catalogues and inventories are iconic signs denoting the cabinet – they depict the physical arrangement of the cabinet. Catalogues consist mostly of symbolic signs (and iconic if images are included) and they often contain an idealized vision that diverges from the contents of an inventory: “inventory quantified the collection, the catalogue as a literary form in its own right served to both present and interpret it” (Bowry 2015: 31). Other textual representations of the cabinet – its verbal descriptions – can be found in various visitors’ diaries and correspondence (Bowry 2015), contemporary travel diaries, itineraries, and guides (Bazin 1969), as well as epistolary queries about specific objects (Findeln 1994; Zytaruk 2011). The earliest surviving illustration of a cabinet of curiosities is the “double-page woodcut frontispiece in Italian apothecary Ferrante Imperato’s Dell’historia naturale libri xxviii, published in Naples in 1599” (Fearrington 2012: 9). It depicts

612

D. Bugno-Narecka

Imperato proudly showing off his collection to some unknown visitors. “The room is crammed with objects housed in cases and hung from the walls and ceiling” (Fearrington 2012: 9). Various items cover any horizontal and vertical surface available. A crocodile skin is suspended from the ceiling, which Fearrington (2012) considers a distinctive feature of the cabinet widely copied by collectors. Similar illustrations include: the museum of Frances Calceolari in Verona – a frontispiece made by Ceruti and Chiocco in 1622; Museum Wormianum (1655), and the museum of Ferdinando Cospiano – a frontispiece in Lorenzo Legatti’s Museo Cospi (1677), all discussed in detail, for instance, by Horst Bredekamp (1995). Numerous drawings of the whole collections as well as individual cabinet specimens, usually plants, included in the preserved catalogues, inventories, and elsewhere testify not only to the encyclopedic impulses of the early modern collectors and their exploration of the ever-expanding world, but also to the considerable size of their collections: “Pictorial representations captured at once the plenitude of such collections and their tacit acknowledgment of nature’s mutability” (Zytaruk 2011: 4). At the same time, all the abovementioned media products are instances of media transformation, and in particular media representation (“a medium representing another medium of a different kind,” Elleström 2021: 81) of the cabinet, with the main change in the technical medium of display (manuscript, print on paper, etc.) and material modality limited to the flat surface of the page. On the basis of the cognitive import transferred by media products that transform and represent the historical cabinet, i.e., inventories, catalogues, and illustrations, it is possible to reconstruct the original cabinet of curiosities and distinguish its basic media traits, which will be the concern of the next section.

The Modalities of the Cabinet Lars Elleström (2010, 2014, 2021) distinguishes four categories or types of media traits which he calls modalities: material, sensorial, spatiotemporal, and semiotic. He explains: For something to acquire the function of a media product, it must be material in some way, understood as a physical matter or phenomenon. Such a physical existence must be present in space and/or time for it to exist; it needs to have some sort of spatiotemporal extension. It must also be perceptible to at least one of our senses, which is to say that a media product has to be sensorial. Finally, it must create meaning through signs; it must be semiotic. (Elleström 2021: 46)

As far as material modality of the cabinet is concerned, “[t]he latent corporeal interface of the medium; where the senses meet the material impact” (Elleström 2010: 36) entails a combination of interfaces. There are surfaces which are flat (e.g., paintings, drawings, tapestries, and fabrics) and not flat (e.g., dried plants, exotic insects, shells, corals, and sponges) and which may (like moving art), though mostly do not, change. There can be sound waves produced by automata and other objects

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

613

or instruments. In general, there can be items in any state of matter: solid, liquid, or gas, which are organic or inorganic. In particular, objects collected in the cabinets of curiosities include specimens belonging to the following categories (cf. Bredekamp 1995; Hooper-Greenhill 1992): natural objects (stones and minerals), organic objects (fossils and stuffed animals), remains of ancient civilizations found in earth (archaeological discoveries), objects of nature processed by man (jewels), works of static art (completely man-made and artificial, yet using natural resources and materials), and specimens of artificial life (moving art, machines, and automata). A separate category of objects entails items with supernatural or magical powers (e.g., unicorn horn or shark teeth, both believed to detect poison, cf. Von Holst 1967). All these objects have their own individual interfaces indicated above. Some objects may share the same kind of interface, e.g., solid flat surface, while others might differ in this respect. The complexity of the cabinet’s material modality stems, on the one hand, from the ambiguity of the cabinet itself, namely whether it is a single showcase, a set of rooms filled with objects, or the collection itself. On the other hand, there are numerous various individual objects (media products) which have their own specific qualities (modes and modalities). The sensorial modality of the cabinet of curiosities amounts to engaging all senses. Sight is primarily involved in the interaction with the collection as visitors first look at the entire cabinet and then focus on individual items. The objects are to be studied more closely, and hence they can be touched, taken into one’s hands, smelled, or even tasted, as was the case, for instance, with the Wormwood sage (Classen 2012). “Manual investigation” of the objects gathered was taken for granted by the seventeenth-century visitors to private collections: “touch supplemented sight as a means of discovering the traits of objects on display. Hearing and smell might also be used for this purpose. Such multisensory investigations were supported by contemporary scientific practice” (Classes 2020: 275–276). Lifting objects to investigate their weight, shaking crystals to see water move inside, smelling scented wood, ingesting ground unicorn’s horn or mummified flesh due to the belief in their potent properties (Classes 2012, 2020), or listening to automata such as an artificial duck or a boy that plays the piano (Bredekamp 1995) “enabled visitors to acquire an embodied understanding of the nature of the display” (Classen 2012: 139). Describing the cabinet’s spatiotemporal modality is equally complex, for on the cognitive and perceptual level it involves, on the one hand, the arrangement of the objects within the cabinet or chamber(s), which depends on the size of the collection and the collector’s idea. On the other hand, there is the visitor who studies the objects, makes connections between the items, and rearranges them according to his or her own ability, taste, or knowledge. In this sense, the visitors to the cabinet of curiosities follow the aesthetics of Merleau-Ponty (2012): once they enter the cabinet they change from a passive viewer into an active explorer. This mode of the cabinet’s spatiotemporal modality is connected with both the cognitive and virtual space and the perceptual time (Elleström 2010), which may vary depending on the individual. Each of the objects, and the entire cabinet, however, has its own individual shape and size, and occupies some demarcated three-dimensional space (Elleström’s “space

614

D. Bugno-Narecka

manifested in the material interface,” 2010: 36) that usually cannot be manipulated.2 What is more, some of the objects, especially automata and other moving machinery, can manifest time in their material interface. The modes of virtual time and virtual space of the cabinet are closely connected with the semiotic modality, i.e., making sense and deciphering the meanings of media products through the signs they constitute. Depending on the perception, experience, and prior knowledge of the perceiver, be that a collector or a visitor, individual items in the cabinet of curiosities can be interpreted as symbols (on the grounds of habits or conventions), icons (on the basis of similarity), or indices (on the grounds of contiguity, cf. Peirce discussed by Elleström 2010, 2014, 2021) of the macro- and microcosm. As already indicated, the cabinet of curiosities as a whole can be regarded as a symbolic sign indicating the owner’s power, social status, and intellectual abilities. It can be understood as an iconic sign, because the cabinet is a universe in miniature, and thus an image or a semblance of the world. The complete picture of the world meant that it was only natural and obvious to include items associated with supernatural powers: “As a symbolic mirror of the world, the collection-as-microcosm also had to depict God’s power to intervene in natural processes and produce miracles, so objects which were rare or ‘monstrous’ were particularly valued because they embodied ‘a world subject to Divine caprice’” (Swann 2001: 18). By the same token, the cabinet can be iconic in that it is an image of the state of knowledge relevant to particular historical period: objects gathered “were placed there in accordance with the opinions and teaching of the time” (Murray 1904 qtd. in Bowry 2015: 61). Consequently, the inclusion of alchemy was also natural and obvious due to the belief that there was no disruption between the physical, the supernatural, and God. As far as indexicality is concerned, the cabinet points to the presence of the world outside the cabinet and to the figure of the owner, a person who put all collected items together on display. The above-listed modalities of the cabinet relate to the historical cabinet, but do not fully describe it as a qualified media type. They merely list essential media traits within the four categories of traits. To provide more detailed description of the cabinet, it is necessary to consider the ambiguities related to the terms “cabinet,” “curiosity,” and “wonder,” which will be followed by the “grounds on which media types are qualified” (Elleström 2021: 60), that is, the discussion of operational and the contextual qualifying aspects.

The Ambiguity of the Cabinet and Curiosity/Wonder As already indicated in the first section, cabinets of curiosities “operated according to different principles of organisation, display and interpretation, and were both conceived and received in entirely different historical, social and cultural contexts” I use the word “usually” having in mind the instances of eating the object or damaging it in a mechanical way by the visitor and his/her handling of the object.

2

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

615

than modern museums (Bowry 2015: 8). To say that they encompassed vast and diverse collections which belonged to private owners is to provide a simplified definition of the phenomenon. As explained by Hooper-Greenhill (1992) and repeated by MacGregor (2007) and Bowry (2015), the word “cabinet” might refer to: (1) the physical showcase or any other item of furniture which housed a particular collection; (2) the room or a series of rooms where the collection was stored and put on display; or (3) the collection itself. Regardless of their size, these collections “were often housed within a private residence, but might also appear in semi-public settings such as churches, gardens, libraries, meeting-places and coffee houses, as well as in purpose-built settings” (Bowry 2015: 45). It is also worth pointing out that cabinets of curiosities were frequently connected to gardens, which naturally extended and expanded the range of the collection. Marjorie Swann (2001: 29), for instance, writes about the Tradescant collection that it “encompassed the extensive grounds of the house, which contained a garden and orchard filled with species of plants and trees, both native and exotic.” Both cabinets of curiosities and gardens contained precious natural and artificial objects which emphasized the link and the constant tension between nature and culture (Reilly 2011). The universe captured within the cabinet of curiosities and represented in the arrangement of the garden “was not so much a static tableau to be contemplated as it was the drama of possible relationships to be explored” (Reilly 2011: 32). These tensions between nature and art/culture were in particular to be celebrated and delved into in the garden (cf. Weiss 1995, 1998). Additionally, Stephanie Bowry (2015, 48–49) following MacGregor (2007) enumerates other functional rooms a cabinet of curiosities could have been adjacent to: A cabinet might consist of or be appended to a library, such as that of Antonio Giganti (1535–1598) in Bologna, a workshop, as did the kunstkammer, or art cabinet of Augustus of Saxony (1526–1586) in Dresden, or even an observatory, as did the cabinet of Emperor Rudoplh II (1552–1612) at the Hradschin Palace in Prague.

The ambiguity of the term “cabinet” points to the blurring of the distinction between form and content, and to the potential problems with describing the cabinet in terms of its technical medium of display, pre-semiotic and semiotic modalities. The other word – curiosity – is also ambiguous. It refers not only to the desire to know or to learn about anything, but also denotes a strange, rare, or novel thing. The sixteenth and the seventeenth century elevated curiosity, “a sensibility formerly viewed with ambivalence as a sign of the prideful nature of humans” (Findeln 1994: 16), to the status of a virtue. “Curiosi or cognoscenti were among the names given to those who sought to expand the boundaries of knowledge through the accumulation and study of collections of objects in a cabinet” (Bowry 2015: 51). Cabinets of curiosities are sometimes called “rooms of wonder” (Fearrington 2012), where wonder as a noun means amazement, admiration, and fascination, but at the same time as a verb it means to doubt or to ponder. Like curiosity, wonder also denotes something that is amazing and fascinating. Thus, wonderful means

616

D. Bugno-Narecka

rare, exotic, and remarkable in form (Swann 2001). Consequently, to collect curiosities is to gather strange, rare, novel objects that satisfy the desire to learn and to know. By the same token, wonders are amazing things that make one ponder over the fascinating objects. Hence, the two terms – curiosity and wonder – refer to both an object and a response, be that emotional or intellectual, to that object (Swann 2001; Findeln 1994). Consequently, the name already indicates that there is more to the cabinet of curiosities than the evident materiality this qualified media product can express. The variety of names the qualified medium in question goes by in academic research points to the technical media of display involved: Kunstkammer, Wunderkammer, cabinet of curiosities, or curiosity chamber. “Kammer” and “chamber” indicate the entire rooms filled with art (Kunst) or wonders (Wunder), while “cabinet” primarily refers to an item of furniture that similarly serves as storage space for this qualified media type (cf. Bredekamp 1995; Schlosser 1908; Bruhn and Schirrmacher 2022). All terms denote a general manner of collecting various objects rather than a particular example thereof. The term consistently and consequently used throughout this chapter is the cabinet of curiosities, i.e., a qualified medium comprised of a collection of diverse objects stored in one place, which represent both the natural world and the world modified by man. On the basis of these objects – repositories of knowledge – one is able to decipher and understand the world (the macro- as well as the microcosm) and determine one’s place in it. The next section will explain this idea in more detail, exploring further what qualifies a particular media product as a cabinet of curiosities.

The Historical Cabinet as a Qualified Medium Lars Elleström distinguishes two aspects as the “grounds on which media types are qualified” (2021: 60): The first is the origin and delimitation of media in specific historical, cultural and social circumstances. This can be termed the contextual qualifying aspect and involves forming media types on the grounds of historically and geographically determined practices, discourses and conventions. [. . .]The second of the two qualifying aspects is the general purpose, use and function of media, which may be termed the operational qualifying aspect. (2021: 60–61)

What qualifies particular media products as cabinets of curiosities in terms of the operational qualifying aspect is the attempt to capture the macrocosm in the microcosm. The latter is created by means of assembling diverse physical objects within the confined space of the cabinet, i.e., the apparatus or room which stores and displays the gathered objects. According to “a traditional cosmology of analogies and correspondences, [. . .] the order of God’s creation – the macrocosm – could be represented in miniature by the arrangement of a group of objects” (Swann 2001: 18). The arrangement of the objects communicated a smooth, fold-

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

617

like transition from nature to culture and from culture to nature, i.e., going back to the origins of man (cf. Deleuze 1993). It also denoted the effort to explain and comprehend the relationship between human beings and their natural surrounding (Bredekamp 1995). Vast collections containing various objects gathered within the cabinet were at first a projection of one’s thinking about oneself, one’s origin, and one’s place in the universe. Through the contemplation of macroscopic/ universal/central relationships between items one could advance and examine one’s own microscopic/local/peripheral (hi)story and place within the greater system (cf. Hooper-Greenhill 1992). As people started thinking about themselves and reflecting upon their lives, they started collecting things and objects which both illustrated and helped them understand their role in this world. Although initially people collected objects that reflected the spirit and the culture of their times, hence, items with no historical value and devoid of reference to the past epochs, soon the seemingly organized space began to represent time and collectors began to appreciate the historical value of their acquisitions (HooperGreenhill 1992). What is also worth stressing is that cabinets of curiosities quickly became vast collections gathered and owned by powerful rulers not only to manifest their interest in the world (of wonders) and knowledge (its complete scope available in their day, cf. Burda 2011), but also to communicate their power and wealth: “In possessing the world in microcosm, he [the ruler] may have symbolically represented his mastery of the greater world. [. . .] [H]e may have believed that in having a theatre of the universe in his collection, he could grasp and control the greater world” (Kaufmann, 1995: 179). Hence, the operational qualifying aspect of the cabinet (and part of the cognitive import on the owner’s/producer’s side) also entailed the emphasis on the power and status of the Baroque ruler who intended to stress the importance of his position from any possible angle by means of any available representation, creating an illusion of the world, in particular of his own absolute power (cf. Miller 1990; Wilson 2000). What strengthened the effect of communicating power and wealth is the fact that as the interest of the collectors turned to the items of historical value, the objects available for collection became more and more expensive, reducing the number of collectors who could afford to purchase these items to the richest people only: princes, merchants, and scholars (Hooper-Greenhill 1992). For nonregal collectors, the cabinet of curiosities frequently emphasized their social position, expressed their political ambition (Swann 2001), and confirmed their belonging to the elite: “Indiscriminant in their choice of objects, collectors were highly selective in their choice of companions. As they demonstrated often in their words and actions, only individuals of privilege and learning had earned the right to collect and classify the world” (Findeln 1994: 15). What is more, it was considered a privilege and an act of a powerful collector’s good will to be able to see the aforementioned collections. There was also a social requirement for a well-born young man to travel across Europe and visit cabinets of curiosities as part of his education, experience, and refinement (Swann 2001). Each visit was carefully recorded both by the collector and the visitor. As a result, young men established their social credentials while

618

D. Bugno-Narecka

“a collector’s reputation was enhanced every time the right sort of person sought to visit his collection” (Swann 2001: 27). As far as the contextual qualifying aspect is concerned, cabinets of curiosities were scattered across Europe of the mid-sixteenth to mid-eighteenth century and, as previously indicated, they belonged to wealthy owners. The most notable examples the cabinet were the Kunstkammer of Rudolph II in Prague (cf. Schlosser 1908; Kaufmann 1978), the Kunstkammer of Duke Ablrecht V of Bavaria in Munich, the Kunstkammer of Duke Ferdinand II at Castle Ambras, the Kunstkammer of the Elector Augustus in Dresden, or studiolo of Francesco I in Florence (cf. Oldenburg 1978; Hooper-Greenhill 1992; Bredekamp 1995). Other “[p]rincely collectors include numerous Medicis and Hapsburgs and Oldenburgs and the like” (Fearrington 2012: 10): Frederick III, Duke of Schleswig-Holstein, and Christian IV and Christian V, Kings of Denmark, to name a few (cf. Swann 2001). A late example of an early eighteenth-century cabinet is Peter the Great’s Kunstkamera in St Petersburg examined, among others, by Olga Baird (2008) and David Mazierski (2012). Surprisingly, French kings “showed less interest in such endeavors” (Fearrington 2012: 10), though there were numerous cabinets owned by “French medical and scholarly men” (Fearrington 2012: 10). Similarly, the Tudor and Stuart rulers of England seemed not to be as interested in cabinet collecting as English scholars of that time (Swann 2001). Notable English collectors include, for instance, John Bargrave, John Tradescant the elder and John Tradescant the younger, Robert Hubert, Hans Sloane, or Elias Ashmole who acquired the enormous collections of his neighbors, the Tradescants, in 1662 (Swann 2001).3 Ashmole later donated the acquired collection to Oxford University, where the Ashmolean Museum became the first truly public museum in Europe (Das 2015), allowing women and members of lower social classes to see the exhibitions upon paying a very low admission fee. Paula Findeln (1994) names a wide range of Renaissance and Baroque naturalists and natural philosophers of Italian origin. Many of them were the owners of enormous collections that contained not only natural specimens but also scientific instruments, such as telescopes or microscopes, and other objects that would always be classified as belonging to one of three categories: inanimate objects, plants, or animals, because of what they were made of (Findeln 1994). Horst Bredekamp (1995) provides an overview of different cabinet owners throughout Europe between 1540 and 1740, while Marjorie Swann (2001) focuses on collectors and collections in England.

3

Although most owners of the cabinet collections were men, there were several female exceptions, the most notable of which was Isabella d’Este Gonzaga. Obsessed with antiquities, she collected mainly coins, statues, books, and paintings (Brown 1996: 55). Florence Fearrington (2012: 11) lists Mary Cavendish Bentinck and Elizabeth Bligh as two examples of female collectors and owners of impressive cabinets of curiosities. According to Fearrington (2012: 11), Horace Walpole claimed that “few men have rivalled Margaret Cavendish in the mania of collecting, and perhaps no woman. In an age of great collectors she rivalled the greatest.”

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

619

Transhistorical Transmedial Cabinets in the Twentieth and Twenty-First Centuries Unfortunately, with the development of science and with the change of scientific method from interpretation based on similitude and resemblance, which allowed for “more imaginative reading of the Book of Nature” (Zytaruk 2011: 2), to ordering based on identity and difference (Hooper-Greenhill 1992: 135), a reduction to monodimensional taxonomies took place (Hooper-Greenhill 1992: 139). In addition, due to experimental sciences gaining importance in the production of life sciences, cabinets of curiosities and the manner of collecting they promoted lost their primary role in the organization of knowledge (Strasser 2012: 311). What is more, such philosophers as Immanuel Kant and Georg Wilhelm Friedrich Hegel meditated profoundly upon the differences between natural wonders and works of man (art). As a result of these changes and influences, cabinets of curiosities were divided into natural history museums which housed natural wonders and public art museums that displayed works of art/man. With time, objects gathered in cabinets of curiosities were further distributed between chemical laboratories, zoos, art galleries, and thematic museums. The idea of displaying monstrosities and abnormalities of the world found later realization in American dime museums and freak shows (McNamara 1974; Murray 2009). Thus, striving for unity led to dispersion. It is claimed by museum skeptics that modern museums have stripped the artifacts of their broad historical context (Carrier 2006; Lawlor 2003). The arrangement in clean, often “empty” spaces, which are devoid of historical and/or cultural context, makes the museums boring to the modern visitor. To remain significant places of cultural and social life, museums have to engage with a wide range of additional activities, including shopping and catering facilities, events in conference halls, and thematic series of lectures as the main projects (Hooper-Greenhill 1992). They also resort to the use of modern technologies in conveying the story they intend to tell (HooperGreenhill 1992). With the advance of the digital era, however, one can observe the return of the curiosity cabinet in an updated and modernized form. Not only has the distance between the image/object and the beholder been considerably reduced, but also the contents of a curiosity chamber have become virtual (digitalized abstract data) and much more individualized/customized, i.e., “tailored precisely to the individual user, based on social proximity and interactions ranked as more relevant or less so” (Burda 2011: 181). Today’s digital cabinets of curiosities, like Google, Facebook, or Twitter, not only “create endless number of possibilities for the global storage, networking and representation of knowledge” (Burda 2011: 182), but also collect and store data concerning different aspects and fields of individual people’s lives in a space which is unlimited and gives endless possibilities of reorganizing and accessing the stored items. Consequently, the technical medium of display has become perhaps less tangible than the original cabinet of curiosities, but definitely limited to an electronic device that displays text and moving or still images, and produces sound. The material modality of the cabinet has been reduced to flat surfaces of various screens and sound waves, and the sensorial modality excludes taste or smell, but still relies

620

D. Bugno-Narecka

on sight, touch, and hearing. Digital cabinet of curiosities emphasizes the virtual aspect of time and space, while semiotic modality still amounts to distinguishing and interpreting signs as icons, symbols, and indices. More often than in the original cabinet, however, the collected items are media representations of physical objects, e.g., images of food one enjoys or works of art one admires. The physical space of the cabinet has been transformed into digital/virtual space. This does not mean, however, that there are no instances of the analogue/material cabinets of curiosities. It is worth mentioning two examples of curiosity cabinet that appeared as attempts to go against the predominant trend of compartmentalizing knowledge and valuable objects: John Soane’s Museum in London and the Garden of Ian Hamilton Finlay in Stonypath. The first instance, John Soane’s Museum in London, uses the very structure of the cabinet of curiosities. It can be read in the museum’s short guide (2013: 1) that every aspect of the museum’s interior “reflects its creator’s architectural ingenuity and extraordinary imagination.” In a relatively small space at 12 and 13 Lincoln’s Inn Fields in London, Soane collected numerous objects that would help his academy students and anyone wanting to learn about and get inspired by the wonders of the world. In establishing the museum there had also been involved the element of displaying Soane’s wealth, power, individuality, and intellectual accomplishment: “If no dynasty of artists should succeed him, at least the Museum, its intended nursery and dowry, should stand as his memorial, and serve the profession which he had loved with such ferocious passion” (Summerson 1983: 22). According to the information available on the museum website,4 John Soane’s Museum contains Egyptian and classical antiquities, medieval and renaissance antiquities, seventeenth- and eighteenth-century and neoclassical sculptures, casts, oriental objects, timepieces, furniture, stained glass, paintings and picture frames, and building fabric, to name the most general categories of the collected items. As described by Donald Preziosi (2003: 89), a visitor to Sir John Soan’s Museum encounters “a series of progressions mapped out throughout the museum’s spaces – from death to life to enlightenment; from lower to higher; from dark to light; from multiple colors to their resolution as brilliant white light.” Not only do the artifacts correspond to one another, but also the architecture interacts with the collected items, creating a peculiar space that entirely absorbs the visitor. The museum approximates the original historical cabinet in its operational qualifying aspect: identical logic of assembling objects, similar manner of displaying the collected items, and analogous purpose of the collection. A different, more contemporary example of a garden constructed and arranged in a cabinet-of-curiosities-like manner involves Little Sparta, the Garden of Ian Hamilton Finlay built in the second half of the twentieth century. Located in the secluded area of Stonypath in the Pentland Hills near Edinburgh, “[i]t is a garden of ideas and poetry, with works in stone, wood and metal almost invariably incorporating words, and set in surroundings which enhance or are actually part of the whole work, so that

4

https://www.soane.org/

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

621

trees and bushes, running water and still ponds, grass and flowers become more significant by their appointment to a place within artistic scheme” (Sheeler 2003: 11). The garden fuses poetic, philosophical, and artistic elements (sculpture, engravings, but also reconstructed paintings) with natural landscape (hills, moors, streams, and ponds) and plants which grow freely and with as little human intervention as possible. “The themes dealt with in the garden are those which underlie the structures of society. The French Revolution, pre-Socratic views of the nature of the world. The Second World War, the sea and its fishing fleets are among the sources of metaphor and image.”5 There are over 275 works of art, man-made objects of culture that constitute an integral part of the garden (Hunt 2008). One can spend endless hours wandering along narrow paths and passages leading to most intimate corners and nooks, deciphering, or making one’s own meanings and connections between the purposefully arranged objects (artifacts and plants), and contemplating the manifold structure of the place. “The garden as a whole discloses to the viewer who walks round it many complexities of meaning, sentiment and wit.”6 The twentieth- and twenty-first-century artists and curators often go back to the notion of curiosity cabinet and consider the possibilities it offers. Frequently, their work takes the form of single exhibits or entire exhibitions. There can be distinguished works of individual artists, the instances of which include but are not limited to: medicine cabinets and formaldehyde tanks by Damien Hirst (Burn and Hirst 2002; Hirst 2017), Mark Dion’s installations exploring the ideas and potentialities of the seventeenth-century cabinet (Sheehy 2006), or Joseph Cornell’s “boxes containing enigmatic assemblies of natural and man-made objects” (Brink 2007; cf. Hoving 2009). There are also historical exhibitions attempting to bring forward the historical cabinet of curiosities. To provide an instance of the latter, the development and cultural importance of curiosity chambers in shaping both knowledge and the institution of museum were well documented in 2012–2013 exhibition “Rooms of Wonder: From Wunderkammer to Museum, 1599–1899” organized and curated by Florence Fearrington at the Grolier Club in New York. Apart from catalogues describing contents of curiosity chambers, the exhibition included objects typically stored in the cabinets of curiosities, namely preserved animals, seashells, models of insects, and many other collectibles (Fearrington 2012). All these examples transform the historical cabinet and adapt it to the present-day sensibility by filling the cabinets with modern objects in the case of contemporary art installations and with relevant remnants of the past in the case of historical exhibitions. In the smooth transition from the cabinet which is a collection of physical/ material/analogue objects to a more virtual collection contained within a book, Guillermo del Toro’s Cabinet of Curiosities: My Notebooks, Collections, and Other Obsessions (2013) can be placed somewhere between the two. The autobiographical work transforms the cabinet of curiosities as it represents and transmediates the director’s films, notebooks, and an extensive collection of objects 5 6

https://www.littlesparta.org.uk/ https://www.littlesparta.org.uk/

622

D. Bugno-Narecka

gathered in Bleak House, all of which are discussed and presented by del Toro on the pages of the book. Interestingly, the fixed sequentiality of the text is disrupted by placing together (side by side on the page) fragments of interview transcript (main text), images showing relevant pages from del Toro’s notebooks with an English translation on the side,7 still images from the discussed films, and photographs of relevant storyboards. The effect created by such accumulation of various media representations and by the repetition of a theme is that of the multiplicity of objects typical of the three- or four-dimensional cabinet. Del Toro’s obsession with the notion of curiosity cabinet and the horror genre is also manifested in his recent movie production Nightmare Alley (2021) and Netflix series Guillermo del Toro’s Cabinet of Curiosities (2022). While the former uses the motives of the cabinet’s American derivatives, namely the freak show and dime museums which are presented with great attention to detail typical of del Toro’s aesthetics, each episode of the latter begins with del Toro himself introducing an object which is stored in the cabinet (the cabinet is embodied in the series by an elaborate prop resembling a Baroque item of furniture used to house collected items) and the director of the particular episode represented by a small figurine. The convention used here is that of a collector taking the objects out of the cabinet, displaying them on a table in front of the viewers and telling them the story of the presented item in the form of a short film that follows. The book, the film, and the series deserve as yet a separate study and will be discussed by me elsewhere for they constitute a prolific material for intermedia scholars. Finally, the two examples of a literary cabinet of curiosities entail Orhan Pamuk’s The Museum of Innocence (2009) and Patrick Gale’s Notes from an Exhibition (2008). The key difference between the two is that while the latter is a self-contained whole, i.e., the cabinet is limited only to the particulars of this one novel, Pamuk has built his cabinet around the novel, an actual museum located in Istanbul, the museum’s catalogue (Pamuk 2012), and a documentary film, the screenplay of which has been published as yet another book (Pamuk 2016). Consequently, Pamuk’s cabinet is additionally transmediated as the story is communicated by different forms of medium: novel, museum, film, screenplay, and catalogue. The main similarity between the two novels is that they emphasize a more spatial, rather than conventionally temporal, approach to the story told and focus on presenting a fictional biography of its main protagonists. The stories told in Gale’s and Pamuk’s novels are derived from objects of questionable quality to a museum curator but significant to cabinet collector, namely from common, ordinary things frequently labeled as “rubbish” but related to the main protagonists. These objects guide both narratives and are “displayed” by means of language which constructs distinctive exhibition labels in Gale’s novel and acquires the form of frequent metafictional references to the exhibited items made by Pamuk’s narrator. Both novels create a

7

The notebooks in question are originally kept by del Toro in Spanish. They contain images (sketches, drawings, photographs), text (loose thoughts, ideas, anecdotes, dialogues, quotes) and some inserted material (tickets, receipts, newspaper clippings), organised in a totally random way.

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

623

microcosm which is a reflection of the macrocosm and emphasize the protagonists’ place in the grand scheme of things by telling their story. Objects are the starting point for stories and while Pamuk tells the story to the reader in chronological order, Gale’s reader needs to reconstruct the chronology on the basis of the objects on display and fragments of the story that follow. As it has been claimed elsewhere (Bugno-Narecka 2022: 13), The fragmentary, multiperspectival and shuffled structure of [Gale’s] narrative also reflects the structure of a cabinet of curiosities by presenting the reader with the deliberate disorder of the scattered pieces of information. Even the characteristics of the book as a material object, the linear progress of a typical narrative and that of the process of reading cannot bring all the introduced chaos under control before the process of reading is finished.

Pamuk’s and Gale’s literary cabinets of curiosities allow to explore imaginary virtual collections by means of reading. They also open the possibility of expanding the traditional understanding of the spatiotemporal and semiotic modality of literature, which is yet to be examined in a separate study.

Conclusion The cabinet of curiosities has been presented here as “the material expression of human knowledge in all its variety, a collection of ways of seeing as much as objects to see” (Bowry 2015: 86). The cabinet thus might be considered as another way of perceiving the world and communicating that perception. As such, it becomes a qualified media product by means of which some cognitive import is transferred. Simultaneously, the cabinet invites the perceiver to establish one’s own cognitive import by making connections between the individual objects gathered. It has been shown in the chapter that the original cabinet of curiosities entails “a set of natural or artificial objects, kept temporarily or permanently out of economic circuit, afforded special protection in enclosed places adapted specifically for that purpose and put on display” (Pomian 1990: 9). Sharing versatile numerous modes of media modalities, the cabinet attempts to combine and integrate media products belonging to different categories and those escaping established categorizations. At the same time, the cabinet of curiosities can be transformed and represented, among others, in catalogues, inventories, and illustrations. Despite historical decline during the Enlightenment, the cabinet is still “prolific as well as diverse in its conception, methods and scope, and subject to individual tastes and predilections as well as changing perceptions of the world and how it could be best interpreted and understood” (Bowry 2015: 18). Contemporary cabinets, like their historical predecessors, can have a considerable size, to mention John Soane Museum or Bleak House. They can be elaborate showcases or boxes, like those of Mark Dijon of Joseph Cornell. Finally, it is possible to indicate instances of media representation and transformation of the contemporary cabinet of curiosities: catalogues and inventories of old collections have now become books that are

624

D. Bugno-Narecka

autobiographies or fictional biographies and point to the figure of the collector/ author/character in question. Illustrations, drawings, engravings, and paintings depicting collections have evolved into moving images contained in films and TV series as a natural consequence of technological development. What all these cabinets share is fascination with ambiguous curiosity, i.e., desire to learn or to know on the one hand, and the rare, odd, marvelous, or novel thing on the other. They provide us with contemporary wonders, i.e., amazing things that make one ponder over the fascinating objects.

References Baird, Olga A. 2008. I want the people to observe and to learn! The St. Petersburg Kunstkamera in the eighteenth century. History of Education Journal of the History of Education Society 37 (4): 531–547. Bazin, Germain. 1969. The museum age. Trans. Jane van Nuis Cahill. New York: Universe. Bowry, Stephanie Jane. 2015. Re-thinking the curiosity cabinet: A study of visual representation in early and post modernity. Thesis submitted for the degree of Doctor of Philosophy at the University of Leicester. Bredekamp, Horst. 1995. The lure of antiquity and the cult of the machine: The Kunnstkammer and the evolution of nature, art and technology. Trans. Allison Brown. Princeton: Markus Wiener Publishers. Brink, Andrew. 2007. Desire and avoidance in art: Pablo Picasso, Hans Bellmer, Balthus, and Joseph Cornell. New York: Peter Lang Publishing. Brown, Clifford M. 1996. A Ferrarese lady and a Mantuan Marchesa: The art and antiquities collection of Isabella d’Este Gonzaga (1474–1539). In Women and art in the early modern Europe: Patrons, collectors and connoisseurs, ed. Cynthia Lawrence, 53–71. University Park: Pennsylvania State University Press. Bruhn, Jørgen, and Beate Schirrmacher. 2022. Intermedial studies. In Intermedial studies: An introduction to meaning across media, ed. Jørgen Bruhn and Beate Schirrmacher, 3–27. London & New York: Routledge. Bugno-Narecka, Dominika. 2022. Beyond language: On intermediality (and multimodality) in literature. Lublin Studies in Modern Languages and Literature 46 (3): 5–14. Burda, Hubert. 2011. The digital Wunderkammer: 10 chapters on the iconic turn. Munich: Petrarca Verlag. Burn, Gordon, and Damien Hirst. 2002. On the way to work. London: Universe. Carrier, David. 2006. Museum skepticism: A history of the display of art in public galleries. Durham & London: Duke University Press. Classen, Constance. 2012. The deepest sense: A cultural history of touch. Urbana/Chicago/Springfield: University of Illinois Press. ———. 2020. Touch in the museum. In The book of touch, ed. Constance Classen, 275–286. London/New York: Routledge. Das, Nandini. 2015. “Wondrous obsessions: The cabinet of curiosities.” Secret knowledge, dir. Katy Homan. BBC four documentary (https://www.bbc.co.uk/programmes/b053pzmb). Del Toro, Guillermo. 2013. Cabinet of curiosities: My notebooks, collections and other obsessions. Titan Books. Deleuze, Gilles. 1993. The fold: Leibniz and the baroque. Trans. Tom Conley. London: The Athlone Press. Elleström, Lars. 2010. The modalities of media: A model for understanding Intermedial relations. In Media Borders, multimodality and Intermediality, ed. Lars Elleström, 11–48. Houndmills, Basingstoke: Palgrave Macmillan.

25

Cabinets of Curiosities as a Transhistorical and Intermedial Phenomenon

625

———. 2014. Media transformation: The transfer of media characteristics among media. Houndmills/Basingstoke: Palgrave Macmillan. ———. 2018. A medium-centred model of communication. Semiotica 224: 269–293. ———. 2021. The modalities of media II: An expanded model for understanding intermedial relations. In Beyond media borders. Vol. 1. Intermedial relations among multimodal media, ed. Lars Elleström, 3–91. Houndmills, Basingstoke: Palgrave Macmillan. Fearrington, Florence. 2012. Rooms of wonder: From Wunderkammer to museum, 1599–1899: An exhibition at the Grolier Club, 5 December 2012–2 February 2013. New York: Grolier Club. Findeln, Paula. 1994. Possessing nature: Museums, collecting, and scientific culture in early modern Italy. Berkeley/Los Angeles/London: University of California Press. Gale, Patrick. 2008. Notes from an exhibition. London/New York/Toronto/Sydney/New Delhi: Harper Perennial. Hirst, Damien. 2017. Damien Hirst: The complete medicine cabinets. London/New York: Other Criteria Books. Hooper-Greenhill, Eilean. 1992. Museums and the shaping of knowledge. London/New York: Routledge. Hoving, Kirsten. 2009. Joseph Cornell and astronomy: A case for the stars. Princeton/London: Princeton University Press. Hunt, John Dixon. 2008. Nature over again: The garden art of Ian Hamilton Finlay. London: Reaktion Books. Kaufmann, Thomas DaCosta. 1978. Remarks on the collections of Rudolf II: The Kunstkammer as a form of representatio. Art Journal 38 (1): 22–28. ———. 1995. Court, cloister, and city: The art and culture of Central Europe, 1450–1800. Chicago: University of Chicago Press. Lawlor, Leonard. 2003. The archive and the museum. In Thinking through French philosophy: The being of the question, 36–39. Bloomington/Indianapolis: Indiana University Press. Lepenies, Wolf. 1992. Melancholy and society. Trans. Jeremy Gaines and Doris Jones. Cambridge/London: Harvard University Press. MacGregor, Arthur. 2007. Curiosity and enlightenment: Collectors and collections from the sixteenth to the nineteenth century. New Haven/London: Yale University Press. Mazierski, David M. 2012. The cabinet of Frederick Ruysch and the Kunstkamera of Peter the great: Past and present. JBC 38: E31–E40. McNamara, Brooks. 1974. ‘A congress of wonders’: The rise and fall of the dime museum. Emerson Society Quarterly 20 (3): 216–232. Merleau-Ponty, Maurice. 2012. Phenomenology of perception. Trans. Donald A. Landes. New York: Routledge. Miller, John. 1990. Absolutism in seventeenth-century Europe. London: Red Globe Press London. Murray, David. 1904. Museums: Their history and their use, with a bibliography and list of museums in the United Kingdom, 3 vols. Glasgow: James MacLehose and Sons. Murray, Lisa Rochelle. 2009. P.T. Barnum presents: The greatest classroom on earth! Historical inquiry into the role of education in Barnum’s American museum. Thesis submitted for the degree of Master of Arts at the University of Texas, Austin. Oldenburg, Peter. 1978. The splendor of Dresden: Five centuries of art collecting. New Haven: Eastern Press. Pamuk, Orhan. 2009. The museum of innocence. Trans. Maureen Freely. New York/Toronto: Alfred A. Knopf. ———. 2012. The innocence of objects: The museum of innocence, Istanbul. Trans. Ekin Oklap. New York: Abrams Books. ———. 2016. The innocence of memories. Trans. Ekin Oklap. London: Faber & Faber. Pearce, Susan. 1995. On collecting: An investigation into collecting in the European tradition. Oxon/New York: Routledge. Pomian, Krzysztof. 1990. Collectors and curiosities: Paris and Venice, 1500–1800. Trans. Elizabeth Wiles-Portier. Cambridge: Polity Press.

626

D. Bugno-Narecka

Preston, Claire. 2006. The jocund cabinet and the melancholy museum in seventeenth-century English literature. In Curiosity and wonder from the renaissance to the enlightenment, ed. Robert Evans and Alexander Marr, 87–106. Aldershot: Ashgate. Preziosi, Donald. 2003. Brain of the earth’s body: Art, museums, and the phantasms of modernity. Minneapolis: University of Minnesota Press. Radden, Jennifer, ed. 2000. The nature of Melancholy from Aristotle to Kristeva. New York: Oxford University Press. Reilly, Kara. 2011. Automata and mimesis on the stage of theatre history. Hampshire: Palgrave Macmillan. Schwenger, Peter. 2006. The tears of things: Melancholy and physical objects. Minneapolis: The University of Minnesota Press. Sheehy, Colleen J. 2006. Cabinet of curiosities: Mark Dion and the university as installation. Minneapolis/London: University of Minnesota Press. Sheeler, Jessie. 2003. Little Sparta: The garden of Ian Hamilton Finlay. London: Frances Lincoln. Sir John Soan’s Museum: A short guide. 2013. London: Sir John Soan’s Museum. Strasser, Bruno J. 2012. Collecting nature: Practices, styles, and narratives. Osiris 27 (1): 303–340. Summerson, John. 1983. Soane: The man and style. Architectural monographs: John Soane, 9–24. London: Academy Editions. Swann, Marjorie. 2001. Curiosities and texts: The culture of collecting in early modern England. Philadelphia: University of Pennsylvania Press. Von Holst, Niels. 1967. Creators, collectors and connoiseurs: The anatomy of artistic taste from antiquity to the present day. London: Thames & Hudson. Von Schlosser, Julius. 1908. Die Kunst- und Wunderkammern der Spätrenaissance: Ein Beitrag zur Geschichte des Sammelwesens. Leipzig: Klinkhardt und Biermann. Weiss, Allen S. 1995. Mirrors of infinity: The French formal garden and 17th-century metaphysics. New York: Princeton Architectural Press. ———. 1998. Unnatural horizons: Paradox and contradiction in landscape architecture. New York: Princeton Architectural Press. Wilson, Peter H. 2000. Absolutism in Central Europe. London/New York: Routledge. Witcomb, Andrea. 2003. Re-imagining the museum: Beyond the mausoleum. London/New York: Routledge. Zytaruk, Maria. 2011. Cabinets of curiosities and the organization of knowledge. University of Toronto Quarterly 80 (1): 1–23.

Crossing Media Borders: From Intermedial Shakespeares to Shakespearean Intermediality

26

Víctor Huertas-Martín

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Shakespeares and Shakespearean Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shakespeare and Intermediality: Scholarly Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Early Research on Intermedial Shakespeares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Social and Political Impact of Intermedial Shakespeares in Digital and Social Media . . . Other Directions in Shakespearean Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

628 629 632 636 642 646 653 654

Abstract

This chapter discusses the relations between Shakespeare and intermediality. Early twenty-first-century Shakespearean criticism has paid attention to an intermedial canon of Shakespearean productions. These productions are characterized by their deployment of digital technology. Yet, scholarly approaches to intermediality in Shakespeare are gradually expanding the range of analysis: Instead of solely approaching intermediality as a feature of digital performances of Shakespeare, intermediality is seen as a critical lens. This lens invites us to think of the multiple border crossings between material, spatiotemporal, sensory, and semiotic traits in Shakespeare’s plays. The chapter distinguishes the concepts of intermedial Shakespeares and Shakespearean intermediality. The former refers to intermediality in artistic practice, and the latter, to the intermedial features of Shakespeare’s plays. After explaining the context in which intermedial Shakespeares are studied, I critically analyze scholarship on Shakespearean intermediality in performance as well as in the plays. The results reveal that uses of intermediality in Shakespearean performance on the stage, on screen, and in social media emphasize and stress the relevance of medial borders. Likewise, V. Huertas-Martín (*) Universitat de València, Valencia, Spain e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_28

627

628

V. Huertas-Martín

these results show that intermediality and transmediality are lenses that may productively help rethink Shakespeare in specific contexts in a systematic and sustained manner. Importantly, it is shown that intermediality in Shakespeare depends on creatives’ interpretive decisions and on receivers’ perceptions. Thus, rethinking Shakespeare in intermedial terms can potentially be achieved by considering verbal language as just one more of the many facets of Shakespeare’s semiotic richness. Keywords

Border crossing · Intermedial Shakespeares · Shakespearean intermediality · Theatre · Screen · Social media · Text

Introduction This chapter will draw scholarly lines between Shakespeare and intermediality. Shakespeare’s plays were conceived, received, and then disseminated as multimedial, transmedial, and intermedial works. Their survival, mutations, and transmission are based on contextual factors of articulation, production, and reception. These have facilitated Shakespeare’s privileged role in worldwide literature. However, the plays’ facility to adjust to the features of media and build bridges between media borders has played an important part in Shakespeare’s hegemony. In the first section, we define the working concepts of intermedial Shakespeares and Shakespearean intermediality. I intend to use these concepts as categories referring to complementary realities. Such categories are helpful in determining Shakespeare’s relations to intermediality as a practice and understanding intermediality as a feature inherent to Shakespeare’s plays. In the second section, I will focus on the scholarly context that has given way to an understanding of intermediality as a practice in Shakespearean adaptation, whether in theatre performance, on screen, or in the digital media. The third section tackles early research on intermedial Shakespeares as a practice. Distinguishing different research paths, including those which focus on non-anglophone to anglophone performances, this chapter looks into the synchronic and diachronic natures of intermediality in Shakespeare. We will look into the potential that intermediality and transmediality have to encourage thinking of Shakespearean performance in terms of the technical media of display, independent of the authority of source texts. In the fourth section, we will examine intermedial Shakespeares and Shakespearean intermediality in social media: The participatory character of social and digital media, as will be shown, lays the ground for thinking of intermediality as a succession of media border crossings. Evaluations of these crossings are not separated from the ethical issues concerning them; yet, such crossings of media borders expand our perceptions of Shakespearean intermediality. These crossings expand and nuance the predominance of theatre as the core metaphor upon which Shakespeare-related meanings in the digital media are grounded. Thus, the theatre – a medium that “combines and integrates a range of basic and

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

629

qualified media” (Elleström 2010: 29) – serves as a runway to pursue Shakespearean intermediality in a variety of artistic languages, including that of the printed book. Therefore, the fifth section is devoted to exploring approaches to intermediality as a quintessential rather than anecdotal feature in Shakespearean plays. Though many of the sources examined are not explicitly formulated as intermedial approaches to Shakespeare, the analysis indicates that the intermediality of the plays is, indeed, palpable in theatrical performance. Nonetheless, as is confirmed by Freda Chapple and Chiel Kattenbelt (2007: 12), intermediality is in no way restricted to stage practice or digital performance. Such logic indeed applies to Shakespeare. Additionally, Shakespearean intermediality is not only a matter of production decisions but also a matter of audience or reader reception.

Intermedial Shakespeares and Shakespearean Intermediality Autumn, 2018. A YouTube trailer announces Shakespeare’s Troilus and Cressida for the Winter Season (October 12–November 17) at the Royal Shakespeare Theatre (Stratford-upon-Avon). An aerial shot: A simulation of a desert in the middle of a postapocalyptic world. A gang of bikers (the Achaeans) riding across this desert en route to Troy. Unlike the luscious city of Homer’s Illiad, the Troy of this production, directed by Gregory Doran, is inspired in Bartertown, the market city in George Miller’s Mad Max: Fury Road (2015). A voiceover delivers the first lines of the play’s prologue, exhibiting the Royal Shakespeare Company’s craft in verse-speaking.1 The reference to the “ships” in the prologue and the aerial shot are also reminiscent of another iconic aerial shot: that of the Greek fleet sailing to Illium in Troy (dir. Wolfgang Petersen, 2004). The eye of the camera flies over the bikers, crosses the city gates, and takes us to the palisade’s parapets. There wait Troilus and Cressida, two avatars of the real human bodies – the bodies of Gavin Fowler and Amber James – which will be later seen on stage, both defiantly standing up to the outsiders. Trailers, according to Susan Bennett, provide an “outer frame” for the “inner frame” of a theatre performance in space (2003: 139). Taking his vantage point from Bennett’s concept of performance frame, Geoffrey Way argues that it is “at the intersection of these two frames–the meeting between these cultural elements and the live dramatic production–that a particular theatrical experience occurs for the audience” (2016). The trailer could also be considered, using Gerard Genette’s words ([1987] 1997) as a “threshold” to the diegetic world of the fictional work. Likewise, it could be seen as one crossing of medial borders among many in Doran’s production. Such media border crossings include, among others, the translation of Cassandra’s lines to British sign language so that actress Charlotte Arrowsmith “In Troy there lies the scene. From isles of Greece, / The princes orgulous, their high blood chafed, / Have to the port of Athens sent their ships, / Fraught with the ministers and instruments, / Of cruel war (. . .) and their vow is made, / To ransack Troy” (Pro. 1–8).

1

630

V. Huertas-Martín

plays the part, the zest added by percussionist Evelyn Glennie’s music throughout the show, the play’s blending of comedy and tragedy contrasted with the epic and sci-fi tones of the trailer, and so on. According to Mark Crossley, the theatre has the capacity to envelop endless profusions of modes, modalities, and media. This, as Crossley continues, complicates and multiplies unpredictability in processes of mediation and representation in the theatre (2021: 97). This richness was appreciated by those who could, on November 14, 2018, see the play “live” from the cinema. An experience different from that of the theatre, it started with a short documentary paying homage to the RSC’s history of productions of Troilus and Cressida, particularly to the one by the iconic directors Peter Hall and John Barton in 1960–1962.2 The overall production emphasized “synchronic” and “genealogical” perspectives (Rajewsky 2005: 47) or, in other words, “synchronic” and “diachronic” perspectives (Elleström 2021: 73) to intermediality in Troilus and Cressida. Gaps between performances were surely produced in the spectators’ cognitive involvement through what are regarded as the sensorial and formal games of intermediality (Ndalianis 2004: 3). Taking this example as our standpoint, we may discern two angles to Shakespeare’s intermediality: One angle centers on intermediality as artistic practice, and the second focuses on its heuristic potential (Rajewsky 2010: 51) as an instrument of analysis and exploration, which may be productively applied to Shakespeare. With their theoretical model to understand intermediality in theatre performance, Chapple and Kattenbelt pointed at the possibilities of interaction between sign systems (word, image and sound) and dimensions of body, space, and time, all of which productively intersect alongside digital analogue as well as mediatized-lived coordinates in complex ways in the contemporary theatre practice (2007: 23–25). According to Kattenbelt, “It is because of its capacity to incorporate all media that we can consider theatre as a hypermedium, that is to say, a medium that can contain all media” (2008: 23). Yet, taking into consideration Shakespeare’s idiosyncratic metatheatricality as vantage point, Aneta Mancewiz defined intermediality “as a productive and self-conscious application of media in performance” (2011: 7). Mancewiz identified this application in a selection of European Shakespearean productions made between 1999 and 2012, coinciding with the heyday of the Digital Age. According to Mancewiz, intermediality “opens the possibility of investigating inter-relationships and inter-exchanges of theatre with other arts” (Ibid.). Mancewiz’s studies were followed by a scholarly corpus developed by Shakespeare adaptation scholars.3 However, intermediality has recurrently been an umbrella term applied to a host of related terms, as Irina O. Rajewsky argues (2005: 44). This has been the case with Shakespeare too insofar as intermediality is used as a tag for

2

I attended the live broadcast of Troilus and Cressida at the MAC Film Theatre of the Midlands Arts Centre in Cannon Hill Park (Birmingham), 21.6 miles away from the theater in Shakespeare’s hometown. 3 For established information in the field of Shakespeare and Adaptation Studies, see Henderson and O’Neill (2022).

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

631

Shakespearean performances which deploy or are wrapped in digital media. According to Sofía Muñoz-Valdivieso, intermediality in Shakespearean performance is inevitable: Not only because of the active role of the non-scholar in the digital distribution of digital analysis and/or some form of performance of the plays, but also because the theatre companies themselves are increasingly seeking the projection of their work through live streaming or recorded broadcasting, forms that blend cinematic and theatrical features. (2017: 72)

Muñoz-Valdivieso endorses a view on intermediality as a practice based on digital performance and the digital appropriation of Shakespeare. Only recently theorized, intermediality may, indeed, be regarded as an artistic practice. However, it may also be regarded as a theoretical lens of study and as intrinsic to the communication process. Such communication processes, according to Elleström’s theoretical model for understanding intermedial relations (2021), include the integration and combination of media, as well as crossings of media borders. At the end of the day, these operations are dependent on both the producers and receivers of communication. Early research on intermedial Shakespeares has led to the development of an alternative canon of Shakespeare in performance and on screen. In light of recent studies, this canon includes, among others, stagings like Erwin Piscator’s The Merchant of Venice (1965), Hansgunter Heyme’s Hamlet (1979), Robert Wilson’s production of Heinrich Müller’s Hamletsmachine (1986), Luigi de Angelis and Chiara Lagani’s Romeo e Giulietta – et ultra (1999–2000), Piotr Lachmann’s Hamlet gliwicki (2006), Stefan Puchner’s Der Sturm (2007), Ivo van Hove’s Roman Tragedies (2007), Grzegor Jarzyna’s Macbeth (2007), Thomas Ostermeier’s Hamlet (2008), the Wooster Group’s Hamlet directed by Elizabeth LeCompte (2008), Armando Punzo’s Hamlice (2010), Punchdrunk’s immersive Sleep No More (2010), Maria Federica Maestri and Francesco Pittito’s Hamlet (2012), Elevator Repair Service’s Measure for Measure (2017), and Forced Entertainment’s Complete Works: Tabletop Shakespeare. When it comes to film, the canon includes works such as Asta Nielsen’s Amleth (dir. Sven Gade, 1921), Prospero’s Books (dir. Peter Greenaway, 1991), Titus (dir. Julie Taymor, 1999), Hamlet (dir. Michael Almereyda, 2000), or Hamlet_X (dir. Herbert Fritsch, 2003). Nonetheless, scholarly works on intermediality in theatre have brought to light Shakespeare’s heuristic qualities as an intrinsically and actively intermedial and transmedial corpus. According to Bryan Reynolds et al., Shakespeare is an articulatory space that encompasses the plurality of Shakespeare-related articulatory spaces and formations (such as theaterspace, lovespace, humanismspace, as well as more specific Freudspace, Marxspace, R&Jspace, and Hamletspace) and the time, speed, and force at which these articulatory spaces transmit, replicate, and reconfigure through places, cultures, and eras. (2017: 28)

632

V. Huertas-Martín

What Reynolds et al. define as Shakespeare’s “articulatory spaces” and successive reconfigurations in space and time may just as well be referred to as Shakespearean intermediality, i.e., Shakespeare’s engagement with different discourses, worldviews, Gestalten, and media formations across time and space. Speaking about Shakespeare’s adaptation to the screen, British director Peter Brook referred to the plays in terms similar to those used by Reynolds et al. Screening Shakespeare, for Brook, involves Finding the ways of shifting gears, styles and conventions as lightly and as deftly on the screen as within the mental processes reflected by the Elizabethan blank verse onto the screen of the mind. (quoted in Reeves 1972: 38)

Brook identifies in the imagery of Shakespeare’s blank verse what seems to be an inner carburant. Such a carburant propels Shakespeare forward in performances across history and across geographies, in anglophone and non-anglophone worlds, on stage, and elsewhere. Shakespeare is, in other words, a heuristic source of media border crossings. The fact is that both Reynolds et al. and Brook use imaginative references to articulations, formations, shifts, and gears that serve as conceits which may be explained using Elleström’s theoretical model of intermediality (2021). Apart from the features already pointed out, such a model involves distinguishing and classifying types of media and modalities and pointing out their varied points of intersection.

Shakespeare and Intermediality: Scholarly Context Shakespeare’s plays have been rewritten and appropriated in all sorts of media; they have become means, vehicles, and channels for expressing and communicating almost every human concern. Regarded as “not of an age, but for all time” (Ben Jonson 1623), labelled as “our contemporary” (Jan Kott, [1965] 1988), Shakespeare’s vitality through every age is attributable, according to Marjorie Garber, to his “capacity to speak directly to circumstances the playwright could not have anticipated or foreseen.” Shakespeare perennially stands, as Garber continues, “[l]ike a portrait whose eyes seem to follow you around the room, engaging your glance from every angle, the plays and their characters seem always to be ‘modern,’ always to be ‘us’” (2004: 19). Reviewing the currency which political criticism gained in Shakespearean scholarship at the end of the twentieth century,4 Ivo Kamps wrote that: It is more important that one writes about Shakespeare than what one writes about him . . . over time, Shakespeare is far more important to criticism as a conduit, as a uniquely powerful academic interface, as that part of the academic body through which the most theoretical innovation and political energy course. (1999: 24; emphasis mine) 4

For established knowledge on political approaches to Shakespeare, see Parvini (2012).

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

633

Kamps’s description of Shakespeare as a conduit is confirmed by the variety of subfields in the discipline of Shakespeare studies.5 Shakespeare is defined as “a conduit,” i.e., an artificial channel; also, as an “interface,” which is described by the OED as “a means or place of interaction between two systems, organizations, etc.; a meeting point or common ground between two parties, systems, or disciplines; also, interaction, liaison, dialogue.” Such definitions emphasize Shakespeare’s capacity to be a mediator for any human situation but, equally, stress Shakespeare’s potential to poetically and creatively build bridges across systems. Since media borders are “enabling structures” (Rajewsky 2010: 65), much of the Shakespearean capacity for survival is explained by the texts’ intermedial features and by the contexts of production, articulation, and dissemination of the plays across time and places. The abovementioned political shift in criticism occurred during the second half of the twentieth century, alongside a proliferation of Shakespearean performance theatre studies and studies of Shakespeare on film and TV.6 Studies on screen-based Shakespeares have pointed out the oscillations between the arts and the media corresponding to each adaptation and the cross-fertilizations between stage and screen languages.7 Despite the heterogeneity of Shakespeare in mass media, we may identify significant clusters in academic studies on Shakespearean practice. One of such clusters is the filmic wave of the 1990s. This period is known as the “Kenneth Branagh Era” due to the cultural dominance of the Shakespeare films produced by the Irish actor and director.8 An alternative canon was added to Branagh’s relatively establishmentarian productions of Shakespeare. This alternative canon consisted of visually daring and experimentally bold films produced by independent filmmakers like Julie Taymor, Baz Luhrmann, Peter Greenaway, Kristian Levring, and others (Cartelli and Rowe 2007; Calbi 2013; Grant Ferguson 2016). The political tools to analyze Shakespeare, which had developed through new historicism, cultural materialism, presentism, feminism, postcolonialism, queer theory, and other lenses, found fertile grounds in the study of Shakespeare in mass media and popular culture.9 This led to the study of Shakespeare as “cultural capital” (Bourdieu [1986] 2002), valued for the marketing and selling of cultural products, such as graphic novels, comics, and a plethora of other goods.10 Shakespeare’s

5

For established knowledge on contemporary criticism on Shakespeare, see Gajowski (2021). For established knowledge on Shakespeare in theater performance, see Bulman (2017), and Kirwan and Prince (2021); for established knowledge on Shakespeare on screen, see Jackson (2020), Díaz Fernández (2021). 7 Davies (1991 [1988]). 8 See Hatchuel (2000), Crowl (2006); Shakespeare at the Cineplex: The Kenneth Branagh Era (2003), Shakespeare’s Hamlet: The Relationship Between Text and Film (2014), Cook (2011); Gerhards (2011), Maerz (2017). 9 See Burt (2002), Lanier (2002), French (2006), Albanese (2010), Brusberg-Kiermeier and Helbig (2010); Thompson (2011), Carson and Kirwan (2014), Fischlin (2014), Blackwell (2018), Balizet (2020), Földváry (2020), Pope (2020), Hartley and Holland (2020), Wilson (2020). 10 On Shakespeare as cultural capital, see Holderness (2001); see collection of essays in Prescott and Sullivan (2015), Bennett (2021). 6

634

V. Huertas-Martín

disseminations increased in what was regarded as a “hybrid” media ecology, particularly noted by scholars dedicated to Shakespeare in the digital world and in live theatre broadcasts. The volume Shakespeare and the “Live” Broadcast Experience (2018), co-edited by Pascale Aebischer, Laurie Osborne, and Susan Greenhalgh, nuances obsolete understandings of “liveness.” For a start, the wasted notion that “liveness” should be solely regarded as “co-presence” is challenged. This is achieved by pointing at the modes of audience participation afforded by social and digital media (Sullivan 2018). It is demonstrated that the recording spaces in the British cinematic and television remediations of Shakespeare during the twentieth century were, as Greenhalgh shows, inspired, modelled, and shaped by the architectonic features of the theatrical venues where Shakespeare was performed in the twentieth-century England (2018). Audiences learned, as shown by Rachael Nicholas (2018), to revel in the intermedial experience of live theatre performances by engaging with social media while enjoying live or asynchronous performances. Likewise, these essays proved that spaces (Aebischer 2018) and the bodies of actors (Sharrock 2018) became interfaces for exchanges between theatre and cinema in live theatre broadcasts. Overall, analyses of hybrid works, such as live theatre or stageto-screen and screen-to-stage hybrid productions of Shakespeare, benefit from combined analyses of the grammars of theatre, cinema, and television (Wyver 2015, 2016; Erin Sullivan 2017; Huertas-Martín 2016, 2020). (For a full discussion on stage-to-screen adaptations and hybridity (including an interview with producer John Wyver, see pages 258–290, in Maurice Hindle, Shakespeare on Film (Second Edition) (London: Palgrave Macmillan, [2007] 2015).) Studies on Shakespeare in participatory culture have augmented with the diversification of Shakespeare Studies in the early twenty-first century. Video performance archives such as MIT Global Shakespeares and Shakespeare-400 prove that global and digital Shakespeares have, to an extent, walked hand in hand, though not unproblematically. Efforts have been made to subvert ideological biases on Shakespeare a universal representative of the Western canon in order to treat it as “a boundless user-driven archive of material to be repurposed and refashioned to suit the tastes of its users in today’s technologically engaged culture” (Fazel and Geddes 2016: 275), as rhizomatic (Lanier 2014), and context-specific worldwide phenomenon.11 As Alexa Alice Joubin argues: The rise of global Shakespeares is inseparable from the prevalence of digital video on commercial and open-access platforms, because these platforms provide inter-connected, instantaneous forms of communication for site-specific epistemologies. (2021: 141)

In Shakespeare and YouTube, Stephen O’Neill regards “Shakespeare” as “an increasingly unbounded category, one extending beyond the corpus of the texts to encompass a range of media forms and cultural stratifications (high, mass, popular)” (2014: 4). Yet, YouTube has also become a site for the audiovisual dissemination of

11

See Burnett (2015), Young (2016); see essays in Desmet and Iyengar (2020).

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

635

Shakespearean works that are otherwise rarely mediated except as printed texts. These examples include videos which intermediate visual languages with recorded versions of narrative poems, like Richard Burton’s reading of The Rape of Lucrece and Jamie Muffett’s reading of Shakespeare Sonnets, or which harbor films like Edward Lui’s Venus and Adonis (2015) or Christina Heyworth’s video The Phoenix and the Turtle. Four years later, O’Neill wrote in his introduction to Broadcast Your Shakespeare that Shakespeare [was] continually being sowed and scattered, planted and dispersed, and. . . it [was] not always possible or desirable to decipher the seed from the human and technological mechanism that cast it so broadly. (2018: 22)

O’Neill’s assertion does not just describe Shakespeare’s embedding in new media as part of convergence culture (Jenkins [2006] 2008). Rather, it suggests that Shakespeare has become an interface that interweaves human creatives and technology, an integration which, as will be shown, poses opportunities as well as challenges. Even running the risk, that almost everything could be regarded as Shakespeare (Lanier 2017; Winckler 2020), it is unquestionable that in current popular culture and mass media, Shakespeare is both verbal and nonverbal, a “thing-Shakespeare,” as expressed by Maurizio Calbi in Derridean terms (2013), a series of iterations which, simultaneously, could be said to be and not to be Shakespeare (Desmet et al. 2017). Conceptualized as a rhizome (Lanier 2014), as a corpus of materials to be broadcast (O’Neill. 2018), as cannibalized (Refskou et al. 2019), as source-code (Winckler 2020), as echoes disseminated in popular culture (Hansen and Wetmore 2015), or as an archive (Huertas-Martín 2022), Shakespeare in adaptation de facto transcends the category of verbal text (Lanier 2011). Furthermore, Shakespeare is irreducible to the predominance of a single medium, albeit this fact remains relatively under-theorized. It is significant that Shakespeare’s shift from cinema to digital platforms has reinvigorated the plays and Shakespeare’s role as an interface for productive exchanges between nonexperts, professionals, amateurs, and academics. For while film or television are only able, as Elleström argues (2010: 31), to mediate theatre to an extent, digital media intensify participation and creative engagements across communities, a fact that multiplies the perception of the intermediality in the plays.12 One side effect of Shakespeare’s embedding in participatory culture is that vernacular adaptations of the plays and, particularly, the poems and less marketable plays are recurrent in the digital agora. Following Jay David Bolter, “The [digital] plenitude easily accommodates, indeed swallows up, the contradictory forces of high and popular culture, old and new media, conservative and radical social views” (2019: 8). This brings another side effect which was evident before the pandemic:

12

This is not to say that television’s mediation of theatretheatre is not illuminating. On the mediation of metatheatricality in TV Shakespeares, see Huertas-Martín (2018, 2019).

636

V. Huertas-Martín

Textual authority is reconfigured (Lanier 2010), rather than revered, in media. Examples such as the well-known live theatre broadcasts of the plays, comic books, graphic novels, manga Shakespeares, or the more recent crowd-produced zoom-broadcast complete canon of Shakespeare’s plays The Show Must Go Online (2020), convened by actor Robert Myles, prove this. What matters for intermedial Shakespeares, whether verbal or nonverbal, faithful, or appropriative, is how intermediality is understood in the different manifestations of mass media. It seems worth examining the conclusions extracted from the analyses of Shakespeare in intermedial performance to rethink Shakespeare’s overall oeuvre as intermedial and transmedial. As we will see, intermediality is not reducible to a selection of digital or mediatized productions. It is, as a matter of fact, part and parcel of Shakespeare.

Early Research on Intermedial Shakespeares After odd publications such as Birgit Wiens’ analysis of Herbert Fritsch’s hypermedial project Hamlet_X (in Chapple and Kattenbelt’s seminal collection Intermediality in Theatre and Performance, pp. 223–236),13 Aneta Mancewiz’s sustained work, Intermedial Shakespeares on the European Stages (2014), posited that intermediality contributed to problematizing received perceptions of Shakespeare’s verbal text as the dominant source of authority on the stage. In the anglophone theatre, as Mancewiz argued, language was the focus of attention with regard to Shakespeare. Her assertions are not misplaced, considering that textcentered anglophone productions “dominate popular understandings of contemporary Shakespearean performance” (Kirwan and Prince 2021: 4). Per contra, the productions in Mancewiz’s corpus challenged Shakespeare’s textual authority, albeit, as she explains, displaying visuals that evoked the plays’ poetic imagery. In Mancewiz’s view, this led to a “nonhierarchical relationship” across media in performance (2014: 23). One of Mancewiz’s strengths in interconnecting these productions to Shakespeare, if not to the authority of the text, was her analysis of the productions’ metatheatricality. Intermediality in this corpus “[made] their specific mechanisms more self-evident, which, in turn, creat[ed] a meta-level that enabl[ed] the transformation of stage codes” (Mancewiz 2014: 4). Mancewiz’s insistence on the metatheatrical features of these productions bridges the metatheatrical cultural valences of Shakespeare’s source texts (Calderwood 1971; Bevington 2007; Escolme 2018; Dustagheer and Newman 2018; Leonard 2018; Purcell 2018) and those of contemporary theatre (Abel [1963] 2003; Freese and Ann 2014). The theatre has been a dominant metaphor when it has come to translating Shakespeare to media other than Hamlet_X transforms the scenes of Hamlet into short film sequences which, in various forms (advertisements, animated clips, trailers, documentaries, etc.), well-known film and theatre German actors and directors take part in a large-scale appropriation of Shakespeare’s play.

13

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

637

the theatre, including television Shakespeares, such as Jane Howell’s First Tetralogy (BBC, 1983), and a plethora of films, such as The Children’s Midsummer Night’s Dream (dir. Christine Edzard, 2001), among others. Yet, while in these film and television productions the theatre was mediated on screen, in Mancewiz’s analyses, we observe the theatre’s embedding in the phenomenon, noted by Philip Auslander, that “live events now frequently are modelled on the very mediatised representations that once took the self-same live events as their models” (2008: 10). This theatrical reappropriation of mass media has implications in the bodies of the actors, which become vessels for the blending of technology and the human (Mancewiz 2014: 5). Such a conceit is not unrelated to Shakespeare’s possible anticipation of posthuman concerns, as recognized by several scholars. For, as Karen Barber asserts, “theatre’s ‘human’ creations are like androids or cyborgs, yet more compellingly viscous (adherent to their environments, including their readers)” (2021: 295–296). Hamlet, the humanist scholar per excellence, speaks of his own posthuman qualities by equating himself with a musical instrument whose “stops” (3.2.330), the “heart of [his] mistery” (3.2.331), his “lowest note to [his] compass” (3.2.332), and “music, excellent voice” (3.2.332–333), are unreachable for Guildenstern (a mere humanist!), who “[has] not the skill” (3.2.328) to play such an instrument. A second strength in Mancewiz’s analysis is her assessment of the spatiotemporal features of intermediality in Shakespearean performance. To conceptualize these, Mancewiz borrowed research tools from the fields of intertextual literary theory and archaeology. With these, she focused on what she described as intermedial interweavings of intermedial performance. Such interweavings are understood as the Multiplicity of media that multiplicitously reflect, revise, and redefine each other within a spatial and temporal framework. It introduces the texture of the weft, which in turn affords a sensual experience of media operating together in the event of performance. (2014: 24)

The concept of “intermedial interweaving” designates the integration of bodies, text, soundtrack, video, and live or recorded music, as well as the variables of time and space. It helps conceptualize the processes of adaptation that take place in mediatized performance. To the identification of citational strategies of mass media seen in the theatrical venues, Mancewiz added that some productions could fully use spatial features to assimilate the language of these mass media, such as that of video games. Video game strategies were successfully transmediated to the multistory setting for Gregor Jarzyna’s Macbeth (2007), producing the impression of levels and a limited set of moves for characters. Productive engagements between the materiality of the book and performance were seen in the book-shaped sets of Stephen Puchner’s The Tempest. The interweavings of text and architecture, paintings, and sculptures were explored in Maria Federica Maestri and Francesco Pittito’s 2012 Hamlet, performed at the Italian Farnesse Palace. Seemingly informed by Michel Foucault’s Archaeology of Knowledge (1969), Mancewiz’s stratigraphic approach facilitates combined analyses of integrated spatiotemporal modalities in the theatre. In the third place, Mancewiz’s stratigraphic approach encourages us to think of Shakespeare as a self-conscious transmedial corpus. This is evident in her analysis of Luigi de Angelis and Chiara Lagani’s Romeo e Giulietta – et ultra (1999–2000).

638

V. Huertas-Martín

Mancewiz stresses, while examining this production, that intermediality becomes, in performance, something more artistically thought-provoking than the mere display of digital media on the stage. Romeo e Giulietta – et ultra took place on two stages separated by a wall, where different versions of Shakespeare’s tragedy were performed: Romeo e Giulietta and Storia infelice di due amanti. When the two simultaneous shows concluded, the wall disappeared; the two audiences became aware of each other’s presence. Then, the wall was raised again, and spectators switched places to watch the other play. In both productions, intermediality was indicated via body movement; video projections and virtual imagery were represented with the help of metal screens, handheld mirrors, veils, and utensils which created the impression of virtual reality and digital projection without the corresponding technical media of display. Following Chapple and Kattenbelt, “Although at first sight, intermediality might appear to be a technologically driven phenomenon it actually operates, at times, without any technology being present” (2007: 12). This point is proven (as also argued by Mancewiz) in the mise en scene of Romeo e Giulietta – et ultra as, seemingly, the creatives were acknowledging that intermediality was already in the audiences’ mindset. More deeply, as it is deduced from Mancewiz’s analysis, they commented on Romeo and Juliet’s overall media history. Dreamlike sequences in lyrical speeches incorporated passages from Romeo and Juliet and other Shakespearean works with which creative convergences could be established. The characters, as Mancewiz’s analysis reveals, were portrayed as symbolic figures, as mythical representations of core values and universal concepts, rather than as psychologically developed individuals. Upon reading Mancewiz’s analysis of Romeo et Giulietta – et ultra, my impressions were that the production thematized Shakespeare as both a synchronically and a diachronically intermedial corpus, a transmedial oeuvre, always mediatized, always in the process of transmediation. Explicitly, this production articulates Romeo and Juliet’s precedents by Dante Alighieri, Masuccio Salernitano, Luigi Da Porto, Matteo Bandello, Arthur Brooke, and William Painter; subsequently, it articulates its retellings by many other artists across the centuries. These retellings often took place, as in these two productions, simultaneously, and, occasionally, they talked to one another, as seen in the cases of West Side Story (1957; later, transposed to film in 1961 and 2021), Franco Zeffirelli’s Romeo and Juliet (1968), or William Shakespeare’s Romeo+Juliet (dir. Baz Luhrmann, 1996). Suggesting that Romeo and Juliet is something larger than the sum of its performances, sources, and editions, Romeo et Giulietta – et ultra’s metaliterary commentary may be illustrated by Elleström’s description of transmediality as an understanding of media involving temporal gaps among modalities, media products and media types–either actual gaps in terms of different times of genesis or gaps in the sense that the perceiver construes the import of a medium based on previously known media. (2021: 74)

Other approaches to Shakespearean intermediality complete, nuance, and expand Mancewiz’s predominantly non-anglophone focus. In her unpublished work, “Strange Bedfellows? Visual Media Use and Intermediality in Shakespeare Productions” (2013), Shari Lynn Foster centers on intermedial anglophone productions

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

639

of Shakespeare. She historicizes Shakespeare’s intermediality from its early performances to the Victorian realistic mise en scene, a historization that may be densified with existing analyses of Victorian Shakespearean portraits and paintings or studies on visual representations of Shakespeare across the centuries.14 In the light of Foster’s analysis, it does not come as a surprise, considering the arguably protocinematic qualities of the Victorian stage, that William Poel’s impulse to return to the performing conventions of the renaissance stage more or less coincided in time with the arrival of Shakespeare to the cinema, for cinema supplied, as deduced from Foster’s analysis, the need which theatre artists tried to fulfil.15 Foster’s work merits attention insofar as she is readier than Mancewiz to include anglophone productions in the intermedial canon. Likewise, Foster dovetails theatre and screen histories, making sense of their proven and plausible cross-fertilizations, particularly in the early decades of cinema. Short films, such as King John (dir. Herbert Beerbohm Tree, 1899) – the first Shakespeare film – were, arguably, paratexts and teasers for their respective stage productions. They were certainly transmedial extensions to the plays themselves insofar as they included scenes which were only alluded to in the plays or which, as the case with King John’s signing of the Magna Carta (1215) – as seen in Beerbohm Tree’s work – were not even part of Shakespeare’s play.16 Challenging established views on the disparate histories of theatre and screen Shakespeares, it has been recently shown that the RSC started its cinematic history in its early stages as Stratford Memorial Theatre, a precedent for what has been analyzed as the company’s screen history running alongside its stage one (Wyver 2019; Huertas-Martín 2020). Another strength in Foster’s analysis is that it historicizes twentieth-century intermedial Shakespeares. To achieve this, she presents Erwin Piscator’s production of Merchant of Venice (1965) as a foundational twentieth-century intermedial Shakespeare on stage. Thus, she distinguishes a stage in a Shakespearean performance characterized by the influence of Eastern European experimental theatrical practice. Her historicizing largely corresponds with twentieth-century screen-stage relations as briefly historicized by Chiel Kattenbelt (2008) or, more thoroughly, by Dominique Goy-Blanquet (2008). Yet, Foster expands on the latter’s overview by inventorying uses of film, video, digital technology, and computer-generated technology in anglophone productions’ depictions of dreams, memories, and the supernatural. Despite her focus on anglophone stagings, it is worth noting that Foster’s most significant examples of uses of metadiegetic screens may be explained using Andrea Virginás’ distinction of diegetic uses of screens. According to Virginás, certain screen uses create “cognitive import” (2021: 153). These examples coincided with those of Mancewiz’s corpus. Both pay attention to Ivo van Howe’s The Roman Tragedies, a six-hour work which comprises Coriolanus, Julius Caesar, and Antony

14

See Poole (2003). See Brewster and Jacobs (1998). 16 For established scholarship on Shakespeare in early cinema, see Hamilton Ball (2016 [1968]), Buchanan (2009). 15

640

V. Huertas-Martín

and Cleopatra. The three works are presented on a stage turned into a television studio. Sofas are placed in the performing area, actor-technicians operate cameras, analogue clocks indicate global time frames, surtitled video projections embed parts of the plays, laptops are available for audiences so that they can check on their emails and on social media to comment on the performance events, and so on. The production itself is meant to embed the Roman plays within the global media apparatus to encourage spectator skepticism on media representations of worldwide events. It seems clear, comparing Mancewiz’s and Foster’s approaches, that the latter’s attempt to highlight intermediality in anglophone productions cannot help but lean toward what, as I have argued, seems to be an undisputable canon of intermedial Shakespeares. But does an intermedial canon necessarily involve an intermedial rethinking of Shakespeare? This seems to be the leading question in Thomas Cartelli’s rethinking of the intermedial canon in Reenacting Shakespeare in the Shakespearean Aftermath (2019). More radically distant from mainstream Shakespeares than both Mancewiz and Foster, Cartelli uses screen versions such as Prospero’s Books (dir. Peter Greenaway, 1991) as the basis for describing what he calls the “intermedial turn” in Shakespearean performance. Works such as these, for Cartelli, anticipate “how Shakespeare on film may productively function more as a database, as archive or ‘montage of attractions’ rather than as a delivery system for dramatic narrative” (2019: 7). The Aftermath is, for Cartelli, constituted by reenactments – as opposed to enactments – of the plays. Reenactments, following Cartelli, give way to A discursive and performative space and condition of awareness theoretically shared by makers . . . of the performances or experiences on offer that all things Shakespearean are present and available for redoing and reenactment differently. (2019: 13)

In Cartelli’s view, one stage which follows the “intermedial turn” consists of a “return to embodiment.” Such a return emphasizes the focus of productions on media that stress (or return to stress) the centrality of the human body in performance. Nonetheless, such a body is, like Shakespeare’s “database,” entangled with technology. According to Andy Lavender, the actors in intermedial theatre are seen as the technical media of display, a feature whose vocal work, gestures, and costumes may be used as variables in intermedial analysis (2021: 114–117). This is self-consciously done, as Cartelli shows, in Romeo Castellucci’s Julius Caesar, in which an aged Mark Antony’s body is thematized as a communicative conduit. Likewise, it is a recurrent feature in the Shakespearean plays Richard III and Hamlet, as produced by the German director Thomas Ostermeier. Mancewiz, Foster, and Cartelli coincide in pointing out how intermediality walks hand in hand with transmediality. Shakespearean fits Marvin Carlson’s description of performance as “a cultural activity deeply involved with memory and haunted by repetition” (2003: 11). As it happens, archive materials are, in fact, used to explore the effects of ghostings creatively. This is best seen in the critics’ appreciation of the 2007 Wooster Group’s Hamlet, directed by Elizabeth LeCompte. An example of the “intermedial mirror,” the Wooster Group’s Hamlet combines “postmodern pastiche”

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

641

with “Brechtian alienation,” as Mancewiz notes (2014: 84). To achieve this, the production’s setting and delivery imitate the decisions made in John Gielgud’s 1964 Hamlet starring Richard Burton, whose film production by Bill Colleran is projected on-screen during the performance. According to Mancewiz, this Hamlet establishes A new understanding of performance genealogy that relies more on audio and video recordings than on text editions . . . [and] draws on the media literacy of contemporary spectators and experience with new technologies. (2014: 85)

An engagement with Burton’s Hamlet – for Foster, a “primary text” for the performance (2013: 208) – rather than with Shakespeare’s, the Wooster Group’s production is, for Cartelli, an archaeological exploration of ruins (2019: 195–196). According to Foster, this performance is an “archaeological excursion into an icon of America’s cultural past” (2013: 208), such as Burton’s Hamlet. Crucially, this archival use of Shakespeare is not isolated but can be appreciated in contemporary performances of Shakespeare, such as Judith Buchanan’s Silents Now project or the cinematically allusive greenscreen film Macbeth (dir. Kit Monkman) (see Buchanan 2020). It may also be perceived in mainstream films, such as the two recent screen Shakespeares: Joel Cohen’s Macbeth (2021) – evocative of Orson Welles’ 1948 film – and Steven Spielberg’s West Side Story (2021), which exhibit self-conscious variations on the cinematic decisions made by Jerome Robbins and Robert Wise in their 1961 film. Archival uses of Shakespeare’s past and past performances have been identified in less obvious works such as Sons of Anarchy (2008–2014) (Huertas-Martín 2022), and, potentially, this could apply to other Serial Shakespeares. Attention to performance ghosting, to large archival serial productions of Shakespeare’s works, and to the self-conscious uses of Shakespearean appropriations point toward national and international cultural pasts – often pasts marked by Shakespeare too. Progressively, these case studies contribute to building a history of Shakespearean transmediality. Mancewiz’s steps in intermedial work on Shakespeare have been continued by other scholars, adding non-European and non-Western touches to her use of intermedial tools for analysis in performance. Tackling site-specific global Shakespeare, Alexa Alice Joubin remarks that [A]s a transhistorical and intermedial practice, global Shakespeares have been deployed to revitalize performance genres, resist colonial appendage and exemplify social reparation. (2021: 132)

Scholars like Hyunshik Ju (2016) and Aleksandra Sakowska (2018) have explored the political impact of intermediality in non-Western Shakespearean productions. Ju’s analysis of Korean documentary theatre explains the intermedial mechanisms used by the Workers Theatre Company’s production of Hamlet to denounce CortCortec’s laying off of workers in 2007. The performance combined conventional acting with the screening of passages of the workers’ lives inside CortCortec. Parallels between the play and episodes concerning the factory’s history

642

V. Huertas-Martín

encouraged the audience to match the world of Shakespeare’s play to the extradiegetic reality. Parallels were established between the CortCortec CEOs and the corrupt rule in the state of Denmark, between Hamlet’s soliloquizing and the workers’ resistance to the employers’ policies. Sakowska’s analysis of Polish director Monika Pecikiewicz’s feminist and intermedial take on Shakespeare in her productions of Hamlet and Titus Andronicus illuminates how intermedial strategies are deployed to raise audience awareness on matters of social injustice and political and chauvinist violence. However, Sakowska’s conclusions that intermediality itself– understood as the use of digital technologies in performance–does not always politically stimulate an audience’s emotive and cognitive responses to what happens on the stage seem to be equally relevant, contrary to what Ju describes. Intermediality is as dependent on the receiver’s as on the producer’s mind (Elleström 2021), a fact indicative of intermediality as a critical tool depends on the very factors which make communication successful.

The Social and Political Impact of Intermedial Shakespeares in Digital and Social Media Shakespeare’s intermediality has been researched in special issues of Borrowers and Lenders (The Journal of Shakespeare and Appropriation) – a journal devoted to Shakespearean appropriation –, Shakespeare Quarterly, which is less inclined to explore Shakespeare in performance, and Cahiers Élisabéthains. These issues have centered on Shakespeare’s dissemination through social media and digital technology and in performance. This phenomenon has given way to an understanding of Shakespeare’s capacity to generate media border crossings with specific ethical and political impacts on the reception of Shakespeare in the digital sphere. By 2010, it was clear, as Laurie Osborne claimed, that the hegemony of Shakespeare on film was coming to an end. This did not mean, as she continued, that Shakespearean films had lost their impact. Rather, it meant that Shakespearean films were forming an archive for creative work in mass media through YouTube mashups and other vernacular artistic practices (Osborne 2010). These relied on Shakespeare’s archive of adaptations rather than on fidelity to the source texts. Consequently, transmedia histories were created – or rather, Shakespearean transmediality became evident to the public eye through the dissemination of the works in the digital agora – by taking known adaptations as the vantage point. For instance, a repertoire of serious, comic, and satiric amateur YouTube Macbeth short films entitled Macmeth resulted from the impact of David Gilligan’s successful Breaking Bad (2008–2013) – a serial adaptation of Macbeth, immersed in the contemporary culture of narco-capitalism in the early twentieth-century United States. These films mix sequences from the popular complex television series with fragments from Shakespeare’s text. Inspiringly, they blended the two works in satiric, palimpsestic, and often illuminating ways. Fanfiction was inspired by Tom Hiddlestone’s participation as King Henry in the “Henry V” episode of The Hollow Crown (dir. Thea Sharrock, 2012; overall series produced by Sam Mendes), suggesting that Serial

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

643

Shakespeares were becoming archival bases for fan writing. Such fan writing is more likely to use popular adaptations of Shakespeare than to use the source texts (Fazel 2016). Examples such as these illustrate that Shakespeare’s transmedial history is the result of ongoing conversations, to use Elleström’s model, between producers’ and receivers’ minds, and of exchanges of cognitive import transferred across modalities and communities of creatives, users, and, in this context, prosumers. Social media’s achievements in blurring social barriers through Shakespeare have been significant. Social media do not only incentivize “user-generated media” (O’Neill 2015: 276) but, above all, as O’Neill continues, give way to “the intermedial nature of contemporary online participation and expressions, where users navigate and connect across a range of platforms and networks” (2015: 277). In Foreword: #Bard: “And Noble Offices Thou Mayst Effect of Mediation,” the first of a collection of essays on Shakespeare and intermediality published in Shakespere Quarterly (2016), Lanier points at social media as preserving both traditional and new forms of disseminating Shakespeare, perceived, in most cases, via contributions by prosumers. The shift from market-based culture, produced from above, to the participatory culture – a grassroots production of popular culture – produced in social media has, as Lanier argues, in the aforementioned article, a democratizing effect in Shakespeare’s professional, semiprofessional, and educational environments.17 Thanks to social and digital media, these have found incentives and opportunities to give voice to countercultures when it comes to Shakespearean appropriation. Several case studies have pointed out such political remapping of Shakespeare through the media. For instance, Hamlet recently became a favorite icon for disseminating Emo narratives through social media (Desmet 2018). More ambivalent are the appreciations of the technosphere’s treatment of feminine characters, traditionally marginalized in performance, for uses of social media often incur in conservative, anglocentric, and male-chauvinistic cultural practices (Moberly 2018; Mullin 2018). Research suggests that some feminine characters are luckier than others in terms of their treatment on social media. Ophelia has been appropriated by prosumers who have found in Ophelia 2.0 a conduit to channel their experience of depression and despair on the web. O’Neill (2015) observes that “Broadcast Yourself” policies lead to risks consequent to the spread of the Ophelia subculture. Such subculture is, as O’Neill continues, developed at the heart of a self-serving industry that homogenizes individuals: The Ophelia subculture might lead to the public exposition and circulation of photographs of girls self-harming. Regarding such risks, in her article “Ana and Mia: Ophelia on the Web” (2016), Remedios Perni describes the use of the Ophelia archetype in blogs and on websites that appropriate the heroine to address experiences of eating disorders. Informed by Michel Foucault’s concept of the “hysteric” as an anti-patriarchal rebellious individual, Perni exposes the perverse use of Ophelia on the web.18 Social media uses of Ophelia as a countercultural icon

17 18

This definition is coincident with that provided by Fiske (2011 [1989]). See Foucault (2006), Psychiatric Power Lectures at the Collége de France (1973–74) (2006).

644

V. Huertas-Martín

are, as can be deduced from Perni’s article, founded upon the premise of excluding members of the female population who are more likely to suffer eating disorders and depressions conducive to suicide from representation on the web. Mixing Ophelia narratives with pseudo-romantic subplots, such reenactments are, as she continues, interpretable as apologies for suffering: “embedding Ophelia in their imaginary worlds helps [users] to justify and encourage their self-destructive drives at the same time that they become cultural producers of the character” (2016: 506–507). Much better health is seemingly enjoyed by Juliet 2.0, appropriated, according to Kirk HendershottKraetzer (2016), by prosumers who rely on her value as an eternal feminine archetype of emancipation. Such an archetype inspires generations across the anglophone and non-anglophone spheres. It is also an archetype recurrent in other popular entertainments such as the West End musical & Juliet (2019), an alternate comic exploration of Shakespeare’s tragedy. Observing the drift of these characters on the web suggests that Shakespeare’s embedding in media is not just influenced by but also influences Shakespeare theatre aficionados and, as will be shown, readers too. Despite Juliet’s success on social media, it is evident that such media bring to the table ethical concerns regarding Shakespearean appropriation, which should not be underestimated. Shakespeare’s embedding in social media is seen to be as democratizing as it is risky. As Kylie Jarrett and Jeneen Naji (2016) observe, Shakespearean content is produced in spaces that result from the interactions between human subjects, the database, and the technological infrastructure. Therefore, the web does not constitute a unified or complete subject nor a singular or static object; rather, it leads to the formation of an unpredictable assemblage whose effects are manifested through protocols such as the request for social acceptance or self-reaffirmation in the media. Also, Jarrett and Naji continue, subjects who personalize Shakespearean contents are at the mercy of regrettable misuses of social media such as trawling, stalking, and other ills. Yet, scholarly works on Shakespearean intermediality in digital and new media are instructive. They point out the risks of social media but, at the same time, suggest routes to learning to dance through the minefields of such media. Perni’s initial observations about Ophelia on the web are recalibrated by her reliance on Georges Didi-Huberman’s concept of the Atlas. Didi-Huberman’s project consisted of collecting more than 1000 images, which were articulated and classified into panels. As Perni argues, such a system of classification revealed the mechanisms of cultural memory (2016: 511). Didi-Huberman’s Atlas permits the reconfiguring of such mechanisms and, therefore, the opening up of overdetermining epistemological views on texts and characters. Observing a selection of Tumblr, Pinterest, and other intermediations of Ophelia on the web, Perni suggests that new conclusions may be drawn from new ramifications and associations that move away from what has operated as a politically stale and dicey overuse of traditional portrayals of the heroine. In her analysis on the Facebook website “Shakespeare’s Friends,” Lisa StarksEstes uses Emmanuel Levinas’s theory on communication to argue that “mediation works in and through human beings in their connections to others, in their intersubjective relations that constitute humanity” (2016). If the subject is, in Levinasian terms, only formed through contact with the other – also, a paradigmatically

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

645

Shakespearean quality –, Facebook websites may, as Starks-Estes observes, provide the mutual care and sincere camaraderie one ideally expects from a grassroots culture produced by the joint efforts of fans, amateurs, professionals, and academics. StarksEstes depicts the Facebook page’s “wall garden,” as she calls it, as “removed from institutional pressures that dictate who does or does not have a say in the academic, educational, or theatrical worlds of Shakespearean study” (Ibid.). The “wall garden” metaphor is, perhaps, not unnaturally applied to a Facebook group whose ambience resembles the life-affirming conclusions of Shakespeare’s comedies and romances. Doubtless, Starks-Estes’ view on Facebook as a life-affirming space for mediation is identifiable in what Ewan Fernie regards as Shakespearean freedom: Shakespeare’s seekers after their own freedom are obliged to seek this via the business of more-or-less successfully integrating with a whole range of other, more-or-less successfully realised, self-integrating selves. (2017: 66)

This quintessentially Shakespearean type of freedom, based on responsibility for the other, might be transferrable to the responsible use of social media. Above all, it is in the responsible use of intermediality in social media Shakespearean prosumerism that a particular ethics of Shakespeare appropriation in the technosphere lies. If Shakespeare’s freedom is not unlimited, the possibilities of social media are also subject to the restricted possibilities of digital infrastructures, which humanists often ignore. Helen Zdriluk (2016) uses the concept of “intermedial play-building” to describe school projects that rethink Romeo and Juliet via Facebook. Participants advance sections of the plot, discuss settings, represent conflicts involved in the play-making, ask audience members to follow up and keep track of the process, and, importantly, use Shakespeare’s tragedy to explore cyberbullying and several mediarelated conflicts. Crucially, Zdriluk tackles the institutional constraints and protocols involved in mounting Facebook performances in a school context. However, these constraints, as shown in Jennifer Roberts-Smith and Shawn de Souza-Coelho’s research (2016), could be taken as enabling sites, for, in Shakespearean appropriation, they could push the boundaries beyond the centrality of the theatrum mundi metaphor as a core vehicle for the transmediation of Shakespeare’s texts. While Roberts-Smith and Shawn de Souza-Coelho (Ibid.) see gameplaying as an alternative to the virtual worlds of Facebook, Tumblr, Instagram, and social media, their results reveal that gameplaying may offer more fertile grounds than theatre for making Shakespeare a conduit for collective engagements. One conclusion to be drawn from this article is that, though the theatre’s possibilities are more open ended than the restricted possibilities of gaming, the roles of spectator, actor, and character may be more fluid in gameplaying than they are in the more hierarchic, professionalized, exclusive, and restrictive organization of the theatre. As we will see in the next section, Shakespeare’s intermediality is conditioned by the theatrical nature of the texts. It is within the multimodal and multimedia features of the theatre that Shakespearean intermediality is richly developed in artistic practice, though, paradoxically, it is by no means reducible to theatrical practice alone.

646

V. Huertas-Martín

Other Directions in Shakespearean Intermediality Martin Wiggins asserts that the death of the actor Richard Burbage marked the end of authenticity in Shakespeare’s early performances (1991: 37). It could also be said that Shakespeare’s transmedial journey began in the Restoration period, when Shakespeare’s plays and characters began to exist outside their texts – a process already begun in Shakespeare’s life, as seen, for instance, in Shakespeare’s own use of Sir John Falstaff in The Merry Wives of Windsor or in John Fletcher’s appropriation of Petruchio in The Tamer Tamed or The Woman’s Prize or of Hamlet in Philostrate. As Wiggins continues, [W]hile some characters grew to be coextensive with the plays in which they appeared, others transcended them. It is striking that the two of the most popular, Hamlet and Falstaff, were remembered as much for their physical qualities as for their parts in their respective plays: not as persons in a story, or even roles performed by an actor, but as embodied beings. (1996: 31)

Restoration productions dispensed with Shakespeare’s texts, whose irregularities of form and content were deemed unacceptable to popular taste. According to Jean I. Marsden, “[Restoration] theatre-goers may have venerated the idea of Shakespeare—but not his text” (1995: 17). Stagecraft, choreography, music scores – like Henry Purcell’s A Midsummer Night’s Dream – and spectacle took precedence over Shakespeare’s language (Dobson 1996). Rewritings of Shakespeare, such as those by John Dryden, William D’Avenant, and others, dominated stage practices during the seventeenth, eighteenth, and nineteenth centuries. Precedence was given to actors’ and actresses’ charisma, musical composition, and architecturally ambitious designs (Holland 1996; Bate 1996; Jackson 1996). It was in the early twentieth century that fidelity to Shakespeare’s original texts gained currency in performance.19 However, by that time, Shakespeare had already gained international prestige, and such prestige had been accumulated with and without the help of Shakespeare’s source texts (Lanier 2002). In light of Shakespeare’s spread in media, adaptation scholars have pointed out the risk of thinking that the plays once existed in a non-mediatized form. As Lanier argues, No adaptation directly engages the Shakespearean text in some pristine state . . . [r]ather, the Shakespeare it adapts–whether a textual Shakespeare or a stage Shakespeare or an audiovisual Shakespeare–is always already itself an adaptation, a Shakespeare that has been edited, performed, transmediated, subjected to multiple prior interpretive operations, incarnated in all manner of formats, from some prior form. (2017: 297)

Lanier’s statement may be expanded by saying that no adaptation engages Shakespeare’s works as either a textual, performance, or screen Shakespeare, for

19

See Speaight (1954).

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

647

Shakespearean practice is irreducible to single-media forms. The processes of editing Shakespeare’s texts, starting with the 1623 Folio, are the consequence of a series of mediations. These mediations impacted the reception of the plays. For instance, the history plays were published as serialized sequences, anticipating contemporary seriality.20 Subsequent printed editions of Shakespeare were, in successive centuries, not purely textual but dependent on each book’s outlook to cosignify with the verbal text (Galey 2014). This intermedial complexity predates the Folio. Shakespeare’s plays, as performed on the Renaissance stage – referred to nowadays as the “original practices” (Dustagheer 2021) – were intermedial. They reached spectacular heights with Court Masques, deus ex machina scenes, or realistic scenes, such as the opening of The Tempest.21 Songs, such as “Full Fathom Five Thy Father Lies,” “Sigh No More, Ladies, Sight No More,” “Willow Song,” “When That I Was a Tiny Little Boy,” “Fear No More the Heat o’ the Sun,” and more, are part of a music archive in Robert Johnson’s remaining scores (an archive, indeed, to match Shakespeare’s contemporary musical canon, which includes Felix Mendelssohn, William Walton, Cole Porter, Leonard Bernstein, Nino Rota, and Patrick Doyle, among others).22 Viols and lutes delighted audiences during masques and serenades. Characters like Benedick and King Richard II – and, arguably, the audiences – demonstrated sensitivity to the stimuli of diegetic music. Trumpets, drums, and pipes indicated the beginnings of performances and added stamina to arresting battles in history plays. Plays ended, as annotated by Swiss traveler Thomas Platter in 1599, with the company dancing “admirably and exceedingly gracefully.”23 Players were expected to dance popular jigs, galliards, voltas, corantos, pavanes, and canarios to make a living in the theatre. Special effects, such as fireworks, bloody deaths, otherworldly presences (armed talking heads, bleeding children, or Hamlet’s Ghost, for example), and objects like handkerchiefs, rings, daggers, and skulls, added meaning to the delivery of the texts. As Marjorie Garber summarizes, “[s]ome portions of Shakespeare’s plays are inaccessible to us because they are made up of spectacles or performances rather than words” (2004: 32). As suggested by theatre historians, the complexity of the spectacles must have required the clockwork precision of the well-oiled machinery of contemporary musicals.24 Furthermore, the poetic language of the play texts is, likewise, itself intermedial. The playwright’s work developed from his readings of several ancient, medieval, and contemporary narrative and dramatic sources. It was also derived from past and contemporary stage practices and topical events. However, evidence of ekphrasis,

20

See Smith (2007). On spectacular features of early modern English theatre, see Orgel (2011). 22 Toffolo (2002), Sanders (2007), Bono (2009), Preston (2009). 23 Ernest Schanzer’s translation of “Thomas Platter’s German text in Thomas Platter’s Observations on the Elizabethan Stage.” N&Q, 201 (1956), 465–7. Quoted in David Daniell’s introduction to the Arden Shakespeare edition of Julius Caesar (Third Series) (1998). 24 See Gurr (2006 [1992]). 21

648

V. Huertas-Martín

metaliterature, and metatheatre in the plays prove that Shakespeare was attracted by the emerging possibilities of Renaissance remediation practices and the creative potential of engaging across the borders of old and new media. Similarly, Shakespeare was interested in the borders between fiction and reality in the theatre. The theatre, as suggested in the opening scene of Timon of Athens, can subsume contending arts, such as poetry and painting (Rouse 2015). Intermedial processes were, more substantially, stitched into the characters’ psychologies. In Twelfth Night, Orsino’s desire for Olivia is expressed through the combined evocation of her pleasant perfume, the transformative impact of her sight on Orsino’s character’s body, the surfeit of taste, and the therapeutic effect of music: If music be the food on love, play on, Give me excess of it. . . [. . .] O, when mine eyes did see Olivia first. Methought she purged the air of pestilence, That instant was I turned into a hart, And my desires, like fell and cruel hounds, E’er since pursue me. (1.1.1–22)

In Antony and Cleopatra, when explaining Mark Antony’s first meeting with Cleopatra, Enobarbus blends the narrative with references to the material environment, colors, smells, sights, touches, and sounds. Cleopatra is, in Enobarbus’ remediation of the Queen’s memory, an improvement on Venus (“. . .she did lie / In her pavilion, cloth-of-gold of tissue, / O’erpicturing that Venus where we see / The fancy outwork nature,” 2.2.208–211). Allusions to classic creatures, such as mermaids and the Nereides, blur the borders of fiction and truth, for it is suggested that Enobarbus’ rich language construes as much as remembers all. These intermedial processes were multidirectional in performance. The spectators were interpellated and summoned to take part in these intermediation processes. The Chorus in Henry V remarks that the audience should actively “[o]n [their] imaginary forces work. / Suppose within the girdle of [those] walls / [Were then] confined two mighty monarchies. . .” (Prol. 18–19). By summoning the audience to set their imagination to work and to suppose, the Chorus shifts the responsibility to make sense of the combinatory possibilities of visual and verbal signs and to make sense of the intra- and extra-communicational domains (identified in Elleström 2021) onto the spectators. The complexity of these processes may, in part, explain the interpretive complexity of the plays. As Hamlet says, “there is nothing either good or bad but thinking makes it so” (2.2.239–240). These words were uttered by the Prince who needs the combined (i.e., the transmedial) experiences of the tale told by the Ghost of King Hamlet, the record of this encounter on his memory tables, and, to find “grounds / More relative” (3.1.556–557) than the Ghost’s word, the simulation of the assassination in the theatre, with the supplement of the dozen lines he provides to the Players. Much of this border crossing and gap filling in Hamlet is materialized in the performance space itself. Such a space energized what William Egginton regarded as a baroque problem of thought: the truth. In Egginton’s view,

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

649

Modernity’s fundamental problem of thought is that the subject of knowledge can only approach the world through a veil of appearances; truth is defined as the adequation of our knowledge to the world thus veiled; hence, inquiry of any kind must be guided by the reduction of whatever difference exists between the appearance and the world as it is. (2010: 2)

However, the Baroque’s minor strategy, as described by Egginton, Take[s] the major strategy too seriously; it nestles into the representation and refuses to refer it to some other reality, but instead affirms it, albeit ironically, as its only reality. (Ibid. 6)

This Baroque logic serves to define the Prince’s relationship to the truth. For, if in this way, the theatre becomes a vehicle for Hamlet to find such truth, this truth lies, it seems, somewhere in between, at a space of transit between all the versions of the King’s assassination that Hamlet gathers, and which he himself contributes toward producing. Making sense of these appearances, superimposed upon one another, implicates an ongoing crossing of medial borders between diegetic fiction and extradiegetic reality. The theatrical space in Shakespeare’s plays is an intermedial site between the fictional and the real. Robert Weimann divided Shakespeare’s theatrical space into locus – the space of authority, coinciding with the diegetic space – and platea, the space for subversion, coinciding with the stage area nearer the groundlings, with the plays giving way to recurrent crossings from one world to the other (1978: 75–76). Recently, Stephen Purcell more poignantly located the fictional space of transit between the abovementioned diegetic and extradiegetic features of the plays in the platea. For Purcell, contrasts the “dissociative” and the “bisociative” manners in which platea and locus are distinguished. The latter suggests that figures in platea “inhabit the worlds of both play and audience at once, both actor and character, real and fictional” (2021: 94). The suggested border crossings are contingent, historically determined, and subject to change. According to O’Neill, “social media [have become] a laboratory, a potentially creative space that prompts responses to Shakespeare in forms yet to be encountered” (2015: 283). This view is extended to our understanding of intermediality as a heuristic mechanism to conceive Shakespeare’s plays. According to Mancewiz, Intermediality offers efficient ways of activating in performance those elements that European artists find fundamental for Shakespearean drama. Thus intermediality does not necessarily entail outright rejection of the script or disregard of its structure and significance. On the contrary, it leads to creative engagements with Shakespeare’s plays, at times bringing out their characteristics while introducing new elements into their interpretation. (2014: 9)

During the last two decades, such intermedial features in Shakespeare’s plays have been observed in piecemeal scholarly works tackling Shakespearean intermediality in traditional media. One instance of this can be appreciated in artistic media character analyses outside the theatre. Erik Heine combines filmic and

650

V. Huertas-Martín

musical analyses of Dimitri Shostakovich’s soundtrack for Hamlet (dir. Grigori Kozintsev, 1964), identifying musical patterns which define the characters. He describes the harpsichord as an “instrument of control” and a symbol of subjection for Ophelia, treated as a puppet by the court (2009: 111). Counterweight to the lofty and epic tones defining the Ghost and Hamlet, Ophelia’s music is disturbing and quasi-mutinous, adding more than just a touch of sedition to Kozintsev’s politically daring film. However, rather than merely following suit with Hamlet’s dissidence, Ophelia’s music critiques the film’s patriarchal order, which is represented by Claudius and the Hamlets. Recently, analysis of intermediality reached the field of Shakespeare in ballet. Iris Julia Bühler (2019) explores character expansion in Rudolf Nureyev’s Romeo and Juliet (1977). Bühler sees in ballet a device to fill “certain gaps in the text” and “expand creatively on hints in the play, thus allowing the audience to fully appreciate Juliet’s exceptional status as an active and selfreflexive tragic heroine” (2009: 443). Background information on Juliet’s life, growth, and other peculiarities strengthen her presence on stage and align her with Shakespearean life-affirming heroines from the comedies and romances. Intermediality implies the integration of sensorial modalities with material, spatiotemporal, and semiotic modalities of communication (see Elleström 2021). This seems to be suggested in Charles H. Frey’s Making Sense of Shakespeare (1999). Frey advocates for examining the bodily experience of Shakespearean reading. Exploring Sonnet 18, he asserts that [C]onsidering how a cherished person might compare to a summer’s day–reaching into memories of summer days and pondering attributes that might link a lover or a friend to light, fragrance, warmth, happy sounds, relaxed postures, or movements through the tender air–may initiate an unaccustomed journey into complex, nonvisual, sometimes disturbing, sensory registers. Visual imagination, in particular, may resist yielding to or adding on nonvisual empathies, preferring to “see” the “darling buds of May” shake at some distance rather than to focus on sensations of bodily movement, the kinesthetic feel, of shaking in “rough winds.” (1999: 9)

What applies to the poems applies to the plays. As Frey argues, Juliet attempts to discover the essence of Romeo by ascribing to him “a nonsensuous essence.” Such a nonsensuous essence – a name – “comes flavored with physicality, [slipping] in a Romeo-mantic sense of smell if not also the “sweetness” of taste” (Ibid: 30). For Frey, such tensions between the abstract and the concrete affect “our “own” experiential imagining (concrete and sensuous as well as abstract and mental)” (Ibid: 13) while reading. This approach proposes the integrated perception of the modalities of media (sensorial and semiotic), as defined by Elleström (2021), in Shakespeare’s playtexts. Frey’s approach finds a similar application in Jonathan Gil Harris’s analysis of smells in Macbeth. Gil Harris hypothesizes that the foul-smelling ingredients used for Macbeth on the Renaissance stage, such as sulfurous brimstone, coal, and saltpeter, and the play’s textual reference to odors contributed to creating “polychronicity” (2007: 476). Such polychronicity at once triggered the audience’s vivid memories of the Gunpowder Plot and of “further back in time, to earlier

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

651

theatrical experiences” (Ibid.), such as the Corpus Christi plays or the Annual plays of Chester Cooks, which must have remained, in Gil Harris’ view, in the memory of a secretly Catholic audience yearning for festivities long suppressed and ridiculed by the dominant protestant ideology. It could be said that by combining the analysis of the sensorial (real smells), the material (smell-producing materials and objects), the spatiotemporal (Shakespeare’s times and his memories of Corpus Christie plays), and the semiotic (the play’s codification of smell-related stimuli, such as icons, indexes, and symbols) modalities in Macbeth, Gil Harris contributes to enhancing the political potential of intermediality in the study of Shakespeare’s stagings and texts. Analysis of material modalities also contributes to enhancing our perception of the plays’ political nuances. This could be said of Margitta Rouse’s analysis of intermediality in Shakespeare. Using Timon of Athens and The Rape of Lucrece as prompts, Rouse throws light on the playwright’s highlighting of creative processes in the plays as explicit, competing fields between the different arts, such as painting and poetry (2015). Maddalena Pennacchia regards props in Shakespeare as sites “of transit of information in an increasingly complex communication system that is made of different, interrelated media” (2019: 336). For Pennacchia, writing and reading the media of display in Roman plays mark the shifts of power in Shakespeare’s dramaturgy, for instance, as she says, by privileging the medium of theatre over the written medium. However, these medial uses are not as predictable as they might seem, for Octavius Caesar, a “writerly character,” is the one who finally triumphs over Antony (Ibid.: 339). Doubtless, this intermedial use constitutes a reflection of the tensions between writing and spectacle and of the shifts in power in the plays, as observable in the arts and fiction of Shakespeare’s time. Intermediality reaches Shakespeare’s bookbinding and reading, too. Sujata Iyengar’s research focuses on the potentialities of intermediality in bookbinding. Exploring the Dove Press’s editing process for Hamlet, directed by T. J. CobdenAnderson, she describes an intermediation of public playgoing and private reading (2016: 482). Speech prefixes and texts were, as Iyengar observed, placed on different sides of the page according to the character’s position on the stage: characters taking part in the main action would take one side of the page; characters taking part in diegetic playing were placed at the center. These decisions led to interpretive challenges. For instance, the speech prefix for Hamlet’s attempt to deliver Pyrrhus’s speech was placed on the left, not the center of the page: the editors meant the speech, as Iyengar said, “as an intervention rather than a set piece, action rather than exposition, poesis rather than ekphrasis” (2016: 500). Metatheatricality was, likewise, a reading practice with early modern playbooks, as proven in Harry Newman’s article, “Reading Metatheatre” (2018). This potential for metatheatrical reading is observable, as Newman explains, in character lists, prefatory materials, and dedications, all of which promote self-conscious engagement with play texts (Ibid.: 93). This is strengthened, as Newman continues, by helping readers orientate themselves within a fictional space and by provoking contemplation about the artificial

652

V. Huertas-Martín

construction of theatrical characters. Newman’s analysis reveals that early readers actually took manuscript notes on playbooks, based on their experience of play reading (Ibid.: 94). Thus, he also proves that in the seventeenth century, the readers’ perception of Shakespeare and his contemporaries was, at least to an extent, intermedial. In a recent article, “Text, Performance, Screen: Shakespeare and Media Literacy” (2021), Lanier shows that the replacement of text with screen in reference to Shakespeare Studies insinuates the current predominance of screens as main Shakespearean conduits or interfaces. In this article, he also warns that this attention to the screen comes at the expense of occluding mediatic features, the differences between media and their corresponding technological apparatuses (120). Lanier’s summons to a historicized study of Shakespeare within the specificities of media echoes Elleström’s warning to study intermediality by first paying attention to proper definitions of media. Furthermore, Elleström’s refusal to regard communication as divided into normal communication and intermedial communication may help explain why a Shakespearean like Lanier would, after more than 40 years of critical studies of Shakespeare on screen and in mass media and popular culture, make such a claim. Yet, as I have been showing, critical approaches to Shakespeare’s intermediality – as well as most critical approaches to the relations between the theatre and the screen (Ingham 2017) – seem to have taken their vantage point from the fundamental oppositions between different media. W. J. Thomas Mitchell proposed investigating the oppositions of word and image differently: Perhaps the redemption of the imagination lies in accepting the fact that we create much of our world out of the dialogue between verbal and pictorial representations, and that our task is not to renounce this dialogue in favor of a direct assault on nature but to see that nature already informs both sides of the conversation. ([1986] 1987: 46)

Such a search for a harmonious encounter between opposite poles can be appreciated in work by scholars concerned with Shakespeare’s intermediality. Christy Desmet’s examination on Shakespearean apps Luminary Shakespeare (Folger Shakespeare Library), Shakespeare, Wordplay Shakespeare, and Shakespeare in Bits seems illustrative. Desmet analyzes the Folger editions’ multimedia materials accompanying the digital reading of plays and the affordances of social media related to these editions. Apps, as Desmet asserts, combine reading and viewing of digital dramatization of the plays in what appears, in Wordplay Shakespeare, to be “a new series of closet drama” (2016: 221). These apps offer a unique experience of enjoying Shakespeare through a combined appreciation of the verbal text, screen work, live performance, and social media. Desmet focuses on the goal of building bridges between the critical camps that have divided Shakespearean studies for years: on one front, critics like Bertrand Evans or Stephen Greenblatt, who value the solitary experience of reading Shakespeare’s sources; and on the other, the adaptation scholars influenced by cultural materialism, presentism, and critical theory who have wanted to radically rethink Shakespeare to best serve the interests

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

653

of society. Desmet’s article throws light upon the fact that building bridges between reading and performance, the screen and the theatre, the text and the page, and the source and adaptation is plausible, exciting, and intellectually challenging. She argues that The real fun actually lies in granting to the Shakespearean apps the literary status they deserve and making them, as well as the bard that lurks beneath their layers of digital data, a subject for analysis and close reading. (226).

Desmet’s exhortation to the close reading of apps involves a call to reading Shakespeare by considering the numberless integrations, combinations, and crossings of borders between the modalities coexisting in digital media. Such a task, systematically extended to Shakespeare and its afterlives, is bound to refine our knowledge of the communicative potential of Shakespeare, perhaps to deepen our understandings of its semiotic and political universe. The abovementioned analyses of Shakespeare’s border crossings of media suggest that it is in those border crossings, rather than in verbal language, that we may find, in a global age marked by plurality and diversity, more meaningful insights on Shakespeare’s work. It is no wonder that Peter Brook intuitively articulated the challenge of mediating the plays via metaphors related to motoring: “shifts,” changes in direction, and “gears,” cogs working with others, reveal a complex machine setting Shakespeare’s expressive (perhaps daunting) apparatus in motion. Such an apparatus produces a series of encounters between modalities unfamiliar with one another. Intermediality, thus, does not merely subvert received notions on the centrality of the texts – such subversions have already been carried to effect by materialist, critical race, intersectional, feminist, performance, screen, and adaptation scholars. Intermediality suggests a further step: creating paths of analysis to enrich knowledge and generate new and fertile paths to new knowledge on Shakespeare’s intermedial communicative encounters. This approach may go further than subverting the textual authority; it may well take an entirely different point of departure.

Conclusion Research on Shakespeare’s intermediality has been, thus far, centered on digital technology in digital appropriations of the plays and on theatre productions that deploy new media and digital technology. Research on these productions has reinvigorated the collective and academic understanding of Shakespeare as an adaptable vehicle for experimentation with new technologies in performance. Likewise, research has thrown light on Shakespeare as a transmedial corpus – often referred to as an archive – undergoing transformation across the ages. However, as I have shown, the most crucial contribution of scholarship on intermedial

654

V. Huertas-Martín

Shakespeares has highlighted what might be regarded as misplaced dichotomic understandings of text and performances or either verbal and/or theatrical languages seen in opposition to intermedial language. One of the most productive contributions brought about by the diversification of Shakespeare Studies in digital media and new media has been the realization that Shakespearean appropriation and practice is not as dependent on emphasizing the differences between the arts and the media as it is dependent on the building of bridges between the producer’s and receiver’s minds, each of these largely consisting of complex intermedial cognitive networks. As also proven, intermediality in Shakespeare may be politically productive and may bring positive or negative effects on communities of users. This contributes to proving, firstly, that Shakespeare’s intermediality involves the construction of channels of communication between the minds of interpreters. Secondly, it reveals that these processes of communication lead to a disparity of results and that success in communicative processes acknowledging intermediality depends on a community’s skills to positively bridge communication gaps, facilitating the crossing of borders between languages and imaginaries. Therefore, as I have shown, intermediality shows us how to conceive Shakespeare’s theatricality as a rich, hypermedial, ongoing communicative engagement with the other arts, not just as an anecdotal episode in Shakespeare’s reception and adaptation history. Rethinking media histories on Shakespeare necessarily involves acknowledging that the media (and corresponding traditionally assigned sign systems) involved cannot do without the systematic (and historically contextualized) study of border crossings, integrations, and combinations taking place in the modalities displayed in Shakespeare. What began as the exploration of contemporary digital Shakespearean performances is slowly turning into a series of text-eccentric forms of rethinking Shakespeare. As, I hope, this study has shown, the task has just started. Acknowledgments This publication is part of the scientific programme of the Research Group «CIRCE: Early Modern Theatre on Screen», developed at the University of Valencia, referenced as CIGE/2021/086, with Víctor Huertas Martín as Principal Investigator, financed by the Conselleria de Innovación, Universidades, Ciencia y Sociedad Digital of the Generalitat Valenciana.

References Abel, Lionel. [1963] 2003. Tragedy and metatheatre (Essays on Dramatic Form). New York: Holmes & Meier Publishers, Inc. Aebischer, Pascale. 2018. South Bank Shakespeare goes global: Broadcasting from Shakespeare’s globe and the National Theatre. In Shakespeare & the ‘live’ theatre broadcast experience, ed. Pascale Aebischer, Susanne Greenhalgh, and Laurie E. Osborne, 113–132. London/New York: Bloomsbury. Albanese, Denise. 2010. Extramural Shakespeare. New York/Basingstoke: Palgrave Macmillan. Balizet, Ariane M. 2020. Shakespeare and girls’ studies. New York/London: Routledge. Barber, Karen. 2021. Posthumanist studies. In The Arden research handbook of contemporary Shakespeare criticism, ed. Evelyn Gajowsky, 292–304. London/New York: Bloomsbury.

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

655

Bate, Jonathan. 1996. The romantic stage. In Shakespeare (an illustrated stage history), ed. Jonathan Bate and Russell Jackson, 92–111. Oxford: Oxford University Press. Bennett, Susan. 2003. Theatre audiences: A theory of production and reception. 2nd ed. New York: Routledge. ———. 2021. Economics: Shakespeare performing cities. In The Arden research handbook of Shakespeare and contemporary performance, ed. Peter Kirwan and Katherine Prince, 99–113. London/New York/Oxford/New Delhi/Sydney. Bevington, David. 2007. This wide and Universal Theater (Shakespeare in performance then and now). Chicago/London: The University of Chicago Press. Blackwell, Anna. 2018. Shakespearean celebrity in the digital age: Fan cultures and remediation. Cham: Palgrave Macmillan. Bolter, Jay David. 2019. The digital plenitude (the decline of elite culture and the rise of new media). Cambridge/London: The MIT Press. Bono, Paola. 2009. Il Bardo in Musical. Riano: Editoria & Spettacolo. Bourdieu, Pierre. [1986] 2002. The forms of capital. In Readings in economic sociology, ed. Nicole Woolsey Biggart. Malden/Oxford, UK: Blackwell. Brewster, Ben, and Lea Jacobs. 1998. Theatre to cinema: Stage pictorialism and the early feature film. Oxford: Oxford University Press. Brusberg-Kiermeier, Stefani, and Jörg Helbig, eds. 2010. Sh@kespeare in the media: From the globe theatre to the world wide web. Revised ed. Frankfurt: Peter Lang. Buchanan, Judith. 2009. Shakespeare on silent film (an excellent dumb discourse). Cambridge: Cambridge University Press. ———. 2020. Collaborating with the dead, playing the Shakespeare archive; or, how to avoid being pushed from our stools. In Adaptation considered as a collaborative art: Adaptation in theatre performance, ed. B. Cronin et al., 323–367. Bühler, Iris Julia. 2019. Julie’s Mute Soliloquies: Visualizing thought processes in Rudolf Nureyev’s Romeo and Juliet. Women: A Cultural Review 30 (4): 440–464. Bulman, James C., ed. 2017. The Oxford handbook of Shakespeare and performance. Oxford: Oxford University Press. Burnett, Mark, T. 2015. Shakespeare and world cinema. Cambridge: Cambridge University Press. Burt, Richard, ed. 2002. Shakespeare after mass media. New York: Palgrave Macmillan. Calbi, Maurizio. 2013. Spectral Shakespeares: Media adaptations in the twenty-first century. New York: Palgrave Macmillan. Calderwood, James L. 1971. Shakespearean Metadrama (the argument of the play in Titus Andronicus, Love’s Labour’s lost, Romeo and Juliet, a midsummer Night’s dream and Richard II). Minneapolis: University of Minnesota Press. Carlson, Marvin. 2003. The haunted stage: The theatre as memory machine. Michigan: The University of Michigan Press. Carson, Christie, and Peter Kirwan, eds. 2014. Shakespeare in the digital world: Redefining scholarship and practice. Cambridge: Cambridge University Press. Cartelli, Thomas. 2019. Reenacting Shakespeare in the Shakespeare aftermath: The Intermedial turn and turn to embodiment. New York: Palgrave Macmillan. Cartelli, Thomas, and Katherine Rowe. 2007. New wave Shakespeare on screen. Cambridge/ Malden: Polity Press. Chapple, Freda, and Chiel Kattenbelt. 2007. Key issues in intermediality in theatre and performance. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 11–27. Amsterdam/New York: Rodopi. Cook, Patrick J. 2011. Cinematic hamlet: The films of Olivier, Zeffirelli, Branagh, and Almereyda. Athens: Ohio University Press. Crossley, Mark. 2021. A recalibration of Theatre’s hypermediality. In Beyond media borders (Vol. 1): Intermedial relations among multimodal media, ed. Lars Elleström, 95–112. Cham: Palgrave Macmillan.

656

V. Huertas-Martín

Crowl, Samuel. 2003. Shakespeare at the cineplex: The Kenneth Branagh era. Athens: Ohio University Press. ———. 2006. The films of Kenneth Branagh. Westport/London: Praeger Publishers. ———. 2014. Shakespeare’s hamlet: The relationship between text and film. London: Bloomsbury. Davies, Anthony. 1991 [1988]. Filming Shakespeare’s plays (the adaptations of Laurence Olivier, Orson Welles, Peter brook, Akira Kurosawa). Cambridge: Cambridge University Press. Desmet, Christy. 2016. Shakespeare and the digitized word. The CEA Critic 78 (2): 213–228. ———. 2018. Emo Hamlet: Locating Shakespearean affect in social media. In Broadcast your Shakespeare: Continuity and change across media, ed. Stephen O’Neill, 107–122. London/New York/Oxford/New Delhi/Sydney: Bloomsbury. Desmet, Christy, Natalie Loper, and Jim Casey. 2017. Introduction. In Shakespeare / Not Shakespeare, ed. Christy Desmet, Natalie Loper, and Jim Casey, 1–24. Cham: Palgrave Macmillan. Desmet, Christy, and Sujata Iyengar, eds. 2020. The Routledge handbook on Shakespeare and global appropriation. London/New York: Routledge. Dobson, Michael. 1996. Improving on the original: Actresses and adaptations. In Shakespeare (an illustrated stage history), ed. Jonathan Bate and Russell Jackson, 45–68. Oxford: Oxford University Press. Dustagheer, Sarah. 2021. Original practices: Old ways and new directions. In The Arden research handbook of Shakespeare and contemporary performance, ed. Peter Kirwan and Kathryn Prince, 65–81. London/New York/Oxford/New Delhi/Sydney: Bloomsbury. Dustagheer, Sarah, and Harry Newman. 2018. Metatheatre and early modern drama. Shakespeare Bulletin 36 (1): 3–18. Egginton, William. 2010. The theater of truth (the ideology of (Neo)Baroque aesthetics). Stanford: Stanford University Press. Elleström, Lars. 2010. The modalities of media: A model for understanding intermedial relations. In Media borders, multimodality and intermediality, ed. Lars Elleström, 11–50. New York: Palgrave Macmillan. ———. 2021. The modalities of media II: An expanded model for understanding intermedial relations. In Beyond media borders (Vol. 1): Intermedial relations among multimodal media, ed. Lars Elleström, 3–92. Cham: Palgrave Macmillan. Escolme, Bridget. 2018. Public eye and private: Intimacy and metatheatre in Pericles and the tempest. Shakespeare Bulletin 36 (1): 111–130. Fazel, Valerie M., and Louise Geddes. 2016. “Give me your hands if we be friends”: Collaborative Authority in Shakespeare fan fiction. Shakespeare 12 (3): 274–286. ———. 2017. Introduction: The Shakespeare user. In The Shakespeare user (critical and creative appropriations in a networked culture), ed. Valerie M. Fazel and Louise Geddes, 1–22. Cham: Palgrave Macmillan. Fernández, José Ramón Díaz. 2021. Shakespeare on screen in the digital era. Cahiers Élisabéthains 105 (1): 128–169. Fernie, Ewan. 2017. Shakespeare for freedom: Why the plays matter. Cambridge: Cambridge University Press. Fischlin, Daniel, ed. 2014. OuterShakespeares: Shakespeare, intermedia, and the limits of adaptation. Toronto: University of Toronto Press. Fiske, John. 2011 [1989]. Understanding popular culture. 2nd ed. London/New York: Routledge. Földváry, Kinga, Cowboy Hamlets, and Zombie Romeos. 2020. Shakespeare in genre film. Manchester: Manchester University Press. Foster, Shari Lynn. 2013. Strange Bedfellows? Visual media use and intermediality in Shakespeare productions. Ph.D. Thesis. https://etheses.bham.ac.uk/id/eprint/4690/1/Foster13MLitt.pdf. Foucault, M. 2006. History of Sexuality, translated by Robert Hurley (Pantheon Books: 1978). ———.Psychiatric Power Lectures at the Collége de France 1973–74 (trans: Burchell, G.). Palgrave Macmillan. Freese, Witt, and Mary Ann. 2014. Metatheater and modernity (Baroque and Neobaroque). Madison/Teaneck: Fairleigh Dickinson University Press.

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

657

French, Emma. 2006. Selling Shakespeare to Hollywood: The Marketing of Filmed Shakespeare Adaptations from 1989 into the New Millenium. Hatfield: University of Hertfordshire Press. Frey, Charles H. 1999. Making sense of Shakespeare. Cranbury: Associated University Presses. Gajowski, Evelyn, ed. 2021. The Arden research handbook of contemporary Shakespeare criticism. London/New York/Oxford/New Delhi/Sydney, Bloomsbury. Galey, Alan. 2014. The Shakespearean archive (experiences in new media from the renaissance to postmodernity). Cambridge: Cambridge University Press. Garber, Marjorie. 2004. Shakespeare after all. New York: Anchor Books. Genette, Gerard. [1987] 1997. Paratexts: Thresholds of interpretation, trans: Jane E. Lewin. Cambridge: Cambridge University Press. Gerhards, Vanessa. 2011. Shakespeare reloaded: The Shakespeare renaissance 1989–2004. Trier: WVT Wissenschaftlicher Verlag Trier. Gil Harris, Jonathan. 2007. The smell of “Macbeth”. Shakespeare Quarterly 58 (4): 465–486. Goy-Blanquet, Dominique. 2008. All the world’s a TV set. In Television Shakespeare (essays in honour of Míchèle Wíllems), ed. Sarah Hatchuel and Nathalie Vienne-Guerrin, 35–60. MontSaint-Aignan-Cedex: Publications des Universités de Rouen et du Havre. Grant Ferguson, Aisla. 2016. Shakespeare, cinema, counter-culture: Appropriation and inversion. New York: Routledge. Greenhalgh, Susanne. 2018. The remains of the stage: Revivifying Shakespearean theatre on screen. In Shakespeare & the ‘live’ theatre broadcast experience, ed. Pascale Aebischer, Susanne Greenhalgh, and Laurie E. Osborne, 19–40. London/New York: Bloomsbury. Gurr, Andrew. 2006 [1992]. The Shakespearean stage 1574–1642. 3rd ed. Cambridge: Cambridge University Press. Hansen, Adam, and Kevin J. Wetmore Jr. 2015. Introduction. In Shakespearean echoes, ed. Adam Hansen and Kevin J. Wetmore Jr., 1–20. New York: Palgrave Macmillan. Hatchuel, Sarah. 2000. A companion to the Shakespearean films of Kenneth Branagh. Blizzard: Winnipeg. Heine, Erik. 2009. Controlling and controlled: Ophelia and the ghost as defined by music in Kozintsev’s Hamlet. Literature/Film Quarterly 37 (2): 109–123. Hendershott-Kraetzer, Kirk. 2016. Juliet, I Prosume? Or Shakespeare and social network. Borrowers and Lenders 10 (1). Henderson, Diana E., and Stephen O’Neill, eds. 2022. The Arden research handbook of Shakespeare and adaptation. London/New York/Oxford/New Delhi/Sydney: Bloomsbury Publishing Plc. Holderness, Graham. 2001. Cultural Shakespeare (essays in the Shakespeare myth). Hertfordshire: University of Hertfordshire Press. Holland, Peter. 1996. The age of Garrick. In Shakespeare (an illustrated stage history), ed. Jonathan Bate and Russell Jackson, 69–91. Oxford: Oxford University Press. Huertas-Martín, Víctor. 2016. Theatrical self-reflexivity in Gregory Doran’s Hamlet (2009). Epos: Revista de Filología 32: 243–262. ———. 2018. Filming Metatheatretheatre in Gregory Doran’s Macbeth (2001): Refracting theatrical crises at the turn of the century. Atlantis: Journal of the Spanish Association of AngloAmerican Studies 40 (2): 101–121. ———. 2020. Hibridaciones de la Royal Shakespeare Company entre el escenario y la pantalla: Revisión crítica y perspectivas metodológicas recientes. Nerter: Revista dedicada a la literatura, el arte y el conocimiento 32–33: 66–78. ———. 2022. Hamlet goes legit: Archaeology, archive and transformative adaptation in sons of anarchy (FX 2008-2014). International Journal of English Studies 22 (1): 41–66. Ingham, Michael. 2017. Stageplay & screenplay (the intermediality of theatre and cinema). London/New York: Routledge. Iyengar, Sujata. 2016. Intermediating the book beautiful: Shakespeare at the doves press. Shakespeare Quarterly 67 (4): 481–502. Jackson, Russell. 1996. Actor-managers and the spectacular. In Shakespeare (an illustrated stage history), ed. Jonathan Bate and Russell Jackson, 112–127. Oxford: Oxford University Press.

658

V. Huertas-Martín

———, ed. 2020. The Cambridge companion to Shakespeare on screen. 2nd ed. Cambridge: Cambridge University Press. Jarrett, Kylie, and Jeneen Naji. 2016. What would media studies do? Social media Shakespeare as a technosocial process. Borrowers and Lenders 10 (1). Jenkins, Henry. [2006] 2008. Convergence culture (where old and new media collide). New York/London: New York University Press. Jonson, Ben. 1623. To the memory of My Beloved, the Author Mr. William Shakespeare. In Mr. William Shakespeares comedies, histories, & tragedies: Published according to the true original copies, ed. John Heminges and Henry Condell. Printed by Isaac Iaggard and Ed. Blount. Printed by Thomas Cotes, for Robert Allot, to be fold at the signe of the Black Beare in Paul’s Chuchyard. Joubin, Alexa Alice. 2021. Global mediation: Performing Shakespeare in the age of networked and digital cultures. In The Arden research handbook of Shakespeare and contemporary performance, ed. Peter Kirwan and Kathryn Prince, 132–150. London/New York/Dublin: Bloomsbury. Ju, Uyunshik. 2016. Becoming hamlet for only nine days: Korean workers and documentary theatre. International Federation for Theatre Research 41 (2): 122–134. Kattenbelt, Chiel. 2008. Intermediality in theatre and performance: Definitions, perceptions and medial relationships. Cultural Studies Journal of Universitat Jaume I (Culture, Language and Representation) 6: 19–29. Kirwan, Peter, and Kathryn Prince, eds. 2021. Introduction. In The Arden research handbook of Shakespeare and contemporary performance, ed. Peter Kirwan and Kathryn Prince, 1–21. London/New York/New Delhi/Sydney: Bloomsbury. Kott, Jan Kott. [1965] 1988. Shakespeare our contemporary, trans: Boleslaw Taborski. London: Routledge. Lanier, Douglas M. 2002. Shakespeare and modern popular culture, 2002. New York/London: Oxford University Press. ———. 2010. Recent Shakespeare adaptation and the mutations of cultural capital. Shakespeare Studies 38: 104–113. ———. 2011. Post-textual Shakespeare. Shakespeare Survey 64: 145–162. ———. 2014. Shakespearean Rhizomatics: Adaptation, ethics, value. In Shakespeare and the ethics of appropriation, ed. Alexa Huang and Elizabeth Rivlin, 21–40. New York: Palgrave Macmillan. ———. 2016. Foreword: #Bard: “And Noble Offices Thou Mayst Effect of Mediation”. Shakespeare Quarterly 67 (4): 401–407. ———. 2017. Shakespeare / not Shakespeare: Afterword. In Shakespeare / not Shakespeare, ed. Christy Desmet, Natalie Loper, and Jim Casey, 293–306. Palgrave Macmillan. ———. 2021. Text, performance, screen: Shakespeare and critical media literacy. Cahiers Élisabéthains: A Journal of English Renaissance Studies 105 (1): 117–127. Lavender, Andy. 2021. Multimodal acting and performing. In Beyond media borders (Vol. 1): Intermedial relations among multimodal media, ed. Lars Elleström, 113–140. Cham: Palgrave Macmillan. Leonard, Nathaniel C. 2018. All “Metatheatre” is not created equal: The knight of the burning pestle, a midsummer Night’s dream, and the navigation of the spectrum of dramatic representation. Shakespeare Bulletin 36 (1): 49–67. Maerz, Jessica M. 2017. Metanarrative functions of film genre in Kenneth Branagh’s Shakespeare film: Strange bedfellows. Newcastle: Cambridge Scholars Publishing. Mancewicz, Aneta. 2011. The stage as a page and the stage as a screen: Intermediality in Stefan Puchner’s the tempest (Münchner Kammerspiele, 2007). Forum Modernes Theater 26: 7–19. ———. 2014. Intermedial Shakespeares on European stages. New York: Palgrave Macmillan. Marsden, Jean I. 1995. The re-imagined text: Shakespeare, adaptation, and eighteenth century literary theory. Lexington: University Press of Kentucky.

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

659

Martín, Victor Huertas. 2019. Theatrum mundi and site specificity in four television Shakespeare films. Cahiers Élisabéthains 99: 76–88. Moberly, David C. 2018. “Once more to the breach!”: Shakespeare, Wikipedia’s gender gap and the online, digital elite. In Broadcast your Shakespeare: Continuity and change across media, ed. Stephen O’Neill, 87–106. London/New York/Oxford/New Delhi/Sydney: Bloomsbury. Mullin, Romano. 2018. Tweeting television/broadcasting the bard: @HollowCrownFans and digital Shakespeares. In Broadcast your Shakespeare: Continuity and change across media, ed. Stephen O’Neill, 207–226. London/New York/Oxford/New Delhi/Sydney: Bloomsbury. Muñoz-Valdivieso, Sofía. 2017. “All the World’s a stage”: William Shakespeare’s cultural capital 400 years after his death. Changing English (Studies in Culture and Education) 24 (1): 67–89. Ndalianis, Angela. 2004. Neo-Baroque and aesthetics of contemporary entertainment. London/ Cambridge: MIT Press. Newman, Harry. 2018. Reading Metatheatre. Shakespeare Bulletin 36 (1): 89–110. Nicholas, Rachael. 2018. Understand ‘New’ encounters with Shakespeare: Hybrid media and emerging audience behaviours. In Shakespeare & the ‘Live’ theatre broadcast experience, ed. Pascale Aebischer, Susanne Greenhalgh, and Laurie E. Osborne, 77–94. London/New York: Bloomsbury. O’Neill, Stephen. 2014. Ophelian negotiations: Remediating the girl on YouTube. Borrowers and Lenders 9 (1). ———. 2015. Shakespeare and social media. Literature Compass 12 (6): 274–285. O’Neill. 2018. Introduction. In Broadcast your Shakespeare (continuity and change across media), ed. Stephen O’Neill, 1–26. London/New York: Bloomsbury. Orgel, Stephen. 2011. Spectacular performances: Essays on theatre, imagery, books and selves in early modern England. Manchester/New York: Manchester University Press. Osborne, Laurie. 2010. iShakespeare: Digital art/games, intermediality, and the future of Shakespearean film. Shakespeare Studies 38: 48–57. Parvini, Neema. 2012. Shakespeare and contemporary theory (new historicism and cultural materialism). London/New York: Bloomsbury. Pennacchia, Maddalena. 2019. Intermedial props in Shakespeare’s Roman plays. Shakespeare 15 (4): 335–343. Perni, Remedios. 2016. Ana and Mia: Ophelia on the web. Shakespeare Quarterly 67 (4): 503–514. Poole, Adrian. 2003. Shakespeare and the victorians. London/New York/New Delhi/Oxford/ Sydney: Bloomsbury. Pope, Jonathan H. 2020. Shakespeare’s fans: Adapting the bard in the age of media fandom. Cham: Palgrave Macmillan. Preston, Kendra. 2009. Shakespeare, madness, and music: Scoring insanity in cinematic adaptations. Lanham: Scarecrow Press. Purcell, Stephen. 2018. Are Shakespeare’s plays always metatheatrical? Shakespeare Bulletin 36 (1): 19–35. ———. 2021. Space: Locus and Platea in modern Shakespearean performance. In The Arden research handbook of Shakespeare and contemporary performance, ed. Peter Kirwan and Kathryn Prince, 82–98. London/New York/Oxford/New Delhi/Sydney: Bloomsbury. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on Intermediality. Intérmedialités 6: 43–64. ———. 2010. Border talks: The problematic status of media Borders in the current debate about Intermediality. In Media borders, multimodality and intermediality, ed. Lars Elleström, 51–68. New York: Palgrave Macmillan. Reeves, Geoffrey. 1972. In finding Shakespeare on film: From an interview with Peter brook. In Film theory and criticism, ed. Gerald Mast & Marshall Cohen, 316–321. Refskou, Anne Sophie, Marcel Alvaro de Amorim, and Vinicius Mariano de Carvalho. 2019. Introduction. In Eating Shakespeare: Cultural anthropophagy as global methodology, ed. Anne Sophie Refskou, Marcel Alvaro de Amorim, and Vinicius Mariano de Carvalho, 1–24. London/New York: Bloomsbury.

660

V. Huertas-Martín

Reynolds, Bryan, Sam Zolodezh, and Kristin Keating Fracchia. 2017. Aspects of Intermediality: Objective agency, wonderment, and transversal refractions from the age of Shakespeare. In Intermedial Theater (performance philosophy, transversal poetics, and the future of affect), ed. Bryan Reynolds, 21–75. London: Palgrave Macmillan. Robert Hamilton Ball. 2016 [1968]. Shakespeare on silent film (a strange eventful history). London/ New York: Routledge. Roberts-Smith, Jennifer, and Shawn DeSouza-Coelho. 2016. Staging Shakespeare in social games: Towards a theory of theatrical game design. Borrowers and Lenders 10 (1). Rouse, Margitta. 2015. Text-picture relationships in the early modern period. In Handbook of intermediality: Literature, image, sound, ed. Gabriele Rippl, 65–81. Berlin: De Gruyter, Inc. Sakowska, Aleksandra. 2018. Feminist Shakespeare and saturation of experience in Monika Pecikiewicz’s Intermedial adaptations. Cahiers Élisabéthains: A Journal of English Renaissance Studies 96 (1): 59–74. Sanders, Julie. 2007. Shakespeare and music: Afterlives and borrowings. Cambridge/Malden: Polity. Sharrock, Beth. 2018. A view from the stage: Interviews with performers. In Shakespeare and the ‘live’ theatre broadcast experience, ed. Pascale Aebischer, Susanne Greenhalgh, and Laurie E. Osborne, 95–102. London/New York: Bloomsbury. Smith, Emma. 2007. Shakespeare serialized: An age of kings, in The Cambridge companion to Shakespeare and popular culture, ed. Robert Shaughnessy, 134–149. Cambridge: Cambridge University Press. Speaight, Robert. 1954. William Poel and the Elizabethan revival. Heinemann Educational Publishers. Starks-Estes, Lisa S. 2016. From Face to Facebook: Levinas’s radical ethics and “Shakespeare’s Friends.”. Borrowers and Lenders 10 (1). Sullivan, Erin. 2017. “The form of things unknown”: Shakespeare and the rise of the live broadcast. Shakespeare Bulletin 35 (4): 627–662. ———. 2018. The audience is present: Aliveness, social media, and the theatre broadcast experience. In Shakespeare & the ‘live’ theatre broadcast experience, ed. Pascale Aebischer, Susanne Greenhalgh, and Laurie E. Osborne, 59–75. London/New York: Bloomsbury. Thomas Mitchell, W. J. [1986] 1987. Iconology: Image, text, ideology. Chicago: The University of Chicago Press. Thompson, Ayanna. 2011. Passing strange: Shakespeare, race, and contemporary America. New York/Oxford: Oxford University Press. Toffolo, Stefano. 2002. Romeo e Giulietta e altri drammi shakespeariani. Musica, Cinema e Letteratura dale origini a Franco Zeffirelli a Nino Rota. Padua: Armelin Musica. Virginás, Andrea. 2021. Electronic screens in film Diegesis: Modality modes and qualifying aspects of a formation enhanced by the post-digital era. In Beyond media borders (Vol. 1): Intermedial relations among multimodal media, ed. Lars Elleström, 141–173. Cham: Palgrave Macmillan. Way, Geoffrey. 2016. Peeking behind the digital curtains: Shakespearean performance institutions, social media, and access. Borrowers and Lenders 10 (1). Weimann, Robert. 1978. Shakespeare and the popular tradition in the Theater: Studies in the social dimension of dramatic form and function, ed. Robert Schwarz. Baltimore/London: Johns Hopkins University Press. Wiens, Birgit. 2007. Hamlet and the virtual stage: Herbert Fritsch’s project Hamlet_X. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 223–236. Amsterdam/New York: Rodopi. Wiggins, Martin. 1996. The King’s men and after. In Shakespeare (an illustrated stage history), ed. Jonathan Bate and Russell Jackson, 23–44. Oxford University Press. Wilson, Jeffrey R. 2020. Shakespeare and game of thrones. London/New York: Routledge. Winckler, Reto. 2020. Hacking adaptation: Updating, porting, and forking the Shakespearean source code. Adaptation 14 (1): 1–22. https://doi.org/10.1093/adaptation/apaa026.

26

Crossing Media Borders: From Intermedial Shakespeares to. . .

661

Wyver, John. 2015. Between theatre and television: Inside the hybrid space of the wars of the roses. Critical Studies in Television 10 (3): 23–36. ———. 2016. An intimate and Intermedial form: Early television Shakespeare from BBC, 1937–1939. Shakespeare Survey 69: 347–360. ———. 2019. Screening the Royal Shakespeare Company: A critical history. London/New York: Bloomsbury. Young, Sandra. 2016. Race and the global south in early modern studies. Shakespeare Quarterly 67 (1): 125–135. Zdriluk, Helen. 2016. “Facebooked: Romeo and Juliet” as educational theatre: An improbable fiction. Ride: The Journal of Applied Theatre and Performance 21 (3): 373–384.

Metareference in the Nineteenth-Century Pictorial Press and Beyond

27

Sonya Petersson

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metareference and Its Intermedial Base in the Nineteenth-Century Pictorial Press . . . . . . . . . . . References to Image and Text Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References to Contrasting Image Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References to Anterior Pictures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion: Beyond the Nineteenth-Century Pictorial Press . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

664 668 672 676 680 686 687

Abstract

Metareference is a medial and semiotic phenomenon that occurs whenever a media product directs attention to its own medial and representational qualities. Usually associated with postmodern art and culture, metareference operates in the margins of media history. This chapter therefore makes a point of examining its presence in the nineteenth-century pictorial press, represented by the Swedish journal Ny Illustrerad Tidning (1865–1900). Here, metareference appears in the form of pictorialized letters, pictures within pictures, and textual elements that are neither inscriptions nor part of the depiction, but are nonetheless integral to pictorial representation. In this capacity, the Swedish journal is like its European and North American counterparts, which were all part of a media culture where the illustrated press boomed. The chapter aims to study how metareference in this context builds on what is introduced as the picture’s inherent, both formal and historical, intermediality. Proceeding from the closely connected fields of intermedial studies, word and image studies, and media history, the chapter demonstrates how metareference is dependent on a web of intermedial relations that tie together media products, mediating practices, and media concepts in the broader nineteenth-century media culture and how S. Petersson (*) Department of Culture and Aesthetics, Stockholm University, Stockholm, Sweden e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_32

663

664

S. Petersson

metareferential explicitness and implicitness results from such meaning-making, historical as well as formal, intermedial relations. Ultimately, and beyond the media culture of the nineteenth century, the chapter’s metareferential examples are highlighted as tools to reexamine past and present historiographically established discourses on media, mediation, and representation. Keywords

Metareference · Intermediality · Image and text · Media history · Nineteenthcentury pictorial press

Introduction The image in Fig. 1, a plate from the Swedish nineteenth-century journal Ny Illustrerad Tidning (1865–1900), shows a set of nine pictures nailed to a board, which represent scenes from the National Library. Such representations of pictures within pictures have the effect of reminding the viewer of the “first” picture’s own pictorial state. In this example, the pictorial illusionism created by the curly edges and the projected shadows suggest a reference to an idea of a real-world board. At the same time, this real-world reference is disturbed by the inscriptions below each picture. These inscriptions are not located in the same space as the depicted pictures but belong to the surface of the first-order picture. Owing to their locations, the inscriptions demand attention. They are suggestive of the artifice and artistry of mediation and representation – of pictorial attempts to convey three-dimensionality. Metareference is an umbrella term for a medial and semiotic phenomenon that occurs whenever a media product directs attention to its status as such, that is, as mediating and representing through medial means (cf. Wolf 2009, 2011; Fricke 2011; Nöth 2007a, b, 2009; Mitchell 1994, 35–82). The term has broadly come to include well-known phenomena such as metalepsis or the transgression of the barrier between two narrative worlds (cf. Thoss 2015, 4, 7–18) and mise en abyme or the recurrence of framing structures on nested levels, such as a picture inside a picture (cf. Klimek 2011, 24). Both metareference and metalepsis are concepts with strong ties to literature and narratology. It is therefore significant that scholars such as Werner Wolf (2005, 2009, 2011), Karin Kukkonen (2011a), and Liviu Lutas (2021) have explored their relevance for other media and representation in general. This chapter builds on that line of research. The chapter’s focus, however, is not the transfer of literary concepts to imagebased media but to investigate metareference in the nineteenth-century pictorial press as inherently intermedial, in order to understand how plates such as the one in Fig. 1, in their intermedial and media-historical specificity, have implications for the discussion of the concept of metareference and for a field of study that is yet to come: a future history of metareferential images. This aim proceeds from three closely aligned fields of research – intermedial studies, media history, and word and image studies – and has two basic objectives. The first is to examine

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

665

Fig. 1 Pictures from the National Library. Drawing by Otto August Mankell, unknown xylographer. Printed in Ny Illustrerad Tidning 1877, no. 52. 22.9  34 cm. (© With permission of the National Library of Sweden, Stockholm. License: CC BY-NC-ND)

666

S. Petersson

metareference as grounded in intermediality. The second is to examine intermediality as simultaneously formal and historical and as both including and transgressing such explicit media combinations as inscriptions on pictures, or combinations where image and text are both materially present in the same media product (as defined in Rajewsky 2005, 51–52). Judging from the recent pile of handbooks, “intermediality” simply means relations between media of any kind, artistic or other (cf., e.g., Glaser 2009; Eilittä et al. 2012; Herzogenrath 2012; Rippl 2015; Elleström 2021a; cf. also the definitions in Wolf 1999, 35–50, and in Lagerroth et al. 1997, two frequently cited texts from the 1990s when intermediality come to the fore as a scholarly field of research). More particularly, and in line with the discussions of Lars Elleström (2010, 2014, 2021b; see also Bruhn and Schirrmacher 2021; Bruhn 2017), Irina O. Rajewsky (2005, 2010), and W. J. T. Mitchell ([2005] 2015; see also Mitchell and Hansen 2010), or scholars associated both with what can be described as aesthetically based intermedial studies and, in Mitchell’s case, media studies more broadly, intermediality can be understood as including, or bridging, two types of differences. First, differences between interrelated media products (or elements of media products), for instance, a journal article that refers to a picture on the facing page. In such cases, both the article and the picture are present. Other cases may include a present media element that refers to an absent one, or a media element on a nested level that refers to media elements on “higher” levels. Second, differences in the nature of the relations that hold the media products (or elements in media products) together. Relations between media products could be described as synchronic and combinatory or diachronic and transmedial; symbolic, iconic, and indexical; and explicit and implicit. All this is explained in greater detail later. For the moment, the point is that intermedial studies formally include both differences between media and differences in the nature of the relations. Additionally, and in line with research taking a more media-historical approach, associated with scholars such as Lisa Gitelman (2006, 2014) and Erkki Huhtamo (2013, 2018), the formal differences should be examined as both entangled in and dependent on a nexus of cultural, aesthetic, epistemic, and technological conditions of meaning and media operability. Metareference in the nineteenth-century press will hence be studied in its historical specificity, in relation to its surrounding media culture. Still, it has the potential for exemplifying, exposing, and nuancing issues of media, mediality, and representation that are not exclusively tied to the past media culture. This is one of the chief potentials for a future history of metareferential images. Studying metareference in the nineteenth-century pictorial press challenges some commonplaces within the contemporary scholarly discourse on the topic. Historically, metareference has often been tied to postmodernity, with the exception of some well-known, and often referred to, predecessors. These include Laurence Sterne’s novel, The Life and Opinions of Tristram Shandy, Gentleman (1759–67), and Lewis Carroll’s Alice’s Adventures in Wonderland (1865) with the sequel Through the Looking Glass and What Alice Found There (1871), both illustrated by John Tenniel, as well as Diego Velázquez’s painting Las Meninas (1656) and René Magritte’s painting La Trahison des Images (1929) with its famous inscription “Ceci n’est pas

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

667

une pipe,” both analyzed by Mitchell (1994, 58–72), among others, as “metapictures” (see also, e.g., Orfila 2015; Grishakova 2010). Notably, the intention here is not to find other predecessors. First, it would be unnecessary since by now classic studies by Victor I. Stoichita ([1993] 1997) and Svetlana Alpers (1983), with or without using the term “metareference,” explore the phenomenon in Early Modern painting. Second, the teleology implied in understanding metareference as a product of postmodernity with occasional precursors, which are examined only to the extent that they confirm the present, deprives the latter of their own mediahistorical specificity (cf. discussions of media teleology in Lyons and Plunkett 2007, part one; Parikka 2012; Huhtamo and Parikka 2011, 3). Wolf rightly notes that there is little research in the media history of metareference (Wolf 2009, 73). Therefore, it seems as a jump to conclusions when he acknowledges Carroll’s Alice stories as early examples of children’s literature where metareferences occur, concluding that “such intensity of metaization was exceptional during the period of publication” (12). If research is lacking, what supports the statement? For metareferential pictorial practices that could be used to frame the Alice stories, one could look at the broader media landscape surrounding Carroll’s novels, including magazines such as Punch (1841–2002) or The Graphic: An Illustrated Weekly (1869–1932), the latter a British equivalent of Ny Illustrerad Tidning. Both journals are representative of a European and North American print media culture where the illustrated press boomed, partly due to the explosive development of new pictorial, both manual and photomechanical, technologies of reproduction (cf. Beegan 2008; Yousif 2012; Korda 2015; Mainardi 2017; Batchen 2018). Within those journals, metareferential devices are both surprisingly common and running in the margins. Pictorial letters, pictures within pictures, and textual elements that, at the same time, strangely appear inside and/or outside pictorial representations belong to the more explicit and repeatedly recurring props that direct attention to media, mediality, and representation. They recur across pictorial pages, headings, captions, and editorial texts that feature and visualize contemporary arts and entertainment, mores and lifestyles, town and countryside, news and politics, and celebrities and criminals. In this latter sense, metareference in the nineteenthcentury press literally runs in the margins: it appears within pages that visualize something else (e.g., views of the countryside or the latest cultural or political events). At the same time, it is far too explicit and recurring to be marginalized as the whim of only some occasional illustrator, especially as the trade in and reuse of clichés, as well as the line work of the printing houses, generally outweigh the importance of individual contributors. The following chapter begins by discussing previous definitions of metareference and outlines how they need to be reviewed in the light of intermediality, both as an inherent state of any media product and as involving the media-historical specificity of the plates of the nineteenth-century press. Next follows three subsections, where four examples from Ny Illustrerad Tidning are examined in detail. The first deals with pictures about image and text, the second with pictures about contrasting image types, and the third with pictures about anterior pictures. These three cases demonstrate how metareference is dependent on formal intermedial relations that are

668

S. Petersson

intertwined with references to historically specific media practices and media concepts. In the framework of intermedial and semiotic theory, the subsections move between the plates in Ny Illustrerad Tidning, contemporary media practices such as magic lantern shows, and nineteenth-century debates about the values and affordances of the illustrated press. The last and concluding section envisions what the study implies for a future history of metareferential images.

Metareference and Its Intermedial Base in the NineteenthCentury Pictorial Press As a scholarly area of research, metareference was broadly introduced in the edited volume Metareference across Media (2009), followed by The Metareferential Turn in Contemporary Arts and Media (2011). To date, Wolf’s introductory chapter to the first volume offers one of the most substantial discussions of metareference within the field. Winfried Nöth’s chapter in the same volume (Nöth 2009) and previous studies of related topics such as self-reference in the arts and media (Nöth 2007a, b) share Wolf’s ambition to define the phenomenon. Wolf’s definition of metareference is outlined with a transmedial goal in view and exemplified with what he repeatedly refers to as “literature and other media.” The special status of “literature” results from the fact that Wolf roots his concept of metareference in literary theory. His strategy is to remodel the concepts of “metafiction,” “metalepsis,” and other cognates to better apply to “other media” (Wolf 2009, 7–8). Throughout Wolf’s introduction, metareference is approached as a theme (i.e., “discursive” in nature) with the potential to be manifested in various media, although his chief examples are novels and paintings. The overall intention is to scrutinize how metareference is evoked in different media through different, media-specific, means of articulation. In this context, “media-specific” means mono-medial: painterly, musical, literary, and so on. Accordingly, Wolf makes a point of formulating three criteria of metareference that are media unspecific or can be applied to various media. In his view, metareference includes (1) a self-reference (a media product that refers to itself as an exponent of media), (2) thematization (the “discursive” nature of the self-reference), and (3) a meta level or “a specific logical origin and content of the self-reflection” (30–31). Metareference thus defined is expected to give rise to some kind of medial “meta-awareness” in the recipient, who becomes aware of the medial status of the media product and “the fact that mediarelated phenomena are at issue, rather than (hetero-)references to the world outside the media” (31). According to the transmedial claim of Wolf’s criteria, music and the visual arts are recognized as being just as capable as literature to evoke explicit and “discursive,” conceptual, metareferences. Nöth’s (2009) approach to metareference does not acknowledge this capability in music and visual arts. His chief examples are linguistic statements, which are used as model cases, and paintings and music, which are used as comparative cases. His argument is that “only language has an explicit metareferential sign repertoire” (my emphasis, 89), since only language

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

669

possesses signs, such as grammatical terms, that are aimed and instituted for metareferential purposes. Following from this, pictorial signs are for Nöth only capable of implicit reference since they refer by way of iconicity and indexicality (i.e., either by being similar to, or sharing properties [such as color] with, their objects, or by being physically [continuously or causally] related to their objects, as in Peirce 1932, 2.247–249). In pictorial media, meta-awareness may be strong or effective, but comes about only implicitly, as pictures function as “signs of signs” but never as “signs about signs” (original emphasis, 99). If “signs of signs” is read as “signs about signs,” it is only dependent on the recipient (100) and the metareference is therefore, according to Nöth, “potential” rather than “actual.” These series of oppositions are tied to examples of media products understood in mono-medial terms, that is, “a painting” referred to solely in its pictorial capacity. In Nöth’s traditional schema, media products are associated with certain types of signs, certain types of relations between signs and referents, and certain types of cognitive modes (implicitness, explicitness). This study relies on a more open-ended definition of metareference than those provided by Wolf and Nöth, in that it rejects formal criteria given beforehand and welcomes investigations of actual media products in their historical and intermedial specificity. It therefore retains the highly general definition of metareference outlined at the beginning of this chapter or metareference as a medial and semiotic phenomenon that occurs whenever a media product directs attention to its status as such (i.e., as mediating and representing through medial means). “Directs attention” here includes any types of references, for example, indexical and iconic as well as symbolic (i.e., references built chiefly on conventional relations between a media element and what it is expected to stand for, as in Peirce 1932, 2.247–249). However, metareference is nevertheless determined in advance by being tied to the image’s inherent intermediality. In this way, it is attached to something already “mixed,” to use Mitchell’s term ([2005] 2015), or, more precisely, something that is media specific in virtue of its “mixture.” Besides the obvious combination of pictorial and textual elements in the nineteenth-century pictorial press, part of the image’s inherent intermediality resides in its endlessly complex combination of different types of signs. As amply demonstrated by word and image scholars and semioticians alike, many kinds of symbolic sign functions are intertwined with iconic and indexical ones in image-based media. These range from traditional iconographic subjects such as The Judgement of Paris and The Dance of Death to the mediating character of the sign process itself, where the relation between the pictorial element and its object is translated, picked out by, or gives rise to an interpretant sign in another – possibly conceptual – sign system (cf. Eco 1976, 68; Bal 1991; Mitchell 1994; Heffernan 2006; Jappy 2013, 91–96; Bateman 2014; Louvel 2011, 2018). It may be right that metareference in the end often acquires a “discursive” character, but any such requirement is unnecessarily restrictive and runs the risk of becoming a guideline for searching only those pictorial elements that are easily conceptualized. Additionally, metareference cannot easily be delimited as a product of one isolated pictorial element, in the same way that a term can be identified as a signifying unit on its own. Large depictional

670

S. Petersson

elements such as figures, things, and scenery are of course easily identifiable and labelled, but are still built up from much smaller elements, from pictorial details to dots and strokes – also signs that can take part in generating metareferential attention. Metareference in the plates of the nineteenth-century press is necessarily a product of different sign types and of a combination of pictorial elements, from dots to figures and scenery. One part of the image’s inherent intermediality thus resides in its mix of different types of signs. Another equally important part resides in the image–picture distinction, elaborated, for example, by Mitchell (2005, 84–86; see also Belting 2005; Dahlgren 2018), which can be compared to Elleström’s distinction between basic and qualified media (Elleström 2010, 11–48). If considered chiefly in terms of visual configurations (still or moving), the image is addressed as a basic medium. If considered as pictorialized in a specific culturally, historically, and socially defined media genre (e.g., nineteenth-century journal illustrations or late twentieth-century Polaroid snapshots) along with its culturally and technologically based operational conditions, the image is addressed as an exponent of a qualified medium. The image is mediated in pictorial media. When the image of the Mad Hatter from Carroll’s illustrated novel shows up in Edwin S. Porter’s silent film (1910) or Tim Burton’s more recent adaptation (2010), it is realized in different qualified media products with different material interfaces: printed, “fixed,” pictures on the pages of the book combined with the text of the novel versus the moving images projected on screen combined with captions and live music, or with the speech and sound of the later film. Relations between media include relations between basic and qualified media, image and picture. Lastly, acknowledging the intermedial and media-historical specificity of qualified media determines how to conceive of metareferential implicitness and explicitness. An example can be seen in a detail from Fig. 2, a print showing juxtapositions of topographical views. In the plate’s upper left corner, one of the framing elements in the margin, a net bag, ends with a circle or a “lens.” The net bag/circle/“lens” is both part of the marginal frame and itself a framing device, showing both a slightly close-up view and a “slice” of a scene. Both alternatives are based on representational conventions associated with some types of photographs (e.g., street or natural views but not studio portraiture) rather than the representational conventions exposed in the other scenes of the plate, the juxtaposed panoramic views. Admittedly, the difference in “zoom” between the “lens” in the corner and the other views on the plate is not radical. The real reason why it is of interest exactly as a “lens” is that anyone familiar with Ny Illustrerad Tidning would know that similar “lenses” occur regularly, often with greatly enlarged pictorial details or additional pictorial information (e.g., NIT 1874, no. 1, 5; NIT 1877, no. 24, 189; NIT 1884, no. 35, 299; NIT 1886, no. 9, 71; NIT 1886, no. 31, 261; NIT 1890, no. 18, 161). It is evident from the illustrations in French and British journals of the same type that this is no isolated phenomenon (cf. reproductions in Mainardi 2017). Rather, it was a newly established pictorial convention both based on and representing a certain photographic genre. The circle’s degree of explicitness as a “lens” is hence supported by its capacity for taking other – anterior – pictures as its object. Put differently, an

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

671

Fig. 2 Views from Mörrum. Drawing by L. Ericson, unknown xylographer. Printed in Ny Illustrerad Tidning, 1876, no. 9. 20.8  30.8 cm. (© With the permission of the National Library of Sweden, Stockholm. License: CC BY-NC-ND)

672

S. Petersson

iconic element of the drawing represents pictorial conventions by mimicking a previously known photographic media genre (as distinct from an individual media product), which thus frames the circle structure of the net bag as a “lens,” and its representation as a “slice” of the “world” that continues outside it. The key point is, however, that degrees of explicitness depend on such culturally situated media conventions and intermedial relations to a much larger extent than the almost mechanical effects of (likewise conventionally instituted) sign systems, conforming to the ability to understand explicit, literal, meanings of terms and what Barthes (1977, 42–43) would have called the denotative meaning of pictorial signs.

References to Image and Text Relations The plate in Fig. 2 is a full-page juxtaposition showing different topographical views from the Swedish country town Mörrum. The lower vista of a waterfall shifts into a closer view of the pier and the stream, which extends into the upper landscape with a silhouette of houses and trees that culminates in another riverside scene. In several ways, ranging from the obvious to the oblique, the plate directs attention to how image and text, in their distinctiveness, constantly border upon each other. Part of the obviousness lies in the combined media character of the page. Besides being composed of traditionally juxtaposed elements, such as picture and caption (cf. Benjamin [1936] 2008, 27), the picture includes a curious text that is neither fully an inscription on the pictorial surface nor fully inserted in the pictorial space. Only the middle letters appear as an inscription. The first and last letters are entangled in not one uniform, continuous, space, but in two types of spaces: the one belonging to the frame in the left margin and the other belonging to the framed space of the different views. The first “M” is caught up in the framing net, which extends into the railing at the bottom right. From outside the railing, one of the two bystanders is looking in and thus enforcing the framing effect by suggesting a difference between the space of the frame and the space of the scenes inside the frame. The difference between these spaces is further emphasized by the illusion of depth that is created between the entangled “M” and the pier and the stream behind. This effect escapes the heteroreferential and representational logic that structures the scenes one by one. The net that grows out of the margin and entangles the “M” thus produces the impression of being in between the space of the scenes and the space of the viewer. The “M” is located in the in-between space of the frame, closer to the surface or in front of the pier and waterfall. The space of the frame is not, however, continuous with the inscription. The right side of the plate shows some inconsistencies to the space of the frame. The fisherman at the back has his leg in front of the stick of the net bag, which extends both in front of, and behind, the last two letters. The last “M” is suddenly inside the space of the scene, constituting a supporting crutch for the stick. What goes on in this part of the plate is not only that the play with back and front makes the representation of pictorial space seem strange. It is also that the subtle play with sign

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

673

types echoes the more obvious insertion of a piece of text in pictorial space. The middle part of the letter “M” resembles a crutch in addition to its being part of an inscription of a proper name referring to a geographical place. This does not mean that the same sign is both iconic and symbolic, but that one and the same media element in different ways relates to different objects (crutch and part of letter), which are just temporarily and interchangeably picked out by an interpretant sign (cf. Bateman et al. 2017, 61). The most salient metareferential feature of the plate is the exposure of the text as a text in the middle. Exposure “as a text” means that one is likely to attend to the text as a sequence of letters rather than as solely a piece of information or a word read as a name referring to a place. It differs from ordinary inscriptions, for example, the signature in the right corner, by being partly inserted in two pictorial spaces: the space of the frame and the space of the views. Finally, the text stands out through its relation to the plate’s pictoriality. On the one hand, it is heightened because of the contrasts between textual and pictorial forms. On the other hand, it plays with the contrasts conventionally associated with image and text by its very insertion in space – which is made salient by the unconventional play with discontinuous spaces – and its switch between crutch and letter. Something similar occurs in Fig. 3, which depicts a juxtaposition of two separate scenes: a stereotypical bourgeois interior and, above it, a likewise generic view of a snowy winter landscape with a house and peasant inhabitants. These are two enclosed worlds. No one from either scene looks at or gestures towards the other. As in the Mörrum plate, both scenes sit firmly within the conventions of pictorial realism and are obviously heteroreferential in that the rendering of people, things, and events is immediately recognizable as representations of the external world. At the same time, a curious text breaks with the conventions of pictorial realism. In the foreground of the top scene, the word “Julafton” (“Christmas Eve”) appears in space, suspended in the air. In this case, there is nothing even slightly reminiscent of an inscription. The word is rendered as a material entity within the same world that is inhabited by people and furnished with a cottage, a church, a sledge, a sheaf, and other mundane things. The word’s materiality is suggested by the fact that each letter casts its shadow on the snow, just like the group of people in the middle and the skis that lean against the cottage wall. Each letter is also a mix of iconic and symbolic signs, but in a more pronounced way than in the Mörrum plate. The letters resemble branches and some have joints fastened together by straps, which further underlines the word’s artefact character as a “thing” in the scene, besides being a combination of letters that refer to a day within the Christmas season. The inhabitants of the scene seem to be unaware of the word as no one gestures towards, looks at, or in other ways interacts with it. The significance of the pictorial text is thus suggested as not residing in any intrusion in what is going on in the scene, but exactly as a play with the (basic) media of image and text. Analogously, the candlestick, which extends from the lower to the upper scene and is crowned by “bulbs” of flames, echoes the form of the middle letter of the monogram in the right corner. The iconic relation between the candles and the monogram exposes pictorial and textual signs that border upon each other.

674

S. Petersson

Fig. 3 Christmas Eve. Drawing and xylography by J. F. Höckert. Printed in Ny Illustrerad Tidning, 1865, no. 2. 21.8  35 cm. (© With the permission of the National Library of Sweden, Stockholm. License: CC BY-NC-ND)

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

675

Importantly, the relation between the candlestick in the picture and the letter of the monogram transgresses the line between what is conventionally counted as central to, or “inside,” the picture and one of its parergonal, “outside” elements, such as captions, headings, and frames (“parergonal” as in Derrida 1987, 37–82). This more oblique image and text relation between the candlestick and the monogram is suggested by the more obvious image and text relation of the pictorialized text in the space of the upper scene. In her study of metalepsis in comics, Karin Kukkonen (2011b) has proposed a concept that captures some aspects of how these plates highlight image and text relations: namely, foregrounding. Kukkonen bases the concept on the figure–ground distinction of Gestalt psychology and broadly explains it as “the cognitive process in which one element is perceived against the context of its background” (217). But as a metaleptic device depending on conventions of representation in comics (for the latter, see also Stein and Thon 2015), Kukkonen further demonstrates how foregrounding involves a switch between figure and ground, where that which is normally taken for granted, such as style, outlines, and the gutter, is foregrounded. In the plates here, the “fictive” elements of pictorialized texts foreground pictorial and textual configurations against the background of an expectation that image and text are separate entities that usually occur together in the forms of inscriptions on pictorial surfaces, captions, and headings, or are located within the heteroreferentially depicted space of an image (e.g., a picture of someone reading a book with a title on the cover). As the letters have pictorial qualities and are inserted in the spaces of the scenes, the texts become the unexpected center of attention (i.e., unexpected in the context of pictorial realism). In a larger historical perspective, the best way of understanding these plates – along with the “tail” page in Carroll’s first Alice story ([1865] 1921, 37), where a pun on a long tail and the telling of a tale is typographically rendered by breaking up the lines of words to form a swirling tail – is as an alternative to the schemes of media separation that largely underpin the nineteenth-century discourses on the cognitive and educational functions of pictorial and textual media. Here, it is important to note that critics, authors, educators, and publishers like Henry Blackburn and Philip Gilbert Hamerton actually took the complementarity of image and text as their starting point. Image and text should be juxtaposed in print for better communicative ends. Within this framework, they nonetheless continue the separatist tradition from Lessing ([1766] 2005) or from the paragone debates rather than the ut pictura poesis paradigm (cf. Kennedy and Meek 2018, 1–24). Blackburn’s argument (1896, 15–39) sides with the picture in the sense that it is expected to step in when the written “word fail[s] to communicate the right meaning” (23) as pictorial expressions clarify by concretizing, spatializing, and schematizing as distinct from the sequential character of writing. The logocentrism implied in “the right meaning” (i.e., the anterior word that takes on lengthy and circumstantial expressions in writing and elucidating, concrete and spatial, expressions in pictures) is both part of Blackburn’s larger argument as the ultimate source behind the picture and a bit beside the point of his separatist discourse. From his stated position as a proponent of illustrated literature,

676

S. Petersson

Blackburn’s strategy builds heavily on separating, in order to compare, the affordances and effects of pictorial and textual media. Hamerton (1882, 34–48) places greater emphasis on pictures as a secondary support of words than on the failure of writing. Image and text complement each other since pictures can deal with the “visible,” whereas words can express the inner, “invisible,” structures of any circumstance; pictures lack the textual ability to generalize (“[t]here is nothing in graphic art corresponding to the word ‘animal’ in language”); and pictures cannot “reason” or carry out language’s ability to sequential arguments (39–40). The concretizing, mimetic, and educative function of pictorial aids to texts assists cognition, but they are still no more than aids. To achieve the greatest efficiency, picture and writing should be juxtaposed on the page, but they are still distinctly separated media with their distinctly separate affordances. It is to these theories, building a well-known system of binaries (including text/ image, abstract/concrete, sequential/spatial, symbolic/iconic, meaning/expression), that the plates here offer an alternative story. It should be emphasized that they do not simply reverse the perspective, but offer another, where the crucial point is to direct metareferential attention to image and text as constantly bordering upon each other. “Constantly” should be understood as a process that coincides with the progression from the obvious to the oblique. The obviousness of the pictorialized texts has to do with their place in the center of the plates and with their intrusion in pictorial space as conventionally defined (in line with Alberti’s window [1435] 2011), which leads on to analogously discover new, more oblique, image and text relations: the “M” as switching to a crutch and the candlesticks as starting to resemble a part of a monogram.

References to Contrasting Image Types Besides exposing image and text relations that are synchronic and present in the same image, the Mörrum plate is also involved in a net of diachronic relations that, in the context of its article and media practices in its surrounding culture, refer to ontological differences between different types of images: moving, fixed, virtual, and graphic. Like most pictorial pages of this kind, the Mörrum plate is accompanied by an article that normally contains few, if any, explicit mentions of the illustration. In this case, the plate is mentioned only at the end of the article when reference, in editorial voice, is made to a particular bridge “that is depicted on our plate” (NIT 1876, no. 9, 76). This is a metareference to the plate on the previous page (rather than to the bridge) in the strict explicit sense of Nöth. Nonetheless, it is such a standard feature of the article’s language and genre conventions that it could very well be expected to pass unnoticed. As argued, the idea of explicitness as merely a “mechanical” effect of sign systems disregards that metareferential suggestiveness is also (and often mostly) produced in relation to a surrounding media culture. From this perspective, the ways in which another passage in the article implicitly represents media that mediate moving images have stronger metareferential bearing on the plate. The

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

677

article, aimed at an urban middle-class reader, introduces its topic by referring to the establishment of railroads and their benefits for tourists. To illustrate the latter by textual means, it goes on to describe the experience of changing landscape scenes viewed from the window of a train in motion. Words that relate to vision, such as “the sight of (åsynen af),” “the views expanding themselves (vidga sig utsigterna),” and “little variety for the eye (för ögat föga omvexling),” alternate with descriptions of the dissolving sights: “barely inviting forests and swamps (föga inbjudande skogs- och myrmarker)” that change to “streaming waterfalls (skummande forsar)” and “verdant shores (grönsakande stränder)” (NIT 1876, no. 9, 73; own translation). The media concept “film” is easily used as a metaphor for that which is described in the article: the impression of changing views from the train compartment’s window, or sense data causing an inner “film,” but it is not fully to the point. I am not so much thinking of the anachronism of the film metaphor before Le Cinématographe (1895), since, for example, zoetropes and zoopraxiscopes before film produced the illusion of moving images (for a critique of pre-cinema studies, see Gaudreault and Marion 2018, 201; Kimby 2018). The real reason why the film paradigm should not be taken too literally is that the article’s description of dissolving images is actually closer to two of the most popular and widespread visual spectacles of the later nineteenth century: the dissolving views of the magic lantern, which originated in the seventeenth century, and those of the moving panorama, which originated in the eighteenth century (Huhtamo 2013, 2020; for different types of magic lanterns, with variations of still, dissolving, and moving images, see Mannoni [1995] 2000, 201–319). According to media archaeologist Erkki Huhtamo, moving panoramas ranged from small peep boxes designed for one person to huge formats made for theatres or other buildings that could house an audience. Regardless of size, the basic features of these panoramas consisted of a roll of canvas whose movement was operated by pulling strings, for the small boxes, or by a mechanical or manual cranking system, for the larger types. Rolls of painted canvas exposed changing views of, for instance, landscapes, biblical stories, the seasons, shipwrecks, and reproductions of artworks, and moved across a “window,” either the hole in the peep box or the frame of the stage (Huhtamo 2013, 6, 38–39). The magic lanterns were likewise of different kinds but basically operated by sets of slides on a transparent medium (e.g., glass) that were lit from behind and projected onto a screen. By using two magic lanterns (sometimes with two or even three objectives) projected on the same spot, an effect of dissolving views was created by switching from slide to slide with a manually operated instrument (a “dissolver”). The result was projections of images that changed from, for instance, daybreak to dawn or summer to fall. As Huhtamo and others have noted, both spectacles were promoted in the press and were well-documented through literature. The Swedish novelist Alfhild Agrell describes a travelling panorama show, including rolls of moving images with the typical accompaniment of music and voice, visiting a local fair (Agrell 1884, 61, 64). Another Swedish author, August Strindberg, engages two of his protagonists in a dialogue touching upon image ontology by comparing the projections of the magic lantern apparatus (“skioptikon”) with specters and mirror images (Strindberg [1907] 1995, 210–211). For these reasons, the

678

S. Petersson

article’s description of the dissolving views from the compartment window coincides with, and is thereby openly evocative of, the media practices of the moving panorama and magic lantern shows. In more technical terms, it should be noted that, through its vivid description of dissolving views, the article represents not so much an individual media product as the qualified media genres of moving panoramas and magic lantern shows. “Represents” in this context conforms to Elleström’s diachronic concept of media representation (Elleström 2014, 16–17). The latter implies that what is represented in a “target medium” is both the form and the content of a “source medium,” that is, both the (moving) image and the apparatus, or the “whole” media type. Notwithstanding the differences between the description of the article and actual panorama and magic lantern practices (e.g., that the viewer is in motion in the former and still in the latter), the all-important connection is the spectacularity of moving images on the threshold of what would later become the film paradigm. This is why moving panoramas and magic lantern shows were likely to be recognized from the description of the article, without being explicitly mentioned. When the article is understood as evocative of the moving images of moving panoramas and magic lantern shows and set against the plate, it further plays a part in suggesting a series of contrasts between ontologically different image types, of the sort Mitchell enumerates in his, by now, classical “family tree of images” (Mitchell 1986, 9–10). These images are, first, the ekphrastic ones, including the mise en abyme relation between the first image of the compartment window and the second, “moving” (mental), images of the different views accessible through the first. In Mitchell’s terms, the family of images thus covers “sense data” that cause “mental images,” as distinct from “graphic images” or pictures. As ekphrastic or accessible through text, images are also what he calls “verbal images” or images in text of the kind Liliane Louvel explores in two studies (Louvel 2011, 2018), examining how texts, even without inserted pictures, are “iconotextual” by means of the “visualizing” roles played by typography, ekphrasis, art-historical references, and references to optical devices such as mirrors and silhouettes (for discussions of “iconotextuality” with a starting point in pictorial media, see Wagner 1995, 2015). The first ekphrastic mise en abyme image of the compartment window gives rise to comparisons – still virtual in nature – with the moving images of the panorama and the magic lantern as source media. This, in contrast to the third image, the Mörrum plate adjacent to the article, evokes the fundamental difference between the temporality of media products such as moving panoramas and magic lantern shows (as well as film) and the static, printed, plate offered as an illustration (cf. Newell 2017a, 11 for comments about media comparison as method). This tour through the moving images of moving panoramas and magic lantern shows, and the contrasts between ekphrastic and graphic images as well as between moving and fixed images, inserts the plate into a web of synchronic and diachronic media relations that continues to direct attention to, and finally to thematize, image and text relations. These were introduced by the text “Mörrum” as a prop that provokes further thinking about image and text, thinking that is also tied to the conventions of the pictorial press in that a plate is expected to refer to a piece of

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

679

adjacent text. The article accompanying the plate offers a textual expression of moving images that contrasts with the pictorial configurations of the plate and its fixed state. In its fixed state, the plate nonetheless attempts to circumvent the distinction between moving and fixed. Unlike the Julafton plate, there are no sharp borders between the views. The sky area above the lower panorama extends seamlessly into the water of the middle view, just like the cloudy skyline behind the silhouette of houses extends seamlessly into the water of the top scene. The views dissolve into one another, with no distinct point that signifies where the water ends and the sky begins. These imprecise areas nurture the idea of dissolving images. Again, in a broader historical perspective, both plates in this and the last section exemplify how image and text are repeatedly highlighted in a metareferential fashion in the nineteenth-century pictorial press. On the one hand, this phenomenon was a product of a print media culture that was occupied with the technologies, aesthetics, and educational functions of image and text as realized and juxtaposed in print. Juxtapositions of image and text on the same page had been made easier by the development of more efficient technologies and new methods of pictorial reproduction. Journals like Ny Illustrerad Tidning (and The Illustrated London News [1842–2003] and Le Monde illustré [1857–1956], all examples of the ubiquitous use of the term “illustrated” in newspaper names) as well as critics, graphic artists, and publishers repeatedly referred to their time as an age of illustration (e.g., Crane [1898] 1902, 300; Blackburn 1898, ix). On the other hand, and more interestingly, the plates here in many ways transgress what could be called the conventional concept of illustration, as standardly defined since the nineteenth century. In Blackburn’s words, the first object of an illustration is “to elucidate the text” (Blackburn 1898, 15). In this concrete sense, the picture is tied to a source text and works in its service: elucidating, representing, adorning, or in other ways keeping path with it or a definition that echoes the entry in Oxford English Dictionary (cf. also Hodnett 1982; for more problematizing discussions, see Hillis Miller 1992; Lund 2009; Newell 2017b, 65–71). It is true that, even on the level of lexical definition, the concept of illustration is ambivalent towards the textual bias that is also a part of its history. As “illustration” was defined to represent a source text, it was both tied to the service of that source and expected to “elucidate” it by pictorial means – by adding something that was not there at the start (therefore, the standard lexical definition is supplemental in the sense of Derrida 1976, 141–164). The metareferential plates in the nineteenth-century press nonetheless transgress this idea of illustration: first, because metareference in this context is largely a pictorial phenomenon independent of the explicit journalistic topics and concerns of the accompanying texts, and second, because the plates cannot be reduced to the function of adding to one particular source text, as they are involved in a much wider web of intermedial relations which, together, produce starting points for metareferential attention to, and thinking about, image and text. The Mörrum plate and its accompanying article depend on references to surrounding nineteenthcentury media practices for their representation of moving images, the ways in which images feature in language and in pictorial media, whether in fixed and in moving states. Simultaneously, both plates actualize issues that are of relevance beyond the

680

S. Petersson

media culture during the last part of the nineteenth century, or issues about how image and text interrelate. The plates have the capacity to prompt past and present viewers to search for the image in the text and the text in the image as two mutual sources of meaning. Even this highly abstracted “iconotextual” terminology is reductive, in the same way as the concepts of ekphrasis and illustration, because it does not account for the fact that all examples here are exponents of yet another layer of mediation – a layer that is subsumed under, or sidestepped by, the concepts of image and text or picture and inscription. The next section will unpack how the medium of reproduction for both historical and theoretical reasons constitutes an inherent condition of pictorial media operability and meaning production.

References to Anterior Pictures The plate showing pictures of library interiors (Fig. 1) and the plate with pictures from Ronneby (Fig. 4), another Swedish town, represent a particular recurring metareferential device in the nineteenth-century pictorial press, namely, pictures of pictures. More precisely, these plates both transmediate and represent anterior pictures in a dual process of mediation and representation where the medium of reproduction as a qualified media category plays a significant part. “Picture,” however, misses the point that both plates show something more specific: depictions of drawings of library interiors and depictions of photographs of topographic views, chiefly to be inferred from common experience of drawings and photographs. This is still not enough: the pictures on the journal pages are neither drawings nor (halftone) photographs but xylographic prints (likely the result of photographic transfers of drawings to wood blocks) with a specific xylographic look. The latter is composed of strokes, lines, and dots that model light and shadow, fill up space, and build up “larger” entities such as figures, things, and scenery. In the scenes from the National Library, these marks are especially salient in the middle and bottom rows, where the roofs, floors, and vaults are horizontally “striped” just like the reverse side of the curly edges of the drawings. Such parallel and “mechanical” black lines do not appear on actual drawings or halftones but are distinct traces of xylographic tools and the pictorial convention of the “black line manner” preferred in the popular press. Xylographic marks differ both from the grid of dots of ink in varying sizes with more or less space in between that constitute the surface of halftone photographs and from the sketchy lines of drawings (which they often imitate). These contrasts between the media of the depicted pictures and the medium of reproduction are important because they establish media awareness: the xylographic print depicts photographs and drawings. The argument now is that the plates in relation to the nineteenth-century print media culture were bound up with ideas about the medium of reproduction as a set of technologies that both transferred and transformed an anterior picture. And insofar as these media contrasts direct attention to the medium of reproduction, with xylography as a prime exponent, it implies that

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

681

Fig. 4 Pictures from Ronneby. Drawing and xylography by Hans Evald Hansen. Printed in Ny Illustrerad Tidning, 1886, no. 17. 21.8  25.5 cm. (© With the permission of the National Library of Sweden, Stockholm. License: CC BY-NC-ND)

682

S. Petersson

the dots, strokes, and lines as traces after tools and technologies (i.e., as indexical signs in the strong casual sense) were open to symbolic sign functions. It should be remembered that xylography in the later nineteenth century kept company with a whole range of other, now extinct or obsolete, technologies of pictorial reproduction. In broad terms, xylography had been the predominant means of illustrating journals, novels, and the whole area of “gray” literature such as manuals, menus, leaflets, tickets, bank bills, and the like since the 1830s (cf. Mainardi 2017, 73–118; Gitelman 2014, 21–52; Curtis 2002) and remained in wide use well into the 1890s when photographic technologies began to dominate (Beegan 2008). What this bigger picture omits, however, is that for a long time, many prints were the result of combined technologies (e.g., photographic transfers to manually executed printing blocks) and many technologies that are not retrospectively remembered were in simultaneous use as distinct media in their own right (referred to by proper names and identifiable by their different appearance). In journals like Ny Illustrerad Tidning or The Graphic, different pictorial media alternated within the same issue. Critical writings on pictorial reproduction from the time abound with terms such as xylography, lithography, chromolithography, zincography, photozincography, photogravure process, autotype process, halftone process, calotype process, and photorelief process. These writings on the merits, values, and manner of “translation” involved in pictorial reproduction do more than suggest competition and refashioning “in the name of the real” (as in Bolter and Grusin’s definition of “medium” as “remediation” 1999, 65), or express the media transparency produced by habitation and familiarity (as described in Gitelman 2006, 5–6), even if such aspects are, for sure, often included in the arguments. What is more striking is the far-reaching preoccupation with different types of “translations” that were understood to cause different changes and involving different values. Blackburn, for instance, claimed that photographic reduction, “for certain styles of drawing especially, can hardly be over-estimated” (Blackburn 1896, 77), whereas William James Linton in “Of the Use and Abuse of Photography” set out to critically meet what he considered to be general misunderstandings concerning the merits of photographic reduction, alternation of colors, “overplus of details,” and therefore lack of emphasis and clarity (Linton 1884, 93–102). This was also John Ruskin’s view ([1865, 1866] 1905, 150) when he reproached the “mechanical artifice” (90) not just of photography but of steel engraving, black-line xylography, and the “whole scheme of modern rapid work” (114). Ruskin equated the latter with the mass reproduction that in capitalist society serves to divide attention, leads to “restlessly” demanding new objects of faint attraction, and weakens “the power of attention by endless diversion and division” (140; cf. Jonathan Crary’s [1999] “suspended perception” as a hallmark of modernity). Ruskin’s persistence in equating some types of reproductive technologies with modernity, capitalism, and industrialization is, in less critical terms, part of Joseph Pennell’s (1895) definition of “modern illustration,” as a practice that began with “the application of photography to the illustration of books and papers” or the possibility of photographing drawings onto wood blocks (33). “Modern” illustration, in Pennell’s view, is characterized by the gain of “greater ease of reproduction, greater speed, and greater economy of

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

683

labor” (35), the benefits of reducing or enlarging the drawing and mechanically reversing the copy, and of getting rid of burins and ruling machines (39). In effect, “modern” illustration was defined as the exchange of one mediating technology for another. Whether valuing the halftone processes as a better means of reproducing tone than the manual “graphies,” Pennell still speaks of it as “translation” and breaking up flat tones “into infinitesimal dots, or squares, or lozenges” (45). In the light of such debates, the pattern of “stripes,” most striking on the plates with library pictures, was, to quote Stephen Bann (2013), part of the “visual economy” of the nineteenth-century print media culture. Different media of reproduction were not only largely recognizable by their distinct appearance, but also tied to different values and associations. The crowd of competing and often intermixed (new) media of pictorial reproduction had the capacity to evoke associations with ideas about the distinctly modern transfer and transformation of pictorial media through technological means. Against this background, the top photograph in the Ronneby plate is important precisely because it is smaller than the rest. In the light of the media awareness described, this difference in scale represents the possibilities of enlargement or reduction associated with the media of pictorial reproduction, the latter in the specific sense of an amendment to or distortion of an anterior picture, along with closely associated ideas about mechanization, industrialization, and speed as signs of modernity. One should, therefore, not underestimate how elements often assumed to be “outside” the picture, such as xylographic marks, and oblique elements, like difference in scale, may turn out to become symbolic signs – in this case, of the metareferential sort representing the medium of pictorial reproduction as a qualified media category. It is also significant that the articles to the plates acknowledge no such metareferential potential. The article to the Ronneby plate describes the history, geography, and building constructions of the town and makes no explicit references to the plate on the cover (NIT 1886, no. 17, 138). The short paragraph about the National Library reiterates the news about its move from the royal castle to a new public place and refers to the second-level scenes only to the extent that they illuminate the interiors of the library’s former residence (NIT 1877, no. 52, 422). The contrast between the expressed heteroreferential interests of the articles, which only capture what is represented within the second-level pictures, and the first-level representation of the plates (explicitly showing a set of different pictures) hence reinforces the mise en abyme character of the plates. Both plates are examples of media representation, depicting a pictorial medium (i.e., drawings and photographs) and its representation (interior scenes and topographic sites). It is worth emphasizing that they are also examples of transmediation. One reason for highlighting this perspective in the context of the medium of reproduction is to counter the latter’s ring of media transparency, which is a lesson from the media culture around the plethora of nineteenth-century technologies of pictorial reproduction. Another is that research on transmediation is closely associated with transformations between source and target medium, albeit of a different kind than described here. Much of the scholarship on transmediation has been devoted to studying “story worlds” or “storytelling,” which transforms when

684

S. Petersson

transferred from, for instance, novels to films, to pop-up books, to theater performances, or to various digital channels of distribution (cf. Salmose and Elleström 2020; Freeman and Gambarato 2018; Freeman 2017; and “convergence culture” in Jenkins 2006). In such examples, media transformation is seen as a crucial and critical part of the transmedial process. In the examples here, media transformation is less sweeping; the source media product of a drawing has the same basic medium as the target media product or the xylographic print of the drawing. Nonetheless, following Elleström’s (2014, 16–17) delineation of the process, that which is transmediated, or the drawing (along with the representation in the drawing), is represented again in new types of sensory configurations and in a new interface. The point to xylography and other reproductive media is that besides changes in size, colors, tones, and pictorial details, they result in different types of marks. These changes in sensory configurations may likewise appear in similar interfaces, such as lead or ink on paper. Transmedial research has highlighted the importance of searching for transformations – slight or radical – in each stage in the interwoven processes of mediation and representation. Even slight media transformations might cause (radical) change in meaning. So far, this section has referred to “anterior pictures” in two senses: the anterior picture that indexically precedes any medium of pictorial reproduction and the anterior picture that is supposed as preceding the prints by virtue of iconicity, which here relates to the drawings and photographs that are represented as “whole” media products on the plates. (In the Ronneby case the photographs are confirmed by the caption as actual models, which is unlikely in the library case.) Yet another type of anterior picture is actualized by the ways in which the Ronneby plate transmediates a genre of paintings (Fig. 5). Here, the reference to the anterior picture depends on a visual quotation, meaning that the anterior picture is not represented as an immediately recognizable media product but depends on a narrower knowledge of art history. Like other plates that depict pictures in Ny Illustrerad Tidning, the Ronneby plate has a wooden background against which the photographs are exposed (cf. NIT 1886, no. 20, 176; NIT 1884, no. 28, 235). Such wooden backgrounds are a standard feature of a seventeenth- and eighteenth-century genre of illusionistic paintings of letter boards that are exemplified in Fig. 5. Other standard features include red ribbons that hold letters, almanacs, sheet music, booklets, sketchbooks (in this case with the painter’s signature beneath the landscape scene), broadsheets, and other print paraphernalia. Among various print miscellanies (playing cards, bills, etc.), many such assemblages contain references to the arts hailed in the tradition of Renaissance humanism (poetry, music, painting, philosophy, philology, history, with variations). Cornelius Gijsbrechts (see Fig. 5) is one painter associated with this type of image, but wooden letter boards assembling inscriptions of image, music, and text (as in Barthes’s famous title 1977) can also be found in works by, for example, Edwaert Collier, Samuel van Hoogstraten, and Johan Klopper. In other words, the plate transmediates the background of the paintings by representing it in new sensory configurations and in a new interface. The crucial point is that the object of the transmedial process can also be described in (metareferential) terms of media representation. Given that the Ronneby plate, as a visual

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

685

Fig. 5 Illusionistic letter board. Cornelius Norbertus Gijsbrechts, 1668. Oil on canvas, 123.5  107 cm. (© With the permission of Statens Museum for Kunst, Copenhagen. License: CC BY-NCND)

quotation of a standard feature of a genre of paintings, brings the genre to mind, it cannot escape conveying another similarity between the painting and the print: the fact that they both depict other media. Further, the relation between the Ronneby plate and the recalled painting genre has implications for how the plate is spatially understood. Without the framework of the painted letter boards (or pictures such as Fig. 1), there is actually nothing to support an understanding of the space of the depiction in the Ronneby plate as vertical; no pins, nails, or curtains indicate that the photographs are hanging rather than being laid out on a horizontal support. Only the perspective fetched from the anterior picture supports such a viewing. And more, if the wooden background and the “montage” character of the Ronneby plate evoke this genre of painting, they also evoke a series of contrasts between the visual configurations of source and target media: between the colors of the painting and the

686

S. Petersson

monochrome tones of the print as well as between the continuous brushstrokes of the painting and the lines, strokes, and dots of the print. Intermedial contrasts like these bring about media awareness.

Conclusion: Beyond the Nineteenth-Century Pictorial Press When metareference is grounded in the image’s inherent, simultaneously formal and historical, intermediality, it exceeds ideas of structural correspondence between media, sign types, and cognitive modes. The most important point resulting from this approach is to understand the relative explicitness or implicitness of metareference as the effects of culturally embedded intermedial relations. Put differently, metareferential explicitness owes more to the historical and intermedial specificity of media practices than to the idea of (likewise conventionally instituted) sign systems (whether “visual” or “verbal”) that comply with certain types of references (symbolic as opposed to iconic and indexical) and certain cognitive modes (explicitness or implicitness). This view of the concept of metareference arises from combining the more formal research traditions of intermedial studies – including word and image studies – with media history as a field oriented towards studying media specificity as modeled by the cultural, aesthetic, epistemic, technological (etc.) particularities of a historically defined media landscape. As to the field of research that is yet to come (i.e., a future history of metareferential images), this chapter has demonstrated why metareference in the nineteenth-century press matters now: because it can provide alternatives to past and present historiographically established discourses on media, mediation, and representation. The Mörrum plate and the Julafton plate do not conform to contemporary theories about the affordances and limits of pictorial and textual means of communication, nor to the much larger separatist tradition in the historiography of word and image studies. Rather, both plates explore how image and text constantly interpenetrate and border upon each other. In that sense they constitute counteractive instances of a thought paradigm, but of a completely different kind than the scholarly treatises – underpinned by academic credentials and propositional and terminological means of communication – that are the standard sources to past and present thinking about the image’s medial and representational capabilities. In similar ways, the plates with library pictures and Ronneby pictures direct attention to a media culture that was saturated with new media of pictorial reproduction. They “propose” a cut through the writings on pictorial reproduction that highlights aspects other than the well-known ideal of media transparency. In that sense, they indicate a counter history of media opacity. The important turning point for a future history of metareferential images is to employ media products, like the plates discussed here, as “voices” in the historiography of media, mediation, and representation to make the latter more multifaceted. This amounts to staging a “discussion” between media products and the concepts that have, in various contexts, been used to frame, analyze, and describe them.

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

687

A future history of metareferential images can provide present phenomena under the umbrella of metareference with new, mutually enlightening, analogies. Based on the image’s inherent intermediality, the study of metareference above has examined how webs of media relations – historical and formal – are tied together in the plates. A central concern has been to acknowledge the plates’ references to extinct or obsolete media – moving panoramas, magic lanterns, and various technologies of pictorial reproduction – and to acknowledge their past distinctiveness. The basic idea was to avoid modeling metareference in the nineteenth-century pictorial press on its present manifestations and hence turn it merely into a precursor for what is often addressed as an avant-garde attitude in postmodern art. Notably, this is not the same as enclosing past metareferential images in history. For the future, the promising move is to search for the ways in which past and present metareferential images, within a history not confined to divides such as “popular culture” and “art,” may be used for mutually reframing ends. For instance, how would a juxtaposition of the media opacity of the nineteenth-century age of various media of pictorial reproduction turn out against the present, digitally saturated, image cultures on the web?

References Agrell, Alfhild. 1884. Från land och stad. Stockholm: Seligmann. Alberti, Leon Battista. (1435) 2011. On Painting: A New Translation and Critical Edition. Edited and translated by Rocco Sinisgalli. Cambridge: Cambridge University Press. Alpers, Svetlana. 1983. The Art of Describing: Dutch Art in the Seventeenth Century. Chicago: University of Chicago Press. Bal, Mieke. 1991. Reading “Rembrandt”: Beyond the Word–Image Opposition. Cambridge: Cambridge University Press. Bann, Steven. 2013. Distinguished Images: Prints in the Visual Economy in Nineteenth-Century France. New Haven: Yale University Press. Barthes, Roland. 1977. Image, Music, Text. Essays selected and translated by Stephen Heath. London: Fontana. Batchen, Geoffrey. 2018. Apparitions: Photography and Dissemination. Sydney: Power Publications. Bateman, John A. 2014. Text and Image: A Critical Introduction to the Visual-Verbal Divide. London: Routledge. Bateman, Johan A., Janina Wildfeuer, and Tuomo Hiippala. 2017. Multimodality: Foundations, Research and Analysis: A Problem-Oriented Introduction. Berlin: De Gruyter Mouton. Beegan, Gerry. 2008. The Mass Image: A Social History of Photomechanical Reproduction in Victorian London. Basingstoke: Palgrave Macmillan. Belting, Hans. 2005. Image, Medium, Body: A New Approach to Iconology. Critical Inquiry 31, no. 2 (Winter): 302–319. Benjamin, Walter. (1936) 2008. The Work of Art in the Age of Its Technological Reproducibility: Second Version. In The Work of Art in the Age of Its Technological Reproducibility, and Other Writings on Media, edited by Michael W. Jennings, Brigid Doherty, and Thomas Y. Levin, translated by Edmund Jephcott, Rodney Livingstone, Howard Eiland, et al., 19–55. Cambridge: Belknap Press of Harvard University Press. Blackburn, Henry. 1896. The Art of Illustration. London: W. H. Allen & Co. Bolter, Jay David, and Richard Grusin. 1999. Remediation: Understanding New Media. Cambridge: MIT Press.

688

S. Petersson

Bruhn, Jørgen. 2017. The Intermediality of Narrative Literature: Medialities Matter. London: Palgrave Macmillan. Bruhn, Jørgen, and Beate Schirrmacher, eds. 2021. Intermedial Studies: An Introduction to Meaning Across Media. London: Routledge. Carroll, Lewis. (1865) 1921. Alice’s Adventures in Wonderland and Through the Looking Glass. New York: The Macmillan Company. Crane, Walter. (1898) 1902. The Bases of Design. London: George Bell and Sons. Crary, Jonathan. 1999. Suspensions of Perception: Attention, Spectacle, and Modern Culture. Cambridge: MIT Press. Curtis, Gerard. 2002. Visual Words: Art and the Material Book in Victorian England. Aldershot: Ashgate. Dahlgren, Anna. 2018. Travelling Images: Looking Across the Borderlands of Art, Media and Visual Culture. Manchester: Manchester University Press. Derrida, Jacques. 1976. Of Grammatology. Translated by Gayatri Chakravorty Spivak. Johns Hopkins University Press. ———. 1987. The Truth in Painting. Translated by Geoff Bennington and Ian McLeod. Chicago: University of Chicago Press. Eco, Umberto. 1976. A Theory of Semiotics. Bloomington: Indiana University Press. Eilittä, Leena, Liliane Louvel, and Sabine Kim. 2012. Intermedial Arts: Disrupting, Remembering and Transforming Media. Newcastle: Cambridge Scholars. Elleström, Lars. 2010. The Modalities of Media: A Model for Understanding Intermedial Relations. In Media Borders, Multimodality and Intermediality, ed. Lars Elleström, 11–48. Basingstoke: Palgrave Macmillan. ———. 2014. Media Transformation: The Transfer of Media Characteristics Among Media. Basingstoke: Palgrave Macmillan. ———, ed. 2021a. Beyond Media Borders: Intermedial Relations Among Multimodal Media. Vol. 2. Cham: Springer International Publishing. ———. 2021b. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In Beyond Media Borders, Volume 1: Intermedial Relations among Multimodal Media, ed. Lars Elleström, 3–91. Cham: Springer International Publishing. Freeman, Matthew. 2017. Historicising Transmedia Storytelling: Early Twentieth-Century Transmedia Story Worlds. New York: Routledge. Freeman, Matthew, and Renira Rampazzo Gambarato, eds. 2018. The Routledge Companion to Transmedia Studies. New York: Routledge. Fricke, Harald. 2011. Pop-Culture in History: Metalepsis and Metareference in German and Italian Music Theatre. In Metalepsis in Popular Culture, ed. Karin Kukkonen and Sonja Klimek, 252–267. Berlin: Walter de Gruyter & Co. Gaudreault, André, and Philippe Marion. 2018. The Double-Birth Model Tested Against Photography. In Photography and Other Media in the Nineteenth Century, ed. Nicoletta Leonardi and Simone Natale, 191–204. University Park: The Pennsylvania State University Press. Gitelman, Lisa. 2006. Always Already New: Media, History and the Data of Culture. Cambridge: MIT Press. ———. 2014. Paper Knowledge: Toward a Media History of Documents. Durham: Duke University Press. Glaser, Stephanie A., ed. 2009. Media inter Media: Essays in Honor of Claus Clüver. Amsterdam: Rodopi. Grishakova, Marina. 2010. Intermedial Metarepresentations. In Intermediality and Storytelling, ed. Marina Grishakova and Marie-Laure Ryan, 312–331. New York: De Gruyter. Hamerton, Philip Gilbert. 1882. The Graphic Arts: A Treatise on the Varieties of Drawing, Painting, and Engraving in Comparison with Each Other and with Nature. Boston: Roberts Brothers. Heffernan, James A.W. 2006. Cultivating Picturacy: Visual Art and Verbal Interventions. Waco: Baylor University Press. Herzogenrath, Bernd, ed. 2012. Travels in intermedia[lity]: Reblurring the Boundaries. Hanover: Dartmouth College Press. Hillis Miller, Joseph. 1992. Illustration. London: Reaktion Books.

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

689

Hodnett, Edward. 1982. Image and Text: Studies in the Illustration of English Literature. London: Scholar Press. Huhtamo, Erkki. 2013. Illusions in Motion: Media Archaeology of the Moving Panorama and Related Spectacles. Cambridge: The MIT Press. ———. 2018. Elephans Photographicus: Media Archaeology and the History of Photography. In Photography and Other Media in the Nineteenth Century, ed. Nicoletta Leonardi and Simone Natale, 14–33. University Park: The Pennsylvania State University Press. ———. 2020. The White Behind the Picture: Toward a Media Archaeology of the Screen. In A Million Pictures: Magic Lantern Slides in the History of earning, ed. Sarah Dellmann and Frank Kessler, 277–304. New Barnet: John Libbey. Huhtamo, Erkki, and Jussi Parikka. 2011. Introduction: An Archaeology of Media Archaeology. In Media Archaeology: Approaches, Applications, and Implications, ed. Erkki Huhtamo and Jussi Parikka, 1–21. Berkeley: University of California Press. Jappy, Tony. 2013. Introduction to Peircean Visual Semiotics. London: Bloomsbury. Jenkins, Henry. 2006. Convergence Culture: Where Old and New Media Collide. New York: New York University Press. Kennedy, David, and Richard Meek, eds. 2018. Introduction: From Paragone to Encounter. In Ekphrastic Encounters: New Interdisciplinary Essays on Literature and the Visual Arts, 1–24. Manchester: Manchester University Press. Kimby, Tim. 2018. Photography, Cinema, and Perceptual Realism in the Nineteenth Century. In Photography and Other Media in the Nineteenth Century, ed. Nicoletta Leonardi and Simone Natale, 176–190. University Park: The Pennsylvania State University Press. Klimek, Sonja. 2011. Metalepsis in Fantasy Fiction. In Metalepsis in Popular Culture, ed. Karin Kukkonen and Sonja Klimek, 22–40. Berlin: Walter de Gruyter & Co. Korda, Andrea. 2015. Printing and Painting the News in Victorian London: The Graphic and Social Realism, 1869–1891. Farnham: Ashgate. Kukkonen, Karin. 2011a. Metalepsis in Popular Culture: An Introduction. In Metalepsis in Popular Culture, ed. Karin Kukkonen and Sonja Klimek, 1–21. Berlin: Walter de Gruyter & Co. ———. 2011b. Metalepsis in Comics and Graphic Novels. In Metalepsis in Popular Culture, ed. Karin Kukkonen and Sonja Klimek, 213–231. Berlin: Walter de Gruyter & Co. Lagerroth, Ulla-Britta, Hans Lund, and Erik Hedling, eds. 1997. Interart Poetics: Essays on the Interrelations of the Arts and Media. Amsterdam: Rodopi. Lessing, Gotthold Ephraim. (1766) 2005. Laocoon: An Essay upon the Limits of Painting and Poetry. Translated by Ellen Frothingham. Mineola: Dover Publications. Linton, William James. 1884. Wood-Engraving: A Manual of Instruction. London: George Bell & Sons. Louvel, Liliane. 2011. Poetics of the Iconotext. Edited by Karen Jacobs, translated by Laurence Petit. Burlington: Ashgate. ———. 2018. The Pictorial Third: An Essay into Intermedial Criticism. Edited and translated by Angeliki Tseti. New York: Routledge. Lund, Hans. 2009. Karen Blixen’s ‘Stork Story’ and the Notion of Illustration. In Media Inter Media: Essays in Honor of Claus Clüver, ed. Stephanie A. Glaser, 369–384. Amsterdam: Rodopi. Lutas, Liviu. 2021. Metalepsis in Different Media. In Beyond Media Borders, Volume 2: Intermedial Relations among Multimodal Media, ed. Lars Elleström, 197–216. Cham: Springer International Publishing. Lyons, James, and John Plunkett. 2007. Multimedia Histories: From the Magic Lantern to the Internet. Exeter: University of Exeter Press. Mainardi, Patricia. 2017. Another World: Nineteenth-Century Illustrated Print Culture. New Haven: Yale University Press. Mannoni, Laurent. (1995) 2000. The Great Art of Light and Shadow: Archaeology of the Cinema. Translated and edited by Richard Crangle. Exeter: University of Exeter Press. Mitchell, W.J.T. 1986. Iconology: Image, Text, Ideology. Chicago: University of Chicago Press. ———. 1994. Picture Theory: Essays on Verbal and Visual Representation. Chicago: University of Chicago Press.

690

S. Petersson

———. 2005. What do Pictures Want? The Lives and Loves of Images. Chicago: Chicago University Press. Mitchell, W. J. T. (2005) 2015. There are No Visual Media. In Image Science: Iconology, Visual Culture, and Media Aesthetics, 125–135. Chicago: University of Chicago Press. Mitchell, W.J.T., and Mark B.N. Hansen. 2010. Introduction. In Critical Terms for Media Studies, ed. W.J.T. Mitchell and Mark B.N. Hansen, vi–xxii. Chicago: University of Chicago Press. Newell, Kate. 2017a. Adaptation and Illustration: A Cross-Disciplinary Approach. In The Oxford Handbook of Adaptation Studies, ed. Thomas Leitch, 1–20. New York: Oxford University Press. ———. 2017b. Expanding Adaptation Networks: From Illustration to Novelization. London: Palgrave Macmillan. Nöth, Winfried. 2007a. Self-Reference in the Media: The Semiotic Framework. In Self-Reference in the Media, ed. Winfried Nöth and Nina Bishara, 3–30. Berlin: Mouton de Gruyter. ———. 2007b. Metapictures and Self-Referential Pictures. In Self-Reference in the Media, ed. Winfried Nöth and Nina Bishara, 61–78. Berlin: Mouton de Gruyter. ———. 2009. Metareference from a Semiotic Perspective. In Metareference Across Media: Theory and Case Studies, ed. Werner Wolf and in collaboration with Katharina Bantleon and Jeff Thoss, 89–120. Amsterdam: Rodopi. Ny illustrerad tidning, 1865–1900. Orfila, Jorgelina. 2015. On Art History and Meta-Images: Art Reproductions, Site Photographs, and Cézanne’s Art. In The Imaginary: Word and Image, ed. Claus Clüver, Matthijs Engelberts, and Véronique Plesch, 301–315. Leiden: Brill Rodopi. Parikka, Jussi. 2012. What is media archaeology? Cambridge: Polity. Peirce, Charles Sanders. 1932. Elements of Logic. In Collected Papers of Charles Sanders Peirce, ed. Charles Hartshorne and Paul Weiss, vol. 2, 1931–1958. Cambridge: Belknap Press of Harvard University Press. Pennell, Joseph. 1895. Modern Illustration. London: George Bell & Sons. Rajewsky, Irina O. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. Intermediality: History and Theory of the Arts, Literature and Technologies/ Intermédialités: Histoire et théorie des arts, des lettres et des techniques, no. 6 (Automne): 43–64. ———. 2010. Border Talks: The Problematic Status of Media Borders in the Current Debate about Intermediality. In Media Borders, Multimodality and Intermediality, ed. Lars Elleström, 51–68. Basingstoke: Palgrave Macmillan. Rippl, Gabriele, ed. 2015. Handbook of Intermediality: Literature – Image – Sound – Music. Berlin: De Gruyter Mouton. Ruskin, John. (1865, 1866) 1905. The Cestus of Aglaia and The Queen of the Air with Other Papers and Lectures on Art and Literature 1860–1870. Edited by Edward Tyas Cook and Alexander Wedderburn. Vol. XIX of The Works of John Ruskin, edited by Edward Tyas Cook and Alexander Wedderburn. London: George Allen, 1903–1912. Salmose, Niklas, and Lars Elleström, eds. 2020. Transmediations: Communication Across Media Borders. New York: Routledge. Stein, Daniel, and Jan-Noël Thon, eds. 2015. From Comic Strips to Graphic Novels: Contributions to the Theory and History of Graphic Narrative. 2nd ed. Berlin: De Gruyter. Stoichita, Victor I. (1993) 1997. The Self-Aware Image: An Insight into Early Modern MetaPainting. Translated by Anne-Marie Glasheen. Cambridge: Cambridge University Press. Strindberg, August. (1907) 1995. Svarta fanor: Sedeskildringar från sekelskiftet. Edited and annotated by Rune Helleday. Stockholm: Norstedt. Thoss, Jeff. 2015. When Storyworlds Collide: Metalepsis in Popular Fiction, Film and Comics. Leiden: Brill Rodopi. Wagner, Peter. 1995. Reading Iconotexts: From Swift to the French Revolution. London: Reaktion Books. ———. 2015. The Nineteenth-Century Illustrated Novel. In Handbook of Intermediality: Literature – Image – Sound – Music, ed. Gabriele Rippl, 378–400. Berlin: De Gruyter Mouton. Wolf, Werner. 1999. The Musicalization of Fiction: A Study in the Theory and History of Intermediality. Amsterdam: Rodopi.

27

Metareference in the Nineteenth-Century Pictorial Press and Beyond

691

———. 2005. Metalepsis as a Transgeneric and Transmedial Phenomenon: A Case Study of the Possibilities of ‘Exporting’ Narratological Concepts. In Narratology Beyond Literary Criticism: Mediality, Disciplinarity, ed. Jan Christoph Meister in collaboration with Tom Kindt and Wilhelm Schernus, 83–107. Belin: Walter de Gruyter. ———. 2009. Metareference across Media: The Concept, its Transmedial Potentials and Problems, Main Forms and Functions. In Metareference Across Media: Theory and Case Studies, ed. Werner Wolf in collaboration with Katharina Bantleon and Jeff Thoss, 1–85. Amsterdam: Rodopi. ———. 2011. Is There a Metareferential Turn, and If So, How Can It Be Explained? In The Metareferential Turn in Contemporary Arts and Media: Forms, Functions, Attempts at Explanation, ed. Werner Wolf, 1–47. Amsterdam: Rodopi. Yousif, Keri. 2012. Balzac, Grandville, and the Rise of Book Illustration. Burlington: Ashgate.

Picturing Music in the Nineteenth Century

28

Ba´lint Veres

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Systematic Aesthetics and Media Hierarchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Absolute Music: Absolute Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Danhauser’s Picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Musical Paintings in the Eighteenth and Nineteenth Century . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Aspirations: Back to Danhauser’s Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

694 697 700 703 705 712 715 718

Abstract

Although music has been conceived as an art of pure emotions and devoid of concepts, it has always gained its significance in intermedial contexts. The sense and impact of music are inevitably articulated through a medial transfer since music contains much more than its sonic aspect. Intermediality, however, that occurs in the interplay of music and its surrounding verbal discourses, remains routine-like or largely unnoticed within institutionalized forms such as music criticism, music history, and the like. By contrast, other intermedial effects of music become highly pronounced when the interplay of musical and linguistic medium is enriched by the involvement of other media, primarily visual ones. If, for example, music becomes the object of the depiction in a painting, we won’t necessarily approach the visual work in the same way as if the depiction was a mere commentary on a preexisting music phenomenon. “You’ll never look at music the same way again!” – as the slogan put it, back in the early days of Music Television. Although music videos provide an ample repository of visually hijacked, exploited, or intensified musical experiences, their medial dynamics B. Veres (*) Moholy-Nagy University of Art & Design, Budapest, Hungary e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_33

693

694

B. Veres

cannot be held as novelties, at least as far as cultural modernity is concerned. This chapter searches for much earlier examples of the audiovisual experience. At the same time, it explores its aesthetic and historical preconditions and the workings of intermediality concerning several eighteenth- and nineteenth-century musical pictures. Keywords

Early modern aesthetics · Intermediality in romanticism · Musical pictures · Audio-vision · Medial transfer

Introduction The primary purpose of this chapter is to provide an insight into the prehistory of aesthetic discourse and the early modern practice of intermediality as it unfolded at the intersection of music and the arts. To achieve this goal, the present procedure unfurls in a two-way elucidation, explicated not only in a strictly consecutive manner but also through alternations from section to section: (1) On the first level, this study presents and interprets a well-known nineteenth-century Biedermeier painting depicting the significant musical hero of the era, Franz Liszt, gleaming in the aura of Romanticism. Romanticism is understood here as a manifestation of a general and all-encompassing worldview that has not sunk into the depths of history in the least, but is still around even today and defines the ways we experience temporal and spatial relations, social habits, and pursuits of existential meaning (Doorman 2004; Millán Brusslan 2020; Taylor 2021; Veres 2021a). In order to attain a deeper understanding of the chosen work of art, it shall be compared to several other musical pictures of the age. Some of them come from an earlier, some from a later period, but each feature a combination of pictorial expression with musical allusion, attaining it in various ways. On the second level (2), this chapter surveys the development of aesthetic ideas that emerged in the early nineteenth-century discourse, which provided incentives for artists to reframe conventional art forms. They not only recognized the potential of artistic abstraction of the form in these ideas but also gained impetus for experimenting with intermedial creation in the sense of emphatically referencing, combining, or transforming diverse medial factors (Rajewsky 2010) and their related modalities (Elleström 2010, 11–48). The central example of this study, Joseph Danhauser’s painting (1804–1845) Franz Liszt, am Flügel phantasierend (1840) (Fig. 1), is shown as an instance of intermedial experimentation in nineteenthcentury visual art. The comparison of this image with other paintings demonstrates that the intermedial ambitions that can be discovered in Danhauser’s picture are far from being an isolated example. Next, an image analysis and research in the history of ideas follow that presuppose each other in close interrelation. This chapter contains sections of theory and analyses that interpret each other and highlight how audiovisual dimensions interact

28

Picturing Music in the Nineteenth Century

695

Fig. 1 Joseph Danhauser: Franz Liszt, am Flügel phantasierend (1840). (Alte Nationalgalerie, Berlin. © Wikimedia Commons)

with each other within a single work, and how they merge in aesthetic perception, as the German terminus technicus of the day says it: in a Totaleindruck. In the case of Danhauser’s picture, the functioning of intermediality is not completely dissimilar to the audiovisual operation of a film, although this “frozen frame” taken out from an imaginary sound movie – if we playfully allow ourselves to regard it as such – fails to follow the established conventions of multimodal merging or media combination as Rajewsky calls it (2010, 55); instead, it offers flexibility for various interpretations. As far as the relationship between music and image is concerned, it is advisable to note from the outset that this relation is almost always complemented by a third, “bridging” dimension, that of language. As it happens in the present case as well: a painting about music is discussed in conceptual terms. Although the linguistic constructions that aim to grasp music gladly use visual materials (illustration, documentation, visualization, etc.), these materials are only seldom interpreted as constituents in the process in which musical meaning or musical experience is created. It is not, however, as if the nature of the musical meaning has ever been definitively clarified (Sheinberg and Dougherty 2020). On the contrary, although music, following the Hegelian heritage (Hegel 1975, 87–88; 888 ff.), is often conceived as an art of pure emotions and devoid of concepts, its significance, not to mention its cultural prestige, always prevails in intermedial contexts (Arvidson

696

B. Veres

2016). The impact of music on life and imagination is exerted from various sociocultural perspectives and is defined by discursive-linguistic conditions (Clayton et al. 2012). Thus, more often than not, the sense and impact of music manifest through a medial transfer or, more precisely, through media clusters, as Ludwig Pfeiffer labeled them (2002, 23), since music contains much more than its sonic aspect. For instance, Richard Leppert showed this too in his research (1993) which demonstrates that the visual spectacle of music performance has become part of the social meaning of music. In full agreement with this view, Danhauser’s painting on Liszt, in fact, exemplifies the working of both latent audiovisual medial transposition and medial referencing, types of which are described and compared to many other types of media border crossing in the synthesizing studies of Lars Elleström, Irina Rajewsky, Werner Wolf, Jerrold Levinson, and others (Elleström 2021, 1–92; Rajewsky 2010; Wolf 1999, 35–50; Levinson 2011). Seeing, however, from the perspective of cultural anthropology, intermediality when it emerges through media clusters is the rule in the long run, not the exception (Pfeiffer 2002, xvii and xxiii; Mitchell 2007). The operation of the intermedial dynamics between music and discourse remains routine-like or largely unnoticed when it happens within institutionalized forms (such as in music criticism, music theory, music history, etc.). By contrast, most often than not, intermedial effects of music become highly pronounced when the interplay of the musical and linguistic medium is enriched by the involvement of other media. If, for example, music becomes the object of depiction in a painting, we won’t necessarily approach the visual work in the same way as if the representation was a mere commentary on a preexisting and intact music phenomenon. We have good reasons to think that we do not look at the painting depicting music as a mere silent spectacle. Instead, we assume that it refers to and is somewhat mixed with the musical experience recalled or imagined by the audience (cf. Kramer 2002). At the same time, we may also acknowledge that the painting can become an inherent factor regarding how we sense music and the formation of its cultural meaning. The depiction does not only imitate music in one way or another but also expands its expression (Rajwesky 2010, 58) – just like a music video that irreversibly defines how a song gets entrenched in the collective experience and memory (cf. Chion 1994, 21; Vernalis 2004, xiv.). Thus, the central thesis of this chapter is that meaning and the experience of the artwork produced within a given medial condition depend largely on the cultural terrain defined by other arts and also on the general aesthetic thinking with regard to artistic mediums (Dayan 2011). In the present case, it depends on the extent to which music, painting, and poetry can or want to navigate as autonomous, “monomedial” entities within the artistic terrain, or whether they favor rather asserting themselves via direct intermedial relations.1 The concluding section of this chapter attempts to return to these issues using the insights gained from the analyses.

Ludwig Pfeiffer relies on Dewey’s aesthetics, noting that “the question looming and lingering in Dewey’s aesthetics [. . .] is [the] one of culturally and therefore also personally significant and attractive media configurations” (2002, xviii).

1

28

Picturing Music in the Nineteenth Century

697

Systematic Aesthetics and Media Hierarchy It is commonly known that systematic philosophical reflection on various art forms, discussions on the general values of art, and on the peculiarities of each branch are a development of the eighteenth century, aided mostly by German aesthetes like Moses Mendelssohn (1997), Lessing (1853), and Kant (2007). In addition to the ontological, epistemological, ethical, and social aspects that were considered at the time concerning the various art forms, the emergent aesthetic thinking touches on issues that specifically came to the fore recently, in the age of new media: these are the questions of medialities in arts. In this regard, Lessing’s Laocoon, originally published in 1767, proved to be the most influential source, although he only discusses the connections between fine arts and poetry putting them into a rudimentary context of medial reflection. Nevertheless, his theoretical plans also included conceptual discernments of music and poetry, dance and music, and literary works of art (cf. Shaw-Miller 2002, 5). It is not a rushed statement to make: had Lessing been able to fulfill his original plan, the model of the Laocoon, that locates art forms in a bipolar way as temporal on the one hand and spatial on the other, would have fared differently in posterity. It is, however, precisely this bipolar model that has had an extraordinary impact on modern art theory – its last major developer being Clement Greenberg through his 1940 Towards a Newer Laocoon – although in recent decades, the model has suffered criticisms that seem irreversible (Sternberg 1999). It is also worth referring to the deconstructive analysis by W.J.T. Mitchell (1986), in which he unraveled political, religious, cultural, national, and gender aspirations hidden in Lessing’s division of the arts. In addition, it is also advisable to emphasize what theorists such as Samuel Weber (2004, 3; 367), Mitchell (2007), and many others have also expressed when they warned that terms such as “television” or “visual culture” are reductive and narrowing as they hide the actual audiovisual and multimodal nature of most cultural products and their reception. “Visual culture,” “material culture,” and other related concepts implicitly affirm the bipolar model inherited from Lessing. On the one hand, there is a sublime, imaginative, eloquent literary field of mental creativity in his theory, and on the other, a seductive, beautiful but wordless visuality-materiality-corporeality that cannot substantially add to the progress of civilization (Mitchell 1986, 112. ff.). Creating an opposition between poetry and art is, of course, unfair and, moreover, is permeated by a kind of overstretched aesthetic hygiene (Han 2018), which has to do presumably with the personal hypochondriac tendencies of influential theoreticians, such as Kant (1996, 318–319). As Jerrold Levinson (2011, 30), an early initiator of the discourse on hybrid art forms emphasizes: two things are immediately evident. One, the notion of a thoroughbred art form is logically secondary to that of a hybrid, being in fact the notion of a nonhybrid. Two, the ordinary categorization of an art form as thoroughbred or nonhybrid will usually be a relative or limited one, not positing an absolute purity reaching back to the dawn of Western art.

698

B. Veres

Ludwig Pfeiffer takes the same view: “Media tend to show up most often in the form of at least implicit or hidden combinations: in ‘intermediality,’ hybridization, in McLuhan’s sense” (2002, xix). Concerning the late eighteenth century, what theoretical criticism today rightly characterizes as simplification, essentialism, and excessive reductionism proved to be, in its time, a courageous act against a baroque culture that showed extraordinary hybridity. For Lessing and also for his main dialogue partner Moses Mendelssohn, the fundamental goal of aesthetics follows a “program” to place every possible art form into a comprehensive system (Kristeller 1951/1952) and define their general and specific essentials (cf. Veres 2021b). Mendelssohn’s 1757 study develops this program on an even broader level compared to Lessing, featuring architecture, music, dance, and sculpture besides painting and poetry, and even hybrid art forms. Although this Aufklärist need to pursue taxonomies reveals the spirit of early modernity, it is also intertwined with another requirement, which turns Lessing’s concept into a genuine modern program and renders it a significant source for the Romantic era as well. It prioritizes the audiences’ response over creation and the experience of comprehending art over the objecthood of the artworks.2 Lessing’s basic observations on the medial aspects of artworks come precisely from the priority of the act of reception: a spatial context is revealed at the hic et nunc moment of viewing the painting, while temporal connections emerge in the process of reading literature. Of course, it is not that he would take the position of radical hermeneutics and attribute the aesthetic experience solely to the act of reception. Lessing interconnects the duality of the point-like and the time-evolving experiences of reception by exploring the specific regularities of spatial and temporal art forms. However, the analysis of artistic creation only confirms the validity of the recipient’s aesthetic experience: a true artist seizes the “fertile moment” because, on the beholder’s side, this is the essence of experiencing a painting. At the same time, a great poet wisely chooses a narrative strategy over descriptive representation because the reader grabs the contents of literary works primarily in the form of a time experience. Lessing, similarly to Mendelssohn, considers the material carriers of the various art forms and the sign systems manifested in them as decisive constituents of the branches of art. In a much-quoted passage, he derives the semiotic differences of the branches of art from the diversity in their spatial and temporal aspects, claiming that “painting and poetry, in their imitations, make use of entirely different media of expression” (Lessing 1853, 101). As a result, due to the medial conditions, different branches of art are not able to represent even the same themes in the same way, and similar aesthetic effects are achieved in very different ways. Adorno repeats this idea in the twentieth century, essentially unchanged: 2

This preference for experiencing artworks is eminently present in Johann Joachim Winckelmann (1972) as well, another defining thinker of modern aesthetics. Winckelmann was concerned, however, less with issues of medialities in arts, thus – along with Hegel, Schopenhauer, and others – he contributed indirectly to the fact that the medial conditions of the arts were brought to light only much later, in the twentieth century.

28

Picturing Music in the Nineteenth Century

699

The different arts may aim at the same subject, but they become different because of the manner in which they mean it. Their substantial content lies in the relation between the what and the how. They become art by virtue of this substantial content. But this needs the how, their particular language; if it went in search of something larger, beyond the particular form of art itself, it would be destroyed. (Adorno 2003, 377)

What Adorno emphasizes here can easily be traced back to Lessing, as the latter following the consequences of the distinctions made between the mode of operations in painting and poetry places each art form under the categories of either (1) natural sign systems or (2) artificial sign systems. The former is meant that representation occurs in the same sensual dimension as what is represented, while the latter represents the material world in a more spiritual, intangible way. The former category includes the fine arts (Mendelssohn includes music, architecture, and dance as well), while the latter category is occupied only by literature. The latent hierarchy that prevails in the system cannot be questioned. It is perhaps unnecessary, however, to speak of latency concerning a text in which it reads, “The poet is as far above the painter, as life is above the painting” (Lessing 1853, 92). As Oscar Kristeller pointed out in his seminal study (1951/1952), there has always been an archetype or a paradigm hidden behind the general attempts to systematize the branches of art since the early eighteenth century, and every element of the system have been related and accorded to this paradigm. Lessing and Mendelssohn are definitely not alone in the sense that literature is at the heart of their system. The tradition of the sectorial conception of the arts has remained the basis of aesthetic thinking to this day, and the entire system of art institutions in modernity has been influenced by this early mediological and aesthetic understanding. Artistic practices, however, have lastingly not confirmed this sectorial thinking, or the presupposition that the central position of literature would be unshakable. Mitchell (1986, 98) writes: the tendency of artists to breach the supposed boundaries between temporal and spatial arts is not a marginal or exceptional practice, but a fundamental impulse in both the theory and practice of the arts, one which is not confined to any particular genre or period.

This stance can be expanded over the modalities of medial articulations (Elleström 2010, 11–50) and the totality of our experiential spectrum, underlining what Steven Connor (2005, 158) sums up, referring to Michel Serres’ ideas: the effort to separate the senses out, displaying them adjacent to each other, like countries on a map, plan or table of correspondences, will be gently and repeatedly precluded by the requirement to knot them together. It will emerge that each sense is in fact a nodal cluster, a clump, confection or bouquet of all the other senses, a mingling of the modalities of mingling.

From this point of view, the main novelty of Romanticism, as it is embodied by figures such as Goya, Friedrich, Liszt, and Wagner, was not the abolition of the systematization of the arts itself but that the central position of literature got

700

B. Veres

successfully challenged by other art forms: by painting and even more so by music. In the last third of the nineteenth century, Walter Pater (1980, 105) thus articulated the epoch-making directive: “all art constantly aspires towards the condition of music.” This directive indicates a significant change compared to the beginnings of systematic aesthetics, in which poetry served as a model for the arts and every creative practice, and now passes on this role to music (Dahlhaus 1989). What happened in less than a century? What led to literature falling out of its central role and music becoming a measure of the arts? And what role did the other elements of the system play in this process? Especially painting, which was also aspiring for the role of being the paradigm from the sixteenth century onwards (cf. da Vinci 2012) by absolutizing the principle of ut pictura poesis (Lee 1967). These questions may come to the fore when taking a closer look at Danhauser’s picture of Liszt and the bouquet of eighteenth and nineteenth century musical images that are to be compared. But before diving in, we shall venture into the opposite direction, the thoroughbred art forms, or put it in a more archaic way: the absolute art forms.

Absolute Music: Absolute Art “Absolute” painting, as a concept, is usually associated with the avant-garde artists of the early twentieth century, like Kandinsky, Mondrian, Malevich, and others (Golding 2000). In parallel, film historians discuss the artistic experiments of the 1920s (works by Ruttmann, Richter, Eggeling, etc.) as the birth of the “absolute film” (Elder 2007). Both terms, more or less consciously, transpose one of the central ideas of nineteenth-century aesthetic thinking into their territory: obviously, the source in both cases is the idea of “absolute music” (cf. Westerdale 2010). Although the latter term was first used – with condemning intent – by Richard Wagner in the middle of the nineteenth century (Dahlhaus 1989, 18–41), its historical development dates back to the end of the eighteenth century (Bowie 2002; Veres 2012). It exerted an invaluable influence on general aesthetic thinking culminating in formulas such as the one of Walter Pater: “all art constantly aspires towards the condition of music” (1980, 105). In his monograph on the idea of absolute music, one of the pioneering scholars in the field, Carl Dahlhaus (1989) convincingly shows how the aesthetic revolution took place with regard to purely instrumental music that freed itself from liturgical, representational, and other social functions, and also from the service of poetry, as well as how this emerging aesthetic paradigm was able to progressively connect to general philosophies of art, while encouraged real musical practice too. Very similar findings emerge from the studies conducted by the eminent Hungarian scholar, László F. Földényi on early Romantic painting (1993) and the art historian, Joseph Leo Koerner on Caspar David Friedrich (2009). In terms of art history, painters like Friedrich (1774–1840), William Turner (1775–1851), Philipp Otto Runge (1777–1810), Carl Gustav Carus (1789–1869), and Adrian Ludwig Richter (1803–1884) are usually not discussed as initiators of absolute painting but are classified as examples of Romantic landscape painting. Although Kandinsky was not a trained art historian, his view on the issue still seems significant, even

28

Picturing Music in the Nineteenth Century

701

decisive: “Friedrich – the beginning of abstract art.” (Guillaud 1984, 19). Consistent with this view is Földényi’s inventive interpretation, which, additionally shows a tint of a transmedial influence from the Romantic philosophy of music: “Friedrich’s or Turner’s ‘cloud paintings’ do not depict the ‘real’ clouds of the ‘real’ world, just as Beethoven’s late string quartets do not depict different mental states or emotions.” (Földényi 2010, 118; own translation). This is important because it is hardly possible to find a more perfect manifestation of the spiritualized conception of purely instrumental music, which Ludwig Tieck hailed as early as 1799 than Beethoven’s late string quartets. As Tieck characterized Beethoven’s works, “they reveal the greatest mysteries in mysterious language, they do not depend on rules of probability, they do not need to ally themselves to narrative or character, they abide in a purely poetic world” (cited in Dahlhaus 1991, 67). Here, the idea of abstract or nonobjective art comes into play, which at the same time is by no means devoid of content. On the contrary, in the absence of any given objective content, the piece of art provides a general position of subjectivity by which it can embrace potentially everything assuming that everything is permeated by a higher order unity, which ultimately eliminates the distinction between subject and object. The discourse of early Romantic music aesthetics is a mutation of the discourse of theology in many ways. Tieck puts it this way: “For music is certainly the ultimate mystery of faith, the mystique, the completely revealed religion” (Dahlhaus 1989, 82). The harmony between the philosophical reflections of the instrumental repertoire of the age and Schleiermacher’s ideas on religious sensitivity (1996) is quite well known to researchers (Seidel 2003; Stoltzfus 2006). However, it is worth pointing to a parallel with the roughly simultaneous ideas at this point regarding painting. Carl Gustav Carus, for example, reports a strange visual experience in one of his letters. Standing on a hilltop, looking down at the landscape below, as in the famous painting of Friedrich’s wanderer, Carus articulates the spiritual ambition of a modern individual from the depth of his soul in an elemental way, which emerges from the opposition between bourgeois consciousness and natural landscape. He writes: “you lose yourself in boundless space, your whole being experiences a gentle elevation and purification, your very self vanishes away, you are nothing and God is everything” (Harrison et al. 1999, 104). The metaphysically tuned concept of the landscape thus indicates a position of subjectivity that can be taken by both the historically and topologically contingent observer and the “all-seeing” Creator. These two options do not exclude, but presuppose each other, since the landscape functions as a medium that transcends perceptible reality. Carus is not talking about a painting but about the landscape itself; however, Simmel rightly emphasizes: “Whenever we really do see a landscape, over and above an aggregate of separate natural objects, then we have a work of art in statu nascendi” (2020, 223). Therefore, in this sense, the landscape is a genuine image, regardless of whether it uses a technical medium or not. It is never merely the order of things beautifully arranged for the eyes but, at the same time, a product of sight, determined culturally and driven by spiritual ambitions. The Romantic generation admires the natural landscape with reverence precisely because it is far from being natural. As a consequence, the landscape which is emancipated from reality can

702

B. Veres

become an independent visual projection devoid of the task of representation. In Friedrich’s paintings, one can no longer see a certain landscape but a sensual visualization of the concept of landscape. As Schelling puts it: “landscape itself possesses reality only in the eye of the observer” (1989, 144). The common feature of the landscape (painting) and purely instrumental music is that they capture the feeling directly instead of depicting concrete things, so their essence is not primarily representation but rather expression; above all, expressing the feeling of infinity. The reverberations between the aesthetic discourses of music and landscape become utterly explicit when Richter declares that a truly great landscape picture is in fact a “deeply captivating music of color tones.” (Földényi 2010, 61). Ludwig Tieck writes in his Franz Sternbalds Wanderungen: It was evening; a beautiful heaven was aglow with its wonderful, brightly colored cloudscapes above them. “Look,” Rudolph continued, “if you painters could present me something like that, I would often pardon you for your moving histories, your passionate and confused representations with all their countless figures. My soul should delight and content itself with these bright colors without connection, these airy images gilt with gold; I would gladly surrender plot, passion, composition, and all [. . .] Oh, my friend, if you could only capture in your paintings this wonderful music which the heavens are today composing! But you have not the proper colors, and meaning in the conventional sense is the condition of your art. (Pape and Burwick 1995, 7)

The way Tieck daydreams of painting strongly resembles the way that the simultaneous Romantic music criticism spoke of the wonderful originality and stirring influence of the symphonies (Dahlhaus 1988, 98–111), which transcends rationality, and which became a model of even poetic work (Dahlhaus 1989, 144–145). Nonfunctional and nondiscursive music that is art freed from representation and figurativeness – and poetry that is increasingly athematic and without references, of course, assess their nature not as deficient but rather as progressive. Runge, Friedrich, and others see arabesque-like, nonobjective art as an “altarpiece,” that is religious although no longer tied to the church or to the traditional religious iconographic tradition. Art can thus become a breed of (non-ecclesiastical) theology, and what is more, the only possible theology. Devotion is more complete and more boundless the less it is defined by any particular object or theme. (Földényi 2010, 124; own translation)

The same (pseudo)theology appears in E.T.A. Hoffmann’s famous 1810 review of the instrumental music of Beethoven, where he extemporizes “the realm of the infinite” and the “endless longing” towards it (1917, 128). Longing for and dissolving into the realm of infinity is a recurring aspect, which characterizes Romantic landscape painting that transcends the naive concepts of visibility but moans with a sense of unhappiness that stems from an insatiable desire to gain vision about what is ultimately invisible, as the “really real” is obscured by sight. Critical voices at the turn of the eighteenth and nineteenth centuries and onwards against both the phenomena of purely instrumental music (Bonds 2006, 6–10) and metaphysical landscape painting (Sumowski 1970, 30–31) were uttered by no less

28

Picturing Music in the Nineteenth Century

703

authorities than Kant, Hegel, and Goethe, who attributed a spiritual emptiness to these art forms. In the meantime, however, the emergence of the idea of absolute art in all creative fields went hand in hand with the progressing predominance of hermeneutic activity. This was realized by the fact that the audience compensated for the decontextualized experience of the absolute artwork – an experience detached from all acknowledged social and functional goals – by constantly repeating the individually executable act of re-contextualization. In this process, the medialities of the various art forms also have themselves become plastic: the absolute painting that liberates vision, the absolute symphony that liberates hearing, and the absolute poem that liberates linguistic meaning give the audience a degree of freedom in which monomedial perception of the work is not guaranteed. Indeed, a plurimedial or intermedial contextualization is available anytime. This contextualization happens, for instance, when a story gets projected into a symphony, when a phonetic poem is set on the horizon of musical experience, or when kinetic imagination is triggered by the painting. Examples of this remediation in the reception process are countless. From this, it can be concluded that, although the idea of absolute art seemed to serve the essentialization of the branches of art and the consolidation of their medial conditions and borders, it did in fact set a dynamic in motion in the opposite direction. The act of spiritualization involved the fading of their materials and also the fading of conventions in their social perception. For the audience, all this resulted in the rise of the individual aesthetic experience. On the creative side, it offered the possibility of reconfiguring artistic media that had become free of constraints in the meantime. One can risk saying, Joseph Danhauser’s painting is an attempt at such a reconfiguration.

Danhauser’s Picture Now let’s take a closer look at Danhauser’s painting itself. The work was not made on canvas, but on a 119  167 cm first-class wooden board provided by the customer, piano builder, and art collector, Conrad Graf (Wythe 1984). Obviously, both artistic and promotional aspects loom behind the choice. The original title of the painting, later forgotten, is Ein Weihemoment Liszts (“The Moment of Liszt’s Consecration”). Most probably, the background to this quasi-sacral title is the fact that Liszt offered the total revenue of his six Vienna matinee concerts in November and December 1839 for the erection of the Beethoven monument in Bonn (cf. Comini 2008, 207). But the original title may also have suggested, of course, a more spiritual meaning: it testified about the fact that Liszt was a worthy continuation of Beethoven’s work who gave him as a child a legendary paternal kiss (Walker 2005, 1–10). What does the picture represent? At first glance: it depicts one of Liszt’s home concerts, more precisely a private musical event akin to the Schubertiades. The execution of the painting is of high quality, but it does not seem to go beyond the Austro-German Romantic-Biedermeier genre painting of the era, of which

704

B. Veres

Danhauser is a noted figure. The painting was immediately well received and many elements of the composition were detailed in an article published in the May 13 issue of the Wiener Zeitung (Kregor 2010, 112. ff). The image was not forgotten later and belongs to the core of Liszt’s iconography. Although it was rarely discussed in art history, it is well known and often quoted in music culture. These initial pieces of information can be perfectly satisfying for a fleeting glance, but in terms of the issues of (inter)mediality indicated above, one cannot be satisfied with a quick look. We have to go back again and again to the various details of the painting in order to eventually learn to immerse ourselves in the work – precisely as dictated by the various hidden guidelines which have been encoded by art philosophy in it. Thus, taking a second glance, one can identify a scene that depicts a somewhat peculiar salon that testifies to a sophisticated taste as well as a bohemian way of life, belonging to the world of the bourgeoisie, rather than the upper circles. A company of seven people crowds together in this room. Compared to its actual dimensions, the density and proximity of the people cry out for an explanation. Especially since the members of this company get intimately close to each other: one figure rests his hand on the other’s arm and another figure hugs his neighbor’s shoulder. The intimacy emanating from the gestures is reminiscent of an everyday group picture where, for example, a family stands as a model in front of the painter’s canvas. However, this is certainly not the case here. The picture falls between a genre scene and a group painting because it does not offer a frontal depiction of the figs. A lady member of the company is pictured with her back to the viewer, seated on a pillow placed on the floor. What’s more, the composition of the group does not fit the rules of stringent and hierarchical social imagery: the posture of each figure, their position as standing or sitting, does not reflect any ordered social structure. Their presence seems informal, their gestures spontaneous. They seem nonchalant or absent-minded in their relations as if they did not care about the codes of social contact. On the other hand, they are all very attentive, floating in reverie. What brings them together if not each other’s presence? What focuses their attention if not their communication? We could hastily answer by saying, it is the sound of music that is the center, as the figure in the center of the picture, Franz Liszt is certainly playing some fantastic music as we see him still in full youth and masculinity. But it is precisely from the exploration of the intermedial aspects of the work that we will learn that this answer would be a hasty one. Whatever actually happens in the room makes the audience socially unstructured but aesthetically and mentally all the more concentrated. By all means, the act of merely listening to music would not explain in itself why we see the six-person audience physically so close to the artist and each other. Three of them are standing and listening to the performance. The abovementioned lady is opting for the uncomfortable floor instead of a convenient seating presumably available in the room but further away from the piano. Wouldn’t it be more suitable for everyone to listen to the performance from a comfortable armchair? It would certainly be if the event taking place in the salon was a mere auditory experience. This is not the case, however. Their sublime listening experience goes along with a visual experience as well, which the painter expresses with the unmistakable indication that he leaves the

28

Picturing Music in the Nineteenth Century

705

eyes of all the characters open, and depicts them with a focused, almost staring look. This behavior is opposed to the idealized mode of listening to purely instrumental music with eyes lowered and immersed inward, as it has been defined by the advanced cultural role models of the era (cf. Dahlhaus 1989, 81). In contrast, Liszt’s audience gets close to something that may not only be audible but also visible, and perhaps even tactile, inhalable, and tastable too. The personal charisma of Franz Liszt, which has already been analyzed in much detail (Leppert 2002; Kramer 2002; Hamilton 2008, 82 ff., 225–254), obviously explains much of their behavior, but not everything. Some do not pay attention to Liszt at all; their attention wanders to somewhere else. What are they immersed in actually? And what is Liszt’s relationship to the visual realm in this image? These questions are only introductory questions to this picture of Danhauser. If we can give answers to them, we can get closer to the content that is not obvious at first glance: to content that the characters themselves are striving to get close to. By this painting, Danhauser in no way confines himself to creating a mere genre painting or portrait but cherishes explicit art philosophical ambitions, for which the genres at hand merely serve as starting points.

Musical Paintings in the Eighteenth and Nineteenth Century Musician and his audience – is a popular theme of the mid-nineteenth century, culminating particularly in Liszt’s extensive iconography. Here, however, it is about more than just a portrait of an artist or a genre painting on a musical event. We realize how obvious this fact is when we compare Danhauser’s painting with a few other examples of the genre, all of which may raise the question: what does the beholder “hear” in a musical picture – understood as a realization of either intermedial reference or medial transposition (Rajewsky 2010)? Not as if every eighteenth and nineteenth century musical picture would raise the above question; most of them do not try to represent music at all, only the musician who appears alone or accompanied by his fellow musicians. The presence of the audience in such portraits of musicians becomes necessary only when the music, rising to the rank of absolute art, offers the audience a specific, new kind of task in which listeners also become, so to speak, co-creators who elaborate their musical imagination through processing ingenious musical forms. Let’s briefly review the development traced through these pictorial representations. First, it is worth having a closer look at two images created before Danhauser’s, then at a couple of later paintings. A permanent reference is the famous 1746 portrait of Elias Gottlob Haussmann (1695–1774), a court painter from Dresden. It is a portrait of Johann Sebastian Bach, 61 years old at the time (Fig. 2). The picture is both ordinary and surprising. It is ordinary in the sense that it depicts a common man with moderate facial features, dressed in an attire that confirms his social status, and showcases important belongings of his profession. Through all this, the depicted person expects recognition from the learned and noble circles. The surprising motive here is that Bach appears in the picture without the symbols that would justify his fame in his own time: his

706

B. Veres

Fig. 2 Elias Gottlob Haussmann: Portrait of Johann Sebastian Bach (1746). (Bach-Museum, Leipzig. © Wikimedia Commons)

outstanding skills as a violinist and organist. The cantor from Leipzig can be seen unadorned, against a dark background, in a simple dress, although in a dignified ricepowdered wig. He faces the beholder of the picture while holding a single music sheet in his right hand. No instrument appears, and there is no obvious reference to the ecclesiastical office of the depicted person or an allusion to his merits achieved in the court. The most dynamic moment of the composition is the sheet music that is composed onto the central axis, counterpointing the master’s illuminated face. Bach holds the music sheet as an invitation to read and sing along. “Canon triplex à 6 Voci” – says the title which is followed only by three lines. At this point, the experiences of the beholders are varying: (1) most of them see only the three lines of music, acknowledging that the person depicted refers to his profession with an obvious gesture. (2) Meanwhile, the few initiated understand the call hidden in the title and recognize that a whole piece of music is hidden in the three lines. It is a kind of music that assumes the highest ability for abstraction and could be experienced instantly by a six-person company of visitors having the willingness to sing the score. It is clear that in Haussmann’s painting, Bach presents himself contrary to contemporary popular belief, primarily not as an organist, violinist, or cantor, but as a composer with a capital C (cf. Wolff 2000). However, he indicates something else too for the initiates: he presents himself as a composer who is a master of musical combinatorics sourced from the eternal order of the cosmos instead of external virtuosity, fashionable sound painting (Tonmalerei), or excessive representation of emotional states. Thus, he shows a different and unusual image of himself compared

28

Picturing Music in the Nineteenth Century

707

to other images of him depicted by his contemporaries. As a matter of fact, Haussmann’s image differs from the emotional, pathos-driven, or sometimes nationalistic Bach images of later generations too. Johann Nepomuk della Croce (1736–1819) signs a painting depicting the Mozart family around 1780 (Fig. 3). The painting strikes a delicate balance between the genres of family portrait and musician portrait. It is important to note that the painting was made not long after the death of the mother, Anna Maria. Her absence, from which all three members of the family suffered greatly, is somewhat compensated by the life-size portrait on the wall behind Nannerl, Wolfgang, and Leopold. The members of the family pose with an instrument in their hands: Nannerl and Wolfgang – looking in a disciplined way at the viewer of the picture – prepare for a piece composed for four hands in a way reminiscent of the times when they performed as virtuoso child prodigies. Now, no longer children, in the current state of their career, this gesture can even be considered anachronistic. Meanwhile, Leopold, in his black overcoat and violin in his hand, seeks to exercise control over the pictorial representation of his children and any questions the observer may have. “We are musicians, but above all a family,” his gaze says. And it is precisely this duality that is reinforced by two elements of the background: the vivid gaze of the mother, who looks at the viewer from the portrait as if she were only watching and supervising events from behind a window; and next to her, in a less exposed way, a small sculpture depicting Apollo playing the lyre (Beck 2017). Anna Maria’s dominance over the muse accurately reflects the intentions of the painting: here we are dealing with an idealized image of a family in which their social activity plays only a secondary role, and the artistic individuality of the young Amadeus has no role at all. It is worth comparing della Croce’s family portrait with Danhauser’s painting in two respects. First, it is not clear in either of the pictures whether music is in the focus or something else: maybe the idea of a family, the heroic artist, or the vision of something sublime. Second, the picture within the picture plays an equally

Fig. 3 Johann Nepomuk della Croce: Portrait of the Mozart family (1780). (Mozarteum Stiftung, Salzburg. © Wikimedia Commons)

708

B. Veres

important role in della Croce’s as well as in Danhauser’s work. The latter painting features, a painted portrait on the wall behind the company, a small sculpture on the left, and most notably the brilliant white marble bust of Beethoven on the right, resting on Liszt’s piano.3 The significance of these is immense, and we shall return to this later. Now, let’s proceed to other musical paintings that were created after Danhauser’s work and are markedly different from the ones preceding it. Teofil Kwiatkowski’s (1809–1891) painting, Polonaise de Chopin – un bal à l’Hôtel Lambert à Paris (1859) (Fig. 4) goes a long way to boldly mix fantasy and reality. It depicts events that took place decades earlier, sometime in the 1840s, in the Great Hall of the Hôtel Lambert, the symbolic center of Polish emigration in Paris. At first glance, Kwiatkowski, a member of the Polish emigrants’ community, assigns a subordinate role in his painting to the great composer, who sits as a small figure almost in the shade at his piano at the bottom right side of the picture. He is presenting his new polonaise composed for the Polish ball at the Hôtel Lambert, but most of the crowd depicted in the painting obviously does not pay attention to his music. The tension created this way, however, is quickly resolved by the painter. The figures behind Chopin are merged into the ball’s multitude, and the crowd does not seem to be a congregation of flesh and blood beings, but rather a spiritual apparition. While activating his decades-old memories of Chopin, through this image, Kwiatkowski expresses a philosophical position on art: before the musician’s eyes, as if an enigmatic historical tableaux vivant came to life, a cavalcade of humans and demigods in an unreal space reminiscent of the distant past. Instead of the score placed on

Fig. 4 Teofil Kwiatkowski: Polonaise de Chopin – un bal à l’Hôtel Lambert à Paris (1859). (National Museum in Poznań. © Wikimedia Commons)

3

Some analysts believe that it shows the Beethoven bust of 1821 by Anton Dietrich (1799–1872), while Alessandra Comini claims it is Danhauser’s own work, as he had the possibility to make Beethoven’s death mask and made several busts from it (Comini 2008, 208).

28

Picturing Music in the Nineteenth Century

709

the holder, the pianist apparently lingers into the vision in a somewhat similar way to Caspar David Friedrich’s wanderer, who is enthralled by his own spiritual projection (cf. Koerner 2009, 210–244). Kwiatkowski’s painting differs from Friedrich’s metaphysical landscape painting in that, instead of nature, it is history, a human vortex of events that plays the medium of infinity to appear. Moritz von Schwind’s (1804–1871) A Schubert Evening at Josef von Spaun’s (1868) (Fig. 5) is not as poetic as Kwiatkowski’s Chopin tableaux, yet it cannot be considered either to be realistic or directly documentary in value. Rather, it is an expression of nostalgia for an “Alt Wien” that had long passed by the time of the picture’s creation and actually never existed in this form (Gibbs 1997, 52 ff.). In the picture, an unlikely large multitude gathers around Schubert and his tiny instrument, making the musician almost lost in the sea of heads. Clearly, the magnetic power of listening to music is exploited here by the artist as a means to represent the culturedriven social bond of the bourgeoisie. Incidentally, this image evokes memories of his own youth and friendship with Schubert. Von Schwind, the former illustrator of Schubert’s sheet music, captures the community experience of the Schubertiads from a half-century perspective, and his drawing features many figures of the Viennese artistic and intellectual life of the time (Gingerich 2014, 70 ff.). Julius Schmid (1854–1935), an Austrian academic artist, is noted for working in the same genre, in a similar retrospective manner. An important difference, however, is that the author of the oil painting titled Ein Schubertabend in einem Wiener Bürgerhause (1897) (Fig. 6) cannot rely on personal memories. Similarly to von Schwind, Schmid turns towards the Schubertiad as a mythical memory and an archetypal vision of his own bourgeois cultural sphere. Schmid’s oeuvre by now has been completely forgotten. Only the large Schubertiad tableau continues to live a

Fig. 5 Moritz von Schwind: Schubertiade (1868). (Wien Museum, Wien. © Wikimedia Commons)

710

B. Veres

Fig. 6 Julius Schmid: Ein Schubertabend in einem Wiener Bürgerhause (1897). (Wien Museum, Wien. © Wikimedia Commons)

life of its own abounding in copies and reproductions after having won the first prize in 1897 at the Künstlerhaus competition. It was announced by a Vienesse magnate, Nikolaus Dumba, on the occasion of celebrating the centenary of Schubert’s birth. Compared to the two previous images discussed above, the great novelty of Schmid’s painting and the main means of creating a powerful atmosphere lies in the unusually intense interaction between the artist and his audience. Schmid depicts Schubert playing the piano, not in an ecstatic and transcendent state, while watched by an audience immersed in meditation, but rather conveys an intense two-way communication. Instead of an individualized and somewhat distanced reception of art, the civic audience here is characterized by an extremely intimate sense of togetherness. Their festive attire shows respect for the rank of the art, but by crowding in the small salon, their unsophisticated behavior indicates that they are attending an event that leads out of the realms of social formalities and navigates in a direction where artists and connoisseurs, men and women can draw as equal partners from the radiation of the spirit. The sharp light projected on the figures in the middle displays this utopia in a very effective way. Schmid not only emphasizes the individuality of Schubert’s traits but also individualizes each member of the audience. Art historians, meanwhile, have successfully identified many of Schubert’s friends among them, including Moritz von Schwind (Wasserman 2003). Through detailed research, Schmid sought to reconstruct the Viennese milieu of the 1820s, the real circumstances of the Schubertiades, and the facial features of the participants. However, realistic demands cannot hide the real ambitions of the image and the

28

Picturing Music in the Nineteenth Century

711

reason for its immediate success: the bourgeoisie of a declining empire can look back with nostalgia on the former potential of art in society and the cultural emancipation of the Bürgertum that has meanwhile become anachronistic (cf. Bürger 1984, 35–54). Henryk Siemiradzki’s (1843–1902) 1887 painting titled Chopin Playing the Piano in the Salon of Radziwili (Fig. 7) is much less phantasmagoric than Kwiatkowski’s work on a similar subject. As for the composition, this is the most comparable to Danhauser’s painting among those reviewed above, and it is from this comparison that the particular “intermedial” concept of the Austrian Biedermeier painter can best be reconstructed. Siemiradzki, known primarily for his large biblical tableaux, celebrates at least as much the sophistication of the prince and the highclass audience in this painting as the presence of Chopin himself, who gives reason to refined aesthetic attention with his play. The artist and his audience are equal partners, although the former’s firm presence (Chopin plays with a straight face and by heart) is in contrast with the latter’s internal dynamics: the gentlemen standing at the back door on the right have apparently not yet matured enough for the deep appreciation of music, while the lady closest to the viewer, sitting half-back, may be focused too much on the artist instead of on the work of art. In any case, the reactions of the company of 15 are much more diverse and diffuse than the gestures of the audience in Danhauser’s painting. In Siemiradzki’s painting, whatever variety of aesthetic behaviors we may register, the loftiness of the aesthetic experience of the prince sitting in an armchair next to the piano, literally “enlightened,” cannot be questioned. He embraces and appreciates music to a degree that perhaps portrays him to be superior even to the artist who produces the sounds. As in the paintings of

Fig. 7 Henryk Siemiradzki: Chopin Playing the Piano in the Salon of Radziwili (1887). (Private collection. © Wikimedia Commons)

712

B. Veres

della Croce, Danhauser, and von Schwind, here too the walls of the salon are decorated with paintings, but – as opposed to the other three – these portraits on the wall are irrelevant to the meaning of the picture. The spatial situation of the musical event is also worth comparing: in von Schwind’s picture, the crowd listens to music behind closed doors having the air of utmost intimacy. In Danhauser’s pictorial space behind the half-open doorway, mysterious darkness lingers. In Siemiradzki’s painting, the doorway is logical and insignificant: it leads towards other, perhaps no less valuable places of social life.

Intermedial Aspirations: Back to Danhauser’s Work In contrast to the visible and invisible outer areas of Siemiradzki’s work that have no real importance, the outdoors in Danhauser’s painting are crucially important regarding the general meaning of the work. The sight that might seem to be a cloudy sunset at first, visible through the window, proves to be a par excellence Romantic imagery of the sublime in a prolonged contemplation. This vista of a pristine natural landscape could have been hardly glimpsed from a Parisian salon in 1840 (Comini 2008, 207) whose milieu might be suspected as the location of the event.4 Rather, this sight outside the window seems to be an artwork, more precisely a metaphysical landscape painting of the kind that has been described earlier in connection to Friedrich, Turner, and others. This renders the image into some kind of a phantasm. And it is indeed a phantasm, not only because of the outdoors visible through the window but also because of the members of the audience. After all, the audience is made up of not just anyone. Liszt is backed by two other not less renowned musicians, Gioachino Rossini who puts his arms around Niccolò Paganini. Further, a triangle of three major figures of Romantic literature appears, George Sand, Alexandre Dumas the elder, and Victor Hugo, all three gesturing that books should be kept closed when the sound of music fills the soul. Finally, a woman, most likely countess Marie d’Agoult, Liszt’s partner and mother of his three children, completes the company, not showing her face. She is the only one of whom the painter had not been able to acquire a portrait beforehand. For, it is hard to imagine that these celebrities were really taking the time to pose for the painter. The whole picture is a montage born of contemporary depictions, containing artistic reproductions (images-within-images) as well. On the left side, on a commode, we can discover a statuette of Joan of Arc (a copy of Marie d’Orleans’ 1837 work), an important reference to Romantic historical sensitivity. In the middle, there is a copy of George Henry Harlow’s prevalent portrait of Lord Byron, the ultimate Romantic. And on the right, most importantly, a gigantesque

4

Paris as the supposed location of the painting was indeed the defining scene of Liszt’s life in the 1830s, but this location is in a somewhat contradicting relation to the aforementioned 1839 Vienna concerts series, which provides the direct cause of the work’s creation.

28

Picturing Music in the Nineteenth Century

713

bust of Beethoven, in a marble-like, Graecized style, the original of which was produced by the painter himself. The bust is placed on the piano whose top is covered with a bunch of scores, including Liszt’s own Fantasia and the third movement of Beethoven’s Piano Sonata in A-flat Major, op. 26.5 The persons appearing in the space depicted in the image are thus much more than a simple audience. They are representatives of the branches of the arts, and they bring the relations of the art forms to the stage. They represent hierarchies between the various artistic and receptive capacities (music as surpassing poetry; spiritual enlightenment as surpassing sensual enthrallment). Besides music and poetry, even history enters the stage, while visuality itself seems to be forgotten. How does the painter position his own art form when staging relations and hierarchies with the other arts? On the one hand, the visual rendering seems impassive while capturing a company that is heated by an artistic rapture. On the other hand, it gives the license to the viewers – both inside and outside of the pictorial space – to get lost in the idea of infinity and metaphysical enchantment through the Friedrichesque landscape that pops up in the window, appearing almost as a fresco, guarded in a sanctuary, which aspect is also emphasized by the monumental curtains. And it is at this point when the question becomes the most acute: whether the oversized bust of Beethoven in the exoteric center of the sanctuary, which opens up via Liszt, is the ultimate value of the picture or the esoteric reference to the Romantic landscape even goes beyond it? Emma Siegmund, an eyewitness to an 1841 concert of Liszt, recorded in her memoir the following: “There is nothing terrestrial in the playing of this artist; I thought I saw open up before me the whole of infinite space” (Leppert 2002, 205). Through Danhauser’s picture, one can raise the question of whether this infinite space is aural or visible, or rather something inaccessible in direct sensory perception. It can be any of these, for as Friedrich Schlegel claims, “The consciousness of the infinite in the individual is the feeling of the sublime. This is present completely unrefined in the individual. And this feeling of the sublime is enthusiasm” (SchulteSasse 1997, 244). In Romanticism, the notion of infinity which evokes a majestic feeling and leads to enthusiasm can be evoked by art, above all by music. To what extent these effects depend on the medial and aesthetic conditions of any given art, however, was not discussed in the first place. A few decades later, these aspects come to the foreground in Walter Pater’s motto. Before him, Schopenhauer thus summed up the privilege music possesses over the other art forms: [Music] stands quite apart from all the other [arts]. In it we do not recognize the copy, the repetition, of any Idea of the inner nature of the world. Yet it is such a great and exceedingly fine art, its effect on man’s innermost nature is so powerful, and it is so completely and profoundly understood by him in his innermost being as an entirely universal language, whose directness surpasses even that of the world of perception itself. (Schopenhauer 1969, 256)

5

Regarding the music Liszt is supposedly playing for the company, cf. Kovács (2014, 122).

714

B. Veres

Danhauser’s picture seems to be a confirmation as well as a contesting of these ideas about music at the same time. While it attempts to capture the sublime essence of music, it also utilizes several painterly methods radical at the time. (1) Following public pictorial representations and portraits (Comini 211 ff.), it composes a collage of prominent artistic personalities of the era who were indeed acquainted with each other but perhaps had never met in the assembly as shown in the picture, and certainly not in 1840, due to various personal reasons and health conditions. Thus, (2) it is a fictitious event that unfolds at a fictitious venue, reminiscent of the Schubertiades, which was a recurring topic of musical images of the time. (3) It is also an idealized pictorial representation of those Parisian Liszt events of the 1830s one of which was recorded by Ernest Legouvé in 1886, as follows (Hamilton 2008, 84): I had invited a few friends around one evening: Liszt, Goubaux, Schoelcher, Sue, and half a dozen others. Berlioz was one of us. “Liszt,” he said, “why not play us a Beethoven Sonata.” From my study we passed into the salon. . . . There were no lights, and the fire in the grate had burned very low. Goubaux brought the lamp from my study, while Liszt went to the piano and the rest of us sought seats. “Turn up the wick,” I told Goubaux. . . . But instead he turned it down, plunging us into blackness, or, rather, into full shadow; and this sudden transition from light to dark, coming together with the first notes of the piano, had a moving effect on every one of us. . . . Whether by chance or by some unconscious influence, Liszt began the funereal and heart-rending adagio of the Sonata in C♯ Minor. The rest of us remained rooted to the spot where we happened to be, no one attempting to move. . . . I had dropped into an armchair, and above my head heard stifled sobs and moans. It was Berlioz.

In sharp contrast to the dark scenery of this memory, Danhauser’s picture allows subjects to be visually contemplated, including the genre of images-within-images. Byron, Jean d’Arc, and most of all Beethoven raise this “phantasy potpourri” (Comini 2008, 207) onto a higher level manifesting the human essence through art. But which of the arts is the most eligible for this rank? The picture can be interpreted both as the exaltation of music and of visual art, as the painting makes an allusion to both Liszt’s playing and the Romantic landscape. However, there is a third option, according to which, music and visual art together in a union can reach the utmost heights. Instead of using a conventional genre painting or a Romantic landscape, it is an “intellectual landscape” involving self-reflection with regard to both music and visual art. In connection to the Liszt phenomenon, Leppert claims that it brings with it “a projection of the self [that] is cast into and onto the sight and sound of the performer” (2002, 222). At the same time, the metaphysical landscape is also a projection of the self, as Földényi and others demonstrated. Ultimately, in Danhauser’s painting, everyone projects themselves: the ladies and gentlemen, the performer and his audience, but also the painter and the spectator as well. The individual projects himself/herself into something or someone else: into the image of the artist who is perceived as a divine being or into nature seen as a divine manifestation. The erudite beholder can easily associate the reminiscent sounds of Beethoven’s Moonlight Sonata or a fierce passage of Liszt’s piano music with the image; and vice versa, while playing the piano, Liszt and his audience may have a

28

Picturing Music in the Nineteenth Century

715

vision of Beethoven or that of Nature with a capital N through its representation by the landscape. But what could the purpose of having these allusions, projections, and phantasms be? In addition to the obvious prestige-oriented aspects of the customer, the composer’s goal can also be grasped quite easily: opposing the ordinary images of mass entertainment, which were connected to his real concerts, and criticizing this imagery with an idealized image of the intimacy of the music. From the painter’s point of view, however, the goal is to show the deep community that exists between music, literature, and painting and to place the intellectual excellence of visual art next to the other two art forms. The work is also a full-fledged demonstration of Romanticism as an artistic stance, involving its “interart aesthetics” first formulated by Robert Schumann: “Die Ästhetik der einen Kunst ist die der anderen, nur das Material ist verschieden” (“the aesthetics of one art is that of the other, only the material is different,” Schumann 1854, 43). Liszt fully shared the same view, as it can be read in his memories from Italy: Day by day my feelings and thoughts gave me a better insight into the hidden relationship that unites all works of genius. Raphael and Michelangelo increased my understanding of Mozart and Beethoven; Giovanni Pisano, Fra Beato, and Il Francia explained Allegri, Marcello, and Palestrina to me. Titian and Rossini appeared to me like twin stars shining with the same light. The Colosseum and the Campo Santo are not as foreign as one thinks to the Eroica Symphony and the Requiem [Mozart’s]. Dante has found his pictorial expression in Orcagna and Michelangelo, and someday perhaps he will find his musical expression in the Beethoven of the future. (Hamilton 2005, 4)

Ultimately, Danhauser’s painting is a utopian montage of a certain kind, attempting to depict a reconciled humanity via the fraternity of artistic mediums.

Conclusion Claiming the brotherhood of the branches of art, if understood as a general idea, might have seemed an ordinary position in the mid-nineteenth century. At the same time, this was a specific and perhaps radical take if “brotherhood” was meant as the transferability of their respective medial traits and a basis for intermedial dynamics. This radicalness has to be seen in the context of the reductive, essentialist nature of the aesthetic views about specific artistic forms in the era, and not against the real experience of art. As Jørgen Bruhn notes, blending is an a priori condition in all [art forms], and [. . .] blending aspects consequently do not constitute a marginal phenomenon or a marginal subgroup: Mixedness characterizes all medialities and all specific [work of art]. Mixedness comes first, so to speak; the monomedial purity of any specific medial object is the result of an active purification – instead of the other way round. (2016, 15)

716

B. Veres

As we have seen, in Hausman’s portrait of Bach, the music is a possible continuation of the painting, and in a particular, intrinsic way. The sheet of music offered to the viewer encourages a medial change and consequently leads to a mixedness that is made emphatic by the overall experience. The nature of its intermediality is a deictic one. Della Croce’s painting also captures the moment before music resounds. However, the increased attention directed to silence that precedes the moment of the musical performing is used here not for a medial change but to comprehend the togetherness of a family for life and death. In contrast to the quiet painting of della Croce, the works of von Schwind and Schmid expect the viewer’s musical contribution in the form of remembrance or inner sonic association. This is all the more possible, because the moment depicted here is the resting point of a musical process in von Schwind’s case, while in Schmid’s painting, it is the echoing seconds between the last sounds of the play and the round of applause. Siemiradzki’s Chopin painting is closer to Danhauser’s, as both depict the performer while playing and the audience in a state of intense listening. At the same time, in Siemiradzki’s painting, the imaginary acoustic homogeneity of the work is violated by the whispering men standing in the doorway, while their function is also to emphasize prosaic realism. In other respects, Kwiatkowski’s painting is even closer to Danhauser’s. Both involve a complex medial reflectivity in trying to capture pure music as it reveals its specific reality. While doing so, they use specific painterly methods and open associations of well-known pieces of music triggering inner sonic memories in the beholder. At the same time, these images show an inverse process also, in which music generates imagery in the listener’s mind. In Kwiatkowski’s case, this image is a historical phantasm, in Danhauser’s case, the landscape “fresco” in the background of the Beethoven shrine. The difference is that in Kwiatkowski’s tableau, similarly to a multichannel audiovisual collage, there is a synchronicity and therefore acoustic conflict between the two realms of the picture: the vivid historical tumult and the passionate piano play of Chopin. In turn, in Danhauser’s Liszt painting, there is no aural competitor against the piano sound. Here, the competitor of music is the art of painting itself, which tries to ride two horses at the same time: demonstrating its capability of conveying music, held as the most spiritual art form, but also demonstrating music’s capacity to engender imagery in those sensitive beings who get in touch with it. However, the painting can only realize this wizardry with the cooperation of a viewer mobilizing proper aesthetic literacy, and this expects the recognition of the deep ambiguity and interrelation between different art forms. Ultimately, let’s consider the consequences one can draw from the analysis of Danhauser’s work, and the questions that arise from all this. The first is the question of whether music can be pictured at all. Is it possible to transplant a work of art into another medium or an art form in its entirety into another? When music is conceived according to the philosophy of absolute art in its utmost purity, this transplantation is doomed to failure from the outset. But it is indeed a real possibility if music turns out to be more than a mere ensemble of acoustic signs. If, however, music is already more than a set of sounds, then what is it that painting can add to this? The first answer that the Danhauser picture gives would be that fine art needs picturing music for the reason because that is precisely how painting can show its power. Fine art thus can

28

Picturing Music in the Nineteenth Century

717

demonstrate that, like music, it is also something more, than might seem at the first sight, more than the sum of its retinal effects. Another answer to the same question would be that picturing music can lend to the visual arts those higher powers and provide access to those invisible realms that are the remnants of metaphysics. We may also consider what hierarchies prevail concerning the medial conditions and interrelations of image, word, and sound in accordance with the Romantic views of the time as manifested in the Liszt phenomenon and in the Danhauser picture. To start with music, this art form seems infinitely strong and incurably weak at the same time. Its strength stems from the intimacy of its experience felt by the audience and the heightened spirituality attributed to it. On the flip side, music is incapable to resist being occupied by external forces, like visuality, materiality, narrativity, and theatricality (Kramer 2002, 81). Images reflect a very similar ambiguity, being strong and weak at once. The sense and meaning of an image are underdetermined unless words and sounds help foster the visual aspects to gain more articulateness. At the same time, an image is utterly suggestive, and in addition to contemplation, it also causes one to stare, suspending the will of the intellect. In terms of sensual effect, the word of poetry proves to be less impressive, unable to express the innermost intimacy. On the other hand, the poetic word is the lightning of the intellect: an opening up of a world, the alpha and omega of any art. The ambiguous position of each art form that we can register from the analysis of Danhasuer’s painting may steer our thinking to the conviction that various art forms are mutually dependent even while competing with each other. Nonetheless, an image does not transfigure into something else, it remains an image, just like the poetic word, or music remains itself (Mitchell 1986, 102–103; Rajewsky 2010). The nature and extent of the intermedial dynamics of them are decided in the act of aesthetic reception, which becomes, at least partly, independent from the intentions of the artists and tends to privilege once the monomodal and other times the multimodal experiences. In this respect, the above prehistory of audio-vision shows that the actuation of the intermedial potentials of any cultural repertoire depends less on the works of art themselves and their primary medial traits than on the intellectual, theoretical, and experiential attitudes that determine the aesthetic reception of the works. Picturing music in Danhauser’s work means picturing something invisible. This task can be approached through a detour by the visible. The visible is depicted by the use of montage and idealization. Meanwhile, the invisible is represented in three ways: through an intermedial reference, a latent narrative, and the metareferentiality of images-within-images. Earlier, Lessing talked about the inferior capacities of painting. Friedrich and the later generations of nineteenth-century visual art sought various solutions for the shortcomings mentioned by Lessing. Some turned towards the panoramic image (Kittler 2002), or the motion picture (Zielinski 1999), others towards hypnotic means (cf. Ruston 2008)6 – and those, who endeavored picturing music in allegoric paintings, were the ones following the idea of the brotherhood of the arts.

6

The hookah at the bottom left, in spatial opposition to that of the Beethovenian shrine, is a telling motif, which might be further discussed in another study.

718

B. Veres

References Adorno, Theodor W. 2003. Art and the arts. In Can one live after Auschwitz? A philosophical reader, ed. Rolf Tiedemann, 368–390. Stanford: Stanford University Press. Arvidson, Mats. 2016. An imaginary musical road movie. Transmedial semiotic structures in Brad Mehldau’s concept album highway rider. Lund: Lund University. Beck, Eleonora. 2017. A new reading of the ‘Mozart family portrait’. Imago Musicae. International Yearbook of Musical Iconographys 29: 159–175. Bonds, Mark Evan. 2006. Music as thought. Listening to the symphony in the age of Beethoven. Princeton: Princeton University Press. Bowie, Andrew. 2002. Music and the rise of aesthetics. In The Cambridge history of nineteenthcentury music, ed. Jim Samson, 29–54. Cambridge: Cambridge University Press. Bruhn, Jørgen. 2016. The intermediality of narrative literature. London: Palgrave Macmillan. Bürger, Peter. 1984. Theory of the Avant-Garde. Trans. M. Shaw. Manchester: Manchester University Press. Chion, Michel. 1994. Audio-vision. Sound on screen. Trans. C. Gorbman. New York: Columbia University Press. Clayton, Martin, Trevor Herbert, and Richard Middleton. 2012. The cultural study of music: A critical introduction. 2nd ed. New York: Routledge. Comini, Alessandra. 2008. The changing image of Beethoven. A study in mythmaking. Santa Fe: Sunstone Press. Connor, Steven. 2005. Michel Serres’ five senses. In Mapping Michel Serres, ed. Niran Abbas, 153–169. Ann Arbor: University of Michigan Press. da Vinci, Leonardo. 2012. A treatise on painting [1632]. Trans. J.F. Rigaud. New York: Dover Publications. Dahlhaus, Carl. 1988. Klassische und romantische Musikästhetik. Laaber: Laaber Verlag. ———. 1989. The idea of absolute music. Trans. R. Lustig. Chicago: The University of Chicago Press. ———. 1991. Ludwig van Beethoven: Approaches to his music. Trans. M. Whittall. Oxford: Clarendon Press. Dayan, Peter. 2011. Art as music, music as poetry, poetry as art, from whistler to Stravinsky and beyond. Farnham: Ashgate. Doorman, Maarten. 2004. De Romantische Orde. Amsterdam: Bert Bakker. Elder, Bruce R. 2007. Hans Richter and Viking Eggeling: The dream of universal language and the birth of the absolute film. In Avant-Garde film, ed. Alexander Graf and Dietrich Scheunemann, 3–54. Amsterdam: Rodopi. Elleström, Lars. 2010. Media borders, multimodality and intermediality. New York: Palgrave Macmillan. ———. 2021. Beyond media borders. Volume 1, Intermedial relations among multimodal media. Berlin: Palgrave Macmillan. Földényi, F. László. 1993. Caspar David Friedrich: Die Nachtseite der Malerei. München: Matthes & Seitz. ———. 2010. Képek előtt állni. [‘Standing in front of pictures’] Bratislava: Kalligram. Gibbs, Christopher H. 1997. “Poor Schubert”: Images and legends of the composer. In The Cambridge companion to Schubert, ed. Christopher H. Gibbs, 36–55. Cambridge: Cambridge University Press. Gingerich, John M. 2014. “Those of us who found our life in art”: The second-generation romanticism of the Schubert-Schober circle, 1820–1825. In Franz Schubert and his world, ed. Christopher Gibbs and Morten Solvik, 67–114. Princeton: Princeton University Press. Golding, John. 2000. Paths to the absolute: Mondrian, Malevich, Kandinsky, Pollock, Newman, Rothko, and still. Princeton: Princeton University Press.

28

Picturing Music in the Nineteenth Century

719

Greenberg, Clement. 1940. Towards a newer Laocoon. Partisan Review 7: 296–310. Guillaud, Maurice. 1984. Caspar Davids theater. In Friedrich Linien und Transparenz, ed. Caspar David, 13–28. Paris: Center Culturel du Marais. Hamilton, Kenneth. 2005. The Cambridge companion to Liszt. Cambridge: Cambridge University Press. ———. 2008. After the golden age. Romantic pianism and modern performance. Oxford: Oxford University Press. Han, Byung-Chul. 2018. Saving beauty. Trans. D. Steuer. Cambridge: Polity Press. Harrison, Charles, Paul Wood, and Jason Gaiger, eds. 1999. Art in theory 1815–1900. Oxford: Wiley-Blackwell. Hegel, G.W.F. 1975. Aesthetics. Lectures on fine art, vols. 1–2. Trans. T.M. Knox. Oxford: Clarendon Press. Hoffmann, E.T.A. 1917. Beethoven’s instrumental music. Trans: A. Ware. The Musical Quarterly 3: 123–133. Kant, Immanuel. 1996. The contest of the faculties [1798]. Trans: M.J. Gregor and R. Anchor. In Religion and rational theology, 233–328. Cambridge: Cambridge University Press. ———. 2007. Critique of judgement [1790]. Trans. J.C. Meredith. Oxford: Oxford University Press. Kittler, Friedrich. 2002. Optische Medien. Berliner Vorlesung 1999. Berlin: Merve. Koerner, Joseph Leo. 2009. Caspar David Friedrich and the subject of landscape. London: Reaktion Books. Kovács, Imre. 2014. The apotheosis of Beethoven in Danhauser’s painting: Liszt at the piano. Studia Musicologica 55: 119–130. Kramer, Lawrence. 2002. Franz Liszt and the virtuoso public sphere: Sight and sound in the rise of mass entertainment. In Musical meaning. Towards a critical history, 68–99. Berkeley: University of Californai Press. Kregor, Jonathan. 2010. Liszt as transcriber. Cambridge: Cambridge University Press. Kristeller, Paul Oskar. 1951/1952. The modern system of the arts: A study in the history of aesthetics. Journal of the History of Ideas 12: 496–527; 13: 17–46. Lee, Rensselaer W. 1967. Ut Pictura Poesis: The humanistic theory of painting. New York: Norton. Leppert, Richard. 1993. The sight of sound: Music, representation, and the history of the body. Berkeley/Los Angeles: University of California Press. ———. 2002. Cultural contradiction, idolatry, and the piano virtuoso: Franz Liszt. In Piano roles: A new history of the piano, ed. James Parakilas and Others, 200–223. New Haven: Yale University Press. Lessing, Gothold Ephraim. 1853. Laocoon; an essay on the limits of poetry and painting [1766]. Trans. E.C. Beasley. London: Longman, Brown, Green and Longmans. Levinson, Jerrold. 2011. Music, art, and metaphysics. Essays in philosophical aesthetics. Oxford: Oxford University Press. Mendelssohn, Moses. 1997. On the main principles of the fine arts and sciences [1757]. In Philosophical writings. Trans. D.O. Dahlstrom, 169–191. Cambridge: Cambridge University Press. Millán Brusslan, Elizabeth, ed. 2020. The Palgrave handbook of German romantic philosophy. New York: Palgrave Macmillan. Mitchell, J.W.T. 1986. Space and time. Lessing’s Laocoon and the politics of genre. In Iconology, 95–115. Chicago: The University of Chicago Press. ———. 2007. There are no visual media. In MediaArtHistories, ed. Oliver Grau, 395–406. Cambridge, MA: MIT Press. Pape, Walter, and Frederick Burwick, eds. 1995. Reflecting senses. Perception and appearance in literature, culture, and the arts. Berlin: Walter de Gruyter.

720

B. Veres

Pater, Walter. 1980. The school of Giorgione. In The renaissance. Studies in art and poetry, ed. Donald Hill, 102–122. Berkeley: University of California Press. Pfeiffer, Ludwig K. 2002. The protoliterary: Steps toward an anthropology of culture. Stanford: Stanford University Press. Rajewsky, Irina O. 2010. Border talks: The problematic status of media borders in the current debate about intermediality. In Media borders, multimodality and intermediality, ed. Lars Elleström, 51–68. London: Palgrave Macmillan. Ruston, Sharon. 2008. ‘High’ romanticism: Literature and drugs. In The Oxford handbook of British romanticism, ed. David Duff, 341–354. Oxford: Oxford University Press. Schelling, F.W.J. 1989. The philosophy of art. Trans. D.W. Stott. Minneapolis: University of Minnesota Press. Schleiermacher, Friedrich. 1996. On religion: speeches to its cultured despisers. Trans. R. Crouter. Cambridge: Cambridge University Press. Schopenhauer, Arthur. 1969. The world as will and representation, vol. 1. Trans. E.F.J. Payne. New York: Dover Publications. Schulte-Sasse, Jochen, ed. 1997. Theory as practice: A critical anthology of early German romantic writings. Minneapolis: University of Minnesota Press. Schumann, Robert. 1854. Gesammelte Schriften. Vol. I. Leipzig: Georg Wigand’s Verlag. Seidel, Wilhelm. 2003. Absolute Musik und Kunstreligion um 1800. In Musik und Religion, ed. Helga de La Motte-Haber, 129–154. Laaber: Laaber-Verlag. Shaw-Miller, Simon. 2002. Visible deeds of music. Art and music from Wagner to Cage. New Haven/London: Yale University Press. Sheinberg, Esti, and William P. Dougherty. 2020. The Routledge handbook of music signification. New York: Routledge. Simmel, Georg. 2020. The philosophy of landscape. In Essays on art and aesthetics, ed. Austin Harrington, 218–230. Chicago: The University of Chicago Press. Sternberg, Meir. 1999. The ‘Laokoon’ today: Interart relations, modern projects and projections. Poetics Today 20: 291–379. Stoltzfus, Philip. 2006. Theology as performance. Music, aesthetics, and god in western thought. New York /London: T&T Clark. Sumowski, Werner. 1970. Caspar David Friedrich-Studien. Wiesbaden: Franz Steiner Verlag. Taylor, Benedict. 2021. The Cambridge companion to music and romanticism. Cambridge: Cambridge University Press. Veres, Bálint. 2012. How did music rise to philosophical Eminence? (and how has it been deprived of it?). In Music semiotics: A network of significations, ed. Esti Sheinberg, 25–34. Farnham: Ashgate. ———. 2021a. Patterns of musical time experience before and after romanticism. ESPES. The Slovak Journal of Aesthetics 10 (1): 64–77. ———. 2021b. Theatricality and the challenge of definition. Towards and alternative perspective. Naxos Musicology International. https://www.naxosmusicology.com/opinion/theatricality-andthe-challenge-of-definition-towards-an-alternative-perspective/. Accessed 24 Aug 2021. Vernallis, Carol. 2004. Experiencing music video: Aesthetics and cultural context. New York: Columbia University Press. Walker, Alan. 2005. Reflections on Liszt. Ithaca/London: Cornell University Press. Wasserman, Janet. 2003. A Schubert iconography: Painters, sculptors, lithographers, illustrators, silhouettists, engravers, and others known or said to have produced a likeness of Franz Schubert. Music in Art: International Journal for Music Iconography 28 (1–2): 199–241. Weber, Samuel. 2004. Theatricality as medium. New York: Fordham University Press. Westerdale, Joel. 2010. The musical promise of abstract film. In The many faces of Weimar cinema. Rediscovering Germany’s filmic legacy, ed. Christian Rogowski, 153–166. Rochester: Camden House.

28

Picturing Music in the Nineteenth Century

721

Winckelmann, Johann Joachim. 1972. Essay on the beautiful in art [1763]. In Writings on art, ed. David Irwin, 89–103. London: Phaidon. Wolf, Werner. 1999. The Musicalization of fiction: A study in the theory and history of Intermediality. Amsterdam: Rodopi. Wolff, Christoph. 2000. Johann Sebastian Bach: The learned musician. New York: W. W. Norton & Company. Wythe, Deborah. 1984. The pianos of Conrad Graf. Early Music 12: 447–460. Zielinski, Siegfried. 1999. Audiovisions. Cinema and Television as Entr’Actes in History. 1999. Amsterdam: Amsterdam University Press.

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht) and Their Resonances in Contemporary Performance

29

Chiel Kattenbelt and Robin Nelson

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Prototypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wagner’s Artwork of the Future as the Prototype of the Dramatic Mode of Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kandinsky’s Stage Composition as the Prototype of the Lyrical Mode of Presentation . . . Brecht’s Epic Theater as the Prototype of the Epic or Rather Dialectical Mode of Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Effects and Affects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Trajectories of Intermediality – Beyond Postmodernism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Resonances of Wagner’s Dramatic Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Resonances of Kandinsky’s Lyrical Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Resonances of Brecht’s Dialectical Theater . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Endnote . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

724 727 727 728 729 731 736 737 740 742 745 746

Abstract

Richard Wagner’s artwork of the future, Kandinsky’s stage composition and Brecht’s dialectical theatre are considered prototype models of intermedial theater, especially where these models relate to the question of inter-art relations in the performing arts. Kandinsky and Brecht oppose the illusionism of Wagner’s dramatic theater, which mainly aims at presenting an imaginary world in which the spectator is completely immersed. In contrast, Kandinsky advocates a lyrical theater in which the artist expresses his vibrations of the soul and thus moves the C. Kattenbelt (*) Utrecht University, Utrecht, The Netherlands e-mail: [email protected] R. Nelson Royal Central School of Speech and Drama, University of London, London, UK Manchester Metropolitan University, Manchester, UK e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_34

723

724

C. Kattenbelt and R. Nelson

spectator in a similar way, and Brecht an epic or rather a dialectical theater in which social issues are raised to motivate the spectator to societal change. Referring to specific examples, it will be demonstrated how the three models still resonate in contemporary intermedial theatre practices. In this context, the intermedial manifests itself mainly in the principles of both-and (e.g., privatepublic, present-absent) and performativity (how the material and sensorial are engaged) as will be explained in more detail and illustrated by the examples discussed. Keywords

Affect · Conjunction · Disjunction · Both-and · Performativity

Introduction The aim of this chapter is threefold. Firstly, it identifies historical prototypes of interrelations between mediums from the late nineteenth to early twentieth centuries, which sought in intermediality new forms of artistic expression consciously aimed at particular ends. Secondly, it explores the effects and affects intended to be achieved in their historical contexts. Thirdly, it updates the trajectories of these prototypes as they have impacted on the late twentieth and early twenty-first centuries at historical moments different from those of their instigation. Following this brief Introduction, the chapter is presented in three sections: “The Prototypes,” “Effects and Affects,” and “Trajectories of Intermediality – Beyond Postmodernism.” Over the past 40 years, the term intermediality has become so widely used in different domains that Irina Rajewsky (2005: 45) suggests each usage should be accompanied by an explanation. Since, however, other chapters in this book explore terminology and its usage, we aim here just to highlight aspects relevant to Richard Wagner, Wassily Kandinsky and Bertolt Brecht and their legacies. In what follows, we mark what we call the prototypes of intermediality of these three theorist-practitioners before exploring shifting contexts post-1960s and jumping to examples of twentyfirst-century practices. We are primarily concerned with two aspects: how might mediums be differently brought together to establish intermedial practices; and what might be the intended effects and affects of different approaches to combining mediums. The trajectory of dispositions to intermediality between our late nineteenth century/early twentieth century starting point and the present time has not been direct. Some aspects of Modernism, particularly in the 1950s/1960s, militated against integrated practices, emphasizing rather a purism of distinctive art forms. It is worth briefly outlining this tendency prior to sketching a range of inters. Reflecting on a marked trend in Modernism for the arts to distinguish themselves from the sciences and from each other, Clement Greenberg, an influential commentator, observed that:

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

725

The arts could save themselves . . . only by demonstrating that the kind of experience they provided was valuable in its own right and not to be obtained from any other kind of activity. Each art, it turned out, had to perform this demonstration on its own account. What had to be exhibited was not only that which was unique and irreducible in art in general, but also that which was unique and irreducible in each particular art. Each art had to determine, through its own operations and works, the effects exclusive to itself. (Greenberg 1960: 5)

In this context, pure Dance was disposed to foreground the body in motion; pure Music focused on sound (pitch, tone, duration); and the pure Visual Arts emphasized line, color, and design. Since we will be concerned in intermediality discourse with affect, it might be noted that the arts thus conceived map on to the senses of feeling, hearing, and seeing – smell playing only a small part in this story. And it is a story in the sense that Greenberg’s account is a percept, a construction of how things might function in the world; while formulated in response to what was happening in culture, it additionally shaped it with a thought. Historically, going back to the Greeks in Western culture, dance has nearly always been associated with music; music has often worked with writings; the visual arts have concrete forms such as sculpture and ceramics, which invoke touch as much as sight. It is nevertheless the case that, counter to the integrative strand of Modernism, each art form sought in the mid-twentieth century to distinguish itself and that when, around the 1960s, they were seen evidently to transgress their Modernist boundaries, Dick Higgins’s percept, intermedia, was coined, picking up on the dispositions to combine mediums of Wagner, Kandinsky, Brecht (see below) and others (e.g., Futurists, Dadaists). Another aspect of backstory is worth noting here, particularly since our emphasis is on intermedial theater and performance: theater and cinema have always been complex forms, combining elements of the supposedly pure art forms. Indeed, coming from an intermedial rather than a purist perspective, Kattenbelt (2006: 37) proposes that theater, above all considered as the art of the performer, might, in its own right, function as a hypermedium embracing all other mediums. One sense of intermediality, then, is the relations between mediums. As noted, up to the Modernist period, the elements in play were sounds, visuals, words, and movements. Formulated as disciplines, these became Dance, Music, Visual Arts, and Writing, with Drama/Theater, as noted, being always a hybrid mix, as was Film as it developed to combine image with sound. Both from the point of view of artistic production and of critical commentary, interest grew in the latter half of the nineteenth century in relations between elements (rather than simply in what they represented) and in the various effects and affects potentially generated. Thus, a second consideration concerns the ways in which discrete mediums might be brought together. “Interdisciplinary,” or (echoing Wagner) “integrated,” arts involved mixes of two or more of these elements. As will be clarified in what follows, two contrasting approaches are taken by Wagner and Brecht. Where Wagner sought to integrate the elements, Brecht sought to separate them and set them in play

726

C. Kattenbelt and R. Nelson

against each other. As a working shorthand, we term these contrasting approaches conjunction (Wagner fusing the elements) and disjunction (Brecht separating them).1 After Greenberg, a gradual slippage between the terms “arts” and “mediums” emerges in the 1960–1970s with “inter-arts” or “integrated arts” gradually being largely displaced by “intermediality” in the 1980s. There are as many points of view on what constitutes a medium as there are conceptions of the arts and some core conceptions may be helpful at this stage. First, in the arts, a medium is the material used in the making of the work (e.g., paint, clay, wood, textiles, metals) and “mixed media” refer to combinations of mediums thus understood. Secondly, conduits (cable for electricity or pipes for water) might be considered mediums along with carriers (air for sound) and transmitters (fiber optics for digital signals). Thirdly, McLuhan’s famous notion that mediums are “extensions of ourselves” (1964) picks up on the idea of medium as something that affords an additional agency (e.g., television afforded seeing at a distance – Fernsehapparat in traditional German). Digital culture has much extended the capacity of various mediums to augment human physical and conceptual capabilities. Indeed, digital culture puts many of the discrete arts discussed above on the same platform through its 0/1 basis, and the convergence (Jenkins 2006) afforded has contributed much to intermediality discourse. In “The Modalities of Media: A Model for Understanding Intermedial Relations,” Lars Elleström responds to the emergent, dense, and multilayered territory, distinguishing between a “medium” understood as a single entity and its “modalities” and “modes.”2 As he puts it, “a medium is a channel, one might say, and of course there are many media, that is, modes of mediating information and entertainment” (2010: 14). Extending human capacities to the Cyborg under digital culture raises profound philosophical questions about what it is to be human – or “posthuman,” as some accounts have it – and our both-and sense of intermediality now is informed by them. Relations between the material and the immaterial such as simultaneously inhabiting both actual and virtual space, or examples of intermedial theater, which are physically based, live in the here and now, but also on-screen, repeatedly pose questions about what is happening in interrelational experience. For some time, and perhaps echoing Samuel Taylor Coleridge’s early formulation (cited by Dick Higgins in Higgens and Higgins 2001), the notion of the in-between has been used to account for phenomena which are neither this nor that but something in the middle. This approach remains useful in some domains, but we ultimately consider that the compound both-and better characterizes contemporary arts and media practices (see also Kattenbelt and Mancewicz 2023) as we shall see in section “Trajectories of Intermediality – Beyond Postmodernism.” But it is now time to turn to the prototypes: Wagner, Kandinsky, Brecht.

1

We wish to acknowledge the seminal work of Liz Stewart and Derek Akers in the 1980s on Wagner/Brecht and conjunction/disjunction. 2 An extended version of this model has appeared in Elleström 2021.

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

727

The Prototypes Wagner’s Artwork of the Future as the Prototype of the Dramatic Mode of Presentation Wagner wanted to revive the unity of the arts of Greek tragedy and conceived his idea of the Gesamtkunstwerk,3 literally the integrated work, or “the artwork of the future” as he also claimed (Wagner 1983 [1849]). Though his operatic oeuvre is often associated with overwhelmingly powerful emotion, Wagner defined music as the heart (rhythm) that reconnects the head (poetry) and the body (dance) with each other. One sense of the Gesamtkunstwerk, bringing together different art forms in powerful combination, is nuanced by Wagner’s view that painting, sculpture, and architecture only fulfill a supportive function, based on the logic that they derive from nature, rather than humankind. Wagner’s ideas were – and still are for many artists from different disciplines – an important source of inspiration for crossing the borders between the individual arts: to reintegrate them in a new total artwork. Art forms, or elements, can be brought together in different ways with various intentions. Wagner’s idea of the Gesamtkunstwerk as an autonomous Totalkunstwerk related very much to the striving for perfect illusion. In the specifically designed Festspielhaus in Bayreuth, which opened in 1876, the orchestra pit is partially under the stage and hidden from the view of the audience by panels to efface the musicians’ physical presence as source of sound. The aim was total immersion into the world that the performance presented. Wagner’s ideal of “the artwork of the future” fits perfectly into the ideal typical category of dramatic presentation, precisely because of the illusionism he pursued and practiced in the Bayreuther Festspielhaus. Peter Szondi (1983 [1956]) characterizes the dramatic mode of presentation in terms of the absolute drama, which refers to nothing but itself, the paradigmatic mode of illusion. Sitting in the darkened auditorium (as in Bayreuth), the audience watches as an invisible witness the actions that take place in an imaginary world with nothing obscuring the view. The narrative is carried by characters who act from recognizable (tangible) psychological motivations. Time and space form a closed continuum, although only those moments are shown that are important to follow the causality of the successive actions within an overarching structure of beginning, middle, and end. In short: the dramatic mode is primarily dominated by the presentation of the actuality of action and, to this end, theater has developed a conventionalized system of interruption techniques (set and light changes, scene divisions according to the presence and absence of characters, etc.). The presented actions are experienced as playing in the present and, ephemeral as they are, oriented to the future.

3

Wagner did not coin the concept Gesamtkunstwerk (as almost none of the concepts he uses). Even though the concept appears only sporadically in his writings, it is mainly associated with him. For a “brief genealogy of the concept” (see Vazsonyi 2016: 21–38).

728

C. Kattenbelt and R. Nelson

Ironically, the ideal of the dramatic mode of presentation has found its full realization only in Hollywood movies. It is characteristic of the conventions of Hollywood film that camerawork and editing are made so invisible that the presented world is optimally accessible to the viewer. This has led to the speculation several times that if Wagner had lived in the twentieth century, he would have travelled to Hollywood to create a furore as a film composer, or perhaps even as an all-round film director or producer pulling all the strings from start to finish (e.g., Künzig 1990). Wagner’s aim was for a psycho-emotional transcendence but, for Friedrich Nietzsche, being completely absorbed in the world presented in the performance was a disconcerting experience of emptiness and complacency. The spectacle as it was offered to him in the Bayreuther Festspielhaus marked a turning point in Nietzsche’s evaluation of Wagner. From that moment onward he considered the man whom he once took to be the greatest genius of his time to constitute the very disease of his time, which needs to be fought against (1899: 16). Above all, Nietzsche opposes the reactionary tendency inherent in Wagner’s approach. To be completely overwhelmed and absorbed and leaving no distance for contemplation and reflection meant for Nietzsche that art, properly understood, is ignored, and reduced to entertainment. This tension is echoed in Walter Benjamin’s later distinction between the “cult value” of art and the “exhibition value” of entertainment (1977 [1936], Sects. VI and VII), and Theodor Adorno’s (1973) advocacy of art as a sanctuary to be defended against its instrumentalization, whether it be for political or economic or for educational or therapeutic reasons. In our both-and understanding of intermedial practice, we also start from the assumption that it concerns a conscious perception of the relations between arts and/or media in contrast with an illusionistic, seamless fusion. In this conception, Wagner’s performances might not be regarded at all as a prototype of intermedial practice. Nevertheless, the illusionism that Wagner pursued with his opera practice is an important point of reference, precisely because illusionism still plays such an important role in the performance practices of our own times (particularly mainstream cinema) though, in today’s media-savvy culture, illusionism might itself embrace an awareness of the performativity of the arts in terms of world-making, and even self-reference and self-reflection. We will discuss examples involving this kind of both-and in section “Trajectories of Intermediality – Beyond Postmodernism.”

Kandinsky’s Stage Composition as the Prototype of the Lyrical Mode of Presentation At the beginning of the twentieth century, a major revision of Wagner’s ideas was initiated. In his essays “Über Bühnenkomposition” [1912] and “Über die abstrakte Bühnensynthese” [1923] Wassily Kandinsky (1979: 49–61, 79–83) critiqued theater in its, by then old formulations. In his view, drama, opera, and ballet were petrified into museum forms, which had lost their pulsating forces. He declared that the theater needed a new form in which the individual arts, as developed over many

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

729

decades according to their own rules of individual languages, be brought together. However, this time in their pure forms as equivalent elements of colors (painting), sounds (music), and movements (dance). Despite their common disposition to integration, there is a fundamental difference between Wagner’s idea of the Gesamtkunstwerk and Kandinsky’s idea of the Bühnenkomposition as monumental art. In his operas, or music dramas as he called them, Wagner strove for a reunification and integration of the individual arts. Kandinsky, in contrast, considered the differentiation and autonomization of the individual arts as a necessary condition for the arts to reach, in the development of their own languages, purity of expression. In this, Greenberg’s conception of Modernism, cited above, is prefigured. According to Kandinsky, however, the specificity of theater is its hidden magnetic power derived from its provision of a stage on which a dynamic play of pure expressive forms occurs as a presentation of inner experiences – of the intensity of the “vibrations of the soul.”4 Whereas Wagner’s Gesamtkunstwerk remains guided by a dramatic mode of presentation (despite the leading role of music as a synthesizing force that connects the head to the body), Kandinsky’s monumental art is preeminently guided by a lyrical mode of presentation, by musical, pictorial, and choreographed movements (Kandinsky 2009: 129). In the lyrical mode, the intensity of experience is uppermost (see Kattenbelt 1994).

Brecht’s Epic Theater as the Prototype of the Epic or Rather Dialectical Mode of Presentation Theater scholars familiar with Brecht’s seminal critique of the dramatic theater and advocacy of the epic theater will recognize, in his famous comparative table (Brecht 1978a: 37), the roots of a distinction between the conjunctive and disjunctive principles of composition noted above.5 The plot and linear development with an eye on the finish, of the dramatic mode, its creation of the illusion of a real (though fictional) world to draw the spectator into an emotional empathy are radically set by Brecht in contrast with the “separation of the elements in the ‘epic theatre’ (his emphasis). The aim is to bring the spectator through critical thought to ‘the point of recognition’ that change is possible that ‘man is a process,’ that ‘he is alterable or able to alter.’” Separation, the state of being apart, implies difference and diversity instead of similarity and unity. As distinct from a classical or traditional conception, separation with a modern inflection draws attention to aspects rather than totality. It may also relate to abstraction and negation, expressing doubt or even denial of conceptual

“The theatre has a hidden magnet, which has the power to attract all these languages, all means of the arts, which together offers the highest possibility of monumental abstract art” (Kandinsky 1979: 80 – Kattenbelt’s translation). 5 See “The Modern Theatre is the Epic Theatre” in Willett (1978 [1930]: 37). 4

730

C. Kattenbelt and R. Nelson

totality. In aesthetic discourses on theater, separation is usually considered to be antithetical to Wagner’s paradigmatic Gesamtkunstwerk, and separation aptly characterizes many artworks of the early twentieth-century avant-garde: the Dadaists and Surrealists, in particular, separated all kinds of artistic procedures from their historically determined stylistic norms. Indeed, they declared the free availability of artistic means as an aesthetic principle, in particular with the artistic intention of shocking their audiences (see Bürger 1974: 22–24). Though Brecht might also be regarded as a seminal Modernist, it has been subsequently suggested – notably by Elizabeth Wright (1989) – that he might be better considered a Postmodernist. As she puts it: If modernism is to be characterized as breaking with tradition while still retaining an individualist stance, then postmodernism can be seen as calling into question both tradition and individualism (1989: 5)

Brecht’s political aims in creating his version of a theater for the future stretched to a communitarian political ambition beyond creating an intense experience for individuals. Going beyond Kandinsky and those Modernists and Russian Formalists whose defamiliarization tactics aimed to enhance perception, Brecht’s interventionist thinking (eingreifendes Denken) sought in the Verfremdungseffekt to foster a specific critical awareness of the social-constructed-ness of the world and a recognition that it was subject to change by human intervention. A number of questions, pertinent to later intermediality debates, arise about the impact of Brecht’s Verfremdungs tactics. If critical awareness is merely a state of mind, how might it transpose into action to change the world? Moreover, since Brecht wanted theater to be an entertaining as well as an educative experience, is there a danger that insights into social function might be lost in pleasure? There is, indeed, in Brecht’s shift to the Lehrstücke, an implication that the audience might need to be educated to a point of disposition toward a new mode of knowing (see Steinweg cited in Wright 1989: 12–13). Ultimately, Brecht preferred the term dialectical theater to epic theater because the revelation of contradictions in the world – typically overlooked in dramatic theater’s presentation of a smooth and coherent artifice – needs to be a shared experience of a production process (Brecht 1978b). Key to Brecht’s approach is that theater is a site of production of what elsewhere Nelson terms “doing-knowing,” a fluid interplay not simply between text and reader (in the poststructuralist formulation) but between (authored) play-inperformance and experiencer (Nelson 2022: 5). The process of knowing is an on-going communal interaction, a both-and construction as in our express approach to intermediality. To Brecht, as a Marxist, the ongoing process of history is more foregrounded, perhaps, than in our approach to intermediality today, but the expanded sense of production is the same. Things have, of course, moved on in the theater and in culture since Brecht, as we shall see, but his seminal essay remains a point of reference in the numerous allusions to him in the context of the debate on intermedial theatre.

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

731

Effects and Affects All three practitioner-theorists above conducted aesthetic experiments with the aim of achieving specific effects upon audiences. In summary, Wagner sought to engage his audiences in an overwhelmingly powerful – and in his terms transcendent – emotional immersion; Kandinsky sought to facilitate inner experiences of the intensity of “vibrations of the soul”; Brecht sought to draw attention, in a Marxian vein, to the socio-structural causes of “the human condition” and of the potential for people to take action to change things. Though their aims differed considerably, they might all be seen to follow a logic of their times, dominant since Newton, that all effects have a cause and, in this sense, new aesthetic arrangements in the arts might result in specific effects. In humanist arts, linear narrative and renaissance perspective organized time and space to make sense of the world in a particular way. The conventions of Western theater, as they emerged and developed from the early seventeenth century to the late nineteenth century, constructed an eye–mind relationship privileging the perspective of the rational individual from a fixed, though unacknowledged, Eurocentric standpoint. The illusionistic representations of real life at the height of late nineteenth century Realism demanded an empathetic emotional engagement with individual character which, on the one hand, Wagner took to its limits and, on the other, Brecht denounced. Hindsight reveals, of course, that the aims of Wagner, Kandinsky, and Brecht were neither fully determined nor achieved. Worldwide Marxist revolution did not come about after Brecht and, where uprisings did occur, their cause cannot be attributed to his Epic theater. However, there is some evidence of the kinds of impact each practitioner-theorist has had (see below). The point here, however, is that it is the deterministic mindset of the historical culture in which they operated which thinks in terms of effects. Even Brecht’s practice, for all his avowed aesthetic and political radicalism, was ultimately logocentric, aiming at persuasion in a scientific-rational tradition (see Auslander 1997: 34ff ). However, as Jens Schröter has noted in an insightful article, sociocultural contexts change over time such that assumptions about the impact of new aesthetic arrangements require adjustment. Capitalism has a habit of appropriating radical practices, which aimed to stand outside market forces, and re-functioning them for commercial purposes – at times for the advertising of commodities. This is not the kind of Umfunktionierung Brecht had in mind, and it may well be, as Schröter proposes, “an apposite moment . . . for us to probe the political implications of our own discourse about intermediality?” (2011: 108). The legacy of the cause-effect mindset is sustained in some of the discourse on intermediality. Boenisch, for example, proposes to define “intermediality” in terms of its “intermedial effects” (2006: 116ff ). In placing different media (actors, pictures, tape) on-stage, Boenisch argues, they are “theatrically reproduced into something beyond their mere (even less: pure) original presence” (his emphases, 2006: 114). The mediums are doubled, becoming both the sign and the thing itself and a gap

732

C. Kattenbelt and R. Nelson

remains perceptible between the sign and the thing.6 Thus, for Boenisch, intermediality is an “effect of performance. . .created in the perception of observers” (2006: 113), because the relational aspect between thing and sign is a matter of experiencing. As Boenisch summarizes: Theatre multiplies its objects in a remarkable way into objects on stage that are present and representations at the same time, and - above all - they are presented to someone who is perceiving and observing them. (2006: 114)

While this account is broadly persuasive, we are not convinced that such disjunction is a necessary condition, inherent in all intermedial practices. Indeed, following the lineages of Wagner, Kandinsky, and Brecht (see examples in section “Trajectories of Intermediality – Beyond Postmodernism” below), we note among today’s intermedial practices, some tending, in Bolter and Grusin’s seminal distinction, to hypermediacy while others tend to immediacy (Bolter and Grusin 1999: 33–34). The assumption of the experience of dislocation arising from semiotic disjunctions does not fully justify Boenisch’s second move to claim that “intermediality offers a perspective of disruption and resistance. . . And create effects of alienation and dys-referential un-realities” (his emphases, 2006: 115). For, if theater in its three inherent layers of “presence, presentation, and representation” (2006: 114) always produces a gap between sign and thing, intermedial theater is not distinguished in these “effects” from other modes. Boenisch, to some extent, anticipates the objection that not all intermedial arts dispose alienation. He points out that: According to the standard hegemonic logic of representation, all these simultaneous, alternative layers, levels and perspectives offered en-route would be homogenized again into a single, closed and coherent final product of representation: in the destination of the ideal viewpoint, the single sharp focused picture of the reading camera-eye, or the one defined meaning of the text. (2006: 114)

If the “effect” is a matter of perception, and that perception is ossified through enculturation into making sense of things, then something is perhaps needed in the principles of composition of the text to mobilize the possibility of a shift in perception. Boenisch proceeds to point out that, “[t]he plurality of the perspectives might also spill over, crack and produce an untidy mess of meaning – either as a calculated result or somewhat subversive side effect” (2006: 114). In other words, some conjunctive compositions might be disposed to immediacy while others are disposed to hypermediacy. It may be that the tendency of nontraditional practice has been toward hypermediacy, rather than the immediacy found in some popular forms, as we shall see, but it is by no means a simple binary distinction. Moreover, the

This evokes the notions of theatrical signs being “signs of signs” rather than “signs of objects” (Bogatyrev 1971 [1938]), and square semiosis (Eco 1977) as inherent to the act of object, actions and events being staged, i.e., framed in a performative situation.

6

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

733

“effect” is also in question since the situated dispositions of the readers/experiencers are also in play.7 In respect of cultural disposition, the dominant cause-effect logic characteristic of the late nineteenth and early twenty-first centuries, though still discernible, has lost much of its force. Today, we think more in terms of “affects” and “affective experiences.” Partly as a result of both conceptual work and audience research since the 1970s, more emphasis is now placed on agency, the active engagement of experiencers in encountering texts or artworks, rather than assuming passive reception. Barthes’s seminal essays, “The Death of the Author” and “From Work to Text,” emphasize a playfulness between a reader and a text no longer determined by an author. Subsequently, Rancière’s emancipated spectator is compelled to “overcome the gulf separating activity from passivity” (2009: 12). In Media Studies, John Fiske (2010 [1987]) in the 1980s, and subsequently Henry Jenkins (2013 [1992]), established the concept of the producerly text whereby the ur-text is fluid and subject to multiple engagements ranging from variable interpretative readings by differently situated members of the audience to literal reconstructions by playful fans. Indeed, contemporary arts and media practitioners have been influenced to adopt more open principles of composition in recognition of, in Barthes’s formulation: [t]he intertextual in which every text is held, it itself being the text-between another text. (Barthes, 1977b: 160)

The ultimate logic of this disposition would deny any influence to the textual construct itself, leading Stanley Fish famously to ask, “is there a text in this class?” Wishing to sustain some sense of the kinds of impact different forms of intermedial text might have (see below), we adopt rather a middle position following Murray Smith’s observation that “whilst no text can determine an effect or affect, certain principles of composition dispose some affects rather than others” (1995: 63), and we speak in what follows of “affects” rather than “effects.” As with many of the terms under discussion in this chapter, “affect” is also contested. Some commentators simply fall back on subjectively felt states of emotion. But it is important in today’s context to avoid a crude Cartesian separation of mind and body, thought and feeling, just as, in different ways, Wagner and Brecht sought to work through both reason and emotion. Patricia T. Clough (2008) notes that there is a lineage from Spinoza through Bergson to Deleuze and Guattari informing a formulation of “affect,” which avoids a separation of mind and body, a separation of sensations/emotions from thinking.8 Indeed, Melissa Gregg and

7 See Bay-Cheng et al. (2010: 45) for the term experiencer, which Nelson coined when it became apparent that the traditional terms “audience” and “spectator” were inadequate to account for how contemporary performance operates. 8 In Massumi’s reading of Deleuze and Guattari), “L’affect (Spinoza’s affectus) is an ability to affect and be affected. It is a prepersonal intensity corresponding to the passage from one experiential state of the body to another and implying an augmentation or diminution in that body’s capacity to act” (1987: xvi).

734

C. Kattenbelt and R. Nelson

Gregory Seigworth (in whose seminal collection Clough’s article is reprinted) suggest that “affect and cognition are never fully separable if for no other reason than thought is itself a body, embodied” (2010: 2–3).9 Because “affect” is a matter of experience in process, they conclude that the concept does not lend itself to instrumental methodological subjugation. They speak instead of “the affective bloomspace of materiality” (2010: 9). We propose that intermedial arts, inter-relating the actual and the virtual in a thinking-feeling process, offers one such bloom-space in which various intermedial compositions dispose, but cannot determine, a range of affects. Dubbed the affective turn, a shift from work to event, away from the “meanings” of aesthetic engagements to “experiences,” has its parallel in the move from “effect” to “affect.” Where, in many popular cultural forms, the principle of conjunctive relations between the elements remains in use, innovative practices have tended toward the disjunctive, if only because experiencers are increasingly media-savvy. In bringing together the discrete elements or media, the disposition is to avoid harmonizing them into a coherent whole (cf. Wagner’s Gesamtkunstwerk) but rather consciously to leave them in play, related to each other merely by articulation and juxtaposition (an extension of Brechtian disjunction). In our own experiences of a range of intermedial art practices, some are dislocating/hypermedial in that one’s bodymind holds two conflicting things simultaneously.10 I am both here and there; I am on-stage and on-screen and in the auditorium. There are others in which attention is drawn to a situation in which I am a (doubly) constructed or in which I am made conscious of being both an individual and part of a collective. Then in other pieces, I feelingly inhabit an actual world (with material consequences) and a virtual world which, though not actual, may feel more intense. I see-hear – am palpably aware of – a person live on stage speaking the words of his father in his father’s accent and I see-hear – am palpably aware of – that person becoming his father in digital transformation. Ultimately, it is a matter, as Massumi puts it, of “perception of the event of perception in the perception [. . .], in the immediacy of its occurrence, as it is felt” (2008: 6). Parataxis, the grammatical term used to signify a lack of joining words between nouns, verbs, and other components of a sentence might usefully be adopted to describe the principle of today’s disjunctive compositions.11 A paratactical approach to making work would be one in which, to a greater or lesser extent, the media, though structured in relation to each other, are consciously left in play rather than fused together or harmonized. It is the lack of seamless connectedness,

9

Perception, here, understood to be a dialogue between sensory perception and concepts, a process of knowing in which knowledge can only be neared but never fully grasped because open to further dislocations just as it is the moment of consolidation. 10 A discomfort (unheimlich) “perversion” (Bishop) produced through two or more mediums in dialogic engagement. 11 For further discussion of parataxis (see Perelman 1993).

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

735

the gaps in the play between two elements juxtaposed rather than fused, which typically requires the experiencer to be active in engaging with the work The pleasure of engagement with this kind of open text is that of a co-producer actively, if playfully, constructing her/his own experience. In contrast, the pleasure of engagement with a conjunctive/immediacy text is the satisfaction of making sense of things which, in the scientific-rational tradition, is said to be part of a hard-wired human disposition to understand an ultimately coherent world. If the work is bound in a sense-making frame, the satisfaction is one of closure, the sense (if illusionary) that we know where we are and that our universe is predictable. Located in today’s culture, which both privileges personal choice and is more media-savvy than its predecessors, there may, as noted, be an awareness even in accepting closure of the constructed-ness of the experience, and perhaps of its political implications, arguably disposed to passivity. But, across a range of cultural forms, today’s preference appears to be for increased textual playfulness and variable affects, rather than singular meanings, though this does not necessarily entail political radicalism. Indeed, it might be seen as a dimension of a dominant neoliberal Western culture, which privileges personal choice and the distinctiveness of individual experience in marked contrast with Brecht’s intention, in radically separating the elements, to dispose a political critique. The alterations of perception as experienced through some of today’s intermedial practice may, then, entail more experienced – philosophical than rational – political dislocations. Since the fall of the Berlin wall in 1989, and an increasing sense of victory of the capitalist West over the Eastern communist bloc as even China embraces free market economics, any address of inequalities in the social process are more likely to be figured in terms of the politics of the personal than of the raising of collective consciousness mobilized into action. Braidotti’s nomadic subject, for example, though indubitably connected to a radical ethics of transformation, involves a force of resistance directed against dualist thought, claims of universality, fixed or stable identities. Nomadism is not about “fluidity without borders but rather an acute awareness of the nonfixity of boundaries” (2011: 66). Digital technologies in intermedial play contribute to shifting experiences in their capacity to afford a range of new perspectives and dynamics. Individual customization is a term closely connected to digital technology, and is used by Lev Manovich (2001: 51) to identify a distinct logic of postindustrial society and digital media: the variability and the modularity of the bits and bytes that distinguish digital or new media from older media allow users to arrange and program media content according to their own preferences although often limited by formats (2001: 29–37). Established conceptions of time and space, in particular, have been roundly challenged through time-space compressions. Telematic practices, for example, might bring into a space here-and-now spaces on other continents and in different time zones. To some, networked cyberspace affords an entirely new spatial dimension, and time has been fundamentally destabilized. As Dixon summarizes, “the juxtaposition of different “simultaneous” temporalities (live and recorded/computer rendered) can complicate the audience’s perceptions of time and space to the extent that, rather than simply “suspending disbelief” and experiencing performance time according to traditional protocols of liveness, a

736

C. Kattenbelt and R. Nelson

different perception of extratemporality is experienced” (2007: 524). In addition, the body has been digitally fragmented and, more than ever, extended through digital devices into that accent of the posthuman, which sees the capacities of digital technologies as adjuncts to human capabilities.12 Cyborg practices and virtual theaters pose even greater challenges to normative perceptual assumptions when, as Foucault puts it, “our experience of living in the world is less of a long life developing through time than that of a network that connects points and intersects with its own skein” (cited in Dixon 2007: 518).

Trajectories of Intermediality – Beyond Postmodernism In this section, we offer a range of examples, which loosely map on to the WagnerKandinsky-Brecht models outlined above. But none of them follows directly either the compositional principles or the intended effects of those models because times and practices have changed, as indicated. Resonances of the models are, however, discernible and may afford a useful lens for making refined distinctions between practices and their implications. Many of the 1990s and Noughties accounts of intermediality construct it in terms of a liminal, in-between space, as noted. Indeed, the in-between-ness of intermediality has been associated with the kind of displacement of perceptual norms in ritual mysteries (theorized by anthropologist Victor Turner) and ascribed to a state of passing through or becoming. Indeed, liminality subsequently became a key concept in Erika Fischer-Lichte’s account of the “transformative power of performance” (2008) and there may be echoes here of Wagner, though with a social function beyond the transforming power of love. In the introduction to Mapping Intermediality in Performance, Nelson suggests that “in-between-ness” is too vague a notion to cover, “the concrete effects of being definitively multiple and interrelational” (2010: 17) in the complex networks of the digital environment. Rather than an in-between state, he suggests that contemporary experience comprises a range of both-and relationships not just “with an ‘Other’ . . . [but] with any number of ‘Also-Others’” (2010: 17). In everyday life, we increasingly inhabit both actual and virtual worlds simultaneously. Though this may occasionally involve a sense of in-between-ness, it is more typically a matter of both-and. As he notes, “spaces may be both public and private and bodies may be both present and absent. . . These are not contradictory qualities but rather, essential, mutually constituting elements” (2010: 17). Traditional bearings have also been dislocated because the capacities of digital technologies have disrupted a sense of geographical, Euclidian, space, and a sequential flow of linear time.

12

Turing famously saw a walking stick as an extension to human capacities and computers as electronic extensions.

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

737

There are echoes of Brecht in the dislocations and disjunctions experienced here but, unlike the Agit-Prop theaters of the late 1960s and 1970s, no overt political outcomes are anticipated – though they may not be precluded. To repeat, while intermediality under digital circumstances as outlined manifests some affordances, not all intermedial theater practices are critical or radical. Indeed, any meta-theory of the effects of intermedial theater remains open to question, so we propose here to work with particular (material) texts in specific contexts, bringing out the resonances of our three models but pointing up the deviations from them as experienced.

Resonances of Wagner’s Dramatic Mode In his performance Are you ready, are you ready for love? (Theater Malpertuis and Productiehuis Brabant, 2005, reprise in 2011) the director Piet Arfeuille forces the spectator into the position of voyeur, of an invisible witness, typical experienced in (mainstream) cinema functioning as a mass medium. The two actors, a man and a woman, never show awareness of the audience’s presence. The fourth wall is not broken at any time. There is a strict separation between stage and auditorium, which is even more noticeable in the small black box theaters for which the performance is made. Arfeuille associates the voyeuristic gaze of mainstream cinema with that of the anonymous, isolated loner in the big city, where people usually live close together but on their own alongside each other. QR code: Are you ready, are you ready for love?: https://vimeo.com/91711993

The stage presents two adjoining apartments that, as it turns out, are separated by a one-way mirror, because of which the audience becomes complicit, as it were, in the voyeurism of the man who spies on the woman next to him. In a sense, the man cinematizes his gaze by using a light filter for his eyes and for the lamp that illuminates his room, and by adding music to the scene. He turns reality, so to speak, into a film. The performance constructs a clash between the reality of illusion and the illusion of reality. In one of the reviews, it is dubbed “webcam theatre in split screen” (de Smet 2005). Indeed, it makes sense to consider the one-way mirror as a screen and the imaginary fixed-frame camera as a webcam. But what is also important here is that the voyeuristic gaze of the webcam, and that of the film spectator who slyly looks on, conflicts with the shared viewing experience of the theater spectators, who are usually aware of being part of the audience as a collective (cf. Kattenbelt 1995: 159–162).

738

C. Kattenbelt and R. Nelson

In the theater, the conditions under which a voyeuristic gaze is permitted, either for the sake of intimacy or immersion, are always in question. By playing with distance and perspective, film can guide the spectator into the world it presents, in that as film, in presenting that world, appeals to the spectator’s own imagination to establish causality and continuity. For example, space is often shown in fragments and the viewer him/herself constructs a spatial totality (cf. Mukarovsky 1974 [1934]). The more the spectator is dependent on his/her own imagination, the stronger the spectator has the experience of being present in the imagined world. But the spectator’s presence does require invisibility. Such ideas can be extended to an immersive theater performance such as The Drowned Man: A Hollywood Fable (2013) by Punchdrunk (directors: Felix Barrett and Maxine Doyle). This company constructs lifelike worlds through which the spectator can move freely to observe what is happening in different locations. They are designed in detail, challenging the spectator really to explore the spaces. Indeed, the spectators are ín a world that is presented (there is no auditorium as such). The price the spectator pays for the experience of proximity and immersion (that is, for being right in the middle of what is going on) however, is that she/he makes her/himself invisible and therefore anonymous by wearing a mask, watching silently without intervention. Those who have come to the performance in a group are requested to split up and follow their own routes. QR code: The Drowned Man: A Hollywood Fable: https://vimeo.com/124304433

The interactivity claimed by the company lies mainly in the possibility to determine your own path through the performance, knowing that you can only experience a part of everything that takes place in the performance and, strictly speaking, everyone takes a unique position (literally and metaphorically). And that is very different from (mainstream) film, in which all spectators, wherever they sit, must identify with the single eye of the camera, if only for the purpose of orientation, which determines the trajectory along which they move through the imagined world. A point of discussion about Punchdrunk’s immersive performances is whether the number of events taking place simultaneously in different locations destroys the basic principles of the dramatic presentation, namely, concentration and selection (see Pfister 1988, Chap. 3). If the overarching coherence is lost, the impact of the dramatic presentation is diminished, even though the sensory overload of excess information remains overwhelming (cf. Machon 2013: 155–165). For Punchdrunk it is, perhaps, more about the intensity of the overall experience, rather than following a story with a beginning, middle, and end.

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

739

The performances by Ivo van Hove and Toneelgroep Amsterdam13 can certainly also be classified within the mode of dramatic theater. In the case of classical repertoire, Van Hove consistently translates the worlds of the past into the world of today, in which screen media are omnipresent. There is therefore often a diegetic motivation literally to stage them. For example, in Roman Tragedies (2007), an adaptation of William Shakespeare’s Coriolanus, Julius Caesar, and Antony and Cleopatra, live video fulfills the role of television as a political arena. QR code: Roman Tragedies: https://tga.nl/en/productions/romeinse-tragedies

Many scenes are played as if they were (live) broadcasts of political debates, press conferences, or breaking news. A news ticker-tape (in surtitles) constantly informs the audience about important events that are yet to happen, which makes the spectators all the more aware of how reality is constructed by media such as television. The intermedial principle of both-and is also at work here: the theater provides the reality of staging to make the staging of reality visible. In the terms of Rajewsky’s intermedial reference (2005: 54), we could say that theater provides a stage for video to play the role of television by imitating its formats. In sum: video is often used in the performances by Ivo van Hove and video designer Jan Versweyveld in a thematic referencing function. In the performance The Fountainhead (2014), for example, frequent use is made of shots straight from above to emphasize the floor plan perspective of the two competing architects, who are the main characters of the performance. QR code: The Fountainhead: https://tga.nl/en/ productions/the-fountainhead

13

In 2018 renamed International Theater Amsterdam in a merger with the Stadsschouwburg Amsterdam.

740

C. Kattenbelt and R. Nelson

Resonances of Kandinsky’s Lyrical Mode Kandinsky’s stage compositions can be regarded as an attempt to transcend the limitations of painting by literally setting forms, colors, and sounds in motion. It is therefore not surprising that contemporary media artists have done this literally by transforming Kandinsky’s paintings into animations as if harking back to the so-called abstract films.14 Kandinsky’s Composition VIII (1923) and some of his other paintings have been adapted by Alexey Berezyuk (Front Pictures Studio) into an animation that was part of the multimedia exhibition Avant-garde – the Space of Colors and Forms (2014 in Kyiv), for which it was made. The projected animation covers the entire wall. QR code: Kandinsky: https://vimeo.com/109119184

Standing in front of it, you enter Kandinsky’s world of colors, forms, and sounds and are set in motion, as it were, by the pan and zoom movements that are inscribed in the moving images. The energetic, kinetic, and perhaps also synaesthetic, experiences evoked by the animation are further enhanced by the music (The Waltz of the Monsters – Toy Piano (1998) by Yann Tiersen) that accompanies the animation. There is something paradoxical in the intended immersion. On the one hand, the extension from painting to animation makes the cooperation and opposition of movements in colors, shapes, and sounds sensory, whereas, on the other hand, the immersive experience of Kandinsky’s world also lends it some narrativity, which Kandinsky himself wanted to avoid, precisely because of the vibrations he aimed for and in which he regarded his abstract art as extremely concrete. In sum, there is a tension between the inwardness of the experience and the outwardness of events. The dynamism suggested by the still images of Kandinsky’s paintings threatens to be lost as soon as the images are set in motion accompanied by music, even more in that the suggested explosion is transformed into flowing movements. Upon entering Kandinsky’s world, the objects in their shapes and colors become real entities belonging to the world by which the experiencer is surrounded. As far as the intensity of an immersive experience is concerned, we can easily imagine that virtual reality can be more e/affective than animation film, particularly

14

For example, Viking Eggeling (e.g., Symphonie Diagonale, 1925), Walter Ruttmann (e.g., Lichtspiel Opus I, II and III, 1921, 1922, and 1924, respectively), Oskar Fischinger (e.g., An Optical Poem, 1938).

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

741

because the immersant is offered the freedom to move and look around by him/herself in that world of moving objects and changing forms, colors, and sounds, instead of being guided by the eye of the fictitious camera. This would result in Kandinsky’s vibrations of the soul becoming even more of a corporeal experience, the physical body constantly interacting with and responding to a virtual world, which is more of a playground than a coherent world in which you are merely present, instead of making something happen or playing along. The both-and principle of the interaction between the physical and the virtual contains enormous potential for playfulness. Insofar as Kandinsky’s vibrations of the soul can be understood as spirituality, it is important to realize that the spiritual in art is always rooted in and remains connected to what Hegel (1986 [1835–1838]) considered to be the proper nature of art as distinct from philosophy and religion), namely, sensualization (Versinnlichung). And it is precisely this sensualization that is inherent in our understanding of intermediality as a specific mode of performativity (see Kattenbelt 2010), an intensification of the material and the sensorial, of the corporeal, of the interaction in the here and now, which is never subordinated to the presentation of an imaginary world.15 Within the lyrical mode of presentation, in line with Kandinsky’s idea of his stage compositions – in which the limitations of painting are transcended by bringing together moving bodies and objects and changing colors and sounds in their own purity to counteract or reinforce each other – we can also argue from the physical body of the performer, extended, and expanded in all sorts of ways in technologically produced images and sounds occurring in the same space. There are many examples of such practice within the domain of dance and technology – particularly performances in which there is actual interaction between dancer and technology. The movements of the dancer are tracked and converted into moving images and sounds in such a way that the expressiveness of the dancer radiates in colors, forms and sounds to the space by which she/he is surrounded. Formulated thus, the dancer is in a sense the instrument or interface that, with its movements, provides the computer with data, which are converted into moving images and sounds. The transposition of movement that takes place is fascinating in the sense that it makes the spectator aware of how much movement, rather than merely the outward appearance, plays an important role in the recognition and identification of human beings and animals. The big challenge is not only to control moving images in projections (whether on screens or in holograms) but also physical objects such as drones. For example, Airman (2018), the opening of the five-part performance Flirt with Reality (2018) by David Middendorp and his company Another Kind of Blue, is a fascinating duet

15

With this phrasing, we explicitly evoke Hans-Thies Lehmann’s (2006) concept of the postdramatic theatre.

742

C. Kattenbelt and R. Nelson

between the dancer and a dozen drones, which are controlled by the dancer in a motion capture suit. QR code: Airman: https://www.youtube.com/watch? v¼NuP4YhMzt3k

For Middendorp, the next step would be for the dancer and the drones to react to each other, thus creating interaction in the true sense of mutuality. Like Kandinsky, the stage compositions that Middendorp creates are also subject to vibrations, this time not from the human soul only, but also from the ubiquitous technology that shapes our current existence and therefore our (post)humanity.

Resonances of Brecht’s Dialectical Theater A number of contemporary intermedial practices are designed to ask questions of the experiencer, sometimes tacitly in the construction of the event and sometimes overtly, putting the participant directly on the spot. At the end of Blast Theory’s Uncle Roy is all around you, for example, participants are requested to mail a postcard to a complete stranger offering to help them, on request, over the next year.16 In Rimini Protokoll’s Best Before, participants are required to make a number of choices to construct an onscreen avatar (see below), but it becomes apparent in the event that the onstage performers might directly identify a participant’s seemingly secret choices.17 On occasion, they even ask individuals to identify themselves. Both these instances, we suggest, echo the Brechtian tactics of interruptus designed to encourage critical reflection through a shock of distanciation. As is well-known, “Rimini Protokoll” is itself dispersed – not so much the name of a theater company as an umbrella title, created in 2002, for the arts projects of Helgard Haug, Stefan Kaegi, and Daniel Wetzel. These projects range from theater to live art, from radio plays to sound design and video installation. In short, Rimini Protokoll seeks the means to the end of the project in hand and the Company has earned a reputation for the development of a Reality Trend (Theater der Zeit) theater initiative, though the term “theater” must be taken loosely in respect of the company’s overall strategies.18

ICA, London, June 2003: “Citywide” is the official title of the project. PuSH/Vancouver, Jan 2010; ICA London, June 2010. 18 The term “Reality Trend” marks a recent tendency in the arts for practitioners and cultural agents of all kinds, actively to intervene in the social process and to re-assert the role of the artist in this context. Documentary and “verbatim” theater are part of this trend, but it extends to a wide range of contemporary arts practices. 16 17

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

743

Created by Helgard Haug and Stefan Kaegi (with collaborators), Best Before is a theater event in that it is performed in recognizable theater spaces and there is a score to guide the events. But it involves more a computer game than a (written) play and there is no fixed script. There are experiencers: performers not actors; participant observers not an audience. Indeed Best Before is a both-and hybrid of a theater event, TV game show, and computer game. It comprises a multiplayer video game played by individuals in the group context of a public theater space. There are five onstage performers, who mediate between the virtual and actual spaces, facilitating the game to construct Bestland in the on-screen, virtual space. An intermittent back projection shows film of an urban landscape. Each experiencer is furnished with a game controller hand-set allowing him or her to activate their on-screen, on-stage avatar (the yellow and pink tennis-ball shapes) in the fictional non-space of Bestland. QR code: Best Before: https://www.rimini-protokoll.de/ website/en/project/best-before

The game involves making choices about constructing individual identity and a competition to gain wealth and power. The loose narrative frame takes avatars from cradle to grave over a couple of hours real-time with a range of choices, both individual and collective, on the way. Early on you choose your gender and establish basic characteristics by making either/or choices in moving your avatar either to the right or left of a dividing line running down the screen and through the actual stage. There is a count-down, cutoff point for making up your mind about each choice, similar to those used in television games shows. Icons are added to distinguish your avatar as a result of your choices. As a teenager, for example, you can opt to smoke or use heroin or break into your parents’ liquor cabinet. In your early twenties you can choose to have a partner and children and divorce. You can opt to legalize abortion and to ban immigration or to carry a gun. You proceed to choose a career path, a home (or not), and can earn currency (Bestos) through speculation. Best Before can claim to be truly interactive. It is fun to play and, in the early stages, there is much intermittent laughter. But the on-stage performers increasingly pose serious actual world questions as part of Rimini Protokoll’s “Reality Trend” initiative. You suddenly realize that your individual avatar profile has been saved electronically and that there have been many actual world ethical implications in your virtual world choices. Best Before reminds you that you can be tracked and made accountable for your choices, subverting internet intimacy by making public

744

C. Kattenbelt and R. Nelson

any private construction of identity. It draws attention to both the liberation and surveillance dimensions of the internet, and its potential for deception. Those who successfully accrue Bestos in the process of the game, for example, are not allowed to forget that the growing unemployed are not able to take advantage of market opportunities. These points are not explicitly made in Best Before by way of narrative or textual argument: the piece simply constructs an experience for you that raises questions and shocks you into thought. Such shocks of recognition mobilize a critical reflectiveness even while participating. The slow, and somewhat clunky, narration militates from the outset against full immersion in the sense of getting lost in the game. But, insofar as players are drawn into the competition to achieve success in Bestland, the revelatory shocks mobilize a broader awareness of what is at stake, which is further fueled by some of the narrative anecdotes that paint, not a world of success achieved through pursuit of individual aspiration, but rather a dystopic environment of displacement and unemployment set against an anonymous cityscape. By means of this intermedial mix, Best Before involves a curious both-and hybrid,19 a juxtaposition of the public and the private. The interrelationships are complex. Best Before breaks down the traditional theater’s stage-auditorium divide; it allows experiencers to be both on- and off-stage; it affords simultaneous inhabitation of both actual and virtual worlds; it draws attention to the mechanisms and implications of both individual and collective choices. It draws attention to the increased porosity of boundaries in digital culture between the actual, the virtual, and marked spaces of play (theater and games spaces). In sum, though engagingly entertaining, it involves numerous interruptus devices, which Brecht might have applauded; it points up numerous social contradictions, and even points in the direction of sociopolitical change. Like Best Before, Uncle Roy is all around you involves a kind of virtual game and it, too, is not strictly speaking a theater piece. Like Best Before, the event it mobilizes involves individual players as in a computer game but here actually located on the streets of city spaces. Furnished with a hand-held computer, players set out to find the elusive Uncle Roy whose situation is a mystery. They have 1 hour to complete the task. Instructions fed to them suggest that people they encounter may be actors who could assist them. But, having no clue as to who is who on the street, they might mistake members of the public for actors and vice versa. Everybody in their world can be constructed as a performer. Extending some sense of a collective-collaborative event, on-street players are helped or hindered by another set of players in a nearby media studio who undertake a virtual search for Uncle Roy. If they discover his location, they might then send messages to assist (or distract) the on-street players to get to their destination.

19 Hybridity and hybridization are concepts that are also used in this context (see Lavender 2016: 59–76).

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

745

The on-street players are inducted into the event and their actual and virtual identities. QR code: Uncle Roy: https://www.blasttheory.co.uk/projects/uncle-royall-around-you/

As in Best Before, the fun of the experience of playing the game is modified by increasing anxieties, if not riskiness, marked particularly at the end of the experience when you are asked to get into a car with a complete stranger in central London. Further, you are requested to make a commitment to a complete stranger, offering help at any time during a year. You are requested to mail your personal contact details on a postcard to this end. Uncle Roy’s tag line is “When can you begin to trust a stranger?” In these two examples, immersion associated with Wagner’s dramatic theater is replaced by immersion – to a greater or lesser extent – in a computer game. But the aim of neither piece is complete illusionism and indeed, as noted, tactics are built in to both pieces designed to shock experiencers out of any dreamworld. Though no specific political outcome is anticipated by these pieces, they demand a level of critical reflection in the manner of Brecht’s dialectical theater.

Endnote When artists today engage in their practices, they typically operate on a project basis and may be more or less conscious about how they do things. Unlike our intermedial prototypes, Wagner, Kandinsky, and Brecht, they are unlikely to have an overarching theory, let alone to write conceptual essays about their approach. Nevertheless, artists today are disposed to achieve certain kinds of affects rather than others and, to whatever ends they aim, principles of composition are still in play, even though, as our examples suggest, they may be less visible in complex outcomes. In this chapter, besides offering a long view of the ways in which mediums have been brought together, we have used our three key theorist-practitioners – who did consciously have principles – as a lens for reviewing practices and the issues involved in taking one approach rather than another. We mark a specific take on intermediality and illustrate a range of both-ands in thinking things together rather than apart, and showing how the whole sensorium of the bodymind, challenging the long-standing theory/practice divide, is increasingly engaged in a playfulness, which might, nevertheless, also be serious.

746

C. Kattenbelt and R. Nelson

References Adorno, Theodor W. 1973. In Ästhetische Theorie, ed. Gretel Adorno and Rolf Tiedemann. Frankfurt am Main: Suhrkamp. Auslander, Philip. 1997. From acting to performance: Essays in modernism and postmodernism. London/New York: Routledge. Barthes, Roland. 1977a. The death of the author. In Image, music, text, ed. and trans. Stephen Heath, 142–148. London: Fontana Press. Online available via https://monoskop.org/images/0/0a/ Barthes_Roland_Image-Music-Text.pdf. Accessed 10 Feb 2023. ———. 1977b. From work to text. In Image, music, text, ed. and trans. Stephen Heath, 155–164. London: Fontana Press. Online available via https://monoskop.org/images/0/0a/Barthes_ Roland_Image-Music-Text.pdf. Accessed 10 Feb 2023. Bay-Cheng, Sarah, Chiel Kattenbelt, Andy Lavender, and Robin Nelson, eds. 2010. Mapping intermediality in performance. Amsterdam: Amsterdam University Press. Benjamin, Walter. 1977 [1936]. Das Kunstwerk im Zeitalter seiner technischen Reproduzierbarkeit. Frankfurt am Main: Suhrkamp. English translation: https://monoskop.org/images/6/6d/ Benjamin_Walter_1936_2008_The_Work_of_Art_in_the_Age_of_Its_Technological_Repro ducibility_Second_Version.pdf. Accessed 10 Feb 2023. Boenisch, Peter. 2006. Aesthetic art to aisthetic act: Theatre, media, intermedial performance. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 103–116. London/New York: Rodopi Publishers. Bogatyrev, Petr. 1971 [1938]. Les signes du théâtre. Poétique 8: 517–530. Bolter, Jay David, and Richard Grusin. 1999. Remediation: Understanding new media. Cambridge, MA: MIT Press. Braidotti, Rosi. 2011. Nomadic subjects: Embodiment and sexual difference in contemporary feminist theory. New York: Columbia University Press. Brecht, Bertolt. 1978a [1930]. The modern theatre is the epic theatre (notes to the opera Aufstieg und Fall der Stadt Mahagonny. In Brecht on theatre: The development of an aesthetic, ed. and trans. John Willett, 33–42. New Delhi: Radha Krishna Prakashan. Online available via https://ia801405. us.archive.org/18/items/in.ernet.dli.2015.150164/2015.150164.Brecht-On-Theatre.pdf ———. 1978b. Appendices to the short organum. In Brecht on theatre: The development of an aesthetic, ed. and trans. John Willett, 276–281. New Delhi: Radha Krishna Prakashan. Online available via https://ia801405.us.archive.org/18/items/in.ernet.dli.2015.150164/2015.150164. Brecht-On-Theatre.pdf. Accessed 10 Feb 2023. Bürger, Peter. 1974. Theorie der Avantgarde. Frankfurt am Main: Suhrkamp. English edition: Bürger, Peter. 1984. Theory of the avant-garde. Trans. Michael Shaw; foreword: Jochen Schulte-Sasse. Minneapolis: University of Minnesota Press. Clough, Patricia T. 2008. The affective turn. Theory, Culture, and Society 25 (1): 1–22. de Smet, Jan. 2005. Webcamtheater in splitscreen. De Morgen, April 7. https://www.demorgen.be/ nieuws/webcamtheater-in-splitscreen~bed190cb/. Accessed 10 Feb 2023. Dixon, Steve. 2007. Digital performance: A history of new media in theater, dance, performance art, and installation. Cambridge, MA/London: MIT Press. Eco, Umberto. 1977. Semiotics of theatrical performance. Drama Review T73: 107–117. Elleström, Lars. 2010. The modalities of media: A model for understanding intermedial relations. In Media borders, multimodality and intermediality, ed. Lars Elleström, 11–48. Houndmills/ Basingstoke: Palgrave Macmillan. ———. 2021. The modalities of media II: An expanded model for understanding intermedial relations. In Beyond media borders, vol. 1: Intermedial relations among multimodal media, ed. Lars Elleström, 4–91. Cham: Palgrave Macmillan. Fischer-Lichte, Erika. 2008. The transformative power of performance: A new aesthetics. Trans. Saska Iris Jain. London/New York: Routledge. Fiske, John. 2010 [1987]. Television culture. London: Taylor and Francis Group.

29

Prototype Models of Intermedial Praxis (Wagner, Kandinsky, Brecht). . .

747

Greenberg, Clement. 1982 [1960]. Modernist painting. In Modern art and modernism: A critical anthology, ed. Francis Frascina, and Charles Harrison with the assistance of Deirdre Paul, 5–10. London: Paul chapman. First appeared as a Forum Lecture radio broadcast (Washington, DC: Voice of America). Also online available via: https://cpb-us-w2.wpmucdn.com/voices. uchicago.edu/dist/9/177/files/2007/10/Greenbergmodpaint.pdf. Accessed 10 Feb 2023. Gregg, Melissa, and Gregory J. Seigworth, eds. 2010. The affect reader. Durham: Duke University Press. Hegel, Georg Wilhelm Friedrich. 1986 [1835–1838]. Band 13: Vorlesungen über die Ästhetik I, eds. Eva Moldenhauer, and Karl Markus Michels. Frankfurt am Main: Surhkamp. Higgins, Dick, and Hannah Higgins. 2001 [1966]. Intermedia. Leonardo 34 (1): 49–54. Jenkins, Henry. 2006. Convergence culture: Where old and new media collide. New York: NYU Press. ———. 2013 [1992]. Textual poachers: Television fans and participatory culture. New York: Routledge. Kandinsky, Wassily. 1979 [1912 and 1923]. Über Bühnenkomposition [1912] and Über die abstrakte Bühnensynthese [1923]. In Kandinsky, essays über Kunst und Künstler, 49–61, 79–83. Bern: Benteli Verlag. ———. 2009 [1911/1912]. Über das Geistige in der Kunst insbesondere in der Malerei. Bern: Benteli Verlag, 1977. English translation: Concerning the spiritual in art. Trans. and intro. Michael T.H. Sadler. New York: Dover Publications. Kattenbelt, Chiel. 1994. The triad of emotion, action and reflection. Kodikas/Code: Ars Semeiotica 17 (1–2): 123–139. ———. 1995. Being there or not being there, that makes the difference. In Towards a pragmatics of the Audiovisual, ed. Jürgen E. Müller, vol. 2, 49–165. Münster: Nodus Publikationen. ———. 2006. Theatre as the art of the performer and the stage of intermediality. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 29–39. Amsterdam/New York: Rodopi Publishers. ———. 2010. Intermediality in performance and as a mode of performativity. In Mapping intermediality in performance, ed. Sarah Bay-Cheng, Chiel Kattenbelt, Andy Lavender, and Robin Nelson, 29–37. Amsterdam/New York: Amsterdam University Press. https://doi.org/10. 5117/9789089642554. Kattenbelt, Chiel, and Aneta Mancewicz. 2023. Intermediality in performance. In Routledge companion to contemporary theatre and performance, ed. Ralf Remshardt, 380–386. London/New York: Routledge. Künzig, Bernd. 1990. Richard Wagner und das Kinematographische. Eggingen: Edition Isele. Lavender, Andy. 2016. Performance in the twenty-first century: Theatres of engagement. London/New York: Routledge. Lehmann, Hans-Thies. 2006. Postdramatic theatre. Trans. and intro. Karen Jürs-Munby. London/New York: Routledge. Machon, Josephine. 2013. Immersive theatres: Intimacy and immediacy in contemporary performance. Houndmills/Basingstoke: Palgrave Macmillan. Manovich, Lev. 2001. The language of new media. Cambridge, MA/London: MIT Press. Massumi, Brian. 1987. A thousand plateaus. Capitalism and schizophrenia. Trans. Gilles Deleuze, and Felix Guattari. Minneapolis: University of Minnesota Press. Massumi, Brian (2008). The thinking-feeling of what happens: A semblance of a conversation. Université de Montréal, Montréal. Online available via: https://inflexions.org/n1_TheThinking-Feeling-of-What-Happens-by-Brian-Massumi.pdf McLuhan, Marshall. 1994 [1964]. Understanding media: The extensions of man. Cambridge, MA: MIT Press. Mukarovsky, Jan. 1974 [1934]. Zur Ästhetik des Films. In Poetik des films, ed. Wolfgang Beilenhoff, 119–130. München: Fink Verlag. Nelson, Robin. 2010. Introduction: Prospective mapping and network of terms. In Mapping intermediality in performance, ed. Sarah Bay-Cheng, Chiel Kattenbelt, Andy Lavender, and Robin Nelson, 13–23. Amsterdam/New York: Amsterdam University Press.

748

C. Kattenbelt and R. Nelson

———. 2022. Practice as research in the arts (and beyond). Principles, processes, contexts, achievements. Cham: Palgrave Macmillan. Nietzsche, Friedrich. 1899 [1888]. The case of Wagner: A musician’s problem. Being a letter from Turin, May 1888. In The works of Friedrich Nietzsche, Vol. III. Trans. Thomas Common. London: T. Fisher Unwin. Online available in a translation by Anthony M. Ludovici via: https:// www.gutenberg.org/files/25012/25012-pdf.pdf. Accessed 10 Feb 2023. Perelman, Bob. 1993. Parataxis and narrative: The new sentence in theory and practice. American Literature 65 (2): 313–324. Pfister, Manfred. 1988. The theory and analysis of drama. Cambridge, UK: Cambridge University Press. Rajewsky, Irina O. 2005. Intermediality, intertextuality and remediation: A literary perspective on intermediality. Intermédialités 6: 43–64. https://doi.org/10.7202/1005505ar. Rancière, Jacques. 2009. The emancipated spectator. Trans. Gregory Elliott. London/New York: Verso. Schröter, Jens. 2011. Discourses and models of intermediality. CLCWeb: Comparative Literature and Culture 13: 3. https://doi.org/10.7771/1481-4374.1790. Smith, Murray. 1995. Engaging characters: Fiction, emotion and the cinema. Oxford, UK: Oxford University Press. Szondi, Peter, and Michael Hays. 1983 [1956]. Theory of the modern drama, parts I–II. Boundary 2 11 (3), The criticism of Peter Szondi, 191–230. Online available via: https://www.jstor.org/ stable/303010. Accessed 10 Feb 2023. Vazsonyi, Nicholas. 2016. Chapter 1: The Plays’s the thing. In The total work of art: Foundations, articulations, inspirations, ed. David Imhoof, Margaret Eleanor Menninger, and Anthony J. Steinhoff, 21–38. New York/Oxford, UK: Berghahn. Wagner, Richard. 1983 [1849]. Das Kunstwerk der Zukunft. In Richard Wagner, Dichtungen und Schriften, Vol. 6, Reformschriften, ed. Richard Borchmeyer, 9–157. Frankfurt am Main: Suhrkamp. Wright, Elizabeth. 1989. Postmodern Brecht: A re-presentation. London/New York: Routledge.

Part III Intermedial Perspectives on Media in the Twentieth Century: New Mediascapes in a Growing World

Intermediality and Liveness at the Turn of the Twentieth Century

30

Mary Simonson

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality and/As Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Impulses at the Turn of the Century . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Media Combination: Theaters of Variety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Media Transposition: Re-presenting the Familiar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Reference: Refusing Medium Specificity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

752 753 756 756 759 762 765 766

Abstract

Though intermediality is primarily associated with digital and new media, the concept is equally productive in theorizing stage performance and other live art and entertainment. As Chiel Kattenbelt, Freda Chapple, Phillip Auslander, and others have argued, theatrical and other live performances also involve mediation: in these genres, too, it is through material forms such as stage technologies, human bodies, sets and costumes, language, and musical instruments, not to mention advertisements, programs, previews, and reviews that meaning is created. While scholars have employed intermedial frameworks to analyze live performances from various periods, particularly rich approaches have emerged in studies of late nineteenth and early twentieth century performance. Indeed, as Klemens Gruber and others have argued, intermedial practices might be understood to be rooted, albeit not fully deployed, in this era. This chapter surveys scholarship on intermedial performance at the turn of the twentieth century, focusing particularly on its invocation in interdisciplinary studies of both theater and pre- and early cinema technologies. At a historical moment in which emergent media and technologies were often placed in dialogue with, staged M. Simonson (*) Colgate University, Hamilton, NY, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_35

751

752

M. Simonson

alongside, or integrated into established genres and performance forms, intermediality serves as a compelling lens through which to engage the evolving artistic and medial landscape, and the broader cultural practices that surrounded it. Keywords

Performance · Hybridity · Theater · Live · Early twentieth century

Introduction Intermediality has often been associated with digital media, with the term deployed most frequently in discussions of the Internet, digitization, and turn of the twentyfirst century media practices and developments. And indeed, our ever-emergent digital media phenomena fairly beg for intermedial analysis, given the extent to which they reference, borrow from, and define themselves in relation to extant media forms. Further, as media scholar Mikko Lehtonnen has noted, the accelerated circulation of technologies and texts as a product of globalized capitalism feed intermedial experimentation and intensify its expression (Lehtonen 2001: 78). As numerous scholars have noted, though, the concept of intermediality is equally productive in theorizing analog media, and more specifically, stage performance and other modes of live art and entertainment. Though live performance is differentiated from many media in its inability to record, it too is laced with mediation. It is through material forms such as stage technologies, human bodies, sets and costumes, language, and musical instruments, not to mention advertisements, programs, previews, and reviews, after all, that meaning is created in live performances (Kattenbelt 2010: 141). Indeed, as a medium through which other media are effectively broadcast, theater might productively be considered as both, to borrow from media scholar Peter Boenisch, a “media technology” and a “semiotic practice” that weaves together and performs “the content and cognitive strategies of other media.” (Boenisch 2006: 113). In recent years, intermediality has been embraced as an especially useful tool for understanding late nineteenth and early twentieth-century media and performance. That a concept associated with the turn of the twenty-first century is equally applicable to this earlier period makes sense. The turn of the twentieth century in the United States saw a similarly rapid adoption of new technologies – recorded sound and amplification, photography and moving images, and radio and other modes of telephony, to name a few – and similar anxieties about the impact of these new phenomena on existing artistic and cultural institutions, industry, education, and human consciousness broadly. Just as turn of the twenty-first-century artists plumbed familiar media for elements and practices that might be integrated into their explorations of new digital media phenomena, so too did early twentieth-century performers and creators borrow nineteenth-century stage traditions, musical and visual artworks, and literary texts as inspiration (and often source material) for

30

Intermediality and Liveness at the Turn of the Twentieth Century

753

their own experiments. Emergent early twentieth-century technologies also accelerated the flow of sights, sounds, and capital that Lehtonen associates with increased intermedial experimentation and expression. This chapter opens with a review of scholarship that recognizes intermediality as a productive lens and framework through which to study and theorize live and hybrid performance as well as recorded media. It then turns to survey the rich and expanding body of scholarship that employs intermediality to understand late nineteenth- and early twentieth-century art and entertainment in the United States, particularly the complex and varied relationships that arose between live performance practices, elements, and aesthetics, and those associated with new technologies such as pre-cinematic devices, photography, recorded sound, and moving images. At a historical moment in which emergent media and technologies were often placed in dialogue with, staged alongside, or integrated into established genres and performance forms, I argue, intermediality serves as a compelling lens through which to engage the evolving artistic and medial landscape, and the broader cultural practices that surrounded it.

Intermediality and/As Performance Intermediality, as this very volume amply suggests, is less a concept than a descriptor that can be applied to a vast range of situations, realities, and approaches. Scholar Irina Rajewsky captures this broadness, situating intermediality as phenomena that “take place between media” or somehow “have to do with a crossing of borders between media” (Rajewsky 2005: 46). In their seminal 2006 collection Intermediality in Theatre and Performance, editors Freda Chapple and Chiel Kattenbelt suggest a subtle reorientation this framework, defining intermediality as the blurring or crossing not only of media boundaries but of “generic boundaries,” and, even more expansively, of different worlds. In doing so, Chapple and Kattenbelt open space for considering performance genres and practices – theater, but also opera, dance, live musical performances, vaudeville, and more – as capable of intermedial interactions. The intermedial, they write, “is a space where the boundaries soften – and we are in-between and within a mixing of spaces, media, and realities. Thus, intermediality becomes a process of transformation of thoughts and processes where something different is formed through performance” (Chapple and Kattenbelt 2006: 12). Alongside their expansions of the term intermediality to include performance genres and practices, scholars have also compellingly argued that these genres and practices themselves might be understood as media. In their book Remediation, media scholars Jay David Bolter and Richard Grusin famously define media as that which is able to remediate: to “appropriate the techniques, forms, and social significance of other media and attempt to rival or refashion them” (Bolter and Grusin 1999: 65). As Peter Boenisch and others writing in Chapple and Kattenbelt’s volume note, theater (and stage performance more broadly) certainly incorporates and broadcasts – that is, remediates – other arts and media. Theater, Boenisch writes,

754

M. Simonson

“incorporates, spatializes, and disseminates in sensorial terms (thus performs) the contents and cognitive strategies of other media by creating multiple channels, and a multimedia semiotic and sensoric environment” (Boenisch 2006: 113). Moreover, as Kattenbelt notes in Mapping Intermediality in Performance, another groundbreaking collection he co-edited with Sarah Bay-Cheng, Andy Lavender, and Robin Nelson, theater and other live performances are not only a “material embodiment which, when the performance is over, deposits cultural expression into cultural memory,” but they also create and are constituted by a rich trove of media traces: posters, programs, previews and reviews, and merchandise, to name a few (Kattenbelt 2010: 141). Indeed, Kattenbelt suggests that theater might more accurately be considered a hypermedium, given its ability to integrate and present so many arts and media (Kattenbelt 2006: 32). Perhaps ironically, embedded in these reconsiderations of theater and other modes of live performance as media is an acknowledgment of intermediality as an inherently performative phenomenon. Through juxtaposition or integration of various media and genres, referencing of techniques associated with one medium in another, or reimagining of works in a new medium or genre (among other possibilities), intermediality effectively stages the media themselves for audiences in a process Kattenbelt describes “conscious self-presentation” (Kattenbelt 2010: 29). Medial mechanics, possibilities, and limitations are revealed, and audiences are invited to consider the ways in which elements and practices specific to a medium have been altered, transferred, and reimagined, or, as Boenisch writes, to dwell in the “gaps, splits, and fissures. . .detours, inconsistencies, and contradictions” (Boenisch 2006: 115). As this reframing emphasizes, we might think of intermediality not as specific configurations of media and technology, but rather, to quote Boenisch, “an effect created in the perception of observers that is triggered by performance – not simply by the media, machines, projections, or computers used in performance” (Boenisch 2006: 114). Intermediality, then, is less a technological status than an experience collaboratively realized by performers and spectators through what Kattenbelt describes as the “exploration of new dimensions of perception and experience,” or simply, the “resensibilisation of perception” (Kattenbelt 2010: 35). The excellent scholarship collected in volumes like Intermediality in Theatre and Performance and Mapping Intermediality in Performance tends to focus on contemporary theater and live performance. Given that intermediality has so often been associated with new media, the digital, and other recent phenomenon, this is logical. Increasingly, however, scholars have argued that the concept of intermediality, particularly as formulated in these collections, is equally useful for understanding nineteenth and early twentieth-century artistic practices and entertainment, as well. As I have argued elsewhere, applying an intermedial lens to cultural productions in this period helps to foreground the ubiquity of various modes of intermediality in this moment (Simonson 2013). Popular modes of performance in this period accommodated – if not necessitated – a range of intermedial practices, including translation or adaptation of works in new genres and mediums (media transposition), experimentation with techniques associated with one genre or medium in another (intermedial reference), and the integration of multiple types of recorded, live, and

30

Intermediality and Liveness at the Turn of the Twentieth Century

755

hybrid media into a single production (media combination) (Rajewsky 2005). Industry structures, too, facilitated intermedial practices: creators and performers often survived by working in and between multiple different genres and media. As they moved between spheres, so too did specific techniques, characters, musical settings, and other elements. That intermediality was a dominant aesthetic in this moment is hardly surprising. The late nineteenth and early twentieth century saw the introduction and development of numerous technologies, including telephony and radio, moving images systems, recorded sound and amplification, and so on. These emergent media, like more recent digital technologies, routinely borrowed, adapted, and dialogued with elements and practices of more familiar genres of art and entertainment, re-presenting them using new technological processes, defining themselves first in relation to and then as distinct from extant media forms. In their essay “A Medium is Always Born Twice. . .,” André Gaudreault and Philippe Marion describe this stage as one of “irrepressible” or “spontaneous” intermediality: a new technology by necessity “insinuates itself into the syntagmatic chain of culturally established genres and media representations” (Gaudreault and Marion 2005: 4, 12). So central is intermediality to the emergence of a medium that film scholar Richard Altman suggests that intermediality should be understood as “a historical stage, a transitional state in the course of which a form in the process of becoming a full-fledged medium finds itself still torn between various existing media, to the extent that its own identity is not yet resolved” (Altman 1999: 38, quoted in Gaudreault 2011: 68). As artists and innovators continue experimenting with new technologies, of course, they develop specific formal strategies, aesthetics, and practices; concurrently, economic and industrial structures are organized, further solidifying the emergent medium’s identity and cultural meaning. Early cinema, Gaudreault and Marion note, serves as a useful example of the process. Early moving image experiments frequently used familiar theatrical entertainments– magic shows, fairy plays, comedies – as source material, borrowing and re-presenting stage genres onscreen. Quickly, though, filmmakers began developing visual language, narrative approaches, and audiovisual aesthetics that capitalized on the possibilities of the new technology, and over time, the filmic medium was constituted. Similarly, Gaudreault and Marion point out that when still photographic technology emerged, it initially used staged poses, expressions, and other pictorial practices that were common in drawing and painting. Yet as camera users began to understand the new technology’s distinct capabilities, new practices began to emerge, and with them, a sociocultural notion of what photography was and how it differed from extant modes of visual representation (Gaudreault and Marion 2005: 10). Certainly, Gaudreault and Marion emphasize, intermediality does not disappear once mediums are constituted. Instead, it ceases to be spontaneous – enacted by default or necessity – and instead becomes an expressive possibility, which the medium “negotiates in its own fashion, in interaction with its own potential” (Gaudreault and Marion 2005: 13). Of course, this sort of intermedial experimentation is hardly a one-way street: early twentieth-century artists and performers working in theater, dance, music, and visual arts eagerly experimented with integrating new technologies – and eventually,

756

M. Simonson

these newly constituted mediums – into their own creative practices, as well. Early twentieth-century theater productions, for instance, routinely incorporated these new technologies sparking lasting innovation (and more short-lived trends). Even those artists who distanced themselves from emergent media might be understood as contributing to the period’s intermedial impulses. As media scholar Klemens Gruber suggests, the emergence of photography and other technologies of reproduction, led some artists working in visual art, theater, and beyond to reject qualities associated with these new verisimilar media: the representation, the mimetic, the referential (Gruber 2010: 248). Instead, they sought to define semiotic and expressive practices specific to their mediums: color and texture in painting, space in theater, and movement in dance. However, this rejection, Gruber notes, also renewed awareness of media specificity and boundaries, instilling in artists the confidence to experiment with and cross these boundaries even while “simultaneously sustaining their independence” (Gruber 2010: 251).

Intermedial Impulses at the Turn of the Century It is in this context that scholars working across and between multiple disciplines – theater, music, dance, performance studies, film, and media studies – have increasingly invoked intermediality in discussions of late nineteenth and early twentiethcentury media objects and performances. Much of this scholarship is quite narrowly focused, examining specific creators or even specific events. However, taken together, these studies point to intermediality’s ubiquity as both a strategy and aesthetic. This body of scholarship also highlights the inherent intermediality of particular entertainment genres and performance practices and gestures toward the extent to which industry structures and economic realities of the period encouraged, if not relied on, intermedial phenomenon. The survey of literature that follows is certainly not exhaustive; rather, I foreground some of the particularly compelling sites in which intermedial aesthetics and practices were at play in this period, and the rich scholarship about these phenomena.

Media Combination: Theaters of Variety Vaudeville shows and revues were fundamental institutions of the American entertainment scene throughout the late nineteenth and early twentieth century. Often highly commercialized, these performances brought together a diverse collection of acts, more or less carefully curated: performances of opera arias might segue into acrobatic acts, followed by a brief narrative skit laced with popular songs, a dance number, and then a pair of comedians might take the stage. Over time, pre-cinema technologies like lantern slides, and then short films, and even recorded music were also integrated into shows. Vaudeville shows tended to feature a series of discrete acts that appeared in succession, ordered based on the type and relative appeal of each act; as shows traversed national circuits of theaters, performers and acts could

30

Intermediality and Liveness at the Turn of the Twentieth Century

757

easily be swapped in and out to accommodate schedules and audience preferences. Revues, which emerged in the first decades of the twentieth century with productions like the Ziegfeld Follies and the Shubert Brother’s Passing Show, also consisted of a diverse series of acts – dance numbers, musical performances, comedy, drama, and so on – but tended to be more carefully crafted, with through-running themes and unifying aesthetics, a cast of performers who reappeared throughout the evening, and lavish sets and spectacular visual effects. Regardless, both of these types of popular performance (and those shows that fell somewhere between) were defined by their combinations and juxtapositions of different media, genres, and modes of expression – in short, by their intermediality. In recent years, scholars have begun using the term intermediality to describe and analyze types of entertainment and predecessors that used similar variety-based formats, such as the minstrel show. Body Knowledge: Performance, Intermediality, and American Entertainment at the Turn of the Twentieth Century, for example, examines several theatrical revues, including J. Leubrie Hill’s My Friend from Kentucky, also known as the Darktown Follies, Florenz Ziegfeld Jr.’s Follies of 1914, and the Shubert Brothers’ Passing Show of 1913 as intermedial performance: in each case, musical performances, theatrical acts, dance numbers, and more were woven together to create spectacular productions (Simonson 2013). Moreover, as the author emphasizes, these shows frequently interpolated or referenced popular songs, narratives, and performers, at times presenting them in unexpected genres or media. Further, songs, acts, and dances were often traded between shows, “reappearing in new venues and distinct contexts, staged by different performers for very different audiences” (Simonson 2013: 3). These sorts of complex flows of material and aesthetics across bodies, genre and medial boundaries, and space and time become particularly visible through an intermedial lens. Similarly, in her article, “Intermediality of Hunger, Intermediality of Effect: Two Commercial Models Developed in Early Twentieth Century New York City,” film and media scholar Sabine Haenni points to the intermediality of vaudeville and variety, noting that these types of shows enabled attendees to engage in various ways with performances and media objects, and in doing so, actively positioning audiences “inbetween media” (Haenni 2012: 405). Yet Haenni is quick to note that the reasons for engaging intermedial aesthetics varied across performances and venues, and offers a useful framework for considering these differences. Examining the variety entertainments staged for working-class Jewish immigrants in theaters like the Houston Hippodrome on New York’s Lower East Side, Haenni argues that the intermedial impulse was driven by economics, or what she calls an “intermediality of hunger.” By integrating as many media forms and genres into their shows as possible – and by staging acts in both Yiddish and English – producers sought to attract the largest, most diverse audience possible. As Haenni writes, in these venues, “the performance of intermediality. . .becomes a crucial vehicle. . .that not only appeals to but helps create a heterogeneous audience and public culture” (Haenni 2012: 409). However, at the more luxurious theaters uptown in Times Square, the intermedial impulse was driven as much by aesthetics as by the market. Producers knit together multiple genres and media in order to offer audience immersive, unified performances that addressed all

758

M. Simonson

senses. This “intermediality of effect,” as Haenni calls it, was intended to “multiply rather than standardize the range of possible spectatorial experiences” (Haenni 2012: 414). And as Haenni points out, the sort of immersive, unified experience was replicated at an architectural level: single buildings in this neighborhood often contained multiple theaters and other venues that presented complimentary entertainment forms – legitimate theater, revues and variety, silent film, and so on. In doing so, these venues supplied audiences with a continuous stream of varied yet resonant options. This is not to say, of course, that the impresarios of Times Square were immune to economic pressures; these performances, too, were designed to draw large audiences. Nor does Haenni intend to imply that performances on the Lower East Side were necessarily less valuable aesthetically than those taking place to the north. Rather, she understands the two phenomena as always overlapping, concluding, “the intermediality of effect was literally erected on top of the intermediality of hunger” (Haenni 2012: 417). As numerous film and theater scholars have noted, the American film industry emerged in close proximity to this early twentieth-century vaudeville and variety scene, and thus, was bound up in this intermedial web. As film scholar Janet Staiger notes, “film as a business and an art was never isolated from other entertainments” (Staiger 2004: 127). Not only did pre-cinema technologies like the magic lantern and early moving pictures find one of their first homes as part of variety entertainments, but many silent film theaters interpolated both variety show aesthetics and actual acts into their exhibitions. By the 1920s, the massive movie palaces in many American downtowns offered elaborate “presentation programs” that cobbled together a series of shorts (a newsreel or current events magazine, a scenic, educational films, a cartoon, a comedy) and the week’s feature film, interspersed with live musical numbers, ballets, comedy skits, and other stage acts. Though much of the scholarship on cinema of this period does not actually use the term intermediality, it nonetheless describes synergies and flows across media borders; indeed, to ignore this is, as Staiger puts it, to “work with blinders on” (Staiger 2004: 127). Film scholars including Tom Gunning and Miriam Hansen, for example, have convincingly argued that films of this era can only be understood in conjunction with the stage presentations and programs that accompanied them: the music, dance, and theatrical moments that audiences witnessed doubtlessly impacted both their experience of the specific films with which these stage acts were paired, and more broadly, their experience of film as a medium (Gunning 2004; Hansen 1991; Musser 1994; Paul 2016). As Hansen writes in Babel and Babylon: Spectatorship in American Silent Cinema, the stage performances effectively lent the films “the immediacy and singularity of a one-time performance,” bestowing the geographic and temporal specificity associated with live performance to the filmic medium (Hansen 1991: 93–4). More recently, scholars have explicitly used the framework of intermediality to examine film exhibition in this period. In his excellent book American Showman: Samuel ‘Roxy’ Rothafel and the Birth of the Entertainment Industry, for example, Ross Melnick traces the career of legendary American theater manager and film exhibitor Roxy Rothafel as both an example and a driver of media convergence and

30

Intermediality and Liveness at the Turn of the Twentieth Century

759

integration in the United States in the 1920s and 1930s. Like other exhibitors of the period, Rothafel integrated film with stage performances, music, and other material, but he also eventually began hosting radio broadcasts of his presentation programs with live narration, as well as “backstage” performances by the theater’s stage performers, punctuated with conversation and commentary. The result was an intermedial feast, with performers and performances migrating from stage to screen to airwaves. Analyzing these multimedia events, Melnick joins Hansen, Gunning, and others in pointing toward the way each of these elements “framed, enhanced, subverted, complicated, and in many other ways altered the original meaning” of the acts and works alongside it (Melnick 2012: 23). Yet Melnick goes a step further, conceptualizing of film presentation programs not just as a descendent of the variety format, but as a “unitary text”: a “collective textual event” in which aesthetics drawn from “classed and disparate arenas” were transformed “into symbiotic elements of a convergent entertainment industry” (Melnick 2012: 14–15).

Media Transposition: Re-presenting the Familiar The generation of intermedial events through media combination and juxtaposition, as in revues and film presentation programs, was hardly the only in which late nineteenth- and early twentieth-century audiences experienced intermediality. Perhaps even more ubiquitous was the transit of narrative tropes, characters, and elements from one medium to another. As mentioned earlier, these sorts of adaptations were supremely practical: not only was restaging familiar, and often well-loved works a quicker and more cost-effective way to introduce new media technologies to audiences, but doing so was also understood as a means of imbuing these new technologies with the cultural capital and status of established genres and media. Conversely, too, transposing works across media boundaries promised to reimagine well-known works, infusing them with excitement and energy, and making space for new interpretation. The body of relevant literature on media transposition and adaptation more broadly is vast; in addition to comprehensive frameworks for theorizing adaptation, such as Linda Hutcheon’s seminal A Theory of Adaptation (2006) and collections like the Oxford Handbook of Adaptation Studies (2017), there are many studies focused on affinities between specific media, as well as common “routes” of media transposition in the late nineteenth and early twentieth centuries. A particularly robust body of scholarship exists on adaptations of novels and other literary works for film, as well as transpositions from the stage to the screen. While much of this scholarship focuses primarily on textual relationships, scholars have increasingly called for more nuanced attention to medial boundaries and relationships. Writing on films by D.W. Griffith, for example, film scholar Rick Altman has critiqued the tendency to compare film narratives with the novels on which they were originally based while ignoring the fact that these novels were often first adapted theatrically, as well as the fact that the film adaptations that followed were often more closely connected to these theatrical versions than to the novel. By skating over stage adaptations and other popular adaptations, Altman argues, we

760

M. Simonson

“abandon the opportunity to understand what is going on beneath and within the classical aspects of Hollywood narrative” (Altman 1996: 158). More recently, musicologist Daniel Goldmark has excavated the rich web of media transpositions surrounding the film The Jazz Singer (dir. Alan Crosland, 1927). Examining how the narrative’s “manner of telling” and soundscape shifts from the original short story to stage adaptations to the film, to the novelization of the film, and then on to radio plays, Goldmark makes a compelling argument about the extent to which Al Jolson’s career and signature performances were intertwined with different creators’ intermedial reimaginings (Goldmark 2017). Magic lantern shows have also been identified as an important site of media transposition in this period, thanks to a flurry of recent scholarship on the form. A mass-produced pre-cinema technology that involved the projection of still images painted or printed on transparent plates, often in conjunction with narration, music, and eventually even moving image projection, magic lantern shows frequently took up familiar narratives and imagery. In her essay “The Lantern Image between Stage and Screen,” media scholar Artemis Willis offers a compelling example, comparing magic lantern versions of Uncle Tom’s Cabin and Ben-Hur as staged by late nineteenth-century lanternist Joseph Boggs Beale, with contemporary theatrical and film versions of each. Lantern shows, Willis notes, often abridged narratives and focused particularly on “known moments,” trading on the fact that audiences would almost certainly be familiar with stories and characters from other media and performances (Willis 2018: 242). These shows also frequently incorporated reproductions of paintings and other art; as media scholar Valentine Robert argues, by “spectacularizing” these paintings and “transforming established images into performed images,” these shows radically altered the way audiences saw and understood these works of art (Robert 2014: 283). While the magic lantern is, as Willis writes, a “medium in its own right, with stylistics, codes, and a language of its own,” it is simultaneously a “translator of other media. . .each of which has its own set of associations” (Willis 2018: 242). Similarly, numerous scholars have explored the unlikely affinity of silent film producers for opera’s narratives and characters. In The Opera Singer and the Silent Film, for example, scholar Paul Fryer points to opera’s ready-made plots and musical scores, as well as its spectacular visual possibilities, compelling characters, and heightened emotions as elements that made the genre attractive source material for silent film (Fryer 2006). Tracing the careers of four opera singers who made their way into silent film, Fryer emphasizes the extent to which opera’s gestural acting styles were also well-suited for the limited camera movement and long shots of early silent film. Musicologist Michel Grover Friedlander, in Vocal Apparitions: The Attraction of Cinema to Opera, acknowledges these practical motives for silent film adaptations of opera as well, before considering the more paradoxical reasons for the affinity, particularly the extent to which we might understand silent film as both emphasizing and attempting to compensate for its own vocal lack through its embrace of opera (Grover-Friedlander 2005). More recently, scholars including Jennifer Fleeger have called for examination of the phenomenon within a more overtly intermedial framework. As she notes in her essay “Projecting an Aria,

30

Intermediality and Liveness at the Turn of the Twentieth Century

761

Singing the Cinema,” considerations of silent film’s embrace of opera must also take into account the variety of other sites and forms in which contemporary audiences encountered operatic elements: as arias performed out of context or available on audio recording, as radically altered stage adaptations, as instrumental excerpts performed in homes, or via radio broadcasts, among other possibilities – the “people, places, and practices” that surrounded specific texts at particular moments in time (Fleeger 2008: 123). As many of these studies hint, the ubiquity of media transposition in this period must be examined in relation to the tendency for performers, directors, and other creators to work in multiple mediums and genres over the course of their careers. As theater directors moved into silent film, for example, they took with them intimate knowledge of their stage productions to reprise onscreen. Similarly, when opera singers and stage performers were offered film contracts, it was often for roles in their repertoire. The fluid movement of personnel across medial borders, in short, facilitated the fluid movement of narratives, characters, and productions across those same borders. As Paul Fryer and others have noted, American opera singer Geraldine Farrar appeared in a film version of her well-known opera vehicle, Carmen (composed by Georges Bizet), and singer Mary Garden’s filmic output included a version of the opera Thaïs (Fryer 2006; Simonson 2012). A number of dancers also made the leap from stage to screen: in the fall of 1915, for example, premiere Russian ballerina Anna Pavlova signed a contract with Universal to appear in an adaptation of D.F.E. Auber’s opera La Muette de Portici while simultaneously launching a national tour of her ballet company and the Boston Opera Company during which they regularly performed the opera. Pavlova and her dancers quite literally moved from stage to screen: much of the film was shot in Chicago’s San Souci Park in between the company’s matinee and evening performances at the nearby Midway Gardens. It is doubtless that the dancers’ experiences on set shaped their performances on stage, and that the opera production’s aesthetics made their way into the film (Simonson 2013: 161–188). As this case illuminates, late nineteenth- and early twentieth-century performers seem to have understood their experiments with new media – and the process of media transposition more broadly – as a site of bidirectional influence and interdependence. In his essay “The Actor as Intermedialist: Remediation, Appropriation, Adaptation,” Ralf Remshardt also points to the extent to which intermediality in the late nineteenth and early twentieth centuries was generated by the movement of actors across medial boundaries, as well as the very different ways in which these “inter-medial voyagers” understood the possibilities of their journeys (Remshardt 2006: 42). Some actors, like Sarah Bernhardt, he argues, made films primarily to cement their stardom on the stage. She chose to reprise successful stage roles without adjusting her acting style in any significant way. Others performers, like Elenora Duse, used film to experiment with new approaches, creating what Remshardt calls a “hybridized cinematic derivative” of her stage style (Remshardt 2006: 49). Strikingly, both strategies draw attention to theater and film as media; even in refusing to adapt to film, Remshardt notes, Bernhardt made film visible “as a secondary medium through which theatre is mediated”; in both cases, moreover, theater is rendered

762

M. Simonson

“medial in a way that it had not previously been. . .film made theater’s theatricality visible, or its putative immediacy visible as theatricality” (Remshardt 2006: 49, 41). As a number of scholars have noted, in addition to considering specific performances as intermedial, it can be useful to understand the careers of late nineteenthand early twentieth-century performers as inherently intermedial. Writing about actor Lydia Borelli’s transition from stage to screen, for example, scholar Ivo Blom, describes an early film, Love Everlasting (dir. Mario Caserini, 1913), in which she is cast as an actress and appears onstage in two of her most famous stage roles; in one of these performances-within-the-film, she even wears costumes from the stage production. These intermedial references, which Borelli or the scriptwriters must have consciously chosen to incorporate, effectively “bridge the gap between stage and screen, by using the iconographic tradition shared between these two media” (Blom 2014: 32). Similarly, Nicholas Sammond has used the framework of intermediality to understand the careers of early twentieth-century American blackface minstrel performers Moran and Mack, and minstrel performance more broadly, noting that it was the movement of these acts between vaudeville and review stages, audio recordings and radio, and silent and eventually sound film and moving between the stage that “produc[ed] and stabiliz[ed] early twentieth-century racial formations” (Sammond 2015: 143). Understanding art and entertainment culture in this period, in short, requires us to consider, to borrow from Sammond, “the layering of persona and self, voice and commodity, that all performers in this period experienced and produced as they transited between media” (Sammond 2015: 144).

Intermedial Reference: Refusing Medium Specificity In addition to re-staging familiar plots, roles, and other elements in new mediums and genres, late nineteenth- and early twentieth-century creators and performers also frequently deployed specific characteristics, aesthetics, and practices of one medium in another. While adaptations provided ready-made material for emerging media, experiments in intermedial reference, to borrow Irina Rajewsky’s term, were often less straightforward: elements and strategies characteristic of one medium or genre would be enacted in another. This process frequently served as a crucial means of defining the possibilities – and as importantly the limits – of both emergent and more familiar genres and mediums. Frequently, media objects and performances experimented with both adaptation and intermedial reference simultaneously. Magic lantern shows offer a prime, though by no means unique, example. As media scholars have uncovered, lanternists devised a number of strategies to imbue the still images they projected with the movement associated with live performance and particularly, with moving images. In addition to incorporating music and narration, lanternists occasionally used multiple lanterns to create what media scholar Ludwig Maria Vogl-Bienek calls “dissolving views”: the image from one lantern would slowly fade as the image from another appeared, much as in moving images (Vogl-Bienek 2014: 223). As early

30

Intermediality and Liveness at the Turn of the Twentieth Century

763

moving picture technologies became available, too, lanternists would integrate moving images into their shows, so that the still images from lantern slides could not only dissolve into other slides, but also into short sequences of moving images, then back into another still slide, and so on. In her examination of the use of paintings and other extant art within lantern shows, Robert also highlights this strategy. Lantern projections of still images of painting (and at times, photographs of actors posing in reproductions of paintings) were overlaid with short sequences of moving images based on the paintings, projected by kinematographs and other early moving picture devices. The result were still images that suddenly seemed to come to life, “transforming imagery into performance” (Robert 2014: 286). While the magic lantern slides themselves were not technically adopting the movement of early film in moments like this, audiences were able to experience a sort of slippage from one medium to the other that created a similar effect: the projection screen became an “infrastructural element connecting traditional and new visual media,” to quote Vogl-Bienek (Vogl-Bienek 2014: 226). Much scholarship has highlighted the frequency with which early cinema adopted and transformed theater aesthetics and elements, particularly as filmmakers sought to define – and indeed, conceptualize of – film as a medium. Indeed, as film scholar Charles Musser writes, “cinema is a form of theatrical entertainment,” and thus, should be examined and understood in relationship with stage histories (Musser 2004: 105). Intermedial explorations of film and theater often reference Nicholas Vardac’s now classic book Stage to Screen: Theatrical Method from Garrick to Griffith, in which he positions cinema as an heir to theater, arguing – without using the term intermediality, of course – that early cinema drew heavily upon the aesthetics of stage melodrama, namely, its aspiration to pictorial realism, and eventually was able to manifest this realism to an extent not possible onstage (Vardac 1949). Theatre to Cinema: Stage Pictorialism and the Early Feature Film, co-authored by scholars Ben Brewster and Lea Jacobs, offers a corrective to this somewhat teleological narrative, while highlighting the extent to which early cinema strove to emulate the stage. As they argue – again, without using the term intermediality – key elements and aesthetics of nineteenth-century theater, including gestural acting and the use of pictorial effects in staging, were borrowed and adapted to suit the capabilities of film technology (Brewster & Jacobs 1997). Film’s intermedial references to a number of specific stage genres have also been explored. Media scholar Dan North has argued that early trick films by Georges Melies and other filmmakers borrowed aesthetics and strategies from nineteenth- and early twentieth-century magic shows. At times, these films even reproduced specific tricks that magicians performed onstage, but using film-specific techniques including cuts, splices, and stop action substitution to achieve the illusion. As North argues, Melies and his fellow filmmakers effectively “re-mediat[ed] his stage illusions with witty embellishments” (North 2007: 184). Other scholars have pointed to continuities between these early trick films and the stage tricks used in the fairy plays ( féeries) regularly performed in France in the late nineteenth century: as Frank Kessler and Sabine Lenk have argued, for example, not only did film tricks borrow setups and mechanics from the stage but they also sought to create the same sort of

764

M. Simonson

expectations and impressions among spectators, balancing the need for illusions to be convincing but also transparent enough that viewers could appreciate the trickery (Kessler and Lenk 2019). Early film’s deployment of aesthetics drawn from nineteenth-century movement and posing practices like tableaux vivants, early modern dance, and Delsartism, in which poses and gestures conveyed specific meanings and emotions, have also garnered significant attention. In Modernism’s Mythic Pose: Gender, Genre, Solo Performance, for example, dance scholar Carrie Preston demonstrates the extent to which the movement principles associated with Delsartism made their way into silent film. Not only is this gestural language visible in the performances of the many female film stars who studied Delsartism (or were exposed to similar techniques outside of formal study), but tableau-like posed scenes reminiscent of these practices were also regularly used by both American early filmmakers including D.W. Griffith and eventually, Russian filmmakers such as Lev Kuleshov as well (Preston 2011). Indeed, as film scholar Daniel Weigand has found, early films often incorporated illusions and ambiguous moments that “hover between volume and flatness,” mimicking the way in which tableaux vivants invite audiences to marvel at the “astonishing resemblance” of three-dimensional performers to familiar paintings (Wiegand 2018: 26). While such techniques were particularly common in the sorts of early trick films discussed above, Wiegand also highlights several film scenes in which models pose right next to a canvas on which an artist is putting the final touches on their portrait. Like tableaux, Wiegand writes, “such films addressed a scrutinizing, maybe even suspicious gaze that enjoyed exploring visual paradoxes and shifts in perception” (Wiegand 2018: 34). As a number of scholars have noted, the intermedial relationships between film and dance in the late nineteenth and early twentieth centuries were complex and multi-direction: while filmmakers often incorporated aesthetics drawn from contemporary movement and dance practices, as described above, dancers also deployed aesthetics that many audiences associated with new technologies like the moving image and even electricity. Indeed, filmmakers frequently turned to dancers as subjects precisely because their choreography showcased film’s capacity to represent movement. In her book DanceFilm: Choreography and the Moving Image, dance scholar Erin Brannigan writes compellingly about the productive co-existence and overlapping impulses between early cinema and early modern dance. Both media, she argues, presented bodies moving through space and time, and helped audiences develop ways of seeing and understanding motion. Loïe Fuller serves as a prime example: in her performances, she swirled large sheets of white fabric around her, alternately appearing and disappearing within it, as colored light and patterns were projected from magic lanterns (and at times reflected off of a glass floor beneath her) onto the drapery. Some scholars, including Tom Gunning, understand Fuller’s dance style and performance strategies as akin to the attraction mode of much early cinema; the “technological art of motion” presented in her performances, he writes, also reappears in avant-garde film in the 1920s (Gunning 2003: 85). Brannigan goes further, arguing that Fuller’s unique combination of choreography and emergent technologies effectively surpassed the capabilities of early cinema. Onstage,

30

Intermediality and Liveness at the Turn of the Twentieth Century

765

Brannigan argues, Fuller effectively transformed her body into a “moving image,” disappearing into “a play between recognition and abstraction where the only continuity is motion itself” (Brannigan 2011: 36). Equally compelling scholarship has focused on the ways in which live performance genres incorporated aesthetics and elements drawn from emerging media like magic lanterns and film. Theater scholar Gwendolyn Waltz has examined the practice of integrating film into stage productions. At times, footage was used to portray dramatic events or locations that would have been difficult or expensive to reproduce on stage: an 1896 production of Theodore Kremer’s drama Carmen, for example, integrated an actuality film called Bullfight, and productions of A Twig of Laurel by the Owen Davis Stock Company replaced an onstage bicycle race with filmed footage of an actual bike race that had taken place in Boston (Waltz 2012). In other cases, though, stage productions attempted to create the illusion that stage actors were interacting with projected images. A segment of the Ziegfeld Follies of 1907 revue, for example, featured stage actors playing in ocean waves that were projected on a drop, and a few years later, the Follies included a comedic sketch about the making of a film in which a film director yelled cues to his actors as cameras rolled – except that the actors following those cues were actually projected onscreen (Waltz 2014). This strategy of narrative integration of the interpolated film – and thematization of film’s relationship to the stage more broadly – appears in other examples Waltz shares, perhaps most strikingly in a stage production called Pay-Day (1916). Here, two stage actors read through a screenplay, trying to decide whether to try their fortune in Hollywood. As they read, the action as they imagine it is projected on a screen, complete with themselves in the roles, and even the intertitles that might grace a finished film (Waltz 2012: 374). In addition to enabling theater to incorporate filmic aesthetics including photographic realism, quickly progressing narratives, and rapid movement between locations, then, these sort of stage-screen hybrids also served as a means of experimenting with, as Waltz writes, “novel means of expression, introducing different modes of perception, and questioning and defining intermedial relationships.” (Waltz 2012: 374).

Conclusion As the rich and growing body of literature surveyed here suggests, intermediality is a compelling framework through which to understand late nineteenth- and early twentieth-century performance, entertainment, and art. As so many of these studies highlight, the same narratives and characters regularly appeared and reappeared – sometimes loosely reinterpreted, sometimes radically adapted – across different media and genres. Careful attention to these moments of media transposition, as demonstrated in many of the studies cited here, reveals much about both the cultural milieu and, to borrow from film scholar Canan Balan, the “collective cultural property” in circulation at the time (Balan 2014: 260). Similarly, identifying the strategies of intermedial reference at play in this period offers clues as to how creators understood the specific mediums in and across which they worked, both

766

M. Simonson

individually and in relation to the broader media ecosystems. Viewing late nineteenth- and early twentieth-century media and performance phenomena through an intermedial lens makes visible the connections and synergies between these phenomena, but simultaneously constructs a rich, surprising, and gloriously messy historical context in which to situate and interpret much more recent intermedial trends and realities.

References Altman, Rick. 1996. Dickens, Griffith, and film theory today. In Silent film, ed. Richard Abel, 145–162. New Brunswick: Rutgers University Press. ———. 1999. De l’intermédialité au multimédia: cinéma, médias, avènement du son. Cinémas 10 (1): 37–53. Balan, Canann. 2014. Between Karagöz and cinema: Connectivity, mobility, collectivity. In Performing new media, 1890–1915, ed. Kaveh Askari, Scott Curtis, Frank Gray, Louis Pelletier, Tami Williams, and Joshua Yumibe, 254–262. Herts: John Libbey Publishing. Blom, Ivo. 2014. Diva intermedial; Lyda Borelli between art, photography, theatre and cinema. In Performing new media, 1890–1915, ed. Kaveh Askari, Scott Curtis, Frank Gray, Louis Pelletier, Tami Williams, and Joshua Yumibe, 22–33. Herts: John Libbey Publishing. Boenisch, Peter M. 2006. Aesthetic art to aisthetic art: Theatre, media, intermedial performance. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 103–116. Amsterdam: Rodopi. Bolter, Jay David, and Richard Grusin. 1999. Remediation: Understanding new media. Cambridge, MA: MIT Press. Brannigan, Erin. 2011. Dancefilm: Choreography and the moving image. New York: Oxford University Press. Brewster, Ben, and Lea Jacobs. 1997. Theatre to cinema: Stage pictorialism and the early feature film. New York: Oxford University Press. Chapple, Freda, and Chiel Kattenbelt. 2006. Key issues in intermediality in theatre and performance. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 11–26. Amsterdam: Rodopi. Fleeger, Jennifer. 2008. Projecting an Aria, singing the cinema. Music, Sound, and the Moving Image 2 (2): 121–125. Fryer, Paul. 2006. The opera singer and the silent film. Jefferson: MacFarland and Co. Gaudreault, André. 2011. Film and attraction: From kinematography to cinema. Trans. Timothy Barnard. Urbana: University of Illinois Press. Gaudreault, André, and Philippe Marion. 2005. A medium is always born twice. . .. Early Popular Visual Culture 3 (1): 3–15. Goldmark, Daniel. 2017. Adapting The Jazz singer from short story to screen: A musical profile. Journal of the American Musicological Society 70 (3): 767–817. Grover-Friedlander, Michel. 2005. Vocal apparitions: The attraction of cinema to opera. Princeton: Princeton University Press. Gruber, Klemens. 2010. Early intermediality: Archaeological glimpses. In Mapping intermediality in performance, ed. Sarah Bay-Cheng, Chiel Kattenbelt, Andy Lavender, and Robin Nelson, 247–258. Amsterdam: Amsterdam University Press. Gunning, Tom. 2003. Loie fuller and the art of motion. In Camera Obscura, Camera Lucida, ed. Richard Allen and Malcolm Turvey, 75–89. Amsterdam: Amsterdam University Press. ———. 2004. Now you see it, now you don’t: The temporality of the cinema of attractions. In The silent cinema reader, ed. Lee Grieveson and Peter Kramer, 41–50. New York: Routledge.

30

Intermediality and Liveness at the Turn of the Twentieth Century

767

Haenni, Sabine. 2012. Intermediality of hunger, intermediality of effect: Two commercial models developed in early twentieth century New York City. In Literatur: Inter- und Transmedial, ed. David Bathrick and Heinz-Peter Preusser, 403–417. Amsterdam: Rodopi. Hansen, Miriam. 1991. Babel and Babylon: Spectatorship in American silent cinema. Cambridge, MA: Harvard University Press. Hanssen, Eirik Frisvoold. 2017. Silent ghosts on the screen: Adapting Ibsen in the 1910s. In The Oxford handbook of adaptation studies, ed. Thomas Leitch, 154–178. New York: Oxford University Press. Hutcheon, Linda. 2006. A theory of adaptation. New York: Routledge. Kattenbelt, Chiel. 2006. Theatre as the art of the performer and the stage of intermediality. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 29–39. Amsterdam: Rodopi. ———. 2010. Intermediality in performance and as a mode of performativity. In Mapping intermediality in performance, ed. Sarah Bay-Cheng, Chiel Kattenbelt, Andy Lavender, and Robin Nelson, 29–37. Amsterdam: Amsterdam University Press. Kessler, Frank, and Sabine Lenk. 2019. ‘Rendre réel aux yeux du public’: Stage craft, film tricks, and the Féerie. In Media archaeology and intermedial performance: Deep time in theatre, ed. Nele Wynants, 83–98. Cham: Palgrave Macmillan. Koszarski, Richard. 1990. An evening’s entertainment: The age of the silent feature picture, 1915–1928. New York: Scribner. Lehtonen, Mikko. 2001. On no man’s land: Theses on intermediality. Nordicom Review 1: 71–83. Melnick, Ross. 2012. American showman: Samuel ‘Roxy’ Rothafel and the birth of the entertainment industry. New York: Columbia University Press. Musser, Charles. 1994. The emergence of cinema: The American screen to 1907. Berkeley: University of California Press. ———. 2004. Historiographic method and the study of early cinema. Cinema Journal 44 (1): 101–107. North, Dan. 2007. Illusory bodies. Early Popular Visual Culture 5 (2): 175–188. Paul, William. 2016. When movies were theater: Architecture, exhibition, and the evolution of American film. New York: Columbia University Press. Preston, Carrie J. 2011. Modernism’s mythic pose: Gender, genre, solo performance. New York: Oxford University Press. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités 6: 43–64. Remshardt, Ralf. 2006. The actor as intermedialist: Remediation, appropriation, adaptation. In Intermediality in theatre and performance, ed. Freda Chapple and Chiel Kattenbelt, 41–53. Amsterdam: Rodopi. Robert, Valentine. 2014. Performing painting: Projected images as living pictures. In Performing new media, 1890–1915, ed. Kaveh Askari, Scott Curtis, Frank Gray, Louis Pelletier, Tami Williams, and Joshua Yumibe, 282–292. Herts: John Libbey Publishing. Sammond, Nicholas. 2015. As the crow flies: The intermediality of Moran and Mack. Studies in American Humor 1 (2): 142–162. Simonson, Mary. 2012. Screening the diva. In The arts of the prima donna in the long nineteenth century, ed. Hilary Poriss and Rachel Cowgill, 83–100. New York: Oxford University Press. ———. 2013. Body knowledge: Performance, intermediality, and American entertainment at the turn of the twentieth century. New York: Oxford University Press. Staiger, Janet. 2004. The future of the past. Cinema Journal 45 (1): 126–129. Vardac, A. Nicholas. 1949. Stage to screen: Theatrical method from Garrick to Griffith. Cambridge: Harvard University Press. Vogl-Bienek, Ludwig Maria. 2014. Screening sensations and live performance: The creative blending of traditional and new projected media at the start of the twentieth century. In Performing new media, 1890–1915, ed. Kaveh Askari, Scott Curtis, Frank Gray, Louis Pelletier, Tami Williams, and Joshua Yumibe, 217–226. Herts: John Libbey Publishing.

768

M. Simonson

Waltz, Gwendolyn. 2012. ‘Half real-half reel’: Alternation format stage-and-screen hybrids. In A companion to early cinema, ed. André Gaudreault, Nicolas Dulac, and Santiago Hidalgo, 360–380. Oxford: Blackwell Publishing. ———. 2014. 20 minutes or less: Short-form film-and-theatre hybrids – Skits, sketches, playlets, and acts in Vaudeville, Variety, Revues, &c. In Performing new media, 1890–1915, ed. Kaveh Askari, Scott Curtis, Frank Gray, Louis Pelletier, Tami Williams, and Joshua Yumibe, 245–253. Herts: John Libbey Publishing. Wiegand, Daniel. 2018. The unsettling of vision: Tableaux Vivants, early cinema, and optical illusions. In The image in early cinema: Form and material, ed. Scott Curtis, Philippe Gauthier, Tom Gunnning, and Joshua Yumibe, 26–35. Bloomington: Indiana University Press. Willis, Artemis. 2018. The lantern image between stage and screen. In The image and cinema, ed. Scott Curtis, 237–246. Bloomington: Indiana University Press.

The Sonification of Modernist Fiction: A Critical Review

31

Niklas Salmose

This chapter is dedicated to my late PhD supervisor, colleague, and friend Laura Marcus, who opened my eyes to the interdisciplinary and intermedial potentials of modernism and English studies. I would also like to thank Thomas Leitch and Kyle Meikle for sharing their extensive bibliography on film and adaptation.

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multimodal Modernism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sonic Modernism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Audio Techniques, Sound, Noise, and Soundscapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Musicalization of Fiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Musicology and Literary Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Music’s Influence on Modernist Style . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Handbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Work on Individual Authors or Specific Fictional Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Popular Music . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sonic Sweden . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Virginia Woolf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . James Joyce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Mann . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

770 775 777 778 781 782 784 785 786 787 788 789 791 794 795 797

Abstract

This chapter contextualizes the historical and aesthetic era of high modernist fiction (1918–1939) by focusing specifically on its multimodal and sonorous qualities. It briefly discusses some major conceptualizations and taxonomies N. Salmose (*) Linnaeus University, Växjö, Sweden e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_36

769

770

N. Salmose

created in the intermedial field of musico-literary theory. This is followed by a summary and discussion of a selection of general musico-literary criticism that focuses on literary texts from the modernist era. This section transitions into a summary of the musico-literary analytical landscape surrounding three modernist authors: Virginia Woolf, James Joyce, and Thomas Mann. The overall aim of the chapter is to identify the different perspectives, methodologies, and concerns that can be found in the vast body of musico-modernist-literary criticism and locate some of the major tendencies in these intermedial ventures. Keywords

Intermediality · Musico-literary · Literary modernism · Music · Modernity · Sound · Noise · Senses

Introduction In the “Present Day” section of Virginia Woolf’s The Years (1937), Peggy, one of the novel’s characters, contemplates the modern experience, that is, what it is like to participate in how everything that is “solid melts into air,” in Karl Marx’s words (Marx and Engels 2012, 37): When the dance music pauses, Peggy listens to the sounds outside. Far away she heard the sounds of the London night; a horn hooted; a siren wailed on the river. The far-away sounds, the suggestion they brought in of other worlds, indifferent to this world, of people toiling, grinding, in the heart of darkness, in the depths of night, made her say over Eleanor’s words, Happy in this world, happy with living people. But how can one be “happy,” she asked herself, in a world bursting with misery? On every placard at every street corner was Death; or worse – tyranny; brutality; torture; the fall of civilisation; the end of freedom. We here, she thought, are only sheltering under a leaf, which will be destroyed. (Woolf 1965, 388)

The fictional present day of the novel, the mid 1930s, and the publication date of 1937 echo the fear of fascist tyranny, economic depression, and a looming war, but the overall time span of the novel, 1880 to the mid 1930s, yields a largely modern anxiety about the idea of progress and an unprecedented artistic and creative scrutiny of the decline of Western civilization. Similarly, in a poem composed in 1919, William Butler Yeats prominently writes that “things fall apart; the centre cannot hold.” With these words, he mourns the perceived collapse of the order that had previously provided structure and meaning to human life. The embattled center in this verse is not doomed for complete dissolution, however, since the poem, entitled “The Second Coming” and written in the wake of the First World War’s destructiveness, anticipates the arrival of a new messianic force that has the ability to restore humankind to a state of spiritual and ontological harmony. The high literary modernism that Woolf and Yeats represent and that existed between the two world wars internalized these anxieties and understood modern time as a rupture, which Andreas Huyssen has framed as “the great divide” (Huyssen 1986).

31

The Sonification of Modernist Fiction: A Critical Review

771

In addition to framing a modernist political, social, psychological, and historical concern, the above quotation from The Years also signifies modernist literary aesthetics and the project of enhancing literary mimesis by employing a preponderant emphasis on the entire sensorial apparatus. As Rishona Zimring posits in relation to The Years, “sound fragments bring to mind disintegration, barriers toppling, and the juxtaposition of worlds crowding up against one another: not only the ‘rupture’ and ‘fragmentation’ with which many literary critics characterize modernism itself” (Zimring 2002, 128). Hence, as Zimring argues, the sonorous aspect of Woolf’s prose is not merely an addition to the dominant visualities in literature; sound is carefully intertwined in the overall project of modernist fiction. “Woolf’s insistent use of sounds to emphasize the porosity of boundaries (between here and far away, this world and others),” she writes, “hints at both the Utopian and dystopian possibilities latent in a modernist sensibility of change and disintegration” (Zimring 2002, 128). But the common insertion of auditive elements in modernist texts is an innate consequence of “the inner turn,” a focus on interior consciousness and subjectivity and a distrust of non-phenomenological realities. “Here and elsewhere in Woolf’s work, flânerie arouses the sense,” Zimring notes, illustrating how interiority and a changed fictional temporality stress a new set of sensations and perceptions (Zimring 2002, 134). The allusion to Joseph Conrad through the reference to the “heart of darkness” in Woolf’s passage echoes Conrad’s aesthetic suggestion in his highly influential 1897 preface to The Nigger of the “Narcissus,” in which Conrad advocates a multisensory writing style (more on this later in this chapter). In addition to the need for other narrative tools to break with Victorian realism, auditive aesthetics is evidently a reaction to the radically changed soundscape of the late industrial world: new studies in acoustics, the widespread use of radio and gramophones, new sounds that were emerging during the rapid urbanization and modernization of the world, and the importance and accessibility of musical experiences. Hence, modernist writers were influenced by sound and music to various degrees and were tempted by the apparent post-Wagnerian idea of Gesamtkunstwerk and yet were blatantly monomodal in their aloof preconception about the power of the literary narrative. The opening passage of Woolf’s The Years appears perfectly in tune with this chapter. It presents how sound creeps into the modernist literary narrative, in terms of both context and aesthetics. Reflecting the novel’s time span, in this chapter I will be concerned with how critics in musicology, intermediality, and the emerging sound studies have approached literary high modernism in relation to the sonic turn in the world it represents and experiences in the years falling between the two world wars, roughly 1918 to 1939. Critical readings of sound and music in Woolf’s literary oeuvre are extensive, and their different focal points (music, sound, speech, acoustics, sound symbolism, onomatopoeia, phonetics) will also figure in this thorough but not exhaustive survey of critical analyses of the intermedial relations between sound, music, and literary text in a symbiosis that can clearly be seen in high modernist fiction. The interrelations between sound, music, and modernist fiction are, obviously, more complex than simply a question of breaking with previous

772

N. Salmose

traditions, and any study of the connections between music and modernist literature, as Nathan Waddell argues, must delineate “how diversely music feeds into, inspires, challenges, and sometimes resembles other kinds of medial experiment throughout the twentieth century” (Waddell 2017, 317).1 I will not be concerned in this chapter with the crucial developments within intermedial and multimodal theories of musico-literary interrelations that have taken place since the late 1960s and that have been strongly supported by advocates such as Steven Paul Scher, Irina Rajewsky, and Werner Wolf (covered in ▶ Chap. 9, “Linnaeus University Center for Intermedial and Multimodal Studies and the Legacy of Lars Elleström,” by Beate Schirrmacher). Instead, I will illustrate the ways in which research has specifically inspected high modernist narratives during the twentieth and twenty-first centuries. I have not limited myself in this regard to explicit intermedial readings, since they are rather scarce, but have included a broader set of analyses of sonic literature. Nevertheless, it is useful to categorize these interpretations within relevant taxonomies concerning the ways in which music (and sound) can be incorporated in a literary text. Scher provides an early definition in his triadic distinction between “literature in music,” “music and literature,” and “music in literature” (Scher 1968). “Literature in music” concerns, for example, program music; “music and literature” involves different combinations of the media types music and literature, as in opera, choral music, and lyrical music. “Music in literature,” my concern in this chapter, which Scher also refers to as “word music,” constitutes diverse ways in which music can be integrated within the media type literature. Werner Woolf expands this category by using two distinctive approaches as “a foregrounding of the acoustic dimension of verbal signifiers reminiscent of musical sound, but also in ‘structural analogies to music’, i.e. the creation of patterns in a verbal text so that they resemble structures in musical compositions” (Wolf 2015, 460). In terms of music in literature, Wolf distinguishes between two overall relations between the two media types: “intracompositional intermediality,” which has a narrow focus on the musicality in literary media from a monomedial point of view, and “extracompositional intermediality,” which considers relations between the media types in terms of their transmedial quality, for example, content, structure, themes, narrativity, and history, that are shared and easily transformed between the different media types (Wolf 2015). However, as Wolf himself notes, there is considerable overlap between these two types of intermediality, and both can also be considered in relation to any study of a purely monomedial character (Fig. 1). Wolf’s typology is fairly straightforward, and it is useful for framing the different sonorous analyses of high modernist texts and slightly amends the definitions in place. Although Wolf’s “plurimediality” category offers an understanding of classical multimedial works such as opera and writing song lyrics, literature can also be

1

Nathan Waddell’s impressive review of recent monographs on the interconnection between music (sound) and modernist literature, “Modernism and Music: A Review of Recent Scholarship” (2017), provides an excellent survey of the current critical musico-literary landscape in modernist studies.

31

The Sonification of Modernist Fiction: A Critical Review

773

Fig. 1 Werner Wolf’s typology of intermedial forms illustrated with musico-literary examples

considered a plurimedial medium because the signifiers of verbal text consist of visual symbols and phonemes. This is why we laugh at Mr. Bloom’s cat in James Joyce’s Ulysses when she makes the noise “Mrkgnao,” and we interpret it as both meow and milk now. At first, this has no textual meaning but is purely sensational and something that has been created by our hearing faculties. The semiotic reflection comes when we perceive that the sensation is indeed similar to the signified sound of

774

N. Salmose

the cat and something similar to English: milk now! In the end, all meaning is semiotic (at least when meaning is identifiable), but the initial phase of sensations that lead to perception varies. There is quite a difference between writing “Mrkgnao” and “The cat meowed as if it wanted milk.” Hence, the category of plurimediality seems important in any sonorous analysis of the actual sound systems of verbal literature. Even if sound is an inherent part of any verbal text, the complexity of triggering sensations beyond the affordances of a media type such as literature should be addressed. It seems, as Lars Elleström confirms, that sensations can be triggered and reactivated in a complex manner: “This is because our mind, to a certain extent, has the ability to perceive resemblances not only within the same but also across different sensory areas and different mental realms” (Elleström 2018, 25). Thus, the visual receptors trigger personal tactile memories that relate to the information that is provided visually and, to some extent, sonically. Obviously, research has shown that sense organs do not operate in isolation from each other. Elleström exemplifies this by explaining how vibrations can be both heard (as sound vibrations) and felt. Furthermore, Elleström claims that “[t]he perception of one sense faculty may be different if combined with perception from another,” which means that senses can fortify each other (Elleström 2018, 25). There also appears to be a certain level of habituation involved. Therefore, even if not every reader experiences similar crossmodal sensations, one can conclude that more often than not, readers might share such experiences, depending on their biographical similarities and experiences. This capacity to experience a reactivation of sensations beyond media borders is central to my construction of “cross-modal stylistics” (Salmose 2020). Building on Elleström’s work, I analyze modernist fiction from this perspective and also consider the complex triggering of sounds and sound memories in a selection of high modernist texts (Salmose 2020). In order to situate the research overview that follows in a proper context, I will contextualize the historical and aesthetic era of high modernist fiction (mainly written during the 1920s) by focusing on its multimodal and sonorous qualities. This will be followed by a discussion of more general criticisms of the musicoliterary stage of modernism; this section then moves on to consider specific authors – predominantly Virginia Woolf, James Joyce, and Thomas Mann, who dominate the field of sonic analysis of the period in question.2 To limit the critical corpus, I largely discuss literary prose. This is not intended to suggest that other literary forms, such as poetry and drama, are not sonic or have not been analyzed from a sound/music perspective. The one exception to this limitation is the inclusion of criticism of Amy Lowell’s poetry, which I believe makes an important contribution to the more general musico-literary deliberations via the way it significantly interacts with

2

I have consciously omitted Proust from this list. Although much has been written, especially in French, on Proust’s use of music in À la recherche du temps perdu, I find that much of the focus has been on the explicit intermedial reference to specific musical (but some less specific, of course) pieces.

31

The Sonification of Modernist Fiction: A Critical Review

775

explicit intermedial terminology. The omission of discussions about Marcel Proust, and French criticism in general, might seem controversial but is largely also due to the need to tighten the corpora used and, unfortunately, to my limited competence in the French language. For the latter reason, I have included only criticism written in one of the Scandinavian languages or English or German. It has been my sincere ambition to include as many trends and theoretical and methodological approaches and subject areas as possible; nevertheless, there is more sonic criticism that relates to work within the discipline of literary scholarship and less that relates to musicology. Admittedly, I have not had space to engage in important debates within emergent fields such as aurality and practices of listening. The section “Sonic Modernism” begins with a review of criticism of high modernism that focuses on sound, noise, the alteration of soundscapes in the modernist period, and the advent of new sonic technologies. It then moves on to sonic analyses, considering phonetic, onomatopoetic, and symbolic qualities of sound, much in the tradition of earlier debates on sound and poetry, before discussing musico-literary work more specifically. Since Wolf’s concept of the musicalization of fiction appears central to many researchers in the field, this part of the section initially surveys discussions intimately related to Wolf’s work. It also considers samples of studies that have predominantly emerged out of the field of musicology rather than literary studies. Next, there are discussions about texts that frame how music contributed to the advent of the high modernist prose style, not only classical music but also contemporary blues and jazz, and Black music. A set of handbooks is then briefly discussed in terms of the areas the handbooks focus on and the intermedial interests that are relevant. Art music is usually at the center of research that delineates origins and inspirations, and is often directly referred to, and the section considers some of the work that explicitly links the literature with specific composers or classical musical works. This discussion is developed via references to the use of popular and folk music in high modernist narratives. Since all of these critical texts that the review deals with are by Anglo-American or Germanspeaking authors, the final section discusses sonic criticism in a Swedish context.

Multimodal Modernism During late modernity, society has been pushed toward the aim of a universally shared world and constant improvement by social affairs and lifestyles being improved; significant technological developments; urbanization; modernized administrations and bureaucracies; new means of transport, communications, and media; the explosive growth of newspapers; and world exhibitions. Some authors tried to discuss the scientific and philosophical advancements and to capture the “new spirit” and the transcendent homelessness that Western humanity experienced as a consequence of these rapid improvements and changes: Freud, Nietzsche, Bergson, and Einstein, to mention a few. In 1895 a new medium, film, was born, and from the early 1930s to 1945, the world witnessed more than ever before how propagandistic art and politics interact. During this intense period, modernist art comingled with,

776

N. Salmose

influenced, and was determined by all these changes, as well as by an increased desire to become more multimodal and multimedial, or, as Richard Begam and Matthew Wilson Smith argue, polyartistic (Begam and Wilson 2016). Modernist fiction, specifically, involves all our senses as part of its reaction to the project of modernity and progress, as well as to Victorian realism; it is not just a response to a heightened sensibility toward new soundscapes, new perceptions of motion, and new olfactory experiences in the aftermath of industrialization and modernization. This “rebellion” involved shifting the focus from being on an outer, rational, and objective reality to being on an inner, irrational, and subjective consciousness, which drove the emphasis on emotional and sensational experience. Further, the birth of a new art form, cinema, in the latter decades of the nineteenth century fundamentally created an intimate intermedial relationship between literature and cinema that involved significant developments in literary style, form, and narration.3 The modernist movement, in general terms, therefore adheres to Ezra Pound’s modernist slogan “Make It New” in that it is self-reflective, experimental, and formalistic and, most importantly for this chapter, occurs in all media. This last element coincides with what Regina Schober articulates as “the questioning and expansion of established media boundaries” (Schober 2010, 164). In a similar vein, Wolf considers that intermediality in modernism is no coincidence, because medial crossing is an integrated part of the experimentation of modernism (Wolf 2002, 21). However, there is a paradox here too if we think about modernist fiction specifically, which occupies an ambiguous space between a desire to be considered monomedial and a desire to be considered multimedial. Modernist authors’ search for a pure, absolute literary form has been well documented since Clive Bell’s Art (1914). As Sara Danius states, there has been a desire among modernist writers “to project themselves as literary equivalents of Wagnerian Gesamtkunstwerk, as synesthetic works of art seeking to transcend genre” (Danius 2002, 3). The difference between opera and literature, though, is that a Wagnerian opera is composed of words, music, acting, and staging and can therefore definitely be considered truly multimedial, whereas a modernist literary work consists of words and sounds only and hence is multimodal. Within the affordances of literature, modernist writing appeals to all the senses and to all the art forms. This in itself signals a departure from more realistic aesthetics to more phenomenological ones that are part of the development of style in modernist writing, and this is demonstrated, as mentioned earlier, in Conrad’s famous preface to The Nigger of the “Narcissus”: “My task which I am trying to achieve is, by the power of the written word, to make you hear, to make you feel” (Conrad 1914, xiv). This seminal line suggests a change in attitude toward the function of prose. Conrad’s “magic suggestiveness” inspires a reaction, perhaps to

Essential contributions to this field that do not involve music per se are those of Lewis (2020), McCabe (2005), Spiegel (1976), Trotter (2007), Seed (2009), Marcus (2008, 2014a), Donald et al. (1998), Danius (2002), Jacobs (2001), Garrington (2013), North (2005), and Cohen (1979). 3

31

The Sonification of Modernist Fiction: A Critical Review

777

viewing realism as a literary movement rather than as a concept. His emphasis on sensations signals a transition in literary focus from outer to inner reality, from perception to sensation, and an indication that the multisensory aspect is needed to grasp the emotional, subjective experience, a process that had already been initiated by the French symbolists. For Baudelaire, there had to be a mingling between the interior and exterior worlds in poetic language. His heightened sensibility, his ecstasy, impelled future poets to use excessive emotional and sensational poetic language as a means to understand the relationship between interior and exterior worlds. This change of perspective, which Marcus Bullock suggests also includes a destabilization of normative distanced objectivity, is why Walter Benjamin considers Baudelaire to be an aesthete that marks “the essential break of modernism with the previous tradition of literary aesthetics” (62). Baudelaire and Rimbaud created a poetic language that lent itself very well to the explorations of inner consciousness that became a trademark of high modernism not only in the obvious examples of the work of Gertrude Stein and James Joyce but also in prose that has less evidently engaged in cross-modal triggering of sensations. In her essay “Modern Fiction,” Virginia Woolf claims that “[l]ife is not a series of gig lamps symmetrically arranged; life is a luminous halo, a semi-transparent envelope surrounding us from the beginning of consciousness to the end” (Woolf 1986, 160). This truly Bergsonian concept of duration and flux is a harsh critique of the realist tradition, ideas about progress and clock time, and the focus on consciousness also suggests a focus on the sensorium of a consciousness. This is an attitude that Merleau-Ponty would later describe as being in the world – the “primacy of perception”: we first perceive the world; then we make sense of it (Merleau-Ponty 1962, 221). As Danius and others have argued so succinctly, the unique style of modernist fiction and its contents grew out of a complex weave of social, historical, scientific, intellectual, and technical contexts that occurred in the Western world from approximately the 1880s until the Second World War. Warren Susman states that, “we are all aware that media somehow affect the general culture while at the same time the general culture shapes the media” (Susman 1984, xxvi). The immense advances in sound reproductive technologies in the postindustrial world would also have a longlasting effect on the literary sonic style of high modernism. This is certainly the case with sonic modernism, which will be developed in the next section.

Sonic Modernism Literary modernism drew much inspiration from contemporary cinema and the visual and performative arts (and vice versa), but perhaps the most proximate relationship existed between modern fiction and music; this is echoed in Walter Pater’s now famous observation that “all arts constantly aspire to the condition of music” (Pater 1919, 111). The literary focus on and fascination with sonorous art stem partly from, as Jacob Harris contends in the case of Woolf, music’s “ability to

778

N. Salmose

evade the grasp of language” (Harris 2017, 58). There is obviously a long tradition of connections between music and literature, especially regarding rhyme and meter in poetry; similarly, modernist fiction reiterates the poetic entanglements with music through its use of prose poetry. As hinted at in Harris’s statement, modernist authors’ appreciation of music has philosophical overtones; music and literature, as Woolf argued, both have the capacity to reveal “the truth about this vast mass that we call the world [. . .] we are the words; we are the music” (Woolf 1985, 72). Werner Wolf frames this discussion in a rethinking of music as fiction’s Other. Basing his argument on persistent historical ideas about music, music as a privileged medium for expressing emotions, the idea of music as a correlative of harmony and cosmic order, music as closer to human imagination and the unconscious, and the idea that music is the original art form, Wolf shows how modernist writers found it very appealing to incorporate this fictional Other in their texts (Wolf 2002, 20–21).

Audio Techniques, Sound, Noise, and Soundscapes Despite these questions about art and truth and about art and realism, the main reason why music had such a precarious position for modernist writers was clearly because the development of modernist prose coincided with an unprecedented technological evolution of music. Music, though, is as unstable a construct as modernism itself, and as Waddell suggests, “modernism in general responded to the rhythms, melodies, harmonies, and textures of very different sorts of music from that of the ancient Greeks onwards, even as it found ways to register how sound seems to split apart from music in the case of noise, with all its acoustic, value-laden complications” (Waddell 2017, 317). As evidenced in most of the studies presented in this chapter, music dominates many of the musico-literary approaches to modernist fiction; with the emergence of sound studies, however, it is becoming increasingly attractive to study sound, rhythm, and noise, especially in terms of the historical parallelization of modernist time and sound contexts. Melba Cuddy-Keane provides a compelling and condensed history of the advent of new sound technologies between 1920 and 1940 in “Virginia Woolf, Sound Technologies, and the New Aurality”:4 The period 1920–1940 was a revolutionary era for the development of sound technologies; it was also the informative moment for the broad impact of these new technologies on the public at large. The nineteenth century had created, through the phonograph, expectations of a new world of sound reproduction, but the early twentieth century realized new possibilities for sound generation. The 1920s saw the development of the first electrical instruments, such

4

There are of course multiple studies that provide overviews of the development of sound and music in the modernist era. A few notable examples are those of Kittler (1990, 1999), Stevens (1990), Schafer (1977), Halliday (2013), Sterne (2012), Thompson (2002), Trotter (2013), and the early chapters in Fekadu (2013). For an excellent review of recent books on sound history and modernism, see Latham (2017).

31

The Sonification of Modernist Fiction: A Critical Review

779

as the theremin, the ondes martenot, and the first electronic organ. Wireless telephony opened up vast new possibilities for sound distribution and for mass communication on a previously inconceivable scale. In 1920, the first radio station to transmit regular broadcasts went on the air in Pittsburgh, Pennsylvania; on 14 November 1922, the BBC launched its first broadcast, establishing itself as the first radio station in Europe. (Cuddy-Keane 2000, 72)

The phonograph, mentioned by Cuddy-Keane, was introduced in 1877, and, as accurately described by Douglas Kahn, it “introduced the fact of worldly sound into culture” (Kahn 2004, 43). Many sonic surveys of the social and cultural contexts of the late nineteenth and early twentieth centuries focus on sound rather than music. The advent of the gramophone contributed to a relationship with music that went beyond occasional visits to concert halls or operas, or limited experiences of music in radio broadcasts. For example, Leonard and Virginia Woolf acquired an Algraphone in 1925, which provided a repetitive evening listening discourse and clearly had an effect on Virginia’s writing. Laura Marcus, on the other hand, argues in “‘A Hymn to Movement’: The ‘City Symphony’ of the 1920s and 1930s” (2014b) that one of the reasons why sound and music creep into modernist fiction is the significant urbanization of fictional settings in modernist novels and poetry. Inspired by the so-called city symphony films (e.g., Walter Ruttmann’s Berlin: Symphony of a Great City and Dziga Vertov’s Man with the Movie Camera), modernist novelists locate their work in the “‘intermediate zone’ of urban experience,” which “chimes with Woolf’s complex sense that future cinema might move to seize the sense-impressions of the city” (Marcus 2014b, 94). The “urban scene” thus evokes urban sounds, such as the discords of traffic, crowds, machines, and factories, as well as street music. “In the novel [Mrs Dalloway],” Marcus writes, “there is a ‘pulse’ or heartbeat of the city as well as of its inhabitants” (Marcus 2014b, 94–95). Philipp Schweighauser comes to a similar conclusion in his analysis of the urban New York soundscape in John Dos Passos’s Manhattan Transfer. Schweighauser calls the strategy Passos uses the noisification of fiction (building on Wolf’s concept of musicalization of fiction) and argues that one of Dos Passos’s strategies is to inject noise into the narrative to further complicate the communication process between text and reader as part of the larger modernist project of communicative detachment (Schweighauser 2017, 501). Katherine O’Callaghan notes that “[m]odernist writers worked in a noisier world than any of their predecessors, and this sociohistorical soundscape reverberates in their work” (O’Callaghan 2018b, 3). Sound, noise, and music are of course not only components of modern urban life but also the consequences of sensorial experiences concerning that great rupture of modern civilization, the First World War. This perspective is evident in Gemma Moss’s musico-literary reading of Ford Madox Ford’s Parade’s End.5 As Moss argues, the destructive sounds of war take the

5

Isabelle Brasme approaches Parade’s End from a more traditional musico-literary perspective, singling out three musicalizations in terms of “symphony, of the structural use of leitmotifs, and of a shift from a contrapuntal mode to one of atonality” (Brasme 2018, 62).

780

N. Salmose

soundscape of modernity to a different level with its very own monstrosity (Moss 2017, 59). But Moss’s focus extends beyond sound; her main focus is on how the deadly soundscapes challenge the transcendental role of music, as argued in many other critical analyses of the modernist use of music.6 Critical appreciations of how sound is foregrounded in literary aesthetics can therefore take many forms. In “The Sounds and Smells of the South: The Auditory and Olfactory in Fitzgerald’s Tarleton trilogy” (Salmose 2022), I assert how Fitzgerald’s sensorial aesthetics use the soundscapes of the American South as a way to create a sense of not only place and local color but also of a mediation of an arcadian, bucolic space of nostalgia and longing as a contrast to the more modernized North. In the same spirit, Karl F. Zender looks at how Southern soundscapes infiltrate Faulkner’s The Sound and the Fury in “Faulkner and the Power of Sound” (1984). He points out that these sounds seem to produce “a heightened awareness of the destructive power of time. Implicit in his [Faulkner’s] effort at preserving the vanishing world of his youth is a sense of the evanescence of reconciliation” (Zender 1984, 91). Zender means that in Faulkner’s prose, these sounds often work like narrative pauses that “intrude” on the narrator because of their hostility and drive him back “to the world of time and loss” (Zender 1984, 92). Hence, sounds in this context are related to both new modernist temporalities and concepts about the discontinuities of narrativity. An original addition to literary attention being paid to sound because of the changed soundscapes of the modern world and the technological advancements is Angela Frattarola’s “Developing an Ear for the Modernist Novel: Virginia Woolf, Dorothy Richardson and James Joyce” (2009). Frattarola positions modern favoritism toward the sense of hearing as a deliberate action against the postEnlightenment idealization of the visual sense. One “reason why modern novelists represent sound and auditory experience to such a great extent,” she notices, is “to counteract traditional, Cartesian concepts of subjectivity, and by extension, rational and linear narrative conventions” (Frattarola 2009, 137). Although this is in line with much of the criticism identifying saturation of sound in modern fiction, Frattarola frames this “break” with tradition in the history of the sensorium specifically. A notable and ambitious study that broadly engages with the tight encounters between music, noise, and the soundscapes of twentieth-century social and cultural life in European modernism is Joshua Epstein’s Sublime Noise: Musical Culture and the Modernist Writer (2014). Epstein’s main argument is that there is a dynamic interrelation between literary modernism and the advent of modernist music and its atonal, dissonant, and noisy formulations by the likes of Alban Berg and Arnold Schönberg. According to Epstein, “[t]he presence of noise in British, AngloAmerican, and Anglo-Irish modernism is the sound of the artwork coming to grips with the failure of its autonomy from social life” (Epstein 2014, xv–xvi). The book’s chapters engage in parallel readings of music and literary texts: Wagner and T. S. Eliot’s The Waste Land, the musical score of Fernand Léger’s 1924 film Ballet

6

For further discussions on the sounds of war and modernist literature, see Connor (1997).

31

The Sonification of Modernist Fiction: A Critical Review

781

Mécanique and Esra Pound’s and James Joyce’s work, and also a comparison between Joyce’s Finnegans Wake and John Cage’s Roaratorio.7 Helen Groth, Julian Murphet, and Penelope Hone present a broad survey of modernism and sound in their introduction to the edited collection Sounding Modernism: Rhythm and Sonic Mediation in Modern Literature and Film (2017), where the work of Samuel Beckett initially proves useful for explaining how modernist literature mediates sounds through language’s acoustic and phonetic qualities (Groth et al. 2017, 3). Although the volume does address the more common denominators of musico-literary analyses of modernist works, Groth, Murphet, and Hone state that it also considers two additional critical components: a more textually focused analyses of sonic modernism (perhaps indicated by using “mediation” in the title) and a rethinking of literary mediations of “rhythms, sonic textures and vocal derangements” (Groth et al. 2017, 4). Hence, onomatopoeia becomes an important ingredient in this textual endeavor, as exemplified by the “speaking” cat in Joyce’s Ulysses (1922) uttering “Mkgnao” instead of “Meow.” This textual focus is also echoed in the concept of listening to the sounds of literary modernism as a new way of reading (this is perhaps most significant in the case of Joyce’s Finnegan’s Wake). The focus on literary mediation is a welcome addition to musico-literary modernism, but also illustrates a tendency in literary studies to become less intermedially analytical and more mono-medially analytical, which means neglecting options involved in rethinking modernist fiction not as a pure media type but as a truly heteromedial art form that not only mimics or produces acoustics but also fully integrates music and sounds as media forms. A somewhat different take on how to interpret sound in literature can be found in Garrett Stewart’s anomalous (in relation to much work in the musico-literary field), phonemic readings of several modernist canonized works: those by Yeats, Eliot, Joyce, and Woolf. In Reading Voices: Literature and the Phonotext (1990), Stewart provides a theoretical, semiotic interpretation of how modernist wordplay (and sound defects8) has a strong acoustic component that sets it apart from other literary traditions. He frames this in his conceptualization of the phonotext, building on Kristeva’s concept of the phenotext but adding material and sensorial qualities to it that only occur in phonetic readings of texts.

Musicalization of Fiction Wolf, moving from sound to music, puts modernist musical aesthetics into five explanatory categories in “Towards a Functional Analysis of Intermediality: The

7

In addition to Epstein’s extensive work on modernist poetry, Claus Clüver addresses concrete sound poetry specifically as an intermediate aesthetic form that sits between music and poetry (Clüver 2002). 8 Stewart defines a “sound defect” as something vaguely askew, indeed unhinged, in the discrete vocabulary of a given syntactic format: a sound effect as sound defect.

782

N. Salmose

Case of Twentieth-Century Musicalized Fiction.”9 The first is a modernist desire to use intermediality to transgress established aesthetic boundaries. As Wolf argues, the use of musicalization in fiction is part of a larger trend that deviates from narrative conventions – this is especially evident in the use of word music and the influence of music on narrative structures. As for the latter, one consequence of it, according to Wolf, is that music inspires a detour from “narrative plausibility and referential or grammatical consistency and hence from aesthetic illusion, a major effect of traditional, especially realist storytelling” (Wolf 2002, 21). The second category involves how suitable music is for both foregrounding the very concept of media and for selfreferentiality as a whole. Music therefore reinforces literary meta-aspects because “one of the essential traits of musical composition is its high degree of auto-referentiality, which surpasses everything possible in literature” (Wolf 2002, 22). The third category is less general than the first two, which concerns more general notions about music in modern fiction, and is about the more specific emotional and sensuous potential of musico-literary intermediality as part of the modernist fascination with consciousness and the unconscious and fantastic (Wolf 2002, 24). This function is part of the multisensory engagement in modern fiction. The fourth category has more philosophical substructures, as music traditionally offers a medial affordance of polyphony, which translates into modernist literature’s preoccupation with the plurality and fragmentation of both internal and external existence and reality (Wolf 2002, 25–26). Fifth and lastly, Wolf situates musicalization as part of a larger historical and cultural context in which traditional notions of meaning are either questioned or disqualified altogether. The loss of belief in master narratives, Wolf argues, co-develops with the advent of new media forms and rapid advancements in diverse media technologies. Hence, integrating music in literature suggests a move away from traditional narrativity and mimetic aestheticization (Wolf 2002, 26–27). It is indisputable, as this brief summary illustrates, that these five categories partly overlap. Many of these categories and concerns have, of course, been considered by other critics and in different circumstances. Brad Bucknell, for example, contends that modernists were preoccupied with both representing inner consciousness and the general problem of representation and that music came in as a “subsidy” to that very project (Bucknell 2001). Regarding music’s emotional character, Wolf’s third category, Bucknell makes a case for rethinking the position of a musical descendent in the hierarchical ranks of emotional art forms, opting instead for reviewing music as one component of framing the emotional and sensorial complexities of the modern world – the musicalization of literature is therefore an attempt to tune in to this intricacy (Bucknell 2001, 37–222).

Musicology and Literary Studies Obvious aspects of an interdisciplinary/intermedial field are the technical and traditional complexities involved in working within such separate fields as musicology

9

It is worth noting that Wolf not only considers modernism to be a literary movement that is transgressive but also includes postmodernism in this genre.

31

The Sonification of Modernist Fiction: A Critical Review

783

and literary/fictional studies. One successful attempt to breach the media borders can be found in Erik Alder and Dietmar Hauck’s Music and Literature: Music in the Works of Anthony Burgess and E. M. Forster (2005). Although the book is not entirely about modernist texts, it explores both the major novels of E. M. Forster and texts by Leo Tolstoy and Aldous Huxley. The authors effectively draw careful parallels between actual notated music and how music affects literary structure, thematics, and content through allusions to it in the relevant texts. Alder and Hauck conclude by identifying four main ways in which music and fiction are correlated, according to their readings: music is linked to the psyche of characters; music is related to characters’ social relations; formal elements of music are represented in narrative structures in the form of aspects such as leitmotifs; and finally, there are mutual inspirations between music and literature (Alder and Hauck 2005, 183). Hence, this book is mostly concerned with various intermedial references and, to a lesser extent, with intermedial transposition. Eric Prieto’s ambitious study of music in modernist fiction, Listening In: Music, Mind, and the Modernist Narrative (2002), looks at the musicalization of fiction in a broad modern period, from the symbolists, such as Mallarme, to Marcel Proust, James Joyce, Thomas Mann, Virginia Woolf, André Gide, and Samuel Beckett and right up to the French Nouveau Roman. Prieto structures each chapter around a specific musical model in which he exemplifies the theoretical discussions in a literary text. He divides the role of music into the three categories: expression, signification, and formalism. Expression is not only the obvious emotional aspect but also cognitive components such as problem-solving. Expression can also be situated in the composer, in the work, or in the listener. Signification relates to music’s symbolic possibilities, and formalism refers to the fact that music often does not refer to any kind of extramusical reality – it is just a pure happening (Prieto 2002, 29–31). Much of Prieto’s discussions are theoretical in a musical sense, such as making further qualifications of the differences between exemplification and essentialism, and are less explicitly about modern fiction as such. Prieto argues that modernist writers’ attraction to music lay partly in how music lacks denotation (representational qualities) and works in terms of exemplification (similarities on nonrepresentational levels) (Prieto 2002, 37–42). One consequence of absence of representational qualities is that modernism stresses language as a suggestive tool with hidden semantic potential; this is particularly evident in the work of Samuel Beckett (Prieto 2002, 49–50). It is impossible in the short space of this chapter to do justice to all of the musico-literary complexities considered in Prieto’s book, since he covers topics as wide as the musicalization of fiction, music as a model for associative progression in narratives, the use of musical repetition and variation, music and mimesis, jazz influences on modern fiction, and music as a moral purpose in literature. Overall, though, much of the intricate analyses build on the general argument that music complements modernists’ aims of breaking rules and traditions, not focusing on plot-based narration, and abandoning ideas about objective realities. Prieto argues that music therefore functions as an inspiration and a model for creating the modernist style. What is engaging in his book is that the discussions about the intermedial combinations between music and literature add to and

784

N. Salmose

complement a general understanding of the development of a unique modernist style of writing. An even more recent essay collection, Essays on Music and Language in Modernist Literature: Musical Modernism (O’Callaghan 2018a), aspires to further emphasize how music merges with the advancement of a specific modernist prose style. “The imprint of musical forms such as sonata, theme and variation, or fugue can certainly be found in modernist literary works,” O’Callaghan writes in her introduction, “but it was primarily the internal structuring principles of music that provoked and compelled a series of radical literary innovations.” Stressing style before direct influence or explicit referencing appears to be a guiding principle in the volume; however, although the editor underlines that the volume explores hitherto unknown authors and perspectives, many essays still consider the usual suspects – Wagner, Woolf, Joyce, Mann, Proust, and Beckett. This collection does add to previous research, though, when it considers less familiar ground in chapters such as Michael Borshuk’s “‘The Blues Always Been Here’: African American Music and Black Modernism in August Wilson’s Ma Rainey’s Black Bottom,” Renée Dickinson’s “The (R)evolution of Olive Moore: Fugue as Bridge to a New Feminist Awakening,” and “Rabindranath Tagore and Musical Modernity in Early TwentiethCentury Bengal” by Suddhaseel Sen.

Music’s Influence on Modernist Style Martin Huber’s Text und Musik: Musikalische Zeichen im narrativen und ideologischen Funktionszusammenhang ausgewählter Erzähltexte des 20. Jahrhunderts (1992) constitutes an ambitious plan to comprehend the musicoliterary entanglements in modernist Austro-German literature. Besides the seemingly mandatory readings of work by Mann, Huber interacts with a variety of modernist authors: Hermann Hesse, Alfred Döblin, Arthur Schnitzler, and Robert Musil, among others. This diversity helps significantly in formulating an understanding of Austro-German modernist fiction in a less monolithic and more holistic way than is usually adopted by authors. Huber is interested in musical influences and representation within the contexts of Wagner and Adorno but more engagingly attempts to trace music’s influence on modernist narratives’ form and technique. Huber’s positions are clearly set out in his first chapter on the foundation of the musico-literary aesthetics of the twentieth century; he divides his discussion into three sections, “Musik und Gefühl” (Music and Feeling), “Musik und Form” (Music and Form), and “Musik und Ideologie” (Music and Ideology). Regarding music and feelings, Huber surveys the history of musical subjectivism and its literary consequences in the romantic era, defining a Sturm und Drang character to the literary use of music that peaks in European decadence with an aesthetics of feeling that embodies musical thematization. In “Musik und Form,” Huber continues the debate on aesthetics but zooms in on how music affects the very formal and structural elements of modernist literature, drawing on the aesthetic theories of Friedrich Schiller, Eduard Hanslick, and Theodor Adorno. The conceptualizations of music

31

The Sonification of Modernist Fiction: A Critical Review

785

as emotion and form triangulate into ideological frameworks, most notably in relation to Wagner’s politicized music, and it is this latter analytical perspective that dominates many of the musico-literary readings, such as those that ascertain the representative quality of Wagner’s Tristan und Isolde in Alfred Döblin’s November 1918 (Huber 1992, 60–63). Huber’s best contribution to the field appears in Chaps. 3 and 4, where the focus areas change from being about representation into debates about structure, musical influence on modernist temporalities, and conceptualizations of text as notation; in summary, he writes about unleashing music’s potential to co-participate in the development of modernist style.

Handbooks A couple of recent handbooks on the relationship between music and literature consider literary modernism and music as well as individual case studies. In The Edinburgh Companion to Literature and Music (2020), the section called “Literature and Music in the Twentieth and Twenty-First Centuries” has almost 200 pages. Stephen Benson summarizes the mutual relationships between music and literature during the relevant period in his “Introduction,” focusing on music history and musical theory. He identifies two specific uses of music in modernist literature: Literature’s writing of music at this time involves thus both a borrowing of specific forms, with especial attention to the interplay of voices in counterpoint and polyphony, and more generally, the practice of leitmotivic composition as allowing a patterning of associations – experiential; memorial; cultural – felt to be more organic, and paradoxically free-form, than the thematic structural machinery of Victorian prose. (Benson 2020, 497)

Benson stresses the importance of musical ekphrasis in the development of literary modernism before discussing in different subsections some aspects of musicoliterary relationships: literature in music, sound, and technology and in the discipline of music generally. The last-mentioned aspect is principally related to theories of music. Prieto picks up the baton from Benson and ventures into the aesthetics of medial combinations in his chapter on the theories of Nelson Goodman, which formed the basis for Prieto’s monograph, mentioned previously. Although both Benson and Prieto touch on the medial intersections between music and literature, it is remarkable that a handbook of this stature does not involve intermedial studies at all. The second handbook, Handbuch Literatur und Musik (2016), carries considerably more theoretical, intermedial weight, perhaps as a consequence of situating the research within the Austro-Germanic intermedial fields, as advocated by Scher and Wolf. The result is a highly systematic presentation of intermedial relations between music and literature that is divided into discussions about combinations, integrations, transmediations, and the difference between telling and showing in musicalization of literature. The handbook also updates theoretical discourses about musico-literature in contemporary, cognitive research fields before providing

786

N. Salmose

a rich enumeration of specific cases and topical studies. Chapters that modernist scholars will be interested in include one that discusses the historical and contextualizing aspects of Gesamtkunstwerk and Wagner’s and Nietzsche’s musical and philosophical influences on literature, another that focuses on music and the literary avant-garde, and a third that considers the expressiveness of modernist musical forms in literature and how the sounds of modernity creep into modernist fiction.10 In a similar fashion, Literatur und Musik in der klassischen Moderne: Mediale Konzeptionen und intermediale Poetologien (2006) formulates a media-oriented approach to musico-literary criticism by stressing the role of media communication and intermedial theory in rethinking the poetics of modern writing. This modus operandi is evident, for example, in Gabriele Rippl’s study of intermedial aesthetics in Anglo-American modernism as a conscious departure from realist fiction. Rippl does not simply illustrate a variety of musical transfers and references in work by Gertrude Stein, H.D., Amy Lowell, and Aldous Huxley but also inspects the medial limitations in literature as a media type and the consequences of these limitations on the exchange with music. Most importantly, in the section “Intermedialitätsforschung” (Intermedial Research), she contextualizes the musico-literary field of modernist fiction in contemporary intermedial theoretical frameworks.

Work on Individual Authors or Specific Fictional Works There are many articles and chapters that specifically consider modernist authors or single fictional works from a musico-literary perspective, but because of the limitation on the length of this chapter, it is impossible to discuss any of these in detail. However, I will aim to briefly discuss in the following paragraphs texts with an intermedial framework or authors who have not received attention in monographs or larger critical works. Regina Schober works engagingly within Wolf’s musico-literary framework (with flashes of Elleström and Rajewsky) in “Translating Sounds: Intermedial Exchanges in Amy Lowell’s ‘Stravinsky’s Three Pieces “Grotesques,” for String Quartet,’” applying the concepts of intermedial references to Igor Stravinsky’s Three Pieces for String Quartet in Amy Lowell’s poem of the same name. Schober’s analysis goes beyond framing music as a reference and also considers the poem a true work of intermedial translation of Stravinsky’s piece in terms of how it contextualizes past and current affairs, situating this particular translation in an overall modernist project of translation, allusion, and intertextuality in the spirit of T. S. Eliot’s concept of the mythological return (Schober 2010, 165). Schober concludes that Lowell’s ambition is not to recreate music in literary text but merely to translate

10

Although the handbook focuses on German and French romanticism, a complement to the musico-literary taxonomies in this handbook is Albert Grier’s “Musik in der Literatur: Einflüsse und Analogien” (1995) in which Grier conducts parallel readings of romantic music and romantic literature within a semiotic framework.

31

The Sonification of Modernist Fiction: A Critical Review

787

the effect of music to her poem. In this reading she targets Elleström’s notion of transmediality, acknowledging that media modalities and modes share similar content and affordances with music and literature (Schober 2010, 169).11 In “Katherine Mansfield and Music: Nineteenth-Century Echoes” (2011), Delia da Sousa Correa argues that nineteenth-century music, Wagner in particular, played a formative part in the development of a distinguished and early modernist style in Katherine Mansfield’s works. Correa notes that although there are few explicit references to music, Mansfield’s prose oozes musical inspiration: “Passages invoking the movement of sea and waves at dawn, noon and night punctuate the story’s opening, middle, and close, suggesting analogies with musical movements as well as with the textures and rhythms of music,” she writes (Correa 2011, 85). What is therefore most arresting about Correa’s article is that she identifies how Mansfield, as a consequence of her passionate relation to nineteenth-century music, constructs a rhythmic and musical prose in which the very modernist language of her writings alludes to the structures (dramatic timing), rhythms, and sounds of nineteenthcentury romanticist music.

Popular Music It is noteworthy that although a massive amount of critical attention has been given to classical music and/or sound, barely any musico-literary attention has been paid to the interrelations between popular music and modernist writing. Ronald Schleifer’s exemplary study, Modernism and Popular Music (2011), is an example of the latter providing a political view of how jazz and popular music engage with modernist cultures in general and occasionally intermedially with literary modernists such as William Butler Yeats, T. S. Eliot, William Carlos Williams, and Virginia Woolf. But the focus of the monograph is still the media type music. Likewise, Christopher Ballantine’s aim in “Modernism and Popular Music” (2014) is to argue that lowbrow culture such as popular music is still an inherent part of the radical project of modernity. Studies of songs in Joyce and Eliot have unleashed the potential of intermedial references to traditional and folk music, and to popular music of course, but not with such popular connotations, perhaps, as popular, contemporary music had in the 1910s and 1920s. It is no surprise, then, that the author who called the 1920s the “jazz age,” F. Scott Fitzgerald, figures in critical studies on the use of popular music in modernist fiction. A groundbreaking analysis that situates Fitzgerald’s prose in the popular music of his times is Ruth Priogozy’s “‘Poor Butterfly’: F. Scott Fitzgerald and Popular Music” (1977). Although it mostly catalogues music that Fitzgerald 11

For a more extensive analysis of Lowell’s musical writing, see Chap. 1 in Fekadu’s (2013) book. Fekadu argues that Lowell charts music as a figure of thought in her approach to a “new poetry” whose (originally) oral, performative character she intends to preserve. In addition to this, Debora Van Durme explores further the musical connection between imagism and Lowell’s poetry with a special focus on the music of Debussy (Van Durme 2018).

788

N. Salmose

refers to, Priogozy also identifies how Fitzgerald uses popular music in his works: (1) to make his characters articulate emotions, (2) to refine characters’ moods and relationships, and (3) to provide background music and intensify settings (see previous discussion in subsection “Audio Techniques, Sound, Noise, and Soundscapes”). In addition to these three uses, popular music, according to Priogozy, also becomes intertwined with themes and leitmotifs in Fitzgerald’s work. T. Austin Graham develops Priogozy’s ideas by offering the term “literary soundtrack” to refer to “written references to specific pieces of music that compel extra-literary responses in readers and thereby heighten, color, or otherwise comment upon the text that contains them” (Graham 2009, 519). By analyzing Fitzgerald’s major novels, Graham not only delivers a detailed survey of Fitzgerald’s use of popular music but also highlights the complexities for later readerships regarding identifying the contemporary contexts surrounding these songs. Critics, he claims, need to resort to “musical archeology” to annotate the meaning of these songs for a modern readership (Graham 2009, 520).12 The latest study to investigate popular musico-literariness in Fitzgerald is the insightful F. Scott Fitzgerald’s Short Fiction: From Ragtime to Swing Time (2019) by Jade Broughton Adams. Chapters on jazz and ragtime dance, musical theatre, and popular music contextualize Fitzgerald’s less researched short stories in terms of the New York music scene. In the chapter titled “Satyre upon a Saxophone: Fitzgerald and Music,” Adams confidently discusses Fitzgerald’s incorporation of jazz in terms of the wider, complex, history of jazz music with tenets of class and race. Adam’s work complicates Fitzgerald’s jazz references but in a positive way, suggesting that they are not only literary soundtracks but integrated facets of his organic writing style. This also entails an added understanding of the often-cited concept of some stories being “jazzy,” which is taken as a sign that jazz also operates on an extracompositional level.

Sonic Sweden Since most intermedial analyses of relations between music and literature have referred to Anglo-American, Austro-German, or French modernist literature, Thomas Seiler’s discussion of music’s role in Swedish modern aesthetics is a welcome contribution. “Instrumental-, Partiturlyrik, Tonpoesi Zur Bedeutung musikalischer Begrifflichkeit im scwhedischen Modernismus” (2006) departs from the influential 1932 essay “Dikt och music” (Poetry and Music) by Swedish poet Gunnar Ekelöf. As Seiler argues, Ekelöf’s essay identifies a rupture in how poetry relates to music. Modernist poetry focuses on the emotions and feelings of music in their transformative state, whereas poetry before that era embraced the sound quality of poetic phonetic qualities (Seiler 2006, 137). Hence, Seiler’s chapter is an

12

For a more extensive discussion on the concept of literary soundtracks, see Graham (2013), who broadens the corpus from Fitzgerald to also include T. S. Eliot and poets of the Harlem Renaissance.

31

The Sonification of Modernist Fiction: A Critical Review

789

important contribution to a modernist meta-text about musico-literary relations and ambitions; it builds on Magdalena Wasilewska-Chmura’s study of musical poetics in Swedish modern literature (Wasilewska-Chmura 2000). Paul Tenngart’s analysis complements these two studies; he analyzes Ekelöf’s agenda of musicalized poetry and its influence on late Swedish modernist lyricism and lyricism since the modernist era (Tenngart 2002).

Virginia Woolf It is a queer experience, lying in the dark and listening to the zoom of a hornet which may at any moment sting you to death. It is a sound that interrupts cool and consecutive thinking about peace. Yet it is a sound – far more than prayers and anthems – that should compel one to think about peace. (Woolf 1940, n.p.)

As is already evident in this chapter, Woolf had an intimate bond with both sound and music in her fiction, which has opened up a critical space for musico-literary readings of her works. It is quite surprising, then, that few critics have paid more attention to her literary soundscapes than to her musical influences and the musicalization of her prose.13 Cuddy-Keane’s chapter “Virginia Woolf, Sound Technologies, and the New Aurality” is a rare example, as are Marcus’s works, of an analytical perspective that specifically discusses the emergence of a new sound media landscape in Woolf’s writing and theorizes the complexities of translating sound to visual media: Besides the problem of mediating sound through a visually oriented discourse, there is the fundamental problem of mediating sound through language at all – the inevitable translation of sound into a conceptual category that takes place in the process of verbalization [. . .] But there is a significant difference between the linguistic representation of sound and the linguistic conceptualization of it [. . .] When Big Ben chimes the hours throughout Virginia Woolf’s Mrs. Dalloway, the words “The leaden circles dissolved in the air” (4) describe the pattern of sound waves, not a human construction of their meaning. (Cuddy-Keane 2000, 70)

Cuddy-Keane legitimately acknowledges that there is a lack of narratological terminology for describing sounds in literature, and she suggests “diffusion” to refer to the emission of sound from a specific source and “auscultation” to refer to the complementary aspect of narrative focalization, for example, who hears rather than who sees (Cuddy-Keane 2000, 70–71). Cuddy-Keane provides an insightful aural analysis of the section “Time Passes” in Woolf’s To the Lighthouse, which stands out since there is no proper auscultation in this passage due to the lack of characters. 13

It is worth mentioning The Reading of Silence: Virginia Woolf in the English Tradition, in which Patricia Ondek Laurence looks at silence in a selection of Woolf’s novels from a predominantly feminist perspective, and Annika J. Lindskog’s two chapters on silence in Woolf’s works, which are part of her doctoral thesis Silent Modernism. Soundscapes and the Unsayable in Richardson, Joyce and Woolf (2014).

790

N. Salmose

Sound is therefore mediated without human consciousness in an attempt to record nonhuman reality. This is a chimera, of course, as Cuddy-Keane is the first to admit, since sound is still mediated through narration and is aimed toward an implied reader. However, as Cuddy-Keane argues, this is a literary attempt to resist “the reduction of sound to an idea, concept, or sign” (Cuddy-Keane 2000, 85). Most critics have focused on music, though they have perhaps been inspired by Woolf’s own statement that she conceives all of her writing as music (Woolf 1975–80, 426). Elicia Clements, in “Transforming Musical Sounds into Words: Narrative Method in Virginia Woolf’s The Waves,” engages in a compelling analysis that not only focuses on musical influence but on formal imitation. After acknowledging how early essays such as “Street Music” (1905), “The Opera” (1906), and “Impressions at Bayreuth” (1909) “deal with music as subject-matter and address its affect on community,” she argues that “Woolf deliberately attempts to reconstitute novelistic methods by looking to the ‘classical’ tradition of music as a potential model” (Clements 2005, 160). In relation to The Waves, Clements provides a thorough parallel reading between how the formal aspects of Beethoven’s late String Quartet in B flat Major, Opus 130, constitute a “new novelistic ‘form’” that reconceptualizes the construction and idea of characterization. Although Woolf evidently listened to Beethoven while writing The Waves, Clements’ analysis goes beyond influence when she maps out the novel’s formal similarities to the quartet and fugue forms. In the novel, Beethoven is considered “signifying” rather than influential, an “intermedial [. . .] link, an exchange of ideas that circulate at a particular historical moment in time facilitated by two different media” (Clements 2005, 163). Clements further connects Woolf’s musicoliterary method to the theories of Wolf, especially regarding the differentiation between a nondominant medium (music) and a dominant medium (literature), where the nondominant medium is an example of a covert intermediality in a dominant medium that does not have aural affordances (Clements 2005, 166–167). Clements compares the similar ways in which late Beethoven interrogates the limits of musical form and Woolf pushes the limits of literary form to new dimensions, and her reading of how Woolf transforms musical form to literary tropes (such as water) and characterization (such as the six interludes) is truly compelling from an intermedial perspective. Similar to Clements’ chapter, Sarah Fekadu’s chapter on Woolf in Musik in Literatur und Poetik des Modernismus (2013) indicates that Woolf writing in music in her narratives signifies a radical break from earlier musical references to formal imitations. Emma Sutton’s 2013 monograph Virginia Woolf and Classical Music: Politics, Aesthetics, Form traces mostly art musical influences in Woolf’s writings chronologically. She discusses the role of Wagner in her “subjective turn” in The Voyage Out and the musical and aural discourses of the Great War as they infiltrate the increasingly interiorized narratives of Jacob’s Room (1922) and Mrs Dalloway (1925). She also examines the use of the “leitmotif” in The Waves, but what she leaves out, “folk music, jazz, recorded sound and ancient Greek music [. . .] rhythm [. . .] and popular music” – in her own words – would perhaps have been more invigorating and novel areas to research (Sutton 2013, 20). A more ambitious attempt to frame Woolf’s writing from an intermedial musicoliterary perspective can be found in Virginia Woolf and Music (2014), edited by

31

The Sonification of Modernist Fiction: A Critical Review

791

Adriana Varga. The essays included in the volume both contextualize and politicize the role of music in the Bloomsbury culture that surrounded several of the London modernists during the 1920s, and music as a source of inspiration to Woolf.14 Early chapters in particular consider Woolfian texts not often discussed from the perspective of music in literature, such as The Voyage Out (1915) and Night and Day (1919). Jim Stewart’s “The Birth of Rachel Vinrace from the Spirit of Music,” for example, investigates the role of intermedial references to songs from Sophocles’ Antigone and Milton’s Comus in The Voyage Out, and Vanessa Mahire argues in ““The Worst of Music”: Listening and Narrative in Night and Day and “The String Quartet”” that Woolf developed her use of music in order to question traditional ways of representation in her quest to embrace subjective consciousness. Trina Thompson, in a similar fashion to Vargas in an earlier chapter, attempts in “Sounding the Past: The Music in Between the Acts” to frame Woolf’s writing in what Wolf would call “implicit references” (evocation, formal imitation, (partial) reproduction). Woolf, according to Thompson, blends media types in an attempt to reach the quality of a Gesamtkunstwerk, to “posit a supersaturated wholeness of experience” (Thompson 2014, 206). Thompson claims that Woolf makes explicit use of music to create fragmentation and motivic repetition in her final novel to erase the borders of media, art, and consciousness (Thompson 2014, 213). Music (and sound) as an ordering function of the chaos of the totality of life experience is also discussed by Sanja Bahun in “Broken Music, Broken History: Sounds and Silence in Virginia Woolf’s Between the Acts.” Wolf’s intricate analysis of Woolf’s short story “The String Quartet” is a prime example of an integration of specific modernist narrative techniques (internal focalization, free indirect discourse, stream of consciousness, sound symbolism, fragmented temporalities) and musicalization of fiction, or, as Wolf demonstrates in the first part of his analysis, a full example of verbal music or simulation of music (Wolf 1999, 148–154). Most importantly, however, and something that is often incomplete in many music-literary analyses, is the section that discusses the functions of musicalization in “The String Quartet.” One fascinating observation made by Wolf is that the musico-literary analysis of the short story also “throws important light on the meaning and functions of” it, thus shifting the position from only looking at the integration of music to also using this as a further enhancement of the story’s construction and meaning overall (Wolf 1999, 154).

James Joyce As can be expected regarding an author with an ear that is sensitive to sound and music, James Joyce’s later work has been scrutinized extensively from a sonic perspective. Hence it is somewhat unexpected that Annika J. Lindskog spends two 14

For a thorough examination of how classical music functions as inspiration as well as reference in the works of Woolf (the Bloomsbury group), T. S. Eliot, Huxley, and D. H. Lawrence, see David Deutsch’s monograph British Literature and Classical Music: Cultural Contexts 1870–1945 (2015).

792

N. Salmose

chapters in her dissertation Silent Modernism. Soundscapes and the Unsayable in Richardson, Joyce and Woolf (2014) investigating Joyce’s use of silence. Lindskog traces silence in Joyce as part of his progression toward a more postmodern, nonrepresentative type of prose. In his quest for a total break with realism, silence found its way into his work in various forms via the playwright and via playfulness (Lindskog 2014, 131). Lindskog’s project, which traces modernist fiction’s distrust of reality and silence as a consequence of this, is original and compelling, and it rewards the reader with close readings of work by Joyce, Woolf, and Dorothy Richardson. One might see Lindskog’s dissertation as a complement to some of the impulses in Danius’s The Senses of Modernism: Technology, Perception, and Aesthetics (2002). Although Danius, in her chapter on Joyce, principally focuses on the visual, she also asserts that “to apprehend the absence of sound is also to rediscover sound, in effect to reinvent it – in its pure and abstract form” (Danius 2002, 149). Danius, importantly, acknowledges that one reason for the insertion of the sound sense in modernist fiction is the emerging phenomenological interest in the differences between sensation and perception, where sound appears stronger than other senses in its sensational stage. This is also the reason why Danius is occupied with the tower scene in Ulysses, since it confirms the autonomy of the ear, a “desire to represent what is heard and, furthermore, to represent it in a register that is radically separate from what is seen” (Danius 2002, 156). Zack Bowen’s Musical Allusions in the Works of James Joyce: Early Poetry through Ulysses (1975) provides a rich but traditional reading of how music functions as explicit, and at times implicit, references (allusions and references) in a longish and detailed chapter on Ulysses that reads like an annotation of music. Bowen’s meticulous work is very useful for tracking down these sources but adds little to the more theoretical, intermedial debate. His work forms part of a larger body of early work examining the musical form and allusions to music and songs in Ulysses that does not have any specific theoretical or aesthetic context.15 A very productive analysis of music and Joyce can be found in the diverse collection Bronze by Gold: The Music of Joyce (1999), edited by Sebastian D. G. Knowles. Because it includes biographical context about Joyce’s musical background and relates Joyce’s work in detail to other modernist impulses and contemporaries and the music of his times, the book is a selection of textual and sonic close readings of Joyce’s main pieces of fiction with a focus on how “[w]ords are multiplied by music,” which is reflected in the title of the collection; there is a particular focus on rhythm in Joyce’s prose (Knowles 1999, xxix–xxx). Knowles illustrates how rhythm informs Joyce’s writing style by citing and framing the end of the short story “The Dead” in rhythmical units: His soul swooned slowly | as he heard the snow | falling faintly through the universe |

15

See White (1936), Dallapiccola (1951), Ross (1936), Boyle (1965), and Sternfeld (1957).

31

The Sonification of Modernist Fiction: A Critical Review

793

and faintly falling, | like the descent of their last end, | upon all the living and the dead. | (Knowles 1999, xxxvi)

Knowles addresses how the measured closure of the short story is rendered through alliteration, phonemes, falling phrases (as the snow in the example above), strong double beats, and spondees in an interpretation that simultaneously pays attention to the specific modernist aesthetics of rhythm and the fictional allusive universe Joyce inhabits. The oppositional relationship between vision and sound and between space and temporality is further enhanced in Susan Mooney’s reading of the Sirens section in Ulysses in “Bronze by Gold by Bloom: Echo, the Invocatory Drive, and the ‘Aurteur’ in ‘Sirens’” (Mooney 1999). Mooney argues against the common interpretation of “Sirens” as a fugue-like literary composition and instead reads the chapter as a fugue-like experience related to the loss of memory in psychoanalysis, which has a strong bearing on how she identifies music and sound in the chapter. Andreas Fischer carries the discourse of text and sound further in “Strange Words, Strange Music: The Verbal Music of ‘Sirens’.” Fischer claims that there are many similarities between language and sound, thus acknowledging the acoustic features of textual language and verbal music (Fischer 1999, 245).16 Bronze by Gold offers a rich and variegated contextualization and reading of Joyce that adheres to the semiotic tradition in how it predominantly investigates specifically how Joyce’s prose triggers synesthetic auditive impulses in a mostly visual medium, thus conceptualizing modern fiction’s transmedial qualities. Bucknell’s chapter on “Sirens” in Literary Modernism and Musical Aesthetics: Pater, Pound, Joyce and Stein (2001) continues the debate on how Joyce fuses music and language but more interestingly how Joyce’s “allusions to music, both within and outside the text, seem designated to call up that art’s unmistakable tangibility, the clear force of its presence” (Bucknell 2001, 121). Bucknell contextualizes the common correlation between musical fugue and “Sirens” and incorporates David Herman’s claim that it is more astute to consider “Sirens” as a representation of the difficulties of representation in the wake of Schönberg’s 12-tone music than a fugue (Herman 1994). Bucknell’s rich chapter on “Sirens” also includes a section on the intertext of songs in Joyce’s chapter, building on Bowen’s work (Bucknell 2001; Bowen 1975). Wolf’s reading of “Sirens” in The Musicalization of Fiction: A Study in the Theory and History of Intermediality (1999) does not differ much from other critical impulses, but he situates the musico-literary analysis in his own intermedial framework, which adds a finer grain to his interpretations. Dividing his text into three sections, Wolf first problematizes the analogue between fugue and the chapter’s construction and instead thinks of this as a translation of features that evokes musical simultaneity by “simultaneously present[ing] characters in the fictional space” (Wolf 1999, 134). Secondly, he considers how the chapter imitates music in the forms of 16 For an additional reading of how Joyce attempts to fuse music with language, see Bucknell (2001, 132–138).

794

N. Salmose

analogies to musical microstructures and the use of word music. In relation to word music especially, Joyce ticks Wolf’s boxes regarding the modernist use of music discussed earlier in this chapter, that is, acoustic foregrounding, selfreferentialization, and a departure from grammatical consistency, but also adds aesthetic musical modes of high frequency and an unusual sound pattern (Wolf 1999, 139). Thirdly, Wolf acknowledges that one reason for Joyce’s extensive references to and translation from music in “Sirens” stems from an overall engagement with Ulysses to an almost encyclopedic approach to the entanglements of world.

Thomas Mann As in much Germanic musico-literary criticism, literary modernism is repeatedly connected to the music and aesthetics of Richard Wagner. Hans Rudolf Vaget argues that Wagner, in fact, is the glue that links Joyce and Thomas Mann’s modernist oeuvres (Vaget 1989, 148–149). This is also supported by Susan von Rohr Scaff, who claims that both authors are writing in the wake of Wagner. “The significance of music in Mann and Joyce,” she writes, “goes beyond the authors’ personal enjoyment, moreover, to become a part of their commentary on modern experience” (Scaff 2009, 440). Both Vaget and Scaff position Joyce and especially Mann in the contexts of Wagnerian and post-Wagnerian music discourse (Adorno, Frankfurt School, Schönberg, Berg); they stress that Wagner is not explicitly notable in their respective work but influences them on more thematic and psycho-character levels. There is a wide range of Wagnerian- or Frankfurt School-inspired musico-literary analyses of Mann. Some of the more prominent readings include H. A. Basilius’s “Thomas Mann’s Use of Musical Structure and Techniques in Tonio Kröger,” Gunilla Bergsten’s Thomas Manns Doktor Faustus. Untersuchungen zu den Quellen und zur Struktur des Romans, Carl Dahlhaus’s “Fiktive Zwölftonmusik: Thomas Mann und Theodor W. Adorno,” Wolf-Dietrich Förster’s “Leverkühn, Schönberg und Thomas Mann. Musikalische Strukturen und Kunstreflexion im Doktor Faustus,” James Northcote-Bade’s Die Wagner-Mythen im Frühwerk Thomas Manns, and George W Reinhardt’s “Thomas Mann’s Doctor Faustus: a Wagnerian Novel.” Noteworthy here, from an intermedial perspective, is Barbara Beßlich’s study of intermedial musical references in Der Zauberberg, which has a theoretical focus on the medialities and mediations involved in these transfers (Beßlich 2009). A seminal early book on musico-literary theory, Verbal Music in German Literature (1968) by Steven Paul Scher, discusses Mann (in addition to E. T. A Hoffmann and Heinrich Heine). Scher too endorses the Mann-Wagner liaison in a close comparison between Wagner’s prelude to act 3 of Die Meistersinger von Nürnberg and the verbal score in Chap. 15 of Mann’s Doktor Faustus (1947). As Scher argues, the comparison yields at least two levels of meaning: (1) it is an “ingenious word picture” of the prelude; (2) the chapter corresponds to major events in the life of Leverkühn, the composer protagonist of the novel (Scher 1968, 109). Scher’s “verbal score” concept is most fully realized in the stylistic analysis of Mann’s

31

The Sonification of Modernist Fiction: A Critical Review

795

novel in the section “Aspects of Style,” where Scher describes how Mann uses instrument analogues, musical terminology, polysyndeton, and asyndeton, among other methods, to create verbal musical movement, orchestration, and dynamics (Scher 1968, 121–126). Building on Scher’s taxonomy differentiating verbal music, word music, and musical form and structure parallels, Johannes Odendahl’s Literarisches Musizieren: Wege des Transfers von Musik in die Literatur bei Thomas Mann (2018) is a significant contribution to the vast musico-literary work on Mann, which previously mostly focused on biographical, psychological, philosophical, and political factors. Like many of his predecessors, his focus is mainly on Doktor Faustus, but he also provides musical analyses of Buddenbrooks and Der Zauberberg. Odendahl suggests that Mann’s descriptions of music are not merely mimetic representations of music but are fully integrated with the plot as well as the themes and motifs (Odendahl 2018, 27). Similar to many interpretations of musicalization in Joyce, Odendahl moves away from the idea of language paralleling music to the very semiotic core of musical descriptions that are reached through linguistic practices such as the use of vowels and consonants, alliteration, onomatopoeia, rhythm, and dynamics. In summary, the analysis shows how language mimics music rather than representing music as such. In his extensive last chapter, Odendahl studies structural analogies to music at different levels of Mann’s literary texts, including an in-depth section on the correlation between the musical and literary leitmotif, concluding that the leitmotif as originated by Wagner is as much music-literature as the one used by Mann (Odendahl 2018, 198). In conclusion, Odendahl’s study is ambitious in tracing several facets of musical transfer into literature; and by avoiding the specific use of intermedial theory, Odendahl constructs his own terminology and categorizations in order to build up his analytical framework. This is, unfortunately, done ad hoc, and he thus never manages to formulate a more coherent theoretical apparatus for contemporary musico-literary analysis. The modernist implications in his analysis mostly concentrate on the contexts of Wagnerism.

Conclusion As evidenced in this chapter, there is a tremendously rich and active body of musicoliterary criticism on modernist writing, but unfortunately I have been unable to mention all of the ambitious and important work in this area. The intermedial critical debates on media affordances, mixed media forms, transmediations, and other exchanges between the arts in the era of modernity and by the modernists themselves were already substantial and dynamic, although these debates have not been a focal point of this survey. Later assessments and analyses of musico-literary modernism show a stubborn and perhaps even passionate conviction that music’s impact and influence on modernist literature have paved the way for the philosophical, aesthetic, political, and stylistic resources that modernist fiction thrusts upon us all. In short, music has had a tremendous repercussion on the development of the originality and experimental character of modernist literary style, and these studies have therefore

796

N. Salmose

illuminated and contributed to the history of modernism. Further, the diversity of this critical work points to the fact that these influences are not to be considered in simplistic terms, or simply as one-to-one relationships between music and literature, but as rather strikingly complex interrelations and entanglements between two historically different art forms. This chapter has favored, for practical reasons, the temporal, synchronic movement of music to literature, but the opposite direction is similarly rich in outcome. I think it is safe to say that there are, paradoxically, both a tendency in modernist literature that the literary form should be guarded as the art form and at the same time a tendency that we should be curious about and radical concerning how we think literature can move beyond media borders of a particular media type. The fascination with music, then, stems from having the utmost respect for this very art form and how it is apparently immensely different from the novel, compared to cinema and theatre. If one were to summarize the various music-literary approaches that are conspicuous in the limited critical corpus of this chapter, the following categories would be exposed (building on Wolf’s taxonomy): approaches that consider (1) the influence of music on the authors, for example, biographical studies; (2) the general context of the different soundscapes of modernity and their effect on modernism overall; (3) studies of explicit intermedial references to music in literature, for example, discussions of music by authors or between characters, or the use of musical themes; (4) probably at least three very different transmedial qualities that occur in both music and literature, such as leitmotifs, accentuation, and drama; (5) implicit references to music in literature, for example, (a) evocation or simulation of music by describing music or music making/performance, (b) formal imitation, for example, how Wagner’s music influences structural analogies in literature, and (c) reproduction of song texts or notation; and (6) a sensorial focus on how modernism is attentive to the sense of hearing, to noise, and to sounds. It is important to acknowledge that these categories overlap, and several are often targeted within one piece of analysis. It is also noteworthy that the first two categories do not explicitly involve theoretical intermedial discussions but create a background for these more technical and formal examinations. I will now make a few final observations. First, it is clear that early musicoliterary analyses of modernist fiction favored titles that were clearly related to music, either through their titles or, to a lesser extent, through biographical evidence or extensive musical discussions within the literary diegesis. Secondly, and related to the first, the vast majority of the critiques have focused on a limited number of authors or novels (Proust, Mann, Woolf, and Joyce). Although there has been an increasing accumulation of the corpora of modernist writers analyzed, there is still scope to expand beyond certain authors and their nationalities and also to investigate how sonic approaches in modernist fiction occur in less straightforward manners. It is obvious that many modernist writers wrote to music (to borrow Woolf’s own wording), even if it was not specifically signaled. Third, many of the musico-literary readings in this review still do not cross disciplinary borders; most come from the field of literary studies, a few are rooted in musicology, and even fewer operate in a true interdisciplinary landscape of merging ideas, concepts, and categorizations from

31

The Sonification of Modernist Fiction: A Critical Review

797

both disciplines. And this, I think, leads to the fourth and final conclusion: there is not a proper intermedial, multimodal, semiotic, or interart framework to support the analyses or discussions. It is true that many critics discussed in this chapter adhere to intermediality and/or reliable musico-literary theories but mostly in a rather superficial manner. This is most obvious in the Anglo-American sphere, where a lack of an intermedial perspective is sometimes remarkable. I do not mean to disrespect in any way these authors’ often splendid and productive findings and discussions, but I still find them surprisingly lacking in theoretical considerations in general as they are rigidly situated in a classic hermeneutic literary tradition. Quick searches in AngloAmerican musico-literary handbooks or monographs yield almost no references to intermediality or practicing and formative critics in this field. German-speaking work is generally more supportive of intermedial and multimodal theory, perhaps in the vein of the determinative theoretical musico-literary work by Austro-German critics such as Scher and Wolf. An indisputable fact arising from their work is that the use of intermedial theory in the analytical process enhances and fine-tunes the understanding of how modernist sonic writing operates. Intermediality is therefore a useful tool for musico-literary analysis, and although several recent and very illuminating studies have illustrated this, they have focused more on contemporary than modernist writers.

References Adams, Jade Broughton. 2019. F. Scott Fitzgerald’s short fiction: From ragtime to swing time. Edinburgh: Edinburgh University Press. Alder, Erik, and Dietmar Hauck. 2005. Music and literature: Music in the works of Anthony Burgess and E.M. Forster. Tübingen: Francke Verlag. Ballantine, Christopher. 2014. Modernism and popular music. Journal of the Royal Musical Association 139 (1): 200–204. Basilius, H.A. 1944. Thomas Mann’s use of musical structures and techniques in Tonio Kröger. The Germanic Review 19: 284–308. Begam, Richard, and Matthew Wilson. 2016. Introduction. In Modernism and opera, ed. Richard Begam and Matthew Wilson Smith, 1–25. Baltimore: Johns Hopkins University Press. Benson, Stephen. 2020. Introduction. In The Edinburgh companion to literature and music, ed. Delia Da Sousa Correa, 495–514. Edinburgh: Edinburgh University Press. Bell. Clive. 1914. Art. Stokes. Bergsten, Gunilla. 1974. Thomas Manns Doktor Faustus. Untersuchungen zu den Quellen und zur Struktur des Romans. Tübingen: Niemeyer. Beßlich, Barbara. 2009. Vom Nutzen und Nachteil des Grammophons für das Leben Hans Castorps: Narratologische, intermediale und reproduktionsästhetische Betrachtungen zum Zauberberg. In Literatur intermedial. Paradigmenbildung zwischen 1918 und 1968, ed. Wolf Gerhard Schmidt and Thorsten Valk, 153–166. Berlin: De Gruyter. Bowen, Zack. 1975. Musical allusions in the works of James Joyce: Early poetry through Ulysses. New York: Gill and Macmillan. Boyle, Robert. 1965. Ulysses as frustrated sonata form. James Joyce Quarterly 2 (4): 247–254. Brasme, Isabelle. 2018. Musicality in ford Madox Ford’s Parade’s end: Towards modernity. In Essays on music and language in modernist literature: Musical modernism, ed. Katherine O’Callaghan, 62–78. New York: Routledge.

798

N. Salmose

Bucknell, Brad. 2001. Literary modernism and musical aesthetics: Pater, Pound, Joyce and Stein. Cambridge: Cambridge University Press. Clements, Elicia. 2005. Transforming musical sounds into words: Narrative method in Virginia Woolf’s the waves. Narrative 13 (2): 160–181. Clüver, Claus. 2002. Concrete sound poetry: Between poetry and music. In Cultural functions of intermedial exploration, ed. Erik Hedling and Ulla-Britta Lagerroth, 163–178. Amsterdam: Rodopi. Cohen, Keith. 1979. Film and fiction: The dynamics of exchange. New Haven: Yale University Press. Connor, Steven. 1997. The modern auditory I. In Rewriting the self: Histories from the renaissance to the present, ed. Roy Porter, 203–223. New York: Routledge. Conrad, J. 1914. The nigger of “Narcissus”. London: Sun Dial Press. Correa, Delia Da Sousa, ed. 2020. The Edinburgh companion to literature and music, Edinburgh companions to literature and the humanities. Edinburgh: Edinburgh University Press. ———. 2011. Katherine Mansfield and music: Nineteenth-century echoes. In Celebrating Katherine Mansfield: A centenary volume of essays, ed. G. Kimber and Janet Wilson, 84–98. London: Palgrave. Cuddy-Keane, Melba. 2000. Virginia Woolf, sound technologies, and the new aurality. In Virginia Woolf in the age of mechanical reproduction, ed. Pamela L. Caughie, 69–96. New York: Garland Publishing. Dahlhaus, Carl. 1982. Fiktive Zwölftonmusik: Thomas Mann und Theodor W. Adorno. In Jahrbuch der Deutschen Akademie für Sprache und Dichtung, 33–49. Heidelberg: Lambert Schneider. Dallapiccola, Luigi. 1951. On the twelve-note road. Music Survey 4: 324. Danius, Sara. 2002. The senses of modernism: Technology, perception, and aesthetics. Ithaca: Cornell University Press. Deutsch, David. 2015. British literature and classical music: Cultural contexts 1870–1945. Bloomsbury: Bloomsbury Academic. Dickinson, Renée. 2018. The (r)evolution of olive Moore: Fugue as bridge to a new feminist awakening. In Essays on music and language in modernist literature: Musical modernism, ed. Katherine O’Callaghan, 79–93. New York: Routledge. Donald, James, Anne Friedberg, and Laura Marcus, eds. 1998. Close up: Cinema and modernism. London: Continuum International. Elleström, Lars. 2018. Identifying, construing, and bridging over media borders. Scripta Uniandrade 16 (3): 15–30. Epstein, Joshua. 2014. Sublime noise: Musical culture and the modernist writer. Baltimore: Johns Hopkins University Press. Fekadu, Sarah. 2013. Musik in Literatur und Poetik des Modernism: Lowell, Pound, Woolf. Munich: Wilhelm Fink. Fischer, Andreas. 1999. Strange words, strange music: The verbal music of “Sirens”. In Bronze by gold: The music of Joyce, ed. Sebastian D.G. Knowles, 245–262. New York: Garland Publishing. Förster, Wolf-Dietrich. 1975. Leverkühn, Schönberg und Thomas Mann. Musikalische Strukturen und Kunstreflexion im Doktor Faustus. Deutsche Vierteljahrsschrift für Literaturwissenschaft und Geistesgeschichte 49: 694–720. Frattarola, Angela. 2009. Developing an ear for the modernist novel: Virginia Woolf, Dorothy Richardson and James Joyce. Journal of Modern Literature 33 (1): 132–153. Garrington, Abbie. 2013. Haptic modernism: Touch and the tactile in modernist writing. Edinburgh: Edinburgh University Press. Gess, Nicola, and Alexander Honold, eds. 2016. Handbuch Literatur & Musik, Handbücher zur kulturwissenschaftlichen Philologie. Berlin: De Gruyter. Grage, Joachim, ed. 2006. Literatur und Musik in der klassischen Moderne: Mediale Konzeptionen und intermediale poetologien, Klassische Moderne. Würzburg: Ergon Verlag.

31

The Sonification of Modernist Fiction: A Critical Review

799

Graham, T. Austin. 2009. The literary soundtrack: Or, F. Scott Fitzgerald’s heard and unheard melodies. American Literary History 21 (3): 518–549. ———. 2013. The great American songbook: Musical texts, modernism, and the value of popular culture. New York: Oxford University Press. Grier, Albert. 1995. Musik in der Literatur: Einflüsse und Analogien. In Literatur Intermedial: Musik – Malerei – Photographie – Film, ed. Peter V. Zima, 61–92. Darmstadt: WBG. Groth, Helen, Julian Murphet, and Penelope Hone. 2017. Introduction: Sounding modernism 1890–1950. In Sounding modernism: Rhythm and sonic mediation in modern literature and film, ed. Helen Groth, Julian Murphet, and Penelope Hone. Edinburgh: Edinburgh University Press. Halliday, Sam. 2013. Sonic modernity: Representing sound in literature, culture and the arts. Edinburgh: Edinburgh University Press. Harris, Jacob. 2017. Modernism and the challenge of aurality. Journal of Modern Literature 40 (3): 56–65. Herman, David. 1994. Sirens after Schönberg. James Joyce Quarterly 31 (4): 473–494. Huber, Martin. 1992. Text und Musik: Musikalische Zeichen im narrativen und ideologischen Funktionszusammenhang ausgewählter Erzähltexte des 20. Jahrhunderts. Frankfurt am Main: Peter Lang. Huyssen, Andreas. 1986. After the great divide: Modernism, mass culture, postmodernism, 1988. Basingstoke: Macmillan. Jacobs, Karen. 2001. The eye’s mind: Literary modernism and visual culture. Ithaca: Cornell University Press. Kahn, Douglas. 2004. Art and sound. In Hearing history: A reader, ed. Mark Michael Smith, 36–50. Athens: University of Georgia Press. Kittler, Friedrich A. 1990. Discourse networks 1800/1900. Trans. Michael Meteer and Chris Cullens. Stanford: Stanford University Press. ———. 1999. Gramophone, film, typewriter. Trans. Geoffrey Winthrop-Young and Michael Wutz. Stanford: Stanford University Press. Knowles, Sebsatian D.G., ed. 1999. Bronze by gold: The music of Joyce. New York: Garland Publishing. Latham, Clara Hunter. 2017. Listening to modernism: New books in the history of sound. Contemporary European History 26 (2): 385–395. Laurence, Patricia Ondek. 1991. The reading of silence: Virginia Woolf in the English tradition. Stanford: Stanford University Press. Lewis, Cara L. 2020. Dynamic form: How intermediality made modernism. Ithaca: Cornell University Press. Lindskog, Annika J. 2014. Silent modernism. Soundscapes and the unsayable in Richardson, Joyce and Woolf. Ph.D. Centre for Languages and Literature: English studies, Lund University, Lund Manhire, Vanessa. 2014. “The worst of music”: Listening and narrative in night and day and “the string quartet”. In Virginia Woolf and music, ed. Adriana Varga, 134–159. Bloomington: Indiana University Press. Marcus, Laura. 2008. The tenth muse: Writing about cinema in the modernist period. Oxford: Oxford University Press. ———. 2014a. Dreams of modernity: Psychoanalysis, literature, cinema. Cambridge: Cambridge University Press. ———. 2014b. ‘A hymn to movement’: The ‘city symphony’ of the 1920s and 1930s. In Dreams of modernity: Psychoanalysis, literature, cinema, ed. Laura Marcus, 89–109. Cambridge: Cambridge University Press. Marx, Karl, and Friedrich Engels. 2012. The communist manifesto. Trans. Samuel Moore. London: verso. McCabe, Susan. 2005. Cinematic modernism: Modernist poetry and film. Cambridge: Cambridge University Press. Merleau-Ponty, Maurice. 1962. Phenomenology of perception. London: Routledge.

800

N. Salmose

Mooney, Susan. 1999. Bronze by gold by bloom: Echo, the invocatory drive, and the ‘Aurteur’ in “Sirens”. In Bronze by gold: The music of Joyce, ed. Sebastian D.G. Knowles, 229–244. New York: Garland Publishing. Moss, Gemma. 2017. Music, noise, and the first world war in Ford Madox Ford’s Parade’s end. Modernist Cultures 12 (1): 59–77. Murphet, Julian, Helen Groth, and Penelope Hone, eds. 2017. Sounding modernism: Rhythm and sonic mediation in modern literature and film. Edinburgh: Edinburgh University Press. North, Michael. 2005. Camera works: Photography and the twentieth-century word. Oxford: Oxford University Press. Northcote-Bade, James. 1975. Die Wagner-Mythen im Frühwerk Thomas Manns. Bonn: Bouvier. O’Callaghan, Katherine, ed. 2018a. Essays on music and language in modernist literature: Musical modernism, Routledge studies in twentieth-century literature. New York: Routledge. ———. 2018b. Introduction. In Essays on music and language in modernist literature: Musical modernism, ed. Katherine O’Callaghan, 1–16. New York: Routledge. Odendahl, Johannes. 2018. Literarisches Musizieren: Wege des Transfers von Musik in die Literatur bei Thomas Mann. Bielefeld: Aisthesis Verlag. Pater, Walter. 1919. The School of Giorgione. In The renaissance, 107–127. New York: Modern Library. Prigozy, Ruth. 1977. “Poor Butterfly”: F. Scott Fitzgerald and Popular Music. Prospects 2: 41–67. Prieto, Eric. 2002. Listening. In Music. Mind and the modernist narrative. Lincoln: University of Nebraska Press. ———. 2020. Nelson Goodman: An analytic approach to music and literature studies. In The Edinburgh companion to literature and music, ed. Delia Da Sousa Correa, 515–521. Edinburgh: Edinburgh University Press. Reinhyardt, George W. 1985. Thomas Mann’s doctor Faustus, a Wagnerian novel. Mosaic 18: 109–123. Rippl, Gabriele. 2006. “If we want pure sound, we want music” (Ezra Pound): Zur intermedialen Ästhetik der angloamerikanischen klassischen Moderne. In Literatur und Musik in der klassischen Moderne: Mediale Konzeptionen und intermediale poetologien, ed. Joachim Grage, 87–105. Würzburg: Ergon Verlag. Ross, Martin. 1936. Music and James Joyce. Chicago: Folcroft. Salmose, Niklas. 2020. Sensorial aesthetics: Cross-modal stylistics in modernist fiction. Text Matters: A Journal of Literature, Theory and Culture 10: 321–335. ———. 2022. The sounds and smells of the south: The auditory and olfactory in Fitzgerald’s Tarleton trilogy. In Southern influences on Scott and Zelda Fitzgerald. A collection of critical essays, ed. Kirk Curnutt and Sara Kosiba. Lexington: Lexington Books. Scaff, Susan von Rohr. 2009. The vitality of music in Mann and Joyce: Wagner in “fullness of harmony” and “sirens”. Orbis Litterarum 64 (6): 439–456. Schafer, Murray. 1977. The soundscape: Our sonic environment and the tuning of the world. New York: Knopf. Scher, Steven Paul. 1968. Verbal music in German literature. Vol. 2, Yale German studies. New Haven: Yale University Press. Schleifer, Ronald. 2011. Modernism and popular music. Cambridge: Cambridge University Press. Schober, Regina. 2010. Translating sounds: Intermedial exchanges in Amy Lowell’s ‘Stravinsky’s three pieces ‘grotesques’, for string quartet’. In Media borders, multimodality and intermediality, ed. Lars Elleström, 163–174. New York: Palgrave Macmillan. Schweighauser, Philipp. 2017. Das Rauschen modernistischer Form: John Dos Passos, Zora Neale Hurston und die Soundscapes der Moderne und frühen Postmoderne. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 495–507. Berlin: De Gruyter. Seed, David. 2009. Cinematic fictions: The impact of the cinema on the American novel up to the second world war. Liverpool: Liverpool University Press. Seiler, Thomas. 2006. Instrumental-, Partiturlyrik, Tonpoesi Zur Bedeutung musikalischer Begrifflichkeit im scwhedischen Modernismus. In Literatur und Musik in der klassischen

31

The Sonification of Modernist Fiction: A Critical Review

801

Moderne: Mediale Konzeptionen und intermediale poetologien, ed. Joachim Grage, 135–153. Würzburg: Ergon Verlag. Sen, Suddhaseel. 2018. Rabindranath Tagore and musical modernity in early twentieth-century Bengal. In Essays on music and language in modernist literature: Musical modernism, ed. Katherine O’Callaghan, 126–139. New York: Routledge. Spiegel, Alan. 1976. Fiction and the camera eye: Visual consciousness in film and the modern novel. Charlottesville: University Press of Virginia. Sterne, Jonathan. 2012. The audible past: Cultural origins of sound reproduction. Durham: Duke University Press. Sternfeld, Frederick. 1957. Poetry and music: Joyce’s Ulysses. In Sound and poetry, ed. Northrop Frye, 16–54. New York: Colombia University Press. Stevens, Garrett. 1990. Reading voices: Literature and the Phonotext. Berkeley: The University of California Press. Stewart, Garrett. 1990. Reading voices: Literature and the phonotext. Berkeley: University of California Press. Stewart, Jim. 2014. The birth of Rachel Vinrace from the spirit of music. In Virginia Woolf and music, ed. Adriana Varga, 110–133. Bloomington: Indiana University Press. Susman, Warren. 1984. Communication and culture. In Mass media between the wars: Perceptions of cultural tension, 1918–1941, ed. Catherine L. Covert and John D. Stevens, xvii–xxxii. Syracuse: Syracuse University Press. Sutton, Emma. 2013. Virginia Woolf and classical music: Politics, aesthetics, form. Edinburgh: Edinburgh University Press. Tenngart, Paul. 2002. Poetry as music: The significance of musicalized poetry in the aftermath of Swedish modernism. In Cultural functions of intermedial exploration, ed. Erik Hedling and Ulla-Britta Lagerroth, 149–161. Amsterdam: Rodopi. Thompson, Emily. 2002. The soundscape of modernity: Architectural acoustics and the culture of listening in America, 1900–1933. Cambridge, MA: MIT Press. Thompson, Trina. 2014. Sounding the past: The music in between the acts. In Virginia Woolf and music, ed. Adriana Varga, 204–226. Bloomington: Indiana University Press. Trotter, David. 2007. Cinema and modernism. Malden: Blackwell Publishing. ———. 2013. Literature in the first media age: Britain between the wars. Cambridge, MA: MIT Press. Vaget, Hans Rudolf. 1989. Thomas Mann und James Joyce: Zur Frage des Modernismus im “Doktor Faustus”. Thomas Mann Jahrbuch 2: 121–150. Van Durme, Debora. 2018. Imagism’s musical sympathies: Amy Lowell and Claude Debussy. In Essays on music and language in modernist literature: Musical modernism, ed. Katherine O’Callaghan, 173–187. New York: Routledge. Varga, Adriana, ed. 2014. Virginia Woolf and music. Bloomington: Indiana University Press. Waddell, Nathan. 2017. Modernism and music: A review of recent scholarship. Modernist Cultures 12 (2): 316–330. Wasilewska-Chmura, Magdalena. 2000. Musik – Metafor – Modernism. En linje i den savenska modernismens poetologiska reflexion. Stockholm: Almqvist & Wiksell International. White, Terence. 1936. Joyce and music. Chesterian 17: 166. Wolf, Werner. 1999. The Musicalization of fiction: A study in the theory and history of intermediality. Amsterdam: Rodopi. ———. 2002. Towards a functional analysis of intermediality: The case of twentieth-century musicalized fiction. In Cultural functions of intermedial exploration, ed. Erik Hedling and Ulla-Britta Lagerroth, 15–34. Amsterdam: Rodopi. ———. 2015. Literature and music: Theory. In Handbook of intermediality: Literature – image – sound – music, ed. Gabriele Rippi. Berlin: De Gruyter. Woolf, Virginia. 1940. Thoughts on peace during an air raid. The New Republic, 21 October. ———. 1965. The years. New York: Harcourt. ———. 1975–80. The letters of Virginia Woolf. 6 vols. New York: Harcourt Brace Jovanovich.

802

N. Salmose

———. 1985. Moments of being. San Diego: Harcourt Brace & Company. ———. 1986. Modern fiction. In The essays of Virginia Woolf, ed. Stuart Nelson Clarke and Andrew McNeillie, 157–165. London: Hogarth Press. Zender, Karl F. 1984. Faulkner and the power of sound. PMLA 99 (1): 89–108. Zimring, Rishona. 2002. Suggestions of other worlds: The art of sound in “the years”. Woolf Studies Annual 8: 127–156.

Adaptation and Sound

32

Kate Newell

Contents Introduction: Why Sound? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sound in Relation: Juxtaposition, Integration, and Accompaniment . . . . . . . . . . . . . . . . . . . . . . . . . . Making Sense of Literary Adaptation: Lecturers, Dialogue, and Words . . . . . . . . . . . . . . . . . . . . . . Literature on the Air: Adaptation and Radio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Audiobook: “A Movie in Your Mind” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion: Sound, Adaptation, and Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

804 808 815 819 824 826 829

Abstract

Virtually all processes of adaptation necessitate some consideration of sound. Some types of adaptation foreground sound as a distinguishing feature, whereas others de- or re-prioritize specific sound technologies. This chapter provides a sampling of sound- and adaptation-related conversations from the fields of adaptation studies, film history, and intermedial studies to demonstrate the range of sound-based topics relevant to adaptation studies and to provide a foundation for future points of inquiry. This chapter is organized primarily according to medial and modal relationships commonly considered within the field of adaptation studies as unfolding dichotomously between word and image, image and image, word and word, and so on. I consider these relationships in terms of sound technologies and the related adaptive engagement, as well as the critical and historical conversation generated in response to those relationships. The first examples of sound technologies are those in which the adaptive relationship becomes evident through the juxtaposition and integration of sound with or as a source text, as in the illustrated songs, sound effects, lecturers, and musical accompaniments that characterize early cinema programs. Writing on such integrations commonly emphasizes K. Newell (*) Savannah College of Art and Design, Savannah, GA, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_39

803

804

K. Newell

juxtaposition, content recycling, source (re)interpretation, and other elements of interest to adaptation studies. Next, I consider adaptive relationships in which sound technologies replace the source in exhibition, as in radio adaptations of literary sources and audiobooks. Writing on such relationships discusses issues of fidelity and abridgement, as well as market influences on adaptation choices and processes. Focusing on adaptation-related scholarship outside of the book-tofilm, word-to-image predominance illuminates considerations specific to certain types or instances of adaptation, as well as those consistent across modes of adaptation; focusing on sound technologies across a range of types of adaptation highlights the inherent intermediality in all adaptive relationships. Keywords

Adaptation · Sound · Soundtrack · Accompaniment · Intermediality

Introduction: Why Sound? Virtually all processes of adaptation necessitate some consideration of sound. Film adaptations of novels, for example, reconceptualize literary acoustics and readers’ mental soundscapes as audible diegetic and non-diegetic sound, whereas novelizations of films follow reverse strategies in reconceptualizing the multilayered soundtrack of the cinematic experience through literary devices. Some types of adaptation foreground sound as a distinguishing feature, as in operatic, balletic, and musical adaptations, whereas others de- or re-prioritize specific sound technologies, as in film and television adaptations of radio plays and podcasts. Despite such ubiquity, considerations of sound as a feature or object of the adaptation process or as an adaptive agent are extremely rare within the field of adaptation studies. Similarly, within sound studies and related fields, the relationship between sound and other modal and medial participants has rarely been conceptualized as adapting, adaptive, or adaptation. This chapter provides a sampling of sound- and adaptation-related conversations from the fields of adaptation studies, film history, and intermedial studies to demonstrate the range of sound-based topics relevant to adaptation studies (and adaptation-based topics to sound studies) and to provide a foundation for future points of inquiry. “Adaptation” takes on many meanings within the field of adaptation studies. Most conceptualizations of adaptation have at their root two of the Oxford English Dictionary’s denotative definitions of adaptation: “[t]he action or process of adapting one thing to fit with another, or suit specified conditions, esp. a new or changed environment, etc.” and “[a]n altered or amended version of a text, musical composition, etc., (now esp.) one adapted for filming, broadcasting, or production on the stage from a novel or similar literary source” (“Adaptation”, 2022). As may be imagined, the act of creation through modification and accommodation and the relationship between two (or more) versions of a creative work conveyed in these definitions has provided grist for amendment, clarification, and complication as

32

Adaptation and Sound

805

adaptation scholars attempt to both narrow and expand understandings of what adaptation is and does1. Dudley Andrew’s description, while focused on film adaptation, is perhaps the most straightforward: “the matching of the cinematic sign system to prior achievements in some other system” (1984, 96). Timothy Corrigan, also writing on film adaptation, labels adaptation “the most common practice in the exchange between literature and film, describing the transposition of a novel, play, or other literary source to film” (1999, 20). For Linda Hutcheon adaptation is a process and product that meets three general benchmarks: “An acknowledged transposition of a recognizable other work or works,” “A creative and interpretive act of appropriation/salvaging,” and “An extended intertextual engagement with the adapted work” (2013, 8–9). Julie Sanders has described adaptation as a “sub-section of the over-arching practice of intertextuality,” noting that adaptation “can be a transpositional practice, casting a specific genre into another generic mode,” and “can parallel editorial practice in some respects, indulging in the exercise of trimming and pruning; yet it can also be an amplificatory procedure engaged in addition, expansion, accretion, and interpolation” (2006, 17–18). Though by no means an exhaustive account of how adaptation has been defined or described, the four share the through-thread of adaptation as an engaged, intertextual relationship between (at least) two entities. Rarely, however, are sound or sound-based technologies considered as one of the entities or agents of engagement. Adaptation studies as a field of inquiry came into being largely through the shared interests and scholarly output of academics interested in examinations and analyses of film, television, and stage adaptations of literary works (Leitch 2007, 3–7; Naremore 2000, 1–3; Cartmell and Whelehan 2007, 3). This priority and its legacy is evident in the titles of foundational monographs and edited collections: George Bluestone’s Novels Into Film (1957), Robert Richardson’s Literature and Film (1969), Morris Beja’s Film & Literature: An Introduction (1979), Brian McFarlane’s Novel to Film: An Introduction to the Theory of Adaptation (1996), Timothy Corrigan’s Film and Literature: An Introduction and Reader (1999), Robert Stam’s Literature Through Film: Realism, Magic, and the Art of Adaptation (2005a), Stam’s and Alessandra Raengo’s Literature and Film: A Guide to the Theory and Practice of Film Adaptation (2005b) and A Companion to Literature and Film (2004), and Deborah Cartmell’s The Cambridge Companion to Literature on Screen (2012). As the field expands to include more diverse forms of adaptation, publication titles reflect more comprehensive conceptualizations of and approaches to adaptation and adaptation processes, among them Linda Hutcheon’s A Theory of Adaptation (2013); Christa Albrecht-Crane and Dennis Cutchins’s Adaptation Studies: New Approaches; Thomas Leitch’s The Oxford Handbook of Adaptation Studies (2017) (see Cartmell 2012, 7); Jørgen Bruhn, Anne Gjelsvik, and Eirik Frisvold Hanssen’s Adaptation Studies: New Challenges, New Directions (2013); Julie

1

Kamilla Elliott provides a comprehensive overview of the range of definitions and terminology used to describe and taxonomize adaptation within adaptation theory (2020, 179–191)

806

K. Newell

Grossman and R. Barton Palmer’s Adaptation in Visual Culture: Images, Texts, and Their Multiple Worlds (2017); my own Expanding Adaptation Networks: From Illustration to Novelization (2017); Kyle Meikle’s Adaptations in the Franchise Era, 2001–16 (2019); and Lissette Lopez Szwydky’s Transmedia Adaptation in the Nineteenth Century (2020). Contemporary studies of adaptation commonly recognize the intertextual, intermedial, and intermodal qualities of adaptive processes and of given examples and test cases. Such recognition notwithstanding, Lars Elleström highlights the tendency to study multimodal and multimedial phenomena via the prioritization of one or another characteristic, as well as the tendency to prioritize language as the “norm” modality. In “The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations” (2021), Elleström identifies problematic “tendencies” in the study of mediality, among them that “[m]edia in general are studied through concepts developed for language analysis” and that “[m]isleading dichotomies,” such as image/text, visual/verbal, “structure the arguments” (5–6). Elleström reasons that, “[b]ecause being visual is a sensorial trait and being verbal is a semiotic trait, it is pointless to oppose the two” (7). For Elleström, applying dichotomies to intermedial productions can generate unproductive and reductive (and, often, historically inaccurate) conclusions. Similarly, Rick Altman finds that media scholarship often ignores the intermedial reality of creative works and their exhibition: “[s]ilent films are interpreted without reference to their musical accompaniment; feature films are treated independently of the full program of music, shorts, and newsreels that originally accompanied them” (1992, 9); and “[i]n spite of cinema’s historical connections to theater and the performing arts in general, critics have preferred to emphasize cinema’s debt to the novel” (10). Considering sound intentionally as an object or agent of adaptation allows for more pluralistic understanding of how features of the adaptation process inflect and inform relationships between texts, expands the intertextual and intermedial network of creative products, and underscores adaptation as unfolding within and through complex networks. Sound serves myriad functions relevant to adaptation and adaptation studies: sound can create the illusion of a multidimensional space, as in the case of radio plays and audiobooks, and can add texture to created worlds in other modalities, as in the case of art installations and film (Gorbman 1987, 37; Altman 1992, 15–16; Hutcheon 2013, 40–42; Geraghty 2008, 180). Sound can direct audience responses and can reinforce, challenge, or destabilize images and other forms of representation. For example, George Bluestone comments on the early sound-image experiments of Sergei Eisenstein and other Soviet filmmakers in which “sound is used to reinforce, comment on, anticipate the film’s visual images” (1957, 30). In part, sound is able to communicate in diverse contexts because, like images, language, and other signification systems, audiences have internalized coding associated with specific sounds and sound patterns. Katherine Barnes Echols extends to radio Brian McFarlane’s discussion of structural and cultural codes in literature and film: “On radio, language, sound codes, and description encourag[e] a particular response to a given situation or

32

Adaptation and Sound

807

character” (2017, 53). Claudia Gorbman likewise comments on coding in the context of film music, explaining that “standard film music efficiently establishes historical and geographical setting, and atmosphere, through the high degree of its cultural coding” (1987, 58). Further, “In emphasizing moods or feelings, in specifying or delineating objects for the spectator’s attention, music enforces an interpretation of the diegesis” and “guides the spectator’s vision both literally and figuratively” (Gorbman 32; 11; see also Anderson 1997, 12). Similarly, Hutcheon avers that “[s] oundtracks in movies [. . .] enhance and direct audience response to characters and action” (2013, 41). In addition to contributing to world building and to audiences’ response to adaptation, sound can impact perceptions of a work’s fidelity to source material. A novel reader, for example, likely develops an impression of the pitch, timbre, and intensity of character’s voice that may or may not align with the actor hired to play that character in the film or audio adaptation. Matthew Rubery highlights this challenge in the context of audiobook adaptations, explaining that, “the spoken narration of a printed text introduces a potentially ideological dimension through its aural rendering of such controversial categories as class, ethnicity, gender, race, and nationality” (2011, 14). Rubery adds that, “[c] haracteristics often thought of in terms of visual representation are inflected in audio formats by the narrator’s voice. [. . .]. Voice actors are often selected as much for their perceived fidelity to the politics of a text as for their vocal talents” (14). As isolated comments pulled from a sampling of writing on adaptation, these observations offer a glimpse of the innumerable directions open to considerations of sound in adaptation. This chapter is organized primarily according to medial and modal relationships commonly considered within the field of adaptation studies as unfolding dichotomously between word and image, image and image, word and word, and so on. Here, however, I consider these relationships in terms of sound technologies and the related adaptive engagements and the critical and historical conversation generated in response to those relationships. The first examples of sound technologies are those in which the adaptive relationship becomes evident through the juxtaposition and integration of sound with or as a source text, as in the illustrated songs, sound effects, lecturers, and musical accompaniments that characterize early cinema programs. Writing on such integrations commonly emphasizes juxtaposition, content recycling, source (re)interpretation, and other elements of interest to adaptation studies. Next, I consider adaptive relationships in which sound technologies replace the source in exhibition, as in radio adaptations of literary sources and audiobooks. Scholars writing on such relationships discuss issues of fidelity and abridgement, as well as market influences on adaptation choices and processes. Focusing on adaptationrelated scholarship outside of the book-to-film, word-to-image predominance illuminates considerations specific to certain types or instances of adaptation, as well as those consistent across modes of adaptation; focusing on sound technologies across a range of types of adaptation highlights the inherent intermediality in all adaptive relationships.

808

K. Newell

Sound in Relation: Juxtaposition, Integration, and Accompaniment Adaptation studies explore creative works “in relation,” that is, how a work in one medium relates to a corollary work in another or the same medium, as well as how media and modalities engage in different contexts. Film historians have well established that, in the first few decades of cinema, films were not the sole attraction of a given exhibition venue – they were but one offering in a menu that included musical performances, vaudeville acts, news reports, and other forms of entertainment (Altman 2004; Gunning 1996, 73; Anderson 1997, 6). Further, due to variables such as venue size, size and talents of staff, and so on, elements of the program were not always discrete entities: concurrency and overlap were not uncommon. As Rick Altman has pointed out, while prior to 1908 films were likely exhibited in silence, commonly the exhibition would unfold within a program that included other soundbased events or within a space in which external sounds (e.g., ballyhoo, hawkers) might be discernable (1996). The silence of the so-called silent film, Julie Hubbert explains, is not “a lack of sound, but a lack of integration” (2011, 15). Silent films ceased to be thought of as such “not with the inclusion of music and sound, but with their mechanization, the technical innovation that allowed music to be represented alongside the images” (Hubbert 2011, 15). In other words, music and sound were part of the film experience from the beginning, whether indirectly, as in the case of audience-wrangling storefront performers and musical interludes, or directly, as is the case of exhibitors who strategically integrated sound to amplify the viewing program, as well as visuals to amplify the sound program. Though not conceptualized as adaptation by scholars of film history, juxtapositions and integrations of sound and image, such as occurs in prologues, illustrated songs, sound effects, and musical accompaniment; are adaptive, adapting, and interpretive; and involve recontextualizations and appropriations consistent with contemporary experiences of adaptation. Prologues and illustrated songs offer two very different uses of juxtaposition in early film programs. Prologues dramatized moments from and provided a theatrical setup for the program’s film and, as their title suggests, occurred immediately prior to the film. A popular feature of exhibitions in the 1920s (Altman 2004, 385–387), prologues offered “a live stage act featuring elaborate sets, narratives, music, and dance that bore a specific connection to the film’s plot characters, or setting” (Simonson 2019, 4). In short, prologues were live-action, stage adaptations of the film program. Given that they “were designed to get the audience in the mood for the film to come” (Altman 2004, 385), we might think of as analogous to contemporary film trailers. Mary Simonson outlines the unique, intermedial viewing experience manifested by the juxtaposition of the prologue and the film: “these acts brought the stage and screen into a sort of mimetic relationship [. . .]. Yet unlike most mimetic acts, which depended on audience familiarity with their origin, here the ‘original’ film sequence only became available to audiences after they had witnessed its double onstage. The restaging primed audiences for ‘the real thing’” (2019, 12). Thus, rather than have the “source” determine the audience’s response to the adaptation, the

32

Adaptation and Sound

809

adaptation shapes and informs audiences’ responses to the source. To further vary the available significations and resulting networks of meaning generated by the juxtaposition of the prologues and the related film, Simonson notes that “the vast majority of prologues were created on a local scale,” meaning that exhibitors “developed acts in-house and used the theater’s resources and personal as well as local talent, to stage them” (2019, 8). Thus, a spectator might at one venue experience a prologue that highlights a moment deemed significant by that exhibitor and experience a completely different prologue for the same film at a different venue. Illustrated songs, which predate prologues, offer another unique example of early adaptation in which meaning-making is generated through the juxtaposition of two creative works. Popular features of the vaudeville and nickelodeon theater from approximately 1904 until 1910, “illustrated song” refers to a type of performance that pairs a song and a series of hand-painted still-frame slides that relate to the content or tone of the song (Abel 2001, 143; Cooke 2008, 7; Hubbert 2011, 19; Trezise 2019, 6–7; Cook 2019, 61–62). As the singer performs, the slides change. Popular or well-known songs were the most likely candidates for adaptation, and the slides offered another modality through which audiences could experience a favorite song and also concretized a particular reading of the song. Commonly, the slides took a literal approach: a song set by a lake about a woman longing for a lost love would be paired with slides that depicted a woman standing at a shoreline gazing into the distance. Singers were also often well-known to their specific audience, as is the case for the film prologues. Richard Abel relates that “the tandem act of illustrated songs and moving pictures” allowed for “a unique mix of national mass culture and local popular culture,” in that the visual components (e.g., slides, film) were being exhibited across the country whereas the performance aspect (e.g., the singer) would often be a local favorite (2001, 147). While the song’s lyrics would remain consistent, the inflection of specific performers coupled with geographical “star power” would likely impact audiences’ understanding of the story being told through the juxtaposition of song and image. The juxtaposition that characterizes the illustrated song allows its modal relationships to inflect the broader program in which the illustrated song is seated. Typically, Abel explains: “illustrated songs and moving pictures [. . .] came in tandem at the end [of the program], probably because they used the same projecting apparatus” (2001, 143–144; also Trezise 2019, 7), and “frequently the picture and songs explicitly complement or ‘counterpoint’ one another” (Abel 2001, 146). That is, the topic of the illustrated song and that of the short film often communicated in interesting ways. Abel shares the example of the pairing of the illustrated song of “The Good Old USA,” a patriotic tune reflecting on the pride of calling the United States “home” with Pay Day, a film in which “a drunken worker unsuccessfully tries to negotiate the streets of Paris” (2001, 147). In such cases, the slide show and song lyrics comment on each other, and the film images then comment on the slide show and lyrics. The audiences’ understanding of the texts as a result of this sign-signifier layering and intertextuality would no doubt be palimpsestuous. The illustrated song offers an instructive example of the inherent intermediality of early exhibition programs. For Altman, the “[f]ailure to recognize the role of illustrated songs within the

810

K. Newell

nickelodeon program has had serious consequences for silent film sound historiography,” as has a general “[i]gnorance or misunderstanding of the role played by illustrated songs” (2004, 193). Further, for adaptation studies, the illustrated song forms a key node in the network of examples of adaptation that utilize strategies of juxtaposition – a network that includes non-audio genres such as illuminated manuscripts and illustrated novels for which the juxtaposition is physical and material and that extends to genres for which the juxtaposition might be more internalized and perceptual, as in the case of film adaptations of popular songs. Illustrated songs serve as precursors for more contemporary forms of visual adaptations of musical source texts. Film adaptations based on hit singles and albums are not uncommon; popular examples include Ode To Billy Joe (1976), based on Bobbie Gentry’s 1967 song, “Ode to Billie Joe”; Take This Job and Shove It (1981), based on a 1977 David Allan Coe song popularized by Johnny Paycheck; Chattanooga Choo Choo (1984), based on the Glenn Miller Orchestra’s 1941 hit; Love Potion No. 9 (1992), inspired by the Clovers’ 1959 hit; and Bad Romance (2011), based on 2009 song by Lady Gaga. In comparing the process of adapting a song to that of a novel, Ian Inglis writes that “the song-as-source could not be more different from the novel-as-source: whereas the contents of the novel are invariably reduced in size to satisfy the demands of the screenplay, those of the song are invariably expanded” (2012, 312). Inglis’s “Music into Movies: The Film of the Song” focuses on a range of popular music and movie intersections, including films that take their titles from popular songs. In such cases, Inglis finds that the film’s “associations” with the song “tend to be general, and are often inferred through the audience’s prior knowledge of the music: the tracks supply broad emotional triggers rather than specific references” (314). Inglis likewise discusses “the adaptation of pre-existing albums into full-length screenplays” (e.g., The Who’s Tommy 1969 rock opera, which was adapted into the 1975 film) and adaptations “where the lyrics of individual songs have been adapted into full-length movie screenplays” (314–316), as is the case for Yellow Submarine (1968), Harper Valley PTA (1978), and others. For Inglis, these films “use music not only for its traditional functions—to establish mood, to reflect emotional and mental states, to locate time and place, to sustain structural unity—but also for its explicit provision of formal narrative content” (317). For illustrated songs and the form’s successors, the nonmusical elements provide a context for reading the song just as the song, and related musical elements provide emotional motivation and context for the images. The discussion of juxtaposition and meaning-making would be incomplete without a consideration of the adaptive role of sound effects in early cinema programs. Sound effects were particularly common after 1906 and through the 1920s, and what is understood as “effects” is broadly conceived (Bottomore 2001, 129). Here, I refer to those sounds exhibitors employed to provide depth and verisimilitude to the film experience, such as pistol shots to accompany a war film or water splashing for a scene by the ocean (Bottomore 1999, 487). Stephen Bottomore comments that “[b]y 1909 the Bioscope was talking of the unnaturalness of seeing events such as explosions, typhoons, and battles without their accompanying sounds, and the need to break this ‘silence of death’ in films” (2001, 130). The effects were, of course,

32

Adaptation and Sound

811

juxtaposed against the images to which they were intended to give voice, and this relationship sparked commentary related to the fidelity or infidelity of that relationship. Trade presses, Bottomore explains, gave “advice on the devices needed to make specific sounds: such as using a rattle to imitate rapid gunfire” (1999, 488), and publications commented on ineffectual effects. An article in Moving Picture World offers: “The imitations should be fairly accurate or they shouldn’t be attempted. Inaccuracy is worse than nothing. It creates wrong impressions and often it wrongly interprets the pictures” (quoted in Bottomore 2001, 130). Not surprisingly, effects men who took “a far too literal approach to their job of reflecting in sound what was on screen” met with criticism, as evidenced by the “Percy Peashaker” moniker (Harrison 2011). Effects men, or “drummers,” who earned this title tended “to make a noise for anything, no matter how unimportant it was within the scene, while failing to take a cue from the mood of the scene” (Bottomore 2001, 132). Such effects are not typically characterized under the rubric of adaptation, yet using the sound produced by one object (e.g., coconut shells) as the source for the sound of another (e.g., horse hooves on cobblestones) is an activity consistent with those commonly theorized as adaptation. When the adapted sounds are then juxtaposed against a silent image as an “effect,” the range of called upon signifiers multiplies. Musical accompaniment includes, developed alongside of, and evolved out of, illustrated songs, sound effects, prologues, and other elements of the exhibition experience and likewise engages in juxtaposition and other adaptation strategies through the interplay between music and other aspects of the program and in the ways in which songs themselves are adapted to meet new contexts. Though perhaps not as explicit as in the illustrated song, the pairing of music and narrative film likewise impacts how both are received and interpreted. Musical accompaniment for early films took on a variety of forms ranging from individual singers and musicians to bands and full orchestras and is difficult to discuss without also discussing synchronization. While the desire for synchronization, Altman points out, became “widespread” around 1910 (2004, 208–209), early examples of music and film alignment are evident in various forms from about 1905 to 1915. Synchronization in this period was less about establishing emotional tension that in establishing action (Altman 2004, 209). Tim Anderson explains that, “As a general rule, the player had to be aware of the on-screen situation and understand all the possible combinations between the music and picture that could be made to make the picture as lively as possible” (1997, 10). Of course, some players crafted purposefully non-synchronized and disruptive aesthetic experiences. Louis Reeves Harrison’s well-known 1911 article “Jackass Music” cites behaviors such as “playing to members of the audience, instead of playing to the film, musicians leaving their post during the most dramatic portion of the film,” and so on (quoted in Anderson 1997, 9; Harrison 2011). Beginning in the 1910s, however, the desire for the musical accompaniment to “illustrate,” align, or synchronize more broadly with the film was strong (Hubbert 2011, 23), and the film and music industries published numerous resources to support this desire. Often, the resources included general music and musical suggestions that could align with a number of visual and narrative contexts, or they included recognizable songs and musical arrangements that would provide

812

K. Newell

additional layers of meaning for audiences. The adaptive relationship that emerges in the early programs’ use of recycled and repurposed songs and musical cues paved the way for similar integrations of recognizable songs and arrangements in contemporary films. Resources, such as industry newspapers and periodicals, musical collections developed specifically for film application, and cue sheets, placed venue organizers, musicians, and others involved with exhibition in a position to make performance choices that shape and inform how audiences responded to the film and that generate new and varied textual and intertextual experiences. Some companies marketed music that aligned with general narrative tensions (e.g., chase scene, dramatic departure) and that could be integrated into any film regardless of overall narrative trajectory (Marks 1997, 10–11). In general, Altman elucidates, musical selections were “labeled for various narrative situations, character types, or source music imitations (bugle call, bagpipes, etc.)” (2004, 258). Other companies marketed pared down and abridged versions of classic and popular songs to meet a broader range of skill sets, and still others marketed original work. Another resource, cue sheets, “offered suggestions,” Kendra Preston Leonard points out, “to cinema musicians on selecting and performing music for silent motion pictures” (2018, n. p.). The suggestions, Preston Leonard explains, were for “specific pairings of an individual piece of music with each scene” (2019, 46). These sheets, developed and circulated from about 1910, “were often modified, used merely as the basis for ideas, or even ignored” in favor of music preferred by the musician (Preston Leonard 2018, n.p.; see also Marks 1997, 9; Anderson 1997, 11; Bottomore 1999, 487; Cooke 2008, 15–18; Hubbert 2011, 20). Similarly, Daniel Goldmark writes that “[m]usicians were in no way obliged to use all or any of such suggestions, but given the often minimal prep time available for a new film score [. . .] having a prefabricated list of music to use would certainly save both time and effort” (2017, 783; Hubbert 2011, 19). Goldmark adds that “[s]ome films even had more than one published cue sheet, providing multiple and possibly conflicting semi-authoritative readings of a film’s musical landscape” (2017, 83). As an example of the preference for more original work, beginning in 1910, the Vitagraph Bulletin “provid[ed] musical accompaniment suggestions for all Vitagraph titles” and “looked to original scores as an appropriate method of emphasizing the ‘dignity’ of their prestige films” (Altman 2004, 256). Regardless of whether the exhibitors integrated musical suggestions or took cue sheet suggestions, the music was already adapted and adaptive prior to entering the exhibition scenario and produced additional layers of adaptation in the new context. That audiences could see the same film with different musical landscapes or hear the same music aligned with different films creates a scenario of an adaptive mix and match and illuminates the range of audiences’ “takeaway” experiences. A common adaptation practice is that of recycling music familiar to audiences to fit distinct films (Altman 2004, 259). For example, many of the musical scores that accompanied early films “were borrowed directly from operas” or composed of recognizable classical pieces that were “reduce[d] [. . .] to a familiar melody (Altman

32

Adaptation and Sound

813

2004, 251, 263; Anderson 1997, 14; also Cooke 2008, 132). In some cases, the musicians hired to provide accompaniment for a film “had experience playing for a melodrama theater, and so might play some of the hurries, agits, or other pieces regularly recycled for melodramatic situations” (Altman 2004, 205; also Cooke 2008, 10). Simon Trezise explains that “composers for theatre could simply transfer their music over to silent film accompaniment” (2019, 10). Recycled sound and musical effects were also used to provide commentary on the action on the screen (Cooke 2008, 13). The impact of this effort could be best achieved through the juxtaposition of narratives and songs familiar to the audience: “[f]amiliar dramas could be undercut by judiciously chosen songs, whose title or lyrics offered everything from veiled commentary to outright lampoons,” and popular songs “were carefully matched to a particular character or situation” (Altman 2004, 220; 221). Altman offers as an example an 1870–1871 production of Hamlet featuring comedian George L. Fox in which “[e]ach song is chosen not because of its musical characteristics, but because its title provides an ironic or comic match to the events it accompanies” (Altman 2004, 221). Goldmark offers a similar observation in his analysis of The Jazz Singer (1927): “[t]he pop songs not used in synchronized sound sequences occur as underscore only, amplifying or reinforcing narrative points” (2017, 83). As Anderson has argued, in the context of “jackass” accompaniment, unanticipated and unsanctioned accompaniments often “yielded [. . .] semantic pleasures,” “unmoored” meaning, and “introduc[e] the chance for a variety of textual discoveries” (1997, 12–13). Recycling continues to shape audio experiences in contemporary films, as well. For example, John Williams’s famous theme music for the shark in Jaws (1975) has been recycled for comedic impact in Close Encounters of the Third Kind (1977), Airplane! (1980), and Spaceballs (1987). Similarly, Bernard Herrmann’s soundtrack for the shower scene in Alfred Hitchcock’s Psycho (1960) has been repurposed in numerous films, such as Tie Me Up! Tie Me Down! (1989), Monsters, Inc. (2001), Finding Nemo (2002), She’s the Man (2006), and many others, as well as in numerous television series, ranging from The Simpsons to Arrested Development to NCIS to Criminal Minds. Less intentional recycling also occurs, as when Williams, working simultaneously on Harry Potter and the Chamber of Secrets (2002) and Star Wars Episode II: Attack of the Clones (2002), included “an unmistakably similar sequence in their soundtracks” (Brooks 2021). As evident in the examples of the illustrated song, early forms of accompaniment, and recycling, “the song form is one of the most naturally adaptive to significant variation” (Ingham 2017, 324). For Mike Ingham, a challenge “that inevitably arises in any discussion of song adaptation [. . .] is how to distinguish between the myriad possible ways in which a particular song may evolve over the longue durée of its life in any given culture” (2017, 325). Familiar melodies, for example, can be adapted across genres and contexts, and artists commonly reinterpret their own songs to allow them to signify in new contexts or to take on new or clarified meaning (Ingham 2017, 325–327). Similarly, song adaptations or covers offer another avenue by which songs come to signify in new contexts,

814

K. Newell

as when, Hutcheon offers, “a female singer [. . .] covers male misogynist songs, the new vocal angle subverts the adapted works’ sexist ideology” (2013, 93). The film soundtrack is the most popular way in which music may be adapted or recontextualized. Of course, not all soundtracks function in the same way. Jerrold Levinson divides film soundtracks into two general camps: composed and appropriated. The former category includes “music composed specifically for the film in question,” whereas the latter, commonly known as a compilation soundtrack, features music composed for another context and “appropriated” to the next context (2004, 483). Levinson explains that “with appropriated scores the issue of specific imported associations, deriving from the original context of composition or performance or distribution [. . .] is likely to arise” (483). Such “imported associations” may complement or comment ironically on the images and actions on the screen. In Levinson’s view, “music composed for a film [. . .] is more likely to be purely narrative in function than preexisting music appropriated by a filmmaker” (483). Stephanie McKnight focuses on both types in her analysis of the soundtrack in Sofia Coppola’s 1999 film adaptation of Jeffrey Eugenides’s novel, The Virgin Suicides. For McKnight, the non-diegetic soundtrack composed for the film by the French duo, Air, supports a dream-like characterization of the four Lisbon sisters as “mere impression[s],” rather than concrete realities (2011, 117). McKnight argues that the music “combined with images of the Lisbon girls [. . .] heightens the subjective male view of the girls as otherworldly beings” (116), “as distant spectacles” (117), and “stresses [the girls’] constructedness, how they’ve been distorted through the eyes of our collective male narrator” (127). The film also integrates compiled soundtrack elements, as well. Heart’s “Magic Man” plays whenever teen heartthrob Trip Fontaine appears on screen (122–123). McKnight, quoting from Rick Altman, relates that “In a sense, we never hear a popular song for the first time; we are always hearing it again, each time with implicit reference to previous hearings” (McKnight 2011, 122). Glenn Jellenik, for example, offers a discussion of the strategic integration of recognizable pop music into the Shrek (2001) soundtrack, including Smash Mouth’s “All Star” and Joan Jett’s “Bad Reputation.” In each case, “the piece of music has a prior existence to its appearance in the film and it functions in certain ways as a signifier that has nothing to do with Shrek. Thus the film adapts the song, pulls it out of its original context and puts it to its own use” (Jellenik 2010, 227). Jellenik avers that “more than simply reflect the adaptive process, the music of a film affects it, and at times acts as one of its prime engines” (225). The synchronization of music and visual and their purposeful (or accidental) asynchronization impact audiences’ experience of a creative work. Like the adaptive relationship developed via the synchronization process, the musical pieces occupy an always-already position of adaptation that further amplifies the significations available. In addition to musical accompaniment, program lecturers and commentators provide another avenue of adaptive and intermedial interplay.

32

Adaptation and Sound

815

Making Sense of Literary Adaptation: Lecturers, Dialogue, and Words Early film programs, be they exhibited in nickelodeons, storefronts, or more expansive venues, were comprised of disparate components and performances, including film, slide shows, skits, songs, and more, that lacked a cohesive narrative, thematic, or chronological through-thread. Film lecturers played an integral role in pulling the program together. Lecturers, Altman relates, “found themselves obliged to weave films and slides into a single cohesive fabric” (2004, 137–138). Judith Buchanan adds that the lecturer helped to “smooth over narrative ellipses and the stop-start progress of successive tableaux by weaving a series of on-screen ‘moments’ into a continuous narrative and so the succession of scenes into a cohesive fictional world” (2012, 28). The film lecturer, Richard Crangle explains, “gave a verbal explication or elaboration of a visual scene, usually to locate the scene in a wider context and especially in a narrative or informative sequence” (2001, 45). For Germain Lacasse, “Film lecturers brought narrative continuity that editing was not yet able to bring; they supplied context, explaining the sources and specific qualities of the film; and they translated the intertitles of imported films [. . .]. They were performers, capable of dramatizing a film’s diegesis and the work of its actors” (2012, 487). In the process of providing cohesion, the lecturer engaged in an adaptive relationship with the text, providing connections between and interpreting the various materials, and this relationship reflects the ways in which content organization and interpretation generated distinct experiences of a text or texts. The lecturer’s role was even more important in the case of film adaptations of literary works, particularly those with which an audience would not be familiar. Buchanan explains that while “catalogues and programs from the 1890s and the early years of the twentieth century reveal that titles with literary and/or theatrical connections were far from dominant in terms of industry output and market exposure,” as the film industry recognized the marketability of narrative film, “it reached gratefully for the library shelf and from about c. 1907 onwards, adapting the work of existing authors became one in a range of standard story-telling production practices for most of the leading film companies” (2012, 20; see also Cartmell 2012, 2; Naremore 2000, 4–5). Denis Gifford’s 1991 Books and Plays in Films 1896–1915: Literary, Theatrical and Artistic Sources of the First Twenty Years of Motion Pictures highlights 861 authors “whose work was adapted to film in the first twenty years of the industry” (Buchanan 2012, 21). The first two decades of cinema saw the release of numerous adaptations, among them Uncle Tom’s Cabin (Edison, 1903), Robinson Crusoe (Vitagraph, 1903), The Adventures of Sherlock Holmes; or, Held for Ransom (Vitagraph, 1905), A Christmas Carol (Essanay, 1908; and Edison, 1910), Frankenstein (Edison, 1910), Vanity Fair (Vitagraph, 1911), As You Like It (Vitagraph, 1912), Hard Times (Trans-Atlantic, 1915), Nosferatu (Prana, 1922), and many, many more. Of course, these films were not full-length adaptations. As

816

K. Newell

Buchanan points out, prior to 1907, “the project was, in effect, to produce cinematically animated, brief, visual quotations from a work” (2012, 21). Later, “when the desire to tell a coherent story had become the usual aspiration for a film, film stories were still typically structured as a medley of strung-together ‘moments’ rather than [. . .] a fluently progressive narrative” (Buchanan 2012, 21). Part of the lecturer’s role, then, was to provide context for these “visual quotations” and a narrative through-thread for these medleys of moments. Simon Trezise writes that, “[i]n small theatres of the nickelodeon period, the lecturer was often the already overworked projectionist, who would have to interpret a new film about a subject he probably knew little about,” and cites adaptations of The Scarlet Letter (1911), Macbeth (1911), and Othello 1909) as examples (2019, 8). Some lecturers would “consult the text of the original literary or theatrical source” and bring that content into the lecture through performance or reading (Altman 2004, 141). After 1907, film production companies issued descriptions of their films to provide lecturers with content to read, yet Altman clarifies that “these were always short and schematic, and often insufficient to guide those charged with bringing meaning to the projected image” (2004, 140). Some lecturers’ contributions became a defining feature of the program, as is evident in the example of “nickelogues” – lectures developed by William G. Morton and delivered at the Nickel Theatre in Providence, Rhode Island, in 1908 (2004, 141). The Providence Tribune, reporting on the first of these nickelogues, delivered for Uncle Tom’s Cabin, comments that Morton “every now and then drew in a bit of dialogue from the play, which he gave in dialect and with much dramatic expression” (quoted in Altman 2004, 141). Similarly, for his lecture on The Scarlet Letter, Morton “explained in a concise and very interesting manner, with interpolations from the successful dramatic version used by the late Richard Mansfield” (quoted in Altman 2004, 141). In engaging audiences actively in the understanding of the literary or theatrical text, “lecturers helped to gentrify moving picture exhibition” (Altman 2004, 142). Looking forward, we can see how aspects of the lecturer’s role morphed into that of the commentator. The popular radio program NBC University Theater: The World’s Greatest Novels, which broadcast from 1945 to 1951, provides a good example. Produced by Margaret Cuthbert, the program aired adaptations of classic novels, such as Charles Dickens’s A Tale of Two Cities, Jane Austen’s Emma, and Herman Melville’s Moby Dick. Guest lecturers hailing from various corners of the literary field – from editors to writers to professors and deans – presented to lectures on the novels “[d]uring the intermission halfway through each episode” (DeForest 2008, 212–13). Tim DeForest clarifies that “[t]hese mini-lectures weren’t dry or overly academic, though, but rather another interesting part of an entertaining show” (2008, 212–213). NBC had brokered arrangements with various universities so that interested audience members could earn college credit through a correspondence course. The tradition of the commentator continues today with literary adaptations broadcast on networks and platforms such as Turner Classic Movies and PBS’s Masterpiece. A key difference, though, between the lecturer and the contemporary commentator, would be that whereas the former functioned as an adapter and actively crafted the “cohesive fictional world” of the program, the latter affirms the

32

Adaptation and Sound

817

cultural value of specific adaptations and, more to the point, the value of specific authors and literary texts. Given the priority adaptation studies have placed on dialogue and other modes of speech in comparative analyses (e.g., how a character’s speech delivered in a novel compares with the corollary speech in an adaptation), this essay’s overview of juxtaposition, accompaniment, synchronization, and adaptation would be remiss without acknowledging the role of speech and words as a form of sound technology. In the early period of cinema, the relationship between image, word, and sound could be real or could be conceptual and perceptual. In terms of the former, actors’ voices, “considered effectively as a class of sound effects,” according to Bottomore, could be used to establish a relationship with the projected image, similar to that of the illustrated song (1999, 488; Altman 1996, 697). As with other types of sound effects, fidelity and verisimilitude were key measures of success. Film history is rife with various examples of exhibition strategies that utilize voice effects, including, as Bottomore notes, “the use of voices, providing dialogue to imitate actors in the scene” (1999, 488). Emerging in 1907–1908 and continuing through about 1910, “talker films,” Jeffrey Klenotic explains, were “a non-mechanical attempt to synchronize voice and image” and “relied on the efforts of live performers rendering character dialogue (and sometimes sound effects) from a station behind the projection drop” (2001, 157; Musser 1990, 439). Isabelle Raynauld has written on more conceptual and perceptual relationships between image, word, and sound, arguing that films produced during the silent period, roughly 1895–1929, “not only represented sound, the act of hearing and of listening in many inventive ways, but also showed silence, as well as noisy and talky situations” (2001, 69). Raynauld’s extensive research into silent screenplays “confirmed that the dialogues in silent screenplays were not just calligraphic entertainment for the reader’s eye but were indeed lines intended to be performed by actors” (70). She writes that “[i]instead of doing away with sound entirely, the early cinema writers and filmmakers found a chorus of strategies to make sound be heard inside the story and be seen on the screen” (Raynauld 70). To establish verisimilitude and allow “spectators to ‘see voices,’ the actors were thus required to move their lips: in short, they had to speak. The spectators were then free to imagine phantasmagorical voices and make the characters say what they felt was appropriate in the context of the scene” (Raynauld 72–73; Simonson 2019, 16)2. Examples of voice actors and other forms of speech notwithstanding, during and prior to 1926, Donald Crafton notes, “‘sound film’ usually meant music, not speech” (1997, 88). Deborah Cartmell’s 2015 Adaptations in the Sound Era: 1927–37 focuses specifically on the late 1920s’ emphasis on speech and on how the strategies deployed in advertisements and pressbooks for early sound adaptation laid the groundwork for expectations for adaptations that persist today. For Cartmell, “the

2

Title cards, used to convey dialogue or exposition, to establish settings, and to mark transitions, offer another technology of perceptual sound and relied on the audience to internally produce the sound.

818

K. Newell

introduction of sound, or spoken words, transformed how an adaptation, in particular, a film adaptation of a well-known literary work, was defined” and led to “the unprofitable approach of comparing the words of a book to those spoken in a film,” which, in turn, set the stage for fidelity-based comparative modes of reading and responding to adaptations (2015, 1). She argues that “the inclusion of speech in screen versions of literary and theatrical works, undeniably revised what it was to be an adaptation: words” (2012, 70). Put another way, “the spoken words, that is how many words were retained and how they were pronounced, were perceived of as increasingly important in respect of the value of the adapted text” (Cartmell 2015, 3). Sound-forward film adaptations were advertised with the promise that they were “more ‘faithful’ to their literary narratives than ever before in that the author’s words were no longer abbreviated and reproduced to be read, but to be spoken” (Cartmell 2015, 9). While filmmakers have always drawn from recognizable sources, as sound options increased, so did the options for texts to be adapted and the adaptive relationships that could be developed. Timothy Corrigan, who likewise notes “the massive impact of the coming of sound on literary adaptations” (1999, 27), drawing from Marie-Claire Ropars-Wuilleumier’s work, explains that “new sound technologies were especially significant in the relationship between film and literature in four far-reaching ways, ways that extended and complicated tendencies already dominating classical cinema” (34–35). These included the “reinforce[ment] and expan[sion] of the possibilities for realism,” as well as “the rapid development of theatrical dialogue within the cinema” and “expanded narrative or novelistic form as the main structural principle for the movies” (17). By the 1930s, the film industry had committed fully to sound and adaptions of literary works, and, for Corrigan, “the spate of adaptations that appeared in the thirties,” which included Frankenstein (1931), Dr. Jekyll and Mr. Hyde (1932), Wuthering Heights (1939), The Wizard of Oz (1939), and Gone With the Wind (1939), “is one indication of the importance of sound in drawing film and literature close together” (35). Corrigan comments on the tendency for filmmakers to adapt works with which audiences were already familiar: “literature provided an abundance of ready-made materials that could be transposed to film” (17). William K. Everson likewise highlights the tendency for Hollywood to draw from familiar sources and to recycle dialogue, much in the way that the industry recycled music. Everson writes that, “In many cases, early sound films were constructed like silents, and dialogue was merely added. Alternately—and more often—Hollywood fell back on novels, plays— and unproduced plays—where the writing and the dialogue were already a fait accompli. Producers owning properties based on novels or plays that had already been made as silents had somewhat of an edge, using the earlier films as models to which pre-written dialogue could be grafted” (1998, 343). As we have seen, the integration of sound technologies in cinema at the turn of the twentieth century was highly adaptive and adapting and impacted contemporary adaptation processes. Adaptive relations and the core questions they generate – e.g., how to represent a work composed in one medium in another, in what ways do juxtaposed media and modalities convey meaning and guide audiences – continue to guide adaptation practices and adaptation studies into the twenty-first century. The

32

Adaptation and Sound

819

next section turns to more integrated forms of adaptation, and I focus primarily on radio adaptations and audiobooks. Whereas the conversation on modes of adaptation that utilize juxtaposition tends to highlight meaning-making, particularly as impacted through synchronicity and asynchronicity, with integrated forms of adaptation, the conversation turns toward medium specificity (e.g., what radio offers that print or film does not and vice versa) and fidelity (i.e., whether the audio iteration is “true” to the print source text), and related topics of abridgement, market concerns, and the efficacy of a work’s message in one medium over another.

Literature on the Air: Adaptation and Radio Radio adaptation entails remediating a visual, audiovisual, or print-based work into an audio work. This process and its effect can be easily underestimated or assumed to be reductive, as when a radio adaptation of an audiovisual work is assumed to be “less than” the “full” sensory experience (Cazeaux 2005, 157). To this, Richard Hand reminds us that “radio is not visionless; rather, it facilitates a particularly rich way of seeing” (2017, 341; also Crook 1999, 7; Cazeaux 2005; Miller 2018; Hand and Traynor 2011, 36). Bonnie Miller offers examples of guides for radio writing in which “the ear had been conceptualized as a mode of seeing” (2018, 323). Referred to as “the theatre of the mind,” radio writers, producers, and performers draw upon “collected ability, styles, tools, and techniques” to “conjure vivid imagery in their audiences’ minds through sound alone” (“Origin” 2017; also Verma 2012, 2; Miller 2018; Huwiler 2010, 135; Crook 1999, 66; Hutcheon 2013, 42). In conceptualizing a literary work as an audio experience, the “conjuring” that had previously taken place through the reader’s mental process gives way to that evoked by the talents and stimulations of the radio program. Radio adaptation is a broad and varied field that includes scripts adapted specifically for radio performance, ekphrastic radio broadcasts of silent films, dramatizations of popular films and novels, and myriad other formats. Within this diversity, sound functions as both adapted and an adapter. That is, sound is adapted when the audiotext is the source for adaptation, as in the case of novelizations of films and visual and audiovisual adaptations of radio plays. Sound is adapting when an existent work in a print-based or non-audio modality is adapted into sound, as is the case for print-based books adapted to audiobooks or to audiovisual modalities. Discussions of radio adaptations and audiobooks tend to take a medium-specific approach and contextualize each on the context of another medium. Perhaps one of the most intriguing aspects of radio adaptation is that, in reception, the experience of the adaptation depends upon the listeners who form a mental “adaptation” of the work in relation to the radio “source” text. Several writers have commented on the particular opportunities and challenges of “writing for the ear, not the eye” (Echols 2017, 5; Hutcheon 2013, 40–41). DeForest, for example, positions radio adaptation as the ultimate opportunity for listeners to “journey to Treasure Island; help solve a murder; face down a gunman on the streets of Dodge City; fight a dinosaur while armed with nothing but a hunting knife; visit alien worlds and lost

820

K. Newell

cities—all without even having to open our eyes” (2008, 228). Further, the circumstance of relying on listening rather than seeing has the benefit of shifting the manner in which audiences might respond to a work or characters. As Katherine Barnes Echols explains, radio “encourage[s] listeners to use their imaginations and to become participants in the story whatever their age, gender, race, or economic distinction” (2017, 1). Laurence Raw considers how this circumstance impacts audience responses to the Creature in adaptations of Frankenstein. Rather than conceptualize him as the antagonist immediately based on visual cues, listeners “are invited to reconsider their prejudices and view him in the same fashion as De Lacey, as an object of sympathy rather than prejudice” (2018, 63). Writing on radio adaptations tends to conceptualize opportunities and challenges as extensions of the medium, as Saturday Review of Literature exemplifies: “The best tales are told tales, designed for immediate effect, where the voice readily does what written style accomplishes with such difficulty” (quoted in Echols 2017, 52). As in the case of the film lecturer, radio writing and narrators can contribute both positively and negatively to the fortification of the “theater of the mind.” Echols comments on the joys and pitfalls of the radio narrators, who “urged [their] listeners to ‘listen’ and to ‘see’ and to imagine [they] too [were] part of the action. Yet a narrator’s interruption could also destroy the illusion of the story” (52). Echols, quoting from a 1939 scriptwriting guide for radio, relates that scriptwriters were “cautioned to remember that a listener’s ‘aural memory [was] not as strong as his visual memory, [that] important material and significant events—items which must be’ remembered and recalled—‘must be firmly fixed so they will’ have the desired effect ‘when their time comes’” (31). For Hand, “radio writing is at its best when it does not obsess about visuals and strikes a balance between exposition and suggestion”; “[l]ess successful examples of radio drama,” he notes, “are often marked by a tendency to over-narrate” (2017, 349). Fortunately, the architecture of the “theater of the mind” is not fortified solely by the strength of the audiences’ imagination nor limited to impressions formed solely by the radio text, but includes myriad other sources including previous experiences with content, as in the case of classic works adapted, previous experiences with voice actors’ work, previous experiences with host programs, and so on – what Miller describes as “associational listening” (2018, 323). Like many forms of adaptation, radio adaptations are commonly discussed in term of their treatment of their source texts, as well as their consideration of market and audience expectations and abridgements necessary to conform to programming standards. Radio adaptions of literary works have long been popular in some form (Rubery 2011, 6). Richard J. Hand and Mary Traynor point out that “[s]ome of the earliest examples of radio drama in the 1920s are essentially broadcasts of stage productions” (2011, 14). In Britain, Matthew Rubery relates, “[e]xcerpts from Shakespearean drama were among the first to be broadcast over the wireless,” followed closely by abridged adaptations of novels, particularly works by Charles Dickens (2011, 6). Lux Radio Theater (1934–55) “produced radio adaptations of classic films, frequently featuring the original movie cast” (Hand and Traynor 2011, 19). The Witch’s Tale (1931–1938), Hand and Traynor write, “had a special interest in adaptation” and

32

Adaptation and Sound

821

adapted a number of classic gothic novels, among them Frankenstein and Dr. Jekyll and Mr. Hyde (107). Orson Welles’s well-known 22-episode radio program, Mercury Theater on the Air (1938), adapted numerous works of classic literature, including 9 created under the subprogram title of “First Person Singular,” among them Bram Stoker’s Dracula, Robert Louis Stevenson’s Treasure Island, and Charles Dickens’s A Tale of Two Cities (also Rubery 2011; Rippy 2003; Hand and Traynor 2011, 22). Several writers discuss radio adaptations of literary works, often focusing on the challenge of adapting a story from literature to radio. DeForest writes, “you can’t just take, say, Treasure Island, assign the dialogue as is to the actors, and have them read it into a microphone. Radio is different from prose—what works dramatically on the page doesn’t necessarily work the same way on the airwaves. Besides, it all had to fit to the second into the proper times slot. Stories had to be abridged—scenes had to be shifted around—dialogue had to be altered or condensed” (2008, 4). Laurence Raw likewise addresses the ways in which radio adaptations were guided more by broadcast time allotments (2018, 67), noting that market interests and “fisca[l] motivat[ions]” outweighed “artistic integrity” (64). Regardless of how a writer might want to bring a literary text to radio, certain factors impact the adaptation process, among them market interests, network norms, and time constraints. Like any form of adaptation, radio adaptations of literary works tend to be evaluated and assessed in terms of their treatment of their source texts. DeForest’s Radio by the Book (2008), for example, focuses primarily on differences between literary sources and radio adaptations, particularly at the character and plot level, with priority placed on the “original.” For DeForest, “[s]traying from the original plots and themes when adapting the classics is dangerous; the classics are classics, after all, because they got everything right the first time” (219). DeForest compliments various programs for their fidelity to their source text. For example, he praises the radio adaptation of “Hold ‘Em Yale” for The Damon Runyon Theater for “not just how much of the original prose it retains, but also how effectively dialogue written for the show compliments that prose” (197). Similarly, he applauds the program writers of NBC Presents: Short Story (1951): “much of the original prose is retained within the narration. The NBC writers were themselves enormously talented men and women, quite able to intelligently insert new dialogue or narration when necessary. But they also knew when to leave well enough alone and let the original author speak for him or herself” (223). Some radio adaptations are more tangentially related to their source texts and use literature “as a jumping off point for something largely original,” as in the case of RCA’s The Weird Circle’s (1943–1953) adaptations of Poe’s “The Fall of the House of Usher” and “The Cask of Amontillado” (DeForest 2008, 218–219). In addition to changes in characterization, the adaptations alter the narrative point of view, add supporting characters, and generally change the story – all of which, DeForest laments, leaves “little opportunity to keep any of Poe’s wonderful prose” (219). The time constraints of radio programming necessitate additional considerations for adaptation. As the run time of most radio programs is 30 or 60 minutes, writers need to adapt literary sources in a manner that condenses plot and character and involves various reframings of content. Commonly, radio programs “utilized

822

K. Newell

familiar story patterns” and adapted works and stories familiar to their audiences and reduced the program to those moments and elements most familiar to the audience, to both positive and negative effect (Miller 2018, 327). In reference to radio adaptations of Robin Hood stories, Echols explains that “[r]adio’s format restrictions [. . .] require[d] scriptwriters to make significant changes to source material [. . .] such as severely compressing for brevity the scope of the tales [. . .] or eliminating material thought inappropriate for a mixed audience” (2017, 37–38). Echols suggests that “the adventures of King Arthur, his Knights of the Round Table, and Robin Hood and his Merry Men all probably made it to radio because they were familiar to listeners of all ages and were easily adaptable to a restrictive format” (35). The resulting adaptations “were based on only the most familiar scenes in the legends and highlighted personal and social morality as important character traits” (3). In compressing the tales, thus “[s]criptwriters were likely working from a broad range of specific sources and from their own familiarity with the legends, which permitted some degree of flexibility” (37–38). Abridgements can result in a cohesive and engaging radio program or in significant plot challenges. The KFI radio program (1946–1949), written by Jerome Lawrence and Robert Lee and produced by Frederic W. Ziv, “consisted of thirtyminute [. . .] adaptations of prestigious classic literature, such as Tom Sawyer, Pride and Prejudice, Great Expectations, and 20,000 Leagues Under the Sea” (Becker 2001, 7). DeForest praises the KFI radio program Favorite Story (1946–1949), which distinguished itself by taking source suggestions from actors and other celebrities, for consistently falling into the first camp: “[t]he stories chosen were nearly always established classics, covering a fairly wide variety of themes and genres. [. . .] the adaptations were skillfully done and nearly always entertaining” (2008, 199). Favorite Story “managed the incredible feat of jamming a classic novel into half an hour and still giving us a rewarding experience” (DeForest 200). Much like the adaptations of Robin Hood addressed by Echols, the classics adapted to the half-hour format of Favorite Story could provide only “the basic story” (200). DeForest offers the example of Oliver Twist, for which “Fagin and the Artful Dodger are largely stripped of their unique personalities and become one-dimensional villains” (200). On Dracula, “To fit the story into the one hour time slot, several of the major characters were dropped. [. . .] The plot was also streamlined, but the simple idea of shrinking the number of protagonists meant that many of the events of the actual story could be saved” (DeForest 205). In some cases, however, the abridgement confused the plot line. In discussing the 1932 adapted-to-radio Tarzan, DeForest writes, “It’s actually understandable that writers for a medium that depends heavily on dialogue wanted to quickly get to the part of the story involving verbal interactions between characters. The trouble with the rushed approach is that it sometimes led to awkward plot holes” (156). Some radio programs prioritized fidelity to the source texts. Welles’s seven-part adaptation of Les Misérables for his program Mercury Theater on The Air is one such example. DeForest writes that “Welles was determined to do a faithful adaptation of the book, lifting passages verbatim to use as narration for the show. Dialogue

32

Adaptation and Sound

823

was also taken directly from the book. Throughout the adaptation, Welles used montages of dialogue to help condense the story without losing track of the plot. [. . .] thirty pages of descriptive prose in the novel could be effectively communicated to the radio audience by replacing it with thirty seconds of dialogue” (204). Another anthology-style program, NBC’s The World’s Greatest Novels, offered half-hour episodes (much like other programs), with longer novels dramatized over several episodes. For DeForest, “[t]his was what was needed—time to do proper justice to the plot and the character; time to leave in the small moments and details that given depth to the story” (211). As with any adaptation, market needs and expectations can play a larger role in adaptation decisions than other considerations. Several writers comment on national and cultural restrictions on programming. Rubery, for example, notes that, unlike “their government-funded British counterparts,” American radio producers faced “commercial restraints” that limited their options for literary adaptations (2011, 7). DeForest also comments on the ways in which commercial restrictions and “broadcast standards of the day” impacted radio scripts (197). For example, in The Damon Runyon Theater’s “Butch Minds the Baby,” “[i]n the original, Butch had retired from safe-cracking and now bootlegged beer. In the radio play, he had gone completely straight and now worked as a plumber” (197). Writers also address the market influence on recycling and refashioning characters to meet expectations for new contexts. Echols, for example, considers Robin Hood’s ability to be removed from an exclusively British context and reimagined as an American ideal. Radio adaptations of Robin Hood, Echols relates, merge the two ideals in that they “illustrate the medieval British ideal of chivalric knighthood and embrace this ideal as a standard of American morality and masculinity” (2017, 2). Echols adds that, “[s]criptwriters couched modern anxieties in a medieval context and reinvented the knight errant as an American cowboy, a superhero, or a soldier” (3). Commonly conceptualized as a mode of entertainment belonging to the past, radio adaptations of literary works continue to be produced and to thrive. The Rockwellian image of the nuclear family gathered around a radio cabinet may be relegated to nostalgia and replaced by images of listeners tuning in to broadcasts and podcasts through smart devices and Apple AirPods (Hand and Traynor 2011, 69–100). Hand and Traynor explain that “[a]n advantage of the internet is that it can provide new depth to the listening experience by presenting supplementary information” (2011, 70). Such information might include interviews with persons relevant to the production of the program, chats and fan forums within the online platform, interactive maps of relevant sites, links to related texts (e.g., films, stage performances), and social media engagement, among others. In an adaptation context, adjuncts to the radio program highlight intertexts and expand the adaptation network in new directions (Newell 2017). This expansion, Bonnie Miller might argue, has always been part of the radio experience, noting that “[o]ther media forms, like novels, magazines, film, comic strips and later television intervened in radio listening and impacted the process of visualization” (2018, 330).

824

K. Newell

Audiobook: “A Movie in Your Mind” Unlike radio adaptations, which, as we have seen, typically reimagine a literary source through strategies of abridgement and in a manner that aligns with network and market needs, ostensibly audiobooks offer listeners a less-interpreted, more faithful version of their source. “By definition,” Iben Have and Birgitte Stougaard Pedersen specify, “an audiobook is a recording of a printed, published book” (2021, 214), which means that, even when abridged, audiobooks purport to offer readings of the source material “as written.” According to Have and Pedersen, “[h]istorically, the audiobook has been described as a kind of by-product of the printed book and as a service for readers who for various reasons have difficulty reading printed books,” but “with the advent of digital media,” “the explosion in usage of audiobooks has caused a detachment from the printed original, so the audiobook is recognised [sic] as a medium in its own right” (2021, 198). Still, audiobooks are generally considered faithful annexes to their nominal source texts. Curiously, while recognized as the audio version of a printed book, audio books are not commonly discussed as adaptations of literary works, save in cases in which the same book has been adapted to audio multiple times with different narrators. Writers commonly situate audiobooks within the context of other modalities, such as radio or film, and highlight the manner in which features such as the voice of the narrator and other sound cues impact listeners’ experience of the story. The tendency to contextualize audiobooks in terms of other media and forms of adaptation is evident in audiobook publishing imprint GraphicAudio’s tagline “A Movie in Your Mind” (an echo of radio’s description as “the theater of the mind”)3. Unlike literature-to-radio adaptations for which the medial comparison is between the print-based source and the audio-based adaptation, here the comparison points to a medial experience beyond that of the adaptation scenario. Sarah Kozloff, writing in 1994 before the advent of Audible and other streaming services, avers that film is one of the audiobook’s strongest antecedents, citing both the significant number of audiobook tie-ins and parallels between the recording and editing processes for audiobooks and films (1995, 86–87). “[N]otwithstanding their literary sources,” writes Kozloff, “audiobooks can only be fully understood in terms of their interrelationships with contemporary visual culture” (83). Kozloff argues that “[f]ilms have accustomed us to the use of novels as source material” and audiences are “not surprised or shocked by radical adaptations or abridgements” (86). While Kozloff’s statement ignores the rich history of adaptation that predates cinema, the general sentiment – that myriad and intersecting modes of adaptation ready audiences for additional modes of adaptation – stands up in scholarship in the field (Hutcheon 2013, 173; Leitch 2003, 2012; Stam 2005a, 31; Newell 2017). The link between

3 Specializing in fantasy and science fiction titles, GraphicAudio utilizes high-quality performances and cinematic music and sound effects in adapting novels to audio and, in this way, creates adaptations much more like radio dramatizations than more traditional audiobooks that feature one or two consistent narrators.

32

Adaptation and Sound

825

audiobooks and film adaptation likewise is evident in the title of Insider.com article: “7 audiobooks you can listen to while you’re waiting for their film adaptations to hit the big screen” (Redkar 2020). Have and Pedersen take a different approach in contextualizing the audiobook, placing its antecedents in the audio realm. “Technologically and materially,” they write, the audiobook “shares its technology and formats with music. [. . .] [t]he technological histories of the audiobook and of recorded music run parallel” (2021, 200). Drawing from the work of Jay David Bolter and Richard Grusin, Have and Pedersen “understand the audiobook as a remediation of the book [. . .], which underlines that the auditive mediation of literature adds substantial new aspects to the work” (1999, 201). Further, “Seen historically, the audiobook is not just a remediation of the printed book but also refers back to the oral tradition of oral tales and the reading aloud of novels, long before literature became an institutional concept” (Have and Pedersen 2021, 202). While the audiobook is understood to be a remediation of a printed book, the tendency to categorize it as a medial event rather than as an adaptation – or, ideally, as both – recalls Elleström’s observation that we tend to study multimodal and multimedial relationships in terms of one characteristic. In the case of audiobooks, however, priority appears to be given to an auxiliary sensorial trait, yet one not present in the immediate adaptation scenario. As is the case in writing on radio adaptations, scholars address topics such as visualization, cognition, and mediation, but, in the context of audiobooks, such topics are tied to the concept of reading. Put succinctly, Have and Pedersen ask, “to what extent can we say that we ‘read’ an audiobook” (2021, 199)? This question alludes, in part, to a history of cultural pushback and criticism of audiobooks for being pale substitutes for “real” literature and “real” reading (Rubery 2011; Have and Pedersen 2016; Kozoff 1995). As Rubery outlines, the charges brought against audiobooks are consistent with those waged against many forms of mass media – that “listening to audiobooks is a passive activity,” that “audiobooks do not require the same level of concentration as printed books,” that “audiobooks distort original narrative through abridgement,” that “the pace of the audiobook is removed from the reader’s control,” that “the audiobook speaker interferes with the reader’s reception of the text,” and so on (2011, 10–15). However, Have and Pedersen’s question also gets at questions of intermediality and cognition; that is, how can we conceptualize an activity such as reading, which is so seemingly tied to lexical and semiotic traits, in contexts that prioritize other traits? As Have and Pedersen explain, “In order to discuss the differences between listening and reading, it is however necessary to speak of the activity of listening as something other than reading understood as a visual decoding of writing” (2021, 202). Further, in their view, the audiobook is “not only a remediation but also an independent medium, which offers other and expanded forms of reading literature” (Have and Pedersen 2021, 214). As in the case of radio adaptations, the tone and cadence of the reader’s voice and pacing, listeners’ familiarity with the reader, and additional layers of production quality can all shape and inform audiobook listeners’ mental conceptualization of the source. Additional voices add additional layers of mediation. Kozloff notes that, in

826

K. Newell

both audiobooks and radio adaptations, “performers and narrators are removed from the listener and mediated by technology” (1995, 83). Furthering their discussion of differences between reading a book and listening to an audiobook, Have and Pedersen explain that, for audiobooks, “The voice is the technical medium of display [. . .] which most clearly points to the difference between the paper book and the audiobook, and an analysis of the audiobook ought therefore to include the role of the voice. The voice delivers an interpretation of the text and in doing so becomes a new medium for literature” (2021, 210). Similarly, Rubery explains that, “[w]hereas printed texts possess a voice in a figurative sense, as a metaphor for the illusion of authorial presence available through the printed word, the situation is very different in the case of literature made audible by an actual speaking voice. The spoken delivery of the audiobook is a departure from the familiar conception of the narrator as an imagined voice in the reader’s mind” (13). Rubery adds that “Voicing an otherwise silent narrative introduces a number of potential interpretive issues. [. . .]. For instance, the narrator’s voice may or may not approximate the imagined sound of the narrative in the reader’s mind. [. . .]. The listener may become too aware of the narrator’s voice if it does not achieve a sufficient level of neutrality, instead becoming an unwelcome third party intruding between author and reader” (14). Whereas strategies of juxtaposition and integration grant visibility to the adaptative relationship (e.g., audiences of the illustrated song see the play between the film image and the singer’s vocals), the adaptive process for radio programs, audiobooks, and other forms of audio adaptation involves a comprehensive reimaging of the source. The former place the source and adaptation “in relation”; the latter, by contrast, substitute the adaptation for the source text. As scholarship on audio adaptation has expressed, the process of adapting a print-based, visual, or audiovisual work into an audio work involves considerations of audience, market, and fidelity that are similar to those of other forms of adaptation but also involves consideration of audiences’ mental processes and application to imagination. Such distinctions in how forms of adaptation align and differ invite reflection on processes commonly associated with a specific type of adaptation and what they say about the process of adaptation more broadly.

Conclusion: Sound, Adaptation, and Intermediality Mediation and medial borders are concepts integral to writing on the adaptive relationships that emerge from sound technologies (Abel and Altman 2001, xii). Discussions of early sound technologies in film exhibitions, such as illustrated songs, sound effects, lecturers, and musical accompaniments, conceptualize those technologies as forging an intermedial bridge between two distinct experiences. Likewise, discussions related to integrated film soundtracks, radio programs, and audiobooks conceptualize the adaptive relationship as a negotiation between two distinct media. As Irina O. Rajewsky describes, “intermediality may serve foremost as a generic term for all those phenomena that [. . .] in some way take place between media. ‘Intermedial’ therefore designates those configurations which have

32

Adaptation and Sound

827

to do with a crossing of borders between media” (2005, 46). The designation “between media” and the metaphor of “borders” need not signify that the adaptive relationship is limited to two fixed points. Rather, through concepts of mediation and intermediality, writers are able to demonstrate the ways in which an illumination of sound technologies heightens the plurality of the available adaptive relationships. For example, as Lars Elleström has outlined, “[a] media product is a single physical entity or phenomenon that enables interhuman communication,” and such products “can be analysed in terms of four types of traits [or modalities]: material, spatiotemporal, sensorial and semiotic traits” (2021, 8). Much of the writing on the adaptive relationships fostered by sound technologies considers those relationships in terms of three or four of these modalities, even as they may prioritize only two. Several writers conceptualize the role of the film lecturer, for example, in terms of mediation between the physical performer and the exhibition, though, of course, this performer can be understood as an entity that enables communication. Germain Lacasse construes the lecturer as “the intermediary for two kinds of encounters: that between the audience in general and a revolutionary technology, and that between specific audiences and the film” (2012, 487). Addressing the latter kind of encounter, Judith Buchanan describes the degree to which the lecturer shapes audiences’ responses to and interpretation of the work adapted to the screen as mediating: “In those cases where the triangulated relationship between players on the screen, lecturer, and picture-goers worked well, the lecturer’s mediating presence could provide a more intimate conduit into the subject viewed [. . .] and make the drama live in ways sensitively attuned to the artistry of the production, the twists and turns of the developing story and the interpretive needs of the particular assembled audience” (2012, 28). By contrast, Buchanan adds, if “the lecturer was less proficient, his mediating presence could sometimes prove an active impediment to an appreciation of the production, an obtrusive obstacle to an audience’s enjoyment of the pictures or simply a diversionary side-show” (2012, 28–29). Richard Crangle sums up the lecturer’s role within the context of the exhibition program, noting that, what the lecturer contributed to was a “hybrid text”: “the verbal element could not logically function without the visual element, and the visual element was perceived to be not fully delivered without some form of additional explication” (2001, 45). Mary Simonson describes Samuel Lionel “Roxy” Rothafel’s ekphrastic radio broadcasts of the Capitol Theater film programming: “As the newsreel and other short films appeared on the Capitol’s screen following the overture, the radio audience heard Roxy’s careful descriptions of each film’s narrative and visual imagery over the accompanying music. [. . .] when necessary, [Roxy] described precisely what the theater’s audience was seeing, offering audiences mental images to compliment the musical performances and other acts that they heard through the ether” (2021, 13). Jean Châteauvert and André Gaudreault comment on the lecturer’s role in engaging the audience in co-producing sound elements for early cinema exhibition: “as a figure of interlocution, [the lecturer] called for and encouraged an (inter-)active participation on the spectators’ part, a participation that could then translate into various

828

K. Newell

forms of sound expressions just when needed” (2001, 188). Crucially, the intermedial lens allows for the role of the lecturer to be understood in all of these contexts simultaneously. Writers likewise conceptualize film music in terms of mediation and medial borders. For example, Claudia Gorbman writes that “[m]usic in film mediates. Its nonverbal and nondenotative status allows it to cross all varieties of ‘borders’: between levels of narration (diegetic/non-diegetic), between narrating agencies (objective/subjective narrators), between viewing time and psychological time, between points in diegetic space and time (as narrative transition)” (1987, 30). Again, the “betweens” Gorbman highlights are not limited to two points, but encompass the full range of possible intersections. Tim Anderson might attribute such border fluidity to “the excessive semiotic possibilities of music” (1997, 16). For Anderson, the “polyvalency” of music “carries with it the potential to create numerous meanings that are neither firm nor arbitrary but rather contingent on multiple performative contextualities” (16). Kay Dickinson likewise touches on this range in her discussion of musical soundtracks for film. Dickinson notes that, in instances of particular empathy between film and music, “these media appear almost on the brink of defying their formal boundaries—as ‘film,’ as ‘music’—so perfectly are they in tune with each other’s registers. At other times, these raw, previously unrelated elements seem to conjure an alchemical transformation, and a fresh approach to understanding materializes, one that could never have been imagined beforehand” (2008, 13). These engagements – the border- and boundary-defying and the alchemical and transformative – highlight the plurality of the adaptive relationships illuminated in considerations of sound technologies. This chapter has sought to provide an overview of existing scholarship on soundbased adaptation strategies not typically characterized as adaptation, but is by no means exhaustive. Certain of the topics highlighted in this contribution have been explored by scholars in the fields of adaptation studies, film studies, sound studies, and intermediality through the lenses of exhibition, technology, and mediality. Rarely are the strategies of juxtaposition and integration contextualized in terms of adaptation within these fields, nor in the field of adaptation studies, save by a handful of adaptation scholars. As Abel and Altman (2001), Crangle (2001), and others have expressed, areas such as the illustrated song, the lantern lecture, and early cinema exhibition programs warrant further study to better understand their inherent intermediality, particularly as relates to the dynamics of sound as adapted and adapting. Similarly, while scholarship on radio programming is rich and diverse, scholarship devoted specifically to examining such programs through a lens of adaptation is more limited, despite the modal richness of such adaptive relationships and intermedial networks. As a last suggestion, discussions of audiobooks likewise prioritize material, temporal, and sensorial questions; asking such questions through co-lenses of adapting and adapted sound may allow for a deeper understanding of the semiotic as well. The inexhaustible ubiquity of adaptations across media and modalities will continue to provide new avenues for exploring sound in relation to other medialities.

32

Adaptation and Sound

829

References Abel, Richard, and Rick Altman, eds. 2001. The Sounds of Early Cinema. Bloomington: Indiana University Press. Abel, Richard. 2001. That Most American of Attractions, the Illustrated Song. In The Sounds of Early Cinema, ed. Richard Abel and Rick Altman, 143–155. Bloomington: Indiana University Press. Adaptation, n. 2022. OED Online. Oxford University Press. www.oed.com/view/Entry/2115. Accessed November 5, 2022. Altman, Rick. 2004. Silent Film Sound. New York: Columbia University Press. ———. 1996. The Silence of the Silents. The Musical Quarterly 80.4 (Winter): 648–718. ———, ed. 1992. Sound Theory/Sound Practice. New York: Routledge. Anderson, Tim. 1997. Reforming ‘Jackass Music’: The Problematic Aesthetics of Early American Film Music Accompaniment. Cinema Journal 37.1 (Autumn): 3–22. Andrew, Dudley. 1984. Concepts in Film Theory. Oxford: Oxford University Press. Becker, Christine. 2001. A Syndicated Show in a Network World: Frederic Ziv’s Favorite Story. Journal of Radio Studies (Summer): 160–174. https://www3.nd.edu/~cbecker1/zivradio.pdf. Beja, Morris. 1979. Film & Literature: An Introduction. New York: Longman. Bluestone, George. 1957. Novels Into Film. Berkeley: University of California Press. Bolter, Jay David, and Richard Grusin. 1999. Remediation: Understanding New Media. Cambridge: MIT Press. Bottomore, Stephen. 2001. The Story of Percy Peashaker: Debates about Sound Effects in the Early Cinema. In The Sounds of Early Cinema, ed. Richard Abel and Rick Altman, 129–142. Bloomington: Indiana University Press. ———. 1999. An International Survey of Sound Effects in Early Cinema. Film History 11 (4): 485–498. Brooks, Nicholas. Harry Potter Contains a Surprising Connection to Star Wars. CBR 08 September 2021. https://www.cbr.com/harry-potter-star-wars-same-music/. Bruhn, Jørgen, Anne Gjelsvik, and Eirik Frisvold Hanssen, eds. 2013. Adaptation Studies: New Challenges, New Directions. London: Bloomsbury. Buchanan, Judith. 2012. Literary Adaptation in the Silent Era. In A Companion to Literature, Film, and Adaptation, ed. Deborah Cartmell, 17–32. Malden, MA: Blackwell. Cartmell, Deborah. 2015. Adaptations in the Sound Era: 1927–37. New York: Bloomsbury. ———. 2012. Sound Adaptation: Sam Taylor’s The Taming of the Shrew. In A Companion to Literature, Film, and Adaptation, ed. Deborah Cartmell, 70–83. Malden, MA: Blackwell. Cartmell, Deborah, and Imelda Whelehan, eds. 2007. The Cambridge Companion to Literature on Screen. Cambridge: Cambridge University Press. Cazeaux, Clive. 2005. Phenomenology and Radio Drama. British Journal of Aesthetics 45 (2): 157–174. https://doi.org/10.1093/aesthj/ayi018. Châteauvert, Jean, and André Gaudreault. 2001. The Noises of Spectators, or the Spectator as Additive to the Spectacle. In The Sounds of Early Cinema, ed. Richard Abel and Rick Altman, 183–191. Bloomington: Indiana University Press. Cook, Malcolm. 2019. Sing Them Again: Audience Singing in Silent Film. In Music and Sound in Silent Film: From the Nickelodeon to The Artist, ed. Ruth Barton and Simon Trezise, 61–75. New York: Routledge. Cooke, Mervyn. 2008. A History of Film Music. Cambridge: Cambridge University Press. Corrigan, Timothy. 1999. Film and Literature: An Introduction and Reader. Upper Saddle River, NJ: Prentice Hall. Crafton, Donald. 1997. The Talkies: American Cinema’s Transition to Sound, 1926–1931. New York: Scribner’s. Crangle, Richard. 2001. ‘Next Slide Please’: The Lantern Lecture in Britain, 1890–1910. In The Sounds of Early Cinema, ed. Richard Abel and Rick Altman, 39–47. Bloomington: Indiana University Press.

830

K. Newell

Crook, Tim. 1999. Radio Drama: Theory and Practice. London: Routledge. DeForest, Tim. 2008. Radio by the Book: Adaptations of Literature and Fiction on the Airwaves. Jefferson, NC: McFarland. Dickinson, Kay. 2008. Off Key: When Film and Music Won’t Work Together. Oxford: Oxford University Press. Echols, Katherine Barnes. 2017. King Arthur and Robin Hood on the Radio: Adaptations for American Listeners. Jefferson, NC: McFarland. Elleström, Lars. 2021. Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In Borders, Volume 1: Intermedial Relations Among Multimodal Media, edited by Lars Elleström, ed. Beyond Media, 3–91. Cham: Palgrave Macmillan. Elliott, Kamilla. 2020. Theorizing Adaptation. New York: Oxford University Press. Everson, William K. 1998. American Silent Film. New York: Da Capo Press. Geraghty, Christine. 2008. Now A Major Motion Picture: Film Adaptations of Literature and Drama. Lanham: Rowman & Littlefield. Goldmark, Daniel. 2017. Adapting the Jazz Singer from Short Story to Screen. Journal of the American Musicological Society 70.3 (Fall): 767–817. Gorbman, Claudia. 1987. Unheard Melodies: Narrative Film Music. Bloomington: Indiana University Press. Grossman, Julie, and R. Barton Palmer, eds. 2017. Adaptation in visual culture: Images, texts, and their multiple worlds. London: Palgrave Macmillan. Gunning, Tom. 1996. ‘Now You See It, Now You Don’t’: The Temporality of the Cinema of Attractions. In Silent Film, ed. Richard Abel, 71–84. New Brunswick: Rutgers University Press. Hand, Richard. 2017. Radio Adaptation. In The Oxford Handbook of Adaptation Studies, ed. Thomas Leitch, 340–355. New York: Oxford University Press. Hand, Richard J., and Mary Traynor. 2011. The Radio Drama Handbook: Audio Drama in Context and Practice. New York: Continuum. Harrison, Louis Reeves. 2011. Jackass Music. In Celluloid Symphonies: Texts and Contexts in Film Music History, ed. Julie Hubbert. Berkeley: University of California Press. Have, Iben, and Birgitte Stougaard Pedersen. 2021. Reading Audiobooks. In Beyond Media Borders, Volume 1: Intermedial Relations Among Multimodal Media, ed. Lars Elleström, 197–216. Cham: Palgrave Macmillan. ———. 2016. Digital Audiobooks: New Media, Users, and Experiences. New York: Routledge. Hubbert, Julie. 2011. Celluloid Symphonies: Texts and Contexts in Film Music History. Berkeley: University of California Press. Hutcheon, Linda. 2013. A Theory of Adaptation. 2nd ed. New York: Routledge. Huwiler, Elke. 2010. Engaging the Ear: Teaching Radio Drama Adaptations. In Redefining Adaptation Studies, ed. Dennis Cutchins, Laurence Raw, and James M. Welsh, 133–145. Lanham: The Scarecrow Press. Ingham, Mike. 2017. Popular Song and Adaptation. In The Oxford Handbook of Adaptation Studies, ed. Thomas Leitch, 324–339. New York: Oxford University Press. Inglis, Ian. 2012. Music Into Movies: The Film of the Song. In A Companion to Literature, Film, and Adaptation, ed. Deborah Cartmell, 312–329. Malden, MA: Blackwell. Jellenik, Glenn. 2010. Quiet, Music at Work: The Soundtrack and Adaptation. In Adaptation Studies, ed. Christa Albrecht-Crane and Dennis Cutchins, 221–243. Madison: Fairleigh Dickenson University Press. Klenotic, Jeffrey. 2001. ‘The Sensational Acme of Realism’: ‘Talker’ Pictures as Early Cinema Sound Practice. In The Sounds of Early Cinema, ed. Richard Abel and Rick Altman, 156–166. Bloomington: Indiana University Press. Kozloff, Sarah. 1995. Audiobooks in a Visual Culture. Journal of American Culture 18 (4): 83–95. Lacasse, Germain. 2012. The Film Lecturer. In A Companion to Early Cinema, ed. André Gaudreault et al., 487–497. Hoboken: Wiley. Leitch, Thomas, ed. 2017. The Oxford Handbook of Adaptation Studies. Oxford: Oxford University Press.

32

Adaptation and Sound

831

———. 2012. Adaptation and Intertextuality, or, What Isn’t an Adaptation, and What Does It Matter? In A companion to Literature, Film, and Adaptation, ed. Deborah Cartmell, 87–104. Malden, MA: Blackwell. ———. 2007. Film Adaptation and Its Discontents: From Gone with the Wind to The Passion of the Christ. Baltimore: Johns Hopkins University Press. ———. 2003. Twelve Fallacies in Contemporary Adaptation Theory. Criticism 45 (2): 149–171. Levinson, Jerrold. 2004. Film Music and Narrative Agency [1996]. In Film Theory and Criticism, ed. Leo Braudy and Marshall Cohen, 6th ed., 482–512. New York: Oxford University Press. Marks, Martin Miller. 1997. Music and the Silent Film: Contexts and Case Studies, 1895–1924. Oxford: Oxford University Press. McFarlane, Brian. 1996. Novel to Film: An Introduction to the Theory of Adaptation. Oxford: Clarendon Press. McKnight, Stephanie. 2011. Happier with Dreams: Constructing the Lisbon Girls through Nondiegetic Sound in The Virgin Suicides. In True to the Spirit: Film Adaptation and the Question of Fidelity, ed. Colin MacCabe, Kathleen Murray, and Rick Warner, 115–129. Oxford: Oxford University Press. Meikle, Kyle. 2019. Adaptations in the Franchise Era, 2001–16. London: Bloomsbury. Miller, Bonnie M. 2018. ‘The Pictures Are Better on Radio’: A Visual Analysis of American Radio Drama from the 1920s to the 1950s. Historical Journal of Film, Radio and Television 38 (2): 322–342. Musser, Charles. 1990. The Emergence of Cinema: The American Screen to 1907. Berkeley: University of California Press. Naremore, James, ed. 2000. Film Adaptation. New Brunswick: Rutgers University Press. Newell, Kate. 2017. Expanding Adaptation Networks: From Illustration to Novelization. Basingstoke, UK: Palgrave Macmillan. Origin of term Theater of the Mind. 2017. https://rpg.stackexchange.com. Preston Leonard, Kendra. 2019. Cue Sheets, Musical Suggestions, and Performance Practices for Hollywood Films, 1908–1927. In Music and Sound in Silent Film: From the Nickelodeon to The Artist, ed. Ruth Barton and Simon Trezise, 45–60. New York: Routledge. ———. 2018. Taking a Cue: Accompanying Early Film. New Music Box. https://nmbx. newmusicusa.org/taking-a-cue-accompanying-early-film/. Rajewsky, Irina O. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. Intermédialités/Intermediality 6: 43–64. https://doi.org/10.7202/1005505ar. Raw, Laurence. 2018. The Gothic imagination in American sound recordings of Frankenstein. In Adapting Frankenstein: The Monster’s Eternal Lives in Popular Culture, ed. Dennis R. Cutchins and Dennis R. Perry, 62–76. Manchester: Manchester University Press. Raynauld, Isabelle. 2001. Dialogues in Early Silent Sound Screenplays: What Actors Really Said. In The Sounds of Early Cinema, ed. Richard Abel and Rick Altman, 69–78. Bloomington: Indiana University Press. Redkar, Surabhi. 7 audiobooks you can listen to while you’re waiting for their film adaptations to hit the big screen. Insider 20 May 2020. https://www.insider.com/audiobooks-you-can-listen-tobefore-film-adaptations-come-out-2020-5. Richardson, Robert. 1969. Literature and Film. Bloomington: Indiana University Press. Rippy, Marguerite. 2003. Orson Welles and Charles Dickens 1938–1941. In Dickens on Screen, ed. John Glavin, 145–154. Cambridge: Cambridge University Press. Rubery, Matthew, ed. 2011. Audiobooks, Literature, and Sound Studies. New York: Routledge. Sanders, Julie. 2006. Adaptation and Appropriation. New York: Routledge. Simonson, Mary. 2021. Finding Meaning in Intermedial Gaps. In Beyond Media Borders, Volume 2: Intermedial Relations Among Multimodal Media, ed. Lars Elleström, 3–31. Cham: Palgrave Macmillan. ———. 2019. ‘Adding to the Pictures’: The American Film Prologue in the 1920s. American Music. 37.1 (Spring): 1–28.

832

K. Newell

Stam, Robert. 2005a. Literature Through Film: Realism, Magic, and the Art of Adaptation. Malden, MA: Blackwell. Stam, Robert. 2005b. Introduction: The Theory and Practice of Adaptation. In Literature and Film: A Guide to the Theory and Practice of Film Adaptation, 1–52. Malden, MA: Blackwell. Stam, Robert, and Alessandra Raengo, eds. 2004. A Companion to Literature and Film. Malden, MA: Blackwell. Stam, Robert, and Alessandra Raengo, eds. 2005. Literature and Film: A Guide to the Theory and Practice of Film Adaptation. Malden, MA: Blackwell. Szwydky, Lissette Lopez. 2020. Transmedia Adaptation in the Nineteenth Century. Columbus: Ohio University Press. Trezise, Simon. 2019. Historical Introduction. In Music and Sound in Silent Film: From the Nickelodeon to The Artist, ed. Ruth Barton and Simon Trezise, 1–21. New York: Routledge. Verma, Neil. 2012. Theater of the Mind: Imagination, Aesthetics, and American Radio Drama. Chicago: University of Chicago Press.

Music Transformation in Literature

33

Beate Schirrmacher

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Field of Word and Music Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedial and Media-Specific Aspects of Music and Literature . . . . . . . . . . . . . . . . . . . . . . . . . Music Representation Transmediates Ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Forms of Music Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aims of Music Transformation in Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Representation of Music in Diegesis and Discourse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Musicians, Conflicts, Ideologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Listening to Music and Sounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Popular Music and Musical Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedial Structural Parallels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metaphorical Frame and Transmedial Common Ground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Repetition and Contrast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Polyphony and Multivoicedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Performance and Transgression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transgressing Borders and Identities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Beyond the Written Text: Literature Performance and Multimodality . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

834 835 837 839 840 841 842 843 844 847 848 849 851 852 853 854 854 855 856

Abstract

This chapter explores the roles of music in literature and how the representation of musical sounds, forms, and genres transforms literature. The chapter surveys how music transformation is used in different literary contexts. Music transformation does not primarily aspire to transform literature into music. Instead, the intermedial relationship with music transforms literary narration and expression and expands the understanding of what kinds of experiences language and written B. Schirrmacher (*) Linnaeus University, Växjö, Sweden e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_42

833

834

B. Schirrmacher

texts can convey. Elleström’s concept of media transformation highlights how the representation of music and the transmediation of structures and ideas are always interrelated. Intermedial relationships with music appear as a performative tool to reach poetological aims. The transgressive potential of music is used not to explain but to express and perform complex relationships. Keywords

Music · Literature · Performance · Musicalization · Auditory experience

Introduction What is the role of music in literature? How is music as an auditory art form represented in written narratives and poems? Integrating words and musical sounds is an intrinsic part of human communication. In every performance of vocal music, words and music integrate in complex ways. Poetry performances or audiobooks offer sites for interaction between musical and verbal sounds, and in some literary and musical genres, the line between speech and singing is difficult to draw. However, we perceive literature and music as two different media types, used differently and qualified by different conventions and forms of meaning-making. The relationship between music and literature has been told as the story of two “sister arts” (Brown 1948) that shared a common origin in oral performance but grew apart – the more words were spread in written form and as instrumental art music gained more independence. The paradoxical tension of contrariness and mutual interdependence is particularly felt in the transformation processes between music and literature. This chapter explores the various roles of music in literature and looks at the specific challenges of how musical sounds and patterns transform literary words, texts, and narratives. The chapter highlights how the descriptions of music or structural analogies in musical genres voice not only a specific kind of music but also connected ideologies, which can be used to both confirm and subvert power relations. Within the endless possibilities of music transformation, Western art music has long had a significant impact. The music of composers such as J.S. Bach, Ludwig van Beethoven, and Richard Wagner is used to voice and stage associated ideas in different socialhistorical contexts. Actual and fictive pieces of music are described extensively or used as structural models, as can be noted in the title of Gabriel Josipovici’s Goldberg: Variations (2002) or Elfriede Jelinek’s Winterreise (2011). Formal composing principles recurringly provide models for dramatic, poetic, or narrative structure, such as in August Strindberg’s Ghost Sonata (1907) or Edgardo Vega Yunqué’s Blood Fugues (2005). In addition, Toni Morrison’s Jazz (1992), Amit Chaudhuri’s Afternoon Raag (1993), Nick Hornby’s High Fidelity (1995), and Rainald Goetz’s Rave (1998) explore a wide variety of musical genres through literature. This chapter begins with a short presentation of the field of word and music studies and looks at different traditions for approaching and analyzing the

33

Music Transformation in Literature

835

representation of musical sounds and structures in literature. More formal analyses of the intermedial relationship are interested in exploring how literary texts bridge the apparent differences in musical and literary modes and conventions. More culturally informed analyses trace how music transformation communicates in a specific literary and socio-historical context. The chapter provides a survey of how music transformation is used in different literary contexts and how music representation and transmedial structural parallels have been explored in word and music research. The chapter stresses how mediaspecific differences between a novel and performed music, between narrative and musical form, between written words and audible sounds are bridged by fundamental transmedial characteristics such as repetition, voice, or performativity that music and literature share. Music transformation does not primarily aspire to transform literature into music. Instead, the intermedial relationship with music transforms literary narration and expression. It expands the understanding of what kinds of experiences language and written texts can convey and are used to communicate aesthetic and political ideas.

The Field of Word and Music Studies The interest in music transformations in literature has been at the core of word and music studies for a long time. When C.S. Brown (1948) provided an early overview of the main historical sites of intermedial relationships between music and literature, he focused on some musical forms that have been influential in literary writing, such as the fugue, the sonata, and the counterpoint. Steven Paul Scher (1968) explored examples of how the detailed descriptions of pieces of music create what he calls “verbal music” in German Romantic literature and offered a first mapping of the field (1982). Werner Wolf embedded word and music studies within a general intermedial framework when he explored how musical characteristics are transformed into so-called musicalized fiction (Wolf 1999). Wolf developed a general intermedial typology that differentiates between extracompositional and intracompositional, direct and indirect, telling and showing forms of intermediality that can be applied to all kinds of intermedial relationships (see also Wolf 2008; Gess and Honold 2017). Another tradition within word and music studies is less focused on formal comparative analysis and approaches the relationship between words and music in a larger cultural context within an expanded intertextual framework. From his early “tandem readings” of music and poetry (1984), the work of Lawrence Kramer, his cultural perspective in musicology (2002, 2011), and the continual exploration of various word and music relationships (see, e.g., 2017) have been influential in this intermedial-intertextual approach. Stephen Benson’s survey of the roles and forms of Western art music in contemporary fiction (2006) applies this cultural perspective, where word and music relationships are examined in a more interdisciplinary musico-literary discourse (see also Chapin and Clark 2013). Since the late 1990s, the Association of Word and Music Studies has provided a platform for studying musico-literary relationships in the form of conferences, the

836

B. Schirrmacher

book series Word and Music Studies, and, since 2010, a network for emerging scholars, Word and Music Association Forum. Word and music studies have often approached the literary transformation of music from one of these two perspectives, either developing or using specific intermedial theoretical frameworks or examining a wide range of case studies from various historical and cultural contexts. The different research traditions are exemplified in two recent handbooks that establish common reference points and provide the scholars of literature and music with relevant concepts and basic disciplinary knowledge for both media involved. The Handbuch Musik & Literatur (Gess and Honold 2017) approaches the history, theory, and mediality of word and music relationships using an intermedial framework. The handbook presents a systematic approach to combinations and transformations of music in literature and literature in music with the support of the theoretical framework developed by Wolf and Rajewsky. As a point of departure the handbook discusses transmedial questions and their media-specific outcomes in literature and music. These intermedial and transmedial theoretical discussions then provide the ground for explorations of specific aesthetic and historic settings as well as cooperation between literary and musical artistic practice in specific historical contexts, from the impact of antique practices and myths to rap and spoken word. The Edinburgh Companion to Literature and Music (da Sousa Correa 2020) anchors word and music studies in this interdisciplinary, intertextual discussion and offers a broad survey and historical overview of the mutual interdependence in literary and musical aesthetics, demonstrating the variety and interdependence of music and literature in specific historical and cultural contexts. The contributions map how theories, aesthetics, and concepts from music and literature have informed musico-literary relationships and strive to map the intermedial web of “textual and cultural points of connection” (da Sousa Correa 2020: 5). Music and literature are brought together to mutually illuminate each other and offer self-reflexive insights into artistic and creative communication. The individual chapters flesh out these points of connections in specific genres or historical contexts as well as in the works of musicians and authors. Another way to approach the intermedial relationship between music and literature takes its point of departure in Lars Elleström’s media modalities (2021). This medial framework starts from the analysis of the material and the spatiotemporal, sensorial, and semiotic characteristics that all media share, even though they draw on different modes. This model provides the basis for intermedial comparisons between literature and music as media types that are characterized by media-specific and transmedial modes, usually qualified differently by use and conventions. This approach enables the analysis of the literary transformation of music based on transmedial similarities (Schirrmacher 2012; Bruhn et al. 2021). Vincent Caers’s detailed analysis of the modes and media types of performed music and the musical score, developed to improve interdisciplinary collaboration between musicians and visual artists, demonstrates how this model can be used in musico-literary contexts (Caers 2022).

33

Music Transformation in Literature

837

Transmedial and Media-Specific Aspects of Music and Literature Although literature and music share a common origin in oral performance, they have developed into distinct art forms or media types that are qualified by different conventions and used in different contexts. This divergence can be seen as a prerequisite for intermedial relationships across conventional media borders. The relationship of literature and music has been told as the story of two “sister arts” (Brown 1948) that shared a common origin before they separated, and yet they remain involved in sometimes conflicting and sometimes more harmonious interdependence. However, this unity is not restricted to a lost origin. The auditory experience continues to provide transmedial common ground that all kinds of musico-literary intermediality draw attention to, though in different ways. When words and music are heard simultaneously in vocal music, they share the same material, sensorial, and spatiotemporal modes but offer different forms of meaningmaking. As performed (instrumental) music and written literature primarily rely on different material, sensorial, spatiotemporal, and semiotic modes, the common ground is more difficult to detect in a written text. A media product such as a printed novel that presents written text on pages or a screen is primarily perceived visually and in a linear sequentiality of sentences and paragraphs, usually with a primary focus on the conventional meaning of words (see also Atã and Schirrmacher 2021). A live performance of a piece of instrumental art music, such as a symphony, is mediated by sound waves, perceived auditorily, but it is also an embodied experience with a specific range and length. This experience offers different possibilities for habitual, symbolic, iconic, and indexical meaning-making (Jensen and Knust 2022: 59–61; Smith 2016: 9–19, see also Kramer 2002). Music transformation in literature draws attention to overlapping modes that bridge the gaps between the texts on pages and screens, representing specific events or experiences and the interconnecting soundwaves and organized sounds of musical experience that open up for meaning-making based on contiguity. These descriptions of differences between specific literary and musical media products, according to Elleström’s media modalities, draw attention to sites and practices that stress the transmedial similarities. For instance, certain literary texts, such as James Joyce’s Finnegans Wake (1939), stress the interplay of the habitual with iconic and indexical meaning-making. Listening to audiobooks instead of reading printed books changes the material, sensorial, and spatiotemporal characteristics of the same verbally constructed narrative (Have and Pedersen 2020). Thus, even written literature and performed music potentially share transmedial modes to which musico-literary research draws attention. For instance, Emilie Crapoulet suggests approaching “musicality” as a travelling concept (Bal 2002) that has traveled across many disciplinary boundaries and is applied to describe different phenomena in different contexts. This nomadic perspective on musicality moves the discussion from correct to relevant use of the term and how it helps the understanding of the literary text (Crapoulet 2009: 82). Lawrence Kramer stresses how the concept of voice is relevant to both speech and music. Compared to how verbal and non-verbal signification meet in the voice, Kramer points out, the acoustic

838

B. Schirrmacher

art form music appears as an “artificial extension of voice, a vocal prosthesis” (2017, vii). In a literary context, Hazel Smith draws attention to how voice is a relevant concept on several levels, such as the auditory voice of performance or the voice of the discourse (Smith 2016: 29–32). While performance and notation play a role in both literature and music, the relationship between performance and notation is different in literary and musical contexts (Previšić 2017). Listening and reading have different functions in music and literature, as fewer people read musical scores than written texts. Generally, even the written score remains more closely connected to performance, as it seldom is thought to replace but instead to enable and coordinate performance (Caers 2022: 46; see also Deeming 2020). However, written text is less specific regarding how it should be pronounced and intonated, and it has greater autonomy regarding its auditory performance. While both the score and the text create linear relationships between signs to represent temporal processes, the score also uses vertical relationships to indicate how different linear sequences are to be performed simultaneously. When authors like Thomas Mann, Claude Lévi-Strauss (2001), or Roland Barthes (1970: 35–36) draw on the musical score, they highlight how existing spatial relationships can be organized to describe texts and narratives as well. Here, as elsewhere, linear temporality and spatial relations connect and replace each other in intermedial relationships (see also Albright 2000). Rhythm is another relevant transmedial category. The “formal integration of rhythmically integrated time,” Kramer points out, is fundamental in both music and poetry (Kramer 1984, viii). However, even on other levels, the regular successions of strong or weak elements can form connections between musical and literary experiences and the rhythm of everyday life (Stougaard Pedersen 2013; Huntingdon 2009). Thus, music transformations in literature exploit the tensions between significant conventional differences and underlying transmedial qualities. However, whereas the concepts of sound, voice, and structure all have transmedial dimensions, Previšić draws attention to their media-specific realizations when he points out that “[l]iterary sound is never musical sound. Literary voice is never musical voice; the literary structure conceived as a score is never a musical score” (Previšić 2017: 52, my transl.). Thus, the media-specific realization of transmedial categories such as rhythm and voice in verbal language provides arguments for or against the “musicality” in language, Hejmej points out (2014: 36–44; “Intermediality and Medium Specificity Picturing Music in the 19th century”). On the other hand, the different ways music has been conceived in relation to language (Hindrichs 2017) highlight certain musical qualities and downplay others. When thinking of how music is produced by manipulating objects that create sound waves, music appears more closely connected to mathematics as applied physics rather than language. The physiological impact of sound waves that listeners respond to suggests an understanding of music as a preconceptive language of affect. Thinking of how music stimulates a subjective, emotional response suggests an understanding of music as a potentially universal language that transcends the conventional signification of

33

Music Transformation in Literature

839

verbal languages. Finally, as musical structure develops specific musical propositions by inferential and syntactic rules, musical structure shares similarities with logic and can be construed as a language of form, of abstraction. From a medial perspective, it should to be stressed how music transformation in literature thus does not borrow specific “musical” traits. Instead, it is a specific use of language that stresses transmedial characteristics that literature and music share but may require references to music to be noticed in literature.

Music Representation Transmediates Ideas From a cultural perspective, it is important to stress that which aspects of music (and language) are valued and chosen to define the concepts such as “musicality” depends on the cultural, social, and philosophical environment. The Romantic idealization of art music in Western thinking can serve as an example (Dahlhaus 1989). As long as the ideal in artistic expression was the mimetic, detailed representation of the physical world, there was little motivation for writers to stress the transmedial similarities with music. However, the more artists in the Romantic period became interested in expressing subjective experiences and emotions, the more the affordances of music to represent motions and developments (Elleström 2021: 22) and the potential for affective, emotional, and associative meaning-making in music became an asset. Philosopher Arthur Schopenhauer construed music as a selfsufficient, timeless, and universal language, as “the only art form that did not merely copy ideas but actually embodied the will itself” (Albright 2004: 39; see also Pankow 2017). Romanticist literary and musical aesthetics were developed in close interdependence (Naumann 2017; Kramer 2020). Music was construed as an ideal that, according to aestheticist Walter Pater, all art should aspire to (Pater 1986: 86). However, the same qualities of instrumental art music, which Romantic aesthetics idealized as transcendent, as “absolute,” or “pure” form, can also be seen within a different and archaizing aesthetic frame, seen as deficient and estranged from a common origin Ancient Greek concept of mousike, which did not differentiate what now is conceived to be music, poetry, and dance (Dahlhaus 1989: 25). Friedrich Nietzsche celebrated music as Dionysian power and Richard Wagner’s music drama as a rebirth of the Greek tragedy. Nietzsche’s celebration of Wagner was highly influential for the broad cultural impact of Wagner’s idea of a Gesamtkunstwerk, a “synthesis of the arts.” Even the formal unity of Wagner’s music dramas that was structured by leitmotifs, recurring motifs connected to dramatic elements, became influential in literary aesthetics as well. Novelists like Marcel Proust or Thomas Mann transformed Wagner’s ideas into a model for writing and reading (Breatnach 2020; Euchner 2005). Intermedial relationships can also be part of artistic self-reflection. Peter Dayan has traced this kind of intermedial thinking when musicians like Frédéric Chopin or Erik Satie, writers like Samuel Beckett, or philosophers like Roland Barthes use “music to compose a definition of literature, and literature to compose a definition of music” (Dayan 2006: 131). Other examples of intermedial self-reflection in aesthetic

840

B. Schirrmacher

and poetological thinking form more triangulating relationships between visual artists, music, and literature (Dayan 2010). Therefore, representing music in literature often includes a meta-referential discussion about the mediality of literature as well (Wolf 1999: 48–51). Yet when music and literature appear as opposing each other in literary texts, these oppositions often go beyond self-reflexively comparing the arts in a competitive aesthetic discourse of paragone. Oppositions between arts and media types in literature are also used to stage varied forms of social or ideological conflicts (Bruhn 2016; see also Bruhn et al. 2021).

Forms of Music Transformation Music can transform literature in different ways. A literary text can explicitly describe music, formally imitate musical structures and techniques, and draw attention to the auditory modes of language. These three transformations have been explored with different terminology and analytical focuses. Scher focuses on the extensive description of pieces of music as “verbal music” (1968, 1982), whereas Claus Clüver approaches poems that referred to pieces of music as “musical ekphrasis” (1999). Wolf explored the “musicalization” of fiction in Romantic, modernist, and postmodernist fiction (1999) as different ways musical structures transform narration. He describes the relationship between elements of “telling” and “showing,” of “thematization” through an explicit description of music and musical pieces, and of “showing” through elements that implicitly transform the literary text based on musical principles. Wolf draws attention to how the forms of structural imitation depend on more explicit strategies of thematization and “auditory highlighting,” the frequent description of sounds in general. Irina Rajewsky’s term “intermedial references” (2002, 2005) stresses the interconnectedness of different strategies even more, as it refers to all ways one medium “with its own means” can thematize, evoke, represent, or simulate another medium, as if the medium referred to was present, although it is absent in the material and sensorial modality. Elleström’s concept of media transformation (2014) further stresses the interdependency of description, imitation, and highlighting of certain modes, and he suggests that elements of media representation in diegesis and discourse and the transmediation of media structures and content are like two sides of the same coin. Therefore, the study of extensive, detailed, and recurring music representations also needs to pay attention to how diegetic representation of music transforms narrative discourse and structure and how it transmediates not only musical form but also aesthetic ideas and social ideologies. The study of how music structurally transforms fiction or poetry further needs to consider how the intermedial relationship between literary and musical form is established with the help of explicit representation. For instance, the structure of repetition and contrast and auditory highlighting in the “Sirens” chapter in James Joyce’s Ulysses is thus intrinsically linked to the overall importance of music as idea, prosodic model and formal principle for narration in Ulysses and Joyce’s writing in general (Weaver 1998, 27).

33

Music Transformation in Literature

841

Consequently, the representation of pieces and sounds in music and the transmediation of structures and ideas are always related. However, this relationship plays out differently, depending on, for example, whether a specific sonata for piano and violin plays a central role in the plot of Marcel Proust’s Du côté de chez Swann (1913, Swann’s Way) or whether the transmediation of sonata forms structures in the Swedish poet Gunnar Ekelöf’s Dada-inspired poem “Sonatform: denaturerad prosa” (1949, “Sonata form: denaturated prose”). Yet both the extensive representation of a specific (although fictive) sonata and a poem that transmediates the ABA pattern of the sonata form bridge the conventional differences between the art forms by drawing attention to what music and literature share.

Aims of Music Transformation in Literature The transformation of music in literature is not an aim in itself, even if within a Romantic and Modernist paradigm, an aspiration to the “condition of music” and the “musicalization of fiction” might be central. Drawing on musical form offers alternative ways to structure and transform narration; it can be seen as a means of transmediating the modern life experience. Thus, authors at different times and in different contexts will have different reasons for looking for alternative models and principles to structure narration and highlight auditory experience. Music thus plays important but different roles in the writings of Marcel Proust, James Joyce, and Thomas Mann, Virginia Woolf (Clements 2019; Varga 2014; Sutton 2013, 2020; Kelley 2010), of Katherine Mansfield (McParland 2009; Sousa Correa 2011), Ezra Pound (Paterson 2020) and Elisabeth Bishop (Cleghorn 2019). Musical form is used to structure the stylistic experiments of the French new novelists like Alain RobbeGrillet, Marguerite Duras, and Robert Pinget (Prieto 2002). The ambiguity of sound and the principle of repetition are central in the fiction and dramatic writing of Samuel Beckett (Prieto 2020; McGrath 2017; Laws 2003) and play a role in Milan Kundera’s, Jeanette Winterson’s, or J.M. Coetzee’s writings (Benson 2006). In writer and composer Anthony Burgess’s work, the intermedial relationship between music and literature works in both directions (Philips 2010; Jeannin 2009). Thus, rather than exploring how well the literary text corresponds with musical principles, focus is better put on how intermedial relationships with music allow writers to transform narration or to stage specific conflicts. Music transformation in novels or poems does not automatically indicate that music is seen as an ideal; it can include more complex and conflicting relationships, as in Paul Celan’s poem the “Death Fugue” (1948), a poem that connects the situation of victims and perpetrators by drawing on contrapuntal compositional form (see Englund 2012). Even postwar German and Austrian writers like Thomas Mann, Günter Grass, Thomas Bernhard, Ingeborg Bachmann, and Elfriede Jelinek draw on music in their struggles with the Nazi past (Lubkoll 2017: 91; Ben-Horin 2016; Schirrmacher 2012; Huber 1992). At the same time, this struggle also reflects the social and ideological role of music in Nazi propaganda and concentration camps. Even in other historical and social contexts, music often appears as a

842

B. Schirrmacher

means to express conflict and trauma. For instance, the representation of jazz music in Toni Morrison’s novel Jazz, is connected to conflict and violence and thus is informed by a drive to witness and testify on the trauma of racism (Visvis 2008; DeCristo 2018). Representing music and structural parallels to music in the context of trauma and violence also draws attention to the history of music as a tool of domination (Johnson and Cloonan 2009) and the ambiguous role of music as a means to escalate or negotiate conflict (Fast and Pegley 2012). Research increasingly draws attention not only to the transformation of literary forms but to how the reading experiences are transformed and highlight a feeling of hearing and listening conditions (Vilmar 2020; Stawiarski 2010). Verbal descriptions of sounds effectively evoke their auditory imagery and contribute to literary soundscapes (Glotova 2021; Frattarola 2018; Schlauraff 2018; Lindskog 2014; Schweighauser 2008; Picker 2003). The representation of music and songs in the diegesis plays a role, on the one hand, in creating the soundscape of urban modernity and, on the other hand, in mediating between orality and literacy, for instance, in African American literature (Schweighauser 2015). In postcolonial and migration literature, representations of music draw on music’s potential for expressing ambiguity as a means to articulate and undermine hegemonic discourses. To summarize, music transformation represents different uses, ideas, and aims connected with music in specific socio-cultural contexts. The intermedial relationship of literature and music allows writers in different ways to realize a sounding and performative poetology that, at different times and places, echoes with different ideas and transgresses different normative borders.

Representation of Music in Diegesis and Discourse The role of music representation comes into focus whenever pieces of music, musicians, listeners, instruments of storage devices, practices, and places are described repeatedly and extensively and have a central role in the plot. Musical characters, objects, and events tend to become involved in dramatic or discursive conflicts that are used to stage narrative, social, or aesthetic conflicts that are intrinsically linked to ideas about music and musical practices. Art, popular, and folk music form different discursive and ideological networks. Particularly in the context of popular music from the twentieth century onwards, these networks also increasingly include devices of storage and display, such as gramophones, tape recorders, records, and cassettes. To catch this complexity and trace the connection between diegetic and discursive acts, dialogue, structure, and socio-historical context can be challenging. These complex entanglements of plot, discourse, and context can be disentangled in a step-by-step analysis of listing which genres, modes, and media types are represented, followed by structuring which qualities are assigned to this kind of music in narrative discourse, and, finally, anchoring this analysis in its textual, literary, and socio-historical context (Bruhn et al. 2021).

33

Music Transformation in Literature

843

Musicians, Conflicts, Ideologies When representations of music play a central role in a literary narrative, the stakes often get high for characters like musicians or composers. These characters are often in conflict with society, weighing society’s norms and its transgressions (Lubkoll 2017). Musical characters try to negotiate these conflicts in ways that often involve violence, mental ill-health, and even lethal consequences. In Romantic literature, the perception of music as a threatening violent force is a recurring motif (Gess 2006). In Heinrich von Kleist’s novella “St. Cecilia, or the Power of Music” (1810), the sounds of an ancient mass appear to strike four brothers with insanity (Krimmer 2014; Gess 2006). The composer Adrian Leverkühn in Thomas Mann’s Doktor Faustus purposely contracts syphilis to increase his artistic inspiration (Fetzer 1990). In Salman Rushdie’s The Ground Beneath Her Feet, the rock music stars Vina and Ormus both suffer violent deaths as their love story is modeled on Western and Asian versions of the Orphean myth (Hoene 2015; Gaultier 2014). The representation of opera offers yet other multilayered possibilities to restage the novel’s conflicts, as can be seen in the writing of Thomas Mann (Tambling 2012) or Marcel Proust (Rushworth 2014). The opera’s dramatic plot works as mise en abyme of the literary plot and at the same time offers a contrapuntal contrast to the everyday diegesis (Halliday 2013; Newark 2011; Skaggs 2010, see also Benson 2006). Still, the kind of conflict and how the literary text engages with it depends on the historical context and the “structural, semantic and aesthetic peculiarities” of the music described (von Ammon 2017: 535). When many nineteenth-century texts, for instance, display a narrative or discursive conflict between instrumental and vocal music, this conflict involves the ideals of so-called “pure” form connected to instrumental music and the ideal of natural sound embodied in the human voice, which draws on Rousseau’s thinking (Lubkoll 2017; see Dahlhaus 1989). The way the description of music engages with other narrative conflicts in Kleist’s novella “St. Cecilia,” such as conventional symbolic writing and embodied performance, can be understood as Kleist’s critical comment to the emerging ideal of absolute music (Gess 2006). In German postwar literature, notably in Thomas Mann’s Doktor Faustus but also in Ingeborg Bachmann’s Malina (1971), the confrontation between the aesthetic concepts of classicist and modernist music is informed by the way Nazi propaganda used music (Lubkoll 2020: 89). In Günter Grass’s The Tin Drum (1959), music and, specifically, the rhythms of drums perform both a manipulation of and a protest against the Nazi ideology (Schirrmacher 2012:95–154). In Jennifer Egan’s A Visit from the Goon Squad (2011) twentieth-century analogue rock music and twenty-first-century, digitally produced music collide in a discourse on the (loss of ) authenticity (Bruhn 2016). These kinds of narrative and discursive conflicts are intrinsically connected to aesthetic ideas about music, which may not primarily connect to contemporary musicological positions. There has been a progression in musicology from emphasizing the autonomy of musical forms and structures to increasingly acknowledging that all musical expression is culturally and socially contingent, autonomous, and socially constructed (Kramer 2002). The modernist and twentieth-century literature uses of premodernist musical forms appear slightly anachronistic when compared to

844

B. Schirrmacher

the contemporary aesthetics in twentieth-century art music, but in a literary context, these forms are used to carry out a modernist poetology. The Romantic idea of music’s aesthetic universality resurfaces in contemporary fiction but often in a way that questions and undermines its universal validity, when other musical traditions such as folk, popular, or other forms of art music complicate commonplace understandings of European art music (Hoene 2020; Dautel 2017). How music is used in a text to stage certain ideas and ideological conflicts connects to the social role and use of specific kinds of music (Deutsch 2015), which often are put in contrast with the perception, practices, and contexts of other musical genres or other media types. Hazel Smith points out that musical meanings unfold in “complex and heterogeneous interactions between listener and work,” which depend on the listener, cultural context, and relationships within and between works (Smith 2016: 18). Similar to the way how the meaning-making potential of music heavily relies upon connotation and association (Jensen and Knust 2021: 59; Deutsch 2015: 230), music representation can be used in literature to carry connotations about characters and situations, and it is loaded with associations to specific social contexts and their ideologies (Schirrmacher et al. 2014). Music representation also highlights the gendered conventions of the music sector and draws attention to how “performing music is performing gender” (Solibakke 2007: 259; see also Kramer 2011). The representation of music transmediates power relationships between class and gender, not only in visual art (Leppert 1993 cross reference to Veres, ▶ Chap. 28, “Picturing Music in the Nineteenth Century”) but also in media representation in literature. Music, sex, gender, and power are often highly intertwined in nineteenth-century literature. Tolstoy’s “Kreutzer Sonata” (1889) is a famous example of how male control over music and female sexuality intertwine (Kramer 2017: 429–450; Dame 2014). Elfriede Jelinek explores the power relationships that involve sex and gender when Erika Kohut ‘s profession as a pianist contributes to her inhibited sexuality in The Piano Teacher (Schirrmacher 2016b; Powell and Bethman 2008). Yet the representation of music often performatively challenges normative binaries. The role of music often remains ambiguous, acting both as part of the norm and as transgressive, as part of the conflict as well as its possible solution (Lubkoll 2017). For instance, when Salman Rushdie rewrites Western rock music history in The Ground Beneath Her Feet, Western rock music remains ambiguous as “a force of political liberation on the one hand and as an agent of cultural imperialism on the other” (Hoene 2015: 6). In Anthony Burgess’s A Clockwork Orange, which explores the relationship between art music and violence, music is described and used in a way that it remains ambiguous, and Ludwig van Beethoven’s Ninth Symphony specifically appears as an “instrument of therapy, of torture – or actually both” (Höying 2011: 172–73).

Listening to Music and Sounds Representing the auditory experience of sound in a visually perceived text is a fundamental change of modes in the sensorial modality. However, even visually

33

Music Transformation in Literature

845

perceived words that describe sound activate the auditory memory and evoke auditory imagery; there is an auditory experience in the absence of actual acoustic stimuli (Reisberg, 1992). The ekphrastic description of pieces of music or repeated references to sounds, noises, acoustics, and lyrics therefore not only foreground the experience of music but also highlight the acoustic qualities of language as well. One way to draw attention to the auditory experience of music in a written text is the quoting of song lyrics. Song lyrics can be approached as a partial reproduction (Rajewsky 2002: 114–117) of a song by quoting part of the lyrics and evoking its sound (Wolf 1999: 67). Parallel to Lawrence Kramer’s concept of speaking melody that “voices [the lyrics] in their absence” (2005: 128), the quoted lyrics are not only written but “singing words” that evoke the auditory images of the melody, especially when fragmentary lyrics can trigger involuntary musical imagery of sticky music (Schirrmacher 2021). Thus, even fragments of familiar song lyrics scattered in the text provide auditory imagery. Familiar songs form a shared, auditory soundscape in Ulysses (Bowen 1975, 1993), in Astrid Lindgren’s novels (Sundmark 2011), and in pop literature (Dautel 2017). The description of diegetic sounds also represents social practices of hearing, auditory meaning potential, and listening as critical practices. As a form of literary performance, verbal descriptions of sounds highlight the diversity of literary voices and a novel’s Bakthinian polyphony (Morrissette 2013) but can also be used to convey the mental disintegration (da Sousa Correa 2021). While the quotation of lyrics depends on the auditory memory, the insertions of score quotation and of musical notation in the text appeal to the reader’s musical literacy. In the Ulysses chapter “Scylla and Charybdis,” the beginning of a Gregorian hymn interrupts the text; the first bars of spirituals head the chapters of African American writer W.E.B. Du Bois The Soul of American People (1903); excerpts of Arnold Schönberg’s melodrama Pierrot Lunaire (op. 21) appear in Austrian Ingeborg Bachmann’s novel Malina (1971); and the first bars of Beethoven’s piano sonata Appassionata (op. 57) frame Swedish Eyvind Johnson’s novel Romantisk berättelse (1953, ‘Romantic tale’). Halliday describes these score insertions as “frozen” music that hovers between sounding and muteness, “a hermetic statement in its own right” (Halliday 2013: 117). Yet, argues Stawiarski, although a paratextual score quotation in the form of paragraphs and titles may appear as “opaque, an unreadable, undecipherable sign,” it functions in the way that suggests even to the unskilled reader a musical approach to the written text (Stawiarski 2010: 102). Moreover, score quotations interact with verbal descriptions of musical sounds as conflicting and complementary ways to represent auditory experiences (Lubkoll 2017: 88–89; see also Schirrmacher 2016a). With elaborate and detailed verbal descriptions of a piece of music, so-called “verbal music” (Scher 1968), “musical ekphrases” (Clüver 1999), “melophrases” (Edgecombe 1993), or “evocations” of music (Wolf 1999, 2017: 101) aim to convey the auditory and subjective experience of a particular piece of music. These elaborate descriptions combine different ways to verbally represent music. Odendahl (2008: 15–17) mentions the description of the auditory experience (the sounds of instruments and voices), the causes and effects of musical sound (the movements of performers and the reactions of the audience), and formal analysis of thematic and

846

B. Schirrmacher

harmonical structure or music by means of musical terminology. References to a particular genre, such as “waltz” or “techno,” already evoke generic auditory imagery. Elaborate music descriptions aim to convey a subjective experience of music and also involve individual associations of the listener, something that Wolf calls “imaginary content analogy” (Wolf 1999: 64). These musical descriptions aim to represent more than just a piece of music and can be used for different narratological ends. In E.M. Forster’s Howards End (1910), the performance of Beethoven’s Fifth Symphony is described from the perspective of different characters, which effectively characterizes the listening characters but also illustrates different aesthetic stances regarding what to listen to in music. Thomas Mann’s elaborate musical descriptions weave together auditory description and formal analysis with dramatic conflict and philosophical ideas (Scher 1968; see also Lubkoll 2017: 85–87). The exalted descriptions of fictive pieces of art music push the reader of Anthony Burgess’s A Clockwork Orange into accepting hooligan Alex’s perspective, which makes it hard to keep a critical distance from his violent excesses (Rabinowitz 2003). Marcel Proust uses the elaborate descriptions of Vinteuil’s sonata for piano and violin in Du coté de chez Swann (1913) to demonstrate the close connection between sensual perception and involuntary memory; Joseph Acquisto (2017) points out how attentive listening becomes a model for how the reader should navigate the meandering narrative threads of the seven volumes of the À la recherche du temps perdu. Thus, music description not only aims to represent sound and a piece of music but also draws attention to its reception and subjective sound experience. In a silent description of Beethoven’s Appassionata in Eyvind Johnson’s Romantic Tale, the absence of sound-based description expresses the young working-class writer Olle’s sense of exclusion. At the same time, his focus on the performer’s movement highlights music as a performed and embodied experience (Schirrmacher 2016a). Narrative description with a focus on sound includes musical sound and spoken language as part of a diegetic soundscape, which is much more than background noise in the plot of the novel. When the descriptions of auditory experience are approached from the perspective of sound studies, music becomes part of a literary acoustics, placed in the social, cultural, and technological contexts of sound production and reproduction. The literary soundscape not only is treated as a representation of sound but performs the text as a sounding object (Schweighauser 2015). In a musico-literary perspective, the connection between music and other sounds in the text has been seen as a form of “acoustic foregrounding” (Wolf 1999: 75) that draws attention to musical parallels. Sound studies invert the relationships: The focus moves from the interart question of music representation to the acoustic experience of a visually perceived text. Philipp Schweighauser therefore points out to how modernist writers not only turn to music to find alternative models of narration, they also respond to the increase of urban noise and the invention of acoustic recording, and they explore the challenge to describe noise with words that always signify and explain. Therefore, one could speak not only of a musicalization but even of a noisification in modern and postmodern literature (Schweighauser 2008; “Sonification of Modernist Fiction”).

33

Music Transformation in Literature

847

Popular Music and Musical Objects While established musical forms of Western art music like the fugue and the sonata remained influential throughout the twentieth century, literary writers’ interests in newer forms of popular music like blues, jazz, rock, pop, rap, and electronic dance music also led to a “pluralization and globalization” (von Ammon 2017: 537) of music transformation in literature. The variety of different styles of popular music as well as new auditory experiences of changing political, cultural, and aesthetic significance have an impact on the strategies of music transformation. Thus, the music transformation of popular music does not necessarily connect to the same avant-garde, experimental aesthetics as the music transformation of Western art music in modernist literature (Höfer 2017). Contemporary poetry represents music as performance and audience-engaging (May 2020). The impact of popular music on literature is not only to be traced in transformations of novels or written poetry, but the impact of jazz, rock, and rap involves collaborations, combinations, and integrations of lyrical and musical performance. Literary genres of jazz fiction (Feinstein and Rife 2009) and jazz poetry (Young 2003, 2006) describe the places, musicians, instruments, and lyrics of different jazz styles. Novels apply structural principles that are characteristic of jazz music such as improvisation, syncopation, call and response patterns, or solos (Petermann 2014; Grandt 2004; Rife 2008; Lock and Murray 2009). Yet jazz representations in literature are part of different discourses. In the works of some writers, including in African American authors’ novels like Ralph Ellison’s Invisible Man (1952) or Toni Morrison’s Jazz (1992), jazz music is rooted in oral tradition, is represented in an ambiguous and transgressive way, and transmediates the experience of racism and slavery. In the writing of others, including F. Scott Fitzgerald, jazz music is more often a sounding signifier for modernity (von Ammon 2017: 537). Outside the United States, the introduction of jazz music and the modern age is closely connected with a radical questioning of social norms and national identities. For example, in the Scandinavian literature between the 1920s and 1950s, jazz music was framed in terms of an “enticing and horrifying” otherness (Fornäs 2010: 219; Strauss 2003). In German literature of the 1920s and 1930s, the literary discourse about jazz connects to the cultural debate on the prospects of the young German democracy (Höfer 2018; Krick-Aigner and Schuster 2013). Different cultural contexts and techniques are at play concerning the representation of pop music in literature. The quoting of titles, names, and lyrics of pop music is characteristic of so-called Litpop (Carroll and Hansen 2016; Viol 2006). The missing commentary or the name- and title-dropping instead of extensive descriptions “seems to leave this intermedial connection at the level of a passing sound only” (Dautel 2017: 176). Yet the titles, lyrics, and performers of pop integrate the literary text in a global media discourse of a contemporary media culture and communicate a sense of global community and belonging (Dautel 2017). They connect the literary texts with a transmedial music discourse that takes place in magazines, on radio programs, and on record sleeves (Carroll and Hansen 2016: 9; see even Arvidson 2009). The universality of pop music in Litpop recalls the

848

B. Schirrmacher

Romantic stereotype of music as a universal language. However, the universality of pop music is not as an ideal but instead described a part of globalized media culture. Moreover, the music repertoire of many contemporary novels is not restricted to one particular genre but comprises all sorts of popular, dance, and art music. The representation of music in contemporary poetry foregrounds the performative and engaging effect of music and sounds (Smyth 2008; see also Hertz and Roessner 2014). The sounds, practices, and aesthetics of electronic dance music get center stage in so-called dance-floor-driven literature, which uses the sounding space of the dance club to structure a disparity of narrative threads and relationships (Morrison 2020). Structural parallels to electronic music productions, such as sampling and loops, can transform the narrative in texts such as Rainald Goetz’s Rave (1998) (Tillmann 2012). In twentieth- and twenty-first-century literature, the technical sounding objects like gramophones, records, singles, LPs, cassette tapes, and tape recorders that store and remediate musical performances tend to play an essential role, especially when popular music is described. In Nick Hornby’s High Fidelity (1995), the life of the record shop owner Rob Fleming revolves around records and popular music. LPs and cassette tapes are discussed as objects and recording formats with aesthetic qualities in ways that intertwine material, aesthetic, and social relationships (Keskinen 2005). Although the sonic experience of live performances differs from the experience of listening to recorded music, Ralf Goebel (2020) points out how literature even describes the recordings of music as intimate and personal experiences. The engagement with music needs a counterpart, and the recorded, “dead acoustic data” (Goebel 2020: 6) are reanimated when played back with technical devices such as gramophones and records. These sounding objects gain transgressive, quasi-human, uncanny, or nostalgic agency in the text. The discourse of quasi-human subjectivity of sound technologies is already present in modernist literature. The descriptions of gramophone recordings in modernist literature create a dialectic discourse between the ideal of (high) fidelity and added value, between the displacement and the equability of recorded music, which is either perceived as distorting or as capable of adding layers of signification to the musical experience (Halliday 2020). The dematerialized presence of recorded and digitally streamed music in everyday life via music apps will eventually transform representations in literature. The description of the constant, instant, and personalized access to streamed music will engage with the different material, sensorial, and spatiotemporal experience of live performance (Goebel 2020: 9).

Transmedial Structural Parallels The verbal description of music includes both a perspective of telling, of speaking about music, and a transformative element of showing, as the words of the text transmediate auditory perception and ideas associated with a specific piece or genre

33

Music Transformation in Literature

849

of music. Yet there are even texts, novels, and poems that display or show a relationship, though the description of the production, experience, and sounds of music is not necessarily in focus. The texts structurally remind the reader of the experience of engaging with music. Often titles of novels or poems refer to musical form in a way that seems to indicate that what readers hold in their hands “is all but a book” (Stawiarski 2010). The titles of Aldous Huxley’s Point Counter Point (1928), Marguerite Duras’s Moderato Cantabile (1958), Kazimierz Brandys’s Rondo (1982), Elfriede Jelinek’s Winterreise (2011), and Tomas Tranströmer’s poems “Ostinato” (1957) or “Allegro” (1962) suggest that readers should approach the text as if it were music. As “imitations” of music they indicate a “musicalization” of literature “that points towards a presence of music in the signification of a text which seems to stem from some kind of transformation of music into literature” (Wolf 1999: 51). Literary texts can refer to the same musical principle, such as the fugue or the sonata form, in innumerable and very different ways. For instance, the chapter “Sirens” in Ulysses primarily appears to repeat and contrast a set of motifs and sounds, so a narrative development may be difficult to trace. There are many approaches to how the structure of Joyce’s chapter is based on contrapuntal or other musical principles (see, e.g., Vilmar 2020: 290–293; Barska 2014; Zimmermann 2002; Ordway 2007; Wolf 1999: 125–146). Virginia Woolf’s The Waves (1931) is another example of the wide variety of different interpretation of how the intermedial relationship to music is best conceived (Crapoulet 2009: 80). Many musicalized novels describe events that connect them to a narrative in a kind of double exposure, where the intermedial reference to music indicates that the reader should understand it in a different manner. For instance, in Günter Grass’s novella Crabwalk (2002) the narrator tries to understand how and why his 17-yearold son Konny became a Neo-Nazi and a murderer. At the same time, the plot is full of repetitive patterns and variations in multiple voices and inversions that form a parallel to patterns of contrapuntal composition and a connection to the retrograde and the so-called crab canon (Schirrmacher 2012: 209–242). Exploring these formal imitations, the challenge becomes how to analyze music formed out of words, events, literary motifs, and the organization of multiple meanings that unfolds between the lines of a text and in the discursive ordering of events.

Metaphorical Frame and Transmedial Common Ground These structural parallels involve a paradox. A first impulse often is to focus on the represented medium, looking for how the musical form of the fugue, the sonata form, or jazz techniques are adapted in the literary text. However, analyzing the structural parallels as an adaptation process between the source and target medium might result in a discussion of literature’s fidelity to musical form. This analysis can also lead to the search for formal analogies, which cultural approaches have criticized as reductive. The relationship between musical and literary forms is more complex and multilayered.

850

B. Schirrmacher

One way to approach this paradox has been to highlight the metaphorical character of intermedial references that point toward structural parallels. Intermedial relationships and metaphors have in common that they both draw attention to similarities between objects that are different on many other levels. The countless musical titles of fiction and poetry (Stawiarski), such as Nancy Huston’s Goldberg Variations; paratextual comments, like when James Joyce refers to the fugue as formal principle for the “Sirens” chapter; or discussions of music in the narrative plot, like when protagonist Philip Quarles in Aldous Huxley’s Point Counter Point reflects on how to structure a novel similar to J.S. Bach’s “Art of the Fugue” (Huxley 1954 [1928], 408), guide the meaning-making process toward these similarities. However, the literary structure cannot be analyzed by identifying the elements of this musical form alone. Using musical terms to analyze the structural parallels also creates problems because they use the metaphor that needs to be analyzed, leading to a vague and impressionist metaphorical use of these terms. When literary analysis tries to adapt a formal analysis of musical form and tries to find the exact correspondences of, for example, the voices of a fugue, key modulations, countersubjects, or codas analysis sooner or later will land on the differences between the text and the musical form. When Eric Prieto explores the role of music in modernist fiction, he stresses this metaphorical perspective that highlights musical form not as the analytical result but as a model, a “starting point for a reflection on the ways that a novel can use repetition and variation to make meaning,” and a structural principle rather than trying to identify “a certain number of fixed traits” (Prieto 2002: 73–74). Metaphorical relationships involve a tension Axel Englund describes as “a conflict between identification and differentiation” (Englund 2012: 15) when he explores the relationship between the role of music in Paul Celan’s poems and how they have been set to music. From a dialogical and metaphorical perspective, the reference to musical terms draws attention to transmedial aspects that connect differently structured phenomena, such as musical polyphony, the Bakhtinian polyphony of a novel, and verbal ambiguity. Erik Redling approaches these transmedial deep-level similarities as “conceptual metaphors” that help to analyze the structural parallels to jazz music in twentieth-century poetry (Redling 2015). Approaching structural parallels from an intertextual perspective highlights the dialogical two-way relationship between music and literature (Höfer 2017; Hejmej 2014; see also Klein 2020). Asking the question how musical polyphony is “imitated” or “simulated” in a literary text might therefore end up putting too much focus on how two different verbal utterances cannot be heard and understood simultaneously in the same way, like the different voices in polyphonic musical composition. Looking for “formal analogies” (Prieto 2002) or “structural parallels” (Sichelstiel 2004) highlights instead how verbal language and narratives have their own means to present different but related voices and structure them in a way that offers multiple perspectives. This is done by exploiting verbal ambiguity, as in Elfriede Jelinek’s prose that resonates with ambiguous meaning potential, or when different characters or narrative strands are perceived as interrelated, as with characters in Huxley’s Point Counter Point or in Grass’s Crabwalk, who at different times and places are confronted with similar questions or

33

Music Transformation in Literature

851

perform similar acts in different ways. The perspective of parallels highlights instead how this simulating effect is achieved. The complex interconnection between how the metaphorical musical frame and the transmedial narrative and musical aspects transform the literary text can be traced in multiple steps. What kind of music is represented and what kinds of ideas, practices, and formal conventions qualify this kind of music? How is music represented, and which transmedial parallels are highlighted? Why is literature transformed in this particular way? What is the narrative and literary effect of the representation of music and the transmedial similarities (Bruhn et al. 2021: 186–188)? An analysis of structural parallels can explore how the explicit representation of a musical genre highlights the significance of transmedial similarities such as repetition, voice, or performance in a specific literary text and create a specific literary form that is used to transmediate ideas, stage conflicts, create ambiguity, and stress the performativity of language.

Repetition and Contrast Repetition and contrast are fundamental structuring principles in music. Language, too, is organized repetitively on the level of sounds, letters, words, and syntax. However, these kinds of repetition are usually not perceived as prominent, as they are usually not central in the process of meaning-making. Even if context matters, the meaning of a word is primarily understood through habit and convention and less by a word’s position in relation to other words. Thus, when verbal communication adopts repetition and contrast as its primary structural principles, it deviates from more common literary and verbal meaning-making principles. The focus on the repetition of sounds and phrases (and events), which Scher described as “word music,” is a language use that often includes a sense of dereferentialization (Wolf 2017: 106). One extreme example is “Sonate in Urlauten” (‘Sonata of primordial sounds’) by the Dada artist Kurt Schwitters (1922–1932). Phrases like “Fümms bö wö tää zää Uu, pögiff, kwii Ee” do not form conventional words. Instead, the sequence of sounds is organized by repetition and contrast, based on contiguity and following rules that are similar to the sonata form (Wolf 2017: 104–105; Lichtenstein 2003). Schwitters’s work highlights the meaning-making potential of non-lexical elements in language, of prosody, and (when performed) of body language, elements that are intrinsic in language but that intermedial references to music draw attention to. When a narrative explicitly refers to musical formalized patterns of repetition and context, such as the sonata form or the ABA repetition pattern, the structural parallels also build on transmedial similarities between narrative and musical structure. The sonata form, the exposition of two musical subjects, and their development and recapitulation bear transmedial similarities with the verbal representation of conflict and dialogue. The ternary structure of the ABA repetition pattern in music displays similarities to fundamental principles of developing a narrative or dramatic conflict through initial balance, imbalance, and restoration of balance. The highlighting of a deep-level form therefore has a metafictional effect. By structuring the plot

852

B. Schirrmacher

according to patterns of repetition and contrast, a narrator not only indicates that “this plot works like music” but stresses at the same time that “this is writing, this is a novel, and I control it” (Shockley 2009: 15). The structural parallels to musical patterns of repetition also rely on the transmedial similarities with the repetition of literary motifs and themes. Literary motifs and themes usually provide a background to the linear and causal connections of specific events in a narrative plot. When a narrative plot forms repetitive patterns of repeated, similar, and contrasting events, linear and causal connections between the events are pushed to the background, a move that undermines or, as Wolf puts it, “negates” narrative development (Wolf 2017: 108). This does not necessarily mean that the narrative is perceived as “dereferentialized” or deviating from narrative practice, Höfer (2017: 3) points out. Even in many musicalized novels, individual events still follow upon each other in a plot that consists of causes and effects. The plot of Anthony Burgess’s novel A Clockwork Orange presents how teenage hooligan Alex, leader of a violent gang, gets into prison, how he after aversion therapy to violence becomes the victim of the violence of others, how he is restored to his former violent self, and how he finally leaves violence behind as he grows up. As the novel is based on the sonata form (Philipps 2010: 88–89), the linear development of a coming-of-age story is less prominent. The ternary ABA structure highlights instead Alex’s opposed yet related roles as perpetrator and victim. Burgess uses the conflicting relationship of violence and music to demonstrate or to perform the conflicting interdependent relationship between individual freedom and social control. The leitmotif is another structural parallel in musicalized texts. The musical leitmotif is a theme that is clearly recognizable, repeated, and used to represent persons, ideas, or objects from a dramatic plot. It was first more widely used to describe the connection between music and dramatic conflict in Wagner’s operas. When used in literature, leitmotif refers to the repetition of words and phrases on the surface level, used to performatively connect and therefore indicate similarities between different scenes or characters. Leitmotifs are not literary motifs that clearly connect to the literary thematic development. Instead the verbal repetition on the surface level draws attention to the repetition of literary themes and motifs on a deep, structural level. For instance, the repeated opening phrase in Burgess’s A Clockwork Orange “What’s it going to be then, eh?” indicates the inverted relatedness between Alex’s life as hooligan, as prisoner, and as discharged.

Polyphony and Multivoicedness While the sonata form bears similarities to narrative patterns of conflict, contrapuntal composition forms like the fugue resemble conversations, discourse, and, thus, fundamental rhetorical structures (Shockley 2009: 15). The counterpoint describes a “combination of simultaneously sounding musical lines according to a system of rules” that relates melodic relationships and harmonic intervals to each other (Sachs and Dahlhaus 2001). These relationships are embedded in a dialogic structure of

33

Music Transformation in Literature

853

question and response, assertion and contradiction that weave together several hierarchically structured levels. Sequences vary and imitate, replace and interrupt each other, resulting in ambiguous oppositions and mirroring inversions. Whenever literary narration and poetry draw on contrapuntal composition, such as the fugue or the canon, they draw on the tension between strictness and creativity, the precision of compository rules, and the complexity of unfolding meaning. Writers draw on contrapuntal principles to organize complex spatiotemporal relationships between narrative strands. The semantic dimension of the term counterpoint and its Latin origin, punctum contra punctum (“note against note”), may be the reason why the ideas of opposition, contradiction, and objection are more important in structural parallels to the counterpoint than they are in musical composition (Honold 2017). Intermedial references to the counterpoint highlight transmedial similarities between musical composition and the literary forms of multivoicedness in dramatic and narrative dialogue. The polyphony of the novel that Bakhtin had in mind takes place in the plurality of contrasting voices. While one cannot understand two verbal sentences in the same way as the simultaneous contrasted voices of polyphonic musical composition, there are other ways to express several verbal meanings simultaneously, such as heightening the semantic ambiguity of words. In James Joyce’s Ulysses and Finnegans Wake, the Nadsat slang in Burgess’s A Clockwork Orange, or Elfriede Jelinek’s prose (Solibakke 2014; Powell and Bethman 2008), there is a constant play and tension between the sound of words and multiple meanings. Structural parallels to musical polyphony thus highlight the verbal capacity for simultaneous multivoicedness and complex relationships within a novel. While diegetic representation of music highlights language as an auditory experience, structural parallels between literature and music draw on patterns of repetition and contrast as a foundation for meaning-making. They undermine causal links between events, a central element in narration, in favor of contiguity, and they bring forth the semantic ambiguity of language and the multi-perspectivity of dialogue. Taken together, the structural parallels to music present literature as a performed practice.

Performance and Transgression The focus on representing performed music also increases different performative aspects in literature and language. Musical titles and paratexts, Stawiarski argues, invite the reader “to consider the reading experience [. . .as] at least partly related to the condition of performativity” (Stawiarski 2010: 105). Delia da Sousa Correa points out a connection between the description of musical performance and a sense of literary performativity, which is characteristic of the writings of, for instance, Katherine Mansfield (da Sousa Correa 2011). Hazel Smith points out how intermedial relationships between words and music often connect cross-cultural exchanges (Smith 2016). The contingent, ambiguous, and performative characteristics of music transformation lend themselves to expressing that which resists

854

B. Schirrmacher

narrative and causal explanations, such as violence and trauma, and to transgressing spatial and normative borders. These transgressive and performative aspects are specifically discussed in postcolonial and migratory contexts. Instead digital media highlight auditory aspects of literary performance that transcend the written text.

Transgressing Borders and Identities The ability of music to transgress spatial and normative borders, the ambiguities and the transgressive potential of music also play an important role in postcolonial writing. Edward Said presents the musical concepts of elaborations and counterpoint as a model for postcolonial reading and writing practices that respond to and subvert Western hegemonies (Said 1991). Postcolonial narratives highlight the transgressive potential of music both to state and to subvert conflicting categories. A contrapuntal presentation of interrelated opposites gives space to marginalized voices in a colonial or global context. Cameron Bushnell (2013) draws attention to how novels such as J. M. Coetzee’s Disgrace or Chang-rae Lee’s A Gesture Life engage with Western art music as an intrinsic part of the hegemonic culture but also use it in oppositional ways. Christin Hoene demonstrates further how British South Asian writers such as Salman Rushdie, Amit Chaudhuri, and Vikram Seth use music to express and transgress ideologies (Hoene 2015). Julie Huntingdon explores the role of sounding phenomena in West African and Caribbean francophone literature. Huntingdon demonstrates how writers such as the Senegalese writer Ousmane Sembène and Martinican Patrick Chamoiseau use music, rhythm, and sound to subvert and transgress and geographical, language, and social borders to create spaces of identity (Huntingdon 2009: 50). In African American contexts, the migratory qualities of music mediate between orality and literacy, between noises, repetitions, and silences in music and languages, for instance in the writings of James Weldon Johnson (Morrissette 2013). Anna Snaith demonstrates how the writing of Jean Rhys explores the sonic dimension of colonial identities and develops an aesthetics of noise, where sounds are disruptive and unsettling (Snaith 2020). The transgressive role of music representation in postcolonial novels connects to the role of music in migration. Research in cultural studies draws attention to how the migratory quality of music that crosses cultural and geographical borders presents music as a mode of translation and mediation between global and local contexts (Fraser 2018; Jaji 2014; Dueck and Toynbee 2011).

Beyond the Written Text: Literature Performance and Multimodality Contemporary writing and reading practices go beyond the visually perceived printed texts. Contemporary reading practices are increasingly listening practices. Digital possibilities of integrating the sounds of words, music, and sounding literary experiences confirm the auditory dimensions of literature that music transformation

33

Music Transformation in Literature

855

always has highlighted. Reading practices that could be described as “reaplaying” and “reastening” are no longer the intermedial effect of musical titles (Stawiarski 2010: 105–107) but everyday practice when engaging with audiobooks (Have and Pedersen 2020) or “soundtracked fiction” (St. Clair 2020). Research on music transformation in literature today can engage with new digital genres and a more diverse interplay between verbal and musical sound in written, performed, and multimodally designed literary experiences. Digital technologies change the music-literature relationship, as works of digital literature can involve silent words on screens as well as spoken words (Smith 2016: 165–189; ▶ Chap. 43, “Interactive and Participatory Sound”). The growing field of audionarratology explores the narrative potential of sounds further (Mildorf and Kinzel 2016). Performance is central not only for musicians but for many literary artists as well. The auditory experience of words plays a significant role in poetry readings, particularly for performing poets like Allen Ginsberg, Ntozake Shange, and Laurie Anderson (Pfeiler 2003; see also Hillebrandt 2017). The concept of song writing highlights how Bertolt Brecht’s early poems are designed to be set to music (Lucchesi 2017). Lyrics have long been neglected by literary studies and musicology alike. An intermedial perspective to lyrics can highlight how the interplay of musical and verbal sounds contributes to meaning-making (Wierød Borčak 2017) and demonstrates how the “arrangement of words as an art form” (Griffiths 2020: 664) takes place within the conventional literary sphere. The Nobel Prize of Literature award to Bob Dylan highlights how popular and literary traditions inform the work of many popular music artists (Otiono and Toth 2019; Gray 2000). Singer-songwriters like Dylan, Leonard Cohen, Nick Cave, Lou Reed, and Patti Smith create literary experiences with their songs (Carroll and Hansen 2016: 33). The lyrics of rap are places of many border zones; they oscillate between the melody of speech and song, between literacy, orality, performance, and social practice (Hörner 2017).

Conclusion The growing possibilities for sounding literary experiences do not mean that music transformation in literature loses its purpose. On the contrary, this recent development only stresses how the music transformation in literature does not primarily aspire to transform a text into music. Instead, the intermedial relationship with music transforms literary narration and expression and expands the understanding of what kind of experiences language and written texts can convey. Music transformation performs paradoxical relationships and connects the sounds and silences of words or the causal and contingent explanations of events. The transgressive potential of music is used not to explain but to express and perform complex relationships. The performance of music in the diegesis and structure of the text create responses and expanded modes of reception. Specifically, structural parallels to music can convey an understanding of contradictory entanglements that cannot be sufficiently grasped by tracing a line of cause and effect. The transgressive yet interconnecting

856

B. Schirrmacher

potential of music is used to express conflicting yet interrelated relationships across borders, identities, and time. Therefore, the comparison of literary music transformations with sounding performed music can only be the first step in exploring how the representation of musical sounds and structural analogies highlights specific musical modes, ideologies, and associations and contributes to the development of literary themes in a specific literary and social context. The transgressive and performative potentials of music transformation in literature offer ways to express and explore the contemporary entanglement of ecological, political, and economic crises as well. In a globalized and digitized society, geographic borders and the materiality of media seem to disappear but, at the same time, create conflicts. Ecocritical research draws attention to how music, sound, and noise have the potential to express the conflicts of ecological crises and, at the same time, transgress the nature/culture divide (Hart 2018; Hart and Schirrmacher 2022). The way written and multimodal literary texts explore verbal performance and the way literary narratives explore the paradoxical relation between digital and embodied presence offer themselves to be explored by intermedial writers and researchers.

References Acquisto, Joseph. 2017. Proust, music, and meaning. Theories and practices of listening in the Recherche. London: Palgrave Macmillan. Albright, Daniel. 2000. Untwisting the serpent: Modernism in music, literature, and other arts. Chicago: University of Chicago Press. Albright, Daniel. 2004. Modernism and Music: An Anthology of Sources. Chicago: University of Chicago Press. Arvidson, Jens. 2009. “To stich the cut. . .”: The record sleeve and its intermedial status. In Changing borders. Contemporary positions in intermediality, ed. Jens Arvidson et al., 55–84. Lund: Lund University Press. Bal, Mieke. 2002. Travelling concepts in the humanities: A rough guide. Toronto: University of Toronto Press. Barska, Johanna. 2014. “New story from old words”: Word and music studies and the poetics of experience. In Ideology in words and music, ed. Beate Schirrmacher et al., 31–48. Barthes, Roland. 1970. S/Z. Paris: Seuil. Ben-Horin, Michal. 2016. Musical biographies. The music of memory in post-1945 German literature. Berlin/Boston: De Gruyter. Benson, Stephen. 2006. Literary music: Writing music in contemporary fiction. Aldershot: Ashgate. Borčak, Lea Wierød. 2017. The sound of nonsense: On the function of nonsense words in pop songs. SoundEffects 7: 27–43. Bowen, Zack. 1975. Musical allusions in the works of James Joyce: Early poetry through Ulysses. Dublin: Gill and Macmillan. ———. 1993. Music as comedy in Ulysses. In Picking up airs: Hearing the music in Joyce’s text, ed. Ruth Bauerle, 31–52. Urbana, IL: University of Illinois Press. Breatnach, Mary. 2020. Music in proust. The evolution of an idea. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 533–544. Edinburgh: Edinburgh University Press.

33

Music Transformation in Literature

857

Brown, Calvin S. 1987 [1948]. Music and literature. A comparison of the arts. Hanover: University Press of New England. Bruhn, Jørgen. 2016. Between punk and PowerPoint: Authenticity versus medialities in Jennifer Egan’s a visit from the goon squad. In The intermediality of narrative literature, ed. Jørgen Bruhn, 103–212. London: Palgrave Macmillan. Bruhn, Jørgen, Liviu Lutas, Niklas Salmose, and Beate Schirrmacher. 2021. Media representation. Film, music and painting in literature. In Intermedial studies: An introduction to meaning across media, ed. Jørgen Bruhn and Beate Schirrmacher, 162–193. London: Routledge. Bushnell, Cameron F. 2013. Postcolonial readings of music in world literature turning empire on its ear. New York: Routledge. Caers, Vincent. 2022. Umbraphonics. An electro-acoustic performance format for mediating musical scores. KU Leuwen. Luca School of arts: Leuwen. Carroll, Rachel, and Adam Hansen. 2016. Litpop: Writing and popular music. London and New York: Routledge. Chapin, Keith Moore, and Andrew Herrick Clark, eds. 2013. Speaking of music addressing the sonorous. New York: Fordham University Press. Cleghorn, Angus. 2019. Elizabeth Bishop and the music of literature. London: Palgrave Pivot. Clements, Elicia. 2019. Virginia Woolf: Music, sound, language. Toronto: University of Toronto Press. Clüver, Claus. 1999. The Musikgedicht: Notes on an ekphrastic genre. In Defining the field. Proceedings of the first international conference on word and music studies at Graz, 1997, ed. Walter Bernhart, Paul Steven, and Werner Wolf, 187–204. Amsterdam: Rodopi. Crapoulet, Emilie. 2009. Voicing the music in literature. European Journal of English Studies 13: 79–91. https://doi.org/10.1080/13825570802708212. da Sousa Correa, Delia. 2011. Musical performativity in the fiction of Katherine Mansfield. In Word and music studies: Essays on performativity and on surveying the field, ed. Walter Bernhart, 71–86. Amsterdam: Rodopi. ———, ed. 2020. The Edinburgh companion to literature and music. Edinburgh: Edinburgh University Press. Dahlhaus, Carl. 1989 [1978]. The idea of absolute music. [Die Idee der absoluten Musik]. Trans. Roger Lustig. Chicago: University of Chicago Press, 1989. Dame, Natalia. 2014. The search for narrative control: Music and female sexuality in Tolstoy’s “Family Happiness” and “The Kreutzer Sonata”. Ulbandus Review 16: 158–176. https://www. jstor.org/stable/24391989. Dautel, Katrin. 2017. The sound of a new generation? Intermedial references to pop music in Judith Hermann’s prose. Forum for Modern Language Studies 53: 163–178. https://doi.org/10.1093/ fmls/cqw089. Dayan, Peter. 2006. Music writing literature, from sand via Debussy to Derrida. Aldershot: Ashgate. ———. 2010. Art as music, music as poetry, poetry as art, from Whistler to Stravinsky and beyond. Farnham, Surrey: Ashgate. DeCristo, Jeramy. 2018. Music against the subject. Theory & Event 21: 169-190. muse.jhu.edu/ article/685975. Deeming, Helen. 2020. Music and the book: The textualisation of music and the musicalisation of text. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 48–64. Edinburgh: Edinburgh University Press. Deutsch, David. 2015. British literature and classical music: Cultural contexts 1870–1945. Historicizing modernism. London: Bloomsbury Academic. Dueck, Byron, and Jason Toynbee, eds. 2011. Migrating music. Culture, economy and the social. London/New York: Routledge. Edgecombe, Rodney S. 1993. Melophrasis: Defining a distinctive genre of literature/music dialogue. Mosaic: A Journal for the Interdisciplinary Study of Literature 26: 1.

858

B. Schirrmacher

Elleström, Lars. 2014. Media transformation: The transfer of media characteristics among media. Basingstoke: Palgrave Macmillan. ———. 2021. The modalities of media II: An expanded model for understanding intermedial relations. In Beyond media borders. Intermedial relations among multimodal media, ed. Lars Elleström, 3–91. Basingstoke/New York: Palgrave Macmillan. Englund, Axel. 2012. Still songs: Music in and around the poetry of Paul Celan. Aldershot: Ashgate. Euchner, Maria. 2005. Life, longing, Liebestod: Richard Wagner in Thomas Mann’s “Tristan” and “Tod in Venedig”. The Germanic Review 80: 187–212. Fast, Susan, and Kip Pegley. 2012. Music, politics, and violence. Middletown: Wesleyan University Press. Feinstein, Sascha, and David Rife. 2009. The jazz fiction anthology. Bloomington/Indianapolis: Indiana University Press. Fetzer, John Francis. 1990. Music, love, death, and Mann’s Doctor Faustus. Columbia: Camden House. Fornäs, Johan. 2010. Exclusion, polarization, hybridization, assimilation: Otherness and modernity in the Swedish Jazz Age. Popular Music and Society 33: 219–236. Fraser, Robert. 2018. Literature, music and cosmopolitanism culture as migration. Palgrave Macmillan. Frattarola, Angela. 2018. Modernist soundscapes: Auditory technology and the novel. Gainesville: University Press of Florida. Gauthier, Tim. 2014. Rock music as cosmopolitan touchstone in Salman Rushdie’s The Ground Beneath Her Feet. In Write in tune: Contemporary music in fiction, ed. Erich Hertz and Jeffery Roessner, 155–168. New York: Bloomsbury. Gess, Nicola. 2006. Gewalt der Musik: Literatur und Musikkritik um 1800. Freiburg i. Br: Rombach. Gess, Nicola, and Alexander Honold, eds. 2017. Handbuch Literatur & Musik. Berlin/Boston: De Gruyter. Glotova, Elena. 2021. Soundscapes in Nineteenth-century Gothic Short Stories. Goebel, Ralf J. 2020. The soul of the phonograph: Media-technologies, auditory experience, and literary modernism in the age of COVID-19. Humanities 82: 1–10. https://doi.org/10.3390/ h9030082. Grandt, Jürgen E. 2004. Kinds of blue. The Jazz aesthetic in African American narrative. Columbus: Ohio State University Press. Gray, Michael. 2000. Song & dance man III: The art of Bob Dylan. London/New York: Continuum. Griffiths, Dai. 2020. Words in popular songs. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 664–672. Halliday, Sam. 2013. Sonic modernity. Representing sound in literature, culture and the arts. Edinburgh: Edinburgh University Press. ———. 2020. “High fidelity”, “added value” and the aesthetics of sound technology in literary modernism. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 655–663. Edinburgh: Edinburgh University Press. Hart, Heidi. 2018. Music and the environment in dystopian narrative: Sounding the disaster. Cham: Palgrave Macmillan. Hart, Heidi, and Beate Schirrmacher. 2022. Music, noise, and nature: Energetic ambiguities in Benedikt Erlingsson’s woman at war. Music and the Moving Image 15: 3–19. Have, Iben, and Birgitte Stougaard Pedersen. 2020. Reading audiobooks. In Beyond media borders: Intermedial relations among multimodal media, ed. Lars Elleström, vol. 1, 197–216. Cham: Palgrave Macmillan. Hejmej, Andrzej. 2014. Music in literature. Perspective of interdisciplinary comparative literature. Trans. Lindsay Davidson. Frankfurt am Main: Peter Lang. Hertz, Erich, and Jeffrey Roessner, eds. 2014. Write in tune: Contemporary music in fiction. New York: Bloomsbury.

33

Music Transformation in Literature

859

Hindrichs, Gunnar. 2017. Sprache und Musik. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 19–38. Berlin: De Gruyter. Hillebrandt, Claudia. 2017. Lyrik als akustische Kunst. In Handbuch Musik & Literatur, eds. Nicola Gess and Alexander Honold, 338-350. Berlin/Boston: De Gruyter. Hoene, Christin. 2015. Music and identity in postcolonial British South-Asian Literature. New York: Routledge. ———. 2020. Music in contemporary fiction. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 577–587. Edinburgh: Edinburgh University Press. Höfer, Hannes. 2017. Jazzmusik in der deutschen Literatur. Ein Beispiel für intertextuelle Steuerung von Intermedialität. Textpraxis 13: 1–24. https://doi.org/10.17879/22259709173. ———. 2018. Jazz als literarisches Symbol der (Unterhaltungs-)Kultur in der Weimarer Republik. The German Quarterly 91: 447–459. Honold, Alexander. 2017. Kontrapunkt. Zur Geschichte musikalischer und literarischer Stimmführung bis in die Gegenwart. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 508–535. Berlin: De Gruyter. Hörner, Fernand. 2017. Rap, orale Dichtung und Flow. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 566–577. Berlin/Boston: De Gruyter. Höying, Peter. 2011. Ambiguities of violence in Beethoven’s Ninth through the eyes of Stanley Kubricks’s A Clockwork Orange. The German Quarterly 84: 159–176. https://www.jstor.org/ stable/41237071. Huber, Martin. 1992. Text und Musik. Musikalische Zeichen im narrativen und ideologischen Funktionszusammenhang ausgewählter Erzähltexte des 20. Jahrhunderts. Frankfurt/ Main: Lang. Huntingdon, Julie. 2009. Sounding off. Rhythm, music, and identity in West African and Caribbean francophone novels. Philadelphia: Temple University Press. Huxley, Aldous. 1954 [1928]. Point Counter Point. London: Chatto & Windus. Jaji, Tsitsi Ella. 2014. Africa in stereo: Modernism, music, and pan-African solidarity. New York: Oxford University Press. Jeannin, Marc. 2009. Anthony Burgess: Music in literature and literature in music. Newcastle: Cambridge Scholars. Jensen, Signe Kjær, and Martin Knust. 2022. Media and Modalities: Music. In Intermedial Studies: An Introduction to Meaning across Media, eds. Jørgen Bruhn and Beate Schirrmacher, 56–69. London: Routledge. Johnson, Bruce, and Martin Cloonan. 2009. Dark side of the tune. Popular music and violence. Aldershot: Ashgate. Kelley, Joyce E. 2010. Virginia Woolf and music. In Edinburgh companion to Virginia Woolf and the arts, ed. Maggie Humm, 417–428. Edinburgh: Edinburgh University Press. Keskinen, Mikko. 2005. Single, long-playing, and compilation: The formats of audio and amorousness in Nick Hornby’s High Fidelity. Critique: Studies in Contemporary Fiction 47: 3–21. https://doi.org/10.3200/CRIT.47.1.3-22. Klein, Michael L. 2020. Intertextuality, topic theory and the open text. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 13–24. Edinburgh: Edinburgh University Press. Kramer, Lawrence. 1984. Music and poetry: The nineteenth century and after. Berkeley: University of California Press. ———. 2002. Musical meaning toward a critical history, 2002. Berkeley/London: University of California Press. ———. 2005. Speaking melody, melodic speech. In Word and music studies: Essays on music and the spoken word and on surveying the field, ed. Suzanne M. Lodato and David Francis Urrows, 127–143. Amsterdam/New York: Rodopi. ———. 2011. Interpreting music. Berkeley: University of California Press. Kramer, Lawarence. 2017. Song acts. Writings on words and music. Leiden/New York: Brill Rodopi.

860

B. Schirrmacher

Kramer, Lawrence. 2020. Music and the rise of narrative. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 395–404. Edinburgh: Edinburgh University Press. Krick-Aigner, Kirsten, and Marc-Oliver Schuster, eds. 2013. Jazz in German-language literature. Würzburg: Königshausen & Neumann. Krimmer, Elisabeth. 2014. Revelation in the public sphere: Religion and violence in Heinrich von Kleist’s “Die Heilige Cäcilie oder Die Gewalt der Musik (Eine Legende)”’. Seminar: A Journal of Germanic Studies 50: 295–313. Laws, Catherine. 2003. Beethoven’s haunting of Beckett’s ghost trio. In Drawing on Beckett. Portraits, performances and cultural contexts, ed. Linda Ben-Zvi, 197–213. Tel Aviv: Assaph Books. Leppert, Richard D. 1993. The sight of sound: Music, representation, and the history of the body. Berkeley: University of California Press. Lévi-Strauss, Claude. 2001[1978]. Myth and meaning. London: Routledge. Lichtenstein, Sabine K. 2003. Listening to Kurt Schwitters’ Ursonate. A Dadaistic-romantic transposition D’arts? Arcadia 38: 276–284. https://doi.org/10.1515/arca.38.2.276. Lindskog, Annika J. 2014. Silent modernism. Soundscapes and the unsayable in Richardson, Joyce and Wolf. Lund: Lund University. Lock, Graham, and David Murray, eds. 2009. Thriving on a riff. Jazz & blues influences in African American literature and film. New York: Oxford University Press. Lubkoll, Christine. 2017. Musik in Literatur: Telling. In Handbuch Literatur & Musik, eds. Nicola Gess and Alexander Honold, 78-94. Berlin/Boston: De Gruyter. Lucchesi, Joachim. 2017. Bertold Brecht und die Musik. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 482–494. Berlin/Boston: De Gruyter. May, Will. 2020. Music and contemporary poetry: Audience, apology and silence. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 616–624. Edinburgh: Edinburgh University Press. McGrath, John. 2017. Samuel Beckett, repetition and modern music. London: Routledge. McParland, Robert P. 2009. Music and literary modernism: Critical essays and comparative studies. Newcastle upon Tyne: Cambridge Scholars Publishing. Mildorf, Jarmila, and Till Kinzel. 2016. Audionarratology : Interfaces of sound and narrative. Berlin/Boston: De Gruyter. Morrison, Simon A. 2020. Dancefloor-driven literature. The rave scene in fiction. New York: Bloomsbury. Morrissette, Noelle. 2013. James Weldon Johnson’s modern soundscapes. Iowa City: University of Iowa Press. Naumann, Barbara. 2017. “Musikalisierung” von Literatur in der Romantik. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 374–386. Berlin/Boston: De Gruyter. Newark, Cormac. 2011. Opera in the novel from Balzac to Proust. Cambridge: Cambridge University Press. Odendahl, Johannes. 2008. Literarisches Musizieren: Wege des transfers von Musik in die Literatur bei Thomas Mann. Bielefeld: Aisthesis. Ordway, Scott J. 2007. A dominant Boylan: Music, meaning, and Sonata Form in the “Sirens” Episode of “Ulysses”. James Joyce Quarterly 45: 85–96. Otiono, Nduka, and Josh Toth. 2019. Polyvocal Bob Dylan: Music, performance, literature. Cham: Palgrave Macmillan. Pankow, Edgar. 2017. Musikphilosophie im 19. Jahrhundert. Unsagbarkeit und Sprache der Musik. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 419–435. Berlin/ Boston: De Gruyter. Pater, Walter. 1986 (1873). The School of Giorgione. In The renaissance, 83–98. Oxford: Oxford University Press. Paterson, Adrian. 2020. Modernist poetry and music: Pound notes. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 587–601. Edinburgh: Edinburgh University Press.

33

Music Transformation in Literature

861

Pedersen, Birgitte Stougaard. 2013. Aesthetic in hip rhythmic potentials skills, hop and music of and conventions, rhythm everyday culture: Life. In Off beat : Pluralizing rhythm, ed. Jan Hein Hoogstad and Birgitte Stougaard Pedersen, 55–70. Amsterdam: Rodopi. Petermann, Emily. 2014. The Musical Novel : Imitation of Musical Structure, Performance, and Reception in Contemporary Fiction. Rochester, New York: Camden House. Pfeiler, Martina. 2003. Sounds of poetry. Contemporary American performance poets. Tübingen: Narr. Phillips, Paul. 2010. A clockwork counterpoint. The music and literature of Anthony Burgess. Manchester University Press. Picker, John M. 2003. Victorian soundscapes. New York/Oxford: Oxford University Press. Powell, Larson, and Brenda Bethman. 2008. “One must have tradition in oneself, to hate it properly”: Elfriede Jelinek’s musicality. Journal of Modern Literature 32: 163–183. Previšić, Boris. 2017. Klanglichkeit und Textlichkeit von Musik und Literatur. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 39–54. Berlin/Boston: De Gruyter. Prieto, Eric. 2002. Listening in. Music, mind and the modernist narrative. Lincoln/London: University of Nebraska Press. ———. 2020. Becket, music and the ineffable. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 565–570. Edinburgh: Edinburgh University Press. Rabinowitz, Peter J. 2003. “A bird of like rarest spun Heavenmetal”: Music in a clockwork orange. In Stanley Kubrick’s a clockwork orange, ed. Stuart Y. McDougal, 109–130. Cambridge: Cambridge University Press. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités/Intermediality 6: 43–64. https://doi.org/10.7202/1005505ar. Rajewsky, Irina O. 2002. Intermedialität. Tübingen: Francke. Redling, Erik. 2015. The Musicalization of poetry. The relation between music and poetry. In The handbook of intermediality: Literature – image – sound – music, ed. Gabriele Rippl, vol. 1, 494–511. Berlin: De Gruyter. Reisberg, Daniel, ed. 1992. Auditory imagery. Hillsdale: Lawrence Erlbaum. Rife, David. 2008. Jazz fiction: A history and comprehensive reader’s guide. Lanham: Scarecrow. Sachs, Klaus-Jürgen, and Carl Dahlhaus. 2001. Counterpoint. Grove Music Online. https://doi-org. proxy.lnu.se/10.1093/gmo/9781561592630.article.06690. Accessed 23 Jan. 2022. Said, Edward W. 1991. Musical elaborations. London: Chatto & Windus. Scher, Steven P. 1968. Verbal music in German literature. New Haven: Yale University Press. ———. 1982. Literature and music. In Interrelations of literature, ed. J.P. Barricelli and J. Gibaldi, 225–250. New York: Modern Language Association of America. Schirrmacher, Beate. 2012. Musik in der Prosa von Günter Grass. Intermediale Bezüge – Transmediale Perspektiven. Stockholm: Acta Universitatis Stockholmiensis. ———. 2016a. Mute performances: Ekphrasis of music and performative aesthetics in Eyvind Johnson’s Romantisk berättelse. In Silence and absence in literature and music. Word and music studies 15, ed. Werner Wolf and Walter Bernhart, 102–116. Leiden/Boston: Brill Rodopi. ———. 2016b. Musical performance and textual performativity in Elfriede Jelinek’s the piano teacher. Word and music studies – new paths, new methods. Danish Musicology Online. Special Edition 2016: 93–108. ———. 2021. Singing words and sticky music. Fragmentary lyrics and acts of violence. In Arts of incompletion: Fragments, drafts, revisions in words and music, ed. Walter Bernhart and Axel Englund, 143–165. Leiden/Boston: Brill Rodopi. Schirrmacher, Beate, Hannah Hinz, Heidi Hart, and Kathy Heady, eds. 2014. Ideology in words and music. Stockholm: Acta Universitatis Stockholmiensis. Schlauraff, Kristie A. 2018. Victorian gothic soundscapes. Literature Compass 15: 11. Schweighauser, Philipp. 2008. The noises of modernist form: Dos Passos, Hurston, and the soundscapes of modernity. In American studies as media studies, ed. Frank Kelleter and Daniel Stein, 47–55. Heidelberg: Winter.

862

B. Schirrmacher

———. 2015. Literary Acoustics. In Handbook of intermediality. Literature – image – sound – music, ed. Gabriele Rippl, 475–493. Berlin: De Gruyter. Shockley, Alan. 2009. Music in the words. In Musical form and counterpoint in the twentieth century novel. Aldershot: Ashgate. Sichelstiel, Andreas. 2004. Musikalische Kompositionstechniken in der Literatur: Möglichkeiten der Intermedialitat und ihrer Funktion bei österreichischen Gegenwartsautoren. Essen: Die Blaue Eule. Skaggs, Carmen Trammell. 2010. Overtones of opera in American literature from Whitman to Wharton. Baton Rouge: Louisiana State University Press. Smith, Hazel. 2016. The contemporary literature–music relationship. New York: Routledge. Smyth, Gerry. 2008. Music in contemporary British fiction: Listening to the novel. New York: Palgrave Macmillan. Snaith, Anna. 2020. Jean Rhys and the politics of sound. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 570–576. Edinburgh: Edinburgh University Press. Solibakke, Karl Ivan. 2007. Musical discourse in Jelinek’s the piano teacher. In Elfriede Jelinek. Writing woman, nation, and identity, ed. Matthias Piccolruaz Konzett and Margarete LambFaffelberger, 250–269. Madison: Fairleigh Dickinson University Press. ———. 2014. Geseire, Geleiere und Gehübungen: Music and sports in Elfriede Jelinek’s Die Klavierspielerin and Winterreise. Austrian Studies 22: 106–120. Stawiarski, Marcin. 2010. This is all but a book: Musicalized paratextuality in literature. Neohelicon 37: 93–112. https://doi.org/10.1007/s11059-010-0054-9. StClair, Justin. 2020. Notes on soundtracked fiction: The past as future. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 672–680. Edinburgh: Edinburgh University Press. Strauss, Frithjof. 2003. Soundsinn: Jazzdiskurse in den skandinavischen Literaturen. Freiburg i. Br: Rombach. Sundmark, Björn. 2011. The sound and music of Astrid Lindgren. In Beyond Pippi Longstocking: Intermedial and international aspects of Astrid Lindgren’s works, ed. Astrid Surmatz, 201–218. New York: Routledge. Sutton, Emma. 2013. Virginia Woolf and classical music: Politics, aesthetics, form. Edinburgh: Edinburgh University Press. ———. 2020. Music in Woolf’s short fiction. In The Edinburgh companion to literature and music, ed. Delia da Sousa Correa, 544–552. Edinburgh: Edinburgh University Press. Tambling, Jeremy. 2012. Opera and Novel Ending Together: Die Meistersinger and Doktor Faustus. Forum for Modern Language Studies 48: 208–221. Tillmann, Markus. 2012. Populäre Musik und Pop-Literatur. In Zur Intermedialität literarischer und musikalischer Produktionsästhetik in der deutschsprachigen Gegenwartsliteratur. Bielefeld: Transcript. Varga, Adriana, ed. 2014. Virginia Woolf and music. Indiana University Press. http://www.jstor.org/ stable/j.ctt16gz62t. Vilmar, Therese Wiwe. 2020. It’s in the silence you feel you hear: Music, literature, and melophrasis. Orbis Litterarum: International Review of Literary Studies 75: 281–297. https:// doi.org/10.1111/oli.12276. Viol, Claus-Ulrich. 2006. Jukebooks: Contemporary British fiction, popular music, and cultural value. Heidelberg: Winter. Visvis, Vikki. 2008. Alternatives to the “talking cure”: Black music as traumatic testimony in Toni Morrison’s “Song of Solomon”. African American Review 42: 255–268. von Ammon, Frieder. 2017. Von Jazz und Rock/Pop zur Literatur. In Handbuch Literatur und Musik, ed. Nicola Gess and Alexander Honold, 535–545. Berlin/Boston: De Gruyter. Weaver, Jack W. 1998. Joyce’s music and noise. Theme and variation in his writings. Gainesville: University of Florida Press.

33

Music Transformation in Literature

863

Wolf, Werner. 1999. The Musicalization of fiction: A study in the theory and history of intermediality, 1999. Amsterdam und Atlanta: Rodopi. ———. 2008. Relations between literature and music in the context of a general typology of intermediality. In Comparative literature: Sharing knowledge for preserving cultural diversity. Encyclopedia of life support systems (EOLSS), ed. Lisa Block, Paola de Behar, Jean-Michel Dijan Mildonian, Djelal Kadir, Alfons Knauth, Dolores Romero Lopez, and Marcio Seligmann Silva, 133–155. Oxford: Eolss Publishers. http://www.eolss.net. ———. 2017. Musik in Literatur. Showing. In Handbuch Literatur & Musik, ed. Nicola Gess and Alexander Honold, 95–113. Berlin/Boston: De Gruyter. Young, Kevin, ed. 2003. Blues Poems. New York: Knopf. ———, ed. 2006. Jazz Poems. New York: Knopf. Zimmerman, Nadya. 2002. Musical form as narrator: The Fugue of the Sirens in James Joyce’s Ulysses. Journal of Modern Literature 26: 108.

Collage as a Creative Act: Emergence, Displacement, and Re-signification

34

Ma´rcia Arbex-Enrico

Contents Introduction: About Scissors and Glue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cubist, Futurist, and Dadaist Collage: Hybrid Forms with a Strategic Role . . . . . . . . . . . . . . . . . . Verbal and Literary Collage: A New Visual Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Surrealist Collage: The Conceptual Turn to a Poetic Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Collage and Intertextuality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion: Toward Hybrid Medial Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

866 870 879 886 891 893 895

Abstract

This chapter aims to present the notion of collage from its emergence in the avantgarde, starting with Cubism, Dadaism, and Futurism, when it played a strategic and subversive role, to surrealism when theorization about collage takes place and marks a conceptual turn: beyond technical aspects, the collage process was raised to the category of poetic device. This conceptual turn enables a greater use of collage throughout the twentieth century, particularly in the 1970s, when it was put in relation to intertextual practices, as citation, by sharing the principles of borrowing and of appropriation. Collage is here understood as a creative act, the gesture of cutting and pasting heterogeneous materials, verbal or iconographic fragments, displaced from their context and, therefore, resignified. Its repercussion is found not only in painting but also in the creation of unprecedented artistic and hybrid forms, made by media combination, such as the object-poem and the collage-novel. The relevance of recontextualizing collage in a diachronic perspective, with its initial potential, in short, may contribute to resignify it as an This work was carried out with the support of CNPq, National Council for Scientific and Technological Development – Brazil. M. Arbex-Enrico (*) Universidade Federal de Minas Gerais – UFMG/CNPq, Belo Horizonte, Brazil e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_37

865

866

M. Arbex-Enrico

intermedial phenomenon, as a form of cultural production marked by the crossing of borders between media, that highlights the materiality and interaction of the objects involved, with relevant implications for the study of contemporary arts and media. Keywords

Collage · Surrealism · Avant-garde · Intertextuality · Intermediality

Introduction: About Scissors and Glue “Si ce sont les plumes qui font le plumage, ce n’est pas la colle qui fait le collage” (If it is the feather that makes the plumage, it is not the glue that makes the collage). This is the formula Max Ernst uses to define collage in “Au-delà de la peinture” (1970: 256), extending its reach beyond the technical aspect. Along the twentieth century, this chameleon word unfolded from individual artistic practices but also as a result of the new technologies within reach of both artists and poets: from the ready-made, the assemblage, and the cut-up to the remix, the sample, and the mash-up, all those different ways of cutting and pasting that the abbreviations CTRL-V + CTRL-C synthetize so well. The word collage is also ambiguous in its etymology, designating both the gesture or action of pasting and the result of this action, embracing the visual as well as the verbal spheres. Pablo Picasso used to paste paper clippings, newspaper fragments, and bits of fabric onto his compositions, while Guillaume Apollinaire made literary collages from phrases and words heard here and there and then inserted into his poems. So, collage is likewise close to intertextual practices, such as citation, or to montage, by the handling of fragments or parts of heterogeneous elements. Thus we could begin by defining the word and briefly recapitulate the history of the practices which preceded the strategic use the artistic avant-garde gave this technique. According to the Centre National de Ressources Textuelles et Lexicales (CNRTL), the word collage (also collage, in French), dating from 1544, derives from the verb “to paste” (Fr. coller) and is defined, first, as “the action of gluing diverse things, or the result of this action” (CNRLT 2012). In this literal sense, it means “‘pasting, sticking, or gluing’, as in the application of wallpaper,” says Marjorie Perloff (1986: 46) in the chapter dedicated to “The invention of collage,” in The Futurist Moment: Avant-Garde, Avant Guerre, and the language of Rupture. And, in this literal and technical sense, the practice of collage was commonplace for centuries all over the world. By extension, the term designates “assemblage, juxtaposition of disparate elements,” when involving a narrative or the character of a personage, a definition which opens the possibility of using the word beyond its first technical aspect. But the third sense ascribed to this term is curious: it is synonymous with the status of a man and a woman who live together without being married (CNRTL 2012). The most eloquent instance of this popular connotation is found in Alphonse Allais’ short novel Collage (1891). This tale, of a markedly black humor, tells of a Dr. Snowdrop, who, upon

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

867

catching his young wife with her lover, takes revenge on the couple performing a delicate surgery to stitch together – literally – the bodies of the two lovers. Waking up from the long anesthesia, Bertha, his wife, is terrified when she sees herself thus united to George, to which Dr. Snowdrop retorts: “ – What are you complaining of, darling? (. . .) I have just made true your dearest wish: Being always with you and never leave you; to make of us a single being!. . . And, with a subtle smile, the doctor added: – This is what the French call collage” (Allais 1891) (Fig. 1).

Fig. 1 Alphonse Allais. Illustration for Collage, published in Gil Blas illustré, n 5, January 31, 1892. https://numerique.banq.qc.ca/patrimoine/details/52327/4075390

868

M. Arbex-Enrico

As a technique used in the field of the plastic arts, popular or decorative, JeanYves Bosseur, in his Le Collage: d’un art à l’autre, remarked that collage is found in practices that could be seen as precursors since the twelfth century, such as Japanese calligraphies on colored, torn, and juxtaposed papers, like the Ishiyama manuscript; the practice of lacca povera in the fifteenth century, which consisted in sticking on furniture patterns cut-out from prints which were then lacquered; the potichomanie, which consisted in gluing pieces of prints over objects such as screens, trays, or boxes; the so-called canivets of the sixteenth century; the woven patchworks; in the late seventeenth century, the playwright Charles Dufresny’s tableautins, pieces of paper and cloths glued on prints, which were called assemblage, perhaps for the first time; and, finally, the relief paintings made from small cards glued and painted in oil by Jean-François Capperon (Bosseur 2010: 15–16). The first instances of what will be called later photomontage appear in the nineteenth century with the English artists Oscar Gustave Rejlander and Henry Peach Robinson and the Parisian photographer Hippolyte Bayard, followed by Francis Bedford, Eugène Appert, and William Notman, whose technique consisted in cutting out photographic images and rearranging them in a new composition. Even Victor Hugo left us examples of collages in his graphic work, and Edgar Degas utilized actual materials, such as tulle skirts, to compose his “little ballerina.” Finally, among the numerous examples that could be mentioned at this turn of the nineteenth to the twentieth century, Paul Gauguin’s Noa Noa voyage journal, with collages made from photographs and scraps of prints, becomes a true forerunner of the contemporary artist’s book (Bosseur 2010: 15). As we can see, this technique is not new and has always been widely used, employing the most diverse materials. It is, however, in the early twentieth century that the practice takes on a “strategic role” (Bosseur 2010: 17) and becomes a “critical tactic” (Quyntin 2007), with enduring consequences for modern and contemporary art, besides innovative literary developments. Perloff stresses that “collage composition, as it developed simultaneously in France, Italy, and Russia (and slightly later in Germany and Anglo-America) is distinguished from the ‘paste-ups’ of the nineteenth century in that it always involves the transfer of materials from one context to another, even as the original context cannot be erased” (1986: 47). This historical moment is particularly significant in the sense of stressing the phenomenon of collage as a form of cultural production marked by hybridism, by crossing boundaries between genres and disciplines, which highlights the materiality and interaction between objects, in short, a practice that, by combining and mixing different elements, breaks with the usual concept of art and points to the concept of media. In 1966, Higgins posed the following question, at the end of his well-known essay “Intermedia”: “Is it possible to speak of the use of intermedia as a huge and inclusive movement of which dada, futurism and surrealism are early phases preceding the huge ground swell that is taking place now?” (2001: 32). Although Higgins announced this possibility, recent scholarship on intermediality theory is not focused on collage as an intermedial phenomenon in itself, as we verify within

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

869

the limits of the scope of this research. The term collage is mentioned by Claus Clüver when he quotes one of Karin Thomas’s meanings of “Mixed Media”: “Collective term for all forms of collage that expand the materials involved, e.g., assemblage, combine painting, environment” (Clüver 2008: 31). However, we agree that this definition seems very vague, so Clüver asks: “That makes installations and environments sub-categories of collage, but is collage a medium, like lithography?” We can ask, indeed, if collage is mixed media as are posters or comic strips (Clüver 2008: 26) or, for example, whether it can be considered “qualified media” that combine, to different degrees, “basic media” and “technical media” (Elleström 2010). Answering these questions would require in-depth studies; nevertheless, we may consider collage as a category of intermediality in the broader sense of “media combination” (Rajewsky 2005: 51) or, to use a different terminology, as a multimedia, mixed media, and intermedial phenomenon (Clüver 2007a: 1137). The relevance of recontextualizing collage with its initial potential, from its emergence in the avant-garde to the 1930s, can, in our opinion, contribute to further research in order to resignify it as an intermedial phenomenon, with relevant implications for the study of contemporary arts and media studies. The large range of collage practices and the notorious differences concerning their medial forms of articulation, of which we intend to give some examples, is a substantial issue. However, this chapter aims to draw attention to the process they have in common: the act of borrowing and appropriation, dislocation and transfer of materials from one context to another, and re-signification. That means that, from this point of view, there’s no difference of process between “pure collage” in a strict and technical sense and “conceptual collage.” We may distinguish indeed “pure collages” – when the borrowed fragment is “perfectly useless in the development of the novel” (Aragon 1980: 127), when a foreign body which causes discontinuity appears explicitly as heterogeneous – from “pseudo-collages,” “metaphorical,” or “conceptual collage,” a “collage effect” (Taylor 2005: 196) that “did not violate the medium directly” (Perloff 1986: 54), that stimulates this effect by other means, which are not material, polyphony, fragments of memory, automatisms, and citation, which leads us to the question of intertextuality. This distinction is not easily perceived or conceptualized, as we hope to demonstrate in the following pages. So, to approach collage in both the historical and theoretical perspectives, involving verbal and visual materiality, either in a strict sense – “pure collage” – or in a conceptual sense, “pseudo-collage,” we will begin by presenting it under the perspective of the cubist and futurist papier collé, the Dadaist’s photomontages, and other visual productions related to it. The second part of this chapter will concern verbal or literary collage. The third part will focus on the conceptual turn operated by the surrealists, when the process was raised to the category of “poetic device.” Finally, this conceptual turn enables a greater approach to collage along the twentieth century, particularly in the 1970s and 1980s, when it was put in relation with intertextual practices, as we intend to show in the fourth part of this chapter.

870

M. Arbex-Enrico

Cubist, Futurist, and Dadaist Collage: Hybrid Forms with a Strategic Role The first examples of collage in modern painting acknowledged by most art historians are Pablo Picasso’s Still Life with Chair Caning and George Braque’s Fruit Dish and Glass, of 1912, “although collage composition is prefigured in Cubist painting at least as early as 1908, when illusionistically painted nails, guitar strings, letters, and numbers were introduced into the otherwise nonrepresentational picture surface with its oscillating and ambiguously defined planes” (Perloff 1986: 46). In Still Life with Chair Caning,1 the artist used an actual rope to “frame” the oval canvas and inserted a piece of waxed fabric representing the woven straw motif, as if to evoke a chair from a Parisian bistro. This example is a good illustration of Picasso’s view “that different textures can enter into a composition to become the reality in the painting that competes with the reality in nature. We tried to get rid of ‘trompe l’œil’ to find a ‘trompe l’esprit’. (. . .) This displaced object has entered a universe for which it was not made and where it retains, in a measure, its strangeness” (Picasso apud Perloff 1986: 44). Braque, in turn, in his Fruit Dish and Glass,2 inserted into the canvas wallpaper clippings simulating wood veins, which could have originated the first papier collé, a terminology that began to be used for the cubist productions which resort mainly to paper cut-outs, in two dimensions, such as newspapers, wallpaper, business cards, stamps, or postcards, among other printed objects. Besides the aesthetic appearance created by the materiality of the new textures, it is “our view of reality” that is deeply changed, remarks Michel Butor, when the artist, instead of representing an object, inserts in his work a piece of this same object (Butor 2003: 34–35). In Les Peintres Cubistes, Apollinaire (1980: 76) says Picasso placed in evidence “authentic objects”: “a two-penny song, a real postage stamp, a piece of newspaper, and a piece of oilcloth imprinted with chair caning. The art of the painter could not add any pictorial element to the truth of these objects. It is impossible to foresee all the possibilities, all the tendencies of an art so profound and painstaking. The object, either real or in trompe-l’œil, will doubtlessly be called upon to play an increasingly important role” (Fig. 2). As a matter of fact, these initial experiments shattered the two-dimensionality of the canvas, moving into the “object-painting” or “relief painting,” the third dimension. The intrusion of fragments of reality in the pictorial space naturally distances the collage from the illusionary principle created by the trompe-l’œil and entails introducing some discontinuity and rupture in the work (Bosseur 2010: 22). This passage to the “object-painting” takes place in Guitar (1912),3 of which Louis Aragon writes in “La peinture au défi”: “The fact is that Picasso did a very serious thing. He took a dirty shirt and attached it to a canvas using a thread and

1

Picasso (1912c). Braque (1912). 3 Picasso (1912a) 2

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

871

Fig. 2 Pablo Picasso (1881–1973). Bouteille de Bass, 1914. Relief painting, oil, fusain, charcoal, pencil, wood, and tack on canvas glued on wood. Musée National Picasso-Paris. (Photo: Márcia Arbex-Enrico, Exposition Picasso-Picabia, Musée Granet, 2018)

needle. And as with Picasso everything ends in guitar, a guitar came up as example. (. . .) He wanted real human life remains, something poor, dirty, disdained” (Aragon 1980: 70). These leftovers generally came from the artist’s immediate context, from what was at hand, such as Picasso, who tore from the walls of his studio in Montparnasse the wallpaper he used in the collage. All of this contributes to the ephemeral, provisional aspect of the work. Aragon analyzes these early cubist collages and identifies two categories: the first includes those in which “the pasted element takes on meaning by the form or representation of the object,” e.g., a fragment cut out in the form of a guitar or a bottle; in the second category, the object becomes meaningful for its materiality, its color, and the “enrichment of the palette” (Aragon 1980: 47). In the latter case, the fragments coming from newspapers or ads contribute with the dynamism of their graphic texture and their typographic diversity. Several critics, however, became interested in the contents of the printed material itself, wondering whether the contents of the newspaper subject or the titles were a determinant factor in the choice. From the analysis of the contents of Still Life with Violin and Fruit (1913),4 Perloff notes: “If we take the trouble to read the newspaper fragments in Picasso’s collage, we discover that their selection is hardly random, for the newspaper ‘circle’ that acts as a kind of ground for the violin-woman shapes (. . .) contains references to most aspects of daily life in the Paris of 1912: sports, finance, sex, crime, religion (. . .), the new technology (. . .) and the latest medical discoveries. Politics, the main subject of newspaper reporting, is the one missing field (. . .)” (Perloff 1986: 50).

4

Picasso (1913).

872

M. Arbex-Enrico

In The Picasso Papers, Rosalind Krauss analyzes the printed subject matter and detects in it a continuous “circulation of signs.” On the one hand, Krauss asks herself whether Picasso, in the autumn of 1912 that saw the birth of collage, had turned the iconic sign, in its condition of similarity, into a “sign without the conventional motivation of language” (Krauss 2006: 46). On the other hand, she examines the signifiers that circulate from one fragment to another to try and identify, besides the texture function mentioned above, or the arbitrary sign value, the content of the news of the time published by newspapers; she asks herself if there had been any intention, ideological or other, in these choices. We bring up a single example among many presented by Krauss which points in that direction, the collage Bottle, Glass and Newspaper on a Table, of 1912, an example which evokes Mallarmé by the cut-out from Le Journal reading: “Un coup de thé,” which undoubtedly alludes to Un Coup de dés jamais n’abolira le hasard (Krauss 2006: 53). Owing to the date of the paper, December 4, 1912, the inclusion of the text might be also interpreted as a political allusion to the Balkan Wars (1912–1913)5 if the letters from “Un coup de thé(âtre)” are reconstituted, and not only as a poetic allusion. Besides the content of the newspaper text itself, it is important to consider the letters of the alphabet, often separated from the words by the cutting, or incomplete words, which contribute not only with iconicity but also to the semantic level. This is the case of the names – here reconstituted – of cities (Sorgues, Avignon, Céret, Nîmes), public places (café, restaurant, bar), and events (the corridas, concerts), and the inscriptions pointing to reading items (the French papers cited are many: L’Indépendant, Le Petit Parisien, Le Quotidien du Midi, etc.), music (instruments, scores, song titles), beverages (Vieux Marc), and labels (cigarette pack, matchbox). Although fragmented, these inscriptions help in the reading of the whole and of the context often related with events taken from the artists’ personal lives. There is also the sense of playfulness of the cut-outs, especially with the title Le Journal, which appears many times just as LE JOU ( jouer, jeu, jouir), opening up to the multiplicity of meanings (Arbex 1992: 37–53). Finally, the choice of papiers collés does not completely pass over the tense political situation of that historical moment. In L’ordre et l’aventure (1984), Pierre Daix reports a conversation with Picasso about that: “When I found out that in one of his first papiers collés, Bottle of Suze,6 Picasso had inserted newspaper bits with Jean Jaurès’ antiwar speech, I asked him whether that had been casual or deliberate. ‘But of course I did it on purpose! It was my way of saying I was against the war’.” In fact, both Picasso and Braque use the pasting of print material to play with several layers of meaning, associating word fragments sometimes with humor, sometimes with 5 The Balkan Wars (1912–1913) were two successive military conflicts that deprived the Ottoman Empire of all its remaining territory in Europe except part of Thrace and the city of Adrianople (Edirne). The second conflict erupted when the Balkan allies Serbia, Greece, and Bulgaria quarreled over the partitioning of their conquests. The result was a resumption of hostilities in 1913 between Bulgaria on the one hand and Serbia and Greece, which were joined by Romania, on the other. Available at https://www.britannica.com/topic/Balkan-Wars. Accessed on October 25, 2022. 6 Picasso (1912b).

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

873

ideological intentions, or both, inviting the spectator to uncover the meaning behind the general fragmentation of the canvas (Arbex 1992: 50). The futurists soon followed up the cubist model in the use of collage, mostly with different objectives, and within a short while, this procedure took on an essential role in the visual and verbal arts, although under different names: montage, construction, and assemblage, with a clear tendency to surpass the two dimension support. Starting in 1912, sculptors suggest introducing additional materials in their works, as Umberto Boccioni states: “The sculptor can use twenty different materials, or more, in a single work, as long as plastic feeling demands it” (Bosseur 2010: 24). A collage of glass, wood, cardboard, cement, leather, fabric, mirrors, electric light, etc. allows for the integration of the most heterogeneous, even if trivial, elements of the modern world. With iconoclastic and subversive intentions, their aim is to distance themselves as much as possible from the art meant for museums and to approach reality, as we can read in Marinetti’s first futurist Manifesto, published in 1909 in Le Figaro. Perloff emphasizes the role of Gino Severini in the use of collage and, more specifically, of the papier collé, as in Still Life with Fruit Bowl, of 1913, made in materials as diverse as colored paper, printed matter, and corrugated cardboard. Collage as practiced by the futurists at first made use of the methods and materials already employed by cubism but also conformed to the theme of the machine, of speed, of dynamism, and of nationalism and to self-advertising: “Either they did not violate the medium directly, but submitted it to a conceptual analysis that transformed it – that is to say, they made works that are collages not literally but metaphorically; or, true to their insistence that the spectator must be placed at the center of the picture, they reconceived collage as propaganda art, an art that directly bombards the senses” (Perloff 1986: 54). As an example of non-violation of the medium, Perloff presents Development of a Bottle in Space, by Boccioni (1912),7 and, as an instance of appeal to the viewer’s senses, Interventionist Manifesto, by Carlo Carrà (1914). The first is not a collage in the strictest sense of the term, but the bottle; although it “is made of one material – bronze – (. . .) it exploits the collage principle of juxtaposition of disparate items without any explanation of their connection” (Perloff 1986: 55). In the Interventionist Manifesto (Fig. 3), the influence of Marinetti’s concepts about the parole in libertà can be observed in the complex strategy of fusing word and image. In this collage, Carrà introduces printed items evoking the noise of a street demonstration mixed with words, onomatopoeias, and numbers, disposed in oblique lines and concentric circles, as if to give the impression of a crowd in movement, so “dizzying a spiral of inciting fragments, sounds, and colors, so shrill a torrent of spectacle and noise, that we can never perceive the picture as a coherent image” (Perloff 1986: 63), an image presented as a protest rally, consistent with a politically disturbed era. Different from the cubist collage, in which the articulation between forms is more structured, the futurist collage provokes, in this example, a

7

Boccioni (1912).

874

M. Arbex-Enrico

Fig. 3 Carlo Carrà. Interventionist Manifesto, 1914. https://www.wikiart.org/en/carlo-carra/ interventionist-demonstration-patriotic-holiday-freeword-painting-1914-1914. Public domain USA

verbal-visual hyperstimulation, calling to mind the idea of simultaneity of sensations and events, of interpenetration of planes. With the progressive abandonment of the still life motif and the application of the theory of the parole in libertà, collage turns into an art of propaganda: “an art that bombards the senses,” as Perloff puts it (1986). Ideological and political discourse fits into the printed material used in collages, which becomes the movement’s spokesperson. Thus, the Lacerba magazine, the movement’s divulgation vehicle, with the plastic quality of the red typography of its logo, is broadly used – cut out and pasted – as material for collages; the posters for the group’s demonstrations, the pamphlets, and the group’s publications themselves are also widely used as support for ideological content and as plastic material. Futurism is reflected, duplicated, and exalted in these collages (Arbex 1992: 126). One paragraph concerning the collage’s strategic role should be dedicated to the peculiar position of the German Kurt Schwitters. As a poet, he published the famous

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

875

Fig. 4 Kurt Schwitters, The And-Picture (Das Undbild), 1919. Centre Georges Pompidou. https:// www.wikiart.org/pt/kurt-schwitters/the-and-picture-1919. Public domain

and controversial Anna Blume (1919), comprising proverbs, songs, and citations. Just as in this poem, Schwitters gathered pieces of paper and other everyday materials for his assemblages Merz, as he prefers to call his collages: “(. . .) Essentially, the word Merz means an artistic assemblage of every material imaginable and, in principle, the equality of each of these materials at the technical level. The Merz painting thus makes use not only of color and canvas but also of all materials the eye can perceive and all tools which may be useful” (Schwitters 1990: 45) (Fig. 4). This is another technique for the appropriation of that which Louis Aragon calls “borrowed reality,” used thenceforth by all painters searching for new means of expression (Aragon 1980: 46). That “borrowed reality” may appear as objet trouvé, which is a privileged means for the insertion of chance and oneirism in surrealist poetry and art. But, at the time of the Dadaist movement and even some time before, the idea of objet trouvé appears under the form of ready-made, the name given by Marcel Duchamp to objects taken

876

M. Arbex-Enrico

from everyday life – bicycles, bottle racks, snow shovels, posters, expressions, etc. – by the act of appropriation which characterizes collage. In his text “Intermedia” (1966), Dick Higgins showed how the ready-made participated in the break with artistic tradition and the separation between “high and low culture”: “Part of the reason that Duchamp’s objects are fascinating while Picasso’s voice is fading is that the Duchamp pieces are truly between media, between sculpture and something else, while a Picasso is readily classifiable as a painted ornament” (Higgins 2001: 30). The object itself becomes the support for collage as in Cadeau (1921) – an iron to which nails had been added – by Man Ray, or as hybrid paintings, such as La Femme aux allumettes (1923–1924) (Fig. 5), by Francis Picabia, whose hair is made of

Fig. 5 Francis Picabia, La Femme aux allumettes (I), 1924–1925, oil and Ripolin on canvas with matches, hairpins, coins, and hair clips. Private collection. Photo: Márcia Arbex-Enrico, Exposition Picasso-Picabia, Musée Granet, 2018

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

877

Fig. 6 Marcel Duchamp. L.H.O.O.Q, Mona Lisa with moustache, 1919. Philadelphia Art Museum. https://www.wikiart.org/pt/marcel-duchamp/l-h-o-o-q-mona-lisa-with-moustache-1919. Public domain USA

matches, functioning as an iconoclastic provocation, or as a one-off intervention in other artworks, such as in Duchamp’s L.H.O.O.Q (1919) (Fig. 6), which takes up a reproduction of Leonardo da Vinci’s La Gioconda. Through the process of

878

M. Arbex-Enrico

appropriating the object, the ready-made calls into question the originality of the artist and of the artistic object, which ceases to be unique; it promotes the denial of the pictorial technique by borrowing from industrial or manufactured material; it institutes the “personality of choice” (Aragon 1980: 53) and consecrates itself as the most important gesture of removing the aura of art in the twentieth century. Along with the ready-made, the photomontage – a collage made mostly of photography prints – would be largely practiced in the early twentieth century, beginning with the Dadaists who, according to Raoul Haussmann, were the first to use photography to create a new object from elements hijacked from their initial meaning, “which extracted from the chaos of wars and revolutions an entirely new image” (Bosseur 2010: 33). László Moholy-Nagy distinguishes the photomontage from what he calls photoplastique, characterized by “cutting out, juxtaposition and the painstaking composition of photographic prints,” by an experimental method of “simultaneous vision that condenses and summarizes the imbrication of the eye and the word,” resulting in a more advanced form than the initial Dadaist photomontages (Bosseur 2010: 32). In Berlin, the group made up of Haussmann, Richard Huelsenbeck, Johannes Baader, John Heartfield, Hanna Höch (Fig. 7), and George Gros, among others, the founders of Club Dada in 1918, elect the photomontage – called tableau collé in the late 1910s in Berlin (Bosseur 2010: 32) – as the most suitable way of manifesting the commotion caused by the First World War, the violent world situation and the resulting fragmentation. Like Duchamp, “by invading the land between collage and photography, the German John Heartfield8 produced what are probably the greatest graphics of our century, surely the most powerful political art that has been done to date” (Higgins 2001: 30). In this context, the procedure takes on a more evident function as political protest: “But the idea of the photomontage was as revolutionary as its content, its form as subversive as the juxtaposition of a photograph and printed texts which, together, turn into a still movie,” says Haussmann (Bosseur 2010: 33). Thus, in these initial decades of the twentieth century, collage is the result of the combination or mixing of both iconic and verbal fragments, objects, reliefs, and prints, proceeding from the cutting and pasting of different materials, more literally than conceptually, that create a “medial complex integrating materiality, perception and cognition,” as Elleström says discussing the modalities of media (2010: 15). The strategic role of collage manifests itself both aesthetically and politically. It challenges the usual concept of art by breaking with the illusionist aesthetics, introducing heterogeneity, disconnection, displacement, by calling into question the hierarchy of materials, of supports, indicating its critical function through the introduction of the perishable: the “extreme, arrogant poverty of materials” (Aragon 1980: 47). This transgressive gesture materializes in political contestation, as a form of protest and intervention in reality.

8

Cf. John Heartfield’s graphic works available at https://www.museumdefundatie.nl/en/johnheartfield/. Accessed on October 25, 2022.

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

879

Fig. 7 Hannah Höch, Cut with the Kitchen Knife Through the Beer-Belly of the Weimar Republic, 1919. https://www.wikiart.org/pt/hannah-hoch/cut-with-the-kitchen-knife-through-the-beer-bellyof-the-weimar-republic-1919. Public domain USA

Verbal and Literary Collage: A New Visual Order In the chapter on the invention of collage, Perloff stresses how much Marinetti’s manifests, such as Destruction of Syntax–Wireless Imagination–Words-in-Freedom (1913) (Fig. 8), “provides an elaborate program of collage aesthetic with respect to literary discourse” (1986: 56), which advocate mainly, as we know, the destruction of syntax, the absence of punctuation or form of verse, the use of uninflected verbs, the substitution of composite names for adjectives, the use of mathematical symbols to replace conjunctions, and the use of typographical resources for emphasis,

880

M. Arbex-Enrico

Fig. 8 Marinetti, example of parole in libertà: Une assemblée tumultueuse (Sensibilité numérique) (Marinetti 1987: 108–109)

onomatopoeias, or phonetic spellings, among other means. In fact, “How can traditional discourse with its complete sentences (. . .) convey this new language of telephones, phonographs, airplanes, the cinema, the great newspaper, wich Marinetti calls ‘the synthesis of a day in the world’s life’” (Perloff 1986: 57)? The example commented by the author is the sound poem Zang Tumb Tuuum (1914), also presented as performance throughout the year of 1913 in several European capitals, which expanded collage as visual or spatial concept paving the way for verbal as well as musical and performatic experiments: “From Marinetti’s Zang Tumb Tuuum, it was just a short and perhaps inevitable step to Apollinaire’s Calligrammes, William Carlos William’s Kora in Hell or T.S. Eliot’s Waste Land, a poem whose collage composition is at least partially the result of the cuts made by Ezra Pound, himself the great master in English of collage form” (Perloff 1986: 72). The example of Apollinaire is often the one cited as the pioneer of verbal collage. In Calligrammes (1925) there are poems created from ads, newspaper titles, scraps of letter envelopes, and others which seem totally made from phrases and bits of conversations captured randomly and stitched together: “Thus, in Lundi rue Christine, the different verses or lines show up as notes of something heard at such a date in such a place; the choice was sufficiently well made to pass the impression” (Butor 2003: 34–35). Regarding “Lettre-Océan” (Fig. 9), Michel

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

Fig. 9 Guillaume Apollinaire, “Lettre-Océan,” Calligrammes (1925), p. 43

881

882

M. Arbex-Enrico

Butor evokes its character of “ideographic poem” for its plastic power, the presentation of ideas in a “visual order,” in which “the relationship between these fragments is no longer that of grammatical logic but that of an ideographical logic, resulting in a spatial display order opposed to that of discursive juxtaposition. (. . .) Revolution: because our intelligence must get used to understanding syntheticideographically instead of analytic-discursively” (Butor 1966: 16). This kind of word-image relationship is often taken as example of “intermedial discourse,” according to Léo Hoek, characterized by the fusion of different elements, or an “intersemiotic text,” as Claus Clüver says, “because such texts will often be housed in one medium [. . .], usually read as literary texts” (Clüver 2008: 25). Tristan Tzara would retake the collage composition process writing the famous “To make a Dadaist poem,” in his “Dada Manifeste sur l’amour faible et l’amour amer” (1920), using scissors, glue, and a good share of chance: Take a newspaper. Take a pair of scissors. Choose an article as long as you are planning to make your poem. Cut out the article. Then cut out each of the words that make up this article and put them in a bag. Shake it gently. Then take out the scraps one after the other in the order in which they left the bag. Copy conscientiously. The poem will be like you. And here are you a writer, infinitely original and endowed with a sensibility that is charming though beyond the understanding of the vulgar.

In the first Manifeste du Surréalisme, André Breton had suggested extending the surrealist means following Tzara’s recipe for the Dadaist poem. He attributes to the papiers collés by Picasso and Braque “the same value as the introduction of a platitude into a literary analysis of the most rigorous sort” and suggests calling a “POEM what we get from the most random assemblage possible (observe, if you will, the syntax) of headlines and scraps of headlines cut out of the newspapers” (Breton 1988b: 341). This proposal was followed by some examples created by himself, with all the “plastic power” brought about by the typographical variety of the printed matter used, with its different colors, shapes, and fonts (Fig. 10). Thus, besides visual collages, verbal or literary collages will be broadly exercised by the surrealists. Adamovicz mentions the “poems made partly or entirely from ready-made materials (Péret, ‘Hier en découvrant l’Amérique’, 1926), collage elements in prose works (Aragon, Le Paysan de Paris, 1926), and collaborative collages such as surrealist games or ‘L’enfant planète’ (1926), a montage text by Robert Desnos and Benjamin Péret,” besides examples of rewriting, “rework” of earlier texts, such as L’Immaculée Conception (1930), by Breton and Éluard (Adamowicz 1998: 8). Lautréamont’s saying – “Poetry must be made by everyone. Not by one (. . .)” – sums up well the idea that art has ceased to be individual (Aragon 1980: 57), an idea that underlies verbal and visual collage in the broader sense, more

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

883

Fig. 10 André Breton, Manifeste du Surréalisme (1924), pp. 342–343

conceptual than technical, as forms of rewriting, parody, and even plagiarism, as much as four-handed or group work. André Breton, who practiced collage in several forms, says that the poem-collage “Le corset mystère” (1919), a title borrowed from a Parisian shop, is “one of the most authentic collages,” made up of parts of ads alternating with ready-made phrases and small interventions (Bosseur 2010: 171) (Fig. 11). Printed items, particularly newspapers, are still a privileged material for verbal collages. Blaise Cendrars composes “Dernière heure” (Dix poèmes élastiques) as a “telegram-poem copied from Paris-Midi,” thus revealing that he had started from information found in a newspaper from 1914, from which he only eliminated some details and organized the sentences on the space of the page as if they were a poetic text (Bosseur 2010: 171). Louis Aragon, in turn, says he dreams about the poster poem (poème affiche), produced from borrowed items, introducing the commonplace and the cliché, aiming at the desacralization of the poetic word under the guise of a manifest (Aragon 1980: 120). This is what is shown by Le Paysan de Paris, with collages in the form of shop signs, posters, billboards, and notices. In his article “Collages dans le roman et dans le film” (1965), in which he analyzes the literary collages of his book Les Beaux Quartiers

884

M. Arbex-Enrico

Fig. 11 André Breton, “Le corset mystère,” Littérature, n 4, p. 7. The International Dada Archive http://sdrc.lib.uiowa.edu/dada/litterature/4/pages/07.htm

(1936), he also evokes the desire to introduce a collage in the novel, as Picasso did introducing a newspaper title or a page from a musical score into his painting, in this instance a phone conversation overheard by chance (Bosseur 2010: 173).

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

885

Borrowing in the form of literary collage is also found in Schwitters’ Anna Blume, “a parody of sentimental German songs,” a “potpourri of different texts, proverbs, Baroque citations, etc., and thus conforming to the Dadaist anti-aesthetics” (Sanouillet 1993: 44); in the Glossaire j’y serre mes gloses (1925–1926), by Michel Leiris; or in the novels by Raymond Roussel, James Joyce, or John Dos Passos, whose work 42e Parallèle (1930) first suggested the expression novel-assemblage, in France, in 1932 (Bosseur 2010: 175). We cannot fail to mention the well-known Walter Benjamin project Passagen-Werk, begun in 1927 and unfinished in 1940, composed of an “astonishing piling up of quoted passages” (Perloff 2010: 25) which method is “literary montage,” as Benjamin explains: “I needn’t say anything. Merely show. I shall appropriate no ingenious formulations, purloin no valuables. But the rags, the refuse – these I will not describe but put on display” (quoted by Perloff 2010: 26). T.S. Eliot’s The Waste Land (1922) and Pound’s The Cantos are also foundational texts of a new poetics which does not seek originality of verses but operates by a collage of quotations.9 The importance of printed materials is also identified in typographic poems, as practiced by Hugo Ball (Karawane, 1917) or Haussmann (Plakatgedicht),10 who play with different types and letter sizes, with repercussions in visual poetry and concrete poetry. Even in some texts intended for the theater, the collage procedure can be found, as a kind of literary collage, made by borrowing printed materials. In the text of the play Victor ou les enfants au pouvoir (1928), Roger Vitrac faithfully reproduced the typography of the newspaper Le Monde which one of the characters in the play is reading. Tzara, in Mouchoir de nuages, includes in act XII of the play 36 replies borrowed from three scenes taken from Hamlet’s act II. In Russia and in Germany, there are also examples in the theater, more specifically of literary assemblage, with strong social and political connotations (Bosseur 2010: 174). Just as Schwitters elected the word Merz to designate his work, Jiří Kolář invented the word chiasmage to express the labyrinthine principle of collage, like the procedure used many years later in his theater play La Peste d’Athènes (1965), composed of long quotations or paraphrases of literary and historical texts. In that regard, the artist clarifies his desire to write “otherwise,” starting from a new principle: “I use the term collage because only collage could ban the fable from the stage, only collage would give polyphony its full reach, as the cornerstone of a new dramatic form” (Bosseur 2010: 178–179). The enumeration of these various examples shows the large range of practices – either “pure” or “pseudo” – that can be considered a verbal or literary collage phenomenon. However, the specificity of each support must be accounted for in process of transfer from the source to the arrival support: in pictorial productions,

9 Still later, Michel Butor in Mobile: Étude pour une représentation des États-Unis (1962) and Georges Perec in La Vie mode d’emploi (1978) would use collage decisively in their narratives, as part of the writing process, a form of “criticism and invention,” as in the title of Michel Butor’s essay (1968). 10 Ball (1917), Haussmann (1918).

886

M. Arbex-Enrico

collage usually leaves “scars,” marks that make visible the cut-and-paste gesture (Adamowicz 1998: 15), unless the artist, like Ernst, uses reproduction techniques that “filter the initial gesture of the collage,” such as lithography or serigraphy (Bosseur 2010: 176). In verbal collages there is a certain invisibility of these edit marks, when the collage can only be detected by the rupture in logical discourse, the introduction of an alien element, or the possibility of identification of the texts which have been appropriated (Adamowicz 1998: 15). In fact, borrowings from printed material, such as newspapers, are generally homogenized when transferred to books, adapting to the format and rules of publication and thus dissipating the gesture and the marks of collage. James Joyce tends to erase the visual features of the media to give the impression of homogeneity, as opposed to what happens in 42e Parallèle or in Le Paysan de Paris. In those books, collage exists not only at the semantic level – considering the displacement of fragments made partly or entirely from ready-made materials – but also at the material modality, for the newspaper cut-outs are inserted just as they appear materially in the press, with typographic variations and formatting typical of the semiotic field to which they belong, thus breaking with the graphic homogeneity of the conventional narrative. Between these two examples, we may place Mobile (1962), by Butor, who subtly marks his collages by using a diversified typography and a novel and heterogeneous mise-en-page, similar to that of Mallarmé’s Un coup de dés (Arbex 2020).

Surrealist Collage: The Conceptual Turn to a Poetic Device Collage as it was exercised by the surrealists deserves to be highlighted, a context in which the practice – either pictorial, cinematographic, theatrical, or verbal – is theorized and raised to the status of poetic procedure. This issue about “pure” or “pseudo-collage,” verbal or visual, will be somewhat outdated by the conceptual turn operated by surrealism. As Max Ernst said, already quoted, “If it is the feather that makes the plumage, it is not the glue that makes collage” (Ernst 1970: 256). The seminal role of Max Ernst’s pictorial collages is, by the way, decisive for the drawing up of the surrealist principles and recognized since the first texts by Aragon on the subject. It is a practice capable of creating new poetic images, and its influence is not only found in painting but also affirmed in the creation of new artistic forms, mixed media, marked by hybridism, pure collages such as the object-poem and the novel-collage, or even collages that do without the use of glue, the pseudo-collages. Once again, the definition of collage is challenged by the variety of practices, as remarks Elza Adamowicz (1998: 13), who quotes from Arp’s papiers déchirés, Ernst’s collages-découpages, Hugnet’s poèmes-découpages, Eluard’s photocollages, Valentin’s photomontages, as well as Dali’s disguised collages, and even Magritte’s mental collages or Ernst’s painted collages. We can add to this list some objects such as the one reproduced photographically in L’Amour fou, a trouvaille, a objet trouvé bought by Breton in the flea markets and described as “a big wooden spoon, rustically made, whose handle, when placed on its convex part, rose to the height

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

887

of a small shoe that was integrated with the spoon” (Breton 1937: 43), an object that reflects well the principle of surrealist collage in its role of catalyzing desire and kindling the poetic spark. At first, the surrealist collage aims to affirm itself against the papier collé. Aragon was the first to stress the essentially modern nature of collage as a poetic mode of production, based on the work of Max Ernst, whose collages had been exhibited in Paris in 1921, at the Au Sans Pareil gallery. In “Max Ernst, peintre des illusions” (1923), he states: “For the cubists, the postage stamp, the newspaper, the matchbox that the painter pasted on his canvas were valid as a test, an instrument for controlling the painting’s reality itself. It was around the object borrowed directly from the outside world that, to employ the cubist vocabulary, gave him a conviction, that the painter established the relationships between the several parts of his painting. [. . .]. In Max Ernst it is wholly different. The elements he borrows are particularly drawn items, and the collage often supplements this drawing. Here collage becomes a poetic procedure, perfectly opposed, in its objectives, to the cubist collage, whose intent is purely realistic” (Aragon 1980: 29, my italics). Aragon relates collage with “magic practices,” with the “miracle,” understood as a property of the marvelous in its irruption into daily life. The author of the Paysan de Paris confers a metaphorical function to Ernst’s collage, for its ability to create visual “illusions,” by the arbitrary and irrational juxtaposition of iconographic elements, by the dislocation of each element from its original meaning to awaken it to a new reality: “Illusions everywhere: illusion this caravan of extraordinary birds crossing a desert, up close these are women’s hats cut from a department store catalog; illusions this glacier, these trees, these characters. Any appearance, our magician recreates it. He diverts each object from its meaning to awaken it to a new reality” (Aragon 1980: 30) (Fig. 12). In 1930, in La peinture au défi (The Challenge to Painting), Aragon reinforces his first approach, now drawing attention to photographic collage and the collage of illustrations, probably because he had in mind Ernst’s novel-collages editions of 1929 and 1930: “Where and when did collage appear? Despite the attempts of several of the very first Dadaists, I think we must pay homage to Max Ernst, at least for the two forms of collage that are furthest from the principles of papier collé, the photographic collage and the collage of illustrations. Immediately, this discovery came into general use, and the German dada publications in particular contain collages signed by at least ten authors. But the success of this procedure resulted more from the astonishment of mastering a new system than from the need to express oneself at all costs. Quickly, the use of collage became limited to a few men, and it is certain that the entire feeling of collages then being made derived from the thought of Max Ernst and Max Ernst alone” (Aragon 1980: 64). Collages had features of what would be the surrealist poetic image, obtained via automatic writing. Pierre Reverdy, to whom Breton refers in the Manifesto, observes the involuntary character of the image – “a pure creation of the spirit” – and situates its origin in the “approximation between two more or less distant realities” (Breton 1988b: 324), an approximation which should produce an effect of contradiction, where uncommon and surprising analogies would arise from the image. The

888

M. Arbex-Enrico

Fig. 12 Max Ernst, Above the clouds (Au dessus des nuages), 1920. https://www.wikiart.org/pt/ max-ernst/above-the-clouds-1920. Public domain USA

dreamlike dimension is essential in this process: “The point is to undertake the most meticulous de-realization possible, to make the irrational and the dream arise through images which seem commonplace and without mystery, to dépayser” (Bosseur 2010: 44). Max Ernst himself draws up a theory of collage, associating this technique with surrealism and giving it an extensive character including the papiers collés. In Au delà de la peinture (1936), a key text for understanding the phenomenon, references to Rimbaud are reaffirmed in the definition of collage: “The simple hallucination, according to Rimbaud, la mise sous whisky marin [“putting under sea whisky,” literally], according to Max Ernst. [Collage] is something like an alchemy of the visual image. The miracle of the total transfiguration of beings and objects with or without changes to their physical or anatomical appearance” (Ernst 1970: 253). As

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

889

for the definition of the “mechanism” of collage, it is to Lautréamont and Breton that the artist resorts to speak of the “exploration of the occasional meeting of its distant realities on an inconvenient plane (that is, paraphrasing and generalizing Lautréamont famous saying: Beautiful as the chance meeting between a sewing machine and an umbrella on a dissecting table) or, in short, cultivating the effects of a systematical dépaysement, according to André Breton’s thesis (. . .)” (Ernst 1970: 253). It is well known that the surrealists had retrieved Lautréamont’s lesson and raised his famous sentence, quoted above, to the category of synthesis of the collage process and model for the new standard of beauty, which favors shock, surprise, the unexpected meeting of distinct realities, a “beauté convulsive” (Breton 1988a: 753). The work of Ernst, a “man of infinite possibilities,” showed Breton this way: in the text of the catalogue, entitled La Mise sous whisky marin, the poet evokes collage as a “marvelous capacity for approximating two distant realities without leaving our field of experience and obtaining a spark from this approximation,” a poetic outbreak obtained by means of visual images, as poets do through verbal images. Therefore, although the point of departure of surrealist collage is the “combinatory technique, namely a material mode of cutting and pasting pre-existing iconic or verbal messages” (Adamowicz 1998: 15), we see the theoretical commitment to a new approach related not to the types of objects involved in the technique or their materiality, but to collage’s inventive capacity of creating new poetic images – pictorial or verbal, or a combination of both. Relying on texts by Aragon, Adamowicz distinguishes four “fields of action” for collage: “As a practice, surrealist collage encompasses a wide range of activities, from encounters with defunct objects at the fleamarket to the transcription of the multiple voices of the unconscious, the fragmentary images of the dream and all modes of production which stage the clash of disparate elements. As a pragmatic act, collage englobes various complementary or conflictual functions – critical, poetic and political – which cohabit throughout the 1920s and 30s. As a technique, collage is a material mode of cutting and pasting distant elements – or indeed a simulation of that process. As a subversive act, it is an instrument of détournement of pre-formed messages (. . .). And as a creative act, it involves the transformation of these messages” (Adamowicz 1998: 13). Surrealist collage confirms it as a powerful instrument for contesting doxa, since, breaking with the order of reality, it inaugurates a criticism of artistic and social codes, calling in question the idea of artistic personality, of talent (Aragon 1980); and, as Walter Benjamin said (1994), it furthers the decline of the aura of the artistic object. Among all these fields of action, surrealist collage stands out as a creative act subverting traditional models of representation, based on the strategies of dislocation and re-signification of signs it employs: “It is also, and more essentially, a creative act of détournement, through the subversive manipulation and creative transformation of ready-made elements, forging the surreal out of fragments of the real, suggesting the merveilleux through the combination of banal and defunct images,

890

M. Arbex-Enrico

clichés and rewritten texts. It is essentially a semiotic practice of transforming pre-formed iconic or verbal messages” (Adamowicz 1998: 17). As a subversive act, collage is a means of liberation from conventional artistic truths, eluding the savoir-faire of pictorial tradition and inventing new medial forms. Photomontage, as a collage procedure, the way it was practiced starting in the 1920s and 1930s, also departs from these criteria. On the one hand, photomontage, for propaganda objectives or as satire, was a “technique which had been favoured by the Berlin dadaists in the early 1920s as a polemical instrument” (Adamowicz 1998: 11). John Heartfield, a member of the German Communist Party, for example, uses it as a way of manifesting his political convictions, in which meaning is the major factor. However, in “John Heartfield et la beauté révolutionnaire” (1935), Aragon includes his collages with photographs among the practices with “poetic objectives,” as a criticism of painting, in which the “vertigo” evoked by Rimbaud is recognized: “the salon at the bottom of the lake of Saison en Enfer became the usual atmosphere of the painting” (Aragon 1980: 84). On the other hand, as it utilizes elements from reality – original photographs or images from the press, that is, figurative images – photomontage has a different impact from collages made from drawings or obtained from reproduction processes such as engraving, owing to its adherence to the referent. Aragon (1980: 83) states that the strength and attractiveness of these new photograph collages are due precisely to the “verisimilitude” ensured by the figuration of actual objects. Seen as a borderline technique in which psychic automatism intervenes solely at the margins of a selective play with some objective materials, collage makes evident an essential aspect of the image: its demystifying and critical value, questioning appearances in name of a hidden, repressed truth (Béhar 1982). In the 1930s and 1940s, André Breton made about ten poems-objects,11 whose origin he places in dreams, combining poetic and plastic resources and aiming at the exploration of their mutual “exalting power,” as he says in Le Surréalisme et la Peinture (1965: 365). Georges Hugnet uses photomontage to create an oneiric and strange universe from images of animals and plants, unlike the Berlin group whose works had a stronger political and protest character. Collage brings out the oneiric and unconscious dimensions, even without the photographic materiality, as we can also see in the three books of Max Ernst’s novelcollages. This hybrid genre – roman-collage – consists of the juxtaposition of a text and visual collages made from engravings from nineteenth-century publications, such as almanacs, illustrated magazines, and serials, placed on the same page. By cutting out these images that refer to a petty-bourgeois collection and pasting it with others, Ernst gives them another meaning, creates tension within the image, and shifts from the real to the imaginary: the reader plunges into a strange and distressing dream universe, ironic or full of humor. For example, in Rêve d’une petite fille qui

11

Breton (1935).

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

891

voulut rentrer au Carmel (1930),12 the collage highlights the “performatic character of the dream by treating the dream theme, linking it to eroticism, ecstasy and hysteria, allows the role of the unconscious to be highlighted, as it brings into play conflicts, images and symbols that it preserves and that dreams can reveal,” at the same time that it exerts its subversive role of criticizing the Church, the clergy and the bourgeois morality (Arbex 2002: 223). Finally, among the surrealist wide range of practices, and although it is not our goal to approach the subject, considering that they involve time and another kind of sensorial perception, we can mention some examples of cinematographic productions. In fact, several films made throughout the 1920s use methods close to pictorial collage: Ballet Mécanique, by Fernand Léger, conceived without a script and based only on the rhythmic relationships between the images; Entr’acte, by René Clair in partnership with composer Erik Satie and artist Francis Picabia; Emak Bakia, by photographer and artist Man Ray, who uses an “automatic” method in the film; La Coquille et le clergyman, by Germaine Dulac with a screenplay by Antonin Artaud, who conceived cinema as the domain of dreams; and, finally, Luis Buñuel, who conceived his films, like Un chien andalou, as an “unrealistic sequence of realistic images,” reinforcing his belonging to surrealist principles (Bosseur 2010: 185).

Collage and Intertextuality As we hope to have shown, the word “collage” is an umbrella term covering different meanings; it has several cognates and posits the issue of generalizations if not duly contextualized. The conceptual turn operated by surrealism, from a technique to a mode of production, a poetic and creative principle, no matter the materiality of the media involved, led to the comparison with concepts from literary theory, especially that of intertextuality in the 1970s. Adamowicz (1998: 15) states that “the specificity of surrealist collage is to have drawn attention to the intertextual process itself, by its deliberate mise-enscène of diverse and often divergent verbal and pictorial texts, which reveal the mechanisms of the assembling process by displaying its breaks.” In fact, studies on intertextuality as defined by Julia Kristeva, based on Bakhtin’s dialogism, recognize the pictorial origin of the word and extend the concept to the field of literature. Nathalie Piégay-Gros designates collage as one of the “procedures which consist in pasting together heterogeneous materials; by extension, it becomes synonymous with quotation and intertext, and refers to any fragment (whether verbal or otherwise) integrated into a new whole” (Piégay-Gros 1996: 179). For the author, intertextuality corresponds to the pictorial exercise of collage by its mode of operation, that is, “to include in the work a heterogeneous piece, to turn the space 12 Some pages of Max Ernst’s roman-collage Rêve d’une petite fille qui voulut rentrer au Carmel are reproduced at https://www.moma.org/collection/works/222806 and https://archives-dada.tumblr. com/post/120184706110/iconophages-max-ernst-from-his-second; examples of La Femme sans têtes (1929) can be seen at https://www.centrepompidou.fr/es/ressources/oeuvre/crbga6a.

892

M. Arbex-Enrico

of invention into the locus of bricolage, of the assemblage of discontinuous elements,” like writers when they summon somebody else’s text to their own (PiégayGros 1996: 4). This is one of the forms of intertextuality, beside citation, allusion, and plagiarism, characterized by the copresence of two or more texts or, more frequently, the actual presence of one text inside another (Genette 1982: 8). Annick Bouillaguet also classifies collage alongside citation, pastiche, and parody for being one of the forms of what she calls “imitative writing” and defines it as a “technique consisting in juxtaposing ( faire cohabiter) the most heterogeneous elements” (Bouillaguet 1996: 125). By approximating citation and collage, Michel Butor draws attention to the display of the fragment’s origin in different degrees. He complements his argument with visual and verbal examples: “If I read the name of the newspaper Le Jour in one of Braque’s papier collés, I know where the fragment comes from. If I recognize the style of the Catalogue de la Manufacture des armes et cycles de Saint-Étienne, I also know the source. But the origin is more or less detailed. I know what newspaper is meant, but not necessarily its date; or I may know the type of newspaper or catalogue, what style of publication. We then have more or less hidden quotes whose origin is sometimes extremely difficult to establish; there are different degrees of citation. In all cases the essential point is that one plays with the colors, the expression, the aspects. In literature, collage, either by Apollinaire or Cendrars, convokes quotations from everyday conversations that one can no longer recover. It must have the appearance of a cut-out, of collection from a raw reality of which we will never recognize the integral record” (Butor 2003: 34–35). Butor particularly emphasizes the dialogical and current nature of collage, insisting on the visual effects it creates: “the chromatic, stylistic and evocative qualities of the fragments brought into contact,” which produce “mosaics,” whose materials form figures though maintaining “their essential distinctions” (Butor 2003: 34–35). Difference, alterity, dialogism, and scars are marks of verbal or pictorial collage. Perloff quotes the definition of Group Mu: “Each cited element breaks the continuity or the linearity of the discourse and leads necessarily to a double reading: that of the fragment perceived in relation to its text of origin; that of the same fragment as incorporated into a new whole, a different totality. The trick of collage consists also of never entirely suppressing the alterity of these elements reunited in a temporary composition” (Perloff 1986: 47). Not everyone identifies collage and citation so easily. For Jean-Yves Bosseur, one should distinguish the two terms usually employed to designate the borrowing procedure. On the one hand, the use of citation betrays a certain concern with “recognition”; it is a critical attitude in the sense that the author tries to insert his thought into a historically wider field, which may be identified by his readers. On the other hand, collage belongs to a broader field of investigation; it is not limited to the elements previously elaborated and may have recourse to raw elements, taken from the immediate environment, sometimes presented as objets trouvés. A surprise effect is thus created, a shock between the pasted elements, which do not always respond to

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

893

a structured organization as in quotation (Bosseur 2010: 8). Collage is marked by the gratuitousness of the grafted text and its uselessness for the narration: the transposed element must appear as a foreign body. On the other hand, citation is a text integrated into the author’s discourse, preserving its global unity. It should be noted that, as mentioned above, Bosseur considers the case of the delimited citation, often indicated by quotation marks, in which the reader is informed of its origin. In this sense it really differs from collage, which contains the idea of relativity and negates the unity and organicity that had long characterized the notion of the work of art (Bosseur 2010: 9). From this same perspective, Henri Béhar considers collage a borderline case of intertextuality (Adamowicz 1998: 14), as it presupposes a double production of meaning, in surface and extension (when the clipped text is identified), distinguished, on one hand, from the quote for keeping neither the marks nor the autonomy of the subject, and, on the other, from montage, which plays with rhythm. In spite of the definitions, many ambiguities persist between citation and collage. Louis Aragon states that every citation may be considered a collage, understood as a poetic device, and presents the example of Marcel Duchamp’s Gioconda with a moustache, a collage which is in the citation of Leonardo da Vinci’s painting and not in the moustache, added to the image later on, moustaches, moreover, which were hand-painted and not glued (Aragon 1980: 113–136). But if we accept that collage is a procedure used in the nonvisual arts, such as the poem or the novel, the confusion between collage and citation is inevitable, as Aragon affirms (1980: 131). Antoine Compagnon may clarify the issue, emphasizing the “archaic gesture” that is at the base of collage and quotation: “The quotation represents the primary practice of the text, the foundation of reading and writing: to quote is to repeat the archaic gesture of the cut-paste, the original paper experience (. . .)” (Compagnon 1996: 31).

Conclusion: Toward Hybrid Medial Forms We should acknowledge that the word collage covers diverse aesthetic paradigms, sometimes ambiguous, beginning with the term itself, which denotes the gesture of cutting-pasting as well as the product of this action. The diversity of procedures covered by the term surely contributes to the lack of a univocal definition. As a technique, collage is the “archaic” gesture of cutting-pasting – literally and materially – heterogeneous materials, whether iconographic or verbal, dislocating them from their context and inserting them, by juxtaposition, combination, or fusion, into another. As an artistic procedure, collage becomes a creative act through the same gesture of cutting and pasting – literally but also conceptually, as “collage effects” (Taylor 2005: 196) – fragments from different contexts (images, texts, objects, etc.) to resignify them, whether with aesthetic, poetic, political, or critical intentions. It results from an aesthetics of “shock” (Quintyn 2007: 20) involving several gestures: selecting, appropriating, cutting out, dislocating, juxtaposing, combining, mixing, reorganizing, accommodating, and pasting.

894

M. Arbex-Enrico

At the time of the avant-gardes, collage was both a technical and pragmatic act, subversive and creative, resulting in the creation of novel hybrid medial forms which caused deep ruptures in the very concept of art and whose repercussions spread along the twentieth century as a principle, a concept, and a structural model. Combine paintings, as Robert Rauschenberg called his constructions, as well as environments, “enveloping collages which surrounded the spectator,” and happenings, collages including “live people” (Higgins 2001: 30), by Allan Kaprow, are examples of this practice. Several artistic procedures derived from collage extend its critical and artistic reach: Ian Hamilton Finlay’s “poems-collages”; Gil. J. Wilson’s “scotch art,” the articulation of images and texts using adhesive tape aiming at a kind of “unwriting” (désécriture); Grupo 70’s “visive poetry,” based on resources borrowed from mass culture, aiming exactly at contesting the instruments of commercial propaganda in contemporary society; and Joan Brossa’s “poem-objects” (Bosseur 2010: 181). The work of John Cage, with his “vocal scores” (Song Books), “concrete poetry,” as well as plastic productions (Bosseur 2010: 177), is frequently cited when speaking about the developments from the principle of collage. The artist moves between all these diverse artistic manifestations and circulates their elements from one medium to another, also exploiting, as Higgins puts it, “the intermedia between music and philosophy” (Higgins 2001: 32). Expanding the term to the sound field and highlighting its quotational aspect, Claus Clüver analyzes what he terms Musico-Verbal Collage in Luciano Berio’s Symphony, particularly in its “third section, which is a carefully crafted collage of quotations, both musical and verbal” (Clüver 2007b: 230). Practiced initially on the level of poetic writing, Brion Gysin’s and William Burroughs’s “Cut-Up Method,” which consists in “cutting newspaper articles into sections and rearranging the sections at random” (Burroughs 2022), extends afterward to the sound level, with magnetic tape recordings, but also into cinema, where collage is often confused with mounting. In the film image, collage materializes above all in the procedure of “found footage” which, since the 1920s, consisted in recycling elements recovered from pre-existing movies, leftover footage found in film laboratories, a procedure which would be later updated by the proposition of experimental methods of “wild” appropriation of non-original material, at the margins of commercial cinema (Bosseur 2010: 192). These are just some examples of how the collage principle materializes, strongly, as a structural model, by exhibiting the mechanisms of verbal or pictorial production, a production which purports to be “uncreative,” as Kenneth Goldsmith demanded for poetry: “a form of copying, recycling or appropriation that obstinately makes no claim on originality” (Perloff 2010: 12) (Fig. 13). These practices became more and more hybrid in their unconformity with the idea of a pure medium and with the arrival of new technologies, by the possibilities of assembling and mixing which reinforce the concept of collage in its broader and nonhistorical sense. The multimedia artist – “the unoriginal genius” (Perloff 2010) – moves between different fields and uses different supports and materials, so that it is possible to resignify

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

895

Fig. 13 Centre Georges Pompidou, 2018. (Photo: Márcia Arbex-Enrico)

collage as an intermedial phenomenon, as a form of cultural production marked by the crossing of borders between the arts but also between media.

References Adamowicz, Elza. 1998. Surrealist collage in text and image: Dissecting the exqisite corpse. Cambridge: Cambridge University Press. Allais, Alphonse. 1891. Collage. À se tordre: histoires chatnoiresques, 31–36. Paris: Paul Ollendorff. André Breton. 1919. “Le corset mystère,” Littérature, n_4, p. 7, The International Dada Archive. http://sdrc.lib.uiowa.edu/dada/litterature/4/pages/07.htm. Apollinaire, Guillaume. 1925. Calligrammes: Poèmes de la paix et de la guerre (1913–1916). Paris: Gallimard. ———. 1980. Les Peintres cubistes. Paris: Hermann. Aragon, Louis. 1980. Les Collages. Paris: Hermann. Arbex, Márcia. 1992. De l’image de la lettre à la poésie peinte. Étude sur la fonction de l’écriture dans les arts visuels (1910–1930). Thèse de Doctorat. Université Sorbonne Nouvelle- Paris 3. ———. 2002. Onirismo, subversão e ludismo no romance-colagem. In Performance, exílio, fronteiras: errâncias territoriais e textuais, ed. Graciela Ravetti and Márcia Arbex. Belo Horizonte: Faculdade de Letras da UFMG.

896

M. Arbex-Enrico

———. 2020. Mesa de montagem: Colagem e montagem em Mobile. Sobrevivências da imagem na escrita: Michel Butor e as artes. Belo Horizonte: Editora Relicário. Ball, Hugo. 1917. Karawane. Available at https://archives-dada.tumblr.com/post/18180886477/ hugo-ball-karawane-1917 Béhar, Henri. 1982. Dictionnaire général du Surréalisme et de ses environs. Paris: PUF. Benjamin, Walter. 1994. A obra de arte na era de sua reprodutibilidade técnica. In Magia e técnica, arte e política: ensaios sobre literatura e história da cultura, ed. Tradução de Sérgio Paulo Rouanet, 165–196. São Paulo: Brasiliense. (Obras Escolhidas, 1). Boccioni, Umberto. 1912. Development of a bottle in space. The Metropolitan Museum of Art. Available at https://www.metmuseum.org/art/collection/search/485529. Accessed 25 Oct 2022. Bosseur, Jean-Yves. 2010. Le collage, d’un art a l’autre. Paris: Minerve. Bouillaguet, Annick. 1996. L’écriture imitative: Pastiche, parodie, collage. Paris: Nathan. Braque, Georges. 1912. Fruit dish and glass. The Metropolitan Museum of Art. Available at https:// www.metmuseum.org/art/collection/search/490612. Accessed 25 Oct 2022. Breton, André. 1935. Poème-objet (Poem-object). National Galleries of Scotland. Available at https://www.nationalgalleries.org/art-and-artists/28925; Poème-objet (Poem-object), 1941, MoMA: https://www.moma.org/collection/works/81121. Accessed 25 Oct 2022. ———. 1937. L’Amour fou. Paris: Gallimard. ———. 1965. Le Surréalisme et la peinture. Paris: Gallimard. ———. 1988a. Nadja (1928). Œuvres complètes. Vol. 1. Paris: Gallimard. (Bibliothèque de la Pléiade). ———. 1988b. Manifeste du Surréalisme (1924). Œuvres complètes. Vol. 1. Paris: Gallimard. (Bibliothèque de la Pléiade). Burroughs, William. 2022. The cut-up method of Brion Gysin. https://www.ubu.com/papers/ burroughs_gysin.html. Accessed 20 Jan 2022. Butor, Michel. 1966. Préface. In Calligrammes: Poèmes de la paix et de la guerre (1913–1916), ed. G. Apollinaire. Paris: Gallimard. ———. 1968. La critique et l’invention. Répertoire III. Paris: Les Éditions de Minuit. ———. 2003. Collage. Michel Butor par Michel Butor, 34–35. Paris: Seghers. (Poètes d’aujourd’hui). Clüver, Claus. 2007a. Inter Textus/Inter Artes/Inter Media. Aletria: Revista de estudos de literatura. Belo Horizonte: POSLIT, Faculdade de Letras da UFMG, v. 14, jul./dez, 11–41. ———. 2007b. The creative flow of a Musico-Verbal Collage: Section III of Luciano Berio’s Sinfonia. In Creations: Medieval rituals, the arts, and the concept of creation, ed. S.R. Havsteen, N.H. Petersen, H.W. Schwab, and E. Østrem, 229–246. Turnhout: Brepols Publishers. ———. 2008. Intermediality and interarts studies. In Changing borders: Contemporary positions in intermediality, ed. Jens Arvidson, Mikael Askander, Jørgen Bruhn, and Heidrun Führer, 19–37. Lund: Intermedia Studies Press, Lund University. Cnrlt – Centre National De Ressources Lexicales Et Textuelles. 2012. https://www.cnrtl.fr/ definition/collage. Accessed 15 Jan 2022. Compagnon, Antoine. 1996. O trabalho da citação. Belo Horizonte: Editora UFMG. Elleström, Lars. 2010. The modalities of media: A model for understanding intermedial relations. In Media borders, multimodality and intermediality, ed. Lars Elleström. Basingstoke: Palgrave Macmillan. Ernst, Max. 1970. Au-delà de la peinture. Écritures. Paris: Gallimard. Genette, Gérard. 1982. Palimpsestes: La littérature au second degré. Paris: Seuil. Haussmann, Raoul. 1918. Plakatgedicht (Poème-affiche). Available at https://www. centrepompidou.fr/en/ressources/oeuvre/cxA45e5. Accessed 25 Oct 2022. Higgins, Dick. 2001. Intermedia. In Multimedia: From Wagner to virtual reality, ed. Randall Packer and Ken Jordan, 27–32. New York: W.W. Norton. Krauss, Rosalind. 2006. Os Papéis de Picasso. Tradução de Cristina Cupertino. São Paulo: Editora Iluminuras.

34

Collage as a Creative Act: Emergence, Displacement, and Re-signification

897

Marinetti, F. 1987. Les Mots en liberté futuristes. Préface de Giovanni Lista. Lausanne: L’Âge d’homme. Perloff, Marjorie. 1986. The futurist moment: Avant-garde, avant guerre, and the language of rupture. Chicago: The University of Chicago Press. ———. 2010. Unoriginal genius: Poetry by other means in the new century. The University of Chicago Press. Picasso, Pablo. 1912a. Guitar. MoMA. Available at https://www.moma.org/collection/works/ 81723. Accessed 25 Oct 2022. ———. 1912b. La bouteille de Suze (Bottle of Suze). The Mildred Lane Kemper Art Museum. Available at https://www.kemperartmuseum.wustl.edu/collection/explore/artwork/1105. Accessed 25 Oct 2022. ———. 1912c. Nature morte à la chaise cannée. Musée National Picasso-Paris. Available at https://www.metmuseum.org/art/collection/search/755493. Accessed 25 Oct 2022. ———. 1913. Still life with violin and fruit. Philadelphia Museum of Art. Available at https:// philamuseum.org/collection/object/53855. Accessed 25 Oct 2022. Piégay-gros, Nathalie. 1996. Introduction à l’intertextualité. Paris: Dunod. Quintyn, Olivier. 2007. Dispositifs/Dislocations. Marseille: Al Dante. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités. Remédier/Remediation (6): 43–64. Ed. Philippe Despoix and Yvonne Spielmann. Sanouillet, Michel. 1993. Dada à Paris. Paris: Flammarion. Schwitters, Kurt. 1990. Merz: Écrits. Paris: Gérard Lebovici. Taylor, Brandon. 2005. Collage: l’invention des avant-gardes. Paris: Hazan. Tzara, Tristan. 1921. Pour faire un poème dadaïste (1920). Dada manifeste sur l’amour faible et l’amour amer. La Vie des lettres, no. 4.

Anthropophagic Appropriation and Intermediality

35

Anna Stegh Camati

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oswald de Andrade’s Concept of Cultural Anthropophagy: Historical Overview and Multiple Interpretations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Local and Global Resonances of Cultural Anthropophagy: Intermedial Relations . . . . . . . . . . . . Cultural Anthropophagy as Part of the Broader Fields of Intertextuality and Intermediality . . . Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

900 901 907 914 917 918

Abstract

The concept of anthropophagic appropriation or cultural anthropophagy was formulated by the Brazilian Modernist poet and thinker Oswald de Andrade in his Manifesto Antropófago in 1928. The notion derives from the anthropophagic rituals of the Tupinambá Indians, who used to devour their enemies to appropriate their intelligence, wisdom, and power. In this light, Andrade’s inventive avantgarde theory, translated as Cannibalist Manifesto, by Leslie Bary, encourages the ritual consumption of foreign artistic manifestations and its fusion with native Brazilian traditions and national themes. This act of ingestion, deglutition, and digestion represents a radical departure from the oppositional binary relations of self and other, allowing the incorporation into the self of what is most excellent in the other to generate something new and distinctly Brazilian. Andrade anticipated important theoretical perspectives on adaptation, appropriation, and transculturation, creative processes which only much later were articulated, according to different perspectives, by poststructuralist and postcolonialist critics. In this sense, he is an important forerunner, whose insights have been readdressed and reconceptualized on a global scale in the twentieth and twenty-first centuries. Considering that the concepts of adaptation and appropriation, which intersect and interrelate at many junctures, have been considered part of the broader A. S. Camati (*) Centro Universitário Campos de Andrade (UNIANDRADE), Curitiba, Brazil © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_38

899

900

A. S. Camati

scholarly fields of intertextuality (Sanders, Adaptation and appropriation. London/New York: Routledge, 2006) and intermediality (Elleström, Adaptation and Intermediality. In The Oxford handbook of adaptation studies, ed. Thomas Leitch, 509–526. Oxford: Oxford University Press. 2017), in the present chapter, I argue that anthropophagic appropriation or cultural anthropophagy is a form of creative non-subaltern intertextuality and that, seen from a diachronic perspective, it is a specific mode of media transformation or transmediation that can be located within the broader scholarly field of intermediality. Keywords

Cultural anthropophagy · Anthropophagic appropriation · Non-subaltern intertextuality · Transmediality · Intermediality

Introduction In 2022, celebrations of the bicentenary of Brazil’s independence and of the centenary of the Semana de Arte Moderna (Modern Art Week) marked the Brazilian cultural agenda with a series of meetings, exhibitions, catalogs, anthologies, and book launches. The Modern Art Week, which took place from February 13 to 17, 1922, at the Theatro Municipal in São Paulo, revolutionized the arts, literature, and culture in Brazil. The polemical and lasting repercussion of Oswald de Andrade’s Manifesto Antropófago (Anthropophagous Manifesto),1 published in the 1st edition of the Revista de Antropofagia in 1928, which established the main premises of creative anthropophagic appropriation of foreign trends – mainly European – and its fusion with native Brazilian traditions and national themes, was widely recognized during the Modern Art Week’s 50th anniversary festivities in 1972 and continues more alive than ever, having widely resounded in Brazilian and international criticism in 2022, when the Modern Art Week reached its centenary. The 100th anniversary celebrations of the Modern Art Week started on January 25, 2022, in São Paulo – date of the city’s 468th Jubilee – with a series of cultural innovations. While most of the members of the first Modernist movement belonged to São Paulo’s cultural elite (descendants of the wealthy coffee barons), who professed fascination for Tupi and Afro-Brazilian traditions but excluded indigenous and black artists, the representatives of the so-called new modernisms in the twentyfirst century come mainly from the periferias – the marginalized communities of the outskirts – most of them of African and Indigenous descent, such as Karol Conká, Linn da Quebrada, Gloria Groove, Jaider Esbell, Denilson Baniwa, Sônia Gomes, Moisés Patrício, Paulo Nazareth, and Tiago Martins de Melo, among others. The intense agenda of celebrations was not restricted to São Paulo but took place in The term “antropófago,” of Greek origin, is constituted by the juxtaposition of the words antropos (man) and phagos (to eat).

1

35

Anthropophagic Appropriation and Intermediality

901

numerous cities throughout Brazil and abroad. Among the events outside Brazil, the University of Oxford’s Brazil Week: 100 Years of Modernism in Brazil (held from January 31 to February 4, 2022) is worth mentioning, since it included a series of lectures discussing the legacy of Brazilian Modernism then and now.2 The present chapter focuses on the importance of the concept of cultural anthropophagy, as formulated by the Brazilian Modernist poet and thinker Oswald de Andrade in his Manifesto Antropófago, and on the insertion of the notion of anthropophagic appropriation into the broader fields of intertextuality and intermediality, offering an in-depth investigation on various issues related to the concept from historical and cross-cultural perspectives. In the first section, titled “Oswald de Andrade’s Concept of Cultural Anthropophagy: Historical Overview and Multiple Interpretations,” the complex theory of the Brazilian Modernist poet and thinker, based on the digestive metaphor of the Tupinambá Indians, which has been interpreted as a creative process for literary and artistic renewal, and as a critical and ideological discourse moving away from colonial essentialist, hegemonic univocality, will be discussed, showing that it has been devoured and digested differently by different theorists in different historical and cultural contexts. The second part, “Local and Global Resonances of Cultural Anthropophagy: Intermedial Relations,” outlines the wide-reaching impact that Andrade’s Manifesto Antropófago had on various aspects of Brazilian culture and society, mainly on concrete poetry, installation art, translation studies, cultural critique, collaborative theater research, and curatorial projects. The recent international recognition of Andrade’s tenets across the Global South shall also be discussed in this part. In the third section, “Cultural Anthropophagy as Part of the Broader Fields of Intertextuality and Intermediality,” I argue that anthropophagic appropriation can be seen as a specific form of non-subaltern intertextuality and that it has been approached as a creative process of transmediation, a mode of media transformation that can be located within the broader scholarly field of intermediality.

Oswald de Andrade’s Concept of Cultural Anthropophagy: Historical Overview and Multiple Interpretations Like most theoretical concepts, Andrade’s notion of cultural anthropophagy is controversial, having been understood and approached from a multiplicity of perspectives. Despite the massive critical attention that the term has received over time, it remains a search concept, as Brazilian scholar Evando Nascimento has aptly put it: It is impossible to confer a strictly conceptual treatment to anthropophagy. Above all, it is inappropriate to simply consider it as another trend of Brazilian Modernism and its 2

Available at https://www.mod-langs.ox.ac.uk/events/2022/01/31/brazil-week-semana-de-artemoderna-100-years-modernism-brazil

902

A. S. Camati

reverberations after the 1922 Modern Art Week. These two arguments inevitably include a third: a single and unequivocal definition of anthropophagy has not yet been worked out in past or present, and probably will never be formulated in future time. [. . .] Trying to define it would lead to an immediate loss of its most important premises: namely, of being one of the guiding principles of Brazilian literature and culture in the 20th century. (Nascimento 2011: 333–334)3

Nascimento further points out that Andrade envisioned “the need to think about European culture installed in Brazilian territory in a non-colonized way. [. . .] through a provocative attitude, adopting a true ‘shock rhetoric’, the Brazilian intelligentsia was invited to rethink the way of relating to the former colonizer, the European [. . .]” (Nascimento 2011: 334). As mentioned above, Andrade rejected the idea that meaningful theoretical production derives exclusively from the Northern Hemisphere, claiming the right of the Brazilian intelligentsia to dialogue with Eurocentric traditions without subservience. While considering alterity a positive value, he sought an alternative place of enunciation which would allow the relativization of Europe as being the central intellectual space of modernity. “His cultural policy was directed against the colonialist mentality of passive acceptance of the values generated by Western civilization, which had been de-historicized and legitimized as universal to sustain colonialist enterprises” (Figueiredo 2011: 290). To gain deeper insight into cultural anthropophagy, as theorized in the Manifesto Antropófago, it is important to realize that this fragmented, thoughtprovoking text reflects the concerns of Andrade’s lifelong aspiration to build an autonomous Brazilian culture, an ideal that dates back to the period of Romanticism. However, different from Romantic thinkers, Andrade endeavored to overcome the excessive idealism and patriotism of the Romantic period. In this regard, he defended a combination of nationalist sentiment and cosmopolitanism, choosing hybridity instead of the mutually exclusive categories current in Brazil at the time which, “on the one hand, embraced the defense of an essentialist nationalism, and, on the other, insisted on a modernizing universalism that implied complete submission to European cultural models” (Figueiredo 2011: 389). In his studies on the genealogy of cultural anthropophagy, João Cezar de Castro Rocha, one of the most prolific scholars addressing the Manifesto Antropófago in Brazil, traces the lineage of the concept through three fundamental historical moments or phases in Brazilian intellectual history, namely, Romanticism, Modernism, and Tropicalism, to highlight the dense web of offshoots that characterize the Brazilian posture of creative and critical appropriation of the contribution of the other without taking on a subaltern position (Castro Rocha 2011). The first phase dates back to José de Alencar (1829–1877), one of the main representatives of Brazilian Romanticism, who was the creator of Indianist litera-

3

Translations from Brazilian Portuguese into English are mine, unless otherwise indicated.

35

Anthropophagic Appropriation and Intermediality

903

ture, a movement that idealizes indigenous identities of the “good savage.” In the novels Iracema (anagram for America) and O guarani, Alencar developed ideas which inaugurate a false “founding myth” for Brazil, which insists on the peaceful interaction and harmonious miscegenation between colonizers and indigenous tribes. Nevertheless, as Castro Rocha reports, in the novel Ubirajara (1874), Alencar anticipates ideas later developed by Oswald de Andrade, when, on the one hand, he created an alternative Indianist narrative based on an anthropophagic reinterpretation of the appalling reports of colonial chroniclers (such as Hans Staden, Jean de Léry, and others) and, on the other, “recovered anthropophagy as an eminently positive idea-force, thus subverting prevalent beliefs” (Castro Rocha 2011: 650). As regards the second phase, the notion of anthropophagy was resumed and reconfigured from a dialectical standpoint in the 1920s. Castro Rocha asserts that “this dialectic method was necessary, because the European vanguards furnished the initial model for the emergence of Brazilian Modernism, since the artistic techniques adopted by the modernists were, initially, derived from European avant-garde artists” (Castro Rocha 2011: 650–651). Brazilian Modernism, under the leadership of the Grupo dos Cinco (Group of Five), which included Oswald and Mário de Andrade, Menotti del Picchia, Anita Malfatti, and Tarsila do Amaral, developed in the first half of the twentieth century. Oswald de Andrade (1890–1954), poet, playwright, and polemicist, was a key figure within São Paulo’s vibrant avant-garde movement. Born into a wealthy family, he traveled extensively in Europe during his youth and had first-hand experience in the vanguard theories emerging in France and Italy. Back in Brazil, at first in close collaboration with his brother Mário de Andrade (1893–1945), he focused his writings on a quest for valuing the uniqueness of Brazilian cultural expression. Considering that intellectual colonialism continued to prevail in Brazil in 1922, 100 years after its independence in 1822, Andrade’s first literary statement of beliefs, the Manifesto da Poesia Pau-Brasil (Brazilian Pau-Brazil Poetry Manifesto), published in 1924, proposed that, besides the devouring, deglutition, and transmutation of the main ingredients of the European vanguards, such as aggressiveness, disruption, intertextuality, collage, parody, and fragmentation, Pau-Brazil Poetry for export4 must relate and incorporate the tropical plurality of Amerindian ancestral traditions, a culture purposely wiped out by the colonizer. Eager to reconcile his country with its own past, Andrade predicated the necessity of “native originality to neutralize academic conformity” (Andrade 1986: 187), denouncing the repressive nature of colonialism to remind his contemporaries that Brazil had been traumatized by colonial constraint and conditioning.

4 Pau-Brazil Poetry for export ironically refers to the Pau Brasil tree [palo-del-Brasil – Brazilian wood], which was one of the main exports during the period of Portuguese colonization.

904

A. S. Camati

Four years later, inspired by Tarsila do Amaral’s painting Abaporu (The man who eats people),5 Andrade formulates the concept of cultural anthropophagy in his Manifesto Antropófago, advancing the conceptual foundations of a Brazilian vanguard theory, “original” and innovative. He derived the notion of anthropophagy from the rituals of the Tupinambá Indians, one of the native peoples living in Brazilian coastal regions at the time of the sixteenth-century colonial explorations, who were reported to devour their enemies to appropriate their strength, intelligence, and power. In his statement of principles, translated as Cannibalist Manifesto by Leslie Bary (1991: 35–47), Andrade, while adopting the metaphor of anthropophagy as his guiding principle, encourages the ritual consumption of foreign cultural capital mixed with native Brazilian ingredients, as an alternative creative process for Brazilian literary and other artistic experimentalisms, since this act of ingestion and digestion allows the incorporation into the self of what is most excellent in the other, generating something new and distinctly Brazilian. In Oswald canibal (1979), Benedito Nunes, one of the first Brazilian theorists to discuss Andrade’s dialogical attitude, conveys that although anthropophagy, a notion closely tied to primitivism, constituted the point of convergence between the European avant-garde movements (among them Futurism, Dadaism, Surrealism, and Cubism) and Brazilian Modernism, the ideas and procedures of the vanguards were reassessed by Andrade when he proposed the devouring, deglutition, and transmutation of foreign models to construct independent conceptual foundations for Brazilian art and literature. Nunes points out that Michel de Montaigne’s essay “On Cannibals,” Levy-Brühl’s theory of primitive mentality, Freud’s views in Totem and Taboo, Keyserling’s ideas on the tecnicized barbarian, and Nietzsche’s assumption that “the conscience of man without resentment is equivalent to his physiological capacity of well digesting” (Nunes 1979: 18) marked the intellectual formation of Oswald de Andrade expressed in the Manifesto. Beatriz Azevedo addresses the Manifesto Antropófago as aesthetic theory and worldview in Antropofagia – Palimpsesto Selvagem [Anthropophagy – Savage Palimpsest], working out a minute exegesis of the 51 aphoristic fragments or axioms that constitute Andrade’s audacious experiment, uncovering some of the multiple meanings that each aphorism engenders. She sustains that in the fragmented sequence of axioms, other literary and philosophical aphorisms are devoured, digested, and reconfigured, since Andrade inscribes “the praxis of his theory into

5

Oswald de Andrade’s insights, developed in his Manifesto Antropófago, emerged after his wife Tarsila do Amaral (1889–1973), one of the leading figures of the Brazilian Modernist visual arts, gave him an oil painting on canvas as a birthday present. Andrade denominated the painting Abaporu (1828), in Tupi-Guarani “abapor’u” – abá (man) + poro (people) + ‘u (to eat) – “the man who eats people.” His fascination with the painting is expressed in the text of the Manifesto illustrated by a line drawing of his wife’s painting, which was published in the first edition of the Revista de Antropofagia (1928). Tarsila do Amaral, who studied in Paris with André Lhote (1885–1962) and Fernand Léger (1881–1955), had learned to devour modern styles of European painting, such as Cubism and others, and to digest them in an anthropophagic manner to produce unique art works.

35

Anthropophagic Appropriation and Intermediality

905

the textual body of the Manifesto, transfiguring anthropophagous taboos into sacred totems” (Azevedo 2018: 101). Hereafter, a reduced number of aphorisms, minutely discussed by Azevedo, shall be briefly looked over to provide insight into some aspects of Andrade’s vanguard theory. In the first aphorism “Cannibalism alone unites us. Socially. Economically. Philosophically” (Andrade, transl. Bary 1991: 38),6 Azevedo reports, Marx and Engels’ 1848 Communist Manifesto is evoked, which ends with the political slogan, translated literally as “Proletarians of all countries, unite!”. This catchphrase is contradicted by Andrade’s assertion that anthropophagy unites us, since the practice of cannibalizing features of the other is a universal tendency that pulls people together globally, for there is a human predilection to ingest, digest, and regurgitate previous human knowledge. Andrade appropriates the word “unite” to convey that “every ending is a starting point, hinting at the cyclical and inconclusive nature of all issues” (Azevedo 2018: 104). The second aphorism, which contends that cultural anthropophagy constitutes the guiding philosophical principle of existence, and that devouring is “The world’s single law. Disguised expression of all individualisms, of all collectivisms. Of all religions. Of all peace treaties” (38),7 suggests that, to some extent, all societies tend to cannibalize aspects of each other. In 1950, Andrade resumes this principle in A crise da filosofia messiânica (The Crisis of Messianic Philosophy), reiterating that “life is devouring, and anthropophagy is a cosmovision centered on ideals of freedom” (Azevedo 2018: 109). Andrade’s parodic rearticulation of Hamlet’s famous dictum, “To be or not to be – that is the question” (3.1.55) into “Tupi, or not Tupi that is the question” (38), expressed in the third aphorism, marks a radical departure from the oppositional binary relations of self and other, conveying that his intention is not to replace one sign with the other, but let them coexist in a radical new interactive relationship, consequently transforming each other, and becoming part of a mutually dependent and mutually empowering act. Andrade’s exchange of “to be” for “Tupi” constitutes “a substitution of Hamlet’s metaphysical anxiety (of patriarchal bias) into an anthropophagic primitive perspective (of matriarchal bias)” (Azevedo 2018: 110). In aphorism 4, Andrade postulates “Down with every catechism. And down with Gracchi’s mother” (38),8 disclaiming the “patriarchal mother” and discrediting the

6

Quotes of aphorisms from Andrade’s Manifesto Antropófago appear in English in the body of the text, selected from the 1991 annotated translation by Leslie Bary. Henceforth, such quotes will be marked only by page numbers. The “original” Brazilian Portuguese version of the axioms will be provided in footnotes. Aphorism 1: “Só a Antropofagia nos salva. Socialmente. Economicamente. Filosoficamente” (Andrade 1928: 3). 7 Aphorism 2: “Única lei do mundo. Expressão mascarada de todos os individualismos, de todos os coletivismos. De todas as religiões. De todos os tratados de paz” (Andrade 1928: 3). 8 Aphorism 4: “Contra todas as catequeses. E contra a mãe do Graccos” (Andrade 1928: 3). In her notes, Leslie Bary explains that, in the Manifesto, Graccos’ mother relates to “the bad mother who (in contrast to the mother-goddesses Jaci and Guaraci) brings her children up as subjects of a ‘civilized’ culture” (Bary 1991: 44).

906

A. S. Camati

myth of the European colonial mission which, by means of violence, was supposed to bring civilization and light to barbarian or primitive people, saving them from darkness. Azevedo explains that the Gracchi’s mother mentioned in the axiom was Cornelia, a prominent Roman matron, “mother of Tiberius (160-33 aC) and Gaius (152-21 aC), who provided them with a strict education and an extremely demanding moral [. . .]” (Azevedo 2018: 112), which constituted attributes Andrade associated with the modes of catechism of the Occidental tradition. To approach modernity, Andrade adopts a new angle of vision directed at the American continent. In aphorism 11, he sustains that “We want the Carib Revolution. Greater than the French Revolution. The unification of all productive revolts for the progress of humanity. Without us, Europe would not even have its meager declaration of the rights of man” (39).9 He is interested in the discontinuity or decentering of history, sustaining the utopian affirmation that the New World, a place of alterity, was instrumental in shaking certainties and suggests alternative worldviews. This decentered gaze contrasts with discourses that assert European superiority. It reveals another face of modernity, opposing Eurocentric discourse and practices, establishing a kind of resistance to European cultural hegemony. In Andrade’s utopian vision, the indigenous people of the Americas “displayed a rich culture, of matriarchal disposition, who lived its heyday in a golden age” (Azevedo 2018: 128). In aphorism 12, Andrade points out the foreign models which he appropriated and reassessed from an anthropophagic perspective “Heritage. Contact with the Carib side of Brazil. [. . .] Montaigne. Natural man. Rousseau. From the French Revolution to Romanticism, to the Bolshevik Revolution, to the Surrealist Revolution and Keyserling’s tecnicized barbarian. We push onward” (39).10 The idea that ritual cannibalism is less barbarous than many European customs considered civilized, set forth by Montaigne (1533–1592) in his essay “Of Cannibals” (c. 1580), appealed to Andrade, and the notion of the “tecnicized barbarian” developed by the German philosopher and Orientalist Hermann von Keyserling (1880–1946) suited his utopian project. He asserts that the emergence of the barbarian is necessary to express Brazilianness, “but not from the perspective of the past, but engaged in a movement looking forward to the future” (Azevedo 2018: 130). The third seminal moment that exerted powerful influence on Brazilian literature and culture in the twentieth century, mentioned by Castro Rocha, is Tropicalism, a cultural movement that emerged in the 1960s and 1970s, at the Teatro Oficina in São Paulo, during Brazil’s dictatorship period. It was not just an artistic movement but also a mode of political protest to express rebellion against dictatorship which had taken power in 1964. Castro Rocha reiterates that Tropicalism “was deeply influenced by Modernism and, above all, by the national/cosmopolitan dialectic Aphorism 11: “Queremos a revolução Carahiba. Maior do que a revolução Francesa. A unificação de todas as revoltas eficazes na direção do homem. Sem nós, a Europa não teria sequer a sua pobre declaração dos direitos do homem” (Andrade 1928: 3). 10 Aphorism 12: “Filiação. O contato com o Brasil Caraíba. [. . .] Montaigne. O homem natural. Rousseau. Da Revolução Francesa ao Romantismo, à Revolução Bolchevista, à Revolução Surrealista, e ao bárbaro tecnizado de Keyserling. Caminhamos” (Andrade 1928: 3). 9

35

Anthropophagic Appropriation and Intermediality

907

that constitutes the axis of the Manifesto Antropófago [. . .]” (Castro Rocha 2011: 652). The initial inspiration for Tropicalism was Andrade’s Manifesto Antropófago, which also served as springboard for other aesthetic innovations, such as concrete poetry, installation art, cannibalistic translation, and Cinema Novo (New Cinema). José Celso Martinez Corrêa (b. 1937), known as Zé Celso, actor, political activist, and theater director of the Teatro Oficina, was one of the first avant-garde artists to rely on the concept of anthropophagy. He revolutionized and contributed to decolonizing Brazilian theater when, in 1967, he appropriated Oswald de Andrade’s anthropophagic agenda for staging O rei da vela (The Candle King), a revolutionary play-text written by Andrade in 1937, which continued to be relevant concerning the Brazilian cultural and political tensions in the 1960s. Zé Celso’s engagement with the tenets developed in Andrade’s Manifesto Antropófago represented a turning point in Brazilian theater and culture. His staging of O rei da vela in 1967 left an important legacy for the Brazilian stage and was the starting point and catalyst of the Tropicália movement. From its beginnings, Tropicalism or Tropicália was seen as a revitalization of cultural anthropophagy. The Tropicalist enterprise involved, in varying degrees, not only popular music but also the visual arts, theater, film, and literature. The movement is mainly associated with a group of musicians, notably Caetano Veloso (b. 1942), Gilberto Gil (b. 1942), Gal Costa (1945–2022), Tom Zé (b. 1936), and a small team of intellectuals and artists, among them Hélio Oiticica (1937–1980), Torquato Neto (1944–1972), and Rogério Duarte (1939–2016). Later Rogério Duprat (1932–2006) and the rock band Os Mutantes joined them as collaborators. Their intention was to create a revolution in musical expression in order to question the fundamental structures of Brazilian society and values. While drawing inspiration directly from Andrade’s Manifesto Antropófago (1928), from concrete poetry (1950–1970), and from Zé Celso’s production of Andrade’s O rei da vela (1967), the musical Tropicalists combined and merged the eclectic heritage of Brazilian and African rhythms (samba, funk, maracatu, etc.) with British and American psychedelic music (soul, blues) and pop rock. The lyrics of Tropicalist composers, besides presenting affinities with the procedures of concrete poetry, also relied on intellectual exchanges among foreign artistic manifestations emerging at the time.

Local and Global Resonances of Cultural Anthropophagy: Intermedial Relations The anthropophagic perspectives, born in Brazilian Modernism and rearticulated in Tropicalism, developed in a series of Brazilian literary and artistic manifestations after the 1950s, such as concrete poetry, installation art, cannibalistic translation, Cinema Novo, postcolonial literary criticism, collaborative theater research, and curatorial projects, among others. This part of the essay explores the impact that

908

A. S. Camati

Oswald de Andrade’s Manifesto Antropófago had on various aspects of Brazilian culture and society and surveys the international reverberations of his avant-garde aesthetic theory in terms of intermedial relations. In the 1950s, motivated by Andrade’s notion of cannibalistic appropriation of foreign cultural capital, three Brazilian poets living in São Paulo – Haroldo de Campos (1929–2003), Augusto de Campos (born 1931), and Décio Pignatari (1927–2012) – started formulating concepts for the creation of a new art form they denominated poesia concreta (concrete poetry). They chose the word Noigandres, borrowed from Ezra Pound’s Cantos (for the identification of the group and for nominating the four issues of a magazine they edited to disseminate their ideas) because the enigmatic name suited their purposes to move across medial borders, freely transiting between different arts forms and media. Claus Clüver has published over 20 essays on concrete poetry and related intermedial products, most of them concerned with the work of the Brazilian Noigandres group. In one of his early reflections, he highlights that, in 1958, the group launched a manifesto entitled plano piloto para poesia concreta (Pilot Plan for Concrete Poetry) in the fourth issue of the Noigandres magazine, to articulate the main ideas disseminated in their earlier publications and to reflect on the plurimedial and multimodal nature of their poetic program: They derived much of the justification of their project from a critical analysis of the work of those (predominantly foreign) writers whom they had chosen as their literary models and whom they introduced to the Brazilian public in their own translations. Many of their ideas were developed from an engagement with constructivist painting and with the techniques of serial music and filmic montage. For theoretical support, these young poetae docti turned to new publications in the fields of general semantics, linguistics, aesthetics, information and communication theory, cybernetics, phenomenology, and Gestalt psychology. (1987: 01)

In an article, titled “Concrete Poetry and the New Performance Arts: Intersemiotic, Intermedial, Intercultural,” Clüver explains that the Noigandres poets adopted the portmanteau word “verbivocovisual” to elicit the plurimedial character of their work. They relied on fragmentation, the displacement of the viewing subject, the defamiliarization of the familiar, the end of rhyme and meter, and its substitution by new structures based on the distribution of words according to geometric alignment, aiming at engaging the reader/viewer in a participatory experience involving spatial, temporal, and material relationships. In their search for a new form of articulating poetic expression, they concentrated on the materiality of language, mainly on its sonorous (musical) and graphic (visual) elements (Clüver 2000). To shed light on the creative processes of concrete poetry, which relies on elements of several sign systems, Clüver categorizes this art form as a syncretic or intermedia construct, which he distinguishes from multimedia or mixed-media texts: “A Concrete poem is always an intersemiotic and intermedia text, i.e., a text that draws on two or more sign systems and/or media in such a way that the visual and/or musical, verbal, kinetic, or performance aspects of its signs become inseparable” (Clüver 2000: 19).

35

Anthropophagic Appropriation and Intermediality

909

In Clüver’s view, the Noigandres group’s Teoria da poesia concreta (Theory of Concrete Poetry) “surpasses in range and acuity any other theory of Concrete poetry.” He further underscores that “the international discussion about the genre and the movement would have gained considerably had those texts, as well as a good deal of the later critical work of the three, been available in English, French or German translation” (1987: 01). In 1968, the musical Tropicalists produced the Album Manifesto of the movement, titled Tropicália ou Panis et Circencis. The term Tropicália, which Caetano Veloso had adopted for his own revolutionary platform song in 1967, was first coined by the Brazilian multimedia artist Hélio Oiticica – sculptor, painter, writer, theorist, performance artist, and filmmaker – for his most influential installation called Tropicália, which was his contribution to the art exhibition Nova Objetividade Brasileira (New Brazilian Objectivity) at MAM (Modern Art Museum) in Rio de Janeiro in 1967. Oiticica is known internationally for his leadership and activism, together with Lygia Clark (1920–1988) and Lygia Pape (1927–2004), in the Neo-Concrete Movement (1959–1961), a Brazilian art tendency which emerged as a reaction to the Concrete Art Movement prevalent at the time in Europe and Latin America. The Neo-Concrete artists rejected rationalistic perspectives and embraced Conceptual Art. The complex media art installation Tropicália, which relates to Oiticica’s innovative “environmental art,” is an intermedial/multimodal experiment involving the creation of a multisensorial interactive space that makes the spectator acutely aware of existence, immediately raising connections with the local context. It was composed of two Penetrables (artifacts you can move into or through), which evoked the precarious architecture of the favelas (slum dwellings in Rio de Janeiro). The material employed was wood, printed cloth, plastic, straw, as well as sand, tropical plants, pebbles, poems written on wood or tiles, and live parrots. In the larger Penetrable, there was a television set, connected to a local channel, placed at the end of its labyrinthine structure, which would be reached by the visitor after entering the structure. The other Penetrable was a simple structure with an open side, painted red in its interior with the written inscription “Purity is a Myth.” On one level, the installation put forward an ironic take on the idea of Brazil as a tropical paradise, suggested by the sand, plants, and parrots. But during the process of penetrating the installation, the web of sensory images produce an intensely intimate confrontation, specially perhaps when the viewer is faced with the innermost image of all, namely the universal switched-on TV set in pitch darkness. Ironically, Oiticica questions the interpretations of Brazil as a tropical paradise. Visitors were invited to walk barefoot in the sand and freely interact with the installation. While making art participatory and inclusive, Oiticica inaugurated a tendency that has inspired visual artists around the world (Hermann 2010). As made evident in the description above, Oiticica’s experiment is a media product that utilizes multimodal interface procedures, engaging the visitors with the material, sensorial, spatiotemporal, and semiotic aspects of his installation (Elleström 2010: 11–48; Bruhn and Schirrmacher 2022: 19–24). Since the early 1960s, the main representatives of Brazilian concrete poetry, Haroldo de Campos and his brother Augusto de Campos, had also been working

910

A. S. Camati

on a Brazilian postcolonial theory of cannibalistic translation. They developed a transcultural, rather than an acculturative, approach, rejecting fidelity practices. The neologisms coined by Haroldo de Campos for his vanguardist poetics of translation, such as “transcreation,” “transtextualization,” “transluciferation” (translation of Goethe’s Faust), “transparadisation” (translation of Dante’s Paradise), and “transhelenization” (translation of Homer’s Iliad), were inspired by Andrade’s anthropophagic view. Later, in the 1980s, in an essay entitled “The Rule of Anthropophagy: Europe Under the Sign of Devoration,” Haroldo de Campos stated that Andrade’s concept of cultural anthropophagy or anthropophagic appropriation encouraged Brazilian artists and writers to value the national in dialogical/dialectical relationship with the universal. Additionally, he expressed his conviction that cultural anthropophagy, as metaphor and methodology, is not limited to a Brazilian context, but potentially present in other cultures as well. In the essay mentioned, Campos defined anthropophagic appropriation as: [. . .] the thought of critical devouring of the universal cultural heritage, formulated not from the insipid, resigned perspective of the ‘noble savage’. . . but from the point of view of the ‘bad savage’, devourer of whites, the cannibal. The latter view does not involve a submission (an indoctrination), but a transculturation, or, better, a ‘transvaluation’: a critical view of History as a negative function (in Nietzsche’s sense of the term), capable of appropriation and of expropriation, de-hierarchization, deconstruction. Any past which is ‘other’ for us deserves to be negated, we could say it deserves to be eaten, devoured. With this clarification and specification: the cannibal was a polemicist (from the Greek polemos), but he was also an ‘anthologist’: he devoured only the enemies he considered strong, to take from them the marrow and protein to fortify and renew his own natural energies. (Campos 1986: 44)

As conveyed in the passage quoted above, the anthropophagic view of translation relates to the notion of deconstruction, proposed by Jacques Derrida during the 1960s. The translated text becomes a new, “original” creation in its own right. Considering that the translator absorbs the strength of the source text and combines it with native input, the target text is energized and revitalized in cultural and linguistic terms. In this way, Brazil found a way to assume a position against European, postcolonial domination, moving away from essentialist approaches and ideologies. Based on cultural anthropophagy, the Brazilian filmmaker Glauber Rocha (1939–1941) proposed a new aesthetic agenda for Brazilian cinema, inaugurating a tendency later denominated Cinema Novo (New Cinema). Opposing the industrial European classic cinema, he encouraged Brazilian filmmakers to operate with low production costs. He was concerned with political and social problems, aiming at denouncing current problems such as hunger, misery, and insalubrity, which until then were not represented realistically by contemporary Brazilian filmmakers. In 1965, as a mechanism for reaffirming the purposes of Cinema Novo and as a way of justifying the aesthetics and resources employed by the representatives of the movement, Glauber Rocha wrote Eztétyka da fome [An Aesthetics of Hunger], his first film manifesto of major impact abroad, which constituted not only a refusal of

35

Anthropophagic Appropriation and Intermediality

911

Western colonizing forms but also an epistemology of the oppressed. In this statement of beliefs, he insisted that, instead of struggling to gain access to a film industry out of reach, Brazilian and Latin American moviemakers should forge their own aesthetics out of the very lack of resources, highlighting hunger as subject and formal principle. He wrote that: [. . .] hunger in Latin America is not simply an alarming symptom, it is the essence of our Society. Herein lies the tragic originality of Cinema Novo in relation to world cinema. Our originality is our hunger, and our greatest misery is that this hunger is felt but not intellectually understood. (Rocha 1995: 68)

Glauber Rocha’s political film manifesto and his movies Deus e o diabo na terra do sol [Black God, White Devil] (1964) and Terra em transe [Entranced Earth] (1967) generated heated debates in Brazil, mobilizing representatives of popular music, theater, and the visual arts, as well as writers and political activists. In the 1960s and 1970s, Silviano Santiago (b. 1936), Brazilian fiction writer, essayist, and literary critic, contributes to the cultural anthropophagy debate, forwarding ideas which comply with several perspectives of Oswald de Andrade’s Manifesto Antropófago. His influential essay “O entre-lugar do discurso latinoamericano” (Latin-American Discourse: The Space In-Between), published in Uma literatura nos trópicos (Literature in the Tropics) in 1978, was originally written in French in 1971, titled “L’entre-lieu du discours latino-américain.” In 2001, the English translation appeared in the anthology The Space In-Between: Essays on Latin American Culture, edited by Ana Lúcia Gazzola. In the introduction, Gazzola and Wander Melo Miranda report that the notions of “hybridity” and “space in-between” have been so completely integrated into contemporary critical and cultural theory that scholars fail to realize that these concepts had been first developed by the Brazilian writer.11 They argue that: Among present-day Brazilian critics, Silviano Santiago occupies a unique place owing to his pioneering development of concepts that have nowadays become current coin in Brazilian and international criticism. Developed in several of his writings since the end of the 1960s and the beginning of the 1970s, concepts such as space in-between and hybridism are central for an innovating thought that managed to articulate from a Brazilian and Latin American perspective the strategic relation between notions of dependency and universality. (Gazzola and Miranda 2001: 1)

In his groundbreaking essay, “O entre-lugar do discurso latino-americano,” Santiago deconstructs the binary relations of center/periphery and original/copy, rejecting notions of hierarchy and hegemony:

While the conceptual vocabulary of “hybridity” and “third space” has been attributed to Homi Bhabha (b. 1936), who elaborates these concepts in The Location of Culture (1994), it is worth acknowledging that the term “space in-between” was coined by Silviano Santiago, who developed the concept by the end of the 1960s and beginning of the 1970s.

11

912

A. S. Camati

The major contribution of Latin America to Western culture is to be found in its systematic destruction of the concepts of unity and purity: these two concepts lose the precise contours of their meaning, they lose their crushing weight, their sign of cultural superiority, and do so to such an extent that the contaminating labor of Latin Americans affirms itself as it becomes more and more effective. Latin America establishes its place on the map of Western civilization by actively and destructively diverting the European norm and re-signifying pre-established and immutable elements that were exported to the New World by the Europeans. (Santiago 2001: 30–31)

The issue of artistic creation is approached from a perspective that evades traditional studies of source and influence. As Santiago has aptly put it, the anthropophagous ritual of Latin American discourse is constructed in the space in-between “sacrifice and playfulness, prison and transgression, obedience and rebellion, submission to the code and aggression, assimilation and expression” (Santiago 2001: 38). In this sense, the decolonized appropriation of foreign cultural capital in Latin America displays immense richness and energy since it contains within itself both a representation of European sources and a response to them. Since the last decade of the twentieth century and after, the concepts of anthropophagy and hybridity mark most of the work of Brazilian theater practitioners operating on a collaborative basis.12 Critical analyses of their anthropophagic adaptation/appropriation processes to stage classical or contemporary theater texts have brought into sharp focus studies in a wider-ranging context of media interrelations theorized by Irina Rajewsky, Lars Elleström, and Jorgen Bruhn, among others. In 1993, Zé Celso presented a cannibalized “medial transposition (Medienwechsel), also referred to as medial transformation” (Rajewsky 2010: 55) of Shakespeare’s Hamlet, retitled Ham-let, at the Teatro Oficina. He reinterpreted Hamlet in a highly anthropophagic fashion, producing, as he himself has put it, Shakespeare in Tupi (referring to the Tupi language, spoken in Brazilian coastal regions during the early colonial period by the aboriginal Tupi Indians, people who belong to the Tupi-Guarani family branch). In an interview for Folha de S. Paulo (1994), Zé Celso claimed that when Shakespeare’s plays arrived in Brazil, the country of anthropophagy, they necessarily gained new life and impetus in the local habitat. The category of media combination (Medienkombination), defined by Rajewsky as a subcategory of intermediality: [. . .] includes phenomena such as opera, film, theater, performances, illuminated manuscripts, computer or Sound Art installations, comics, and so on, or, to use another terminology, so-called multimedia, mixed media, and intermedia. The intermedial quality of this

12

Collaborative Brazilian theater groups include Zé Celso and Teatro Oficina, Antunes Filho and Centro de Pesquisa Teatral, Gerald Thomas and the Dry Opera Company, Antônio Araújo and Teatro da Vertigem, Enrique Diaz and Cia dos Atores, Sérgio Carvalho and Companhia do Latão, Eduardo Moreira and Grupo Galpão, Marcio Meirelles and Bando de Teatro Olodum, and Guti Fraga and Nós do Morro, among many others.

35

Anthropophagic Appropriation and Intermediality

913

category is determined by the medial constellation constituting a given media product, which is to say the result of the very process of combining at least two conventionally distinct media or medial forms of articulation. These two media or medial forms of articulation are each present in their own materiality and contribute to the constitution and signification of the entire product in their own specific way. (Rajewsky 2005: 51–52)

Like Andrade, Zé Celso professed total freedom of expression. By adopting a hybrid performance aesthetics which combined and fused multiple theater expedients and qualified media, such as music, dance, circus techniques, slapstick, carnivalization in the Bakhtinian sense, theatricality inspired by the American Living Theatre, as well as strategies developed by Brecht and Artaud, reconfigured in the light of the Manifesto Antropófago, he explored the fundamentally plurimedial structure of the theater. The tenets of Oswald de Andrade’s Manifesto continue to inform the performance aesthetics of Brazilian group theater to date. In 2004, the anthropophagic experiment developed by the theater ensemble Cia. dos Atores, titled Ensaio. Hamlet (Rehearsal. Hamlet), directed by Enrique Diaz, featured global and local issues emerging in the twenty-first century expressed through postdramatic theatrical signs, such as parodic play, emphasis on theatricality and performativity, alternation of narration and performance, non-hierarchical use of signs, division of stage-time into minimal sequences or quasi-filmic takes, rhizomatic connections of heterogeneous components, musicalization, physicality, and irruption of the real (Lehmann 2006). Specific topics present in Hamlet, such as power, corruption, incest, fratricide, and madness, were selected, explored, expanded, and cannibalized by the actors/performers, who used Shakespeare’s tragedy as material to create new scenes pregnant with contemporary signifiers. In a personal pronouncement, Diaz claimed that he viewed the production as “an anarchical and anthropophagic re-reading” (Santos 2004) of Shakespeare’s Hamlet. In 1998, cultural anthropophagy was chosen as the curatorial theme by Paulo Herkenhoff, the organizer of the 24ª Bienal de São Paulo (24th São Paulo Biennial), an event that sought to locate Andrade’s anthropophagic theory in contemporary art tendencies and media products, among them Brazilian Anti-Art (inaugurated by Lygia Clark in the 1960s), whose main contemporary representatives are Antonio Manoel, Artur Barrio, and Cildo Meirelles. Herkenhoff’s intention was to present a non-Eurocentric version of art history in the most important exhibition in South America and thereby put an end to the lingering implication that artists in Brazil and other non-European countries still needed to catch up with the latest artistic trends. The 24ª Bienal de São Paulo employed the Brazilian notion of anthropophagy as both concept and methodology, encouraging contamination and cannibalization of the canon. By doing so, it proposed a new model for large-scale curatorial projects that could effectively address non-specialist audiences (Lafuente et al. 2015). As predicted by Haroldo de Campos in the 1980s, Andrade’s notion of anthropophagic appropriation has recently been valued and adopted for its rich potential for subversion and renewal in the non-traditional staging contexts of the Global South, not only in Brazil but also in numerous locations across the Indian and

914

A. S. Camati

the Atlantic Ocean worlds. In Shakespeare in the Global South: Stories of Oceans Crossed in Contemporary Adaptation (2021), Sandra Young argues that Andrade’s Manifesto Antropófago: [. . .] functions as a riposte to coloniality as it continued to structure sociality in Brazil in the twentieth century. The Manifesto asserts with relish the right of Brazilians to consume the cultural forms they have encountered and to make them their own entirely, a mode of cultural appropriation that has powerful resonance for Global Shakespeare. Cultural anthropophagy offers a compelling example of insurgent possibilities of southern theory. (Young 2021: 129)

Similarly, the collection Eating Shakespeare: Cultural Anthropophagy as Global Methodology (2021) highlights the global importance and the transformative possibilities of Andrade’s Manifesto. In the introduction of the book, the editors Anne Sophie Refskou, Marcel Alvaro de Amorim, and Vinicius Mariano de Carvalho underscore that cultural anthropophagy has become an important alternative point of view and methodology within contemporary theater and performance studies. They argue that, in Amerindian cosmology, the act of “devouring” to incorporate otherness is never unilateral: As the Tupinambá ate their enemies, and kept them alive in their own bodies, they knew that they too would be eaten if captured by the enemy. The violation of their being would ensure their survival, which helps Cultural Anthropophagy to formulate how one may deconstruct an(other) cultural sign – even violently so – but without negation. [. . .] signs are not replaced but coexist in the creation of new meaning. The seed of this relation is again to be found in Andrade’s eating of Shakespeare in the ‘Cannibalist Manifesto’. It seems that Andrade’s intent in appropriating ‘To be or not to be’, to create ‘Tupi or not Tupi’, is not to replace one with the other. Instead, he lets signs (and sounds) coexist in a radical new relationship, concomitantly transforming themselves and each other: Shakespeare and the Tupi become part of a mutually dependent and mutually empowering performative act. (Refskou et al. 2021: 10)

In the introduction of the book, the editors elucidate that among the Tupinambá Indians, the practice of anthropophagy “was not a dietary condition, nor a form of religious sacrifice, nor, strictly speaking, an act of revenge. Instead, it was a complex physical and metaphysical way of incorporating otherness. [. . .] In other words, the cannibalist ritual demonstrates signs of being fundamentally inclusive rather than exclusive, even to the point of significantly blurring the distinction between ‘self’ and ‘other’” (Refskou et al. 2021: 4–5), thus shedding light on fundamental aspects of cultural anthropophagy.

Cultural Anthropophagy as Part of the Broader Fields of Intertextuality and Intermediality The previous sections have familiarized readers with the different textual-discursive processes and practices entailed by anthropophagic appropriation, showing that cultural anthropophagy is in itself a vast field of transdisciplinary research that has

35

Anthropophagic Appropriation and Intermediality

915

been expanding ever since the publication of Oswald de Andrade’s Manifesto Antropófago in 1928. Considering that the concepts of adaptation and appropriation have recently been placed into the wider-ranging context of intertextuality (Julie Sanders 2006), and that the notion of adaptation has been positioned within the broader frame of intermediality (Lars Elleström 2017), in the present section, I argue that anthropophagic appropriation can be seen as a specific form of non-subaltern intertextuality and as a creative process of transmediation, a mode of media transformation located within the broader scholarly field of intermediality. Adaptation and appropriation have been approached from dissimilar and, at times, contradictory theoretical positions by scholars associated with different critical perspectives, such as poststructuralism, postcolonialism, and postmodernism. Julie Sanders (2006) provides insight and an extended discussion on adaptation and appropriation which, as she has put it, tend to converge, in many respects, “with the critical and cultural movements of postmodernism and postcolonialism” (Sanders 2006: 18). She argues that adaptation and appropriation “are in many respects a sub-section of the over-arching practice of intertextuality” (17), contending that “there are many ways in which both the practice and the effects of adaptation and appropriation intersect and interrelate, yet it is equally important to maintain some clear distinctions between them as creative activities” (26). Sanders assumes that while “an adaptation signals a relationship with an informing source-text or original,” it remains ostensibly a specific version of its source “albeit achieved in alternative temporal and generic modes” (26), hence preserving the canon instead challenging it. Conversely, she sustains that appropriation displays a more subversive character, questioning and confronting the ideologies and values of its source texts, frequently pursuing “a more decisive journey away from the informing source into a wholly new cultural product or domain” (26). In A Theory of Adaptation (2006), Linda Hutcheon extended the field of adaptation studies, including in its range not only literature and film but also opera, theater, theme parks, and other medial configurations. She turns adaptation into an umbrella concept, including in its scope other modes and methods of creativity, among them appropriation and transculturation. Hutcheon approaches the phenomenon of adaptation from three interrelated perspectives: as “an acknowledged transposition of a recognizable other work or works” into a new medial product; as “a creative and an interpretive act of appropriation/salvaging” or process of creation; and as “an extended intertextual engagement with the adapted work” or process of reception (Hutcheon 2006: 7–8). While Sanders inserts adaptation and appropriation into the broader field of intertextuality, Hutcheon extends the scope of adaptation including in its range a vast bulk of medial types and products. Although both Sanders and Hutcheon discuss intermedial relations between literary texts and other medial configurations in their theoretical frameworks, they never mention the notion of intermediality. Despite the fact that the concepts of adaptation and appropriation have been approached from different theoretical positions, it has been generally agreed that both terms can be associated with creative, dialogic processes of textual composition in the Bakhtinian sense, since all forms of discourse, without exception, are

916

A. S. Camati

re-appropriations of the discourse of (an)other. Mikhail Bakhtin (1895–1975) started to develop his theories on verbal creativity and textual dialogism in the Soviet Union in the 1920s, but his works were introduced into the West much later. His most influential book was first published in Russian in 1929, under the title Problems of Dostoevsky’s Creation, later revised and retitled as Problems of Dostoevsky’s Poetics when republished in Moscow in 1963. The first English translation appeared in 1973; however, as it is out of print, Caryl Emerson’s 1984 version is used for academic purposes. Inspired by the Bakhtinian concept of dialogism, theoretical perspectives on intertextual practices started to emerge in academic circles in the 1960s, after Julia Kristeva’s first mention of the term “intertextuality” in her essay “Word, Dialogue and Novel” (1980 [1966]), followed by developments in structuralist and poststructuralist criticism by Roland Barthes (1977 [1967]), Michel Foucault (1984 [1969]), and Gérard Genette (1997 [1982]), who set in motion mechanisms that led to the crisis of the concept of authorship. In the essay “The Death of the Author” (1977), Barthes argues that any text is an intertext, “a multi-dimensional space in which a variety of writings, none of them original, blend and clash. The text is a tissue of quotations drawn from the innumerable centers of culture” (Barthes 1977: 148). In a similar vein, Michel Foucault demystifies the concept of authorship in “What is an Author?” (1984), claiming that all “discourses are objects of appropriation” (Foucault 1984: 108), so that the author is no longer the origin of writing, but a function of discourse. Considering that Andrade’s vanguard concept of cultural anthropophagy, devised in his 1928 Manifesto Antropófago, defies authorship and subalternity, the notion of anthropophagic appropriation has been seen by Brazilian scholars as an instance of “creative non-subaltern intertextuality” (Fernandes 2019: 8). In this regard, I argue that the Brazilian theorist and thinker anticipates insights developed much later by Bakhtinian-inspired theorists, such as Kristeva, Barthes, Foucault, Genette, and others. Cultural anthropophagy or creative cannibalism, as a sui generis non-subaltern intertextual practice, entails generative power, since the foreign input, far from being denied or rejected, is incorporated and transformed into a new interactive relationship, bringing the concepts of Bakhtinian dialogism and cultural anthropophagy close together. The rich and complex Brazilian translation theory, devised by Haroldo and Augusto de Campos, which involves processes of anthropophagic transcreation of the Western canon, can also be approached in terms of “creative non-subaltern intertextuality.” While disrupting dichotomous views of source and target, the application of cultural anthropophagy to translation unsettles the primacy of origin, rejects fidelity discourses, and energizes the target text with native input in cultural and linguistic terms, so that the translated text becomes a new original creation in its own right. In this regard, “creative non-subaltern intertextuality” encompasses both anthropophagic creative writing and anthropophagic translation or transcreation. In the article “Adaptation and Intermediality” (2017), Lars Elleström situates the notion of adaptation within the broader academic field of intermediality. He predicates that the connection between adaptation and intermediality is not “a relation

35

Anthropophagic Appropriation and Intermediality

917

between two equal parts,” but “a relation of subordination” (Elleström 2017: 509). To his mind, adaptation studies are part of “a broad intermedial research context” and need to be approached from “well-developed intermedial concepts” (509). To support his thesis, he presents two main arguments. In the first, based on a synchronic perspective, he maintains that different kinds of media can “be understood, analyzed and compared in terms of the combination and integration of fundamental media traits” (510), thus providing “an understanding of media as co-existing media products, media types and media traits” (510). In the second, conceived from a diachronic perspective, he sustains that the compositional elements of media in general are transmedial and, therefore, can be transferred to different media types and products (after short or long temporal time gaps). As the general term for designating this process of transfer from one media product to another is adaptation, Elleström argues that adaptation can be seen as a specific phenomenon of media transformation or as a sort of transmediation, inasmuch as one “medium represents again, but in a different way, some media characteristics that have already been represented by another kind of medium” (512), thus involving “a diachronic, rather than a synchronic view of intermedial relations” (524). Drawing on Elleström’s view that adaptation is a mode of media transformation or transmediation situated within the realm of intermediality, I argue that anthropophagic appropriation, conceived from a diachronic perspective, is also a specific form of creative media transformation or transmediation which can equally be located within in the broader field of intermediality. Anchored in anthropophagous epistemologies, Brazilian collaborative theater groups engage in critical and creative appropriation of classic and contemporary texts. They associate and combine European and North American international theater practices with local trends, “drawn from a native Indian tradition (anthropophagia) with strong references to black African rituals (candomblé) and popular culture song and dance” (Dundjerovic and Ramos 2017: 11), thus creating “original” forms of Brazilian stagecraft. The stage productions of Brazilian ensemble theater practitioners can be seen as specific forms of creative transmediation and approached from the perspective of medial transformation and media combination. Hence, anthropophagic performance practices can be accommodated in the realm of transmediality and in the broader field of intermediality.

Concluding Remarks Oswald de Andrade’s innovative theory of cultural anthropophagy has become prominent in Brazilian and international criticism in our time, displacing outdated notions of cultural purity and influence. In an interview conducted by Arnaldo Bloch, entitled “The world eats Oswald” (2008: 272–277), first published in the newspaper O Globo in 2005, the theater practitioner José Celso Martinez Corrêa argues that Andrade can be considered the first postmodernist theoretician, since his innovative avant-garde concept of anthropophagic appropriation is one of the most fruitful aesthetic propositions of the twentieth century which has been illuminating

918

A. S. Camati

Brazilian and Latin American cultural, literary, and artistic history ever since its inception and continues to wield an enduring impact after the turn of the millennium. As has been pointed out throughout this chapter, Andrade’s notion of cultural anthropophagy has been differently reinterpreted by different theorists and thinkers. The language of his Manifesto Antropófago, “with its taboo-breaking provocations, does not only signal out-and-out resistance to colonialist diminishments [. . .]. It also establishes a strategy – absorption into the self – for the radical transformation of objectification of any kind. [. . .] It collapses the division between the ‘indigenous’ and the ‘cosmopolitan’, which so much of intellectual Western tradition has treated as wholly separate [. . .]” (Young 2021: 131). The Brazilian poet and thinker contributed to the destruction of the “colonizer-colonized” hierarchical order that prevailed in academic and literary circles in the 1920s and after. His theory of anthropophagic appropriation of foreign cultural capital mixed with local input presupposes not just exchange, or dialogue open to difference, but implies in a critical and creative ingestion and digestion of tradition without taking on a subaltern position. It does not operate by hiding the incorporation of the foreign but showcases the self-conscious nature of the appropriation process openly. The pioneering theoretical premises developed by Andrade in the Manifesto Antropófago, which by now have become current in Brazilian and international criticism, anticipated contemporary reflections on creative processes and practices, such as adaptation, appropriation, and transculturation. The notion of cultural anthropophagy or anthropophagic appropriation developed by Andrade, which defies authorship and subalternity, can be seen as a case in point of “creative non-subaltern intertextuality,” and the practice of anthropophagic appropriation or transcreation of theater texts can be approached from the perspective of medial transformation and inserted in the realm of transmediality and in the broader field of intermediality. In this sense, Oswald de Andrade is an important forerunner, whose concepts have currently been valued for their rich potentials of subversion and renewal in non-traditional contexts in the Global South, encompassing numerous locations across the Indian and the Atlantic Ocean worlds.

References Andrade, Oswald de. 1928. Manifesto antropófago. Revista de Antropofagia 1 (1): 3,7. ———. 1986. Manifesto of Pau-Brazil poetry. Trans. Stella M. de Sá Rego. Latin American Literary Review 14 (27): 184–187. ———. 1991. Oswald de Andrade’s “Cannibalist manifesto”. Trans. Leslie Bary. Latin American Literary Review 19 (38): 35–47. Azevedo, Beatriz. 2018. Antropofagia – Palimpsesto selvagem. São Paulo: SESI-SP Editora. Bakhtin, Mikhail. 1984 [1929]. Problems of Dostoevsky’s poetics. Ed. and trans. Caryl Emerson. Minneapolis: University of Minnesota Press. Barthes, Roland. 1977 [1967]. The death of the author. In Image music text. Trans. Stephen Heath, 142–148. London: Fontana. Bhabha, Homi. 1994. The location of culture. London/New York: Routledge.

35

Anthropophagic Appropriation and Intermediality

919

Bruhn, Jorgen, and Beate Schirrmacher. 2022. The four modalities of media. In Intermedial studies. An introduction to meaning across media, ed. Jorgen Bruhn and Beate Schirrmacher, 19–24. London/New York: Routledge. Campos, Haroldo de. 1986. The rule of anthropophagy: Europe under the sign of Devoration. Trans. María Tai Wolff. Latin American Literary Review, Special Issue: Brazilian Literature 14 (27): 42–60. Castro Rocha, João Cezar. 2011. Uma teoria de exportação? Ou: “Antropofagia como visão de mundo”. In Antropofagia hoje? Oswald de Andrade em cena, ed. João Cezar de Castro Rocha and Jorge Ruffinelli, 647–668. São Paulo: É Realizações. Clüver, Claus. 1987. Iconicity and isomorphism in Brazilian concrete poems. In Proceedings of the conference on Visuelle Poesie im Historischen wander – Changing forms of visual poetry, ed. Jeremy Adler and Ulrich Ernst, 1–10. Wolfenbüttel: Herzog August Bibliothek. ———. 2000. Concrete poetry and the new performance arts: Intersemiotic, Intermedial, intercultural. In East of west: Cross-cultural performance and the staging of difference, ed. Claire Sponsler and Xiaumei Chen, 33–61. New York: Palgrave. Dundjerovic, Aleksandar, and Luiz Fernando Ramos. 2017. Brazilian collaborative theatre: Interviews with directors, performers and choreographers. Jefferson: McFarland and Company Inc, Publishers. Elleström, Lars. 2010. The modalities of media: A model for understanding Intermedial relations. In Media Borders, multimodality and Intermediality, ed. Lars Elleström, 11–48. New York: Palgrave Macmillan. ———. 2017. Adaptation and Intermediality. In The Oxford handbook of adaptation studies, ed. Thomas Leitch, 509–526. Oxford: Oxford University Press. Fernandes, Fabiano Seixas. 2019. Antropofagia, Brasilidade and translation in recent international scholarship. Santa Barbara Portuguese Studies 3: 1–24. Figueiredo, Vera Follain de. 2011. Antropofagia: uma releitura do paradigma da razão moderna. In Antropofagia hoje? Oswald de Andrade em cena, ed. João Cezar de Castro Rocha and Jorge Ruffinelli, 389–397. São Paulo: É Realizações. Foucault, Michel. 1984 [1969]. What is an author? In The Foucault reader, ed. Peter Rabinow. Trans. Josué V. Harari, 101–120. New York: Pantheon Books. Genette, Gérard. 1997 [1982]. Palimpsests: Literature in the second degree. Trans. Channa Newman and Claude Doubinski. Lincoln/London: University of Nebraska Press. Hermann, Carla. 2010. A cultura popular vai ao museu: a Tropicália de Hélio Oiticica. Textos Escolhidos de Cultura e Arte Populares 12 (1): 213–224. Hutcheon, Linda. 2006. A theory of adaptation. New York/London: Routledge. Interview for Folha de S. Paulo. 1994. Zé Celso põe Hamlet no Carandiru. Folha de S. Paulo. Caderno Mais. https://www1.folha.uol.com.br/fsp/1994/9/18/mais!/21.html. Accessed 20 Nov 2022. Kristeva, Julia. 1980 [1966]. Word, dialogue and novel. In Desire in language: A semiotic approach to literature and art, ed. Leon S. Roudiez. Trans. Thomas Gora, Alice Jardine and Leon S. Roudiez, 64–91. New York: Columbia University Press. Lafuente, Pablo, Lisette Laguado, and Mirtes Martins de Oliveira. 2015. Cultural anthropophagy: The 24th Bienal de São Paulo 1998. London: Afterall Books. Lehmann, Hans-Thies. 2006. Postdramatic theatre. Trans. Karen Jürs-Munby. London/New York: Routledge. Martinez Corrêa, José Celso. 2008. O mundo come Oswald. In Encontros – Zé Celso Martinez Corrêa, ed. Karina Lopes and Sergio Kohn, 270–277. Rio de Janeiro: Azougue Editorial. Nascimento, Evando. 2011. A antropofagia em questão. In Antropofagia hoje? Oswald de Andrade em cena, ed. João César de Castro Rocha and Jorge Ruffinelli, 331–361. São Paulo: É Realizações. Nunes, Benedito. 1979. Oswald canibal. São Paulo: Perspectiva. Rajewsky, Irina O. 2005. Intermediality, intertextuality and remediation: A literary perspective on Intermediality. Intermédialités 6: 43–64.

920

A. S. Camati

———. 2010. Border talks: The problematic status of media Borders in the current debate on Intermediality. In Media Borders, multimodality and Intermediality, ed. Lars Elleström, 51–68. New York: Palgrave Macmillan. Refskou, Anne Sophie, Marcel Alvaro de Amorim, and Vinicius Mariano de Carvalho. 2021. Introduction. In Eating Shakespeare: Cultural anthropophagy as global methodology, ed. Anne Sophie Refskou, Marcel Alvaro de Amorim, and Vinicius Mariano de Carvalho, 1–24. London/New York: The Arden Shakespeare. Rocha, Glauber. 1995. An esthetic of hunger. Trans. Randal Johnson and Burnes Hollyman. In Brazilian cinema, ed. Randal Johnson and Robert Stam, 69–71. New York: Columbia University Press. Sanders, Julie. 2006. Adaptation and appropriation. London/New York: Routledge. Santiago, Silviano. 2001. Latin American discourse: The space in-between. Trans. Ana Lúcia Gazzola and Wander Miranda. In The space in-between: Essays on Latin American culture, ed. Ana Lúcia Gazzola. Durham/London: Duke University Press. Santos, Valmir. 2004. Cia. dos Atores desmonta tragédia de Hamlet. Folha de S. Paulo. https:// www1.folha.uol.com.br/fsp/acontece/ac1311200401.htm. Accessed 23 Aug 2022. Young, Sandra. 2021. Shakespeare in the global south: Stories of oceans crossed in contemporary adaptation. London/New York: Bloomsbury Publishing.

Late Twentieth-Century Intermedia Poetry in the Americas

36

Rebecca Kosick

Contents Visual Poetry (Including Concrete Poetry) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Object Poetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sound Poetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Video Poetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Action Poetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Postal Poetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Concept Poetry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Some Poetry Intermedia Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

923 928 932 936 939 942 946 949 950

Abstract

This chapter provides an overview of significant intermedia poetic activities taking place in the Americas during the late twentieth century. Borrowing from Dick Higgins’ proposals for “Some Poetry Intermedia,” it examines poetic practices that intersect with his categories of object, video, visual and concrete, action, sound, concept, and postal poetry. Each section discusses possibilities for defining the category at hand and explores examples corresponding to its understanding of intermedia poetry. Examples include work by practitioners including Brazil’s Noigandres group, the Cuban publishing collective Ediciones Vigía, Chilean poet/artist Cecilia Vicuña, Mexican artist Ulises Carrión, US poet N.H. Pritchard, and Brazilian artist Leonara de Barros, among others. The chapter ultimately suggests that the range of intermedia poetic examples identifiable at the end of the twentieth century speaks to the malleability of poetry during the era and to the generative effect of changes introduced to poetic form and language by prior waves of vanguard experimentation.

R. Kosick (*) University of Bristol, Bristol, UK e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_45

921

922

R. Kosick

Keywords

Intermedia poetry · Concrete poetry · Sound poetry · Object poetry · Video poetry · Postal poetry After its use by the poet Samuel Taylor Coleridge in 1812, the concept of “intermedia” was reintroduced in a 1966 essay by artist and publisher of Something Else Press, Dick Higgins. Although Coleridge is acknowledged in Higgins’ reflections, poetry is mentioned only minimally in the initial “Intermedia” essay that appeared in the Something Else Newsletter. There, Higgins briefly comments that “the constructed poems of Emmett Williams and Robert Filliou certainly constitute an intermedium between poetry and sculpture” (Higgins 1966: n.p.). More focused considerations of poetry intermedia follow in Higgins’ 1976 “Some Poetry Intermedia.” Encircling “poetry,” printed in the center of the poster-sized diagram, is the word “metapoetries,” and out from this core emanate a number of media vectors and accompanying types of intermedia poetry. These are “visual poetry (including concrete poetry),” “action poetry,” “sound poetry,” “concept poetry,” “postal poetry,” “object poetry,” and “video poetry” (Higgins 1976). The poster also advances parts of Higgins’ initial commentary on intermedia. Whereas W.J.T. Mitchell argues that “the interaction of pictures and texts is constitutive of representation as such: all media are mixed media” (Mitchell 1994: 5), Higgins draws a distinction between mixed and inter-medias, arguing that the latter implies “a fusion conceptually of the elements” (Higgins 1976).1 With fusion in mind, my application of the term “intermedia” here also aligns with Irina Rajewsky’s subcategory of “media combination” (as opposed to medial transposition or intermedial references) (Rajewsky 2005: 51–52). Higgins’ poster indicates that “anything” could be combined with poetry in the form of endless intermedia, but this chapter will take up the seven original categories named by Higgins’ 1976 poster, in order to sketch out a map of intermedia activity taking place across Hemispheric American poetry at the end of the twentieth century. The temporal framing of this chapter is bookended by two eras of surging intermedia activity within poetry. On the one hand, the 1950s and 1960s witnessed the rise and consolidation of the international concrete poetry movement. This movement and the forms of poetry it generated are often taken as exemplar of poetic intermedia. And indeed, “concrete poetry” and its associated practitioners like Emmett Williams are specifically acknowledged in Higgins’ writing on intermediality, generally, and with particular regard to poetry. At the other end of this chapter’s timeline is the rise, and eventual ubiquity, of the “digital sphere” which has spurred additional intermedia innovations in poetry and catalyzed a complementary resurgence of attention to these developments within academic studies of poetry and poetics. But, as Rajewsky indicates, “while it is true that some new aspects and problems have

1

This document is a single-leaf poster and thus has no page numbers. It will be cited by year throughout.

36

Late Twentieth-Century Intermedia Poetry in the Americas

923

emerged, especially with respect to electronic and digital media, intermedial relations and processes per se remain phenomena which have been recognized for a long time. This fact is easily overlooked on account of those approaches to intermediality that concentrate specifically on so-called New Media” (Rajewsky 2005: 44). Taking Rajewsky’s point, this essay will draw its examples primarily from the interval between the two recent peaks of intermedia fervor – roughly the 1960s through the 1990s. This period overlaps with both concrete poetry’s articulation after the 1950s and the arrival and expansion of Web 1.0. But the affordances of intermediality yielded vital opportunities for poets and artists working in between the two mobilizing moments at mid- and end-of-century as well.2 Given the range of practices encapsulated (or potentially encapsulated) by Higgins’ open understanding of poetry intermedia, this study will inevitably leave out a number of relevant examples. Thus, the aim of this chapter is not a comprehensive accounting of all intermedia poetry produced within the American hemisphere at the end of the twentieth century. Instead, by using Higgins’ categories, this essay intends to frame and discuss a set of key examples and polemics of poetic intermediality across a range of American localities and language contexts – from Chile to Brazil, Mexico, and Canada, among other places. To this purpose, this chapter will demonstrate both the breadth of such practices within the Americas and the elasticity of American poetry as it has been practiced in recent history. As Friedrich W. Block indicates, “from today’s view [at the end of the twentieth century], we find that the literary avant-gardes have hardly left out a technical medium, the list comprising far more than the regular range from paper to the new media” (Block 1997: 723). Higgins’ “Some Poetry Intermedia” offers a useful framework for examining poetryengaged media and widening the scope of our understanding of poetry when it is constituted by such extrapoetic relationships. The following sections address each category of intermedia poetry suggested by Higgins in 1976. Each section posits a definition of the intermedium under examination and discusses its development during the late twentieth century. References to illustrative examples of poetry corresponding to each category are included in each section. Readers interested in one particular type of intermedia poetry may wish to jump ahead to the relevant section. This chapter may be read as a whole, or in parts, according to the reader’s interest.

Visual Poetry (Including Concrete Poetry) Visual and concrete poetry have been understood, especially in retrospect, as key examples of intermedia poetry in the twentieth century. Though these terms are sometimes used interchangeably, concrete and visual poetry (or Vispo) had different orientations and self-definitions in the American hemisphere during the twentieth century. The concrete poetry of the Brazilian Noigandres group (Décio Pignatari and 2

This chapter takes a geographic focus on the Americas, but I welcome further such studies of late twentieth-century intermedia poetry in other regions.

924

R. Kosick

brothers Haroldo and Augusto de Campos), particularly during its “heroic” phase at the end of the 1950s, involved, as Claus Clüver describes, “a ‘verbivocovisual’ intermedial text operating with highly reduced verbal material (but operating usually with the entire word as its minimal semantic unit) and relying on carefully organised structures achieved by interrelating the verbal units according to a spatial syntax that also semanticises the white of the page” (Clüver 2019: 72). Concrete poetry of this type was nondiscursive, but did not, as was sometimes the case with later variants of visual poetry, refuse the semantic function of poetic language. Nor did it generally privilege the visual mimeticism of shaped poetry in the vein of George Herbert’s “Easter Wings” (1633). In the latter decades of the twentieth century, a wider, less rigorously defined “concrete” or visual poetry movement spread across the Americas (and beyond). This more diffuse category came to stand for a range of diverse productions. As Lars Elleström comments, “the term visual poetry” arose in response to the “need to find a generic term for the new forms of poetry (such as concrete poetry)” circulating at the time (Elleström 2016: 444). Elleström takes issue with the term “visual poetry,” arguing that “all written poetry is visual as such” and makes the case that forms corralled as “visual poetry” might better be understood as marked by “a high degree of iconicity” (Elleström 2016: 440). Still, it remains the case that the general understanding of visual poetry is any poetry that emphasizes its visual activity, and the category, such as it is, continues to welcome an array of related but distinct practices. Amanda Earl, in the twenty-first-century anthology, Judith: Women Making Visual Poetry, defines visual poetry as “a global term used for all work that integrates elements of language with another medium or engages with the graphical elements of text and mark making. This integration of language has to be an integral part of the work, not just companion to a poem or an illustration” (Earl 2021: 20). Both Clüver and Earl, writing in the twenty-first century, emphasize intermediality as a feature of concrete and visual poetry. Earl, like Higgins (1976), specifically excludes more “mixed” media approaches where visual art and poetry might accompany one another. As Block describes, the key element is a fusion of two media, whereby “the text is visualized concretely and not as a trope, and the image is verbalized in a nonadditive way” (Block 1997: 715). Given the diversity and robustness of the movements in visual and concrete poetry, there are many examples that might be included here, and generally speaking, it is difficult to select just a small sample of intermedia poems for inclusion in this study – something that speaks to the extent of these practices during the era in question.3 Even outside movement poetics, visual and concrete poetry were widely encouraged by growing access to personal printing technologies, including new

3

Three are too many examples of poets to name here, but readers interested in visual and concrete poetry might consult the following anthologies: Anthology of Concretism (1967), Bann (1967). Balgiu and de la Torre (2020), Hill and Vassilakis (2012), Perloff (2021), Solt (1968), Williams (1967) (Also republished by Primary Information in 2013).

36

Late Twentieth-Century Intermedia Poetry in the Americas

925

typewriter models and associated word processing hardware.4 As Sarah Wells argues, “the typewriter existed as a much more privatized technology than either cinema or the radio, and one much more intimately connected to writing itself” (Wells 2017: 85). Such technologies, once brought into the home and the act of writing, afforded poets new tools for working with the page and encouraged experimentation with line and spacing, even among lyrical poets. Opportunities for experimentation facilitated by typewriters were further supported by new technologies of reproduction such as the Xerox machine. In this environment, visual poetry flourished throughout the Americas. Canadian poet Steve McCaffery’s Carnival (initially made at the end of the 1960s; first published in 1973) used the typewriter to create his panel poem’s distinctive swirls of text and whitespace (Fig. 1). The poet would cover blank pages with variously cut

Fig. 1 Steve McCaffery (1973). (Reprinted with permission)

4

For more on typewriter art, specifically, see Tullett (2014) and Sackner et al. (2015).

926

R. Kosick

“masks,” feeding both into the typewriter together and typing onto the paper assemblage. Later, the overlaid sheets would be removed to reveal distinctive shapes of text, cut at random by the mask overlay. McCaffery writes that “the panels grew directly through the agency of the typewriter and through the agency of marginal link-ups. . .. As a mask bled off a page I would devise another shape that picked up the bleed of the text at the margin. . .. the mask came about as a way to create a painterly shape by censoring the flow of typewritten line” (McCaffery 1987: 72–73). Considered by McCaffery to be “concrete,” this example of text-art-meets-visualpoetry demonstrates the breadth of formal possibilities that concrete poetry absorbed into itself as the initially more focused movement was taken up, and taken in new intermedia directions, by poets around the hemisphere. Another facet of visual poetry that followed the more specific mid-century “concrete” movement had to do with the increasingly political tendencies of the form, particularly among Latin American practitioners. This was evident in both movement poetry (for example, the process poets in Brazil, or for those associated with the neovanguardia in Spanish America), as well as among practitioners working outside of these circles, stirred by political upheaval and technological proliferation. In Guadalajara, for instance, Jesus Arellano5 created his “poelectrones” (electropoems), as he tells readers, on an MT72 model of the IBM Selectric typewriter (Arellano 1975: 6). Many of the poems are set in verse line and in visually imaginative layouts including iconic forms mimetic of objects such as dice, scarecrows, butterflies, and political figures, as well as nonmimetic visual forms that make use of novel spacing or defy the horizontal tendencies of print in Spanish. When set in verse line, the verses themselves are typically long, extending past the margins of the print edition and continuing onto subsequent lines with a hanging indent. The political figures depicted in type include Ché Guevara and Salvador Allende. One such poem, published in Mexico the same year as Allende’s assassination on September 11, 1973, would be Arellano’s “De puro Chile” (From/Of Pure Chile). The poem is an ode addressed directly to the socialist president as the informal second-person “tú” (you). It includes a direct quote from a speech Allende had given the year before at the Universidad de Guadalajara, in the poet’s home state of Jalisco. Allende’s quote signals international solidarity in its support of “los que caen en otras partes por hacer de sus patrias países independientes como ocurre en Vietnam” (those who fall fighting to make their countries independent in other parts of the world, as is happening in Vietnam)6 and Arellano extends this gesture to include solidarity with Chile, from Mexico, in the aftermath of the golpe de estado that installed dictator Augusto Pinochet (Arellano 1975: 96–99). Arellano does this both in content and in visual form, and the poem’s visual arrangement – which approximates a bust of Allende’s head and upper shoulders – concentrates the gesture by adopting a style of visual art typically reserved for persons honored and revered. The

5 6

I am grateful to Nohelia Meza for introducing Jesus Arellano’s work to me. All translations are the author’s unless otherwise indicated.

36

Late Twentieth-Century Intermedia Poetry in the Americas

927

visual and textual aspects of the poem fuse intermedially in what Block describes as a “nonadditive way” (Block 1997, 715). This is the case, even if Arellano did not refer to the concept of “intermedia” poetry as he was generating his innovative and politically engaged “poelectrones” on his new typewriter. In fact, the intermedia framework was not a primary touchstone for many early practitioners of concrete and visual poetry, with the Noigandres group being the most prominent American example. As Greg Thomas writes, “in their emphasis on semantic communication, the first concrete poets were [. . .] anxious that their work, for all its medium-defying influences, be recognised as a legitimate literary genre rather than as a mixed-media or an intermedia art practice” (Thomas 2019: 26). The category of “intermedia” arose and found its affiliations with concrete and visual poetry later in the century, and retrospective accounts tended to position concrete and visual poetry as intermedia whose sturdy belonging to the literary arts was in question. As Marjorie Perloff writes, “one would be hard put to find an English or Comparative Literature department that offers courses in concrete poetry. Doesn’t the subject belong more properly, if at all, in the art department, my colleagues ask, specifically in courses on graphic design?” (Perloff 2010: 50) The institutional situation that Perloff describes may be changing with the rise in intermedia arts (and scholarship into the same) 20 years into the new millennium, but she raises a point about concrete and, by extension, visual poetry’s generic position and lack of fit with traditional media categories. While Higgins emphasizes the character of “fusion” that defines intermedia art forms’ relationship with traditional media categories, Perloff acknowledges an inverse effect, whereby mediatic fusions on the part of artforms themselves can sometimes be received with confusion by traditional aesthetic categories. Intermedia poetry is a category that both joins and disjoins. Yet for many practitioners associated with intermedia poetry, it is precisely this dual movement that facilitates the productive energies of their work. With regard to the concrete and visual, these early-identified forms of intermedia poetry also contributed to a broadened scope of possibilities for intermedia poetry as the century went on, as this essay will explore. Craig Saper draws attention to Öyvind Fahlström, who along with the Noigandres group in Brazil and Bolivian-Swiss poet Eugen Gomringer, was one of the earliest founders of twentieth-century concrete poetry. Upon his 1961 move to New York, Fahlström came into contact with Fluxus, an intermedia art movement with which Dick Higgins was also intimately involved. Saper writes that “even as [Fahlström] directly imported Concrete poetry into the Fluxus scene, he began to see this type of poetry as part of what Dick Higgins called intermedia rather than exclusively part of a literary tradition” (Saper 2015). As this example suggests, many of the poets who were initially associated with the practice of concrete poetry extended their own work to include increasingly intermedial engagements, something true of the movements in visual and concrete poetry as well. While the concrete poets in Brazil, for example, emphasized the “verbivocovisual” character of their work – meaning it operated semantically, aurally, and visually – later developments increased certain of these emphases and

928

R. Kosick

their own work expanded to overlap significantly with other of the intermedia poetry categories in Higgins charts. Within Brazil, movements including neoconcretismo and poema-processo also took the concrete strategies of the Noigandres group in new intermedia directions, including further work with Higgins’ categories of object and action poetry. Thus, while concrete (and later on, visual) poetry may not have arisen with the concept of intermedia front of mind, their experiments with the materiality of language and poetry were fertile ground for the kinds of transformative challenges to and fusions of traditional media that would take hold across the arts of the Americas in the late twentieth century.

Object Poetry There are functionally two distinct understandings of “object poetry” – poetry about objects and poetry that is itself an object. The prior can include a variety of textual approaches to objects including description, ekphrasis, and the plain language associated with the North American “objectivists” of the early twentieth century who sought “to use language more literally than figuratively” (Berry 2012: 963). Ravid Rovner considers this type of object poem (a poem about an object) as having a set of related tendencies: “the poem deals with one object; the object’s name is the poem’s title or a part of it; the poem creates the feeling that the poet was looking at the object while writing, so the object is revealed before our eyes as the poem unfolds” (Rovner 2018: 7). While there is a long tradition of this kind of object poetry, this chapter will, for the most part, set aside that understanding of object poetry in favor of poetry that is itself an object. This second understanding – of poetry that is objects – is likely nearer to what Higgins had in mind in identifying this category of intermedia poetry, as for him, object poetry is an intermedium between poetry and sculpture, suggesting a poem that takes on three-dimensional characteristics. Within the American hemisphere of the twentieth century, object poetry of this type grew in prominence following the more often two-dimensional experiments of concrete and visual poetry. Many associated with concrete poetry began making three-dimensional poems following the movement’s mid-century origins, some of which included sculptural versions of prior poems initially rendered on a page or plane. Noigandres group poet Augusto de Campos, for example, created a number of object poems, such as, in the 1970s, the Poemobiles in collaboration with the Spanish-born artist Julio Plaza. And Augusto’s initially print-based poem “cidade city cité,” from 1963, went on to take part in numerous other intermedia constellations. As Charles Perrone writes, “one of [“cidade city cite’s”] most impressive renderings came in the domain of installationsculpture: a 70-meter row of red meter-high letters gracing an entire exterior side of the exhibition hall of the Bienal de Arte in São Paulo” (Perrone 2002: 67) (Fig. 2). The notion that poetry could be an object was already key to early concrete poetry before it grew to encapsulate 70-m-long sculptural texts. Concrete poetry’s emphasis on poetry’s materiality, its verbivocovisuality, helps to call attention to the fact that language, like other objects, is material. Likewise, many of the concrete poets’

36

Late Twentieth-Century Intermedia Poetry in the Americas

929

Fig. 2 Photo of Augusto de Campos with “cidade city cité,” by Augusto de Campos (here, with the collaboration of Julio Plaza), displayed as part of “Trama do Gosto,” São Paulo Biennial, 1987. (Photo by Lygia de Azeredo Campos. Reprinted with permission)

theoretical claims anticipate later definitions and theories of nonlinguistic objects (Kosick 2020b), particularly the notion that concrete poetry is “an object in and of itself, not the interpreter of exterior objects” (de Campos et al. 2007: 218). This claim on the autonomy of the poetic object underscores the fact that the material poem does not defer to the objects it might name or represent, but rather is itself an object like any other. To return to the fusion of poetry and sculpture, key to their mediatic differentiation is the latter form’s claim on three-dimensional space. This is evident even when acknowledging that a painting, too, is a three-dimensional object. To extend that analogy, then, it can be acknowledged that a flat poem may be an object, as the early concrete poets suggested, but also that object poetry affords a special privilege to three-dimensional poetry beyond the page. Object poetry was particularly widespread throughout Latin America during the late twentieth century, as scholars such as Rachel Robinson have noted. Numerous practitioners, with footholds in both poetry and the visual arts, constructed sculptures that either incorporated text or made the text itself into a three-dimensional structure. In the immediate post-concrete context, artists and poets associated with Brazil’s neoconcrete movement were especially interested in object poetry. Practitioners such as poet Ferreira Gullar and artists Hélio Oiticica and Lygia Pape each made poems with sculptural qualities, some of which were explicitly known as “object poems.” Both Gullar and Oiticica, for example, created boxlike structures that required

930

R. Kosick

readers to physically open and handle the objects, which also included language. In Gullar’s case, these were often single words – for instance, “noite” (night) – that isomorphically related to physical aspects of the object poem. In “noite,” this would include the object’s incorporation of dark blue, a color suggestive of the night sky. For Oiticica, brief lyrical poems by Roberta Camila Salgado painted on threedimensional objects are scattered throughout his famous installation Tropicália and multiple “caixa-poemas” (box poems) feature language among other materials such as raw pigment, photography, and plastic that, together, comprise the object that is also the poem. Both Gullar and Pape created book-based works that also made a claim on object poetry – not because they left the book in favor of three-dimensional space, but because the books were sculpted in innovative ways, calling the readers’ attention to the object-nature of the codex form.7 A related strand of influence informing object poetry is thus the artist book, which Johanna Drucker suggests is a “quintessential 20th-century artform” (Drucker 2004: 1). She writes that “books have served to extend the possibilities of visual arts, performance, and music” and, in the other direction, “have also offered a unique conceptual possibility to the poet” (Drucker 2004: 10). Drucker indicates that many concrete poets take up this possibility inherent in artist books. However, even lyrical poets can benefit from reformed opportunities to engage with the three-dimensional object that is the book. One brief example would be the Cuban bookmaking collective, Ediciones Vigía (see Fig. 3). As Ruth Behar describes, founders “[Alfredo] Zaldívar and [Rolando] Estévez wanted their books to be readable and to be fine art. Estévez designed Vigía books that consisted of scrolls to be hung on the wall as well as three-dimensional works to be placed on display” (Behar et al. 2020: 5). Many of Vigía’s handmade books incorporate poetry and provide for it a highly decorated, expanded material support that approaches readers not merely as an inconsequentially three-dimensional book container, but as an object of art/poetry, all of whose elements contribute to its construction and communicative capabilities. Object poetry can thus exceed and incorporate the book form, particularly when it is rearticulated as an overtly sculptural and three-dimensional object. Many object poems were interested in inviting, and indeed requiring, sensory participation on the part of their reading or perceiving audience. And an appeal to tactility and sensation is a further characteristic beyond three-dimensionality that unites object poetry across examples. As Robinson and Felipe Cussen write with reference to the poetry of Argentine poet Ana María Uribe, in participatory object poems, the reader “does not interact solely with a reference to an object off the page, but also with an object itself. The poem affects the reader and is affected by the reader” in a mutual and bidirectional process that exceeds hermeneutic interpretation (Cussen and Robinson 2022). As Robinson suggests, this also elevates poetry’s potential to intervene politically (Robinson 2021).

7

For more on neoconcrete poetry, see Kosick (2020b).

36

Late Twentieth-Century Intermedia Poetry in the Americas

931

Fig. 3 Víctor Rodríguez Núñez, cover of El cuaderno de la rata almizclera (2019). (Design and illustration by Marialva Ríos. Reprinted with permission)

Brazilian artist Regina Silveira extended sensory participation to include taste with her Biscoito Arte (Art Biscuit/Cookie) from 1976. As the title suggests, the work consisted of a cookie in the shape of the word “arte” (art) that could be, and was, eaten. Biscoito Arte was at once an object poem and a “happening” which saw the work consumed; it may thus also qualify as an “action poem,” as this essay will soon discuss. As Patricía Figueiredo Pedrosa writes, Biscoito Arte is a “testament to [the artist’s] interest in conceptualism and the value of the graphic sign” (Pedrosa 2019: 993). Silveira shared this combination of interests with many Brazilian artists working at the time, in the wake of concrete poetry on the one hand and amid the rising global movements in conceptual and participatory art on the other.8 Object poetry is one category that can sit at this nexus, both building on a variety of intermedia influences and contributing to still further developments in media fusion to follow.

8

See Camnitzer (2007), Hilder (2016), Kosick (2020a).

932

R. Kosick

Sound Poetry Just as Elleström points out that “all written poetry is visual as such” (Elleström 2016: 440), the term “sound poetry” might be considered redundant. Poetry has long been a medium created, materialized, and recorded in sound. Higgins charts sound poetry as being an intermedium between poetry and music, specifically, but it is hard to suggest this relationship can be traced to “the social problems,” which Higgins suggests “no longer allow a compartmentalized approach” to aesthetic media in the twentieth century (Higgins 1966: n.p.). As Jahan Ramazani points out, “song has long been conceived as poetry’s closest generic kin” (Ramazani 2014: 184). And Higgins himself stresses this point in other writing, acknowledging the “folk roots of sound poetry,” though he later lays out “five relatively modern classes of sound poetry: (1) works in an invented language, (2) near-nonsense works, (3) phatic poems, (4) unwritten-out poems, and (5) notated ones” (Higgins 1984: 50). Connecting each of these, per Higgins, is a broad view which suggests sound poetry is “poetry in which the sound is the focus, more than any other aspect of the work” (Higgins 1984: 40). Others, such as Phildelpho Menezes, take a narrower view, suggesting that sound poetry should not be understood as “a declamation of a written poem, even if the declamation is an oralization of an experimental written text” (Menezes 2001: 66). Preferring the term “intersign poetry,” Menezes emphasizes poetry where “sounds of any species (phonetic, bodily noises, especially from a vocal tract, noises of natural or artificial origin, daily life auditory elements), are put together, formally related to each other, constructing a whole meaning or sense of the poem to be understood by the audience” (Menezes 2001: 68). Much of what Menezes emphasizes here echoes what we have seen with definitions of other intermedia poetries – where the sonic (in this case) elements are not mere accompaniments to an otherwise text-based poem, but integral components of it. Relatedly, Richard Kostelanetz suggests what he calls “text-sound” “characterizes language whose principle means of coherence is sound, rather than syntax or semantics” (Kostelanetz 1977: 61). However, for the purposes of this overview, it can be acknowledged that there are a range of ways for poetry and sound (or music, in Higgins’ framework) to fuse together under the banner of “sound poetry.” From the perspective of the twentieth century, a poetic emphasis on sound was not entirely new, but it did stand out during a time when poetry favored the page and privileged the visual media of print and text. As Ramazani comments, “the question of poetry’s kinship with song may seem an unlikely framework within which to explore modern and contemporary poems” and he locates, with support from Giorgio Agamben,9 a graphic turn within the Western lyric occurring “around the twelfth century” (Ramazani 2014: 184). That said, this is not a uniform timeline across all contexts, as Ramazani acknowledges, citing “African and Caribbean societies which had rich traditions of oral poetry but took up literary verse to a

9

See Agamben (1999).

36

Late Twentieth-Century Intermedia Poetry in the Americas

933

significant degree” in the twentieth century (Ramazani 2014: 185). As McCaffery points out, “sound poetry,” as it came to be in the twentieth century, draws from varied influences and its “attempts to recover lost traditions mix with attempts to effect a radical break with all continuities. What is referred to by ‘sound poetry’ is a rich, varied, inconsistent phonic genealogy” (McCaffery and Nichol 1978: 6). Part of this twentieth-century genealogy includes early avant-garde movements such as Dada. One example would be Hugo Ball’s “abstract phonetic poems” which Ball likened to “the liturgical chanting that wails through all the Catholic churches of East and West” (Richter 1997: 43). As the century continued, the visual began to outcompete the aural among America’s experimental poets. Hilary Kaplan describes how there was “a visually-dominated period in Brazilian experimentalism that extended from the beginning of the Concrete poetry movement in the 1950s until the 1990s,” noting that during this time “avant-garde poetic sound was concentrated instead in music, though the musicalization of Concrete poems and through the projects and influence of MPB poet-composers such as Caetano Veloso” (Kaplan 2010). Kaplan’s point here is important in order to consider how we can conceptualize intermedia poetry as coming into being. Although Higgins’ diagram shows a direction of travel for poetic intermedia as beginning from poetry and heading toward other media, we can assume that intermedia poetry arises from other media’s, other disciplines’, and other genres’ fusions with poetry as well. Kostelanetz acknowledges this, writing that “as text-sound is an intermedium located between language arts and musical arts, its creators include artists who initially establish themselves as ‘writers,’ ‘poets,’ ‘composers,’ and ‘painters,’” among other possibilities (Kostelanetz 1977: 61). Keeping this in mind, sound poetry may come more prominently into focus as an active category of intermedia poetry at the end of the last century, even in locales where it was predominantly affiliated with media beyond poetry, including music and other experimental arts. Concrete and visual poetry, as the latter’s name suggests, did tend to be understood as prioritizing play with the visual material of language. But this priority may be in part attributable to the reception and circulation of such poems, which were sometimes displayed in art galleries and museums and were inviting to look at, drawing the attention of readers’ eyes. As Antonio Sergio Bessa comments, “the poems from the so-called heroic phase of concretism display a heightened sense of design that seems to overwhelm other aspects of the text” (Bessa 2009: 219). That said, both the early concrete poets working in the mid-century and later iterations of the broader movement considered sound to play a key role, on par with visual material, when it came to generating poetry. As Bessa indicates, for the Noigandres group, “sound was submitted to as rigorous a program as the written text” and in their writings “one finds repeated references to sound” (Bessa 2009: 219–20). Such references include direct inspiration from Austrian composer Anton Webern. In fact, as Clüver outlines, “the first of what were soon to be called ‘concrete poems,’ the six polychromatic poetamenos texts by Augusto de Campos (1953), represent, with all their striking visual qualities, a successful attempt at translating a musical device into the verbal medium” (Clüver 1982: 4). Picking up Kaplan’s comment about musician

934

R. Kosick

Caetano Veloso, the reverse process took place as well –for example, in a 1979 recording Veloso translated Augusto de Campos’s multicolored poetry back into music. Further experiments with sound followed, such as Augusto and Cid Campos’s collaboration, Poesia é risco.10 Within the later, more elaborated, and looser concrete poetry movement, visual experimentation was also tightly tied to sound. McCaffery’s Carnival, visually striking both in its arrangement of text on the page and by its largess when arranged into panels as the poet encouraged, can also act as a score for an equally striking vocal performance piece. Fiona McMahon comments that the project “encapsulates an ideal of poetry as an object of performance and of the poet as a polyartist plotting experience in new spaces” (McMahon 2007). These new spaces included intermedia ones and McCaffery’s version of sound poetry aligned with aspects of the intermedium that prioritized a non-semantic series of vocal utterances tracing paths through the movements of type. Similarly, US poet Norman (N.H.) Pritchard, associated with the Umbra poets in New York City, thought of his visually experimental work as intricately tied to sound. He reflects that “visual and sound are the two elements that have been most important in my work. [They] always run parallel. Because that is. . . that is the essence of poetry” (Tully and Pritchard 1978).11 As Aldon Lynn Nielsen describes, Pritchard’s work also includes “chanted versions of the poems that he recorded for the 1967 Broadside album New Jazz Poets” and, as Nielsen goes on to say, Pritchard’s poems, overall, “are virtual catalogues of jazzy rhythmic effects, virtuoso free rhyming, hyperbolic and metaphysical imagery, understatement, compressed and cryptic imagery” that take “black music as poetic reference” (Nielsen 1997: 13–14). Beyond the relationship to jazz that scholars have noted, Pritchard’s poems are emphatic in the way they shape the process of poetic enunciation and pronouncement. In “La alba” (Dawn) from The Matrix, for instance, words are extended and expanded across the page in ways that undeniably provoke elongated pronunciation on the part of readers. Its first lines read, The dawning cloak of heat enlists its remembrance of light aaaaaaaaaaccccccccccrrrrrrrrrrooooooossssssss vvvvvvvvvaaaaaaaaaaaasssttt sssssssppppppppppaaaaaaaaaaaacccccccccceeeeeeesssssss (Pritchard 2021, 33)

Notable in this excerpt is the fact that the repeated letters in the words “across,” “vast,” and “spaces” are not uniform in their number, but vary in length from three to

10

This is a CD and intermedia performance/screening of Augusto’s poetry set to music with other accompanying elements. 11 A recording of this interview is available at https://jacket2.org/commentary/pritchard-1978.

36

Late Twentieth-Century Intermedia Poetry in the Americas

935

twelve, suggesting, as in a musical score, different spans of time associated with the pronunciation of each letter/sound. Although there is disagreement about this, some poets do understand the relationship of sound poetry to visual poetry as one in which the written text acts as a score for vocal performance, and this approach also carries implications for the innovative possibilities of text on the page. As Drucker writes, “the idea of visual scoring, or the manipulation of phonetic components of poetry in a manner analogous to a musical piece, has a history which far precedes the twentieth-century avantgarde, but in the mid-century reinvestigation of musical form new expressions of this sensibility came to the fore” (Drucker 1996: 47). Such scoring also extended beyond music to include other modes of capturing sound and speech in type. Natilee Harren makes such an argument about US poet Charles Olson, describing how “Olson’s imperative [in his essay ‘Projective Verse’] that poetry ought to be read aloud and thus written to facilitate its oration called for renewed attention to its graphic layout” (Harren 2020: 34). A key example of a sound poet known for scoring his works would be US poet Jackson Mac Low, who, as Sylvia Mae Gorelick writes, “made a series of 501 poetic works called the Asymmetries” in the 1960s, each of which was “written as a score for its performance” (Gorelick 2017: 20). And, more contemporarily, this framework may be useful in considering the sound poetry of intermedia Chilean artist/poet Cecilia Vicuña. As Kenneth Sherwood argues, Vicuña’s work is “poetry that proceeds through writing to performance” (Sherwood 1997: 74), making orality an end point beyond the text’s graphic imprint. For Vicuña herself, sound is also present at the origins of this sequence. In an interview, she reflects on childhood play, where “there was all this water that was moving, playing with light and shadow. I think it’s true that these were languages. They were speaking, and I was part of it, part of those sounds. That is the source of my art and my poetry” (Harvey and Vicuña 2020). Like McCaffery, Sherwood points to the intersecting vectors of indigenous orality and Western tradition for sound poetry. For José Felipe Alvergue, Vicuña’s poetry “communicates indigenism as an alternative site,” one that is a “combination of orality — and all the narratological elements of language as a spoken and not written medium — installation, and performance” (Alvergue 2014: 90). In Vicuña’s case, then, sound poetry may not just arise as what Higgins suggests is an intermedium between poetry and music,12 though this is perhaps one way it emerges for other poets such as Pritchard. But sound poetry might also flow from fusions between poetic language and environmental aurality, or from encounters between avant-garde and indigenist poetic practices. As we have seen, it can even arise, perhaps idiosyncratically, out of impetuses to turn the visual poetic text into 12

In later writing, Higgins complicates and expands on sound poetry’s relationship to music. He writes, “one thing sound poetry is not is music. Of course it has a musical aspect—a strong one. But if one compares typical sound poetry pieces with typical musical ones, music is usually the presentation or activization of space and time by means of occurrences of sound [. . .] sound poetry points in a different direction, being inherently concerned with communication and its means, linguistic and/or phatic” (Higgins 1984, 51)

936

R. Kosick

(or use it to record) a vocal performance. All of this is part of what McCaffery calls the “varied, inconsistent phonic genealogy” of sound poetry at the end of the twentieth century (McCaffery and Nichol 1978: 6).

Video Poetry Like sound poetry, video poetry’s lineage may also be inconsistent – influenced by a series of convergent evolutions growing from practices in film, the visual arts, and poetic experimentation with emerging technologies. Per Higgins’ classificatory schema, video poetry is the fusion of poetry and video, but as Sarah Tremlett suggests, video poetry (or “videopoetry,” or “poetry film,” etc.) can take a variety of forms and refer to differently organized constellations of poetry and video (or film). She writes: A “film poem” historically abjuring verbal language for experimental cinematic language [. . .] depends on its audio-visual rhythms and tonal qualities to create aesthetic effects. It goes without saying that there are crossovers between definitions. The poetry film often contains and capitalizes on the same aesthetic effects as the film poem, so that where I discuss structure, rhythm, tone or mood, the term film poem can, where applicable, be transposed for poetry film. The description “video poem” is often used as another term for a poetry film, whereas a “videopoem” does not illustrate a separate poem, but in the combination and composition of the elements it is in itself an original screen-based poem. (Tremlett 2021, xxi)

Tremlett’s final category of “videopoem,” as a single word, shares with certain definitions of visual and sound poetry the conviction that one medium not merely accompany, or in Tremlett’s words “illustrate,” the other. Instead, for the “videopoems” Tremlett describes and the “video poems” (two words) appearing in Higgins’ framework, mediatic fusion is a key element in the poem becoming an intermedia work. In line with Higgins, this essay will use the terminology of “video poem,” but with the recognition that the meaning of this term may have evolved over the years, as Tremlett suggests, to incorporate what Higgins might have identified as more mixed media tactics.13 Like sound poetry, video poetry shares a relationship with concrete and visual poetry, borrowing from the strategies of these earlier-established practices, but shifting them into sonically, temporally, and kinetically distinct forms. These 13

As Ben Bollig points out, beyond video poetry there exists a diffuse and at times contradictory conversation around films considered to be “poetic.” So-deemed “poetic films” can diverge dramatically from the intentionally hybrid or media-fusing approach to aesthetic making that would generally characterize video poetry on Higgins’ terms. As Bollig writes, critics apply the term “poetic” to “films that are quirky, ambiguous, or subtle. For others, this is a classification that may be used to refer to films that do not favour narrative or plot. Others use the term to indicate a preference for the long take over frequent cutting or various forms of montage [. . .] The ‘poetic film’ might be seen as one that is simply creative,” among other possibilities (Bollig 2021, 17).

36

Late Twentieth-Century Intermedia Poetry in the Americas

937

transformations arose in part thanks to poets’ adoptions of new, or more massively available, technologies, as well as from impetuses already embedded in concrete or visual poetry that may not have been fully realizable on the two-dimensional plane. Kaplan describes how, following the concrete era anchored in the 1950s and 1960s, “poets and artists experimented with video to create the next generation of visual poems, advancing poetry in an intermedia direction” (Kaplan 2010). This was the case for early concrete poets such as the Noigandres group, who were cultivating broader practices by the latter decades of the twentieth century, as well as for the poets whose work was generated in the years following earlier experiments in the visual sphere. The Brazilian poet and musician Arnaldo Antunes, for example, was the author of poetry that moved between the page and other intermedia environments. As Lívia Ribeiro Bertges and Vinícius Carvalho Pereira describe, the late twentieth century witnessed “the technological development of new supports, as both integral to and determining of unprecedented aesthetic experiences.” New technologies like video fostered “different ways of interacting with and of reading emerging poetic textualities” (Bertges and Pereira 2017: 492–493). One example by Antunes might be his poem “Dentro” (Inside) whose lines play with the title word and sonically related signs in an approach similar to early concrete poetry: de dentro entro centro sem centro entro dentro de dentro entro centro sem centro dentro14

As Patrícia Ferreira da Silva Martins and Goiandira de Fátima Ortiz de Camargo describe, “Antunes’s poem ‘Dentro’ was initially published in graphic form, in the poet’s book Tudos (Antunes 1993). The black and white poem centers a distortion of the signifiers ‘dentro,’ ‘entro,’ ‘centro,’ etc. In transposing this text to video, the poet needed to make a series of adaptations. Thanks to the (im)possibilities of the available technology, a totally new version of the poem emerged” (Martins and Camargo 2005: 111). In the initial visual poem, the letters were arranged in a globe

14 Literally, “of/inside/I enter/center/without/center/I enter/inside/of/inside/I enter/center/without/ center/inside.”

938

R. Kosick

shape that produced a visually distorted image of the word “DENTRO.” As Ricardo de Araújo Pereira writes, in this version, the poem’s boundaries “are a complete union of concave and convex planes” (Pereira 1997: 42). The song and the “videopoema” that “Dentro” became introduced further visual and audio distortions of the text, as well as, in the case of the video, footage of an endoscopy encircled in a round frame.15 As Clüver notes, Antunes has also “gone on tour performing the sound score live against a projection of the video” (Clüver 2000: 51). One significant departure from concrete poetry that video poetry introduces is a lengthened (or perhaps restored) sense of poetic temporality and movement. While early concrete poets emphasized their desire to create a “dynamic structure” that would involve a “multiplicity of concurrent movements” (de Campos et al. 1965: 156), such movements were often implied in page-based concrete or visual poetry. Laura Lopez-Fernandez recognizes that “the poetic value of movement was already established in Western literature through works such as Simmias of Rhodes’s pattern poems and technopagneia, ‘The Axe’, ‘The Wings’ and ‘The Egg’, all requiring a non-sequential type of reading” (Lopez-Fernandez 2011: 439). This is true of some early and mid-century avant-garde poetry as well, but video poetry allows for actual, continuous movement of text and images. As Lopez argues, “more recently, different media and technologies have supported innovative kinetic effects in poems” distinct even, from the kinds of reading processes spurred by visual poetry (LopezFernandez 2011: 439). Such distinctions are evident when considering concrete poetry’s “appeal to non-verbal communication” and its desire to break with a “merely temporalisticlinear” poetic structure (de Campos et al. 1965: 157, 156). Whereas early waves of concrete poetry sought an immediacy of poetic perception akin to the way viewers might take in a painting all at once, for video poetry, a linear experience of time on the part of the viewer overlies even the most nonsequential or nonnarrative “content” it might convey. In this way, video poetry returns the visual/spatial aspects of concrete poetry to a tradition perhaps more temporally akin to literature or film than to the visual or plastic arts. Yet this same fact can offer opportunities to the intermedium of video poetry. As Tremlett explains, “what makes a poetry film so unique is that the spatio-temporal visual surface or monstration, descending from the graphic arts, is as powerful or more powerful than the sequential trajectory inherited from the traditional dramatic film” (Tremlett 2021: 79). Thus, while a lengthened temporality may seem to be restored from the perspective of visual poetry’s turn toward video, from the perspective of film, poetry – particularly poetry related to the visual arts – offers alternatives to the narrative sequentiality dominant in more mainstream cinema. Although video poems like those by Antunes contain words, or grow from visual experiments with graphic text, other approaches to video poetry include films (or videos) that do not necessarily involve words, but specifically exploit poetry’s

15 At the time of this essay’s writing, the video poem of “Dentro” is available at https://www. youtube.com/watch?v¼PLgGhrf9NMc.

36

Late Twentieth-Century Intermedia Poetry in the Americas

939

lack of requirement for sequential or linear development, with or without actual words. These include experimental films such as those by US filmmaker Stan Brakhage. Film and video offer numerous opportunities to poets seeking to write outside the strictures of verbal syntax. Poetry filmmaker and co-curator of the VideoBardo poetry film festival, Marisol Bellusci has argued that, for Latin American video poets in particular, making poems without words, but via visual communication instead, offers an alternative to writing in languages imposed by the colonizer (Bellusci 2020). From the perspective of the twenty-first century, video poetry is now a robust and heterogenous category that, like visual poetry, continues to evolve and permit new poetic opportunities across a range of approaches. And although the variety of these approaches prevents absolute terminological clarity from taking hold among the various iterations of video poems, poetry films, etc., it can be seen as a strong and vibrant example of poetry intermedia that has only grown in scope since Higgins’ charted its potential in the 1970s.

Action Poetry Visual, sound, and video poetry are admittedly sprawling categories that gather a range of related, if not uniform, practices, but these categories of intermedia poetry are more consolidated than what Higgins calls “action poetry.” This branch of intermedia poetry that Higgins identified in the 1970s did not go on to become as established as certain other arms of intermedia poetry and the category itself is marked by aesthetic trends particular to the time of Higgins’ writing. In some ways, this may suggest that “action poetry” remains an intermedium while other more established poetic fusions no longer occupy the “inter-” space. Higgins writes that “with familiarity each intermedium becomes a new medium,” pointing to the constant flux of his model and the ways in which time, exposure, and public perception influence the consolidation, or sometimes the forgetting, of a nascent media category (Higgins 1976). Action poetry may have dipped below the radar in the years since Higgins suggested its emergence between poetry and the “happening” – another intermedia category set within a triad of mediatic influences drawing from music, theater, and the visual arts. The term “happening” was initially coined by US artist Allan Kaprow, and as Jillian Suarez notes, it “emphasized the organic connection between art and its environment” (Suarez 2015). Kaprow describes happenings as originally entailing presentations made to: small, intimate gatherings of people in lofts, classrooms, gymnasiums and some offbeat galleries, where a clearing was made for the activities. The watchers sat very close to what took place, with the artists and their friends acting along with assembled environmental constructions. The audience occasionally changed seats as in a game of musical chairs, turned around to see something behind it, or stood without seats in tight but informal clusters. Sometimes, too, the event moved in and amongst the crowd, which produced some movement on the latter’s part. (Kaprow 2006: 102)

940

R. Kosick

As Kaprow indicates, happenings at first maintained the separation of audience and performer – albeit with modifications as compared to mainstream theater – but, as he reflects, this “proved to be a serious drawback [. . .] to the plastic morphology of the works” and happenings began to move toward an ideal of eliminating the audience entirely (Kaprow 2006: 102). This meant an increasing indistinction between audience and performer, where everyone involved would be “willing and committed participants who have a clear idea what they are to do” (Kaprow 2006: 103). Action poetry, as a relative of happenings, might also be pinned to the moment in time when happenings were coursing through the vanguard arts of the American hemisphere. But, because this category is less established, the term has been used in other, or less specific ways in the years since. Francois Victor Tochon defines action poetry as “poetry put into action, a kind of poetic action-research that intends to change social life in a poetic way” (Tochon 2000: n.p.). Others are developing new theories under the name of action poetry that emphasize the body, vocality, and performance.16 There are echoes here with considerations likely to have informed Higgins’ understanding of action poetry, particularly around the ways action poetry utilizes the body in poetic creations that directly engage an audience of “readers” in wider society. With all this in mind, this essay will take embodied, participatory poetry as the frame in which action poetry can be explored. Even though poets may not have always had a clear intent to make action poetry by that name, it is possible to highlight a number of examples that might correspond with this category of intermedia poetry. And there are precedents for related uses of the term beyond Higgins. Pauline Bachmann, for instance, uses the term “action poetry” to describe certain poetic activities of Uruguayan poet Clemente Padín and Argentinian poet Edgardo Antonio Vigo that prioritize readerly involvement. Bachmann describes Vigo’s score-like use of the “poema para realizar” (poem to be performed), which asks readers to pose and respond to their own questions in writing. Here, she argues, “readers were not simply asked to be inventive in the literary and graphic domains but also called upon to use their body to participate in the creative process” (Bachmann 2018: 22–23). Other score or instruction-based poetry that ignites readers’ action includes Canadian visual poet bpNichol’s booklet Cold Mountain which, as Karl Young describes, “contained instructions for folding the book and burning it. In this case the real visual poem was something you could only see for a few minutes: the book burning in front of you” (Young 1998: n.p.). Throughout the American hemisphere, other visual poets took up participatory poetry as well. Mary Ellen Solt – who is principally known today as the editor of the important anthology, Concrete Poetry: A World View – was herself a visual poet who worked with other related intermedia practices over the course of her career. One example would be her poem “PEOPLEMOVER,” first made, as she notes, in the summer of 1968, “when the mood of frustration and anger against policies relating to the Vietnam War and the rapidly deteriorating quality of American Life and

16

See Walker (2019).

36

Late Twentieth-Century Intermedia Poetry in the Americas

941

institutions in general was about to erupt in total disorder: the violent confrontation of police and demonstrators during the Democratic National Convention in Chicago” that summer (Solt 1978: unpaginated preface). The poem drew from found language in the public and political spheres, functioning as a script for a series of demonstrations/performances held in Bloomington, Indiana, that had the character of happenings. There are multiple iterations of the poem, including a published version in codexical form, but it also took shape in what we might call object poetry form, on placards and posters carried in staged, but perhaps genuinely felt, protests near Indiana University. One such performance of “PEOPLEMOVER” took place in 1970 “as part of a program devoted to expanded forms of concrete poetry” (Solt 1978). The event entailed a group reading that also involved the projection of “historical names, titles, dates and places on a screen” in the same room (Solt 1978). During the reading and visual display, “a quiet demonstration took place in the aisles. When a particular poster was being featured in the reading, that demonstrator stood facing the audience. Fragments of patriotic songs and marches were played at the beginning and end when the members of the audience were invited to join the demonstration” (Solt 1978). Solt admits, though, that few audience members actually took up this invitation, suggesting this intermedia spectacle shared the challenges of early happenings as identified by Kaprow. “PEOPLEMOVER” provokes multiple intermedia considerations. As mentioned, the poems took shape as three-dimensional objects – protest posters – whose physical structure contributed a key, not ancillary, function to the poems’ purpose and semiotic capabilities. These were held by participating performers whose embodied engagement with the poems was crucial to their display. The event itself also involved vocal performance, incorporated music, displayed text on screen, and invited (if not necessarily successfully) audience participation beyond what would have traditionally been asked of a spectator or reader. If, per Higgins, the happening fuses theater, music, and visual art, all of that is evident here, with the added participation of poetry in object, voice, and script form. Saper calls this work an “art of infrastructure,” in part because of its ironic reference to Disneyland’s transport system (also called a people mover), commenting that “Solt saw visual poetry in terms of sociopoetic activities” (Saper 1997: 19). In this way, too, Solt’s “PEOPLEMOVER” aligns with later understandings of action poetry that take it as intervening in social space, and here, the character of that intervention is expressly political. A political message also permeates Brazilian poet Paulo Bruscky’s “Poesia viva” (live poetry), performed in collaboration with Unhandeijara Lisboa on March 14, 1977. This was the International Day of Poetry, and the poetic performance/ happening took place during the Brazilian military dictatorship, though after its most repressive years. During those years, Bruscky had suffered arrests, torture, and threats to his life from the regime in power, and the participatory poem – whose words of “live” poetry were spelled out on wearable smocks – attests to Bruscky’s endurance amid years of state-inflicted violence. A flyer accompanying the event urged readers to “do something (whatever you can) so that poetry doesn’t suffer

942

R. Kosick

inside literature courses or die in literary supplements.” As it goes on to say, poetry “lives through you, poetry lives in the street” (Bessa 2015). Thus, the poem/ happening, or action poem in Higgins’ terms, like other examples of the loose genre, makes a direct entreaty to its readers and participants, with the aim of changing both the parameters of the poem and the society in which it appears. Fluxus, a movement with which Bruscky collaborated internationally, often made use of poetic scores that, as Saper describes, did “not depend on the voice but on creating situations,” meaning poetry no longer functioned as “documentation of past reverie” but as a textual catalyst for experiences of all kinds, reverie potentially included (Saper 1997: 10). This is somewhat in line with Solt’s use of poetry in “PEOPLEMOVER” in that it precedes and is read/displayed during the event itself (as well as being a textual record of it). But for “Poesia viva” the poetry arises from the fusion of the wearable letters with the participants’ bodies and movements, as well as from the situation more generally, suggesting that poetry, within or as part of a happening, need not function just as a textual score or record of action, but can also be generated within, and by, the action itself. Brazilian artist Leonora de Barros’s Poema (1979) illustrates another mode of embodied poetic action that, as Alex Balgiu and Mónica de la Torre write, took the form of “genre-defying word-based art at the intersection of concrete poetry and conceptual art” (Balgiu and de la Torre 2020: 14). Distinct from the United States where, as Jaime Hilder describes, “conceptual artists were eager to dissociate themselves” from concrete poetry (Hilder 2016: 153), conceptual art and concretestyle poetry were often blended in Brazil during the late twentieth century, fostering, perhaps, a more fertile field of works that could be categorized retrospectively as action poetry. In the case of de Barros’s Poema, the “text” involved was neither poetic score nor record – instead, the artist engaged directly with the technology of print, pressing her tongue into the keys and typebars of a typewriter in a performance stressing the overlap between tongue (language) and tongue (organ). This performance may be without the same invitation for audience engagement characteristic of happenings and other examples of action poetry closer to this tradition. Nevertheless, it is certainly active and participatory, even if it is primarily the artist’s body taking part, and the work significantly reorganizes readers’ relationship to textual production and reception by comparison with traditional page-based poetry. Thus, while “action poetry” is less codified than other intermedia types suggested by Higgins, it can be seen as a dynamic category that fostered new developments for poetry and its many contextual relationships.

Postal Poetry Sound, video, and object poetry have a strong evolutionary relationship with earlier concrete and visual poetry practices. This is in some ways true of postal poetry as well – Isabel Lázaro and César Reglero, addressing the Spanish context, write that “visual poetry’s boom coincid[ed] with the beginnings of mail art’s consolidation” (Lázaro and Reglero 2014: 199), suggesting mutual influence and benefit across the

36

Late Twentieth-Century Intermedia Poetry in the Americas

943

two intermedia practices. Mail art, itself, is already an intermedia category involving, as Mexican mail artist Ulises Carrión writes, “any postal sending that incorporates one, several or all the elements of the actual mailing as part of the transmitted message.” Mail art, as he says simply, is “the art of using the mail” (Carrión 1980: 25). While its own intermedium, mail art is linked with a variety of related twentieth-century aesthetic practices. Fabiane Pianowski includes among these “the early twentieth-century avant-gardes, the Nouveaux Realistes, Fluxus artists, and conceptual art” (Pianowski 2014: 329). Mail art also has a relationship with artists’ books; as Carrión argues, “one of the decisive factors for the world-wide proliferation of artists’ books (and of artists’ book shows) was their ability to be distributed by means of the mail” (Carrión 1980: 25). In the Americas, a variety of factors contributed to the development of mail art and, by extension, postal poetry. These included a DIY ethos on the part of practitioners, a growth in small press publishing, and – particularly in Latin America where significant numbers of artists and poets were working under dictatorships – the need to distribute artworks and share poetry outside of official channels. As Bellusci writes, mailing works of art and poetry “meant avoiding the art institutions (authorities) [. . .] as a means to connect with other like-minded thinkers” (Tremlett 2021: 266). Saper also calls attention to “the availability of industrial production techniques (photo-copiers, low-end printing presses, increasingly affordable film, video, and photographic equipment, and so on) and the urge for democratization of art making” that, in general, “fueled an alternative art scene since the 1950s” (Saper 1997: 12). Postal poetry did, but also did not, privilege the materiality of the sign in line with other intermedia poetries. While Saper highlights how groups like the Nouveaux Realistes’ “play with the postal system” included “graphic sculptural poetry rather than poetry closely connected to voice” (Saper 1997: 24), other postal poets specifically took up the subjective affordances of the epistolary in their mailed verses. For Higgins, postal poetry is the fusion of poetry and mail art and, in practice, that fusion involved the mobilization of a range of poetic types, from lyrical forms to alreadyintermedia categories such as visual poetry. In terms of format, Seeta Peña Gangadharan points to Ray Johnson’s New York Correspondence School, founded in 1962, which “had the effect of clarifying the mail art practice, its revaluation of the postal system and the envelope” (Gangadharan 2009: 286). Artworks were often mailed inside envelopes and envelopes likely have a more important role in mail art than in postal poetry, which often moved by postcard as with Nicanor Parra’s image-text postcard series Artefactos,17 but a number of poets and artists who took part in the wider mail art movement did make use of envelopes as both poetry and poetic support. In addition to the action poems and happenings Paulo Bruscky staged as part of his varied career, the Brazilian artist/poet created both mail art and postal poetry. Bruscky was affiliated with the wider mail art movement and with Fluxus, a movement that frequently

17

See Kosick (2018).

944

R. Kosick

intersected with mail art and whose associates included Ray Johnson, Dick Higgins, and John Cage, among other international artists. The distinction between the categories mail art and postal poetry is not always pressing in Bruscky’s case, but of note is his work with the envelope, specifically, which comprised an important strand of his posted productions. Bruscky’s “envelopoemas” (envelopoems) were decorated works of postable art and poetry that frequently bore the marks and stamps of the postal channels through which they travelled. These marks included chance imprints of that travel itself, such as stains or tears, as well as official material contributions in the form of things like cancellation stamps and air mail stickers. Like many mail artists at the time, Bruscky’s work also satirized these same official conventions with its own use of pasted images and rubber stamps. This relates to Saper’s claim that practitioners employed “the trappings of bureaucracies for poetic ends” in networked art, where, as he argues, “poets and artists shift the use and tone of the bureaucratic images from signaling authority to participation in ironic satire, parody, and inside jokes” (Saper 1997: 16). In Bruscky’s case, his envelopoemas were frequently stamped with the bilingual message “hoje a arte é este comunicado/today art is this communicated” – a tautological (re-)inscription of the artwork itself that mimics a line of practice common across his work of the era.18 This text, as well as the variety of other images, drawings, and inscriptions that decorated his work, can be understood as a kind of visual poetry, which “along with artists’ stamps and seals,” as Pianowski notes, was prominent in the earlier years of mail art (Pianowski 2014: 343). Carrión describes the ways in which rubber stamps “function in our social reality,” writing that “they are a symbol of power—their role is to validate or invalidate something” (Carrión 1980: 33). While Carrión suggests the opposite is true of artists’ stamps – “they are incapable of transmitting power” (Carrión 1980: 33) – stamps can offer a variety of functions which include, in the case of Bruscky and other postal poets working under authoritarian conditions, a critique of, and challenge to, the absolute reach of dictatorial repression and censorious assertions of power that might otherwise affect, limit, or foreclose poetic communication. In the inverse, this communicative potential was also important for Latin American artists and poets working in exile, such as those associated with Beau Geste Press in Devon, England. Mexican artists Martha Hellion and Felipe Ehrenberg published numerous works of poetry via Beau Geste including innovative, intermedia works by Cecilia Vicuña and Ulises Carrión that benefitted from postal circulation. As Carrión suggests, the mail means that “an artist doesn’t need to live in an ‘art-capital’ to have his voice heard” (Carrión 1980: 30) and communication across far-flung places around the globe becomes possible. Thus, on the one hand, intermediality is characteristic of these books’ contents – Vicuña’s Sabor a Mi (Taste of Me) includes leaves, paintings, and verse poetry, among other things, and Carrión’s Tras la poesía/Looking for Poetry combines visual and textual semiosis.

18

See Kosick (2020a).

36

Late Twentieth-Century Intermedia Poetry in the Americas

945

Fig. 4 Ken Mikolowski, “Fear Art” postcard (Ann Arbor: The Alternative Press, late twentieth century). (Reprinted with permission)

On the other hand, these works are also shaped by what Eric Vos calls the “fundamentally intermedia” concept that is mail art’s “Eternal Network” (Vos 1997: 331). As Vos describes, this network aims at “achieving communio, a global community of artists, sharing information and work” (Vos 1997: 331). Such a network benefitted poets publishing outside of well-resourced “centers” of culture, as Carrión suggests. In the United States, this included the “Alternative Press” founded by artist Ann Mikolowski and poet Ken Mikolowski. Together, the couple letter press printed bumper stickers, bookmarks, broadsides, and postcards from their home in Michigan (see Fig. 4). Their work is especially remarkable for its deployment of the poetry postcard, which was itself a mailable format, but also circulated in mailed packets to the Press’s subscribers around the United States via the postal service. As contributor, US poet Alice Notley describes, the Alternative Press had an ongoing project of sending a set of five hundred [blank] postcards to select people in the hopes of having the cards returned to them with the “picture” side filled with holograph poems and other materials. Individual cards were then included in packets of printed broadsides, bumper stickers, and so on, sent to the Mikolowskis’ “subscribers” (the packets were free) [later the Press charged $10]. (Notley 2005: 67)

Notley’s reference to holopoetry is maybe not exactly in line with the immaterial poetry by that name created by Brazilian/US artist and poet Eduardo Kac which “remains invisible until the viewer/reader moves it into a particular position” revealing a “three-dimensional text unattached to any surface” (Clüver 2000: 53). However, the postcards’ relationship to their readers was distinct from traditional forms of printed poetry – while the poets filled in the faces of each card, the backs remained blank, awaiting other authorial interventions from writers of any, or no, training in the craft. Kac writes that holopoetry “integrates dynamically the three dimensions of space with the added dimension of time” (Kac 2007: 130), and this point introduces further intermedia considerations into the postal poetry of the Alternative Press, beyond the mere fact that poetry circulated by and as mail. While the poetry shared on each postcard’s face ranged in character, from short lyrical poems to haiku to handwritten epistolary works, for poets creating sets of 500 “Multiple Originals” as Notley describes, the poems became both spatial

946

R. Kosick

constructs that often included text and visual input and components of a durational procedure that sometimes took poets years to complete. As with many of the intermedia poetry categories that Higgins diagrams, postal poetry is informed by numerous vectors of intermediality beyond an apparently simple one-to-one fusion of mail art and poetry. As Vos notes, mail art is already a multiply-intermedia category, and postal poetry incorporates elements from a variety of media – from painting and collage, as with Notley’s Alternative Press postcards, to drawing and handwritten text in the case of Parra’s Artefactos, to rubber and postal stamps for Bruscky, among other possibilities.

Concept Poetry Like action poetry, the category of concept poetry did not go on to become as consolidated as many of the intermedia poetries discussed here. Furthermore, from the perspective of the twenty-first century, “concept poetry” may suffer from a nomenclature overlap with the more recent movement calling itself “conceptual poetry.” The latter prizes “appropriation, citation, copying, [and] reproduction” (Perloff 2010: 23), culling poetic material from found texts, rather than creating “original” works of poetry. Concept poetry, on the other hand, was understood by Higgins to be a fusion of poetry and philosophy. Understood this way, concept poetry might also not be as firmly bound to the vanguard energies of many of the other kinds of intermedia poetry that Higgins charts. For example, video poetry or twentieth-century sound poetry grew in part from cutting-edge poetic experiments with newly available technologies by poets already engaged with the avant-garde. Recording technologies facilitated a movement from the page into sound and moving images but maintained an already-established tendency to favor a concentration of poetic materiality over communicative transparency.19 Many poets associated with these intermedia poetries and related avant-garde movements also engaged in extensive theorization of the nature of language, poetry, and perception – the Noigandres concrete poets are prime examples. However, it is not the case that poetry’s fusion with philosophy must take form in what are, effectively, already intermedia environments. The lyric and other traditional poetic types can catalyze such fusions too, as can prose poetry, or even poetic prose. Concept poetry is also distinct from other forms of intermedia proposed by Higgins in that most others involve the fusion of poetry – traditionally a languagebased medium – with media that are not traditionally (or primarily) linguistic in nature such as sculpture or music. In the twentieth century, most instances of 19

The Language poetry movement might sit at the overlap between conceptually and materially invested poetics. This theoretically informed movement “insisti[ed] on the materiality of language—the thing-character of language—by retarding or congesting or displacing language (what we understand as the communicative func- tion of language) with work made from fragments, syllables, letters, &c” (Brown 2014: 524).

36

Late Twentieth-Century Intermedia Poetry in the Americas

947

philosophy and poetry would be likely to develop via, and register in, the medium of text. In this way, Higgins’ framework elides distinctions between media, as the plastic or material form in which a work of art can take shape, and genre, as the mode or style to which a medium might conform or correspond. Philosophy and poetry might be considered two genres of writing; thus, their fusion may be less about the ways in which linguistic material combines with nonlinguistic and more about the ways in which two genres of writing can mix or approximate one another. Yet even when this happens, not every poet working at the intersection of poetry and philosophy takes for granted a fusion of these two genres. Figures such as Argentine poet, critic, and student of philosophy Tamara Kamenszain raise this issue.20 Kamenszain, who wrote poetry and theorized the avant-garde, reflects on the relationship between her two types of writing, commenting: Until recently I alternated: when I was writing poetry, I couldn’t write essays, and vice versa. Now I’m writing in both genres at once and they end up contaminating each other. I don’t mean that they turn into the same thing. The opposite, actually, the more I mix them, the clearer their differences are to me. I don’t believe in the avant-garde hybridity of genres. I believe in pushing boundaries of their differences. (Kamenszain and Ferrero 2013: 62)

Alternately, understanding poetry, in general, as already a mode of philosophy is a widely held position. As Gustavo Ortiz Mill writes, “many agree that poets practice philosophy through their poems, even that we can find more philosophy, as well as distinct and more profound truths, in poetry than we can in many works of philosophy proper” (Mill 2017: 144). For Ortiz Mill, “there are different ways of understanding” what he calls “philosophical poetry” but, as he writes, “without a doubt, one of the most interesting is the notion that the poet philosophizes through poetry” (Mill 2017: 144). Ortiz Mill discusses the late 1950s poem “Piedra de sol” (Sunstone) by Mexican poet Octavio Paz as an example of philosophical poetry. But he draws distinctions between the ways in which poetry and philosophy address or express philosophical concepts. Paz’s poem, Ortiz Mill suggests, “captures in its verses ideas that we can find discussed by philosophers” such as Martin Heidegger and Emmanuel Levinas but the poem does not, and perhaps should not, make or justify an argument in the way philosophy must (Mill 2017: 169). And this is both one potential distinction between poetry and philosophy and one potential contradiction inside which the intermedium of concept poetry might shape itself. As US poet Rainer Diana Hamilton suggests, it is possible for poetry to argue, particularly when it appeals to intermedia affordances. In what Hamilton calls a “poem-essay,” she constructs an argument whose shape is familiar to reasoning, with a premise that is increasingly complicated and eventually overturned by the very conceptual practice the poem itself engages in. Once, I wrote that poetry doesn’t make arguments: it does

20

Thank you to Edmundo Paz Soldán for introducing me to Kamenszain’s work.

948

R. Kosick

something else instead, I argued, in an essay that was not a poem. If I had argued it in an essay that was a poem, it might have been “interesting,” where interesting just means “there’s something to be said about that,” and what I might say about that would include some acknowledgment of the intentional contradiction meant to complicate a clearly reductive idea of the work poetry can or cannot do with something “clever.” But I wasn’t being clever, when I wrote that. I was being naive, so I was wrong, when I wrote that, and I’m probably wrong still, I was defensive, when I wrote that, I was defending “poetry,” I thought, but Poetry doesn’t need my defense. It can make its own arguments. (Hamilton 2018: 87)

By the time readers find themselves ready to agree with Hamilton that poetry can make arguments, it is clear this poem already has. Argument need not be the only thread facilitating the fusion of poetry and philosophy. Higgins’ use of the word “concept” suggests that something key to the fusion of philosophy and poetry may be ideas. This carries echoes with analogous assertions in the field of contemporary art around the time Higgins was developing his theories of intermedia. As US artist Sol LeWitt writes (initially in 1967), “In conceptual art the idea of concept is the most important aspect of the work. When an artist uses a conceptual form of art, it means that all of the planning and decisions are made beforehand and the execution is a perfunctory affair. The idea becomes a machine that makes the art” (LeWitt 1999: 12). Here, as LeWitt indicates, the material outcome of a work of art is of less importance than its conceptual impetus or development. One consequence of this turn was that language – involved in the scores, planning, and recording of these works of art – came to take on new levels of importance within the broadly understood visual arts. Going further, as Pedro Erber writes, “notions of conceptual art and conceptualism expanded their explanatory power to the point of including almost any artwork that happened to cross the boundary between visual and verbal discourses” (Erber 2012: 74). Transposing this situation to poetry, then, it may be the case that its structuring ideas, rather than the material of language itself, take on expanded significance within concept poetry. As a result, this intermedium might be understood as inverse to many of the others that Higgins names in his 1976 diagram, where the opposite – poetry’s material support or basis is elevated – tends to occur.

36

Late Twentieth-Century Intermedia Poetry in the Americas

949

Yet taking poetry as a materially emphatic medium can also contribute to the possibilities it opens for philosophy when the two combine. In an interview with Canadian poet Lisa Robertson, Lebanese-US poet and artist Etel Adnan, who also studied philosophy, remarks on the ways that poetry can connect thinking with feeling, opening up affective and subjective possibilities for writing across the two genres. She says, Philosophy is freer now, and for that reason Heidegger could say that the great philosophers were the poets. That a real, trained philosopher like Heidegger would come to that is very important to poets. Poets were afraid to think and philosophers were afraid to let go, to let loose and speak of themselves as part of their thinking. This boundary has been broken down. I love contemporary poetry because it moves between what we call poetry and what we call philosophy. It joins these fields and makes writing more natural, as in how it is lived in the person. We don’t separate thinking from feeling in real life, so why should we separate it in writing? The life of the mind is one and the boundaries and the categories are useful tools. We made them realities, but they are not realities—they are only tools, categories. (Adnan and Robertson 2014: 105–106)

Some of the points that Adnan here raises bring to light other interventions in what we might retrospectively call concept poetry, including that of neoconcrete poet and critic-theorist of art and culture, the Brazilian poet Ferreira Gullar. His sensorially engaging work could be understood as action or object poetry, or as a materialpoetic exploration of phenomenological principles that, like Adnan suggests, erode the distinction between thinking and feeling, mind and body. During the late twentieth century, numerous other poets around the American hemisphere also worked at the spot where a loosened philosophy opened itself to poetic input (or vice versa), including Martinican poet and philosopher Édouard Glissant, whose “philosophical thinking,” John Drabinski writes, “gives way, or ought to give way, to poetry—the language of tout-monde, the language of archipelagic thinking, the language of paradox that resolves and dissolves” (Drabinski 2019: xvii). For Drabinski, it is precisely the coming together of poetry and philosophy that permits Glissant to challenge the presumptions of a white, European philosophical tradition, allowing us to read “Glissant’s motifs against white Western philosophy and its pretensions or efforts to neutralize the geography and universalization of historical experience” (Drabinski 2019, xviii). Thus, though it may be a less well-known intermedium than some of the others here addressed, concept poetry, like each of Higgins’ poetic intermedia, affords poets and philosophers (and poet-philosophers) a set of different, new, or loosened praxes with the potential to invigorate both poetry and philosophy.

Some Poetry Intermedia Conclusions Charting with Higgins the late twentieth-century intermedia poetry of the Americas, as this chapter has done, a clear takeaway is the breadth of conceptual and material variety suggested by these seven exemplar categories. Higgins’ categories are both a

950

R. Kosick

record of vanguard disruptions to traditional media ongoing around the 1970s and speculative proposals for how poetry might develop in future years, what forms it might take, and what media it might fuse with and incorporate. It is also clear that between Higgins’ original categories lies significant further overlap – visual poetry bleeds into sound poetry, object poetry facilitates participatory action poetry, and, as Higgins anticipates, some poetry intermedia consolidate into more familiar media forms, spurring new “betweens” and seeding the ground for the continued proliferation of novel mediatic fusions in the years to follow. The sheer number and range of intermedia poetic examples identifiable at the end of the twentieth century speaks to the malleability of poetry during the era and to the generative effect of changes introduced to poetic form and language by prior waves of vanguard experimentation. Here, many of the intermedia poetries discussed particularly capitalize on the materiality of language, which has been transhistorically important to poetry but saw itself concentrated, distilled, and further elevated by poetic experimentalisms of the twentieth-century Americas. The headway made by movements such as early practitioners of concrete poetry gives way, around the time of Higgins’ writing, to further developments benefitting from a rearticulated poetry open to comingling with other media at the level of form, matter, and (as is the case in concept poetry) idea. Some of the intermedia categories suggested by Higgins are more historically contingent than others, and some have faded or have not gone on to especial prominence (at least by the name Higgins gave) in the years since “Some Poetry Intermedia” appeared in print. But the overall lesson we might absorb is that poetry, then and now, is open to fusion with companion and contrasting media forms, to the great benefit of both. Acknowledgments I am grateful to the Eccles Centre for American Studies at the British Library, the Special Collections Research Center at the University of Michigan, and Craig Saper for their support of this essay’s research. Thanks also to Steve McCaffery, Augusto de Campos, Víctor Rodríguez Núñez, and Ken Mikolowski for permission to reproduce their work here.

References Adnan, Etel, and Lisa Robertson. 2014. Interview with Etel Adnan. BOMB Magazine 127 (Spring): 102–107. Agamben, Giorgio. 1999. The End of the Poem: Studies in Poetics. Translated by Daniel HellerRoazen. Stanford Stanford University Press. Alvergue, José Felipe. 2014. The Material Etymologies of Cecilia Vicuna: Art, Sculpture, and Poetic Communities. The Minnesota Review 2014: 59. https://doi.org/10.1215/002656672412901. Anthology of Concretism. 1967. Chicago Review 19 (4): 3–144. Antunes, Arnaldo. 1993. Dentro. https://www.youtube.com/watch?v¼PLgGhrf9NMc Arellano, Jesús. 1975. El canto del gallo. Mexico City: Universidad Nacional Autónoma de México. Bachmann, Pauline. 2018. Action Poetry: Clemente Padín and Edgardo Antonio Vigo’s Experimental Poetic Practices in the 1970s. Hispanic Journal 39 (2): 11–30. https://doi.org/10.1017/ s000316150007694x.

36

Late Twentieth-Century Intermedia Poetry in the Americas

951

Balgiu, Alex, and Mónica de la Torre, eds. 2020. Women in Concrete Poetry: 1959–1979. New York: Primary Information. Bann, Stephen, ed. 1967. Concrete Poetry: An International Anthology. London: London Magazine. Behar, Ruth, Juanamaría Cordones-Cook, Kristin Schwain, and Ruth Behar. 2020. Handmade in Cuba. Handmade in Cuba (January): 1–16. https://doi.org/10.5744/florida/9781683401520. 001.0001. Bellusci, Marisol. 2020. Latin_no American VideoPoetics. Bristol: Bristol Poetry Institute and Liberated Words. Berry, E. 2012. Objectivism. In Princeton Encyclopedia of Poetry and Poetics, ed. Roland Greene, 963–964. Princeton: Princeton University Press. Bertges, Lívia Ribeiro, and Vinícius Carvalho Pereira. 2017. Uma proposta de análise do gênero GIF poem a partir de ‘Asas’ e ‘Volve’, de Arnaldo Antunes. Revista desenredo 13 (2): 492–507. https://doi.org/10.5335/rdes.v13i2.7250. Bessa, Antonio Sergio. 2009. Sound as Subject: Augusto de Campos’s Poetamenos. In The Sound of Poetry / The Poetry of Sound, ed. Marjorie Perloff and Craig Dworkin, 219–236. Chicago: University of Chicago Press. ———, ed. 2015. Poesia Viva: Paulo Bruscky. São Paulo: APC/Cosac Naify. Block, Friedrich W. 1997. The Form of the Media: The Intermediality of Visual Poetry. In Semiotics of the Media, ed. Winfried Nöth, 713–730. Berlin and New York: De Gruyter Mouton. Bollig, Ben. 2021. Moving Verses: Poetry on Screen in Argentine Cinema. Liverpool: Liverpool University Press. Brown, Bill. 2014. [Concept/Object] [Text/Event]. ELH 81 (2): 521–552. Camnitzer, Luis. 2007. Conceptualism in Latin American Art: Didactics of Liberation. Austin: University of Texas Press. Carrión, Ulises. 1980. Second Thoughts. Amsterdam: VOID Distributors. Clüver, Claus. 1982. Concrete Poetry into Music: Oliveira’s Intersemiotic Transposition. The Comparatist 6 (May): 3–15. ———. 2000. Concrete Poetry and the New Performance Arts: Intersemiotic, Intermedia, Intercultural. In Cross-Cultural Performance and the Staging of Difference, ed. Claire Sponsler and Xiaomei Chen, 33–61. New York: Palgrave Macmillan. ———. 2019. Exploring the Structures of Chance: Transcreating Noigandres Ideogramas into English. In The Translation and Transmission of Concrete Poetry. New York/Abingdon: Routledge. Cussen, Felipe and Rachel Robinson. 2022. The Spirituality of Language in Cecilia Vicuña’s and Ana María Uribe’s Visual Poetry. Bulletin of Spanish Studies 99 (5): 885–906. de Campos, Augusto, Haroldo de Campos, and Décio Pignatari. 1965. Plano-Pilôto Para Poesia Concreta. In Teoria da poesia concreta: Textos críticos e manifestos 1950–1960, ed. Augusto de Campos, Haroldo de Campos, and Décio Pignatari, 154–156. São Paulo: Edições Invenção. de Campos, Haroldo, Augusto de Campos, and Décio Pignatari. 2007. Pilot Plan for Concrete Poetry. In Novas: Selected Writings, ed. Antonio Sergio Bessa and Odile Cisneros, translated by Jon Tolman,, 217–219. Evanston: Northwestern University Press. Drabinski, John E. 2019. Glissant and the Middle Passage: Philosophy, Beginning, Abyss. Vol. 1, 397. Minneapolis: University of Minnesota Press. https://doi.org/10.5840/pga20211226. Drucker, Johanna. 1996. Experimental, Visual, and Concrete Poetry: A Note on Historical Context and Basic Concepts. In Experimental – Visual – Concrete: Avant-Garde Poetry Since the 1960s, ed. Vos Eric, K. David Jackson, and Johanna Drucker, 39–61. Atlanta/Amsterdam: Rodopi. ———. 2004. The Century of Artists’ Books. New York: Granary Books. Earl, Amanda, ed. 2021. Judith: Women Making Visual Poetry. Malmo: Timglaset Editions. Elleström, Lars. 2016. Visual Iconicity in Poetry. Orbis Litterarum 71 (6): 437–472. https://doi.org/ 10.1111/oli.12112. Erber, Pedro. 2012. The Word as Object: Concrete Poetry, Ideogram, and the Materialization of Language. Luso-Brazilian Review 49 (2): 72–101.

952

R. Kosick

Gangadharan, Seeta Peña. 2009. Mail Art: Networking without Technology. New Media & Society 11 (1–2): 279–298. https://doi.org/10.1177/1461444808099581. Gorelick, Sylvia Mae. 2017. Gesture and the Politics of Communication in the Drawings of Jackson Mac Low. In Low: Lines-Letters-Words, edited by Brett Littman, ed. Jackson Mac, 18–26. New York: The Drawing Center. Hamilton, Rainer Diana. 2018. God Was Right. New York: Ugly Duckling Presse. Harren, Natilee. 2020. Fluxus Forms: Scores, Multiples, and the Eternal Network. Chicago: University of Chicago Press. Harvey, Sarah Timmer, and Cecilia Vicuña. 2020. An Interview with Cecilia Vicuña. Asymptote (January). Higgins, Dick. 1966. Intermedia. Something Else Newsletter 1 (1). ———. 1976. Some Poetry Intermedia. New York: Unpublished Editions. ———. 1984. Horizons: The Poetics and Theory of Intermedia. Carbondale/Edwardsville: Southern Illinois University Press. Hilder, Jamie. 2016. Designed Words for a Designed World: The International Concrete Poetry Movement, 1955–1971. Montreal: McGill Queen’s University Press. Hill, Craig, and Vassilakis, eds. 2012. The Last Vispo Anthology: Visual Poetry 1998–2008. Seattle: Fantagraphics Books. Kac, Eduardo. 2007. Media Poetry: An International Anthology. Bristol: Intellect. Kamenszain, Tamara, and Adrián Ferrero. 2013. Interview with Tamara Kamenszain. Hispamérica 42 (125): 59–62. Kaplan, Hilary Beth. 2010. Poetic Sound Work 1980-2010. Jacket 2: 1–12. https://jacket2.org/ commentary/poetic-sound-work-1980-2010. Kaprow, Allan. 2006. Notes on the Elimination of Audience // 1966. In Participation: Documents of Contemporary Art, ed. Claire Bishop, 102–104. Cambridge: M.I.T. Press. Kosick, Rebecca. 2018. The Postcard Poetics of Nicanor Parra’s Artefactos. In Latin American Textualities, ed. Heather J. Allen and Andrew R. Reynolds, 128–150. Tucson: University of Arizona Press. ———. 2020a. From Repetition Poem to ‘Envelopoemas’: Concrete Tautologies in Paulo Bruscky’s Political Poetics. Journal of Lusophone Studies 5 (1): 117–139. ———. 2020b. Material Poetics in Hemispheric America: Words and Objects 1950–2010. Edinburgh: Edinburgh University Press. Kostelanetz, Richard. 1977. Text-Sound Art: A Survey. Performing Arts Journal 2 (2): 61–70. Lázaro, Isabel, and César Reglero. 2014. Mail art y poesía visual: Un parto natural. Tintas. Quaderni di letterature iberiche e iberoamericane Marzo (0): 197–235. https://riviste.unimi.it/ index.php/tintas/article/view/3964/4070. LeWitt, Sol. 1999. Paragraphs on Conceptual Art. In Conceptual Art: A Critical Anthology, ed. Alexander Alberro and Blake Stimson. Cambridge: M.I.T. Press. Lopez-Fernandez, Laura. 2011. Reflections on Intermedia Poetic Modes: Bartolome Ferrando and Eduardo Kac. Forum for Modern Language Studies 47 (4): 429–441. Martins, Patrícia Ferreira da Silva, and Goiandira de Fátima Ortiz de Camargo. 2005. O Ver Poético: Arnaldo Antunes e Eduardo Kac. Signótica 17 (1): 103–118. McCaffery, Steve. 1973. Carnival: The First Panel 1967–70. Toronto: The Coach House Press. ———. 1987. The Annotated, Anecdoted, Beginnings of a Critical Checklist of the Published Works of Steve McCaffery. Open Letter 6 (9 (Fall)): 67–92. McCaffery, Steve, and Bp Nichol, eds. 1978. Sound Poetry: A Catalogue for the Eleventh International Sound Poetry Festival Toronto, Canada October 14 to 21, 1978. Toronto: Underwhich Editions. McMahon, Fiona. 2007. Iconicity and Typography in Steve McCaffery’s Panel-Poems. Revue LISA / LISA e-Journal 5 (2): 133–152. https://journals.openedition.org/lisa/1237?lang¼en. Menezes, Phildelpho. 2001. Experimental Poetics Based on Sound Poetry Today. Visible Language 35 (1): 64–75.

36

Late Twentieth-Century Intermedia Poetry in the Americas

953

Mill, Gustavo Ortiz. 2017. La poesía como filosofía. 8 (1): 143–173. https://doi.org/10.5281/ zenodo.998163. Mitchell, W.J.T. 1994. Picture Theory: Essays on Verbal and Visual Representation. Chicago: University of Chicago Press. Nielsen, Aldon Lynn. 1997. Black Chant: Languages of African-American Postmodernism. Cambridge: Cambridge University Press. Notley, Alice. 2005. Coming After: Essays on Poetry. Ann Arbor: The University of Michigan Press. Núñez, Víctor Rodríguez. 2019. El cuaderno de la rata almizclera. Matanzas: Ediciones Vigía. Pedrosa, Patrícia Figueiredo. 2019. Paraler: Palavras de Regina Silveira. Encontro de História Da Arte XIV: 993–1004. https://doi.org/10.20396/eha.vi14.3402. Pereira, Ricardo de Araújo. 1997. La nueva poesía visual brasileña. Cuadernos hispanoamericanos enero (559): 39–43. Perloff, Marjorie. 2010. Unoriginal Genius: Poetry by Other Means in the New Century. Chicago: University of Chicago Press. Perloff, Nancy, ed. 2021. Concrete Poetry: A 21st-Century Anthology. London: Reaktion Books. Perrone, Charles A. 2002. Performing São Paulo: Vanguard Representations of a Brazilian Cosmopolis. Latin American Music Review 23 (1): 60–78. Pianowski, Fabiane. 2014. Reflexiones acerca del mail art latinoamericano. Barcelona investigación arte creación 2 (3): 327–348. https://doi.org/10.4471/brac.2014.16. Pritchard, N.H. 2021. The Matrix: Poems: 1960–1970. New York: Primary Information and Ugly Duckling Presse. Rajewsky, Irina O. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. Intermédialités Fall (6): 43–64. https://doi.org/10.7202/1005505ar. Ramazani, Jahan. 2014. Poetry and Its Others: News, Prayer, Song, and the Dialogue of Genres. Chicago: University of Chicago Press. Richter, Hans. 1997. Dada: Art and Anti-Art. Translated by David Britt. London: Thames and Hudson. Robinson, Rachel. 2021. ‘Printed Matter’: Rodrigo Lira’s Visual Antipoetics. Hispanic Review 89 (3): 265–291. Rovner, Ravid. 2018. ‘Object Poetry’ as a Source of Inspiration for Design Studies. In Poetry across the Curriculum: New Methods of Writing Intensive Pedagogy for U.S. Community College and Undergraduate Education, ed. Frank Jacob and Shannon Kincaid Amy E. Traver, 3–24. Leiden/ Boston: Brill. Sackner, Marvin, Ruth Sackner, and Steven Heller. 2015. The Art of Typewriting. New York: Thames and Hudson. Saper, Craig. 1997. Networked Art. Minneapolis: University of Minnesota Press. ———. 2015. Concrete Poetry in America: A Story of Intermedia Performance, Publishing, and Pop Appeal. Coldfront. http://coldfrontmag.com/concrete-poetry-in-america/%0A. Sherwood, Kenneth. 1997. Sound Written and Sound Breathing: Versions of Palpable Poetics. In The Precarious/QUIPOem: The Art and Poetry of Cecilia Vicuña, ed. M. Cathering de Zegher, 73–93. Hanover: University Press of New England. Solt, Mary Ellen. 1968. Concrete Poetry: A World View. Bloomington: Indiana University Press. ———. 1978. The Peoplemover: A Demonstration Poem. Reno: West Coast Poetry Review. Suarez, Jillian. 2015. How Allan Kaprow Helped Create ‘Happenings.’ Guggenheim.org. July 9, 2015. https://www.guggenheim.org/blogs/findings/how-allan-kaprow-helped-create-happenings. Thomas, Greg. 2019. Border Blurs: Concrete Poetry in England and Scotland. Liverpool: Liverpool University Press. Tochon, Francois Victor. 2000. Action Poetry as an Empowering Art: A Manifesto for Didaction in Arts Education. International Journal of Education & the Arts 1 (2) http://www.ijea.org/v1n2/. Tremlett, Sarah. 2021. The Poetics of Poetry Film: Film Poetry, Videopoetry, Lyric Voice, Reflection. Bristol/Chicago: Intellect.

954

R. Kosick

Tullett, Barrie. 2014. Typewriter Art: A Modern Anthology. London: Laurence King Publishing. Tully, Judd, and N. H. Pritchard. 1978. Interview with Norman (N.H.) Pritchard. CETA Cultural Council Foundation Artists Project. Vos, Eric. 1997. The Eternal Network: Mail Art, Intermedia Semiotics, Interarts Studies. In Interart Poetics: Essays on the Interrelations of the Arts and Media, ed. Ulla-Britta Lagerroth, Hans Lund, and Erik Hefling, 325–336. Amsterdam: Rodopi. Walker, Nathan Ashley. 2019. Action Poetry: The Vocal-Body in Performance Art, PhD thesis. University of Salford. Wells, Sarah Ann. 2017. Media Laboratories: Late Modernist Authorship in South America. Evanston, IL: Northwestern University Press. Williams, Emmett, ed. 1967. Anthology of Concrete Poetry. New York: Something Else Press. Young, Karl. 1998. The Visual Poetry of BpNichol: A Brief Sketch. Thing.Net. 1998. http://www. thing.net/~grist/l&d/bpnichol/ky-e-bp.htm.

Photojournalism and Beyond

37

Jan Baetens and Domingo Sa´nchez-Mesa

Contents Part One: Reworking the Definition of Photojournalism as an Institutionally Embedded Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Question 1: Photojournalism and Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Question 2: Who Is Showing, and Who Is Talking? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Question 3: Photojournalism Outside Photojournalism? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part Two: A Short History of Photojournalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part Three: Intermediality and Transmediality as Gateways to Medium Specificity in Photojournalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

956 958 961 964 969 971 975

Abstract

Photojournalism is doing journalism with photographs. A typical example of intermediality, since images can never be separated from words in journalism, it also involves many elements of transmediality, embedding photojournalism in broader medial networks. This chapter does not only discuss definitions of photojournalism, it also tackles more historical and theoretical questions, focusing on three major fields of interest, which are always discussed in relationship with the existing literature on the topic. The chapter further addresses questions raised by the shift to digital media and their impact on issues of realism, truth, and (post-) truth. More specifically, the chapter deals with the following three questions: (1) What is the relationship between words and pictures in photojournalism. Thus it asks whether a photojournalistic picture is only an image, that is a silent and only metaphorically speaking object, or also a sign that functions in J. Baetens (*) Cultural Studies, KU Leuven, Leuven, Belgium e-mail: [email protected] D. Sánchez-Mesa Literary Theory, Universidad de Granada, Granada, Spain e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_40

955

956

J. Baetens and D. Sa´nchez-Mesa

combination with other signs, in this case verbal signs, (2) Who is the actual agent of photojournalism, that is, who is doing the “talking,” even when the images are not accompanied, supervised, or marshalled by spoken or written words? (3) Can there be photojournalism outside journalism, for instance in literary texts or in the art circuit – and if so, what is the journalistic values of these practices? In addition, this chapter also discusses questions of intermediality (the copresence and interaction of different media within a single medium) and transmediality (both in the sense of the shift from one medium to another and the distribution of a story in different media). Keywords

Agency · Authorship · Aaption · Crowdsourcing · Documentary style · Host medium · Illustration · Intermediality · Photojournalism · Remediation · Transmediality

This chapter combines historical questions on the transformations of photojournalism and questions having to do with the authorship, agency, and institutional position of photojournalism. One of the general issues that appears throughout the various parts concerns the twin notions of intermediality (within photojournalism) and transmediality (between photojournalism and other media, journalistic or not).

Part One: Reworking the Definition of Photojournalism as an Institutionally Embedded Practice Photojournalism is doing journalism with photographs, and the emphasis in the composite word is on journalism, not on photography, with the latter always at the service of the former. In other words, photojournalism is not only a certain type of photography (and we will have to come back on what distinguishes it from other types of photography such as “documentary style photography”), it is also part of the more comprehensive task of journalism. If we adhere our approach to journalism, as verified news in the public interest, to the recent UNESCO’s report “Journalism is a Public Good” (2021), we are entitled to say that journalism, like any other public good, “plays a critical role in promoting a healthy civic space. It does so by providing citizens with trusted and fact-based information that they need to participate in a free and open society. Journalism simultaneously acts as an independent watchdog and agenda-setter. But for journalism to function as a public good, it needs to operate under politically and economically viable conditions so that it can produce independent, high-quality, and trustworthy news and analysis” (UNESCO 2021: 8). As for photojournalism, it entails a certain type of images, mostly linked to current events and facts and taken with the purpose of giving “trusted and fact-based information.” On the other hand, it also supposes a certain number of other subsequent operations. The pictures have to be made, which is often a great challenge,

37

Photojournalism and Beyond

957

edited, for instance cropped and captioned, and eventually embedded in another medium, ranging from classic print media, for fixed photographs, to newer media such as television, film, and the Internet, also for moving images. This very simple and straightforward take on the field, which may apply from the very first photojournalistic practices till more contemporary experiments, does not address, however, the many issues, tensions, and paradoxes that a more complete and global approach should take into account. Although it is not our goal in this article, we cannot avoid mentioning the impact on photojournalism that the current crisis of veracity, or the so-called post-truth phenomenon is having on the perception of audiences and the business strategies and policies of media themselves. William J. Mitchell, in his very influential book The Reconfigured Eye. Visual Truth in the Postphotographic Era, already warned us that digital photography was meant to have an immediate repercussion on institutions like journalism and science itself (1992). Among others, Marzal & Casero point to what we could call an acceleration of the loss of credibility of photography in photojournalism 2.0. The fact that “editing” digital images is a widespread accepted practice to make them more aesthetically appealing and spectacular (2017: 14) has reached and changed also the public perception about photojournalism. The aura or mystery traditionally attached to analogical photography has been lost, while professional mastery in the production of informative photographs gave way to almost unlimited possibilities for masses of amateurs registering “my now and here” of the world (Sousa 2011: 8). Although it is very clear that this postphotographic scenario makes professional photojournalism even more relevant, it is also a fact that, quoting Lev Manovich title, software has taken command (Manovich 2013). In his postphotographic decalogue, the artist and essayist Joan Fontcuberta locates the function of images more in their circulation than in their content (2016: 39–40); more in their communicative present than in their evocation or memory of the past (114). The triumph of the public over the private and the ludic over the “formal” or “serious” functions of photography completes the cultural slime where Instagram, as the “perfect” postphotographic medium establishes its new reign. From a similar stance, Del Campo and Ernesto emphasize that: “Rapid advances in photographic technology – cell phones with high-quality cameras, immersive 360 photography and video, geolocation, powerful digital retouching on the mobile device – add numerous aesthetic and narrative possibilities, coupled with immediate dissemination on the web” (2017, 32; own translation). We can recall, among many others, a case like the images photographed by Anastasia Taylor-Lind in Naidan (Kiev 2014)1 superimposing an iPhone on the viewfinder of a Hasselblad camera and then uploading the photos to her Instagram account. Anyway, even if Instagram might be a good platform for photojournalism and seemed to announce a major change in the practice of photography, its potential has been proven (so far) more as a social network (a young culture full of self-identifying homogenizing models) and a marketing tool than as a photographic creative one (Cantón 2019).

1

http://www.anastasiataylorlind.com/maidan/h9ekevwjfwmsx41jebkf5k9j4jj5zl

958

J. Baetens and D. Sa´nchez-Mesa

In order to better grasp the very complexity of what photojournalism actually is, more precisely in light of the notion of intermediality, it is key to discuss the following problems, which are at the same time autonomous and tightly intertwined: (a) What is the relationship between words and pictures in photojournalism? (b) Who is the actual agent of photojournalism? (c) Can there be photojournalism outside journalism? A formula like: “Can the photojournalist talk? a nod to the famous postcolonial essay “Can the subaltern speak?” by Gayatry Chrakravoty Spivak (1988) might bring all these questions together in one single glance: (a) It asks whether a photojournalistic picture is only an image, that is a silent and only metaphorically a speaking object, or also a sign that functions in combination with other signs, in this case verbal signs. (b) It invites to reflect on who is doing the “talking,” even when the images are not accompanied, supervised, or marshalled by spoken or written words. (c) It opens a window to uses of photojournalism that may exceed or bypass the usual communication format of the press. In what follows, we will discuss these three angles, already including some historical examples that will be further developed in the second half of this chapter.

Question 1: Photojournalism and Intermediality As the very concept of photojournalism makes clear, this type of photography is inextricably linked with a very specific medium environment, that of the press. In that sense, all photojournalism is by definition intermedial (Baetens and SánchezMesa 2015), in the general sense of the word, since press pictures are always surrounded by words (titles, credits, captions, comments, and more generally the totality of the news medium that publishes it). Yet this intermediality cannot be reduced to the mere relationship between the image of the picture and the words that identify or use it, such as the caption or the article that uses the image as evidence. The scholarship on this type of intermediality is long-standing and very rich (see Scott 1999; for practical guidelines, see Smock 2018). However, it is possible to broaden the almost exclusive focus on the dialogue between the single image and the single caption. First of all, it is important to stress that in most news media, there exists an implicit or explicit “traffic” between the various parts or elements of the medium. This is for instance the case for the relationship between journalism and publicity: the way we read photojournalistic pictures is not only influenced by the words that accompany it and that helps (or force us to) interpret the picture’s meaning in this or that sense; it also depends on the presence of, and relationship with, non-journalistic material.

37

Photojournalism and Beyond

959

The necessity of having words next to pictures, which has always been a basic component of photojournalism, has been underscored by Bertolt Brecht, among others, and imposed as a “must” in photography studies by the comments of Walter Benjamin in his influential Short History of Photography (1931): The creative in photography is its capitulation to fashion. The world is beautiful – that is its watchword. In it is unmasked the posture of a photography that can endow any soup can with cosmic significance but cannot grasp a single one of the human connections in which it exists, even when this photography’s most dream-laden subjects are a forerunner more of its salability than of any knowledge it might produce. But because the true face of this kind of photographic creativity is the advertisement or association, its logical counterpart is the act of unmasking or construction. As Bertolt Brecht says, ‘The situation is complicated by the fact that less than ever does the mere reflection of reality reveal anything about reality. A photograph of the Krupp works or the AEG tells us next to nothing about these institutions. Actual reality has slipped into the functional. The reification of human relations – the factory, say – means that they are no longer explicit. So something must in fact be built up, something artificial, posed.’ (Benjamin 2005: 526)

But since most news media are commercial enterprises that cannot survive without advertisements, the role of publicity, which is always seen next to journalistic content, is absolutely crucial (and we know that the general organization and layout of news is made in such a way that consumers are either encouraged to watch and read also the advertisements or are actually prevented from skipping it). The point has been made by Marshall McLuhan, in his emblematic style (and typically taking as its starting point ads, not news): “Ads are news. What is wrong with them is that they are always good news. In order to balance off the effect and to sell good news, it is necessary to have a lot of bad news.” (1964: 210, emphasis in original; for a comment, see Curley 2013: 68). Photojournalism’s intermediality is therefore a practice that is radically plural. Instead of focusing on one specific type of word and image interaction (namely, the picture and its caption), we should envisage a wide range of intermedial relationships that progressively appear during the reading of the picture as it appears in a hybrid, if not properly heterogeneous context: intermediality is networked. The meaning of a picture certainly depends on what that image shows (it would be absurd to claim that the visual dimension of the message is totally framed by its verbal surroundings), but it depends no less on two other types of aspects. First the general way in which a certain news medium is practicing photojournalism: readers or consumers are “trained,” albeit often unconsciously, to make sense of this by their frequent and repetitive use of that medium: the more one knows how a certain medium works, the easier it becomes to interpret its way of using images (of course we should add that according to whether one feels sympathy or distrust toward a certain medium, the deciphering will be more or less critical). Second, the interaction between image and the photojournalistic and non-photojournalistic parts of the medium. The use of a politically committed image (and text, provided both reinforce each other) can be enhanced but also destroyed by the presence of other material. The opening word of the editor can reframe the journalistic content that follows, or readers can be shocked

960

J. Baetens and D. Sa´nchez-Mesa

or upset by the presence of advertisements they consider inappropriate, for instance publicity for risk investments next to an illustrated article on ecological topics. In the case of photojournalism, intermediality should, thus, be a word that only exists in plural: it always involves certain models (that is the paradigmatic dimension of intermediality) and, at the same time, it also involves a broad array of relationships with other words, but also with other images within the same medium context (that is the syntagmatic dimension of intermediality). Finally, and to come back to the initial question of where the image’s intermediality is situated, it is also important to underline that even uncaptioned images or images that are not otherwise framed (e.g., by the voice of a life presenter during a slideshow lecture) do have an intermedial dimension by themselves, either at the level of the pictured content (countless photographs include written or printed words or forms of oral utterances: we “hear” Lenin when we “see” him address the masses on the Red Square, we “read” the billboards that are massively shown in the photographs of the Depression era in the United States) or at the level of the viewer’s reception (even if a picture has no title, we are so used to the fact that images are not “untitled” – in painting, the word “untitled” is a perfectly accepted title – that we crave to produce one for ourselves, while the way we interpret visual signs is definitely, although never completely, determined by the language(s) we speak: Innuit people do not look at pictures of snow and ice the same way as people living in the desert. And what applies to intermediality in photojournalism more generally helps rethink similar issues in photography as a whole, where there is now an increasing interest for questions having to do with the host medium, that is the material support of the images (hence, for instance, the now well-institutionalized emphasis on the photobook, see Parr and Badger 2004, 2006, 2014, Berghmans 2019). Second, it is also possible to interpret photojournalism’s intermediality in a more radical way. In the traditional analysis, this intermediality is approached in a static way: given a certain photojournalistic use of an image, one asks which are the types of intermediality that can be disclosed. But as we have seen, there is often a temporal gap between the taking of the picture and the making of a photojournalistic news item. The image first exists as just that, an image, and after that it is turned into part of a news item. Yet even in the case of a reportage in real time, for example when we follow a photographer during a Facebook event, a similar gap exists, although not in the temporal sense, since the connection is so fast that we have the impression that there is no gap at all, but more importantly in the mediological sense of the word, since there are isolated images and images which are part of some type of journalistic communication. More generally speaking, the very existence of technological recording and publishing has radically deconstructed the very notion of “live” itself (see the pioneering work by Auslander 1999). The whole set of these shifts, displacements, and reframings, which can be very radical, makes clear that any form of photojournalism equally presents an aspect of transmedialization, that is the transfer from one medium to another, and that intermediality (in the general sense of copresence of media) and transmedialization (in the equally general sense of medium transfer) are not always easy to tell apart.

37

Photojournalism and Beyond

961

Intermediality is therefore not only plural because one should always read it as “intermedialities,” but also because it touches, in many respects, on processes and practices of moving between media which are properly transmedial.

Question 2: Who Is Showing, and Who Is Talking? Photojournalism is capable of making use of all kind of pictures, including those, to quote once again our starting definition, that do not refer to “current events based on facts and [that are] supported with proof or evidence,” such as family snapshots or other forms of vernacular photography. Yet the traditional photojournalist is himself a professional (himself, and not herself, since the number of female photojournalists has been, until recently, very low –interesting counterexamples are Margaret Bourke-White or Lee Miller, see below), generally working on a freelance basis for various types of media formats and organizations (for the example of the Life photographers, see Loengard 1998). Once again, this very simple statement – a photojournalist on the one hand, a publishing medium on the other hand – raises complex questions concerning both authorship and agency of the pictures that are taken. Although authorship (not to be confused with ownership) and agency (that is the principle that autonomous beings, agents, are capable of acting by themselves, thus asking questions such as: who decides to take the picture and to do it in this particular way?) are strongly intertwined, we will address the “who” question of photojournalism by sticking to the very general distinction between these two notions. As far as authorship is concerned, even a very superficial encounter with the photojournalistic corpus immediately discloses two striking features. First, not all pictures are credited. Second, the name that is credited is not always the name of the actual photographer. Examples of the first practice have always been – and are still – widespread, but the reasons of this anonymity can be various: photographers can be unknown; they may have good or personal (safety) reasons to withdraw their identity from the public eye; they may have no particular desire to see their name in print; the contract with the publishing agency may stipulate that by giving or selling their images the photographers also abandon their moral and commercial rights; the name of the photographer may have been forgotten or overlooked; and not last but certainly not least: the pictures may have been reused without authorization, not to say “stolen,” since copyright law is not always applied or interpreted in similar terms in all places of the world. Historically speaking, one may have the impression that photographers have become more and more successful to claim and keep both authorship and ownership (and the copyright that goes along), but this general impression is somewhat misleading or at least simplifying. Countless recent pictures still circulate uncredited, while we do know quite well the names of many nineteenth century photojournalists, including very famous ones such as Roger Fenton, who documented the Crimean War (for a discussion of the staged character of the most famous of his images, “Valley of the Shadow of Death,” taken 23 April 1855, see Dicker 2012).

962

J. Baetens and D. Sa´nchez-Mesa

On the other hand, authorship can also be transferred to photo agencies catering to newspapers and magazines, both print and online, and all kinds of broadcasters and platforms. In that case, credits are only given to the agency – although not systematically, since certain agencies are owned by the photojournalists themselves who then use a double signature, that of the collective that of the individual. The history of photojournalism presents however many examples of pictures where the functions of commissioner, photographer, and developer were still somewhat confused – as a matter of fact, similar questions appear each time a new medium appears (for the example of cinema, see Jeancolas et al. 1999). Take for instance the Civil War photographs of Mathew Brady (Trachtenberg 1990), a photographic entrepreneur documenting various “events” (read: battles) that were reprinted as engravings in Harper’s Weekly (Davis 1991), but that were actually taken by his main field photographer (read: “employee”), Alexander Gardner, who will eventually publish his images under his own name in a different format (see below). The French journal Le Miroir (“The Mirror”), a news magazine that turned into a picture magazine exclusively devoted to the Great War in the years 1914–1918, is an interesting historical point in case (Baetens 2022). Well known for its spectacular and often very gore action photographs, Le Miroir generally refrained from crediting its pictures, which were sent to the magazine by eye witnesses, sometimes soldiers working for the photography service of the French army, in the context of a wellfunded photo contest, with important financial rewards for the winning pictures. Contributing photographers had however to be “amateurs,” who were described by the weekly as “our reporters” or “the operator,” since participating in the contest involved the complete and unconditional transfer of all copyrights (moreover, one easily imagines that not all “amateurs” wanted to publicize the fact that they were reusing images shot on duty, that is for military purposes). In order to stress its claims to the absolute truth of its material, the weekly also explicitly excludes from publication “artistic” pictures, that is, manipulated or composite pictures. Things went however differently each time the magazine published pictures coming or taken from the enemy, which are then credited as “photo handed over by a German prisoner” or “taken by a German officer” in order to stress not only the exceptional character of these images but above all their authenticity. Le Miroir’s practice of not crediting the photographers was all the more conspicuous, since the drawn illustrations that it also printed, generally to visualize events that were too difficult or impossible to photograph, were all meticulously credited. Photographs and (spectacular) drawings communicated in more than one way, as hinted above in our presentation of networked intermediality: drawings did not only replace pictures, they also emulated them, and vice versa. The consequences of unstable and varying authorship for the question of intermediality cannot be neglected. The more authorship is fixed and the more copyright legislation defines the maker of a photograph as the agent having full rights on ownership, use, and reuse of this picture, the photographer will also be entitled to write also the corresponding captions, and perhaps even the text that accompanies it and that may constitute with the images a hybrid photo-essay (see below for the example of the Magnum reporter Raymond Depardon). In that regard,

37

Photojournalism and Beyond

963

the difference between contracted photographers well protected by modern legislation that grants important rights to actual makers and totally independent and selfpublished photojournalists has certainly reduced. But one should not forget that in today’s journalism, the role of the photojournalist is under high pressure, and the increased rights of some “stars,” who can be seen as the equivalent of auteurs in cinema, are not compensated by the difficult labor conditions of the “unhappy crowd” of photojournalists, generally underpaid and only allowed to make pictures as a side activity to other journalistic activities. Many questions of authorship already touch upon questions of agency. Indeed, there is much more in taking a photograph than “going there” and “pushing the button.” The former is often only possible if someone allows the photographer to do so (and in many cases, this “going there” also involves previous research, which is time-consuming and thus expensive and which cannot always be taken care of the photographer himself or herself). The latter is no more than a tiny part of the general process that in most cases completely escapes the photographer’s power and control. Someone else – an editor, but behind each editor there is a medium; and behind each medium there is a board; and behind each board there are investors and other stakeholders – takes over, choosing this image and discarding that one, cropping the image, adding or rewriting captions, composing and lay-outing the reportage and the page, framing the word and image combination. Even very successful and highly appreciated photojournalists relying on excellent personal contacts with the editorial staff and, thus, largely in control of their own agency, never cease to be confronted with this problem. Let us take for instance the example of Eugene Smith, whose famous reportage, “Country Doctor,” published in Life, September 20, 1948, and covering the difficult work of a physician in a thinly populated environment, Dr. Ernest Ceriani, working in the town of Kremmling, Colorado, has been described as the first extended editorial photo story (O’Hagan 2017). Even in spite of his unchallenged prestige and the good contacts with the Life picture editor, Smith was permanently struggling for control. He could not always chose his own themes and certainly not all specific details (in “Country Doctor,” Dr. Ceriani had been casted by Life whose editorial staff was looking for a physician resembling Frank Sinatra), some of his pictures were heavily manipulated, while accompanying texts and page layout were made or changed without previous discussion by the weekly. After his resignation from Life, where he had worked from 1939 to 1954, and joining in 1955 the Magnum agency, a cooperative owned by its photographer-members, Smith’s “agency” problems will not disappear (the funding of his Pittsburgh project – 3 years of work and some 21,000 pictures – was stopped by Magnum before the actual finishing, which did not produce the originally commissioned book). The example of Smith is telling, because it reflects the eventual failure of the photographer of staying in control, and extreme, for other photographers than Smith may be less demanding but therefore no less successful: complying with what the executives and the customers want is not necessarily a synonym of selling out. In conclusion, the fundamental intermediality question of showing and talking cannot be distinguished from larger issues of authorship and ownership, on the one

964

J. Baetens and D. Sa´nchez-Mesa

hand, and agency, on the other. Only photojournalists who really own their pictures can try to negotiate the way their work is published. But even then their control is never complete, unless of course they move their status from contracted or freelance author (and in both cases they can be seen as “subcontractors” of a larger assignment) to independent and self-publishing author. And even in this case, it is generally impossible to put between brackets the demands, concerns, and constraints of other stakeholders, such as for instance photobook publishers or curators. Intermediality is therefore much more than a technical or formal problem of word and image combinations, it is in the first place a political and institutional problem, having to do with the fundamental question of who owns the pictures and who has the power to use them.

Question 3: Photojournalism Outside Photojournalism? Since photojournalism is so hard a job, certainly in current days when the world of global and digitized journalism has difficulties in maintaining the specific position of a photojournalist (and the position of his or her previous supervisor, the picture editor, is going through similar issues), many photographers have started looking for alternatives. Generally speaking – and putting aside the possibility of just leaving the profession – the main options are open. First the transformation of photojournalism from within, actually a permanently ongoing process that will be briefly sketched in the next section. Second, the migration of the photojournalist techniques and ethos to other domains, either as a replacement for no longer existing opportunities in the traditional journalistic venues or in combination with such a career (a way of trying to have one’s cake and to eat it, at least from a financial point of view). Actually there exists two-way traffic: photojournalism migrates to other disciplines and contaminates non-journalistic work and vice versa. More and more photojournalistic work is done in non-journalistic fields. One may think here of the crucial role played by photography in disciplines such as visual anthropology and visual sociology, either directly, as a way of producing and documenting evidence, or indirectly, as raw data (see Gitelman 2013 on the pitfalls of the concept of “raw data” open to scientific scrutiny). A breathtaking example of such a migration is Michael Lesy’s Wisconsin Death Trip, the first-ever visual essay cum creative writing style PhD in history: First published in 1973, this remarkable book about life in a small turn-of-the-century Wisconsin town has become a cult classic. Lesy has collected and arranged photographs taken between 1890 and 1910 by a Black River Falls photographer, Charles van Schaik. Against these are juxtaposed excerpts from the writings of Hamlin Garland and Glenway Westcott. (Lesy 2000, n.p.)

Photojournalist techniques and approaches are widely used in all these disciplines (history, sociology, anthropology, creative writing, visual arts, publicity), although one has to stress that their very use in these new contexts supposes a shift from

37

Photojournalism and Beyond

965

photojournalism to photography. Thus, The Sage Handbook of Visual Research Methods (Pauwels and Many 2020) makes a clear distinction between photography as a “method” (and photojournalism is not included in the quasi-exhaustive list of possible visual methods) and photojournalism as a “subject,” which can be used for scientific investigation such as quantitative analysis and data-mining, with precise protocols on how to approach the material from an intermedial point of view. Here is the list of features to be checked as proposed in the Sage Handbook by Parry (2020: 359–361) in an example of quantitative content analysis of news items in the daily press: (1) newspaper, (2) date, (3) page, (4) size, (5) color, (6) top of page, (7) relevant headline?, (8) caption?, (9) location, (10) graphic nature?, (11) number of subjects, (12) gender, (13) age, (14) distance, (15) eye contact?, (16) angle, (17) photo subject. In this list, word and image are strongly intertwined, which draws attention to a specific problem that pops up when photojournalistic images are moving – and this move is another form of transmedialization – from a journalistic to another context (in this case a scientific context): the necessity of sharply defining the metadata that will be included and the policy that will be followed to tag the images. Tagging is a form of applied intermediality, which can be seen as a form of captioning in the expanded field and which has become absolutely crucial in the digital era where database-related queries are still largely dependent on verbal properties (although technological changes are evolving at high speed, current image searches on Google still need a strong verbal input; in order to find an image, one needs to know the relevant keywords; for a practical guide, see Russell 2019). The power and relevance of many visual databases is flawed or biased by the use of the wrong metadata (in most cases, by the lack of those metadata whose importance only appears as useful or imperative once the database starts to be really used) or by an erroneous, incomplete, or infelicitous use of tagging (it is not difficult to imagine how the metadata of archival collections can bear the influence of various forms of prejudices). In addition, tagging and metadata as applied intermediality are also an example of the radically open and, thus, permanently ongoing dimension of these word and image combinations. A phenomenon such as crowdsourcing, that is the invitation to users and audiences to add supplementary information or to enhance the quality of the already available information, has a huge impact on the way we should address this type of intermediality in photojournalism (Brabham 2013; Baetens 2017). The relationship between photojournalism and other forms of photography using photojournalistic techniques and protocols is not limited to scholarly or scientific uses. It equally involves the world of photography as art, in the very broad and general sense of photography as an independent way of making images (the relationship between photography and art remains extremely complex and controversial, as well as definitely unfinished one, as can be seen in the place that photography occupies in the modern museum, see Moschovi 2020). Here as well, it is possible to make a fundamental distinction between two forms, which we can call direct and indirect. The former designates the attempts to turn photojournalism away from its purely instrumental use and thus turn it into what we call today “art” (quotation

966

J. Baetens and D. Sa´nchez-Mesa

marks intended). The latter labels various types of artistic appropriation and reuse of photojournalistic material. In the examples that follow, it should not come as a surprise that these two types are strongly intertwined. Either of them proposes however new challenges to the question of intermediality, both at the internal level of word and image combinations and at the external and transmedial level of media shifts and combinations. Concerning the first option, the shift from photojournalism to photography tout court, the most important evolution has probably been the interwar emergence of a “documentary style” in photography. As argued by Olivier Lugon (2001), the primary theoretician and historian of this global trend in the years around 1930, this style is both a reaction against photojournalism, accused of being manipulative and propagandistic, and a creative reappraisal and transformation of its possibilities. In this debate intermediality was key, since it was mainly the use of biased captions – and in the case of the largely produced film documentary, the use of authoritative text panels or a didactic voice-over, as promoted by documentary film pioneer John Grierson (Nichols 2017) – that was seen as an attack on the image’s integrity and purity. “Documentary style” in the technical sense of the word was an attempt to liberate the photographic image from this verbal tutelage as well as to develop a new type of visual rhetoric more respectful of both the image and reality, and relying on five major techniques: (1) brightness, (2) clarity, (3) anonymity of the viewpoint, (4) frontality of the motif, and (5) density of information for the sake of readability (for a discussion of Lugon’s ideas and the distinction he defends between this “documentary style,” as represented by photographers such as August Sander and Walker Evans, and contemporary but not totally similar trends such as the “New Objectivity,” see Brückle 2006). The impact of the documentary style, not to be confused with documentary photography in general, has been huge and longstanding, including in the field of more ambitious forms of photojournalism. The case of Walker Evans, who is credited of having coined the notion of “documentary style” (for a full archeology of the term, see Lugon 2001), shows the complex intermediality of this practice. Although working for the photographic propaganda unit of a F.D. Roosevelt’s New Deal agency created to combat rural poverty, the Resettlement Administration (1935–1937), later called Farm Security Administration or FSA (1937–1942) and eventually turned into the Office of War Information Service (1942–1944), and thus having to comply, more or less willingly, with the overtly propagandistic aims of this employer, Evans managed to strengthen and develop his documentary yet non-propagandistic way of photographing during his FSA employment (actually: 1935–1938). The two major publications that came out of his documentary style photography of the 1930s, namely, the photobook American Photographs (1938) and the “illustrated” nonfiction book Let Us Now Praise Famous Men (1941, in collaboration with writer James Agee), are indeed two landmarks that foreground the effort to highlight the intrinsic power of pictures, independently of the verbal tutelage jeopardizing the value of most traditional documentary photography. American Photographs, actually the companion volume of a show at the MOMA in 1937, is a book that radically splits words and images in such a way that the former becomes almost invisible. The

37

Photojournalism and Beyond

967

images are still captioned, but these captions do not appear next to the pictures, but only at the end of each of the two parts of the book. Moreover, the captions in question are extremely simple and neutral: a brief identification of the subject, the location where the picture (although not systematically) was taken, and the date (the very first caption goes as follows: “1. License Photo Studio, New York, 1934,” but caption 19 is simply: “42nd St., 1929”). The equally famous photobook of Robert Frank, American Photographs (1958), will later follow and even radicalize this model (Frank’s captions systematically drop the date, for instance). A similar phenomenon can be observed in Let Us Now etc., a coauthored book that grew out of a (never published) assignment on the predicaments of three impoverished tenant farmer families in Alabama for Fortune, a deluxe yet socially committed and upscale magazine launched in 1929 by Time cofounder Henry Luce (the dates are important here: The Fortune project as commissioned in 1936, that is the year Henry Luce restyled Life magazine in order to make it a photo-essay weekly). Let Us Now Praise Famous Men is a groundbreaking example of literary nonfiction. Agee’s prose is notably lyrical and tackles human and metaphysical problems. The writer defines the project in terms of “an independent inquiry into certain normal predicaments of human divinity,” 1960: xiv), while stressing the autonomy of the images that accompany it, his preface calling words and photographs “coequal, mutually independent, and fully collaborative” (xv). Although the presentation of Evans’s images can vary according to the various editions (some editions break them up in different parts, not all of them insert them in the same place, not to speak of the hugely varying print quality of the pictures in a work that was often reprinted as a cheap pocket book), these very variations prove the radical autonomy of the pictures, bringing to the fore a kind of “obstructed intermediality”: words and images are put together between the two covers of the book, but the reader is warned to keep them apart. Next to the first option, the direct move of photojournalism to photography, there exists also a second, indirect type of “artification,” that is the transformation of a nonartistic item or practice into an artistic one, namely, the artistic appropriation of “real” photojournalistic work (Heinich and Shapiro 2012). Let us start here with a very simple example: Land of the Free (1938), a collection of poetry by Archibald MacLeish, a poet and writer, also collaborator to Luce’s Fortune, who was to become the ninth Librarian of Congress in 1944. In the spirit of the New Deal and relying on the FSA material (freely made available by the FSA agency for promotion purposes), this book both documents Depression poverty and acclaims the American value of equality. A great public success, the book attracted disparaging reviews all coming from academic circles, which drastically rejected what was the absolute novelty of this long collective poem: its innovative use of the word and image association. Yet unlike what happens in the work by Evans, pictures and texts lose here all autonomy. At the level of the captions, the principle does not change and is even more radical than in American Photographs, for even the names of the photographers are only revealed in the final credits. Many of these photographers are still illustrious today, yet more as artists than as photojournalists: among them Dorothea Lange, Arthur Rothstein, Russell Lee, Ben Shahn of Margaret

968

J. Baetens and D. Sa´nchez-Mesa

Bourke-White as well as Walker Evans, this last one only in a small proportion which may hint at the rather low propagandistic character of his FSA images which made them less appropriate for propaganda. Yet the pictures in this book, also exclusively laid out on the right pages like in American Photographs, are not deprived of a textual component as in Evans’ book (all one finds there on the left pages are page numbers), but combined with a stanza of the long epic poem that strongly reframes the pictures and that presents itself as the “sound-track” of the photographs. Despite of the fact that MacLeish’s name is the only one to appear on the cover as well as on the title page, quite a rude interpretation of the collective dimension of the work, Land of the Free was rapidly recognized as the pioneer of a new kind of intermediality. After having sketched a well-documented overview of writers and photographers collaborations in photobook format, the today’s prefacer of the poem, A.D. Coleman, notes that, contrary to all existing books defined by the domination of one of their communicative modes (be it the text or the images), Mac Leish’s work was “virtually sui generis” (Coleman 1977, n.p.), the artist being more or less the first of his kind to produce a book “in which words and images are merged in vital, organic ways so that they nourish and amplify each other, joined in such a fashion that the work (whether or not it ultimately succeeds) could not be considered complete if one or the other were removed.” (ibid.). There is certainly room for debate on whether or not Land of the Free can be considered an artistic success, but it cannot be denied that it successfully managed to appropriate journalistic images in a literary context. This kind of organic intermediality, where the very distinction between words and images tends to vanish or more precisely to be dialectically superseded by a new synthesis, has become a model of many artistic projects reusing photojournalistic elements, certainly when the basic materials have a strong intermedial dimension. The visual component has priority in much installation art reworking photojournalism, such as the already mentioned Wisconsin Death Trip. In both strands of appropriation, intermediality is less a challenge or opportunity than a mere instrument. Other artistic projects remain faithful to the classic model of internal hierarchy. The (political) textual message is clearly dominating in Hans Haacke’s documentary photo project “Shapolsky et al. Manhattan Real Estate Holdings, a Real-Time Social System, as of May 1, 1971,” an attempt to unmask an NYC real estate slum landlord, which has also become a staple in “institutional critique” (Buchloh 1999) using or creating a visual archive (the show, initially prepared at the Guggenheim Museum, was canceled some weeks before its opening, but later integrated in a larger MOMA exhibition (Haacke 1986). The “archive” itself, a concept that has been under close scrutiny since the early 1980s (see Sekula 1986; Tagg 1993; for a recent overview see Wells 2016), has become a staple feature of urban photography and documentary. A fascinating example of contemporary online activism is J.R. Carpenter’s 2008 “in absentia” project on Montréal real estate policies and gentrification (see also Baetens and Truyen 2013). Photojournalism outside photojournalism helps thus question the new doxa introduced by Roland Barthes in his first writings on photography. Taking stock in 1960 of modern photojournalistic practices, he famously noted:

37

Photojournalism and Beyond

969

(. . .) the text constitutes a parasitic message designed to connote the image, to ‘quicken’ it with one or more second-order signifieds. In other words, and this is an important historical reversal, the image no longer illustrates the words; it is now the words which, structurally, are parasitic on the image. The reversal is at a cost: [. . .] in the relationship that now holds, it is not the image which comes to elucidate or ‘realize’ the text, but the latter which comes to sublimate, patheticize or rationalize the image. (1977: 25)

What these transmedializations and appropriations tell us is that the traditional hierarchy of words dominating images is not simply reversed, but opened to a large spectrum of possibilities, which foster a creative clash between the sign types.

Part Two: A Short History of Photojournalism The history of photojournalism is well documented (Gunthert and Poivert 2007; Gervais and Morel 2020), and there is a wide consensus on its successive forms and phases, at least as far as Western (and today global) photojournalism is concerned, although, as usual, things are less clear-cut when we come closer to our present. The so-called Golden Age of journalism was generally situated in the period 1930–1960, triggered by some key technical developments (the compact commercial 35 mm cameras, Ermanox and Leica, becomes available in 1924 and 1925, as some years later the first flash bulbs – all these innovations allow the photojournalist to work “on the spot”) as well as the appearance of a new form of magazines that do no longer use pictures to illustrate texts but that experiment forms of telling that mainly rely on photographs – and of course on their layout, page design, and accompanying verbal elements. The major magazines were, in chronological order of appearance: Arbeiter Illustrierte Zeitung (Germany, 1924–1933), Vu (France, 1928–1940), Life (USA, 1883–1972, but with its photojournalistic era starting in 1936), Look (USA, 1937–1961) or Picture Post (UK, 1938–1957). Newspapers such as The Daily Mirror (London) and The New York Daily News (New York) followed this trend. Both magazines and newspapers of the Golden Age had specialized staff, freelance collaborators, and picture editors, whose contribution to the final output of the “photo-essay,” as the new way of photographically enhanced journalism was called, has always been crucial – sometimes at the expense of the photographers themselves who felt instrumentalized by their employers. As a reaction, the former will try to self-organize and to create photographer-owned agencies, the most famous example being Magnum, founded in 1947 by Robert Capa, Henri CartierBresson, David Seymour, George Rodger, and William Vandivert. It with this Golden Age that we associate names such as Robert Capa, Margaret BourkeWhite, Eugene Smith, and the other grand maverick, Henri Cartier-Bresson, the man who coined the notion of the “decisive moment” (1952), a notion that was then considered the alpha and omega of photojournalism. The Golden Age actually represents the acme of a longer evolution that is almost as old as photography itself, first in the context of war journalism, later in that of general information and actualities. From the late 1870s on, various press media

970

J. Baetens and D. Sa´nchez-Mesa

aimed at reinforcing and innovating journalism with the help of photographs. Here as well, a mix of technical developments and new editorial policies were at the cradle of this evolution. The first halftone reproduction of a news photography was published on March 4, 1880, in the New York-based The Daily Graphic, while flash powder was invented in 1887. Editorially speaking, the German Berliner Illustrierte Zeitung (1892–1945) is often seen as the first illustrated news magazine (corollarily, its star photo reporter, Erich Salomon, is frequently cited as one of the architects of modern photojournalism). From 1901 onwards, it printed photographs not just on the cover but inside the magazine, a model that will soon be followed by other magazines (see the already mentioned example of the French Le Miroir). There were however other magazines that had already introduced similar design and layout formulas, often in the emerging field of movie actualities (a genre that from the very start blurs the boundaries between information and entertainment, while being also a good example of “sponsored content”) or sports journalism (see Gervais (2007) for the example of La Vie au grand air [“Outdoor Life”], published between 1898–1914, with stunning, pre-constructivist compositions that will be a great source of inspiration for later general magazines such as, in France, Vu, which in its turn influences Life and other similar publications (see Frizot and de Veigy 2009; Leenaerts 2010; Sattertwaithe 2020). The acme of photojournalism was however a non-photojournalist event: the immensely successful and eventually globally touring “The Family of Man” 1955 photo show at the MOMA, curated by Edward Steichen, who designed the exhibition as a gigantic photo-essay, thus demonstrating as well as imposing photojournalism as the hegemonic way of doing photography in the 1960s (Moschovi 2020). The decline of photojournalism, which eventually resulted in the disappearance of most photo magazines in the 1970s as well as the progressive dismantling of the specialized picture units in magazines and weeklies, is mainly explained by the rise of new visual media, television in the first place, that offered a more vivid representation of the events. Susan Sontag’s collection of essays On Photography (1977), which highlights the power of the photojournalist fixed image, can be read as perhaps the last great defense of this approach of photography. In that sense, television was the perfect example of what Bolter and Grusin (1999) have coined as remediation, that is, the replacement of an old medium by a new one, the latter presenting a “stronger,” that is more realistic, more direct, more appealing take on the world than the former – a never-ending process of the quest of an increased reality effect. The Vietnam war was perhaps the swan song of photojournalism: it is still a moment that revealed unforgettable pictures and amazing photographers (see McCullin 2015), yet it also was the first war to be shown on television and the impact of the daily news on television definitely superseded the influence of the illustrated magazines. Although we seem to be very far from Eddie Adams’ (Associated Press) photo from February 1, 1968, of the assassination of Nguyen Van Lem (Viet Cong agent) by the chief of the Vietnamese national police, Nguyen Ngoc Loan, we have to acknowledge Kevin Maloney’s claim on the durability of this photograph compared with the same scene captured by the cameraman Vo Su (NBC TV). Even if these images were broadcasted the day after the photo was taken, according to Kevin

37

Photojournalism and Beyond

971

Moloney “The photograph is much more memorable of the horror of the moment than the 9 seconds of shooting (shock and gore)” (2019: 173). Remediation is, however, not only a technological or aesthetic phenomenon. It also has economic consequences, in this case the shrinking advertising revenue of photo magazines and newspapers in general and thus the rapid disappearance of photojournalism as the new form of journalism that was triumphant during the Golden Age. Subsequent evolutions such as citizen journalism and mobile telephone photojournalism will further reduce the importance of specialized photoreporters, who will then, as sketched above, look for other yet much more selective horizons. Many may feel called to restart a career in the museum or gallery circuit, but very few will be chosen. In spite of that it is possible to identify successful endeavors in this evolution such as the National Geographic Society. At the turn of the century, it revolutionized the use of photography in print media to the point (1930s) of replacing verbal text as the core of the narrative form. In the 1960s, the NGS began its conversion into a transmedia empire that now publishes content in more than 40 channels, digital, analog, and fully solid media, with photography always at the center (the 2016 National Parks series; 2014’s “The Future of Food” Project with 3555 photos, professional and amateur, between April and December). Their mastery and acknowledged efficiency in printing photographs in magazine format did not prevent their successful jump to Instagram, circulating the photojournalists’ photos in the field before the lavish reportage was published in the magazine’s pages (and with much more social interaction).2 While for commercial cinema this medium (Instagram) is nothing more than a marketing platform, in journalism those photos contribute to the development of storytelling (Moloney 2019: 181).

Part Three: Intermediality and Transmediality as Gateways to Medium Specificity in Photojournalism The history, as drafted in the previous paragraph, reflects a widely shared vision of what photography was but no longer is, and this outline is not really challenged today. We now know always “more” about this history, but its general structure and internal evolutions continue to be analyzed in the same terms by all historians. It is however possible to somewhat reframe this material by trying to bring to the fore the specifically intermedial aspects of it, which once again cannot be reduced to the mere debate on the relationship between pictures and captions or, more generally, on the power balance between word and image (are the pictures just illustrations or are they the driving force of the story?). These questions are of course vital, but there should be room as well for a different approach, of which we will briefly discuss some examples. 2

Kevin Moloney inquired in this successful evolution at his PhD Diss. Future of Story: Transmedia Journalism and National Geographic’s Future of Food Project, University of Colorado (quoted in Moloney 2019: 181).

972

J. Baetens and D. Sa´nchez-Mesa

In order to make possible “the production and distribution of reports on current events based on facts and supported with proof or evidence,” three major conditions must be fulfilled: (1) one must have the technology that is needed to take this kind of pictures, (2) one must have the possibility to be present at the event, (3) one must make sure that the picture taken will also be published by a news organization. Each of these questions rapidly proves to have a strong intermedial as well as transmedial component.3 As far as the technology is concerned, and this is question number one, the history of photojournalism could be described in function of the different problems, obstacles, or thresholds that have been solved and superseded since the first attempts to record an event till the current situation, still characterized by a strong distance in time and place between event and observer (photographer as well as viewer/reader), which Lev Manovich would describe as a fusion of body and screen (2001, 103–111), with an almost imperceptible distance in time, but still a critical distance in place, between photographer, event, and audience. Picturing an event was far from easy, if not impossible, before the invention of more advanced photographic technologies (faster films and lighter cameras, for instance), as can be seen in the fact that the first war reportages did not document actual events but the aftereffects of these events: this type of photojournalism, think of Fenton and Gardner, did not bring images of actual fighting, but images of corpses after the battle. In addition, one had also to take into account the time that was needed to transfer the locally taken picture to the print or broadcast organization, a problem that was not solved before (wirephotos only became possible in 1921). Today, almost “everything” seems technically possible, think for instance of the real-time viewing of an event filmed though a bodycam – and there are no indications that this type of remediation has yet reached its end (there is certainly work to do for those eager to develop the interactivity of these systems). Yet, even today the technical possibility to shoot is not always present (“embedded” photoreporters are certainly not allowed to go wherever they want. . .). What is interesting from the intermediality as well as transmediality point of view are the alternatives that have been explored in order to solve some of these problems.

Our concept of transmediality was established in Baetens and Sánchez-Mesa 2015: “[. . .] we suggest to use it to refer to the fact that more and more works tend to appear in various media. The key word in this depiction of transmediality is ‘tend to’: the reappearance of a work in another medium is a possibility, not a general law; moreover, its presence and impact depend on its historical context.” Deeply rooted in digital culture, transmediality is not a direct consequence of it and it must be considered in a double dimension: “On the one hand, transmedialization is the mechanism or process that adapts a work that exists in a given medium to another medium. We could call the result of these operations snowball transmedial narrative worlds (as for instance in many Hollywood franchises). [. . .] On the other hand, there is also the fact that certain works are not first elaborated in a given medium and then adapted into other media, but produced more or less simultaneously in various media, none of them being the ‘source’ of the other ones, by means of a multiplatform production design where a fictional/narrative world emerges through a complex planning of multiple media contributions” (292–293); this is Henry Jenkins’ transmedia storytelling (2006).

3

37

Photojournalism and Beyond

973

Chief in this regard is the use of engravings in the period that newspapers and magazines could not yet rely on halftones: instead of being printed, photographs were copied as engravings (woodcuts or other) reproducing authentic events. As convincingly argued by Martha A. Sandweiss (2002) and Davis (1991), these engravings did however not simply copy or reproduce photographs. They did not even try to emulate the “new” medium of photography or to hide their “old” status of engravings. They remediated photography, yet not in the conventional meaning of this term, but by foregrounding aspects of the old medium that were considered superior to those of the new one. Engravings were judged capable of making distinctions between essential and less relevant information, on the one hand, and to make the image tell a story, on the other hand. Instead, photographs blurred the distinction between what really mattered and what was not pertinent. For this reason, photography was less useful in producing the decisive moment that enabled the public to imagine what came before and what was about to follow –and this increased narrative power is obviously a strong intermedial feature: it is the (verbal) story one can infer from the engraving, whereas the photographic document seems just capable of showing a set of visual data and nothing more. The second question, that of the physical presence of the photographer, displays the strongly gendered aspects of photojournalism. Indeed, presence is not just a matter of speed, and thus of communication technology (in the tradition of medium theorist Harold Innis, one should always think together communication as the transfer of information and communication as the transfer of goods and persons, see Berland 2009). However, it is also a matter of daring and bravery, more precisely of the social permission to be courageous and to take subsequent risks –hence Sontag’s emphasis on the multiple meaning of “shooting.” Photojournalism is not necessarily male, as demonstrated by the vital contributions of female daredevils such as Margaret Bourke-White and Lee Miller, yet in practice the almost automatic link between photojournalism and dangerous working conditions, ranging from street photography in unpatrolled areas to all types of war photography, has given birth to the myth of the photojournalist as the acme of toughness and fearlessness, two properties traditionally seen as “essentially” male. Robert Capa’s often quoted maxim4 “If your pictures aren’t good enough, you aren’t close enough” (meaning: close to violent action) is as characteristic of our way of thinking the profession as Cartier-Bresson’s “decisive moment.” Hence for instance, in the 1950s, the key role of photojournalism as a role model of masculinity. As argued by Patricia VettelBecker in Shooting from the Hip (2005), “photography helped to reconstruct and redefine the American idea of masculinity after the traumas of World War II. [. . .] from 1945 to 1960 photography became increasingly concerned with restoring the male body and psyche, glorifying traditional masculinity – cowboys, boxers, athletes, military men – while treading carefully in a homophobic Cold War climate” (Vettel-Becker 2005, book blurb). In practice, photography stands here for

For some comments on this “travelling” quote, probably from the post-D-Day period, see https://www.buzzfeednews.com/article/katebubacz/get-closer

4

974

J. Baetens and D. Sa´nchez-Mesa

photojournalism, as articulated in opposition to less male (read: homophobic) forms of photography as exemplified by fashion and publicity photography, which often also need a verbal complement that “real,” quasi-combatant journalism is strong enough to do without. Critics of this perhaps naïve belief will reply by saying: “the first to go, the last to know,” and this critique has developed a strong transmedial dimension, as can be seen in the use of drawings replacing photographs in various forms of contemporary “slow journalism” (In’t Veld, n.p., 2023). The third question, that of publishing and distribution, has also a vital intermedial and transmedial dimension, which may remain hidden if one exclusively focuses on the relationship between the photojournalist and the organizations that reproduce their work or not. The shifting power balance between reporter and publisher is certainly an interesting way of going through the history of photojournalism, but it is no less important to stress the medium transformations caused by the possible frictions between both. The photobook for instance is not only a natural continuation of the commercial activities of a news agency (Life had an important book publishing branch for instance), but also a more or less viable alternative to journalism. The already mentioned conflict between Mathew Brady, commissioner and photographic entrepreneur, and Alexander Gardner, the immigrant and lower class photographer who struggled to get acknowledged for his pictures (Kelsey 2007) did not end with Brady’s selling of Gardner’s pictures to Harper’s Weekly, where they were published as engravings and attributed to Brady. It also resulted in the publication, 1 year after the hostilities, of the two volumes Gardner’s Photographic Sketchbook of the Civil War (Gardner 1959), an ambitious but commercially unsuccessful deluxe publication mixing words and images (with the pictures on the right pages and a heavily moralizing commentary on the left pages). As seen above, photobooks – a midway solution between journalism and the gallery and museum circuit – has often been a substitute for print or other forms of press distribution. As the Gardner example shows, this transmedial shift has crucial intermedial consequences, since it may allow the photographer to become in charge of her or his own verbal commentary, which can also be expanded extensively (as in the case of Frank’s The Americans, which included an important foreword by Jack Kerouac) or, on the contrary, strongly condensed (as in the case of American Photographs, Evans shying away from too strong editorial accompaniments). The work of the contemporary French reporter Raymond Depardon foregrounds examples of either of these possibilities. A fascinating and trendsetting example, adopted today by a large number of photojournalists is Depardon’s Correspondance New-Yorkaise (Depardon 1981). The gallery show of the book in 2018 presents it a way that clearly highlights the premier role of writing in photojournalism: In 1981, Raymond Depardon’s La [sic] Correspondance New-Yorkaise was published in the Libération newspaper in France. Composed of humorous, observational, photographic notes – and no topical news – the newspaper dedicated a full page to this correspondence every day for a month. It was a pivotal moment both in French photography and in Depardon’s career. La [sic] Correspondance New-Yorkaise marked a turn toward the “new journalism” of the era, which fed on daily life and featured first-person, subjective writing. (Magnum 2017, n.p.)

37

Photojournalism and Beyond

975

To conclude, this chapter has shown that photojournalism and intermediality are intimately related, yet not always in the traditional sense of the inevitable link between picture and caption. In the case of photojournalism, where images move between various host media, intermediality cannot be dissociated from transmediality. In addition, it also suggests that the generic concept of intermediality has to be nuanced with the help of concepts such as applied, networked, obstructed, or organic intermediality. Finally, the existence of photojournalism as an autonomous field, more precisely as a subfield of documentary photography, has also to be challenged in order to better understand how photojournalism works in the broader field of the media ecology.

References Agee, James, and Walker Evans. 1960 [1941]. Let us now praise famous men. Boston: HoughtonMifflin. Auslander, Philip. 1999. Liveness: Performance in a mediatized culture. New York/London: Routledge. Baetens, Jan. 2017. Crowdsourcing: A critical discussion on some issues of a challenging new practice in the digitization of photographic heritage. In Recherches en communication 42, 203–211. https://ojs.uclouvain.be/index.php/rec/article/view/48563. Last accessed 19 May 2023. ———. 2022. On journals and photographs: One-way mirrors? In Edinburgh companion to first world war periodicals, ed. Marysa Demoor et al., 2022. Edinburgh: Edinburgh University Press. Baetens, Jan, and Truyen, Fred. 2013. Hypertext revisited. Leonardo 46:5: 477–480. Baetens, Jan, and Domingo Sánchez-Mesa. 2015. Literature in the expanded field: Intermediality at the crossroads of literary theory and comparative literature. Interfaces 36: 289–304. Barthes, Roland. 1977 [1961]. The photographic message. In Image music text. Trans. Stephen Heath. London: Fontana. Benjamin, Walter. 2005 [1931]. Little history of photography, in selected writings, volume 2.: 1931–1934, ed. Michael W. Jennings, Gary Smith, and Howard Eiland, 507–530. Cambridge: Belknap. Berghmans, Tamara, ed. 2019. Photobook belge 1854–Now. Veurne: Hannibal. Berland, Jody. 2009. North of empire. Essays on the cultural technologies of space. Durham: Duke University Press. Bolter, Jay David, and Richard Grusin. 1999. Remediation. Understanding new media. Cambridge, MA: MIT Press. Brabham, Daren C. 2013. Crowdsourcing. Cambridge, MA: MIT Press. Brückle, Wolfgang. 2006. On documentary style: Anti-graphic photography between the wars. History of Photography 30 (1): 68–79. https://doi.org/10.1080/03087298.2006.10442840. Buchloh, Benjamin. 1999. Conceptual art 1962–1969: From the aesthetics of administration to the critique of institutions. October 55: 105–143. Cantón, Javier. 2019. Sociabilidad digital y creatividad digital. Construcción social de la imagen a través de Instagram. Granada: Universidad de Granada, 2019. Online: https://digibug.ugr.es/ handle/10481/56491. Last accessed 12 Jan 2021. Carpenter, J.R. 2008. In absentia. Online: https://luckysoap.com/inabsentia/jrcarpenter.html. Last accessed 13 Aug 2021. Cartier-Bresson, Henri. 1952. The decisive moment. New York: Simon and Schuster. Coleman, Archibald. D. 1977. Introduction (non paginated, but 6 pages) to MacLeish, Archibald. 1977 [1938]. Land of the free. New York: Da Capo.

976

J. Baetens and D. Sa´nchez-Mesa

Curley, John J. 2013. A conspiracy of images. Andy Warhol, Gerhard Richter, and the art of the cold war. New Haven: Yale university Press. Davis, Keith F. 1991. A terrible distinctness: Photography of the civil war era. In Photography in nineteenth-century America, ed. Marta A. Sandweiss, 130–179. Fort Worth/New York: Amon Carter Museum and Harry N. Abrams Publ. Del Campo, Elpidio, and Leónidas Ernesto. 2017. Fotoperiodismo contemporáneo: entre el documento y el arte. The Aftermath Project. AdComunica, Revista Científica de Estrategias, Tendencias e Innovación en Comunicación; n 13, January, 25–49. Online: https://raco.cat/ index.php/adComunica/article/view/343059. Last accessed 12 Jan 2021. Depardon, Raymond. 1981. Correspondance new-yorkaise. Paris: éd. de l’Étoile. Dicker, Ron. 2012. Valley of the shadow of death, famous early war photo, a staged fake, investigator says (PHOTOS), Huffington Post 1 Oct. 2012. Online: https://www.huffpost.com/ entry/valley-of-the-shadow-of-death-fake_n_1928760. Last accessed 13 Aug 2021. Evans, Walker. 2012 [1938]. American photographs: Seventy-fifth anniversary edition. New York: MOMA. Fontcuberta, Joan. 2016. La furia de las imágenes. Notas sobre la Postfotografía. Barcelona: Galaxia Gutenberg. Frank, Robert. 2008 [1958]. The Americans. Götttingen: Steidl. Frizot, Michel, and Cédric de Veigy. 2009. Vu. The story of a magazine. London: Thames and Hudson. Gardner, Alexander. 1959 [1866]. Gardner’s photographic sketchbook of the civil war. New York: Dover. Gervais, Thierry. 2007. La photographie mise en page dans La Vie au grand air (1898–1914). Etudes Photographiques 20: 34–49. Gervais, Thierry, and Gaëlle Morel. 2020. The making of visual news: A history of photography in the press. New York: Routledge. Gitelman, Lisa. 2013. Raw data is an oxymoron. Cambridge, MA: MIT Press. Gunthert, André, and Michel Poivert, eds. 2007. L’Art de la photographie. Paris: Mazenod & Citadelles. Haacke, Hans. 1986. Unfinished business. Cambridge, MA: MIT Press. Heinich, Nathalie, and Gisèle Shapiro, eds. 2012. De l’artification. Enquêtes sur le passage à l’art. Paris: EHESS. In’t Veld, Laurike. 2023. Graphic journalism. In The Cambridge companion of the graphic novel, ed. Jan Baetens, Hugo Frey, and Fabrice Leroy. New York: Cambridge University Press. Jeancolas, Jean-Pierre, Jean-Jacques Meusy, and Vincent Pinel. 1999. L’Auteur du film. Nîmes: Actes Sud. Jenkins, Henry. 2006. Convergence culture: Where old and new media collide. New York: New York University Press. Kelsey, Robin. 2007. Archive style. Photographs and illustrations for U.S. surveys, 1850–1890. Berkeley: The University of California Press. Leenaerts, Danielle. 2010. Petite histoire du magazine VU. Brussels: Peter Lang. Lesy, Michael. 2000 [1973]. Wisconsin death trip. Albuquerque: University of New Mexico Press. Loengard, John. 1998. LIFE photographers: What they saw. New York: Bulfinch Press. Lugon, Olivier. 2001. Le Style documentaire: d’August Sander à Walker Evans (1920–1945). Paris: Macula. MacLeish, Archibald. 1977 [1938]. Land of the free. New York: Da Capo. Magnum. 2017. Raymond Depardon: Correspondance new Yorkaise 1981–2017. Online: https:// www.magnumphotos.com/events/event/correspondance-new-yorkaise-1981-2017/. Last accessed 13 Aug 2021. Manovich, Lev. 2001. The language of new media. Cambridge, MA: MIT Press. ———. 2013. Software takes command. New York/London: Bloomsbury. Marzal Felici J, and Andreu Casero. 2017. El fotoperiodismo en la era de la posverdad. In AdComunica. Revista Científica de Estrategias, Tendencias e Innovación en Comunicación;

37

Photojournalism and Beyond

977

n 13, enero, 2017, 1–17. Online: https://raco.cat/index.php/adComunica/article/view/343057. Last accessed 12 Jan 2021. McCullin, Don. 2015 [1980]. Hearts of darkness. The new definitive edition. London: Jonathan Cape. McLuhan, Marshall. 1964. Understanding Media. New York: McGraw-Hill. Mitchell, William J. 1992. The reconfigured eye. In Visual truth in the postphotographic era. Cambridge, MA: MIT Press. Moloney, Kevin. 2019. Transmedia photography. Implicit narrative from a discrete moment. In The Routledge companion to transmedia studies, ed. M.Y. Freeman and R. Rampazo, 173–183. London: Routledge. Moschovi, Alexandra. 2020. A gust of photo-philia. Leuven: Leuven University Press. Nichols, Bill. 2017. Introduction to documentary. 3rd ed. Bloomington: Indiana University Press. O’Hagan, Sean. 2017 (August 6). W. Eugene Smith, the photographer who wanted to record everything. The Guardian. Online: https://www.theguardian.com/artanddesign/2017/aug/06/ w-eugene-smith-photographer-record-everything. Last accessed 13 Aug 2021. Parr, Martin, and Gerry Badger. 2004, 2006, 2014. The photobook. A history, Vols. I, II and III. New York: Phaedon. Pauwels, Luc, and Dawn Many, eds. 2020. The Sage handbook of visual research methods. 2nd ed. London/Thousand Oaks/New Delhi/Singapore: Sage. Russell, Daniel M. 2019. The joy of search. A Google insider’s guide to going beyond the basics. Cambridge, MA: MIT Press. Sandweiss, Martha A. 2002. Print the legend. Photography and the American west. New Haven: Yale University Press. Sattertwaithe, Tim. 2020. Modernist magazines and the social ideal. London: Bloomsbury. Scott, Clive. 1999. The spoken image. Photography & language. London: Reaktion Books. Sekula, Alan. 1986. The body and the archive. October 39: 3–64. Smock, John. 2018. Writing phot captions, in journalism basics. Online: https://ijnet.org/en/ resource/writing-photo-captions. Last accessed 16 Aug 2021. Spivak, Gayatri Chakravorty. 1988. Can the subaltern speak? In Marxism and the interpretation of culture, ed. Cary Nelson and Lawrence Grossberg, 271–313. Basingstoke: Macmillan. Tagg, John. 1993 [1998]. The burden of representation. Minneapolis: The University of Minnesota Press. Taylor-Lind, Anastasia. 2014. Maidan – portraits from the black square. Online: http://www. anastasiataylorlind.com/maidan/h9ekevwjfwmsx41jebkf5k9j4jj5zl. Last acessed May 15th 2023. Trachtenberg, Alan. 1990. Reading American photographs: Images as history, Mathew Brady to Walker Evans. New York: Hill and Wang. UNESCO. 2021. Journalism is a public good: World trends in freedom of expression and media development, global report 2021/2022. Paris: UNESCO. Vettel-Becker, Patricia. 2005. Shooting from the hip. Photography, masculinity, and postwar America. Minneapolis: The University of Minnesota Press. Wells, Liz, ed. 2016. Photography. A critical introduction. 5th ed. London: Routledge.

Media Borders in a Post-Media Age: The Historical and Conceptual Co-evolution of Cinema, Television, Video, and Computer Screens

38

Andrea Virgina´s

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . From “Broad” to “Genealogical” Intermediality and Back . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Historical Genealogy: A Sample in McLuhan’s Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Being a Screen, Framed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . After the Screen, Comes the Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

980 980 983 989 992 995 996

Abstract

This chapter offers an overview of literature theorizing our condition defined by electronic screens, often called a post-cinema age, the age of expanded or fragmented cinema, or indeed named the spatial turn in the analysis of electronically mediated audiovisual communication. With a faraway starting point in Marshall McLuhan’s Understanding Media and relying on Lars Elleström’s media theory throughout, the overview covers comparative theorizing of cinema and television, cinema and video, and cinema and digital screen(s). Such monographic titles are referred to as David Jay Bolter and Richard Grusin’s Remediation, Lev Manovich’s The Language of New Media, Sybille Krämer’s Medium. Messenger. Transmission, André Gaudreault and Philippe Marion’s The End of Cinema, as well as referring interventions by Roger Odin, Francesco Casetti, Giuliana Bruno, Thomas Elsaesser, Erika Balsom, or Irina Rajewsky and Laura Mulvey.

This work was supported by a grant of the Ministry of Research, Innovation and Digitization, CNCS – UEFISCDI, project number PN-III-P1-1.1-TE-2021-0613-P, within PNCDI III, and by the Bolyai János Research Fellowship of the Hungarian Academy of Sciences (2021–2024). A. Virginás (*) Faculty of Theatre and Film, Babeş-Bolyai University, Cluj-Napoca, Romania e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_46

979

980

A. Virgina´s

The media borders between cinema, television, video, and streaming are shown to be conditioned by historical developments in electronic communication technologies, by the fictional filmic representation of such developments, and finally by the critical-theoretical conceptualization of their codependencies. The concepts of broad intermediality (Elleström) and genealogical intermediality (Rajewsky) are proposed as denoting the default experience of our era of media convergence on the all-engulfing digital platform. The suggestion is made that the electronic screen has been existing as a messenger of medium specificity in the pre-1990s era, keeping its status amid the changed circumstances of the digital era too. Keywords

Screen · Electronic · Intermediality · Broad · Genealogical

Introduction The hybrid or the meeting of two media is a moment of truth and revelation from which new form is born. For the parallel between two media holds us on the frontiers between forms that snap us out of the Narcissus-narcosis. The moment of the meeting of media is a moment of freedom and release from the ordinary trance and numbness imposed by them on our senses. (McLuhan 1994: 55)

From “Broad” to “Genealogical” Intermediality and Back It is based on the four modalities – “the material, the sensorial, the spatiotemporal and the semiotic,” building “a medial complex integrating materiality, perception and cognition” (2010: 15) – and the “two qualifying aspects” – “historical, social, cultural circumstances” and “aesthetic and communicative characteristics” (2010: 24) – that Lars Elleström differentiates between basic media, qualified media, and technical media. Meanwhile, “[b]asic and qualified media must be understood as abstractions that need technical media to be materially realized” (Elleström 2010: 36, emphasis in the original). “Broad intermediality” – when border(s) between “dissimilar qualified media types based on similar basic media types” are crossed (Elleström 2020: 49) – could be the default experience of our era of media convergence on the digital platform. The icon of this condition is the electronic screen, and electricity is an element engendering the phenomena described thereinafter, in its capacity to “offer a means of getting in touch with every facet of being at once,” an existence” “only incidentally visual and auditory,” and “primarily tactile” as we read in Understanding Media (McLuhan 1994: 249). Brian Ott also emphasizes that electricity is “a structural feature of digital media,” adding to the “speed with which people process and manage information” (Ott 2007: 159). Electronic screens cue a high number of “qualified media types” (Elleström 2020: 49): analogue and digital television, analogue and digital video, and desktop and

38

Media Borders in a Post-Media Age: The Historical and. . .

981

mobile computer interfaces for streaming. As a matter of fact, Sylvie Leleu-Merviel enumerates seven types of screens defining our present age, arguing that “for a long time, we had two [screens] in our lives – cinema and television,” to which further ones have been added: “three, with the computer screen that opens up access to the immense resources of the Web and the Internet, . . . four with mobile tools – smartphones. Some are pursuing inflation by adding five (the intermediate terminal that is the tablet), six (the games console) and seven (the immersive helmet)” (LeleuMerviel 2020: xiii). Leleu-Merviel’s categorization is enabled by her primary interest in projection mapping – a practice and ensuing phenomena definitely beyond the tipping point of an electronic screen-based existence, composed of “screenscapes,” to use Francesco Casetti’s term (2019: 46). Defined by these enumerated elements and aspects, this chapter offers an overview of literature theorizing our condition defined by screens, often called a post-cinema age, the age of expanded or fragmented cinema, or indeed named the spatial turn in the analysis of electronically mediated audiovisual communication. Marker of media convergence and of fuzzy cases of broad intermediality as we will see, the electronic screen also mobilizes the arsenal for discussing (electronic) medium specificities. This may be identified as a constant need even if less and less occasions for the material actualizations of electronic medium specificities are available for being experienced by large contemporary audiences – who are exclusively linked to the digital platform. Watching an outdoor electronic screen of advertisements at a train station, a television screen at home, a translucent window screen in a museum, or immersing in the (once) silver screen at the movies are common situations for us all. Even if fully digitally mediated, such occasions recall the media history and specificities of the respective media – somewhat similarly to watching such fictional moments in film diegeses. Having the capacity “to nonneutralize” television, cinema, or video (art) – to use Sybille Krämer’s terminology (2015: 31) – our encounters with electronic screens might make these media “visible,” evoking their historical lineage to us, as spectators involved. According to the main argument developed currently, “non-neutralizing” and “making visible” electronic screen-based, historically developed media occurs not only on the level of materiality but also concerning the perceptual and cognitive processes involved. Or as Torben Grodal observes: “[t]he use of technical devices [to represent these processes in an intersubjective form] makes the mental processing visible; the gadgets are real, but they are also an extension of the way normal perception and cognition work, and are therefore mental models of perception and cognition” (Grodal 1997/2002: 242). Yet, modelling analogue television “mentally” while (being) cued by an electronic digital screen to do so definitely differs from “non-neutralizing” analogue television as such for the viewer of an analogue television set. As so adequately summarized by Irina O. Rajewsky in her influential 2005 article, we have reached the stage when “a medial difference is no longer a given, i.e., is no longer discernible” (Rajewsky 2005: 62–63) – at least not in the classical, analogue sense of medium specificities defined and intermedial relations developed among them. Moreover, “computer technology, with its increasing capability to (more or less) ‘perfectly’

982

A. Virgina´s

simulate earlier media forms, does not quite fit into a division of different intermedial subcategories . . ., and even more generally challenges any definition of intermediality that is based on medial differences” (Rajewsky 2005: 32). Lars Elleström’s already defined concept of broad intermediality, combined with issues of medium specificity in reference to electronic screen-based media, definitely initiates a framework for more nuanced descriptions and subsequent analyses, yet further specifications are needed. Although not negating the need to concentrate “on concrete medial configurations and their specific intermedial qualities” as Rajewsky so relevantly formulates (2005: 51), and evidently containing such instances too, the focus of this handbook entry lies elsewhere. Thanks to the ever-present formation of electronic screens and their cueing, occasionally even simulating medium specificities pertaining to pre-digital film, video, and television as well as the digital formations enumerated by Leleu-Merviel (2020: xiii), one is constantly reminded of the new situation that digitally coded audiovisual media, or new media have induced. Obviously, this is a contemporary subcase or actualization of Marshall McLuhan’s fundamental thesis, advanced in his paradigmatic 1964 Understanding Media, namely, that “media as extensions of our senses institute new ratios, not only among our private senses, but among themselves, when they interact among themselves,” with “[r]adio [having] changed the form of the news story as much as it altered the film image in the talkies” (McLuhan 1994: 53). Therefore, the author’s hope that the slice of examinations below is relevant enough to be classified as such “new aspects and problems” “under the heading of intermediality” that “have emerged, especially with respect to electronic and digital media,” while not just repeating truisms about “intermedial relations and processes per se . . . which have been recognized for a long time” (Rajewsky 2005: 44, emphasis in the original). Since the stakes are high: does the digital encoding transform the previously intermedial references from and to film, video, television, and the pre-digital image to intramedial references, allusions, and figurations (Rajewsky 2005: 54, emphasis in the original)? Or should we rather categorize these as cases of “virtual intermediality” (Rajewsky 2005: 62–63) within what Friedrich Kittler describes as “a total connection of all media on a digital base” (Johnston 1997: 5–6)? This chapter strives to show that in the context of twenty-first-century media convergence as described by Kittler, the concept of broad intermediality is essential because of an interconnected chain of causes. Even if encountering film, television, or video in their own pre-digital, distinct materialities, and medialities is an evershrinking or even nonexistent possibility on the large scale of global audiences, the intense and constant “intermedial referencing” of television or video from within the digital (streaming) platform is more vivid and flourishing than ever, including the “virtual simulation of a perceptible medial difference, in order to create a discernible effect of intermediality” (Rajewsky 2005: 62–63). Erika Balsom also identifies the phenomenon from within her field of study, contemporary gallery art working with film and moving images. Observing that convergence and the “dissolution of the boundaries of individual media has been met by a reassertion of medium specificities produced out of intermedial tension” (2013: 14), Balsom opens a whole new field

38

Media Borders in a Post-Media Age: The Historical and. . .

983

where the abovementioned examples populated by electronic screens will find their peers. Of course, considering the important differentiation signaled by Rajewsky – namely that “in intermedial references . . . only one conventionally distinct (either monomedially or, as in the case of dance theatre, plurimedially constituted) medium is present in its own specific materiality and mediality” (2005: 59, emphasis in the original) – is paramount from this perspective. Yet, the question remains: when a characteristically digitally produced feature film in the tradition of analogue filmic narratives includes various electronic screens as meaningful surfaces in their diegetic worlds – television sets like in Lost Highway (1997, David Lynch), Irma Vep (1996, Olivier Assayas) or Maps to the Stars (2014, David Cronenberg); computer or mobile screens like in Arrival (2016, Denis Villeneuve) or The Square (2017, Ruben Östlund); or indeed outdoor environmental screens like in Blade Runner 2049 (2017, Denis Villeneuve) – which or what are the media that are “present in its [their] own specific materiality and mediality”? It is evident that understanding the processes of mediation happening in such cases calls for what Irina Rajewsky describes as “genealogical approaches to intermediality,” the characteristic of which is their “main focus” “on the fundamental interrelatedness of earlier and newer media” (2005: 63), in the best McLuhanian traditions, we can add. This method – as hopefully demonstrated in what follows – enables us to describe and theorize our electronic media history conditioned by Elleström’s broad intermediality, actualized in the constant gestures of intermedial and/or intramedial referencing and (most conveniently) cued by the formation of the (electronic) screen. This can be argued to be the case since “genealogical intermediality” reinserts meaning into discussions “once a medial difference is no longer a given” – instead of making such discussion “pointless” (Rajewsky 2005: 62–63). Therefore, in what follows the “constructed and perceptual” media borders (Newell 2020: 3) between cinema, television, video, and streaming are shown to be conditioned by historical developments in electronic communication technologies, by the fictional filmic representation of such developments, and, finally, by the critical-theoretical conceptualization of their codependencies. Or as summarized by Balsom, [R]ather than buy into the notion that all media will converge into a homogeneous digital field, it is necessary today to interrogate the ways in which the boundaries between media are both articulated and blurred, to see the pair convergence/specificity as existing in a dialectical tension with one another that allows for a new thinking of historicized ontologies rather than a dissolution, or even disappearance, of a given medium. (2013: 17)

Historical Genealogy: A Sample in McLuhan’s Steps The first in the line of electronic screens related to qualified media is the projectionbased screen of film, conditioned by the analogue film platform, with its celluloidbased material modality, to use Lars Elleström’s terminology (2014: 37). It is in Understanding Media that McLuhan introduces his famous description of cinema

984

A. Virgina´s

and film based on the view projected and experienced, describing it as “a hot, highdefinition medium” (McLuhan 1994: 318), with its “message” being “that of transition from lineal connections to configurations” (McLuhan 1994: 12). In their capacity to generate classical narrative filmic diegesis, the analogue platform and the celluloid-based material modality have implied the usage of film genres, while “impos[ing] masculinity as ‘point of view’” (Mulvey 1989: 29), thus giving birth to what Laura Mulvey famously named “visual pleasure in mainstream film” in her pivotal 1975 essay (2005). Another concept fundamentally dependent on the projection of a flawless filmic image on the best possible (silver) screen, visual pleasure describes how “[i]n a world ordered by sexual imbalance, pleasure in looking has been split between active/male and passive/female. The determining male gaze projects its fantasy onto the female figure, which is styled accordingly. In their traditional exhibitionist role, women are simultaneously looked at and displayed, with their appearance coded for strong visual and erotic impact so that they can be said to connote to-be-looked-at-ness” (Mulvey 2005: 62–63, emphasis in the original). The emergence of Mulvey’s deep-impact critical concept has been conditioned by what we can retrospectively assess as the first technological challenge addressed to the analogue, celluloid, film-stock-based paradigm, and its screenic endpoint. Mulvey positions her insights into the way “the symbolic unconscious of the patriarchal order” (2005: 59) is constituted as being enabled by the development of “an alternative cinema” made possible by such “[t]echnological advances (16 mm and so on) [that] have changed the economic conditions of cinematic production, which can now be artisanal as well as capitalist” (Mulvey 2005: 59). We can no doubt add video technology mainstreamed in the next decade, the 1980s, described by Lars Elleström as “first launched as a technical medium” which “eventually gave birth to a qualified medium with specific aesthetic qualities” (2010: 44). Resulting in Mulvey’s creating such powerful, enduring concepts as visual pleasure, male gaze, and female to-be-looked-ness this smaller revolution on the borderland between the analogue paradigm and forthcoming digitalization is most suggestively staged but also narratively capitalized upon in a 1992 film directed by Lucian Pintilie, The Oak. The female protagonist, Nela lies in an unmade bed with her terminally ill father, and a celluloid Super 8 home movie projector is in function, placed between them on the bed, and projecting its content on the opposite wall in the cramped little room. The home video presents a long-gone St. Nicholas/Christmas party, with high-ranking communist army and party officials celebrating in an elite communist mansion, persons we possibly identify as Nela’s former entourage, while the little girl taking center stage in the events seems to be herself, several decades ago. Pintilie’s screen inserted in The Oak’s diegetic world is not an electronic one, but a projected smaller frame, showing a strange coincidence with the “triumph of projection over monitor-based presentation” brought forth by the 1990s (Balsom 2013: 20). The alternation of projected, monitor-based, and again projected screens within the electronic realm incorporates cinema, video, television and then the digital possibilities. Equipped with a monitor-based screen, television is the first example

38

Media Borders in a Post-Media Age: The Historical and. . .

985

of “cool” audiovisual media that “is visually low in data” (McLuhan 1994: 312–13). According to McLuhan’s conception, this means that television leaves “much more for the listener or user to do than a hot medium” (McLuhan 1994: 319) also because the television screen offers “some three million dots per second to the receiver” and they accept “only a few dozen each instant, from which to make an image” (McLuhan 1994: 312–313). This characteristic of electronic screens – as part of complex media apparatuses, cool or hot – is to be linked with a further important differentiation that also allows today’s audiences to construct and envisage pre-digital medium specificities based on the principle of Rajewsky’s genealogical intermediality. Namely, that these devices “hosting impermanent images” (Casetti 2019: 46) allow, on the one hand, for smooth “media representation” defined by Lars Elleström as being “at hand whenever a medium presents another medium to the mind: a medium, which is something that represents, and becomes itself represented” as in ekphrasis (Elleström 2014: 15). Or, on the contrary, a “nonneutralization” of the medium/media involved is just as possible, making these “visible” through glitches and noises – evidently more and more frequently perceivable on cool television screens. This is an idea developed in Sybille Krämer’s media theory, where the constant neutralizing of media is suggested for the sake of the message to be revealed – as “[t]he message is . . . considered primary, while the medium itself is secondary; it neutralizes itself, becomes invisible and disappears in its (noise-free) use” (Krämer 2015: 35). David Cronenberg’s 1984 film Videodrome sets up the rules of its diegetic electronic screen use aiming at making the medium visible and filling it with noises of all kinds already in the introductory credit sequence. First, animated letters fill the cinematic screen, their candy colors and rudimentary design disturbing, evidently, the cinematic immersion and recalling the “cool” television screen in a gesture of “non-neutralizing media representation,” to combine McLuhan’s, Elleström’s, and Krämer’s concepts. Then, a shortly visible screenic glitch of a black-and-white nonfigurative formation informs the actual viewer that the sensible surface of this screen does not bear messages as usual, as normatively should be the case, emphasizing thus its “cool” nature “visually low in data” (McLuhan 1994: 312–13). Already in these introductory “media representational” credit sequences, we are presented with what Sybille Krämer names “the medium’s inherent features” (Krämer 2015: 31): television’s specific framing techniques, the striking visual glitches and its two-dimensional, flat liveness. In their monumental Remediation Bolter and Grusin observe that McLuhan and British television theorist Raymond Williams “have both suggested that the poorer resolution or different lighting robs television of visual depth,” with “the flatness and coarseness of the traditional televised image” making “it harder to remediate the perspective techniques of photography and film” (Bolter and Grusin 2000: 186–187). Employing these “inherent” characteristics to full effect, the introductory sequences of Videodrome also “transmediate” (Elleström 2014: 15) television’s “functional logic” which “only takes effect when media are in use, “its (media) performance” (Krämer 2015: 31), thanks to the cinematic screen. In this case, we see the first character – Bridey, the secretary – appear after the mentioned “noisy” and “cool” title sequence as having

986

A. Virgina´s

been recorded, meanwhile, by a small video camera, a method flagging television’s less success in achieving “perceptual transparency” (Bolter and Grusin 2000: 186–187). In their capacity of material and technical platforms for the classical narrative film the “celluloid analogue,” the “artisanal 16 mm” (Mulvey), “cool” television (McLuhan) and 1980s video technology may be categorized as phenomena of what John Johnston – summarizing Friedrich Kittler’s poststructuralist media historical framework – names the “past modernity” when the “data flows of optics, acoustics and writing [have been] autonomous” (Johnston 1997: 5–6). Interestingly enough – and supporting this chapter’s claim that the electronic screen has been existing as a messenger of medium specificity in the pre-1990s era, keeping its status in the changed circumstances of the digital era too – in Understanding Media McLuhan also deduces the developments in the future along this formation. Film (and cinema, or moving image-based representation) will abandon “its manuscript phase,” pressured by television “to go into its portable, accessible, printed-book phase” (McLuhan 1994: 291–292). The result? Soon everyone will be able to have a small, inexpensive film projector that plays an 8-mm sound cartridge as if on a TV screen. This type of development is part of our present technological implosion. The present dissociation of projector and screen is a vestige of our older mechanical world of explosion and separation of functions that is now ending with the electrical implosion. (McLuhan 1994/1964: 291–292)

This McLuhanian vision is in perfect synchronization with the artworld processes summarized by Erika Balsom and referencing art historian Bill Horrigan’s retrospective analysis of how the 1990s brought “the end of a ‘golden age’ of video art and the advent of a different, more cinematic paradigm of moving images within the gallery,” thanks to the “triumph of projection over monitor-based presentation” (Balsom 2013: 20). The era starting with the 1990s offered us a glimpse of “the/[this] future,” when “a total connection of all media on a digital base will [have] erase[d] the very notion of a medium” in Kittler’s system too (Johnston 1997: 5–6). In his theory “of media relations,” Lars Elleström considers the digital base primarily a technical medium of “distribution” (2014: 14) with no specific “aesthetic or communicative” features (2010: 24). According to Elleström, and as far as the “sensorial modality” is concerned, the analogue and the digital cannot (should not?) be differentiated, as “[t]he procedural difference between analogue and digital technologies has no importance in itself when focusing on how the senses meet the material impact” (2010: 31). Sean Cubbitt expresses a similar opinion in his essay “Digital Aesthetics.” Cubbitt writes that “[d]eriving from the Greek, the term ‘aesthetics’ refers to the study of sensory or sensory-emotional values” (2009: 23), and he argues that “this poses a first problem in digital aesthetics” as “many aspects of digital media simply cannot be sensed”; moreover, “what you cannot see is often the most significant thing about digital aesthetics” (2009: 23–24; emphasis in the original). Cubbitt’s examples are, among others, the unseen,

38

Media Borders in a Post-Media Age: The Historical and. . .

987

unheard (pieces of) information going through wires and sent to space via satellites and then back. We might accept Elleström and Cubbitt’s media theoretical standpoints of not seeing a fundamental difference between the analogue and the digital “when they impact upon the senses” (Elleström 2010: 31). Claiming, therefore, that “the digital cannot be sensed” (Cubbitt 2009: 23) supports the argument of Bolter and Grusin that the digital media may be conceived of as a “remediation” of older media – “we call the representation of one medium in another remediation, and we will argue that remediation is a defining characteristic of the new digital media” (Bolter-Grusin 2000: 47, emphasis in the original) – while also introducing “a new ratio among the senses,” as McLuhan states (1994: 53). However, not losing sight of the argument advanced currently is paramount too: namely, proposing genealogical intermediality as the standard condition of intermedial effects in our era of “electronic-digital implosion” (McLuhan 1994: 291–292). Consequently, we need to assess digital filmmaking and cinema as yet another phase in what Kittler describes as “technological advance” (Johnston 1997) when the “symbolic unconscious” (Mulvey 2005) becomes, again, exceptionally visible as existing through the analogue celluloid technical medium’s symbolic legacy: classical narrative cinema and diegesis, and the (digitally mutating) cinematic apparatus, also displaying an on-living “patriarchal order” (Mulvey 2005). Both Elleström and Cubbitt might be thus quite right on the conceptual level of their respective theoretical frameworks. However, within the qualified medium of “the motion pictures” (Elleström 2010: 19), we may discern a deep interest and concern for the cohabitation possibilities of the analogue and the digital “media systems,” which are definitely considered both different and also to be sensed quite differently – in contrast to Elleström and Cubbitt’s theoretically generated standpoints. The “hot, high definition” cinema medium, introducing “configurations” (McLuhan 1994: 12), receives the role of bridging between the analogue and the digital. As Laura Mulvey so eloquently puts in a new millennium piece, “the cinema belongs as much, or even more, to the past as to the present” being “[o]vertaken by the novelty of the latest technologies, the electronic and the digital which create their own dimensions of time and space” (2004: xvii). The extraordinary powers of transformation of the cinematic apparatus’s have been noted by many, with [E]lements of the cinematic apparatus break[ing] out of the previously fixed network of relations of which they were once a part to now appear far from their usual configuration in new constellations that inhabit a murky interstitial space between cinema and its various others – television, the Internet, video games, mobile phones, and, of course, media art. (Balsom 2013: 14)

Cinema’s versatility definitely serves as a model in our present when the culturally and, perhaps, also cognitively funded differences among the mentioned technical and electronic media are being indexed by the various electronic screen(ic) formations. Olivier Assayas’ 2014 The Clouds of Sils Maria offers itself as an exquisite training ground about “cinema and its various others” (Balsom 2013: 14) in

988

A. Virgina´s

the context of “‘partially connected media systems’” (Johnston quoting Kittler 1997: 5–6). The French director juxtaposes the generation of digital non-natives to that of digital natives in a diegetic world centered on contemporary theatre and filmmaking, both spheres shown as deeply embedded with(in) the online internet sphere. The Clouds of Sils Maria introduces Maria Enders, a world-famous actress with a venerable career on the stage but also performing in action blockbusters “hanging from wires in front of a green screen”: she embraces the analogue and is cautious with the digital. Maria is positioned in contrast to both Valentine – her young “android” assistant whose two smartphones and tablet are fully integrated into her flesh-and-bone existence – and Jo-Ann Ellis, a hypothetical Hollywood blockbuster superstar, famous for her private scandals gone viral on the internet. The cinematically “transmediated” (Elleström 2014: 15) version of a theatrical performance constitutes the final part of the film: the play entitled Maloja Snake stages the bitter love story of a powerful firm executive (Helena as played by Maria Enders) with her ruthless young assistant (Sigrid as played by Jo-Ann Ellis). At the end of the theatrical scene – and that of the filmic sequence – Sigrid exits the geometrical, sterile office space toward the audience and stops at the extreme outer edge of the stage. The camera focuses on Jo-Ann-as-Sigrid’s angry, disillusioned, tired, and sad face: this female face is both filmed in real time – connoting the analogue system of representation – and simultaneously projected digitally on the huge canvas of the stage in magnified proportions. The view created is that of a beautiful female head filmed but also squeezed through the grid of pixels and geometrical lines, “cooling down” the “high-definition” cinematic medium (McLuhan 1994: 318). The analogue narrative filmic image of an actress performing a role in the sketchy environment of a theatre play is transmediated into the digital filmic image of the same theatre actress in front of our very eyes. The hybrid representation emerging – Jo-Ann Ellis playing as Sigrid was filmed and screened digitally and all throughout transmediated within the genealogically analogue conventions of the narrative fiction filmic close-up full with Mulvey’s “visual pleasure” (2005) – is neither an analogue filmic image nor a filmed theatre scene, or digital filmic image, but all at the same time. According to the main argument of this article, media differences in our era need the genealogical examinations of the nature presented above, since in the electronic digital twenty-first century, the most frequent instances of intermediality belong to broad intermediality as defined in Elleström’s media theory (2020: 49) and, thus, need a constant, also latent and genealogically inclined self-reflexivity to be identified as such. This need and cognitive process is proposed and demonstrated to be sustained by such constructions in film diegetic worlds where these various media traditions, indexed by corresponding screens, are present as apparently afilmic but actually profilmic objects with serious functions in the narrative development. The “constructed and perceptual” media borders (Newell 2020: 3) between cinema, television, video, and computer will be surveyed in the last part of this chapter throughout the critical-theoretical conceptualization(s) of their codependencies as far as their (electronic) screenic endpoints are concerned.

38

Media Borders in a Post-Media Age: The Historical and. . .

989

Being a Screen, Framed Francesco Casetti summarizes a possible genealogy by now famous: “[f]rom denoting a protection against fire or air, or a divide that splits a room, or a filter that sieves grain, the term ‘screen’ begins to designate in the early nineteenth century, in connection with the emergence of the Phantasmagoria, a surface that hosts impermanent images” (Casetti 2019: 33). Within this context of historically developing framed spectacles, screens currently are related to electronic and technical media: film, video, television, and computer or mobile (phone). These media not only produce or store but also distribute content, in accordance with Lars Elleström’s notion of a technical medium that “should consistently be understood not as a technical medium of production or storage but of ‘distribution’ in the precise sense of disseminating sensory configurations” (Elleström 2014: 14). Friedrich Kittler also emphasizes that storage and information manipulation are interweaving with transmission in the case of media as “[t]here are, first of all, media of transmission such as mirrors; secondly, storage media, such as film; and thirdly . . . machines that manipulate words or figures themselves” (Kittler 1997: 132–133). These definitions allow one to fix the screens in the moment of “distributing/disseminating sensory configurations” according to the various media apparatuses they are the endpoint of – [as] screens “define the way in which the visible is distributed, to use Jacques Rancière’s felicitous term” (Casetti 2019: 46, emphasis in the original). The dividing aspect of screens – their power to intervene in continuous space and introduce surfaces that “host impermanent images” (Casetti 2019: 46) – has been repeatedly commented upon by Lev Manovich too in The Language of New Media. For example, he draws attention to the fact that “a screen’s frame separates two spaces that have different scales–the physical and the virtual” (2001: 112, emphasis in the original). Manovich’s observation definitely reminds one of the generative dichotomy structuring Bolter and Grusin’s theory of remediation (1999/2000): immediacy, cued by what Manovich describes as “the scales of the physical space,” and hypermediacy – or “the scales of the virtual space,” separated, and also constituted by the frame. Considering it one type of parergon, like the title or a signature on a painting, or indeed the clothing on a statue, Jacques Derrida writes of the frame in The Truth of Painting that it “is the decisive structure of what is at stake, at the invisible limit to (between) the interiority of meaning (. . .) and (to) all the empiricisms of the extrinsic (. . .)” (Derrida 1987: 61, emphasis in the original). The French philosopher repeatedly returns to conditions of consistency for the parerga, and consequently for the frame, and his observations may be extrapolated to the case in point of the various electronic screens under scrutiny presently: Parerga have a thickness, a surface which separates them not only (as Kant would have it) from the integral inside, from the body proper of the ergon, but also from the outside, from the wall on which the painting is hung, from the space in which statue or column is erected, then, step by step, from the whole field of historical, economic, political inscription in which the drive to signature is produced . . .” (Derrida 1987: 61, emphasis in the original)

990

A. Virgina´s

The power of the frame “to create a de-contextualized zone” (Peretz 2017: 36) cannot be overemphasized in the present discussion of electronic screens, being the main constituent of the screen(s/ic) structure in its capacity of “mark[ing] an absolute cut between what it shows and its surroundings” (Peretz 2017: 36). In Ruben Östlund’s 2017 The Square, the museum curator, Christian, commissions the artwork of the same title, which is further qualified “as a sanctuary, a zone of trust and care,” being actually nothing else but a frame placed on the ground. The artwork entitled The Square creates, therefore, “a different scale and order of reality” (Manovich 2001; Odin 2016) within the confines of the museum space, to be filled by the spectator if they have the courage to cross over the frame and step inside. Visiting this artwork in the light-filled Scandinavian museum offers the occasion to experience “the screen . . . [as] a mental operator, a filter that produces distance and changes the perception of reality as it introduces points of reference (the edges of the frame) that lead us to build relationships that do not exist in reality” (Odin 2016: 83). This strange generative power of the frame will definitely have a role to play in The Square’s dramatic arch, the main character, Christian, being literally challenged “to build relationships that do not exist in reality” to protect this strange work of art, simultaneously restoring the balance in his personal life which has gone amok after he loses his mobile phone. This framing activity is the common characteristic of all screen-based media, from the painting to the mobile screen, with “the screen . . . ha[ving] been used to present visual information for centuries– from Renaissance painting to twentiethcentury cinema,” including of course the “computers” that “have become a common presence in our culture only in the last decade” (Manovich 2001: 94). In his computer and film history of intermedial composition – foundational for how we conceive old media as being influenced by new media nowadays – Lev Manovich repeatedly positions screens as devices that have been making possible experiences of virtuality, a characteristic that might be just as important as their relying on frames. It is by looking at a screen – a flat, rectangular surface positioned at some distance from the eyes – that the user experiences the illusion of navigating through virtual spaces, of being physically present somewhere else or of being hailed by the computer itself. (Manovich 2001: 94)

The specificity of the film or cinematic screen compared to other types of screens, such as computer monitors, becomes, however, evident too if we evoke the comparison of off-screen spaces in the case of film, and, respectively, computer interface. Persson observes that in cinema – which he also names a “realistic space” in contrast to the “abstract space” of interfaces – “the space ‘stretches out’ beyond the frame; concepts of left-right/up-down/off screen space are meaningful; objects look and behave more or less like everyday objects” (Persson 1999: 204). In contrast, “many (if not most) interfaces are not realistic in this sense,” as “[t]he space off screen (right-left or below-above) does not contain anything in particular and does not trigger any particular off-screen space expectations” since “[e]verything of interest is

38

Media Borders in a Post-Media Age: The Historical and. . .

991

contained within the frame. The landscape does not ‘stretch out’ into the distance in any direction” (Persson 1999: 204). Thus, the real cinematic space borders, on the one hand, on ungrounded computer screenic space, as we have seen. On the other, it touches on painterly space, described by Eyal Peretz in relation to the “outside of the frame [that] seems to be a (nonspatial) part of the painting, belonging to something we might call the fictional realm of the painting (a realm that is ‘larger’ or ‘more’ than what the painting makes visible” (Peretz 2017: 4). Interesting aspects emerge in its comparison to the art and practice of projection mapping, where framing exists, but it is not fixed throughout. Or as theorized by Leleu-Merivel, “[i]n cinema or video, the reported scene that fits into the window is dynamic and has movement, but the window itself is motionless” (Leleu-Merviel 2020: xiv), just like in the case of painting in its quality of “a window open onto a landscape and/or a scene to be inserted” (ibid.). These traditions and practices may be contrasted “to projection mapping, which brings out the image on the set, and where the specific geometry of the projection medium reappears” (ibid). This tension between meaningful cinematic off-screen space – grounded in the painterly off-screen – and computer presence ungrounded outside the frame of screen – interestingly mutated in projection mapping – is at the heart of recent digital melodramas like the 2002 S1mOne (Andrew Niccol), the 2013 Her (Spike Jonze), the 2015 Ex Machina (Alex Garland) or indeed Blade Runner 2049 (Denis Villeneuve, 2017). In these fictional creations, the analogue filmic traditions are hybridized with digital audiovisualizations within the same time-space continuum of film scenes. Men exist in permeable “real” cinematic spaces, endowed with analogue bodies, sense perceptions, and active trajectories in the diegetic worlds. Meanwhile, digital and computer screenic presences are feminized. Therefore, the low possibilities of actual romance define melodramatic narrative outcomes as in the case of Theodore Twombly’s meeting Samantha, the operation system in Her; Caleb’s and Nathan’s falling for the artificial intelligence’s bluish wires in Ex Machina; or the new generation blade runner greeted by the marvelous yet digital wife escort (see Virginás 2017). Similarly, the particular moments of detecting figures searching through and with the help of diegetic electronic screens in narrative – fictional or documentary – creations dramatize the tension between the realistic screen of cinema, where off-screen space is full with meaning, and the abstract space of interfaces, where (the) off-screen is devoid of meaning. In titles as varied as the 1984 Blade Runner (Ridley Scott), the 2002 Minority Report (Steven Spielberg) or Niels Arden Oplev’s 2009 The Girl with the Dragon Tattoo, the investigative documentary Collective (Alexander Nanau, 2020) or the 2021 Netflix-hit Don’t Look Up (Adam McKay), we can survey the modes of constructing such scenes. Therein, major data are represented and analyzed by investigators – be they policemen, journalists, or scientists – on televisual, video, computer, and mobile screens, in such modes as to influence the course of fictional or documentary diegetic events that we, spectators, always encounter on the “cinematic” screens. Such scenes showcase aspects of how we are able to conceive of our digitally interconnected human existence, contrasting but also hybridizing the electronic and the cinematic screens. For

992

A. Virgina´s

example, when in Oplev’s film the content of the laptop screen that Blomkvist, the investigating journalist watches, will cover the entire cinematic screen in an extreme close-up of long disappeared victim Helen Vanger’s digitalized analogue photograph in sepia shades. Such constructions unsettle the conception of what Manovich names the tradition of “the screen as representation” and “being marked by a rectangular frame,” in favor of the “screen as simulation” (2001: 112). The film’s spectator watching the protagonists navigate through computer screens arrives to experience an “aim[ing] to blend virtual and physical spaces rather than to separate them,” with “the two spaces hav[ing] the same scale; their boundary [is] de-emphasized,” and “the spectator [is] free to move around the physical space,” even if just hypothetically, given the cinematic dispositif (Manovich 2001: 112). The representation vs. the simulation tradition of screens evokes another fundamental difference between pre-cinematic and cinematic versus post-cinematic electronic screens, identified by Vivian Sobchack in a 1990 essay based on their capacity of generating or not a diegetic reality through establishing a point of view in unfolding space. Sobchack observes that “ungrounded and uninvested as it is, electronic presence has neither a point of view nor a visual situation, such as we experience, respectively, with the photograph and the cinema,” (Sobchack 2000: 80–81). This is a pertinent observation actualizing another of McLuhan’s visionary prophecies, namely, that “[t]he partial and specialized character of the viewpoint, however noble, will not serve at all in the electric age” (1994: 5).

After the Screen, Comes the Noise Cinematic and film cultural contexts have collided abruptly with convergence culture since the 1980s and 1990s, being more weakly equipped in dealing with multiscreen realities, yet obviously in need of reacting to global trends in media technological developments. This line of analysis luckily blends with the post-2015 resurgence of interest in screens and their theories, while being an appendix to Gaudreault and Marion’s theory of how filmic diegesis exists and exerts its influence in the digital, and even in the post-digital era. It is of the latter that Florian Cramer observes that “[i]t is an approach to digital media that no longer seeks technical innovation or improvement, but considers digitization something that already happened and can be played with” (Cramer 2015 n.p.). In their co-authored volume The End of Cinema?, André Gaudreault and Philippe Marion set up a system based on twentieth-century media history, taking as a principle the substitution of the cinema silk screen by the electronic cathodic television screen, and then by the electronic portable small computer screen. They observe that “[o]ne of the principal effects of the digital shift has been the big screen’s loss of hegemony,” with “projection onto a movie screen ha[ving] become just one way among others to consume images. The screen may have a greater aura, but it is now just one means of consumption among others” (Gaudreault and Marion 2015: 9, emphasis in the original). Gaudreault and Marion ultimately argue that “[w]e might even view the emergence of the small (but highly cathodic) screen as the point of rupture between a ‘hegemonic cinema’ and

38

Media Borders in a Post-Media Age: The Historical and. . .

993

this ‘cinema in the process of being demoted and shared,’ which is often called ‘expanded cinema’ but which we believe would be more appropriately described as ‘fragmented cinema’” (2015: 11, citing Guillaume Soulez’ conference intervention). Thus, hegemonic cinema would denote the first part of the twentieth century when the cinema theatre silk screen was the sole framed surface which displayed electronically mediated and also always pre-recorded moving images. Expanded cinema should denote developments of the second part of the twentieth century, when television and then video camera and screen appeared as electronic surfaces where previously exclusively cinematic worlds and narratives would expand, altering the nature and the significance of framed storytelling based on moving images. Finally, the twenty-first century brought us into the era of what Gaudreault and Marion name fragmented cinema, with the same cinematically constructed narrative worlds scattering further on “the electronic portable small computer screen,” and also becoming compatible with such surfaces. The analyses presented previously demonstrate that cinematic diegetic worlds, and other electronic screen-based audiovisual representations, react with mutual flexibility to these mutations. Filmic diegeses incorporate smaller electronic, usually portable screens in the modes already described, and all kinds of screenscapes are also eager to overtake the content of once hegemonic cinema screens, in a successful survival strategy of what Gaudreault and Marion call “non-hegemonic-cinema-in-the-digital-era” (2015: 14). It is in this respect that examining the role of electronic screens at large and/or their embedding in film diegetic worlds can be said to belong to what Thomas Elsaesser names “the archaeology of the screen and the frame,” (Elsaesser 2016: 112) performed while these ruins are not fully covered by layers of earth. We also need to mention that the various instances of electronic screens analyzed beforehand along the principles of a broad and genealogical intermediality, and in their capacity of standing in for non-hegemonic cinema in the digital age, contradict the widely held idea that contemporary screens need to “mask[ing] the mediate (that is, not immediate) conditions of [their] working” (Rubio Marco 2016: 222, emphasis in the original). Relying on the historical development of the object and the denomination of “screen,” Rubio Marco highlights that “every screen is, in a way, a ‘masking screen,’” with “the history of screens . . . [equaling] the history of the naturalization of that mediation to the point at which screens lose their excess baggage (‘masking’) in order to become just ‘screens,’ allegedly enabled to show every bit of the reality they refer to” (Rubio Marco 2016: 222, emphasis in the original). The examined sequences when (diegetic) electronic screens are scattered within the non/fictive spaces are aiming for non-neutralizing the media involved, making them visible primarily not for the diegetic spectator but rather (for) the actual viewer, thus offering such “cognitive representational models” (Grodal 1997/2002: 242) that allow us to decipher or construct medium specificity/ies in our post-digital age. This often happens through introducing media and/or technical noise as a spectacle in the functioning of electronic screens. In the 1997 Lost Highway (David Lynch), possibly as an effect of the noisy video/televisual set the two characters watch in their living room, the whole cinematic screen becomes blurred, and is covered with nonfigurative patches of light and dots – recalling “the

994

A. Virgina´s

low-definition, cool medium” of McLuhan. Noise introduced makes us realize a case of broad intermediality to be uncovered through genealogical excavations referring to the cinematic medium, “un-aisthecizing” it, to refer back to Sybille Krämer’s thesis. Namely, that “[t]he implementation of media depends on their withdrawal,” in a process that the German media philosopher calls “aisthetic self-neutralization” and that “belongs to the functional logic of media,” not being “an inherent feature of the medium itself, but rather it only take[ing] effect when media are in use” (Krämer 2015: 31, emphasis in the original). Consequently, “[t]he invisibility of the medium – its aesthetic neutralization” can be said to be “an attribute of media performance” (Krämer 2015: 31, emphasis in the original). This performative aspect has been reappearing along the preceding examinations of electronic screens, and it must have happened so, since, as Francesco Casetti so adequately puts it, “contexts,” “set[s] of operations,” and “basic operations” “allow a screen – whatever its materiality and substance – to perform as a screen” (Casetti 2019: 29, emphasis in the original). Such “an archaeology of the screen and the frame” (Elsaesser 2016: 112) definitely belongs to Manovich’s representation tradition, since the virtual and the physical spaces brought to life by the framed screen have (a) clear demarcation (line), “with an emphasized boundary” along the “rectangular frame” evident, and “the spectator” not “free to move around the physical space” (2001: 112) – unlike in what Manovich names the “simulation tradition” (ibid.). Fifteen years after Manovich’s fundamental distinction, Roger Odin denoted the vanishing point of screen- and frame-based audiovisualizations. In his important 2016 essay he theorized such inspiring concepts as mental and dream screens, suggesting that “[t]he notion of a mental screen corresponds to physical screens (cinema, television) that have become mental spaces” while “[d]ream screens are mental screens waiting for physical manifestation; one must note that this type of screen is the source of certain inventions: cinema and television have been dream screens before being invented” (Odin 2016: 185). Odin concludes that “this trivialization of the frame-screen [e.g., its proliferation though the small mobile screens] should not hide the opposite trend, even if it is still marginal: its disappearance with the emergence of virtual reality” (Odin 2016: 185). The proliferation of multiple screens in our everyday has been usually theorized in their capacities of gadgets or “vertical viewing dispositifs” with practical purposes and displaying Harun Farocki’s “functional images” or Wanda Strauven’s “image+”-s that need to be manipulated (Strauven 2016:144). From the standpoint made evident by Odin, and adding the metaphorical usage of such electronic screens with the aim of building narratively complex worlds, may be named the swan song of a past system of representation based on a framing activity. This suggestion is definitely built upon in film historian Thomas Elsaesser’s 2016 monograph Film History as Media Archaeology: Tracking Digital Cinema, where he proposes a cinema that “does not project itself as a window on the world nor requires fixed boundaries of space like a frame,” but “it functions as an ambient form of spectacle and event, where no clear spatial divisions between inside and outside pertain” (Elsaesser 2016: 133). With several such theoretical and artistic cases enumerated in this chapter, closing the line with some installations that literally and metaphorically

38

Media Borders in a Post-Media Age: The Historical and. . .

995

strain the frame of the electronic screenic constructions serves as demonstration and conclusion too. American art historian Giuliana Bruno describes the case when “[w]e no longer face or confront a screen only frontally but rather are immersed in an environment of screens” (Bruno 2014: 102), making a reference to Pipilotti Rist’s 2010 installation Layers Mama Layers. Another 2007 installation of the Swiss artist is on permanent display in the Arhus Museum of Modern Art. Dawn Hours in the Neighbour’s House definitely fits Bruno’s description of the process “where one becomes an integral part of a pervasive screen environment in which it is no longer preferable or even possible to be positioned in front of the work” (Bruno 2014: 102). From the window panes of the terrace, on to the plasma TV screen, through the floor and the edge cover of the books on the shelf: in Dawn Hours, all mentioned objects function as screens that light up and then fade in the dark, creating a “a fluid, haptic world of surrounding screens,” (Bruno 2014:102) from where there is no escape. Another example could be Apitchatpong Weerasethakul’s 2013 single-channel silent video One Water that the author had the chance to see exposed at the Vilnius Museum of Modern Arts in early 2018. What one sees is a naked electronic screen on a metal frame at a height less than human eyesight and placed in an immense and otherwise dark hall, in the company of further electronic screens. This screenscape, together with One Water’s exhibition guidelines – “Tilda Swinton, a British actress who became a good friend of Weerasethakul, organized a gathering of friends in the Maldives. Weerasethakul asked Swinton to recall her dreams in front of his camera” – introduce the audience in electronic screenic gallery art on its way to the cinematic dispositif. One Water retrospectively can be seen as a stage in the creation process of the director and the actor’s 2021 fictional feature film presented with great success at art film festivals, Memoria. The eerie, low-definition, cool image of a Swinton lost in her own dreamy processes (and) as seen on a mesmerizing electronic screen may be named a par excellence manifestation of “the mind and the gaze being captured as in hypnosis” (Chateau 2016: 197), enacting the framed screen’s capacity to offer “an experience of close relations between subject and object” in Bruno’s words (2014: 86). This last aspect of the framed screen, including the diegetic electronic screens in fiction filmic worlds, their “capturing our wandering minds,” is theorized extensively by Bruno in her 2014 Surface: Matters of Aesthetics, Materiality and Media as “an actual projective surface onto which an experience of close relations between subject and object is inscribed, in a way that overcomes divisions between outside and inside, inward and outward” (Bruno 2014: 86).

Conclusion All the analyzed sequences and examples, including those when diegetic electronic screens are woven into the filmic narratives and within the diegetic spaces, dramatize the barely palpable threshold between what Lars Elleström names “mediation” and, respectively, “representation” in the 2014 Media Transformation: The Transfer of Media Characteristics Among Media. In his formulation, “mediation is a presemiotic

996

A. Virgina´s

phenomenon and should be understood as the physical realization of entities (with material, sensorial, and spatiotemporal qualities, and semiotic potential) that human sense receptors perceive within a communication context,” like hearing “the sound of a voice” (Elleström 2014: 12). In contrast, “representation is a semiotic phenomenon and should be understood as the core of signification,” since “[a]s soon as a human agent creates sense, sign functions are activated and representation is at work;” thus, “one may interpret the sound of a voice as meaningful words” (Elleström 2014: 12). The fictional filmic characters, but also us, as real-world spectators, in our practices of “looking at a rectangular frame” (Manovich 2001) – which is hypnotic in Chateau’s sense and which has a different scale than the firstlevel diegetic reality, emphasizing both their separation and difference – are in the process of “creating cognitive import.” This, then, goes beyond/above “human sense receptors’ [perception]” (Elleström 2014: 12), in a mode homologous to One Water’s Tilda Swinton recalling her dreams in front of a camera and experienced by the audience on a museum electronic screen. These examples are also instances of technological interactivity within the diegetic filmic worlds for the fictive characters involved, and even more so for the actual spectators interpellated by the various cool, rather than hot media asking for heightened participation. Through their role of questioning, expanding, erasing, or simply dispersing the narratively valid information on various electronic surfaces, they also force what Gaudreault and Marion name “passive viewer” watching “linear cinema” (2015: 10) into a highly active one. She must energize her mental screen of electronic small portable screens within a filmic diegetic world, or vice versa, simultaneously while constructing the narrative world(s) as having multiple, intersecting levels. Throughout these processes, electronic screens related to various media have been existing as messengers of medium specificity in the pre-1990s era, keeping their status in the changed circumstances of the digital era too. Media differences call for the genealogical examinations of the nature presented above, since in the electronic digital twenty-first century, the most frequent instances of intermediality belong to broad intermediality as defined in Elleström’s media theory (2020: 49) and, thus, need a constant, also latent genealogically inclined selfreflexivity to be identified as such. This chapter developed these interrelated claims while suggesting that broad and genealogical intermediality (Rajewsky 2005) need to be positioned as the standard condition of intermedial effects in our era of “electronic” and also digital “implosion” (McLuhan 1994: 291–292).

References Balsom, Erika. 2013. Introduction – The Othered cinema. In Exhibiting cinema in contemporary art, 9–26. Amsterdam: Amsterdam University Press. Bolter, Jay David, and Richard Grusin. (1999) 2000. Remediation. Understanding new media. Cambridge, MA/London, UK: MIT Press. Bruno, Giuliana. 2014. Surface: Matters of aesthetics, materiality, and media. Chicago/London: University of Chicago Press.

38

Media Borders in a Post-Media Age: The Historical and. . .

997

Casetti, Francesco. 2019. Primal screens. In Screen genealogies. From optical device to environmental medium, ed. Craig Buckley, Rudiger Campe, and Francesco Casetti, 27–50. Amsterdam: Amsterdam University Press. Chateau, Dominique. 2016. Between fascination and denial: The power of the screen. In Screens. From materiality to spectatorship – A historical and theoretical reassessment, ed. Dominique Chateau and José Moure, 186–199. Amsterdam: Amsterdam University Press. Cramer, Florian. 2015. Post-digital aesthetics. Le Magazine Jeu de Paume 1st May 2015. http:// lemagazine.jeudepaume.org/2013/05/florian-cramer-post-digital-aesthetics/. Last downloaded 5 February 2018. Cubbitt, Sean. 2009. Case study: Digital aesthetics. In Digital cultures. Understanding new media, ed. Glenn Greeber and Royston Martin, 23–29. Maidenhead: Open University Press. Derrida, Jacques. 1987. The truth in painting. Trans. Geoff Bennington and Ian MacLeod. Chicago/London: The University of Chicago Press. Elleström, Lars. 2010. The modalities of media: A model for understanding intermedial relations. In Media borders, multimodality and intermediality, ed. Lars Elleström, 11–50. Houndmills, Basingstoke: Palgrave Macmillan. ———. 2014. Media transformation: The transfer of media characteristics among media. Houndmills, Basingstoke: Palgrave Macmillan. ———. 2020. The modalities of media II. An expanded model for understanding intermedial relations. In Beyond media borders: Intermedial relations among multimodal media, ed. Lars Elleström, vol. 1, 3–91. Houndmills, Basingstoke: Palgrave Macmillan. Elsaesser, Thomas. 2016. Film history as media archaeology: Tracking digital cinema. Amsterdam: Amsterdam University Press. Gaudreault, André, and Philippe Marion. 2015. The end of cinema? A medium in crisis in the digital age. Trans. Timothy Barnard. New York: Columbia University Press. Grodal, Torben. (1997) 2002. Moving pictures. A new theory of film genres, feelings and cognition. Oxford: Clarendon Press. Johnston, John. 1997. Friedrich Kittler: Media Theory after Poststructuralism. In Friedrich A. Kittler Literature, media, information systems. Essays, ed. John Johnston, 1–26. Amsterdam: G+B Arts International. Kittler, Friedrich A. 1997. The world of the symbolic – A world of the machine. In Literature, media, information systems. Essays, ed. John Johnston, 130–146. Amsterdam: G+B Arts International. Krämer, Sybille. 2015. Medium, messenger, transmission. An approach to media philosophy. Trans. Anthony Enns. Amsterdam: Amsterdam University Press. Leleu-Merviel, Sylvie. 2020. Foreword. In Image beyond the screen. Projection mapping, ed. Daniel Schmitt, Marine Thébault, and Ludovic Burczykowski, xiii–xvi. London/Hoboken: ISTE Ltd, Wiley. Manovich, Lev. 2001. The language of new media. Cambridge, MA/London: The MIT Press. Marco, Salvador Rubio. 2016. Dos Passos’s USA trilogy: Current answers for the eyeminded public. In Screens. From materiality to spectatorship – A historical and theoretical reassessment, ed. Dominique Chateau and José Moure, 203–222. Amsterdam: Amsterdam University Press. McLuhan, Marshall. (1964) 1994. Understanding media. The extensions of man. Cambridge, MA/London, UK: The MIT Press. Mulvey, Laura. 1989. Afterthoughts on ‘visual pleasure and narrative cinema’ inspired by King Vidor’s duel in the sun (1946). In Visual and other pleasures, 29–38. Basingstoke: Palgrave. ———. 2004. Foreword. In The Couch and the Silver Screen. Psychoanalytic reflections on European cinema, ed. Andrea Sabbadini, xiii–xvi. Hove and New York: Brunner-Routledge. ———. (1975) 2005. Visual pleasure and narrative cinema. In Feminist film theory. A reader, ed. Sue Thornham, 58–69. Edinburgh: Edinburgh University Press.

998

A. Virgina´s

Newell, Kate. 2020. Transferring handmaids: Iconography, adaptation and intermediality. In Beyond media Borders: Intermedial relations among multimodal media, ed. Lars Elleström, vol. 2, 33–57. Houndmills, Basingstoke: Palgrave Macmillan. Odin, Roger. 2016. The concept of the mental screen: The internalized screen, the dream screen, and the constructed screen. In Screens. From materiality to spectatorship – A historical and theoretical reassessment, ed. Dominique Chateau and José Moure, 176–185. Amsterdam: Amsterdam University Press. Ott, Brian L. 2007. The small screen. How television equips us to live in the information age. Blackwell Publishing. Peretz, Eyal. 2017. The off-screen. An investigation of the cinematic frame. Stanford, CA: Stanford University Press. Persson, Per. 1999. Understanding representations of space: A comparison of visualisation techniques in mainstream cinema and computer interfaces. In Social navigation of information space, ed. Alan J. Munro, Kristina Höök, and David Benyon, 195–218. Springer PH. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on Intermediality. Intermédialités 6: 43–64. Sobchack, Vivian. (1990) 2000. The scene of the screen: Envisioning cinematic and electronic ‘presence’. In Film and theory: An anthology, ed. Robert Stam and Toby Miller, 66–84. Blackwell Publishers. Strauven, Wanda. 2016. The Screenic image: Between verticality and horizontality, viewing and touching, displaying and playing. In Screens. From materiality to spectatorship – A historical and theoretical reassessment, ed. Dominique Chateau and José Moure, 143–156. Amsterdam: Amsterdam University Press. Virginás, Andrea. 2017. Gendered transmediation of the digital from S1m0ne to ex Machina: ‘visual pleasure’ reloaded? European Journal of English Studies 21 (3): 288–303.

The Qualified Medium of Computer Games: Form and Matter, Technology, and Use

39

Ida Kathrine Hammeleff Jørgensen

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Games as a Formal Mode of Expression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Games and Transmedial Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Technical Media and Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Games as Participatory Media Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1000 1003 1011 1015 1019 1023 1026

Abstract

This chapter considers computer games as a qualified medium situated in an intermedial network of other qualified, technical media and media cultures. The aim of this chapter is to offer an overview of state-of-the-art research, which in four different ways sheds light on the relations between games and other media, supports intermedial analysis of computer games, and informs our understanding of computer games as a qualified medium that favors certain modes of expression, technologies, content matter, and modes of use. The chapter offers a broad review of literature originating in the field of game studies and studies of games and gaming technologies from the fields of media studies. This research is categorized into four different categories (comprising four parts of the review), each dealing with a different aspect of the intermedial network in which the qualified medium of computer games is situated. Part one concerns existing theoretical approaches to computer games as a distinct formal mode of expression along with critiques of such approaches. Part two reviews research that considers the content of games through the lens of adaptation, transmedial storytelling, and convergence culture. Part three discusses media historical and archeological approaches to the I. K. Hammeleff Jørgensen (*) Department of Media, Design, Education and Cognition, University of Southern Denmark, Kolding, Denmark e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_47

999

1000

I. K. Hammeleff Jørgensen

technical media and platforms employed by games. Finally, part four is concerned with empirical studies on games as a media culture that is practiced through the means of other qualified media (such as cinema and film, fanfiction, and play guides). Keywords

Computer games · Media · Gaming technologies · Playing practices · Transmedial storytelling · Media archeology · Platform studies · Paratext

Introduction Computer games are significant intermedial phenomena. The game The Witcher 3: Wild Hunt (CD Project 2015) offers a good illustration of this. The game is the third installment of a game series that adapts a series of literary fantasy novels authored by Polish author Andrzej Sapkowski. Following the success of the games, a television series has been aired which is greatly informed by the iconography of the game series. The game itself employs text, imagery, and sound in configurations reminiscent of other media. It contains cinematic sequences, some of which are comprised of voice-over narration and still imagery similar to comic book illustrations. Meanwhile, others are rendered in a 3D environment or even take place as performances on a theatrical stage. The game also contains a variety of sounds and musical scores, some of which function as background music during gameplay, whereas others appear in cinematic sequences during the game and performed by characters in the game. The Witcher 3 also contains written text in the form of transcribed dialogue reminiscent of subtitles in cinematic works, a quest log, a bestiary represented as an old book, runestones, written letters that can be collected from notice boards throughout the game, in-world books, as well as a wealth of written labels in the game interface. The game even contains another game. A simulation of the card-game Gwent, described in the book series, can be played at various points throughout the game. Finally, the games’ protagonist and playable character, Geralt of Rivia, may, using his special Witcher senses, enhance his sensory apparatus and perceive past events, which are represented to the player as visual marks on the screen. The Witcher 3 is also intermedial in a different sense. The game is materialized on, and has travelled across, technical platforms. The game was first released as a software package for Microsoft Windows as well as the dedicated gaming consoles Xbox One and PlayStation 4. Later, the game was ported to the Nintendo Switch, which offers the player a significantly smaller, handheld screen. In addition to the main game, the card game Gwent was also released as a stand-alone software package for Microsoft Windows, Xbox One, and PlayStation 4, as well as for mobile platforms iOS and Android with touchscreen interfaces. As this example suggests, intermedial theory offers a highly productive perspective on computer games. First, many games employ a great set of different sign types, each with different aesthetic and sensorial qualities. Second, different

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1001

(concepts of) media have greatly informed theoretical accounts of the expressive form and function of games. Many game scholars consider games to be a medium capable of conveying messages between game designer or player, or a platform for communication between players. While there are also several game scholars who resist the idea of games as a medium, all of them acknowledge that computer games require mediation. Third, the emergence of computer games has made the importance of studying the material basis of games – and the technical media employed by games – apparent to game scholars. This has not only resulted in studies of different gaming technologies, but also of their history and the consumption cultures that surround them. Fourth, and finally, empirical studies of games and how players are involved with them must consider how the content of games travels across qualified media borders. Characters and plots from popular movies or television shows are frequently appropriated by games, whereas characters and background stories of games are often expanded upon in novels, graphic novels, television series, and movies. Further, the content of games themselves can be remediated in other qualified media, such as when players write detailed walkthroughs or publish video recordings of their play sessions online. Despite this, the very concept of media occupies an ambiguous position in the field of game studies, and the idea that games constitute a distinct medium, while accepted by some, has also been criticized. This is mainly due to a conflation between the concept of media as a mode of expression and as a technical channel of transmission. Therefore, intermedial theory does not only afford inquiries between games and other media types, such as literature or cinema, but also affords explorations of the intermediality of games themselves. This chapter employs the terminology of Lars Elleström’s model of the modalities of media. Elleström (2010, 2020) distinguishes between three types of media: basic, qualified, and technical media. Basic media are media products that are primarily understood in terms of their basic medial traits (of which Elleström proposes four modalities: a material, sensorial, spatio-temporal, and semiotic modality). Qualified media are media products that are understood in terms of aspects that lie beyond the basic medial traits, such as how they are produced or by whom. Technical media are the objects (or bodies) that mediate the basic medial traits of a media product. In this chapter computer games are considered qualified media (Elleström 2010, 2020). The term game covers a wide range of phenomena from soccer over quizzes, jigsaw puzzles, tabletop roleplaying games, card games, and casino slot machines to first-person shooters, massively multiplayer online games, and augmented reality games. Theorizing language rather than games, Wittgenstein (2009 [1953]) made the important observation that it is difficult to find one overarching feature common to all phenomena that we call games, and it might therefore be more useful to think of them as phenomena with family resemblances, where several features are recurrent to several phenomena but no single feature is prevalent in all. Although the problem that Wittgenstein describes is by no means unique to games, many game scholars have found use in this notion of family resemblances. Computer games have at least one obvious, common feature: that they are played on a family of computational media technologies (be it in the form of a PC, a smartphone, or a dedicated gaming

1002

I. K. Hammeleff Jørgensen

console such as a PlayStation, Xbox, or Nintendo Switch). Still, the term “computer games” covers a diverse set of games characterized by different basic medial qualities and playing practices (c.f., Aarseth and Calleja 2015). This may potentially make it difficult to propose any general characteristics of games as a distinct medium beyond the most elementary. To accommodate this problem, Ida Jørgensen (2018, 2020) suggests considering computer games as a qualified medium. Elleström (2010, 2020) considers qualified media as multistable media types that are defined by appealing to features that lie beyond the basic, salient qualities. What features are used to qualify a given media type differ, but may include their production, consumption and use, the historical period in which they emerged and became popular, and so on. By considering games as a qualified medium, it is possible to describe how games are situated in an intermedial network along with other qualified media, technical media, and media cultures, and explore how discourses on games are influenced by this intermedial network. This chapter offers a review of state-of-the-art research on the qualified medium of computer games and the intermedial relations between games and other qualified, technical basic medial aspects. Although the research reviewed in this chapter may not necessarily identify itself as interart, inter-, and transmedia studies, it still refers to other media when theorizing the structural, material, semiotic, aesthetic, and phenomenal qualities of computer games. It also examines their history and the use practices associated with them by comparing, contrasting, or otherwise exploring their interactions with other media. The chapter is structured as follows: the first part of the review focuses on theoretical approaches to games as a formal mode of expression, or simply a symbol system or language. In this view, it is typically the rules of the game and its underlying computational process that are considered capable of conveying messages. The analogy in these approaches is typically language, although it varies how far this analogy goes. Some scholars argue that it is the player’s interpretation of her interaction with the game rules that allows her to fill in the blanks and compose an actual argument out of this experience, whereas other scholars argue that the game system by itself can make propositions (like verbal language). Most scholars, however, acknowledge that to understand the expressive capacity of games, one must move beyond media essentialism to study how games employ existing symbol systems such as spoken and written language, moving and still imagery, sound, music, as well as conventional modes of expression from other media. Part two discusses media relations between games and other media through the lens of adaptation, transmedial storytelling, and convergence culture. The research reviewed in this part will mostly be concerned with the content of games and how it travels between media products. In the study of games, the concept of transmedial worlds or storyworlds has gained particular popularity. The reason for this is that this concept shifts the focus from the transposition of narratives supposedly implied in theories of adaptation and transmedial storytelling to a focus on narrative constituents such as characters, settings, objects, world mythos, physical and social laws, etc. Many scholars agree that, as machinic systems or simulations, it is not straightforward to consider games as a narrative medium. Instead, games can be thought of as

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1003

story systems employing the building blocks of the transmedial worlds. The notion of convergence culture typically implies that there is a recognizable construct of a storyworld that is shared between players, and game scholars have theorized how such a construct can be sustained based on the otherwise often highly individual practice of play. The third part of this review will shed light on the technical media and platforms employed by games through the lens of media history and archeology. While some scholars argue that games are in principle transmedial, abstract systems of means and ends (such as rules of play, winning and losing conditions, and goals), empirical computer games are nevertheless intimately related to their technical medium of storage, arithmetic operation, and display. Media archeological approaches to games typically study how games appropriate existing media technologies for the purpose of gaming, and how this process of appropriation is informed by existing use practices. These are often studies informed by Friedrich Kittler and his attention to media as materiality and its influence on how we organize and structure our knowledge of the world. Inspired by media archeology, platform studies conduct close analyses of the technical gaming platforms and how technical constraints and possibilities affect and shape the design of games. Part four will discuss studies of games as a participatory culture that is practiced through media objects such as player-produced paratexts (a concept that departs from the original term coined by Gérard Genette, and that will be introduced in greater detail later in this chapter). In studies of games culture, the notion of paratext has gained popularity as a means to theorize the meaning-making processes and products that surround games. In this context, paratexts are not only the official texts published by game companies but are also player-made texts such as walkthroughs and game guides, modifications (mods) and add-ons to the game software, fan fiction, Let’s Play videos, and cosplay activities. Additionally, these paratexts can also be found in discussions of both the games themselves, as well as the playermade texts on games, such as comments to videos on YouTube, discussions on Reddit, and so forth, which constitutes a significant aspect of engagement with computer games. This four-tier review aims to show that intermediality is an indispensable condition in all aspects of computer games, not only in terms of how they represent or express content but also regarding how they constitute material cultures and in terms of the cultures of use and participation that surround them. As the review will also show, the intermediality of games is reflected in the interdisciplinarity of game studies where both theoretical accounts and empirical studies of games draw on concepts, methods, and perspectives from a wide set of disciplines.

Games as a Formal Mode of Expression The expressive capacities and practices of games have been theorized with roughly two different approaches. First, scholars have theorized games as an expressive medium by looking for medial differences, that is, ways that the expressive quality

1004

I. K. Hammeleff Jørgensen

of games differ from those of other media. Second, scholars have theorized how gameplay is mediated by looking for media similarities. This line of research has studied how games employ basic medial traits that are also found in other media; how games may simulate qualified media such as photographs, cinema, and maps; and how theories of other media (film studies, musicology, theories of iconic depiction, and narrative analysis) can be employed and harnessed by game scholars. These are two very different ways of understanding the relationship between games and media, though they are not necessarily mutually exclusive. As Fig. 1 shows, the former approach considers computer games as a communication medium capable of transmitting messages between a sender (game designer) and receiver (player). The latter approach, on the other hand, considers computer games as a mediated experience. In this approach, games are formal systems that are mediated, not only by a technical medium such as a computer but also by different signs that make up the surface layer of the game and communicate the game state to the player. As the following review will show, this is of course a gross simplification of these two approaches, but it serves to illustrate the overall difference. The idea that games constitute a medium with a distinct expressive power has enjoyed significant popularity in game studies, with the most influential proponents of this idea being Janet Murray (2012, 2016) and Ian Bogost (2010). According to Murray games are media for a specific type of dramatic storytelling that she calls “multiform stories.” These are stories that “presents a single situation or plotline in multiple versions” (Murray 2016, 30). While multiform stories can be told by other media (Murray offers examples such as Borges’ novel The Garden of Forking Paths or Ramis’ movie Groundhog Day), electronic media, such as computer games, afford effective ways of conveying multiform stories because of what she argues to be four essential properties of electronic media, namely, that they are procedural, participatory, spatial, and encyclopedic. According to Murray, the procedural property of digital media relies on the fact that the computer is essentially an engine that embodies and represents the behavior of phenomena. Designing electronic media products, therefore, involves translating the behavior of represented phenomena into a string of rules that can be executed by a computer. Electronic media are not only procedural, argues Murray, they are also responsive to the user. In other words,

Fig. 1 Two approaches to the relation between games and media

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1005

electronic media becomes participatory when the procedural model that Murray described also includes rules about how to respond to user input. One of the ways that electronic media affords participation is by making it possible for the user to navigate them spatiality. While a novel can describe space, and an image may depict space, Murray observes that electronic media is capable of modeling space in a way that allows the user to navigate it. Finally, Murray characterizes electronic media by the computer’s capacity to store and retrieve great quantities of information and represent them in a single environment. While each of the properties that Murray proposes may also be found in other media products, it is their combination in what Murray argues to be one artistic medium that makes electronic media significant. Murray observes that electronic media often employ representational strategies of other, older media. To her, this suggests that electronic media are still in an immature state. According to Murray, the development of a medium into its mature form involves “seizing on the unique physical properties” of the representational technology until the artistic medium finds its own distinct expressive power (Murray 2016, 82). In other words, when electronic media relies upon conventions of earlier media, this is a sign that its users are still not sophisticated enough to be able to comprehend the capacity of the medium by itself. As it should be clear, Murray does not simply describe electronic media, but also offers a judgment of what she thinks are their unique qualities that designers should put to use. A similar discourse can be found especially in the early days of game studies. At the turn of the millennium, scholarly interest in games was characterized by an aspiration to find and define the unique representational capacity and use of games vis-à-vis other media. Games were compared to and contrasted with other qualified media, especially narrative media (the literary novel, cinema, and theater). Some scholars, such as Murray (2016 [1997]) and Brenda Laurel (2013 [1991]), have argued that games can be considered a narrative medium with ways of narrating that should be distinct from other narrative media. Other scholars, most notably Markku Eskelinen and Gonzalo Frasca, have argued that games essentially do not constitute a narrative medium. According to Eskelinen, games are systems of means and ends that afford configuration and manipulative action, and, just like Murray, Eskelinen also offers a value judgment when he admits that although games may employ story elements, these are “uninteresting ornaments or gift-wrappings to games” (Eskelinen 2001). While Frasca, like Eskelinen, does not consider games to be a narrative medium, Frasca still argues that games have a representational function, as he thinks of them as simulations. In this view, games do not represent sequences of events, but rather models the behavior of phenomena. While Frasca does not consider games as a narrative medium, to him they nevertheless constitute a distinct expressive medium. This view has informed the highly influential concept of procedural rhetoric. Procedural rhetoric is a concept coined by Ian Bogost (2010) to account for the representational capacity and potential of computer games. Bogost draws on Murray (2016) and Frasca (2003) when he considers games to be a distinct medium capable of conveying messages by means of rule-based representations. According to Bogost computer game designers may employ the procedural properties of the computer to author arguments through processes. In simple words, the practice of translating the behavior of phenomena into a procedural model allows

1006

I. K. Hammeleff Jørgensen

game designers to make claims about how things work. Bogost convincingly shows how rule-based processes underlie all areas of human life, from return policies to bureaucratic organizations. Whereas narrative literature and cinema may describe processes through written text or moving imagery, only games can represent processes with processes, Bogost argues. In other words, according to Bogost, procedures become the artistic material that a game designer can use to craft her expressions in much the same way that a painter would use oil on a canvas. Bogost grants a special role to the player. By comparing procedural models to the Aristotelian notion of the enthymeme, Bogost notes that it is sometimes necessary for the player to close what he calls “the simulation gap” by critically assessing the rules of the game and what is included and excluded in the model. In other words, the processual model of the game only offers some, but not all, necessary premises, and it is up to the player to supply the remaining premises of an argument (Bogost 2010). Although procedural rhetoric has become an influential theory in the study and design of games, it has also received critique. The idea that processes represent processes and that the meaning of these processes can be determined through player interaction is problematic. First, computational processes are only made perceptual to the user through the game’s interface, which consists of a variety of sign types, including static and moving imagery, sound, verbal language, and so on. On the most fundamental level, computational processes, as Kittler (1995) reminds, can be reduced to signifiers of voltage differences. Therefore, it is not so much the processes as the signs of the game interface that communicate information to the player, as Sebastian Möring (2013) has observed. This is in line with Espen Aarseth, who theorizes games as cybertexts (Aarseth 1997) or cybermedia objects (Aarseth and Calleja 2015). Aarseth has proposed a model of games according to which the game object can be divided into a mechanical layer (the rule-based engine) and a semiotic layer (Aarseth 2009). He stresses that these two layers exist independently of each other and should not be confused with the distinction between signified and signifier in the Saussurian semiotic tradition. The semiotic layer is the game interface and that which is perceptible to the player, but it is the mechanical layer that contains the rules of the player’s manipulation of the game. Although game mechanics and semiotics do not make up two parts of a formal system of sings, both layers affect the player’s experience of play and inform how she makes sense of that experience. Aarseth’s model of games is visualized in Fig. 2.

Fig. 2 Aarseth’s ontological model of games

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1007

Aarseth’s distinction between the mechanical and semiotic layer of games departs from the aim of defining a unique expressive power of games and instead makes it possible to consider how, on a semiotic level, representational modes and means of expression (which can also be found in other qualified media) intersect with game mechanics, inform the player about the game state, and add meaning to the game beyond play itself. In computer games studies, Aarseth’s notion of a semiotic layer is often discussed under the term “interface.” Interface is defined by Jose Zagal et al. (2007) as the means by which the player experiences the game and takes action within it. For Zagal, the interface can be divided into two main categories, namely, the presentational means that provides the player with a sensory experience of the game, and the input devices that provide mechanisms for operating the game. Input devices also translate the player’s input into game action. Beyond the material input and output devices in the form of game controllers, keyboard and mouse, monitors, speakers, and so on, Kristine Jørgensen (2013) describes a range of different virtual interfaces – that is, interfaces that are mediated by the physical devices of the computer. On Jørgensen’s list we first find the WIMP interfaces employed in a range of software beyond games, which refers to a human-computer interaction paradigm that employs windows, icons, menus, and pointers. According to Jørgensen, the WIMP interface superimposes information as a separate layer on top of the visual representation of the game world. However, she also observes how many contemporary computer games aim to integrate information into the game world. This is the case in many first-person games where the traditional WIMP interface is represented as a military heads-up display. Here, the superimposed information layer is presented to the player as if projected to the helmet visor of the player’s avatar. Drawing on Jay Bolter and Richard Grusin (2000), Jørgensen identifies this as a design trend that aims to achieve greater invisibility and transparency of the medium and give the player the feeling of interacting with game objects in a “natural way,” as akin to objects in the physical world. Jørgensen, however, also notes that this trend is problematic in how it ignores the fact that human interaction with the physical world is already highly mediated. In addition to these graphical interface types, Jørgensen (2013) stresses the that the auditory features of the game also provide important information to the player. However, her main thesis is that the audio-visual representation of game world by itself should also be considered an interface, or what she calls an information space that characterizes a reification of the underlying game system. Importantly, the tension that Jørgensen describes between interface and game world is a consequence of the expressive modes of the games she discusses. Her discussion revolves mainly around games employing iconic depiction and moving imagery, and she does not discuss the many text-based games, so-called text-based adventures or interactive fiction, whose popularity peaked in the 1980s. While Jørgensen frames the game world as an information space and studies it in relation to how it communicates to the player, other scholars have approached game worlds from a perspective informed by phenomenology. Espen Aarseth (2003) describes the semiotic surface layer of computer games as a virtual environment. In an analysis of the spatiality of computer games, Michael Nitsche (2008)

1008

I. K. Hammeleff Jørgensen

characterizes the prevailing spatial paradigm of games as a hybrid between an architecturally navigable and a cinematically represented space. The games of Nitsche’s analysis all rely on iconic depictions of a navigable game space. The player is granted access to this space in part like a movie spectator who perceives moving images as they are projected onto the screen, and in part as situated within a three-dimensional space in which the player can move around. Rune Klevjer (2012) argues that this access to the game space is mediated by the player’s avatar (a game character that is controlled by the player). Klevjer characterizes this avatar as a prosthetic extension of the player’s body, but also notes that from a phenomenological point of view it is not the avatar’s body that offers this prosthetic telepresence in the virtual space, but rather the so-called virtual camera. In first-person games, this camera coincides with the player’s point of view of the game space, while in thirdperson games the camera hovers somewhere behind and above the avatar, thus offering the player a view of the player avatar as situated in the game space. Because of this representational paradigm, many game scholars (King et al. 2002; Klevjer 2002, 2012, 2019; Backe 2018) have often examined relations between games, cinema, and the visual arts, to explore the applicability of cinematic theory and the philosophy of depiction to games. Such research has studied how the representation of game spaces, and the player’s experience of it, rests on conventions associated with cinema in particular and visual arts more generally. Text-based adventure games offer another kind of mediation of spatiality that more clearly demarcates the spatiality of the medium (the computer screen) and the simulated game space. Although text-based adventure games often contain no or few depictions of space, they nonetheless simulate spaces that are just as navigable as the depicted spaces in modern computer games. These spaces are presented to the player via passages of linguistic text, which often describe spaces as the player moves through them. To this day, research on the spatiality of text-based adventure games, the dialectics between the space of the interface and the simulated space, and the phenomenology of these spaces remains sparse. Instead, scholarly discourse on text-based adventure games has been more concerned with theorizing their formal properties from the perspective of narrative theory, structuralism, and poststructuralism. Another line of research into the semiotic layer of games is concerned with the auditory features of games, including the sound effects and musical scores designed into games and the sounds accompanying them. Karen Collins (2008) offers a general and extensive overview of game sounds, including their history, production, and function in games. Collins discusses game sound from the perspective of technical restrictions and affordances, commercial motivations, limitations behind choices of music in games, and the roles sound and music play in games from both a communicative and aesthetic point of view. The double function of game sounds as both informational and aesthetic elements is often addressed in research on game sounds. Kristine Jørgensen (2006, 2008) studies the function of game sounds and how they inform the player about the game system, arguing that game sounds create a sense of presence in the game world. Drawing on film theory, Jørgensen (2006) distinguishes between three types of sounds. Diegetic sounds are represented as if naturally belonging inside the represented environment. These include guttural

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1009

sounds from monsters or other antagonists in the game as well as dialogue between characters. Extra-diegetic sounds are represented as imposed on this environment from the outside and include background music and sounds of the superimposed user interface. Jørgensen, however, argues that in games these two categories must be supplemented by a third category: transdiegetic sounds. These are sounds that overcome the divide between the proposed diegetic and extra-diegetic space and merge the usability function of game sounds with the function of providing a sense of presence in a narrative universe. Collins proposes yet another significant feature of game sounds, namely, that they are often not only listened to by players, but are also interacted with. According to Collins, when players interact with games, they are also triggering certain sounds, from jumping sounds while controlling an avatar to ambient music procedurally generated while players traverse the game world. To Collins, game sounds exist in a network that includes the physical body of the player as she manipulates a controller, the position of the avatar in game space, the events that takes place on this site, and the moving imagery of events unfolding on the screen. The musical sounds accompanying games are studied by scholars under the notion of “ludomusicology.” Game music has been studied in how it defines a genre of music-based computer games (Pichlmair and Kayali 2007), but also in mainstream games, where music is but one among many aesthetic and semiotic devices. While music in these games is often used as extra-diegetic background music, Ian Hart (2014) argues that it is still a key element in how the player makes sense of the game. He compares the player’s engagement with computer games with a person playing a musical instrument. In both cases, he argues the player performs a series of physical manipulations of an object (such as a game controller or a musical instrument) in a certain order to achieve some goal (playing a given score or performing an action within the game), and that this manipulation will have an effect (perceivable music is created or the player can see her character move on the screen). Building on this, Hart then identifies two ways of analyzing the semiotics of game music. The first considers game music in its original composition as a work produced by a composer and received by a listener together with the other medial aspects of the game. The second analyzes the interactive configuration of this composition by the player and takes into account that the player is not only receiving and interpreting the music, but also that her interpretation will cause her to react to and modify the music. Finally, game music has also been studied in relation to how it communicates the overall spatio-temporal setting of a game, often by appropriating musical scores from different media. Stevens (2020), for example, explores how the soundtrack of The Witcher 3 (CD Project 2015) adapts a diverse set of Eastern European musical traditions to evoke a sense of exotic otherness to the game’s Western audience. Comparing the music of The Witcher 3 with that of The Elder Scrolls V: Skyrim (Bethesda Softworks 2011), Lamb and Smith (2018), in an analysis of neo-medieval aesthetics in the musical score of the games, observe how the score of Skyrim strongly rests on an American tradition of musical scores for cinematic works, whereas The Witcher adopts a more local musical tradition. Similarly, Hart (2015) observes how L.A. Noire (Rockstar Games 2011) employs jazz music and the musical tradition of film noir to convey a particular cultural, spatiotemporal atmosphere.

1010

I. K. Hammeleff Jørgensen

In addition to what Elleström has characterized as basic medial traits, games also frequently simulate other qualified media. Game scholars have, for example, studied the spatial quality of games through the lenses of maps. This is an especially common feature in games played from the point of view of a player character in a three-dimensional game space, where a mini-map shows the position of the player character from a top-down perspective. However, as a review by Dugas Toups et al. (2020) argues, the relation between maps and games not only can be traced all the way back to early tapletop games such as the military training game Kriegsspiel from 1824, but is also a common aspect of text-based adventure games and tabletop roleplaying games where they are typically hand-drawn by players and extrapolated from the way the game space is communicated by the game (or game master). Mathias Fuchs (2019) compares game maps with a range of historical maps and mapping practices and discusses how they are not just representations of space, but tools for orientation and navigation. He discusses how maps, including game maps, employ to different degrees a mix of text and imagery to convey not only the topography of the game space, but also descriptions about the ‘tour’ that the user needs to take and the landmarks that she needs to navigate by. Hans-Joachim Backe (2018) analyzes pictures that the player can find in video games. Backe compares the use of in-game pictures in the two games Dishonored 2 (Arkane Studios 2016) and Prey (Arkane Studios 2017) and discusses how the pictures in the game (in the former representational visual art, and in the latter an image technology creating an immersive environment around the player figure) function as meta-reflective devices that comments on the relationship between games and reality on several layers, including the role of game imagery as both depictive interface for the player and reified game object, and the status of game simulations, more generally, as either fictional or real. Möring and Mutiis (2019) discuss simulations of photography in games. Some games allow players to employ the “virtual camera” of the game to produce a still image of the game space, such that the authors distinguish four ways that games remediate photography. Simulated photography denotes cases where photography is a central aspect of gameplay and key to the game’s objective. What the authors call additional photo-modes are cases where the player can halt gameplay in order to take a photo of game imagery and save it on their computer. In this mode, photography is used to celebrate the aesthetic qualities of the game rather than used as part of achieving a game objective. While not all games offer simulated photography, the technical medium (such as the computer) may allow the player to take screenshots of what appears on the computer monitor at a given time. This constitutes a third mode of game photography that the authors call artistic screenshotting, whereas what they name photo mod(ification)s is a variant of this where players are not just using the functionality of the computer but are making a deliberate modification to the software code of the game, so that it affords taking screenshots (despite this not being originally intended by the game designers). Armin Lippitz (2019) discusses different ways that games incorporate visual and aesthetic aspects of comics. His discussion covers a wide range of examples for how still imagery is used in non-playable sequences to evoke the visual motif of the comic strip – on-screen panels, superimposed images, graphical techniques to make

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1011

images look hand-drawn, and the like. Lippitz also discusses examples where the conventions of comic book design gain a functionality in games, such as allowing the player to rearrange a sequence of events in order to achieve a specific end-state in the game Framed (Loveshack 2014) (an example also discussed by Ida Jørgensen (2018) and Backe (2020)). Michael Hancock (2016) studies the representation of books in computer games, particularly regarding three different strategies for representing books in computer games (beyond paratextual, physical game books and booklets that may accompany games). In one approach, the book functions as an overall frame. Representing books as the frame of the game can be done by displaying the material book, or showing somebody reading in the book, in the game’s opening sequence (a practice that can also be found in cinema), but Hancock also observes how some games divide gameplay into chapters. While the events of the game in fact unfold as the player operates it, the use of books as a frame represents these events as something that has already taken place, and implies that the player is cast in a double role as both reader and player. In another use of books as a frame, books are used at specific points in the game as portals that transport the player character to another “world” (i.e., another part of the simulated game space that can only be accessed through this particular portal). This is not the only way that games represent books, however. Hancock also observes how some games represent the informational interface (c.f. K. Jørgensen 2013) as a book. This, he argues, is often the case in games representing quasi-medieval settings. In addition to this, the author discusses simulated books that are typically scattered across a game world, which can often be (at least partially) read. Finally, he observes how games may not necessarily represent physical books, but also allude to actual books, characters, and writing practices. To sum up on this section, game scholars have employed concepts from disciplines such as film studies, musicology, and the visual arts and approaches such as textual analysis and phenomenology to theorize and study the expressive capacity and practices of games. Of particular interest in the context of intermediality is that several scholars have explored how games also remediate and simulate other qualified media and integrate the use of these media, and the discourses that surround them, into gameplay.

Games and Transmedial Content A second line of intermedial research on games considers how content migrates across media. The use of titles, characters, and settings from popular books, movies, and TV series in games is in no way a new practice that emerged with the advent of computer games, but can be found throughout the history of modern proprietary games. Games appropriating different source material, typically from cinema and television (and to a lesser degree, novels), have been studied under the moniker of adaptation. While the notion of adaptation is often associated with transpositions of narrative content from one qualified media to another, it is possible to see adaptation more widely as any practice of representing a source material in a given material or

1012

I. K. Hammeleff Jørgensen

with a given set of communicative and aesthetic conventions. In this broad conception, simulation essentially becomes adaptation. While scholars, as discussed in section “Games as a Formal Mode of Expression,” have explored how computer games may adapt the expressive modes of cinema, when it comes to adaptations of narrative content – a story – from one medium to the next, adaptation remains an under-theorized concept in relation to the study of games, with a few notable exceptions. Cathlena Martin (2010) offers a close reading of the game American McGee’s Alice (American McGee 2000) and how it adapts its source material of Lewis Carroll’s popular children’s book Alice in Wonderland, as well as its antecedent adaptations in the form of the Disney animated movie and an related computer game also released by Disney (many years after their animated movie). Mark Rowell Wallin (2007) analyzes how two different game adaptations of The Lord of the Rings discursively construct their relation to their source material (Tolkien’s book series and Peter Jackson’s film series, respectively) as a way to claim authenticity and legitimacy. Game designer Pippin Barr (2020) takes a more practical approach, as he uses the concept of adaptation to explore new game design ideas. Barr choses a single central scene, from a set of canonical cinematic works, which he tries to adapt to a single, stand-alone game. He reflects on what aspects of the cinematic scene he tries to capture in his game work and how these ideas are reflected on the level of the underlying game code as well as on the representational layer of the game. Barr observes how the differences between the qualified medium of cinema and games lead to a productive tension that allows him to consider new ways of thinking about the game player’s agency. His adaptation of a central scene from Coppola’s 1974 The Conversation subverts what he claims is the conventional way that games position the player as the source of agency in the game. Barr describes how his game does not give the player a visible representation in the game space, and reduces her to a listener rather than the committer of the violent act that is carried out in the scene. Scholars have also discussed cinematic adaptations of computer games. Betty Kaklamanidou and Maria Katsaridou (2013) analyze the movie adaptation of the horror game series Silent Hill (Konami 1999–2014), and Mizsei Ward (2018) discusses the reception of the movie adaptation of Doom series (Carmack et al. 1993-present). Apart from adaptations of particular games, Mike Sell (2021) discusses a concept he calls the video game movie. This concept does not only include cinematic works that offer explicit adaptations of actual, named games (e.g., Doom), or pastiches over types of games (such as Disney’s Wreck-It Ralph 2013), or movies that have games as a central motif, but also movies where representations of games and gameplay may seem more incidental. Sell identifies six tropes that can be found in video game movies. Fictive adaptations are movies whose narratives take place within the fictional parameters of a specific computer game or game genre. Supplementary adaptations are movies that fill in the narrative gaps of computer games by, for example, offering a background story for game characters. Diegetic representations are representations of games and the playing of games serving a direct narrative function in a movie. Figurative representations are depictions of games and gameplay that serve a symbolic or metaphorical function in the game. Intertextual references are fragments of actual computer games that appear in the game in a way

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1013

where the audience must know the source material in order to become aware of it in the movie. Finally, procedural adaptations are movies that in part rely on a game logic by employing structural features of games such as respawns (fictional characters that re-emerge after they have died), time loops (representing the same sequence over and over again, with some variation), and what he calls visual procedures such as the above-and-over-the-shoulder camera found in most third-person shooter games. However, when it comes to the transposition of content specifically, the concept of transmediality has proved much more productive to game scholars than adaptation. Through the concept of convergence culture, media scholar Henry Jenkins (2006) describes what he calls a paradigm shift in how media industries produce and deliver content on the one hand and how the audience uses it on the other. Jenkins argues that this is a shift from a medium-specific way of thinking about the production, distribution, and consumption of content towards “content that flows across multiple media channels, towards the increased interdependence of communications systems, towards multiple ways of accessing media content, and toward ever more complex relations between top-down corporate media and bottom-up participatory culture” (2006: 243). Jenkins describes how content of different media franchises, such as Star Wars, is designed to be commercialized across technical platforms and types of qualified media, such as comic books, games, zines, cinema, and so on. In this culture of convergence, Jenkins argues that a new aesthetic mode has appeared. Transmedia storytelling describes this new way of conveying stories across media. Inspired by Jenkins’ concept, Lisbeth Klastrup and Susana Tosca (2004) have coined the notion of transmedial worlds. This term has been particularly influential in the study of games. Klastrup and Tosca promote the notion of worlds, rather than narratives, because they want to emphasize the abstract properties that cut across individual stories or mediated expressions from which multiple narratives may emerge. Klastrup and Tosca theorize transmedial worlds as abstract content systems that have three core features. The mythos of the transmedial world describes the backstory of the world, or “the central knowledge one needs to have in order to interact with or interpret events in the world successfully” (2004: 4). The topos describes the setting and historical period of the world, such as a futuristic technological world. Finally, the ethos describes the codex of behavior that exists in the world, which makes the audience capable of assessing if characters behave in accordance with the logic of the world. As such, the authors argue that designers and audience share a mental representation of the world and its distinguishing features. Although the world may change over time, there is a shared sense between audience of what constitutes the world and how to make (correct) interpretations about it. Klastrup and Tosca note that although many transmedial worlds have a foundational text, such as Tolkien’s Lord of the Rings series, they are not a fixed product of a single text. According to the authors, computer-mediated cyberworlds (coinciding with what were earlier called virtual environments) function as particularly good manifestations of these transmedial worlds, as they provide the player with a navigable space in which they can interact (with other players) as a character from the fictional universe. While Klastrup and Tosca’s concept of transmedial

1014

I. K. Hammeleff Jørgensen

worlds describes fictional worlds that are products of multiple texts, Marie-Laure Ryan and Jan-Noël Thon (2014) proposes her somewhat similar concept of storyworld to make sense of the distinction between the sign and the signified in narrative media. Ryan uses the concept of storyworld to distinguish between the world that a text presents and its mode of presenting it (its discourse). According to Ryan, this makes it possible to avoid medium-specific claims about the capacity of media to convey stories. She argues that storyworlds consist of existents such as characters and objects with special significance to the plot, a setting, physical laws (often dependent on the genre), social rules and values that apply to the world, and events as well as mental events (such as motivations, emotional responses, and so on). Ryan argues that a single text can convey multiple storyworlds on the one hand, and several texts can rely on the same storyworld on the other. Ryan also applies her storyworld concept to computer games, which she characterizes as multiworld narratives. In Ryan’s account, the simulation engine (which coincides with what was earlier called the mechanical layer of the game) is capable of generating a new storyworld each time the game is played. Neither Jenkins’s concept of transmedia storytelling, nor Klastrup and Tosca’s notion of transmedial worlds, nor Ryan’s idea of storyworlds significantly theorizes the capacity of games to participate in the transposition of content and how the mechanical level of games affects this capacity. This, however, has been the focus of a couple of game scholars. These scholars have used the notion of transmedia worlds, to come to grips with the relation between stories and games. The underlying problem is, as briefly discussed earlier in this chapter, that many scholars consider the representational capacity of games to be more akin to scientific simulations than narrative texts. Although the outcomes of game simulations can be a sequence of events, this outcome is a result of player manipulation and, therefore on the material level of the sign, locally produced by the player rather than transmitted to her. On a cognitive level, this results in the player’s idiosyncratic interpretation of what she has done in the game and why. Jan-Noël Thon (2009), for example, observes that there is a tension between the fictional world of the game and the rule-based simulation. This is a particularly critical issue in multiplayer games. Unlike single player games where the player can traverse through a single sequence of events, in multiplayer games many players may play a game simultaneously, each making different choices with different outcomes. In addition to this, multiplayer games – especially massively multiplayer online roleplaying games (MMORPGs) – typically rely on so-called grinding, which is the performance of repetitive tasks such as killing non-player characters to level up the player character’s skills, gather resources, or obtain certain prestigious in-game items. Thon argues that such games are nevertheless capable of conveying a fictional world that is shared by all players of the game. According to Thon (2009, 2014), players build a mental representation based on their engagement with the game, but he also argues that there are parts of the simulation that players must disregard in order to adhere to the fictional world of the game (which can be considered common to the game’s players). In summary, Thon’s answer to the problem imposed by computer games is to distinguish between the local, subjective mental construct of a world and the global, intersubjective, shared

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1015

fictional world. The former is evoked by the “player story,” which is a result of the individual player’s local actions in the game. The latter is evoked by what he calls the “designer-story,” which is conveyed through predetermined, unplayable cinematic sequences and voice-over narration and through separate, accompanying media products such as books, comics, and films that are part of the same franchise. By contrast, Blom (2018) problematizes how games have been addressed in transmedia research. According to Blom, transmedia research has been too focused on the transposition of content and has neglected the role of the medium. Although Blom does not define what she means with the notion of medium, like other game scholars, she is critical of the idea that games convey stories. Some games, she argues, have more in common with sports than with comics and short film. In her analysis of the Overwatch franchise – and the relationship between the multiplayer game, comics, and short films released under the same franchise – she even observes how some elements of the game outright contradict the story world as it is presented in the comics and short films. In the game, several manifestations of the same character, operated by different players, can be presented in the virtual world at the same time, while this would not be possible in the story world of the comics and films. Similarly, a team of Overwatch players can freely choose characters even though some of these characters are presented as enemies in the comic books. Blom therefore distinguishes between the virtual world of the game in which players play together and the mental construct of the story world evoked by the comic books and films. According to Blom, these two unlike worlds make up a universe shared by fans of the franchise when they engage with the various products. Paratext is another important concept in relation to the transmediality of game content. While game scholars often consider paratexts to be any texts surrounding game (which will be discussed in detail in the last section), Jan Švelch, drawing on Gérard Genette, reserves the concept to study the “official” paratexts that are published either by the game companies themselves or by other companies who have licensed the intellectual property of the game. Using this concept, Švelch (2015) discusses the different functions of game trailers. He distinguishes between performance trailers which represent a player’s performance from a spectating, third-person perspective, transmedia trailers that aim to expand the fictional world of the game, and interface trailers that recreate the interactivity of computer games by offering the spectator limited viewer agency.

Technical Media and Games Disregarding discussions of whether games constitute a distinct artistic medium or not or how content moves between media, the relationship between games and their mediating objects constitutes another trajectory of intermedial scholarship. While games can be described as abstract systems of means and ends of engagement (Parlett 1999; Eskelinen 2001), most games are materialized by some media object. Jesper Juul (2018) describes games as transmedial phenomena. But while Juul’s argument is that games are not bound to a specific medium, this does not mean that

1016

I. K. Hammeleff Jørgensen

studying the material medium of games is not relevant. Games have a long cultural history involving multiple, dissimilar materialities: from the human and non-human animal bodies involved in the Ludi Romani chariot races of ancient Rome, the cardboard boardgames that emerged with the nineteenth century’s print industry to the silicon chips running modern-day computer games. Throughout this long history, engagement with games has always been a material practice: dice are cast, buttons are pressed, tokens are moved, and fingers, arms, or entire bodies are in motion, while electronic beams shoot within cathode ray tubes, and forces are transmitted by moving mechanical gears. Several game scholars have studied the relationship between games and the technical medium of the computer. Others have explored how various technical media have been appropriated for gaming purposes, how they afford types of expressions and player experiences, and how residues of earlier media have shaped the qualified medium of games. Tom Apperley and Darshana Jayemanne (2012) have called this a “material turn” of game studies. The authors describe this as a turn away from studies of the formal aspects of games towards the “contexts, uses and material qualities of games.” One of the ways the relationship between games and the technical media they employ has been studied has been under the headline of media archeology. Media archeology offers diachronic studies of media concerned with understanding present media through earlier media and, in the words of media archaeologist Jussi Parikka (2013: 2), often with a preference towards “the forgotten, the quirky, the non-obvious apparatuses, practices and inventions.” Media archeology is informed by Kittler and his focus on media technologies and material cultures. Media archeologists, therefore, typically approach computerized, digital technologies not as novel or unprecedented phenomena, but rather as grounded in existing, earlier modes of use, discourses, and cultures. In relation to games, this is evident in studies that seek to understand the relationship between computer games and preceding technologies, media, and use cultures. The work of Erkki Huhtamo offers one example of this. Most notably, Huhtamo has shed light on intermedial relations pertaining to the visual aspect of many computer games, and intermedial relations pertaining to the interactive or operational aspect of computer games. Regarding the former, Huhtamo has explored various forms of screen practices that also characterize the playing of modern computer games (2006a, b). Of particular interest is his (2006b) discussion of what he calls “peep media,” small screens that typically afford looking into rather than simply looking at. Although he also observes that peep media was used for didactic, enlightening purposes, Huhtamo primarily associates peep media with a nonelitist culture of attractions, with practices of “armchair travelling,” and with erotic lurking. The devices that Huhtamo describes can be related to multiple modern media, but Ida Jørgensen (2020) notes how the aspects of peep culture described are reflected in discourses on computer games. Jørgensen argues that like Huhtamo’s peep media, computer games are also often considered camp, low-cultural products for mindless entertainment. Similarly, computer games are often either condemned or praised for being an escapist medium that allows the player, while sitting in a comfortable chair in front of her computer, to escape her

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1017

immediate, social reality and travel to fantastic realms designed to cater for all her needs. Finally, discussions about problematic and sexist gender representation in games reflect how games, like Huhtamo’s peep media, are also associated with a sexual gaze upon their in-game characters. Huhtamo’s archeology of slot machines is more directly linked with modern computer games. In his archeology of gaming, Huhtamo (2005) analyzes the prehistory of games as an interactive medium. He focuses on the material and technical aspects of gaming media and on the apparent relations between human user and machine, deliberately leaving out other aspects of the prehistory of gaming such as storytelling traditions and earlier play practices. According to Huhtamo, the foundation on which early interactive machines – coin-operated slot machines and vending machines – were built was the emerging industrialization of society and its subsequent mechanization, turning not only factory workers into mechanical gears, but also tying office workers to mechanical apparatuses like telephone switchboards, typewriters, and mechanical calculators. Huhtamo notes how machines also emerged outside the spheres of the production hall or the office, and he conceives of the amusement machines of the Penny Arcade as the antithesis of serious production machines. He observes two kinds of amusement machines. Automatic machines were coin-operated mechanisms such as vending machines, miniature theaters, and mechanical phonographs. For these machines, the user’s operation was limited to putting in a coin, turning a lever, and either picking up the product from the vending machine or perceiving the automated visual or auditory experience. The human was the initiator but cannot affect the experience once it has been initiated. What was nevertheless significant about these machines, argues Huhtamo, was that they required close physical contact between human and machine. The other category of proto-interactive machines required continuous input by the human user, and Huhtamo exemplifies these with the gambling machine, or the “one-armed-bandit,” and the mutoscope. The gambling machines were simple and did not rely on the user’s skill. What sets them apart from the automatic machines was mainly that they invited repeated use. The mutoscope, on the other hand, required the user to continuously turn a lever in order to see an animated sequence unfold. Huhtamo finds in this prehistory of interactive media many aspects that anticipate modern gaming technologies, modes of use, as well as the culture and discourses around this use. While Huhtamo focuses on the public gaming machines of the Penny Arcade and gaming halls, Bernadette Flynn (2003) offers a discussion of the domestication of gaming technologies, a process she parallels, first, with the appropriation of the radio, then later the television set as the hearth of the family living room, but also the mechanization of the home as a whole. This domestication of gaming, she argues, does not only represent a shift in the spaces in which gaming takes place, but also affects who has access to these experiences, the social relations that form around these technologies, and their patterns of consumption. Jussi Parikka and Jaakko Suominen (2006) focus on the archeology of mobile devices and gaming technologies. The authors see mobility as a key aspect of not only the modern experience of subjectivity but also the means of expressing this experience. Building on a broad notion of media reminiscent of McLuhan’s idea of media as the extension of man,

1018

I. K. Hammeleff Jørgensen

Parikka and Suominen argue that technologies for movement (such as the railway) created the need for portable entertainment media. Initially this need could be met in the railway stores where pocket novels were sold – often low-brow “pulp” genres such as horror, thrillers, and later detective stories. Pocket novels, newspapers, and other media objects designed to be used on the train represent, according to Parikka and Suominen, the first stages of mobile media entertainment. Like modern mobile games, they afforded mobility in a double sense. They were portable due to their weight and size, but they were also capable of transporting the user away from their immediate surroundings and into another imaginary or social realm. In this sense, these objects functioned like mobile versions of Huhtamo’s aforementioned peep media. Inspired by media archeology, Nick Montfort and Ian Bogost (2009) have established the research field and book series “Platform Studies,” which aims to “investigate computational systems and how they enable, constrain, shape and support the creative work that is done on them” (2009: VII). Platform studies scholarship tends to focus on close examinations of single proprietary platforms, such as the Atari video game console (Montfort and Bogost 2009), the Nintendo Entertainment System (Altice 2015), or the Commodore Amiga 1000 (Maher 2018). In their study of the Atari video game console, Montfort and Bogost demonstrate how the platform shapes the creative products developed to be run on this machine. They define the platform as an abstract standard or specification that is materialized by means of chips, boards, wires, and other components and discuss how a platform’s hardware and software influence each other to afford or constrain the artistic expressions made for this technical medium. What Montfort and Bogost show is that individual games are often intimately related to specific technical media. This, of course, does not mean that individual titles do not travel across technical media. Pawel Grabarczyk and Espen Aarseth (2019) offer a framework for a nuanced analysis of what happens when a game transitions from one technical medium to the next. The first problem the authors wish to tackle is inconsistency in what are colloquially described as different “versions” of a computer game. Second, they discuss how the relationships between games and the technical media they employ are highly complex and involve numerous different levels, including the physical interface of the computer, various levels of code, abstracted rules of operation, and finally how all these prior levels are presented to the player. They suggest distinguishing between game versions via the following parameters: physical platforms (e.g., arcades, consoles, card decks, and boards), physical interfaces (e.g., mouse and keyboard, controllers, VR headset), computational cores (the game code), computational shells (operational systems and other software frameworks), mechanical layers (the set of actions that the user can perform to change the state of the game), and finally presentational layers. With this framework, it becomes possible for Grabarczyk and Aarseth to distinguish between different game versions, such as between what they call a “game conversion” (resemblant physical platform, different computational core, and similar mechanical layer and presentational layer) and a “game clone” (resemblant physical platform, different computational core, similar mechanical layer but different presentational layer). Fourteen different forms

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1019

of game versions emerge from this matrix, ranging from adaptations, emulations, and reskins to software and hardware ports. While Grabarczyk and Aarseth’s focus is on how games travel and develop and are thus “kept alive” across individual versions, Raiford Guins (2014) offers an analysis of what he calls the afterlife of games, that is, when games cease being games and are turned into material objects in museum exhibitions, private collections, or even landfills. Guins demonstrates another aspect of games as an intermedial phenomena, namely, that games are multistable objects whose ontological status may change over time. First, they are objects to be played with – machines that offer a spectacle of sound and vision, as players navigate a virtual space in real time. Then they may be abandoned objects, physical arcade cabinets, and game cartridges left in the corner of the arcade hall, turned off, yet still trying to attract players with bright imagery outside of the cabinet. Then, finally they may be discarded, left to deteriorate on a dumpster somewhere, or maybe they may be deemed important cultural objects, put on display in a museum.

Games as Participatory Media Culture The fourth and final line of research that will be presented in this chapter concerns games as a participatory media culture (Jenkins 2006) that is practiced across qualified and technical media. To Jenkins, the notion of participatory culture denotes a media culture in which the audience actively participate, not only as recipients but also as producers and distributors of content and meanings. Most game scholars describe play as an active process, both in terms of the players’ physical manipulation and configuration of the game material and in terms of the interpretive process and sense-making that come before and after play. However, several scholars have explored how these playing practices lay the foundation for other forms of participation that are not restricted to the game itself. Discussing different ways in which players engage with games, Gordon Calleja (2011) distinguishes between micro- and macro-involvement. Micro-involvement describes the moment-by-moment engagement of gameplay, whereas macro-involvement describes what takes place in between these play sessions. Macro-involvement includes the feeling of belonging to a player community, laying out strategies, planning character development, or working on community web sites. This participation is often practiced by means of other qualified media such as the comic book, or short story, as well as through technical media like the computer. Hanna Wirman (2009) identifies five different productive practices centered around the playing of computer games. The first category considers play by itself as a co-production of the original media text. The second consists of what she calls “instrumental productivity.” These are playerdriven practices that produce materials that aid gameplay and inform the player in how to overcome difficult challenges. These materials are materially separate from the game. They come in many forms, including written “walkthroughs,” which are texts that describe in detail what the player should do to overcome a specific part of the game, obtain a given object, beat a particular non-playable enemy, etc. Instrumental products may also come in database form, cataloguing game items,

1020

I. K. Hammeleff Jørgensen

characters, and quests, which enable players to get an overview of individual parts of the system and how they are found. Interface modifications and so-called add-ons or mods represent another type of instrumental product. Mods refer to player-made modifications of the game software. Add-ons are software that can be installed on the player’s computer and make instrumental changes to the game by organizing the game’s information in different ways, providing additional information, or adding functionality to the game. Wirman’s third category consists of what she calls “expressive productivity.” Expressive productivity refers to practices that serve a self-expressive function for the product’s creator, rather than an immediate instrumental value for other players. These can come in the form of skins – player-made modifications to the surface appearance of game characters and objects rather than of functional aspects of the game. These skins are therefore tightly integrated into the game. However, expressive productivity can also refer to the production of literary texts, poems, drawings of game characters, and scenes from the game. It may also come in the form of more external media products, such as the so-called machinima, videos of recorded gameplay that is typically subsequently edited into a stand-alone product and published on video-sharing platforms such as YouTube. There exist plenty of research on machinima, and the phenomenon has emerged as a qualified medium in its own right. An overview of the diverse trajectories of research on this phenomenon can be found in an anthology on machinima, edited by Henry Lowood and Michael Nitsche (2011), which includes discussions of the history of the phenomenon, its relationship with cinema, its media characteristics, and its social significance. In her fourth category of player-made products, Wirman considers how games are sometimes used to express fandom that is not centered on the games themselves, but on other cultural products, such as TV series, movies, music videos, and so on. She offers examples of machinima imitations of music videos, where each scene in the video is recreated in the virtual space of The Sims 2 (Maxis Redwood Shores 2006) using The Sims 2’s characters, and then subsequently edited to imitate the original video. Finally, Wirman observes how many games have integrated player productivity into the game itself. She offers examples such as Spore (Maxis 2008) and Hello Kitty Online (Sanrio Digital and Typhoon Games 2009), but in addition to this one can also mention a hugely popular game such as Roblox (Roblox Corporation 2006). Roblox is a platform that not only allows players to program games and play the games that others have made, but also makes it possible for players to sell and distribute virtual products they have made in the game. In addition to these four categories, it is also relevant to consider participatory player practices that are not tied to specific commercial game titles, but rather to gaming more generally. This could be the hobby practices of making the so-called homebrew games described by Melanie Swalwell (2007, 2008, 2021) and Jaroslav Švelch (2013a, b, 2018, 2021). Homebrew games are amateur production of games, which according to Swalwell are often made in a domestic sphere, by self-taught programmers who are making games to explore the possibilities of computer technology. Švelch (2013b) offers a study of amateur game making in former Czechoslovakia in the 1980s and observes several motivations for these practices. Among other things he notes that amateur developers were making conversions of

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1021

Western games in order to make them available in a region without a commercial software market, but also that the games could function as carriers of messages from personal rants and jokes to explicit political commentary and satire. When considering these player-made products an interesting question emerges regarding their connection to the specific games and gaming technologies. The participatory practices described by Wirman all fall under Mia Consalvo’s (2009) notion of paratext. Like Jan Švelch (discussed earlier in this chapter) Consalvo appropriates this concept from literary scholar Gérard Genette. She considers paratexts to be “all the elements surrounding a text that help structure it and give it meaning” (Consalvo 2009, 21). Paratexts are therefore a broader category than the player-made texts described by Wirman and include third-party content like gaming magazines as well as supplementary, promotional media products related to the game published by the game company (such as trailers, comic books, short films, etc.). In a review of how the concept of paratext has been applied by game scholars, Jan Švelch (2020) notes that Consalvo’s notion of paratext is much broader than Genette’s original concept and includes not only official material but also fan art, transmedia storytelling, and the like. He also observes that Consalvo’s notion of paratext has become much more influential in the field of game studies than Genette’s more limited concept. Švelch relates the popularity of Consalvo’s concept to what he describes as a turn away from formalism in game studies towards more cultureoriented research. Another reason for the popularity of Consalvo’s concept may be found in the precarious status of the game as a text in the first place. As Espen Aarseth (1997) describes, games are not texts but rather machines for the production of texts. It is therefore possible to claim that all the meaning-making processes of games are not to be found in the game as “text,” but rather in all the texts that surround games. A related observation has been made by Ida Jørgensen (2020), who argues that as simulations, games are not inherently representational, but must rely on textual material spelling out the relationship between the game elements and the target elements. While Jørgensen does not refer to Consalvo’s concept of paratext, it is a useful concept to account for meaning-making practices around games. Indeed, in a more recent article, Consalvo (2017) analyzes situations where games themselves are transformed from the “central text” to paratext. One example of this can be found in streaming practices on the live-streaming platform Twitch. Twitch is a popular online site where players may live-stream their gameplay. Gameplay footage is often accompanied by live footage of the player’s face and upper body. Spectators may subscribe to their favorite channels and chat with other spectators as well as with the streamer. According to Consalvo, although Twitch at first blush seems to be a site for viewing gameplay, it is actually not the games that are the central media products, but the streamers themselves – the streamer’s behavior during the streaming session, their opinions about the game, and the overall persona and brand they have created for themselves. Closely related to live-streams on Twitch are so-called “Let’s Play” videos. Like Twitch streams, Let’s Play videos feature footage of gameplay coupled with overlaid footage of the player’s head. Although Ingrid Richardson (2018) also considers Let’s Play videos paratextual products to games, her analysis of the ways in which the audience engage with the videos makes evident

1022

I. K. Hammeleff Jørgensen

that they are also stand-alone media products in their own right. Alex Gekker (2018) discusses Let’s Play videos from the position of the audience and considers spectating practices as a form of delegated play that resists and subverts the constant call for engagement in computer games and digital media more generally. Another notable study of the participatory game culture from a perspective of intermediality is Nicolle Lamerichs’ study (2018) of expressive fandom and how it is practiced especially in fanfiction, which she describes as derivative, self-published works, roleplaying activities (based on other media products) and cosplay, which denotes the fanish practice of dressing up like a character from media products such as games and movies. The focus of Lamerichs’ studies of these phenomena is on how media content is appropriated and reworked by fans. Related to Consalvo’s discussion of the performative aspect of Twitch streaming, Lamerichs (2011) shows another way in which the physical body interrelates with games in participatory media culture. In her study of cosplay, she notes that characters are embedded in media and that these media may impose constraints on cosplayers when they appropriate a character from one medium (e.g., a movie or a computer game) to the medium of the cosplayer’s own body. A character from a movie may be strongly associated with a specific real-world actor, and the cosplayer needs to refer not only to the character but also to the actor playing this character. Similarly, animated characters, such as those often found in computer games, may be represented in ways that make it difficult for the cosplayer to imitate with their physical bodies. On the other hand, she argues that the aim of cosplayers is not to create complete look-a-likes, but to express one’s own identity through the practice of dressing up. As such, the performative aspect of cosplay – such as posing in front of a camera for photoshoots or acting on stage or, more generally, using their body to display a character – can be seen as a process of media transformation and in the terminology of Consalvo a paratextual practice that adds meaning to the original media product. Jan Švelch and Tereza Krobová have also significantly contributed to the study of paratextual fan works. The authors have explored how fan works may become part of the official promotion of a game on the social-media platform Facebook. Interestingly, the authors observe how visual fan culture (including the production of drawings and imagery of cosplay activities) is most frequently picked up by administrators of the profiles and featured on the games’ official Facebook page (Jan Švelch and Krobová 2016b). In another article, Švelch and Krobová (2016a) have analyzed how fan-made drawings (including their authorial captions and other written comments) reflect and negotiate the events that take place in a given game or game series, the production history of the game(s), as well as the player’s own personal memories with the game(s). Švelch and Krobová describe the paratextual fan practices that they analyze as multimodal amalgams. Although the authors do not employ the terminology of Elleström, their analysis shows how these fan practices involve the full spectrum of the modalities of media. The amalgams do not simply imitate the iconography of specific scenes from a computer game. Instead, they are intermedial sites where a multiplicity of personal memories of engaging with the materiality of the game as well as its characters, events, and settings becomes intertwined with visual cues and symbols from the game, which in turn reflect the fans’ appreciation of not only specific games but the entire game series.

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1023

Research on players’ participatory, paratextual practices is too rich and varied to be discussed at length within the scope of this chapter. Instead, this section has aimed to give an outline of how player engagement with games may also be practiced outside of the game in a narrow sense. The review has focused on research that identifies different forms and functions of productive player practices, delineating the game itself from its paratexts and offering a few examples of studies of different forms of player production. While this research is seldom carried out under the moniker of intermediality, it nevertheless addresses medial relations because the participatory practices it describes are communicational, expressive, and/or performative by nature. While this section has studied playermade works, it is important to stress that between the official game and the surrounding official paratexts, and the player-made works there is a great deal of additional texts, in the form of journalistic articles, television programs about games, and so on. Studies of these aspects are still sparse, but not completely missing. Notable examples of this perspective on games include Graeme Kirkpatrick’s (2012, 2015, 2017) studies of the construction of gaming culture through UK gaming magazines and Tero Kerttula’s (2022) exploration of the broadcasting of play in television game shows and journalistic magazine programs, and how this form of broadcasting compares to the modern Let’s Play videos mentioned earlier in this section. To understand the intermedial network in which game culture is practices, it is necessary to also include these paratexts.

Conclusion This chapter aims to offer a broad overview on research addressing the qualified medium of games as an intermedial phenomenon. Most of the works included in this review identifies itself as belonging to the field of game studies. As this research shows, scholarly engagement with computer games almost unavoidably involves considering basic, qualified, and technical media. Still, a few remarks about the concept of intermediality in the field of game studies must be made. When game studies emerged as a research field in the late 1990s and early 2000s, games were seized as an object of study by scholars from other fields such as literary studies, the arts, and media studies. This had consequences for how games were conceptualized and studied as a medium. Scholars sought in different ways to delineate games from existing media either by claiming games to constitute novel, narrative media or by resisting the concept of media altogether. Still, as scholars such as Tom Apperley and Darshana Jayemanne (2012) and Jan Švelch (2020) have observed, the early years of games studies were focused on the formal aspects of games, such as their ludic or narrative constituents and their function. In this research, intermediality rests as an underlying concept that motivates the aim of finding the unique qualities of games in an ever-increasing landscape of old and new media. When the intermediality of games was addressed, it was through the overall paradigm of “remediation” (Bolter and Grusin 2000). This is particularly clear in Screenplay (King et al. 2002), an anthology on the relationship between

1024

I. K. Hammeleff Jørgensen

games and cinema. In the introduction the editors are careful to emphasize that the anthology aims to explore points of contact and diversion between two distinct media. As the first cohort of researchers born from the field of game studies emerged, this rhetoric of media differentiation was replaced by a readiness to consider appropriate the concepts and theories of other qualified media. Early game studies research further sought to understand games as mediated phenomena that offer the player not only ludic experiences but also narrative, social, creative, and aesthetic pleasure. More recently, more and more game researchers embrace games as phenomena between media. Rather than considering games as a qualified media object among other qualified media, Aarseth (2017) suggests considering games as a perspective that can be imposed on a range of other phenomena. As a result, game scholars are also increasingly framing their research in intermedial terms and exploring the extent to which games studies may contribute to intermedial theory. Backe (2020) has recently suggested the notion of hypercomplex objects to describe “media objects which combine traits of so many qualified and multimodal media that they cannot be meaningfully identified with conventional media terms” (2020: 3). Backe argues that such hypercomplex objects necessitate a hermeneutic orientation towards play as a meaning-making practice. This also calls for new ways of thinking about the intermediality of games. The anthology “Intermedia Games—Games Inter Media. Video Games and Intermediality” (2019) is an example of how game scholars have embraced the concept of intermediality. According to the editors Michael Fuchs and Jeff Thoss, computer games is one of the richest areas in regard to intermedial phenomena, and in the 12 chapters of the book, the contributors consider aspects of games and phenomena around games (many of which have also been touched upon in this chapter) as examples of intermedial exchange. In comparison, the focus of this chapter has been to discuss how intermedial theory has underlain existing game research. This research has been divided into four main areas. The first section discussed first how theories of games as an expressive medium have demarcated games from other media, and second how theories of games as a mediated phenomenon have then considered how games appropriate what Elleström (2010, 2020) calls basic media. This research showed that the spatial and operational quality of games affects the expressive qualities of games to a large extent. The next section discussed how games may be part of extensive transmedial worlds or universes. The notion of transmedial worlds was used by scholars to theorize the role of narrative elements in games, but also how these narrative elements may be repurposed and expanded on in other media. As this chapter aims to go beyond traditional divisions between form and content, the following sections considered, first, the role of media technologies in relation to games, and, second, how player participation extends beyond the main “text” of the game. While Juul (2018) has characterized games as “transmedial” by principle, the section on games and media technologies reviewed research establishing the intimate links between games and their mediating technologies: first, in the form of media archeological approaches discussing how earlier technologies and their use cultures have shaped the

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1025

qualified medium of games – its aesthetic and usage as well as its discursive construction; second, in the form of platform studies analyzing how the technical platforms of games constrain possible game design; and third, regarding the alterations of games that take place, when a game travels from one technical platform to the next. The final section discussed the intermedial aspect of player engagement with games, not only as practiced when games are played but also in relation to a range of other qualified media and with different purposes. This section also discussed player-made products as paratexts and discussed a few examples of player-led paratextual practices such as gameplay streaming on Twitch, Let’s Play videos, and cosplay. What emerges from this is an overview of how the qualified medium of games is situated in an intermedial network of other qualified media, technical media, expressive and aesthetic conventions, and practices of appropriation, use, and reuse. A sketch of this network is offered in Fig. 3, which also includes selected references to works discussed in this chapter. While the research in all four areas is much richer and more nuanced than this chapter has accounted for, this summary offers a good starting point for understanding the qualified medium of computer games as a fundamentally intermedial phenomenon.

Fig. 3 An intermedial network of the qualified medium of computer games

1026

I. K. Hammeleff Jørgensen

References Aarseth, Espen. 1997. Cybertext: Perspectives on Ergodic Literature. JHU Press. ———. 2003. Playing Research: Methodological Approaches to Game Analysis. In Proceedings of the Digital Arts and Culture Conference, 28–29. http://www.bendevane.com/VTA2012/ herrstubbz/wp-content/uploads/2012/01/02.GameApproaches2.pdf. ———. 2009. Define Real, Moron! Edited by Stephan Günzel, Michael Liebe, and Mersch. Some Remarks on Game Ontologies. In DIGAREC Keynote-Lectures 10 (6). ———. 2017. Just Games. Game Studies 17 (1). Aarseth, Espen, and Gordon Calleja. 2015. The Word Game: The Ontology of an Undefinable Object. In FDG. Altice, Nathan. 2015. I Am Error: The Nintendo Family Computer/Entertainment System Platform. MIT Press. American McGee. 2000. American McGee’s Alice. Apperley, Thomas, and Darshana Jayemane. 2012. Game Studies’ Material Turn. Westminster Papers in Communication and Culture 9 (1): 5–25. Arkane Studios. 2016. Dishonored 2. ———. 2017. Prey. Backe, Hans-Joachim. 2018. Metareferentiality through In-Game Images in Immersive Simulation Game. In Proceedings of the 13th International Conference on the Foundations of Digital Games, 1–10. ———. 2020. Beyond Dominance, Mixture, and Hybridity. On the Challenges of Hypercomplex Objects. Between. Journal of the Italian Association for the Theory and Comparative History of Literature 10 (20): 1–26. Barr, Pippin. 2020. Film Adaptation as Experimental Game Design. Arts 9: 103. Multidisciplinary Digital Publishing Institute. Bethesda Softworks. 2011. The Elder Scrolls V: Skyrim. Blom, Joleen. 2018. Overwatch as a Shared Universe: Game Worlds in a Transmedial Franchise. In DiGRA Conference. Bogost, Ian. 2010. Persuasive Games: The Expressive Power of Videogames. Cambridge, MA: The MIT Press. Bolter, Jay David, and Richard Grusin. 2000. Remediation: Understanding New Media. MIT Press. Calleja, Gordon. 2011. In-game: From immersion to incorporation. MIT Press. Carmack, John, John Romero, Adrian Carmack, Kevin Cloud, and Tom Hall. 1993. Doom (Series). CD Project. 2015. The Witcher 3: Wild Hunt. Collins, Karen. 2008. Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. MIT Press. Consalvo, Mia. 2009. Cheating: Gaining Advantage in Videogames. MIT Press. ———. 2017. When Paratexts Become Texts: De-Centering the Game-as-Text. Critical Studies in Media Communication 34 (2): 177–183. Elleström, Lars. 2010. The Modalities of the Media. In Media Borders, Multimodality and Intermediality, edited by Lars Elleström, 11–48. Palgrave Macmillian. ———. 2020. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. Beyond Media Borders, Volume 1 Intermedial Relations among Multimodal Media, 1. Eskelinen, Markku. 2001. The Gaming Situation. Game Studies 1 (1): 68. Flynn, Bernadette. 2003. Geography of the Digital Hearth. Information Communication & Society 6 (4): 551–576. Frasca, Gonzalo. 2003. Simulation versus Narrative. The Video Game Theory Reader: 221–235. Fuchs, Mathias. 2019. Itineraria Picta. In Ludotopia, 215–230. transcript-Verlag. Fuchs, Michael, and Jeff Thoss. 2019. Intermedia Games—Games Inter Media: Video Games and Intermediality. Bloomsbury Publishing USA.

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1027

Gekker, Alex. 2018. Let’s Not Play: Interpassivity as Resistance in ‘Let’s Play’videos. Journal of Gaming & Virtual Worlds 10 (3): 219–242. Grabarczyk, Pawel, and Espen Aarseth. 2019. “Port or Conversion? An Ontological Framework for Classifying Game Versions.” In DiGRA Conference. Guins, Raiford. 2014. Game after: A Cultural Study of Video Game Afterlife. MIT Press. Hancock, Michael. 2016. Games with Words: Textual Representation in the Wake of Graphical Realism in Videogames. Hart, Iain. 2014. Meaningful Play: Performativity, Interactivity and Semiotics in Video Game Music. Musicology Australia 36 (2): 273–290. ———. 2015. Hard Boiled Music: The Case of LA Noire. Screen Sound 5: 19–35. Huhtamo, Erkki. 2005. Slots of Fun, Slots of Trouble. In The Handbook of Computer Game Studies. Cambridge, MA: MIT Press [Google Scholar]. ———. 2006a. Elements of Screenology: Toward an Archaeology of the Screen. NavigationenZeitschrift Für Medien-Und Kulturwissenschaften 6 (2): 31–64. ———. 2006b. The Pleasures of the Peephole: An Archaeological Exploration of Peep Media. In Book of Imaginary Media: Excavating the Dream of the Ultimate Communication Medium, 74–155. Jenkins, Henry. 2006. Convergence Culture. New York University Press. Jørgensen, Kristine. 2006. On the Functional Aspects of Computer Game Audio. ———. 2008. Left in the Dark: Playing Computer Games with the Sound Turned Off. Ashgate. ———. 2013. Gameworld Interfaces. MIT Press. Jørgensen, Ida Kathrine Hammeleff. 2018. Media and Games: An Intermedial Framework. In Proceedings of the 13th International Conference on the Foundations of Digital Games, 1–10. ———. 2020. Games as Representational Artifacts: A Media-Centered Analytical Approach to Representation in Games. PhD Thesis, IT-Universitetet i København. Juul, Jesper. 2018. The Game, the Player, the World: Looking for a Heart of Gameness. PLURAISRevista Multidisciplinary 1 (2). Kaklamanidou, Betty, and Maria Katsaridou. 2013. ‘Silent Hill’: Adapting a Video Game. Literature/Film Quarterly 41 (4): 266–277. Kerttula, Tero. 2022. (Broad) Casting the Game: The Spectacle of Real in Representing and Narrating Video Game Play. JYU Dissertations. King, Geoff, Tanya Krzywinska, and Geoff Siegel. 2002. Screenplay: Cinema/Videogames/Interfaces. Wallflower Press. Kirkpatrick, Graeme. 2012. Constitutive Tensions of Gaming’s Field: UK Gaming Magazines and the Formation of Gaming Culture 1981–1995. Game Studies 12 (1): 3. ———. 2015. The Formation of Gaming Culture: UK Gaming Magazines, 1981-1995. Springer. ———. 2017. How Gaming Became Sexist: A Study of UK Gaming Magazines 1981–1995. Media, Culture & Society 39 (4): 453–468. Kittler, Friedrich. 1995. There Is No Software. Ctheory: 10–18. Klastrup, Lisbeth, and Susana Tosca. 2004. Transmedial Worlds-Rethinking Cyberworld Design. In 2004 International Conference on Cyberworlds, 409–416. IEEE. Klevjer, Rune. 2002. In Defense of Cutscenes. In CGDC Conf. Citeseer. ———. 2012. Enter the Avatar: The Phenomenology of Prosthetic Telepresence in Computer Games. In The Philosophy of Computer Games, 17–38. Springer. ———. 2019. Virtuality and Depiction in Video Game Representation. Games and Culture 14 (7–8): 724–741. Konami. 1999. Silent Hill (Series). Lamb, Brendan, and Barnabas Smith. 2018. From Skyrim to Skellige: Fantasy Video Game Music within a Neo-Mediaevalist Paradigm. Musicology Australia 40 (2): 79–100. Lamerichs, Nicolle. 2011. Stranger than Fiction: Fan Identity in Cosplay. Transformative Works and Cultures 7 (3): 56–72. ———. 2018. Productive Fandom. Amsterdam University Press. Laurel, Brenda. 2013. Computers as Theatre. Addison-Wesley.

1028

I. K. Hammeleff Jørgensen

Lippitz, Armin. 2019. Lost in the Static? Comics in Video Games. Intermedia Games—Games Inter Media: Video Games and Intermediality: 115–132. Loveshack. 2014. Framed. Lowood, Henry, and Michael Nitsche. 2011. The Machinima Reader. MIT Press. Maher, Jimmy. 2018. The Future Was Here: The Commodore Amiga. MIT Press. Martin, Cathlena. 2010. Wonderland’s Become Quite Strange: From Lewis Carroll’s Alice to American Mc-Gee’s Alice. Beyond Adaptation: Essays on Radical Transformations of Original Works, 133–143. Maxis. 2008. Spore. Maxis Redwood Shores. 2006. The Sims 2 - Open for Business. Mizsei Ward, Rachel. 2018. ‘Was the 2005 Doom Film ‘Doomed from the Beginning?’: FirstPerson Shooters, Subjective Cameras and Intertextuality in the Critical Reception of the Film Adaptation Doom (2005). Journal of Adaptation in Film & Performance 11 (1): 57–70. Montfort, Nick, and Ian Bogost. 2009. Racing the Beam: The Atari Video Computer System. MIT Press. Möring, Sebastian. 2013. Games and Metaphor – A Critical Analysis of the Metaphor Discourse in Game Studies. IT University of Copenhagen. Möring, Sebastian, and Marco de Mutiis. 2019. Camera Ludica: Reflections on Photography in Video Games. Intermedia Games—Games Inter Media: Video Games and Intermediality. New York: Bloomsbury Academic, S, 69–94. Murray, Janet H. 2012. Inventing the Medium: Principles of Interaction Design as a Cultural Practice. MIT Press. ———. 2016. Hamlet on the Holodeck. Simon and Schuster. Nitsche, Michael. 2008. Video Game Spaces: Image, Play, and Structure in 3D Worlds. MIT Press. Parikka, Jussi. 2013. What Is Media Archaeology? John Wiley & Sons. Parikka, Jussi, and Jaakko Suominen. 2006. Victorian Snakes? Towards a Cultural History of Mobile Games and the Experience of Movement. Game Studies 6 (1). Parlett, David. 1999. The Oxford History of Board Games. USA: Oxford University Press. Pichlmair, Martin, and Fares Kayali. 2007. Levels of Sound: On the Principles of Interactivity in Music Video Games. In DiGRA Conference. Citeseer. Richardson, Ingrid. 2018. Participatory Media and the Lusory Turn: Paratextuality and Let’s Play. In The Routledge Handbook of Digital Writing and Rhetoric, 389–398. Routledge. Roblox Corporation. 2006. Roblox. Rockstar Games. 2011. L.A. Noire. Ryan, Marie-Laure, and Jan-Noël Thon. 2014. Storyworlds across Media: Toward a MediaConscious Narratology. University of Nebraska Press. Sanrio Digital, and Typhoon Games. 2009. Hello Kitty Online. Sell, Mike. 2021. What Is a Videogame Movie? Arts 10: 24. Multidisciplinary Digital Publishing Institute. Stevens, Joshua. 2020. Child of the Elder Blood: A Semiotics of Folklorism in the Soundtrack of The Witcher 3: Wild Hunt. Games and Culture, 1555412020914701. Švelch, Jaroslav. 2013a. Indiana Jones Fights the Communist Police: Local Appropriation of the Text Adventure Genre in the 1980s Czechoslovakia. Gaming Globally, 163–181. Springer. ———. 2013b. Say It with a Computer Game: Hobby Computer Culture and the Non-Entertainment Uses of Homebrew Games in the 1980s Czechoslovakia. Game Studies 13 (2). Švelch. 2015, January. Towards a Typology of Video Game Trailers: Between the Ludic and the Cinematic. G| A| M| E, the Italian Journal of Game Studies, (4). Švelch, Jaroslav. 2018. Gaming the Iron Curtain: How Teenagers and Amateurs in Communist Czechoslovakia Claimed the Medium of Computer Games. MIT press. Švelch, 2020, January. Paratextuality in Game Studies: ATheoretical Review and Citation Analysis. Game Studies 20 (2).

39

The Qualified Medium of Computer Games: Form and Matter, Technology,. . .

1029

Švelch, Jaroslav. 2021. Adventures in Everyday Spaces: Hyperlocal Computer Games in 1980s–1990s Czechoslovakia. In Game History and the Local, 17–35. Springer. Švelch, Jan, and Tereza Krobová. 2016a. Historicizing Video Game Series through Fan Art Discourses. Transformative Works and Cultures 22 (September). ———. 2016b. Who Is the Note-Worthy Fan? Featuring Players in the Official Facebook Communication of Mainstream Video Games. Replay. The Polish Journal of Game Studies 3 (1): 81–100. Swalwell, Melanie. 2007. Independent Game Development: Two Views from Australia. Videogames and Art: 160–180. ———. 2008. 1980s Home Coding: The Art of Amateur Programming. Aotearoa Digital Arts New Media Reader: 192–201. ———. 2021. Homebrew Gaming and the Beginnings of Vernacular Digitality. MIT Press. Thon, Jan-Noël. 2009. Computer Games, Fictional Worlds, and Transmedia Storytelling: A Narratological Perspective. In Proceedings of the Philosophy of Computer Games Conference. ———. 2014. Subjectivity across Media: On Transmedial Strategies of Subjective Representation in Contemporary Feature Films, Graphic Novels, and Computer Games. In Storyworlds Across Media: Toward a Media-Conscious Narratology, ed. Marie-Laure Ryan and Jan-Noël Thon, 67–102. University of Nebraska Press. Toups, Z. O., Nicolas LaLone, Katta Spiel, and Bill Hamilton. 2020. Paper to Pixels: A Chronicle of Map Interfaces in Games. In Proceedings of the 2020 ACM Designing Interactive Systems Conference, 1433–1451. Wallin, Mark Rowell. 2007. Myths, Monsters and Markets: Ethos, Identification, and the Video Game Adaptations of the Lord of the Rings. Game Studies 7 (1). Wirman, Hanna. 2009. On Productivity and Game Fandom. Transformative Works and Cultures 3: 8–8. Wittgenstein, Ludwig. 2009. Philosophical Investigations. John Wiley & Sons. Zagal, José P., Michael Mateas, Clara Fernández-Vara, Brian Hochhalter, and Nolan Lichti. 2007. Towards an Ontological Language for Game Analysis. Worlds in Play: International Perspectives on Digital Games Research 21: 21.

Part IV Intermedial Perspectives on Media in the Twenty-First Century: Challenges in Contemporary Society

The Ecological Crisis and Intermedial Studies Jørgen Bruhn

40

, Matilda Davidsson, and Niklas Salmose

Contents Part 1 Introduction: Stating the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 2 Media-Related Approaches in the Environmental Humanities . . . . . . . . . . . . . . . . . . . . . . . . . Part 3 Intermedial Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 4 Transmediations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 5 Comparative Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part 6 Case Study: Future Food Cultures Across Media Borders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eatforum.org . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bladerunner 2049 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Comparative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1034 1036 1041 1043 1047 1048 1049 1051 1053 1054 1056

Abstract

Taking the human-induced ecological crisis as our starting point, in this chapter we discuss the implications of how this crisis is communicated. The knowledge and data produced by the natural sciences is mediated and communicated in different ways to the public and after a review of central traditions relating to this question, we argue that there is a need for a theory of comparison that can encompass different disciplines and aesthetic forms of media. We combine an intermedial toolbox of terms introduced by Lars Elleström, most notably his concept of “transmediation,” with the field of ecocriticism that originates in literary studies but today encompasses a broader definition of media representing the ecological crisis. This we call intermedial ecocriticism, which has theoretical and methodological implications for the analysis and comparison of ecomedia. We exemplify this by discussing existing research from intermedial scholars as

J. Bruhn (*) · M. Davidsson · N. Salmose Linnaeus University, Växjö, Sweden e-mail: [email protected]; [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_52

1033

1034

J. Bruhn et al.

well as with a case study regarding representations of future food cultures from the website Eatforum.org and in the sci-fi movie Blade Runner 2049. Keywords

Intermediality · Ecocriticism · Environmental humanities · Ecological crisis · Intermedial ecocritism

Part 1 Introduction: Stating the Problem The planetary ecological crisis that humanity is facing is not a natural phenomenon. It is a threat produced by human beings living in societies characterized by damaging structures that originate in historical, economic, political, cultural, and geographical factors. The idea of the Anthropocene, a geological definition of a modernity defined by massive human influence, to the extent that the cultural history of humanity has put its mark on the supposedly non-human natural history of the earth, is among the most widespread, and most criticized, terms for the planetary situation we are in. No matter what term is used, the Plantationocene, the Capitalocene, the Anthropocene, or any other, it is founded in a consciousness about a current and future crisis in several interrelated areas that includes global pollution, major species extinction events, and global warming (Ellis 2018; Demos 2017; Malhi 2017; Bonneuil and Fressoz 2016; Haraway 2016; Moore 2016). The most important symptoms of this crisis are investigated and described first of all in the natural sciences, including biology, geology, atmospheric climate science, and oceanography: these academic fields collect, dissect, document, archive, and disseminate material concerning the crisis that makes it possible for citizens and decision makers in private or public businesses to navigate in the crisis and to take stock of the kind of measures they need to take. The natural sciences meticulously collect data, which are often based on measurements taken by sophisticated technical apparatuses in and outside laboratory facilities. The data are, since the massive breakthrough of immensely strong computing powers, analyzed by way of the now ubiquitous computer simulations (Roundtree 2014) which enable scientists to produce extremely complex data sets. These data sets are representations of aspects of the outer world, not only now, at the present moment but also in relation to the past, and the data are used to predict future scenarios, for instance regarding ecological issues. Most of such work, and the discussions around it, takes place in the confined spaces of academic journals and scientific conferences, and this is how most non-specialists get to know about scientific results and research about the ecological crisis – via printed or digital or other media forms, often in relation to breakthroughs or particularly frightening predictions. This natural scientific production of knowledge is a complex, multifaceted social process that has been keenly investigated using anthropological methods (Latour et al. 2013) as well as via the strong tradition of science and technology studies (Sismondo 2010).

40

The Ecological Crisis and Intermedial Studies

1035

This is a rather obvious but potentially very important insight: for the natural scientific results to make any impact outside the relatively narrow circles of scientific publications and debate, the results need to be communicated. And communication per se involves media products with different possibilities and limits. Newspapers, popular science journals, and documentaries are some of the well-known media types that communicate information about scientific research, but poetry, science fiction novels, feature films, and art exhibitions can do this too. That the results of scientific research need to be communicated to the public is, of course, a known fact in several well-developed disciplines, including risk communication, health communication, science communication, and climate change communication studies. These fields are based on a tradition of communication studies which is particularly well suited to analyzing the history and impact of journalistic media and mass medial forms: but traditionally, they have been less interested in aesthetic communication. We briefly consider this in the first part of the literature review in Part 2. A classical entry point to the question of representing the ecological crisis is ecocriticism, which we will examine in the literature review in Part 2. We consider ecocriticism to be a more specialized but comparable approach to the many competitors that deal with understanding the communication of climate change, risk communication, disaster studies, and many other fields. However, there is a problem with the heterogeneity of this immense field of widely different communicative forms. An illustrative example (which we will discuss in more detail at the end of this chapter) is how ideas and concepts concerning sustainable future food cultures are communicated through many different media types and products; hence a complete understanding of the effect of food media requires an analysis of several different media types. In our example we will compare food communication on a website, Eatforum.org, dedicated to scientific discussions of the food transformation with a scene from the dystopian sci-fi film Blade Runner 2049. The immediate problem that arises from analyzing and comparing the two examples is that they fall into two very different categories. In our intermedial terminology that we will briefly introduce later, these two specific “media products” belong to two different “qualified media types” (Elleström 2010, 2020). The fundamental differences between qualified media types make it difficult to form wellfounded critical comparisons between them, unless the two media products are simply, and quite crudely, considered to be different containers enclosing the same content. This is probably why the fields invested in discussing how ecological questions may be or should be communicated have mostly refrained from making more substantial comparisons across media borders. In general, the environmental humanities tradition is haunted by a methodological impasse in that it often tends to do one of two things. The first is that in terms of media types, and thus academic disciplines, there is a tendency to “compartmentalize” ecological questions, as if the ecological crisis respects the borderlines defined by media types or academic departmental traditions. In other words, and this will be a recurrent observation in the literature review, in principle all researchers offer monomedial approaches to the ecological discussions – or when different media

1036

J. Bruhn et al.

types are included, they are not systematically compared. This is a problem, because the ecological crisis as it is communicated in different media types does not respect the media borders that have been set up by different academic traditions. This leads us to another impasse, namely that when cultural theorists or communication scholars with broader interests span different media types and academic disciplines dealing with ecological representations, they often do so without a methodological and theoretical grounding that makes it possible to make good, comparative analyses. This is a problem because it would be very helpful to compare different media types’ abilities to represent ecological questions; such comparisons are scientifically interesting and might be very valuable for other academic fields as well as for practical communicators and thus for decision makers. This is why we argue that the environmental humanities could be supplemented with an analytical tool that is sufficiently broad to analyze several media types and sufficiently finegrained to do this in a detailed way. This position is what we call intermedial ecocriticism. In what follows, therefore, we investigate which options are available if the ideas concerning the ecological crisis, as these are represented in a wide array of media forms, are to be analyzed in a fruitful way. To begin, we will present a literature review in Part 2, where we give an overview of the most important discussions relating to communicating the ecological crisis, with a particular focus on methods or theories that are not monomedial in their approach. Here we also discuss the notion of ecomedia. After this very general overview we will discuss two positions that most directly point in the direction that we find productive. In Part 3 we introduce aspects of intermedial studies and in Part 4 we introduce the idea of transmediations/media transformations of scientific material into different media. This will pave the way for describing in Part 5 some of the comparative position combinings including the notion of “intermedial ecocriticism,” which in Part 6 will be exemplified in a comparative analysis of two media products that deals with future food cultures. Part 7 briefly summarizes the chapter.

Part 2 Media-Related Approaches in the Environmental Humanities Before we survey the critical literature in the area of environmental humanities, we will briefly discuss the very field of environmental humanities. The broad field of environmental humanities has had several different goals in its history but one way to define it is as an interdisciplinary approach towards ecological problems, including the climate emergency. As Ursula Heise has stated in the introduction of The Routledge Companion to the Environmental Humanities, “The environmental humanities do not so much propose a new object of study, a new humanistic perspective on a nonhumanistic field, or a particular set of new methods, as they combine humanistic perspectives and methods that have already developed in half a dozen or so disciplines over the last four decades” (Christensen et al. 2017, 1). Often, the goal has been to raise awareness about ecological issues and to come up with ways in which the humanities can contribute to potential solutions. By

40

The Ecological Crisis and Intermedial Studies

1037

investigating how we as humans make sense of, value, and interact with the world around us, subfields of the environmental humanities conduce new perspectives on and understandings of the planetary-wide ecological issues, and they often do so in collaboration with the natural sciences. An important task of humanities work confronting the environmental crisis is to understand how we make sense of information relating to the crisis, but also understand how to communicate these questions to individuals and social groups. Climate change communication and nearby fields such as science-, risk-, and environmental communication all deal with processes of communication within the scientific community as well as how to convey scientific data to the public (Doyle 2011). Needless to say, communication studies often overlap with intermedial studies in that the process of communication is also one of mediating a message or an idea, and communication studies shares with intermedial studies the conviction that communication is a complex process consisting of several interdependent and heterogeneous dimensions. According to the Yale Program on Climate Change Communication, the classic and rather simplistic model of communication as messenger–message–audience needs to be rebuilt into a more complex and accurate model that is based on how communication actually works: In many situations there are a variety of messengers, who craft and transmit different and sometimes opposing messages, through an ever-growing number and complexity of channels, to diverse audiences who have their own pre-existing beliefs, attitudes and values, and who actively interpret and construct their own meanings from the messages they receive, which they in turn communicate through their own networks. Thus communication occurs within a rich, highly complex, and dynamic system of individuals, organizations, and institutions, with sometimes widely divergent knowledge, politics, and cultures. And it is through these dynamic processes that societies develop climate change awareness, (mis) understanding, concern, and action. (Yale Program on Climate Change Communication n.d.)

The environmental emergency is a complex global phenomenon consisting of sometimes very abstract issues that can be hard to grasp or even to imagine in everyday life. That is what made Timothy Morton (2013) construct the term “hyperobjects,” signifying objects or processes that are intangible in time and space, such as global heating and ocean acidification, and phenomena that we can only face through diverse mediations. The question of comparison of representations across media is one of the issues that an intermedial approach to the ecological emergency can be beneficial for, because it is not hindered by the form or content of the individual media types. With such complex and intricate communication processes, questions regarding truth claims, the transmediation of factual statements into different media types and the mediation of information about the climate emergency to many different audiences require equally equipped methods of analysis. The fields of environmental communication and science communication more generally have the task of examining how research results are communicated to the public; this is often done through the transformation of scientific data into other media forms and popular culture. They also examine how the dissemination of information is implemented and how different forms of communication between

1038

J. Bruhn et al.

different parts of society work. Environmental communication is also a tool for creating awareness of changes in the environment and provoking action, as well as analyzing framings of and responses to the available information (Slovic et al. 2019). The communication of ecological questions has also been researched within ecocriticism. Originally based in literary studies, ecocriticism is the study of literature and the environment surrounding humans, how human and non-human relationships are (re)presented in media products, and discussions of philosophical alignments towards the concept of “nature.” Initially, ecocritical scholars often worked as advocates for nature, but over time their more central focuses have become environmental threats, crises, or emergencies (Clark 2015; Barry and Welstead 2017; Zapf 2016; Iovino and Oppermann 2014). The field deals with questions that have been acknowledged and discussed by philosophers, artists and writers, scientists and politicians throughout many epochs and in many geographical areas, but Rachel Carson’s Silent Spring and Leo Marx’s The Machine in the Garden: Technology and the Pastoral Ideal in America, both published in the 1960s, are often referred to as important works that started the environmental movement from which ecocriticism sprang. The starting point for ecocriticism as an analytical discipline has been identified as Rueckert’s 1978 text Literature and Ecology: Experiment in Ecocriticism that coined the term that named the field (Rueckert 1978). Nowadays there are plenty of academic journals, research programs, and academic networks working with an assortment of associated questions, but scholars such as Glotfelty and Fromm (1996) and Buell (2005) divide ecocriticism into two waves loosely separated by time and different approaches to deciding which texts to analyze and by which parameters. The first wave, they argue, took place during the 1980s and 1990s and focused predominantly on nature writing as the object of study. Nature was also seen as something separate and distinguished from culture and the human, and the literary objects of study were first and foremost North American and Euro-centered. The second wave, starting in the late 1990s, moved beyond the notion of separation into dichotomies and strode away from the romanticizing of nature. In the past two decades, what can be titled a third wave, a myriad of approaches has appeared and developed into a broad discipline with an interest in everything from Animal Studies to dystopias. Posthuman orientations such as New Materialism (Bennett et al. 2010) highlight the interconnectedness and co-dependence of ecosystems that humans as well as non-humans are a part of. By questioning the ontology of concepts like “nature” and “wilderness” and the inherent dichotomy between “human” and “culture,” they show how agency can be described as a force rather than an action and how multispecies relations work by focusing on networks between entities. One such example is perceiving the human body as an intersection where nature and culture meet and are inseparable since our bodies are home to millions of microbes (Alaimo 2010). Other important branches that have developed in parallel, or grown out of or borrowed ideas from ecocriticism, are the often related ideas of Ecojustice (Moore 2015; Ghosh 2016), Ecofeminism (Plumwood 1993; Grusin 2017), Queer Ecology (Mortimer-Sandilands and Erickson 2010), and Environmental Postcolonialism

40

The Ecological Crisis and Intermedial Studies

1039

(Nixon 2011). All of them highlight the intersections between environmental destruction and social oppression. Many of these branches have roots in the environmental movement, the feminist movement and other activist movements, but their approaches have changed and grown throughout the decades. Ecofeminist scholars highlighted the connection between man’s subordination and objectification of nature as a tool to the patriarchal subordination of women (Plumwood 1993), and Queer Ecology has developed out of it as a response to ecofeminism’s heteronormativity (Gaard 1997). Ecocriticism today thus not only involves the analytical study of texts about nature or about the ecological crisis, but it is also the study of how the very concept of nature is defined in philosophical, cultural, social, and political senses. One influential definition is that of Greg Garrard, who states that “the widest definition of the subject of ecocriticism is the study of the relationship between the human and the non-human, throughout human history and entailing critical analysis of the term ‘human’ itself” (2012, 5). With this third wave or turn, new scholarly attention has also been given to indigenous and non-Western-centered perspectives of the relationship between nature and culture, and this has made room for a broader range of genres, texts, and mediums to be analyzed (Adamson and Michael 2017, 763). One important addition to the scholarship, with clear connections to communication studies (in particular regarding methodology), is so-called empirical ecocriticism, as presented by, among others Małecki (2019) and Schneider-Mayerson, Weik von Mossner, and Małecki (2020). As an interdisciplinary approach, empirical ecocriticism investigates the influence of environmental narratives on audiences, taking inspiration from reception studies and using experimental settings and theories of cognition and psychology to measure the affective or other responses of consumers of ecomedia. One important goal for empirical ecocriticism is to present proof for claims made by ecocritical scholars about the importance of environmental media as a tool for changing people’s opinions (and possibly behavior) regarding the climate emergency and its influences on the world at large. Affect theory and cognitive science in relation to environmental issues have in themselves become substantial branches of study since they deal with both positive and negative emotional responses and experiences (Weik von Mossner 2017; Bladow and Ladino 2018). While all the aforementioned perspectives can be and have been applied to a broad range of media forms, ecocriticism was originally a literary and text-focused theory, and none of the perspectives offer a method for comparing media forms like the one we suggest: the crucial distinction is that intermedial ecocriticism can both focus on and compare transformation between and within different media types. The transformation of scientific work into aesthetic and more easily comprehensible media types is often analyzed in relation to either one media type, like film, literature, or art, or in adaptation studies that are examining two forms, e.g., a book and a movie (Kaplan 2015). Although there are some research areas that deal with similar questions, they usually negotiate the differences in media types through a broader cultural perspective or through the use of a theme or trope that converges. For example, Andersen (2020) uses the Anthropocene as an umbrella term for the

1040

J. Bruhn et al.

ongoing environmental changes in his monograph, Climate Fiction and Cultural Analysis: A New Perspective on Life in the Anthropocene, which examines the Anthropocene from a cultural perspective in literary fiction with a focus on climate fiction novels. Another example is Davis and Turpin’s (2015) book Art in the Anthropocene: Encounters among Aesthetics, Politics, Environments and Epistemologies, which captures the intersections between contemporary art and the new conditions that the Anthropocene as a supposed epoch brings. A third example is Adam Trexler’s (2015) Anthropocene Fictions: The Novel in a Time of Climate Change, which stresses the novel as a medium that can be used to represent environmental change. While none of these examples explicitly use intermedial theory, all three focus on different ways to mediate and communicate the same issues that the term Anthropocene evokes. There is, however, still a tendency to compartmentalize or divide academic disciplines as well as different media types in the research. Trexler, in particular, deals only with narrative in anglophone literature, and Davis and Turpin with art in a broad sense of the word, whereas Andersen deals with both literature and film. There have of course been endeavors within ecocriticism to connect with media studies (of the humanities tradition – not the media and communication tradition most often associated with the social sciences) or intermedial and transmedial approaches. One example is the idea of having a “cultural ecology of literature” that can be used as a tool to explore how humans adapt to their surrounding world through culture and how culture is simultaneously transformed by changes in the environment, as suggested by Hubert Zapf (2021). Another important example of a comparative approach to mediations across media types is that of Rust, Monani, and Cubitt (2016). In Ecomedia: Key Issues, they discuss many different media types and their forms of representation and also investigate production and cultural perceptions through theoretical introductions and case studies. Even though the term ecomedia is ambiguous in that both “eco” and “media” can have many possible definitions, it is a handy term that we will use to mean any media that deals with ecological issues. Ecomedia is a useful term because it includes any form of cultural expression used to convey a message, as opposed to only social media or news media, etc. (For a critical discussion of ecomedia, see Bruhn 2021). Ecomedia studies and green cultural studies are open to conducting interdisciplinary investigations into how different ecomedia products present complex ideas regarding ecological issues, and it is our intention to develop this approach further with intermedial studies by looking into the affordances of different media types. Aesthetics as a philosophical concept of beauty frequently focuses on art, but in some cases it is also used as a broader notion or theme that can be explored in different media types, such as an “aesthetic of ecology,” which combines ideas about beauty within a framing of the health of the environment (Ballard 2021; Goniorava 2019). Another approach with intermedial potentials is the comparative study of narrative and storytelling in different media types. Expressions of culture from an ecological perspective are researched, for example, in Caracciolo’s (2021) Narrating the Mesh: Form and Story in the Anthropocene. A changing world requires new

40

The Ecological Crisis and Intermedial Studies

1041

ways to tell stories, and the development of new mediums requires new ways to analyze how these stories are told and how they impact the receivers of said stories.

Part 3 Intermedial Studies After the literature review of media-related approaches to the ecological crisis in Part 3 we now move towards intermedial studies. Media studies can be understood as the theoretical and historical study that aims to understand the material aspects of human and non-human communication, whereas media theory relates to a “natural history of media” (Eliassen et al. 2017). It is becoming more and more common to refer to and understand media as the natural “environment” of humanity, thus striking up a strong link between media studies and environmental humanities (notably in Peters 2015). With Lars Elleström providing the main theoretical inspiration, an intermedial theory of media has been developed at the Linnæus University Centre for Intermedial and Multimodal Studies in Sweden. An overview of the main ideas is found in Elleström’s recent work (Elleström 2021) or in an introductory version offering a broad scale of analytical tools in Bruhn and Schirrmacher’s Intermedial Studies: An Introduction to Meaning across Media (Bruhn and Schirrmacher 2022). The intermedial approach that we pursue here is developed as both a general theory of signification and communication and a methodology that is directed towards better understanding a wide range of different media types across the conventional borderlines of, for instance, artistic and non-artistic media types, or fictive or non-fictive types, or historical versus contemporary or analogue and digital media types. This wide-ranching perspective of the intermedial approach is what makes it particularly useful in the context of ecocritical and environmental questions. An important idea is that all media types may be defined according to what Elleström defines as their modalities (material, sensorial, temporal/spatial, and semiotic), and furthermore that any given media product necessarily has three media dimensions (basic media, technical media of display, and qualified media) (Bruhn and Schirrmacher 2022; Elleström 2021). An example may clarify the terminology, so let us take an example that relates to our discussions. AR6 Climate Change 2021. The Physical Science Basis is better known as the IPCC report. It is an unprecedented international and collective scientific effort involving the work of thousands of scientists assessing the global research output in several scientific fields relevant to climate change: this army of scientists peer review, edits, writes, and peer review again in principle all existing current research related to the planetary climate. Reading all of the 3900-page report is not a realistic option for non-expert readers, so the 42-page “Summary for Policy Makers” is a more relevant source of information about the panel’s work for the majority of journalists, citizens, and other decision makers. This is the report we will describe in our intermedial terminology. We access the report online on our desktop computer; the computer screen and navigating instruments are thus the technical media of display aspect of the report

1042

J. Bruhn et al.

(in this particular version: paper versions exist too). The material modality of the report, when accessed on a computer screen, is the two-dimensional screen, which uses lively colors. The spatio-temporal modality consists in an ordered sequence of pages that can be read in potentially different sequences (if one moves back and forth, like in a paper version). The dominating sense modality is sight: the readers look at the pages – there is no sound. There is a very limited aspect of physical interaction when pressing keys or moving a cursor to navigate the report digitally. The three Peircean semiotic modalities are activated. The reader understands the report, reading the mostly symbolic letters of the words, and combines this with iconic interpretation of the many visual illustrations; the indexical relation is present too, in the sense that all the data lying behind the scientific discussions are seen as real traces, real connections, from an outside world, the climate changing world. In basic media terms, the report consists of written language, several visual diagrams, and other illustrations that aim to clarify either the scales or the time frames of the scientific arguments. Most readers, based on their experience and by relating this media product to other communicative forms, will understand it in qualified media terms as a form of scientific communication or popular science communication. All media products, we argue following Elleström, may be described according to such a schematics, and furthermore we postulate that media products tend to cluster into relatively homogeneous groups, what we prefer to call media types: these would include, for instance, written narrative literature, live-action cinema, digital popular science writing, and Braille or mathematic formulae. The list is of course more or less endless. Each media type can be described as having a relatively clear medium specificity, but an important caveat is that media types change over time and space. The notion of popular science communication, for instance, was different in the nineteenth century from how it is perceived in the twenty-first; the notion of what a sculpture is has changed from antiquity to the present, and the type of sign language used by hearing impaired people differs according to their geographical location. Medium specificity is thus contextually defined. A third important idea is to specify the general starting point of much of the thinking in intermedial and multimodal theory, namely that all media products and all media types are characterized by being mixed, that is, by not being monomedial or unimodal. This mixedness can be conceptualized in different ways (Wolf 1999; Rajewsky 2002; Bruhn 2010; Elleström 2021; Bruhn and Schirrmacher 2022), but the two main definitional axes are heteromediality and the transformation. These have to do with, respectively, synchronic media combination (for instance of text and music in an opera aria, or text and image in a scientific article) and diachronic media transformation (for instance a novel turned into a film or a poll or survey of political sympathies that is turned into a diagram in a TV program). As will be discussed soon, the broad notion of media transformation as adaptation was introduced and richly exemplified in Linda Hutcheon’s A Theory of Adaptation (Hutcheon 2006) and the theory has been expanded and developed, for instance, in more strict intermedial directions. Consequently, the idea of seeing intermedial studies as a discipline “supporting” ecocritical investigations may lead to two possible approaches: the first is a

40

The Ecological Crisis and Intermedial Studies

1043

transmediation position which is in many ways comparable to existing positions, except for the fact that the investigation of the communication of ecological material is based on more solid, methodological ground (transmediation). This is a diachronic approach, and the essential question is: which differences occur when eco-related form and content is transferred from one media type to another? The second position is more synchronic and comparative: it involves an attempt to formulate a method of describing, comparing, and criticizing different media products with comparable eco-content or eco-form? A core question of such an investigation is therefore: how does the affordances of different media types affect two or more media products that tries to represent similar material?

Part 4 Transmediations Transmediation as an intermedial phenomenon, theory, and method in the sense that we use it was thoroughly established by Lars Elleström in Media Transformation: The Transfer of Media Characteristics Among Media (Elleström 2014), but the term itself was coined by Charles Suhor (1984). It was subsequently further defined and developed in the anthology Transmediations. Communication Across Media Borders (Salmose and Elleström 2020). Elleström defines transmediation as “repeated representation of media traits” among media (Elleström 2020, 3). Transformations between media products occur everywhere, all the time, although we are not always aware of this fact. When we read the latest Nordic noir novel on our surf pad as an e-book, the novel has been transmediated from one media product to another. As we browse through our (now antique) CD collection, we are choosing analogue music from a magnetic tape that has been transmediated into a digital CD format. At the cinema, we might decide to watch a zombie adaptation of Jane Austen’s Pride and Prejudice. The example of the Austen novel frames the historical backdrop of the development of transmediation from adaptation studies which have focused on the transformation from one specific media type to another, most often from a novel to a film (Boyum 1985; McDougal 1985; Corrigan 2011; Leitch 2007), but also from a text to the stage and from the theatre to film (Elliott 2020). Linda Hutcheon has executed more ambitious, and less constrained, work on adaptation successfully in A Theory of Adaptation and Thomas Leitch’s criticism of the fallacies of adaptation studies in “Twelve Fallacies in Contemporary Adaptation Theory” (Leitch 2003). Hutcheon’s and Leitch’s works have been especially crucial in initiating a focus on media transfer that does not simply consider media specificity alone but opens up transmedial aspects such as ideas, concepts, themes, structural patterns, and aesthetics, and in particular Hutcheon’s work has advanced a much wider field of investigations of popular cultural phenomena and the arts. Transmediation, as developed within intermedial studies, builds on a more medially open theory than adaptation studies. Studies in transmediation investigate media-specific qualities (including media qualifications, notions of a specific technical medium of display, media types, and products) in any kind of transformational

1044

J. Bruhn et al.

process. Further, transmedial aspects and concepts of representation, mediation, and re-mediation, are assembled in a fine-tuned tool to understand complex sets transmediations (Elleström 2020, 5).1 At a very basic level, much of the aesthetic work on the mediations of the Anthropocene implicitly engages in transfers of either scientific realities or future simulations and speculations into aesthetic and more accessible media types. Philosophical debates on the challenges of representing the Anthropocene with objectoriented ontological or new materialist agendas (Morton 2013, 2007; Nixon 2011), or politicized deliberations (Ghosh 2016; Moore 2015), have not to a larger extent considered how to more exactly describe, analyze and possibly compare the nature of these transfers, even though Morton’s concepts of dark ecology and ambient poetics and his recent attempts to define art experiences clearly have aesthetic configurations (Morton 2021; 2010). There has been a myriad of excellent studies on representations of the Anthropocene, but they have been monomedial (focusing on one specific media type only) or without any significant consideration of the functions of media transformations (Weik von Mossner 2017; Trexler 2015; Murray and Heumann 2014, 2009; Svoboda 2014). Separating transmediations from conventional adaptations of art products opens up this area to an understanding of the complex transmediations involved within the broad medial field of the Anthropocene. Jørgen Bruhn’s analysis of the intricacies of transmediating Anthropocene temporality avoids the common media-specific discussion and unfolds ideas concerning transmediation as essential aspects of how people experience the Anthropocene: “the term Anthropocene reaches the average reader [. . .] in the forms of transmediations” (Bruhn 2020a, 217). Similarly, Niklas Salmose, in “Three Transmediations of the Anthropocene: An Intermedial Ecocritical Reading of Facts, Sci-Fi, PopSci and Eco-Horror” offers an investigation of a broad-ranging chain of transmediations involving specific media products as well as qualified submedia types (or genres) in order to reveal the complex network of transmediations of ecocritical material during the three decades before, during and after the publication of the seminal environmental Silent Spring (Rachel Carson 1962). The proposal that is built on as the book progresses treats “the transmediation of the scientific discourse of the Anthropocene as a common ‘source’,” arranging the Anthropocene as a set of critical values and discourses on our environmental crisis that are conceptualized as a transmedial aspect and a similar shared media content (Salmose 2020, 257–258). It is evident from the title of Salmose’s chapter that an additional transmedial focus in ecocriticism is how the scientific facts are transmediated from either the raw material of scientific facts (as in academic journal articles, conference proceedings, diagrams, and maps, environmental simulations) into more digestible media types such as popular science, documentaries, and news. These debates are at the heart of

For a comprehensive overview of the intermedial field of transmediation, see Chap. 9, “Transmediation”, in Intermedial Studies: An Introduction to Meaning Across Media (Bruhn et al. 2022, 138–161).

1

40

The Ecological Crisis and Intermedial Studies

1045

Emma Tornborg’s analysis of how scientific facts are transmediated into the ecocritical poetry of Swedish poet Jonas Gren in “Transmediations of the Anthropocene: From Factual Media to Poetry.” Tornborg argues that there is a question that is intimately bound to the very idea of transmediations, perhaps even more so in a discussion about transformations of scientific facts to aesthetic media, which is “what happens with truth claims usually associated with [factual] media” when they are transmediated through media types not historically considered as trustworthy or truthful as scientific media (Tornborg 2020, 235)? Tornborg succinctly examines source media such as scientific articles, documentaries, and news programs. In her thorough analysis of the target media of the two poems by Gren, she reveals how several scientific media characteristics of the source media transmediate into the aesthetic form of the ecocritical poems. These questions of truth claims and authenticity are central when Bruhn and Gjelsvik direct their attention to the media type documentary film in “Cinematic Representations of a ‘Super Wicked Problem’: Climate Change in Documentary Film (Ice and the Sky and Chasing Ice” (2018). Discussing both the challenges of making the climate crisis visible (as in Morton’s concept of the hyperobject) and the challenges of transmediating scientific truth claims, they scrutinize the two cinematic documentaries Ice and the Sky (2015) and Chasing Ice (2012). The focus on documentary as a specific genre is highly relevant, since the media type involves exceptional media characteristics that are bound to accessible Anthropocene representations and questions of truthfulness. The intermedial consequences of the concepts of truthfulness and truth claims have been thoroughly considered in Elleström’s article “Coherence and Truthfulness in Communication: Intracommunicational and Extracommunicational Indexicality” (2018) and in the chapter “Truthfulness and Truth Claims as Transmedial Phenomena” (Bruhn et al. 2022). In this chapter, the authors develop Elleström’s theorizations concerning truthfulness defined in intermedial and semiotic terms and propose the following definition of these two terms: We understand truthfulness, in the most general sense, as a reliable representation of the world around us: the social world as well as the physical world. The concept of truth claims refers to the reasons why we should trust a media product. (Bruhn et al. 2022, 226)

Working from these premises, the chapter includes analyses on less researched specifics of the interrelation between scientific media and aesthetic representations of the climate crisis, with a particular focus on how truthfulness transmediates into climate fiction in terms of various truth claims. In three brief analyses, the authors study the truth claims of popular science in Rachel Carson’s Silent Spring, so-called poetic truthfulness in ecopoetry and the limitations of Hollywood transmediations and depictions of dystopic realities and end-of-the-world narratives, with a focus on Roland Emmerich’s feature film The Day After Tomorrow (2004). These issues are also central to Elleström’s study “Representing the Anthropocene: Transmediation of Narratives and Truthfulness from Science to Feature Film” (2020), in which he performs a source-target analysis of the transmediation with Anthropocene with a

1046

J. Bruhn et al.

focus on truthfulness from a specific scientific article, “The ‘Anthropocene’,” written by Paul J. Crutzen and Eugene F. Stoermer in 2000 (Crutzen and Stoermer 2000), to The Day After Tomorrow. Elleström concludes by speculating about the possibility of works of art and entertainment being even more directly scientifically truthful than scientific articles such as Crutzen and Stoermer’s “The ‘Anthropocene’” that depend on associated media products. Strongly index-based media types, such as various forms of films and photography, may well be relevant for science and thus truthfully represent those aspects of the extracommunicational domain that one expects scientific communication to reveal. (Elleström 2020, 47)

The controversy concerning the representation and transmediation of the Anthropocene (Ghosh 2016; Morton 2013; Nixon 2011) is considered in AnnaSofia Rossholm’s brief but thought-provoking study of how geological and temporal facts about mountains are transformed into a time-limited model of cinematic representations. In “Moving Mountains: Cinema, Deep Time and Climate Change in Swedish artist Hanna Ljungh’s I am Mountain, To Measure Impermanence” (2018), she contextualizes Ljungh’s installation (which includes a five-hour-long film) within the cinematic history of representations of mountains in genres ranging from adventure films to nature documentaries and the cinematic subgenre of mountain films (Bergfilme). She sees the transmediation of the reality of mountains to art as a study of different temporalities, of humans as agents and of agency as well as of minimalist aesthetics. A further focus on the transmedial aspects of cinema and the Anthropocene can be found in two articles by Salmose: “The Apocalyptic Sublime: Anthropocene Representation and Environmental Agency in Hollywood ActionAdventure Cli-Fi Films” (2018) and “Behemoth, Nostalgia and Ecological Agency” (2019). These two articles together identify both the limitations and the possibilities inherent in various cinematic genre formulas as correlated with industry practices and diverse levels of commercialization. In traditional Hollywood spectacles, as well as in auteur cinema, nostalgia operates as an affective constituent on top of the scientific discourse around the Anthropocene, but has diametrically different results. While Hollywood resorts to pure sentimentality and conservatism in its depictions of cataclysmic futures, albeit using effective aesthetics of embodiment and what Salmose terms the apocalyptic sublime (Salmose 2018, 1418–1424), Zhao Liang uses nostalgia as an exposing strategy in Behemoth, resulting not in a drift towards the past but in a “clairvoyant” representation of the present in opposition to the lost past.2

2

Consequently, a new nostalgia, or an Anthropocene nostalgia, is gaining momentum as a useful method and art style for interrogating and exposing the Anthropocene condition. This is discussed in Alicja Relidzyńska’s “The Nature of Irrevocability: Anthropocene Nostalgia in Hayley Eichenbaum’s Photography Series The Mother Road (2015–19)” (2022), in which she convincingly frames nostalgia as a critical method for investigating the post-human imagination in the wake of the changes that have taken place and are still doing so in the Anthropocene.

40

The Ecological Crisis and Intermedial Studies

1047

Part 5 Comparative Approaches The studies of transmediations of the ecological crisis is, as shown in Part 4, covers a very wide field and it attempts to investigate the intermedial discussions about how aspects of the ecological crisis can be thought of as media transformations. However, in order to understand how the ecological crisis is most effectively communicated, there is a need to thoroughly and methodologically investigate and critically compare ecomedia products. In the following part, we review intermedial work that has taken one further methodological step. It does not just investigate transmediations of the ecological crisis (or of the Anthropocene, or of more or less synonymous notions); in the following examples, the different transmediations are not only seen as representations of originating material but are also compared to each other. As seen in the literature review in Part 2, ideas concerning a comparative intermedial approach to ecological representations exist in different academic discourses and fields but have only seldom been developed systematically. But such notions were beginning to gain ground in the mid-2010s, when a research group in Växjö developed ideas in this direction. This led to an entire issue of a journal (Ekfrase, in 2016 (in Norwegian)) being devoted to the first ideas that were raised about the topic, and later the group also co-organized an international conference that took place in Romania in August 2019. The conference led to the publication of a thematic issue of a Romanian journal, Ekphrasis. Images, Cinema, Theory, Media, in 2020 (Bruhn and Lutas 2020 – see also Bruhn and Salmose, forthcoming 2023). In the introductory article to the thematic issue, Bruhn (2020b, 8), in “Intermedial Ecocriticism: The Anthropocene Ecological Crisis Across Media and the Arts,” argues that there is a lack of work on ecocriticism and related areas, noting “that no sustained attempts have been made to analyse and systematically discuss and compare the different types of media that deal with the ecological crisis.” Bruhn (2020b, 12) also points out that “[a] major analytical advantage of the intermedial studies approach is that it enables comparison of form and content elements in different media types, including media types that are normally considered to be so far apart that they are seen to be incomparable: a scientific article and a novel, a documentary film and a poem, or an advertisement campaign and a classical symphony.” It is this comparative approach, enabled by intermedial terminology, that Bruhn and his colleagues in Växjö wanted to advocate, even going so far as to suggest a research agenda with three targets. They suggest that intermedial ecocriticism achieves the following functions: (a) facilitates a broad, theoretically and methodologically solid approach to the very wide field of representations of the environmental crisis in many different media types; (b) enables comparisons across different media types, with both form and content issues including but not limited to narrativity, scientific truth claims, toxicity, and the ephemeral nature of the threat (to mention merely some of the issues discussed in this issue of Ekphrasis); (c) thereby makes it possible to reach a better understanding of the mediation of the ecological crisis in society, and, in the final instance, to better understand how to

1048

J. Bruhn et al. represent the crisis in communication in different media, and thus work against misunderstandings or plain, cynical disinformation. (Bruhn 2020b, 13)

Several of the papers given at the symposium, and subsequently published in the Romanian Ekphrasis volume, methodologically investigated transmediation issues (written by Elleström, Kato, Skiveren, Sternudd, Tornborg; see previous section), whereas other contributions moved towards the more comparative intermedial approach. These included Liviu Lutas’s comparison of feature film, animation film, and literature (Lutas 2020) and Alexa Weik von Mossner’s discussion of representations of extinction (Weik von Mossner 2020); Simon Estok’s more cultural and critical discussion took in several media types (Estok 2020). Even though these articles are quite divergent, taken as a whole, they theoretically and methodically nevertheless show that there is potential for a comparative intermedial approach. In subsequent work, Jørgen Bruhn has tried to streamline the research program mentioned above. In “Towards an Intermedial Ecocriticism” (Bruhn 2021), Bruhn compares and discusses two very different media products: an online popular science article by CarbonBrief and a Danish novel about climate change, and he suggests a three-step analytical methodology. Intermedial ecocritism, as a subcategory of ecocriticism, has also been described in a Swedish textbook anthology in which Jørgen Bruhn and Niklas Salmose offer a student-oriented methodological introduction followed by a discussion of a case study, referred to below (Bruhn and Salmose 2022). To exemplify the possibilities of an intermedial comparative approach to representations of the ecological crisis, we will paraphrase the example of that article.

Part 6 Case Study: Future Food Cultures Across Media Borders Intermedial ecocriticism, then, is a method for comparing different media products by focusing on a specific ecocritical theme. The theme of the following analysis is future food cultures, more specifically: how are ideas about future food cultures represented through different media products, and what is the potential of the media products for environmental agency directed towards a changed food culture? In January 2019, The Lancet published a call to action named Food in the Anthropocene: the EAT–Lancet Commission on healthy diets from sustainable food systems. According to the commission, one of the reasons for our climate crisis is the way we eat. The Lancet commission observes: [c]ivilisation is in crisis. We can no longer feed our population a healthy diet while balancing planetary resources [. . .]. The dominant diets that the world has been producing and eating for the past 50 years are no longer nutritionally optimal, are a major contributor to climate change, and are accelerating erosion of natural biodiversity. Unless there is a comprehensive shift in how the world eats, there is no likelihood of achieving the Sustainable Development Goals [. . .] or of meeting the Paris Agreement on climate change. (Lucas and Horton 2019, 386)

40

The Ecological Crisis and Intermedial Studies

1049

The goal of the commission is to help outline how this shift in “how the world eats” can come about. In its summary report it launches a very ambitious program aimed at making people eat in ways that are more sustainable both for their bodies and for their environment. This program is scientifically very well grounded and a necessary step towards addressing both poor diets and climate change. Part of the EAT–Lancet Commission’s dedicated and ambitious work, which includes spreading the word about the need for a changed food culture, is their website, Eatforum.org. The commission clearly realizes that scientific reports have a limited impact on potential changes and grassroots work. Eatforum.org contains popular science and targets a general audience interested in the connection between environmental questions and food, so its intentions are clear. Below, Eatforum.org, will be examined and then compared to another media product, namely Blade Runner 2049, directed by Denis Villeneuve in 2017. This post-apocalyptic film paints a picture of a sinister future world where borders between humans and non-humans, life and machines, are blurred. It belongs to the speculative climate fiction genre and has different goals and ambitions than the website. Despite its commercial and entertaining aspirations it also clearly intersects with ecological concerns such as protein production and synthetic agriculture. But despite huge differences in the medial form and the plausible audiences of such diverse media, we nevertheless argue that they can and indeed should be compared and that the tools for doing this are intermedial methods and terms.

Eatforum.org The media product Eatforum.org belongs to the media type website. It is a heteromedial media type, consisting of the basic media types of words, images, illustrations, animations, and moving images. It can be experienced through several technical media of display, but whichever is used, the experience is highly tactile. In terms of spatio-temporality, it is two-dimensional and consists of a non-linear temporality – even though many websites are designed to provide information in a particular order. The website is easy to navigate and offers an opportunity to explore it in different ways with different focus points: it is highly interactive. Its design is simple, but lively and full of contrasts and its layout feels spacious. But it is also unexpectedly complex as one starts to explore it, with many subpages and different and new categories. Eatforum.org is an informative and popular scientific website that aims to represent the basic scientific idea that a radical change in food culture and production is essential. Hence, the site has a clear and explicit goal which influences the layout and content, its eclectic mix of information, news, shorter press releases and longer scientific or popular scientific articles and reports, colorful images, diagrams, and a few short films and animations. The website is organized via five categories: “Learn & Discover,” “Knowledge,” “Initiatives,” “Events,” and “About EAT.” In “About EAT” (which is perhaps the first section some viewers will look at) we are presented with an image to the left of the page of green seedlings; their color is

1050

J. Bruhn et al.

vividly emphasized against the dark soil in which they have been planted. The color green is a common symbol for ecology, life, and health. The layout in Eatforum.org is mostly constructed of images and text alongside each other so that the text is complemented by visual imagery and vice versa. The text to the right of the image on the “About EAT” page is “Who we are,” and the use of the third-person pronoun evidences a sense of community and agency. This is further developed by the text beneath it, which reads, “EAT is a global, non-profit startup dedicated to transforming our global food system through sound science, impatient disruption and novel partnerships,” and in combination with the image of the seedlings, this suggests a grassroot, non-profit movement that engages in activity that is separate from that of established institutions, something that is new and exciting, immersive and inclusive. This is a good example of how image and text successfully complement each other across the entire website (Image 1). If we scroll down to beneath the picture of the seedlings just described, we will see the heading “People,” which is on the opposite side of the page to the “Who we are” heading, for contrast and variation in design. Underneath this heading, it says “Our people are our greatest strength,” which further cements the sense of crossing borders between different cultures, countries, and professions. The idea of “People” is completed by a photo to the right of this heading of a group of smiling EAT administrators, researchers, and other collaborators standing on a conference stage against a huge backdrop with a photo of London on it. The group photo looks up to date in a digital world, and the text above the photo is emphasized by the use of all capital letters: “WHAT WILL YOU DO?” The effect of all this is that we feel part of the EAT concept, even without knowing its exact contours. The interactive aspect is inclusive in itself and the choice to zoom in on EAT’s independence and freedom adds to this sense of community and participation. This is extended in the categories “Knowledge” and “Initiatives.” In “Knowledge” we are met with a saturated photo of a woman selling spices and vegetables at a market or bazaar. Taken from above, it signals objectivity, which

Image 1 “Who we are” on Eatforum.org. (CREDIT LINE: https://eatforum.org/about/. Courtesy of Eatforum.org)

40

The Ecological Crisis and Intermedial Studies

1051

connects to scientific overviews. The subtitle to the left of the photo emphasizes a global focus, which is suggested by the non-Western look of the photo: “Can we feed a future population of 10 billion people a healthy diet within planetary boundaries?” This can also be seen as a response to the often mentioned criticism of the EAT–Lancet report that it does not value food traditions and food culture, seeing food culture as being about feeding rather than eating. Here we can see the possibility of changing the food industry and still valuing food as tradition, ritual, and culture. If we click on “Read more” underneath the heading “The EAT-Lancet Commission on Food, Planet, Health,” we are given new choices: “The Science,” “The Planetary Health Diet,” and “The Commission.” “Science” allows us to take part of a popular scientific summary of the scientific report (available in nine languages). The photo above the “The Planetary Health Diet” heading is of an inviting pink veggie smoothie instead of the common image of meals made of insects. Clicking here reveals a future, sustainable dinner plate diagram. Further down the page, we can see the so-called Briefs, which focus on professions and more specific themes such as cities, farmers, food professionals, health care, administration, leadership, and people in general. This is an opportunity, then, to zoom in on specific interests, and it includes links to recorded lectures, hands-on advice, condensed scientific reports, videos, and podcasts.

Bladerunner 2049 Bladerunner 2049 belongs to the qualified media type film and its basic media are moving 2D images and sound that are structured in a decisive chronology (although, depending on the technical media of display, one can control this to some extent). Film as a media type obviously has different objectives and ambitions from a website; regarding the sensorial modality, even if both media types are audiovisual, the impact of film as a cinematic medium is to create an immersive emotional and sensorial experience. Alexa Weik von Mossner has convincingly shown how film has the ability to create embodied experiences which have the potential to affect our opinions on the climate and climate crisis in films within the cli-fi genre (2017, 3). Film as a cinematic medium is always 2D and always has a linear temporality (even though watching film on other technical media of displays can offer opportunities to alter this – which is what we call contextualized media specificity). The dinner scene in Blade Runner 2049 shows K., who is a “replicant” (a biotechnically constructed human), eating dinner in his apartment (Image 2). The setting outside the window is grey and gloomy; an eternal acid rain falls, only occasionally interrupted by sudden bursts of snow. Through the window we notice several similar apartment blocks. The diegetic music plays Frank Sinatra’s “Summer Wind,” creating a stark contrast with the dreary setting and the cyberpunk-like style of the film. The lyrics of the song create a nostalgic background that evokes an ephemeral summer and provides a contrast to the film’s eternal winter. It also alludes to the conservative American dream about life after World War II. This is further

1052

J. Bruhn et al.

Image 2 K’s artificial girlfriend Joi serves dinner in Blade Runner 2049. (CREDIT LINE: Screenshot, Bladerunner 2049, Dir. Denis Villeneuve, Warner Bros., 2017)

enhanced by K.’s artificial girlfriend, Joi, serving dinner in a 1950s waitress’s dress with a high-waisted skirt, white shirt, and pink apron. K.’s meal consists of a bowl of unidentifiable brown slush with noodles, but over this real food, an image is projected of a steak, fries, and a green salad. Food in this scene is problematized for the viewer in terms of future food cultures in an emotional and aestheticized way. The brown slush is feed, whereas the projected image is a mediation of food. Consider the difference between feeding and eating, as described by Claude Fischler: Food is central to our sense of identity. The way any given human group eats helps it assert its diversity, hierarchy and organization, and at the same time, both its oneness and the otherness of whoever eats differently. Food is also central to individual identity, in that any given human individual is constructed, biologically, psychologically and socially by the food he/she choses [sic.] to incorporate. (Fischler 1988, 275)

For the contemporary viewer, the dinner scene becomes a potent commentary on a contemporary food discourse which creates a time tunnel in the otherwise stark gothic and present (future) reality of the film. We realize that in the grey and dim post-apocalyptic future, food is feed, which is further stressed by the projection of a steak. The scene elucidates two different reactions. The connotations of 1950s food cultures become reminders of how the Anthropocene condition is ultimately the result of the culture represented in the scene: patriarchy, stereotyped gender roles, the sexualization and objectification of women, and industrially produced food. The second reaction concerns the brown goo being seen as a warning sign, a ticking clock, about the future of our food cultures unless we change our production and consumption patterns right now. The scene in Bladerunner 2049 displays a future scenario where food is feed but where the culture around it is still active and meaningful, even if artificially so. How food is essentially feed is clarified in an earlier scene in the film that takes place at a

40

The Ecological Crisis and Intermedial Studies

1053

protein farm. K. asks the farmer what is cooking in a pot and the reply is “garlic,” for private use. “Oh is that what smells” says K., who has never come across garlic. This stresses the discrepancy between consumer and producer in the commercialized food industry. Although the focus in this analysis is future food cultures, it is worth remembering that the representation of the food of the future is starkly contrasted with food that is represented with a nostalgic shimmer as part of a broader Anthropocene deconstruction; it is a swansong about lost humanity and culture. The film is packed full of cultural and material artefacts and media types that have lost their physical, intellectual, and emotional connection to humans: books, music, and various cultural practices. Just as the projected steak is nostalgic and emblematic of non-sustainable food, the deserted city of Las Vegas in the film creates a complex idea about a breaking point between utility on the one hand, and all that is destructive but also fantastic about humanity. The remains of past civilization and culture, no matter how destructive they were (gambling, drugs, prostitution, violence, fat-filled food) still appears as “human proper” and hence is longed for. In this former youthful and luxurious desert city, holograms of Presley and cancan dancers lag and remind us that the Anthropocene is more ambivalent than we might think. The film accentuates our deep longing for the authentic no matter what its ethical values are.

A Comparative Analysis To conclude, let us return to the thematic question for the comparative intermedial and ecocritical analysis: how are ideas about future food cultures represented and can we draw any preliminary conclusions from them? We have noticed both similarities and differences in how these two different media products communicate the topic. Eatforum.org has several affordances that are specific to the media type website: interaction, freedom of choice, and multimedial qualities. The website hence satisfactorily includes users and invites them to deepen their knowledge about a changed food culture as well as to become actively engaged in sustainable food activism and criticism on many different levels. Eatforum.org does not shy away from the truth about the climate crisis but in the end provides a hopeful view on the possibility of change if we engage in these questions globally, collectively, and in solidarity. The website is rather uncritical of some more complex issues around future food cultures; for example, it does not provide a clear image of what future food will actually look like and it does not problematize the relations between culture, tradition, identity, and food. Regarding these latter concerns, the affordances of climate fiction (novels, poetry, and podcasts) and audiovisual media types, including cinema, have a huge advantage in that they build advanced multisensory worlds. Eatforum.org, however, has a significant impact on creating agency and activism in users through its interactive possibilities (including the user being able to act at their own pace and in relation to their own interests), knowledge gathering, simple audiovisual experiences, and inspiration regarding owning initiatives on different levels. The scientific backdrop displays pragmatic options for turning our climate in

1054

J. Bruhn et al.

a more positive direction, which infuses a strong sense of hope and community. Eatforum.org is inclusive of people from very different backgrounds, but, as stated earlier, it builds on a specific scientific thesis rather than on culture and tradition. This risks it becoming overtly didactic and only speaking to those already involved in these debates. Blade Runner 2049 offers a different, more critical, multifaceted, and open entry to questions on sustainable future food cultures by creating a truly emotional experience through the film’s audiovisual design and the powerful, immersive qualified Hollywood narrative spectacle. As Gregers Andersen has shown, climate fiction has the ability to represent hope and fear about the climate crisis in a deeper and more emphatic way since climate fiction tends to investigate what daily life might be like in a future scenario (Andersen 2020). Fiction, as empirical ecocriticism has shown, produces identification that can lead to cognitive effects and affects and that can even lead to changed personal behavior. On the other hand, these experiences are singular and individual and do not bring a sense of collectiveness to them (which the website does). These emotions, though, can be created in post-film debates, in dialogues and in teaching situations where the film would be discussed. The speculative nature of the future vision built up in Bladerunner 2049 is in stark contrast to the more rational, balanced, and explicitly science-based communication in Eatforum.org. Obviously, fiction gives the reader certain freedoms, but the scene we have focused on is in no way alien to a potentially close future reality. Meaningful elements in the film, such as artificial intelligence, avatars, digital manipulations, and protein farms, are all more or less realities in 2021. Bladerunner 2049 auspiciously places the theme of future food cultures in a broad discourse concerning what is human and authentic. Hence, the film involves a larger network of philosophical and existential ideas than Eatforum.org. To sum up, the film forces the viewer to relate to how food specifically can be contextualized in a future perspective through its audiovisual attraction and immersion, but it cannot channel these emotions into a concrete and explicit engagement and agency. Thus, the experience gained from watching Bladerunner 2049 can be seen as superficial and hence as not having any particular impact on how we deal with and think about food in the climate crisis. Empirical research has unfortunately shown that even if engagement with environmental questions increases after an experience of climate fiction, these engagements are only temporary (Małecki et al. 2018; Schneider-Mayerson 2018). Maybe the reality is that we need to think of these different media types as complementary to each other in order to create a more holistic vision of the role of different media in understanding and even battling the climate crisis.

Conclusion In this chapter, we have pointed to some of the relations between intermedial studies and ecocritical questions, interrelations that have only in recent years been developed as a possible research agenda. After the introduction, where we presented the

40

The Ecological Crisis and Intermedial Studies

1055

research problem we wanted to discuss, we offered a literature review of important contributions to academic fields that communicate the ecological crisis and provided a summary of the tradition of ecocriticism. Connected to that overview we gave a brief sketch of the intermedial position that we find is most relevant in this context. After that, we mentioned and discussed the attempts to connect ecocritical questions to transmediation, followed by a more elaborate and explicit attempt to formulate what we call an intermedial ecocriticism. To exemplify the method of intermedial ecocriticism, a case study comparing how a website and a sci-fi film scene communicate ideas of a future food culture was supplied. Based on this we conclude that intermedial studies and ecocritical themes share important potential contact points: we are convinced that the highly topical and timely societal questions raised in ecocriticism can benefit from an intermedial perspective. Our own suggested attempt to combine intermedial studies and ecocritical issues we call Intermedial ecocriticism: this newly launched research paradigm considers both transmediations of scientific results to wider audiences – and the usefulness to pursue systematic and critical comparisons of different representations related to the ecological crisis not only in isolated media types, but across media borders. Given its short history intermedial ecocriticism needs to be theoretically developed as well as methodologically enhanced. The intermedial ecocritical analytical model could, for instance, be tested on media types that have not yet been considered in communication research: when reviewing the research that more or less explicitly deals with intermedial aspects of ecocritical questions, a large number of media types have been analyzed including different genres of poetry and narrative literature, essays, different cinematic genres, games, art, music, popular science in different technical media of display. In the intermedial research environment in Växjö that we are part of, ideas of how to widen the scope of intermedial approaches to ecocritical questions have been ventilated for the past few years. As an example, artist and cultural theorist Ola Ståhl, has investigated art exhibitions (and not only individual art works or performances): the art, or art-science exhibition, could be seen as a specific media type with its own affordances, and here for instance Bruno Latour and Peter Weibel’s so-called “thought-exhibitions” Reset Modernity and Critical Zones would be valuable media types for scrutinization. Another example – and this is something that social scientists Henrik Eneroth och Malin Henriksson have considered – would be to take the highly complex UN COP-meetings under intermedial ecocritical scrutiny. Further, descending one or two steps in physical size and in cultural capital, but not necessarily in social importance, it would be essential to consider some of the social media practices that relate to ecological issues: art historian Hans T. Sternudd, for instance, has been interested in relating the media affordances of YouTube-videos to the representation among young people suffering from climate anxiety, thus bridging eco-questions in social media with intermedial theory. The subject matter of ecological emergency is, unfortunately, more or less infinite: possibly intermedial studies may offer ways to better understand and, in the broadest sense of the word, cope with it.

1056

J. Bruhn et al.

References Adamson, Joni, and Michael Davis, eds. 2017. Humanities for the environment: Integrating knowledges, forging new constellations of practice. London: Routledge. Alaimo, Stacy. 2010. Bodily natures: Science, environment, and the material self. Bloomington: Indiana University Press. Andersen, Gregers. 2020. Climate fiction and cultural analysis: A new perspective on life in the anthropocene. New York: Routledge. Ballard, Susan. 2021. Art and nature in the Anthropocene: Planetary aesthetics. New York: Routledge. Barry, Peter, and William Welstead. 2017. Extending ecocriticism: Crisis, collaboration and challenges in the environmental humanities. Manchester: Manchester University Press. Bennett, Jane, et al. 2010. New materialisms: Ontology, agency, and politics. Durham, London: Duke University Press. Bladow, Kyle, and Jennifer Ladino. 2018. Affective ecocriticism: emotion, embodiment, environment. Lincoln: University of Nebraska Press. Bonneuil, Christophe, and Jean-Baptiste Fressoz. 2016. The shock of the anthropocene: The earth, history and us. London: Verso. Boyum, Joy Gould. 1985. Double exposure: Fiction into film. New York: New American Library. Bruhn, Jørgen. 2010. Heteromediality. In Media borders, multimodality and intermediality, ed. Lars Elleström, 225–236. Basingstoke: Springer. ———. 2020a. “We’re Doomed–Now What?”: Transmediating temporality into narrative forms. In Transmediations. Communication across media Borders, ed. Niklas Salmose and Lars Elleström, 217–234. New York: Routledge. ———. 2020b. The Anthropocene ecological crisis across media and the arts. Ekphrasis 24 (2): 5–18. ———. 2021. Towards an intermedial ecocriticism. In Beyond media borders: Intermedial relations among multimodal media, ed. Lars Elleström, 117–148. Cham: Palgrave Macmillan. Bruhn, Jørgen, and Liviu Lutas, eds. 2020. The Anthropocene and intermedial ecocriticism. Ekphrasis, 24 (2). Bruhn, Jørgen, and Niklas Salmose. 2022. Intermedial ekokritik. In Ekokritiska metoder, ed. Camilla Brudin Borg, Jørgen Bruhn, and Rikard Wingård. Lund: Studentlitteratur. ———. 2023. Intermedial ecocriticism. The climate crisis through art and media. Lanham: Lexington Books. Bruhn, Jørgen, and Beate Schirrmacher. 2022. Intermedial studies: An introduction to meaning across media. Abingdon, Oxon: Routledge. Bruhn, Jørgen, Niklas Salmose, Beate Schirrmacher, and Emma Tornborg. 2022. Truthfulness and truth claims as transmedial phenomena. In Intermedial studies: An introduction to meaning across media, 225–254. New York: Routledge. Buell, Lawrence. 2005. The future of environmental criticism: Environmental crisis and literary imagination. Malden: Blackwell. Caracciolo, Marco. 2021. Narrating the mesh: Form and story in the Anthropocene. Charlottesville: University of Virginia Press. Carson, Rachel. 1962. Silent spring. Boston: Houghton Mifflin Co. Christensen, Jon, Ursula K. Heise, and Michelle Niemann. 2017. The Routledge companion to the environmental humanities. New York: Routledge. Clark, Timothy. 2015. Ecocriticism on the edge: The Anthropocene as a threshold concept. London: Bloomsbury. Crutzen, Paul, and Eugene Stoermer. 2000. The anthropocene. Global change newsletter 41: 17–18. Corrigan, Timothy. 2011. Film and literature: An introduction and reader. London: Routledge. Davis, Heather, and Etienne Turpin. 2015. Art in the Anthropocene: Encounters among aesthetics, politics, environments and epistemologies. London: Open Humanities Press.

40

The Ecological Crisis and Intermedial Studies

1057

Demos, T.J. 2017. Against the Anthropocene: Visual culture and environment today. Berlin: Sternberg Press. Doyle, Julie. 2011. Mediating climate change. Farnham: Ashgate. Eatforum.org. Website. https://eatforum.org/. Accessed 20220124. Eliassen, Knut Ove, Anders Skare Malvik, Kim Menage, and John Peters. 2017. “Det finnes mening i naturen” – medienes naturhistorie. Journal for metafysisk spekulasjon 2–3: 290–311. Elleström, Lars. 2010. Media borders, multimodality and intermediality. Springer. ———, Lars. 2014. Media transformation: The transfer of media characteristics among media. Basingstoke: Palgrave Macmillan. ———. 2018. Coherence and truthfulness in communication: Intracommunicational and extracommunicational indexicality. Semiotica 225: 423–446. ———. 2020. Representing the Anthropocene: Transmediation of narratives and truthfulness from science to feature film. Ekphrasis 24 (2): 36–48. https://doi.org/10.24193/ekphrasis.24.3. ———. 2021. Beyond media borders, Volume 1 [electronic resource] intermedial relations among multimodal media. Cham: Springer International. Elliott, Kamilla. 2020. Theorizing adaptation. New York: Oxford University Press. Ellis, Erle C. 2018. Anthropocene: A very short introduction. Oxford: Oxford University Press. Emmerich, Roland, dir. 2004. The day after tomorrow. 20th Century Fox. Estok, Simon C. 2020. Intermedial apocalypticism and the growing Anthropocene crises. Ekphrasis 24 (2): 208–224. Fischler, Claude. 1988. Food, self and identity. Social Science Information 27 (2): 275–292. Goniorava, Matthew. 2019. Bleak joys: Aesthetics of ecology and impossibility. London: University of Minnesota Press. Gaard, Greta. 1997. Toward a queer ecofeminism. Hypatia 12 (1): 114–137. Garrard, Greg. 2012. Ecocriticism. London: Routledge. Ghosh, Amitav. 2016. The great derangement: Climate change and the unthinkable. London: The University of Chicago Press. Glotfelty, Cheryll, and Harold Fromm. 1996. The ecocriticism reader: Landmarks in literary ecology. Athens: University of Georgia Press. Grusin, Richard. 2017. Anthropocene feminism. Minnesota: University of Minnesota Press. Haraway, Donna Jeanne. 2016. Staying with the trouble: Making kin in the Chthulucene. Durham: Duke University Press. Hutcheon, Linda. 2006. A theory of adaptation. New York: Routledge. Iovino, Serenella, and Serpil Oppermann. 2014. Material ecocriticism. Indiana University Press. Jacquet, Luc, dir. 2015. Ice and the sky. Paris: Pathé Distributions. Kaplan, E. Ann. 2015. Climate trauma: Foreseeing the future in dystopian film and fiction. New Brunswick: Rutgers University Press. Latour, Bruno, Steve Woolgar, and Jonas Salk. 2013. Laboratory life: The construction of scientific facts. Princeton: Princeton University Press. Leitch, Thomas. 2003. Twelve fallacies in contemporary adaptation theory. Criticism 45 (2): 149–171. ———. 2007. Film adaptation and its discontents: From gone with the wind to the passion of the Christ. Baltimore: Johns Hopkins University Press. Lucas, Tamara, and Richard Horton. 2019. The 21st-century great food transformation. The Lancet 393 (10170). Lutas, Liviu. 2020. Friends or foes–representations of the relationship between man and nature in three different media types: Literature, live action film and animated film. Ekphrasis 24 (2): 243–262. Małecki, Wojciech. 2019. Experimental ecocriticism, or how to know if literature really works. In Routledge handbook of ecocriticism and environmental communication, 211–223. New York: Routledge.

1058

J. Bruhn et al.

Małecki, Wojciech, Bogusław Pawłowski, Marcin Cieński, and Piotr Sorokowski. 2018. Can fiction make us kinder to other species? The impact of fiction on pro-animal attitudes and behavior. Poetics 66: 54–63. https://doi.org/10.1016/j.poetic.2018.02.004. Malhi, Yadvinder. 2017. The concept of the Anthropocene. Annual Review of Environment and Resources 42 (1): 77–104. https://doi.org/10.1146/annurev-environ-102016-060854. McDougal, Stuart Y. 1985. Made into movies: From literature to film. New York: Holt, Rinehart, and Winston. Moore, Jason W. 2015. Capitalism in the web of life: Ecology and the accumulation of capital. New York: Verso. ———. 2016. Anthropocene or capitalocene?: Nature, history, and the crisis of capitalism. Oakland: PM Press. Mortimer-Sandilands, Catriona, and Bruce Erickson. 2010. Queer ecologies: Sex, nature, politics, desire. Bloomington: Indiana University Press. Morton, Timothy. 2007. Ecology without nature: Rethinking environmental aesthetics. Cambridge, MA: Harvard University Press. ———. 2010. The ecological thought. Cambridge, MA: Harvard University Press. ———. 2013. Hyperobjects: Philosophy and ecology after the end of the world. Minneapolis: University of Minnesota Press. ———. 2021. All art is ecological. Dublin: Penguin Books. Murray, Robin L., and Joseph K. Heumann. 2009. WALL-E: from environmental adaptation to sentimental nostalgia. Jump Cut: A Review of Contemporary Media 51: 1–13. ———. 2014. Film and everyday eco-disasters. Lincoln: University of Nebraska Press. Nixon, Rob. 2011. Slow violence and the environmentalism of the poor. Cambridge, MA: Harvard University Press. Orlowski, Jeff, dir. 2012. Chasing ice. New York: Submarine Delux. Peters, John Durham. 2015. The marvelous clouds: Toward a philosophy of elemental media. Chicago: University of Chicago Press. Plumwood, Val. 1993. Feminism and the mastery of nature. New York: Routledge. Rajewsky, Irina O. 2002. Intermedialität. Tübingen: Francke. Rossholm, Anna-Sofia. 2018. Moving mountains: Cinema, deep time and climate change in swedish artist Hanna Ljungh’s I am mountain, to measure impermanence. In Nordic narratives of nature and the environment: Ecocritical approaches to Northern European literatures and cultures, ed. Peter Degerman, Anna-Karin Jonasson, and Hennig Reinhard, 39–48. Lanham: Lexington Books. Roundtree, Aimee Kendall. 2014. Computer simulation, rhetoric, and the scientific imagination: How virtual evidence shapes science in the making and in the news. Lanham: Lexington Books. Rueckert, William. 1978. Literature and ecology: Experiment in ecocriticism. The Iowa Review 9 (1): 71–86. Rust, Stephen, Salma Monani, and Sean Cubitt. 2016. Ecomedia: Key issues. New York: Routledge. Salmose, Niklas. 2018. The apocalyptic sublime: Anthropocene representation and environmental agency in Hollywood action-adventure cli-fi films. The Journal of Popular Culture 51 (6): 1415–1433. ———. 2019. Behemoth, nostalgia and ecological agency. In Futures worth preserving: Cultural constructions of nostalgia and sustainability, ed. Andressa Schröder, Nico Völker, Robert A. Winkler, and Tom Clucas, 239–256. Bielefeld: Transcript Verlag. ———. 2020. Three transmediations of the anthropocene: An intermedial ecocritical reading of facts, Sci-Fi, PopSci and eco-horror. In Transmediations: Communication across media Borders, ed. Niklas Salmose and Lars Elleström, 254–273. New York: Routledge. Salmose, Niklas, and Lars Elleström. 2020. Transmediations. Communication across media borders. New York: Routledge. Schneider-Mayerson, Matthew. 2018. The influence of climate fiction: An empirical survey of readers. Environmental Humanities 10 (2): 473–500. https://doi.org/10.1215/220119197156848.

40

The Ecological Crisis and Intermedial Studies

1059

Schneider-Mayerson, Matthew, Alexa Weik von Mossner, and W.P. Małecki. 2020. Empirical ecocriticism: Environmental texts and empirical methods. ISLE Interdisciplinary Studies in Literature and Environment 27 (2): 327–336. https://doi.org/10.1093/isle/isaa022. Sismondo, Sergio. 2010. An introduction to science and technology studies. Chichester: WileyBlackwell. Slovic, Scott, et al. 2019. Routledge handbook of ecocriticism and environmental communication. New York: Routledge. Suhor, Charles. 1984. Towards a semiotics-based curriculum. Journal of Curriculum Studies 16 (3): 247–257. Svoboda, Michael. 2014. A review of climate fiction (cli-fi) cinema. . . past and present. Yale Climate Connections: 1–20. Tornborg, Emma. 2020. Transmediations of the Anthropocene: From factual media to poetry. In Transmediations. Communication across media Borders, ed. Niklas Salmose and Lars Elleström, 235–253. New York: Routledge. Trexler, Adam. 2015. Anthropocene fictions: The novel in a time of climate change. Charlottesville: University of Virginia Press. Villeneuve, Denis, dir. 2017. Bladerunner 2049. Warner Brothers. Weik von Mossner, Alexa. 2017. Affective ecologies: Empathy, emotion, and environmental narrative. Columbus: The Ohio State University Press. ———. 2020. Larger than life: Endangered species across media in Louis Psihoyos’s racing extinction. Ekphrasis 24 (2): 19–35. Wolf, Werner. 1999. The musicalization of fiction: A study in the theory and history of intermediality. Amsterdam: Rodopi. Yale Program on Climate Change Communication. n.d. What is climate change communication? https://climatecommunication.yale.edu/about/what-is-climate-change-communication/. Accessed 220125. Zapf, Hubert. 2016. Handbook of ecocriticism and cultural ecology. Berlin: De Gruyter. ———. 2021. Cultural ecology, the environmental humanities, and the ecological archives of literature. Colloquia Germanica 53 (2–3): 121–139.

Simulated Climate in Ecological Games: Mediating Climate Change to Endow Players with Transformative Agency

41

Pe´ter Kristo´f Makai

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Games as Thought Experiments in Climate Change Communication . . . . . . . . . . . . . . . . . . . . . . . . True-Life Adventures: How Truthfulness and Truth Claims Are Mediated in Gaming . . . . . . How Board Games Signify: Interaction Between Mechanics and Theme . . . . . . . . . . . . . . . . . . . . God Games and Grand Strategy: An Immortal Bird’s-Eye View of Climate Change . . . . . . . . “Is This a Random World, or Did You Planet?”: SimEarth’s Designer Climate System . . . . . It Never Rains But It Pours: Sid Meier’s Civilization as an Environmental History Primer . . . Negotiation Is a Two-Way Street in Climate Change Board Games . . . . . . . . . . . . . . . . . . . . . . . . . . Development as a Highway to Hell in Tipping Point and CO2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Coda: Fixing Games – The First Step to Fixing the Climate? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1062 1063 1066 1070 1071 1072 1075 1079 1081 1083 1084

Abstract

The chapter introduces the reader to the challenges of using games as media for communicating climate change and to ecocritical work done on climate change games. It begins by summarizing the state of the wider field of media studies on ecological media and highlights the usefulness of games as thought experiments that promote systems thinking. The chapter then proceeds to give an intermedial account of how scientifically produced knowledge is transposed from the realm of fact to that of fiction, specifically, on how truth claims migrate to algorithmic environments. Understanding video games as tools for representing complexity, it then focuses on two classes of digital games, so-called god games and grand strategy games, and uses SimEarth and Sid Meier’s Civilization VI: Gathering Storm as case studies for discussing how game designers can use the medial

P. K. Makai (*) Rachel Carson Center for Environment and Society, München, Germany, and Kazimierz Wielki University, Bydgoszcz, Poland e-mail: [email protected]; [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_51

1061

1062

P. K. Makai

properties of video games to simulate human agency’s effects on the climate system. Additionally, the chapter demonstrates the capabilities of modern hobby board gaming to provide a different perspective on the same representational problem, using mechanics more congenial to board gaming, such as negotiation. Using Kyoto, Keep Cool, Tipping Point, and CO2: Second Chance as illustrations, the chapter establishes ludo-textual analysis and ludic discourse analysis as key scholarly methods for investigating how scientific knowledge is transmitted in games but also for exposing the ideological assumptions behind various game mechanics. Keywords

Climate change · Video games · Board games · Ludic narratives · Artistic science communication

Introduction Planet Earth is a complex system of interwoven physical, chemical, biological, and sociocultural processes, from the workings of the geosphere to that of the biosphere, from industrialization to environmental regulation. Climate science is dedicated to mediating observations of climate phenomena to model and predict future climate trends. This mediated information, however, does not often make an impact on the lay public, partially due to the counterintuitive nature of how Earth systems work. As entertaining working simulations, computer games are uniquely poised to present the mechanics of earth science to the public. Intermedial research can indicate how truthfully coded discourses about the climate transfer to the realm of fiction and how ludic simulations straddle the representational boundaries of fact and fiction when it comes to future climate scenarios. This chapter surveys ecocritical game studies and the problems of simulating climate systems in digital and analogue games. The chapter also indicates how the principles of climate science are mediated as information via systems design, noting their effect on gameplay and player agency. However, we must be aware that “digital climate models simulate environmental scenarios as indistinguishable from social scenarios” (Konior 2020: 58), which compels us to consider that science mediation also entails the mediation of the social sphere and social negotiations in climate change games. The chapter summarizes and details ecocritical work done in games studies, with an eye on game analytical approaches. In particular, ludo-textual analysis and ludic discourse analysis form the methodological kernel of this chapter. On the landscape of digital games, the discussion extends to two genres of life simulations: (a) “god games” that simulate long-term changes to the climate of their gameworlds and (b) grand strategy games, which portray more immediate threats to human civilizations. On the analogue front, the chapter deals exclusively with modern hobby board games, arising in the late twentieth and early twenty-first centuries in the wake of the German board gaming renaissance across the globe.

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1063

The goal of this survey is to demonstrate the remarkable flexibility of game designers in their repertoire of representations and their choice of game mechanics to meet the climate crisis head-on but without losing a sense of dark humor that pervades climate discourse today. It also serves as a contribution to the program of intermedial ecocriticism, defined by Bruhn as a research agenda that: (a) facilitates a broad, theoretically and methodologically solid approach to the very wide field of representations of the environmental crisis in many different media types; (b) enables comparisons across different media types, with both form and content issues including but not limited to narrativity, scientific truth claims, [etc.]; (c) thereby makes it possible to reach a better understanding of the mediation of the ecological crisis in society, and [. . .] to better understand how to represent the crisis in communication in different media [. . .]. (2020: 13)

The video game case studies are selected for analysis in the understanding that they are interactive digital narratives which represent complex topics (Koenitz et al. 2020). With regard to board games, the chapter follows in the footsteps of games scholars Stewart Woods (2012) and Paul Booth (2021), who treat the board game as a qualified medium of its own right, with a shared language of expressive mechanics. This work also draws on intermedial theories of truthfulness (Elleström 2018, 2020) to indicate how discourses of truthfulness and truth claims migrate from the realm of the natural sciences to the simulated worlds of video games. Such an intermedial perspective is essential, because “media are always involved in transmedial processes in a dynamic, reciprocal and complex network” (Tornborg 2019: 132).

Games as Thought Experiments in Climate Change Communication Make no mistake: despite the ongoing coronavirus pandemic, the horrors of an imperialist Russia reawakening the spirit of Cold War enmities with its attack of Ukraine, and the continuing ravages of economic opportunism, the greatest threat humankind faces is still global warming and its attendant climate change. As of early 2022, the world has been given a taste of how to cooperate on the global scale, including the punishment of rogue states, but political and economic action of a heretofore unseen magnitude must be taken to ensure the continued existence of humankind. Still, it is not the lack of scientific consensus that is hindering us: we know what we should be doing, yet we continue to do precious little. As Bushell and their colleagues note, a key solution is the development of a unifying, strategic narrative that gives meaning to the problem of climate change and inspires action (2017). Arnold also indicates that narratives serve an essential role as an analytic device in making sense of and adapting to climate change, since narratives are tools for structuring complex information and social organization (2018: 57–82). The recent emergence of climate change sci-fi, called cli-fi, created textual narratives and films (Kaplan 2015) that illustrate the complexities of the

1064

P. K. Makai

Anthropocene through traditional representational means: throwing fictional characters into a storyworld affected by a changing climate (Andersen 2019; Milner and Burgman 2017). These narratives are protean in their incorporation of other genre elements (as readings in Goodbody and Johns-Putra 2019 attest), which might make them more engaging for various audiences. In spite of that, they all share a core trait: their narratives are set in stone, and whatever agency the characters exhibit is circumscribed by the author. In a survey of several cli-fi novels, Mertens and Craps hypothesize that “it might actually be possible to write ‘the great climate change novel.’ Or perhaps it will not be a novel: [. . .] other or hybrid art forms might be the way forward” (2018: 151). This chapter demonstrates that interactive ludic narratives have several advantages over other traditional textual works in simulating the complexities of climate change. Digital environments have several essential properties by the virtue of the fact that they run on a computer: they handle vast amounts of data (i.e., they are encyclopedic) that they display spatial relationships (spatiality); the digital programs work algorithmically to simulate cause and effect between their elements (thus, its behavior is procedural), and they require nontrivial actions and decision-making from a human interactor to alter the state of the program (allowing active participation). Thereby they provide certain aesthetic effects, namely, the perceptual and cognitive immersion of the interactor within the storyworld, wherein they may exercise their agency to influence the outcome of the narrative, which ideally results in a mental and emotional transformation of the interactors, who change their attitudes and promote prosocial actions in the outside world (Murray 2017: 123–230). Because the climate is a vast phenomenon distributed across the globe, affected by a myriad of environmental factors, working as a chaotic system into which humankind is immersed as agents who have an outsized impact on it, and climate change transforms their whole lives, ludic media are in an ideal position to use simulated microcosms to highlight the workings of the climate system and defamiliarize our conventional perceptions of human agency. On the other hand, in issuing a call for an ecological ethical examination of board games, Paul Booth has stated the once-true consensus among gamers that “No one wants to be reminded of climate change when they’re sitting down to fight some orcs or take over the galaxy” (2021: 218). Recent years of game development have put paid to that assumption, as there has been a tendency in the new millennium to produce board games that “can help teach people about the dangers of climate change” (2021: 222). Unlike computer games with vast processing power, board games rely on humans to do all computation (Bellomy 2017), and their strategies for representing the game state are much more modest: one tabletop must be enough to contain all relevant information. Nonetheless, their materiality remains one of their main draws, alongside social interaction (Rogerson et al. 2016). As such, board games under survey here are much more likely to feature some form of direct or indirect competition between human players, versus the digital games’ tendency to

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1065

represent conflict as occurring between the solitary player and AI “opponents” or the game system itself. The mediality of board games, therefore, make them more suitable to represent dramas of agency, such as the antagonisms inherent in climate policymaking or the competing demands of different geopolitical actors in energy transition. Contrariwise, they simplify the complexities of climate to a far greater degree, which throws the effects it does simulate into sharp relief. It should be noted, though, that despite their differences, analogue and digital games use many of the same resources to make climate change a felt and integral part of their play experience. Rules of the game translate how the microcosmic system of the game changes states and how players may affect them through their actions; such rules are lawful even in analogue games. These mechanics and the emergent behaviors they elicit are the main “arguments” the game (or, rather, its designer) makes in favor of how they envision the human-climate interaction to occur, which the players have to adopt for felicitous play. This opens the door for tacit learning and for shifts in consciousness to take place. In spite of this, as Abraham points out, the “persuasion through mechanics approach faces significant barriers when dealing with highly contentious (even ideological) issues” such as climate change, since if interactors are “ideologically predisposed to reject belief in the phenomenon, then it is hard to see how they could be convinced by any number of simulations, no matter how great their degree of fidelity to the real world” (2015: 74, 83). While one can certainly remain resistant to the designed models of individual games, it is much more likely that players have simply not experienced a working model of the climate, and consequently have not thought of the cascading effects of human action. Therefore, this chapter adopts a view that conceptualizes climate games as “executable thought experiments,” which may in fact prove to be superior because they add elements missing from philosophical thought experiments, such as probability calculations, whole-world simulations with more contextual factors, thus eliminating the bias of abstraction, and players meet these scenarios as “interested actors, rather than as disinterested spectators” (Schulzke 2014: 259–260). Together, these facets can expose the simulation’s faulty premises and implementational defects, which player feedback and further updates or editions can correct. A fruitful strategy to explain how playing climate change games impacts their players is to harness the power of board and computer games as objects-to-thinkwith (Holbert and Wilensky 2019) by interrogating the process of model-building, situating the knowledge they produce in their cultural contexts, and challenging their assumptions. Environmental or ecocriticism is the traditional approach in the humanities for doing so, and as Alenda Chang has recently written, “game studies and environmental criticism have had little congress to date,” but “we will be amply repaid if we move past the assumption that the natural and the digital are realms inherently inimical to each other” (Chang 2019: 10). The goal of the rest of the chapter is to survey the field for evidence that the scholarly community has already done so.

1066

P. K. Makai

True-Life Adventures: How Truthfulness and Truth Claims Are Mediated in Gaming Scientific ideas have long fascinated the audiences of popular culture, and scientific concepts regularly percolate down to popular media products (Buckland 2013; Johns-Putra 2019). Modernity served as a springboard for a new vocabulary and mode of articulation for living in a culture steeped in science, giving birth to a new genre of its own: science fiction. Sometimes called “the other side of realism” (Clareson 1971), science fiction served as an important vehicle for making sense of scientific advancement and bringing the future closer to the common people of the twentieth century. Creators of fiction rely on a whole host of assumptions and expectations shared across human communities when they are at work. Things begin to get more complicated when contentious issues or scientific controversies become the subject of popular culture, especially in a discourse context that is adverse to evidence and reasoning, as is the case with the media landscape today. Climate change communication is rising to the task of understanding who, when, why, and how are resistant to the idea of accepting climate change as real, a problem in need of a global solution, and how to transform their attitudes (Arnold 2018; Ballew et al. 2019; Ranney and Clark 2016). Art – understood in the broadest sense of human, mediated aesthetic expression that engages an audience’s mind-body psychologically and viscerally, often (but not exclusively) using the fictional mode of addressing them – appears to solve several of the big problems that purely information-based communication might have in breaking down the barriers of imaginative resistance. The “deficit-based model” of science communication holds that, when people are given the right facts, they should perceive them as sensible additions to their views of the world (Suldovsky 2017); and while the approach has its merits (Simis et al. 2016), there has been a bifurcation of worldviews on scientific issues that divide the public into people who trust in science and people who do not (Rughiniş and Flaherty 2022), which cannot be explained by a simple lack of knowledge. Storytelling, in particular, seems to punch through the defenses of skeptics and to foster agency (Veland et al. 2018). As the leading medium of artistic depictions of climate change, climate fiction excels in reaching out to moderates and to “effectively nudge” them “and remind concerned [readers] of the severity and urgency of anthropogenic climate change” (SchneiderMayerson 2018). Still, even in research that investigates artistic media as a vehicle of climate change communication (Andersen 2019; Doyle 2011; Bentz 2020; Kaplan 2015; Nurmis 2016), scholars tend to focus on the traditional arts, like literature, film, and visual art – that is to say, qualified media with a fixed representation – to the detriment of interactive media. And yet, digital simulations of climate and interactive ludic narratives (in our case, board and computer games) provide working models of Earth systems, albeit of varying accuracy. Even today, scientific climate models of various stripes from the 1970s up

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1067

until 2013 have been able to accurately predict changes in global temperatures, showing “outcomes reasonably close to what has actually occurred” (Hausfather 2017). Nonetheless, a certain skepticism is in order. As Bruhn indicated, “these models-turnedsimulations are undoubtedly the best possible ways of producing scientific predictions about the future, but commentators have noted that simulations are characterized by a certain ‘fictionality’ and ‘virtual indexicality’” (2020, citing Pias 2008: 108–115). Clearly, climate simulations are man-made constructions with inevitable human biases, and the kinds of uncertainty that scientists must always account for in their modelling do not usually translate well to headlines; simulations, then, are best understood as predictive models that work within a certain confidence range that can also be utilized to educate people and to galvanize them into action – playful simulations known as “serious games” can familiarize people to the contingencies of weather and climate and allow players to gain insight into how uncertainty must be factored into decision-making (van Pelt et al. 2015). There is even evidence that such simulations, like World Climate, transcend political barriers, and people of opposing political persuasions can be convinced by a sufficiently authentic climate simulation game (Rooney-Varga et al. 2018). Games both digital and analogue must be informed by truthful discourses of climate science in order for players to accept them as authentic arenas for meaningful player action. The algorithmic nature of tabletop and computer gameplay, with a heavy focus on rules that determine the consequences of player actions, are natural allies to the rule-bound discourse of science. Truthfulness, therefore, is essential for a designer to create believable and impactful gameplay. In this chapter, I treat truthfulness as a scalar concept of how representational systems model propositional statements about complex systems of the Earth onto the simulated microcosm of a game. Elleström identifies indexical references as central to claims of truthfulness. He observes that media with truth claims “might be assumed or even required to be truthful to objects that are both material and mental” (2020: 39), depending on their domain. Nonfiction works are not the sole province of truthfulness, because “narration and truthfulness exist [. . .] both in science and in art and entertainment [. . .]. Therefore, to some extent, narration and truthfulness can be transmediated among the different media types” (2020: 40). Tornborg (2019) instructively distinguishes between objective and subjective truthfulness. Objective truthfulness maps onto the rule-based system of games, which is equally binding to all interactors, whereas subjective truthfulness is an aesthetic impression on the individual interactor that tracks how well the simulation matches up with their expectations of the extent to which the simulated target domain corresponds to real-world scenarios of the thing simulated. She notes that “the transmediation of scientific or factual media characteristics generally has two functions: it affects both how we perceive the [artwork] aesthetically and how we regard its claims to truthfulness” (2020: 135). Here, transmediation is a term that describes the process of mapping and adapting content and form from one medium to that of another. This process must contend with the medial characteristics of each form of media. For example, a scientific paper

1068

P. K. Makai

uses a particular kind of argumentative prose, graphs, charts, and bibliographic entries to demonstrate practical or theoretical principles, whereas a computer game uses algorithmic rules, a changing database, and runtime environment to model a virtual gameworld, an audiovisual system for displaying that simulated world, and an input system which enables the player to affect it. In contrast, board games use printed booklets, game boards, and cards to simulate the gameworld, and human players calculate the effects of their actions using the descriptions they get in a printed rule book. In fact, Booth argues that “board games allow scholars to open up media studies from the screen and focus instead on the activity of mediation, regardless of medium” (2021: 9). For the purposes of this chapter, understanding medial changes means a transmedial analysis, which is always medium-specific. In the case of climate games, the chapter investigates the one-way transmission of real-world knowledge from the source domain of scientific models and reports to the two target domains of analogue and digital games. It also highlights instances where the two ludic traditions have influenced each other’s design. Although materially quite different, both playful media share an important distinction from traditional forms of art: the constitutive use of player agency to influence the game state. Human action’s imbrication into the events of a simulated world is crucial to drive home the point that humankind is collectively responsible for both causing and counteracting the effects of climate change. On the truthfulness of this statement, and its implications, hangs the future of humanity. Ultimately, because games are human constructions of complex systems, the agency of the designers must not be discounted either, but on the level of felt experience, this means that players must experiment with the designed rule systems to bring about desired changes to a lawfully operating environment that acts as an impersonal arbiter of the player’s actions – a challenging game would counterbalance easy, one-size-fits-all solutions by throwing complicating subsystems at the player, such as a tight in-game economy, a limitation of action points, AI agents with conflicting demands, or other players working at cross-purposes. All these serve to bring the drama of collective action closer to home. The Anthropocene is a conflux in global history of both actions and occurrences, an Earth-historical first. “Telling it as it is” is not only an ethical imperative, but a matter of group survival. Communicative media of all stripes have long relied on representational indexicality to make their cases about the world we live in. However, simulationist media, like games, also have algorithmic indexicality. That is to say, they not only make a claim about how truthfulness and truth claims are represented in static, fixed, authoritative models, there is an added layer of dynamic, open-ended processes, which affect a complex, adaptive system. As Cross argues, “Climate change is the sort of thing that, precisely because it’s so well-modelled, lends itself to interactive representation” (2019: n.p.). For example, a game might recognize that, if a gameworld registers an increase of greenhouse gas emissions by a set amount of parts per million, it must raise the global temperatures accordingly, or that a player’s implementation of carbon capture and storage technology ought to

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1069

reduce their CO2 emissions by so much, but it should keep other factors of pollution intact, in accordance with what the best models tell us the likely consequences of these events might be. Scientists of climate change communication have demonstrated the usefulness of games to mediate the ecological crisis, suggesting that they do so in four, interlocking ways: 1. By combining and incorporating different sources of (scientific) knowledge about uncertainty and translating or simplifying the knowledge to make it accessible to the target group. 2. By connecting the abstract descriptions of uncertainty to the tacit knowledge of the target group by providing a real life experience. 3. By directly showing the consequences of policy or individual decisions. A game exposes users to different conditions, settings, and renderings of the future. The game allows to present and calculate the effect of users current decisions. 4. By using subject matter as a vehicle for learning about the influence of different forms of uncertainty. Simulation games stimulate thinking about the long term in an experimental setting. (van Pelt et al. 2015: 43) Skeptics might object to a simplistic adoption of climate games as somehow “true” by contesting their use on several fronts. They could argue that, as human constructs, they are fallible and likely to contain the (un)conscious biases of their creators. They can claim that the translation from scientific data to game mechanics overly simplifies complex processes, to the point where they become divorced from reality altogether, or that scientific knowledge about climate change evolves too fast and such simulations can become quickly obsolete. They even have to option to outright reject them as rhetorical constructions that serve ulterior motives, such as green propaganda. Fortunately, none of the representational strategies identified by van Pelt’s group require simulation games to be slavishly realistic, faithful to particular scientific models of climate change, or to be predictive of actual conditions, all of which contribute to their use as fictional experiments that generate thought-provoking discourse on climate change. The potential mitigation of the games’ climate models as “just a game” also disarms demands of meticulous simulation while at the same time prompting players to temporarily suspend disbelief and adopt the game’s (pseudo-)scientific model of climate change to succeed. With these preliminary discussions in place, we shall now turn to the question of media specificity and genre. How do modern board games use their medial affordances to tell stories? What tools do they have at their disposal to depict climate change? How do god games and grand strategy simulations represent their gameworlds? Which elements lend themselves to playful interventions into environmental discourse? Attending to the conventions that make these qualified media work as conveyors of information and fun at the same time shall give us perspective on where each case study innovates in presenting the environmental crisis of our time.

1070

P. K. Makai

How Board Games Signify: Interaction Between Mechanics and Theme The primogeniture of board games grants them primacy in our discussion, too. Traditional board games (like chess, Parcheesi, backgammon, or games of chance) enjoyed millennia-long popularity with the elite and the common people alike. The nineteenth century served as the staging ground for the explosion of parlor games that members of the growing leisure class could while away their time with. The twentieth century saw the rise of proprietary, mass-market games from giants like Hasbro, 3 M, Ravensburger, Milton Bradley, or Mattel. The Second World War’s end brought on a renaissance of wargaming in the USA and, in a humbled Western Germany, a new trend for quality, nonviolent games focusing on building, trading, and optimizing abstract systems, called Eurogames. After reunification, Germany became a springboard for the popularization of a new era of board games whose auteurs (or créateurs, to use Booth’s terminology in 2021: 77–100) left their mark on their designs, starting with titles like Die Siedler von Catan, Tikal, and Carcassonne, which became widespread in Europe and North America alike. (This short history is adapted from Woods 2012: 15–78.) Climate change board games are largely the descendants of this Eurogaming tradition. Eurogames are recognizable from their relatively abstract boards, stylized game pieces (such as the meeple, a wooden, five-leaf cloverlike representation of a person, which became an icon of Eurogames), Victory Point-based scoring, themes based on exotic locations or cities known for particular trades, and game mechanics that foster indirect competition. They are systemic games, in the sense that players succeed by understanding how to extract the most points from the limited number of actions they have – they must be capable of extracting more resources for faster growth by optimizing their strategy and ought to come up with synergies to speed ahead of the competition. To unearth the implied and embedded narratives of board games under discussion, this work combines what Paul Booth calls a “ludo-textual analysis” and “ludic discourse analysis”: an LTA investigates “how similar textual elements combined with differing player interaction styles can lead to a greater understanding of the underlying values and beliefs that are present within the board game,” or, to put it in another way, “how player interactivity augments pure textuality” (Booth 2021: 19), whereas LDA is a “methodology for investigating the different meanings determined by particular game texts,” (58) primarily by analyzing how “meanings that are created via game play must function in combination between the game system (which sets up the theme and rules of the game) and the player engagement (which determines how those rules are enacted and that theme understood)” (60). In taking the very words board games use to describe their setting and mechanics, and in dissecting the way themes, mechanics, and player agency are set up to create conflicts of will, we can expose ideologically coded ways of understanding and interacting with game objects as media products.

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1071

God Games and Grand Strategy: An Immortal Bird’s-Eye View of Climate Change Digital games have a rich tradition of complex simulated systems, from simulations of cityscapes to football leagues, or warfare to agricultural production. Among the most challenging systems to convincingly simulate are the chaotic processes that govern biological life on Earth, including climate simulations, as well as artificial life and natural selection. Because these processes take place over geological times and span across the globe and interact in unforeseen ways, any simulation that hopes to make a credible case of some degree of accuracy must sacrifice a great deal of detail on the altar of computational feasibility and playability. The reduction of complexity is also necessary for a game to be learnable, and for the players to be empowered to experiment with different strategies of progressing within a game. The two most prominent genres of digital games that take on the challenge of representing climate change are so-called god games and city-building games, since they already simulate larger social communities and their complex interplay with the natural world. Examples of god games include Populous (Bullfrog 1989), Black & White (Lionhead Studios 2001), or The Universim (Crytivo n.d. Interactive n.d.), while city-building games are most recognizable as descendants of SimCity (Maxis 1989), Caesar (Impressions Games 1992), The Settlers (Blue Byte 1993), or Anno 1602: Creation of a New World (Max Design 1998). Mark Hayse defines the god game as one “in which players assume an explicitly divine role in the emergent growth and development of a simulated life system. [They] share some characteristics with other video game genres [. . .] in which players construct and manage the emergent growth of other systems, such as cities, civilizations, neighborhoods, and nations” (2012: 429). Makai has defined god games as digital games that: (a) Grant the player simulated powers that ordinary living organisms cannot exert over their life-worlds; (b) This power is exercised over a group of living beings whose interactions and social organization are complexly modelled with a degree of computercontrolled autonomy (c) Over the course of represented game time spanning whole historical or geological periods, (d) Pitting the player-controlled group against out-groups controlled by an artificial intelligence or a competing human player, (e) Where the death of an individual member of the controlled group does not constitute a fail state for the player and control is then given over another group member, (f) Where the betterment or continued survival of the group, tribe, species, etc. is the overarching goal of the player, (g) Served by dedicated game mechanisms that quantify the group’s adaptive traits or quality of life, and the player’s impact on the game-world. (Makai 2023)

1072

P. K. Makai

Because god games enable control over the natural world, they were among the first games to feature climate models. An early example, SimEarth, will be examined later on. Even today, most city-building games limit the presence of weather and climate to being visual effects, rather than as sustained factors in a city’s energy production or their agricultural yields, but even the earliest examples featured natural disasters, either occurring randomly (and not as a consequence of the player’s mismanagement of the climate) or initiated by the player to see their destructive powers in a more controlled manner. In what follows, our investigations turn to how climate system simulation evolved over the course of the history of game design in a sample of board and computer games that were particularly influential or innovative. By considering the implications of both ludo-textual analysis and ludic discourse analysis, the reader will gain a more nuanced understanding of the transmediation of scientific knowledge, and how the two playful media alter scientific models to make climate science playable and fun.

“Is This a Random World, or Did You Planet?”: SimEarth’s Designer Climate System To this date, there has never been a more ambitious project of climate game design than that of SimEarth (Maxis 1990). Fred Haslam and Will Wright’s vision was to create a living-breathing simulation of a whole planet and its climate systems, to model the creation of continents in geological time, the emergence of life, the birth of sentient animals, the rise of civilization, and its demise. SimEarth differed from games of their era because it lacked “what we might call a satisfying ‘win condition’ that terminates its algorithm” (Wark 2007: }215); in common gamer parlance, it is a sandbox game, which can be played indefinitely. Scenario play suggests some ways to beat a level, but the main “goal” of interactors is to tinker with the Earth system in a way that most satisfies them. Released after a year of development, the game was meant to serve as an illustration of the ideas of James Lovelock, author of the 1979 book Gaia: A New Look at Life on Earth. In it, Lovelock posited that the Earth is an actual living being, a “complex entity involving the Earth’s biosphere, atmosphere, oceans, and soil; the totality constituting a feedback or cybernetic system which seeks an optimal physical and chemical environment for life on this planet” (1979: 11). Lovelock formulated his hypothesis in a way congenial to computer programmers at the time, so when the team at Maxis was inspired to design what they would call a “software toy” (Rouse 2001: 439) or “philosophy toy” (Ching 2012), they effectively created a game that was thoroughly intermedial. If it is indeed philosophy that Maxis was after, it was not a slavish adoption of Lovelock’s book, albeit he did write a Foreword summarizing his theory in the game manual (Maxis and Bremer 1990: 3–4). Maxis themselves have been forthright about the fact that the astonishing complexity of the biosphere cannot be accurately

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1073

modelled; they wrote that SimEarth is “a rough caricature–an extreme simplification” (Maxis and Bremer 1990: 6). Even so, the simulation they have built is based on four models that capture the Earth systems at work in making the planet habitable. And it still remains one of the most complex playful simulations of the Earth to date. Even a decade after its publication, Will Wright expressed pride at what they achieved: “I have still never seen anyone do an integrated model with an integrated lithosphere, hydrosphere, and atmosphere [. . .]. And we were getting some effects in the model that were real effects, that really show up, that even some of the more elaborate models that NCAR [National Center for Atmospheric Research] makes weren’t capturing” (Wright interviewed in Rouse 2001: 444). Climatic conditions (e.g., heat, rainfall, winds, and sea heat) create different biomes that have temperature and precipitation preferences. The game tracks things like continental drift and air currents, as well as the movements of the sea – even the globe’s axial tilt figures into the calculations. Unlike many later climate simulations, the game not only follows the changes in CO2 and O2, but CH4 and N2 (in direct adaptation of the table in Lovelock 1979: 68), water vapor, dust particles, and air pressure as well, each with a decisive influence on the fate of Gaia. However, as Laura op de Beke states, “atmospheric and geospheric settings are best left alone after a habitable climate has been achieved because even small alterations can be catastrophic for the lifeforms” (2020, 245). As such, the game is one long argument in favor of the interconnectedness of all existence. SimEarth featured heavily into games scholar McKenzie Wark’s arguments about the need for a renewal of critical theory based on gaming (2007). She correctly identifies SimEarth as an illustration, or, in Galloway’s terms, an “allegorithm” (2006: 90) of political implications of the “theory of global warming” (Wark 1994: 127). Wark has also argued that SimEarth had something real-world global climate models of the time lacked, namely, “an understanding of the socio-technical processes which create it in any other terms than their physical inputs and outputs,” that would display “an understanding of interests which might link the global scenario back to the local and the domain of experience” (1994: 123). This refers to video games’ ability to radically compress time and space to convert the geological timescales into manageable play sessions and the random ebb and flow of epochs into perceivable events within the gameworld that the player is emotionally invested in, a barrier that would be harder to break down in real-world climate action without these philosophical toys. A ludic discourse analysis illuminates the game’s focus on control: the player exercises their agency chiefly by changing the parameters of the planet on Control Panels, and they are encouraged to reach the kind of homeostasis on analogy with the cybernetic control systems that Lovelock posits to be Gaia’s main mechanism of existence (1979: 48–63). Ludo-textual investigation reveals that the present-day scenario identifies problems, such as “pollution, nuclear war and global warming” in the in-game dialogue box, and the briefing insists the player “must solve each problem” (emphasis added) or leave the planet to its fate by becoming a spacefaring civilization. . .It seems, in climate games, insurmountable challenges must be beatable so as to avoid unduly frustrating the player.

1074

P. K. Makai

The Manual also sets out Things To Do in each timescale simulated, for example, “design continents,” or “burn off oceans,” and this discourse also exhorts the player to experiment, like “test the greenhouse effect” (Maxis and Bremer 1990: 102, 108). In actual play, however, there are few hindrances to the player’s actions, and achieving goals set out by the manual or the scenario might not nearly be as fun as to create a semi-stable world and then proceeding to annihilate it using one’s godly powers. It is not for naught that Adams calls a god game a “destruction and management simulation” (2010: 581, emphasis in original). Thus, the tension between what is written and what is possible becomes evident in the course of play. To their credit, the designers sought to infuse their work with discursive elements that foster life-promoting and life-preserving play. The abstract concept of Gaia is embodied in a pixel art avatar of a cartoon globe, complete with an uncanny facial expression that oscillates between constant astonishment and utter dismay. In Gaia, we see the epitomization of the principle identified by Kokonis that “in God Games, the hero is the map itself” (2015: 180). The expressive face of Gaia gives the interactor some emotional reflection that their actions have consequences to the living planet, and it reaffirms their status as the globe’s steward rather than king in a way not inimical to Wark’s assertion that ultimately, control belongs to the game rather than the player. Another gesture that promotes the transmission of scientific ideas is the dedication of a hefty chunk of the manual to describe the state of earth systems sciences at the time. Spanning 70 pages and featuring a 13-item bibliography, it outlines the model-building assumptions Maxis resorted to in working out their planetary model and serves as an ars geopoetica for SimEarth. Lovelock’s initial philosophical object, called Daisyworld, outlined in his collaborative article with Andrew Watson, posited a thought experiment – or “biological parable,” in their words – in which only two living life-forms, a black daisy and a white daisy, flourished, with different capabilities to reflect sunlight (known as albedo). Their argument was that the hypothetical planet would enact some form of spontaneous self-regulation due to the “close-coupling of the biota and the global environment” (1983: 284). These principles would be hard to extend to Earth as such, but they made the case that a temperature regulation system would work on similar principles on our Blue Marble. In a direct act of ludic transmediation, SimEarth made the Daisyworld thought experiment playable. It functions in terms very similar as those laid out by Lovelock, but the game experiment can be reset and altered to see how the simulation parameters would change with different initial conditions. However, instead of two types of daisies, there are eight, each with a different degree of sunlight reflected. The player is given the option to plant any kind of daisy in any combination and density and then observe how the conditions of life change as a result. According to the manual (and thus the designer’s intentions), it is meant to serve as a demonstration that “the life on the planet affected the climate of the planet in a way that is beneficial to life” and of “the two-way connection between life and environment” (Maxis and Michael Bremer 1990: 121). While the purpose of this chapter is not to be an arbiter of scientific truth, it is instructive to see that computational software can serve as an effective transmediator

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1075

of philosophical thought experiments with a basis in scientific models about the once and future climate. SimEarth presents the interaction with a designer climate system than can be fine-tuned or fiddled with, driven by the speculative science of thinkers with controversial but influential ideas of how the world works. In the case of SimEarth, the interactivity of the simulation closely mimics the pseudo-sapient, godlike Gaia, which might never exist in a way recognizable to its inventor, James Lovelock. Still, the transmission of scientific ideas from Watson and Lovelock’s (1983) paper to Gaia: A New Look at Life on Earth and thence from book to simulation indicates a willingness unparalleled in the history of computer gaming to create a mimetic, working model of Earth systems and, as a result, provides an allegorithm of climate change.

It Never Rains But It Pours: Sid Meier’s Civilization as an Environmental History Primer In 2021, civilization turned 30 years old. That is to say, the computer game Sid Meier’s Civilization (MicroProse 1991) turned 30 years old. Conceived of as an epic history of humankind in the wake of the success of god games like Populous (Bullfrog 1989) and management games like SimCity (Maxis 1989), developer MicroProse wanted to marry strategic planning with the feeling of divine powers in a turn-based game that would span from the founding of the first settlements to our future in the stars. The father of SimCity, Sid Meier, teamed up with Bruce Shelley, a noted board game designer at Avalon Hill, a wargaming company, where he created 1830: The Game of Railroads and Robber Barons (1986). Francis Tresham’s board game Civilization (1980) served as a primary inspiration for Sid Meier’s Civilization (henceforth, “Civ”). Tresham’s game followed the rise of civilization from 8000 BC to 250 BC in the Mediterranean and the Levant. It featured a complex system of, yes, warfare but also resource exploitation and trading, cultural achievements, plus technological innovation. It is credited for being the first game with a technological tree, whereby players could transform resources into a series of concatenated inventions, such as Mysticism, Coinage, Metalworking, or Democracy. Furthermore, it also featured natural and social disasters, called Calamities, which would impact the players’ civilizations, like Civil Disorders, Epidemics, Famines, Floods, and Volcanic Eruptions or Earthquakes. In their transmedial adaptation, one of the single biggest changes Meier and Shelley made was to shift the timeframe to encompass the modern era and the near future, too; Civ 1 began in 4000 BC and lasted until 2020–2100 depending on the game’s difficulty settings. Because of the vast temporal scope of the game, weather was never simulated, but since its first iteration, climate mechanics were implemented. Thus, in the game, it never rained on civilizations, but when it did, it poured, with floods soon following. Turn-based progression remained, the technological tree was vastly expanded over the board game, and cities now proved to be resource-gathering engines that fuelled the growth of empires. Cities could build

1076

P. K. Makai

21 edifices, and civilizations could construct 21 wonders of the world, which went beyond buildings to also include social projects like Magellan’s Expedition, Women’s Suffrage, and the Cure for Cancer. Disasters were also adapted from Tresham’s board game, with Earthquakes, Floods, Pirate Raids, or Plagues and other calamities bedeviling a civilization. Civ was recognized by educators as instructive and applicable in the classroom, and they have lauded its inclusion of environmental history into its gameplay, stating that it “gives the student an opportunity to consider the geographical ramifications of technological diffusion” and they help “to eliminate a teleological understanding of history” (Welchel 2007). However, the technological trees still betray elements of teleology (Ghys 2012). Adam Chapman observes that the games’ technology trees are more than a mere concatenation of history as technology but instead become “a procedural map of the links between particular tools, knowledge and, most importantly, collective action, in the games’ history of human civilization(s). Accordingly, the tech tree is a core aspect of the game’s ecological arguments” (2013: 67). Although Chapman uses ecology to mean the sum of actions afforded to the player in the historical possibility space of the game, this observation remains valid in its more literal sense: the technology trees define how players may use their produced units to shape the biomes of the game map. For example, Bronze Working allows the chopping down of forests, Iron Working enables the same for jungles, Agriculture opens up farming, researching Calendar unlocks the building of plantations, etc. For our interest in the climate, Civ 1 also introduced mechanics that promoted “planetary caretaking,” as well as climate change, and the game’s manual did not shy away from naming the culprits. They wrote: “One cost of heedless industrial growth is a gradual polluting and poisoning of the environment. Of the many dangers posed by pollution, the greatest may be global warming. An unchecked rise in the planet’s atmospheric temperature threatens catastrophic geographic changes including melting polar ice caps, rising sea levels, and parched farmlands” (Microprose 1991: 123). The designers implemented mechanics for industrial pollution, which reduces produced goods on tiles it has spread to. Industrial pollution’s appearance depends upon the city’s production and, once you research the automobile, its population. Pollution can be reduced by building Nuclear and Hydro Power Plants, as well as Recycling Centers, and the impact of cars can be eliminated with building Mass Transit. Pollution can be cleared up by the backbreaking labor of the Settlers unit. The progress of global warming is indicated as a sun icon, whose color changes from dark red to light red and thence to yellow and white in order of severity. Global warming may be triggered after nine tiles become polluted. Once global warming sets in, tiles may change from one biome to another: coastal lowland tiles may become swamps, forests might become jungles, and inland tiles have a chance of becoming deserts. The risk of environmental disasters rises, and after the resolution of the disaster, “the cycle starts over again. The planet has achieved equilibrium at the new higher temperatures. If pollution continues or increases once more to high levels, another bout of environmental problems may occur. This cycle may repeat

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1077

endlessly if pollution is not controlled,” the manual ominously warns (Microprose 1991: 125). The simulation of anthropogenic climate change was groundbreaking at the time, and few games have reciprocated its mechanics ever since to quite the same degree. Although digital Civ games have always featured pollution and global warming as key mechanics in the late game, the two board game adaptations (Drover 2002; Wilson 2010) have omitted them from their mechanics. Ostensibly, The latter’s design decisions must have taken into consideration the already voluminous in-person calculations needed to operate a physical copy of their games, but it does go to show that a series focused on resource extraction, expansion, and conquering of rival civilizations can still be counted as more of a wargame than a climate-conscious simulation at heart. This changed significantly with the new downloadable content (DLC)-based expansion pack for the series’ latest installment, Sid Meier’s Civilization VI: Gathering Storm (henceforth: Gathering Storm). For the first time, the game tackled climate change head-on and in depth. Each civilization’s technological advancement directly contributed to the globe’s compounded emissions of greenhouse gases (with CO2 standing in as an avatar for all GHGs), and emissions directly influenced the worsening of the global climate. Even if the player somehow managed to build a civilization with zero emissions, other AI- or human-controlled players could still pollute the planet into becoming a hothouse. In Gathering Storm, designers have created a climate system which is capable of simulating climate change and mechanics for mitigating said change. The climate system recognizes seven Phases of environmental destruction, with I equating to pre-industrial levels and VII being the most polluted atmosphere. There are two key prongs to making climate effects feel impactful: disasters and CO2 emissions. Power plants introduce CO2 into the air, and each unit that consumes the resources Coal, Oil, and Uranium is factored into global emissions, as are railroad tiles. Deforestation also impacts simulated CO2 emissions. Map size determines how much CO2 needs to be emitted for the climate system to shift to the next level of pollution, and once the requisite units are in the atmosphere, the damage is irreversible: one can never go back to a less dynamic state of the in-game climate. In practice, a shift from one state to the next is precipitated by an accumulation of Climate Change Points, each representing a 0.5
C increase in global temperatures. The transition to the next phase involves the following disastrous consequences, including but not limited to Droughts, Dust storms, Floods, Forest fires, Hurricanes, Meteor showers, Tornados, and Volcanic eruptions in a clear transmedial lineage of descent from the board game. To avoid the certain annihilation of the planet, players must maintain standards of good Earth stewardship and reduce factors leading to climate change. Besides not exploiting natural resources and becoming an energy powerhouse, they can do so through adopting the Global Warming Mitigation civic, which unlocks a project called Carbon Recapture. This is the only way to remove other civilizations’ emissions from the game’s climate system and is key to prevent the pandemonium of an overheating planet.

1078

P. K. Makai

To turn more analytical, the ludic discourse of the Civilization series has primarily posited global warming as a challenge to be met by the players. This challenge arises due to the inevitable technological development of civilizations, which concurrently seek technological solutions. Disasters strike randomly, they harm the player’s people, and they require the sacrifice of additional resources to research technologies that will fix the climate – often the same ones whose exploitation led to the disasters in the first place. Cameron Kunzelman also notes that the inexorable march toward global warming is a procedural argument in these games: “conceptually [. . .] our fast climb toward a fundamentally altered climate is the telos of the earth, as inevitable as guns and computers and democracy” (2022, 154), despite the historical flexibility of their narrative framework. Climate change becomes a problem in the late game: this is crucial because canny players who can exact a quick victory might not even see its effects, and also, if the game stretches out that long, the player must have built up a significant technological and resource advantage that can easily mitigate the worst excesses of destruction the game can throw at you. Because Civilization is a turn-based game with close to perfect information on the state of the gameworld’s climate, anticipating climate change is easy, and assessing the human impact is not much harder, either. However, as op de Beke points out, this works against the argument the game is trying to make: “the problem is that climate change does not function like a turn-based phenomenon at all. It does not patiently await its turn while we consider our options; and it certainly cannot be anticipated and outmanoeuvred like an opponent in chess” (2021). Further ludo-textual analysis also considers how different playstyles might affect the meaning of global warming scenarios. The intended effect of disasters and climate change mechanics are to throw curveballs at the player and to engender reflection on the collective (in)actions of humankind to secure safe climate futures. However, aided and abetted by omnipotence, dark play might emerge as a desired playstyle. Dark play is defined as a concept that deals with “issues such as [. . .] game actions and behaviours that are deviant and controversial content in games,” with the darkness denoting “content, themes, or actions that occur within games” that are “problematic, subversive, controversial, deviant, or tasteless” (Linderoth and Mortensen 2015: 5). In the case of climate change games, such actions might include simply doing nothing and watching the world burn, or even exterminating all human life by accelerating the climate crisis. Especially when the rhetoric of the game is so heavily biased in favor of an optimal route to mitigating climate change, it seems to taunt and beg a certain gamer demographic to wreak havoc. Players flood the world with glee (The Spiffing Brit 2019) and race to reach Apocalypse mode the fastest (RTGame 2020), and one nightmarish game of Civ II, dubbed the Eternal War, which has been running for 10 years around the time it became famous, had the polar ice caps melt 20 times due to continued nuclear war (Eternal War (Civ2) 2022). Such dark play can be both foreboding and a way to preprocess the trauma of the climate crisis; it is also a nihilistic celebration of the computer game medium, which enables the creation of thought experiments that can go spectacularly wrong.

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1079

Negotiation Is a Two-Way Street in Climate Change Board Games One of the things that climate video games have a hard time simulating is the sad fact that real-life climate change mitigation strategies often fail, or are too incremental, because political negotiations between human beings and communities are seldom driven by enlightened cooperation. Activists, industrialists, nations, and trading blocs find themselves at cross-purposes, and at the negotiating table, people with power and advantages do not want to give an inch if it benefits their adversaries. Because AI in digital games work algorithmically and are executed without a psychological impetus behind the code, computer-controlled agents react inflexibly and irrationally to player input, and programming quirks are incredibly easy to exploit by a skilled player. There is no room for such shenanigans in board game play. Environmental themes have experienced an upsurge in popularity as the climate crisis has become lodged in popular consciousness. Games about the rise of industry, global warming, melting ice caps, and the rise of atmospheric CO2 have become successful, each treating climate change as a serious consequence of development (Makai 2020). Environmental board games have been assessed for their ability to transmit knowledge and systems thinking about climate change and were found useful in inculcating the central premises of Earth systems interacting with human activity (Castronova and Knowles 2015; Eisenack and Reckien 2013). Scholars note that “it is important to strike a balance between quantitative and qualitative game components to create the right level of quantitative scientific detail,” because a “greater interrogation of quantitative game elements prolongs gameplay, and may create tension between retaining player interest and ensuring sufficient time for evaluation” (Flood et al. 2018: 18). In other words, abstraction makes games more engaging and memorable, so that educators may concentrate more on salient takeaways. Keep Cool (Eisenack and Petschel-Held 2004) is a negotiation board game that simulates the social dynamics of climate talks. Intended to raise awareness of the geopolitical landscape behind climate change mitigation, the game puts the players in the position of global powers, and they are tasked with decarbonizing their economy. Each players’ income is tied to the number of factories they own. To decarbonize, they need to phase out black, polluting factories for green factories using renewables. Black factories directly contribute to the heating up of the planet, represented by the Carbometer, a tower of tokens that symbolize GHG emissions – whenever a black factory is built, a carbon chip is removed from the Carbometer. If enough chips are removed, the planet begins experiencing devastating natural disasters, whose relief efforts must also be financed by the players. Each great power is further beholden to several lobbyist groups: hidden from other players, players receive cards signifying that they also represent the interests of, say, insurance companies, who benefit from a more turbulent climate, or the climate skeptics, who would like to see the green power plants kept to a minimum. Whoever first reaches a predetermined amount of green power plants plus satisfies

1080

P. K. Makai

one of their lobbyist groups’ goals is the winner. In a direct transmediation of an active climate policy, the latest Paris-Glasgow extension also introduces the mechanics of Nationally Determined Contributions. With NDCs, players may pledge to support other players’ green factories, or finance climate change adaptation, or vouch to remove X black factories in the upcoming turns. These pledges are entirely voluntary, but they serve as declarations for intended gameplay, and during debriefing or postgame talk, players are encouraged to reflect on how gameplay changed due to their NDCs being broadcast to other players. In actual gameplay, it is easy for the players to collectively reach the fail condition of the game, which is using up all the carbon chips on the Carbometer and thereby dooming the environment. Similarly, because natural disasters only impact some countries but not others, such as “Droughts in India,” “Floods in Central Europe,” or “Forest Fires in the US,” players who are not affected end up unwilling to contribute to relief efforts, which further drives players apart. The ludic discourse of the game switches the focus from an optimistic, every-country-will-collaborate vision of mitigating climate change to a more Realpolitik-based outlook of backroom deals and perverse incentives. However, an empirical study has shown that “playing climate-friendly is not to a necessary prerequisite for learning and changing beliefs about international climate politics. Instead, the possibility to experiment with alternative in-game decisions, either cooperative or non-cooperative, matters” (Meya and Eisenack 2018). This finding is consistent with the hypothesis that games are most effective when they function as replayable thought experiments that support dark play. Comparing Keep Cool to a more recent climate negotiation game, Kyoto: Money Makes the World Go Down (Harrer and Krenner 2020), is a revealing exercise in how the central mechanics of the game can stay relatively similar with vastly different consequences for its ludic discourse. To begin with, it is surprising how much of the essence of Keep Cool is retained in Kyoto: players represent global powers like Russia, China, or the EU, each seeking to lose less from climate mitigation policies than their rivals. Global impacts of development are represented on a thermometer next to a stylized board of Earth. The players are supported by lobbyists with agendas, which are dealt out on cards hidden from other players, and provide unique end-of-the-game scoring. But in Kyoto, players want to keep as much Affluence for their own countries as possible, represented by unique cards with titles like “Amusement Parks,” “Smartphones,” or “Burgers,” which they will need to sacrifice to reduce emissions, since each Affluence card has an associated set of environmental impacts, quantified as tons of CO2 but also as icons like animal skeletons for biodiversity loss or an atom for the use of nuclear power in sustaining that luxury. At the climate summit, each player takes a turn being a speaker and draws a study of climate change’s impact (e.g., “study on the reduction of biodiversity,” or “study on the melting of the polar ice caps and thawing of permafrost”), which require the players to collectively reduce emissions by discarding Affluence cards. Note that this is one of the very few games where studies are directly transmediated qua scientific papers. Should the players not be willing to sacrifice enough, each study has expected impacts revealed to all players, but the speaker also knows the hidden,

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1081

true impact, which changes the way they negotiate. These impacts range from species going extinct through the worsening of air pollution to a rise in global temperatures. Should any natural damage reach a critical threshold, all players lose, but if they manage to mitigate the impact of the 24 studies, they carry out a final scoring, determining the winner. Although both Keep Cool and Kyoto present the selfishness and economic pressures on decision-makers, and they both emphasize the role of multilateral contributions in mitigating climate change, Kyoto positions scientific studies as catalysts of climate negotiations and offers particular policymaking strategies for reducing impact, whereas Keep Cool already assumes that decarbonization is a worthy effort in and of itself, and insofar as this incontestable science is addressed at all, it is built into the fabric of the rules (global warming is caused by polluting industries). Another important innovation in Kyoto is that not only do the countries have hidden agendas, but the studies’ real impacts are also hidden from all but one person during negotiations, which (a) underscores the unequal power relations between governments and (b) serves as a critique of the transmediation of scientific papers, pointing the finger at how science communication and journalists can emphasize some more eye-catching aspect of a study while neglecting other, perhaps more consequential but less attention-grabbing results. Kyoto also makes a more forceful connection between global development and country prestige, as affluence is hard to give up for any country. Even so, the ludic discourse of the game makes it clear that “the country that best maintained its wealth wins” (Kobilke 2020: 2), rather than the one who did most to save the environment except if the players lose. In that scenario, the player with the most points is excluded from winning, and the person with the second most points becomes the “winner,” albeit that is a relative term at that stage, since Earth is ravaged. The ludo-textual implications of this means that a player may change their strategy mid-game if they see that the game state is trending toward global warming rather than successful mitigation and will start contributing more in an effort to shed excess affluence. This dynamic is entirely missing from Keep Cool.

Development as a Highway to Hell in Tipping Point and CO2 Where Kyoto reinvented Keep Cool’s negotiation mechanics, another game, Tipping Point (Smith 2020), scrapped the geopolitical squabbles and shifted its attention to the energy transition process. In Tipping Point, players become mayors of independent city-states (obviously transmediating the narrative situation of city-simulation video games), who must support the development of their cities, feed their evergrowing population, and protect them from the environmental impacts of climate change – but to do so, they must also curtail their GHG emissions. Like in Keep Cool, city developments are either black (polluting) or green (eco-friendly), and one of the goals is to meet the demands of the population while balancing black emissions with green developments. However, in this game, players with soldiers or military bases (black cards) may attack others and capture infrastructure cards,

1082

P. K. Makai

thereby simulating the incentives of the military-industrial complex to gain advantage over others at the cost of war. After everyone developed their cities, the Weather phase simulates the impact of CO2 emissions on the cities. The more is emitted, the more using the Weather cards are drawn. Any 2 cards drawn are translated into climactic events using Weather Almanac chart, and if 1 of the 9 combinations (out of a possible 16) is identified, players will have to face the consequences of extreme weather, such as Freezing Rains (Snow+Rain), Forest Fires (Sun+Wind), Blizzards (Wind+Snow), or Droughts (Sun+Sun). The mechanics mean that at the end of a decade, players may face multiple disasters: in a Forest Fire, the player with the largest Forest loses all Forest cards; then a Tornado wipes out one infrastructure and citizen for each player, followed by a Blizzard, where everyone loses a number of citizens again. Even when compared to other climate change games, disasters are truly devastating in Tipping Point. The ludic discourse of the game posits climate change as something that can be managed on the municipal level, which is an approach that appears short-sighted, especially in comparison with the other games taking the global view. The procedures of the game make the case for the inevitability of population growth, the necessity of innovation to meet challenges, and the causal effect they have on the weather. While the mechanism for determining the exact nature of the disasters is unique to the game, there is too much randomness involved in it, whereas real-life climate change is more predictable. Tipping Point does not take into account that citizens are not the main cause of GHG emissions, and it only distinguishes their carbon footprint based on their professions rather than their consumption habits. Finally, the game’s discourse argues that it is technology rather than politics and policy implementation that will ultimately secure the fate of the world. Finally, no discussion of climate change games would be complete without Vital Lacerda’s CO2: Second Chance (2018), which itself is a reworking of the original 2012 game. Much like in Keep Cool, in CO2, players take on the role of decisionmakers who try to innovate their way out of the climate crisis: as energy company executives, the players’ goal is to invent new technology, disseminate scientific knowledge, build green power plants, and fulfil goals set by the UN, each of which scores victory points – however, as in Tipping Point, should the collective CO2 emissions ever reach 500 ppm, the game ends with a loss for everyone. The player board is a stylized representation of the globe – perhaps the most clichéd icon of global change but one that remains a mainstay of climate games (op de Beke 2020). CO2 differs from other energy transition board games (and takes a leaf out of the playbooks of video games like Civilization) by including a greater diversity of power plants the players may build. At the start of the game, the globe is bespeckled with fossil fuel plants, including coal, oil, and natural gas plants. The way to replace them with green plants, like Hydro, Wind, and Solar power plants, as well as Recycling and Reforestation projects (which also contribute to net-zero emissions) is to propose projects across the globe and fulfil the UN’s goals for energy transition. To raise

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1083

money for these projects, the players exchange Carbon Emission Permits on the open market, with a dynamic change of prices as supply and demand fluctuate. The ludic discourse of the game could be criticized on a number of fronts. To begin with, the game is heavily biased in favor of technocratic solutions and imagines the mitigation of climate change in a capitalist framework, as the duty of innovating companies rather than as a concerted effort of governments to transform their energy sectors. In a highly significant act of bringing real-life discourses into the gameworld, the game manual is written to inform the players of the real-world science behind climate change (Lacerda 2018), since the developers included vignettes on the 2015 Paris summit, the 2030 UN goals for sustainable energy, and even a graph on CO2 emissions in recent history, using data from the Carbon Dioxide Information Analysis Center and the National Oceanic and Atmospheric Administration’s Office of Oceanic and Atmospheric Research’s Earth System Research Laboratory, which serves to bolster the game’s claims of scientific accuracy. Unfortunately, in practice, all energy projects are taken to have the same impact on the climate in terms of emissions reduction.

Coda: Fixing Games – The First Step to Fixing the Climate? The original CO2’s shortcomings were significant enough that noted games economist Edward Castronova and Isaac Knowles picked up on them and declared that “while CO2 does a good job of teaching players the difficulty of developing clean technologies, as well as the dangers of humanity’s collective failure to do so, it is less effective at teaching players the difficulties that policy-makers face in dealing with the CO2 problem” (Castronova and Knowles 2015: 45). To remedy these shortcomings, they have developed house rules for the game, calling it Climate Policy, which uses all the original game’s pieces but also includes mechanics such as the establishment of carbon regulations by governments, and loopholes to get out of said regulations, carbon taxes, technology grants, and property rights, to name just a few. Their design goes to show that by modifying existing games, the ludo-textuality of the game can be significantly altered, thereby interrogating the ideological assumptions of a game’s authors, and that games have the potential to be “a tremendous asset for exploring serious issues like public policy, but it is extraordinarily difficult to make a good one” (Castronova and Knowles 2015: 51). If this survey of board and computer games has taught us anything, it is that the environmental crisis makes the need to create thoughtful and enlightening games about the wicked problem of climate change more pressing every year and that thoughtful integration of real-world discourses into mechanics and aesthetics of the game create a truly intermedial web of signification and simulation. Newer games are often iterations and modifications of extant game mechanics (Gathering Storm, after all, was an expansion of the sixth Civilization game, and nearly every board game about climate change inherited something from Keep Cool), and with the never-ending creativity of game designers, they combine them

1084

P. K. Makai

to create more complex representations of the challenges of today and tomorrow. One such, highly anticipated board game still in development is Daybreak, a cooperative game from Matt Leacock, the designer of Pandemic (2008), and Matteo Menapace, a game designer and educator who creates games about social issues. Daybreak promises to be an even bigger hit than Pandemic – which put cooperative games on the map and proved to be a massive success during the Covid-19 crisis – and a timely intervention in the ludic discourse on climate change. One can only hope that it would turn more players toward issues like energy transition, climate justice, and the resilient societies we need to build to survive without Earth becoming an uninhabitable hothouse.

References Games Cited Blue Byte. 1993. The Settlers. Blue Byte. Bullfrog Productions. 1989. Populous. Electronic Arts. Crytivo Interactive. n.d. The Universim. (Forthcoming) Drover, Glenn. 2002. Sid Meier's Civilization: The Boardgame. Eagle-Gryphon Games. Eisenack, Klaus, and Gerhard Petschel-Held. 2004. Keep Cool. Spieltrieb. Firaxis Games. 2001. Sid Meier’s Civilization III. Infogrames. ———. 2019. Sid Meier’s Civilization VI: Gathering Storm. Expansion Pack. 2K Games. Harrer, Sabine, and Johannes Krenner. 2020. Kyoto: Money Makes the World Go Down. Deep print Games. Impressions Games. 1992. Caesar. Sierra On-Line. Leacock, Matt. 2008. Pandemic. Z-Man Games. Lionhead Studios. 2001. Black & White. Electronic Arts. Max Design. 1998. Anno 1602: Creation of a New World. Sunflowers Interactive. Maxis. 1999. SimCity. Maxis: Design by Will Wright. ———. 1990. SimEarth. Maxis: Design by Will Wright. MicroProse. 1991. Sid Meier’s Civilization. MicroProse. Smith, Ryan. 2020. Tipping Point. Treeceratops Games. Tresham, Francis. 1980. Civilization. Avalon Hill. ———. 1986. 1830: The Game of Railroads and Robber Barons. Avalon Hill. Wilson, Kevin. 2010. Sid Meier's Civilization: The Board Game. Fantasy Flight Games.

Works Cited Abraham, Benjamin. 2015. Video Game Visions of Climate Futures: ARMA 3 and Implications for Games and Persuasion. Games and Culture 13 (1): 71–91. https://doi.org/10.1177/ 1555412015603844. Adams, Ernest. 2010. Game Design Fundamentals. Berkeley, CA: New Riders. Andersen, Gregers. 2019. Climate Fiction and Cultural Analysis. New York/London: Routledge. https://doi.org/10.4324/9780429342493. Arnold, Annika. 2018. Climate Change and Storytelling: Narratives and Cultural Meaning in Environmental Communication. Cham: Palgrave Macmillan.

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1085

Ballew, Matthew T., Matthew H. Goldberg, Seth A. Rosenthal, Abel Gustafson, and Anthony Leiserowitz. 2019. Systems Thinking as a Pathway to Global Warming Beliefs and Attitudes through an Ecological Worldview. Proceedings of the National Academy of Sciences of the United States of America 116 (17): 8214–8219. https://doi.org/10.1073/pnas.1819310116. Bellomy, Ian. 2017. What Counts: Configuring the Human in Platform Studies. Analog Game Studies 4 (2) http://analoggamestudies.org/2017/03/what-counts/. Bentz, Julia. 2020. Learning about Climate Change in, with and through Art. Climatic Change 162 (3): 1595–1612. https://doi.org/10.1007/s10584-020-02804-4. Accessed March 12, 2022. Booth, Paul. 2021. Board Games as Media. 1st ed. London: Bloomsbury Academic. https://doi.org/ 10.5040/9781501357206. Bruhn, Jørgen. 2020. Intermedial Ecocriticism: The Anthropocene ecological crisis across media and the arts. Ekphrasis. Images, Cinema, Theory, Media 24 (2): 5–18. Buckland, Adelene. 2013. Novel Science : Fiction and the Invention of Nineteenth-Century Geology. Chicago and London: University of Chicago Press. Bushell, Simon, Geraldine Satre Buisson, Mark Workman, and Thomas Colley. 2017. Strategic Narratives in Climate Change: Towards a Unifying Narrative to Address the Action Gap on Climate Change. Energy Research & Social Science 28 (June): 39–49. https://doi.org/10.1016/j. erss.2017.04.001. Castronova, Edward, and Isaac Knowles. 2015. A Model of Climate Policy Using Board Game Mechanics. International Journal of Serious Games 2 (3 SE-Articles). https://doi.org/10.17083/ ijsg.v2i3.77\. Accessed March 12, 2022. Chang, Alenda Y. 2019. Playing Nature: Ecology in Video Games. Minneapolis: University of Minnesota Press. Chapman, Adam. 2013. Affording History: Civilization and the Ecological Approach. In Playing with the Past: Digital games and the simulation of history, ed. Matthew Wilhelm Kapell and Andrew B.R. Elliott, 61–74. New York: Bloomsbury Academic. https://doi.org/10.5040/ 9781628928259.ch-004. Accessed March 12, 2022. Ching, Dixie. 2012. Passion Play: Will Wright and Games for Science Learning. Cultural Studies of Science Education 7: 767–782. Clareson, Thomas D. 1971. SF: The Other Side of Realism; Essays on Modern Fantasy and Science Fiction. Bowling Green, OH: Bowling Green Univ. Popular Press. Cross, Katherine. 2019. Opinion: Civilization VI’s (Green) New Deal. Gamasutra, February 26, 2019. https://www.gamedeveloper.com/design/opinion-i-civilization-vi-i-s-green-new-deal. Doyle, Julie. 2011. Mediating Climate Change. New York: Routledge. Eisenack, Klaus, and Diana Reckien. 2013. Climate Change and Simulation/Gaming. Simulation & Gaming 44 (2–3): 245–252. https://doi.org/10.1177/1046878113490568. Elleström, Lars. 2018. Coherence and Truthfulness in Communication: Intracommunicational and Extracommunicational Indexicality. Semiotica 2018: 423. https://doi.org/10.1515/sem2017-0001. ———. 2020. Representing the Anthropocene : Transmediation of Narratives and Truthfulness from Science to Feature Film. Ekphrasis. Images, Cinema, Theory, Media 24 (2): 36–48. https:// doi.org/10.24193/ekphrasis.24.3. Eternal War (Civ2). 2022. Civilization Wiki. https://civilization.fandom.com/wiki/Eternal_War_ (Civ2). Accessed March 12, 2022. Flood, Stephen, Nicholas A. Cradock-Henry, Paula Blackett, and Peter Edwards. 2018. Adaptive and Interactive Climate Futures: Systematic Review of ‘Serious Games’ for Engagement and Decision-Making. Environmental Research Letters 13 (6): 063005. https://doi.org/10.1088/ 1748-9326/aac1c6. Galloway, Alexander R. 2006. Gaming: Essays on Algorithmic Culture. Minneapolis, MN: University of Minnesota Press. Ghys, Tuur. 2012. Technology Trees: Freedom and Determinism in Historical Strategy Games. Game Studies 12 (1) http://gamestudies.org/1201/articles/tuur_ghys. Accessed March 12, 2022.

1086

P. K. Makai

Goodbody, Axel, and Adeline Johns-Putra. 2019. Cli-Fi. A Companion. Oxford, UK: Peter Lang Verlag. https://www.peterlang.com/document/1056436. Hayse, Mark. 2012. God Games. In Encyclopedia of Video Games: The Culture, Technology and Art of Gaming, ed. Mark J.P. Wolf, vol. 1, 429–431. Santa Barbara: Greenwood. Hausfather, Zeke. 2017. Analysis: How Well Have Climate Models Projected Global Warming? – Carbon Brief. Carbon Brief, October 5, 2017. https://www.carbonbrief.org/analysis-how-wellhave-climate-models-projected-global-warming. Accessed March 12, 2022. Holbert, Nathan, and Uri Wilensky. 2019. Designing Educational Video Games to Be Objects-toThink-With. Journal of the Learning Sciences 28 (1): 32–72. https://doi.org/10.1080/10508406. 2018.1487302. Accessed March 12, 2022. Johns-Putra, Adeline. 2019. Climate and Literature. Cambridge\New York: Cambridge University Press. Kaplan, Elizabeth Ann. 2015. Climate Trauma: Foreseeing the Future in Dystopian Film and Fiction. New Brunswick\New Jersey\London: Rutgers University Press. Kobilke, Viktor. 2020. Kyoto: Money Makes the World Go Down Rule Book. Berlin: Deep Print Games. Koenitz, Hartmut, Mirjam Palosaari Eladhari, Sandy Louchart, and Frank Nack. 2020. INDCOR White Paper 1: A Shared Vocabulary for IDN (Interactive Digital Narratives), October 2020. http://arxiv.org/abs/2010.10135. Konior, Bogna. 2020. Modelling Realism: Digital Media, Climate Simulations and Climate Fictions. Paradoxa 31 (1): 55–76. Kunzelman, Cameron. 2022. Video Games as Interventions in the Climate Disaster. Paradoxa 31 (1): 105–122. ———. 2022. The World is Born From Zero. Oldenbourg: De Gruyter. Lacerda, Vital. 2018. CO2: Second Chance Rulebook. Roma: Giochix. Linderoth, Jonas, and Torill Elvira Mortensen. 2015. Dark Play: The Aesthetics of Controversial Playfulness. The Dark Side of Game Play, November, 3–12. https://doi.org/10.4324/ 9781315738680-1. Accessed March 12, 2022. Lovelock, James E. 1979. Gaia: A New Look at Life on Earth. Oxford: Oxford University Press. Makai, Péter Kristóf. 2020. Climate Change on Cardboard: Ecological Eurogames. Paradoxa 31 (1): 77–104. ———. 2023. Red in Bits and Bytes: Evolutionary Conflicts in Biological God Games. In Representing Conflicts in Game: Antagonism, Rivalry, and Competition, ed. Björn Sjöblom, Jonas Linderoth and Anders Frank. New York. Routledge. 13–29. ———. 2023. Red in Bits and Bytes: Evolutionary Conflicts in Biological God Games. In Representing Conflict in Games: Antagonism, Rivalry and Competition, ed. Björn Sjöblom, Jonas Linderoth, and Anders Frank, 13–29. New York: Routledge. Maxis and Michael Bremer. 1990. SimEarth User Manual. Moraga, CA: Maxis. Mertens, Mahlu, and Stef Craps. 2018. Contemporary Fiction vs. the Challenge of Imagining the Timescale of Climate Change. Studies in the Novel 50 (1): 134–153. https://doi.org/10.1353/ sdn.2018.0007. Meya, Jasper N., and Klaus Eisenack. 2018. Effectiveness of Gaming for Communicating and Teaching Climate Change. Climatic Change 149 (319–33). https://doi.org/10.1007/s10584-0182254-7. Accessed March 12, 2022. Microprose. 1991. Civilization Manual. Alameda, CA: Microprose USA. Milner, Andrew, and J.R. Burgmann. 2017. Climate Fiction: A World-Systems Approach. Cultural Sociology 12 (1): 22–36. https://doi.org/10.1177/1749975517725670. Murray, Janet Horowitz. 2017. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. 2nd ed. Cambridge, MA: The MIT Press. Nurmis, Joanna. 2016. Visual Climate Change Art 2005–2015: Discourse and Practice. WIREs Climate Change. Wiley. https://doi.org/10.1002/wcc.400. Op de Beke, Laura. 2020. Anthropocene Temporality in Gaia Games. KronoScope 20 (2): 239–259. https://doi.org/10.1163/15685241-12341470. Accessed March 12, 2022.

41

Simulated Climate in Ecological Games: Mediating Climate Change to. . .

1087

———. 2021. Procedural Futurism in Climate Change Videogames. Alluvium Journal 9 (3) https:// www.alluvium-journal.org/2021/06/04/procedural-futurism-in-climate-change-videogames/. Accessed March 12, 2022. Pias, Klaus. 2008. Klimasimulation. In 2: Das Wetter, der Mensch und sein Klima, ed. Petra Lutz and Thomas Macho, 108–115. Dresden: Wallstein Verlag. Ranney, Michael Andrew, and Dav Clark. 2016. Climate Change Conceptual Change: Scientific Information Can Transform Attitudes. Topics in Cognitive Science 8 (1): 49–75. https://doi.org/ 10.1111/tops.12187. Rogerson, Melissa J., Martin Gibbs, and Wally Smith. 2016. I Love All the Bits’: The Materiality of Boardgames. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 3956–3969. CHI ’16. New York, NY: Association for Computing Machinery. https:// doi.org/10.1145/2858036.2858433. Rooney-Varga, J.N., J.D. Sterman, E. Fracassi, T. Franck, F. Kapmeier, V. Kurker, E. Johnston, A.P. Jones, and K. Rath. 2018. Combining Role-Play with Interactive Simulation to Motivate Informed Climate Action: Evidence from the World Climate Simulation. PLoS One 13 (8): e0202877. https://doi.org/10.1371/journal.pone.0202877. Accessed March 12, 2022. Rouse, Richard. 2001. Game Design: Theory & Practice. Plano, TX: Wordware. RTGame. 2020. I Caused The Apocalypse As Fast As Possible in Civilization. YouTube, May 29, 2020. https://www.youtube.com/watch?v¼3EYD7V1OH_Q. Accessed March 12, 2022. Rughiniş, Cosima, and Michael G. Flaherty. 2022. The Social Bifurcation of Reality: Symmetrical Construction of Knowledge in Science-Trusting and Science-Distrusting Discourses. Frontiers in Sociology. https://www.frontiersin.org/article/10.3389/fsoc.2022.782851. Accessed March 12, 2022. Schneider-Mayerson, Matthew. 2018. The Influence of Climate Fiction: An Empirical Survey of Readers. Environmental Humanities 10 (2): 473–500. https://doi.org/10.1215/220119197156848. Schulzke, Marcus. 2014. Simulating Philosophy: Interpreting Video Games as Executable Thought Experiments. Philosophy & Technology 27 (2): 251–265. https://doi.org/10.1007/s13347-0130102-2. Simis, Molly J., Haley Madden, Michael A. Cacciatore, and Sara K. Yeo. 2016. The Lure of Rationality: Why Does the Deficit Model Persist in Science Communication? Public Understanding of Science (Bristol, England) 25 (4): 400–414. https://doi.org/10.1177/ 0963662516629749. Suldovsky, Brianne. 2017. The Information Deficit Model and Climate Change Communication. In Oxford Research Encyclopedia of Climate Science. Oxford University Press. https://doi.org/10. 1093/acrefore/9780190228620.013.301. The Spiffing Brit. 2019. Can You Flood The World In Civilization 6 Gathering Storm? Breaking The Perfectly Balanced Game. YouTube, February 28, 2019. https://www.youtube.com/watch? v¼_9iyGXfLsoQ. Accessed March 12, 2022. Tornborg, Emma. 2019. Transmediations of the Anthropocene: From Factual Media to Poetry. In Transmediations: Communication Across Media Borders, 235–253. Taylor and Francis. https:// doi.org/10.4324/9780429282775-12. van Pelt, S.C., Marjolijn Haasnoot, Bas Arts, Fulco Ludwig, Rob Swart, and Robbert Biesbroek. 2015. Communicating Climate (Change) Uncertainties: Simulation Games as Boundary Objects. Environmental Science & Policy 45: 41–52. https://doi.org/10.1016/j.envsci.2014. 09.004. Veland, S., M. Scoville-Simonds, I. Gram-Hanssen, A.K. Schorre, A. El Khoury, M.J. Nordbø, A.H. Lynch, G. Hochachka, and M. Bjørkan. 2018. Narrative Matters for Sustainability: The Transformative Role of Storytelling in Realizing 1.5
C Futures. Current Opinion in Environmental Sustainability 31: 41–47. https://doi.org/10.1016/j.cosust.2017.12.005. Accessed March 12, 2022. Wark, McKenzie. 1994. Third Nature. Cultural Studies 8 (1): 115–132.

1088

P. K. Makai

———. 2007. Gamer Theory. Cambridge, MA/London, UK: Harvard University Press. https://doi. org/10.4159/9780674044838. Watson, Andrew J., and James E. Lovelock. 1983. Biological Homeostasis of the Global Environment: The Parable of Daisyworld. Tellus B: Chemical and Physical Meteorology 35 (4): 284–289. https://doi.org/10.3402/tellusb.v35i4.14616. Accessed March 12, 2022. Whelchel, Aaron. 2007. Using Civilization Simulation Video Games in the World History Classroom. World History Connected 4.2 (February 2007) https://worldhistoryconnected.press. uillinois.edu/4.2/whelchel.html. Accessed March 12, 2022. Woods, Stewart. 2012. Eurogames: The Design, Culture and Play of Modern European Board Games. Jefferson, NC: McFarland & Co.

Intermediality in Theme Parks

42

Florian Freitag

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Plurimediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Paramediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1090 1095 1100 1104 1109 1110

Abstract

This chapter provides readers with a broad overview of the various ways in which intermediality has been conceived in the interdisciplinary field of global theme park studies. Building up on established research strands, the discussion takes the form of a critical review of secondary literature from various disciplines and surveys three main areas: (1) The plurimediality of theme parks: in order to create multisensory, immersive experiences for visitors, theme parks have combined and fused individual art forms and media such as architecture, landscaping, music, etc. Within this plurimedial mix, however, critics have identified distinct roles for specific media such as film, performance, or language. (2) The transmediality of theme parks: from their beginnings, theme parks have been firmly integrated into chains of remediations. Research has particularly investigated how transmedia conglomerates have used theme parks as one among many platforms for the distribution of transmedia franchises, but has also become interested in the alternative strategies that smaller companies have developed in order to offer recognizable media content at their parks. (3) The paramediality of theme parks: guidebooks, maps, apps, and other media paratexts have been considered as

F. Freitag (*) University of Duisburg-Essen, Essen, Germany e-mail: fl[email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_53

1089

1090

F. Freitag

medial interfaces that provide visitors with filters and guidelines on how to anticipate, experience, and remember the park landscape and thus script the theme park experience even beyond the theme park. Keywords

Disney · Media · Paratext · Theme parks · Transmediality

Introduction Research on (inter)medial aspects of modern theme parks forms an integral and paradigmatic part of theme park studies, going back to the very beginnings of scholarly discussions of the parks in the late 1960s and subsequently displaying many of the general characteristics of the field. In The Disney Version: The Life, Times, Art and Commerce of Walt Disney (1968), one of the first serious critical studies of Disneyland and published even before the term “theme park” had established itself in the scholarly discourse,1 Richard Schickel not only described the then 13-year-old park in Anaheim, California, as a “new and unique medium” (18) in itself, but further characterized it as “one of the best mixed-media shows ever devised” (325) and located it within a transmedial network where all parts “interlock and are mutually reciprocating” (18). Hence, Schickel simultaneously established the mediality of the theme park (i.e., the notion that the theme park constitutes a medium), its plurimediality (i.e., the idea that as a medium, the theme park combines and references other media), as well as its trans- and paramediality (i.e., the notion that the theme park is firmly embedded within networks of media artefacts that refer to and promote each other). Schickel thus laid the groundwork for research on intermediality in theme parks and provided a rough map of the field that will also serve as the underlying structure of this chapter. Subsequent scholarship would further explore and greatly refine Schickel’s ideas about the (pluri-, trans-, and para)mediality of theme parks but would also contribute to establishing – and in turn be influenced by – certain critical paradigms that are only beginning to be questioned and overcome in more recent work. With respect to scope, for example, theme park criticism has often focused on Western (and specifically US) as well as large destination (and specifically Disney and, to a lesser extent, Universal) parks. Writing in 1968, Schickel had few other parks to draw on, either in the USA or elsewhere – and, seeking to publish an “analytic biography” (10) of Walt Disney, may have also had little interest in them. Over the years, however, the Disney parks’ enormous impact on the industry and their easy accessibility (in the

The Oxford English Dictionary dates the first use of the term “theme park” to the year 1960 (in the American Peoples Encyclopedia Year Book). However, the first critical studies of Disneyland still referred to the place as an “‘atmospheric park’” (Schickel 1968: 22) or an “amusement park” (Finch 1973: 396); by the mid-1970s, however, the term “theme park” appears to have established itself in scholarly publications as well (see Hall 1976).

1

42

Intermediality in Theme Parks

1091

geographic, linguistic, and cultural sense of the term) to Western scholars has led to a serious critical imbalance, with smaller and non-Western parks having been comparatively neglected. Recent studies such as Maribeth Erb and Chin-Ee Ong’s Theming Asia (2017), Crispin Paine’s Gods and Rollercoasters (2019), Filippo Carlà-Uhink’s Representations of Classical Greece in Theme Parks (2020), and Florian Freitag2 and Chang Liu’s Cultural History and Heritage in Chinese Theme Parks (2022) have explicitly gone beyond the established canon of theme parks, with, for example, the contributions in Erb and Ong as well as in Freitag and Liu deliberately concentrating on theme parks and themed environments in the AsiaPacific region, certainly one of the most dynamic markets in the global theme park industry during the last years before the pandemic.3 Yet also and especially with respect to (inter)medial aspects of theme parks, scholarship has continued to focus on Western and destination parks, thereby neglecting how, for example, strategies of transmedia storytelling have been adapted elsewhere and have spread beyond the big parks. By focusing on a specific theme and how the latter is portrayed in smaller, local parks in the USA, Europe, and Asia, the studies by Paine and Carlà-Uhink, but also those by Sabrina Mittermeier (A Cultural History of the Disneyland Theme Parks) and Florian Freitag (Popular New Orleans; both 2021), which trace the development of specific themes and theme park formats across time, simultaneously indicate a paradigmatic change in the methodology of theme park studies. Generally, theme park scholars have preferred synchronic over diachronic approaches and have focused on individual parks rather than, for example, comparing parks or examining specific topics and phenomena across parks. The Journal of Popular Culture’s summer 1981 special issue on theme and amusement parks had, in fact, featured both a diachronic study (of Cedar Park in Sandusky, Ohio; see Hildebrandt) as well as a comparative investigation (of the Main Street, U.S.A. sections in Disneyland and Walt Disney World’s Magic Kingdom; see Francaviglia). Nevertheless, the scarcity of historiographical approaches to theme parks in subsequent research would prompt Alan Bryman in Disney and His Worlds (1995) to remind his fellow critics that “the parks are not inert texts” (83) but are constantly changing and evolving; and single-park studies such as Stephen M. Fjellman’s Vinyl Leaves: Walt Disney World and America (1992) or Susan G. Davis’s Spectacular Nature: Corporate Culture and the Sea World Experience (1997) would for a long time remain the norm. While such in-depth textual analyses of individual sites have greatly contributed to understanding the role of individual media within the parks’

2

Throughout this chapter, the author will refer to his own research in the third person. Before the pandemic, the Chinese theme park market had long been predicted to become the largest in the world by 2020 (see e.g., Rubin 2018: 41). While this may not have happened (yet), it is true that with their mostly local or regional catchment area and market, Chinese parks were hit much less severely by the pandemic, with the People’s Republic of China’s OCT parks reporting an average attendance decline of only 14% and attendance at Fantawild parks merely dropping an average of 20% in 2020 (see Anton Clavé et al. 2023).

3

1092

F. Freitag

complex plurimediality, they have also prevented us from seeing the bigger picture as well as long-term developments. Yet another recent paradigmatic methodological change concerns questions of the production of theme parks by employees and their reception by visitors, fans, and artists. Early pioneering studies such as Michael R. Real’s Mass-Mediated Culture (on theme park visitors; 1977) or John van Maanen’s “The Smile Factory: Work at Disneyland” (on theme park employees; 1991) had long remained largely without any emulators, although scholars had repeatedly called for a “triangulated” or “integrated” theme park studies that simultaneously takes into account the parks themselves as well as their production and reception (see, for example, Raz 1999 and Lukas 2016). A rising interest in theme park fandom, exemplified by such studies as Janet Wasko’s Understanding Disney (revised edition 2020) and especially Rebecca Williams’s Theme Park Fandom (2020) and Abby S. Waysdorf’s Fan Sites (2021), has recently contributed to complicating the traditional notion of theme park visitors as passive and easily controlled by the park designs and landscapes, a notion that goes back as far as Royston Landau’s 1973 “Mickey Mouse the Great Dictator: The Disney Game as a Control System” (“Central to the Disney programme there is a singleminded, undeviating goal which is: to entertain, to please and to provide fun for all those who wish to be placed in the role of passive fun lovers,” 592; emphasis original). Perhaps somewhat paradoxically, however, recent scholarship on theme park fandom has also shown that it is not only through their designed landscapes that theme parks seek to control the ways they are anticipated, experienced, and remembered but also through ancillary or paramedia such as guide apps and social media. Seeking to debunk the myth of passive, controlled audiences, critics have thus also contributed to further exploring the issue of control through theme park paramediality. Finally, research on theme park paramedia and their effects on visitors has also profited from the general increase of economic perspectives on theme parks. Whereas early scholarship on theme parks mainly came from cultural and media studies, semiotics, and human geography, the parks’ growing economic relevance eventually led to an understanding of the theme park sector as a discrete branch of industry, first on a national (see Judith A. Adams’s American Amusement Park Industry: A History of Technology and Thrills, 1991), then on a global scale (see Salvador Anton Clavé’s The Global Theme Park Industry, 2007). Among the various issues discussed in recent economic theme park research, it is especially studies on capacity management, visitor satisfaction, and marketing and branding that have contributed to a deeper understanding of theme park paramedia. As it is frequently published in Chinese, however, much recent work on theme park economics remains inaccessible to a non-Chinese-speaking audience (see Zhang and Shan 2016). Many classic studies of theme parks had, in fact, originally appeared in French or Italian – from Louis Marin’s Utopiques: Jeux d’espaces (1973) and Umberto Eco’s “Viaggio nell’iperrealtà” (1975) to Jean Baudrillard’s “La précession des simulacres” (1978) – but were soon translated into English, something which due to the sheer volume of work published in Chinese today is no longer an option, however. At the beginning of the 2020s, then, research on theme park paramediality – and research on intermediality in theme parks in general – is thus also paradigmatic of theme park

42

Intermediality in Theme Parks

1093

studies in the sense that it illustrates the growing multidisciplinarity and multilingualism of the field. By further exploring early ideas about the pluri-, trans-, and paramediality of the theme park, scholars have also refined the notion of the theme park as a medium in itself, although here, too, well-established critical paradigms have contributed to underlining certain issues while obscuring others. For example, according to Lisa Gitelman, a medium is a technology that enables communication and around which certain social and cultural practices or “protocols” have grown (see Gitelman 2006: 5–6). Of course, what exactly is communicated via the “technology” of the theme park, how it is communicated, as well as the accompanying “protocols” of the theme park visit have changed dramatically since the beginnings of the modern theme park in mid-1950s’ southern California.4 Thus the opening of EPCOT at Walt Disney World (Orlando, Florida) in 1982 marked the advent of the “multigate” theme park resort with several parks, water parks, and entertainment or shopping districts, hotels etc., which would transform the “regular” theme park visit from a day trip to an entire holiday lasting several days or even more. With respect to content, the opening of entire lands based on transmedia franchises in the 2010s such as The Wizarding World of Harry Potter at Universal parks (since 2010) or Star Wars: Galaxy’s Edge at Disney parks (since 2019) has raised IP-based theming – heretofore mainly restricted to individual rides, shows, restaurants etc. – to a new level. Finally, the development of new media technologies such as onboard audio systems, mist screens, dome theaters, or the video game have left their traces in both continuously updated (e.g., Disney’s Space Mountain or Pirates of the Caribbean) as well as newly developed theme park attractions (e.g., Universal’s Men in Black: Alien Attack interactive shooter ride). However, due to the field’s traditional preference for synchronic and rather than historical perspectives (or, perhaps more precisely, for synchronic perspectives and the larger history of immersive spaces rather than for micro-historical approaches to the development of the theme park form and individual parks), such developments have largely been ignored so far. Instead, critics have further qualified the mediality of the theme park by stressing its pluri-, trans-, or paramedial specificities. For example, writing only five years after Schickel and seeking to highlight the 4

The questions of when, where, and how theme parks originated are both complex and contested, especially because the answers to these questions are of popular and academic but also commercial interest in an industry where novelty, innovation, and uniqueness are paramount. While there is reason to argue that the contemporary theme park as it has appeared since the 1950s in southern California with Disneyland (Anaheim; 1955) and Pacific Ocean Park (Santa Monica; 1958) displays singular characteristics that define it as something new and different to older forms of entertainment (Anton Clavé 2007: 21–27), scholars such as Robert Riley and Terence Young (Theme Park Landscapes: Antecedents and Variations, 2002), Norman Klein (The Vatican to Vegas: A History of Special Effects, 2004), Angela Ndalianis (Neo-Baroque Aesthetics and Contemporary Entertainment, 2004), and Miodrag Mitrasinovic (Total Landscape, Theme Parks, Public Space, 2006) have situated the theme park within much older traditions, reaching as far back as ancient Rome. Ultimately, both claims of newness as well as teleological historical narratives should be viewed critically.

1094

F. Freitag

plurimediality of Disneyland and Walt Disney World’s Magic Kingdom, Christopher Finch, in The Art of Walt Disney from Mickey Mouse to the Magic Kingdoms (1973), employed the term “total theater”: The theme parks exploit all kinds of technical novelties – from the audioanimatronic figures to a 360-degree wraparound cinema screen [. . .]. [. . .] Disneyland and Walt Disney World are shows – a kind of total theater which exceeds the wildest dreams of avant-garde dramatists. (421)

In The Total Work of Art: From Bayreuth to Cyberspace (2007), Matthew Wilson Smith even more explicitly refers to Richard Wagner and describes Disneyland as a “Gesamtkunstwerk” (a total work of art; 126). Other scholars investigating the roles(s) of individual media in theme parks or their multisensoriality and immersivity have drawn on Irina Rajewsky’s and Werner Wolf’s typologies of intermediality to describe the parks as “hybrid” or “composite” media (Freitag 2017a: 706; see Rajewsky 2002: 203 and Wolf 2007: 253) or have simply spoken of the “hypermediacy” of theme parks (Bolter and Grusin 2000: 169). In turn, critics interested in the transmediality of theme parks – whether in the more economic, synergistic side of transmedial networks or in narrative questions about transmedia storytelling – have employed terms such as “synthetic media” (Davis 1996: 399) or “narrative medium” (Baker 2018: 1). “Total medium” and “participatory medium,” finally, have been used by scholars examining the relationship between the theme park and its audience, with, for instance, Real noting that Disneyland is “a total medium exercising an awesome degree of social control” (1977: 85) and Mittermeier using “participatory medium” to stress the theme park’s “status as a medium in a larger participatory culture” (2021, 8). The contrast between Real’s 1977 reading of Disneyland as a “total medium” and Mittermeier’s 2021 identification of the theme park as a “participatory medium” shows how dramatically the debate has changed over the decades and simultaneously testifies to the fact that aspects of intermediality have continued to be of central relevance to the field. Indeed, while theme park studies are still in the process of consolidation and institutionalization,5 scholarly discussions about theme parks have been concerned with questions of intermediality from the very beginning. To

5

In addition to the publication of several more or less comprehensive introductions to theme parks, including Anton Clavé’s The Global Theme Park Industry (2007), Scott A. Lukas’s Theme Park (2008), David Younger’s Theme Park Design and the Art of Themed Entertainment (2016), and, most recently, Anton Clavé et al.’s Key Concepts in Theme Park Studies: Understanding Tourism and Leisure Spaces (2023), the foundation of the Journal of Themed Experience and Attractions Studies in 2018 as well as the establishment of specialized degree programs and tracks of study – e.g., at the University of Central Florida, the University of Indianapolis, and the California Institute of Arts (all in the United States); Staffordshire University and Falmouth University (U.K.); the Baden-Wuerttemberg Cooperative State University Ravensburg (Germany); or Breda University of Applied Sciences (the Netherlands) – have marked significant steps toward the recognition of the theme park as a unique, distinct form worthy of its own academic field.

42

Intermediality in Theme Parks

1095

argue, then, as Abby Waysdorf has done in 2021, that “the theme park can be seen as a medium” that “has its own specificity” (106), is to simply state one of the foundational tenets of the field that has since been greatly refined. In the following, this chapter will provide readers with a broad overview of the various ways in which intermediality has been conceived in the interdisciplinary field of global theme park studies. It will do so by critically discussing significant contributions to the topic – journal articles, book chapters, edited collections, and monographs – from over 50 years of research on theme parks from a large variety of disciplines. The discussion will be structured according to the three categories introduced (in spirit, if not in name) by Schickel in 1968: plurimediality, transmediality, and paramediality. Both on the macro- and on the micro-level, the discussion will mostly follow a chronological order to showcase the broad development of the debates, also and especially against the background of the changing paradigms of theme park research.

Plurimediality With respect to the plurimediality of theme parks, scholars have investigated both the direct and the indirect participation of individual media in the parks’ complex semiotic or medial mix – that is, in Werner Wolf’s terminology, scholars have examined both the theme park’s “multimediality” or “plurimediality” as well as its various “intermedial references” (see Wolf 2007: 254). The fact that film and architecture have played particularly prominent roles in this subfield of intermediality in theme parks is certainly no accident: not only did one of the first and most influential theme parks – namely, Disneyland in Anaheim – emerge from the world of Hollywood, with Disney recruiting many of the park’s original designers from the staff of his own movie studio. Perhaps somewhat inevitably, these filmmakers cum “imagineers”6 would leave their cinematic imprint on the park’s overall layout and design: as many critics have pointed out, Disney’s designers did not think as architects; they were filmmakers. And what they designed was not a group of buildings or a park but an experience. They thought in very literally cinematic terms as they designed the place as a movie that could be walked into (Hine 1986: 151)

In turn, scholars such as Richard V. Francaviglia (“Main Street Revisited,” 1974; “Main Street, U.S.A.: The Creation of a Popular Image,” 1977; “Main Street U.S.A.: A Comparison/Contrast of Streetscapes in Disneyland and Walt Disney World,” 1981; and Main Street Revisited: Time, Space, and Image Building in Small-Town

A portmanteau of “imagine” and “engineer,” “imagineer” is Disney’s term for its in-house park designers.

6

1096

F. Freitag

America, 1996) and Karal Ann Marling (“Disneyland, 1995: Just Take the Ana Freeway to the American Dream,” 1991; and especially Designing Disney’s Theme Parks: The Architecture of Reassurance, 1997) have ensured that the study of theme park architecture has figured conspicuously in critical debates almost from the very beginning. These two strands of research have recently been fused in Dave Gottwald’s work on Disneyland and the cinematic aspects of its (landscape) architecture (see “The End of Architecture: Theme Parks, Video Games, and the Built Environment in Cinematic Mode,” with Greg Turner-Rahman, 2019; and “Cinematography in the Landscape: Transitional Zones in Themed Environments,” with Benjamin George, 2020). Yet critics have pointed out the direct and indirect participation of other media in the theme park’s plurimedial mix as well. In his various contributions, Francaviglia is primarily concerned with the Main Street, U.S.A. sections of the American Disney parks and the way their overall layout and façade design relates to real, historical main streets in the USA, which he has also extensively researched in Main Street Revisited (1996) – he even identifies real-life buildings that inspired individual façades in the theme parks (see 1996: 148–49). At the same time, however, Francaviglia is also interested in the more general representational strategies used to transform real-life main streets into carefully designed theme park spaces and, in turn, in the reception of Disney’s Main Streets in real-life built environments. Disneyland, he already notes in “Main Street, U.S.A.: The Creation of a Popular Image” (1977), “provides students of popular culture with a clear example of how the American landscape is idealized, abstracted, and, in this case, rendered into commercially viable symbolism” (19–20). More specifically, Francaviglia views Disney’s Main Street as a nostalgic symbol of prosperity and community: In Disney’s Main Street USA, architecture becomes the façade that creates the impression that all was right with the world in the small town at the turn of the century; it implies that commerce (and merchants) thrive along Main Street, and that society and a community are working together in harmony. (1996: 156)

This vision of main street was subsequently translated back into non-theme park environments, Francaviglia argues, via shopping malls and especially downtown revitalization programs, a process he refers to as the “Disneyfication of Main Street” (1996: 169). Overall, then, Francaviglia’s research uses the example of Disney’s Main Street, U.S.A. to show how architecture has served as a medium for the creation and perpetuation of popular images of urbanity both within and beyond the (Disney) theme park. Karal Ann Marling, too, is interested in the real-life sources of Disneyland in general and of its Main Street, U.S.A. section in particular, in the transformation of these sources into the theme park’s symbolic landscapes, and in the subsequent impact of theme park architecture on urban planning and design. Identifying Disneyland’s Main Street as Kansas Street in Walt Disney’s hometown of Marceline, Missouri, the way “Walt Disney remembered it from a distance of half a century”

42

Intermediality in Theme Parks

1097

(1991: 194), Marling describes the “land” as “a memory first softened by the blur of nostalgia and then coaxed into a three-dimensional existence, just that way – pristine and eternally lovely – in the architecture of Disneyland” (195–6). Like the park as a whole, this “city of dreams” (1997: 176) with its “spatial reassurances and human scale” (143), Marling argues, not only served as a critique of the surrounding city of Los Angeles but would later also inspire the festival marketplaces of James Rouse and the “new urbanism” of Seaside, Florida, and Disney’s own Celebration, Florida (170–71). However, Marling is also concerned with the creative process of designing (Disney) theme park architecture, characterizing it as “picture architecture” (“Everything starts from an illustration, a concept picture, a slice of an animation storyboard showing characters, or settings, or props, or, best of all, their interaction in space and time,” 1997: 77) and as a collaborative effort (“Main Street is not an architectural masterpiece attributable to a single genius [. . .]. Like a Hollywood film, it is a collaborative effort between Walt Disney and his art directors,” 1997: 61–62). Finally, and notwithstanding the reference to Hollywood film, Marling uses her discussion of the direct contribution of architecture to the theme park media mix to also examine the indirect contribution of (or intermedial reference to) television, comparing the park’s various sections to different television channels: Like a Wednesday-night viewer, the tourist standing in the Hub at Disneyland was presented with a whole range of possibilities. Like an impatient viewer in front of the set, the tourist could switch from one channel – oops! land – to another in just a few steps. (1997: 74)

A very similar point had already been made by Judith A. Adams in The American Amusement Park Industry (1991) with respect to theme parks in general: The sector structure [of theme parks] can be seen as a reflection of the segmented nature of television entertainment. [. . .] Just like television programs, which last for a half hour or an hour, then make way for a different setting and dramatic situation, the theme park creates a series of startingly diverse environments to be experienced in a single day. (109–10)

And the very concrete structural parallels between Disneyland (the park) and Disneyland (the TV show; 1954–58), with the latter’s segments bearing the same titles as the former’s sections, would later inspire Jay P. Telotte in Disney TV (2004) to describe both as part of a larger “Disney media project,” a transmedial “vision machine” (9; see also below). Yet the majority of critics have been concerned with the theme park’s intermedial references to movies rather than to television. Marling had insisted that the “switch” from one theme park section to the next was too abrupt to be considered cinematic: [I]n the movies, the experience is continuous and unbroken, but in Disneyland, it is discontinuous and episodic, like watching television in the privacy of one’s own home – each ride a four- or five-minute segment, slotted in among snacks, trips to the rest room, and “commercials” in the form of souvenir emporia. And it is always possible to change the channel. (1991, 205)

1098

F. Freitag

Somewhat ironically, it is precisely the transition zones between different themes, as well as other nodal points in the theme park layout such as the entrance and decision points,7 that later studies have identified as implicitly referencing cinematographic techniques. Thus in his “‘Like Walking into a Movie’: Intermedial Relations between Disney Theme Parks and Movies” (2017), Florian Freitag not only discusses the use of projection techniques and moving images in (Disney) theme park rides and shows (i.e., the direct role of cinema in the parks’ media mix), but also the implicit references to the medium of film that can be found in individual attractions as well as in the parks at large, from the filmic “cuts” evoked by sharp turns in dark rides to the “sequential” design of the park entrance: “the experience of entering Disneyland and walking down Main Street to Central Plaza, through formal imitation, implicitly references the experience of going to a movie theater and watching the beginning of a movie (an establishing shot)” (715; see also Gottwald and Turner-Rahman 2019). Similarly, in “Cinematography in the Landscape: Transitional Zones in Themed Environments” (2020), Dave Gottwald and Benjamin George identify various filmic editing techniques that have been imitated in transition zones between differently themed areas through careful landscape design: cuts, fades, ellipses, wipes, irises, and dissolves. With respect to the iris, for example, Gottwald and George argue that whereas it may have gone largely out of fashion in film, in “thematic design, its translation lives on in the extensive use of tunnels and bridge underpasses to stage emerging viewpoints. Texts in landscape architecture admire space modulation as expressed by the compression and release afforded by tunnels” (57). At the same time, providing detailed insights into the original landscape design of Disneyland in Anaheim and the people involved in it, Gottwald and George also put landscape architecture on the map of the theme park’s plurimedial mix. The role of at least three other “traditional” media within this mix have been examined in greater detail by theme park critics so far: music, performance, and language. Regarding music, a special section in Ethnomusicology Forum on “Music and the Disney Theme Park Experience” (see Carson 2004; and Nooshin 2004) as well as Gregory Camp’s “Mickey Mouse Muzak: Shaping Experience Musically at Walt Disney World” (2017) provide musicological perspectives on the use and experience of music in Disney parks, while Freitag’s 2017 interview with artist David Tolley, who scored the background music for the Discoveryland section of Parc Disneyland in Paris, offers insights into the creative process of composing music specifically for theme parks. Whereas Camp once again highlights the impact of cinema on theme parks and their music by identifying “specific musical tropes” shared by Disney park music and film music (65), Carson, Nooshin, and Freitag examine how music contributes to communicating the atmospheres, worldviews, and (temporal) settings of specific rides and themed “lands,” from the nostalgia of Main Street, U.S.A. and the exoticism of Epcot’s various pavilions (Carson) to the

7

Decision points are nodal points in a theme park where visitors are offered a choice between two or more options (see Younger 2016: 298).

42

Intermediality in Theme Parks

1099

imperialism of “it’s a small world” (Nooshin) and the steampunk future of Discoveryland (Freitag). Carson, moreover, offers a typology of theme park music (229) and also reflects on the role of music-related products sold in the parks (such as CDs) as paramedia (233–34; see also below). The contribution of performance to the theme park’s plurimediality has only recently been examined by scholars such as Ariane Schwarz (“Spieglein, Spieglein an der Wand – wer ist das echte Schneewittchen im Land?” 2016; “Staging the Gaze: The Water Coaster ‘Poseidon’ as an Example of Staging Strategies in Theme Parks,” 2017) or Jennifer A. Kokai and Tom Robson (Performance and the Disney Theme Park Experience: The Tourist as Actor, 2019; “Disney during COVID-19: The Tourist and the Actor’s Nightmare,” 2022). Following the tenets of Performance Studies and in the tradition of Finch’s and Smith’s earlier characterization of theme parks as “total theater” and a Wagnerian “Gesamtkunstwerk” (see above), Schwarz and Kokai and Robson use performance as a an analytical lens, viewing the theme park as a performance by visitors, robots, and employees on the “stage” of the park landscape (see Schwarz 2017; Kokai and Robson 2019) and thus responding to the call for “triangulated” or “integrated” theme park studies. Yet they also examine the logistics and functions of “actual” performances by performance artists in theme parks, with, for instance, Kokai and Robson discussing the impact of COVID-19 on stage shows at Walt Disney World and the more general role of live entertainment as providing structure to the theme park visit and offering an interactive element to visitors (see Kokai and Robson 2022). The use of language as a medium in the theme park has mainly been discussed by two scholars so far, Filippo Carlà-Uhink (Representations of Classical Greece in Theme Parks, 2020) and Florian Freitag (“‘This Way or That? Par ici ou par là?’: Language in the Theme Park,” 2021b). It is certainly no coincidence that CarlàUhink and Freitag both focus on Europa-Park in Rust (Germany): with its individual sections themed to various European countries, this park heavily relies on foreign languages in the shape of written signs and recorded dialogues to convey the idea of a particular nation to its visitors. For example, Europa-Park’s Greek section, Carlà-Uhink shows, features multiple signs in both ancient and modern Greek, but also in English and German written in either “simulation” typefaces or Greek letters (where e.g., Σ needs to be read as E; see 49). Freitag, in turn, suggests adopting the “linguistic landscape” or “semiotic landscape” approach from sociolinguistics to study (written) language in the theme park as it closely corresponds to the notion of the theme park as a plurimedial medium: “Drawing on various interacting semiotic systems, art forms, or media to convey a particular sense of place, story, or theme, theme parks constitute a prime example of such multimodal or semiotic landscapes in which language in the shape of written signs collaborates in meaning-making” (78). These and other recent studies – such as Vanessa Schwartz’s Jet Age Aesthetic: The Glamour of Media in Motion (2020), which examines the interaction between transport and communication media in the theme park (11) – have greatly contributed to the prominence of plurimediality in recent theme park scholarship.

1100

F. Freitag

Transmediality Whereas the previous section was concerned with analyses of the direct or indirect participation of specific media in the plurimedial spectacle of the theme park – that is, with phenomena of intermediality “that can be documented within a given ‘work’” or “intracompositional” cases of intermediality (see Wolf 2007: 253) – the following two sections will focus on phenomena where two or more distinct artefacts are intermedially linked to each other. One of the most widespread and popular cases of such “extracompositional” intermediality (253) is that of adaptation, the “deliberate, announced, and extended revisitation” of a prior work (the adapted work) in a later work in a different medium (the adaptation; see Hutcheon and O’Flynn 2013: xvi). Several critics have investigated theme park adaptations of specific literary and other works or genres on a variety of levels, ranging from the individual ride and the themed section to entire parks or the industry as a whole. In “‘Being Inside the Movie’: 1990s Theme Park Ride Films and Immersive Film Experiences” (2019), for example, Angela Ndalianis and Jessica Balanzategui discuss rides such as Back to the Future: The Ride (Universal Studios, Florida, 1991), Terminator 2 3D: Battle across Time (T2-3D; Universal Studios, Florida, 1996), and The Amazing Adventures of Spider-Man (Universal’s Islands of Adventure, Florida, 1999), which “revisit” specific movies and simultaneously incorporate “screen technologies into traditional ride formats” (20), thus characterizing the “ride film” as both a plurimedial and an adaptive media artefact. Also focusing on ride adaptations of movies, Suzanne Rahn (2000) generally distinguishes between rides that retell the story of a movie and others that use the characters and the setting of a movie to tell a new story and then examines Disneyland’s Snow White’s Scary Adventures dark ride as an example of the latter case. Stijn Reijnders and Abby Waysdorf similarly downplay the role of plot in theme park adaptations, arguing that Universal’s Harry Potter-themed area, called the Wizarding World of Harry Potter, is rather “understood by its visitors as an adaptation of the series” (2018, 173) because of its general “ability to represent the storyworld in physical space” (180). Yet for “all the emotionality of ‘being there,’” Reijnders and Waysdorf report, the visitors they interviewed for their study also appreciated and remained aware of all “the work put in to making the experience” (185) and thus consciously experienced the themed area as an adaptation. Marty Gould and Rebecca N. Mitchell (2010), in turn, explore Dickens World (Chatham, UK; 2007–2016), a short-lived indoor theme park based on the literary oeuvre of nineteenth-century British writer Charles Dickens, as a showcase of “adaptive techniques,” from structure, nostalgia, and spectacle to narrative and commodification (164). In Fairground Attractions: A Genealogy of the Pleasure Ground (2012), finally, Deborah Philips traces the successive adaptation of such (literary) genres as the fairy tale, the Western, or science fiction from folk history and oral tradition to (illustrated) chapbooks, dioramas and panoramas, theater plays, cinema, and eventually to theme parks. Rather than as an adaptive medium, however, most scholars have conceived of the theme park as one of multiple nodes within transmedial networks that develop

42

Intermediality in Theme Parks

1101

specific story-worlds across different media in order to maximize profits. This specific perspective has arguably been fostered by developments both within the theme park industry and in media theory. In the 1950s and the 1960s, Disney’s “machine” with “its parts – movies, television, book and song publishing, merchandising, Disneyland – [that all] interlock and are mutually reciprocating” could still be seen as something that was unique to the company and that “made it superior to all its competitors” (Schickel 1968: 18), and as something that could only be expressed through traditional metaphors of mechanical production (“machine”). By the mid-1990s, however, the competitors, if not the terminology, had caught up: in “The Theme Park: Global Industry and Cultural Form” (1996), Susan G. Davis observes a dramatic concentration of ownership in the (American) theme park sector during this time: “a striking development of the early 1990s has been the acquisition of large theme park chains by media conglomerates” such as MCA, Viacom, Blockbuster, and Time-Warner (405). Davis offers three explanations: theme parks’ short-term profitability; theme parks as an entryway into the mass tourism sector; and, most importantly, theme parks as offering seemingly limitless opportunities to cross-promote goods and imagery produced in other parts of the conglomerate or acquired elsewhere. [. . .] The cross-promotional possibilities have been evident for at least three decades, but only now, with the recent merging of huge media companies, has the full promise become clear. (406)

More than 20 years later, Reijnders and Waysdorf would describe Davis’s argument as “as a version of ‘media convergence’ before this term was in vogue” (2018: 176). The term did come in vogue in 2006 with Henry Jenkins’s Convergence Culture: Where Old and New Media Collide.8 Given her primarily economic perspective on theme parks, one may wonder whether Davis would indeed have used the term “convergence” or whether she would have stuck to the term she actually did use, namely, “synergy” (1996: 406).9 A third term popularized by Jenkins’s Convergence Culture, but also by the publications of Marie-Laure Ryan, is that of “transmedia

In his book, Jenkins defines “convergence” as a recent process that involves both dramatic changes in media production (technological changes and changes in media ownership) and in media consumption (2006: 15–16) and that denotes “the flow of content across multiple media industries, and the migratory behavior of media audiences who will go almost anywhere in search of the kinds of entertainment experiences they want” (2). Hence, for Jenkins “convergence” is a term “that manages to describe technological, industrial, cultural, and social changes” (2–3). 9 Davis uses “synergy” more or less synonymously with cross-promotion, an economic strategy in which each “product [adds] value to the others” and where “marketing, advertising, and content” overlap (1996: 406). Similarly, Jenkins identifies “synergy” as an industry concept to refer to “the economic opportunities represented by [the media company’s] ability to own and control all of those manifestations [of content across different delivery systems]” (2006: 19). 8

1102

F. Freitag

storytelling.”10 Other terms that have been used to describe transmedial networks involving theme parks are Bolter and Grusin’s “(networks of) remediation”11; Telotte’s “integration,” an economic “sense of combined purposes” (2004: 4); and Dan Herbert’s “industrial intertextuality,” a concept that takes into account both the production and reception side, that is, “the work that occurs in the minds of the readers and viewers” (2017: 25). Whatever terminology they have used, critics have generally focused on transmedia networks that involve large destination parks owned by international multimedia entertainment conglomerates such as Disney and Universal, even though the practice of incorporating theme parks into such networks has also spread to smaller and independently run theme parks and even though the destination parks not only participate in transmedia franchises that have been developed by their parent company (as in the case of Disney’s Pirates of the Caribbean, where the franchise even originated in a theme park) but also work with licensing (as in the case of Universal’s Harry Potter-themed areas, which were designed and have been operated under an exclusive license from Warner Bros., who produced the movies). Moreover, scholars have used their investigations of transmedia networks to further specify either the particular role of theme parks within these networks or to examine the workings of transmedia storytelling in general. Cases in point are Rebecca Williams’s work on Theme Park Fandom: Spatial Transmedia, Materiality and Participatory Cultures (2020) and Sabrina Mittermeier’s “Theme Parks: Where Media and Tourism Converge” (2020) on the one hand, and Jason Sperb’s study on Disney’s Most Notorious Film: Race, Convergence, and the Hidden Histories of Song of the South (2012) and Bobby Schweizer and Celia Pearce’s “Remediation on the High Seas: A Pirates of the Caribbean Odyssey” (2016) on the other hand. Focusing more on the reception of transmedia franchises, Williams develops the concept of “spatial transmedia” in order to “account for these moments of narrative extension and world-building that take place within specified rooted locations” such as theme parks (2020: 12). Due to their rootedness, spatiality, and materiality, Williams notes, theme parks in turn allow for what she refers to as “haptic fandom,” a concept that underlines “the centrality of the physical and experiential in understanding fan engagement with(in) the contemporary transmedia spaces of the theme park” (13). Likewise, Mittermeier argues that theme parks have become “spaces of fandom, where visitors engage in

Seeking to put narratology – i.e., the formal study of narrative – back “on the transmedial and transdisciplinary track,” Ryan defines “transmedial storytelling” as a practice in which a “core of meaning may travel across media, but its narrative potential will be filled out, actualized differently when it reaches a new medium” (2005: 1). Likewise, according to Jenkins, a transmedia story “unfolds across multiple media platforms, with each new text making a distinctive and valuable contribution to the whole” (2006: 95–96). 11 Bolter and Grusin are interested in the ways in which media interact: what is new about newly emerging media, they argue, “comes from the particular ways in which they refashion older media and the ways in which older media refashion themselves to answer the challenges of new media” (2000: 15). Accordingly, they use “remediation” to refer to “the formal logic by which new media refashion prior media forms” (273). 10

42

Intermediality in Theme Parks

1103

cosplay and feel a sense of community, similar to that of fan conventions” (2020: 30). By contrast, Sperb is more interested in the management of and thus in the production side of transmedia franchises and offers an intriguing case study of Disney’s Song of the South, which originated in 1946 with the live-action/animated musical film Song of the South (itself an adaptation of Joel Chandler Harris’s 1886 story collection Uncle Remus: His Songs and His Sayings) and from 1989 onwards appeared in several of Disney’s theme parks in the shape of the log flume ride Splash Mountain. Sperb describes the ride as an example of “strategic remediation,” a process in which media companies use only carefully selected parts of a given text for further distribution across media platforms, with the result that “intellectual property diffuses across the dispersed texts of media convergence culture” (2012: 21). As Sperb explains, Splash Mountain’s designers not only completely dropped the live-action frame narrative of the movie (which had been greatly expanded from the literary original, where it consisted of just one short paragraph), they also changed what had come to be regarded as the most offensive element of the animated core segments, namely, the “Tar Baby” storyline (see Mauro 1997; Sperb 2012: 163–164). Whereas transmedia storytelling is usually thought of as expanding and extending a story-world, Sperb argues that Disney has employed “transmedia dissipation” to purge the franchise of its most controversial elements and to thus keep it palatable—and profitable—in the post-Civil Rights movement era (see also Bringardner 2019: 116).12 Similarly, Schweizer and Pearce (2016) have used the example of Pirates of the Caribbean to show how rather than continuously expanding, a story-world may also undergo a “deliberate integration,” a process in which “a conscious effort is being made to construct a cohesive story-world from a vast collection of components across multiple media” (99). In their case studies, Sperb as well as Schweizer and Pearce also revisit the early days of Disneyland, where most scholars have located the beginnings of transmedia networks involving theme parks – hence, 1950s and 1960s Disneyland has prominently served to illustrate not only the plurimedial (see above) but also the transmedial aspects of theme parks. Sperb, for example, attests the Walt Disney Company a “trailblazing” role in the history of transmedia storytelling and synergy (2012: 24) and describes the 1950s as an “early moment of literal convergence” when “the word ‘Disneyland’ simultaneously signified (1) a television show on the ABC network, (2) a theme park in Anaheim, California, (3) a profitable record company, and (4) a series of successful books” (90). Whereas for Sperb and Schweizer and Pearce, however, this moment merely constitutes a prehistory for later developments, the transmedial beginnings of Disneyland provide the very context for Christopher

12

Yet, in 2020, in what seems to be a direct reaction to the Black Lives Matter movement and ongoing discussions about structural racism in the USA and abroad, Disney announced that it would re-theme the Splash Mountains in its US-based parks to the animated, Louisiana-set The Princess and the Frog (2009). Hence, it seems that transmedia dissipation cannot entirely erase the story elements it seeks to leave behind.

1104

F. Freitag

Anderson’s and J.P. Telotte’s studies of the Disney studio’s early forays into television in Hollywood TV (1994) and Disney TV (2004), respectively. Both Anderson and Telotte discuss the spread of story-worlds across media in general and across television and the theme park in particular but also emphasize the promotional function of television vis-à-vis the park. Thus, Telotte describes Disney’s strategy as a “vision machine” (2004: 9) in which story-worlds and genres were distributed across media and in which the individual media artefacts also sought to advertise each other. For example, not only did the structure of the park mirror that of the TV show (see above) and characters such as Davy Crockett made appearances both on screen and on site (2004: 34), Disney also used the TV show to “update viewers on the state of Disneyland, later to introduce new rides and attractions or herald the openings of companion parks” and thus to directly promote the theme parks (74). Likewise, Anderson considers Disney’s decision to structurally link the theme park and the TV show as “one of the most influential commercial decisions in postwar American culture” (134) and notes: In effect, Walt identified the program with the park in order to create an inhabitable text, one that would never be complete for a television-viewing family until they had taken full advantage of the postwar boom in automobile travel and tourism and made a pilgrimage to the park itself. (152–53)

Hence, Anderson and Telotte simultaneously identify the Disneyland TV show as a central node in Disney’s transmedial network as well as a promotional vehicle for the park. The latter perspective serves as a segue to the second type of “extracompositional” intermediality and, at the same time, the third subcategory of intermediality to be discussed here: namely, theme park paramediality.

Paramediality The term paramediality (or paratextuality)13 has only quite recently started to be used in connection with theme parks. In Jonathan Gray’s Show Sold Separately: Promos, Spoilers, and Other Media Paratexts (2010), theme parks themselves are identified as paratexts for other media texts (namely, in the example of “a new ride at an amusement park” which promotes e.g., a movie; see 29); likewise, in her Theme Introduced by French literary critic Gérard Genette in his Palimpsestes (1982), “paratextuality” originally described “verbal or other productions, such as an author’s name, a title, a preface, illustrations” that accompany and surround a literary text “in order to present it, in the usual sense of this verb but also in the strongest sense: to make present, to ensure the text’s presence in the world, its ‘reception’ and consumption in the form (nowadays, at least) of a book” (Genette 1997: 1). Genette himself later opened the concept to other media, granting that “some, if not all, of the other arts have an equivalent of our paratext,” naming music, sculpture, painting, and film as examples (407). This idea was then taken up in 2010 by media scholar Jonathan Gray in his Show Sold Separately, where he argues that media paratexts like ads, previews, trailers, posters, billboards, bonus materials, and spin-offs “establish frames and filters through which we look at, listen to, and interpret the texts that they hype” (3).

13

42

Intermediality in Theme Parks

1105

Park Fandom (2020), Williams describes the themed food and drinks sold in theme parks as “hyperdiegetic paratexts” for the media franchise they are based on (163; emphasis original). It was not until 2022, in fact, that Freitag has argued that like other media texts, theme parks, too, are regularly accompanied by a wide array of paramedia: ads, signs, announcements, maps, leaflets, videos, websites, apps, and other representations of the theme park or its parts that are produced by the park itself and that serve as a medial interface between the park landscape and its visitors, providing the latter with “frames and filters” or “scripts” for how to experience the park (2022). Of course, the notion of theme park paramediality, that is, the idea that the theme park seeks to control the ways in which it is anticipated, experienced, and remembered through a variety of ancillary media texts, has been circulating for much longer already. As early as 1992, for example, Richard Findlay noted that most visitors arrive at Disneyland already “school[ed] in the Disney way” through extensive advertising (79; see also Mitrasinovic 2006: 119). Since then, several critics have provided in-depth analyses of particular medial artifacts that could be classified as paramedia (albeit without identifying them as such and thus also without considering them as part of the larger field of theme park paramediality) while some others have sought to examine the entire paramedial arsenal of one specific (part of a) park. For example, some in-depth studies of individual parks have also included detailed looks at the latter’s marketing materials. Vinyl Leaves: Walt Disney World and America (1992), Stephen M. Fjellman’s magisterial monograph of Disney’s Florida-based resort, contains an entire chapter on “Marketing the Magic Mall,” in which the author discusses both Disney’s promotional strategies for its Orlando parks as well as, anticipating Williams’s classification of food and drinks as paramedia (see above), merchandising in the parks. Following overviews of the various company subdivisions involved in the promotion of Walt Disney World (152–56), Fjellman argues that until Frank Wells and Michael Eisner joined the company in 1984 as COO and CEO, respectively, Disney (also) used a “‘Tom Sawyer’ approach to marketing: Get the other guys to do it by convincing them that it’s a good deal” (159). As an example, Fjellman cites the opening of EPCOT in 1982, for which a “twenty-four-page magazine supplement,” jointly funded by Disney and some of the corporations that had signed sponsorship deals with EPCOT, was distributed in such periodicals as Newsweek, Time, Life, or People (159). Post-1984, Fjellman notes, the company has increasingly relied on print, radio, and television advertisements “placed by Vista Advertising (an in-house arm of the creative support division) rather than by an exo-Disney company” to allow for more control over the “symbolic information” associated with the parks (160). Fjellman even complains about the “[m]edia blitzes [that] have accompanied the accelerated openings of new Disney theme parks” (160). Perhaps the most intensive of these “blitzes” came in the same year that Fjellman’s book was published: namely, the pre-opening marketing campaign for Disney’s new Parisian resort, to which Andrew Lainsbury has dedicated an entire chapter in his Once upon an American Dream: The Story of Euro Disneyland (2000). Apart from the medial variety of Euro Disney’s promotional material – which included print ads, TV commercials and shows, and even themed spaces and performances (in the shape of the “Espace Euro Disneyland” preview center and public appearances by Disney

1106

F. Freitag

characters) – Lainsbury particularly emphasizes how paramedia such as the park’s televised opening ceremony sought to “forg[e] an all-new identity for the Euro Disney Resort” (101). Perhaps in response to the virulent public debate about American cultural imperialism sparked by the announcement of the Euro Disney project (see Kuisel 2012: 167–69), the Grand Opening TV gala featured several segments that stressed the European roots of Disney in general and Euro Disney in particular, including a “film clip of Roy Disney on location in the small village of Isigny-surMer (from which the Disney name was derived) [that] dramatized his family connection with France” (101). In a subchapter of his history of New Orleans-themed spaces in Disney theme parks and resorts, in turn, Freitag (2021a: 160–64) examines the various TV specials dedicated to the openings of Disneyland’s New Orleans Square (1966), Pirates of the Caribbean (1968), and the Haunted Mansion (1970), arguing that each addressed a specific market segment: After Disney had targeted mature and primarily female visitors looking for high-end shopping and sophisticated dining in [the opening special for New Orleans Square] and a predominantly male audience looking for action and adventure in [the opening special for Pirates of the Caribbean], [the opening special for the Haunted Mansion] [sought to] position the [ride] – and, in fact, Disneyland in general – as an attractive destination for a gender-neutral and young (child and teenage) demographic. (164)

Besides television specials and ads (see Lam [2010] on Hong Kong Disneyland’s Halloween TV campaign), critics have also examined such quintessential theme park paramedia as souvenir maps and guides tours, apps, websites and social media. Interestingly, the “case study” approach that many scholars have taken to theme park paramediality, with individual essays each discussing one particular paratext in one particular medium, somewhat mirrors the structure of research on theme park plurimediality, where individual publications focus on the role of specific media in the theme park (see above). Thus in “Mapping the Happiest Place on Earth: Disney’s Medieval Cartography” (2012), Stephen Yandell looks at the large drawn souvenir maps sold at Disneyland from 1958 onwards, arguing that they “manipulate [visitors] to preserve an illusion of perfection” (24). Indeed, whatever visitors are not supposed to notice during their visit – from construction and maintenance works to backstage areas and the outside world – has simply been eliminated from these maps or hidden behind innocuous “groves of trees, the default symbol used [on theme park maps] to indicate off-limit spaces” (24). Thus, the maps’ “carto-reality,” which Yandell compares to that of Medieval mappaemundi, offers post-visit customers an idealized memory of the park that supports “the park’s simulation more effectively than the park itself” (25). At the same time, Yandell notes, the “same maps that tell us so insistently what to look at also generate a desire to peek at what is forbidden, out in the imperfect margins” (22) – a desire that has been commercialized and fulfilled through yet another paramedium: the guided backstage tour. Two critics have examined this particular paramedium in detail. In “The Walt Disney World Underground” (2005), David L. Pike focuses on the last part of a backstage tour of the Magic Kingdom at Disney’s Floridian resort, which leads visitors through the underground tunnels known as “utilidors.” Promising a look

42

Intermediality in Theme Parks

1107

behind the scenes, the visit to the utilidor system rather complements a visit to the park above, Pike argues, not just because “one space is predicated upon the other” (56), but also because both spectacularize the concept of the vertical city: The backstage tour takes us no closer to the real life of the Disney city [. . .]; what it does provide is a sense of the totality of the ideology, all the distorted pieces rather than just the half of them available to the spectator above. (56)

Likewise, in “Behind the Behind the Scenes of Disney World: Meeting the Need for Insider Knowledge” (2012), Mathew J. Bartkowiak emphasizes the staged character of the backstage tours and views them as Disney’s successful attempt to bring potentially deviant readings of the parks back under its control: “The successful reincorporation of the challenging of the fantasy Disney has created in these behind-the-scenes tours marks the fact that, yes, the hegemonic impulse is still alive” (957). Yet the tours do not only address Disney skeptics, but also fans. Indeed, Bartkowiak compares the tours to other “behind-the-scenes” paramedia such as coffee-table books or DVDs on theme park design and operation by arguing that they commercialize supposed insider knowledge that circulates as cultural capital among fans: “Depending on interests, there is cultural capital to be amassed for [. . .] the Disney fan garnering insider knowledge” (946–47). “Sneak peeks” of, for example, the theme park design process have also been frequently offered on the blog of the Christian young earth creationist theme park Ark Encounter (Williamstown, Kentucky), which James S. Bielo has examined in one of the extremely rare studies of theme park paramediality about parks other than Disney (2016: 5). In the 2010s, theme park critics like Bielo have increasingly turned to digital paramedia, much like the parks themselves. Carol J. Auster and Margaret A. Michaud (2013), for example, have examined the photographs on the official websites of Disney’s “Magic Kingdom” parks in order to analyze the portrayal of race and gender with respect to the roles of the individual human beings in the theme park (visitors vs. employees) and the roles of the particular settings (i.e., the various themed sections of the parks) and divisions (e.g., rides, entertainment, dining etc.). Regarding visitors, they found that while there is a general “preponderance of samesex pairings in parent-child combinations in the images,” “the images of some theme parks displayed more racial diversity among their guests than others, in some images, individuals of different races were shown interacting whereas in others they were not” (1). Hence, whereas images promoting Frontierland frequently showed fathers and sons, images of Fantasyland more often depicted mothers and daughters; and overall the visitors depicted in the photographs appear to have been chosen to portray “the racial group representing the statistical majority in each country in which the Disney resort was located” (11). Regarding employees, in turn, Auster and Michaud found that, for instance, the majority of food servers shown in images were female, especially so on Tokyo Disneyland’s website, and that as in the case of visitors, certain themed sections were promoted in a heavily gendered way. Whereas blogs and websites are usually consumed before or after the visit to the theme park and thus primarily constitute, in Freitag’s typology, “entryway” or “in

1108

F. Freitag

medias res” paramedia, respectively (2022), visitors often use “in situ” digital paramedia such as apps or social media while they are in the park as well. In “Linking Emotion and Place on Twitter at Disneyland” (2018), for example, Park, Kim, and Ok combine social media analytics and GIS analysis to examine the emotions expressed in Twitter posts sent from Disneyland in California. While their study (also) captures postings by regular visitors rather than exclusively focusing on the company’s own tweets, the fact that “Disney has increasingly tried to engage with bloggers and other social media personalities” (Mittermeier 2020: 32) illustrates that the lines have become blurry. This also applies to Instagram, of course, as Arthur Soto-Vásquez has shown in his 2021 study on the intersections between the popular photo and video sharing site and Disneyland: A few Instagram users in this study noted that their posts were sponsored, suggesting that Disney paid for their travel. [. . .] Savvy promotion on the part of Disney, this practice allows for a more authentic-seeming marketing of the parks to consumers. (12)

Yet Soto-Vásquez also speculates about how social media formats and their medial exigencies may influence the parks themselves, suggesting a feedback loop that leads from the text to the paratext and back to the text: What might an attraction or theme park built around the logic of social media look like? It would probably focus less on dark or fast-moving attractions where photography is difficult. Instead, spaces where one can wander around and take photos with colorful sets might take priority. (16)

In “Digital Disneyland” (2021) Soto-Vásquez elaborates on his prediction: “The future of theme parks will be increased integration between mobile apps, social media representation, and perhaps gamified integration with attractions” (n.p.). Theme park apps that allow visitors to navigate the park, make reservations for rides and restaurants, and virtually queue for experiences, have significantly altered the dynamics of the theme park visit, sometimes requiring visitors to invest a significant amount of “fun work” (Kagelmann 2004: 175; own translation) or “anticipatory labor” (Williams 2020: 67–99); and according to Soto-Vásquez, this trend “will likely continue to increase under social distancing protocols due to the [COVID-19] pandemic” (2021: n.p.). Another interesting link between COVID-19 and theme park paramediality has been provided by Williams (2021), who considers paramedia as one way for fans and researchers to keep in touch with the objects of their fandom and studies even when the latter are closed down – a strategy that may be continued even beyond the pandemic, she suggests, in the interest of sustainability: One way forward for those who research mediated places or fan tourism is to reimagine a more ethical and ecologically friendly way of conducting this work, ensuring that such journeys are made carbon-neutral or that potential environmental harm is offset. In this landscape, physical visits to sites of fan tourism and pilgrimage (whether theme parks, museums, filming locations, and beyond) may no longer be possible, for both environmental and health reasons. (142)

42

Intermediality in Theme Parks

1109

It is thus not only the form(s) of theme park paramediality that has dramatically changed over the years, but also their (perceived) function: whereas formerly theme park paramedia were examined according to their capacity to entice a physical visit to the site, they are now also considered as an alternative to travelling (see also Moulton 2020).14

Conclusion As studies on the paramediality of theme parks illustrate, recent research on intermedial aspects of theme parks has continued to both perpetuate and challenge central critical paradigms of the field: on the one hand, scholars interested in theme parks paramedia have increasingly adopted “triangulated” or “integrated” approaches that take into account not just the parks themselves but also their employees and especially their visitors, whether by investigating how the latter are represented on theme park websites or by examining how they, in turn, and perhaps with funding from the park, represent the sites on social media. On the other hand, the idea that theme parks may increasingly reflect the “logics of social media” perhaps to some extent also neglects the fact that from the beginning, theme parks have been designed for the camera eye – by intermedially referencing cinema, for example, but also by identifying “photo spots” (suggested locations for taking souvenir pictures) or offering on-ride photos (even on “dark or fast-moving attractions”) – and thus continues theme park studies’ traditional preference for synchronic over diachronic, historical perspectives. To be sure, while scholars are acutely aware of the potential shortcomings of specific research traditions, they also remain bound to the realities of the industry and to the practical limits of doing on-site research: Williams, for example, agrees with David Bell both on the fact that it is “important [. . .] to move away from the shadow of Disney” and on the fact that it is important to simultaneously acknowledge “the shadow that the Mouse’s ears still cast over theme parks design, philosophy, management and experience” (Bell qtd. in Williams 2020: 43), while at the same time she also cites “pragmatic” reasons for her choice to focus on Western destination parks owned by multimedia conglomerates: “these are the parks most visited by the researcher in the course of undertaking this study” (Williams 2020: 42). There is no doubt, however, that innovations in the field of digital theme park paramedia (also and especially during and in the aftermath of the COVID-19 pandemic) as well as the increasing involvement of all kinds of theme parks in transmedial networks have generated new and exciting material for investigations into intermedial aspects of theme parks. What these developments also offer are opportunities for overcoming more of the traditional critical paradigms of the field 14

The impact of the COVID-19 pandemic on the theme park industry in general and on theme park paratexts in particular are still being examined. In addition to Williams (2020) and Freitag (2020), see also the articles collected in Anton Clavé and Freitag (2022).

1110

F. Freitag

(e.g., the focus on large Western parks), for introducing new methods and approaches into theme park studies (e.g., social media analytics), and for establishing new research collaborations – collaborations across disciplinary boundaries (see Anton Clavé et al. 2023), collaborations between scholars and the industry (“creative research”; see Carlà et al. 2016: 109), and collaborations between scholars and fans (“co-creative research”). Apart from showing how much the theme park and its plurimediality, transmediality, and paramediality have (or, in at least some cases, haven’t) changed over the past seventy years, such innovations in research design and methodology would contribute to keeping the field of intermediality as integral and paradigmatic to theme park studies as it has been from the very beginning.

References Adams, Judith A. 1991. The American amusement park industry: A history of technology and thrills. Boston: Twayne. Anderson, Christopher. 1994. Hollywood TV: The studio system in the fifties. Austin: University of Texas Press. Anton Clavé, Salvador. 2007. The global theme park industry. Wallingford: CABI. Anton Clavé, Salvador, and Florian Freitag. 2022. Theme parks and COVID-19. Journal of Themed Experience and Attractions Studies 2.1. https://stars.library.ucf.edu/jteas/vol2/iss1/. Anton Clavé, Salvador, et al. 2023. Key concepts in theme park studies: Understanding tourism and leisure spaces. Cham: Springer. Auster, Carol J., and Margaret A. Michaud. 2013. The internet marketing of Disney theme parks: An analysis of gender and race. SAGE Open 3 (1): 1–16. Baker, Carissa. 2018. Summary of exploring a three-dimensional narrative medium: The theme Park as “de sprookjessprokkelaar,” the gatherer and teller of stories. Journal of Themed Experience and Attractions Studies 1(1): 1–9. Bartkowiak, Mathew J. 2012. Behind the behind the scenes of Disney world: Meeting the need for insider knowledge. The Journal of Popular Culture 45 (5): 943–959. Baudrillard, Jean. 1994 [1978]. The precession of simulacra.” In Simulacra and simulation, ed. Jean Baudrillard, 1–43. Trans. Sheila Faria Glaser. Ann Arbor: The University of Michigan Press. Bielo, James S. 2016. Materializing the bible: Ethnographic methods for the consumption process. Practical Matters Journal. http://practicalmattersjournal.org/2016/05/04/materializing-thebible/. Bolter, Jay David, and Richard Grusin. 2000. Remediation: Understanding new media. Cambridge, MA: MIT Press. Bringardner, Chase A. 2019. Disney-fying Dixie: Queering the ‘laughing place’ at Splash Mountain. In Performance and the Disney theme park experience, ed. Jennifer A. Kokai and Tom Robson, 107–125. Cham: Palgrave Macmillan. Bryman, Alan. 1995. Disney and his worlds. London: Routledge. Camp, Gregory. 2017. Mickey mouse muzak: Shaping experience musically at Walt Disney World. Journal of the Society for American Music 11 (1): 53–69. Carlà-Uhink, Filippo. 2020. Representations of classical Greece in theme parks. London: Bloomsbury. Carlà, Filippo, Florian Freitag, Gordon Grice, and Scott A. Lukas. 2016. Research dialogue: The ways of design, architecture, technology, and material form. In A reader in themed and immersive spaces, ed. Scott A. Lukas, 107–112. Pittsburgh: ETC. Carson, Charles. 2004. ‘Whole new worlds’: Music and the Disney theme park experience. Ethnomusicology Forum 13 (2): 228–235.

42

Intermediality in Theme Parks

1111

Davis, Susan G. 1996. The theme park: Global industry and cultural form. Media, Culture & Society 18: 399–422. ———. 1997. Spectacular nature: Corporate culture and the sea world experience. Berkeley: University of California Press. Eco, Umberto. 1986 [1975]. Travels in hyperreality. In In travels in hyperreality: Essays, Umberto Eco, 1–58. Trans. William Weaver. San Diego: Harcourt Brace Janovich. Erb, Maribeth, and Chin-Ee Ong, eds. 2017. Theming Asia: Theme park experiences in east and southeast Asia. Tourism Geographies 19 (2): 143–300. Finch, Christopher. 1973. The art of Walt Disney from Mickey Mouse to the Magic Kingdoms. New York: Harry N. Abrams. Findlay, John M. 1992. Magic lands: Western cityscapes and American culture after 1940. Berkeley: University of California Press. Fjellman, Stephen M. 1992. Vinyl leaves: Walt Disney World and America. Boulder: Westview. Francaviglia, Richard V. 1974. Main Street revisited. Places 1 (3): 7–11. ———. 1977. Main Street, U.S.A.: The creation of a popular image. Landscape: A Magazine of Human Geography 21 (3): 18–22. ———. 1981. Main Street U.S.A.: A comparison/contrast of streetscapes in Disneyland and Walt Disney World. The Journal of Popular Culture 15 (1): 141–156. ———. 1996. Main street revisited: Time, space, and image building in small-town America. Iowa City: University of Iowa Press. Freitag, Florian. 2017a. ‘Like walking into a movie’: Intermedial relations between Disney theme parks and movies. The Journal of Popular Culture 50 (4): 704–722. ———. 2017b. Scoring the future that never was: An interview with David Tolley. In Time and temporality in theme parks, ed. Filippo Carlà-Uhink, Florian Freitag, Sabrina Mittermeier, and Ariane Schwarz, 259–263. Hanover: Wehrhahn. ———. 2020. Theme parks and COVID-19: The return of metatourism. Balliol Interdisciplinary Institute blog: Studying theme parks. https://studyingthemeparks.com/2020/10/20/theme-parksand-covid-19-the-return-of-metatourism/. ———. 2021a. Popular New Orleans: The Crescent City in periodicals, theme parks, and opera, 1875–2015. New York: Routledge. ———. 2021b. ‘This way or that? Par ici ou par là?’: Language in the theme park. Visions in Leisure and Business 23: 1. https://scholarworks.bgsu.edu/visions/vol23/iss1/7/. ———. 2022. The happiest virtual place on earth: Theme park paratextuality. In Virtual interiorities: The myth of total virtuality, ed. Dave Gottwald, Greg Turner-Rahman, and Vahid Vahdat, 97–121 Pittsburgh, PA: ETC. Freitag, Florian, and Chang Liu, eds. (2022). Cultural history and heritage in Chinese theme parks. Cultural history11 (2): 121–218 Genette, Gérard. 1982. Palimpsestes: La littérature au second degré. Paris: Seuil. ———. 1997 [1987]. Paratexts: Thresholds of interpretation. Trans. Jane E. Lewin. Cambridge: Cambridge University Press. Gitelman, Lisa. 2006. Always already new: Media, history, and the data of culture. Cambridge, MA: MIT Press. Gottwald, Dave, and Benjamin George. 2020. Cinematography in the landscape: Transitional zones in themed environments. Landscape Research Record 9: 49–64. Gottwald, Dave, and Greg Turner-Rahman. 2019. The end of architecture: Theme parks, video games, and the built environment in cinematic mode. International Journal of the Constructed Environment 10 (2): 41–60. Gould, Marty, and Rebecca N. Mitchell. 2010. Understanding the literary theme park: Dickens world as adaptation. Neo-Victorian Studies 3 (2): 145–171. Gray, Jonathan. 2010. Show sold separately: Promos, spoilers, and other media paratexts. New York: New York University Press. Hall, Millicent. 1976. Theme parks: Around the world in 80 minutes. Landscape: A Magazine of Human Geography 21 (1): 3–8.

1112

F. Freitag

Herbert, Dan. 2017. Film remakes and franchises. New Brunswick: Rutgers University Press. Hildebrandt, Hugo John. 1981. Cedar point: A park in progress. The Journal of Popular Culture 15 (1): 87–107. Hine, Thomas. 1986. Populuxe. New York: Knopf. Hutcheon, Linda, and Siobhan O’Flynn. 2013. A theory of adaptation. London: Routledge. Jenkins, Henry. 2006. Convergence culture: Where old and new media collide. New York: New York University Press. Kagelmann, H. Jürgen. 2004. Themenparks. In ErlebnisWelten: Zum Erlebnisboom in der Postmoderne, ed. H. Jürgen Kagelmann, Reinhard Bachleitner, and Max Rieder, 160–180. Munich: Profil. Klein, Norman. 2004. The Vatican to Vegas: A history of special effects. New York: New Press. Kokai, Jennifer A., and Tom Robson, eds. 2019. Performance and the Disney theme park experience: The tourist as actor. Cham: Palgrave Macmillan. ———, eds. 2022. Disney during COVID-19: The tourist and the actor’s nightmare. Journal of Themed Experience and Attractions Studies 2: 1. https://stars.library.ucf.edu/jteas/vol2/iss1/5. Kuisel, Richard. 2012. The French way: How France embraced and rejected American values and power. Princeton: Princeton University Press. Lainsbury, Andrew. 2000. Once upon an American dream: The story of Euro Disneyland. Lawrence: University Press of Kansas. Lam, Sunny S.K. 2010. ‘Global corporate cultural capital’ as a drag on glocalization: Disneyland’s promotion of the Halloween festival. Media, Culture & Society 32 (4): 631–648. Landau, Royston. 1973. Mickey Mouse the great dictator: The Disney game as a control system. Architectural Digest 43 (9): 591–595. Lukas, Scott A. 2008. Theme park. London: Reaktion. ———. 2016. Research in themed and immersive spaces: At the threshold of identity. In A reader in themed and immersive spaces, ed. Scott A. Lukas, 159–169. Pittsburgh, PA: ETC. Maanen, John van. 1991. The smile factory: Work at Disneyland. In Reframing organizational culture, ed. Peter J. Frost et al., 58–76. London: Sage. Marin, Louis. 1984 [1973]. Utopics: The semiological play of textual spaces. Humanity: Trans. Robert A. Vollrath. New York. Marling, Karal Ann. 1991. Disneyland, 1995: Just take the Santa Ana freeway to the American dream. American Art 5 (1–2): 168–207. ———, ed. 1997. Designing Disney’s theme parks: The architecture of reassurance. Paris: Flammarion. Mauro, Jason Isaac. 1997. Disney’s Splash Mountain: Death anxiety, the tar baby, and rituals of violence. Children’s Literature Association Quarterly 22 (3): 113–117. Mitrasinovic, Miodrag. 2006. Total landscape, theme parks, public space. Burlington: Ashgate. Mittermeier, Sabrina. 2020. Theme parks: Where media and tourism converge. In The Routledge companion to media and tourism, ed. Maria Månsson, Annæ Buchmann, Cecilia Cassinger, and Lena Eskilsson, 27–34. London: Routledge. ———. 2021. A cultural history of the Disneyland theme parks: Middle class kingdoms. Chicago: Intellect. Moulton, Carter. 2020. ‘Welcome (or stay at?) home’: Theme parks and Covid-19. Media commons. http://mediacommons.org/imr/content/welcome-or-stay-home%E2%80%9D-themeparks-andcovid-19. Ndalianis, Angela. 2004. Neo-baroque aesthetics and contemporary entertainment. Cambridge, MA: MIT Press. Ndalianis, Angela, and Jessica Balanzategui. 2019. ‘Being inside the movie’: 1990s theme park ride films and immersive film experiences. The Velvet Light Trap 84: 18–33. Nooshin, Laudan. 2004. Circumnavigation with a difference? Music, representation and the Disney experience. Ethnomusicology Forum 13 (2): 236–251. Paine, Crispin. 2019. Gods and rollercoasters: Religion in theme parks worldwide. London: Bloomsbury.

42

Intermediality in Theme Parks

1113

Park, Seunghyun, and B., Hyung J. Kim, and Chihyung M. Ok. 2018. Linking emotion and place on twitter at Disneyland. Journal of Travel and Tourism Marketing 35 (5): 664–677. Philips, Deborah. 2012. Fairground attractions: A genealogy of the pleasure ground. London: Bloomsbury. Pike, David L. 2005. The Walt Disney world underground. Space and Culture 8 (1): 47–65. Rahn, Suzanne. 2000. Snow White’s dark ride: Narrative strategies at Disneyland. Bookbird 38 (1): 19–24. Rajewsky, Irina O. 2002. Intermedialität. Tübingen: Francke. Raz, Aviad E. 1999. Riding the black ship: Japan and Tokyo Disneyland. Cambridge, MA: Harvard University Press. Real, Michael R. 1977. Mass-mediated culture. Englewood Cliffs: Prentice-Hall. Reijnders, Stijn, and Abby Waysdorf. 2018. Immersion, authenticity and the theme park as social space: Experiencing the wizarding world of Harry potter. International Journal of Cultural Studies 21 (2): 173–188. Riley, Robert, and Terence Young, eds. 2002. Theme park landscapes: Antecedents and variations. Washington, DC: Dumbarton Oaks Research Library and Collection. Rubin, Judith, ed. 2018. 2017 theme index and museum index: The global attractions attendance report. Burbank: Themed Entertainment Association and AECOM. Ryan, Marie-Laure. 2005. On the theoretical foundations of Transmedial narratology. In Narratology beyond literary criticism: Mediality, disciplinarity, ed. Jan Christoph Meister, Tom Kindt, and Wilhelm Schernus, 1–23. Berlin and New York: Walter De Gruyter. Schickel, Richard. 1968. The Disney version: The life, times, art and commerce of Walt Disney. New York: Simon and Schuster. Schwartz, Vanessa. 2020. Jet age aesthetic: The glamour of media in motion. London: Yale University Press. Schwarz, Ariane. 2016. Spieglein, Spieglein an der Wand – wer ist das echte Schneewittchen im Land? In Where the magic happens: Bildung nach der Entgrenzung der Künste, ed. Julia Dick, Thorsten Meyer, and Peter Moormann, 149–154. München: Kopaed. ———. 2017. Staging the gaze: The water coaster “Poseidon” as an example of staging strategies in theme parks. In Time and temporality in theme parks, ed. Filippo Carlà-Uhink, Florian Freitag, Sabrina Mittermeier, and Ariane Schwarz, 97–112. Hanover: Wehrhahn. Schweizer, Bobby, and Celia Pearce. 2016. Remediation on the high seas: A Pirates of the Caribbean odyssey. In A reader in themed and immersive spaces, ed. Scott A. Lukas, 95–106. Pittsburgh, PA: ETC. Smith, Matthew Wilson. 2007. The total work of art: From Bayreuth to cyperspace. New York: Routledge. Soto-Vásquez, Arthur. 2020. Digital Disneyland. Media Commons. http://mediacommons.org/imr/ content/digital-disneyland. ———. 2021. Mediating the magic kingdom: Instagram, fantasy, and identity. Western Journal of Communication 85: 5. https://doi.org/10.1080/10570314.2021.1970797. Sperb, Jason. 2012. Disney’s most notorious film: Race, convergence, and the hidden histories of song of the south. Austin: University of Texas Press. Telotte, Jay P. 2004. Disney TV. Detroit: Wayne State University Press. Wasko, Janet. 2020. Understanding Disney: The manufacture of fantasy. Second ed. Cambridge: Polity. Waysdorf, Abby S. 2021. Fan sites: Film tourism and contemporary fandom. Iowa City: University of Iowa Press. Williams, Rebecca. 2020. Theme park fandom: Spatial transmedia, materiality and participatory cultures. Amsterdam: Amsterdam University Press. ———. 2021. Theme parks in the time of the Covid-19 pandemic. In Pandemic media: Preliminary notes toward an inventory, ed. Philipp Dominik Keidl, Laliv Melamed, Vinzenz Hediger, and Antonio Somaini, 137–143. Lüneburg: Meson.

1114

F. Freitag

Wolf, Werner. 2007. Intermediality. In Routledge encyclopedia of narrative theory, ed. David Herman, Manfred Jahn, and Marie-Laure Ryan, 252–256. London: Routledge. Yandell, Stephen. 2012. Mapping the happiest place on earth: Disney’s medieval cartography. In The Disney middle ages: A fairy-tale and fantasy past, ed. Tison Pugh and Susan Aronstein, 21–38. New York: Palgrave Macmillan. Younger, David. 2016. Theme Park design & the art of themed entertainment. N.P: Inklingwood. Zhang, Wen, and Shilian Shan. 2016. The theme park industry in China: A research review. Cogent Social Sciences 2: 1–17.

43

Interactive and Participatory Sound Vadim Keylin

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mediality of Own Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ecologies and Affordances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interactivity and Presence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Social Immersion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1116 1117 1120 1124 1128 1130 1131

Abstract

Whether singing, playing an instrument, or dancing, sound has been interactive through most of human history. While the twentieth century has been dominated by linear media such as radio or sound recording, digital technologies reintroduced interactivity into sound culture, offering a plethora of new ways of interacting with sound. The chapter offers an overview of contemporary research into technological and aesthetic aspects of interactive audio from the fields of sound studies, video game studies, media studies, and, to a lesser extent, music sociology and HCI. It concentrates on three principal domains: • Interactive and participatory sound art • Sound in immersive media, such as video games and VR (virtual reality) • Sound practices of online participatory cultures While at its most basic level interactive audio involves an interplay of sound and movement, sonic interactions necessarily spill into the broader sociocultural context, progressing from interactivity to participation. The chapter thus furthermore explores the ways material (sensory, spatiotemporal, technological) aspects V. Keylin (*) Institute for German Language and Literature, Universität Hamburg, Hamburg, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_55

1115

1116

V. Keylin

of sound necessarily influence and are in turn influenced by and reimagined through the social processes of participation, approached as a medium of its own. This dialectic is addressed through a media-ecological approach emphasizing the sonic affordances of interactive media works. Keywords

Interactivity and participation · Sound art · Sonic and audiovisual media · Immersion · Digital culture

Introduction In principle, sound has always been interactive. It is either an action in itself – when we speak, strike the strings of a guitar, or ring a doorbell – or a “sonic trace” (Schulze 2018, 111) of an action, like footsteps of traffic noise. In its acculturated forms – most prominently, music – sound has also been participatory for longer than not. In most non-Western and pre-modern Western music cultures, music-making is a social activity that brings whole communities together as players or dancers (Turino 2008). Even the linear forms of music and sound performance that modernity has instituted leave space for sonic participation in the form of clapping, booing, or whistling the performers. The necessity to qualify sound as interactive or not comes with the audio landscape of the twentieth century dominated by linear sonic media such as radio or sound recording. However, participatory and interactive cultural practices have been making a steady return in the past decades. Participatory art and theater emerged in the 1960s–1970s as a way of integrating the audience into artmaking and transcend the boundaries of autonomous art institutions (Bishop 2006). At the same time, technological advances have driven the development of interactive digital media and art that situate themselves in opposition to traditional, linear media by virtue of offering the user agency in navigating their content and “freeing them from the passive experience of simply watching” (Dyson 2009, 2). This chapter discusses interactive sonic and audiovisual media and art forms ranging from participatory music and sound art to video games and VR (virtual reality) to participatory sound culture online. My central contention is that the experience of such cultural forms is necessarily intermedial, operating through the interplay between action – in its physical-social unity – and sound. The first half of the chapter investigates the aesthetic mechanisms of this interplay and its principal non-linearity, which differentiates interactive sound from other kinds of intermediality. The second half explores the two directions in which sonic interactions can occur: between the participant (or player, or user)1 and the artwork

1

Since the nomenclature differs significantly between different media and art forms, in the following these forms are used somewhat interchangeably.

43

Interactive and Participatory Sound

1117

(or game, or platform) and between the participants themselves, mediated by the artwork.

Mediality of Own Actions Before discussing the intermediality of interactive audio, the sides of this equation must be elaborated. One of them is relatively straightforward (to the extent that defining something as a medium can be straightforward at all): sound being one of basic sensory modalities, its mediality seems self-evident. Calling participation or interactivity a medium is somewhat more counterintuitive, exacerbated by the fact that no agreement can be found on how to define and distinguish interactivity and participation themselves (see, e.g., Kester 2011; Bishop 2012)2 While action is a fundamental modality of performing arts, it is traditionally an observed action, not own action. Interactivity is typically described as a feature of a medium; the users may furthermore interact with media, irrespective of interactive or not (Eichner 2014). The notion of mediality describes the way actions are structured by media (Jäger 2010), further driving the conceptual separation between them home (▶ Chap. 5, “Intermediality and Medium Specificity”). Finally, own actions are typically situated on the sender side of the sender-receiver model, so how can one attend to them as part of media experience of the receiver?.3 At the same time, interactive sonic and audiovisual artworks and media differ unambiguously from non-interactive ones, even where the sonic material stays the same. For example, playing a video game creates a fundamentally different listening experience than watching a stream of the same game: sounds change their meaning, or even lose it completely, when being uncoupled from actions that trigger them. Similarly, art theorists such as Nicolas Bourriaud (2002), Grant Kester (2011), or Claire Bishop (2012) go as far as to downplay the sensory materiality of participatory artworks, claiming instead that they unfold in the medium of social forms and social interactions, “where people constitute the central artistic medium and material” (Bishop 2012, 1–2). In this sense, participation emerges as a medium retroactively. While, as noted in the introduction, many pre-modern cultural practices were inherently interactive or participatory without necessarily becoming multimodal or intermedial, the emergence of interactive media and participatory art forms that position themselves against their twentieth-century linear counterparts demands a rethinking of mediality of participation. Interactive art forms appear in intermediality studies predominantly in the context of theater (see, e.g., Crossley 2021). Theater studies in general form one of the major 2

Again in this chapter, the two terms are used somewhat interchangeably, as distinguishing them is not necessarily meaningful in the context of sound art and sonic media (see Keylin 2019). 3 Lars Elleström’s (2021) revised model of intermediality circumvents this problem by redescribing the situation in terms of interacting minds rather than sender and receiver. However, this has the consequence of reinforcing the Cartesian mind-body dualism.

1118

V. Keylin

discourses on interactivity and participation, conceptualizing them as aspects or forms of performance rather than media in their own right (see, e.g., White 2013; Machon 2017). However, this approach is not without limitations. While video games can in some cases be regarded as performative in the theatrical sense (Neitzel 2015), it is difficult to apply this perspective to, e.g., abstract puzzle games like Tetris, which are nevertheless fully interactive (▶ Chap. 39, “The Qualified Medium of Computer Games: Form and Matter, Technology, and Use”). Similarly, many open-ended works of media art – and particularly sound art – eschew narrative and temporal structures, making the concept of performance of little analytical value. Furthermore, my contention is that participation exists as a continuum encompassing one-on-one interactions of the user with the artwork or a media work (e.g., singleplayer video games), social interactions in participatory art and multiplayer games, as well as creative practices of participatory culture. For these reasons, this chapter adopts a disciplinary background from art theory and media studies with their more open-ended concepts of interactivity and participation (Jenkins 2009; Kester 2011; Bishop 2012; Collins 2013; Eichner 2014; Heinrich 2014). Of course, own actions have a sensory modality inherent to them in proprioception, which can be manipulated, for example, in VR works that disorient the user and alter their spatial perception (see Dyson 2009, Chap. 5). However, proprioception cannot account for the social or cultural meaning ascribed to own actions, nor for the “social forms,” whose emergent character does not allow to easily place them within a sensory modality, but that nevertheless constitute the medium of participatory art. Exploring the mediality of own actions in his book Performing Beauty in Participatory Art and Culture, Falk Heinrich (2014) approaches them as a continuity between three layers: physical (“to do”), social (“to act”), and discursive (“to perform”). Furthermore, actions are perceived as aesthetically meaningful when they “join three dimensions – sensuous appreciations (including proprioception), agency within scripted realms, and conceptual understanding – into an experience of unity in that it affords meaning through agency” (Heinrich 2014, 178). Heinrich’s three layers of act find a certain parallel in Lars Elleström’s (2010, 35) distinction between “basic” (determined by a single modality of a combination of modalities) and “qualified” (culturally or contextually determined) media. To clarify the ambiguous notion of the medium, Elleström further distinguishes “technical media” as technologies and devices through which both basic and qualified media unfold. On a physical level (“to do”), own actions may thus be said to possess a basic mediality, a combination of material (movement) and sensory (proprioception) modalities. However, in specific cultural practices – such as video games or sound artworks – they may also be organized socially (“to act”) and/or discursively (“to perform”), often with the help of technical media, to attain the quality of a qualified medium. Approaching the conundrum of mediality of own actions from a different perspective, Susanne Eichner (2014, Chap. 1) proposes the notion of agency as a separate modality of experience. She takes as her starting point Janet H. Murray’s (1998, 126) definition of agency in video games as “the satisfying power to take meaningful actions and see the results of our decisions and choices.” Eichner argues,

43

Interactive and Participatory Sound

1119

however, that this experience of agency is not limited to video games and can be found, on the one hand, already in linear media (in the user’s power to switch the TV channel or to disagree with the idea of the text) and on the other hand, in media practices – particularly, in participatory culture – that are not centered around a specific media work. She further distinguishes between technical interactivity as a feature of a technical medium and perceived interactivity as a specific capability of a (typically, qualified) medium to foster and foreground the experience of agency (Eichner 2014, Chap. 2). This understanding of agency can be connected to that of flow state – a state of hyperfocus characterized by the merging of action and awareness that is perceived as intrinsically gratifying (Csikszentmihalyi 2009). A characteristic trait of the flow state is that it is self-sustaining: the action becomes its own reward and thus the main content of the experience. At the same time, as Heinrich notes in the quote above, agency exists in interactive media and participatory art “within scripted realms.” The experience of agency is thus always necessarily dialectical, oscillating between exercising agency and being denied it. The player’s actions in a video game are limited by what the interface allows them to do and guided by the game’s rules and narrative. Similarly, the capabilities of online platform and the genre expectations of their communities direct the creative agencies and practices of participatory culture (Jenkins 2009; Goriunova 2012). The dialectics of agency and non-agency operating across the levels of physical, social, and conceptual acts serves as a mechanism through which the mediality of own actions unfolds. However, a media work must communicate these limitations of agency through the feedback systems that necessarily involve other modalities than agency and proprioception and thus other basic or even qualified media than own actions – which means that own actions can operate as a medium only in intermedial constellations. In other words, they can be said to possess a mediality without constituting a medium in the proper sense (unless understood in the narrow sense, as the artistic medium of participatory art). At the same time, Elleström’s (2010, 28) model of intermediality distinguishes between “combination and integration of (basic or qualified) media and, on the other hand, mediation and transformation of (basic or qualified) media.” This is an important distinction for interactive sound as well since not all own actions result in a sound. For example, a VR work may possess both a soundscape and interactive qualities (e.g., free navigation) that do not influence each other directly but work in accord to create a sense of immersion. On the other hand, sounds in a video game or participatory sounds art are often unambiguously related to actions – they can even be approached as a direct sonification of the player’s or the participant’s conduct (Grimshaw 2008). This relation is reciprocal: sounds may be translated into own actions as well, such as when a video game provides gameplay cues through sound. A different classification of intermedial relations that observes the borders between individual works more so than between media modalities is offered by Irina Rajewsky (2011, 51–52). She proposes a distinction between “media transpositions” such as film adaptations, “media combination [. . .] or, to use another

1120

V. Keylin

terminology, so-called multimedia, mixed media, and intermedia,” and “media references” – a more subtle form of intermediality by which one medium takes on structural qualities of another without directly engaging with it. This latter concept offers an explanation of intermediality of sonic practices in participatory culture: while monomedial in the strict sense, being realized as audio or video recordings, they carry the references to the participatory processes and experiences that led to their creation (▶ Chap. 47, “The Recommended Experience: Engaging Networked Media Platforms with Intermediality”). To sum up the above, the intermediality of interactive sound works and media can be situated within the interplay between the experience of agency emerging from own actions and the listening experience. Both components of this equation unfold simultaneously on physical (elementary actions and individual sounds) and social levels (social forms and forms of sound culture), joined together at the conceptual level. Different (qualified) sonic and audiovisual interactive media prioritize different levels of this interplay: single-player video games and VR works tend to prioritize the physical and conceptual, while in participatory culture the physical level becomes largely irrelevant, abstracted to social actions, which take center stage. Participatory sound artworks and multiplayer video games, depending on precise configurations, may run the whole gamut.

Ecologies and Affordances Unlike the relationship of, for example, audiovisuality, the analysis of intermediality of soundmaking actions in interactive and participatory media is complicated by the fundamentally non-linear character of such media, with makes the intermedial relationships themselves non-linear. Various concepts have been proposed to describe this non-linearity. Umberto Eco (1989) in his classic treatise The Open Work describes the way such works are structured as “the field of possibilities” for the performer or the participant to explore. Heinrich (2014) uses the notion of the “topology” of participatory artwork to refer to the way the materialities of the artwork guide the participant’s actions and structure the relationships between the three layers of the act. Along the same lines run the conceptualizations of sound art pieces not as works but as situations that emphasize their embeddedness in the spatiotemporal and sociocultural contexts that they unfold in (Sanio 1999; Krogh Groth and Samson 2017). This non-linearity is exacerbated by the fact that both the composition of the parties engaged in interaction and the site of the interaction are left somewhat indeterminate. For example, media art theorist Beryl Graham (2010) suggests that “reactive” might be a better term for artworks that simply connect user input to a number of pre-defined or randomized responses, while reserving the category of “interactive” to artworks that provide platforms through which users can interact with each other. At the same time, Susanne Eichner (2014) argues that the user’s agency in approaching a piece of media is not limited to the possibilities nor sites

43

Interactive and Participatory Sound

1121

explicitly contained in that piece of media – as evidenced, for example, by the creation of memes from video game clips in participatory culture. Karen Collins (2013) explores this indeterminacy in relation to video game sound with a multidimensional model of sonic interactions, classifying them into several categories. In “multimodal/perceptual interactions” several sensory modalities converge into a perception of the player-controlled avatar’s actions and the gameworld, such as pressing a “shoot” button coupled with animation and sound of a gunshot. “Physical interactions” occur between the player and the technical medium of the game audio. “Interpersonal interactions” use the audio capabilities of the game – such as voice chat – to communicate between the players. Finally, “sociocultural interactions” are a continuation of interpersonal but occur on a larger spatial and temporal scale – for example, in ROM-hacking music scene or at conventions where participants may perform cover versions of video game music. Importantly, only the perceptual interactions are limited to the game itself. Physical interactions occur simultaneously within the game and outside of it (in the “metagame” in Collins’ terms), within the physical space of the player, while interpersonal interactions evolve diachronically between the two. Sociocultural interactions transcend the context of the game by definition as they manifest in such practices as hacking the game cartridges to use as music instruments or performing cover versions of video game music at conventions. Collins’ model, while developed in relation to video game specifically, can be easily expanded onto other interactive and participatory media4 and shows the complexity of sonic interactions afforded by them. By emphasizing their sonic character, it also shows how intermediality manifests in different forms at the same time in one-on-one interactions with the media piece and in the social interactions both within the media piece and outside of it, but nevertheless mediated by it. The complexity and open-endedness of intermedial relations characteristic of interactive and participatory media make it useful to approach them as ecologies. A particularly useful concept within the ecological thinking is that of affordance. Originally coined by the psychologist James Gibson (1979), it refers to a relationship between an actor and an environment that delimits what actions – whether intentional or unintentional, positive or negative – could be performed. As it accounts for the whole “field of possibilities,” the notion of affordance is best suited for the analysis of the non-linear relationship between own actions, their sonic counterpart, and the media or artwork that facilitates their connection. Affordances furthermore apply to all three layers of act. In their study of social media affordances, Taina Bucher and Anne Helmond (2018) distinguish between “low-level” affordances for elementary actions and “high-level” affordances that facilitate complex behaviors. This distinction can also be mapped quite precisely onto Elleström’s dichotomy of basic and qualified media. Most evidently, basic or low-level affordances of interactive audio describe elementary actions that produce individual sounds, like shooting a gun in a video

4

Keylin (2019) discusses such applications to participatory sound art.

1122

V. Keylin

game or plucking a string on a musical instrument-like sound sculpture. However, sound can also be an action in itself that produces other sorts of feedback, for example, in a variety of voice-controlled media (e.g., Amazon Echo) or artworks (e.g., Max Neuhaus’ Auracle discussed below). Finally, a variety of affordances for elementary action can be found in sound: video games may use audio cues to signal the timing of player actions; sonic movement in dance music can inform physical moves of the dancers (Keylin 2020a). In other words, basic affordances of interactive sound can be classified into soundmaking actions, sound actions, and sound-facilitated actions. In many cases, however, they are intertwined: a sound that is being produced by a soundmaking action may contain cues that facilitate further actions, and both sides of this equation may be sound actions. Their interplay thus shapes the experience of agency in interactive sound artworks and media. Unlike low-level affordances that are applicable more or less universally throughout the whole spectrum of interactive audio, high-level affordances vary significantly between different qualified sonic media. These differences can be explicated by comparing two art forms: participatory music and participatory sound art. On the level of basic affordances, both operate roughly the same: they engage the participants in soundmaking actions using their voices or soundmaking tools provided to create an aesthetic and social experience. However, on the level of complex behaviors, these elementary actions combine into vastly different patterns. Participatory music performance, as Thomas Turino (2008) argues, relies on familiarity and repetition. Forms of participatory music are typically characterized by constant unchanging rhythms and a general lack of melodic or timbral variation. Individual virtuosity and self-expression are discouraged. Participation relies on cultural memory and tradition and serves to affirm a collective identity. In other words, the high-level affordances of participatory music practices facilitate the kind of music-making behavior that is determined by convention, whose rules are followed by all the participants, and results in a repetitive and familiar sound, which, in a feedback loop, helps guide the participants. Conversely, participatory sound art does not have traditions or conventions that would guide the participants. The rules of participation – as well as the sonic palettes – vary greatly from artwork to artwork and are at times purposefully obscured. A participant thus approaches such an artwork without knowing what to expect. This brings about an exploratory ethos as the participants are encouraged to embrace their ignorance and experiment. Similarly, the sociality that sound artworks facilitate emerges from a shared sonic experience of strangers rather than from reinforcing pre-existing relationships. The high-level affordances of participatory sound art thus include exploration and experimentation – something that is discouraged by the structures of participatory music – and exclude tradition-guided routine. This is again reinforced by the sonic feedback loop prioritizing unusual and unexpected sounds (Keylin 2020a). Even within a given qualified medium, sonic affordances, and thus the intermedial relations of sound and agency, can vary significantly. Here Elleström’s (2010) notion of technical medium is helpful to explain how the technical

43

Interactive and Participatory Sound

1123

affordances underlie the aesthetic ones. This is most evident, again, in participatory sound art whose technological base runs the gamut of low- or even no-tech pieces such as soundwalks to technologically sophisticated digital artworks. Keylin (2019) classifies sound art practices into three environment types based on their technical media: local, networked, and augmented. The term “environment” is not accidental here as it underscores, on the one hand, the ecological organization of such artworks, and on the other hand, the prominence of spatiotemporal modality in structuring their affordances. Local environments are continuous spaces that are not technologically extended in any way5. Networked environments, which could be also called distributed or telematic, are characterized by the flow of information between several physical locations connected through technological channels – typically, the Internet, although in some cases it might be phone lines or other means of communication. Finally, augmented environments are comparable to AR works in that they overlay a physical space with a virtual one, placing the user in the center of both at the same time. These technical media and their affordances exert a significant influence on how the intermedial relationship between agency and sound is structured. Sound art in local environments relies on the continuity of its spatial structure and immediacy and corporeality of its sound-producing and listening mechanisms to encourage the participants to interact with the works (physically) and each other (socially). For example, the classical sound sculptures by Bernard and François Baschet operate akin to traditional musical instruments. The explicit materiality of the soundmaking mechanism makes immediate and visceral the connection between the participants’ soundmaking action on the sonic result, emphasizing the materiality of sound. At the same time, placing several such sculptures into an exhibition space where the sound made by one is heard by all the others encourages a sonic dialog between them. Networked sound art, on the other hand, employs the network’s ability to connect spaces and actors over various barriers, both physical and institutional, to create participatory structures that would be impossible in local environments, for example, due to geographical distance. At the same time, digitizing the sound, algorithmically processing it, and locating it in the virtual space make the action-sound connection a lot more obscure than in the case of local environments, amplifying the experimental character of interaction. A classic example of such environments is Max Neuhaus’ online installation Auracle. It offers the participants a chance to connect over distance in a collaborative improvisation using their voices to control sound synthesis processes. The participants are not seen, and their voices are not heard by the others – only the synthetic audio results, piquing curiosity both towards each other and towards the soundmaking mechanisms of the artwork. Whereas networked environments connect physical spaces to virtual, augmented environments merge them. They introduce a new sonic layer to the existing sites,

5

At least as far as the artwork is concerned, since nothing stops the participants from using their mobile devices to share their experience on the Internet, but that would be beyond the borders of the artwork.

1124

V. Keylin

enhancing the participants’ perception and making them engage deeper with the site and its natural, cultural, and social aspects. For example, Kaffe Matthews’ sonic bikes are equipped with loudspeakers, a GPS system, and a sound samples bank that the artist maps onto various urban spaces. Riding a sonic bike, the participants activate the pre-recorded sounds based on their location, direction, and speed, connecting their movement through a physical space to a movement through a non-linear sound composition and situating the listening experience within a specific site and vice versa. The three environment types show how the interplay between different basic, qualified, and technical media, both in sound and participation, leads to the emergence of different constellations of affordances. In turn, those affordances structure in a non-linear fashion the intermedial experience of interactive and participatory sound. The following two sections focus on two aspects of this experience to which both sound and interactivity contribute the most: immersion and communality.

Interactivity and Presence The concepts of immersion and presence have long been at the center of the discourse of digital media. Most of the time, they are used somewhat interchangeably, to designate “a process or condition whereby the viewer becomes totally enveloped within and transformed by the ‘virtual environment’” (Dyson 2009, 1). Where the distinction between immersion and presence is made, the latter generally refers to the user’s experience of being immersed in an environment, while the former, to the conditions that facilitate this experience (Garner 2017, Chap. 4). Presence is also sometimes approached as the highest degree of immersion. On the other hand, immersion is sometimes considered to be possible without presence (Glassner 2017). Additionally, concepts such as “telepresence” or “virtual presence” are used to emphasize that the environment in question is located elsewhere and particularly in the “virtual,” computer-generated space (Sheridan 1992). This section first addresses immersion as presence, as defined by Frances Dyson’s quote above, and then considers other forms of immersion and their relation to sound and interactivity. The primary material of this section is immersive media – video games and in particular virtual reality (▶ Chap. 39, “The Qualified Medium of Computer Games: Form and Matter, Technology, and Use”). It is little wonder that the rhetoric of immersion has come to dominate the VR discourse as the creation of realistic simulated environments is the whole raison d’être of such technologies and media. On the one hand, VR works are necessarily multimodal and hypermedial as they simulate the multimodality of human experience. On the other hand, they simultaneously aim for the immediacy of the VR experience. Accordingly, the concept of presence is sometimes defined as the “perceptual illusion of non-mediation” (Lombard and Ditton 1997, 9). Finally, VR applications most commonly include artworks and games, which makes them more focused experiences emphasizing their artifice (Rajewsky 2010). In other words, immersive media are

43

Interactive and Participatory Sound

1125

characterized by the paradoxical dialectic of foregrounding their intermediality and interactivity and at the same time striving to appear immediate and natural. At the same time, as Frances Dyson (2009, 3–4) argues, the rhetoric of immersion has been present in the sound and music discourse long before VR became a reality: [T]he diverse technologies we associate with new media reconstitute experiences characteristic of the aural, for sound is the immersive medium par excellence. Three-dimensional, interactive, and synesthetic, perceived in the here and now of an embodied space, sound returns to the listener the very same qualities that media mediates: that feeling of being here now, of experiencing oneself as engulfed, enveloped, absorbed, enmeshed, in short, immersed in an environment.

Dyson furthermore directly connects the ideas heralded by VR to earlier acoustic media. Telepresence – being remotely present – was first promised by telephony. Audio recording technology promulgated the idea of fidelity – of the medium that facilitates immediacy. Finally, the roots of notion of virtual space, according to Dyson, can be directly traced to the ethereal space of radio. Unlike audio, however, the immersiveness of digital media is underscored by their interactivity, defined at the minimum level as “the user’s navigation of and engagement with digital content [that] is said to give the users agency” (Dyson 2009, 2). Whereas, as noted at the beginning of this chapter, sound itself has always been interactive, the sound theory has to this day heavily prioritized listening over soundmaking (see Keylin 2021). So how do sound and interactivity interweave in an experience of immersive media? One of the aspects where the two act in accord is the creation of the sense of space. Spatiality is, understandably, a crucial aspect of presence (see Dyson 2009, Chap. 5; Garner 2017, Chap. 4; Nordahl and Nilsson 2014). VR experience relies on surrounding the user in computer-generated imagery, placing them at the center. This goal, however, is challenged by the directedness of vision, allowing us to view only a limited segment of surrounding space at a time. Here the spatial qualities of sound – its ability to be heard from all directions, without its source being seen – help maintain the illusion that the virtual space and the objects that fill it continue to exist. Sound’s ability to immerse the listener in a three-dimensional acoustic space in fact precedes VR and digital media. As Dyson (2009, 3) notes, “The desire for total immersion, as well as the use of audio to simulate this condition, is certainly evident in old media such as cinema.” Three-dimensional sound of cinema gives depth to the two-dimensional picture, immersing the viewer in the fictional world of the frame. Interactivity offers a similar transposition, “with the body providing the here-andnow being, while the simulated environment stands in as the ‘there’” (Dyson 2009, 3). Being able to traverse the virtual space seen and heard, and possibly interact with the objects in it, supports the illusion of presence in the temporal dimension. In that respect, video games occupy an interesting position between screen media and immersive media. On the one hand, much like VR they emphasize the player immersion in the virtual environment. On the other hand, they do not simulate the

1126

V. Keylin

embodied experience of presence, situating the player in front of the screen representing visual space rather than inside it. Here the reactivity and responsiveness of the virtual sonic environment to player actions becomes particularly important, as the action-sound feedback loop helps connect the physical space to the virtual. Marc Grimshaw and Tom Garner (2014, 373) use the concept of “virtual acoustic ecology” to describe “an intricate relationship between the player, the audio engine (virtual soundscape), and the resonating space (real acoustic environment) [. . .] with the explicit purpose of representing a virtual environment that players will find immersive, irrespective of the fact that they are not physically situated within that world.” As discussed in the previous section, the term “ecology” underscores the salience of player participation to this relationship. In other words, even when they do not necessarily mediate each other directly, the media combination of sound and participation amplifies the experience of immersion and presence. Interestingly, the influence of self-produced sounds – that is, sounds effected by the player or the participant – on the immersive experience has been assessed as somewhat ambiguous. On the one hand, such sounds have been found to interfere with the sense of presence, drawing the user back into the physical space (Larsson et al. 2010). Karen Collins (2014) similarly notes that in multiplayer video games, user-generated sonic content (such as voice chats or streaming of player-owned music into the game) may lead to “breaking the fourth wall.” On the other hand, in a study by Rolf Nordahl (2005) the participants assessed virtual environments that included self-generated sounds of footsteps as facilitating significantly stronger sensations of presence. One explanation of this ambiguity may have to do with the distinction between diegetic – ascribed to objects in the fictional or virtual environment – and non-diegetic, firmly situated outside of it, sounds. Indeed, the sounds of footsteps in Nordahl’s experiment could have been ascribed to the user’s virtual “avatar,” while streaming a pop song into a fantasy multiplayer RPG would clash with the setting. Importantly, diegetic sounds in this sense do not have to be generated by the media work – it may also incorporate the sounds of the user’s physical body. For example, in the classic VR installation Osmose by the Canadian artist Char Davies, the viewer’s breath was used to control their traversal through ephemeral surrealistic landscapes. The self-produced sound of breath thus became a part of the soundscape of these environments, connecting the viewer’s physical body to them and enhancing the sense of presence. Collins (2014, 359) also argues that “while we may feel less present in virtual worlds in which we partake in user-generated content sharing, the act of creation within that space may lead to a more engaging experience.” In other words, while disturbing presence, such self-produced sounds may enhance immersion. Collins thus makes a case for differentiating between immersion and presence. This distinction, as noted at the beginning of this section, is not without precedent. Andrew Glassner (2017, 81–82), for example, distinguishes between four levels of immersion: curiosity, sympathy, empathy, and transportation, only the latter of which is equatable with presence. Similarly, Jennett et al. (2008) distinguish three kinds of immersion: sensory, imaginative, and challenge-based. Again, only sensory

43

Interactive and Participatory Sound

1127

immersion in this model somewhat corresponds with presence, while the other two describe more intellectual or affective kinds of engagement. Self-produced sounds may enhance these kinds of engagement by providing feedback on the success or failure of action (Collins 2013) or by sonifying the player’s emotional state (Grimshaw 2008). Tom Garner (2017) posits a circular, dialectic relationship between immersion and flow state. On the one hand, flow state is immersive par excellence. It makes the user hyperaware of the performative space of their actions while simultaneously dulling their awareness of the physical environment and experiences that are unrelated to play. On the other hand, “feeling present (via ludic immersion) within a virtual environment draws all user focus towards play” (Garner 2017, 97). The affordances for action found in sound become particularly important for this loop as they direct play, providing important gameplay cues, facilitating the rhythm (and thus, the flow) of action, or affecting the user’s emotional state. In case of music-based games, such affordances and sounds may even constitute the core of gameplay (Fritsch 2014). The most famous example of such games is the Guitar Hero series which uses guitar-shaped controllers to challenge the player to play lead, bass, or rhythm guitar parts in a number of songs of increasing complexity. The player has to learn to respond to both visual (score) and sonic (music) cues to achieve success. Most of the songs included in Guitar Hero packages are written specifically for the game; however, they may also contain cover versions of existing songs and users can in principle create and add their own custom songs (although the use of external software is necessary for that). Apart from such straightforward music-playing games, more experimental ones also exist, such as Vib-Ribbon (1999) that generates platforming-style gameplay levels from the musical tracks provided by the player. The use of custom tracks in Guitar Hero or Vib-Ribbon is a relatively rare example of straightforward “media transpositions” (Rajewsky 2011, 51) in interactive sound media. Unlike most such media, where own actions (which, as noted above, cannot constitute a medium outside of intermedial constellations) are remediated by sound, in music-based games pre-existing sounds and music are remediated by the user’s participation. A few similar cases can also be found in sound art, such as Benoît Maubrey’s Speaker Sculptures – site-specific urban space installations built of active loudspeakers that allow the participants to connect their smartphones via Bluetooth and play their tracks through the sculpture. Given the importance of music tastes to one’s identity-building, this kind of participation provides a way of immersing in art and media that foregrounds personal emotional connection to it. But it also fosters a different way of engaging with and immersing in music itself through interactivity and participation. Games like Guitar Hero foreground the game-like quality of virtuosic playing, while VibRibbon highlights the structuring of musical time by translating it into a structuring of the game’s performative space. At the same time, artworks like Speaker Sculptures make music a multimodal immersive experience by connecting it (and the participants through it) to local sites and communities (Keylin 2020b).

1128

V. Keylin

Social Immersion As Collins’ (2013) model suggests, there exists a continuity between the sonic interactions that occur fully or partly within the media work and those that occur fully or partly outside of it. This section thus discusses a kind of high-level intermediality that occurs between participation as a sociocultural phenomenon and the sonic media in and through which it manifests. Whereas the fundamental goal of one-on-one interactions with the media work addressed in the previous section is the experience of immersion and presence, the key aspect to sonic participation in the broader cultural context is the sociality that emerges from it and crystallizes in aesthetic artifacts and practices. That is not to say that this sociality and immersion in a media work are mutually exclusive. On the contrary, social aspects, whether real or simulated, have been named repeatedly as a key component of immersive experience (see Garner 2017, 92–94). Presence itself may be defined as “social richness” characterized “by the extent to which individuals engaged in some form of mutual interaction, find the medium facilitating the interaction sociable, warm, sensitive, and personal” (Nordahl and Nilsson 2014, 216). Even where self-produced sounds can break the illusion of presence in multiplayer games, as discussed in the previous sections, they nevertheless work towards the deeper engagement with the medium on other levels of immersion. In this broader sense, immersion in the medium can also be facilitated by sonic interactions that occur outside the medium itself. Among such interactions, Collins (2013) names various video game fandom activities. For example, players may perform cover versions of in-game songs at conventions or record and share them with online game communities. Such activities promote identification with in-game characters and social groups, thus deepening the players’ immersion in the game narrative (where the game has one) but also reinforce the identities of gamers and their communities. Conversely, outside-of-game immersion may also engage mechanical or technical rather than narrative aspects of the game, such as when musicians and sound artists use video game hardware as sound-producing instruments. However, similar participatory processes of immersing in a sound media aesthetic and building communal identities around occur also around musical and sonic practices in general, without a specific video game or other kind of media work to ground them. In their article “From Microsound to Vaporwave: Internet-mediated musics, online methods, and genre,” Georgina Born and Christopher Haworth (2017) show that a whole music genre may function in the same way. They invoke the extended definition of the genre as being constituted not only by “formal musical and technical criteria” (Born and Haworth 2017, 608), but also “a community whose agreement forms the basis for the definition of [the] genre” (Fabbri 1981, 2). Furthermore, “the conventions governing the existence of music genres may include performance and gestural styles as well as social, spatial, ideological, behavioural, economic, and juridical rules” (Born and Haworth 2017, 608). In other words, their

43

Interactive and Participatory Sound

1129

definition of the genre already contains the interplay between the social and the sonic. In participatory online culture, however, this social-medial unity of the genre is further mediated in three specific ways: material, social, and discursive. Materially, online music sharing platforms simplify both the artists’ access to audiences and the audiences’ access to music. Discursively, the Internet fosters a participatory discussion around music in such forms as reviews, comments, recommendations, etc. posted on blogs and social media, stimulating the “democratization of this discursivity [. . .] and speeding up its production and circulation” (Born and Haworth 2017, 611), thus making genre definitions more fluid and dynamic. Finally, social mediations refer to the ease with which online platforms afford the formation of communities around specific music genres and practices. In other words, in online culture, the social experience of genre-defining participation becomes an equally – or sometimes even more – important aspect of a music genre as its sonic and formal features, making the whole experience intermedial. Two examples that Born and Haworth analyze make this point particularly salient: microsound and vaporwave. Before it spread online, microsound was not even a genre in the first place. The term originally referred – and in its strict sense, still refers – to an academic technique of electronic music composition using extremely short (below 100 ms) “grains” of sound. However, as the software tools for granular synthesis became easily obtainable, both legally and otherwise, and a microsound mailing list was established in 1999, an online community including academic, non-academic, and amateur soundmakers emerged around it. This in turn led to the continuous development and redevelopment of a particular microsound aesthetic. Another example, vaporwave, is a musical (and often audiovisual) genre that originated fully in the online culture. Vaporwave’s musical identity is centered around it sounding ironically cheap – for the most part repurposing samples of functional background music from the 1980s and 1990s (muzak, advertisement music, etc.), often haphazardly processed with simple audio filters. Despite unfolding in fully realized music tracks, Born and Haworth argue, vaporwave operates not so much as a music genre as a form of meme culture. Its sonic experience takes a backseat to the discursive act of uncreative appropriation and the sense of communality stemming from ironically sharing a common frame of musical reference. Both microsound and vaporwave examples reveal an intermedial relationship at the core of online musical practices that unfolds through an interaction of sonic aesthetic and community building around it. Olga Goriunova (2012) underscored the dialectic quality of this relationship with her notions of autocreativity and operational aesthetics. She argues that for online creative practices it is impossible to determine whether it is a particular aesthetic that gives rise to a community around it or it is a community that gives rise to a particular aesthetic. Rather, both processes occur simultaneously, with both community and aesthetics being in constant flux. The aesthetic resonance that emerges from this medial-social dynamic may either give rise to “aesthetic brilliance” (Goriunova 2012, 10) or fizzle out into irrelevance.

1130

V. Keylin

Importantly, autocreativity and operational aesthetics may occur even in the absence of a genre or even a defined artistic medium. Many of the sonic practices in participatory culture defy the traditional designations of art disciplines. Some examples include various creative uses of autotune – applying it to animal voices or to storytelling videos – or combining cutups of news clips and other documentary speech recordings with electronic music to produce news remixes. Such practices do not possess a genre identity, nor the required discursive framings necessary to call them a genre. Some of them may have names, like “autotune stories,” but most operate anonymously within the broader meme culture. However, they engage in the same interplay between the experience of social unity and the listening and soundmaking as the musical genres discussed above. Interestingly – though unsurprisingly, given the fundamental referentiality of digital culture (Stalder 2017, Chap. 2) – one of the reasons that many of these practices evade genre definitions is that the aesthetic artifacts they produce are in themselves intermedial, recombining found material from different media into novel constellations. Furthermore, participation even suspends the familiar aesthetic expectation, making highly experimental, in a sense avant-garde, listening experiences palatable to the general public that would often reject them in the context of institutionally defined arts.

Conclusion The analysis of intermedial relationship between sound and participation undertaken in the above sections reveals a number of paradoxes. The first of these concerns the question of whether interactive sound is indeed intermedial. If sound has always been interactive, it would be common sense to consider interactivity its essential feature and not a separate medium with which sound interacts. However, the intermediality of interactive sound is conditioned – in many cases retroactively – by the historical context. While pre-modern musical and sonic practices were indeed largely participatory, modern era was characterized by the movement towards non-participation and professionalization of arts, culminating in linear media such as radio or audio recording that separated sound from participation. The interactive sonic media and arts that emerged – or, in a sense, resurged – in recent decades have to be considered against this linear mediascape, as intermedial: sonic and interactive. The experiential modality of participation as a medium can be defined as the sense of agency, “the satisfying power to take meaningful actions and see the results of our decisions and choices” (Murray 1998, 126). Furthermore, participation itself is already necessarily intermedial. On the one hand, own actions unfold simultaneously in several dimensions: physical, social, and discursive. As a result, the category of interactive and participatory sound spans a wide continuum that encompasses such diverse phenomena as individual sound acts, acoustically mediated social interactions, and sonic practices of participatory culture. On the other hand, as some sort of framing is needed for own actions to become a part of media experience, participation may only exist as a medium in intermedial

43

Interactive and Participatory Sound

1131

constellations. At the same time, the breadth of the continuum of interactive sound means that some of such constellations may occur within a media work, some outside, and some operate across its borders. The non-linearity of interactive and participatory sonic media makes it difficult to analyze them in a traditional way. Instead, the media-ecological approach becomes more salient. The notion of affordance can be applied across the spectrum of participation to shed light onto the reciprocal sound-action relationship on the levels of both basic and qualified media. While all sonic activities can be broken down into elementary soundmaking or sound-facilitated actions, the high-level affordances of qualified sonic media explicate the different ways they may structure and be structured by participation. A characteristic trait of interactive sound is the increased sense of immersion, facilitated on the one hand by responsiveness of interactivity and on the other hand by the immersive, three-dimensional character of sound that reinforce the participants’ connections to and through media. Much like action itself, this immersion also operates on several levels. Most prominent of those is immersion as presence, the feeling of being transported into a virtual environment. Here sound and interactivity may act in accord to enhance the sensation of presence without even necessarily interacting with each other. Self-produced sounds may enhance this sensation when they can be placed inside the virtual environment (i.e., when they are or can be perceived as diegetic), such as sounds of footsteps, or they may “break the fourth wall” (Collins 2014), when they contradict the setting such as streaming pop songs into a fantasy multiplayer game. In any case, however, they strengthen the engagement with the artwork or media work unfolding in a more intellectual-, challenge-, or exploration-based dimension of immersion. Interestingly, such interactions may happen fully outside of the media work in question – such as the musical practices of video game fandoms – yet lead to the players becoming more immersed in the fictional world. A third dimension to immersion is the social immersion characteristic of sound in participatory culture. Already the above example of fandom practices possesses this social dimension as they lead to the increased sense of community between the game’s players. Here intermediality takes the high-level abstracted form of autocreativity – a dialectic between an aesthetic practice and its community that co-constitute each other as both develop. Such sociocultural processes fully transcend individual works and/or media channels yet their traces are fully present in the sonic artifacts that they produce.

References Bishop, Claire, ed. 2006. Participation. Whitechapel. https://mitpress.mit.edu/books/participation. ———, ed. 2012. Artificial hells: Participatory art and the politics of spectatorship. Verso Books. Born, Georgina, and Christopher Haworth. 2017. From microsound to vaporwave: Internetmediated musics, online methods, and genre. Music and Letters 98 (4): 601–647. Bourriaud, Nicolas. 2002. Relational aesthetics. Les Presses du réel.

1132

V. Keylin

Bucher, Taina, and Anne Helmond. 2018. The affordances of social media platforms. In The SAGE handbook of social media, ed. Jean Burgess, Alice Marwick, and Thomas Poell, 233–253. London: SAGE Publications. Collins, Karen. 2013. Playing with sound: A theory of interacting with sound and music in video games. MIT press. https://mitpress.mit.edu/books/playing-sound. ———. 2014. Breaking the fourth wall? User-generated sonic content in virtual worlds. In The Oxford handbook of virtuality, ed. Mark Grimshaw, 351–363. Oxford: Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199826162.013.028. Crossley, Mark. 2021. A recalibration of theatre’s hypermediality. In Beyond media borders, ed. Lars Elleström, vol. 1, 93–112. Cham: Palgrave Macmillan. Csikszentmihalyi, Mihaly. 2009. Flow: The psychology of optimal experience. Harper Collins. Dyson, Frances. 2009. Sounding new media: Immersion and embodiment in the arts and culture. Berkeley: University of California Press. Eco, Umberto. 1989. The open work. Cambridge, MA: Harvard University Press. Eichner, Susanne. 2014. Agency and media reception: Experiencing video games, film, and television. Springer Fachmedien Wiesbaden. Elleström, Lars. 2010. The modalities of media: A model for understanding intermedial relations. In Media borders, multimodality, and intermediality Hrsg, ed. Von Dems, 11–48. Basingstoke und New York: Palgrave Macmillan. ———. 2021. The modalities of Media II: An expanded model for understanding intermedial relations. In Beyond media borders, volume 1: Intermedial relations among multimodal media, ed. Lars Elleström, 3–91. Cham: Springer International Publishing. https://doi.org/10.1007/9783-030-49679-1_1. Fabbri, Franco. 1981. A theory of musical genres: Two applications. In Popular music perspectives, ed. David Horn and Philip Tagg, 52–81. Gothenburg and Exeter: International Association for the Study of Popular Music. Fritsch, Melanie. 2014. Worlds of Music: Strategies for creating Music-based experiences in videogames. In The Oxford handbook of interactive audio, ed. Karen Collins, Bill Kapralos, and Holly Tessler, 168–177. Oxford: Oxford University Press. Garner, Tom Alexander. 2017. Echoes of other worlds: Sound in virtual reality: Past, present and future. Springer. Garner, Tom Alexander, and Mark Grimshaw. 2014. Sonic virtuality: Understanding audio in a virtual world. In The Oxford handbook of virtuality, ed. Mark Grimshaw. Oxford University Press. Gibson, James Jerome. 1979. The ecological approach to visual perception. Boston, MA: Houghton Mifflin. Glassner, Andrew. 2017. Interactive storytelling: Techniques for 21st century fiction. CRC Press. Goriunova, Olga. 2012. Art platforms and cultural production on the internet. New York/London: Routledge. Graham, Beryl. 2010. What kind of participative system? Critical vocabularies from new media art. In The ‘do-it-yourself’ artwork: Participation from fluxus to new media, ed. Anna Dezeuze, 281–305. Manchester and New York: Manchester University Press. Grimshaw, Mark. 2008. Sound and immersion in the first-person shooter. International Journal of Intelligent Games and Simulation 5 (1): 119–124. Heinrich, Falk. 2014. Performing beauty in participatory art and culture. Abingdon: Routledge. Jäger, Ludwig, ed. 2010. Media culture, and mediality. New insights into the current state of research. Bielefeld: Transcript. Jenkins, Henry. 2009. Confronting the challenges of participatory culture. Media education in the 21st century. Cambridge, MA: MIT Press. Jennett, Charlene, Anna L. Cox, Paul Cairns, Samira Dhoparee, Andrew Epps, Tim Tijs, and Alison Walton. 2008. Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies 66 (9): 641–661. https://doi.org/10.1016/j.ijhcs.2008. 04.004.

43

Interactive and Participatory Sound

1133

Kester, Grant. 2011. The one and the many: Contemporary collaborative art in a global context. Duke University Press. Keylin, Vadim. 2019. Medialities of participation in sound art. In Cultures of participation: Arts, digital media and cultural institutions, ed. Birgit Eriksson, Carsten Stage, and Bjarki Valtýsson, 129–146. New York: Routledge. ———. 2020a. Crash, boom, bang: Affordances for participation in sound art. SoundEffects-An Interdisciplinary Journal of Sound and Sound Experience 9 (1): 98–115. https://doi.org/10. 7146/se.v9i1.118243. ———. 2020b. Postcritical listening: Political affordances in participatory sound art. Organised Sound 25 (3): 353–361. ———. 2021. Sound acts: Towards a sonic pragmatism. Sound Studies 7 (1): 83–99. https://doi. org/10.1080/20551940.2020.1857622. Krogh Groth, Sanne, and Kristine Samson. 2017. Sound art situations. Organised Sound 22 (1): 101–111. https://doi.org/10.1017/S1355771816000388. Larsson, Pontus, Aleksander Väljamäe, Daniel Västfjäll, Ana Tajadura-Jiménez, and Mendel Kleiner. 2010. Auditory-induced presence in mixed reality environments and related technology. In The engineering of mixed reality systems, Human-Computer Interaction Series, ed. Emmanuel Dubois, Philip Gray, and Laurence Nigay, 143–163. London: Springer London. https://doi.org/10.1007/978-1-84882-733-2_8. Lombard, Matthew, and Theresa Ditton. 1997. At the heart of it all: The concept of presence. Journal of Computer-Mediated Communication 3 (2): JCMC321. https://doi.org/10.1111/j. 1083-6101.1997.tb00072.x. Machon, Josephine. 2017. Immersive theatres: Intimacy and immediacy in contemporary performance. Bloomsbury Publishing. Murray, Janet H. 1998. Hamlet on the holodeck: The future of narrative in cyberspace. MIT Press. Neitzel, Britta. 2015. Performing games: Intermediality and videogames. In Handbook of Intermediality: Literature – image – sound – music, ed. Gabriele Rippl, 584–603. Berlin: De Gruyter. Nordahl, Rolf. 2005. ‘Self-induced footsteps sounds in virtual reality: Latency, recognition, quality and presence’. In The 8th annual international workshop on PRESENCE, PRESENCE 2005, conference proceedings, 21–23 September 2005, London, United Kingdom, ed. Mel Slater, 353–55. London: University College, London. Nordahl, Rolf, and Niels Christian Nilsson. 2014. The sound of being there: Presence and interactive audio in immersive virtual reality. In The Oxford handbook of interactive audio, ed. Karen Collins, Bill Kapralos, and Holly Tessler. Oxford Handbooks. Oxford University Press. https://doi.org/10.1093/oxfordhb/9780199797226.013.013. Rajewsky, Irina O. 2010. Border talks: The problematic status of media borders in the current debate about intermediality. In Media borders, multimodality and intermediality, ed. Lars Elleström, 51–68. London: Palgrave Macmillan UK. https://doi.org/10.1057/9780230275201_3. ———. 2011. Intermediality, intertextuality, and remediation: A literary perspective on intermediality. Intermédialités: History and Theory of the Arts, Literature and Techniques 6 (August): 43–64. https://doi.org/10.7202/1005505ar. Sanio, Sabine. 1999. Autonomie, Intentionalität, Situation. Aspekte Eines Erweiterten Kunstbegriffs. In Klangkunst. Tönende Objekte Und Klingende Räume, ed. Helga de la Motte-Haber, 67–118. Laaber: Laaber-Verlag. Schulze, Holger. 2018. The sonic persona: An anthropology of sound. The Sonic Persona. London: Bloomsbury. Sheridan, Thomas B. 1992. Musings on telepresence and virtual presence. Presence: Teleoperators and Virtual Environments 1 (1): 120–126. https://doi.org/10.1162/pres.1992.1.1.120. Stalder, Felix. 2017. The digital condition. Hoboken, NJ: Wiley-Blackwell. Turino, Thomas. 2008. Music as social life: The politics of participation. Chicago: University of Chicago Press. White, Gareth. 2013. Audience participation in theatre: Aesthetics of the invitation. New York: Palgrave Macmillan. https://doi.org/10.1057/9781137010742.

Intermediality and Computer Simulation

44

Jens Schro¨ter

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Computer Simulation and Transmaterial Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedial Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmedial Forms in Monsters, Inc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Transmaterial Visual Imagery in Monsters, Inc. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . On the Reflexivity of Monsters, Inc., I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . On the Reflexivity of Monsters, Inc., II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Short Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1136 1136 1139 1140 1142 1142 1143 1145 1145

Abstract

The topic of this chapter relates to widespread phenomena in digital media culture, as will be demonstrated by a close analysis of a popular movie – Monsters, Inc. It will be shown that computer simulation allows for a new form of intermediality, namely, transmaterial forms. These transmaterial forms are often used in digital media products, and there they are combined with transmedial forms, which are a more classical form of intermediality. Monsters Inc. will be analyzed to demonstrate the entanglement of transmaterial and transmedial forms. Keywords

Intermediality · Computer Simulation · Computer Graphics · Animation · Transmaterialization

J. Schröter (*) University of Bonn, Bonn, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_48

1135

1136

J. Schro¨ter

Introduction The topic of this chapter relates to widespread phenomena in digital media culture, as will be demonstrated by a close analysis of a popular movie – Monsters, Inc. (Docter 2001; unfortunately, Disney didn’t respond at all, so no images could be included). It will be shown that the comparatively new technology of computer simulation allows for a new form of intermediality, namely, transmaterial forms (see section “Computer Simulation and Transmaterial Forms”). These transmaterial forms are often used in digital media products, and there they are combined with transmedial forms, which are a more classical form of intermediality (see section “Transmedial Forms”). In sections “Transmedial Forms in Monsters, Inc.,” “Transmaterial Visual Imagery in Monsters, Inc.,” “On the Reflexivity of Monsters, Inc., I,” and “On the Reflexivity of Monsters, Inc., II,” Monsters Inc. will be analyzed to demonstrate the entanglement of transmaterial and transmedial forms. Section “Short Conclusion” is a very short conclusion.

Computer Simulation and Transmaterial Forms 1967, J. C. R. Licklider referred to the development of computer simulations as the most significant event for science and technology since the invention of writing (cf. Licklider 1967, 289). The computer simulation of a real object or process consists of “detaching” mathematically describable structures from the materiality of the object. We can adopt Helmut Neunzert’s definition: In simulations, a “real process [. . .] is represented as mathematics [. . .] so as to be simulated in the computer using algorithms” (Neunzert 1995, 44). This means that based on different kinds of gathered or sampled data, the rules governing (or at least regularities in) the behavior of an object or process, that is, a theory, can be derived. The basic model is translated into a formalized model able to be carried out by a computer. To the extent that – to use Deleuze’s (1994, 209) formulation1 – the structure is the reality of the virtual, simulation models of objects are virtual objects. Take the example of a rubber ball: The bouncing behavior of a ball is abstracted from the material ball made of rubber by observing and measuring it and then putting these data into mathematical equations, in order to then serve as the basis for a model. A “virtual ball” is created. This model can then be represented on a display showing a point that bounces “like a ball.” Other kinds of display are possible, for example, an auditive representation. Often such simulations are not done to produce images but to deliver data that can be used for analytical purposes or to train machine learning systems, etc. The virtual objects can now be used for virtual experiments in different ways: We could allow the model to develop more or less independently (guided by theoretical extrapolations), which permits time to be compressed considerably, so as to see what 1

“The reality of the virtual is structure” (Deleuze 1994, 209).

44

Intermediality and Computer Simulation

1137

the modelled phenomenon presumably will be like, and/or certain parameters are modified, so as to see how the object would behave under different conditions (our virtual ball under different gravity etc.). We could also change the parameters at will just for surprising or aesthetically interesting effects. After 1945, simulations were used to develop the hydrogen bomb (cf. Galison 1997, Chap. 8) and today are omnipresent in natural and even social and economic sciences,2 as well as in the so-called interactive simulations used to train pilots, highspeed train drivers, the staff of nuclear power stations, and other high-risk technologies. They are also present in many forms of computer games, e.g., in the form of “physics engines” that control the behavior of certain objects in the game. But for the discussion here, it is more relevant that computer simulation techniques can also be used to simulate other media. There are many examples of this – one important example is “photorealism” in computer graphics, which made some of its early spectacular appearances in the early 1990s with movies such as Terminator 2 or Jurassic Park. Photorealism is simulation, as the characteristics (of certain aspects) of photographic media are measured and the resulting data and their theoretical descriptions are used as the basis of computer models. A simulated or virtual camera is a real (but not actual, material) camera (cf. Hughes et al. 2014, Chap. 13), which, depending on the data and theories available, can approximate its material model (actual, material cameras with their optical and material properties) as closely as desired (although a more detailed model needs more processing time and resources and that can be a hindrance). This virtual camera is used to virtually take a photograph of a virtual object field, which is lit by a virtual source of light (cf. Mitchell 1992, 117–135). As far as their visual appearance is concerned,3 the pictures generated in this way correspond to the basic characteristics of chemical photography (with the exception of photograms,4 although these can, of course, also be simulated in principle): firstly, the wealth of unintended details. Many generated graphics are classed as not (photo)realistic enough precisely because they are too “clean”; their surfaces do not have enough scratches, spots, and other similar flaws (cf. Newell and Blinn 1977, 445–446). Secondly, there are the effects created through camera optics.5 Computer-generated images could follow any other or indeed no projection logic, but if they want to be photorealistic, they will adopt the linear perspective that is traditional in photography

2 See on the central role of computer simulation for science, among others: Gramelsberger (2010), Lenhard (2019), and Varenne (2019) (esp. Chap. 8). 3 This applies only to their photographic look, not to their arrangement via montage, or to additional sound etc. 4 A photogram is an image formed by light but without lens optics, e.g., objects are placed directly on the photosensitive sheet of paper, and their shadow is recorded. 5 On the simulation of cameras and motion blurring (cf. Potmesil and Chakravarty 1982).

1138

J. Schro¨ter

and film (cf. Foley et al. 1995, 230–237, esp. 231), but also disturbances created by the optical system, if that is desired.6 Thirdly, computer-generated images can attempt to model the qualities of the photographic emulsion itself, such as the grainy structure of the image that is visible particularly in blow-ups or films that are highly sensitive to light (cf. Geigel and Musgrave 1997). The virtual camera, which ultimately is a mathematical construct, can also be changed in whichever way is desired, for example, in order to be able to produce pictures which could not be produced by actual, material cameras. This graininess brings us to our key point: In traditional media, we conventionally distinguish between the materiality of a medium and the forms generated in or through it. These forms can be transmedial,7 such as narrative structures in literature, comics, films, television, and also computer games (see section “Transmedial Forms”). By contrast, materiality was always seen as something that remains more or less invisible “behind” the specific configuration of transmedial forms. Materiality is said to become perceptible in faults or malfunctions, like film tears, crackles, white noise, distortions, motion blurring, or lens flares. Often such signs of the materiality were used in pre-digital media-based art forms, usually explicitly, so as to emphasize the material specificity of the medium “behind” the form (Mersch 2006, 225–226). The simulation models of media are, depending on how detailed they are and what purpose they serve, able to simulate the faults of the very same media and – most importantly – transfer these faults from one medial context to another: In one scene of Monsters, Inc. (Docter 2001), we can see some lens flares, which are created in lens optics when photographs or films are shot against the sunlight – but this film is completely computer generated, and of course, no material camera was used. A photographic fault appears detached from materiality in this otherwise more painting- or cartoon-like animated film.8 In Monsters Inc., as in many other similar films of recent years – we recall Ants (Eric Darnell, Tim Johnson, USA 1998), Toy Story (John Lasseter, USA 1995), Finding Nemo (Andrew Stanton, Lee Unkrich, USA 2003), The Incredibles (Brad Bird, USA 2004), Shrek (Andrew Adamson, Vicky Jenson, USA 2001), and many more besides – photorealistic techniques are used in 6

But it is important to add that computer graphics doesn’t have to be photorealistic in the sense that it does try to simulate the look of photographic images, although that might be a goal especially if computer-generated graphics are to be integrated in an otherwise photographically recorded movie (as is often done in popular cinema). Computer graphics can be abstract or highly stylized (cf. Gooch and Gooch 2001). If the goal is to model objects without clear-cut boundaries (like fire, fog, or clouds), agent-based simulations like particle systems are used (cf. Reeves 1983). And having said all this, we’re still not talking about many different methods about lighting the scene in object space, of texture mapping, and so on (see the comprehensive overview in Hughes et al. 2014). Computer graphics has many different roots and forms that cannot be reduced to one medial or pictorial “specificity” (cf. Schröter 2018). 7 See Schröter (1998) for a clarification of the notion of transmediality. Transmedial forms are those that are not specific for a given medium, e.g., central perspective which can appear in painting, analogue or digital photography, analogue or digital film and video, etc. 8 There is specialized software only to generate lens flares; see, e.g., LensFlareStudio, http://www. brainfevermedia.com/lensFlareStudio.html

44

Intermediality and Computer Simulation

1139

combination with cartoon-style images to create a new intermedial type of image that falls between the imagery of animation and photography. I propose calling this process transmaterialization; it produces forms that are not only transmedial but transmaterial. While transmedial forms (such as narrative techniques) cannot be attributed to any medium in particular (see section “Transmedial Forms”), transmaterial forms emphatically refer to the respective “specific” materiality (e.g., a lens flare refers to “photography” or at least to photographic optics) of a medium, but in a different medial context.9 Such forms do not refer selfreflexively to a basic materiality (such as the forms that modernist aesthetics is concerned with, Greenberg 1993), but rather digitally repeat selected and isolated materiality forms of previous media and make them available for different uses.

Transmedial Forms As already mentioned in section “Computer Simulation and Transmaterial Forms”: Transmedial forms are forms which appear in artifacts of varying media provenance. A relatively simple example is central perspective, which was developed around 1425 and codified for the first time (in mathematical terms as well) in Alberti’s De Pictura in 1435. It is a procedure for the pictorial representation of space, which is available to painting, but does not necessarily have to be followed (other cultures have favored other procedures, e.g., parallel perspective in Asian painting; see below). In technical visual media which follow geometrical optics, such as photography, film, or video, this mode of representation must be followed (borderline cases occur when certain kinds of telephoto lenses are used). In digitally generated images, on the other hand, central perspective is optional, since, as Friedrich Kittler (2001, 35) once put it, “computer graphics make optic modes optional at all” (in computer graphics you can use every mode of representation that can be calculated). Central perspective is a mode of composition found in images across different media and can be formally identified in a comparison of images by means of the diagonal vanishing lines, which lead to one or sometimes more vanishing point(s). In the history of computer graphics, incidentally, some of the computer scientists who developed algorithms for representation using central perspective are known to have studied the relevant textbooks from the Renaissance (and later) and the instructions given there – some of which were already formulated in mathematical terms. Another important transmedial form that can be used by very different media is narrative structuring in time. Thus, for example, the narratologist Seymour Chatman (1981, 117, emphasis mine) once noted: “One of the most important observations to come out of narratology is that narrative itself is a deep structure quite independent of its medium.” Admittedly this thesis has repeatedly been subjected to critical discussion, but it does seem to have some validity at least: If it were not so, there would be no film adaptations of literature. Another example from narratology is Bordwell 9

“The computer marks a certain dissociation of media from technics” (cf. Hansen 2010, 178).

1140

J. Schro¨ter

(1993, 51), who writes: “As a distinction the fabula/syuzhet pair cuts across the media. At a gross level, the same fabula could be inferred from a novel, a film, a painting, or a play.” Fabula is his expression for story, and syuzhet, his expression for the plot (more or less, in any case). We can refer to such forms as “transmedial intermediality” (cf. Schröter 1998, 136–143). This does not mean, however, that all forms are transmedial and equally available to all media. Painting and drawing have always also included modes of representation using parallel perspective, which have no vanishing point and which are still preferred in technical drawing and architectural drafting because they avoid changes in angle and relative changes in length. Photographic media cannot represent such forms (they can only approximate them in the borderline case of certain telephoto lenses), since they follow the behavior of the light, whether their mode of recording is chemical, electronic, analogue, or digital. Computer-generated images, on the other hand, since they can represent anything which is computable within a reasonable time, can also use forms based on parallel perspective (cf. Beil and Schröter 2011). This means that it is necessary to analyze precisely, in each specific case, which forms have been connected with which other forms and in what way – and to which media these forms are available or unavailable. A concrete example is analyzed in the following parts. In section “Transmedial Forms in Monsters, Inc.,” the transmedial forms in Monsters, Inc. are described, and in section “Transmaterial Visual Imagery in Monsters, Inc.,” it is shown how this film utilizes transmaterial forms. In sections “On the Reflexivity of Monsters, Inc., I” and “On the Reflexivity of Monsters, Inc., II,” the reflexivity of Monsters, Inc. produced by the entanglement of transmedial and transmaterial forms is analyzed.

Transmedial Forms in Monsters, Inc. If we first consider the structure of Monsters, Inc. on the level of the narration, we can note that the film follows the “classical Hollywood mode of narration” as described by Bordwell, Thompson et al. for Hollywood cinema from about 1917 up to today. Bordwell (1986, 18) writes: “The classical Hollywood film presents psychologically defined individuals who struggle to solve a clear-cut problem or to attain specific goals.” Clearly this also applies to Monsters, Inc. There is no indistinct blurring of objectivity and subjectivity, as in many forms of what Bordwell (1993, Chap. 10) refers to as the “art cinema mode of narration.” Instead, a clear situation is established at the outset: Sulley and Mike work at the company Monsters, Inc., after which the film is named, and are depicted as successful and, in this sense, careeroriented monsters; a subtle rendering of their facial expressions shows a psychological inner life which, for example, clearly associates success with enjoyment. And then, with the accidental entry of the small child (Boo) into the monsters’ world, a problem arises which upsets the stable situation. For the remainder of the film, Sulley and Mike try to solve the problem, i.e., to return Boo to her world, facing various complications on the way. In the end, they succeed. The whole construction of the film serves to build up the causal steps of this chain of action as clearly and

44

Intermediality and Computer Simulation

1141

unambiguously as possible. According to Bordwell (1986, 27–28): “Most explicitly codified into rules is the system of classical continuity editing. The reliance upon an axis of action orients the spectator to the space [. . .]” and: “[M]ost Hollywood scenes begin with establishing shots, break the space into closer views linked by eyelinematches.” This classic structure can be found in precisely this form in Monsters, Inc. There is, e.g., a dialogue sequence, which follows exactly the codified rules of Hollywood narration. Before beginning their work at Monsters, Inc., Sulley, the furry monster, and Mike, his round, green friend, get ready in a sort of changing room: 1st shot: Establishing shot, the space is established, along with a line of sight (eyeline match) between Mike and Sulley; 2nd shot: The antagonist, Randall, is introduced, and a new line of sight is created between him and Mike; Mike gets a fright and jumps over the bench to Sulley’s side (incidentally, the psychological construction of the characters can be studied particularly well by watching Mike’s face here) – the eyeline match remains in place, however; 3rd–6th shots: A classic sequence of shot/reverse shot begins here, whereby the virtual camera always remains on this side of the eyeline; i.e., it follows the traditional 180-degree rule; 7th shot: There is another long shot which makes the spatial configuration absolutely clear again. In short, the construction of the space is completely focused on consistency. The space is intended to be the stable background for the development of the causal chains of action by the protagonists and antagonists and is not supposed to confuse matters by intervening itself. This is typical of the classic Hollywood film. Deviations from this, such as a conspicuously tilted line of sight, are at some points in the film only permissible because they are connected with a hectic chase situation, Bordwell (1986, 27): “Stylistic disorientation, in short, is permissible when it conveys disorienting story situations.” In short, the film confirms the assertion that “classical narration quickly cues us to construct story logic (causality, parallelisms), time, and space in ways that make the events ‘before the camera’ our principal source of information” (ebd., 24). But in a computer-generated film, there actually is no “before the camera” (unless we count the virtual space ‘in front’ of the virtual camera, but that’s quite metaphorically). It is significant that during the closing credits of the film, very amusing “bloopers” are shown, constructing “pre-film events” with an ironic wink: the clapper board, a microphone in the picture, and finally an out-of-control machine which knocks over the “camera.” Here Monsters, Inc., is of course ironizing its own mode of narration (and its “production culture,” cf. Caldwell 2008) – in one of the “blooper” scenes, a monster botches a dialogue and is berated by his monster colleague: “You’re messin’ up this scene, we’re never gonna work in Hollywood again” – precisely classical Hollywood narration. In short, Monsters, Inc., although completely digitally simulated, follows this classic narrative tradition – a set of transmedial forms that

1142

J. Schro¨ter

intermedially connect photographic and computer-generated film but can also be related to narrative forms in comics or literature.

Transmaterial Visual Imagery in Monsters, Inc. The transmedial forms appear also in the visual imagery, the look of Monsters, Inc. The images are organized using central perspective. Now computer graphics do not have to have central perspective, of course; unlike photographic media, the choice of central perspective in computer graphics is always a conscious stylistic decision – and here, or course, its purpose is to make the cartoon image seem photorealistic at the same time. The reference to the simulation of photography is obvious in many respects: For example, when Sulley observes Randall pursuing his machinations from under a table, and the table legs and edges in the foreground are out of focus; or the other way around, when the background is out of focus. In photographic optics (be it photography, film or analogue, or digital video), such varying levels of focus are part of the dispositive and can mess up an image, consciously be used, or carefully avoided, in simulated images; on the other hand, they have to be wanted and brought about deliberately, e.g., in order to achieve a photorealistic effect. Many images also show another typical way of getting closer to the visual imagery of photography, marking a considerable difference from many cartoon styles – that is, the numerous apparently random surface details. So, the point is: faults which result from the material specifics of media technologies “behind” the transmedial forms become transmaterial forms themselves. Here, it is faults in photographic optics which are transferred into a completely different context, in this case the cartoon. For the visual imagery of Monsters, Inc., is not simply photorealistic: on the contrary, the film links photographic with cartoonish visual imagery, as can be seen in, amongst other things, the extreme colorfulness, especially of the shadows in many scenes. This hybrid form of image is the actual new visual/aesthetic achievement of the Pixar films (I exclude a few marginal predecessors in computer graphics research). This role of drawing and painting, the tradition to which cartoons and animation belong, is thematized intradiegetically at various points in the film, for example, when the childish drawings produced by Boo point directly to the potential of non-photorealistic rendering (cf. Strothotte and Schlechtweg 2002). This points to the reflexivity of Monsters, Inc.

On the Reflexivity of Monsters, Inc., I It is hardly necessary to point out that the photographic monocular is evoked even in the logo of Monsters, Inc., which is also one-eyed. Again, this underlines the highly reflexive character of this and other Pixar movies. That is no coincidence. Pixar, the firm behind Monsters, Inc., was substantially built up with money from Steve Jobs and does not only make money with films. Since 1989, it has also been selling software, PhotoRealistic RenderMan, based on the RenderMan standard. Pixar also

44

Intermediality and Computer Simulation

1143

defines the cutting edge of the computer graphics industry standard (cf. Pixar n.d.). Seen in this light, the films are also advertising for the graphic achievements of Pixar. The technical state of the art determines the choice of subject of the films. Hence, Friedrich Kittler (2001, 36) noted that in 1998 “Not coincidentally, computergenerated films like Jurassic Park do not even attempt to compete with the fur coats [sic] in Hans Holbein’s The Ambassadors; they content themselves with armored and thus optically unadorned dinosaurs.” But in Monsters, Inc., 2001, it was the rendering of fur and hair which was foregrounded, precisely because it had previously been difficult to simulate such complex structures convincingly (cf. Failes 2021). This is the reason for the narrative digression of Mike and Sulley’s banishment to the Himalayas: When Sulley attempts to reach a nearby village, he falls from the sled and lies in the snow, and his fur is blown about by the harsh wind and gradually covered by snowflakes. This scene demonstrates what was then the state of the art in the simulation of moving furlike surfaces. Knowledge of this function of the Pixar films can in itself become an attraction for viewers. Thus, neo-formalist film theoretician Kristin Thompson commented: “For me, part of the fun of watching a Pixar’s film is to try and figure out what technical challenge the filmmakers have set themselves this time. Every film pushes the limits of computer animation in one major area, so that the studio has been perpetually on the cutting edge” (Thompson 2006). Certain elements of the film, then, are not simply subordinated to the narrative process. The lens flares, for example, have no function in the development of the causal chain, nor does Sulley’s elaborate fur; furthermore, this – in the words of David Bordwell – could at best be transtextually motivated, as something borrowed from a knowledge of the design of monster films. But they represent elements which can be understood in Kristin Thompson’s terms as excess, or in Bordwell’s terms as purely “artistically motivated” (Thompson 1986; Bordwell 1993, 36, 53, 164 and passim). These are elements which display their own fabricated nature and thus form a discourse about the state of development of the computer image, over and above the narrative. In this respect, the new visual imagery of the Pixar films does in fact change the narrative. Transmaterial forms transform transmedial forms. Although the narrative largely conforms to the “classical Hollywood mode of narration,” it is – to use another term of Bordwell’s (1993, 58–59) – more “self-conscious” or “self-referential,” since it does not merely conceal itself in order to seamlessly convey the story/fabula/information, as is usually the case in this mode of narration. Attention is increasingly focused on its own fabricated nature, to the point where one wonders whether the film’s subject was chosen as a showcase for a specific new accomplishment of simulated visual imagery.

On the Reflexivity of Monsters, Inc., II The increased self-referentiality or self-consciousness of the narrative, which arises from its interference with the hybrid, transmaterial, simulated visual imagery (and the discourses surrounding this) in Pixar films and in particular in Monsters, Inc.,

1144

J. Schro¨ter

reveals itself in the many self-reflexive references, some of which have already been mentioned. There are many more levels and ways in which the film is reflexive. Thus, in Film Theory. An Introduction through the Senses, Thomas Elsaesser and Malte Hagener (2010, 170–187) explicitly pointed out the role of the doors through which the monsters can enter the children’s world in order to frighten them. This evokes discourses about the “portal to another world” which have, since the 1990s, referred directly to cyberspace and virtual reality (see Schröter 2004, 227). Furthermore, Monsters, Inc., i.e., the company our monsters work for – is an industry for the production of terror and (at the end of the film) laughter; so in this sense, it is a reflection of the production of affect by the film industry. All that can be added to this precise analysis is that the motif of the door later expands into a massive archive of doors, a database; this in turn, to paraphrase Lev Manovich, introduces a new theme to the digital film: the logic of the database, which is typical of the new media (cf. Manovich 2001, 212ff.). Furthermore, in the chase at the end of the film, the doors operate, as it were, as shortcuts through the diegetic space, which is at the same time global space, and allow a sort of montage within the image, which in turn displaces and reflects the forms of spatial construction in classical Hollywood cinema. I would like to finish, however, by discussing something much more straightforward. Monsters, Inc., begins in a simulator. The sequence is established with sounds off camera, indicating that parents have put their child to bed; in the establishing shot (which is in fact the third shot), we see the child sleeping. The door – that portal to the monsters’ world – opens. A monster has entered. It rears up to frighten the child, the child screams – and what happens? The monster gets the most dreadful fright itself, trips over a football, hurts itself: in short, messes everything up. Then the light goes on. A technical voice off camera repeats again and again: “Simulation terminated,” and we learn that the child was only a machine. And in a further doubling of the theme of the door to another world, one wall of the apparent child’s bedroom slides up, and we see the trainer as she tries to explain to the monster-in-training (and to the other monster trainees who are watching) what he has done wrong – in the first instance, this is an allusion to the diegetic 4th wall. More important still, it is a simulator, just like those flight simulators which, in some respects at least, stood at the beginning of the development of certain forms of photorealistic computer graphics (cf. Schröter 2003). And one of the reasons why the simulator is established here is because it appears again later on. Sulley and Boo, on the run from the evil boss of the company – the classical evil capitalist of Hollywood cinema, later to be replaced by Sulley as the good capitalist – have apparently fled through a door into a child’s bedroom. The evil boss, who is also behind Randall’s machinations, wants to get hold of Boo, but when he reaches out to seize her from the bed, it turns out that they are in the simulator. The evil boss is utterly confused. But that is not important anymore, because he has just revealed his sinister plans to Sulley while in the simulator, thinking it was a child’s bedroom. However, Mike was controlling the simulator and has recorded the boss’s crucial confession on a sort of videotape. This representation not only reflects back to another predigital visual form, in that the interlace lines are part of the simulation.

44

Intermediality and Computer Simulation

1145

More importantly, a turning point in the narrative is explicitly connected with the theme of simulation here. Here, the interference between the narrative and the simulative visual imagery in Monsters, Inc., is itself thematized intradiegetically.

Short Conclusion I tried to argue that computer simulation allows for the new form of transmaterial intermediality and my detailed reading of Monsters, Inc., tried to demonstrate the entanglement of transmaterial and (more traditional) transmedial forms in a concrete text. This combination of transmaterial and transmedial forms is very characteristic for the formal strategies in many texts of digital media culture and, therefore, represents a central form of intermediality today.

References Beil, Benjamin, and Jens Schröter. 2011. Die Parallelperspektive im digitalen Bild. Zeitschrift für Medienwissenschaft 4: 127–138. Bordwell, David. 1986. Classical Hollywood cinema. Narrational principles and procedures. In Narrative, apparatus, ideology, ed. Phil Rosen, 17–34. New York: Columbia University Press. ———. 1993. Narration in the fiction film. London: Routledge. Caldwell, John. 2008. Production culture: Industrial reflexivity and critical practice in film and television. Durham/NC: Duke University Press. Chatman, Seymour. 1981. What novels can do that films can’t (and vice versa). In On narrative, ed. W.J.T. Mitchell, 117–136. Chicago/London: University of Chicago Press. Deleuze, Gilles. 1994. Difference and repetition. New York: Columbia University Press. Docter, Pete, director. 2001. Monsters, Inc. Walt Disney Pictures/Pixar Animation Studios, 1 hr., 32 min. Elsaesser, Thomas/Hagener, Malte. 2010. Film theory. An introduction through the senses. New York: Routledge. Failes, Ian. 2021. ‘Monsters, Inc.’ is 20 years old – look back at Pixar’s fur tech breakthroughs. Accessed January 9, 2021. https://beforesandafters.com/2021/11/01/monsters-inc-is-20-yearsold-look-back-at-pixars-fur-tech-breakthroughs/ Foley, James D., Andries van Dam, Steven K. Feiner, and John F. Hughes. 1995. Computer graphics. Principles and practice. 2nd ed. Reading, MA: Addison-Wesley Professional. Galison, Peter. 1997. Image and logic. A material culture of microphysics. Vol. 50, 65. Chicago/London: Chicago University Press. Geigel, Joe, and F. Kenton Musgrave. 1997. A model for simulating the photographic development process on digital images. SIGGRAPH Proceedings: 135–142. Gooch, Bruce, and Amy Gooch. 2001. Non-photorealistic rendering. Natick, MA: A. K. Peters. Gramelsberger, Gabriele. 2010. Computerexperimente: Zum Wandel der Wissenschaft im Zeitalter des Computers. Bielefeld: Transcript. Greenberg, Clement. 1993. Modernist painting. In Clement Greenberg. The collected essays and criticism, Modernism with a vengeance, 1957–1969, ed. John O’Brien, vol. 4, 85–93. Chicago and London: University of Chicago Press. Hansen, Mark B.N. 2010. New media. In Critical terms for media studies, ed. W.J.T. Mitchell and Mark B.N. Hansen, 172–185. Chicago/London: University of Chicago Press.

1146

J. Schro¨ter

Hughes, John F., Andries van Dam, Morgan McGuire, David F. Sklar, James D. Foley, Steven K. Feiner, and Kurt Akeley. 2014. Computer graphics. Principles and practice. 3rd ed. Upper Saddle River, NJ: Addison Wesley. Kittler, Friedrich. 2001. Computer graphics. A semi-technical introduction. Grey Room 2 (winter): 30–45. Lenhard, Johannes. 2019. Calculated surprises. A philosophy of computer simulation. Oxford: Oxford University Press. Licklider, J.C.R. 1967. Interactive dynamic modeling. In Prospects for simulation and simulators of dynamic systems, ed. George Shapiro and Milton Rogers, 281–289. New York/London: Spartan. Manovich, Lev. 2001. The language of new media. Cambridge/MA: MIT Press. Mersch, Dieter. 2006. Medientheorien zur Einführung. Hamburg: Junius. Mitchell, W.J. 1992. The reconfigured eye. Visual truth in the post-photographic era. Cambridge, MA: MIT Press. Neunzert, Helmut. 1995. Mathematik und Computersimulation. Modelle, Algorithmen, Bilder. In Simulation. Computer zwischen Experiment und Theorie, ed. Valentin Braitenberg and Inga Hosp, 44–55. Reinbek bei Hamburg: Rowohlt. Newell, Martin E., and James F. Blinn. 1977. The progression of realism in computer-generated images. ACM 77. Proceedings of the Annual Conference: 444–448. Pixar. n.d. ReenderMan. Accessed January 9, 2021. http://renderman.pixar.com/view/renderman Potmesil, Michael, and Indranil Chakravarty. 1982. Synthetic image generation with a lens and aperture camera model. ACM Transactions on Graphics 1 (2): 85–108. Reeves, William T. 1983. Particle systems-a technique for modeling a class of fuzzy objects. Computer Graphics 17 (3): 359–375. Schröter, Jens. 1998. Intermedialität. Facetten und Probleme eines aktuellen medienwissenschaftlichen Begriffs. montage AV 7 (2): 129–154. ———. 2003. Virtuelle Kamera. Zum Fortbestand fotografischer Medien in computergenerierten Bildern. Fotogeschichte 23 (88): 3–16. ———. 2004. Das Netz und die Virtuelle Realität. Zur Selbstprogrammierung der Gesellschaft durch die universelle Maschine. Bielefeld: transcript. ———. 2018. Especially the ‘ambient term,’ was a terrible thing. Ambient und Atmosphäre in der Computergraphik. In Ambient. Ästhetik des Hintergrunds, ed. Jens Schröter, Gregor Schwering, Dominik Maeder, and Till A. Heilmann, 167–184. Wiesbaden: Springer VS. Strothotte, Thomas, and Stefan Schlechtweg. 2002. Non-photorealistic computer graphics: Modeling, rendering, and animation. San Francisco/CA: Morgan Kaufmann. Thompson, Kristin. 1986. The concept of cinematic excess. In Narrative, apparatus, ideology, ed. Phil Rosen, 130–142. New York: Columbia University Press. ———. 2006. Reflections on CARS. Accessed January 9, 2021. http://www.davidbordwell.net/ blog/2006/10/08/reflections-on-cars/ Varenne, Franck. 2019. From models to simulations. London/New York: Routledge.

Intermediality and Digital Fiction

45

Bartosz Lutostański

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hypertext and Hypermedia Fiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Interactive Fiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cybertext . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Network Fiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Small-Screen Fiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1148 1149 1151 1154 1156 1158 1161 1163 1164

Abstract

From hypertext to app fiction, intermediality has been a key concept in the 40-year-or-so evolution of digital fiction. A good case in point is hypertext fiction which straddles two distinct media: print-based literature and digital environment. The goal of the following chapter is to summarize and synthesize information derived from primary and secondary sources with a view to exploring digital fiction as inherently intermedial, that is, generated at the nexus of multiple media. I discuss hypertext, hypermedia, cybertext, interactive fiction, network fiction, and small-screen fiction, and indicate what role and function intermediality has played in their construction and consumption. To achieve that goal, I first employ theories and typologies of major literature and media scholars to map out my own conceptual territory and subsequently reference established and renowned theoreticians and practitioners of digital fiction. Keywords

Intermediality · Media · Digital fiction · Electronic literature · Genre B. Lutostański (*) University of Warsaw, Warsaw, Poland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_56

1147

1148

B. Lutostański

Introduction The history of digital fiction is typically divided into six phases or, as Astrid Ensslin (2007) puts it, generations (cf. van Looy and Baetens 2003; Rettberg 2019).1 To briefly recapitulate, hypertext fiction, constructed by means of hyperlinks and lexias (textual nodes), belongs to the first phase. The integration of multimodal and multimedia resources with hypertextual architecture results in hypermedia fiction. It “investigates the borders between different modalities and art forms, such as the borders between literature, music, graphics, and photography” (Bell et al. 2013, 9). Interactive fiction constitutes the third phase (Hayles 2008); it draws on ludic affordances by combining verbal narrative with various game devices or genres, such as “Choose Your Own Adventure” or role-playing. The fourth generation is what Ensslin (2007), adapting from Espen Aarseth (1997), calls cybertext; it is fiction with a diminished readerly agency and the underlying machine code either fully or partially in control. Network fiction, a term which defines, broadly speaking, “electronic literature created for and published on the Internet” (Rettberg 2019), belongs to the fifth wave. The term refers both to the multiple platforms and websites (network sites) designed for creation and dissemination of fiction and to the collaborative and communal practices revolving around fiction writing (social networks). More recently, there has been research into digital fiction delivered on mobile devices, for example, smartphones or tablets. Ensslin et al. (2017) conceptualized new storytelling practices as “small-screen fiction” (the sixth generation of digital fiction). Considering this term too general and vague, Magdalena RembowskaPłuciennik and myself (2020 and 2022 [forthcoming]) have proposed to construe the fiction in the form of a downloadable application for mobile devices as app fiction. In the following chapter, I discuss the six abovementioned genres of digital fiction from the context of intermediality. I commence by addressing hypertext and hypermedia as intermedial in the consequence of the “convergence” (Landow 1991) of the literary and the digital. A hypertextual text is uniquely constructed with nodes of text (dis)connected by hyperlinks strewn across them. Interactive fiction, too, integrates the literary and the digital in its narrative structure relying in turn on ludic conventions which gear the audience with the capacity to “interact” with(in) the storyworld. An autonomous “text machine,” cybertext highlights the technological component of text production and consumption due its bespoke code and operates with relatively minimal audience input. Network fiction draws from Internet-related writing and reading practices, and encompasses curated websites which foster collaboration and cooperation in text production process; consequently, this genre appears heavily determined by websites’ ecology and affordances. Small-screen

Importantly, I exclude from my ensuing discussions multiple hybridic subgenres of digital fiction due to space limitations of this publication. A good case in point of my exclusion is Twine, a subgenre hinged on hypertext and interactive fiction and network fiction (for research on Twine, see, for example, Ensslin and Skains 2017).

1

45

Intermediality and Digital Fiction

1149

fiction, finally, is intermedial in multiple ways thanks to the flexibility of digital application environment; as illustrated below, the app environment allows for a production of fictional storyworld tailored and geared for a specific narrative project by exploiting any literary and nonliterary conventions required. Simultaneously, app fiction motivates the imbrication of fiction within the contexts of mobile media culture and mobile technology practices of the audience.

Intermediality It can be argued that we are now going through the “media turn” (Bewell et al. 2011; McDowell 2017), having succeeded the linguistic, narrative, cultural, digital, and other turns of the recent decades. The concepts of mediality, mediation, intermediality, transmediality, plurimediality, multimediality, and even multimodality have become a staple of a plethora of research fervently carried out in various humanities departments. Such terminological abundance necessarily breeds “fuzziness” (Grishakova and Ryan 2010), and the key term of this collection frequently appears fuzzy, too. In order to both map out the conceptual terrain for my chapter as well as avoid any misunderstandings of my discussions, I would like to succinctly define intermediality of most use to me. To put it crudely, intermediality, not to be confused with Hayles’s “intermediation” (2008), refers to the presence of medium 1 within medium 2; it can thus be understood as the practice of integrating one medium (or more) within another, dissimilar medium. As Gabriele Rippl (2015) posits in the introductory chapter to the Handbook of Intermediality, “Intermedial literary texts transgress their own medial boundary – writing – in many creative ways by including pictures and illustrations or by referring to absent (static and moving, analog and digital) pictures, by imitating filmic modes or by mimicking musical structures and themes” (np.). This explanation implies Irina O. Rajewsky’s (2005) and Werner Wolf’s (2011) general differentiation of intermediality into its “narrow” and “broad” senses; the former “focuses on the participation of more than one medium within a human artefact,” while the latter “applies to any transgression of boundaries between conventionally distinct media” (Wolf 2011; see Grishakova and Ryan 2010). Elsewhere, Wolf (2005) develops the notion of “broad intermediality” and makes detailed distinctions into extracompositional intermediality and intracompositional intermediality (cf. Elleström 2014, 8–9). The latter includes: transmediality and intermedial transposition, while the former consists in plurimediality and intermedial reference which further subsumes thematization, evocation, and imitation (Wolf 2005, 253–255; see Rajewsky 2005). In this chapter, Wolf’s intracompositional intermediality, one that refers to a single literary work or genre, will be of primary use. Hence, it seems prudent to briefly characterize plurimediality and intermedial reference. Plurimediality occurs “whenever two or more media are overtly present in a given semiotic entity,” asserts Wolf (2005, 254), and goes on to explain, “This co-presence implies that the components of the medial mixture are discernible on the level of the signifiers without being semiotically dependent on each other . . . Such discernibility

1150

B. Lutostański

can be assumed as given even when the medial components are not ‘quotable’ separately” (Wolf 2005, 254). Hypermedia fiction and app fiction, for instance, will be described as intermedially plurimedial inasmuch as they are, to adapt Wolf’s phrase, “complex syntheses of medial components” with signifiers independent of each other (2005, 254). In intermedial reference, works “seem to be medially and semiotical homogenous, for the involvement of another medium here takes place only covertly or indirectly: through signifiers and sometimes also signifieds pointing to it” (Wolf 2005, 254). Specifically, intermedial thematization is present whenever another medium is mentioned or discussed in a text; evocation refers to the imitation of “the effects of another medium or heteromedial artefact by monomedial means”; and, finally, formal imitation “consists in the attempt to shape the material of the semiotic complex . . . (its signifiers, in some cases also its signifieds) in such a manner that it acquires a formal resemblance to typical features or structures of another medium” (Wolf 2005, 255). To illustrate, hypertext fiction, cybertext fiction, and interactive fiction employ intermedial reference in miscellaneous ways, in particular formal imitation: these genres transform their textual substance by adapting it to a code-driven presentation. Quite aptly then, in the 2015 ELO Conference’s CfP, the organizers construed electronic literature as a whole as “an intermedial field of practice” (Baetens and Sánchez-Mesa 2015). In other words, digital fiction is inherently intermedial, conceived at the nexus of distinct media and their multifarious and multifaceted semiotic, cultural, and technological dimensions (Ryan 2014). The semiotically oriented research into media products has been carried out under the umbrella term of multimodality (Kress and van Leeuven 2001; Gibbons 2011; Hallet 2014; see Rajewsky 2010). I repeatedly use it in my discussions below when pointing out the copresence of multiple semiotic resources at work (words, images, sounds, videos, and haptics). In addition, digital fiction exists within a complex cultural dimension. Following Ryan (2014), the cultural dimension of digital fiction “addresses the public recognition of media as forms of communication and the institutions, behaviors, and practices that support them” (30). In other words, any explorations of digital fiction cannot avoid addressing the issue of how it is used and perceived by authors and users alike. Intermediality of digital fiction in the technological dimension refers primarily to the entanglement of literature, traditionally determined by the print and paper format since Gutenberg, and digital architecture of new means of communication. Digital fiction is dependent on the hardware and software of a digital device used to consume a work of fiction (Ryan 2006, 126). In this way, the technological is “marked” in the sense of being “thematized and explored creatively” as opposed to the “invisible” print format of conventional literature (Atã and Schirrmacher 2021, 45; Ensslin 2007, 37; O’Sullivan 2019, 23–25). Far from being deterministic, digital fiction has exploited and adjusted technological innovations of the last several decades (be it hypertext, Flash, HTML, specific software and hardware, or online services) in pursuing innovative and exciting avenues for literary expression. As Marie-Laure Ryan posits, “It is evident

45

Intermediality and Digital Fiction

1151

that developments on the level of hardware had a crucial impact on the features of digital texts: for instance, faster processors and expanded storage capabilities allowed the integration of word, image, and sound, while the creation of large computer networks allowed communications between multiple users and the collaborative construction of the text. But the form and content of digital texts, as well as the reader’s experience, are also affected by the underlying code” (2006, 126). In short, to paraphrase Alex Goody (2011), digital technology has been a key factor in contemporary culture and literature. Accordingly, digital technology should be treated as an ontologically distinct medium due to the fact that it provides a distinct type of mode of production and material support: from a PC, laptop, tablet, or smartphone as well as the Internet to specific online services. To reiterate, intermediality is central to digital fiction. As a matter of fact, in the case of this type of fiction, we can actually be talking about, following Wolfgang Hallet (2015), “genre-specific intermediality,” a specific kind of intermediality which overtly and covertly contributes to the fiction’s ontology and semiotics (see Douglas 2000, 6–10). In what follows, I succinctly explore the six major digital fiction genres in the context of Wolf’s typology of intracompositional intermediality and Ryan’s complex aggregate of media dimensions.

Hypertext and Hypermedia Fiction As Rettberg asserts (2019), “Hypertext fiction was the first form of electronic literature to garner sustained critical interest during the late 1980s and early 1990s. A small but dedicated group of writers began to work seriously in the genre at the same time as the personal computer and then the Internet were becoming widely adopted, writing stories designed as interlinked fragments of text, with multiple possible reading sequences to be navigated through the reader’s selection of links between them” (see also Nielsen 1990a; Delany and Landow 1994; Joyce 1996; Bolter 2001; Ciccoricco 2007; Bell 2010). Major formal and aesthetic implications of such fiction are multilinearity, metafictional, and self-reflexive potential, and emphasis on reader’s engagement with the system of hyperlinked textual units (lexias) (Ensslin 2007; Nielsen 1990a; Bolter 2001, ch. 7). As David Ciccoricco (2007) contends, following Landow (1994), “Hypertext documents fall into three basic categories: axial, arborescent, and networked. In general terms axial denotes a structure situated along an axis; arborescent denotes a branching structure resembling a tree; and networked denotes an interconnected system of nodes in which there is no dominant axis of orientation” (5; italics in original). The last subtype of hypertext (network) realizes the “promise of hypertext technology” to the fullest: it is “dispersed, multiply centered networked organization inherent in electronic linking” (Landow 1994, 23), rhizomic and ramifying without one “authoritative” narrative structure, indefinitely deferring closure and signification (Douglas 1994; Ensslin 2014a, 259), and “ergodic” (Aarseth 1997). Quoting from Theodor H. Nelson, who coined the term in his 1965 lectures at Vassar College, hypertext refers to “nonsequential writing – text that branches and

1152

B. Lutostański

allows choices to the reader, best read at an interactive screen. As popularly conceived, this is a series of text chunks connected by links which offer the reader different pathways” (qtd. in Landow 1991, 4; on a critique of nonsequentiality and nonlinearity, see, for example, Aarseth 1997, 41–47; Landow 2006, 41–43). The combination of text and links generates intermediality of hypertext fiction whose ontological and hermeneutic status is grounded in the nexus of literary fiction (“text”) and electronic technology (“interactive screen”). In other words, the affordances of hypertext technology are generated by its specific structuring of chunks of texts and links that connect them. The intermedial status of hypertext fiction informs some of its key formal and semantic features: blurring “the boundaries between reader and writer” and leading to the so-called “(w)readership” (Landow 1991, 5; Coover 1992; Delany and Landow 1994, 9–19; see the critique in Simanowski 2004 and Ensslin 2007); multiplicity of readings, multilinearity and fragmentariness, multiple readings (“aesthetics of revis(itat)ion” (Ensslin 2007; Ciccoricco 2007, 11); spatialized or visual text (Bolter 1990, 1994; Bukatman 1994; Joyce 1996); intertextuality, pastiche and collage (Rettberg 2019); alternative narrative structures, complications of character development and chronology (Ryan 2001); or ergodic design (the reader’s onus to unwind the story by selecting links) (see also Nielsen 1990b; Simanowski 1999; Glazier 2002). A good case in point of the significance of the technological dimension of hypertext are hyperlinks which have been an object of a number of sustained researches and intricate typologies (Bernstein 1998; Vandendorpe 2009 [1999]; Bolter 2001, 14–26 and 40–44; Ryan 2006, 110–111; Landow 2006, 13–29; Barnet 2013). In his (in)famous “convergence thesis” (1991), George P. Landow seems to rather precisely infer intermediality of hypertext fiction. The scholar points out to the profound and transformative relation of “literary theory of the past three of four decades” integrated with electronic technology (1991, 27; Lanham 1993, ch. 4; Ryan 2001, ch. 6). Effectively, hypertext is intermedial in its functional entanglement of the literary (realization of postmodernist and deconstructionist theories) and the digital (see Ensslin 2014a, 260), and it is this very entanglement which ultimately breeds a “differently constituted whole” (Glazier 2002, 78–83): the genre of hypertext fiction. One of the first and most famous examples of hypertext fiction, Michael Joyce’s afternoon, a story (1987), employs the Storyspace platform, which facilitates reading progression of textual units by means of links. The platform’s algorithm determines a “digressive structure” and detouring “without coming to a clear end” (Rettberg 2019). Stuart Moulthrop’s Victory Garden (1992) was created with Eastgate Systems, which offered multiple endings, shifts in perspectives, temporal back-and-forths, and the so on. (Moulthrop is well-known for his extensive employment of various technological tools; he has worked with HyperCard and Storyspace through HTML, Flash, JavaScript and, of course, the Internet.) Perhaps the best example of hypertext novel is The Unknown (1998–2002) by William Gillespie, Frank Marquardt, Scott Rettberg, and Dirk Stratton (Fig. 1). As “a sprawling network-based form” (Rettberg 2019), the novel unabashedly mixes

45

Intermediality and Digital Fiction

1153

Fig. 1 The opening page of The Unknown

genres, styles and forms, includes documentary material, “metafictional bullshit,” correspondence, art projects, and recordings of live performances of reading the sections of the novel. The technological dimension, the links-and-lexias, is made manifest by cross-linking scenes and providing other indices and apparatuses for navigation, indicative of database (Manovich 2001). Also, the links serve multiple functions: “referential, line break/double entendre, point of view shifting, comic subversion, and chronological” (Rettberg 2002). The multimodality and multimediality of The Unknown accounts for its actual status as hypermedia novel. Considered as a type of “second-generation electronic

1154

B. Lutostański

literature” (Hayles 2002, 27), hypermedia, a term also coined by Nelson as a blending of “hypertext multimedia,” used innovative coding conventions (such as HTML, XML, or JavaScript) which worked with a variety of modalities (text, image, and sound) (Nielsen 1990a; Delany and Landow 1994; Joyce 1996, 19–21; Bolter 2001; Calvi 2001; Ensslin 2007). As Ensslin observes, “From an aesthetic point of view, hypermedia readers are confronted not only with interlinked text lexias but a wider semiotic variety, for example, image-text, image-image, and text-image links, as well as dynamic and interactive elements such as film clips and drag-and-drop mechanisms” (2014a, 262; Glazier 2002, 84–88; Hayles 2008, 6–8). Consequently, hypermedia turns out to be inherently and explicitly intermedial in many new and exciting ways: not only a combination of literature and technology but also of audio recordings, films, photographs, newspaper clippings, and many more (see Ensslin 2007, 2014a). A good case in point is one of the first hypermedia novels, Mark America’s Grammatron. To conclude, hypertext’s intermediality stems from the integration of literature and technology. It is the latter medium which profoundly transforms the former one generating its specific aesthetic properties and, eventually, leading to the constitution of ontologically distinct genre within literary studies. The code-driven hypertext technology motivates an original and ephemeral structure of hypertext fiction, along with its hermeneutic and semiotic properties. As a result, hypertext fiction is intermedial in the sense of formally imitating hypertext technology and adapting textual content to its unique architecture and mechanics affecting the narrative in both its story and discourse. As for hypermedia, it acquires its intermedial status on account of combining not only text and code but also diverse multimodal and multimedia resources. As Ensslin puts it, “As opposed to first-generation hypertexts, which use images sparsely and mainly as illustrative or decorative means, hypermedia writings form a coherent intertextual, intermedial, and multimodal whole, which is more than the sum of its constituent parts” (2014a, 262). Consequently, hypermedia fiction does not only formally imitate hypertext technology but also offers means for overt inclusion of multiple media within a given work of fiction. It is, therefore, intermedial in the sense of juxtaposing separate, nonliterary (and nonverbal) media and generating a complex synthesis of media components (i.e., plurimediality).

Interactive Fiction To paraphrase Nick Montfort, interactive fiction (henceforth abbreviated as IF) is a computer program that displays text, accepts textual responses, and then displays additional text in reaction to what has been typed (2003; Montfort 2011; JacksonMead and Wheeler 2011; Plotkin 2011; Montfort 2013). The key feature of this genre of digital fiction is text-based interactivity between a human and a machine: the program “must be able to react to [human] input meaningfully” (Montfort 2003). The input is not about following randomly associated lexias or links, as in hypertext and hypermedia fiction that also exhibit a degree of interactivity (Douglas 2000;

45

Intermediality and Digital Fiction

1155

Bolter 2001), but about a reader’s contributing a text which subsequently needs to be understood by a machine and reacted to in a relevant and appropriate manner (for discussions of interactivity in hypertext, see, for example, Bates and Smith 1989; Aarseth 1997; Murray 1997; Douglas 2000; Bolter 2001; Ryan 2001, 2006; O’Sullivan 2019). IF is usually described as a “text-based adventure game,” and the ludic component appears essential for the ontology of this fiction (see Ensslin 2014b); it is a genre that is “directly derived from some of the earliest games made for personal computers” (Rettberg 2019) and that focuses “heavily on puzzles and exploration” (Short 2014, 289). In fact, as Montfort admits elsewhere (2007, 7), interactive fiction is often called a “game” due to the fact that it uses graphics and even sound or animation, although textual input and output form the basis of the genre: “The player would interact with the game by typing simple commands, such as ‘go east,’ ‘take sword,’ or ‘kill the troll with the sword’” (Rettberg 2019). The integration of literary and ludic media in interactive fiction results in its intermedial status. In the context of literature, there have been discussions of interactive fiction as the detective novel (Mary Ann Buckles), the riddle, and the “potential literature” (Montfort 2011, 33). As for the game context, irrespective of manifold controversies regarding whether “a work with aspects of a game . . . should be thought of in literary terms” (Montfort 2003, n.p.), this digital fiction entails a technological component (software, program, and code) in order to constitute a textual storyworld dominated with adventures, mystery, and fantasy (Douglas 2000). Terms such as “interactor,” “parser,” “session,” “player character,” and “interaction” all indicate the gaming context, so do theoretical and methodological discussions ranging from literary studies to game studies to IT studies, among others (Montfort 2007, 277–280). In analogy to hypertext and hypermedia fiction, IF relies on a specific technological dimension in constituting its structural, aesthetic, and ontological properties. For the previously described genres of digital literature, the technological affordances originated in software (Eastgate and Storyspace, for instance), but for IF, they stem from games mechanics and architecture. One of the most famous and celebrated example of IF is Photopia by Adam Cadre (1998). In this short work, interactor solves puzzles in order to complete the “literary game” and engages with other characters. As Rettberg explains, “Photopia is comparatively complex from the standpoint of narrative style. While the action is focused on one central character, Alley, the author employs shifts in point-of-view and in time frame between a ‘real world’ and a series of imaginative story environments. These small dream worlds are each named after a color (red is a Mars-like extraplanetary environment, while yellow is a golden beach). Shifts in the color of the texts and the background mark ontological shifts in the story: the real/present world has black text on a white background while the other texts are colored. These environments are poetic in the sense that each in some way serves as a metaphor for a character’s relationship with Alley, who will die in a car crash during the course of the story” (2019). Jordan Magnuson claims that Photopia is essential for the interactive fiction genre due to its “experiments with nonlinear presentation of

1156

B. Lutostański

time, menu-based conversation, and constrained game-play to support a specific plot” (quoted in Rettberg, 2019, 219). Another relevant example of the intermedial status of IF in its employment of literary and ludic components is Emily Short’s well-known work Galatea (2000). As the interactors, we visit an art gallery and converse with a marble statue by the famous Pygmalion of Cyprus. The work does not offer any conclusion or puzzle solution at the end, while the mere conversation with the statue might follow multiple trajectories. When replaying and taking alternative runs, we learn more about Galatea (some information transpire as self-exclusive): in one version, Galatea admits her unrequited love for Pygmalion, and in another, she strangles him for having been offended. Along the way and through replay, we learn about Galatea herself and her relationship with her creator. For this reason, the key point of the work (“winning” is this literary game) is to unpack and unwind the storyworld; as Rettberg explains, “the play is much more about finding the right tone and approach to entice Galatea to reveal story to you” (2019). All in all, IF transgresses the boundaries between distinct media. Specifically, it is plurimedial (potential juxtaposition of separate media) and formally imitative. IF imitates game mechanics and architecture by implementing interactive tools in storyworld construction. Finally, in contrast to hypertext and hypermedia fiction, IF is intermedial in one more sense. Games are considered autonomous media because of their vital cultural dimension. They are publicly recognized as forms of communication and expression, as art forms, as institutions, or even as consumer products. They implicate specific social practices, behaviors, and expectations. Accordingly, synthesizing two culturally distinct media, literature and games, results in the intermediality of IF on a cultural level, too (“cultural intermediality”).

Cybertext For Ensslin, digital cybertext constitutes a subtype or “third-generation” hypertext (2007, 6). It is produced by “algorithmic text production processes, which automatically generate uncountable variants of a previously entered textual body. In cybertext, textons generate scriptons perceived and responded to by the reader, a process which, in its entirety, could be seen as a neo- or cyber-deistic concept, with the programmer ‘creating’ an organic system and, once it is finished, leaving it to its own mechanistic devices” (Ensslin 2007, 23; on a more extensive cybertext theory, see Eskelinen 2012). In other words, the reader is “coded” into a text world (Dovey and Kennedy 2006, 108). Cybertexts are “programmed hypertexts” in such a way as to become autonomous “text machines” (Aarseth 1997, 21), which assume power over the reader by literally “writing themselves” rather than presenting themselves as an existing textual product (Ensslin 2014a, 262; see Eskelinen 2012, 30–46). The reader is relegated to the role of a “spectator at a soccer game,” reduced to speculation, conjecture, extrapolation, “even shout[ing] abuse” (Aarseth 1997, 4). Effectively, cybertext undermines both the author- and reader-driven fiction, as in classical and hypertextual or

45

Intermediality and Digital Fiction

1157

interactive species; “the operator [cybertext reader – BL] [is] a constitutive yet somewhat disempowered element of (cyber)textual performance. Put differently, readers become part of a cybernetic feedback loop, which operates on the basis of mutual stimulus and response between machine and operator” (Ensslin 2014a, 262). As Marie-Laure Ryan observes, cybertexts “highlight the dynamic production of text, turning this production into a spectacle. Experiencing the text means watching words and meaning emerge and evolve on the screen, animated by the invisible code of a computer program” (Ryan 1999, 9). An early example of cybertext is Stuart Moulthrop’s Hegirascope (first version in 1995 and a revised one in 1997). A parody of hypertext (Aarseth 1997, 81), it features over 700 tightly timed links, which cause 175 lexias to change at a rapid pace (every 18 s in the 1995 version) without giving the reader a chance to take control via mouse click or to even read them completely (Fig. 2). And although “Hegirascope’s text nodes contains normal links, which give the reader some slight sense of control, . . . he is left with the feeling of rowing against the current in a mighty river” (Aarseth 1997, 80). (Interestingly, probably due to reception challenges, Moulthrop reduced the power of the machine considerably in the 1997 version. So much so that if one manages to read the lexias relatively quickly, the impression of being faced with a cybertextual machine may not even occur. “The intention behind this move was to give more weight to content and language, which was, in Moulthrop’s case, a provident thing to do, as both his style and treatment of subject matter are worthy of close reading” (Ensslin 2007, 106)).

Fig. 2 An example of a lexia and four links of Hegiroscope. © Stuart A. Moulthrop

1158

B. Lutostański

A unique example of cybertext is Kate Pullinger’s Breathing Wall (2004). It consists of two key components: a hypertext-based, largely sequential narrative, organized into five parts, or “daydreams,” and a set of “nightdreams,” which reveal the protagonist Michael’s nightly conversations with his deceased girlfriend Lana. In the former, sequential text-based lexias are presented by a machine and the readers cannot opt for speed of reading; they can only move from subchapter to subchapter within a daydream; however, they lose control once the text film has started. In the nightdreams, the pace of information transfer is entirely dependent on the reader’s breathing rate. It is registered by means of a microphone attached to a headset and placed directly underneath the reader’s nostrils, and processed by a special software (Hyper Trans Fiction Matrix). As Ensslin explains, “Depending on the rate and depth of inhaling and exhaling, the text (in the combined sense of program code and user interface) will release either more or less information essential for solving the riddle imposed by the story” (2009, 160). Next to the two major components, The Breathing Wall features auditory material (distinct sounds for dreams and daydreams; voices) and visual resources (background images and colors; animation). Still, the verbal substance remains the decisive medium of information retrieval. Nonetheless, it is the breathing process included in cybertextual mechanics of the work that appears most exciting and leads to a variety of conceptual endeavors (Ensslin’s “physio-cybertext” (2007, 111; 2009) and one reviewer’s “hyperventiliterature” (in Ensslin (2007, 112)). To conclude, cybertext acquires its intermedial status due to its integration of the literary and the technological. This integration is of a significantly different nature from hypertext or hypermedia. Now the technological determines the “cybernetic feedback loop” by emphasizing the key role of a tailored code and its effects on interface, structure, and ontology of this genre (Ensslin 2014a, 262–263). Cybertext fiction defamiliarizes the reading experience of digital text by supplanting the author and the reader (or interactor or player) with the emphasis on the technological dimension of text production and consumption. Therefore, cybertext is intermedial in the sense of radical representation (or formal imitation, to use Wolf’s term) of an algorithm-driven structure of communication by fictional text.

Network Fiction In contradistinction to, for instance, Ciccoricco’s (2007) use of the term, network fiction is today defined as a genre of digital fiction which references a number of Internet-related writing and reading practices. It entails a rich, complex, and expanding body of texts, some of which can be subsumed under the common types of social media fiction (Rustad 2012; Vlieghe et al. 2016; Thomas 2020), Twitter fiction (Anciman and Rensin 2009; Thomas 2013), email novels (Walker Rettberg 2014), and collective/social narratives (Rettberg 2004; Klaiber 2013; Pianzola 2021). Network fiction, therefore, is an umbrella term for any fiction written on and for specific Internet websites, usually online platforms which are set up mainly for literary purposes (blogs, collaborative writing platforms, such as Wattpad) or not (social media platforms). It accounts for the fact that, firstly, a

45

Intermediality and Digital Fiction

1159

specific platform generates a “significant part of the aesthetic expression and the meaning potential” (Bell et al. 2013, 10); secondly, it can “use the Internet’s potential for collaboration, or use the network as a site for performance” (Rettberg 2019). Effectively, network fiction fuses two distinct although interrelated semiotic domains: “network” standing for the use of one or more Internet websites or platforms, and “network” implying collaboration, community-building, and co-construction by multiple users-authors. In his Electronic Literature, Scott Rettberg (2019) enumerates major network fiction subgenres: codework, flarf, home page fiction, email novels, fictional blogs, Twitter fiction, online writing communities, collective narratives, and Netprov. Due to space limitations of this publication, I cannot explore all of these subgenres in detail; instead, I would like to focus on two of them: Twitter fiction (as an example of a broader phenomenon of social media fiction) and collective narratives. My presentation of these two subgenres will mostly regard their status as intermedial digital fiction. Twitter fiction’s heyday happened roughly 10 years ago. In 2012, The Guardian commissioned a number of “well-known writers – from Ian Rankin and Helen Fielding to Jeffrey Archer and Jilly Cooper – to come up with a story of up to 140 characters.” Two years later, The Guardian’s culture editor, Claire Armitstead, was asking in a column “Has Twitter given birth to a new literary genre?” (2014). And one year later occurred the third edition of #TwitterFiction Festival, including contributions from, among others, Margaret Atwood and Lemony Snicket. However, that third edition was the last one, and in the ensuing years, Twitter seems to have lost its mysteriously innovative and charmingly challenging appeal for fiction writers. In the meantime, some esteemed narratives have been created on and with Twitter, for example, Jennifer Egan’s “Black Box” (2012) and Teju Cole’s “Hafiz” (2014). In a 2013 article, “140 Characters2 in Search of a Story,” Bronwen Thomas claims that Twitter fiction “challenge[s] basic concepts of order, chronology, and causality in a number of ways” (96). It segments content and unfolds in “real time” in the form of concurrent narration (in the vein of “processurality” (Ensslin 2007, 27)); defers the narrative closure; is susceptible to revisions and negotiations of meanings by readers; and, more importantly, “destabilizes the relationship between author and reader” (Thomas 2013, 96). These formal and ontological properties of Twitter fiction result from the platform’s affordances, including character limitations, interface and interactive, and multimodal architecture. They motivate the fact that Twitter fiction recasts “the reader/viewer/user not as passive consumer of already-formed objects but as a participant in an ongoing experience in which his or her contribution is key. We can see this at work with Twitterfiction, both in the interactions that take place between author and readers and also in the activities of retweeting, sharing, and linking” (Thomas 2013, 105). In addition, it recalibrates reception in such a way as to make the serialized content turn into “a part of the daily ritual of users, where it is not so much the quality or even the quantity of output that matters most, so much as

2

Since 2017, users can use up to 280 characters per tweet.

1160

B. Lutostański

the reassurance that there will be constant updates, and that followers will be kept ‘in the loop’ about any new developments” (Thomas 2013, 105). Ultimately, fictional narratives on Twitter display a substantial degree of intermediality: they are conceived and received at the nexus of the literary (fiction) and the technological (the social media platform). For Thomas, the former is at first glance a “rather conventional” set of narrative (and narratological) features (2013, 96) with antecedents in the form of flash fiction and microfiction, but disrupted or defamiliarized by the latter and its microblogging ecology of tweeting and retweeting, using and following, on the go and in participation with others. Twitter fiction encompasses (at least) two distinct media. Specifically, it exhibits plurimediality (a literary text enmeshed within the multimedial and multimodal Twitter interface), formal imitation (text adapted to a website’s or platform’s ecology), and thematization (Twitter use potentially referenced in text’s creation and dissemination). The imitation, as in the case of previous digital fiction genres, is of paramount importance: the process of adapting a fictional text to the platform’s interface, architecture, and mechanics heavily and profoundly affects the ontology of a narrative the readers read from the screens. In addition, Twitter fiction is entangled within a complex mesh of Twitter’s technological, social, and cultural features informed by ubiquitous use of the Internet, prevalence of mobile technology, and online community practices. Accordingly, Twitter fiction (and the remaining two genres, too) partakes in Jenkins’s participatory culture, O’Reilly’s Web 2.0 and van Dijck’s culture of connectivity, among many other present-day Internet-related phenomena. In this way, Twitter fiction, similarly to IF, is a case of cultural intermediality beside technological intermediality (as hypertext, hypermedia, and cybertext fictions are). The second type of network fiction, collective narrative, fully partakes in the extensive and expansive “Internet culture,” too, by the dint of tapping into and making use of the Internet’s ecology, especially its fostering online community of like-minded individuals. In the case of collective narrative, sociality comes into the spotlight; participants build upon the contributions of previous authors in order to further develop a sequential narrative. This subgenre includes works such as A Million Penguins (a 2007 web-based collaboration of Penguin Books and De Montfort University), netprov works such as Grace, Wit & Charm (Wittig 2011), or Kate Pullinger and Chris Joseph’s Flight Paths: A Networked Novel (2007–2012), in which “Readers were invited to discuss the development of the project with Pullinger and Campbell at each stage of the project’s release, to contribute personal stories and images that could be woven into the story, remaining in contact with the authors throughout the process” (Rettberg 2019). These works hinge on cooperation and collaboration of multiple users who effectively become (co-)authors (or “wreaders,” as Klaiber, borrowing from Landow (1994), puts it (2013)). Herein lies the main ontological difference between collective narrative and Twitter fiction: the former employs the Internet’s architecture to invite and incite others to create and critique fiction communally. Recently, the collective writing has been made more accessible and available due to the commonality of websites geared to one goal: enhancing writing in groups. One of the most popular collective fiction writing platform is Wattpad (other well-known

45

Intermediality and Digital Fiction

1161

platforms, some of them inactive today, have included Ficly and Protagonize). The writing process occurs “publicly” so that other Wattpad users read and comment upon text, providing support and encouragement. The most popular Wattpad writers boast millions of followers (that is, readers and contributors). The platform exemplifies a contemporary Internet phenomenon called “digital social reading” (DSR), which encompasses “a wide variety of practices related to the activity of reading and using digital technologies and platforms (websites, social media, mobile apps) to share with other people thoughts and impressions about texts” (Pianzola 2021, 2). Collective narrative’s intermedial status originates in its integration of literature and a platform’s or website’s technological ecology. Hence, this subgenre of digital fiction operates by formally imitating and harnessing the Internet’s affordances in collaborative literary expression.

Small-Screen Fiction For several years now, a number of scholars and critics have attempted to explore what seems to be latest genre of digital fiction. “Touchscreen fiction” (Bouchardon 2013), “app-driven” and “app-based fiction” have been proposed, while Ensslin et al. dedicated a special issue of Paradoxa (2017) to the storytelling practices “on the smaller screens of laptops, tablets, and even mobile phones” (8). The so-called “small-screen fiction” conceptualizes a type of digital fiction whose narrative and storyworld construction heavily rely on the affordances of mobile devices. However, an extensive scope of the collection leads to multiple controversies with regards to the object of study. It remains unclear whether the small-screen fiction utilizes standalone apps or intermediary apps (such as web browsers, for instance, Google and its “Editions at Play”). Another shortcoming is a lack of clear distinction as to the exact technology in use since, arguably, a tablet and a smartphone generate dissimilar formal, aesthetic, and cultural implications. The term “small-screen fiction” also seems to include e-readers, which offer a different technological environment for literature and, more importantly, beg different ontological questions. In a recent project, Magdalena Rembowska-Płuciennik and myself (2020) have tried to demonstrate a more sustained and nuanced exploration into fiction delivered via applications for mobile devices – smartphones and tablets – and christened this type of digital fiction “app fiction” (see Lutostański 2023). Selected examples include: The Pickle Index by Eli Horowitz (2015), Arcadia by Iain Pears (2015), and Belgravia by Julian Fellowes (2016). Mobile phone apps, offering a host of medial and modal possibilities, are used in various ways in these three works. Horowitz’s novel, next to being a literary work of fiction, exploits ludic components (it is a role-playing game, quantifying our performance and enabling players to collaborate within the storyworld), social media components (as storyworld participants, we are asked to post profusely, share others’ posts, and read news in the app), and mobile media properties (embodied and situated consumption) (Fig. 3). Pears writes a hypertext novel in which textual nodes offer a variety of subsequent nodes to choose from a menu at the bottom of the screen. The whole multilinear plot is visualized by means of a colorful

1162

B. Lutostański

Fig. 3 Home page of The Pickle Index. © Sudden Oak

arborescent diagram (upside-down, to be exact; that is, the initial node is located at the top of the screen, and subsequent lexias flow down in various directions). By doing so, Pears evokes hypertext fiction integrated with new mobile technology. Fellowes’s work provides (infrequent) links in (very long) stretches of text (nodes), echoing the hypertext architecture. However, it does not much disrupt the sustained reading experience and opts for all the non- and parafictional details to be included in separate sections; these contain photographs of, say, characters’ clothes on a ball, map of Belgravia, paintings of historical characters, and even recordings of music performed during social events.

45

Intermediality and Digital Fiction

1163

App fiction relates to app as a distinct medium in itself and embedded with an intricate medial relations of mobility, smartphones, algorithms, and postdigital phenomena (Lutostański 2023). At any rate, intermediality plays a vital role for this fiction’s ontology and reception. As can be seen above, it not only combines two or more media in one artifact but in fact displays a project-specific approach to freely employ, thematize, evoke, and imitate other media. The Pickle Index, the most complex example of the three mentioned above, is a very good case in point.

Conclusions The aim of this chapter was to approach the major genres of digital fiction from the perspective of intermediality. Drawing on the theoretical discussions of Irina Rajewski, Marie-Laure Ryan, Wolfgang Wolf, and others, I have conceptualized hypertext, hypermedia, IF, cybertext, network, and small-screen fictions as transgressing boundaries between literature and technology. These genres hinge and thrive on existing in-between media. For hypertext, hypermedia, and cybertext, the intermediality functions essentially in a technological manner; digital technology provides means of production and consumption of these types of fiction. The literary medium formally imitates the electronic ecology by being adapted to its code-driven architecture (e.g., hypertext technology segmenting the textual content into lexias connected with links). Hypermedia, in addition to technological medium, employs and juxtaposes separate media, such as music, film, or photography. IF is codedriven, too; however, it is simultaneously imbricated in the context of gaming. It employs specific game genres and mechanics in creating storyworld and, in particular, its interactive qualities, unlike any other digital fiction genre. In addition, IF partakes in a game medium’s cultural dimension, which includes specific social practices and behaviors as well as institutional organizations that support and foster them. Network fiction and small-screen fiction build on the technologically dominated intermedial status of prior genres; they, too, employ the electronic medium’s affordances in literary expression and communication. In addition, they partake in a variety of the “Internet cultures,” such as participation, Web 2.0, and social network sites. Network fiction owes its name to the very fact of coming to existence on (literature writing) online networks (e.g., WattPad or Tumblr) as well as to the social networks of “wreaders,” communities of (non)professional writers, fans, and literature aficionados, ready to read, comment, critique, or support you. On the other hand, small-screen fiction, and app fiction in particular, seems the most advanced genre of digital fiction due to the fact that it offers the most flexible and hybridic ontology of all digital fiction genres. It can borrow from game conventions, social media netiquettes, or hypertextual topography in storyworld construction. To conclude, whereas hypertext, hypermedia, and cybertext are intermedial primarily due to their plurimediality and formal imitation (let me conceptualize it with “technological intermediality” for lack of a better term), IF, network fiction, and small-screen fiction additionally entail a cultural dimension by tapping into a

1164

B. Lutostański

complex aggregate of practices, behaviors, and institutions imbricated in games, mobility, and the Internet (conceptualized as distinct media for reasons other than purely technological). Admittedly, my discussions might seem deterministic and simplifying as I repeatedly referred to technology as a determining and defining criterion for differentiating and identifying major digital fiction genres (cf. Rettberg 2019; O’Sullivan 2019, 35; Hayles 2008, 159–165). However, my point was solely to highlight how evolving electronic technology (software and hardware alike) has been influential for and harnessed in creating new fiction (Hayles 2008, 1–3). What comes as a surprise, there is no book-length and sustained study exploring complex intermedial relations within digital fiction in their technical, semiotic, and cultural dimensions.

References Aarseth, Espen J. 1997. Cybertext. Perspectives on ergodic literature. Baltimore/London: The Johns Hopkins University Press. Anciman, Alexander, and Emmet Rensin. 2009. Twitterature: World’s greatest books retold through Twitter. London: Penguin. Armitstead, Claire. 2014. Has Twitter given birth to a new literary genre? The Guardian, January 10. https://www.theguardian.com/books/booksblog/2014/jan/10/twitter-birth-new-literarygenre. Accessed 15 June 2022. Atã, Pedro, and Beate Schirrmacher. 2021. Media and modalities – Literature. In Intermedial studies. An introduction to meaning across media, ed. Jørgen Bruhn and Beate Schirrmacher, 42–55. London/New York: Routledge. Baetens, Jan, and Domingo Sánchez-Mesa Martínez. 2015. Literature in the expanded field: Intermediality at the crossroads of literary theory and comparative literature. Synaesthesia 36: 289–304. https://doi.org/10.4000/interfaces.245. Barnet, Belinda. 2013. Memory machines. The evolution of hypertext. London/New York: Anthem Press. Bates, Joseph, and Sean Smith. 1989. Towards a theory of narrative for interactive fiction. Carnegie Mellon: University Pittsburgh, School of Computer Science. Bell, Alice. 2010. The possible worlds of hypertext fiction. New York: Palgrave Macmillan. Bell, Alice, Astrid Ensslin, and Hans Kristian Rustad, eds. 2013. Analyzing digital fiction. New York/London: Routledge. Bernstein, Mark. 1998. Patterns of hypertext. Watertown: Eastgate Systems. http://www.eastgate. com/patterns/Print.html. Accessed 30 Apr 2022. Bewell, Alan, Christina Lupton, John Klancher, and Ted Underwood. 2011. Review of ‘This is enlightenment’. Studies in Romanticism 50: 531–543. Bolter, Jay David. 1990. Writing space: The computer, hypertext, and the history of writing. Mahwah: Lawrence Erlbaum Associates. ———. 1994. Topographic writing: Hypertext and the electronic writing space. In Hypermedia and literary studies, ed. Paul Delany and George P. Landow, 105–118. Cambridge, MA/London: MIT Press. ———. 2001. Writing space: The computer, hypertext and the history of writing. 2nd ed. Hillsdale: Lawrence Erlbaum. Bouchardon, Serge. 2013. Figures of gestural manipulation. In Analyzing digital fiction, ed. Alice Bell, Astrid Ensslin, and Hans Kristian Rustad, 159–175. New York/London: Routledge. Bukatman, Scott. 1994. Virtual textuality. The Free Library. https://www.thefreelibrary.com/Virtual +textuality.-a015143604. Accessed 06 May 2022. Cadre, Adam. 1998. Photopia. https://adamcadre.ac/if/photopia.html. Accessed 04 July 2022.

45

Intermediality and Digital Fiction

1165

Calvi, Licia. 2001. Hypertext and comics: Towards an aesthetics of hypertext. In Proceedings of the twelfth ACM conference on hypertext and hypermedia, ed. K. Grenbaek, 135–137. Aarhus: ACM Press. Charney, David. 1994. The impact of hypertext on processes of reading and writing. In Literacy and computers, ed. Susan J. Hilligoss and Cynthia L. Selfe, 238–263. New York: Modern Language Association. Ciccoricco, David. 2007. Reading network fiction. Tuscaloosa: The University of Alabama Press. Coover, Robert. 1992. The end of books. The New York Times Book Review, June 21. http://www. nytimes.com/books/98/09/27/specials/coover-end.html. Accessed 26 May 2022. Delany, Paul, and George P. Landow, eds. 1994. Hypermedia and literary studies. Cambridge, MA/London: MIT Press. Douglas, J. Yellowlees. 1994. ‘How do I stop this thing?’: Closure and indeterminacy in interactive narratives. In Hyper/text/theory, ed. George P. Landow, 159–188. Baltimore: Johns Hopkins University Press. ———. 2000. The end of books – Or books without end? Reading interactive narratives. Ann Arbour: University of Michigan Press. Dovey, Jon, and Helen Kennedy. 2006. Game cultures: Computer games as new media. Maidenhead: Open University Press. Elleström, Lars. 2014. Media transformation: The transfer of media characteristics among media. New York: Palgrave Macmillan. Ensslin, Astrid. 2007. Canonizing hypertext. Explorations and constructions. New York: Continuum. ———. 2009. Respiratory narrative: Multimodality and cybernetic corporeality in ‘Physiocybertext’. In New perspectives on narrative and multimodality, ed. Ruth Page, 155–165. New York/London: Routledge. ———. 2014a. Hypertextuality. In The Johns Hopkins guide to digital media, ed. Marie-Laure Ryan, Lori Emerson, and Benjamin Robertson, 258–265. Baltimore: The Johns Hopkins University Press. ———. 2014b. Literary gaming. Cambridge, MA/London: The MIT Press. Ensslin, Astrid, and Lyle Skains. 2017. Hypertext: Storyspace to twine. In The Bloomsbury handbook of electronic literature, ed. Joseph Tabbi, 295–310. London et al: Bloomsbury. Ensslin, Astrid, Lisa Swanstrom, and Paweł Frelik. 2017. Introducing small screen fiction. Paradoxa 29: 7–18. https://paradoxa.com/wp-content/uploads/2020/06/1-Introduction-EnsslinSwanstrom-Frelic-pp7-18.pdf. Accessed 04 Mar 2022. Eskelinen, Markku. 2012. Cybertext poetics. London/New York: Continuum. Gibbons, Alison. 2011. Multimodality, cognition, and experimental literature. New York/London: Routledge. Glazier, Loss Pequeño. 2002. Digital poetics. Hypertext, visual-kinetic text and writing in programmable media. Tuscaloosa: University of Alabama Press. Goody, Alex. 2011. Technology, literature and culture. Cambridge/Malden: Polity Press. Grishakova, Marina, and Marie-Laure Ryan, eds. 2010. Intermediality and storytelling. Berlin/New York: De Gruyter. Hallet, Wolfgang. 2014. The rise of the multimodal novel: Generic change and its narratological implications. In Storyworlds across media. Towards a media-conscious narratology, ed. MarieLaure Ryan and Jan-Noël Thon, 151–172. Lincoln/London: University of Nebraska Press. ———. 2015. A methodology of intermediality in literary studies. In Handbook of intermediality, ed. Gabrielle Rippl, 605–618. Berlin/Boston: de Gruyter. Hayles, Katherine N. 2002. Writing machines. Cambridge/London: The MIT Press. ———. 2008. Electronic literature. New horizons for the literary. Notre Dame Indiana: University of Notre Dame Press. Jackson-Mead, Kevin, and J. Robinson Wheeler, eds. 2011. IF theory reader. Boston: Transcript On Press. Joyce, Michael. 1996. Of two minds. Hypertext pedagogy and poetics. Ann Arbor: The University of Michigan Press.

1166

B. Lutostański

Klaiber, Isabell. 2013. Wreading together. The double plot of collaborative digital fiction. In Analyzing digital fiction, ed. Alice Bell, Astrid Ensslin, and Hans Kristian Rustad, 124–140. New York/London: Routledge. Kress, Gunther, and Theo van Leeuven. 2001. Multimodal discourse: The modes and media of contemporary communication. London: Arnold. Landow, George P. 1991. Hypertext. The convergence of contemporary critical theory and technology. Baltimore: The Johns Hopkins University Press. ———., ed. 1994. Hyper/text/theory. Baltimore: Johns Hopkins University Press. ———. 2006. Hypertext 3.0. Critical theory and new media in an era of globalization. Baltimore: Johns Hopkins University Press. Lanham, Richard. 1993. The electronic word: Democracy, technology, and the arts. Chicago: Chicago University Press. Lutostański, Bartosz. 2023. A poetics of app fiction. Berlin/Boston: De Gruter. [in review]. Lutostański, Bartosz, and Magdalena Rembowska-Płuciennik. 2020. Powieść aplikacja – między wolnym czytaniem a szybkim Internetem. Teksty Drugie 5: 53–72. http://journals.openedition. org/td/15838 Manovich, Lev. 2001. The language of new media. Cambridge, MA/London: The MIT Press. McDowell, Paula. 2017. Media and mediation in the Eighteenth Century. Oxford Handbooks Online, Mar 2017. https://doi.org/10.1093/oxfordhb/9780199935338.013.46. Montfort, Nick. 2003. Twisty little passages. An approach to interactive fiction. Cambridge, MA/London: The MIT Press. ———. 2007. Generating narrative variation in interactive fiction. https://citeseerx.ist.psu.edu/ document?repid=rep1&type=pdf&doi=9697c926377de131c9de3d478c350a795d13d0a8. Accessed 28 July 2022. ———. 2011. Toward a theory of interactive fiction. In IF theory reader, ed. Kevin Jackson-Mead and J. Robinson Wheeler, 25–58. Boston: Transcript On Press. ———. 2013. Riddle machines: The history and nature of interactive fiction. In A companion to digital literary studies, ed. Ray Siemens and Susan Schreibman, 267–282. Chichester: Blackwell Publications. Murray, Janet. 1997. Hamlet on the Holodeck: The future of narrative in the cyberspace. New York: Free Press. Nielsen, Jakob. 1990a. Hypertext and hypermedia. San Diego: Academic. ———. 1990b. The art of navigating through hypertext. Communications of the ACM 33(3). https://dl.acm.org/doi/pdf/10.1145/77481.77483. Accessed 27 May 2022. O’Sullivan, James. 2019. Towards a digital poetics. Electronic literature and literary games. Cham: Palgrave Macmillan. Pianzola, Federico. 2021. Digital social reading: Sharing fiction in the 21st century. https://wip. mitpress.mit.edu/digital-social-reading. Accessed 01 June 2022. Plotkin, Andrew. 2011. Characterizing, if not defining, interactive fiction. In IF theory reader, ed. Kevin Jackson-Mead and J. Robinson Wheeler, 59–66. Boston: Transcript On Press. Rajewsky, Irina O. 2005. Intermediality, intertextuality, and remediation: A literary perspective on intermedialit. Remédier/Remediation 6. https://doi.org/10.7202/1005505ar. Rajewsky, Irina O. 2010. Border talks: The problematic status of media borders in the current debate about intermediality. In Media borders, multimodality and intermediality, ed. Lars Elleström, 51–68. New York: Palgrave Macmillan. Rettberg, Jill Walker. 2004. Distributed narrative: Telling stories across networks. https://jilltxt.net/ txt/Walker-AoIR-3500words.pdf. Accessed 13 July 2022. ———. 2014. E-mail novel. In The Johns Hopkins guide to digital media, ed. Marie-Laure Ryan, Lori Emerson, and Benjamin Robertson, 178–179. Baltimore: Johns Hopkins University Press. Rettberg, Scott. 2002. The pleasure (and pain) of link poetics. Electronic book review. http://www. electronicbookreview.com/thread/electropoetics/pragmatic. Accessed 16 Apr 2022. ———. 2011. All together now: Hypertext, collective narratives, and online collective knowledge community. In New narratives. Stories and storytelling in the digital age, ed. Ruth Page and Bronwen Thomas, 187–204. Lincoln/London: The University of Nebraska Press. ———. 2019. Electronic literature. Cambridge/Medford: Polity. Ebook edition.

45

Intermediality and Digital Fiction

1167

Rippl, Gabriele (ed.). 2015. Handbook of intermediality. Berlin/Boston: De Gruyter, ebook edition. Rustad, Hans Kristian. 2012. Digital litteratur. Oslo: Cappelen Damm Akademisk. Ryan, Marie-Laure. 1999. Cyberspace textuality: Computer technology and literary theory. Bloomington: Indiana University Press. ———. 2001. Narrative as virtual reality. Immersion and interactivity in literature and electronic media. Baltimore/London: The Johns Hopkins University Press. ———. 2006. Avatars of story. Minneapolis/London: University of Minnesota Press. ———. 2014. Story/worlds/media: Tuning the instruments of a media-conscious narratology. In Storyworlds across media. Towards a media-conscious narratology, ed. Marie-Laure Ryan and Jan-Noël Thon, 25–49. Lincoln/London: University of Nebraska Press. Short, Emily. 2000. Galatea. In Electronic literature collection. Volume one. https://web.archive. org/web/20090127161422/. http://collection.eliterature.org/1/works/short__galatea.html. Accessed 03 June 2022. ———. 2014. Interactive fiction. In The Johns Hopkins guide to digital media, ed. Marie-Laure Ryan, Lori Emerson, and Benjamin Robertson, 289–292. Baltimore: Johns Hopkins University Press. Simanowski, Roberto. 1999. Hypertext, cybertext, digital literature, medium: An interview with Espen Aarseth. Dichtung Digital 7: 1–5. https://doi.org/10.25969/mediarep/17331. ———. 2004. Death of the author? Death of the reader! In p0es1s: Ästhetik digitaler Poesie – The aesthetics of digital poetry, ed. Friedrich W. Block, Christiane Heibach, and Karin Wenz, 17–92. Ostfildern-Ruit: Hatje Cantz. The Guardian. 2012. Twitter Fiction: 21 authors try their hand at 140-character novels. The Guardian, October 12. https://www.theguardian.com/books/2012/oct/12/twitter-fiction-140character-novels. Accessed 15 June 2022. Thomas, Bronwen. 2013. 140 characters in search of a story. Twitterfiction as an emerging narrative form. In Analyzing digital fiction, ed. Alice Bell, Astrid Ensslin, and Hans Kristian Rustad, 94–108. New York/London: Routledge. ———. 2020. Literature and social media. London: Routledge. Van Looy, Jan, and Jan Baetens, eds. 2003. Close reading new media: Analyzing electronic literature. Leuven: Leuven University Press. Vandendorpe, Christian. 2009. From papyrus to hypertext. Toward the universal digital library. Trans. P. Aronoff and H. Scott. Chicago/Springfield: University of Illinois Press. Vlieghe, Joachim, Muls Jaël, and Kris Rutten. 2016. Everybody reads: Reader engagement with literature in social media environments. Poetics 54: 25–37. https://doi.org/10.1016/j.poetic. 2015.09.001. Walker Rettberg, Jill. 2014. E-mail novel. In The Johns Hopkins guide to digital media, ed. MarieLaure Ryan, Lori Emerson, and Benjamin Robertson, 178–180. Baltimore: Johns Hopkins University Press. Wolf, Werner. 2005. Intermediality. In The Routledge encyclopedia of narrative theory, ed. David Herman, Manfred Jahn, and Marie-Laure Ryan, 252–256. London/New York: Routledge. ———. 2011. (Inter)mediality and the study of literature. CLCWeb: Comparative Literature and Culture 13(3). http://docs.lib.purdue.edu/clcweb/vol13/iss3/2

Intermediality and Metamediality: From Analog Representations to Digital Resources

46

Klaus Bruhn Jensen

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metamedia: The Very Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Three Decades of Digital Media Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Beyond Cyberspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Communications Across Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Speaking into the System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The World as a Medium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metacommunication in and by Metamedia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metacommunication in the Flesh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Models of Mediation from Bateson to Barthes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Metacommunication as Process and Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Politics of Metamedia and Metacommunication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Big Science for Big Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Habeas Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1170 1172 1175 1175 1175 1178 1179 1180 1180 1181 1183 1185 1185 1186 1189 1190

Abstract

The digital computer constitutes a metamedium, hosting all previous media of human communication – speech, writing, printing, cinema, radio, television – while simultaneously supporting new media forms and communicative practices, from the World Wide Web and smartphone apps to the home appliances and traffic systems of an emerging Internet of Things. Digitalization is reemphasizing the point that, more than representations of reality, media afford resources for acting in and on reality as well. This chapter takes stock of research on the ongoing transition to digital infrastructures of human communication and culture, and its implications for the understanding and study of intermediality. The K. Bruhn Jensen (*) Department of Communication, University of Copenhagen, Copenhagen, Denmark e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_57

1169

1170

K. Bruhn Jensen

chapter, first, situates metamedia within the long history of human communication, characterizing media as technologies, institutions, as well as discourses. The second section reviews digital media and communication studies since the 1990s, with special reference to the diverse flows of one-to-one, one-to-many, and manyto-many communication on digital platforms. Third, the chapter elaborates on the relationship between metamedia and metacommunication: the many implicit meanings that both anticipate and accompany the transmission or sharing of information. Whereas all human communication is framed and facilitated by metacommunication, the concept has acquired a distinctive and strategic importance in the context of digital media, as their users communicate many-to-one, into the system, leaving behind bit trails with unforeseen, and unforeseeable, applications and abuses. The final section, accordingly, addresses some of the ethical and political implications of metamedia, considering a principle of habeas data to complement the classic principle of habeas corpus. Keywords

Metamedia · Metacommunication · Metalanguage · Internet of things · Habeas data

Introduction Throughout recorded – mediated – history, the introduction of new media has entailed scholarly and public debate on the distinctive features, interrelations, and relative qualities of different media. Plato, in the Phaedrus, had worried that writing might lead to forgetfulness among readers. Print media, for centuries, have been censored on the assumption that at least some writings, if mass-reproduced, would lower public morality and fuel popular unrest. Broadcasting, throughout the twentieth century, came to be associated with escape and entertainment diverting listeners and viewers from the education and enlightenment on offer through the printed word. Most recently, digital computing has brought together all previous media on shared platforms of hardware and software, prompting utopias and dystopias of digital media as a force for either liberation and empowerment, or surveillance and discrimination. This chapter takes stock of digital media as a new category of media – metamedia – placing metamediality in the longer genealogy of intermediality, and addressing some of the ethical and political implications that follow from an ongoing society-wide and worldwide process of digitalization. The first section characterizes metamedia and their recombination and reconfiguration of the many and diverse forms of representation and interaction enabled by analog media, including human beings as media in their own right, who are being complemented and challenged by artificial intelligence. The terminology of metamedia goes back to the early days of desktop and laptop devices, when computers could be seen to integrate previously separate modalities of experience and expression, subsequently facilitating networked collaboration and coordination.

46

Intermediality and Metamediality: From Analog Representations to. . .

1171

Given the generic and flexible capacities of digital computers – their programmability – a great variety of digital media have emerged, which in recent decades have been consolidated as critical infrastructures. Looking beyond metamedia as material technologies, the chapter further considers the social institutions that digital media are variously reproducing and transforming, as well as the discursive genres that metamedia both inherit and innovate. Most important, metamedia underscore the insight that communications – the social uses of words, images, numbers, and other signs and symbols – constitute actions, in the context of their users’ private and public lives, and feeding into broader and deeper cultural practices and social structures. Second, the chapter reviews the main lines of research on digital media and communicative practices since the 1990s, including some of its intersections with studies of traditional or legacy media and cultural institutions. In a perspective of intermediality, the sorts of legacies and links between media, genres, and discourses that used to exist, above all, as the intentions and interpretations of authors and audiences, have increasingly been externalized and operationalized, programmed as part of the system. Alongside studies of particular digital media, research has taken on board the many interconnections and interdependencies of contemporary culture, suggesting a reorientation of the field of media and communication research, from media as delimited organizations and messaging systems toward communication as practices that flow across platforms, devices, and contexts. Whereas research has commonly centered on prototypes of one-to-one, one-to-many, and many-to-many communication, the review includes a fourth prototype of many-to-one communication, as manifested by the data and metadata that ordinary users communicate into the system, in and of their use of digital media. Such data and metadata lend themselves to empirical analysis in terms of what linguistics and semiotics have referred to as connotation languages and metalanguages. One legacy of intermediality studies, these categories provide a point of departure, among other things, for historical and culturally comparative studies of both intermediality and metamediality. The third section elaborates on the relationship between metamedia and metacommunication: the many and mostly implicit messages that facilitate and frame the transmission or sharing of information. Metacommunication is a constitutive part of all human communication, but it has acquired strategic importance in the case of digital media. Because the users of digital media feed selected aspects of the wider contexts of their interactions into the system, this information remains available and accessible – as data and metadata – for subsequent processing, analysis, and application in opaque chains and networks. Personalized advertising and recommendations from services such as Amazon, Netflix, and Spotify are among the familiar end products of metacommunication in metamedia – programmable forms of intermediality with structural consequences. As such, metamediality represents a necessary addition to the agenda of intermediality studies in digital times, while simultaneously raising contested issues for public debate and policy development.

1172

K. Bruhn Jensen

The fourth and final section of the chapter, accordingly, takes up some of the normative issues arising from digitalization. Whereas classic freedoms of information and communication remain essential concerns, metamedia enable novel forms of action, at a distance and over time, and unprecedented forms of archiving – practices which bring up precarious human rights of participation and privacy. The resulting challenges are taken to a new level by an emerging Internet of Things (IoT), which promises to link and network elements and processes of the natural environment, too, beyond the institutions and infrastructures of human societies. It is still an open question how IoT will be developed, administered, and regulated, with which kinds and degrees of transparency and oversight, so that the aggregation and recycling of vast quantities and diverse qualities of data may serve the interests of the users who both supply and demand the data that are the raw materials of IoT. To illustrate the interests at stake, the chapter refers, in conclusion, to a principle of habeas data to complement the classic principle of habeas corpus.

Metamedia: The Very Idea A lesson from the long history of human communication as well as the much shorter history of what, today, is commonsensically understood as media, is that changing technologies of representation and interaction have stimulated shifting conceptions of both media and communication (Park and Pooley 2008; Simonson et al. 2013). In his agenda-setting history of the very idea of communication, John Durham Peters (1999: 6) documented the paradoxical point that “mass communication came first”: It was only after the emergence of electronic media of communication – from the telegraph via telephones, radio, and television to the internet – that people came to think of communication as a generic form of human and social interaction that would unfold face-to-face as well as through a proliferating set of tools and technologies. It was not until 1960s that an understanding of “the media” as a delimited and distinctive social institution became part of common parlance (Scannell 2012). Originating from Harold A. Innis (1951, 1972[1950]) and popularized by Marshall McLuhan (1962, 1964), the tradition of medium theory has highlighted media as enabling as well as constraining conditions of representing and reflecting on reality. This process – of reinterpreting the media and modalities in and through which individuals and collectives will interpret the human condition – parallels and extends the practice of what the sociologist, Anthony Giddens (1979), summed up as double hermeneutics. Compared to natural, material, and technical sciences, social sciences and humanities interpret a reality that has already been interpreted by the subjects inhabiting it, and when offered alternative interpretations, people may think and act differently, sometimes radically so. Classic instances have been the works of Karl Marx and Sigmund Freud, as disseminated far and wide through

46

Intermediality and Metamediality: From Analog Representations to. . .

1173

any number of media, which led large numbers of people to think of wage labor in terms of exploitation and oppression, and of the workings of something called the unconscious as a real and present force shaping both private and public life. Modern media of communication have become increasingly prominent carriers of a continuous process of interpreting and reinterpreting common concerns in public. As consolidated throughout the twentieth century and into the twenty-first century, media have come to constitute institutions-to-think-with (Jensen 1991) that admit and involve, in principle, anyone as someone (Scannell 2000), as participants in societies that could be seen to run on information and communication. The idea of a media, communication, or information society has been a long time coming. Following the modern turn of science and scholarship, which subordinated theory to practice and redefined practice as production rebuilding the natural environment for human and social ends (Lobkowicz 1967), industrial and other material production took on a scale and scope that presented major challenges of coordination and regulation, accelerating during the nineteenth century and into the twentieth century. Matter was getting out of control, and had to be informed by novel, immaterial means. Summed up as a control revolution (Beniger 1986), solutions were devised from the late nineteenth century, relying on systems of bureaucratization and rationalization, and deploying technologies such as telegraphy, telephony, and punch-card processing. After 1945, a consensus took hold that industrial society had transitioned into an information society (Daniel Bell 1973; Porat 1977), in which, quantitatively, a majority of the workforce is employed in information and other service occupations, while, qualitatively, information has become a key means of production in its own right (and an increasingly profitable consumer product) and, hence, a strategic element of private business as well as public governance. As information and communication technologies came to enable economic globalization across long lines of production and distribution (Baldwin 2016), the public and popular side of these same technologies entailed variable measures of cultural globalization as well – more media and more intermediality for more people across more social and cultural contexts and networks (Castells 1996). A decisive juncture between the nineteenth-century control society and the twentieth-century information society was marked by the digital computer, less as a material device than as an immaterial, but extremely practical and productive principle extending, once again, the transformative capacities of humans vis-à-vis their natural and cultural environments. Grounded in pioneering work by the British mathematician and philosopher, Alan M. Turing (1965[1936]), digital computers started out as behemoths the size of family homes with specialized military, scientific, and business applications. Already in an early stage, however, the idea of computers as personalized resources of information and communication was outlined in a 1945 article by the US wartime science administrator, Vannevar Bush (1999[1945]), still awaiting the development of a range of enabling technologies as well as mass consumer markets. Whereas the term “personal computer” was claimed

1174

K. Bruhn Jensen

by IBM for its 1981 product line, it was the Apple Macintosh that came to symbolize the breakthrough of computers into the everyday lives of the general public. Referring back to the principle of representative democracy – one person, one vote – Apple advertised its innovation on a complementary principle of one person, one computer, adding in television commercials during January of 1984 that, thanks to the Macintosh, “1984 won’t be like 1984” (Jensen 1993). It was during the emergence of personal computing that the concept of metamedia was articulated by Alan Kay and Adele Goldberg (Kay and Goldberg 1999[1977]). Members of the Xerox Palo Alto Research Center (PARC), which hosted key developments at the intersection of digital computing and human communication, they challenged the widespread understanding of computers as basically technical instruments and tools of engineering, exploring instead the embedding of computing in social practices spanning work and education, arts and entertainments, public and private life. As means of representation, computers join text, image, and sound in some new and many old forms and contents, as inherited from both mass media and face-to-face encounters: narratives, debates, games, and much else. As modes of interaction, computers support one-to-one, one-to-many, as well as many-to-many communication. In the longer perspective of media history (Briggs et al. 2020), computers integrate certain media that were born digital (social network sites, search engines, blogs, and some still taking shape or not yet thought of), while others have been adopted and adapted onto digital platforms (books, newspapers, film, broadcasting). This is on top of computers being the source of a more or less personalized artificial intelligence, as currently illustrated by personal digital assistants (Gunkel 2020). In the perspective of intermediality, the digital computer reproduces and recombines all currently conceivable media, from human bodies to tools and technologies. At the time of Kay and Goldberg’s (1999[1977]) intervention, digital computers were still commonly understood as stand-alone resources of information, rather than as networked media of communication and other social interaction. In the intervening decades, another kind of metamedium has emerged – the Internet with its many constituent media, genres, and discourses – which merits sustained attention from intermediality studies. The next section reviews the main lines of three decades of research on the Internet and other digital media and communication, through the conceptual lens of metamediality (see further Humphreys et al. 2018; Jensen 2022). The review is structured according to prototypes of one-to-one, one-to-many, manyto-many, and many-to-one communication, each of which has given rise to significant research programs redeveloping the agenda of media and communication studies. More than the presentation of information and the exchange of communication, digital media enable action at a distance – interaction – in a wide variety of discursive genres and across the full range of social institutions. Anticipating the following section on metacommunication – communication about communication – the review highlights the interlinked nature of the data and metadata that metamedia circulate. In doing so, metamedia greatly extend what people can do with media, but also what media may do to people (Katz 1959).

46

Intermediality and Metamediality: From Analog Representations to. . .

1175

Three Decades of Digital Media Studies Beyond Cyberspace Research on the digital computer as a media of communication gathered momentum from the 1990s, when the Internet (which had originated as a military invention in the shape of the 1969 ARPANET (Abbate 1999)) became freely accessible to the general public in the form of the World Wide Web with its interlinked homepages and audiovisual affordances (Berners-Lee 1999). From the outset, the basic architecture of many distributed clients accessing the same servers offering factual information as well as far-fetched fictions, stimulated a widespread understanding of the Internet as a world apart, soon to be widely celebrated both in popular culture and by public opinion. Deriving from William Gibson’s (1984) science fiction novel, Neuromancer, the idea of a cyberspace invited identity experiments and political utopianism, and both perspectives were taken to heart by a good number of scholars, too, in the window of intellectual opportunity between the 1989 fall of the Berlin wall and September 11, 2001. Book titles by the psychologist, Sherry Turkle, for instance, suggested the ambivalent attractions of migrating between realities: The Second Self (Turkle 1984) was followed by Life on the Screen (Turkle 1995) with a subtitle wondering and worrying about the state of “identity in the age of the Internet.” The titles of numerous other contributions to an early mainstream of “new media” studies sported references not just to cyberspace, but to presumed cybersocieties and cybercultures as well (e.g., David Bell and Kennedy 2000; Benedikt 1991; Jones 1998).

Communications Across Media As another millennium dawned, it was still possible, in 2003, to interest millions of people in participating in a vast virtual environment entitled Second Life, featuring elaborate representations and complex functionalities covering diverse social institutions and cultural practices (and still in operation, https://secondlife.com, accessed January 15, 2023). But, as the Internet gradually came to be taken-for-granted (Ling 2012) as part of offline institutions and practices, its forms, contents, and uses – and the corresponding theoretical and empirical approaches – were differentiated accordingly. Figure 1 lays out six prototypes of human communication which, historically, have been hosted by different technologies and institutions, but which have been recombined and reconfigured in and through metamedia (Jensen and Helles 2011). Along the vertical dimension, the figure divides communicative practices according to their number of participants: How many are in a position to address how many others? Along the horizontal dimension, the question is whether their interaction occurs immediately, in a shared time, even if, sometimes, relating far distant places. One-to-one interaction is the evolutionary and historical prototype of human communication. Whereas the points of access to cyberspace had typically been

1176

K. Bruhn Jensen

Fig. 1 Six communicative prototypes

visualized as desktop screens, from the 2000s access to the Internet and, thus, to other individuals and institutions was increasingly mediated by mobile devices, first basic cell phones, but soon smartphones featuring a broad range of media and genres as applications – apps, for short. It was the global diffusion of mobile phones that explained why and how, by the early 2020s, more than half of the world’s population had ended up connected to the Internet (ITU 2021). And yet, most of the one-to-one communications linking billions of people to each other would still flow quite close to home. In principle, anyone might speak to and with anyone else, anywhere in the world; in practice, mobile and other digital media studies documented that, as a social rule, people will predominantly communicate with others they already know and typically have face-to-face contact with, too (Baym 2015; Ling et al. 2020). In parallel, writing came back in style as a common mode of interaction, as email and, more pervasively, as texting, which became the first killer application of mobile phones (Castells et al. 2007), followed by a range of digital messaging services. In the perspective of intermediality, metamedia, mobile or not, offered a wide variety of media and genres for ubiquitous communication – spoken, written, and multimodal – which each could be selected, but also combined, depending on context and purpose. One-to-many or mass communication, next, for centuries had been an essential vehicle of fact and fiction, for instrumental and reflective ends, initially for small reading minorities. With mass markets and comprehensive schooling in place in a growing number of countries, audiences came to include majorities of national populations in the Global North (Briggs et al. 2020). With personal computing and the Internet ascending, an early and popular prediction, advanced by both pundits and scholars, was that traditional mass communication was about to be replaced by much more personalized and networked interactions. In a terminology expounded by Nicholas Negroponte (1995), push communication belonged to the past, and pull

46

Intermediality and Metamediality: From Analog Representations to. . .

1177

communication would be the wave of the future, as people could pick and choose from great masses and varieties of contents. Already early web pages, however, hinted that users might encounter a relatively standardized fare, in a rather static Web 1.0 universe. As a so-called Web 2.0 added more interactive functionalities, users still voted with their digital footprints to remain, to a considerable extent, recipients of already familiar genres of news and entertainment. The established senders of broadcasting, but also of cinema, literature, and, importantly, gaming, moved their operations online, streaming tried-and-tested formats, which remained immensely popular. Comparative studies have found that mass communication is alive and doing surprisingly well, in very different cultural settings, such as mainland China, Europe, and the United States (Jensen and Helles 2023). A plausible historical explanation is that mass communication, once introduced and adopted as an everyday staple, will remain attractive across social segments and cultural traditions, a global common denominator of sorts. In sum, although classic mass media may have begun a gradual decline, with the future of news media as a fourth branch of governance hanging in the balance, mass communication appears as lively and sturdy as ever. For intermediality and other media studies, the lesson appears to be a shift of research agendas from media towards communication – the communicative prototypes and practices that different media and metamedia can be expected to keep on carrying. Many-to-many communication represents the first of two distinctive components of contemporary media environments, as associated with digitalization, and with implications for intermediality both as a theoretical concept and as an analytical category. Whereas anyone can address anyone else, anywhere and anytime, not just individually, but collectively, as parties to a new global communication infrastructure, it remains an open question how this prototype may evolve over the coming decades, as it is adapted to variable social institutions and cultural contexts. The so-called social media – which are no more social than other media, if certainly conducive to other forms of sociality than the mass media of the past four centuries – are no more than two decades old. The subfield studying social media is itself still getting its bearings and beginning to compare and contrast both variants and ancestors (Bruns 2008; Highfield 2016; van Dijck 2013; van Dijck et al. 2018). Among users’ as well as scholars’ favorites have been social network sites such as Facebook, LinkedIn, and TikTok (Boyd and Ellison 2007), video services such as YouTube (Weller et al. 2014), and blogs (Rettberg 2013) and microblogs such as Twitter (Weller et al. 2014). Across these variants, and in contrast to the media systems of the twentieth century, it is society’s full range of individual and institutional voices that speak up on social media, from ordinary citizens and influencers in cultural and lifestyle domains, via corporate business and legacy media organizations, to governmental agencies addressing citizens within and beyond national borders, including for purposes of misinformation, disinformation, or computational propaganda (Woolley and Howard 2018). Also on social media, however, some are evidently more equal than others, in terms of the material and immaterial resources at their disposal, and hence their agency, in and by communication. Compared to early hopes vested in cyberspace and, subsequently, more participatory forms of

1178

K. Bruhn Jensen

“connective action” (Bennett and Segerberg 2012), research increasingly has approached social media, and the Internet as such, as arenas of classic social struggles, as part and parcel of the real world of power and privilege (Rosen 2022). It should be added that, like the rest of the communications field, digital media studies have largely focused on public forms of interaction, whether regarding political and cultural or personal and existential matters. Technologies and practices of communication in other social domains – notably private business, but also healthcare, education, and further sectors – have been examined in parallel and comparative isolation, in fields such as human-computer interaction (HCI) (Jacko 2012) and computer-supported cooperative work (CSCW) (Schmidt and Bannon 2013). This is in spite of sustained efforts in organizational communication research to bridge sectoral divides (Scott and Lewis 2017). With digitalization, public and private communications have come to constitute, in significant ways, the front offices and back offices, respectively, of common infrastructures and political economies. Intermediality pervades the things that people do with media at work as well as at home, because the technologies are generic, just as the social institutions that they support are interdependent. Commercial investment in and social planning of communication infrastructures are both premised on the insight that citizens and consumers will navigate professional, public, and private spheres of their lives through the same networks and devices, and within single sessions on one desktop, laptop, or smartphone.

Speaking into the System Across sectors and spheres, digital media have reemphasized the point that communication, more than a representation of reality, is a mode of acting in and on reality. The interactive functionalities for presenting oneself, addressing others, selecting goods and services, or contacting public authorities, are short-term manifestations of long-term processes of becoming and remaining a participant in a society and a culture. Communication theory is heir to a central insight of twentieth-century philosophy, from the late Wittgenstein (1953) to speech-act theory (Austin 1962; Searle 1969) and beyond, namely, that both senders and receivers do things with words and other signs and symbols. In this regard, digital media stand out from previous analog media forms. Not only does digital computing extend the human capacities for acting in and on natural and social environments in decisive ways, it also facilitates the recording and recycling of individual interactions and transactions. Alongside many-to-many communication, it is such many-to-one communication – spoken into the system (Jensen and Helles 2017) – that represents the second historically distinctive component of contemporary communication systems. Many-to-one communication represents an addition and amendment to the six prototypes laid out in Fig. 1. Certainly, many-to-one communication is not without precedents; historical examples include the census, tax registers, and military drafting. But, compared to these public and nominally regulated systems, digital media are being consolidated as privately operated, commercial entities that

46

Intermediality and Metamediality: From Analog Representations to. . .

1179

capitalize on the resulting data streams. In response, the 2010s witnessed a substantial growth in critical studies of the structural consequences of what has been referred to, variously, as platformization or infrastructuralization (for overviews, see van Dijck et al. 2018; Plantin et al. 2018). On the one hand, platforms such as the so-called Big Five – Alphabet (Google), Amazon, Apple, Meta (Facebook), and Microsoft – have become critical conditions of numerous private and public endeavors, from everyday traffic navigation and home entertainment to personal banking and corporate finance. On the other hand, these same quasi-monopolies have been supplanting classic infrastructures of the past century and more – post, telephone, and telegraph, historically organized as public services. As detailed by van Dijck et al. (2018), platformization and infrastructuralization depend on additional practices of both datafication – rendering as data what used to be spoken into the air or kept as private notes and reflections – and commodification, as content data as well as user data are rendered as commodities that can be bought and sold. The critique of the resulting social and cultural inequalities, in addition to likely continuing abuses, has been summed up, in the terminology of Shoshana Zuboff (2019), as surveillance capitalism.

The World as a Medium The last step of digitalization, so far, is a so-called Internet of Things, currently emerging at the intersection of technological innovations and political economies (for overview, see Bunz and Meikle 2018; Greenfield 2006; Howard 2015). The preferred terminologies still vary, but the shared assumption is that digital media of the future will be categorically different from current distinct devices with interfaces – workstations, laptops, smartphones – with which users interact in dedicated fashion. Instead, information may be ingrained in all manner of natural objects, everyday artifacts, and social infrastructures. Early versions of virtual media environments, during the 1980–1990s, had been based on a model of “the world in a medium” – one local, stationary, multimodal interface, including goggles, gloves, and a treadmill enabling users to enter a different, separate, virtual reality (for overview, see Levy 1992). In contrast, ubiquitous and pervasive computing proposes an embedding of diverse communicative interfaces into natural objects and social artefacts – “the world as a medium.” In homes, for example, appliances are being automated and integrated through personalized and localized, yet globally connected digital assistants. In public spaces, an Internet of Things has been associated with the development of smart cities, managed and regulated through interconnected networks, and promising environmentally sustainable solutions in areas such as public utilities and transportation systems. A key aspect of these developments has been a return, in recent decades, of the idea of artificial intelligence to scientific as well as public agendas, suggesting that the imagined boundaries and interrelations between humans and their machines may be shifting once again (Gunkel 2020). Metamedia in general, the Internet of things in particular, have served as reminders that communication as most commonly understood – the transmission

1180

K. Bruhn Jensen

or sharing of information – is accompanied by an additional range of intimations and implications: metacommunication. Indeed, the concept of metacommunication provides an interdisciplinary interface between evolving metamedia and classic approaches to languages as means of both communication and analysis. Whereas metamedia constitute a recent addition to the repertoires of representation and interaction, metacommunication is a constant partner in human communication. This is in spite of the fact that, in and by metamedia, metacommunication has come to be recorded and recycled in unprecedented ways.

Metacommunication in and by Metamedia Metacommunication in the Flesh The very idea of metacommunication was first articulated with reference to embodied face-to-face interaction by the polymath, Gregory Bateson, who suggested that “human verbal communication can operate and always does operate at many contrasting levels of abstraction” (Bateson 1972[1955]: 150). With reference to a philosophical standard – “the cat is on the mat” – Bateson noted that this statement, most basically, holds a denotation or literal meaning regarding an actual state of affairs: the position of a furry four-legged organism in space (on a mat that the person speaking can point to) and time (is, not was). But, beyond the level of denotation, speakers typically rely on two additional categories of information in conversation: metalinguistic information, for example, to clarify that the referent of the word “cat” includes tigers, and metacommunicative information to suggest a particular relationship with the other participants in the conversation, “e.g., ‘My telling you where to find the cat [tiger] was friendly,’ or ‘This is play.’” Operating at several levels at once, communication delivers a whole range of implicit information that must be inferred from various material and discursive aspects of the context of the interaction: “[T]he vast majority of both metalinguistic and metacommunicative messages remain implicit” (p. 151). And, to complicate matters further, one of Bateson’s earlier publications had added that “a majority of propositions about codification are also implicit or explicit propositions about relationship and vice versa” (Ruesch and Bateson 1987[1951]: 209). The meaning of what we say to each other simultaneously implies the meaning of our relationship. It is because of such textual as well as contextual complexities that human communication so often gives rise to uncertainty about what people are really saying, and why, feeding confusion and conflict. In a related argument, Bateson (1972[1956]: 173–198) found that schizophrenic disorders could be approached as the outcome of a “double bind” in which a person is unable to resolve several conflicting levels of representation and interaction. Most people, however, are masters at navigating these complexities, at least most of the time. We recognize that “this is information” (a signal of something else), and that “this is this kind of information” (a message in a specific code, with a particular reference to reality, and a likely relevance for us, then and there). Through socialization and acculturation,

46

Intermediality and Metamediality: From Analog Representations to. . .

1181

we build and rework our informational and communicative relationships to other people and the world as such, relying on elaborate conventions of expression, turntaking, and role-playing. In doing so, we maintain contexts that are experientially and socially real, what Bateson described as frames (Bateson 1972[1955]: 157). This concept was fed into wider humanities and social sciences by, among others, Erving Goffman (1974) in analyses of how frames are observed, but also continuously modified or replaced, as part of everyday social interaction. Media and communication studies have gone on to explore how, for instance, journalists depend on frames to find and report “the news” (Hanusch and Marres 2021), and how their readers, listeners, and viewers, in turn, make sense of the news on offer, partly with reference to conflicting or competing frames (Jensen 2021a: 169–171).

Models of Mediation from Bateson to Barthes Like Goffman, Bateson had stayed focused on face-to-face verbal interactions. But the two aspects of metacommunication – the codification of messages and the articulation of communicative relationships – equally apply to writing and print, and to audiovisual modalities, too. The genre systems of literature, cinema, and broadcasting all codify their messages – anticipate and instruct their interpretation – while also articulating discursive positions for their audiences to occupy. Writing in the tradition of cybernetics, Bateson (1972) recognized the relationship between human communication and social control – signs and actions – as a general and close one. It fell upon another tradition with a family resemblance to cybernetics, however, to elaborate models of metalinguistic and metacommunicative analysis: semiotics, specifically the structuralist variant developed by Roland Barthes (1967[1964]; 1973 [1957]). Since the 1960s, especially one of his models has become a mainstay of media studies, proving applicable in analyses of the cultural implications of the mass media that came of age concurrently, during the second half of the twentieth century (at the same time proving itself teachable to growing contingents of media students around the world). A second and complementary model has remained marginal in work on media and intermediality, but has acquired renewed relevance for the understanding of metacommunication with the global diffusion of metamedia. Barthes was standing on the shoulders not of Bateson or cybernetics, but of linguistics in the tradition of Louis Hjelmslev (1963[1943]). Barthes’ (1973[1957]) most widely referenced model proposed to account for the micro-mechanics of how multiple layers of meaning come to make up worldviews or ideologies – what an early collection of articles termed mythologies. In formal terms (Fig. 2), the combination of the expressive form (signifier) and the conceptual content (signified) of a word, statement, or image – a language, broadly speaking – constitutes a denotation or basic meaning. At a second level, this first sign makes up the form of a secondorder sign, and when a further content is added, this generates what Barthes (and Hjelmslev) referred to as a connotation language. As connotations enter into still more complex configurations of meaning, they amount to comprehensive views of the world with normative implications. One of Barthes’ famous examples was a

1182

K. Bruhn Jensen

Fig. 2 Language and connotation language. (Adapted from Barthes 1973[1957])

magazine cover (form) with an image of a young black man in a uniform saluting the French flag (content), which could be seen to carry a second-level, viciously ideological content, namely, that French imperialism was not a discriminatory system: Under the flag and in uniform, all French are created equal. The immediate point of Barthes’ analysis was ideology critique; its further contribution was a robust model of how codification and articulation may naturalize certain worldviews, while delegitimating and silencing others. Barthes’ application of Hjelmslev’s formal concepts and original terms was debatable. Nevertheless, his appropriation of Hjelmslev’s basic figure of thought and analysis proved massively influential in research and education. Being an archetypal twentieth-century linguist, Hjelmslev had approached languages as systems – systems of communication and second-order systems that either build on or describe such systems. Barthes’ accomplishment was to transpose this logic to communication as a practice and a process: Connotation languages can be examined not only through formal analysis and in the rearview mirror, but as they are articulated and take effect in social and cultural practices. Connotations represent codes that have been accumulated in the sorts of representations and interpretive frames that students of intertextuality and intermediality track across historical time and cultural space (Bakhtin 1981; Fiske 1987; Kristeva 1984). And, with a retrospective and interdisciplinary nod to Bateson’s contribution, connotations could be conceived as codifications – the metalinguistic aspect of metacommunication – suggesting inferences and implications of the words or images being said or shown here and now. What Hjelmslev (1963[1943]) and, subsequently, Barthes (1967[1964]) termed metalanguages corresponded to the other aspect of Batesonian metacommunication: the articulation of communicative relationships. Metalanguages have a different, if complementary relation to the common reference point: first-order languages or languages as intuitively understood by the individuals speaking them. Whereas connotation languages are themselves languages or modes of expression, metalanguages rather describe languages: They are not languages in themselves, but languages about languages, for instance, syntactical or semantic descriptions of the English language. Figure 3 lays out the logic of metalanguages. Compared to Fig. 2, this last figure reverses the relation between signifier (form) and signified (content) at

46

Intermediality and Metamediality: From Analog Representations to. . .

1183

Fig. 3 Language and metalanguage. (Adapted from Barthes 1973[1957])

the second level. A metalanguage constitutes a form that serves to configure a particular social relationship between the people communicating and with reference to the content in question. Linguists will take the analyst’s rather than the user’s stance vis-à-vis the language that they speak, but also study as a scholarly community. Hjelmslev (1963[1943]) had made an antecedent distinction between scientific and nonscientific languages. In his vocabulary, metalanguages would be scientific languages, as defined by formal procedures, but presumably accessible, above all, for expert users of languages such as linguists. In some media and modalities, however, metalanguages are as accessible as connotation languages. Ordinary users of metamedia effortlessly employ a whole range of metalanguages, customizing their profiles at social network sites; tagging and commenting on posts by others; forwarding news stories to family and friends from websites via embedded messaging services; and pulling a later push of information to themselves through alerts and newsletters. Much more than analytical systems, metalanguages serve as practices of communication – metacommunication – in metamedia. Responding to the experience of a cultural environment suffused by mass communication, Roland Barthes (1973[1957]) had mobilized linguistic resources for analytical and critical purposes. Responding to another historical reconfiguration of communication systems, current research has been recruiting additional interdisciplinary resources – from Bateson to Barthes and beyond – to account for the several senses in which people can be said to communicate with metamedia (Jensen 2013, 2022; Lomborg and Frandsen 2016). Metacommunication is key to what people do with metamedia, but also to what metamedia do to people. Metacommunication is both an inherent aspect of the universal human practice and process of communication and a source of contested products with ethical and political implications.

Metacommunication as Process and Product An early insight of communication studies, duly inspired by Gregory Bateson, was that human beings “cannot not communicate” (Watzlawick et al. 1967: 49). In faceto-face settings, people will constantly try to make sense of what they are (meta)

1184

K. Bruhn Jensen

communicating to each other. Goffman (1959) went on distinguish between the signals that people intentionally give and those that they happen to give off, which may be as important, even decisive, both for the process of interaction and for its outcomes. In metamedia as well, users cannot not communicate. It is by now common knowledge that users leave behind bit trails that are systematically harvested, and which are analyzed and recycled through nebulous networks, and applied by private as well as public operators. Illustrious examples have included the 2013 revelations by Edward Snowden of the global surveillance of citizens by the US National Security Agency, and the Cambridge Analytica incident relating to the 2016 US presidential election and centering on the business practices of Facebook (now Meta) and its associates. Most important, such practices have become standard operating procedure, not a bug, but a feature of metamedia. As exemplification of metacommunication at work, consider search engines (Halavais 2018), with Google as the near-monopoly on the web in the Global North. Search engines both codify information and enact communicative relationships. Depending on the coding of the available information, people gain access to it (or not) as information and as this kind of information with that likely relevance for them in a given context. In searching, people simultaneously establish a communicative relationship, not necessarily with identifiable individuals or institutions, but with a distributed resource of information, which supports further communication and other social interaction. And, in and of their searches, users provide input to and reconfigure the search engine (however minimally in each instance) regarding which information may be of interest, to whom, related to what other information, and in which contexts. Users’ metacommunication thus prefigures many subsequent communications, by themselves and by others, whether searching for more information or interacting about, and perhaps acting on, either single searches or long sequences of inquiries. In face-to-face metacommunication, people listen for tones of voice and look out for the faintest of bodily signals. In classic mass media, audiences will read between the lines of texts and the sutures of images. In metamedia, codifications and social relationships are documented and accumulated through fine-grained technological and institutional architectures, which are tapped by a few owners and operators. Returning to the typology of Fig. 1, as amended to include many-to-one communication, a concrete example of synchronous many-to-one interaction is the commercial practice of real-time bidding for advertising space, in which the available data on users and their trajectories inform a personalized push of promotional messages. A corresponding example of asynchronous many-to-one interaction is the accumulation of user preferences over an extended period that underlies recommendations regarding particular genres or works (and other products and services) in systems such as Amazon, Netflix, and Spotify. The audience is the product, as argued by critical media and communication researchers since the heyday of commercial flow television (Smythe 1977), and a notion extending to a digital marketplace of attention (Webster 2014). Metacommunication in digital systems yields at least two categories of products. First, at the interface between user and system, interactions are registered, for example, as

46

Intermediality and Metamediality: From Analog Representations to. . .

1185

time stamps and as records of user flows across wider environments of information and communication: Who attended to what, when, where did they come from, and where did they go next, with what implications for their commercial profiles as consumers or their political and cultural identities as citizens? Second, in the interchange between users, the interactions comprise both codifications and social relationships, communication as well as metacommunication. Emojis, tags, and other commentary contribute to the meaning of a message (through connotation languages) and, simultaneously, to the experience of being a self, an other, and, perhaps, a community of sorts (through metalanguages). In the aggregate, and in both respects, realities are produced, shared, and reaffirmed. In a benchmark article, James W. Carey (1989[1975]) had compared and contrasted two perspectives on human communication, as transmission in a technical sense and as ritual in a cultural sense, respectively. Citing the philosopher, John Dewey, Carey found it essential to recognize that society exists not only by communication, as an end product, but in communication, as it unfolds. He went on to characterize the latter perspective on communication as the process in which reality is produced, maintained, repaired, and transformed – what other traditions have termed the social construction of reality (Berger and Luckmann 1966). In hindsight, Carey and Bateson offered complementary perspectives on the common practice of communication and metacommunication. Also metacommunication is a continuous process which, in present technological and institutional circumstances, generates distinctive products of great economic value and with contested ethical implications. Referred to, in a technical vocabulary, as metadata – data about data – the products of metacommunication potentially carry rich information, which is recommunicated and recycled by system administrators, advertisers, and numerous third parties. The further question is who will be in a position to codify what – and whom – resulting in which social relationships, in subsequent unforeseen and unforeseeable contexts. It is this question, in particular, which has motivated a recent wave of research and debate on the normative implications of metamedia and metacommunication (for overview, see Jensen 2021b).

The Politics of Metamedia and Metacommunication Big Science for Big Data Digitalization of diverse human endeavors, across social sectors and around the world, is currently consolidating a version of the information society that had been anticipated in academic research and public debate for half a century (Daniel Bell 1973; Porat 1977). Roughly since the turn of the millennium, a complementary terminology of big data has become widespread, highlighting the scale of digital infrastructures and the scope of their embedding into the basic institutions of modern societies (Mayer-Schönberger and Cukier 2013). Being essential parts of the world economy and geopolitics, global data flows have become subject to intense scholarly as well as political interest.

1186

K. Bruhn Jensen

Configured as information and institutionalized as knowledge, data have been important raw materials throughout history, as increasingly recorded in more or less durable media. Institutions of knowledge (from scientific academies to research universities) and institutions of communication (from postal systems to networked databases) have served as curators and disseminators of what, in a given time and place, could be known and shared – by anyone as someone (Scannell 2000) with rights and responsibilities. For centuries, the task of preserving and preparing extant information for later use, first by small elites and then, very gradually, entire populations and publics, had been entrusted to libraries, museums, and other archives. Whereas current public and private initiatives range from national digital archives (e.g., https://dp.la) to Google Books (https://books.google.com), sustainable infrastructures for an age of exponentially growing quantities and qualities of data, information, and knowledge have yet to emerge. It is such infrastructures that may support informed consent and public participation in working out the future trajectory of information societies. The scale of the challenge, at once intellectual and political, recalls what came to be known, after the 1945 caesura, as big science (Weinberg 1961): grand, collaborative, basic research that prepares the ground for many more specialized and applied projects, some or, likely, most of them still waiting be imagined. The massive funding and machineries required for big science to operate are illustrated by CERN, the European Organization for Nuclear Research, which spawned a seemingly unrelated invention: the World Wide Web (Berners-Lee 1999). Big science has been characteristic of military research and product development, as exemplified by the projects of the US Defense Advanced Research Projects Agency, which initiated what developed into the Internet, the so-called ARPANET launched in 1969 (Abbate 1999). In both instances, the outcome was a generic technology with applications and implications for diverse institutions of private and public life, from the local to the global level. For the first time in human history, there are local and global institutions of knowledge and communication in place to support the design of infrastructures that may carry information across generations, according to explicit and contestable criteria of curation and dissemination. The examination of criteria and ends – the common good to be served by information and communication across generations – suggests itself as a candidate for big science. The object of inquiry would be data as a necessary, if far from sufficient condition of eudaimonia – the good life of individuals and collectives. Who has the right to hold which kinds of data, about what, and about whom?

Habeas Data Metamedia have entailed a shift in conceptions of privacy. What previously required dedicated activities can now be accomplished through “automatic realtime data collection and analysis” (Agre 1994: 103) of user interactions with digital

46

Intermediality and Metamediality: From Analog Representations to. . .

1187

systems. The final word of this last quotation – analysis – is key, suggesting how the configurations of digital systems prefigure the analysis of who did what. Under the banner of big data, a notion of “collect now, analyze later” has been emerging. The common suggestion is that, instead of sampling data to support inferences to a wider population, research ought to gather “as much [data] as possible, and if feasible, getting everything: N¼all” (Mayer-Schönberger and Cukier 2013: 26). And, importantly, it is not just large quantities, but diverse qualities of data that are of value, in both economic and ethical senses of the word. One of the most elaborated and influential responses to the ethical and political implications of digital data has been outlined by Helen Nissenbaum (2011). Her theory of contextual integrity develops a communicative understanding of the principles governing the availability of data and the procedures for regulating the accessibility of data. With specific reference to personal data, but with wider normative implications, Nissenbaum’s (2011) argument is that the ontological status and potential practical ramifications of data cannot be decided once and for all, as they are captured in the architecture of devices and networks, and according to the general terms of service accepted by their users. Information travels through unpredictable contexts, in which it may be accessed and acted upon by third parties for entirely unforeseen and, perhaps, unacceptable purposes. Users cannot – ought not to – waive, in advance and as a default, the right to decide or influence what later happens to what is, in some sense, their data. It is, arguably, in both the personal and public interest that the social uses of data – as information and knowledge, in various media, some still to be invented – be regulated according to certain “context-specific substantive norms” (p. 32). The problem, in Nissenbaum’s words, is that “the dominant approach to [. . .] achieving privacy online is a combination of transparency and choice.” The unrealistic premise is that complex terms of service will be transparent to ordinary users as they click and accept. With a view to future and likely delicate contexts, so-called “notice-and-consent, or informed consent” (p. 34) cannot be considered informed. (For a critical discussion of Nissenbaum’s position, see Rule 2019.) The principles at stake were illustrated, in a longer historical perspective, by early responses to media environments of the late nineteenth century. Two American lawyers, Samuel D. Warren and Louis D. Brandeis (1890: n.p.), worried about the impact of “instantaneous photographs and newspaper enterprise” on “the sacred precincts of private and domestic life,” and advocated measures “for the protection of the person, and for securing to the individual what Judge Cooley calls the right ‘to be let alone’.” Personal information and the sense of having an individual identity have remained two sides of the same coin. In a philosopher’s perspective, Maurice Merleau-Ponty (1962[1945]: 174) summed up the point in saying that whereas people typically say that they have a body, they also are a body. We have information, about ourselves and others, but we also are information, which others may access and act upon, depending on the historical institutions and social practices regulating appropriate representations and interactions.

1188

K. Bruhn Jensen

Habeas data has been advanced, by analogy to habeas corpus, as a legal principle addressing the relevant concerns (for overview, see Guadamuz 2001). Habeas corpus, which derives from the English Magna Carta of 1215, and which became integral to modern legal systems around the world, literally means “so that you may have the body.” The principle is meant to protect individuals from unlawful imprisonment through the possibility of appeal, so that the courts must establish the whereabouts of prisoners – their corporeal existence. Habeas data – “so that you may have the data” – is intended to grant individuals access to archives or registers holding information about them and, if needed, to have information corrected or removed. Subject to various interpretations, the principle was first implemented in the constitution of Brazil in 1988 (Doneda and Mendes 2014). The European General Data Protection Regulation (GDPR), which took effect in 2018, addresses such issues, and is becoming a standard for digital privacy in other regions of the world. The idea of habeas data suggests that human beings both exist and coexist in and by information and communication. Online as offline, individuals and collectives are social in distinctive ways, depending on how data about them are recorded, analyzed, and applied. Communities are imagined (Anderson 1991), in part, with reference to the data that citizens leave behind as users of metamedia. Rights of data protection or privacy, accordingly, must be considered on a par with classic rights of information and communication. Like habeas corpus, of course, habeas data is no absolute principle; societies will assign only certain rights for individuals to have, hold, and use their information as well as their bodies. But the conversations fleshing out these rights are only just beginning. Media users increasingly speak into systems that not only structure their communications, but which condition their agency far beyond “the media” as traditionally understood (Jensen and Helles 2017). Beyond personal data (socially but also biologically identifying data, including personal genomes), information, its communication and enactment, present a range of contested and conflicted issues that are being underscored by an emerging Internet of Things (IoT). Information everywhere and in everything have been the commercial and policy buzzwords of the information society. In intellectual and public debate, the question remains open to what extent, and in which sense, metamedia and the IoT may serve the common interest. The currently available information has been accumulated through centuries and millennia of human inquiry and enterprise, resulting in a “communicative commons” (Murdock 2013: 160) that enables the cultivation of ever more information for humans to be and do things they have reason to value, and to coexist across time and space. Earlier debates have noted that information may “want to be free,” but it also “wants to be expensive because it can be immeasurably valuable to the recipient” (Brand 1987: 202), individuals as well as entire societies and civilizations. The emergence of metamedia and their reprogramming of metacommunication both suggest the importance of having studies of intermediality and its present and future infrastructures go on.

46

Intermediality and Metamediality: From Analog Representations to. . .

1189

Conclusion Digitalization invites reconsideration of the very ideas of media and communication. The digital computer constitutes a metamedium that recombines and reconfigures all previous media, while at the same time challenging received definitions of communication as the transmission or sharing of information. More than representations of reality, information and communication afford resources for acting in and on reality. Humans do things with words and other signs and symbols, as extended in tools and technologies enabling communities and civilizations. As foregrounded by the tradition of medium theory – from Harold A. Innis (1951, 1972) via Marshall McLuhan (1962, 1964) to John Durham Peters (1999, 2015) – the universal practice of communication is circumscribed by and inscribed in the media of a given historical time and cultural place. Metamedia constitute a significant component of contemporary repertoires of human communication and an altered condition of intermediality. With metamedia, the interrelations between media, genres, and discourses that used to exist, above all, in the imagination of authors and audiences, senders and receivers, now manifest themselves as technological and institutional infrastructures. Intermedia have become hypermedia, joined by clicks on links. Metamediality represents a new category of intermediality and, hence, a necessary addition to the theoretical and methodological agendas of research on intermediality. Media and communication studies in recent decades have contributed to the understanding of an ongoing transition from analog to digital media environments, examining the multiple flows of communication – one-to-one, one-to-many, and many-to-many – that different media enable, each in its own right and in configured combinations. Responding to the performative potentials of metamedia, the field has recognized communicative practices as forms of social action. People do things with media, by navigating interfaces and selecting contents, but also in carrying on ordinary and, occasionally, extraordinary aspects of their lives in networked interactions with other individuals and institutions. In doing so, they participate in the long-term structuration of societies (Giddens 1984) and the cultivation of local and global communities. Beyond the one-to-one, one-to-many, and many-to-many prototypes, metamedia host a great deal of many-to-one communication, which qualifies as metacommunication. Whereas all communication is accompanied and enabled by metacommunication, metamedia register metacommunication as products, which lend themselves to a wide range of social uses, for better and worse. To address the multiple levels and flows of communication that digitalization facilitates, research can stand on the shoulders of classic models of media and intermediality, from Bateson to Barthes, approaching data and metadata as connotation languages and metalanguages. As an Internet of Things is consolidated, studies of media and intermediality need all the models they can recover or reinvent to account for the theoretical as well as practical implications of a new category of communication infrastructures.

1190

K. Bruhn Jensen

The ethical dilemmas and political choices that human communication presents have been reemphasized by metamedia and digitalization writ large. The challenges have been studied and debated, so far, with special reference to issues concerning personal data and privacy. But intermediality and metamediality invite normative reflection and deliberation, in professional, public, and policy contexts, about all the things that media still do to people in different political economies and distinctive cultural settings. Media are constitutive of the human condition. Much more and, at best, big science is called for to analyze and assess what media are, but also what media ought to be, and what media could be.

References Abbate, Janet. 1999. Inventing the Internet. Cambridge, MA: MIT Press. Agre, Philip E. 1994. Surveillance and capture: Two models of privacy. The Information Society 10 (2): 101–127. Anderson, Benedict. 1991. Imagined communities: Reflections on the origin and spread of nationalism. 2nd ed. London: Verso. Austin, John L. 1962. How to do things with words. Oxford: Oxford University Press. Bakhtin, M.M. 1981. The dialogic imagination. Austin: University of Texas Press. Baldwin, Richard. 2016. The great convergence: Information technology and the new globalization. Cambridge, MA: The Belknap Press. Barthes, Roland. 1967. Elements of semiology. New York: Hill and Wang. 1964. ———. 1973. Mythologies. London: Paladin. 1957. Bateson, Gregory. 1972. Steps to an ecology of mind. London: Granada. Baym, Nancy K. 2015. Personal connections in the digital age. Cambridge: Polity Press. Bell, Daniel. 1973. The coming of post-industrial society. New York: Basic Books. Bell, David, and Barbara M. Kennedy, eds. 2000. The cybercultures reader. London: Routledge. Benedikt, Michael, ed. 1991. Cyberspace: First steps. Cambridge, MA: MIT Press. Beniger, James. 1986. The control revolution: Technological and economic origins of the information society. Cambridge, MA: Harvard University Press. Bennett, Lance W., and Alexandra Segerberg. 2012. The logic of connective action: Digital media and the personalization of contentious politics. Information, Communication & Society 15 (5): 739–768. Berger, Peter L., and Thomas Luckmann. 1966. The social construction of reality. London: Allen Lane. Berners-Lee, Tim. 1999. Weaving the web: The original design and ultimate destiny of the world wide web by its inventor. San Francisco: Harper. Boyd, Danah M., and Nicole B. Ellison. 2007. Social network sites: Definition, history, and scholarship. Journal of Computer-Mediated Communication 13 (1): 210. http://jcmc.indiana. edu/vol13/issue1/boyd.ellison.html. Brand, Stewart. 1987. The media lab: Inventing the future at MIT. New York: Viking. Briggs, Asa, Peter Burke, and Espen Ytreberg. 2020. A social history of the media. 4th ed. Cambridge: Polity. Bruns, Axel. 2008. Blogs, Wikipedia, Second Life, and beyond: From production to produsage. New York: Peter Lang. Bunz, Mercedes, and Graham Meikle. 2018. The internet of things. Cambridge: Polity. Bush, Vannevar. 1999. As we may think. In Computer media and communication: A reader, ed. Paul A. Mayer. Oxford: Oxford University Press. Original edition, 1945.

46

Intermediality and Metamediality: From Analog Representations to. . .

1191

Carey, James W. 1989. A cultural approach to communication. In Communication as culture, ed. James W. Carey, 13–36. Boston: Unwin Hyman. Original edition, 1975. Castells, Manuel. 1996. The rise of the network society. Oxford: Blackwell. Castells, Manuel, Mireia Fernández-Ardèvol, Jack Linchuan Qiu, and Araba Sey. 2007. Mobile communication and society. Cambridge, MA: MIT Press. Doneda, Danilo, and Laura Schertel Mendes. 2014. Data protection in Brazil: New developments and current challenges. In Reloading data protection: Multidisciplinary insights and contemporary challenges, ed. Serge Gutwirth, Ronald Leenes, and Paul de Hert, 3–20. New York: Springer. Fiske, John. 1987. Television culture. London: Methuen. Gibson, William. 1984. Neuromancer. New York: Ace. Giddens, Anthony. 1979. Central problems in social theory. London: Macmillan. ———. 1984. The constitution of society. Berkeley: University of California Press. Goffman, Erving. 1959. The presentation of self in everyday life. New York: Anchor Books. ———. 1974. Frame analysis. Cambridge, MA: Harvard University Press. Greenfield, Adam. 2006. Everyware: The dawning age of ubiquitous computing. Indianapolis: New Riders. Guadamuz, Andrés. 2001. Habeas data vs the European data protection directive. Journal of Information, Law & Technology (3). https://warwick.ac.uk/fac/soc/law/elj/jilt/ Gunkel, David J. 2020. An introduction to communication and artificial intelligence. Cambridge: Polity. Halavais, Alexander. 2018. Search engine society. 2nd ed. Cambridge: Polity. Hanusch, Folker, and Phoebe Marres. 2021. News production. In A handbook of media and communication research: Qualitative and quantitative methodologies, ed. Klaus Bruhn Jensen, 93–111. London/New York: Routledge. Highfield, Tim. 2016. Social media and everyday politics. Cambridge: Polity. Hjelmslev, Louis. 1963. Prolegomena to a theory of language. Madison: University of Wisconsin Press. 1943. Howard, Philip N. 2015. Pax Technica: How the internet of things may set us free or lock us up. New Haven: Yale University Press. Humphreys, Lee, Veronica Karnowski, and Thilo von Pape. 2018. Smartphones as metamedia: A framework for identifying the niches structuring smartphone use. International Journal of Communication 12: 2793–2809. Innis, Harold A. 1951. The bias of communication. Toronto: University of Toronto Press. ———. 1972. Empire and communications. Toronto: University of Toronto Press. 1950. ITU. 2021. Individuals using the Internet, 2005–2021. https://www.itu.int/en/ITU-D/Statistics/ Pages/stat/default.aspx. Accessed 1 June, 2022. Jacko, Julie A., ed. 2012. The human-computer interaction handbook: Fundamentals, evolving technologies, and emerging applications. 3rd ed. New York: CRC Press. Jensen, Klaus Bruhn. 1991. When is meaning? Communication theory, pragmatism, and mass media reception. In Communication yearbook, ed. James Anderson, 3–32. Newbury Park: Sage. ———. 1993. One person, one computer: The social construction of the personal computer. In The computer as medium, ed. Peter Bøgh Andersen, Berit Holmqvist, and Jens F. Jensen. Cambridge: Cambridge University Press. ———. 2013. How to do things with data: Meta–data, meta–media, and meta–communication. First Monday 18 (10). https://doi.org/10.5210/fm.v18i10.4870. http://journals.uic.edu/ojs/ index.php/fm/article/view/4870 ———. 2021a. The humanistic sources of media and communication research. In A handbook of media and communication research: Qualitative and quantitative methodologies, ed. Klaus Bruhn Jensen, 25–53. London/New York: Routledge. ———. 2021b. A theory of communication and justice. London/New York: Routledge. ———. 2022. Media convergence: The three degrees of network, mass, and interpersonal communication. 2nd ed. London/New York: Routledge.

1192

K. Bruhn Jensen

Jensen, Klaus Bruhn, and Rasmus Helles. 2011. The internet as a cultural forum: Implications for research. New Media & Society 13 (4): 517–533. ———. 2017. Speaking into the system: Social media and many-to-one communication. European Journal of Communication 32 (1): 16–25. ———, eds. 2023. Comparing communication systems: The internets of China, Europe, and the United States. London/New York: Routledge. Jones, Steven G., ed. 1998. Cybersociety 2.0. Thousand Oaks: Sage. Katz, Elihu. 1959. Mass communication research and the study of popular culture: An editorial note on a possible future for this journal. Studies in Public Communication 2: 1–6. Kay, Alan, and Adele Goldberg. 1999. Personal dynamic media. In Computer media and communication: A reader, ed. Paul A. Mayer, 111–119. Oxford: Oxford University Press. Original edition, 1977. Kristeva, Julia. 1984. Revolution in poetic language. New York: Columbia University Press. 1974. Levy, Mark. 1992. Symposium: Virtual reality: A communication perspective. Journal of Communication 42 (4): 4–172. Ling, Rich. 2012. Taken for grantedness: The embedding of mobile communication into society. Cambridge, MA: MIT Press. Ling, Rich, Gerard Goggin, Leopoldina Fortunati, Sun Lim, and Yuling Li, eds. 2020. The Oxford handbook of mobile communication and society. New York: Oxford University Press. Lobkowicz, Nicholas. 1967. Theory and practice: History of a concept from Aristotle to Marx. Notre Dame: University of Notre Dame Press. Lomborg, Stine, and Kirsten Frandsen. 2016. Self-tracking as communication. Information, Communication & Society 19 (7): 1015–1027. Mayer-Schönberger, Viktor, and Kenneth Cukier. 2013. Big Data: A revolution that will transform how we live, work, and think. Boston: Houghton Mifflin Harcourt. McLuhan, Marshall. 1962. The Gutenberg Galaxy. Toronto: University of Toronto Press. ———. 1964. Understanding media: The extensions of man. New York: McGraw-Hill. Merleau-Ponty, Maurice. 1962. Phenomenology of perception. London: Routledge Kegan Paul. 1945. Murdock, Graham. 2013. Communication in common. International Journal of Communication 7: 154–172. Negroponte, Nicholas. 1995. Being digital. New York: Alfred A. Knopf. Nissenbaum, Helen. 2011. A contextual approach to privacy online. Dædalus 140: 32–48. Park, David W., and Jefferson Pooley, eds. 2008. The history of media and communication research: Contested memories. New York: Peter Lang. Peters, John Durham. 1999. Speaking into the air: A history of the idea of communication. Chicago: University of Chicago Press. ———. 2015. The marvelous clouds: Toward a philosophy of elemental media. Chicago: University of Chicago Press. Plantin, Jean-Christophe, Carl Lagoze, Paul N. Edwards, and Christian Sandvig. 2018. Infrastructure studies meet platform studies in the age of Google and Facebook. New Media & Society 20 (1): 293–310. Porat, Marc. 1977. The information economy: Definition and measurement. Washington, DC: Government Printing Office. Rettberg, Jill Walker. 2013. Blogging. 2nd ed. Cambridge: Polity. Rosen, Devan, ed. 2022. The social media debate: Unpacking the social, psychological, and cultural effects of social media. London/New York: Routledge. Ruesch, Jurgen, and Gregory Bateson. 1987. Communication: The social matrix of psychiatry. New York: Norton. 1951. Rule, James B. 2019. Contextual integrity and its discontents: A critique of Helen Nissenbaum’s normative arguments. Policy & Internet 11 (3): 260–279. Scannell, Paddy. 2000. For-anyone-as-someone structures. Media, Culture & Society 22 (1): 5–24. ———. 2012. History, media, and communication. In A handbook of media and communication research: Qualitative and quantitative methodologies, ed. Klaus Bruhn Jensen, 219–234. London/New York: Routledge.

46

Intermediality and Metamediality: From Analog Representations to. . .

1193

Schmidt, Kjeld, and Liam Bannon. 2013. Constructing CSCW: The first quarter century. ComputerSupported Cooperative Work 23 (4–6): 345–372. Scott, Craig, and Laurie Lewis, eds. 2017. The international encyclopedia of organizational communication, 4 vols. Hoboken: Wiley-Blackwell. Searle, John R. 1969. Speech acts. London: Cambridge University Press. Simonson, Peter, Janice Peck, Robert T. Craig, and John P. Jackson, eds. 2013. The handbook of communication history. New York: Routledge. Smythe, Dallas W. 1977. Communications: Blindspot of Western Marxism. Canadian Journal of Political and Social Theory 1 (3): 1–27. Turing, Alan. 1965. On computable numbers, with an application to the Entscheidungs problem. In The undecidable, ed. Martin Davis. New York: Raven Press. Original edition, 1936. Turkle, Sherry. 1984. The second self: Computers and the human spirit. New York: Simon & Schuster. ———. 1995. Life on the screen: Identity in the age of the Internet. New York: Simon & Schuster. van Dijck, José. 2013. The culture of connectivity: A critical history of social media. Oxford: Oxford University Press. van Dijck, José, Thomas Poell, and Martijn de Waal, eds. 2018. The platform society: Public values in a connective world. Oxford: Oxford University Press. Warren, Samuel D., and Louis D. Brandeis. 1890. The right to privacy. Harvard Law Review IV (5). http://www.lawrence.edu/fast/BOARDMAW/Privacy_brand_warr2.html Watzlawick, Paul, Janet H. Beavin, and Don D. Jackson. 1967. Pragmatics of human communication: A study of interactional patterns, pathologies, and paradoxes. New York: Norton. Webster, James G. 2014. The marketplace of attention: How audiences take shape in a digital age. Cambridge, MA: MIT Press. Weinberg, Alvin M. 1961. The impact of large-scale science in the United States. Science 134 (3473): 161–164. Weller, Katrin, Axel Bruns, Jean Burgess, Merja Mahrt, and Cornelius Puschmann, eds. 2014. Twitter and society. New York: Peter Lang. Wittgenstein, Ludwig. 1953. Philosophical investigations. London: Macmillan. Woolley, Samuel C., and Philip N. Howard, eds. 2018. Computational propaganda: Political parties, politicians, and political manipulation on social media. Oxford: Oxford University Press. Zuboff, Shoshana. 2019. The age of surveillance capitalism: The fight for a human future at the new frontier of power. New York: Public Affairs.

The Recommended Experience: Engaging Networked Media Platforms with Intermediality

47

Maja Bak Herrie

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermediality and Multimodality and Their Relevance for Engaging Networked Media Experiences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Medial Transposition, Medial Compositions, and Medial Reference . . . . . . . . . . . . . . . . The Medial Modalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Two Illustrative Cases: Spotify and Kindle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Spotify’s Musical Advice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kindle’s Popular Highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1196 1200 1200 1202 1205 1205 1207 1209 1210

Abstract

Recommendation algorithms play an important role in our reception of digital media products. Whether we listen, watch, or read online, the content we receive is often pre-filtered based on calculated relevance or on the similar taste of others. As such, our reading of a thriller or watching of the newest romantic comedy contributes to a network of previous experiences supplying media producers with valuable data for an even more optimized future product. Engaging the aesthetics of networked media, this trend is important to consider, as it plays a vital role in the communication situation. From the spatiotemporal and sensorial affordances of Spotify’s musical advice to the complex semiotic modalities of the Popular Highlights in the Kindle interface, the recommended experience of online media content calls for an intermedial vocabulary to be adequately described and understood. This chapter offers a general discussion of the concepts of intermediality and multimodality focusing on their relevance for engaging with networked media experience. The chapter is divided into two parts: the first part M. Bak Herrie (*) Postdoc at School of Communication and Culture at Aarhus University, Aarhus, Denmark e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_59

1195

1196

M. Bak Herrie

presents a broader introduction to existing intermedial scholarship as it relates to networked media, while the second part exemplifies ways to approach this topic through two illustrative analyses of the interfaces of Spotify and Kindle. Keywords

Recommendation · Networked media · Aesthetics · Intermediality · Multimodality

Introduction It is an early morning. The coffee machine is hissing away, while raindrops drum on the windows. As you squint at your phone and rub your eyes, Spotify recommends a Soulful Morning Playlist as the backdrop of your morning soundscape, starting out with Gladys Knight and the Pips’ 1973 hit “Midnight Train to Georgia.” 70,327 other listeners liked this experience. Maybe you will as well. Recommendation algorithms play an important role in our reception of digital media products. Whether we listen, watch, or read online, the content we receive is often pre-filtered based on calculated relevance or on the similar taste of others (Prey 2018; Seaver 2015). As such, our reading of a thriller or watching of the newest romantic comedy contributes to a network of previous experiences supplying media producers with valuable data for an even more “optimized” future product in terms of competing for consumer attention (Citton 2017; Rossiter and Zehle 2015). Engaging the aesthetic experience of networked media products – i.e., content provided by media and streaming services such as Netflix, Disney+, HBO Max, Hulu, YouTube, Tidal, and Spotify – or e-reading platforms such as Amazon Kindle, Kobo, and Nook, this trend is important to consider, as it plays a vital role in the reception. Rather than simply listening to a song or reading a poem, we encounter networked media products on highly contextualized platforms1 with automatically generated playlists, continuously updated recommendation feeds, ubiquitous comment sections, and annotations showing signs of previous receptions. This chapter focuses on the aesthetic consequences of this contextualization of networked media experience. It provides an overview of some of the key theoretical concepts and distinctions to understand and adequately describe the recommended experience of today. To do so, valuable resources from the fields of intermediality and 1

In choosing the word platform, attention is brought to some of the broader changes linked to the new mass-cultural industries providing platforms for cultural consumption, social interaction, advanced advertisement, and much more. As Christian Ulrik Andersen and Søren Pold convincingly argue in their 2018 book The Metainterface: The Art of Platforms, Cities, and Clouds, the omnipresence, invisibility, and algorithmic power of platforms represent a marked change in our environment and culture. While the predefined “sameness” of a mass-cultural industry may seem different than the highly personalized interfaces of today, users are nonetheless enrolled in the modes of the production of what Andersen and Pold call the metainterface industry (Andersen and Pold 2018, 76).

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1197

multimodality2 is introduced and operationalized in relation to the new territories of networked media platforms. While it is nothing new that artists and producers use various strategic mechanisms to contextualize works and generate interest, the multimodal platforms surrounding networked media products shape and guide the aesthetic experience to a previously unseen degree. According to literary theorist Gérard Genette, what we usually understand as the main text (the story, the narrative) is often infolded in other paratextual materials that shape our experience (as, e.g., cover art, front matter, foreword, back matter, colophon, and many other materials not crafted by the author) (Genette 1997). In the context of networked media, one could in principle add any media product to this distinction, arguing, e.g., for a similar interpretation of music – where the “main text” would be the musical piece itself, while other materials would include the album cover or the latest review in Pitchfork – or of TV series where a similar distinction between the images and narrative of the series itself and the countless visual and textual contextualizations of it would be central.3 Engaging content provided by media and streaming services, such paratextual materials are numerous and ubiquitous. Inhabiting a “zone between text and off-text, a zone not only of transition but also of transaction,” the paratexts of networked media products operate in a similar way as the literary paratexts introduced by Genette in the 1980s (Genette, 2). They are designed to persuade their recipients to keep listening, watching, and reading, and in doing so, they can aptly be said to represent a “privileged place of pragmatics and a strategy, of an influence on the public” (2). While the concept of paratexts provides a tentative explanation of the persuasive nature of current platforms surrounding networked media products, several key factors on both the production and reception side require further development. One of the main changes is that the recommended experience received on these new platforms expands beyond traditional notions of communication as a two-way transaction, between, e.g., an addresser and an addressee (Jakobson 1960) or a communicator and a receiver (Schramm 1971).4 While there is a clear sender in the form of the platform owner, the content provided by these platforms is dynamically adapted to a receiver defined in terms of data models of taste and preference rather than a traditionally delineated market segment (Konstan and Riedl 2012).

The term “multimodality” is perhaps best known in relation to its educational and socio-semiotic potentials; see, e.g., Kress (2010) and Jewitt and Henriksen (2016)). In this chapter, however, the conception of multimodality is inspired by Lars Elleström (Elleström 2020, 8–9) and an approach more closely tied to aesthetics and to the multisensorial dimensions of, e.g., reading and listening; see Pedersen et al. (2021). 3 Earlier efforts to translate Genette’s classical conception of the paratext into other media forms are plentiful; see, e.g., Kümmerling-Meibauer (2013) and Sutton (2015). 4 (For a discussion of the problems with the crude nature of these traditional notions, see Elleström (2020, 9–14). 2

1198

M. Bak Herrie

These models are collected into personalized5 and derived6 profiles to which consumers are matched to successfully recommend content. As such, (at least) four important features of the recommended experience of networked media products should be considered in analytical endeavors: 1. The recommended experience is presented as a uniquely personalized product that becomes more accurate to your taste as you continue to interact with the algorithms of the system. As such, Spotify presents playlists with recommendations “made for you” that “match your mood,” and collections based on your previous listenings that are “uniquely yours.”7 2. At the same time, the recommended experience is built on aggregates8 of previous experiences of other recipients. While uniquely personalized content is the bread and butter of networked media platforms, the content recommended to any one particular receiver is typically based on their similarity to other previous receivers who have engaged with that content. Its input data being the likes, views, clicks, shares, and other digital interactions of others, the recommended experience is based on a collaborative formation of taste. As such, the Kindle interface not only suggests the next book match for its reader based on previous interactions but features the “remnants” of these very interactions via the so-called “Popular Highlights” function. 3. In the recommended experience, the consumer is encouraged to engage with the paratext by generating additional metadata through digital interactions. Following the terminology of new media scholar Wendy Hui Kyong Chun, this social dimension can be understood as a “promiscuous exchange of information” (Chun 2016, 12), where “once silent and private acts – such as reading a book – [are turned] into noiselessly noisy ones, eroding the difference between reading, writing, and being written” (94). Rather than just framing and contextualizing the media product, the paratext is interwoven in and in some cases becomes inseparable from the content itself. This changes not just the content, but also the aesthetic experience in anticipation of this engagement. This mechanism is particularly prevalent in YouTube videos where “calls to action” (CTAs) such as requests for liking, subscribing, and sharing videos are not just featured in but

(In the first chapter of her seminal book (co-edited with Serge Gutwirth) on the societal impacts of data-driven profiling, Mireille Hildebrandt defines personalized profiling as the process of “discovering correlations between data in databases that can be used to identify and represent a human or nonhuman subject (individual or group) and/or the application of profiles (sets of correlated data) to individuate and represent a subject or to identify a subject as a member of a group or category” (Hildebrandt and Gutwirth 2008, 19). 6 That these profiles are derived from empirical data is central to studying them and marks an important shift in perspective from traditional market segmentation to data-driven consumer profiling. For an in-depth discussion of notions of data derivatives, see, e.g., Amoore (2011). 7 Spotify’s home screen as of January 10, 2022. 8 For a discussion of the term aggregation, see, e.g., Lehmann (2021). 5

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1199

also significantly shape the content. (see, e.g., Fraitag 2020).9 On other platforms, similar “calls” appear as, e.g., Spotify’s promise that you “get better recommendations, the more you listen.” 4. Finally, the recommended experience is characterized by a dynamic and continuous flow of content mingling in auto-generated mixes and playlists targeting specific data models of taste. Numerous examples of this can be found, but Spotify’s “Discover Weekly” and “Daily Mix”10 and YouTube’s profiled preference categories11 are noticeable ones. This mixing of music tracks, video clips, and texts not just affects the content – e.g., when a pop hit is separated from its original affiliation with an album and instead is promoted in a group of similar, danceable hits – but also the aesthetic experience.12 The platforms encourage a particular way of listening, watching, or reading via their strategic, multimodal compositions. Plenty of thorough and convincing analyses of such issues and their relation to algorithmic transformations of cultural content and its consumption are available: on political issues, see, e.g., Martin Scherzinger’s work on the economics and politics of streaming (Scherzinger 2019) or Taina Bucher’s writings on algorithmic power (Bucher 2018); on the more technical specifics, see, e.g., Tarleton Gillespie’s comprehensive analyses of algorithms (Gillespie 2014, 2018) or the work of Nick Seaver and K. E. Goldschmitt on recommendation (Goldschmitt and Seaver 2019; Seaver 2015); and on broader social and cultural issues, see, e.g., the work of David Beer, John Cheney-Lippold, Nick Couldry, Aristea Fotopoulou, Luke Dickens, and Ted Striphas (Beer 2017; Cheney-Lippold 2017; Couldry et al. 2016; Striphas 2015). While these perspectives are valuable for informing interdisciplinary research in networked media platforms, aesthetic studies of this new algorithmic condition (Colman et al. 2018) are needed as well. From the spatiotemporal and sensorial affordances of Spotify’s musical advice to the complex semiotic modalities of the “Popular Highlights” in the Kindle interface, the recommended experience of online media content calls for a vocabulary that takes aesthetic aspects seriously. Particularly relevant ideas for analytically approaching and understanding this landscape 9

CTA, or call to action, is a marketing term that covers any design to prompt an immediate response from a user (such as liking, subscribing, or sharing) or encourage an immediate sale. While some uses of CTAs are simply strategic add-ons to promote extra sales, it can be argued that creative uses of CTAs in, e.g., YouTube videos have turned into a genre itself. CTAs are used strategically on a range of social media platforms, e.g., YouTube, Facebook, Instagram, and Pinterest. See, e.g., Fraitag (2020). 10 Spotify’s home screen as of January 10, 2022. 11 YouTube’s home screen as of January 10, 2022. 12 Notably, this discussion was displayed in the Twitter debate between English singer and songwriter Adele and Spotify on November 21, 2021, about the shuffle button on album pages at the Spotify interface. Adele criticized the button, objecting that artists “don’t create albums with so much care and thought into [. . .] track listing for no reason.” “Our art tells a story and our stories should be listened to as we intended,” she ends. Responding directly to Adele’s tweet, Spotify decided to remove the shuffle button from album pages. See, e.g., Vinter (2021).

1200

M. Bak Herrie

are Irina Rajewsky’s medial transpositions, compositions, and references and Lars Elleström’s conceptions of medial modalities. The following pages introduce these concepts, focusing on their relevance for engaging with networked media experience. The chapter is divided into two parts: the first part presents a broader introduction to existing intermedial scholarship as it relates to networked media, while the second part exemplifies ways to approach this topic through two illustrative analyses of the Spotify and Kindle interfaces.

Intermediality and Multimodality and Their Relevance for Engaging Networked Media Experiences The Medial Transposition, Medial Compositions, and Medial Reference In her influential 2005 article “Intermediality, Intertextuality, and Remediation,” literary scholar Irina Rajewsky divides the conception of intermediality into two senses: a broad one that covers all phenomena that in some way take place between media and a narrower one covering only specific subcategories of intermediality. Choosing the latter as her point of departure, she concentrates on three concrete medial configurations and their specific intermedial qualities. Her tripart terminology thus consists of medial transposition, medial combination, and medial reference. The first of the three subcategories, medial transposition, has to do with the way in which a media product comes into being via a process of transformation from one medium (e.g., a book) into another (e.g., a film). Rajewsky describes this transposition as “genetic,” as it involves an “original” source, which is transformed into a new media product (Rajewsky 2005, 51). Her key examples are film adaptations and novelizations, yet other examples closer to the context of networked media could include the medial adaptation of a paper book into an e-format. Here, the source would be the paper book with its medium-specific13 characteristics and the transformed “new medium” would be the e-format with its specific digital affordances. Here, in the first of the three subcategories, focus is on the medial transformation process itself, i.e., on what Rajewsky calls the “production-oriented” conception of intermediality. Yet, it is important to note that a single medial configuration may fulfill the criteria of two or even all three of the subcategories (53), making the classification of an e-book, for instance, a question of more than just its transposition from analogue paper to digital screen. The e-book can also make use of, e.g., strong visual or auditive components in its communication, pointing to the medial combinations it is composed of. Under this subcategory, Rajewsky lists multimedial phenomena as “opera, film, theatre, performances, illuminated manuscripts, computer- or sound art installations, and comics” – in short, any “medial constellation “Medium specificity” here refers its particular technical features and affordances of a medium (and its associated context of use and/or production).

13

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1201

constituting a given media product,” which is to say the very process of combining “at least two conventionally distinct media or medial forms of articulation” (52). Crucially, these two media are “each present in their own materiality and contribute to the constitution of signification of the entire product,” Rajewsky states. Turning again to the e-book example, we are not so much concerned, here, with the development of electronic literature, i.e., with general questions of media history, but with the coming together of different medial forms and the aesthetic effects this new constellation creates: what is it like to “turn” the pages of a digital book? How has the communicative-semiotic situation of reading changed? Rajewsky points to the “fundamental interrelatedness of earlier and newer media” (63) in her conception of new media forms, yet advocates for situating the reading in concrete, medial configurations (51). The last category of Rajewsky’s tripartite division is the medial reference. Under this category, one finds medial imitations: different strategies in which a media product evokes certain techniques of another medium by using its own mediaspecific means, either to refer to a specific, individual work produced in another medium or to refer to a specific medial subsystem. Rajewsky’s example is literature that imitates filmic techniques as zoom shots, fades, dissolves, and montage editing (52). An example closer to the context of networked media could be the so-called Popular Highlights at the Kindle interface. Appearing as underlined text in your book, Popular Highlights are passages that have been marked by more than ten people. The underlined text comes with a little note saying that there are, e.g., “415 Highlighters,” indicating that this particular passage is indeed a “popular” one. As such, the Kindle interface not only suggests what book to read next based on previous interactions, but also features the “remnants” of these very interactions as new kinds of digitized marginalia or dog-ears.14 Understood in relation to Rajewsky’s medial references, the Kindle interface imitates reading and noting techniques established in relation to the paper book: it borrows well-known practices of private marking and memorizing yet translates them into its own digital form. “Rather than combining different medial forms of articulation,” Rajewsky writes, “the given media-product thematizes, evokes, or imitates elements or structures of another, conventionally distinct medium through the use of its own media-specific means” (53). The media-specific means, here, is defined by the data-driven platform with its uniquely personalized, yet inherently “noisy” offers (Chun, 94). While it feels like a private act to read a book (as it indeed once was15 the data gathered about such readings are now incorporated in a new reading experience. “Never has so 14

For an in-depth discussion of Amazon Kindle’s Popular Highlights, including their status as remediated marginalia, see Cameron (2012). 15 As the American media scholar Jim Collins already proposed in 2010, an increasingly digitized literary culture is now gaining ground: “What used to be a thoroughly private experience in which readers engaged in intimate conversation with an author between the pages of a book has become an exuberantly social activity” (Collins 2010, 4). One must remember, however, that this private and silent reading experience is indeed a modern phenomenon, and that public reading practices are anything but new.

1202

M. Bak Herrie

much cultural content been so readily available, on so many cultural platforms,” media scholars Søren Pold and Christian Ulrik Andersen write, yet “the amazing efficiency of the new metainterface industry comes at the expense of monitoring, control, and strict licensing” (Pold and Andersen, 76). This monitoring is not always (completely) hidden under the seamless interface, however, but is often integrated in the experience itself: as an imitation or evocation of another medium, or as what Rajewsky calls an “illusion-forming quality” of a medium acting “as if” it had the techniques of, e.g., the paper book at its disposal (Rajewsky, 55).

The Medial Modalities In 2010, Lars Elleström published the first version of his influential article “The Modalities of Media: A Model for Understanding Intermedial Relations,” in which he presented a systematic description of media and their modal composition (Elleström 2010). The overall argument is that all media are both similar and different and that they cannot be compared without clarifying which aspects are relevant to the comparison and how these aspects can relate to each other (Elleström 2020, 43). Focusing on these medial aspects – the medial modalities – Elleström opens a fruitful field of possibilities for discussing both medium specificity and intermediality. In 2020, a modified and expanded version of the original article was published, pushing the work even further in the direction of medial integration and media transformation. In the context of networked media, the expanded discussions of virtual representations of materialities, sensory perceptions, time, and space are especially helpful. For this chapter, the expanded 2020 model is used. Central in Elleström’s model are the four medial modalities which constitute the fundamental architecture. He calls them the material, the spatiotemporal, the sensorial, and the semiotic modality and refers to them as “basic” in the sense that all media products have traits belonging to all four modalities (46). The material modality refers to the physical, perceptible nature of the media product, e.g., the “solid road sign made of painted metal, liquid water used in an art installation, gas in the form of vibrating air (sound waves) produced by vocal cords and plasma in a television screen or other device” (48). The spatiotemporal modality points to the four-dimensional mode of every media product, i.e., its extension in width, height, depth, and time. Thus, temporality is an aspect of songs, speeches, gestures, and dance, but not of stills and most sculptures. In a similar way, width and height represent central aspects of photography, whereas a sculpture has three spatial dimensions. A dance has four dimensions – it begins, extends, and ends in time – while the time of a photograph seems absent as soon as it is developed. The sensorial modality refers to ways in which a media product affects the senses. As media products are always perceived (Dufrenne 1973, 218), the five external sense faculties of humans are central to this modality: seeing, hearing, feeling, tasting, and smelling influence the ways in which a media product is grasped. Finally, the semiotic modality points to the representational aspects of media communication. While this modality may seem less palpable than the three others, it is equally important

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1203

for realizing communication: for communication to happen, interpreting minds – perceiving and conceiving subjects situated in social circumstances – are needed, Elleström states (Elleström 2020, 50). Following the American philosopher and logician Charles Sanders Peirce, meaning can be described as the result of sign functions, i.e., the ways in which something comes to mean something to someone in a way that is either iconic (because it is similar to something), indexical (because it is connected to something), or symbolic (because it is related to a certain habit or convention) (Peirce 1994 [c.1897], 2.228). To Elleström, iconicity, indexicality, and symbolicity are three main media modes, without which no communication could occur. Thus, he suggests annexing them into the media model by referring to them as depiction (iconicity), deiction (indexicality), and description (symbolicity) (Elleström 2020, 51). Depiction, deiction and description are not mutually exclusive: as modes of the other modalities, they are often combined to create multimodal media, i.e., media that are both visual and auditory, spatial and temporal, iconic and indexical, and so forth. As they mix with each other, they must be read in connection, yet often one of them will dominate the relation. In, e.g., written texts, the symbolic sign functions of the letters and words will dominate the signification process, whereas for, e.g., instrumental music, it will be iconic signs that generally dominate. Whereas the three first modalities are presemiotic in the way they physically mediate communication, the fourth semiotic modality points to the representation met by a receiving mind. Both types of modalities are equally important to Elleström, who stresses the significance of both the physical mediation and the mind-work of communication. Yet some media products are closely tied to the appearance of the medium, whereas others are more a result of interpretation. With regard to networked media products and the platforms hosting them, both types of modalities relentlessly interrelate. With a platform such as Spotify, the listening experience has become markedly different from the pre-streaming era due to material, spatiotemporal, and sensorial changes in the mediation: – Materially, music is no longer encountered at physical or digital storage media bought or borrowed in a shop or at a library but is featured in applications on mobile devices and at computer screens. – The spatiotemporal manifestation of the single track is transformed by automatically generated playlists and continuously updated recommendation feeds. We seldom encounter a pop hit in its album context, but rather unbundled and reassembled in mixes and playlists (Hagen 2015; Prey 2020) that catch attention and stimulate taste. – Sensorially, recorded music is of course still primarily auditive, yet the relation to other sense faculties has changed: the paratext of the CD or LP featuring, e.g., artist information, lyrics, and cover art, has been replaced by highly contextualized and commodified interfaces presenting a myriad of multimodal inputs – band information is updated live with tour dates and merchandise offers, cover art is collected and presented for all releases, and the previous experiences of other consumers, including their activities and preferences, are visualized in different ways. This persuasive yet visually dense interface is an important layer of communication in the context of music streaming.

1204

M. Bak Herrie

The multimodal mediation of streamed music thus plays an integral role in semiotic questions as well: given these transformations of the more “physical” communication, the receiver’s interpretation changes too. With the introduction of new modes of data collection and visualization to the interface, a more quantitative decoding of the signification process is encouraged: a progress bar shows the length of the track, a sidebar tells on the activity of friends and followed profiles, and a running counter registers and displays every “play” and “like.”16 In Elleström’s terminology, music – and instrumental music in particular – is traditionally dominated by the iconic sign function (depiction).17 Yet with a central progress bar, for instance, this depiction also becomes inherently visual: the duration of a musical piece is not just visually demarcated with a timestamp, and the recipient’s power to pause or “jump” between internal sequences in the track is made clear and easy. Not only is the track visually depicted, but it is also symbolically described as part of a highly textual (and informational) steaming context: are your friends liking this too? How many times was this track played? Does it fit any of your playlists? Unlike playing a CD or an LP, experiencing music through a streaming platform is necessarily entangled with an online social context that extends beyond the confines of the home. Recalling the terminology of Chun, this social dimension has eroded the difference between “reading, writing, and being written” (Chun, 94), and one could add listening, speaking, and being listened to18 rather than just framing the media product, the multimodal mediation of the musical “content” is interwoven in and in some cases becomes inseparable from the content itself. As such, the deictic signification can be seen as a surprisingly dominating function within this platformized19 context with the indexical marks and traces – the likes, views, clicks, and shares – from previous listenings becoming a main currency of the platform owners.20

16

Spotify’s home screen as of January 10, 2022. For an in-depth discussion of the varying levels of iconicity related to sound, see Santaella (2015). 18 Most recently, Spotify filed a patent to use artificial intelligence for emotional surveillance and manipulation. The patent concern, e.g., listening to users’ conversations, analyzing the sound of their voice, and curating targeted ads and music for their emotional state. The technology claims to identify “emotional state, gender, age, or accent” in forming its recommendations (Matta 2021; Yoo 2021; Savage 2021). 19 Recently, scholars have warned about critical influence platforms have over the industries they enter, and the commodities they exchange; see, e.g., Langley and Leyshon (2017) and Srnicek (2019). In relation to the cultural industries, media scholars David B. Nieborg and Thomas Poell have defined the concept of platformization as “the penetration of economic, governmental and infrastructural extensions of digital platforms into the web and app ecosystems” (Nieborg and Poell 2018, 2). This is a process that is seen to fundamentally affect, and in some cases even compromise, cultural producers and the content they create (Prey 2020). 20 Music streaming services such as Spotify generate revenue from advertising. To do so, they need to precisely target ads at listeners to increase ad rates by use of competing techniques to “see” the individual listener. As media scholar Robert Prey writes: “Every interaction a listener has with a musical item – including the listener’s music tastes (selected artists and songs) and music behavior (favorites, ratings, skips, and bans) – is captured and recorded in real-time. The Taste Profile 17

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1205

A similar situation is at play for other networked media products and platforms, i.e., media and streaming services such as Netflix, Disney+, HBO Max, Hulu, and YouTube, or e-reading platforms such as Amazon Kindle, Kobo, and Nook, where complex material, spatiotemporal, sensorial, and semiotic modes call for multimodal analyses and interpretations to be described and understood. Part II of this chapter illustrates what such an intermedial and multimodal analysis and interpretation can look like.

Two Illustrative Cases: Spotify and Kindle Spotify’s Musical Advice Founded in 2006 by Daniel Ek and Martin Lorentzon, Swedish Spotify was one of the first audio streaming and media service providers and today has the biggest share of the market with 381 million monthly active users.21 As a platform it offers music and podcasts22 and currently has more than 70 million songs in its database. Choosing the so-called freemium version, customers have access to basic features for free with advertisements and limited control, while additional features, such as offline listening and commercial-free listening, are offered via paid subscriptions. Such a contextualized and commodified listening environment is interesting in the perspective of intermediality and multimodality for several reasons. Spotify’s recommended experience is designed to capture the fleeting attention of its users. To this end, the platform presents a potentially unending flow of content from different sources expertly knitted together to match the users’ individual tastes, inviting them to keep listening. As talented players in the attention economy, Spotify’s developers build more than just platform functionality: they design the recommended music experience, an experience that relies on a deep understanding of intermedial and multimodal affordances. In an intermedial perspective, their strategic employment of users’ expectations and trained use of both visual and auditive media is central. Recalling the vocabulary of Rajewsky, it is relevant to address the interrelatedness of earlier and newer media when analyzing the Spotify interface: while music streaming services are (relatively) new, they to a large degree act their part by thematizing, evoking, and imitating [Spotify’s preference analytics and visualization tool] is thus a dynamic record of one’s musical identity and ‘the foundation of personalization at Spotify’ according to Ajay Kalia, who oversees the project at the company” (Prey 2018, 1091). 21 As of January 2022, see Spotify (2021). 22 These years, Spotify is gaining power over podcast distribution by either buying production directly or striking exclusive deals with content providers, as, e.g., Joe Rogan in 2020; see Steele (2020) and Stoller (2020). With platforms as Spotify, traditional circuits of production and publishing have been replaced by more complex circuits, introducing new actors and paths. For an in-depth discussion of these new digital production and publishing circuits, see Have and Pedersen (2020).

1206

M. Bak Herrie

elements of other, conventionally distinct media using their own media-specific means. Like reading a book, listening to music within the confines of the home was up until recently a private affair. As such, the Spotify interface may at first seem rather individualized in its setup with its personal navigation functions (imitating the navigation bottoms known from personally operated reel-to-reel tape machines) and home screen providing personalized recommendations. While these uniquely personalized elements may be the bread and butter of a platform such as Spotify – and, accordingly, a highly prioritized part of its design23 – the algorithmic recommendation behind is typically based on the similarity to other previous receivers’ engagements with the content. Its input data being the likes, clicks, shares, and other digital interactions of recipients, the recommended experience is based on a collaborative formation of taste. As such, the interface designers could as well have hidden these social (and, to some observers, problematic24 dimensions of the listening experience, focusing on the private nature of music consumption. Yet, the monitoring is not always hidden under the seamless interface, but is, interestingly, often integrated in the experience itself. What are your friends listening to? What do they like? What playlists are trending, and do they feature the artists you are listening to yourself? Mimicking the social aspects of musical culture by integrating, for example, the activities of friends, Spotify draws on our expectations of other, older media forms in the broadest sense of the term. A streaming platform such as Spotify is not just a digitized tape machine: it is also a highly personalized, yet socially and collaboratively organized record store. It is a radio broadcasting station playing your favorite music day and night, and it is an early 2000s schoolyard filled with cool kids crowding together around an iPod holding the latest and hottest rap hits. Spotify does not provide its users with direct medial transpositions from one existing medium to another, nor does it simply combine medial forms in new constellations comprised of clearly identifiable “older” media. Yet it builds on an interrelatedness or referentiality to more wellknown media forms. Not just to their technical functionality (mimicking, e.g., the navigation of a tape machine), but also to the expectations and sociality built into the use of these older media forms. In a multimodal perspective, this important referentiality can be analyzed in more detail. Considering the spatiotemporal and sensorial encounters with the platform, e.g., it is relevant to look at the persuasive yet visually dense paratext mediating the tracks. Just like in the record store, you are faced with an attractive flow of multimodal content: beautiful cover art, band information, and – as an addition in the digital marketplace – tempting playlists promising to set the mood for

23

See, e.g., Robert Prey’s informative analysis of the individuation processes taking place on music streaming platforms (Prey 2018). 24 See, e.g., Taina Bucher’s analyses of the specific uses of algorithms and the social and cultural forms they generate in her 2018 book If. . . Then: Algorithmic Power and Politics (2018) or David Beer’s reflections on how we might approach algorithms from a social scientific perspective (Beer 2017).

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1207

“Tranquility,” “Creative Focus,” or a “Quiet Moment.”25 As in the record store, you not only listen to the music in your individual listening booth, but you also see others listen (and, in many cases, also what they are listening to), knowing that they in turn see you. The sensorial environment surrounding musical consumption, in this perspective, is far from private. In a semiotic perspective, this social aspect is reinforced by various strategic design choices made by the platform designers: new modes of data collection and visualization have been introduced to the listening experience, making it a more quantitative one. A sidebar constantly tells on the activity of friends, and a running counter registers and displays every “like” and “play.” Unlike listening to a CD or an LP, experiencing music through a streaming service like Spotify is necessarily connected to an online social context. As such, the deictic signification described by Elleström has become surprisingly dominant within this context. Indexical marks and metadata traces from previous listenings not only create value for the platform owners, but also play into the aesthetic experience of the users, because they are integrated in the mediation and signification processes. Rather than just framing the musical track, the social context of listening is interwoven in and in some cases becomes inseparable from the musical experience itself.

Kindle’s Popular Highlights A similar social tendency can be found on the e-reading platform Amazon Kindle. Hosting their services digitally, networked media platforms like Amazon harvest large amounts of metadata about the habits and preferences of their customers – metadata that feeds into advanced data analyses providing recommendations for the next purchase. While Kindle offers a wide range of devices that enable users to browse, buy, and read e-books, newspapers, magazines, and other digital content via the online Kindle Store, it is especially the so-called Popular Highlights function that is interesting in the context of this chapter. As explained earlier, Popular Highlights are underlined text in your e-book that appears when more than ten people have marked a particular passage. The underlined text thus refers directly to the experiences of previous readers and, hereby, indexes the platforms’ medium specificity as a digital host for literary products and their consumption. Not only does Amazon provide its users with digitized books and other digital content through their devices, it also provides them with a new reading environment and, through this, new ways of reading and experiencing literature. As such, the highlights can be seen as emblematic for this chapter’s discussion of the social aspects of the recommended experience. As mentioned earlier, Rajewsky’s conception of the medial references is particularly relevant to discuss this phenomenon. Imitating the paper book by “referring” to the spatiotemporality of texts and their ability to be read and actualized repeatedly by different readers, Popular Highlights directly visualize the “remnants” of previous interactions with a particular book. The result of this visualization is a sort of digital

25

Spotify’s home screen as of January 10, 2022.

1208

M. Bak Herrie

marginalia or dog-ear, reminding the reader that previous receivers found this passage interesting enough to highlight it digitally. In this way it borrows wellknown practices of private marking and memorizing but converts them into its own digital form, acting as if it had the techniques of, e.g., the paper book at its disposal (Rajewsky, 55). Rajewsky’s conception of intermediality in the narrow sense of the medial reference helps identifying the interrelatedness of the paper book and the e-book and hypothesizing about our expectations of different media forms and their technical qualities. Yet the point that intermedial perspectives can work as a fruitful prism to understand more general cultural and aesthetic issues related to datafication and digitization can be pushed even further with Elleström’s idea of medial modalities. In an analytical perspective, it is of course important to stress that Kindle – as all other media products – encompasses all four basic medial modalities (Elleström 2020, 46). Yet when it comes to the Popular Highlights, particularly the spatiotemporal and semiotic modalities are relevant. Both pointing to the transcendence of the singular book and reading experience, the spatiotemporal and semiotic modalities can help illuminate how the singular, seemingly private, reading experience feeds into and potentially transforms future readings. Even if a reading situation feels private, it is indeed “noisy,” as Chun phrases it. Spatiotemporally, the Popular Highlights function is interesting because they exceed the particularity of the individual reading. If you had borrowed a book from the local library, you would not be puzzled by minor signs of use such as a dog-ear or some doodles in the margins, but when buy your book from an e-book platform such as Amazon, you do not just expect the product to be new – unused and untouched – but also that it will remain so due to its digital nature. This spatial relation to what Rajewsky calls the medial “source” (i.e., paper book with its material advantages and disadvantages), however, is not maintained with the Popular Highlights function resulting in widespread criticism from customers asserting that their freshly purchased book has been violated by emphases unendorsed by the author or publisher.26 Also the temporal aspects of this intermedial relation are interesting. As a continuous, dynamically updated function, Popular Highlights are designed to integrate the individual reading into a network of readings, directly marking and changing future readings. Such a dynamic operationality is ubiquitous in the recommended experience on networked media platforms. Just as Spotify’s paratextual framing of its content constantly contributes to the social context of the platform, your reading of the latest thriller from the Kindle Store also feeds into experiences yet to come. In this way, datafication radically changes the ways in which we consume media products. Not only has the content itself – the latest comedy or the danceable pop hit – changed spatiotemporally with pervasive digitization and datafication processes, but the For instance, Virginia Heffernan writes: “Stumbling on a passage that other people care about, framed as though you should care about it too, can seem like a violation of virgin text. It’s bad enough that vandals have gotten to your ‘new’ edition before you have and added emphases unendorsed by author or publisher. What’s worse is that they invariably choose the most Polonius-like passages” (see Heffernan 2010). 26

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1209

platforms have also been altered, in turn changing the paratextual framing of cultural products such as music, film, and literature and so our aesthetic encounters with them. In a semiotic perspective, Popular Highlights emblematically point to a central development in and around networked media, namely, that of the datafication of the signification processes themselves. Just as new modes of data collection and visualization have been introduced to Spotify, quantifying the listening environment and socializing the previously private aesthetic experience of facing recorded music, datafication has also changed how we meet, buy, read, and share literature. Whereas Spotify simply tells what your friends are listening to in a sidebar next to your navigation panel, Kindle takes a step further by slipping traces of previous consumptions into the very pages of your book and intercepting every marking you make yourself. From a semiotic perspective, the highlighting of selected passages points to their inherent importance to other readers: this word or these sentences are something special and should be noticed. Unlike the pages you have just read, this page contains something of real importance. However indirectly, readings and readers “collaborate” in ways that lie beyond the intentions of authors and publishers. As such, the highlighted passages become some sort of deictic indication of preference without any context or explanation situating the purpose of or meaning behind the highlight: whereas Spotify informs you about the behavior of friends, Kindle features the tastes of strangers, and whereas Spotify brings in the social aspects of listening as a paratextual framing displayed next to the musical piece, Kindle integrates it into the media product itself. In this way, Kindle represents an even more radical version of datafied and quantified cultural consumption than Spotify, interweaving indexical marks and traces of others into the very experience itself.

Conclusion This chapter has discussed the role played by recommendation in our aesthetic reception of digital media productions. It has provided an overview of some of the key theoretical concepts and distinctions to understand and adequately describe the recommended experiences of today. To do so, it has drawn on resources from fields of intermediality (via Irina Rajewsky’s conception of intermediality in the narrow sense of medial transposition, medial combination, and medial reference) and multimodality (via Lars Elleström’s four medial modalities) and operationalized them in relation to the new territories of networked media platforms. While aesthetic encounters with networked media platforms give rise to many questions and discussions, particularly four important features of the recommended experience of networked media products have been considered in the chapter: (1) the recommended experience is presented as a uniquely personalized product that becomes more accurate to your taste as you continue to interact with the system, yet (2) concurrently, it is built on aggregates of previous experiences of other recipients. Its input data being the likes, views, clicks, shares, and other digital interactions of others, the recommended experience is based on what has been termed here a

1210

M. Bak Herrie

“collaborative” formation of taste. As such, it is (3) deeply connected to the contextualized and commodified platforms that technically host and stream the content, but also – more importantly – aesthetically and culturally frame and transform it. Finally, (4) the recommended experience is characterized by a dynamic and continuous flow of content met in, e.g., auto-generated mixes or playlists targeting specific data models of taste. As such, the platforms treated in this chapter can be said to encourage particular ways of listening, watching, or reading via their strategic intermedial and multimodal compositions and affordances. Engaging the aesthetics of networked media, these features are important to consider, as they play vital roles in the communication situation. From the spatiotemporal and sensorial affordances of the Spotify platform’s musical advice to the complex semiotic modalities of the Popular Highlights in the Kindle interface, the recommended experience of online media content calls for an intermedial vocabulary to be adequately described and understood.

References Amoore, Louise. 2011. Data Derivatives: On the Emergence of a Security Risk Calculus for Our Times. Theory, Culture & Society 28 (6): 24–43. Andersen, Christian Ulrik, and Søren Bro Pold. 2018. The Metainterface: The Art of Platforms, Cities, and Clouds. Cambridge: The MIT Press. Beer, David. 2017. The Social Power of Algorithms. Information, Communication & Society 20 (1): 1–13. Bucher, Taina. 2018. If...Then: Algorithmic Power and Politics. New York: Oxford University Press. Cameron, Lauren. 2012. Marginalia and Community in the Age of the Kindle: Popular Highlights in ‘The Adventures of Sherlock Holmes’. Victorian Review 38 (2): 81–99. Cheney-Lippold, John. 2017. We Are Data: Algorithms and the Making of our Digital Selves. New York: NYU Press. Chun, Wendy Hui Kyong. 2016. Updating to Remain the Same: Habitual New Media. Cambridge: The MIT Press. Citton, Yves. 2017. The Ecology of Attention. Trans. B. Norman. Cambridge: Polity Press. Collins, Jim. 2010. Bring on the Books for Everybody: How Literary Culture Became Reading Culture. Durham: Duke University Press. Colman, Felicity, Vera Bühlmann, Aislinn O’Donnell, and I. van der Tuin. 2018. Ethics of Coding: A Report on the Algorithmic Condition. European Commission. Couldry, Nick, Aristea Fotopoulou, and Luke Dickens. 2016. Real Social Analytics: A Contribution Towards a Phenomenology of a Digital World: Real Social Analytics. The British Journal of Sociology 67 (1): 118–137. Dufrenne, Mikel. 1973. The Phenomenology of Aesthetic Experience. Trans. E.S. Casey. Evanston: Northern University Press. Elleström, Lars. 2010. The Modalities of Media: A Model for Understanding Intermedial Relations. In Media Borders, Multimodality and Intermediality, ed. Lars Elleström and Jørgen Bruhn, 11–50. London: Palgrave Macmillan UK. ———. 2020. The Modalities of Media II: An Expanded Model for Understanding Intermedial Relations. In Beyond Media Borders: Intermedial Relations among Multimodal Media, ed. Lars Elleström, vol. 1, 3–91. Cham: Springer International Publishing. Fraitag, Madison. 2020. CTAs: Why, Where, and How to Use Them. Animoto Blog, December 21, 2020. https://animoto.com/blog/video-marketing/why-where-how-ctas. Accessed 27 Jan 2022.

47

The Recommended Experience: Engaging Networked Media Platforms. . .

1211

Genette, Gerard. 1997. Paratexts: Thresholds of Interpretation. Trans. J.E. Lewin. Cambridge: Cambridge University Press. Gillespie, Tarleton. 2014. The Relevance of Algorithms. In Media Technologies: Essays on Communication, Materiality, and Society, ed. Tarleton Gillespie, Pablo J. Boczkowski, Kirsten A. Foot, Leah A. Lievrouw, Ignacio Siles, Finn Brunton, Gabriella Coleman, Geoffrey C. Bowker, Jonathan Sterne, and Lucy Suchman, 167–194. Cambridge: The MIT Press. ———. 2018. Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media. New Haven: Yale University Press. Goldschmitt, K.E., and Nick Seaver. 2019. Shaping the Stream: Techniques and Troubles of Algorithmic Recommendation. In The Cambridge Companion to Music in Digital Culture, ed. Nicholas Cook, Monique M. Ingalls, and David Trippett, 63–81. Cambridge: Cambridge University Press. Hagen, Anja Nylund. 2015. The Playlist Experience: Personal Playlists in Music Streaming Services. Popular Music and Society 38 (5): 625–645. Have, Iben, and Birgitte Stougaard Pedersen. 2020. The Audiobook Circuit in Digital Publishing: Voicing the Silent Revolution. New Media & Society 22 (3): 409–428. Heffernan, Virginia. 2010. E-Readers’ Collective. New York Times Magazine, October 15, 2010. https://www.nytimes.com/2010/10/17/magazine/17FOB-medium-t.html. Accessed 27 Jan 2022. Hildebrandt, Mireille, and Serge Gutwirth. 2008. Profiling the European Citizen: CrossDisciplinary Perspectives. Dordrecht: Springer Netherlands. Jakobson, Roman. 1960. Closing Statement: Linguistics and Poetics. In Style in Language, ed. Thomas A. Sebeck, 350–377. New York and London: The Technology Press of Massachusetts Institute of Technology and John Wiley & Sons. Jewitt, Carey, and Berit Henriksen. 2016. Social Semiotic Multimodality. In Handbuch Sprache im multimodalen Kontext, ed. Nina-Maria Klug and Hartmut Stöckl, 145–164. Berlin, Boston: De Gruyter. Konstan, Joseph A., and John Riedl. 2012. Recommender Systems: From Algorithms to User Experience. User Modeling and User-Adapted Interaction 22 (1–2): 101–123. Kress, Gunter R. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge. Kümmerling-Meibauer, Bettina. 2013. Paratexts in Children’s Films and the Concept of MetaFilmic Awareness. Journal of Educational Media, Memory, and Society 5 (2): 108–123. Langley, Paul, and Andrew Leyshon. 2017. Platform Capitalism: The Intermediation and Capitalisation of Digital Economic Circulation. Finance and Society 3 (1): 11–31. Lehmann, Sune. 2021. Aggregation. In Uncertain Archives: Critical Keywords for Big Data, ed. Nanna Bonde Thylstrup, Daniela Agostinho, Catherine D’Ignazio, and Kristin Veel, 47–55. Cambridge: The MIT Press. Matta, Natasha. 2021. Spotify Is Listening to You in More Ways Than You Think. Medium, April 19, 2021. https://medium.com/encode-justice/spotify-is-listening-to-you-in-more-ways-thanyou-think-d490132577d6. Accessed 27 Jan 2022. Nieborg, David B., and Thomas Poell. 2018. The Platformization of Cultural Production: Theorizing the Contingent Cultural Commodity. New Media & Society 20 (11): 4275–4292. Pedersen, Birgitte Stougaard, Maria Engberg, Iben Have, Ayoe Quist Henke, Sarah Mygind, and Helle Bundgaard Svendsen. 2021. To Move, to Touch, to Listen: Multisensory Aspects of the Digital Reading Condition. Poetics Today 42 (2): 281–300. Peirce, Charles Sanders. 1994. In The Collected Papers of Charles Sanders Peirce. Electronic edition, ed. Charles Hartshorne and Paul Weiss, vol. 5. Cambridge: Belknap Press of Harvard University Press. Prey, Robert. 2018. Nothing Personal: Algorithmic Individuation on Music Streaming Platforms. Media, Culture & Society 40 (7): 1086–1100. ———. 2020. Locating Power in Platformization: Music Streaming Playlists and Curatorial Power. Social Media + Society 6 (2): 1–11.

1212

M. Bak Herrie

Rajewsky, Irina O. 2005. Intermediality, Intertextuality, and Remediation: A Literary Perspective on Intermediality. Intermédialités (Montréal) 6: 43–64. Rossiter, Ned, and Soenke Zehle. 2015. The Aesthetics of Algorithmic Experience. In The Routledge Companion to Art and Politics, ed. Randy Martin and Victor J. Peterson, 214–221. London: Routledge. Santaella, Lucia. 2015. Sound and Music in the Domain of Rhematic Iconic Qualisigns. Signata 6: 91–106. Savage, Mark. 2021. Spotify Wants to Suggest Songs Based on Your Emotions. BBC News, January 28, 2021. https://www.bbc.com/news/entertainment-arts-55839655. Accessed 27 Jan 2022. Scherzinger, Martin. 2019. The Political Economy of Streaming. In The Cambridge Companion to Music in Digital Culture, ed. Nicholas Cook, Monique Marie Ingalls, and David Trippett, 274–297. Cambridge: Cambridge University Press. Schramm, Wilbur. 1971. The Nature of Communication Between Humans. In The Process and Effects of Mass Communication, ed. Wilbur Schramm and F.D. Roberts, 3–53. Urbana: University of Illinois Press. Seaver, Nick. 2015. The Nice Thing About Context Is That Everyone Has It. Media, Culture & Society 37 (7): 1101–1109. Spotify. 2021. Company Info. Spotify For the Record, 27 October 2021: https://newsroom.spotify. com/company-info/. Accessed 27 Jan 2022. Srnicek, Nick. 2019. Platform Capitalism. Cambridge: Polity Press. Steele, Anne. 2020. Spotify Strikes Podcast Deal with Joe Rogan Worth More Than $100 Million. The Wall Street Journal, May 19, 2020. https://www.wsj.com/articles/spotify-strikes-exclusivepodcast-deal-with-joe-rogan-11589913814?mod¼hp_lead_pos11. Accessed 27 Jan 2022. Stoller, Matt. 2020. The Death of Independent Podcasting: What Spotify Is Trying to Do With the Joe Rogan Deal. ProMarket: The Publication of the Stigler Center at the University of Chicago Booth School of Business, May 27, 2020. https://promarket.org/2020/05/27/the-death-of-independentpodcasting-what-spotify-is-trying-to-do-with-the-joe-rogan-deal/. Accessed 27 Jan 2022. Striphas, Ted. 2015. Algorithmic Culture. European Journal of Cultural Studies 18 (4–5): 395–412. Sutton, Matthew. 2015. Amplifying the Text: Paratext in Popular Musicians’ Autobiographies. Popular Music and Society 38 (2): 208–223. Vinter, Robin. 2021. Spotify Hides Shuffle Button after Adele Says Albums Should ‘Tell a Story’. The Guardian, November 21, 2021. https://www.theguardian.com/music/2021/nov/21/spotifyhides-shuffle-button-adele-albums-should-tell-a-story. Accessed 27 Jan 2022. Yoo, Noah. New Spotify Patent Involves Monitoring Users’ Speech to Recommend Music. Pitchfork, January 28, 2021. https://pitchfork.com/news/new-spotify-patent-involves-monitor ing-users-speech-to-recommend-music/. Accessed 27 Jan 2022.

48

Posthuman Intermedial Semiotics and Distributed Agency for Sustainable Development Asun Lo´pez-Varela Azcárate

Contents Introduction: Everything Is Intermedial. Media Modalities Extended . . . . . . . . . . . . . . . . . . . . . . . . The Path to Posthumanism and 4E Cognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Semiotic Modality and Peirce’s Sign Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . From Habits to Distributed Agencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intermedial Semiotics. Analogue Versus Digital . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Extended Cognition and the Problem of Agency Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1214 1218 1224 1228 1231 1234 1236 1238

Abstract

This chapter brings Intermedial Studies out of bounds by introducing intermediality in a discussion on the possible scaling of agentive capacities beyond the human and onto the material medium itself. In order to do this, the chapter begins with an introduction to Lars Elleströn’s media modalities (the material, the sensorial, the semiotic, and the spatiotemporal) and how they relate to neo-materialism and Peircean semiotics. The chapter shows how some of the main ideas present in the neo-materialist mind-matter continuum of philosophical posthumanism are also in close relation to Charles S. Peirce concept of synechism. The chapter discusses this continuum and its connection to 4E cognition in order to offer a theorization of process ontology that helps envision the need for systemic process epistemologies that, with a proper body of practices, might align Intermedial Studies with Sustainable Development through the idea of agency distribution across material media, including human and nonhuman agencies.

A. López-Varela Azcárate (*) English Studies, Comparative Literature, Semiotics, Universidad Complutense, Madrid, Spain e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2_61

1213

1214

A. Lo´pez-Varela Azcárate

Keywords

4E Cognition · Distributed agency · Intermedial studies · Nonhuman · Posthumanism · Peircean semiotics · Sustainable development

Introduction: Everything Is Intermedial. Media Modalities Extended The so-called “material turn” (Miller 1998; Latour 2007) is an extensive interdisciplinary conversation involving complex interrelations between networks of users, the material artifacts and tools they use, and their living environments. These relations are simultaneously material (physical exchanges) and semiotic (conceptual and communicative exchanges), and they implicate intermediality at various levels. The discussion is relevant because at the heart of the debate is the possible scaling of agentive capacities beyond human users, connecting the debate to environmental action and Sustainable Development in general. The Oxford Dictionary informs that the Latin prefix “inter” has two meanings: on the one hand, it signifies “between and among.” At the same time, it means “mutually and reciprocally.” In turn, “mediality” has been seen as designating the interaction of technology, society, and cultural factors. Institutionalized media produce, circulate and transform what Lars Elleström has termed “media modalities”: the material, the sensorial, the semiotic, and the spatiotemporal. Elleström’s “sensorial modality” includes the physical and mental acts of perceiving the medium, achieved with the intervention of the “spatiotemporal modality,” that structures perception according to the four physical axes of width, height, depth, and time (Elleström 2021b: 20 ff). Sensory experience is materially realized in all species. For instance, all animals have eyes that allow them to see and detect light, color, and movement. Their ears enable animals to hear sound and detect vibrations. Through the respiratory orifices, animals can detect smells, while the tongue allows taste of different flavors. The skin serves to sense touch, pressure, and temperature. Depending on their complexity, animals’ nervous system might have specialized nerve endings that respond to pressure, vibrations, and electrical fields (i.e., some fish like sharks and mammals like whales) (Alcock 2013). Similarly, plants have specialized cell structures that allow them to perceive changes in their environment and respond to stimuli. For example, photoreceptor cells are affected by light and help regulate processes such as phototropism (bending toward light) and flowering. Some plants, like mimosa, also respond to touch, folding their leaves when poked. Like animals, plants are affected by the Earth’s gravity and direct their roots downward and their stems upward. Their chemical receptors respond to water, nutrients, etc., and thermoreceptors to changes in temperature, even opening and closing their leaves in response to heat or cold (Karban 2015). Cognition refers to the processes and activities associated with acquiring, processing, storing, and using information that becomes meaning and functional

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1215

(Elleström 2021b: 13, 43, 51, 85, 115, 116, 127, 152, 199, 226). In humans and animals, these processes are mostly mental, because brain regions have the largest number of connecting neurons and sensorial neurotransmitters. However, various disciplines have contributed to 4E cognition, demonstrating that meanings are not only in our heads, and they are embodied, embedded, enacted, and even extended beyond the (human) body. In this sense, certain animals and plants may have neurotransmitters that are distributed through their physiological frameworks. Although experienced as an integrated process, cognition involves many aspects, from the physical materials that enable certain kinds of perception, to the elements that turn them intelligible as well as sharable with other individuals through various forms of information transfer. This coded information can take the form of verbal and gestural communication proper of humans and some animals. But it can also involve other forms of coding such as chemical coding, present in DNA. For instance, in recent years, the so-called “DNA of Things” is a storing method of digital information in DNA molecules that can then be embedded into almost any object. Intermediality is involved at all these levels for, as Lars Elleström once pointed out, everything is intermedial (2020). The perceptive, connected to the immediate material world, and the cognitive realms communicate semiotically, that is, by means of signs systems. The semiotic modality encompasses the coding and decoding of signs. Although both signs and signals convey information, some scholars claim that they are different depending on how they are perceived and interpreted. Signs are often seen as operating in a given community of individuals who rely on prior knowledge and cultural convention. Signals, on the other hand, are mostly considered in response to immediate specific situations, coded by chemicals for instance. Thus, linguists consider that human communication operates by means of signs, whereas animal and plant communication relies on signals. In Peircean semiotics this difference does not exist. According to Harvard pragmatist Charles Sanders Peirce, a sign is anything that appears as meaningful to an individual (human, animal, or vegetable). A sign is a mere carrier of a potential meaning, which is not part of the sign itself. For instance, plants can communicate through the release of volatile organic compounds (VOCs) serving various purposes, such as attracting pollinators, deterring herbivores, or attracting symbiotic fungi. Below the ground, their roots can release chemicals to attract beneficial microbes or deter harmful ones. Interplant communication is also a growing area of research whose ecological implications still have to be discovered. In 1791 American naturalist William Bartram published his Travels and his observations led him to conclude that “vegetable beings are endued with some sensible faculties or attributes, similar to those that dignify animal nature.” (1791: 18). Sensory modalities determine the way signs are decoded in intermedial configurations (Elleström 2021b: 22, 40, 46). In Peirce’s classification, the visual modality can distinguish between a gesture (an indexical or iconic sign, or both, depending on the type of gesture), an icon (signs that resemble the real such as maps, diagrams, and images) and symbols (abstract and arbitrary signs in a community of users, for example, human discourse). To read Braille, the tactile modality is used. Thus, Braille uses a specific language-code made up of symbols learnt by the users of

1216

A. Lo´pez-Varela Azcárate

the community of visually impaired people. The material modality of pottery, for instance, or of sculpture, consists of solid materiality that can be realized by different technical media such as clay, plaster, stone, bronze, and so on, and perceived visually, by touch, or even smell. While the materiality of a photograph might be considered static, once the picture forms part of a larger composition of moving images, the dynamic and temporal axis is introduced into what we know as cinema or motion pictures. Temporal structures include aspects such as sequence, present in sound (music), in cinematic formats (television, cinema, etc.), as well as the syntactic patterns of human discourse. Another important aspect is the relationship between cognition and consciousness. Some theories (for an overview, see Baars and Geld 2021) suggest that consciousness arises from and is dependent on cognitive processes. In other words, conscious experience is thought to be a result of brain activity that supports cognitive processes such as attention, perception, and memory. In this view, consciousness is a unique aspect of humans because they are able to demonstrate awareness of their own perceptual and cognition processes, and can claim any actions performed as their responsibility. The development of psychology, however, has shown that cognition also involves unconscious experiences, often related to emotions, including clusters of individual and cultural aspects such as motivations, values, etc., not always aligned with one’s actions and agency (LeDoux 1996). Antonio Damasio has explained that there are forms of nonconscious neural signaling in organisms, which give rise to what he terms the “proto-self,” “the nonconscious forerunner for the levels of self which appear in our minds as the conscious protagonists of consciousness: core self and autobiographical self” (1999: 22), which permit extended consciousness and, finally, consciousness. Developmental psychology (see Shaun Gallagher distinction between the “minimal self” and the “narrative self”) has explored the progress toward self-awareness in human infants. In legal terms, minors under a certain age cannot be held accountable and fully responsible for certain actions (Lagercrantz and Changeux 2009). Research that associates consciousness with specialized neural structures, chiefly neuroanatomical, neurochemical, and neurophysiological, links cognition to complex organisms (for an overview, see Cohen and Schooler 1997). For a very long time, it was believed that both brain size and connectivity contributed to make perception and sensation more discriminating in cognition. Contemporary evidence indicates that there are problems with these assumptions (Harley 2021). For instance, although it would seem that conscious experiences would be more likely associated to brain regions with a large number of connecting neurons, for example, in the cerebellum, a part of the human brain that contains almost three-quarters of the neurons, people born without a well-functioning cerebellum suffer some problems, but lacking consciousness is not one of them. In contrast, other brain regions like the thalamus seem to be closely related to consciousness even if they might not generate it (apparently, the thalamus seems to work like hub that facilitates interconnections). Besides, it has been claimed that humans are not unique in possessing complex neurological substrates. Animals, including all mammals and birds, and many other creatures, for example, dolphins and octopuses, also possess these systems, and there

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1217

are bats that can orientate themselves acoustically by echolocation (sonar), emitting short high-frequency pulses of sound well above the range of human hearing. The fact that humans cannot perceive or make sense of certain sign systems does not mean that they do not exist. In consequence, cognition is shaped and structured by complex dynamic intermedial interactions. These relations occur between the brain, the body, their forms of enhancement (in humans the use of glasses, a walking stick, or a prosthesis, or forms of neuro-stimulation, like Transcranial Magnetic Stimulation TMS), connected to the physical and social environments, with different effects on the brain and the body. For instance, temperature has been seen to affect body metabolism as well as brain activity and concentration in humans, while nonhumans, like plants, are very sensitive to extreme temperatures, fluctuations that can affect their growth and survival. As noted, intermedial relations involve both the material and the conceptual, so that, semiotics is one of the spheres within the intermedial continuum. All interactions are sustained by particular material means, which range from the physical body (including cell types, tissues, organs, etc.) to the tools, artifacts, and complex machines used. Technical media also require forms of energy to function, and all these interactions take place in a particular spatiotemporal context that changes over time; for example, home appliances, personal computers, laptops, televisions, all electricity, smartphones that need it to charge their batteries. Since the discovery of electricity, technical media, from microscopes to telescopes, have moved from manual adjustment to being powered by electrical means. Any instrument that conveys information via integrated digital circuits requires some kind of powering. Finally, social aspects like economy, or the institutions that support the social context, are all involved in this complex process, validating the media being used, introducing new models, etc. These changes are also aligned with interests, motivations, choices, desires, or limitations. For instance, since the 1990s, the rapid digitalization of all technical media introduced challenges such as accessibility to particular population groups. Thus, the expeditious development of digital and cybernetic machines has encouraged an inquiry into the forms of agency enhanced by their use, particularly after the spread of intermedial digital technologies (see, for instance, Bruhn and Schirrmacher 2022: 69 ff & 219–20). Alongside these changes in the uses of technology, there is a growing concern for the environment and for Sustainable Development; most fundamentally, the consequences of greenhouse gas emissions, which trap heat in the Earth’s atmosphere, leading to a rise in temperatures, causing widespread changes in climate patterns as well as extreme weather events. The melting of polar ice and rise of sea levels, ocean acidification or the decline of biodiversity, with the spread of viruses across animal populations and humans are among the issues, also related to pollution and health problems. There are also significant economic impacts, such as increased costs for energy and water, reduced crop yields, as well as massive migrations of all sorts of human and animal populations associated to changes in their habitats. In this panorama, there is a need for urgent actions, but first people need to understand and be aware that human activity is part of these complex networks of human and

1218

A. Lo´pez-Varela Azcárate

nonhuman actants where intermediality is enmeshed. Jørgen Bruhn and his co-authors affirm that “this ‘cultural and spiritual’ transformation is closely connected to concepts of agency,” (Bruhn et al. 2022: 236) identifying the crucial importance of agency in ecological discourse. Posthumanism envisions a systemic and process ontology that questions the concept of anthropocentric agency, a concept that has been appropriated as an unalienable right of one sole species, the human, positing its superiority over other animate beings and even inanimate things. This ontology was already present in the works of Harvard semiotician and pragmatist, Charles Sanders Peirce. The following section introduces posthumanism and 4E Cognition so that in later sections their tenets can be compared to those of Peirce.

The Path to Posthumanism and 4E Cognition Neo-materialism is at the heart of the posthuman debate (Miller 1998; Latour 2007; Caracciolo et al. 2022). It is an ongoing discussion that contemplates the possibility that agential capabilities might be distributed beyond humans. Philosophical posthumanism explores how occurrences in the world are “enacted” by means of intermedial interactions between humans and nonhuman actants. The category of nonhumans includes living forms other than humans (microorganisms, plants, animals), advanced forms of machine intelligence, such as Artificial Intelligence (AI), as well as inanimate matter, traditionally considered inert and passive. In the last decades, neo-materialism has acquired new importance because of two concurring circumstances: the rapid development of digitalization and cybernetic machines, and the growing concern for greenhouse gas emissions, with severe consequences for the environment, human health, and the global economy. In both cases, new forms of symbiosis, enacted from the encounters between human and nonhumans, with emergent and unexpected effects, are extending the debate on intermediality. To address and to challenge the human connection to things, the Earth, and life, posthumanists have turned toward vitalism as well as semiotics. Posthumanism is an intellectual and cultural movement that challenges traditional human-centered perspectives and aims to develop a more holistic understanding of the world. It criticizes the idea of human exceptionalism and asserts that humanity cannot be understood as separate or distinct from other nonhuman entities. The French sociologist and semiotician, Bruno Latour has been influential in the development of posthumanism with his concept of actor-network theory (ANT), which questions the separation of nature and culture and the division between subjects and objects. In his 1991 book We Have Never Been Modern, Latour argued that there is no clear distinction between human and nonhuman actors, and that all entities, including technology, humans, animals, other bio-entities, and even inanimate objects, can be considered as active agents in the world. He suggests that we need to understand the relationships between these “actants” as networks, rather than simple linear causal chains, in order to fully understand the intermedial and relational dynamics of the world.

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1219

Following Latour, scholars like N. Katherine Hayles have explored the intersection of digital technology and the humanities and social sciences. With a background in chemistry and Dynamic Non-Linear Systems Theory, Hayles’ approach to posthumanism in her 1999 book emphasized the importance of maintaining a connection to the human body in the face of technological advance. Hayles connects the crisis of Cartesian humanism to the development of cybernetics since the 1940s. The first gatherings, known as the Macy Conferences, included a wide range of scientists like Alan Turing or NASA expert John von Neumann, anthropologists like Margaret Mead and her husband Gregory Bateson, electronic engineer Claude Shannon, responsible for the binary theory of information, or mathematician and philosopher Norbert Wiener, who in 1948 coined the term “cybernetics,” drawing on the Greek word for “steersman” (kubernētēs). The term described the scientific study of control and communication in the animal and the machine. This first wave focused on the study of “homeostasis”; that is, the ability of organisms to maintain steady states regardless of environmental changes and external agents interfering in the dynamics of the system. A system can be defined as a nonlinear dynamic set of actors, relations, objects and things, and all their intra and inter connections. Systems can be biological, for instance, an ecosystem, but also cultural, situated in a particular environment, place, and time. Systems can be open or closed to their surroundings. Closed systems have boundaries or walls, sometimes defined artificially (i.e., territorial borders). Although finite to a certain extent and with degrees of closeness, different systems are interconnected. For instance, in the human body, the digestive system functions in relation to the respiratory system, the circulatory and all other bodily systems. Operating in a sort of network, the distribution of system components can vary, which means that a given system can acquire different states in a short time-span, while remaining the same in a longer time-span. Thus, a state is understood as the momentary position of a system in space and time. This position depends on physical properties (space-context) as well as on the distribution of these properties in a particular time. To have a complete understanding of how a system works, it needs to be seen from an integrated approach that looks at the full spectrum of scales, networks, states, and multiple spatiotemporal dimensions, considering the intraactions and inter-actions of all system components, because the whole can be other than the sum of the parts. In the 1950s, research by Nobel laureate Ilya Prigogine on chemical dissipative structures and entropy led to pioneering studies in self-organizing systems, as well as philosophical inquiries into complex systems and irreversible processes in the natural sciences. These findings implied that absolute determinism was no longer possible. Mathematical probability and big data analysis were required to monitor the evolution and dynamics of complex systems where random encounters between the parts of the assemblages yielded unexpected results. Concepts like “enaction” or “emergence,” explained below, gained popularity. In her inquiry into How we became posthuman, Hayles noted that second wave of cybernetics turned to the study of how component elements of a given system work together and replicate. The collection Automata Studies appeared in 1956 with contributions from most

1220

A. Lo´pez-Varela Azcárate

members of the Macy’s group, including Claude Shannon, W. Ross Ashby, John von Neumann, Marvin Minsky, Edward F. Moore, and Stephen Cole Kleene. Iterative forms of computation enabled forms of accounting for replication mechanisms in all kinds of systems, including biological ones. The idea that the human mind operated as if it were a computer, able to perform automatic operations, began to influence many theories of cognition. Languages were also seen as replicative structures. In the 1960s, Noah Chomsky began to develop his first ideas on generative grammar, a theory that explains human language by means of iterative units. It was not until the 1980s, coinciding with the third wave, that scientists from various disciplines, including, for instance, linguist George Lakoff and philosopher Mark Johnson in their well-known volume Metaphors We Live By, challenged the notion that meanings are just in the head. During this decade, the focus shifted toward embodiment as well as context-related aspects (that is, embedding, as part of what later became 4E cognition). In 1984, Humberto Maturana and Francisco Varela published their study The Tree of Knowledge: the Biological Roots of Human Understanding. Their research claimed that biological systems are “autopoietic,” meaning that they are proactive, self-organizing, and self-regulating in particular spatiotemporal spans. “Autopoïesis” went beyond homeostasis in arguing that organisms respond to their environments in ways that align to the changing needs of the organism as part of a larger ecosystem. Since these needs are specific to each organism, interactions within the system can “bring forth” or “enact” unexpected effects. In the field of bio-semiotics, work by Jakob von Uexküll, Thomas A. Sebeok, or Jesper Hoffmeyer and his concept of “emergence” (López Varela 2012: 115–119) come very close to the considerations present in “enactivism” (the third E present in 4E cognition). Enactivism has been seen to rely on the concept of “affordance” (Gibson 1977; Gallagher 2014); in Peircean terms, a set of stimuli that an object provides an interpretant so that it acquires the character of a sign representamen, that is, it stands for something in some respect or capacity. The concept of “affordance” was developed by psychologist James J. Gibson (1977), who argued that humans can modify affordances in their environment to their benefit. According to the authors of The Embodied Mind: “The key point, then, is that the species brings forth and specifies its own domain of problems [. . .] this domain does not exist ‘out there’ in an environment that acts as a landing pad for organisms that somehow drop or parachute into the world. Instead, living beings and their environments stand in relation to each other through mutual specification [. . .] the result of a conjoint history, a congruence that unfolds from a long history of codetermination” (Varela et al. 1992: 198–199). Enactivism also emphasized that incidental and fortuitous changes occur, and that organisms absorb these changes, incorporating them as meaningful aspects. Accumulation and replication of changes provides a sort of direction to evolutionary variation. In terms of Norbert Wiener’s theory of information, what this means is the any form of “noise” that enters a communication channel by chance, might acquire some form of meaning. In other words, a signal can become a sign. Signals, in the sense I use the term here, depend on physical properties (electrical or radio waves) transmitted or received by technical media. In organisms, signals operate at the level

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1221

of perception (i.e., color in an object is a sensation on the eye, which is the result of light reflection or emission in relation to the object). The simplest signal takes the form ‘exist/non-exist’ (or on/off and 0/1), used at the lowest level of digital programming). It is important to emphasize that in perception, a signal is a very basic form of “sign,” if we follow Peirce’s theories. By the 1990s, “enactive” and “emergence” were used interchangeably. The term “emergent” appeared for the first time in work Problems of Life and Mind, by the British philosopher George Henry Lewes. He affirmed that “instead of adding measurable motion to measurable motion, or things of one kind to other individuals of their kind, there is a co-operation of things of unlike kinds. The emergent is unlike its components insofar as these are incommensurable, and it cannot be reduced to their sum or their difference” (Lewes 1875: 369). In other words, emergent properties are more than the initial properties attached to the units/elements/substances of the system or assemblage. In the twentieth century, many of these finding were tied to situations encountered in Quantum Physics. Hence, in the 1990s, American physicist Karen Barad put forth her theory of “agential realism” as applied beyond the field of physics. Agential realism presupposes the ontological inseparability of intra-acting agencies, positing that the world is not made up of preexisting objects, but rather that objects and subjects are co-constituted through their mutual entanglement and performative inter- and intra-actions. By intra-action, Barad means that “specific material (re)configurings of the world through which local determinations of boundaries, properties, and meanings are differentially enacted” (2003: 828). In Barad’s account, everything is intimately intertwined in the process of becoming: “matter is substance in its intra-active becoming – not a thing but a doing, a congealing of agency. And performativity is [. . .] iterative intra-activity” (2003: 828). Furthermore, for Barad, agency is a distributed condition where we are not outside observing the world. “Nor are we simply located at particular places in the world; rather, we are part of the world in its ongoing intra-activity” (2003: 828 emphasis in the original). This form of distributed agency opens the door to considering different agential levels, in which nonhumans might also be implicated. Agency is also related to the notion of “homeostasis” or forms of self-regulation, which are really a form of what Peirce would describe as “taking on habits” (see below); habits that might be internalized and occur out-of-awareness, that is, in the absence of human consciousness. This notion comes to alter the idea of what is an agent and who can act as such. Thus, from the perspective of Non-Linear Dynamic Systems Theories, it seems clear that intermediality needs to be seen as a complex system of relations because of the large number of variables (or media modalities) involved. In 1991, eight years before Hayles’s publication, Donna Haraway approached posthumanism with her concept of cyborg as a figure that signals the blurring of the boundaries between humans and technology. In her “A Cyborg Manifesto: Science, Technology, and Socialist-Feminism in the Late Twentieth Century,” (1991) Haraway uses the concept to challenge binary oppositions such as human/machine or nature/culture, which in turn contain the female/male separation. Her article closes with the lines “I would rather be a cyborg than a goddess,” meaning that the cyborg

1222

A. Lo´pez-Varela Azcárate

challenges conventional notions of identity (i.e., women associated to nature, not to culture or technology). In When Species Meet (2007), Haraway moves on to a holistic understanding that recognizes the agency of nonhuman entities, including animals, machines, and ecosystems. Her feminist approach to Otherness is expanded to the study of species relationships, examining the ways in which gender, race, and class shape human relationships with nonhumans, thus pioneering ecofeminist perspectives. Also from a ecofeminist approach, Rosi Braidotti’s Nomadic Subjects (1994) and The Posthuman (2013) explore alternative models of subjectivity and argue that the current dominant Western understanding of the human subject is deeply flawed and in need of radical transformation. Braidotti defends a process of redefinition of one’s sense of self, through her notion of “nomadic subjectivity” in connection to a shared habitat. This redefinition includes an extension of the classical Greek notions of “bios” and “zoe,” that distinguish two different aspects of human existence. While “bios” is defined by biological and cultural attributes, encompassing the material, physical aspects of life, such as personal experiences, relationships, and social norms, “zoe” refers to the essential, universal aspects of life that are shared by all living beings. Zoe is often used to refer to the fundamental nature of life itself, beyond the specifics of individual existence, standing for the enactive transversal vitality that connects all kind of bodies and things. In her view, individuals never merely inhabit a place, in the sense of taking possession of it by means of habit. Instead, their individuality is in the making, so to speak, embodied, embedded, and enacted within the networks and practices of the group, cluster, or community (including nonhuman groups). Jane Bennett’s approach to posthumanism asserts that all matter, including nonhuman entities, have agency and capacity to affect change in the world. Like Latour, Haraway, or Braidotti, Bennett also criticizes the dualisms that have long dominated Western thought, such as the separation of nature and culture, and the division between subjects and objects. Bennett argues that these forms of ontology prevent the recognition of the agency of nonhuman entities. She argues that inanimate matter like rocks, water, or waste have impacts on human lives and that this odd form of agency need to be accounted for in terms of ethical responsibility toward other beings and entities in the material world. In The Whole Creature: Complexity, Biosemiotics and the Evolution of Culture (2006), Wendy Wheeler claimed that meaning making is a bio-semiotic process that involves not just conceptual but also material aspects. According to Wheeler, organic and nonorganic matter have porous borders and are informed by a sort of drive to establish contact and communication with the rest of the world. This dynamics engages all beings “from the humblest forms of single- cell life upwards” (2006: 271), a transit across bodies that Stacy Alaimo has termed “transcorporeality” (2010) and Jane Bennett “vibrant materiality” (2010). Likewise, Haraway speaks of all bodies as being “semiotically active,” no matter if they are biological, abiotic, or artificial (2007: 250). Trying to make sense of embodiment after the onset of digitalization, Hayles (1993; 2002) noted that different dimensions of inscription

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1223

(writing, representation, etc.) and decoding (reading, interpretation, and so on) concerned themselves with sensing as well as making sense of things mediated through embodied actions (incorporation; notice the etymology of the term). From a political and ethical dimension, Stacy Alaimo has stressed the importance of recognizing the role of power, oppression, and injustice in shaping the relationships between humans and nonhumans, defending that a posthumanist framework must take into account these aspects. Similarly, Serenella Iovino and Serpil Oppermann (2014) have highlighted the fact that the world takes the form of signs constantly inscribed in different forms of matter. These signs of the world, present in many different intermedial forms, can be translated into human languages, thus voicing and giving expression to all entities (see also Von Mossner 2017; Caracciolo 2022). For “every being that matters” is “a congeries of its formative histories” (Haraway 2007: 2) in the way that it “joins text and body in [. . .] material semiosis and semiotic materiality” (Haraway 2007: 163). In the course of time, materiality is punctuated with meanings, becoming storied matter. Even unintended interventions leave traces that “speak” back to humans, as pandemics might. Naturally, Anthropocene histories appear in all sorts of intermedial formats, for instance, in contemporary cinema (for an exploration of agency in environmental cli-fi films, see Salmose 2018). Once again, the recognition of the semiotic continuum between nature and culture, including within the cultural and technological change, brings awareness to the fact that everything is intermedial, and that there is a need for the development of a holistic approach, when facing global problems and the implementation of Sustainable Development goals. Nevertheless, it is not entirely clear how this sense of total awareness can be put into action to challenge the neoliberalist approaches that threaten the planet. This is where Peircean semiotics might help in providing an ontological framework that offers a foundation for an epistemology and a body of practice that establishes new underlying assumptions and beliefs about the nature of reality and existence. These assumptions shape the way in which we approach what cognition is, informing the means by which information is intermedially acquired, used, and applied. For example, in the Cartesian ontological framework, the nature of reality was fixed and unchanging, leading to an epistemology that emphasized anthropocentric interpretation, objectivity, and the pursuit of universal truth. This, in turn, informed a body of practice that valued empirical observation, experimentation, and evidence-based decision-making. Conversely, Peirce’s process ontology contemplates reality as dynamic, leading to an epistemology that emphasizes the importance of perspective, with a body of practice that values open-mindedness, collaboration, and interdisciplinary inquiry. By articulating these underlying assumptions, Peircean semiotics provides a consistent framework for understanding action and agency. Thus, the following sections introduce Peirce’s basic formulations and sign categories as well as his vision of process philosophy and of synechism (from Greek “syn,” together and “échein” to hold), that is, the mind–matter continuum. These formulations seek to illuminate the way in which “the transfer of cognitive import from a producer’s to a perceiver’s mind” through “the intermediate stage” (Elleström 2021a: 13) takes place.

1224

A. Lo´pez-Varela Azcárate

The Semiotic Modality and Peirce’s Sign Categories A treaty entitled Perì semeîon kai semeióseon (On Signs and Sign Inferences) also known as De Signis by the Epicurean philosopher Philodemus of Gadara seems to be the source from which Peirce finally derived the term “semiotics” (Castañares 2012: 21–23). The older etymological roots of the term occur in relation to the ancient Greek term semeîon and the sem-family. Sêma was used to refer to the markings of death, meaning “tomb” and “burial mound,” as seen in writings by Hesiod or Homer. In writings by Aeschylus, Aesop, Hecataeus of Miletus, Anaxagoras, or Cleostratus, semeîon was used to refer to a technical medium generally interpreted as a “sign” of the gods, for instance, Neptune’s trident, which produced rays and thunder. While Peirce, from his background as logician, focused on the cognitive and phenomenological aspects of what he termed “semiotics,” contemplating the diversity of communication modes (not just discursive), the Suisse linguist Ferdinand de Saussure established the European study of “semiology” based on the interactions between a system of signs or signifiers in relation to signified or concepts. The phonetic signs used in discourse and their connections to written forms established the ground for the interpretation of other signs, like images and gestures. The human linguistic sign was the basis for the entire system. In contrast to the Saussurean dyadic model, Peirce’s model was triadic and dynamic; not a fixed structural model, but a dynamics of relations. For Peirce, a material object determines a sign (“representamen”) in a process called “firstness,” which in turn determines another sign or “interpretant” in “secondness.” The “interpretant,” fulfilling its function as sign of the object, determines a further “interpretant sign” in “thirdness.” Peirce defines sign as follows: A sign, or representamen, is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the interpretant of the first sign. The sign stands for something, its object. It stands for that object, not in all respects, but in reference to a sort of idea, which I have sometimes called the ground of the representamen. “Idea” is here to be understood in a sort of Platonic sense, very familiar in everyday talk; I mean in that sense in which we say that one man catches another man’s idea, in which we say that when a man recalls what he was thinking of at some previous time, he recalls the same idea, and in which when a man continues to think anything, say for a tenth of a second, in so far as the thought continues to agree with itself during that time, that is to have a like content, it is the same idea, and is not at each instant of the interval a new idea. (“On Signs” CP 2.228, 1897)1

A sign is something that stands for another (Peirce calls this “object”) so that an experience of the former affords knowledge of the latter in some respect or capacity. This includes sounds, images, gestures, scents, tastes, textures, words, etc. Peirce

1

Adhering to scholarly tradition, Peirce’s Collected Papers are cited CP, followed by volume and paragraph number as well as original date of publication.

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1225

insists that the interpretant is a sort of “sign vehicle” (he describes it as a “Quasimind”) and not a person or living entity: “the interpretant is nothing but another representation to which the torch of truth is handed along; and as representation, it has its interpretant again, leading to another infinite series” (Peirce 1931: 58). That is, the sign creates in the mind a more developed sign, a mental effect or thought that Peirce calls “interpretant” and which gives the sign significance or meaning, becoming in turn a sign in a dynamic process ad infinitum. The “interpretant” is “that which the Sign produces in the Quasi-mind that is the Interpreter by determining the latter to a feeling, to an exertion, or to a Sign, which determination is the Interpretant.” (CP: 4.536, 1906) Peirce makes clear that the Interpretant is not a person/agent but a result of sign semiosis. The Interpretant approximates the object–sign relationship through a representation informed by the object and directly brought about by the sign (CP: 5.475, 1905; CP: 5.484, 1907; CP: 6.347, 1909). Thus, a signing action can be itself a sign for the next, since signs have their own agentive properties, becoming a “more developed sign” (CP: 2.228, ca.1897). Thought [. . .] is in itself essentially of the nature of a sign. But a sign is not a sign unless it translates itself into another sign in which it is more fully developed. Thought requires achievement for its own development, and without this development it is nothing. (CP 5.594, 1903)

In his paper “Questions Concerning Certain Faculties Claimed for Man,” published in Journal of Speculative Philosophy 2 (1868: 103–114), Peirce tried to identified the major issues the cognitive sciences face in determining what is consciousness and how it is connected to perception and mental representations or cognition. Consciousness presupposes a certain awareness, while unconsciousness, present in dreams, intuitions, and certain emotional states, points to something beyond the level of self- awareness: Every cognition involves something represented, or that of which we are conscious, and some action or passion of the self whereby it becomes represented. The former shall be termed the objective, the latter the subjective, element of the cognition. The cognition itself is an intuition of its objective element, which may therefore be called, also, the immediate object. The subjective element is not necessarily immediately known, but it is possible that such an intuition of the subjective element of a cognition of its character, whether that of dreaming, imagining, conceiving, believing, etc., should accompany every cognition. The question is whether this is so. . . Thus, the arguments in favor of this peculiar power of consciousness disappear, and the presumption is again against such a hypothesis. (CP 5.238-5.243, 1868)

Peirce’s classification of sign relations moves from monadic relations, expressing quality, to dyadic, expressing reaction and sometimes resistance, to triadic (symbolic) relations involved in representation and mediation. The category of “firstness” relies on qualities (he uses the term “qualisign” to a primordial quality, which can be a simple feeling that stages the possibility of an object functioning as a sign). Qualisigns are, in some sense, primary or irreducible, leaning toward physicality, and dependent on emotional content. Peirce emphasized that a quality cannot act as a

1226

A. Lo´pez-Varela Azcárate

sign until it is embodied, that is, materially realized; but embodiment has nothing to do with its character as a sign. General laws are embodied in actual things or events (themselves manifesting a range of possible qualities). Thus, the fact that something is interpreted as a sign depends on embodiment. The kind of sign it is, however, depends on the mode of being of whatever is, in a given instance, functioning as a sign. Peirce’s well-known classification of signs -icons, indexes, and symbols is based on how signs are related to their objects. An icon signifies its object by virtue of shared qualities, an index by virtue of a causal relation, and a symbol by virtue of an action ruled by a norm or habit, without resemblance or real connection to the denoted object but agreed by convention; by virtue of the fact that it will be interpreted to do so. It denotes by virtue of its “interpretant” (“the idea to which a sign gives rise,” CP 1.339, 1885; emphasis added; “give rise” is equal to the concepts of “enaction” and “emergence”). The process of “secondness” initiates relations between domains and is essentially dyadic. “Sinsigns” are signs that consist in reaction/resistance, or an actual singular thing, occurrence, or fact. Peirce also held that an index can be a general thing (not only singular; the etymology of “sêma” points in this direction). For instance, a symptom of a disease, a label, a diagram (which can be both iconic and indexical), a proper name, a pronoun, etc. Indexes or pointers make connections by means of spatiotemporal proximity or contiguity, crucially bound up with the situation or context. If interpreted as linguistic signs in triadic symbolic relations, pointers become personal pronouns (I, you, we, they, etc.), deictic adverbs (here/ there, now/then), demonstratives (this/that), and grammatical categories of tense and aspect, all of which that are situationally contingent. The cognitive operations originating in bodily experience pass through processes of metaphorical and metonymic projections based on recognition of patterns presented in experience (Lakoff and Mark Johnson 1980). Once corresponding qualities between material and mental spaces have been mapped, their integrative projections yield symbolic signs. As structures become more symbolic, their connection with bodily experience turns more indirect and cultural particularities emerge. Particularities are filled in with salient “situated” cultural content within specific population groups. Conceptual integration includes out-of-awareness forms of cognition, such as tacit knowledge of what is possible, permissible, and acceptable within a community. Three fundamental relations occur between the representamen and its object: iconicity, indexicality, and symbolicity. These relations are based on fundamental cognitive operations. Iconicity seems to be a single property, related to varying grades of semblance/similarity with what is perceived. An icon is a sign that denotes its object by virtue of a quality that resembles or imitates its object. Iconic signs do not possess the properties of the object but reproduce some conditions of common perception. Depending on material aspects, Peirce established three types: (a) the image, which depends on a simple quality; (b) the diagram, whose internal relations, mainly dyadic or so taken, represents by analogy; and (c) the metaphor, which represents by drawing a parallelism to something else; for instance, an abstraction represented by physical resemblance.

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1227

Indexicality, is related to a “ground” of contiguity (CP 2.228, ca.1897) (i.e., conceptual proximity) and to factorality (i.e., metonymic part/whole relations) since experience can occur in terms of parts and totalities. An index denotes its object by virtue of an actual connection, irrespective of interpretation. In contrast to the icon, which has only a “ground” for denotation of its object, and in contrast to the symbol, which denotes by an interpretive habit or law, an index compels attention without conveying information about its object. This form of semiotic “ground” or relevance applies not just to indices but also to icons, for it is endorsed by perceptual aspects such as those mentioned above (similarity, contiguity, and factorality). The distinction of “ground” in Peirce’s definition is crucial because it recognizes that the sign perceived is relevant to its semiotic object only in a particular respect or capacity. Peirce used the “immediate” and “dynamical” interpretants to signal different levels of interaction and partial aspects of the objects (CP 8.183, 1909). The notion of “ground” is also important from the perspective of evolutionary anthropology, for instance, in that it emphasizes that what is cognized is a thematic aspect of what preceded it (whether a physical thing or a previous thought). In other words, while some signs are readily perceived, others require prior familiarity with their sign function, often established in social communities (not necessarily just human). Finally, symbolicity, refers to shared conventionalities and habits established in languages (whether human discourses, visual language, mathematical and artificial, code, and so on). For instance, iconicity (resemblance relation) appears as an instant recognition of something; indexicality (proximity/contiguous) incorporates additional aspects in a dialogical relation between the sign (“representamen”) and its object, presupposing earlier experiential connections; symbolicity (habit) is activated when common “ground” is articulated in the construction of signs in a given context. The leading principle is that the “habit” of thought determines the passage from a premise to a conclusion (CP 3.160, 1880) even in out-of-awareness circumstances. Thus, Peirce explained that habits rely on processes of inference that make sense of information by drawing conclusions via various mechanisms. Induction (making generalizations based on specific observations or experiences); deduction (reaching a conclusion based on logical reasoning); abduction (making an educated guess based on the best explanation for a set of observations); causal inference (making a conclusion about a cause-and-effect relationship); and analogical inference (making a conclusion based on similarities between things or occurrences). Accordingly, some of these processes of inference making can be performed by nonhumans. In line with Peirce’s idea about things in the world creating habits by means of repetition, the founder of cybernetics, Norbert Wiener, speaks of the recurrence of patterns becoming internally meaningful even out of awareness. He explains that “organism is opposed to chaos, to disintegration, to death, as message is to noise. To describe an organism, we do not try to specify each molecule in it, and catalogue it bit by bit, but rather to answer certain questions about it which reveal its pattern; a pattern which is more significant and less probable as the organism becomes, so to speak, more fully an organism” (Wiener 1950/1989: 95). He goes on to add that “we

1228

A. Lo´pez-Varela Azcárate

are not stuff that abides, but patterns that perpetuate themselves. A pattern is a message (Ibid., 96). Thus, any information carried by a set of messages is a measure of organization in an organism, whether this organism is human or nonhuman. This measure of organization or habit shows forms of agency that might not be directly linked to consciousness and intentional action. For instance, exploring nonhuman agency in trees, Owain Jones and Paul Cloke (2008) speak of several forms of agency: “agency as routine action,” associated with ongoing process of life existence; “agency as transformative action,” involving natural fields of relations often bound up with geo-transformation; “agency as purposive action,” beyond human intentionality, for nonhumans can influence courses of action through the encoded blueprint present in their DNA. Finally, “agency as non-reflexive action,” recognizing that nonhumans have the capacity to engender affective and emotional responses from humans. In this sense, Wiener explains that making meaning out of signals and signs is important as a stage in the continuous process of being and acting in the world, and that “to be alive is to participate in a continuous stream of influences from the outer world and acts on the outer world, in which we are merely the transitional stage” (Wiener 1950/1989: 122). Like Peirce, these authors consider that only final causation, which involves complex semiosis, yields the human “experience of agency,” which relies on self-consciousness, and is different from simple “agency.”

From Habits to Distributed Agencies In the nineteenth century, the debate on vitalism included scientists, philosophers, and artists. Samuel Butler, mostly known for his work as novelist, published four books on evolution in which he harshly criticized Charles Darwin’s theory. Without scientific evidence from the existence of DNA, which was only discovered in the 1950s by James Watson and Francis Crick, Darwin’s model involved a basic form of genetics in which variation was presumed to contain random aspects combined with the notion of natural selection driven by adaptive behavior to the environment. According to Butler, in the case of complex animals and humans, there were aspects of inheritance that were unaccounted for in Darwin’s theory. In particular, Butler was adamant to explain that memory and knowledge were also part of directed ongoing evolution, which could not depend on fortuitous events driving change. He argued that the process of adaptive behavior was related to repetitions creating habits (where a habit could be a perception, a conception or thought, and an action). He explained that these habits lie in the deepest parts of the biological system of organisms, thus remaining unconscious or “out-of-awareness.” Butler insisted that the better an organism “knows” something, the less conscious this knowledge becomes. In other words, there is a mechanism in cognition that operates on a sort of economy principle; pushing conscious memories to the unconscious level in a similar way to the way perception works. For instance, aspects of size, depth, length, etc., are often beyond the level of conscious awareness when it comes to processing three-

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1229

dimensional visual images. In this way, organisms can know and do things without consciously thinking about them: “No organism can afford to be conscious of matters with which it could deal at unconscious levels” (Bateson 1972: 152). As a vitalist, Butler also made claims that blurred the distinction between organic and inorganic, anticipating contemporary findings as well as some of the issues involved in posthumanism. In his treaty Unconscious Memory, Butler writes The only thing of which I am sure is, that the distinction between the organic and inorganic is arbitrary; that it is more coherent with our other ideas, and therefore more acceptable, to start with every molecule as a living thing, and then deduce death as the breaking up of an association or corporation, than to start with inanimate molecules and smuggle life into them; and that, therefore, what we call the inorganic world must be regarded as up to a certain point living, and instinct, within certain limits, with consciousness, volition, and power of concerted action. (1880: 23)

It is clear that Butler sees inanimate matter as capable of “concerted action,” that is, of some form of agency. In his closing chapter, Butler again asserts that “We should endeavour to see the so-called inorganic as living in respect of the qualities it has in common with the organic, rather than the organic as non-living in respect of the qualities it has in common with the inorganic” (1880: 275). Posthumanist and environmentalist scholars Serenella Iovino and Serpil Oppermann discussed the distribution of agency in their 2014 volume Stories Come to Matter, defending that Agency assumes many forms, all of which are characterized by an important feature: they are material, and the meanings they produce influence in various ways the existence of both human and nonhuman natures. Agency, therefore, is not to be necessarily and exclusively associated with human beings and with human intentionality, but it is a pervasive and inbuilt property of matter, as part and parcel of its generative dynamism. From this dynamism, reality emerges as an intertwined lux of material and discursive forces, rather than as complex of hierarchically organized individual players. (2014: 3)

As aforementioned, this distribution of agencies has a basis in Peirce’s concept of “synechism,” “the tendency to regard everything as continuous” (CP 7.565, ca.1892), which presupposes “a continuity between the characters of mind and matter” (CP 6.277, 1893). Peirce explains that This hypothesis might be called materialistic, since it attributes to mind one of the recognized properties of matter, extension, and attributes to all matter a certain excessively low degree of feeling, together with a certain power of taking habits. But it differs essentially from materialism, in that, instead of supposing mind to be governed by blind mechanical law, it supposes the one original law to be the recognized law of mind, the law of association, of which the laws of matter are regarded as mere special results. (CP 6.277, 1893)

Peirce’s ideas on the “Quasi-mind” extend not only to animals, but also to inanimate things like crystals, pointing to the questions discussed in the posthuman debate today. He argues that

1230

A. Lo´pez-Varela Azcárate

Thought is not necessarily connected with a brain. It appears in the work of bees, crystals and throughout the purely physical world; and one can no more deny that it is really there, than that the colours, the shapes, etc., of objects are really there. [. . .] Not only is thought in the organic world, but it develops there. But as there cannot be a General without Instances embodying it, so there cannot be thought without Signs. We must here give ‘Sign’ a very wide sense, no doubt, but not too wide a sense to come within our definition. Admitting that connected Signs must have a Quasi-mind, it may further be declared that there can be no isolated sign. (CP: 4.551, 1906)

Some of the questions that can be raised from Peirce’s assumptions include the following: How do the “couplings” between organisms, entities, and their environments take place? How can they be scaled in order to explore a potential differentiation in the agentive qualities of humans and nonhumans, animate and inanimate? Where are these agential structures located? Are they in the brain as well as embodied, embedded, and enacted from other encounters and, if so, can they be considered cognitive “extensions”? Finally, where does intermediality come into this discussion? As mentioned, Peirce explained that cognitive functions are simultaneously materialized in the brain and in the material artifacts used in meaning making. Peirce’s pragmatism defines semiotics as “the action of signs” (Peirce, CP: 5.473, 5.484, 1907), and explains that signs are both generated and generative, and that something (potentially anything) acquires the function of sign in the process of meaning making (CP 2.59). As aforementioned, “the action of signs” only “enacts” some aspects in a particular space-time, within the continuum of experience. This continuum problematizes the relationship between interiority and exteriority with respect to the locus of the mind. Peirce writes: The psychologists undertake to locate various mental powers in the brain; and above all consider it as quite certain that the faculty of language resides in a certain lobe; but I believe it comes decidedly nearer the truth (though not really true) that language resides in the tongue. In my opinion, it is much more true that the thoughts of a living writer are in any printed copy of his book than that they are in his brain. (CP 7.364, 1902)

He goes on to add: A psychologist cuts out a lobe of my brain (nihil animale me alienum puto) and then, when I find I cannot express myself, he says, 'You see your faculty of language was localized in that lobe.' No doubt, it was; and so, if he had filched my inkstand, I should not have been able to continue my discussion until I had got another, Yea, the very thoughts would not come to me. So my faculty of discussion is equally localized in my inkstand. It is localization in a sense in which a thing may be in two places at once. (CP 7.366, 1902)

The quotations above show that Peirce does not merely understand cognition as embedded, involving a combination of inter- and intra-actions. According to him, representations are not necessarily autonomous or internal. Cognition is constituted by inferences that rely on external elements like the act of writing (CP 7.364-6, 1902). The North American pragmatist considers that the mind (and not only the human mind) is “enactive” and “extended.”

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1231

Continuity and generality are the same thing [. . .] Time and space are continuous because they embody conditions of possibility, and the possible is general, and continuity and generality are two names for the same absence of distinction of individuals. (CP 4.172, 1897)

According to Peirce, as experience and learning merge, becoming “habits” embedded in particular contexts, it becomes almost impossible to establish a vertical hierarchy of influence strata because the entire exchange occurs in a continuum that also involves the materiality of things: “We ought to suppose a continuity between the characters of mind and matter” (CP 6.277, 1893). To bring the discussion closer to intermediality, I remind my readers that media configurations carry implicit interventions (agencies). The “qualifying aspects” that Elleström mentions are determined by the origin, delimitation, and use of media in specific sociocultural circumstances. In the case of human aesthetic regimes, they impose a certain order on the sensible so that artistic practices can be seen as intervening politically at the conceptual level imposing certain distributions of power relations and agencies. Indeed, creativity is an expression of agency through a process of recombining and altering elements. Finally, a technical medium includes any object or body that realizes, mediates, or displays both basic (often given by nature), as well as what McLuhan termed “media extensions,” (1964) tools or complex machines artificially created to extend interactional capacities. However, there are important differences in the machines we use. Analogue and digital machines enable different forms of sign making.

Intermedial Semiotics. Analogue Versus Digital As seen, semiotics or “the action of signs” (Peirce, CP: 5.473, 5.484, 1907) has recognized the inter-actions and intra-actions of anything acting as a sign; present also within the framework of the 4Es (embodied, embedded, enacted, extended). At a systemic scale, semiotic interactions include specific media channels, with the term “medium” understood in its broadest sense, including bio-entities along with artifacts and technologies that can be material, and thus physically perceived by humans and animals, but also digital, which can lie beyond the scope of perception. Naturally, depending on the materiality, there may be different types of semiotic relationships linked to diverse forms of agency. We could define a medium as a system (or assemblage) of components (natural or technological) used to store information and mediate its propagation. Information is perceived by another entity (human or otherwise) by means of a sort of sensing apparatus, which also depends on the materiality of the medium and on the channels used for the transfer of information. A medium can appeal to optics (the specific wavelength of light can pass through some materials such as water, air, glass, etc.); to sound (by means of sound waves, the vibration of matter, and so on); to individual sensorial organs in bio-entities or sensors in machines. Finally, the transfer can be done synesthetically across sensing domains (i.e., sound waves can become

1232

A. Lo´pez-Varela Azcárate

vibration, thus rising the temperature of a given material, which can impact on tactile experience). Bio-entities experience the different materials and the information they carry in terms of analogue forms. As mentioned, the complex relation between perception and cognition is mediated by “the action of signs,” so that information becomes meaningful to an interpretant. However, it is important to note, once again, that signs do not represent reality in their totality. They only “bring forth” or “enact” some aspect of reality to a particular “interpretant” in a given moment and time. Semiotician Winfried Nöth highlights the fact that “the agent in the process of semiosis in which the sign creates an interpretant, is the sign, not the addresser, and the agency of the sign is one of final causality: it is the purpose of the sign to create an interpretant.” (2009: 19) Signs are not mere instruments, but semiotic agents acting with a semiotic autonomy of their own. They mediate relations between things in the world and operate by final causality, even though they cannot do without efficient causes to convey their messages. Final causality involves triadic interaction: “it is the long term causality of purposes, intentions, ideas, signs, and general laws, all of which belong to the Peircean category of thirdness” (Nöth 2009: 19). Noting that the term “organ,” which refers to bodily parts, comes from the Greek form órganon, meaning “tool,” Nöth insists: “the object and the interpretant of a machine qua sign are the ways in which the machine has been produced and used and in which it may be used in the culture to which it belongs” (Nöth 2010: 47). This is relevant to Elleström’s notion of “qualified media.” Objects, tools, and machines have practical functions when used to transform directly the environment, but they also have semiotic functions in individual indirect interaction with the environment by means of them: “serving a practical purpose thus does not preclude an object from serving semiotic purposes at the same time” (Nöth 2010: 48). When the medium is a machine, we need to differentiate between analogue and digital. In analogue machines, information is converted into pulses of different amplitude that can be represented as a continuous wave on a graph. Thus, analogue media carry information in the form of signs that are depictable, that is, they can have physical equivalent and they are geometrically continuous. Examples include inscribed media (letters, and other types of inscriptions, including cuneiform, hieroglyphs, etc.); print media (book, newspaper, etc., after the advent of the printing press); broadcast media (radio, television); telegraph, telephone; sonic recording (gramophone and phonograph records, vinyl discs, audiocassettes); analogue image processing (engraving, daguerreotype, photography); finally moving pictures or early cinematography, and video (created by scanning an electron beam across a phosphor). Until the invention of digital computers at the beginning of the twentieth century, analogue instruments, including some ancient computing tools like the Antikythera mechanism, dominated the fields of science and technology. The material distribution of the sensible in a particular medium, whether this is a natural channel (i.e., the human vocal tract), or a technical apparatus (i.e., a

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1233

recording machine), shapes the way signals are received and become meaningful signs through sense experiences (in Firstness and Secondness) as well as sociocultural conditions of reception (laws, habits, in Thirdness). Even “What we call matter is not completely dead, but is merely mind hidebound with habits.” (CP: 6.158, 1892). Several issues are brought to the fore here. First, the inquiry as to whether or not events in the world exceed human intentionality; in other words, if there are inherent potentials in specific material situations that become implicated in their unfolding, creating enactions beyond human control. Second, the issue of how Intermedial Studies are involved in the discussion of a possible scaling of agentive capacities beyond the human. After all, intermediality, as defined by Elleström, involves the combination of biological perceptual modalities, enhanced by technical media combination, as “qualified” by sociocultural requirements in a particular spatiotemporal span. Thus, the question posed at the beginning of this chapter as to whether or not intermediality could be brought further out of bounds to interrogate interrelations between humans and nonhumans, subjects and objects, animate and inanimate, active and inert. Transgressing these boundaries means, to a certain extent, to question a certain conception of the human, hence the prefix “post” in posthumanism. In Robert Pepperell’s words: The upshot is that individual humans in the sense of isolated, separate objects do not really exist, other than in our imaginations. What exists instead are non-contained beings who, in numerous ways, are distributed far beyond their local space and time, caught in an infinite chain of events without beginning or end. Each act I make, whether trivial or expansive, has further consequences that will ripple through infinity, just as each act is the extension of an indeterminate number of prior events. [. . .] The result is that our conception of human beings must include our wider cultural environment as well as our physical structure, and in particular, our technological environment, not just as an external adjunct to the human condition but as an inherent part of what constitutes us in the first place. To put it succinctly: Humanists might regard humans as distinct beings, in an antagonistic relationship with their surroundings. Posthumanists, on the other hand, regard humans as embodied in an extended technological world. (2005: 34)

Pepperell’s assertion comes close to Peirce’s “continuum” of experience, to Latour’s Actor-Network-Theory and, more recently, to the “Extended Mind” hypothesis. To quote Peirce once again, sign relations include an object that determines the sign in a process called “firstness,” which in turn determines another more developed sign or “interpretant” in “secondness.” The “interpretant,” fulfilling its function as sign of the object, determines a further “interpretant sign” in “thirdness.” This is a process logically structured to perpetuate itself. The “interpretant” is a meaning maker only in the sense that it engages in sense making via dynamic “couplings,” to use Andy Clark’s terms. The “interpretant” is itself a sign, in a process representing an object, and carries agency in the sense that it can be understood as a sign’s effect on the mind, or on anything that acts like a mind, that Peirce calls a Quasi-mind. A signing action can become itself a sign for the next, becoming a “more developed sign” (CP 2.228, ca.1897).

1234

A. Lo´pez-Varela Azcárate

Extended Cognition and the Problem of Agency Distribution In a similar way to the Extended Mind Hypothesis, developed by Andy Clark and David Chalmers (1999), Material Engagement Theory, an approach developed by cognitive archaeologist Lambros Malafouris, draws upon Peirce as well as upon enactivism in order to support the idea that engagement with material things/signs brings forth semiotic activity. MET agrees with Chalmers and Clark’s theory but tries to go beyond, claiming that is “simply an expansion of the ontological boundaries of the res cogitans rather than the dissolution of those boundaries altogether” (Malafouris 2013: 65). Malafouris refers to intermedial relations in the following way: The functional anatomy of the human mind (which includes the whole organism, that is, brain/CNS and body) is a dynamic bio-cultural construct subject to continuous ontogenetic and phylogenetic transformation by behaviourally important and socially embedded experiences. These experiences are mediated and sometimes constituted by the use of material objects and artefacts (e.g., the blind man’s stick) which for that reason should be seen as continuous, integral, and active parts of the human cognitive architecture. (Malafouris 2013: 244)

As to agency, rather than seeing it as the result of prior intention, Malafouris conceives it in a similar way to Karen Barad and other posthumanists, as the emergent product of material and semiotic activity: “meaning is not the product of representation but the product of a process of conceptual integration between conceptual and material domains” (Malafouris 2013: 90). In Theory as Praxis: The Poetics of Electronic Textuality, Johanna Drucker had also insisted that we have a plastic mind inextricably intertwined with the plasticity of forms that we make so that “things actively participate in human cognitive life or that human thinking is better described as thinging. We think with and through things, not simply about things” (2019: n.p), Drucker believes that “making things [. . .] pushes the horizons of one’s understanding” (2002, 684). So how does the functioning of our purely material brain generate phenomenal experiences and their subjective and agentive feel? Are we phenomenally aware of more things than what we can consciously cognitively access? Posthumanism is an attempt to decouple agency from human consciousness and to modulate intentionality. Bodies are not inert matter and have the power to affect others and to be affected by them. In this regard, artifacts have some degree of “operational intentionality” (Gallagher 2017: 67) in that they can establish “couplings” with other material bodies. Thus, “while agency and intentionality may not be properties of things, they are not properties of humans either; they are the properties of material engagement, that is, of the grey zone where brain, body and culture conflate” (Malafouris and Knappett 2008: 22). Accordingly, agency can be de-coupled from subject–object distinctions and dissociated from intentionality as a unique human property. Appealing to John Searle’s (1969) distinction between “prior intention” in premeditated or deliberate action, and “intention in action,” where no intentional state is formed in advance of the action, Malafouris notes that

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1235

in “intention in action” the internal intentional state and the external movement become indistinguishable, but still have a pragmatic effect in the world. This shows that agency might be an emergent product of mediated activity in material engagement. Agency is a temporal and interactively emergent property of activity not an innate and fixed attribute of the human condition. The ultimate cause of action in this chain of micro and macro events is none of the supposed agents, humans or nonhumans; it is the flow of activity itself. (Malafouris and Knappett 2008: 35, emphasis added)

Returning to Nöth’s notion of complex machines, instruments that convey information via digital artifacts connected to them are strong candidates for extended cognition where the integration between the biological and the digital creates concerns regarding agency and consciousness (i.e., AI). Indeed, the use of contemporary technology involves a chain of human agency as well as nonhuman agents. Algorithms attached to search engines provide information of the places humans visit online to help target advertising to users’ interests. In these cases, agency is distributed across intermedial encounters between humans and machines. The Belgian philosopher of technology, Mark Coeckelbergh, explains that consciousness aligns with intentionality only in individuals “able to explain decisions to someone who rightfully and reasonably asks ‘Why?’” (Coeckelbergh 2020: 2061). In The Fragile Species, Lewis Thomas claimed that this sense of what is reasonable, useful, and has value drives human social relations, having an impact on how we relate to the rest of the world. One human trait, urging us on by our nature, is the drive to be useful, perhaps the most fundamental of all our biological necessities. We make mistakes with it, get it wrong, confuse it with self-regard, even try to fake it, but it is there in our genes, needing only a better set of definitions for usefulness than we have yet agreed on. (Thomas 1992: 26)

According to philosopher Shaun Gallagher (2006, 2014, 2017), intentional agency requires a self who masters explanation and, conscious of the actions performed, tries to justify them by telling others about them expecting a sense of worth and validation. Gallagher calls it the “narrative self” and explains that some animals possess only partial consciousness as “minimal selves.” Responsibility and value attribution is given in language and communication, with “the question who is responsible to whom” (Coeckelbergh 2020: 2061), and adds that “the category of responsibility patients does not necessarily exclude nonhumans” (Ibid., 2062). This discussion is central in ethics because many believe that phenomenal consciousness is the main source of responsibility, and that only phenomenally conscious or “sentient creatures” possess moral status. Berenike Jung, Klaus Sachs-Hombach, and Lukas R.A. Wilde (2021) also note that medially distributed agency is essential in contemporary intermedial processes in order to determine the mutual influences among system components (modalities). Under the conditions of “post-digitality,” the assumption is that there are hardly any “non-digital” media areas left, and the potential for action can also be attributed to

1236

A. Lo´pez-Varela Azcárate

newly created nonhuman entities. This is the case of “conversational digital agents,” that is, AI programs that use natural human language in spoken form to respond to users’ inquiries (i.e., GPS, ChatGPT, etc.; see Lotherington in Elleström 2021: 217–236). In these cases of digital intermediality, chains of semiotic action are increasingly characterized by available or limited access to resources as well as the affordances of digital media configurations. Thus, the development of cybernetics has a direct impact upon how we understand intermedial encounters beyond technical media. To close this inquiry into the possible vindication of a view of agency that is distributed between humans and nonhumans, let us return to our first tenet that a theorization of process ontology may help envision the need for systemic process epistemologies that, with a proper body of practices, might align Intermedial Studies with Sustainable Development. The possibility of such radical revision raises the problem of determining possible alternative sources for an ethics of responsibility, questioning the claim of the superior value of human consciousness and action on the material world. For what is worth, if consciousness and responsibility are only human traits, the need arises to have them aligned with the value of nonhuman agents. After all, sustainability is a term derived from the Latin verb sustentāre, which means to maintain, support, uphold, as well as endure. As Norbert Wiener wrote in 1950, To those of us who are aware of the extremely limited range of physical conditions under which the chemical reactions necessary to life as we know it can take place, it is a foregone conclusion that the lucky accident which permits the continuation of life in any form on this earth, even without restricting life to something like human life, is bound to come to a complete and disastrous end. Yet we may succeed in framing our values so that this temporary accident of living existence, and this much more temporary accident of human existence, may be taken as all-important positive values, notwithstanding their fugitive character. (Wiener 1950/1989: 40; emphasis added)

On March 16, 2016, the Open Culture blog published a post with a photograph and a small commentary entitled “The “Shadow of a Hiroshima Victim, Etched into Stone Steps.” The note explains that the shadow was left by a person, who seems to be holding a walking stick, on the morning of August 6, 1945, at the entrance of the Sumitomo Bank in Hiroshima, Japan. Seconds later, one of the atomic bombs dropped over Japan exploded. The person vanished, leaving the body silhouette inscribed on the steps as a marker and last form of agency. And one wonders, is it the agency of the person or of the bomb?

Conclusions This chapter has argued that concern with intermediality entails complex relations, both physical and conceptual (semiotic), between actants (human and nonhuman) and material aspects –tools, artifacts, and technology– within a given environment. The diverse forms of materiality and media modalities determine these relations. For instance, music historians and music theorists, like experts on theatrical and

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1237

cinematographic representation, have acknowledged that there are aspects of performance that cannot be reproduced in analogue musical scores or in play scripts. Similar restrictions apply to turning them digital. The particular media distribution, for example, on a comic strip in a graphic novel, or the more complex intermedial environment of computer screens, allow specific forms of inscription, perception, and cognitive engagement and agency. In terms of intermediality, “the disruption of cognitive import is amplified when the capacity for participant agency is factored in, as we make unpredictable interventions into the processes of mediation” (Elleström 2021b: 105). The development of cybernetics has confirmed the long beheld notion that technology has a direct impact upon sign production, distribution, and reception and, thus, upon the entire system of human perception, cognition, and communication. The advent of digital machines brought about a revolution in the way information is encoded, converted into data units, bits at the lowest level of machine code. All areas of human activity have been affected. Most analogue information recorded for years in various formats is still in the process of digital conversion. New forms of online inscription have emerged, with electronic literature, Net.Art, or computer games merging human language and machine code. With the development of the World Wide Web, the degree of interactivity has continued to increase across all kinds of screen devices (computers, smartphones, tablets, augmented reality systems, cloud storage and, more recently, the multiverse). Networked media platforms are constantly inducing novel forms of engagement with online media products whose materially is hidden in the hyperspace. There is also the question of the agencies of nature, the organic and the inorganic. How the organic is made up of atoms and molecules of inorganic matter; clay particles encapsulated in vesicles that contribute to the concentration of RNA subunits, according to 2009 Nobel Prize winner Jack Szostak. Flesh “which holds the dust/That measures all our time; which also shall/Be crumbled into dust” again (George Herbert “Church Monuments,” The Temple 1633); dust that allow us “To see a World in a Grain of Sand/And a Heaven in a Wild Flower/Hold Infinity in the palm of your hand” (William Blake Auguries of Innocence 1863). In a world that promotes assertion and empowerment, the future of Intermedial Studies passes by a reassessment of agencies that bring forth the worth and value of care; of the kind of semiotic relations that operate in a mind-matter continuum where sustainability and ecological awareness unfold on progressively from the intra- to the inter-relations in ever-growing scales. The Cartesian dualist view that justified intentional action upon the world in terms of the supremacy of human consciousness manifested not just actions, but also thoughts, ideas, and even moral purpose requires a radical reexamination. Advances in semiotics as well as the cognitive sciences show that things, whether natural or artificial, shape the human mind through its embodied extensions. Multiple aspects related to the environment, such as the physical ecosystem, climate, habitat conditions and so on, have an impact on human lives. In the contemporary scenario of existential threat, human agency needs to hold back in humbleness, giving way to the material multiplicity of nonhuman agencies. Thus, the future of Intermedial Studies is to become a sphere of knowledge metamorphically out of bounds.

1238

A. Lo´pez-Varela Azcárate

References Alaimo, Stacy. 2010. Bodily natures: Science, environment, and the material self. Bloomington Indiana, Indiana University Press. Alcock, John. 2013. Animal behavior: An evolutionary approach. Sunderland, Massachusetts, Sinauer Associates/Oxford University Press. Baars, Bernard J., and Natalie Geld, eds. 2021. On consciousness: Science & subjectivity. Oxford. Massachusetts. The Nautilus Publishing Company. Barad, Karen. 2003. Posthuman performativity: Toward an understanding of how matter comes to matter. Sings: Journal of Women in Culture and Society 28 (3): 801–831. Bartram, William. 1791. Travels. Philadelphia: James & Johnson. Internet Archive.org. Bateson, Gregory. 1972. Steps to an ecology of mind. San Francisco. California, Chandler Publishing. Bennett, Jane. 2010. Vibrant matter: A political ecology of things. Durham: Duke University Press. Braidotti, Rosi. 1994. Nomadic subjects. Embodiment and sexual difference in contemporary feminist theory. Cambridge: Columbia University Press. ———. 2013. The posthuman. Cambridge: Polity Press. Bruhn, Jørgen, and Beate Schirrmacher, eds. 2022. Intermedial studies. An introduction to meaning across media. Oxon: Routledge. https://doi.org/10.4324/9781003174288. Bruhn, Jørgen, Niklas Salmose, Beate Schirrmacher, and Emma Tornborg. 2022. Truthfulness and truth claims as transmedial phenomena. In Intermedial studies. An introduction to meaning across media, 225–254. Oxon: Routledge. https://doi.org/10.4324/9781003174288. Butler, Samuel. 1880. Unconscious memory. London: David Bogue. Caracciolo, Marco. 2022. Storying the Anthropocene: Narrative challenges and opportunities in times of climate change. In The Routledge companion to narrative theory, ed. Paul Dawson and Maria Mäkelä, 542–555. New York: Routledge. https://doi.org/10.4324/9781003100157-50. Caracciolo, Marco, Kristin Ferebee, Heidi Toivonen, Gry Ulstein, and Shannon Lambert. 2022. Climate fiction: A posthumanist survey. Interférences Littéraires 27 (2): 6–21. Castañares, Wenceslao. 2012. Lines of development in Greek semiotics. Special issue semiotics of world cultures, ed. Asunción López-Varela. Cultura. International Journal of Philosophy of Culture and Axiology 9(2): 33–44. http://connection.ebscohost.com/c/articles/85244953/linesdevelopment-greek-semiotics Clark, Andy, and David Chalmers. 1999. The extended mind. Analysis 58 (1): 10–23. Coeckelbergh, Mark. 2020. Artificial intelligence, responsibility attribution, and a relational justification of explainability. Science and Engineering Ethics 26: 2051–2068. https://doi.org/10. 1007/s11948-019-00146-8. Cohen, Jonathan D., and Jonathan W. Schooler. 1997. Scientific approaches to consciousness. Routledge Psychology Press. https://doi.org/10.4324/9781315806563. Damasio, Antonio. 1999. The feeling of what happens body and emotion in the making of consciousness. New York: Harcourt Brace & Company. Drucker, Johanna. 2002. Theory as praxis: The poetics of electronic textuality. Modernism/Modernity 9 (4): 683–691. Elleström, Lars. 2021a. Beyond media borders. Intermedial relations among multimodal media, vol. 1–2. Berlin/Heidelberg, Germany, Springer International Publishing. https://doi.org/10. 1007/978-3-030-49679-1. https://doi.org/10.1007/978-3-030-49683-8. ———. 2021b. The modalities of media II: An expanded model for understanding intermedial relations. In Beyond media borders, ed. L. Elleström, vol. 1. Palgrave Macmillan. https://doi. org/10.1007/978-3-030-49679-1_1. Elleström, Lars, Massimo Fusillo, and Mattia Petricola. 2020. Everything is intermedial: A conversation with Lars Elleström. Between 10 (20). https://doi.org/10.13125/2039-6597/4443. Foucault, Michel. 1970. The order of things: An archaeology of the human sciences. New York: Pantheon Books.

48

Posthuman Intermedial Semiotics and Distributed Agency for. . .

1239

———. 2006. The narrative alternative to theory of mind. In Radical enactivism: Intentionality, phenomenology, and narrative, ed. Richard Menary, 223–229. Amsterdam: John Benjamins. ———. 2014. Pragmatic interventions into enactive and extended conceptions of cognition: Pragmatic interventions into enactive and extended conceptions of cognition. Philosophical Issues 24 (1): 110–126. https://doi.org/10.1111/phis.12027. ———. 2017. Enactivist interventions: Rethinking the mind. Oxford: Oxford University Press. Gibson, James. 1977. The theory of affordances. In Perceiving, acting, and knowing: Toward an ecological psychology, ed. Robert E. Shaw and John Bransford, 67–82. Hillsdale: Lawrence Erlbaum Associates. Haraway, Donna. 1991. A cyborg manifesto: Science, technology, and socialist-feminism in the late twentieth century. In Simians, Cyborgs and women: The reinvention of nature, 149–181. New York: Routledge. ———. 2007. When species meet. Minneapolis, Minnesota, University of Minnesota Press. Harley, Trevor. 2021. The science of consciousness: Waking, sleeping and dreaming. Cambridge: Cambridge University Press. https://doi.org/10.1017/9781316408889. Hayles, N. Katherine. 1993. Virtual bodies and flickering signifiers. Vol. 66, 69–91. Boston. Massachusetts, MIT Press. ———. 1999. How we became posthuman: Virtual bodies in cybernetics, literature, and informatics. Chicago/London: Chicago University Press. ———. 2002. Deeper into the machine: Learning to speak digital. Computers and Composition 19 (4): 371–386. Iovino, Serenella, and Serpil Oppermann. 2014. Material ecocriticism. Bloomington, Indiana, Indiana University Press. Jones, Owain, and Paul Cloke. 2008. Non-human agencies: Trees in place and time. In Material agency towards a non-anthropocentric approach, ed. Carl Knappett and Lambros Malafouris. Berlin/Heidelberg, Germany, Springer. Jung, Berenike, Klaus Sachs-Hombach, and Lukas R.A. Wilde. 2021. Agency postdigital. Köln, Germany, Herbert von Halem Verlag. Karban, Richard. 2015. Plant sensing and communication. Chicago, Illinois, University of Chicago Press. Lagercrantz, Hugo, and Jean-Pierre Changeux. 2009. The emergence of human consciousness: From fetal to neonatal life. Pediatric Research 65: 255–260. https://doi.org/10.1203/PDR. 0b013e3181973b0d. Lakoff, George, and Mark Mark Johnson. 1980. Metaphors we live by. Chicago: Chicago University Press. Latour, Bruno. 1991. Nous n’avons jamais été modernes. Essai d’anthropologie symétrique. Paris: Editions La Découverte. English Trans. We have never been modern. Harvard University Press, 1993. ———. 2007. Can we get our materialism back, please? Isis 98 (1): 138–142. LeDoux, Joseph E. 1996. The emotional brain: The mysterious underpinnings of emotional life. Manhattan, New York, Simon & Schuster. Lewes, George H. 1875. Problems of life and mind first series: The foundations of a creed. Vol. 2. Boston: Osgood. López Varela, Asun. 2012. Emergence. In A more developed sign: Advancing the work of Jesper Hoffmeyer, ed. Doland Fabereau, Paul Cobley, and Kalevi Kull, 115–119. Tartu. Estonia, University of Tartu. Malafouris, Lambros. 2013. How things shape the mind: A theory of material engagement. Massachusetts, Cambridge: MIT Press. Malafouris, Lambos, and Knappett Carl. 2008. Material agency towards a non-anthropocentric approach. New York: springer. Maturana, Hurberto, and Francisco Varela. 1984. The tree of knowledge. Biological basis of human understanding. El árbol del conocimiento: las bases biológicas del entendimiento humano. Santiago de Chile: Editorial Universitaria.

1240

A. Lo´pez-Varela Azcárate

McLuhan, Marshall. 1964. Understanding media: The extensions of man. Introduction by Lewis H. Lapham. Cambridge, Massachusetts: MIT Press. Miller, Daniel. 1998. Material cultures: Why some things matter. Chicago, Illinois. University of Chicago Press. Nöth, Winfried (2009). On the instrumentality and semiotic agency of signs, tools, and intelligent machines. Cybernetics & Human Knowing 16, 3–4, pp. 11–36. ———. 2010. Machines of Culture – Culture of Machines? In Analisi delle Culture – Culture dell’analisi, ed. M. Leone, 41–58. Lexia: Rivista di Semiótica, nuova serie. Università di Torino, Centro Interdipartimentale di Ricerche sulla Comunicazione 5–6. https://doi.org/10. 4399/97888548345903. Open Culture Blog. 2016. https://www.openculture.com/2016/03/the-shadow-of-a-hiroshimavictim-etched-into-stone-steps-is-all-that-remains-after-1945-atomic-blast.html Peirce, Charles S. 1931–35/1958. Collected papers of Charles Sanders Peirce, ed. Charles Hartshorne and Paul Weiss, vol. 1–6, 1931–35, ed. Arthur W. Burks, vol. 7–8, 1958. Cambridge, Massachusetts, MA: Harvard University Press. Pepperell, Robert. 2005. Posthumans and extended experience. Journal of Evolution and Technology 14: 27–41. Salmose, Niklas. 2018. The apocalyptic sublime: Anthropocene representation and environmental agency in Hollywood action-adventure cli-fi films. Journal of Popular Culture 51: 1415–1433. Searle, John. 1969. Speech acts. Cambridge, UK, Cambridge University Press. Thomas, Lewis. 1992. The fragile species. London: Scribner. Simon & Schuster. Varela, Francisco J., Evan Thompson, and Eleanor Rosch. 1992. The embodied mind: Cognitive science and human experience. Cambridge Massachusetts, MIT Press. Von Mossner, Alexa. 2017. Affective ecologies empathy, emotion, and environmental narrative. Columbus, Ohio: Ohio State University. Wheeler, Wendy. 2006. The whole creature: Complexity, biosemiotics and the evolution of culture. London: Lawrence & Wishart. Wiener, Norbert. 1950. The human use of human beings: Cybernetics and society. Cambridge, MA: Riverside Press. Revised edition. London: Free Association Books, 1989.

Index

A Aarseth’s ontological model of games, 1006 Abridgement, 807, 819, 820, 822, 824, 825 Absolute music, 700–703, 843 Abstract phonetic poems, 933 Abu Ghraib, 117–118, 125 Academic journals, 165–169, 1034, 1038, 1044 Accompaniment, 438, 677, 806–814, 817, 826, 932, 974 Action poetry, 922, 928, 939–942, 946, 950 Adaptation, 3, 8, 20, 24, 25, 32, 35, 38, 40–42, 49, 74, 86–89, 139, 140, 151, 163, 166, 176, 194, 204, 207, 215, 288, 290, 291, 293–300, 310–312, 315, 349, 366, 368, 377, 378, 410, 416, 422, 423, 427, 428, 448, 628, 630, 632–635, 637, 642, 643, 646, 652–654, 670, 739, 754, 759–762, 803–828, 849, 899, 900, 912, 914–918, 937, 1002, 1011–1013, 1019, 1039, 1042–1044, 1073, 1075, 1077, 1080, 1100, 1103, 1119, 1139, 1200 audiobooks, 806, 807, 819, 824–826, 828 definition, 804 film, 804–820, 824–828 literacy, 815–819 practice, 805, 812, 815, 818 radio, 804, 806, 807, 816, 819–828 sound, intermediality and, 826–828 studies, 804–808, 810, 817, 818, 828 Addi Køpcke, Danes, 434 Aesthetics experience, 1196–1210 Aesthetics thinking, 696, 697, 699, 700 Affect, 7, 22, 24, 25, 43, 52, 56, 60, 78, 81, 83, 84, 90, 91, 109, 148, 151, 177, 180, 196, 200, 255, 257, 262, 269, 312, 344, 346, 381, 394, 424, 440, 441, 449, 451, 467, 479, 482, 487, 490, 650, 724, 725, 731–736, 745, 777, 783, 784, 790, 814, 838, 930, 944, 1003, 1006, 1014, 1017,

1024, 1039, 1043, 1051, 1054, 1065, 1067, 1068, 1078, 1144, 1160, 1190, 1202, 1204, 1217, 1222, 1234 Affective experiences, 733 Affordances, 9, 23, 24, 188, 370, 406, 424, 431, 652, 668, 676, 686, 737, 774, 776, 782, 787, 790, 795, 839, 923, 943, 947, 1008, 1040, 1043, 1053, 1055, 1069, 1120–1124, 1127, 1131, 1148, 1152, 1155, 1159, 1161, 1163, 1175, 1195, 1199, 1200, 1205, 1210, 1220, 1236 Agency, 9, 154, 297, 341, 405, 429, 473, 474, 726, 733, 848, 926, 956, 961–964, 966, 967, 974, 1012, 1015, 1038, 1046, 1048, 1050, 1053, 1054, 1061–1084, 1116, 1118–1120, 1122, 1123, 1125, 1130, 1148, 1177, 1184, 1186, 1188, 1213–1237 AI systems, 260, 280 Albright, Thomas, 443, 838, 839 Allegorical figure, 529 Allegorical poetry, 580 Allegorism, 530 Allegory, 6, 20, 107, 346, 354, 522–545 Alternative histories, 128 Amazon Kindle, 1196, 1201, 1205, 1207 Analepsis, 369, 375 Analogue, 25, 26, 82, 85, 108, 109, 193, 259, 260, 262, 265, 270, 280, 325, 326, 336, 620, 621, 630, 640, 793, 795, 843, 980, 981, 983, 984, 986–988, 991, 992, 1041, 1043, 1062, 1065, 1067, 1068, 1138, 1140, 1142, 1200, 1213, 1231, 1232, 1237 Analysis art history and literary, 469 cultural, 425, 426, 478, 479, 1040 iconographic, 468 rhetorical, 483 for semiotics, 464 and teaching intermediality, 489–490

© Springer Nature Switzerland AG 2024 J. Bruhn et al. (eds.), The Palgrave Handbook of Intermediality, https://doi.org/10.1007/978-3-031-28322-2

1241

1242 Andrade, Oswald de, 209, 901–907, 917, 918 Animation, 14, 17, 180, 186, 280, 289, 310, 312, 527, 559, 740, 1048, 1097, 1121, 1142, 1143, 1155, 1158 Anna Blume, 875, 885 Anterior pictures, 667, 680–686 Anthropophagic appropriation, 8, 899–918 Anthropophagic translation, 916 Anticipatory labor, 1108 Antiquity, 50, 58, 74, 137, 237, 342, 353, 469, 529, 540, 618, 620, 1042 Applied arts/artefacts, 14, 16, 22, 34, 372–374, 523, 537, 539, 540, 542, 543, 1053, 1090, 1100, 1104, 1179, 1234 Appropriation, 8, 23, 206, 281, 298, 323, 334–337, 596, 631, 636, 637, 641–646, 653, 654, 761, 805, 808, 865, 869, 875, 876, 894, 899–918, 946, 966–969, 1003, 1017, 1025, 1129, 1182 Armchair travelling, 1016 Artefactos, 943, 946 Artificial intelligence, 67, 280, 991, 1054, 1170, 1174, 1179, 1204, 1218 Artificial sign systems, 699 Artist’s books, 206, 438, 868 Art music, 775, 834, 835, 837, 839, 844, 846–848, 854, 1055 Associational listening, 820 Asymmetries, 935 A Theory of Semiotics, 261, 262 Audiobook, 193, 195, 295, 804, 806, 807, 819, 824–826, 828, 834, 837, 855 Audiovision, 717 Auditive aesthetics, 771 Auditory experience, 246, 704, 780, 837, 841, 844–847, 853, 855, 1017 Auricularization, 370, 374 Authenticity, 63, 65, 79, 86, 117, 120, 196, 246, 360, 646, 843, 962, 1012, 1045 Authorship, 23, 110, 296, 313, 916, 918, 956, 961–963 Autobiography, 392, 402, 566 Automata, 559, 610, 612–614, 1219 Automatism, 269, 869, 890 Automaton, 559 Avalon Ballroom, 446 Avant-garde, 8, 14, 15, 22, 23, 80, 81, 114, 124, 687, 700, 730, 740, 764, 786, 847, 866, 869, 894, 903, 904, 907, 908, 917, 923, 933, 935, 938, 943, 946, 947, 1094, 1130 Ay-O, 434

Index B Bach, Johann Sebastian, 558, 705–707, 716, 834 Balanced transmedia, 383 Bamboo, 232, 497, 498, 503, 507–510 Baroque, 7, 22, 357, 446, 485, 523, 531, 545, 552–574, 578–599, 604, 608–610, 617, 618, 622, 648, 649, 698, 885, 1093 Barthes, Roland, 20, 74, 82–84, 86, 87, 127, 128, 262, 267, 271, 310, 356, 357, 367, 397, 470, 541, 672, 684, 838, 839, 916, 968, 1181–1183, 1189 Bateson, Gregory, 256, 1180–1183, 1185, 1189 Becker, Wolfgang, 435 van Beethoven, Ludwig, 270, 410, 701–703, 708, 713–716, 790, 834, 844, 845 Bell, Daniel, 440, 1173, 1185 Berger, Rene, 435 Beuys, Joseph, 20, 434 Biedermeier painting, 694 Big data, 1185–1187, 1219 Big science, 1185–1187, 1190 Biographic space, 402 Bipolar model, 697 Biscoito Arte, 931 Black line manner, 680 Black music, 775, 934 Block, René, 443 Blumer, Herbert, 441 Board games, 1062–1065, 1068–1070, 1075–1077, 1079–1081 Border, 7, 8, 14, 37, 66, 107, 163, 172, 190, 191, 215, 333, 345, 349–352, 356, 421, 425, 440, 462, 463, 465, 466, 470, 471, 478, 571, 572, 605, 609, 610, 628–654, 672, 673, 679, 686, 696, 703, 727, 735, 753, 758, 761, 774, 783, 791, 796, 826–828, 837, 842, 854–856, 895, 908, 980–996, 1001, 1035, 1036, 1043, 1048–1050, 1055, 1119, 1131, 1148, 1177, 1219, 1222 Bory, Jean-Francois, 443 Bottle, Glass and Newspaper on a Table, 872 Branagh, Kenneth, 447, 448, 633 Braque, George, 870, 872, 882, 892 Brazil, 34, 48, 203–223, 334, 564, 900–903, 906, 909–914, 923, 926–928, 942, 1188 Brazilian translation theory, 916 Brecht, George, 442, 443 Breder, Hans, 439 Broad intermediality, 980–983, 994, 996, 1149

Index Brockman, John, 438 Bruhn, Jørgen, 1–10, 15, 16, 18, 26, 49, 55, 57, 66, 89, 99, 104, 166, 185–200, 209, 213, 247, 349, 350, 416, 418, 425–429, 596, 616, 666, 715, 805, 836, 840, 842, 843, 851, 909, 912, 1033–1055, 1063, 1067, 1218 Bush, Vannevar, 443, 1173 C Cabinets of curiosities, 7, 603–624 Cadeau, 876 Cage, John, 20, 439, 442, 443, 781, 894, 944 Caixa-poemas, 930 Calligrammes, 880, 881 Calligraphy, 22, 226, 228, 232, 239, 241, 357, 496–499, 501, 502, 505, 507, 511, 514–517 Camera Lucida, 262 “Caméra stylo”, 407 Canon, 62, 65, 122, 383, 384, 532, 631, 633, 634, 636, 639, 640, 647, 849, 853, 913, 915, 916, 1091 Caption, 303, 538, 672, 684, 958, 965, 967, 975 Carnival, 115, 925, 934 Cartesian perspectivalism, 561, 564 Castells, Manuel, 120, 440, 1173, 1176 Cave, 5, 16, 100, 105–107, 116, 354, 855 Celluloid analogue, 986 Centre de Recherche sur l’intermédialité (CRI/CRIalt), 137, 138, 140–142, 153, 156 Channel, 24, 25, 66, 104, 111, 152, 191, 245, 262–264, 276, 277, 292, 295, 302, 305, 375, 377, 380, 382, 385, 444, 447, 526, 632, 633, 643, 654, 684, 716, 726, 754, 909, 943, 944, 971, 995, 1001, 1021, 1037, 1054, 1097, 1119, 1123, 1131, 1220, 1231, 1232 Characters, 15, 21, 34, 38, 42, 55, 83, 86, 109, 147, 154, 164, 179, 194, 197, 254, 256, 259, 261, 266, 267, 275, 281, 287, 289, 290, 292, 293, 299, 304, 310, 313, 327, 328, 330, 334, 336, 347, 356, 360, 370, 372–375, 378–382, 384, 385, 394, 399, 401, 403, 404, 408, 409, 424, 445, 469, 470, 499, 500, 503, 505, 530, 532, 541, 556, 565, 567, 568, 573, 587, 588, 603–605, 624, 628, 644, 645, 649–651, 669, 672, 673, 675, 683, 685, 701, 731,

1243 772, 782, 784, 787, 794, 795, 807, 812, 813, 821–823, 850, 866, 882, 887, 888, 890, 891, 908, 915, 927, 941, 945, 946, 961, 962, 968, 985, 990, 992, 1000, 1008–1011, 1013, 1015, 1019, 1022, 1107, 1118, 1120, 1121, 1123, 1131, 1142, 1152, 1155, 1159, 1220, 1225, 1226, 1236 Chattheater, 278 Chinese Room Argument, 265 Chinese traditional culture, 497, 510 Christian Metz’s film semiotics, 262 Christiansen, Henning, 434 Cinema, 8, 17, 19–22, 36, 37, 107, 139–141, 151, 152, 154, 161, 166, 167, 173, 191, 210, 211, 215, 222, 232, 263, 286–288, 294, 311, 340, 369, 405, 407, 418, 425, 438, 468, 470, 630, 634, 635, 639, 725, 728, 737, 755, 758, 763, 764, 776, 777, 779, 796, 807, 808, 810, 812, 815, 817, 818, 824, 827, 828, 880, 891, 894, 907, 910, 911, 925, 938, 962, 963, 971, 979–996, 1001, 1004–1006, 1008, 1011–1013, 1020, 1024, 1042, 1043, 1046, 1047, 1053, 1094, 1098, 1100, 1109, 1125, 1138, 1140, 1144, 1177, 1181, 1216, 1223 Citation, 5, 6, 8, 321–337, 436, 866, 869, 875, 892, 893, 946 Clark, Colin, 440 Classical music, 241, 775, 787, 790 Climate change, 3, 9, 57, 186, 195, 196, 1035, 1037, 1040, 1041, 1045, 1046, 1048, 1049, 1061–1084 Clüver, Claus, 4, 19, 32–45, 54, 55, 86, 99, 187, 194, 204, 207–209, 211, 213–215, 261, 290, 291, 293, 299, 322, 329, 330, 349, 350, 496, 502, 518, 527, 781, 840, 845, 869, 882, 894, 908, 909, 924, 933, 938, 945 Cognitive ekphrasis, 16 Cognitive process, 57, 675, 981, 988, 1216 Cognitive representational models, 993 Cold Mountain, 940 Coleridge, Samuel Taylor, 20, 347, 437, 726, 922 Collages, 15, 22, 43, 866, 868, 869, 871, 873–875, 882, 883, 886, 887, 890, 894 composition, 866, 868, 870, 880, 882 cubist, futurist and dadaist, 870–878 effects, 869, 893

1244 Collages (cont.) and intertextuality, 891–893 pseudo-collage, 869, 886 pure collage, 869, 886 surrealist, 233, 886–893 verbal and literary collage, 879–886 Collective narrative, 1159–1161 Combinatorial approach, 438 Comenius, 586 Comics, 17, 23, 24, 38, 86, 139, 211, 220, 244, 326, 328, 329, 333, 334, 352, 356, 368, 371–374, 382, 383, 385, 392, 397, 403–406, 408, 410, 421, 430, 633, 675, 912, 1010, 1015, 1138, 1142, 1200 Commodore Amiga 1000, 1018 Communicate, 6, 7, 81, 90, 107, 147, 175, 188, 226–228, 247, 291, 351, 372, 445, 449, 464, 566, 599, 606, 607, 617, 675, 806, 809, 823, 835, 848, 935, 944, 962, 1004, 1006, 1007, 1009, 1035–1037, 1040, 1053, 1055, 1119, 1121, 1170, 1171, 1176, 1183, 1184, 1215 Communication, 3, 6, 8, 14, 19, 24, 25, 34, 38, 39, 66, 80, 83, 87, 88, 90, 98, 99, 103, 104, 108, 128, 144–148, 150, 152–155, 161, 162, 166, 172, 186, 189, 191–193, 195–199, 208, 210–212, 216, 229, 230, 232, 233, 240, 241, 243, 255–257, 259–263, 267, 268, 270–272, 274–278, 281, 292, 322, 337, 340, 348, 351, 357, 395, 396, 404, 405, 417–422, 425, 428, 429, 431, 436, 439, 441, 443, 444, 447, 449–451, 463, 478, 527, 605–609, 631, 634, 642, 644, 650–652, 654, 686, 704, 710, 779, 786, 827, 834, 836, 851, 908, 927, 935, 938, 939, 944, 958, 960, 973, 981, 983, 996, 1001, 1004, 1035–1043, 1045, 1046, 1048, 1054, 1055, 1063, 1066, 1069, 1081, 1093, 1099, 1123, 1150, 1156, 1158, 1163, 1171–1186, 1188, 1189, 1197, 1200, 1202, 1203, 1210, 1215, 1219, 1220, 1222, 1224, 1235, 1237 across media, 1043, 1175 theory, 195, 197, 256, 268, 275, 449, 908, 1178 Comparative literature, 33, 34, 36, 43, 161, 166, 167, 186, 187, 204, 208, 212, 214, 230–232, 350, 416, 420, 439, 927 Complementation, 289, 290, 313, 316 Comprehensive analysis, 597 Computer games, 8, 193, 278, 288–290, 301, 343, 397, 743–745, 999–1025, 1062, 1064–1068, 1072, 1075, 1078, 1083, 1118, 1124, 1137, 1138, 1237

Index formal mode of expression, 1003–1011 intermedial network of the qualified medium, 1025 participatory media culture, 1019–1023 technical media, 1015–1018 transmedial content, 1011–1015 Computer graphics, 1137–1139, 1142–1144 Computer simulation, 9, 1034–1145 Computer-supported cooperative work (CSCW), 1178 Concept poetry, 922, 946–950 Conceptual art, 20, 437, 438, 442, 909, 942, 943, 948 Concrete poetry, 26, 43, 187, 188, 301, 438, 496, 885, 894, 901, 907–909, 922, 930, 933, 934, 937, 938, 940–942, 946, 949, 950 Conjunction, 17, 65, 592, 593, 726, 758, 760, 879 Connective action, 1178 Connotation languages, 1171, 1181–1183, 1185, 1189 Consalvo’s notion of paratext, 1021 Contextual integrity, 1187 Contextual qualifying aspect, 614, 616, 618 Continuum, 9, 66, 379, 405, 408, 583, 610, 727, 991, 1118, 1130, 1131, 1217, 1223, 1230, 1231, 1233, 1237 Contrasting image types, 676–680 Control revolution, 1173 Convergence, 2, 7, 23–25, 105, 111, 192, 205, 210, 211, 226, 227, 274–276, 278, 375, 382–384, 430, 438, 440, 443, 499, 580, 592, 595, 596, 635, 638, 684, 726, 758, 904, 980–983, 992, 1002, 1003, 1013, 1101, 1103, 1148, 1152 culture, 211 Cool, 385, 789, 985, 986, 988, 994, 996, 1206 Copernican revolution, 555 Counterpoint, 44, 207, 233, 308, 706, 785, 809, 835, 852–854 Counter-reformation, 554–555, 563, 564, 582 Course in General Linguistics, 256 Covert narrator, 373 COVID-19 pandemic, 45, 1108, 1109 Crane, Michael, 443, 679 Creoleness movement, 401 Crisis, 165, 196, 243, 245, 258, 369, 555, 581, 905, 916, 957, 1033–1055, 1063, 1069, 1078, 1079, 1082–1084, 1219 Critical media literacy, 419, 427 Critical writings, 682 Crossing, 7, 37, 87, 145, 213, 333, 349, 350, 358, 449, 605, 609, 627–654, 696, 727, 753, 776, 827, 868, 887, 895, 1050

Index Cross-media narrative, 240, 242–246 Crowdsourcing, 965 Cubism, 8, 43, 873, 904 Cultural analysis, 425, 426, 467, 478, 479 Cultural anthropology, 696 Cultural anthropophagy Andrade’s notion of, 901–907, 913, 918 anthropophagic theatre performance, 912–913 Cinema Novo, 910–911 cultural anthropophagy as methodology in the Global South, 913–914 curatorial projects, 913 installation art, 909 intertextuality and intermediality, 914–917 local and global resonances of, 907–914 poesia concreta, 908 postcolonial literary criticism, 911–912 postcolonial theory of cannibalistic translation, 910 Cultural forms, 14, 18, 19, 22, 734, 735, 914, 1101, 1116, 1206 Cultural memory, 25, 57, 62, 126, 196, 644, 754, 1122 Cultural noise, 278 Cultural production, 37, 86, 139, 140, 142, 143, 149–152, 154–156, 168, 169, 172, 214, 243, 275, 754, 866, 868, 895, 1071 Cultural semiotics, 122, 262–264 Culture of excess, 557 Curiosity, 121, 371, 380, 408, 587, 605, 607, 609, 611, 614–616, 619–622, 624, 1123, 1126 Cursive Script, 505 Cybermedia objects, 1006 Cybernetic principles avant la lettre, 256 Cybernetics, 256, 257, 261, 908, 1181, 1219, 1227, 1236, 1237 Cyberspace, 735, 1144, 1175, 1177 Cybertext, 1006, 1148, 1156–1158, 1160, 1163 D Dadaism, 8, 330, 904 Dadaist photomontages, 869, 878 Dance, 32, 36, 37, 40, 41, 114, 204, 212, 214, 216, 228, 235, 243, 245, 248, 254, 287, 313, 340, 438, 439, 486, 487, 565, 566, 568, 593, 644, 647, 669, 697–699, 725, 727, 729, 741, 753, 755–758, 764, 770, 788, 808, 839, 847, 848, 913, 917, 983, 1122, 1202 Danhauser, Joseph, 694, 695, 700, 703–705, 708, 712–714, 716, 717 Deconstructive analysis, 697

1245 Deep-impact critical concept, 984 Deep Nostalgia, 280 Della Croce, Johann Nepomuk, 707, 708, 712, 716 Derrida, Jacques, 215, 273, 349, 358–360, 419, 468, 470, 472, 473, 580, 675, 679, 910, 989 de Saussure, Ferdinand, 255, 258, 259, 280, 353, 354, 1224 de Saussure’s semiology, 256–258, 263, 355 Description, 15, 16, 40–42, 48–54, 56–59, 61, 63, 64, 66, 76, 102, 122, 173, 193, 199, 233, 236, 266, 274, 279, 288, 290, 331, 341, 343, 346, 450, 465, 489, 499, 513, 527, 529, 531, 532, 563, 565, 566, 581, 588, 590, 592, 598, 611, 614, 633, 638, 640, 677, 678, 695, 805, 806, 816, 824, 827, 834, 835, 837, 840, 842, 843, 845–849, 853, 909, 928, 936, 982, 983, 995, 1010, 1068, 1069, 1137, 1182, 1202, 1203 Designer-story, 1015 Development of a Bottle in Space, 873 Diagrammatic analysis, 174, 175, 181 Dialectical theatre, 730, 742–745 Dialogism, 355, 356, 470, 891, 892, 916 Diasemiotic translation, 292 Diegesis, 370, 393, 394, 422, 796, 807, 815, 840, 842, 843, 855, 984, 987, 992 Digital age, 23–26, 48, 53, 55, 58–60, 65, 108–111, 117, 118, 148, 154, 630, 993, 1236 Digital archaeology, 450 Digital computer, 442, 559, 1140, 1171, 1173–1175, 1189, 1232 Digital convergence, 440, 443 Digital culture, 23, 262, 726, 744, 972, 1130 Digital ekphrasis, 16, 58 Digital fiction, 9 cybertext, 1156–1158 hypertext and hypermedia fiction, 1151 interactive fiction, 1154–1156 network fiction, 1158–1161 small-screen fiction, 1161–1163 Digitalization, 984, 1170, 1172, 1177–1179, 1185, 1189, 1190 Digital literacy, 197, 417, 428, 430 Digital media, 9, 23, 25, 36, 38, 60, 85, 103, 108, 118–120, 166, 192, 198, 205, 206, 211, 248, 375, 377, 416, 418, 420, 428, 429, 628, 631, 634, 635, 638, 643, 653, 735, 752, 824, 854, 923, 980, 982, 986, 987, 992, 1004, 1022, 1048, 1116, 1124, 1125, 1136, 1138, 1145, 1170, 1171, 1174–1176, 1178, 1179, 1189, 1196, 1209, 1236

1246 Digital paramedia, 1107, 1108 Digital social reading (DSR), 1161 Dilthey, Wilhelm, 449 Dine, Jim, 446 Disjunction, 17, 726, 732, 734, 737 Disney, 237, 289, 375, 383, 941, 1012, 1090–1108, 1136, 1196, 1205 Dispositif, 101, 102, 106–108, 116, 123, 142, 145, 176, 177, 179, 180, 264, 268, 270, 276, 277, 281, 992, 994, 995 Documentaries, 5, 109–111, 113, 152, 636, 1035, 1044, 1045 Documentary style, 956, 966 Dorfles, Gillo, 435 Double hermeneutics, 1172 Double marriage in Mantua, 565–568 Dramatic mode of presentation, 727–730 Dupuy, Jean, 434, 442, 443, 450 Dynamic interaction between sister arts, 507–510 E Early cinema, 7, 107, 141, 167, 470, 639, 755, 763, 764, 807, 810, 817, 827, 828 Early modernity, 698 Early Modern Times, 121–123 Early twentieth century, 7, 77, 107, 226, 335, 346, 466, 558, 642, 646, 700, 724, 730, 752–759, 761–766, 778, 779, 784, 868, 928, 943 artists and performers, 755 audiences experienced intermediality, 759 theater productions, 756 Eco, Umberto, 259, 261, 262, 288, 308, 535, 536, 584, 586, 669, 732, 1120 Ecocritical, 9, 65, 856, 1038, 1039, 1041, 1044, 1045, 1048, 1053–1055, 1062 Ecocriticism, 5, 9, 47, 57, 166, 1035, 1036, 1038–1040, 1044, 1047, 1048, 1054, 1055, 1065 empirical, 1039, 1054 intermedial, 57, 113–114, 166, 1036, 1044, 1047, 1048, 1055, 1063 Eco-ekphrasis, 57 Ecological crisis, 9, 196, 1034–1055, 1063, 1069 Ecological emergency, 196, 1037, 1055 Economic depression, 770 Economies of attention, 101, 110–116, 123 The Edinburgh Companion to Literature and Music, 785, 836 Education, 82, 190, 194, 226, 418–420, 423, 426–428, 439, 443, 444, 447, 451, 476, 533, 542, 617, 752, 906, 1170, 1174, 1178, 1182

Index Ekphrasis, 3, 5, 6, 15, 16, 33, 35, 38–40, 42, 47–67, 76, 77, 79, 88, 173, 233–237, 290, 299, 305, 322, 326, 349, 424, 502, 514, 516, 526, 527, 530, 532, 539, 544, 565, 566, 598, 647, 651, 678, 680, 785, 840, 928, 985, 1047 functions of, 40, 53, 57, 60–66 histories of, 50–53 theories and definitions of, 53–58 typologies of, 58–60 El cuaderno de la rata almizclera, 931 Electronic literature, 23, 60, 65, 66, 214, 1148, 1150, 1151, 1159, 1201, 1237 Electronic media, 983, 987, 1004, 1005, 1172 Electronic screens, 8, 980–983, 985–996 Elleström, Lars, 2–5, 9, 10, 17, 18, 57, 86–90, 98, 99, 102, 127, 128, 139, 143, 162, 166, 167, 186–200, 209, 210, 212, 213, 235, 236, 247, 270, 277, 279, 280, 293, 295, 322, 330, 331, 336, 349, 350, 372, 392, 396–400, 406, 407, 409, 410, 416–418, 421, 422, 425, 426, 428, 429, 431, 462–464, 469, 472, 474, 478, 480, 481, 522, 528, 537, 566, 583, 588, 596, 598, 604, 605, 607–609, 612–614, 616, 629–632, 635, 638, 642, 643, 648, 650, 652, 666, 670, 678, 684, 694, 696, 699, 726, 772, 774, 786, 787, 806, 825, 827, 836, 837, 839, 840, 869, 878, 909, 912, 915–917, 924, 932, 980, 982–989, 995, 996, 1001, 1002, 1010, 1022, 1024, 1035, 1041–1046, 1048, 1063, 1067, 1117–1119, 1121, 1122 concept, 8, 18, 194, 198, 407, 840, 1200 contributions, 187 early publications, 187 general medial framework, 186 intermedial framework, 195 intermedial studies, 188 media modalities, 192, 193, 195 media products, 192 media transformation, 194 systematization, 186 visual poetry, 188 Emanata, 374 Emblematic allusions, 543 Emblems, 6, 35, 39, 117, 279, 328, 346, 444, 496, 521–545, 556, 562, 579, 581, 583, 584, 586, 587, 589–591, 595–599 allegorical, 523, 528, 532 applied, 539, 542–543 to epigram, 538 nature, 529 printed, 539–540

Index as Renaissance, 531 symbolic and allegorical character of (see also Epigram, epigrammatic; Allegory; Symbol) Enactivism, 1220, 1234 Enargeia, 40, 50, 51, 58–60, 77, 79, 527 Engraving, 122, 147, 206, 207, 289, 305, 311, 502, 529, 563, 581, 598, 605, 611, 621, 624, 682, 890, 962, 973, 974, 1232 Enhanced communication, 447 Enigmatic creativity, 587 Entertainment, 7, 19, 22, 104, 115, 118, 230, 366, 375–377, 379, 385, 440, 443–445, 447, 451, 551–574, 596, 631, 644, 667, 715, 726, 728, 752–759, 762, 763, 765, 808, 817, 823, 970, 1016, 1018, 1046, 1067, 1093, 1094, 1097, 1099, 1101, 1102, 1107, 1170, 1174, 1177, 1179 Entertainment industry, 366, 375–377, 379, 383, 385, 440, 573, 758, 759 Environment, 25, 58, 90, 91, 106, 142, 143, 145, 146, 154, 156, 174, 186, 187, 196, 200, 206, 267, 268, 272, 274, 304, 328–330, 332–334, 336, 353, 429, 471, 479, 499, 507, 648, 736, 744, 754, 804, 839, 869, 892, 925, 939, 958, 963, 988, 995, 1005, 1007, 1008, 1010, 1038, 1040, 1041, 1049, 1055, 1068, 1072, 1074, 1076, 1080, 1081, 1096, 1121, 1123–1125, 1127, 1131, 1149, 1155, 1161, 1172, 1173, 1175, 1183, 1196, 1205, 1207, 1209, 1214, 1217–1220, 1228, 1232, 1233, 1236, 1237 Environmental humanities, 1035–1041 Ephemeral architecture, 7, 577–599 Epic theatre, 729–731 Epigram, epigrammatic, 523–534, 537–540, 542, 543, 545, 586, 589, 591, 592, 595, 598 Epistemic consequences, 168–170 Epistemic options within inter-and transdisciplinarity conceptual modelling of intermedial relations, 162 critical perspectives on conditions of intelligibility, 163 Essential, 76, 78–83, 85, 87–89, 100, 102, 108, 116, 121, 137, 142, 143, 146, 165, 173, 174, 192, 267, 272, 276, 291, 296, 301, 342, 343, 351, 367, 368, 375, 376, 421, 424, 463, 465, 471, 535, 544, 557, 558, 604, 614, 736, 777, 782, 848, 873, 888, 890, 892, 973, 982, 1004, 1043, 1044, 1049, 1055, 1063, 1064, 1067, 1130, 1155, 1158, 1172, 1176, 1185, 1222, 1235

1247 Event structures, 442 Expanded cinema, 20, 438, 993 Experience, 7, 9, 16, 17, 19–21, 24, 26, 43, 58, 76, 78–81, 84, 86, 90, 91, 102, 112, 121, 125, 126, 143, 148, 151, 152, 156, 163, 167, 169, 172, 179, 180, 192, 206, 210, 213, 227, 246, 259, 267, 273–275, 296, 314, 341, 343, 345, 347, 348, 350, 351, 353, 372, 374–377, 379–381, 383, 402, 403, 440, 449, 490, 511, 526, 527, 556–558, 561, 563–565, 614, 617, 629, 630, 634, 637, 641, 643, 650, 652, 677, 680, 694–696, 698, 701, 703, 704, 709, 711, 715–717, 725, 726, 728–730, 732, 734–738, 740, 741, 744, 745, 754, 758, 763, 770, 774, 776, 777, 779, 780, 789, 791, 793, 794, 804, 808–811, 813, 814, 819, 822–824, 837, 841, 844–849, 949, 990, 995, 1042, 1054, 1065, 1068, 1098, 1109, 1116–1120, 1158–1160, 1170, 1195–1210, 1214, 1216, 1222, 1224, 1226, 1227, 1230–1232 Experiments in Art and Technology (E.A.T.), 450 Expressive productivity, 1020 Extended mind hypothesis, 1233, 1234 Extracompositional intermediality, 772, 1100, 1104, 1149 F Face-to-face metacommunication, 1184 Fandom, 1020, 1022, 1092, 1102, 1105, 1108, 1128, 1131 Fan fiction, 378, 384, 1003 Far-off effect, 511 Feasts, 126, 553, 564, 565, 568, 572, 581, 759 Festivals, 116, 438, 439, 545, 553, 566, 567, 995 Filliou, Robert, 434, 442, 992 Fillmore Auditorium, 446 Film, 14–21, 32, 36–38, 49, 54, 55, 58, 66, 75, 82, 86, 107–109, 111, 126, 194, 207, 208, 211, 243–245, 247, 257, 258, 262, 263, 288, 290, 291, 294–297, 299, 300, 302, 304, 308, 310–313, 368–370, 374, 385, 398, 404–410, 422–425, 428, 430, 431, 439, 444–446, 479–482, 485–487, 538, 566, 622, 630, 633, 635, 636, 670, 677, 678, 728, 738, 755, 758, 760, 761, 765, 780, 806–820, 823–828, 883, 891, 894, 910, 911, 915, 936, 939, 943, 966, 981–986, 988–993, 1000, 1004, 1008, 1009, 1011, 1012, 1015, 1040, 1042, 1043, 1045–1049, 1051–1055, 1066, 1095, 1097, 1144, 1163, 1200, 1209

1248 Film adaptation, 20, 215, 290, 448, 805, 814, 818, 825 Filmic diegesis, 984, 992, 994, 996 Filmic enunciation, 370 Filmmakers’ Cinematheque, 438 Film music, 245, 807, 828, 1098 Film soundtrack, 814, 826 Fluxus, 6, 20, 21, 148, 433, 434, 439, 442, 446, 450, 927, 942, 943 Flynt, Henry, 437, 438, 442 Focalization, 369, 370, 374, 385, 394, 399, 400, 407, 408, 789, 791 Fold, 180, 358, 610, 616 Földényi, F. László, 700–702, 714 Folk music, 775, 787, 790, 842 Formal (transmedial) intermediality, 21 Foucault, Michel, 85, 87, 103, 211, 358–360, 554, 556, 558, 637, 643, 736, 916 4E (embodied, embedded, enacted, and extended) cognition, 1215, 1218–1223, 1231 Four Figures of Moral Integrity, 507 Franchise, 9, 23, 366, 375, 376, 378, 379, 381–385, 972, 1013, 1015, 1093, 1102, 1103, 1105 Freemium version, 1205 Freudian psychoanalysis, 263 Friedman, Ken, 6, 434, 435, 438, 440–442 Fruit dish and glass, 870 Fugue, 558, 784, 790, 793, 835, 847, 849, 850, 852 Functional memory, 125, 126 Futurism, 8, 868, 874, 904 G Galatea, 65, 1156 Game clone, 1018 Game conversion, 1018, 1163 Game imagery, 1010 Game mechanics, 1006, 1007, 1063, 1069, 1070, 1083, 1156 Gameplay footage, 1021 Gaming technologies, 1001, 1017, 1021 Gance, Abel, 449 Garden/gardens, 560, 566, 568, 573 Gaulli’s Triumph of the Name of Jesus, 553, 562, 563 Genealogical intermediality, 980–983, 985, 993, 996 Genette, Gérard, 353, 354, 368–370, 372, 373, 393–395, 397, 399, 402, 404, 406, 411, 892, 916, 1021, 1104, 1197

Index Genre, 107–110, 113, 120, 121, 128, 1148, 1150–1155, 1158–1160 Gesamtkunstwerk, 14, 21, 22, 348, 727, 729, 734, 771, 776, 786, 791, 839, 1094, 1099 Getty Art & Architecture Thesaurus Online, 436 Gift of Tao Gu, 506 Global village, 3, 117 Global warming, 425, 1034, 1063, 1073, 1076–1079, 1081 Glusberg, Jorge, 435, 443 God games, 1062, 1069, 1071, 1072, 1074, 1075 von Goethe, Johann Wolfgang, 269, 273 Grand image-maker, 405 Graphic narratives, 371, 374, 375, 397 Grinding, 770, 1014 Grooms, Red, 446 Gugelberger, 443, 444 Gugong bowuyuan lidai huihua tishi cun (Palace Museum Anthology of Tihuashi of Past Dynasties), 500 Guitar, 870, 871, 1126, 1127 H Habeas corpus, 1172, 1188 Habeas data, 1172, 1186–1188 Habits, 15, 193, 331, 464, 466, 470, 526, 614, 694, 793, 1082, 1207, 1221, 1227–1231, 1233 Handbuch Literatur und Musik, 785 Al Hansen, 4, 24, 353, 434, 442, 446, 635, 666, 681, 758, 759, 847, 855, 1139 Happenings, 442, 446, 894, 939–943 Haptic fandom, 1102 Hard intermediality, 279 Harrison, Helen, 442, 701, 811 Haussmann, Elias Gottlob, 705–707, 878, 885 Heart Beats Dust (Cone Pyramid), 450 Hegelian heritage, 695 Hegirascope, 1157 Hendricks, Geoffrey, 434 Hendricks, Jon, 439 Henry V, Play (theatre), 447 Hermeneutics of platforms, 166 Hermeticism, 582 Heterogeneous elements, 866, 895 Heteromediality, 15, 350, 1042 Hierarchy between media, 382–384 Hieroglyphs, 523, 530, 534, 535, 539, 540, 583, 584, 589, 591, 595, 599, 1232

Index Higgins, Dick, 6, 19–21, 204–206, 215, 349, 434, 437, 438, 442, 443, 446, 451, 725, 726, 868, 876, 878, 894, 922–924, 927, 928, 932, 933, 935, 936, 939–944, 946–950 Hispanic American baroque celebrations allegorization and ekphrastic relations, 583–584 intermedial way of thinking and creating, 590–593 medial relations, context of, 581–583 story of a misreading, 584–587 symbolic programs and commissions, 587–590 Historical narrative as an act of mediation, 177–179 Homebrew games, 1020 Hompsom, Davi Det, 434 Host medium, 960, 975 Houston, Jean, 443, 757 Huaba, 499 Huaji, 499 Huaming, 499 Huazan, 499 Human communication, 186, 194, 196, 199, 420–422, 428, 429, 441, 827, 834, 1041, 1171, 1172, 1174, 1175, 1180, 1181, 1185, 1189, 1190, 1215 Hybrid, 8, 14, 15, 17, 21, 22, 57, 110, 116, 128, 196, 214, 349, 357, 378, 402, 434, 439, 442, 443, 445, 566, 610, 634, 697, 698, 725, 743, 744, 753, 755, 870–879, 890, 893–895, 913, 936, 959, 962, 988, 1008, 1064, 1094, 1142, 1143 Hybridity, 4, 106, 116, 238, 351, 360, 522, 527, 569, 634, 698, 752, 902, 911, 912, 947 Hybrid media, 17, 116, 196, 634, 755 Hyperdiegesis, 376 Hyperdiegetic paratexts, 1105 Hypermediacy, 22, 275, 732, 989, 1094 Hypermedia fiction, 1148, 1150–1156 Hypertext fiction, 86, 1148, 1150–1152, 1154, 1162 Hyposemiotic translation, 292 Hypotyposis, 50, 59, 78, 79, 526 I Icon, 18, 254, 259, 270, 272, 275, 279, 281, 331, 467, 480, 483, 486, 641, 643, 980, 1070, 1076, 1086, 1215, 1226, 1227 Iconographic program, 562, 589

1249 Iconography, 42, 466–469, 471, 523, 530, 540, 543, 592, 1000, 1022 Iconotext, 16, 40, 46, 59 Iconotextuality, 678 Illustrated song, 807–811, 813, 817, 826, 828 Illustration, 35, 39, 41, 42, 44, 211, 239, 240, 290, 293, 307, 308, 331, 333, 505, 506, 528, 529, 534, 538, 540, 543, 544, 605, 611, 612, 623, 624, 670, 676, 678–680, 682, 683, 696, 887, 962, 971, 1000, 1042, 1046, 1072, 1073, 1097, 1104, 1149 Images, 15–18, 23, 48, 56–61, 109, 114, 117, 120, 125, 145, 147, 152, 177, 198, 234, 240, 290, 322, 324–333, 351, 353, 354, 385, 399, 404, 405, 444, 463, 474, 478, 501, 509, 525–528, 531, 532, 534, 544, 558, 584, 586, 590, 611, 622, 644, 664, 666, 676, 677, 686, 707, 712–716, 740, 741, 752, 755, 760, 763, 806, 809, 811, 814, 817, 823, 888–891, 893, 909, 938, 944, 957–960, 962, 965, 967, 968, 972, 982, 985, 986, 989, 992–994, 1008, 1010, 1046, 1049–1051, 1098, 1107, 1136–1140, 1142, 1150, 1154, 1158, 1160, 1171, 1182, 1184, 1216, 1224, 1229 analysis and research, 694 images-within-images, 712, 714, 717 and text, 16, 666, 667, 672–676, 678, 679, 686 and word, 17 Imagination, 51, 57, 66, 80, 101, 120, 123, 230, 233, 245, 246, 268, 344, 346, 352, 371, 376, 472, 509, 527, 535–536, 539, 571, 593, 598, 604, 620, 648, 650, 652, 703, 705, 738, 820, 826, 879, 1046, 1189, 1233 Immersion, 229, 376, 727, 731, 738, 740, 744, 745, 1054, 1064, 1119, 1124–1128, 1131 In-betweenness, 5, 105, 339–361 Index, 18, 259, 270, 271, 274, 275, 279, 480, 483, 1046, 1161–1163, 1226, 1227 Industrial intertextuality, 1102 Information society, 440, 1173, 1185, 1188 Information theory, 261, 262 Innis, Harold, 440, 973, 1172, 1189 Instagram, 645, 957, 971, 1108, 1199 Instrumental music, 122, 245, 371, 373, 409, 410, 700–702, 705, 837, 843, 1204 Instrumental productivity, 1019 Intellectual landscape, 714

1250 Intellectual property, 379, 1015, 1103 Interactive and participatory sound, 9, 855, 1115–1131 Interactive fiction, 1148, 1154–1156 Interactive simulations, 1137 Interactivity, 1117–1119, 1124, 1125, 1127, 1130, 1131 Interart aesthetics, 715 Interart poetics, 187, 227, 233, 236, 237, 350, 507 Interarts studies, 33–37 Interdisciplinarity, 2, 162, 165–167, 169, 215, 340–342, 348–353, 356, 463, 464, 1003 Interface, 1000, 1006–1011, 1015, 1018, 1020, 1105, 1119, 1158–1160, 1179, 1180, 1184, 1189, 1196, 1198, 1200–1206 Interlingual translation, 41, 287, 288, 290–294, 298, 300, 302, 303, 305, 310, 314, 315 Intermedia, 6, 8, 19–21, 33–39, 41, 42, 44, 100, 102, 103, 105–108, 110–114, 116, 118, 120–127, 207–210, 212, 213, 242, 243, 312, 316, 428, 433–452, 481, 496, 586, 590, 622, 725, 868, 876, 894, 921–950, 1024, 1120, 1189 first history, 437 fourth history, 441 second history, 439 third history, 440, 441 three directions for, 442–443 Intermedial/intermediality/intermédialités, 6, 9, 14, 19–21, 37, 48, 55, 75–77, 79, 86–90, 160, 204, 205, 207, 209, 212, 322, 323, 328–330, 332, 340–343, 349, 351–357, 360, 416–428, 462, 463, 465, 467, 469, 473, 474, 477, 479, 481, 488, 553, 561, 572, 573, 593, 605, 666, 667, 670, 694–696, 698, 703, 704, 711, 715–717, 752–759, 761–763, 765, 771, 776, 782, 790, 797, 809, 825, 826, 828, 868, 869, 914–917, 958–961, 966, 968, 1136, 1149–1151, 1196, 1200, 1202, 1205, 1208, 1209 analysis and teaching, 489–490 cabinet of curiosities, 609–612 canonical phase, 276 case studies, 161, 164, 170, 171, 175 definition, 20 diagrammatic analysis, 175–180 digital age, 23–26, 118–119 within disciplinary academic journals, 165 ecocriticism, 1036, 1039, 1047, 1048, 1055 economies of attention, 110–116 extracompositional intermediality, 772 historical genres, 107–110 impulse of technical functionalities, 278, 279

Index in and for the world, 485–487 within interdisciplinary academic journals, 167 Intermédialités Journal 5, 135-158, 159-184 intermedial power nodes, market places, 120–121 interweavings, 637 intracompositional intermediality, 772 ‘laterality’ of technics, 279 literacy, 425–428 materialities, 105–107 as a material practice and artistic event, 21–23 media differentiation, 277 media qualification, 279 metarepresentation, 17 network archaeology, 101 photojournalism, transmediality and, 971–975 political powerplays, 116–120 Object #1, 451 power-nodes, 121 process, 323 serializing the case, 171, 173, 174 Shakespeare, 628, 631, 636, 639, 640, 654 social memories, 125–127 studies, 1–5, 7, 9, 10, 226, 230–232, 235–238, 240, 241, 243, 248, 664, 666 symbolic universes, 121–125 references, 332, 705, 717, 762–765 transposition, 40, 139 turn, 420 types of, 21 Intermedia poetry action poetry, 939–942 concept poetry, 946–949 object poetry, 928–931 postal poetry, 942–946 sound poetry, 932–936 video poetry, 936–939 visual and concrete poetry, 923–928 Intermediary operators, 147 Intermediary references, 139 Intermedia transformations, 294 Internal variation of the Montreal school, 153–155 “The international collective”, 138 International Society for Intermedial Studies, 190, 213, 440 Internet, 5, 18, 48, 65, 112, 114, 117, 118, 267, 278, 419, 420, 445, 447, 451, 552, 743, 823, 957, 987, 988, 1129, 1148, 1151, 1158–1161, 1163, 1174–1176, 1178, 1179, 1186, 1188

Index Internet of Things (IoT), 1172, 1179, 1188, 1189 Intersemiotic aspects of translation, 301–304, 314, 316 Intersemiotic dominant, 300, 301, 305, 307, 315 Intersemiotic interpretation, 292 Intersemioticity, 286, 301 Intersemiotic translation, 5, 35, 36, 39–41, 54, 205, 212, 214, 220, 261, 286–288, 290–294, 296–298, 300–303, 313–316 Intersemiotic transposition, 36, 39–41, 54, 261, 287, 290, 291, 293 Inter-ship, 463, 474 Intersign poetry, 932 Intertextuality, 8, 19, 36, 99, 233, 293, 330, 350, 353, 355–357, 421, 423, 466–471, 477, 786, 805, 809, 869, 891–893, 903, 914–917, 1102, 1152, 1182, 1200 Interventionist Manifesto, 873, 874 Intracompositional intermediality, 772, 1149, 1151 J Jackson, Michael, 113, 114 Jakobson, Roman, 41, 256, 257, 260, 261, 273 Jakobson’s mixed heritage, 260–261 Japanese calligraphies, 868 Jordan, Ken, 443 Joyce, James, 52, 242, 373, 770, 773, 777, 780, 781, 784, 787, 791–794, 796, 840, 841, 850, 853, 885, 886 Judge, A.J.N., 441, 443 K Kabbalah, 582 Kahn, Douglas, 443, 779 Kaprow, Allan, 442, 446, 939–941 Kindle, 9, 1196, 1198–1201, 1207–1210 Kittler, Friedrich, 4, 172, 255, 258, 264, 266–274, 278, 778, 982, 986–989, 1003, 1006, 1016, 1139, 1143 Kittlerian turn, 266–269 Knizak, Milan, 435, 442, 446 Knowledge economy, 440 Knowles, Alison, 20, 434, 435 Kosugi, Takehisa, 434 Kosuth, Joseph, 437 Krämer, Sibylle, 89, 266, 269, 273, 274, 696, 705, 717 Krämer’s media theory, 985

1251 Kristeva, Julia, 355, 356, 360, 423, 781, 891, 916, 1182 Kubota, Shigeko, 434 Kulterman, Udo, 435 L Lacanian imaginary, 263 Lacuna, 255, 257, 276 Lady with Fan in Autumn Breeze, 503 Landscape, 700–702, 709, 712–716 Language, 256, 258, 259, 261, 266, 268, 269, 275, 281, 1182, 1183 Lecturers, 807, 814–817, 820, 826–828 Legisign, 260 Leppert, Richard, 696, 705, 713, 714, 844 Les Peintres cubistes, 870 “Let’s play” videos, 1021 Library interiors, 680 Liminal space, 439, 443 Linguistic inquiry, 259, 281 Linguistic landscape, 1099 Linguistic theory, 188 Linguists, 1183, 1215 Linnaeus University Centre for Intermedial and Multimodal Studies (IMS), 186, 194–197, 200 Liszt, Franz, 694, 696, 699, 700, 703–705, 708, 712–715, 717 Literacy, 416–422, 425, 427, 428 adaptation, 815–819 modernism, 770, 777, 780, 781, 785, 794 Literary theory, 234, 396, 420, 637, 668, 770, 794, 891, 1152 Literature, 3, 4, 8, 14, 18–21, 23, 26, 32–37, 43, 44, 49, 50, 54, 58, 81, 140, 149, 167, 170, 173, 186–188, 191–195, 197, 205, 208, 212–214, 230–234, 236, 238, 243, 245, 286, 288, 304, 310, 340, 347, 348, 360, 371, 392–401, 403, 404, 406–411, 420–425, 439, 468, 522, 523, 557, 569, 667, 699, 712, 772, 774, 776, 778, 780–785, 788, 789, 805, 819–826, 833–856, 904, 911, 915, 927, 938, 955, 975, 981, 1001, 1006, 1035, 1036, 1048 Literature in music, 772, 785, 836 Literatur und Musik in der klassischen Moderne: Mediale Konzeptionen und intermediale poetologien, 786 Live performance, 101, 752–754, 758, 762, 765, 848, 1153 Living together, 136, 142, 143, 145, 148–153, 155, 156

1252 Lockean thesis, 256 Logo, 522, 542–545, 874, 1142 Logocentrism, 85, 354, 357, 675 Lotman, Yuri, 262–264, 266, 276, 278, 478, 479 Ludic discourse analysis, 1062, 1070, 1072, 1073 Ludic narratives, 1064, 1066 Ludification, 102, 110–112, 114–116, 129 Ludo-textual analysis, 1062, 1070, 1072, 1078 Lyrical mode, 728–729, 740–742 M Machines, 144, 258, 265, 266, 268, 274, 275, 566, 572, 581, 586, 610, 613, 779, 989, 1017, 1019, 1021, 1049, 1156, 1206, 1217, 1222, 1231, 1232, 1235, 1237 Machinima, 1020 Machlup, Fritz, 440 Maciunas, George, 434, 438, 439, 442, 446 Mac Low, Jackson, 434, 935 Magic lantern, 141, 668, 677, 678, 687, 760, 762–765 Mail art and video art, 438 Manifesto Antropófago, 90–902, 904, 905, 907, 908, 911, 913–916, 918 Manipulation, 43, 117, 262, 266–268, 270, 271, 281, 470, 558, 582, 843, 889, 935, 989, 1006, 1009, 1014, 1019, 1054, 1204 Mann, Thomas, 774, 783, 784, 794–795, 838, 839, 841, 843 Manovich, Lev, 268, 276, 278, 281, 957, 972, 989, 990, 992, 994, 996, 1144, 1153 Many-to-many communication, 1171, 1174, 1177, 1178 Many-to-one communication, 1171, 1174, 1178, 1184, 1189 Mapping, 142, 163, 173, 254, 315, 643, 736, 754, 835, 981, 991, 1010, 1067, 1106, 1138 Maravilla, 557 Marcus, Laura, 66, 352, 776, 777, 779, 789 Market places, 120, 121 Markov-chains of letters, 272 Marx, Karl, 770, 1172 Mass communication, 233, 779, 1172, 1176, 1177, 1183 Massively multiplayer online roleplaying games (MMORPGs), 1014 Masters, R.E.L., 443 Material culture, 697, 1003, 1016

Index Materiality, 15, 22, 54, 61, 85, 86, 89, 102, 106–108, 122, 139, 142, 146, 192, 214, 229, 268, 272, 275, 280, 304, 327, 337, 349, 396, 421, 485, 526, 616, 673, 697, 717, 734, 856, 866, 868–871, 878, 889, 890, 928, 943, 946, 981–983, 995, 1003, 1016, 1064, 1102, 1120, 1123, 1136, 1138, 1201, 1216, 1222, 1223, 1231, 1236 Material modality, 18, 192, 270, 279, 281, 544, 612, 613, 619, 681, 886, 983, 984, 1042, 1202, 1216 Materials, 14, 15, 18, 23, 24, 40, 43, 110, 113, 114, 167, 191, 275, 296, 379, 580, 588, 589, 606, 609–614, 620–623, 640, 651, 695, 702, 815, 818, 868, 945, 1019, 1186, 1197, 1231, 1232 The Matrix, 375, 378, 380, 383, 934 Matter, 15, 78, 82, 83, 89, 136, 144, 151, 175, 180, 257–259, 315, 342, 351, 394, 397, 474, 483, 527, 535, 569, 612, 629, 636, 732, 734, 811, 870, 872, 950, 962, 973, 999–1025, 1053, 1065, 1068, 1128, 1151, 1197, 1222, 1229, 1231, 1234, 1237 Matthaei, Renate, 436 Maxfield, Richard, 442 McKeich, Murray, 443 McLuhan, Marshall, 3, 4, 20, 25, 74, 82–84, 198, 255, 266, 268, 271, 274–278, 281, 282, 351, 385, 419, 426, 440, 445, 449, 541, 698, 726, 959, 980, 982–988, 994, 996, 1017, 1172, 1189, 1231 Meaning-making concept, 464, 479 Mechanics, 754, 763, 1006, 1007, 1062, 1063, 1065, 1069, 1070, 1075–1078, 1080–1083, 1154–1156, 1160, 1163, 1181 Media, 2–9, 14–26, 32–39, 41–45, 48, 49, 51, 52, 55–57, 60, 64, 75, 77, 80, 85, 136, 138–152, 154–156, 322, 324, 326, 429, 1090–1096, 1098–1101, 1103–1106, 1108, 1110, 1149, 1150, 1154–1156, 1160, 1161, 1163, 1232 archaeological and intermedial perspectives, 580 archeology, 1003, 1016, 1018 awareness, 683 borders, 191 combination, 756–759, 869, 912–913 conscious narratology, 370, 372, 385 convergence, 382, 1101, 1103

Index differentiation, 277, 278 economy, 111 history, 664, 667, 686 literacy, 430 media-centred approach to communication, 190–194 “media-conscious” tools of analysis, 376 modalities, 279, 280, 604, 623, 1214–1218 repertoire, 328, 337 representation, 678, 683, 684, 985 semiotic convergence, 274 studies, 34, 37, 348, 666 transformation, 295 transposition, 759–762 Mediality, 54, 56, 62, 90, 91, 102, 111, 136, 138, 143–146, 149, 155, 156, 163, 164, 168, 169, 172, 349, 416, 417, 420, 421, 424–428, 469, 666, 667, 704, 806, 828, 836, 840, 983, 1065, 1090, 1093, 1117–1119, 1149, 1214 Medial modalities, 1200, 1202, 1208, 1209 Medial properties, 52, 162, 175–177 Medial reference, 1200–1202, 1208, 1209 Medial transfer, 696 Medial transposition, 20, 24, 38, 41–42, 286, 294, 299, 300, 314, 496, 696, 705, 912, 1200, 1206, 1209 Mediation, 5, 7, 18, 87, 89, 140, 142, 144, 145, 154, 155, 167, 169–171, 175, 177, 179, 181, 192, 195, 196, 259, 277, 290, 294, 295, 304, 310–312, 316, 351, 357, 359, 360, 421, 426, 630, 643, 644, 680, 684, 686, 780, 781, 825–828, 854, 983, 993, 995, 1001, 1008, 1037, 1044, 1052, 1062, 1063, 1068, 1119, 1124, 1149, 1181–1183, 1203, 1204, 1207, 1225, 1237 Medieval times, 114, 115, 438, 526, 545 Medium, 17–19, 37, 39–41, 75, 77–81, 83–86, 88, 349, 351, 354, 355, 359, 959, 960, 963, 973, 1090, 1093, 1094, 1096, 1098–1100, 1106 centred model of communication, 605, 607–609 of reproduction, 680, 683 specificity, 74–76, 79, 80, 82, 84–90, 981, 986, 996 Mega-narrator, 405, 406, 410 Mekas, Jonas, 434 Memories, 117, 125–127, 196, 476, 526, 639, 650, 651, 708, 709, 715, 716, 774, 1022, 1228

1253 Mendelssohn, Moses, 647, 697–699 Meraviglia, 557 Meta-awareness, 668, 669 Metacommunication, 9, 1171, 1174, 1180–1186, 1188, 1189 Metalanguages, 1171, 1182, 1183, 1185, 1189 Metalepsis, 664, 668, 675 Meta level, 624, 636, 668 Metamedia, 9, 1170–1174, 1176, 1177, 1179–1181, 1183–1186, 1188–1190 Metamediality, 9, 1169–1190 Metaphor, 4, 15, 25, 42, 189, 261, 262, 265, 272, 280, 383, 487, 525, 537, 539, 572, 593, 599, 608, 621, 628, 636, 645, 677, 826, 827, 850, 901, 904, 910, 1155, 1226 Metapictures, 16, 17, 667 Metareference, in nineteenth-century pictorial press anterior pictures, 680–686 contrasting image types, 676–680 image and text relations, 672–676 postmodernity, 666 Metarepresentation, 17 Meta-theatre, 572, 648, 651 Metonymy, 525 Metz, Christian, 262–264, 407 Mid-Ming Dynasty, 502 Milieu, 136, 138, 143–146, 148–151, 153–156, 163, 169, 171, 174–176, 181, 582, 592, 710, 712, 765 Milieux, 164, 170, 177 Milman, Estera, 439 Mimetic/mimesis, 368–370, 372–374 Mind, 35, 49, 64, 80, 81, 84, 85, 103, 105, 118, 119, 121, 125, 162, 199, 212, 226, 228, 229, 247, 260, 265, 277, 278, 331, 332, 337, 342, 344–346, 348, 356, 396, 397, 399, 419, 429, 448, 472, 477, 483, 485–487, 498, 501, 533, 534, 545, 556, 561, 584, 607, 608, 614, 632, 642, 685, 716, 730, 731, 733, 743, 771, 774, 783, 819, 820, 824, 826, 853, 874, 887, 917, 922, 928, 933, 940, 949, 985, 995, 1066, 1203, 1220, 1221, 1223–1225, 1229, 1231, 1233, 1234, 1237 Mind-matter continuum, 1233, 1237 Misty River and Layered Peaks, 511, 512, 514, 515 Mixed media, 4, 18, 20, 34–37, 42, 43, 49, 55, 191, 350, 351, 430, 496, 517, 726, 795, 869, 886, 908, 912, 922, 924, 927, 936, 1090, 1120

1254 Mobile media, 1018, 1149, 1161 Modality/modalities, 18, 55, 188, 189, 191–195, 197, 199, 200, 270, 277, 279, 281, 295, 333, 465, 527, 528, 530, 535, 541, 544, 561, 886, 983, 984, 986, 1042, 1051, 1117, 1118, 1123, 1130, 1202, 1203, 1214–1216, 1224–1228 Modelling, 162, 163, 166, 256, 261, 480, 981, 1067 Modern art theory, 697 Modern illustration, 682, 683 Modernity, 173, 236, 342, 353, 358, 360, 545, 561, 569, 682, 775, 776, 780, 786, 787, 795, 796, 842, 847, 902, 906, 986, 1034, 1055, 1066, 1116 Modern times, 5, 39, 97, 100, 115, 121–123, 232, 328, 450 Moffett, Kenworth W., 443 Moles, Abraham, 435 Monarchical absolutism, 554–555, 564–565 Monogram, 673 Monstration, 147, 370, 371, 404, 405, 938 Monstrator, 370, 373 “The Montreal crucible”, 138 Montreal school affinities with other intermedial perspectives, 150, 151 of intermediality, 166 of intermedial studies, 5 intermedial transposition, 139 intermediary references, 139 internal variation of, 153–155 and living together, 142–148, 150 media, 140 reflexive definition of the contours of, 136–138 technological innovation, 141 Moving images, 385, 406, 624, 670, 676–679, 740, 741, 752, 753, 755, 762, 763, 946, 957, 982, 986, 993, 1008, 1049, 1098, 1216 Moving panorama, 677, 678, 687 MT72 model, 926 Multimedia, 14, 34–37, 121, 149, 205, 206, 293, 305, 311, 347, 348, 385, 420, 433–452, 496, 517, 559, 588, 645, 652, 740, 754, 759, 869, 894, 908, 909, 912, 1102, 1109, 1120, 1148, 1154 Multimediality, 14, 106, 214, 347, 350, 351, 355, 1095, 1149, 1153 Multimodal and multimedial phenomena, 806 Multimodality, 14, 17, 26, 186, 188, 195, 197, 214, 220, 300, 304, 350, 366, 404, 854–855, 1124, 1149, 1150, 1153, 1197, 1200–1202, 1205, 1209

Index Multimodal modernism, 775–777 Multimodal platforms, 1197, 1199, 1203–1206, 1210 Multimodal work, 287, 291, 295, 297, 299, 301, 305, 315, 316 Museum, 45, 53, 54, 63, 64, 126, 178, 233, 237, 239, 298, 329, 410, 437, 445, 450, 500, 502, 504, 505, 511–513, 515, 517, 563, 604–612, 615, 618–623, 685, 706, 708–710, 873, 877, 909, 933, 965, 968, 971, 974, 981, 990, 995, 1019, 1108, 1186 Museum of Modern Art, 63, 329, 437, 450, 995 Music, 3, 6–9, 14–16, 18, 20, 24, 32, 36, 37, 40, 41, 43, 44, 54, 55, 66, 106, 113, 186, 208, 227, 232, 241, 244, 304, 398, 409, 410, 443, 565, 593, 630, 668, 693–717, 725, 755, 756, 771, 772, 775–783, 785–790, 808, 812, 817, 834, 835, 840, 844, 845, 851, 856, 1009, 1098, 1116, 1121, 1122, 1127, 1129, 1148, 1162, 1163, 1197, 1199, 1203–1207, 1209, 1236 aesthetics, 701 listening to, 844–846 literary transformations of, 841–842 modernist style, influence on, 784–785 popular, 787–788, 847–848 representation, 839–840 studies, 835–836 transgressive role of, 854 transmedial and media-specific aspects, 837–839 Musicalization, 14, 19, 41, 55, 292, 314, 775, 779, 781–783, 785, 789, 791, 793, 795, 840, 849, 933 Musical paintings Della Croce, Johann Nepomuk, 707, 708 Haussmann, Elias Gottlob, 705, 706 Kwiatkowski, Teofil, 708 Schmid, Julius, 709 Siemiradzki, Henryk, 711, 712 Von Schwind, Moritz, 709 Musical pictures, 694 Musico-literary, 8, 772–775, 778, 779, 781–787, 789–791, 793–797, 835–837, 846 Musicology, 186, 348, 430, 771, 775, 782, 796, 835, 843, 855, 1004, 1009, 1011 and literary studies, 782–784 Mystery-plays, 5, 114, 115 N Narrative environments, 25, 328 Narrative fiction, 48, 49, 53, 247, 988 Narrative medium, 398, 1002, 1005, 1094

Index Narrative theory, 240, 241, 244, 370, 371, 470, 1008 Narrativity, 24, 75, 86, 109, 196, 244, 367–373, 380, 385, 392, 393, 397, 402, 403, 407–409, 421, 463, 483, 717, 740, 772, 780, 782, 1047, 1063 Narrativization, 373 Narratological tools, 371, 402 Narratology, 6, 86, 240–242, 365–386, 392–394, 396–398, 402–405 Narrator, 6, 178–180, 247, 368–370, 372, 373, 375, 385, 392–397, 399–411, 470, 474, 622, 780, 807, 814, 820, 826, 828, 852 National Library, 664, 665, 671, 674, 680, 681, 683 Natural sign systems, 699 Neobaroque, 573 Neptuno alegórico, 588, 593–596 Networked media, 9, 1120, 1174, 1195–1210, 1237 Network fiction, 1148, 1158–1163 Neuromancer, 1175 Neutralizing of media, 985 New knowledge system, 555–556 New media, 22, 23, 25, 26, 39, 44, 57, 82, 104, 110, 140, 152, 156, 210, 275, 278, 322, 375, 441–443, 449, 450, 635, 648, 653, 654, 735, 759, 782, 923, 982, 989, 990, 1023, 1101, 1102, 1125, 1144, 1170, 1175, 1200, 1201 News reporting, 392, 400–402 Newton, 269, 442, 731 Nineteenth-century pictorial press, see Metareference, in nineteenth-century pictorial press Nintendo Entertainment System, 1018 The Noble Features of Forest and Streams, 511 Nodes, 5, 120–121, 145, 175, 181, 195, 197, 1100, 1148, 1151, 1157, 1161, 1162 Noise, 107, 144, 146, 164, 246, 263, 270–274, 278, 773, 775, 778–780, 796, 811, 846, 854, 856, 873, 932, 985, 992, 993, 1116, 1138, 1198, 1220, 1227 Non-discursive music, 702 Nonhuman, 154, 156, 217, 790, 1036, 1198, 1218, 1222, 1223, 1227–1230, 1233–1237 Nonsequential writing, 1151 Non-subaltern intertextuality, 8, 915, 916, 918 Novohispanic baroque system, 593 Ny illustrerad tidning, 7, 664, 667, 668, 670, 671, 674, 679, 681, 682, 684

1255 O Object-painting, 870 Object poetry, 928–931, 941, 942, 949, 950 Ocularization, 370, 371, 374 Oldenburg, Claes, 442, 446, 608, 618 One and Three Chairs, 437 One-to-many communication, 1176 One-to-one communication, 1175 Ono, Yoko, 20, 434, 442 Ontological intermediality, 21 Opera, 18, 20, 21, 36, 38, 49, 124, 140, 176, 233, 241, 243, 245, 289, 290, 294, 296, 297, 301, 371, 445, 447, 553, 559, 565, 568, 569, 573, 728, 753, 756, 760, 761, 772, 776, 790, 810, 843, 912, 915, 1042, 1200 Operational qualifying aspect, 605, 614, 616, 617, 620 Oral language, 269, 281 Oral literature, 392, 400, 401, 406 Oral storytelling, 86, 400, 401 Organon model, 256 Ornamental Impulse, 557–558, 573 P Packer, Randall, 89, 443 Paik, Nam June, 434, 435, 440, 442 Paintings, 41–43, 48, 51, 54, 56, 61–63, 76, 100, 105–107, 116, 122, 125, 139, 228, 280, 289, 298, 299, 326, 334, 344, 356, 359, 373, 385, 398, 407, 408, 444, 471, 472, 482, 483, 496, 499–501, 503, 511, 563, 573, 589, 595, 604, 611, 612, 624, 637, 639, 668, 684, 701, 702, 705, 708, 711, 740, 760, 763, 868, 879, 894, 944, 1162 “Palace Lament”, 505 Paragonal conflict, 52, 342, 345, 348, 358, 361 Paragone, 51, 53, 56, 59, 64, 113, 226, 341, 343, 346, 360, 675 Paramediality, 1090, 1092, 1104–1110 Paratexts, 36, 379, 380, 383, 639, 853, 1003, 1011, 1015, 1021–1023, 1025, 1106, 1108, 1197, 1198, 1203, 1206, 1208, 1209 Paratextuality, see Paramediality Paratextual materials, 1197 Participation, 25, 44, 48, 57, 108, 109, 213, 371, 375, 383–385, 1116–1118, 1122, 1124, 1126–1131, 1149, 1160, 1163, 1172, 1186

1256 Participatory culture, 9, 634, 635, 747, 1003, 1013, 1019, 1020, 1022, 1023, 1118–1121, 1130, 1131, 1160 Participatory medium, 1094 Pathos formula, 146, 147 Patterson, Ben, 434 Pedagogy, 243, 340, 417, 419, 423, 426, 428, 430, 431 Peirce, Charles S., 9, 188, 193, 255, 258–260, 265, 268, 281, 331, 478, 1215, 1218 Performance, 3, 7, 14, 20–22, 26, 32, 37, 40, 86, 101, 106, 107, 114, 140, 148, 179, 192, 239, 296, 304, 327, 401, 430, 433, 440, 442, 448, 497, 505, 523, 525–528, 531–534, 542, 545, 553, 559, 564–569, 572, 753, 754, 756–762, 764–766, 838, 846, 848, 851, 853–856 Performativity, 154, 473, 487, 560, 561, 728, 741, 835, 853, 913, 1221 Personification, 20, 61, 308, 468, 483, 526–530, 542, 562, 566 Pfeiffer, K. Ludwig, 696, 698 Phaedrus, 359, 1170 Phantasm, 712, 716 Phillpot, Clive, 439 Photographic prints, 878 Photographs, 14, 16, 18, 19, 21, 22, 40, 43, 48, 52, 82, 84, 149, 152, 155, 173, 193, 198, 259, 262, 280, 356, 373, 399, 406, 434, 622, 643, 670, 680, 683, 684, 763, 890, 957, 960, 962, 964, 966–971, 973, 974, 1004, 1107, 1138, 1154, 1162, 1187 Photojournalism definition, 956–958 history of, 969–971 and intermediality, 958–961, 971–975 to photography, 966 transmediality, 971–975 Photomontage, 868, 869, 878, 886, 890 Photopia, 1155 Picasso, Pablo, 866, 870–872, 876, 882, 884 The Picasso Papers, 872 Pictorial illusionism, 664 Pictorial realism, 673, 675, 763 Picture architecture, 1097 Picture of the Elegant Gathering in the Western Garden, 516, 517 Pipa, 507 Platforms, 5, 9, 14, 24, 25, 86, 89, 112, 118, 119, 165, 166, 206, 211, 375, 526, 634, 635, 643, 816, 962, 986, 1000, 1013, 1018, 1020, 1102, 1103, 1120, 1129, 1148, 1158, 1159, 1161, 1170, 1171, 1174, 1195–1210, 1237 Player-made products, 1020, 1021, 1025

Index Player story, 1015 Playing practices, 1002, 1019 The Pleasures of the Enchanted Island, 566–568 Plot, 42, 61, 62, 207, 245, 289, 296, 311, 313, 328–330, 342, 373, 378, 380, 381, 384, 569, 644, 650, 702, 729, 760, 783, 795, 808, 821–823, 842, 843, 846, 849–852, 934, 936, 1001, 1014, 1100, 1140, 1154, 1156, 1161 Plurimediality, 9, 139, 351, 772, 774, 1090, 1092, 1094, 1095, 1099, 1106, 1110, 1149, 1154, 1160, 1163 Plurisemiotic environments, 329 Poelectrones, 926, 927 Poem-essay, 947 Poesia é risco, 934 Poesie visive, 438 Poetry, 15, 17, 20, 23, 38, 48, 49, 51–54, 56, 57, 66, 76, 77, 80, 81, 91, 187, 188, 226–229, 231, 233, 235–237, 239, 242, 247, 271–273, 277, 301, 342–347, 402–403, 446, 448, 496–503, 505, 507, 511, 514, 516, 517, 565, 568, 577–599, 651, 696, 700, 713, 717, 774, 775, 778, 779, 838, 840, 847, 850, 853, 885, 894, 907, 908, 921–950, 967, 1035, 1045, 1053, 1055 Political and economic power, 117, 119 Polyphony, 126, 355, 470, 782, 785, 845, 850, 852, 853, 869, 885 Polysemiotic translation, 300–302, 304, 314, 315 Polysystem theory, 299, 315 Popular music, 113, 304, 787–788, 842, 847, 848, 855, 907, 911 Postal poetry, 922, 942–946 Postcolonial narratives, 854 Posthuman, 9, 637, 726, 736, 1038, 1213–1237 Postmodernity, 666, 667 Poststructuralist, 5, 6, 88, 339–361, 730, 916, 986 Post-truth, 248, 957 Power definition, 103 as typical forms and processes, 103 Practice, 4, 6, 8, 19–22, 75–78, 80, 83, 84, 86, 88, 89, 111, 166, 170, 197, 212, 237, 239, 243, 254, 267, 280, 281, 308, 324, 326, 330, 337, 357, 366, 372, 417, 419, 423, 425, 450, 451, 462, 464, 468, 472, 473, 490, 500, 511, 525, 526, 528, 538, 542, 561, 565, 592, 609, 613, 628, 630, 631, 633, 645, 647, 651, 654, 694, 699, 700, 728, 731, 732, 752, 765, 785, 805,

Index 836, 852, 853, 868, 886, 893, 894, 914–916, 918, 927, 943, 944, 947, 957, 959, 966, 967, 973, 981, 991, 1003, 1005, 1011, 1016, 1022, 1024, 1077, 1083, 1102, 1108, 1131, 1149, 1150, 1172, 1173, 1176, 1182–1183, 1185, 1189, 1223 Prague School, 256, 257 Prehistoric caves, 116 Prehistoric times, 116, 606 Pre-posterous history, 481, 483, 485 Presence, 15, 16, 50, 60–62, 76, 107, 122, 136, 138–140, 143, 150, 152–154, 156, 173, 176, 180, 181, 212, 255, 257, 264, 304, 314, 325, 330, 337, 359, 360, 370, 372, 392, 393, 397, 425, 470, 477, 478, 488, 526, 527, 562, 563, 582, 610, 614, 634, 638, 650, 704, 711, 727, 731, 738, 780, 793, 827, 848, 849, 892, 958, 959, 972, 990, 991, 1008, 1009, 1072, 1104, 1124–1128, 1131, 1149 Pre-text, 305–307, 462, 467, 470, 472, 476, 477, 479, 480 Print, 17, 18, 22, 23, 25, 37, 84, 86, 118–120, 262, 305, 312, 354, 427, 430, 488, 524, 534, 541–545, 612, 667, 670, 675, 679, 680, 684–686, 819, 824, 826, 872, 916, 926, 928, 932, 942, 950, 957, 961, 962, 967, 971, 972, 974, 1016, 1105, 1150, 1170, 1181, 1232 Printed to emblem, 539–543 Printer, 523, 524, 531, 537–543, 545 Printer’s marks, 524, 537, 540–543 Programmed hypertexts, 1156 Prologues, 808, 809, 811 ProQuest database, 165, 168 Psychic automatism, 890 Psychoanalysis, 262, 263, 482, 793 Publications, 31–33, 50, 54, 100, 105, 137, 165, 166, 170, 187, 195, 204, 205, 208, 213, 216, 234, 235, 237, 241, 244, 248, 436, 467, 611, 636, 811, 874, 887, 890, 908, 966, 970, 1035, 1090, 1101, 1106, 1180 Public pictorial representations and portraits, 714 Q Qualified media, 193–195, 197, 198, 279, 280, 282, 331, 422, 424, 429, 604, 605, 609, 614, 616, 623, 629, 670, 678, 680, 683, 869, 913, 980, 983, 1000–1002, 1004, 1005, 1007, 1010, 1011, 1013, 1019, 1024, 1025, 1035, 1041, 1042, 1051, 1066, 1069, 1118, 1119, 1121, 1131, 1232

1257 Qualisign, 260, 1225 Quasisemiosis, 259 Quotation, 6, 40, 43, 74, 142, 152, 272, 324–326, 328, 336, 462, 464, 466–474, 476–478, 481, 483, 490, 684, 685, 771, 816, 845, 885, 891–894, 916, 965, 1187, 1230 R Rachmaninov, Sergei Vasilyevich, 270 Radio adaptation, 804, 807, 819–825 Radio programming, 821, 828 Rajewsky, Irina O., 8, 19, 20, 38, 49, 55, 99, 167, 190, 194, 204, 211, 213, 276, 286, 290, 305, 332, 349, 350, 417, 421, 426, 428, 496, 630, 633, 666, 695, 696, 705, 717, 739, 753, 755, 772, 786, 826, 840, 845, 869, 912, 913, 922, 923, 981–983, 985, 1042, 1094, 1124, 1127, 1149, 1150, 1200–1202, 1207–1209 Ravicz, Marilyn Ekdahl, 443 Reading media, 417, 419, 427, 428 Reception, 14, 32, 35, 36, 39, 57, 109, 112, 196, 198, 211, 238, 255, 262, 265, 296, 301, 302, 322, 349, 376, 377, 436, 524, 582, 588, 629, 642, 647, 654, 697, 698, 703, 710, 717, 733, 819, 825, 846, 855, 915, 933, 942, 960, 1012, 1039, 1092, 1096, 1102, 1104, 1157, 1159, 1163, 1196, 1197, 1209, 1233, 1237 Receptor, 180, 328–331, 333, 334, 337 Recommendation, 1196, 1198, 1199, 1203, 1206, 1207, 1209 Recommended experience, 1120, 1195–1210 Re-creation, 293–297, 315, 327, 503 hypertextual re-creation, 296 mediated re-creation, 294 Recycling, 43, 108, 110, 113–115, 118, 119, 123, 127, 464–466, 469, 475–477, 894, 1076, 1082, 1172, 1178 Reduction of signs, 274 Regular script, 505, 509 Relief painting, 868, 870, 871 Remediation, 21, 24, 25, 105, 106, 113, 114, 116–120, 126, 127, 151, 180, 205, 224, 275, 278, 293, 295, 322, 332, 350, 421, 634, 648, 682, 753, 761, 825, 970, 971, 985, 987, 989, 1102, 1103, 1200 Renaissance, 20, 22, 51, 77, 226, 227, 343, 360, 399, 444, 522, 531, 534, 536, 539–541, 545, 552, 553, 555, 556, 561, 562, 568, 569, 584, 604, 608–610, 618, 620, 639, 647, 648, 650, 684, 731, 788, 990, 1062, 1070, 1139

1258 Research groups, 204, 205, 207, 208, 210, 213, 215, 216, 221 Research media, 165, 167–170 Re-signification, 8, 865–895 Revues, 161, 756–759 Rhetoric, 4, 16, 40, 62, 76, 89, 236, 241, 244, 344, 350, 354, 360, 522, 523, 525–527, 532–534, 537, 587, 588, 592, 902, 966, 1005, 1006, 1024, 1078, 1124, 1125 Roblox, 1020 Romantic/romanticism, 79, 86, 88, 347, 348, 351, 394, 471, 524, 536, 572, 694, 699–702, 712, 713, 715, 717 Romantic landscape, 700, 702, 713, 714 Rossini, Gioachino, 712 Roth, Dieter, 434 Running Script, 505, 507, 509, 515 S Salmose, Niklas, 7, 9, 87, 90, 189, 195, 684, 769–797, 1033–1055 San Francisco State University Experimental College, 439 Sapir-Whorf hypothesis, 449 Sassen, Saskia, 440 Schirrmacher, Beate, 3, 5, 8, 18, 49, 55, 185–200, 416, 418, 425–429, 616, 666, 688, 772, 833–856, 909, 1041, 1042, 1150, 1217 Schmid, Julius, 19, 100, 125, 128, 446, 709, 710, 716, 1178 Schmidt-Burkhardt, 446 Schweighauser, Philipp, 779, 842, 846 von Schwind, Moritz, 709, 710, 712, 716 Scopic regime, 66, 358, 561, 564 Screen, 7, 17, 19, 41, 85, 139, 143, 144, 166, 198, 211, 262, 280, 295, 305, 311, 367, 368, 443, 448, 496, 507, 628, 631–634, 637, 639–641, 652–654, 670, 677, 726, 734, 737, 739, 743, 759–763, 765, 805, 808, 811, 813–815, 817, 825, 827, 941, 972, 981, 983, 985, 986, 989–996, 1000, 1008–1010, 1016, 1041, 1068, 1094, 1100, 1104, 1125, 1148, 1152, 1157, 1161–1163, 1175, 1198–1200, 1202, 1204, 1206, 1207, 1237 Screenplay, 622, 765, 810, 891, 1023 Scripts, 295, 351, 498, 572, 819, 823, 1105, 1237 Seal, 496–498, 501–503, 505, 507, 509, 510, 517 Second-generation electronic literature, 1154

Index Self-reference, 668, 728 Self-reflexivity, 553, 558, 988 Self-taught programmers, 1020 Semiosis, 212, 214, 258–260, 262, 265, 266, 269, 272, 274, 281, 360, 478, 486, 732, 944, 1223, 1225, 1228, 1232 Semiosphere, 275, 276, 464, 470, 477–483, 485 Semiotics, 5, 9, 87, 90, 99, 103, 166, 188, 210–214, 236, 240, 253–282, 293, 350, 354, 368, 385, 464, 478, 479, 1171, 1181, 1215, 1224, 1230–1233 asynchronous confluences in the formation of Saussure’s semiology, 256, 258 A Theory of Semiotics, 261, 262 blindness, 274 codes, 292, 295, 301, 316 eco’s advances and sidesteps, 261 emergence of media difference in film and cultural semiotics, 263 machines, 275 media-semiotic convergence, 274, 276 modalities, 193, 279, 280, 609, 611, 612, 614, 620, 623, 1199, 1201–1205, 1207–1210 Peircean way, 258, 260 semiotic process in Peirce, 258, 260 symbolic, sign and signal, 269, 270 three aspects of ‘sign, 264, 265 Senses, 20, 35, 76, 84, 90, 101, 191, 192, 200, 229, 245, 246, 331, 344, 359, 427, 443, 535, 560, 582, 596, 609, 612, 613, 684, 699, 725, 758, 774, 776, 780, 792, 873, 874, 982, 986, 987, 1000, 1039, 1144, 1149, 1183, 1187, 1200, 1202 Sensorial modality, 192, 200, 277, 527, 609, 613, 619, 840, 844, 986, 1051, 1202, 1214 Serializing, 171–174, 181 Shakespeare, William, 287, 297, 447, 448, 527, 572, 628–647, 649–654, 912, 914 Shakespearean intermediality, 627–654 Shannon, Claude, 270, 271, 273, 447, 1219, 1220 Sharits, Paul, 442 Shiomi, Mieko, 434, 442 “Shu caopuzhi zhi suocang yuke huazhu sanshou”, 510 Siemiradzki, Henryk, 711, 712, 716 Sign, 34, 35, 41, 52, 54, 61, 109, 125, 162, 191, 193, 236, 255–257, 259, 261–263, 1220, 1224–1228, 1230 Signified, 43, 69, 256, 354, 355, 359, 367, 368, 370, 480, 536, 773, 1006, 1014, 1103, 1141, 1182, 1224

Index Signifier, 43, 256, 257, 263, 354, 355, 358, 359, 368, 480, 536, 809, 814, 847, 1006, 1181, 1182 Sign machines, 275 Silent films, 808, 819 The Sims 2, 1020 Simulation, 9, 16, 85, 570, 629, 648, 791, 796, 889, 982, 992, 994, 1000, 1002, 1005, 1006, 1010, 1012, 1014, 1021, 1067, 1069, 1071–1075, 1077, 1081, 1083, 1099, 1106, 1135–1145 Singularity, 136, 146, 148, 163, 164, 169, 323, 353, 595, 758 Sinsign, 260 Six communicative prototypes, 1176 Small-screen fiction, 1148, 1161–1163 Social immersion, 1128–1130 Social media, 7, 48, 60, 192, 195, 198, 262, 279, 377, 604, 605, 628, 634, 640, 642–645, 649, 652, 823, 1022, 1040, 1055, 1092, 1106, 1108–1010, 1121, 1129, 1158–1161, 1163, 1177, 1178, 1199 Social memories, 125 Social network sites, 1163, 1174, 1177, 1183 Something Else Press, 434, 439, 922 Sonata, 410, 713, 714, 784, 835, 841, 847, 849, 851, 852 “Song of a Round Silk Fan”, 505 Song of Regret, 505 Sonic and audiovisual media, 1116 Sonic modernism, 777–778 audio techniques, sound, noise and soundscapes, 778–781 handbooks, 785–786 musicalization of fiction, 781–782 musicology and literary studies, 782–784 music’s influence on modernist style, 784–785 popular music, 787–788 Sonic Sweden, 788–789 work on individual authors/specific fictional works, 786–787 Sound, 3, 8, 9, 14, 17, 18, 38, 44, 100, 105, 106, 146, 152, 180, 187, 194, 227, 229, 245, 246, 248, 256, 258, 270, 277, 322, 371, 385, 400, 403, 406, 409, 443, 450, 463, 479, 480, 486, 527, 559, 567, 612, 619, 630, 670, 695, 704, 706, 712, 714, 716, 717, 725, 726, 742, 752, 753, 755, 762, 766, 771, 772, 774, 775, 777–781, 787, 789–793, 804–828, 838, 844–846, 853–855, 880, 894, 932–936, 938, 996,

1259 1006, 1008, 1051, 1115–1131, 1151, 1154, 1155, 1174, 1200, 1202, 1204, 1217, 1231 adaptation, intermediality and, 826–828 archaeology, 100, 106 art, 1116, 1118–1120, 1122, 1123, 1127, 1128 film, 817 functions, 806 poetry, 932–936 in relation, 808–814 studies, 846 Soundtrack, 370, 637, 650, 788, 804, 813, 814, 1009 Spatiotemporal modality, 18, 192, 199, 528, 609, 612, 613, 623, 1123, 1202, 1214 Spenser, Edmund, 437, 527 Spotify, 1171, 1184, 1196, 1198–1200, 1203–1207, 1209, 1210 Stage composition, 728–729, 740–742 Stamp art, 438 Still Life with Chair Caning, 870 Still Life with Violin and Fruit, 871 Stofflett, Mary, 443 Storage memory, 125, 126 Storyworld, 330, 366, 371, 374, 376–378, 380–382, 384, 398, 409, 1003, 1014, 1064, 1100, 1148, 1149, 1155, 1156, 1161, 1163 Strategic remediation, 1103 Streaming, 445, 631, 677, 824, 981–983, 1021, 1022, 1025, 1026, 1131, 1177, 1196, 1197, 1199, 1203–1207 Studies of intermediality, 18, 32–36, 39, 40, 44, 45, 128, 332, 334, 337, 350, 425, 1188 Sub-genres, 106, 421, 422, 500, 571, 1148, 1159 Sui generis, 264, 281, 916 Supersemiotic translation, 292 Surrealism, 8, 330, 868, 886, 888, 890, 891, 904 Surrealist collage, 233, 886–891 Su’s Misty Poem, 515 Sustainable development, 9, 1048, 1213–1237 Švankmajer’s stop-motion animation technique, 280 Symbol/symbolic, 6, 18, 20, 22, 43, 79, 103, 113, 114, 121–125, 137, 145, 153, 162, 168, 170, 173, 176, 189, 192, 193, 228, 243, 258, 259, 262, 263, 265–267, 269, 270, 275, 278, 522, 524, 527, 530–537, 540–541, 543, 544, 563, 564, 572–573, 579, 581–584, 587–589, 591, 592, 595–599, 606, 611, 614, 617, 620, 638,

1260 650, 651, 666, 669, 673, 676, 683, 686, 705, 708, 771, 773, 775, 777, 783, 791, 837, 843, 879, 891, 944, 984, 987, 1002, 1012, 1022, 1042, 1050, 1079, 1096, 1105, 1106, 1171, 1174, 1178, 1189, 1203, 1204, 1215, 1225–1227 Symbolic program, 583, 584, 587, 592, 595–598 Symbolic universes, 121–123 Symbolism, 22, 99, 353, 447, 501, 534, 535, 771, 791, 1096 Symbolon, 534, 544, 545 Synchronization, 811, 814, 817, 986 Synthetic intermediality, 21 Synthetic media, 1094 Systematic aesthetics, 697–700 and media hierarchy, 697–700 Systematizations for educational purposes, 422–427 T Talker films, 817 Teaching, 6, 32, 44, 45, 190, 197, 208, 209, 215, 233, 333, 415–431, 439, 478, 479, 489–490, 586, 587, 614, 1054, 1083 intermediality, 417, 420–423, 426 and literacy, 6 Technical media, 8, 128, 140, 143, 198, 211, 255, 264–276, 278, 279, 331, 417, 607, 616, 628, 638, 640, 869, 980, 1000–1003, 1015–1017, 1019, 1023, 1025, 1041, 1049, 1051, 1055, 1118, 1123, 1124, 1216, 1217, 1220, 1233, 1236 Technological blindness, 281 Technological innovation, 141, 1075, 1150, 1179 Technological media, 265–267, 269, 274, 281 Technology, 8, 18, 19, 66, 101, 120, 146, 152, 180, 193, 205, 211, 243, 255, 258, 261, 267, 268, 270, 275–277, 280, 322, 351, 361, 419, 436, 440, 442–445, 447–451, 534, 541, 545, 549, 635, 637–640, 653, 683, 735, 741, 742, 752, 754, 755, 760, 763, 792, 817, 825–828, 925, 942, 957, 972, 973, 984, 986, 999–1025, 1034, 1068, 1076, 1082, 1083, 1092, 1093, 1125, 1136, 1149, 1151, 1152, 1154, 1160–1164, 1186, 1204, 1214, 1217–1219, 1221, 1222, 1232, 1235–1237 Techno-text, 25 Teleology, 667, 1076

Index Telepresence, 1008, 1124, 1125 Television, 8, 17–19, 32, 37, 48, 91, 101, 103, 109, 117–119, 139, 152, 153, 166, 211, 212, 236, 241, 262, 311, 322, 326–328, 356, 376, 377, 383, 418, 442, 444–446, 634, 635, 637, 640, 642, 697, 726, 739, 743, 805, 813, 823, 909, 957, 970, 979–996, 1000, 1001, 1023, 1097, 1103–1106, 1138, 1172, 1174, 1184, 1202, 1216, 1217, 1232 Terminology, 3, 4, 6, 13–26, 35, 44, 145, 186, 194, 296, 297, 406, 418, 420, 421, 425, 428, 462, 554, 724, 789, 795, 840, 846, 869, 870, 912, 936, 981, 983, 1001, 1022, 1035, 1041, 1047, 1070, 1095, 1101, 1102, 1120, 1170, 1176, 1179, 1185, 1198, 1200, 1204 Text-based adventure game, 1007, 1008, 1010, 1155 Theall, Donald, 443 Theatre, 22, 35, 37, 38, 53, 86, 121, 139, 149–153, 166, 170, 171, 179, 192–194, 207, 208, 216, 232–235, 238, 239, 244, 247, 279, 286, 288, 311, 406, 430, 439, 442, 446–448, 523, 538, 552, 553, 559, 560, 567–572, 580, 599, 609, 617, 628, 630, 634, 636, 637, 639, 641, 646, 647, 649, 652, 684, 725–732, 737–739, 742–745, 752–759, 761, 763, 765, 788, 806, 809, 813, 816, 819–822, 824, 827, 885, 901, 907–915, 917, 918, 939, 940, 983, 988, 993, 1005, 1043, 1094, 1098, 1100, 1116, 1117, 1200 Theatricality, 552, 553, 560, 561, 571, 717, 762, 913 Thematization, 55, 279, 668, 765, 784, 840, 1149, 1150, 1160 Theme parks, 9, 915, 1090–1109 The three perfections, 502 Thriller, 113, 114, 1208 Tihuashi, 226, 229, 497, 499, 500, 503, 511, 516 Tofts, Darren, 443 Total medium, 1094 Total translation, 292 Trademark, 542, 777 Traditional Chinese painting, 6, 227, 496–518 Tragicomedy, 553, 569–572 Transfictionality, 376, 378 Transformational intermediality, 21 Transgression, 14, 348, 349, 462, 476, 477, 664, 853–854, 912, 1149 Transhistorical transmedial cabinets, 605, 619–623

Index Translated texts, 300–305, 313, 315 Translation diasemiotic translation, 292 hyposemiotic translation, 292 intermediality in translated texts, 300 intersemiotic aspects of translation (proper), 303 intersemiotic translation, 298, 300 polysemiotic translation, 300–302, 304, 314, 315 supersemiotic translation, 292 total translation, 292 translation proper, 298, 300 translation proper (interlingual translation), 292–294, 298, 300, 302, 303, 305, 310, 314, 315 Translational-proper aspects of intermediality, 312 Translation proper, 287, 292, 298, 299, 313–316 Transmaterialization, 1139 Transmedia dissipation, 1103 Transmedial forms, 1136, 1138–1143, 1145 in Monsters, Inc., 1140–1142 visual imagery in Monsters, Inc., 1142 Transmediality, 6–8, 24, 26, 32, 74, 75, 86, 87, 211, 351, 355, 366, 392, 393, 397, 400, 410, 522, 528, 666, 668, 684, 917, 918, 972, 1013, 1015, 1100, 1101, 1104, 1149 content, 1011 intermediality, 1140 narration, 400 narratology, 6, 366, 367, 372, 373, 376, 385 remediation, 295 snowball narrative worlds, 24 storytelling, 1002 Transmedia marketing, 379–380 Transmedia storytelling, 6, 86, 105, 215, 220, 365–386, 972, 1013, 1014, 1091, 1094, 1102, 1103 Transmediation, 8, 23, 49, 55, 86, 87, 194, 293, 295, 296, 315, 526–528, 540, 545, 596, 598, 599, 683, 785, 795, 840, 841, 900, 901, 915, 917, 1036, 1043–1048, 1055, 1067, 1072, 1074, 1080, 1081 Transmissions, 143 Transmutation, 41, 42, 286–288, 290, 291, 294, 297, 302, 310, 312, 314–316, 359, 903, 904 Transparenticization, 277 Transposition, 16, 20, 24, 32, 33, 36–42, 54, 139, 205, 207, 261, 278, 286, 287, 290, 291, 293, 294, 297–300, 312, 315, 326,

1261 329, 330, 350, 378, 384, 421, 496, 517, 696, 705, 741, 754, 759–761, 765, 783, 805, 912, 922, 1002, 1011, 1013–1015, 1119, 1125, 1127, 1149, 1200, 1206, 1209 Trauma, 62, 65, 842, 854, 903, 973, 1078 Triumph of the Name of Jesus, 553, 562, 563 Tropicalism, 902, 906, 907, 909 Turing, Alan, 261, 271, 274, 1173, 1219 Twitch, 1021, 1022, 1025 Twitter fiction, 1158–1160 U Unbalanced transmedia, 383 Urban and village nodes, 121 Ut pictura poesis, 5, 6, 51, 54, 75, 77, 339–360, 583, 588, 675, 700 V Vautier, Ben, 434, 442 Verbal analepsis, 369 Verbal and literary collage, 879–886 Verbivocovisuality, 928 Victorian realism, 771, 776 Vicuña’s poetry, 935 Video games, 3, 9, 17, 24, 86, 139, 247, 248, 298, 371, 376, 380, 383, 637, 987, 1010, 1024, 1063, 1073, 1079, 1081, 1082, 1116–1122, 1124–1126, 1128, 1131 Video poetry, 922, 936–939, 946 Virtual camera, 1008, 1010, 1137, 1138, 1141 Virtual presence, 1124 Virtual reality (VR), 91, 143, 206, 216, 248, 278, 638, 740, 994, 1116, 1118, 1119, 1124–1126, 1144, 1179 Virtual space, 44, 399, 400, 429, 613, 614, 620, 726, 743, 989, 1008, 1019, 1020, 1123, 1125, 1141 Visual arts, 14, 20, 32, 37, 51, 56, 57, 59, 65, 187, 204, 208, 244, 287, 289, 372, 397, 502, 518, 559–561, 595, 668, 717, 725, 755, 904, 907, 911, 929, 930, 936, 938, 939, 948, 964, 1008, 1011 Visual citation, 321–337 Visual culture, 48, 50, 84–86, 113, 162, 167, 215, 222, 233, 474, 541, 573, 579, 580, 697, 824 Visual economy, 683 Visual poetry concrete movement, 926 definition, 924 Visual rendering, 713

1262 Voice actors, 807, 817, 820 Vostell, Wolf, 434, 442, 446 W Watergate Model, 262 Watts, Robert, 442 Web 2.0, 113, 118, 377, 1160, 1163, 1177 Webster, Frank, 435 Western civilization, 770, 902, 912 Western humanity, 775 Williams, Emmett, 434, 435, 922 Window screen, 981 Wolf, Werner, 14, 19–21, 48, 55, 166, 187, 191, 210, 276, 277, 293, 349, 351, 352, 366, 371, 372, 376, 392, 397, 407, 408, 420–422, 428, 429, 431, 434, 442, 446, 664, 666–669, 696, 772, 776, 778, 782, 785, 791, 793, 794, 797, 835, 836, 840, 845, 846, 849, 851, 852, 1042, 1094, 1095, 1100, 1149, 1150, 1163 Wonder, 7, 66, 67, 178, 311, 551–574, 605, 614–616, 653, 1101, 1124 Woolf, Virginia, 770, 771, 774, 777–792, 796, 841 Word and image, 15, 17, 33, 36, 48, 49, 51, 52, 57, 64, 167, 206–208, 226, 236, 243, 346, 347, 352, 354, 356, 357, 360,

Index 497, 501, 510, 517, 522–524, 526–530, 533–537, 540–543, 586, 652, 664, 669, 686, 807, 873, 959, 963–967, 971 Word music, 772, 782, 794, 795, 851 Work of art, 14, 22, 23, 34, 49, 53, 54, 56, 65, 75, 82, 233, 279, 347, 358, 427, 558, 586, 596, 610, 694, 701, 711, 715, 716, 893, 947, 948, 990, 1094 Works of artifice, 559–560 World Wide Web, 198, 444, 451, 1175, 1237 X Xylographic prints, 680 Xylography, 674, 680–682, 684 Y The Years, 770, 771 Yeats, William Butler, 237, 770, 781, 787 Yongwushi, 499 Youngblood, Gene, 436 Z Zen Buddhism, 357, 442 Zurbrugg, Nicholas, 443