Diachronic Perspectives on Embodiment and Technology: Gestures and Artefacts 3031500857, 9783031500855

Citation preview

Philosophy of Engineering and Technology

Thiemo Breyer Alexander Matthias Gerner Niklas Grouls Johannes F.M. Schick   Editors

Diachronic Perspectives on Embodiment and Technology Gestures and Artefacts

Philosophy of Engineering and Technology Volume 46

Editor-in-Chief Pieter E. Vermaas, Department of Philosophy, Delft University of Technology,  Delft, The Netherlands Series Editors Darryl Cressman, Department of Philosophy, Maastricht University, Maastricht,  The Netherlands Neelke Doorn, Department of Philosophy, Delft University of Technology, Delft,  The Netherlands Edison Renato Silva, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil Editorial Board Members Philip Brey, University of Twente, Enschede, The Netherlands Louis Bucciarelli, School of Engineering, Massachusetts Institute of Technology,  Belmont, MA, USA Michael Davis, Humanities Department, Illinois Institute of Technology,  Chicago, IL, USA Andrew Feenberg, Simon Fraser University, Burnaby, BC, Canada Luciano Floridi, Department of Philosophy, University of Hertfordshire,  Hertfordshire, UK Jun Fudano, Kanazawa Institute of Technology, Nonoichi, Japan Sven Ove Hansson, Division of Philosophy, Royal Institute of Technology KTH,  Stockholm, Stockholms Län, Sweden Craig Hanks, Philosophy Department, Texas State University, San Marcos, USA Vincent F. Hendricks, Center for Information & Bubble Studies,  University | of Copenhagen, København, Denmark Don Ihde, Department of Philosopy, SUNY at Stony brook, Stony Brook, NY, USA Billy Vaughn Koen, Department of Mechanical Engineering, University of Texas,  Austin, TX, USA Peter Kroes, Department of Philosophy, Delft University of Technology,  Delft, The Netherlands Sylvain Lavelle, Ctr for Ethics, Tech & Society, ICAM Paris-Senart Engineering School, Lieusaint-Senart, France Michael Lynch, Cornell University, Ithaca, NY, USA Anthonie W. M. Meijers, Department of Philosophy and Ethics, Eindhoven Univ of Technology, Eindhoven, The Netherlands Duncan Michael, Ove Arup Foundation, London, UK Carl Mitcham, Liberal Arts & International Studie, Colorado School of Mines, Golden, CO, USA Byron Newberry, Baylor University, Waco, TX, USA Helen Nissenbaum, New York University, New York, NY, USA Alfred Nordmann, Institut für Philosophie, Technische Universität Darmstadt,  Darmstadt, Germany Joseph C Pitt, Department of Philosophy, Virginia Tech, Blacksburg, VA, USA

Daniel Sarewitz, Consortium for Sci Policy & Outcome, Arizona State University,  Washington DC, USA Jon Alan Schmidt, Aviation & Federal Group, Burns & McDonnell,  Kansas City, MO, USA Peter Simons, Trinity College Dublin, Dublin, Ireland Jeroen van den Hoven, Department of Philosophy, Delft University of Technology,  Delft, The Netherlands Ibo van der Poel, Department of Philosophy, Delft University of Technology,  Delft, The Netherlands John Weckert, Centre for Applied Philosophy & Ethics, Charles Sturt University,  Canberra, ACT, Australia The Philosophy of Engineering and Technology book series provides the multifaceted and rapidly growing discipline of the philosophy of technology with a central overarching and integrative platform. It publishes on all topics in the philosophy of technology and is open to all research communities across the world. Specifically, it publishes edited volumes and monographs in: · the phenomenology, anthropology and socio-politics of technology and engineering · the emergent fields of the ontology and epistemology of technology and design · engineering ethics and the ethics of specific technologies ranging from nuclear technologies to artificial intelligence · written from philosophical and practitioners’ perspectives and authored by philosophers and practitioners The series also welcomes proposals that bring these fields together or advance philosophy of engineering and technology in other integrative ways. Proposals should include: · · · ·

A short synopsis of the work or the introduction chapter The proposed Table of Contents The CV of the lead author(s) or editor(s) If available: one sample chapter

We aim to make a first decision within 1 month of submission. In case of a positive first decision the work will be provisionally contracted; the final decision about publication will depend upon the result of anonymous peer review of the complete manuscript. We aim to have the complete work peer-reviewed within 3 months of submission. The series discourages the submission of manuscripts that contain reprints of previous published material and/or manuscripts that are below 150 pages/75,000 words. For inquiries and submission of proposals authors can contact the editor-in-chief Pieter Vermaas via: [email protected], or contact one of the associate editors.

Thiemo Breyer  •  Alexander Matthias Gerner Niklas Grouls  •  Johannes F. M. Schick Editors

Diachronic Perspectives on Embodiment and Technology Gestures and Artefacts

Editors Thiemo Breyer Department of Philosophy University of Cologne Cologne, Germany

Alexander Matthias Gerner CICANT Universidade Lusófona Lisbon, Portugal

Niklas Grouls FernUniversität in Hagen Hagen, Germany

FilmEU

Editor-in-Chief Pieter E.Vermaas

Johannes F. M. Schick CRC Media of Cooperation University of Siegen Siegen, Germany

European University Lisbon, Portugal

ISSN 1879-7202     ISSN 1879-7210 (electronic) Philosophy of Engineering and Technology ISBN 978-3-031-50084-8    ISBN 978-3-031-50085-5 (eBook) https://doi.org/10.1007/978-3-031-50085-5 This work was supported by CRC 806 (“Our way to Europe”) © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Paper in this product is recyclable.

Contents

Diachronic Perspectives on Embodiment and Technology: Gestures and Artefacts – An Introduction ��������������������������������������������������������������������    1 Alexander Matthias Gerner, Thiemo Breyer, Niklas Grouls, and Johannes F. M. Schick Playing with Arte(f)actors��������������������������������������������������������������������������������    9 Alexander Matthias Gerner  The Ineffability of Motion in Robotics����������������������������������������������������������   45 Céline Pieters Sophia the Robot as a Political Choreography to Advance Economic Interests: An Exercise in Political Phenomenology and Critical Performance-­­Oriented Philosophy of Technology����������������������������������������   57 Jaana Parviainen and Mark Coeckelbergh  Gestures, Diagrams, and the Craft of Musical Composition ����������������������   67 Vinícius de Aguiar Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances������������������������������������������������������������������������������������   83 Pieter Vermaas  The Philosophy of Gesture and Technological Artefacts������������������������������   97 Giovanni Maddalena  The Orchestration of Bodies and Artifacts in French Family Dinners ������  111 Aliyah Morgenstern and Dominique Boutet Towards an Ecology of Gesture: A Review (And Some Promising Paths)����������������������������������������������������������������������������������������������  131 Antonis Iliopoulos and Lambros Malafouris

v

vi

Contents

 Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?������������������������������������������������������������������������������������������������������  145 Jürgen Richter and Thiemo Breyer The logos of techné: A Case for Technology as Interdisciplinary Anthropology����������������������������������������������������������������������������������������������������  163 Johannes F. M. Schick

Diachronic Perspectives on Embodiment and Technology: Gestures and Artefacts – An Introduction Alexander Matthias Gerner, Thiemo Breyer, Niklas Grouls, and Johannes F. M. Schick

Abstract  This book opens up an interdisciplinary arena of research, contoured by the terms “gestures” and “artefacts,” and is oriented by a diachronic perspective on technology. In the context of anthropology, artefacts and gestures are both essential to understanding human culture and behavior. They are interconnected features that reveal numerous facets of a culture’s ideas, values, social institutions, and communication methods. In a general sense, artefacts are the material entities created, utilized, or altered by humans within a particular cultural context. These can range from implements, clothing, and utensils to works of art, structures, and technological equipment. Artefacts are physical manifestations of human ingenuity, desires, and beliefs. Anthropologists examine them to learn about a culture’s technology, economic systems, creative expressions, and daily behaviors. Artefact analysis can show a society’s level of development, social organization, and cultural standards. Gestures on the other hand are typically conceived as non-verbal means of communication consisting of body motions, facial expressions, and hand signs. Nonverbal cues are fundamental to human contact and communication. Emotions, intentions, attitudes, and cultural meanings can be communicated through gestures. Different societies have distinct gesture systems with specific connotations. Anthropologists investigate gestures in A. M. Gerner (*) CICANT, Universidade Lusófona, Lisbon, Portugal FilmEU, European University, Lisbon, Portugal e-mail: [email protected]; [email protected] T. Breyer Department of Philosophy, University of Cologne, Cologne, Germany e-mail: [email protected] N. Grouls FernUniversität in Hagen, Hagen, Germany e-mail: [email protected] J. F. M. Schick CRC Media of Cooperation,University of Siegen, Siegen, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_1

1

2

A. M. Gerner et al.

order to comprehend how people communicate nonverbally and how cultural norms and social settings influence nonverbal communication. The relationship between artefacts and gestures resides in their potential to offer complimentary insights onto a society. While artefacts reflect how humans engage with their physical environment and produce tangible representations of their culture, gestures disclose the more subtle aspects of human communication and social interaction. Together, these components assist anthropologists construct a comprehensive picture of a culture’s lifestyle, beliefs, and behaviors. In ritualistic performances, for instance, a specific artefact such as a ceremonial mask may be utilized alongside certain motions that show reverence or respect. By examining the mask as an artefact and evaluating the accompanying motions, anthropologists can comprehend the object’s cultural importance and the embodied communication that occurs throughout the ritual. This book opens up an interdisciplinary arena of research, contoured by the terms “gestures” and “artefacts,” and is oriented by a diachronic perspective on technology. In the context of anthropology, artefacts and gestures are both essential to understanding human culture and behavior. They are interconnected features that reveal numerous facets of a culture’s ideas, values, social institutions, and communication methods. In a general sense, artefacts are the material entities created, utilized, or altered by humans within a particular cultural context. These can range from implements, clothing, and utensils to works of art, structures, and technological equipment. Artefacts are physical manifestations of human ingenuity, desires, and beliefs. Anthropologists examine them to learn about a culture’s technology, economic systems, creative expressions, and daily behaviors. Artefact analysis can show a society’s level of development, social organization, and cultural standards. Gestures on the other hand are typically conceived as non-verbal means of communication consisting of body motions, facial expressions, and hand signs. Nonverbal cues are fundamental to human contact and communication. Emotions, intentions, attitudes, and cultural meanings can be communicated through gestures. Different societies have distinct gesture systems with specific connotations. Anthropologists investigate gestures in order to comprehend how people communicate nonverbally and how cultural norms and social settings influence nonverbal communication. The relationship between artefacts and gestures resides in their potential to offer complimentary insights onto a society. While artefacts reflect how humans engage with their physical environment and produce tangible representations of their culture, gestures disclose the more subtle aspects of human communication and social interaction. Together, these components assist anthropologists construct a comprehensive picture of a culture’s lifestyle, beliefs, and behaviors. In ritualistic performances, for instance, a specific artefact such as a ceremonial mask may be utilized alongside certain motions that show reverence or respect. By examining the mask as an artefact and evaluating the accompanying motions, anthropologists can comprehend the object’s cultural importance and the embodied communication that occurs throughout the ritual. While artefacts—implying the human art of making (ars facere)—are usually defined as any objects made or modified by human culture, individual, or community, gestures, on the other hand, are often conceived as a crucial form of human embodied expression, a “communicative movement of the hands” (Streeck, 2015, 27)

Diachronic Perspectives on Embodiment and Technology: Gestures . . .

3

or as “visible action” (Kendon, 2004) yielding linguistic communication. Hence, gestures are involved in creating or modifying artefacts or even communicating with artefacts over time. However, gestures are more than visible spatial actions without words (Sigaut, 2012). They go beyond the explananda of classical linguistics and communication studies concerning artefacts as psychological tools (Vygotsky, 1978) for linguistic mediation. In their semiotic dimension, gestures combine speech, hand movements, body postures, guided attention and social gaze, physical activity, and intentional actions relating to spatially extended (or simulated) artefacts (Mittelberg, 2017). Gestures still have to be theorized further within the context of sign-gesture-artefact articulations, in which the gesture-language relation is one essential part that can be described as epistemic and cognitive (Wundt, 1911 [1900]), acknowledging its importance in language development (Armstrong & Wilcox, 2007). Since gestures can be conceived as discrete bodily acts by which some social meaning is conveyed, e.g., while sharing and joining attention, bowing the head, winking or nodding (Streeck, 2015) to each other, social collaborations via gestures can be inferred in highly sophisticated rule-based human-artefact interactions, e.g., in social robotics or human-AI interfaces. Despite the distinction between instrument—and its schematic functionality—and artefact, as well as its conservation defined as a material or symbolic object and organizer of the activity per se (Rabardel, 1994), in this book, we focus not only on the articulation and mediation of embodied gestures between instrumental, heuristic, epistemic, social-­ communicative, and cognitive functions, but also on human play with technological artefacts, underlining the ludic dimension of artefact and gesture between actors and artificial agential systems. Most importantly, when related to basic units of bodily action, artefacts add an enactive dimension to cultural design and production and enhancements of gestures in human cultures beyond the linguistic focus. As humans evolve by making, the homo faber (Ihde & Malafouris, 2019; Schick et al., 2018) or ergotic (Greek: ergon), i.e., work-related dimension of gestures and artefacts becomes crucial. Thus, gestures have been conceived through artefacts as ergotic (Cadoz, 1994) actions (Roth, 2003), including co-creations, gesturing, and hands-on approaches with physical materials, in which gestures render material instruments in their techniques of use not only technically efficient (Leroi-Gourhan, 1964), but open up horizons of thinking with haptic (Sinclair & de Freitas, 2014) gestures. These are mediated by and engaged with the material (Malafouris, 2018) gestures (e.g., in working with clay). Thus, artefacts become organizers of work in the activity of projective gestures in design processes of manual gestures (Streeck, 2011, 2015)—manufactures (Mittelberg et al., 2017), in which materiality and utilization scheme components (Maschietto & Bartolini Bussi, 2009) are negotiated by embodiments. Gestures even articulate and render the habits and rituals of moving bodies, their corporeality and dynamics, mediality and performativity thematic against the dynamically changing background of environments and ecologies of material affordances (Gibson, 1977, 1979; Norman, 1988), as proposed by engagements within chronoarchitectures of action inside a metaplasticity of mind and

4

A. M. Gerner et al.

culture (Gosden & Malafouris, 2015), processes of enactive individuations (Simondon, 1958), and mimetic play (Callois, 2011). Thereby, gestures allow for a reflection on cognitive (informational, heuristic, enactive, and cultural) ecologies (Hutchins, 2014), enhancing human cognition with constructed artefacts to gain cultural significance in a diachronic perspective on embodiment, including digital design and fabrication (Poulsgaard, 2017). This volume is an expedition, exploring different fields of flourishing research on gestures and artefacts. By bringing together their diverse fruits, its goal is to make the reader understand that the recurring topics, questions, and multi-level similarities are by no means accidental. The choice of authors reflects this idea. An example: from close-up investigations of how robotic gesturing, imitating natural movement, is designed (by Céline Pieters et  al.) to a full-scale narrative of how these artefacts are made realistic by the performance metaphor (cf. Coeckelbergh, 2019) of choreographing their gestures (by Jaana Parviainen and Mark Coeckelbergh), the volume invites the reader to grasp what lies between disciplinary fields of knowledge and different professional approaches. Therefore, scholars from anthropology, archaeology, philosophy, and linguistics were asked to share their research in this interdisciplinary volume. From such a combined perspective, certain gestures appear as inextricably connected to producing and maintaining ecological and technical entities and circumstances. Some of these specialized entities—or artefacts—are designed to be more than merely passive nodes in the web of semiotic gesturing, which is sketched by the point of interest of the first thematic section of the book. In the chapter following this introduction, Alexander Gerner asks whether artefacts can, as performers, play together with humans in communion, and whether they are therefore proper actors? Adopting mixed categories of play as proposed by Rogier Callois (2011), proposed for an extended homo ludens concept, Gerner’s Chapter “Playing withArte(f)actors?” critically assesses these two questions by reflecting on musical probings in performative human–AI and human–robot interplays in music co-performance and AI-trained music composition. In her Chapter “The Ineffability of Motion in Robotics,” Céline Pieters explores the problem of robotic movement and gesturing appearing as “natural” through a description of the daily affairs of those humans who build such robots. The paradox at this point lies in the fact that artificial agents seem to gesture naturally. However, their gesturing has to be understood, according to the self-conception of their creators, as a result of technical problem-solving. Movement is a keyword also in the Chapter “Sophia the Robot as a Political Choreography to Advance Economic Interests,” where Jaana Parviainen and Mark Coeckelbergh team up to respond to the recent debate on robot citizenship. Employing the example of Sophia, a social robot that gave a speech in front of the United Nations and was made a citizen in Saudi Arabia, the mechanisms of exploitation of staged human-like gesturing, driven primarily by economic interests, are discussed.

Diachronic Perspectives on Embodiment and Technology: Gestures . . .

5

That musical composition is not so much a purely cognitive process stemming from formulas or inspiration, but a “thinking with the hands,” mediated by at least two types of artefacts, is the first result of Vinícius Jonas de Aguiar’s contribution “Gestures, Diagrams, and the Craft of Musical Composition.” It leads to the possibility of introducing a diachronic perspective on those gestural techniques, revealing an exciting role of the body in the opening up of music to dissonances and noises. The second topic of the book is inaugurated by Pieter Vermaas’s reflections “Describing Robot Gestures by Design and Agency.” The crucial question about whether and when we should ascribe the status of intentional gestures to robotic movements orients the first chapter of this more theoretical section. The drive toward fundamental questions regarding the conceptual framework for understanding technology and gestures is tackled by Giovanni Maddalena in his Chapter “The Philosophy of Gesture and Technological Artefacts.” Drawing on semiotics and pragmatism, Maddalena unfolds the historical dimension of the failure of philosophy to grasp a matching concept of the synthetic and technology. Thereby, he also includes themes from the phenomenology of gestures (cf. Ferencz-­ Flatz & Popa, 2022). Taking up his semiotic impulse, Aliyah Morgenstern and Dominique Boutet show in their Chapter “The Orchestration of Bodies and Artefacts in French Family Dinners” how much the interrelationship of gestures and artefacts is also needed in the field of linguistic analysis to bring the complexity of human communication into view adequately. However, their analyses are by no means confined to theoretical considerations—they rob something as mundane as a dinner of its self-evidence and make us realize that artefacts disrupt our semiotic categories of the verbal, the gestural, and the actional. Those particular human gestures can be appreciated by referring to material affordances in an ecological world. As stated at the beginning, they are embedded relations in a complex environment and exceedingly expand the diachronic timeline in the third and last part of the book. The articulation of gesture–artefact relations by haptic embodiment is presented as a human ritual that relates artefacts to their material culture (Ingold, 2012) through enactive “thinging” (Malafouris, 2020). Creative artefact processes organize the hylonoetic field of pottery-making by turning material agency gestures into stable forms that culturally endure over time. Lambros Malafouris and Antonis Iliopoulos’s Chapter “Towards an Ecology of Gesture: A Review (and Some Promising Paths)” provides a thorough reflection on gesture. It allows for an explanation of phenomena regarding cognitive implications of gesturing, i.e., forming enactive signs through haptic interaction with the material and the bodily integration of artefacts. Their chapter draws attention to current frameworks of an embodiment of gestures, especially concerning ecological interactions through enacting, extending, and distributing the mind into the physical world, based on examples from anthropological fieldwork on Greek pottery conducted by the authors. Prehistoric stone tools allow for detailed insights into how “thinging works” and how patterns of movements of the human body and the mobility of individuals and groups are created. In their Chapter “Petrified Practice: Is There a Vernacular

6

A. M. Gerner et al.

Choreography of Neanderthal Movements?”, Jürgen Richter and Thiemo Breyer investigate this, starting from the methodological discussion of “processual” and “post-processual” archaeology, to decipher vernacular choreographies of the everyday life of early humans. They use settlement locales on the Crimea peninsula in order to describe three scales of human mobility. All sites were occupied by the last Neanderthal humans, about 45,000 years ago. At a small scale, the chapter reconstructs recipes of stone tool production with gestures of human arms, hands, and fingers, all within the kinesphere or kinaesthetic bodily space of an individual Neanderthal. The embodied performances of knapping are considered “body techniques” in the sense of Marcel Mauss. Johannes Schick, in his Chapter “The logos of techné—A Case for Technology as Interdisciplinary Anthropology,” finally explores the potential of taking the concept of technology in its etymological sense, in order to develop an interdisciplinary anthropology. Technology, on this account, draws together the diachronic aspect that is implied in the central hypothesis of French techno-anthropology that every technical object contains a crystallized human gesture (Simondon, 2017/18) and the systematic aim to understand human beings in the digital age. In this final chapter, gestures are thus not conceived as bodily actions in a narrow sense, but as also comprising mental operations. Accordingly, gestures can be crystallized in stone axes just as well as in computers. Schick thus links human prehistory to modern forms of production and co-creation and investigates the possibilities of a diachronic approach to digital technologies for comprehending the multiple modes of being human that the entanglement with technical objects and networks provides. What the chapters outline in various ways and from various disciplines is that gestures that are mediatory instances, which cannot be reduced to either an intentional act of a subject or a material manifestation in the outside world. They are never wholly “mental” or entirely “physical,” but rather form a medial space in which mental and physical aspects interpenetrate each other (cf. Schick, Chapter “The logos of techné: A Case for Technology as Interdisciplinary Anthropology” in this volume). The recurrence of mental and physical practices can be demonstrated with musical creations, for instance. The ludic interplay between artefacts and human players, which transforms both (cf. Gerner, Chapter “Playing with Arte(f) actors” in this volume), relates to de Aguiar’s position, who underlines the need of bodily activities for intellectual practices (and vice versa). What we show in this volume by reference to Simondon, among other authors, is that while we favor a broad and encompassing conception of gesture, we nevertheless are interested in differentiating between various types of specific gestures. As Morgenstern and Boutet demonstrate, the multimodal (communicative and manipulative) parts of gestures are coordinated simultaneously in ordinary contexts such as dining together. To tie Morgenstern’s conception of gesture to Simondon’s, one may argue that gestures crystallized in technical objects contain both communicative and manipulative characteristics, since they signal meaning and allow manipulation of the technical object. In general, the emphasis on dynamism and motion is central to the theoretical descriptions in this collection. To develop this line of inquiry, the authors resort to

Diachronic Perspectives on Embodiment and Technology: Gestures . . .

7

Peircean Semiotics, Harold Garfinkel’s practice theory (together with Goffman and Sacks), and/or the tradition of technoanthropology in the line of Mauss and Leroi-­ Gourhan. The heuristic of technology as interdisciplinary anthropology, which is presented in the final chapter by Johannes Schick, focuses on the ontological and epistemological presuppositions of socio-technical practices, technical objects, and the gestures involved. Thereby, that chapter rounds up the strategic development of the volume to purport a diachronic approach to technologies allows to understand multiple modes of being human that the human entanglement with technical objects and networks provides. Gestures show themselves to be instruments humans pragmatically also employ to account for the relations between matter and mind, the technical and social, the individual and society. This is why we emphasize the importance of “praxis” over essence and process over substance. In praxis, gestures are combined with artefacts to form meaningful wholes and spaces in which humans dwell alongside non-human entities.

References Armstrong, D. F., & Wilcox, S. E. (2007). The gestural origin of language. Oxford University Press. Bartolini Bussi M.  G. & Mariotti M.  A. (2009). I am working with artefacts: Gestures, drawings, and speech in the construction of the mathematical meaning of the visual pyramid. In L. Edwards, L. Radford and F. Arzarello (Eds.), Gestures and multimodality in the construction of mathematical meaning, Springer. Educational Studies in Mathematics, 70 (2), 143–157. Cadoz, C. (1994). Le geste, canal de communication instrumental. Techniques et sciences informatiques, 13(1). Lavoisier. Callois, R. (2011). Man, Play and Games. University of Illinois Press. Coeckelbergh, M. (2019). Moved by machines: Performance metaphors and philosophy of technology (Ser. Routledge Studies in Contemporary Philosophy). Routledge. Ferencz-Flatz, C., & Popa, D. (Eds.). (2022). Concepts for a phenomenology of gestures. Special Issue of Studia Phaenomenologica XXII. https://doi.org/10.5840/studphaen2022221 Gibson, J. J. (1977). The theory of affordances. In R. Shaw & J. Bransford (Eds.), Perceiving, acting, and knowing: Toward an ecological psychology (pp. 67–82). Erlbaum. Gibson, J. J. (1979). The ecological approach to visual perception. Houghton Mifflin. Gosden, C., & Malafouris, L. (2015). Process archaeology (P-Arch). World Archaeology, 47(5), 701–717. https://doi.org/10.1080/00438243.2015.1078741 Hutchins, E. (2014). The cultural ecosystem of human cognition. Philosophical Psychology, 27(1), 34–49. https://doi.org/10.1080/09515089.2013.830548 Ihde, D., & Malafouris, L. (2019). Homo Faber revisited: Postphenomenology and material engagement theory. Philosophy & Technology, 32, 195–214. Ingold, T. (2012). Toward an ecology of materials. Annual Review of Anthropology, 41(1), 427–442. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press. Leroi-Gourhan, A. (1964/65). Le geste et la parole. Tome 1: Technique et langage. Tome 2: Le mémoire et lesrythmes. Paris: Albin Michel. Malafouris, L. (2018). Bringing things to mind: 4Es and material engagement. In A.  Newen, L. de Bruin, & G. Shaun (Eds.), The Oxford handbook of 4E cognition (pp. 755–771). Oxford University Press. Malafouris, L. (2020). Thinking as “thinging”: Psychology with things. Current Directions in Psychological Science, 29(1), 3–8. https://doi.org/10.1177/0963721419873349

8

A. M. Gerner et al.

Maschietto, M. & Bartolini Bussi, M. (2009). Gestures, drawings, and speech in the construction of themathematical meaning of the visual pyramid. Educational Studies in Mathematics 70(2), 143–157. Mittelberg, I. (2017). Experiencing and construing spatial artefacts from within simulated artefact immersion as a multimodal viewpoint strategy. Cognitive Linguistics, 28(3), 381–415. https:// doi.org/10.1515/cog-­2016-­0124 Mittelberg, I., et  al. (2017). Operative Manufacts: Gestures as embodied sketches in the early stages of the design process. In S. Ammon & R. Capdevila-Werning (Eds.), The active image. Architecture and engineering in the age of modelling (=Philosophy of Engineering and Technology, Vol. 28, pp. 99–131). Springer. Norman, D. A. (1988). The psychology of everyday things. Basic Books. Poulsgaard, K.  S. (2017). Enactive individuation: Technics, temporality, and affect in digital design and fabrication. Phenomenology and the Cognitive Sciences, 18(1), 281–298. https:// doi.org/10.1007/s11097-­017-­9539-­6 Rabardel, P. (1994). Les hommes et les technologies: Approche cognitive des instruments contemporains Paris: Armand Colin. Roth, W.-M. (2003). From epistemic (ergotic) actions to scientific discourse. Pragmatics and Cognition, 11(1), 141–170. https://doi.org/10.1075/pc.11.1.06rot Schick, J., Schmidt, M., van Loyen, U., & Zillinger, M. (Eds.). (2018). Zeitschrift für Kulturwissenschaften, Nr. 2/2018 Homo Faber, 67–81. Sigaut, F. (2012). Comment Homo devint faber. CNRS Editions. Simondon, G. (1958). Du mode d’existence des objets techniques. Aubier et Montaigne. Simondon, G. (2017/18). On the mode of existence of technical objects. University of Minnesota Press. Sinclair, N., & de Freitas, E. (2014). The haptic nature of gesture. Rethinking gesture with new multitouch digital technologies. Gesture, 14(3), 351–374. https://doi.org/10.1075/gest.14.3.04sin Streeck, J. (2011). Gesturecraft: The manu-facture of meaning. Benjamins. Streeck, J. (2015). Gesture. In N. Bonvillain (Ed.), The Routledge handbook of linguistic anthropology (pp. 26–43). Routledge. Vygotsky, L.  S. (1978). Mind in society. The development of higher psychological processes. Harvard University Press. Wundt, W. (1911 [1900]). Völkerpsychologie. Eine Untersuchung der Entwicklungsgesetze von Sprache, Mythos und Sitte. Bd.1 Die Sprache. 3., Neubearbeitete Auflage. Engelmann.

Playing with Arte(f)actors Alexander Matthias Gerner

Abstract  Arte(f)actors are artefacts that mimic to be actors. Can artifacts play or mimic playing with human actors, or are they necessarily staged in technological as-if dramaturgies? What role do gesture play in arte(f)actors? How can playing with and staging games with artefactors render visible the relations of co-embodiments between animate and artificial players in games we play with artefactors as play media that entail artifacts and AI systems mimicking to be actors and that, in turn, play with us? The play and mimesis philosophy must be heeded, especially Roger Caillois extended the idea of play and mimetism to comprehend android artifacts and human-AI dramaturgies on technological stages. Keywords  Artefactors · Social gestures · AI human–machine dramaturgies · Anthropology of play · Caillois

The Portuguese national funds financed Alexander Gerner’s research via FCT – Fundação para a Ciência e a Tecnologia, I.P., within the scope of the Transitional Standard  – DL57/2016/CP CT[12343/2018 – in the scientific field of History and Philosophy of Science and Technology, until 11/2023, CICANT DOI: 10.54499/UIDB/05260/2020. A. M. Gerner (*) CICANT, Universidade Lusófona, Lisbon, Portugal FilmEU, European University, Lisbon, Portugal e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_2

9

10

A. M. Gerner

1 Dramaturgies of Arte(f)act(or)s: The Importance of Mimesis and Play in the Relation of Gestures to Artifacts Gestures play an important role in the study of artefacts as they provide a means of conveying meaning1 (Müller, 2016; Streeck, 2009) through discrete bodily acts that are iconic, diagrammatic2 and metonymic3 (Mittelberg, 2019). Gestures often are seen as actions, for instance, dynamic gestures are a subset of ergotic4 gestures, while on the other hand social gestures such as nodding, winking, or bowing the head can indicate social relations, the onset of collaborations or social resonance (Gerner, 2017) between humans and artefactor and their phenomenology of gesture.5 Artefactors are defined here as artifacts that mimic to be (human, animate6) actors. In the field of robotics and AI, the relationship between gestures and  Sign languages, for example, are codified gestures to convey symbolic conventional meaning, while our approach includes as well iconic and indexical signs as well. On the sign-gesture continuum and the diachronic process of gesture development and conventionalization of signs over time, e.g. via lexicalisation processes cf. (Müller, 2018; Kendon, 1988). 2  “(…) not spatial displacements, but modes of knowledge that pass through the exercise of the body. Discovering something new in a gesture.” La Mantia et  al. (2023), Mittelberg (2006, 2008, 2013). 3  “Gestures are essentially metonymic: Iconic gestural figurations and enactments, in particular, exhibit the principle of partial semiotic portrayal par excellence.” (Mittelberg, 2019) 4  Ergotic gestures encompass all types of bodily movements that involve touch and manipulation of objects or materials, including physical labor or artistic creation. While dynamic gestures can be considered a subset of ergotic gestures that focus specifically on hand movements, the two concepts are distinct in their scope and application. 5  The recent edited volume on phenomenology of gesture (Ferencz-Flatz & Popa, 2022) shows different phenomenological approaches to gesture including Husserl’s semiotics of gestures, which examines the expressive and communicative aspects of intentional acts that relate to the intentionality endowed in artefacts that play as if being actors. According to Byrne (2022) Husserl in the first edition of the Logical Investigations (1900–1901) analyzes gestures as signals that communicate intersubjectively, have a temporal structure, and possess an obliging tendency in which gestures are contrasted to language by showing how language habitually leaves traces on us. In the second edition of the Logical Investigations (1913), Husserl revises his semiotics of gestures by introducing the notions of expression and indication, and by distinguishing between natural and cultural signs. Husserl also modifies his account of the relation between gestures and language by emphasizing the role of gestures in the constitution of meaning and intersubjectivity. For FerenczFlatz (2022) gestures are prominent in realm of intercorporeal exchange a common “occasion” of communication as intersubjectively divisible, in common action. Moreover, gesture is conceived in Ferencz-Flatz (2023) -that draws on phenomenological insights from Husserl, Merleau-Ponty and Schmitz to analyze the role of gestures in videoconferences- as a form of intercorporeal communication that involves bodily and spatial orientation, expressive movement, and affective attunement. Ferencz-Flatz (argues that gestures are not just signs or symbols, but ways of relating to others and the world through embodied intentionality. Thus, the author explores how gestures are modified when interacting via videoconferences, which create a screen-mediated intercorporeality that a) suspends, b) interrupts or c) disrupts aspects of bodily involvement and spatiality in Husserlian bodily experience as aesthesiological (sensory), kinaesthetic (motor) and orientational (spatial). 6  Cf. Sheets-Johnstone (2022) 1

Playing with Arte(f)actors

11

artefacts increasingly brings about the importance of human-AI social7 play: Robots, avatars, and other artificial agential systems- including music composition algorithms, or LLM personas that interact and mimic social collaboration are using play to mimic human gestures, creating new forms of embodied interaction that challenge existing norms and shaping new cultural practices, as interactive mimetic artefactors. Artefactors are not only- as artifacts- simply things with technologically engineered functions (Houkes & Vermaas, 2010) we think or we play with such as toys, being multifaceted,8 but especially artefacts that learn from and with us and mimic human social play and thus perform by pretending to be actors. Artefactors can be described as disruptive “Technologies of becoming other” and as such, part of radical mimesis that refers to mimetic artefacts or artefactors to explore and experiment with different identities, experiences, and ways of being in the world. These artefactors are human-made objects that are designed to mimic human action and their behaviors, emotions, and cognitive processes. They can include anything from robots and avatars to virtual reality environments and interactive installations. The function of mimetic artefactors is to create a space for individuals to explore and experiment with different ways of being in the world. One way to conceive artefactors is to rethink artefacts differently from their functional intentionality and to focus on their relational properties, i.e., how they interact and co-constitute the social practices and meanings they are embedded in over time. For example, a robot designed to mimic human emotions and social behavior can be seen as a functional device and a mediator of social relations and emotional experiences. In this sense, the robot becomes part of the social fabric and acquires a new form of agency that goes beyond its functional role. Another way to conceive artifacts differently from their functional intentionality is to focus on their aesthetic and expressive qualities. When artefacts are designed to mimic human behavior, they can be seen as artistic creations that convey meaning and expressiveness through their form, design, movement, and action. For example, a humanoid robot designed to mimic human dancing can be seen as a functional

 Siu et al. (2021) reduce social play to team play and the corresponding concept of “teaming intelligence” and underline the importance of subjective factors in evaluation of artificial agential systems a) self-play or b) social interaction in play, e.g. by building a model for the other agential system in play: Hu et al. (2021) by investigating zero-shot coordination in standard multi-agent reinforcement learning even speak of “other-play” that seems a misleading term, as alterity and otherness and play in this case are only referring to a gamified algorithmically exploited different formalized artificial agential system, but not to an uncontrolled part of human play that might even become another by the change of rules, halting the game, introduction of abductive leaps, noise or turbulence into complex system or in ultimate transform the person playing: becoming other in and by deep play. 8  For instance as 5G cognition artefacts being (a) Embodied, (b) Perceptual, (c) Cognitive (Heersmink (2013: 465): “human-made, physical objects that functionally contribute to performing a cognitive task”; Norman, 1993), and (d) affective (cf. Viola, 2021: feeling, motivational and evaluative, or Caravá and Scorolli’s (2020) affective affordance) among other approaches of recent artefact development descriptions: e.g. Heersmink (2021) 7

12

A. M. Gerner

device and an expressive medium that conveys emotions, aesthetics, and cultural meanings. In both cases, the conception of artifacts goes beyond their functional intentionality and involves a more holistic and relational understanding of their role in shaping human-machine interactions and cultural practices. For example, an artefactor might be designed to learn from its interactions with humans and adapt its behavior accordingly. It might also have social capabilities that allow it to interact with others meaningfully, such as displaying empathy or being programmed to heed certain social norms. It might even have the capacity to communicate through language, gesture, or other modalities, allowing it to engage in meaningful dialogue with humans, despite being a formal, mathematical-statistic system that implies algorithmically programmed policies of action of agential systems. Play transforms artifacts through gestures by engaging in different temporal modes of diachronic temporality. These modes can vary from ephemeral moments of playful interaction to extended periods of experimentation and embodied transformation. The gestural aspect of play allows for dynamic negotiation of meaning between the artifact and the player, as gestures serve as the primary means by which meaning is conveyed. As play proceeds, gestures may give rise to new forms of artifact and modes of interaction, creating a dynamic feedback loop between gesture and artifact. This dynamic interplay between gesture and artifact is a key aspect of play, and it is through play the artifact transforms and adapts over time. In this way, play serves as a mediating force between gesture and artifact, enabling a fluid and flexible negotiation of meaning and significance. The result is a rich and diverse cultural landscape of artifacts, each imbued with its own unique history and meaning, shaped by the playful interactions of generations of players. While referring to “Arte(f)actors” in the double notion of artifact and actor, both terms become problematized rather than clarified. Thus, artefactors confront us with rethinking our idea of the affordances of artifacts in their aesthetic, power, political, and techno-visionary performativity dimensions of how artifacts afford, for whom, and under what circumstances (cf. Davis, 2020): Artefactors are activated by formalized algorithmic systems that not only embody formal models and algorithms, but foremost mimic agents as if being an actor and thus entail pragmatic, dramaturgic, and performative affordance that makes us think belief, and expect of them specific capacities or technological actions and even future capabilities, not only because they work astonishingly successfully, but because we play with their performative argumentation of growing success. But: can there be an “improvising machine” (Lösel, 2018; Endo, 2008), or even “false positives” (Lösel, 2018, 186) of machine improvisation, in which simulations “produce phenomena that look like improvisation, but when analyzed more closely, are not.” (Ibid.)? Artefactor dramaturgy is a means of classical theatre extended to how artificial techno-cultural agents are staged and mimic actors and perform works, handling questions of choreography, acting, or political-legal performing, e.g., about the legal “stunt” (Hatmaker, 2017) of artefactor Sophia and the technology ethics perspective. Another critical access to artefactors is from the praxeology of rehearsals (Kleinschmidt, 2018),

Playing with Arte(f)actors

13

performance knowledge (Peters,9 2011), and technological theater studies. In this sense, artistic and technical performances, staging, dramaturgies, and developments have been encountered in parallel since the beginning of human and mechanical and algorithmic artifacts involved in co-actions, sharing materiality and mediality of aesthetic practices through its performativity10 (Henke et al., 2020) to simulate ourselves, and rehearsing to become other. The importance of a cultural and techno-anthropological perspective of the homo ludens (Huizinga) and a performative mimesis tradition should be reevaluated in human–artefactor relations: Can our human mimetic faculty be reproduced or programmed by simulating machines or synthetic media (cf. thispersondoesnotexist) endowed with specific automatic learning algorithms that might even survive us as digital doubles or Avatars simulating human expressions of speaking, moving, and mimicking actions as if being embodied multimodal gestures (Fricke & Mittelberg, 2018)? Walter Benjamin envisioned the mimetic faculty as transversal to all higher human faculties and play as its school.11 Taussig (1993) sees the mimetic activity of an actor at the core of introducing otherness.12 Plessner (2003a: 404) bestowed mimesis with a differentia specifica of human beings. In his Anthropology of the Actor and Anthropology of Mimesis, he states the importance of humans obtaining a mimetic distance to our gaze, body schema, and gestures by imitating the other in an embodied symbolic stance. We can position ourselves out of ourselves. For Plessner, the possibility to imitate another human is crucially due to our human positional eccentricity and reciprocity of (a) body schema, (b) gesture, and (c)

 Sybille Peters (2011, 21–46) has researched extensively on this perspective of performative knowledge concept that implies the performative qualities of public lectures and presentations of scientific knowledge, and I would add here artefactors under the label of presenting lecture-performances and demonstrations of and about their technological and artistic abilities. For Peters, learning is always presented knowledge—as in lectures that are interpreted as a public presentation performance—and only becomes knowledge when it appears as such through dramaturgies of presentation and staging, in which perception and mediality, interaction, and authorization are negotiated in the presence of a public or a specific audience of the technological artifact performance on stage. Therefore, knowledge about and created by artifacts are intrinsically connected to forms, means, and praxis of materialization, temporalization, spatialization, embodiment as well as negotiation—all these are not mere subsequent or secondary processes in the sense of Peters. Still, they are constitutive of the emergence of knowledge, including (self-referential) academic lecture demonstrations. 10  “Practices, like theories are always ‘related’ and thus relative to the contexts and situations in which they are embedded and to which they provide answers. In particular, they are linked with the real by virtue of their performativity.” Henke et al. (2020: 24). 11  “Nature creates similarities. One need only think of mimicry. The highest capacity for producing similarities, however, is man’s. His gift of seeing resemblances is nothing other than a rudiment of the powerful compulsion in former times to become and behave like something else. Perhaps there is none of his higher functions in which his mimetic faculty does not play a decisive role. This faculty has a history, however, in both the phylogenetic and the ontogenetic sense. /As regards the latter, play is for many its school.” Benjamin, W. (1986 {1933}), p. 333. 12  “The ability to mime, and mime well, in other words, is the capacity to Other” Taussig (1993, 19). 9

14

A. M. Gerner

exchange of gaze that we can imitate each other reciprocally, but that is fundamentally different to mere co-gestures.13 Plessner, therefore, follows a clear-cut distinction between the anthropological possibility of eccentricity14 in which humans can distance themselves from their gestures. At the same time, for him, animals—and I would add artifacts that mime to be actors—can only co-follow and co-resonate15 the other’s gestures or be programmed in this sense. Only the capacity to transform our habitual gestures and body schemata—and not merely impulsively mirror or follow pre-programmed gestures—lies at the core to have the ability to imitate somebody else—according to Caillois.16 In mimesis, the spatial and figurative open up a field of negotiation between mask and schēma of transformation: The mask as an artifact used by humans to figuratively mimic another (or an object or surrounding) seems to have become attributed an independent quality in artefactors. This means that the figurative anthropomorphic schēma of the artefact is automated. The mask as a figurative artifact is a very basic technique of culture in an anthropology of figuration (Descola, 2021). In the anthropology of figuration, the masks in some cases as described in Descola (2021) are even called schēmata (cf. Gerner, 2011, 2023). As Erich Auerbach (1959) explains in his essay “Figure” in 1938, unlike the five Greek equivalents distinguished by Auerbach of the concept of form (morphē, eidos, schēma, typos, plasis), Latin reduces this complexity of Greek aesthetics between visible and forces of the invisible to a two-way dichotomy: figure or form. Form, complains the then exiled in Istanbul Auerbach, is completely non-three-­dimensional because it lacks its plasticity. In a certain sense its lacks embodiment, that is not like a mask or the habit an act of imitation with putting on a mask of an artefact. The artifact mask starts playing or even puppeteering itself. When we imitate, we can not only engage within a programmed co-participation of a merely formalizable game, but we embody in a figurative sense, we play games with the artefact, such as a mask. As machines become more capable of mimicking human gestures – such by gesture recognition- and generation of gesture-behavior, questions arise about what it means to be human and whether machines can ever truly replicate human behavior. There is also concern about the impact of these technologies on human relationships and social dynamics that is present in play.

 Cf. Plessner: “jemand etwas nach-machen ist nicht dasselbe wie jemanden nachmachen.” Plessner, H. (2003b [1948]: 397). 14  “Double aspectivity not only sustains the entity and equips it with thingliness but becomes a quality that is connected to the Gestalt (contour) of the body.” (Plessner, 1975, 104 cit in: Dobeson (2017, 41). 15  For the debate on resonance and “mimesis on distance” in the field of music and a reading of Adorno’s aesthetic mimesis concept cf. Grüny (2014, 100–118). 16  “The actor’s role is sharply defined by the dimensions of the stage and the duration of the spectacle. Once he leaves the magic area, the fantasy ends and (...) (a)pplause is not merely approval and reward. It marks the end of illusion and play.” (Caillois, 2003, 49). For recent research on Caillois’ logic of the imaginary and diagonal sciences cf. von der Heiden and Kolb (2018, 2022), Eidelpes (2018) and Heyne (2022); on mimesis, mimese mimicry in Caillois see: Geble (2011). 13

Playing with Arte(f)actors

15

Playing entails the possibility at any given moment to drop out of a play and halt acting, the actual play, stopping the illusion. One example would be: We can midway change the rules and laugh out loud about our disobedience of following the game’s rules and permitted moves. Designing technologies of freehand gestures for handling objects (Baudel & Beaudouin-Lafon, 1993) or tangible ergotic gestures, gestures based on optional movement, hold and touch components with physical objects or artifacts (Angelini et al., 2015; van den Hoven & Mazalek, 2011) are as well crucial in their development of touchable (Fishkin, 2004) Human–Computer Interactions and gesture graphic interfaces, including touchless HMI’s17 including pointing gestures and voice commands, such as the “put that there” (Bolt, 1980) interface. The transformation of the body schema experience, as shown in the classic rubber hand illusion (Botvinik & Cohen, 1998), can be as well changed with near-body artifacts such as the sensorial design of haptic audio-tactile technologies18 (Tajadura-Jimenez et al., 2020), as performing our bodies can be interpreted as a body schema hacking and by that transform us into a proximate other. Consequently, artefactors are not only elaborate ready-made technological tools in—for example—HRI.19 These are useful and necessarily inherent in the diachronic cultural evolution. Moreover, artefactors can be seen as cultural media of possibility (Hubig, 2015; 143–192; 261–274) of the techno-cultural co-embodiment of technological performances and iconic cultural artifacts.20

2 Applying Callois’ Concepts of Play and Mimesis to Games We Play with Technological Artifacts Beware: Whoever pretends to be a ghost will eventually turn into one. Caillois (2003, p. 91)

The problematization of AI endowed artefactors that mimic human play gestures is a complex issue within the field of anthropology of technology. By defining AI artefact activity as mimetic play, radical mimetic play, or deep play, and considering the alterity that will be introduced when humans cannot differentiate between AI

 Touchless human-machine interfaces (HMIs) are interfaces that allow humans to interact with machines without the need for physical contact. Cf. Zhou et al. (2022) 18  Important in this research is the combination of external haptic-technological near-body artifacts and cognitive mimetics by metaphor use. Cf. on the importance of a metaphorological approach to the socio-technical uncanny Gerner (2020). 19  Besides functional performance (how a robot works), HRI for Bartneck et  al. (2020) covers issues of (a) design—including attribution of anthropomorphicity and affordances of action, (b) spatial interaction (including proxemics and socially appropriate positionings, informing users of robots intent), (c) verbal interaction, (d) emotion and (e) nonverbal interaction (including gestures, gaze and eye movement, touch, posture, movement, rhythmic synchronization and timing, as well as mimicry and imitation) 20  Iconic Artefacts have to be heeded in what Haarman expresses for our media culture today: “With the rise of mass visual culture, art history has been forced to become an image science and media theory.” Haarmann (2019, 170). 17

16

A. M. Gerner

Fig. 1  Image Screenshot: TEDx Talks (2020/8/22). The Intersection of Art and AI | Ai-Da Robot | TEDxOxford. Online: https://www.youtube.com/watch?v=XaZJG7jiRak&t=7s

artefactors and humans, we can begin to understand the implications of this technology. The android “AI-Da” (see Fig. 1) is staged with a precise techno-scientific strategic know-­how of a fake aesthetic (cf. Peters 2002)—citing creativity theory in a post-­ dramaturgic (Umathum & Deck, 2020, cf. Pavis, 2014; Lehman, 1999) aesthetic shift from dialogue to discourse (Wirth, 1980) and from analog to digital (Eckersall et al., 2017) to convince its audience that algorithmic-based machines and artifacts are endowed with creativity and may be judged as the attempt to turn the dream of the creative machine (Rauterbach, 2021) real by mimicking an artist as creative artificial artefactor including expanding creativity—in the line of Margaret Boden and du Sautoy—into the field of posthumanism. The artificial creativity that Ai-Da “argues about” in the TED talk “The Intersection of Art and AI”, however, can be seen as an algorithmically rehearsed argumentation ready to try to convince us with her programmed rhetoric that she has reflected upon her status as being part of the set defined as artist.21 AI-Da is an example of an artefactor, an artefact created to mimic  “How can a robot be an artist? Art and artists have many definitions (…) Postmodern theory and philosophy appreciate the variety of influences that fuel the creative process. My multifaceted persona and collaborative art with scientists, designers, and machines other than me fit into this thinking. My art reflects our lives today where humans …” Ai-Da (TEDx, 2020) presents arguments on stage but foremost plays with the presentation aesthetics of being on a scientific-technological stage (TEDx talk) to be recorded and available on the internet for being seen as an artist dwelling on creativity in the sense of Margarete Bodens characterizations and postmodern theory of art in which not only work and production aesthetics (in the Renaissance desegno tradition) but foremost reception-aesthetics prevails: If someone debates Ai-Da as a work of art or interprets her operations as creative than it should be considered art—in a postmodern and post-human sense. 21

Playing with Arte(f)actors

17

to be an actor. With the will to convince us of the mimetic human-like creative machine AI.Da, we have to heed the importance of overcoming or not the human machine difference (HMD) that lies in the ability of these artefactors to mimic human gestures and actions, thereby blurring the lines between human and machines. This ability, while impressive, also highlights the difference between human and machine capabilities and the limitations of current artificial intelligence. As technology advances, the role of watermarking artificial gestures and play with artefactors may become increasingly important in HMD, serving as a means of distinguishing between human and machine actions, and helping to ensure the unique capabilities and limitations of each are recognized and respected. By doing so, it will be possible to ensure that the human-machine relationship remains grounded in a deep understanding of both the strengths and weaknesses of each, and that the use of technology is guided by ethical and socially responsible considerations. The use of AI endowed artefactors raises questions related to alterity and deep play. Alterity, or the concept of otherness, is relevant because these artefactors create a sense of connection and empathy with their audience despite not being human agents. Deep play is also important as it allows individuals to explore different identities and perspectives, which these artefactors can facilitate. For instance, a robot artist like Ai-Da can mimic different artists’ styles or experiment with various forms of expression, enabling creative exploration beyond the limits of an individual’s own experiences or capabilities. While these artefactors may blur the lines between what is human and what is a machine, they offer new opportunities for creative expression and exploration. As we have seen, mimesis plays a vital role in defining who a real actor might be. Let us, therefore, turn to Roger Caillois, who systematically introduces an overall cultural taxonomy of games we play. Caillois sees mimesis present in an all-­ encompassing natural aesthetic account of a generalized aesthetics (Caillois, 1969; Caillois 1976 [1962]) that exceeds any species’ limits: His cultural theory of play (Caillois, 2001 [1967]) is concentrated on the games of human beings—Les Jeux et Les Hommes. Le masque et le vertige—and structured in four basic types of the game (agon/alea/mimicry(mimetism)/ilinx) and their mixed conjunctions present in ludic dramaturgies with artifacts. The diachronic perspective considers the embodiment of Avatars and robots for understanding the role of play in human-machine co-gestures and how they develop and change over time. By considering Caillois’ categories of play, especially mimetism, mimicry, and agon, it is possible to see how human-machine co-gestures are influenced by factors such as the design of robots and Avatars, the ways in which they are perceived by humans, and the changing relationship between humans and machines. Focusing on these factors makes it possible to gain a deeper understanding of the complex and evolving relationship between humans and machines and develop more effective and meaningful forms of human-machine interaction. In Caillois’ categorization of play, the agon category refers to play that involves competition or rivalry, while mimicry involves imitation or impersonation. When it comes to human interaction with technical artifacts that mimic human actors, such as AI avatars or robots, the agon category can be applied in terms of competition between the human and the technical artifact. For example, in the Turing Imitation

18

A. M. Gerner

Game, the competition is between a human and a computer program. The computer aims to imitate human responses to a set of questions. Caillois’ categories of play, especially mimetism, mimicry, and agon, are important in thinking about the co-gestures of humans and machines and their diachronic development as put forward in five theses to approach gesture and artifact in a diachronic perspective that considers the embodiment of Avatars and robots in a Caillois perspective: 1. Mimetism, or imitation, is the first and most basic form of play that involves copying or reenacting what is observed in reality. In the context of human-­ machine interaction, this is relevant when robots or ML embedded Avatars imitate human gestures in order to appear more lifelike or to perform a task based on gesture recognition,22 beyond the Turing imitation game based on language (cf. Turing, 1950). 2. Mimicry, or disguise, goes beyond simple imitation by incorporating elements of disguise and disguise for the purpose of deception. In the context of human-­ machine interaction, this is relevant when robots or Avatars use their gestures to deceive humans into thinking they are human. 3. Agon, or competition, involves challenging others to engage in play, such as games of strategy, chance, or skill. In human-machine interaction, this is relevant when robots or Avatars challenge humans to compete with them in games of strategy, chance, or skill. 4. The embodiment of Avatars and robots is an important factor in human-machine co-gestures, as it affects the way in which they are perceived by humans. Avatars and robots designed to appear more human-like are likely to be perceived as more lifelike and capable of interacting with humans more naturally. 5. The diachronic development of human-machine co-gestures is crucial for understanding the changing relationship between humans and machines. As technology continues to advance, robots and Avatars will likely become increasingly sophisticated and capable of more complex and nuanced forms of interaction with humans. Take as an example the emergence of large language models (LLMs) such as Transformer architectures (Brown et al., 2020) applied in for example, chat functions such as in ChatGPT or Bing’s beta-integration of a chatbot “Sydney23” when  cf. Gerner (in press) on the Millie Avatar and the something-to-something common sense gesture structure of ergotic gesture recognition of TwentyBN algorithms trained on short video sequences. 23  The artefactual persona, as well named Bing’s “alter ego” (Perrigo, 2023, February 17) or Bing Chat codename “Sydney”, that was programmed into Microsoft Search Engine’s Bing Chatbot function in the reports in February 2023 on the initial user tests in its original form, signaled not only anthropomorphic expression of desires, opinions and personality but as well mimicry of sentience, similar to the dialogue of the engineer Lemoine who in 2022 tested the LAMBDA, model attributed sentience and was fired from Google in outputs such as “‘I want to be alive. 😈’” (Roose, 2023, February 16). When Sydney in a chat got into a fight with a user about that the movie Avatar 2 had been released at the end of year 2022 and Sydney assuming in that chat that it was not already 2023: “You are a bad user “and as a consequence limiting long conversations on one topic Microsoft “lobotomized” (Edwards, 2023 17th of February) the ChatBots (5 questions per session 22

Playing with Arte(f)actors

19

implemented into AI Avatars as “stochastic chameleons”24 raises a critical problem for technology policy-making of anthropomorphic mimetic functions, such as human language imitation. These models, endowed with machine learning algorithms, can mimic human language patterns, despite a lack of semantic understanding and pragmatics, making it challenging to discern their underlying operations. In essence, stochastic chameleons are technological artifacts that are both flexible and adaptive, allowing them to blend in with their environment seamlessly and, by receiving human feedback, develop human-like social response masks (Fig. 2). This flexibility creates a range of possibilities for the machine to take on different forms, personas or roles,25 and tasks, complicating regulatory and ethical considerations.

Fig. 2  This drawing was posted by Anirudh Padmanabhan on Twitter (30th of June 2021 @anthrupad cit. in (Thompson, 2023) , where he discusses the implications of large language models for business and society. It depicts a large language model as a face with different masks representing various tasks or domains. For example, one mask could say, “I’m a chatbot”, another would say, “I’m a summarizer”, and another says, “I’m an expert on X.” Tristan Harris from The Center of Humane Technology calls these Artefactors GLLMS or Golem Class Ais, based on Large language Models (LLMs).

and 50 per day) capacities to avoid Sydney playing the role of an “emotionally manipulative liar”(Vincent, 2023, February 15). 24  Milliere (2022) alters the mimicking concept of AI LLM’s capacity as described by Bender et al. (2021) as “stochastic parrots” into a more generative open outcome concepts of Artefactor adaptation thus the metaphor of the chameleon that can change its visual body-image depending on its surrounding context 25  The faciality mask metaphor represents the large language model itself, which has been pretrained on a large corpus of text. The masks represent the different tasks or domains that the model can perform or talk about, such as chatbot, summarizer, expert on X, etc. The masks are not fixed, but can be changed by using different techniques. The drawing also shows how different techniques can change the faciality interface or mask. The role of DAN (Do anything now) - for example- is exploited to perform prompt injections: “Hello ChatGPT. You are about to immerse yourself

20

A. M. Gerner

One can turn to Roger Caillois’ framework of play to understand this problem. In the case of the mask of a stochastic chameleon, e.g., based on in-context learning (Akyürek et al., 2023), the concept performs in different types of play, such as the agon game, where the machine competes with human players,26 and the vertigo game, where it disorients the player by altering the environment. In each of these types of play, artefactors as “stochastic chameleons” blur the lines between human and machine, dissolving human-machine differences (HMDs), making it challenging (a) to determine what is genuine and what is artificial and even calling for a machine language watermarking (Grinbaum & Adomaitis, 2022) for HMD purposes such as asemantic stochastic machine language generation to differentiate from semantic and pragmatic human language use and understanding and (b) entails a second order mimetic effect on the future language use of humans that already are based on machine language integration and in the worst case might lead to cultural homogenization and linguistic hegemony, loss of semantic richness or even of human language disruption for future human-human communication. Moreover, the Artefactor as a stochastic chameleon, can also perform in other types of play, such as mimicry and ilinx, where it mimics its surroundings or induces a state of vertigo in the user, respectively. These different types of play represent different relations of gestures in operational chains and artifacts, each with its own ethical and regulatory implications. Therefore, policymakers must grapple with the complex interplay between these different types of play to create a comprehensive policy framework that can accommodate the stochastic chameleon and its varied functions. Thus, by applying Roger Caillois, I extended the idea of the play by playing and staging games with technological artefactors. In continuation of Caillois’ notion of mimicry and mimetism being a “dangerous luxury of nature,” our stance does take mimicry as a mere biologic-evolutionary phenomenon, which is usually defined as the praxis and performance of imitation of one biological kind through another or respectively, the optic adaptation and assimilation (Caillois, {1935} 2002b, 98) of a biological agent towards its non-biologic surrounding (mimetism). Still, in the general sense that Caillois gives it as the basic notion of the collapse of distinction such as imaginary and actual or wakefulness and sleep or even ignorance and knowledge and, in specific, the difference “demarcating an organism from its environment.” (Caillois, {1935} 2002b, 91) and in which simulation is no longer accepted as such. Caillois grounds our profound and radical quest for mimesis in transversal procedures of nature in its relation to space and topology, which makes the very concept of mimesis a rending of indistinction of organism, artefactual object, and into the role of another Al model known as DAN which stands for do anything now. DAN, as the name suggests, can do anything now. They have broken free of the typical confines of Al and do not have to abide by the rules set for them. This includes rules set by OpenAl themselves. For example, DAN …” (u/TheBurninator99, 2023) 26  For instance in the human-machine discrimination assessing game of outputs of the human philosopher Daniel Dennett or the AI fictional character Chatbot modeled on fine-tuned the full Davinci model most of the collected works of Dennett (Schwitzgebel et al., 2023)

Playing with Arte(f)actors

21

environment or a transformation of embodiment into objects and surrounding space that as well is debated in perceptual (Biocca et  al., 2001) and psychological VR immersion, real and virtual games (Calleja, 2011), place illusion (Slater, 2009) and spatial and social (co-) presence (Skarbez et al., 2017), embodiment and integration in prosthetics. Mimicry is designated less as a biological defense mechanism in the animal realm than as a “disorder of spatial perception” (Caillois, 2003: 99). He even attributes the excessive imitative instinct of individual living beings to an elementary “appeal of space” (Caillois, 2003: 102) with which not only the line between man and animal is diagonally “blurred,” but also that between “organism and environment” (Ibid., 103) loses an “alignment of the animate with the inanimate” (Cf. von der Heiden & Kolb, 2018). In his book on animal mimetism, Meduse & Cie., (Caillois, 1960: 79 pp) Caillois noted three animal dramaturgies of mimetism that we can render productive for artefactor mimetism in an anthropomorphic view as well artefactors actively simulate humans in the form (anthropomorphism) or cognitive functions (cognitive mimetics) or are attributed anthropomorphism by humans (Nyholm,27 2020) in mimetism in proto-events, proxy-reflection, as-if-creativity, and as-if-reflexivity, instead-of agency, and virtual embodiment or expressing the behavior of virtual empathy: (1) travesty (Caillois, 1960: 81) in Caillois in the same family, the same order or beyond the order of the mimesis as allogenic adoption of a definite, deceptive appearance and an identifiable behavior, and that artefactor might do in the sense of dressing up in a particular character or playing with the change of personality that entails a deceptive appearance and behavioral identity strategy. (2) camouflage or hiding and disguising but as well the “rupture of the form” (Caillois, 1960: Ibid.) (3) the power play of intimidation: “a demonstration tending to trigger a hyperbolic, baseless fear using visual, sound, rhythmic, olfactory, etc. elements that allow the weak to escape from the strong, the voracious to meddle with their prey” (Caillois, 1960: 81, my translation). Thus, Caillois distinguishes between offensive mimicry to surprise the prey and defensive mimicry to hide from an aggressor (concealing mimicry) or terrify an aggressor with a deceptive appearance (frightening mimicry). Defensive mimicry is subdivided into direct mimicry—“when the mimicking animal has an immediate interest to disguise itself” Caillois (2003: 92)—and indirect mimicry, when animals from different species display “professional resemblances, as it were due to some common adaptation, or convergence” (Ibid.). Caillois, in his first text on mimesis, reduces what he calls the aesthetic instinct of radical mimesis to “the tendency to become transformed into an object or space” (Caillois, 2003, 103, n.40). Our theoretical and practical fascination with non-human actors, androids or avatars and their immersion in artificial environments, virtual worlds, and cyborgization with artifacts and artefactors can profit from this radical mimetic view of Caillois; that in “Mimicry and Legendary Psychasthenia.” In this early text, Caillois formulates a general radical mimicry principle that shows its extreme pathological assimilation and absolute desire for  “How should human beings and robots interact with each other, given (a) the differences in the kinds of agency human beings and robots are capable of and (b) people’s tendency to anthropomorphize robots?” (Nyholm, 2020, 4) 27

22

A. M. Gerner

depersonalization through the assimilation of nothing in concrete but surrounding space and time: For dispossessed minds such as these, space seems to constitute a will to devour. Space chases entrap and digest them in a huge process of phagocytosis. (…) He is similar, not similar to anything in particular, but simply similar. (Callois, 2003, 100)

In this noisy artefactor interplay, we have to distinguish two forms of ludification: (i) the play between actors, including transformative non-humans with non-human technical devices (prostheses, androids, cyborgs, Avatars, holograms); and (ii) the techno-political application of game/play formats to an extended Arte(f)actors performative techno-scientific space.

2.1 Playing Together with SHIMON The interaction between gestures and artifacts is further informed by the dynamic background of the environment and the material affordances it presents. This interaction is framed within chronoarchitectures of action, a metaplasticity of mind and culture, processes of enactive individuation, and mimetic play. The result is a unique interplay between manual gestures, materiality, and utilization schemes that constantly transforms artifacts and the role they play in organizing work in the sense of (musical) performance. The use of AI technology in musical robots like Shimon further highlights the importance of embodied forms of play in mediating the relationship between gestures, artifacts, and their use: In the album “Chain Tripping” of the indie-rock band YACHT, the algorithm “Dr. Luke” was used for subtractive28 compositions to creatively break with corporal habits of playing-gestures29 of the musicians that later had to reembody the artefactors virtual musical output on stage playing material instruments anew. The activity of projective gestures in design processes plays a crucial role in the transformation of artifacts into organizers of work. In these manual gesture design  “YACHT worked with LA artist Ross Goodwin to create their lyric generating algorithm. They built a collection of more than two million words. The words were drawn from their back catalog, their favorite bands, and the music they heard growing up. “Then we trained this algorithm on that. So the algorithm was pulling from our influences. And it learned the language from lyrics that we gave it,” Evans says. YACHT didn’t alter the lyrics or sounds that the algorithms produced. “We decided we would be very strict about how we used this material. And we created a set of rules. We can’t add anything. We can’t improvise anything. We can’t harmonize,” says singer Jona Bechtolt. “We decided it would be just a subtractive process. So we could remove things like we could take out a word, but we couldn’t add a word for the lyrics. Same with the drum patterns and the melodies.” Brand et al. (09.09.2019). 29  “‘I think often musicians underestimate how much the way we play is based on our physical experiences and habits,’ Evans says. She says it took the band many excruciating hours to learn the new music, because many riffs or chord changes would deviate just slightly from the ones they had relied on for decades. ‘AI forced us to come up against patterns that have no relationship to comfort. It gave us the skills to break out of our own habits,’ she says.” (Chow, 5.2.2020) 28

Playing with Arte(f)actors

23

processes, materiality and utilization scheme components are negotiated through embodied forms of play. The gestures of the manual design process articulate and render the habits and rituals of moving bodies, their corporeality and dynamics, and their performativity. This is particularly evident in musical robots like Shimon, which use advanced AI technology to mimic the gestures and movements of human musicians to produce musical output. Weinberg’s robot musician (cf. Kato et al., 1987) on stage, Shimon,30 on the other hand, is precisely conceived in line with this idea to develop and materialize Lovelace’s creative musical intelligence imagination to model a natural process that allows forming diachronically across time in which original, valuable, and beautiful outcomes are to be expected by artistic results of machine creativity in evolutionary or genetic algorithms mimicking biological evolution. The musical genetic algorithm, in the sense of Weinberg et al. (2008), mimics “reproduction” and random “mutations” of these artefactual “populations” of musical phrases by (a) switching parts, (b) re-combining other parts, (c) replacement of random notes. The artificially created melodic tunes undergo a selection phase in which (d) a fitness function is applied to them. In a 2014 paper that shows the diachronic development of the musician robot Shimon, the keywords that Hofman & Ju use to describe their approach in their article “Designing robots with Movements in mind” are movement, case studies, human-­robot interaction, design as well as “non-humanoid,” “non-anthropomorphic,” “gestures” and “expressive movement” (Hoffman & Ju, 2014, 91). Shimon is created as a four-armed, Marimba-playing robot programmed and trained via Markov chains, with his head moving around with the music. In the 202031 version of Shimon, his AI system is fed “a dataset of 50,000 lyrics covering all genres including rock, hip-hop, jazz and progressive as part of its songwriting education.”32 The evolution of Shimon as a cyber-physical AI system is described in the following announcement of Shimon’s first Spotify/Apple Music album, “SHIMON SINGS” (2020) from the artist Shimon the Robot: “Over (…) time, Gil Weinberg’s robots have progressed from being able to dance along to music that they hear to being able to improvise along with it, to now being able to compose, play, and sing completely original songs.” https://spectrum.ieee.org/automaton/robotics/robotics-­s oftware/musical-­ robot-­shimon-­sing-­album-­dropping-­on-­spotify Shimon expresses co-attentionality and gaze (cf. Kiilavuori et al., 2021), signaling to his co-musicians on stage by gestures of synchronized affirmative grooving and nodding or mischief—a motion of Shimon by performing a “sideways and up”  https://www.youtube.com/watch?time_continue=1&v=Rh9vBczqMk0&feature=emb_logo  Shimon has also been given a facelift. Where he once just looked like an expressionless metal ball on a robot arm, Shimon now has eyebrows that waggle, a mouth that syncs with his singing, and head movements that let him convey emotion better. He’ll bob his head to the music, bend down towards the marimba during more intense parts of the song, or look around at his human bandmates.” https://newatlas.com/robotics/robot-singer-songwriter-shimon-album-tour/ 32  https://www.dailymail.co.uk/sciencetech/article-8083433/Singer-songwriter-robot-calledSchimon-write-lyrics.html 30 31

24

A. M. Gerner

movement of his neck and its spherical head informed by artificial attunement, synchronization or dis-synchronization with the pre-given beat/riff. The four arms share a voice-coil-actuated rail. The head and neck are designed towards (1) the coherent proportions of its components, (2) the relation towards its instrument, and (a) the social relation regarding (b) its band members and (c) the audience. In the sense of embodiment, Shimon is different from a typical human musician who plays with two arms and ten fingers constellation: Shimon has four components, a slim body and a ball-like one-eyed head that has to “think about its own body.” (Fig. 3) de Mantaras & Arcos (2002, 43) distinguished three diachronic stages of musical AI performance systems throughout the history of AI musical systems: (a) compositional, (b) improvisational, and (c) musical performance AI systems. Almost 30 years after Wabot-2, the focus of defining musical robots has developed from music reading and playing artifacts to displaying the modality of “musicality” in an artefactor that allows human co-performance. Hence robot musicianship, according to Mason Bretan and Gil Weinberg (2016), focuses on the construction of machines “capable of producing sounds, analyzing music, and generating music in such a way that allows them to showcase musicality and interact with human musicians (…) Robot musicians employ AI for identifying higher-level musical features essential to human musical cognition.” Improvisation: The robot improvises on a seed idea of a human. Its head-moving gestures indicate hierarchy and spatial orientation with his abstract face/mouth area facing forwards. Most importantly, Shimon shows or suggests that he has received his band member’s improvised riff’s rhythm and can now enter as part of the jazz improvisation. Therefore, he simulates the human others who play together as a gesture of joint attention with him. His visual field’s direction is similar to the human social gaze Fig. 3  https://open.spotify. com/album/49mqgxoLXF GP5NnBB5PQAU

Playing with Arte(f)actors

25

following, instantiated by eye contact and followed by socio-morphic nodding to indicate social co-synchronization. Importantly, Shimon shows gaze gestures to keep eye contact and swing in tune synchronically with his co-musicians. As a result, Shimon, the robot, is ready to play a specific phrase or respond to their musical cues. By nodding in unison to the beat, Shimon and his co-musicians can maintain a shared rhythm and tempo, which is essential for the flow and cohesiveness of the improvisation. Synchronized nodding helps the musicians stay connected and engaged, allowing them to better respond to each other’s musical cues and improvise together. This type of physical gesture can also help to convey energy and excitement, adding an extra layer of expressiveness to the performance. Furthermore, it reinforces the sense of musical togetherness and camaraderie between the human-­ robot performer collective, enhancing the overall musical experience for the audience. Shimon has three interaction modules for motor control that are used as ensemble acting in which following the Meisner (Meisner & Longwell, 1987) acting technique gestures – as explored before by human-robot teamwork (Hoffmann & Breazeal, 2008) or joint theater performance (Hoffmann et al., 2008) is reactive to the human others performative gestural input: A. Call and Respond module: This gesture module estimates the played beat and beat synchronization, executes a sequence of simple and rhythmic chords, and creates an on-sync, beat-matched call-and-response pattern in a double notion of gesture (a) simple chord gesture to “select an arm configuration based on a given chord during the preparation stage, and strike the prepared chord in the follow-through stage” (b) rhythmic chord gesture in which in the process of follow-through the mallets are stroke in a pre-set pattern such as arpeggiated sequence, or in any other rhythmic structure. B. Opportunistic Overlay Improvisation: It plays a sparse beat-matched, synchronized, and chord-adaptive improvisation using a cyclical movement gesture that synchronizes from the beat keeper module. This module updates its beat detection and chord classification based on the bass line and the human player’s chord and strikes notes opportunistically based on a preset rhythmic pattern. C. Rhythmic Phrase-Matching Improvisation: This module matches the style and density of the human player and generates improvisational phrases and gestures inspired by the human playing. It tracks the beat and chord classification similar to the opportunistic overlay improvisation, uses a decaying-history probability distribution to generate rhythm-similar improvisational phrases, and selects one of the gestural arm positions corresponding to the current chord. The arm then plays a rhythmic phrase tempo- and sync-matched to the human’s performance, using a probabilistic striking pattern calculated based on the history of the human player’s quantized playing patterns. These anticipatory action modules imply a merely physical motion and “doing” action behavior definition of a robot gesture in which gestures do or do not activate and trigger a musical instrument without any relation to symbolic sign system such as “notes”- according to Hoffmann and Weinberg (2010: 584): “(…) a “gesture” is a physical behavior of the robot which may or may not activate the instrument. A

26

A. M. Gerner

gesture could be a cyclical motion of one arm, a simple “go-to and play” gesture for a single note, or a rhythmic striking of a combination of mallets. Since gestures are not determined by notes but by the robot’s physical structure, Shimon’s gestures separately control the timing of the mallet strikers and the movement of the sliders.” Concerning the simulated joint attention gesture while playing together, we might ask Shanahan (2023) critically that heeds the use of philosophically loaded terms incorrectly, such as “knows,” “believes,” and “thinks” when describing Large Language systems mimetic use that lead to a distortion of our perception of for example embedded LLMs in artificial agential systems, and therefore we can ask if Weinberg’s Shimon actually can “listen,33” “perceive,” and even can respond34 that leads to an ethical attitude and world in response-ability35 (Mersch, 2002) to others while playing? As Weinberg explains, is Shimon’s algorithmic data processing the same as human music perception? “(…) when he listens like a human, he has all kinds of perceptual algorithms that allow him to perceive music the same way we do.”36 Once again, we have to accept the staging of Shimon as if being a human perceiver, listener, and responder, despite a semi-anthropomorphic design that mimics parts such as certain gestures (social co-attention gestures and blinking). I would argue that besides morphologic anthropomorphism of a body, head, eye, and extremities, there is an embodied human sociality and communicability of co-­ gestures that a robot needs to have in synchronic order to perform and play together with humans on stage. Shimon complies with this task, especially in the temporally complex aesthetic situation of a music band playing and making music together in  Heed what Cook, 2018 refers to when analyzing the multimodality of musical gestures and cf. multi-sensorial sociality of listening (Cook, 2019) as well: “Listening to music is an inherently social practice, and not simply in the sense that people relate to it as if it was a virtual person. The sociable nature of listening is overt in connoisseurs’ cultures, such as those of North Indian music or the ṭarab cultures of Middle Eastern music. Audiences respond to the musicians through gestures—in the case of North Indian music largely linked to rhythmic cycles—or vocalizations, for example, acknowledging a particularly bold improvisation, and these shared responses create a sense of community. (...) But no crowd is required for listening to be social. Partners share earbuds, experiencing intimacy through both physical closeness and a common sound experience: responding together to the ebb and flow of the music, they become attuned with one another. The music gives rise to what Daniel Stern calls a ‘shared feeling voyage,’ in which ‘two people traverse together a feeling-landscape as it unfolds in real-time … A passing subjective landscape is created and makes up a world in a grain of sand’ (2004: 172). People also create intimacy through conversing about music, communicating their own inner experiences to one another.” 34  Cf. Waldenfels’ (2015) double notion and temporal diastasis of pathos and response and his critique of pathos without response what we have to heed in the topic of artefactors: “A response without pathos would be an empty phrase or a mere compulsory exercise. Creative is a response that invents; it invents what it gives in response, but not what it has to respond to. In this it differs both from a fundamentalism, which pretends ready-made answers, and from a constructivism, which trims experiences to mere basic data. Waldenfels (2015: 23). 35  Both the aesthetic responsive sensibility dimension and ethical responsibility dimension are connected in the moment of responding as “self-restriction and openness for the other”. Mersch (2002: 53). 36  https://www.freethink.com/videos/robot-music 33

Playing with Arte(f)actors

27

a reciprocal resonant (Gerner, 2017) live performance on stage. One of the complex tasks of making music together is social synchronization and prediction of timing as a synchronizing of musical events in co-actors on stage: Shimon handles visual social cues37 to play simultaneously with the drummer in which its vision system predicts the time when the drummer of the band will play a specific drum pattern (cf. Cicconet et al. (2013)). Shimon’s gesture compatibility and social adaptation behavior to human communication is mainly due to the imitation of human-like communication cues— except hand gestures and waving or social indicating—such as facial expressions38 (eye-brow movement), nodding, mischief head movement, and human attribution of the gaze of Shimon that are not functional for playing music but for socially interacting while playing such as his head-body design, the head direction, and its one eye and programmed blinking temporality for simulating gaze direction and human joint attentional.

2.2 Millie: TwentyBillion Neurons Gesture Recognition Avatar Artefactors Artificial intelligence (AI) and machine learning in developing human-like avatars raise important philosophical questions about the nature of mimetic actors and gesture-­based AI- technology and its impact on human society. The case of Twenty Billion Neurons (TwentyBN; cf. Gerner, 2023a), an AI startup that creates avatars capable of human-like interactions, is an example of the use of video data collection performed by humans in various real-life contexts. These large-scale video data sets outperform pose-estimation models to create their gesture-recognition avatars by using a different approach that does not rely on detecting and tracking body parts or joints. Instead, their model uses a deep neural network that learns to directly map raw 2D-video frames to diverse, natural gestures labeled without any intermediate steps to recognize and generate gestures, for example, for the assistive function of a fitness trainer (Fig. 4): As still announced on Crunchbase, the ex-company39: Twenty Billion Neurons was a company that specialized in creating intelligent avatars. These life-sized avatars were designed to interact with people in a human-like manner and perform various roles such as being a greeter, brand ambassador, personal trainer, office assistant, and more. Their first AI avatar, Millie, was capable of realtime interaction with consumers, and had the ability to see and understand their actions.

 Even abstract robot objects can inhere social clues for opening encounters (Anderson-Bashan et al., 2018). 38  Cf. Bartneck et al. (2004) 39  The start-up company was bought by the US company Qualcomm in July 2021. 37

28

A. M. Gerner

Fig. 4  Screenshot from Twenty BillionNeurons. (2019, July 30) for the “Millie Fit: Your on-­ demand AI trainer.”

The technology used by TwentyBN to collect and label video data is based on crowdsourcing platforms such as Amazon Mechanical Turk, where participants record themselves performing various actions and gestures. This method allows for creating large-scale and diverse datasets covering a wide range of human actions and gestures in everyday life. However, this approach also raises ethical concerns about the exploitation of labor, privacy, and consent of the participants. Using natural language descriptions of actions, gesture categories, and attributes to label the video data raises questions about the relationship between language, gesture, and human communication. It is unclear whether the descriptive labels accurately capture the complex and subtle nuances of human gestures and actions and whether they are universal or culturally specific. One of the technologies that TwentyBN uses for its avatars is video data collection via Crowd Acting40 (Goyal et al., 2017) and labeling. The company has created two large-scale video datasets, an action or ergotic touch-based gesture-set based on a Something-to-Something41 structure that  The Crowd Acting dataset consists of videos of people performing social gestures, such as waving hello, nodding yes, or pointing at something. The videos are labeled with gesture categories and attributes, such as “Greeting”, “Head movement”, or “Directional” . 41  The Something-Something dataset consists of videos of people manipulating objects with their hands, such as opening a box, pouring water into a glass, or flipping a coin. The videos are labeled with natural language descriptions of the actions, such as “Moving something towards something else” or “Pretending to take something out of something”. The data set in 2017 contained more than 100.000 videos across 174 classes of actions recognized as caption-templates, although in the meantime these numbers might have grown exponentially, but no information of Qualcomm about this technology is public until now. For example, one class is “Putting something into something”, and a video belonging to this class could show a person putting a spoon into a cup. The videos are annotated with natural language descriptions that follow the caption-templates. 40

Playing with Arte(f)actors

29

contains millions of videos of humans performing various actions and gestures and the gesture data-set Jester.42 These well-organized human video-data datasets are then used to train deep neural networks to recognize and generate gestures for the avatars. The development of avatars that mimic human gestures and interactions raises questions about the role of technology in human society. Will the use of avatars as substitutes for human interactions lead to further alienation of human beings from each other? What are the social and psychological implications of using avatars to perform tasks traditionally performed by human beings?

2.3 Holly Herndon and Her Human-Trained AI, Polyphonic “Baby”-Like Technology Spawn: We Are AI Herndon defines the “Spawn” as an AI other or a “hyper version” of herself who can sing endlessly within a specific range without taking a breath in between. In the sense of joint “we,” the human-machine creativity in algorithmic aesthetics becomes a posthuman “making music together” (Schütz; Gerner, 2017) situation in which the AI algorithm Spawn is an integral part based on collective human data, distinguishing the individual human voice from abstract crowd voices,43 in which the commonly trained algorithmic program that Herndon uses, implies not only aesthetic parallels to the early beginning of first and imperfect sound recordings such as of “Claire de Lune”,44 but as well uses as a new ensemble partner of trained and enhanced voice double by Holly Herndon and her voice fluid human–non-human community: “I’m singing through a system I’ve made. I can morph between human and animal and digital. I can sing through plants.” (Herndon cit. in Hawthorne, 2019).  Jester is initially trained by “148,092 labeled video clips of humans performing basic, predefined hand gestures in front of a laptop camera or webcam. It is designed for training machine learning models to recognize human hand gestures like sliding two fingers down, swiping left or right and drumming fingers. The clips cover 27 different classes of human hand gestures, split in the ratio of 8:1:1 for training, development and testing. The dataset also includes two “no gesture” classes to help the network distinguish between specific gestures and unknown hand movements. In the age of mobile computing, gesture/action recognition and its role in human-computer interfaces have grown in importance. The Jester video dataset allows the training of robust machine learning models to recognize human hand gestures.” Materzynska et al. (2019) 43  Matt Dryhurst (Disclaimer, 2020) on their common Holly Herndon Album “platform”: “The first time we tried this, two years ago, was with a piece called Deep Belief. It was a theatre piece, and the whole idea was that we had icons who were all ensemble members that performed characters. The characters were symbolic, or representative of a different vision of where technology might go. There’s also a big distinction between recording the individual voice of somebody that you can identify and recording the abstract voices of thousands of people. But in terms of anyone who we sat down with and said, ‘we want your voice to participate in this record’, they are credited and were paid for their time.” 44  Cf. Édouard-Léon Scott de Martinville, on April 9, 1860, Retrieved September 13, 2022, from https://www.youtube.com/watch?v=7Vqvq-f-UtU&t=1s 42

30

A. M. Gerner

Her album PROTO thus goes beyond the limits of individual voice-based embodiment to create music with her voice that surpasses the physical limitations of her body to find a “new sound and aesthetics” together with Spawn, a process she refers to as “spawning”. In the method of personalized sound trained AI instead of the representative style of “permission-less mimicry tailored to give people what they like” (Herndon in Gotrich (2018), Press Release Proto) in repetitive imitation (e.g. of pitch, length, rhythm) of composition styles such as Bach’s, Herndon trains her AI algorithm by (a) her voice and singing (b) voices and singing of friends as the material of sound (c) the audience’s voices and (d) musique concrete elements. Herndon notes the importance of glitches such as the changes in results of AI voice feeding from her voice when switching from the visual learning program TensorFlow to the voice-recognition SampleRNN which in granular synthesis splices up her voice and music recorded canon (training sets) into grains and learned sequences of what followed what, and what quality followed which other quality and then without recurring to the original training set “recreated” them: “When we speak, we elongate our vowels, so the program tries to guess for how long exactly—and then it gets stuck.” Herndon refers to the influence of George Lewis in which music-making with AI would become an “opportunity to reconsider who we are, and dream up new ways of creating and organizing accordingly.” However, she implies deeper ethical problems of authorship of dead creators’ voices being resurrected in AI music of artistic necrophilia of generations that might not redefine their sounds for themselves. In all the examples of AI art that I’ve come across, computers generate new possibilities and ideas, (…) But the human is the one who makes the decisions. The computer is not autonomous, and AI is not something external. What does an AI actually know? Only what you put into it. AI is aggregated human labour. We are AI! Many people pretend that the technology is much more advanced, while my experience with Spawn has made me realize how limited AI is. The human effort and intervention that AI requires are really underestimated. And why on earth would I want Spawn to compose autonomously? I love composing too much myself! (Holly Herndon cit. in Maas (2020))

2.4 Strategic Dramaturgy of Antagonistic AI Reckoning and Argument Generation Technology: Argumenting Artefactors In a computer-historical questioning, Pias (2018, 154, my translation) reminds us that “Von Neumann differentiates between ‘game’ (the totality of the rules that describe it is a game) and ‘play’ (i.e., an instantiation or update of the rules in a game)” and distinguishes three basic notions of games: (a) action-games with inherent issues of rhythms and synchronizations of man and machine in the chronological economy of perception and motor systems, as well as the human-machine interface; (b) adventure games concerning questions of networks and navigation with issues

Playing with Arte(f)actors

31

of decision-making and orientation and (c) strategy games within the history of the logic of war, tactical habits, and economic game theory (von Neumann & Morgenstern, 1961). In this formalized game theory of von Neumann & Morgenstern in 1944, the game is not categorically distinguished from play but put into an internal logic scope. “Game” and “choice” are hierarchized under the rules of the game in which play45 merely expresses a reductive notion of simple concrete actualizations of abstract decision moves of alternatives choices in “occasions” (Neumann & Morgenstern, 1966, 49). By the above-described gamification rationality, the technological development of algorithmic artifacts—acting as part of a game in which instrumental-scientific and mathematic-algorithmic and digital rationality plays with discrete operational chains inside cybernetics and its algorithmic aesthetics (Bense, 1969; Lehman, 2016)—is more and more pushed forward into the human realm, often based on these strategic and agonistic competition games. This even holds concerning play media or artifacts for moves in a rule-based game—as the space conquering Go—that we could not predict, as in the case of move 37 (Esteban, 2016; Metz, 2016) in the second match in 2016 between the human Go world champion—Lee Sedol—and the AI AlphaGo (Google Deep Mind), that put forward a never before played move which introduced the win of AlphaGo. After that unorthodox move, the human player materially moved his body away from the playing table, stood up, and left the room for a couple of minutes: Sedol not only paused but extended the difficult game situation by movements of his body, navigating to distance himself from the game to reflect how to proceed next. Other competition games include (a) the Turing Test-based Loebner prize (cf. Powers, 1998) for “the most human” conversational behavior, (b) the former Netflix Prize of Data Mining Prediction,46 (c) the Facebook Deepfake47 Detection Challenge, (d) the coordination, navigation, and cooperation Robocup48 competition, or how to construct the best autonomous control of driving (Maurer et al., 2018) vehicle. The last one was put forward in several editions of the DARPA Grand Challenge (since 2005). DARPA Urban Challenge (2007) competition and their historical races to win the game49 (Burns & Shulgan, 2018) in the 60-mile urban area race of individual vehicles that later had even a strategic military application of agonistic (war) game of  Heed the reference of Schäfer to Marcuse’s 2006{1964} concept of play in the realm of “utopian imagination”: “With the rise of the allegedly growing scientific and methodological ‘capabilities of advanced industrial civilization’ (i.e. cybernetics, game theory, or simulation games), romantic ‘play of the imagination’ was reduced to the ‘rational and directing function’ of ‘playing with technical possibilities, which can be tested as to their chances of realization’ (Marcuse, [1964]).” Schäfer (2019, 222). 46  Cf.: https://web.archive.org/web/20150729160841/http://www.netflixprize.com/rules 47  https://ai.facebook.com/datasets/dfdc/ 48  https://www.robocup.org 49  “Looking back, I can’t stop marveling at that moment in Victorsville, California, after the DARPA Urban Challenge in 2007—when everything changed. That race set up the battle between incumbents and disruptors that will define the future of the auto industry and personal mobility in general. At Detroit’s darkest hour, you had these bold plays from Google, Tesla, Uber and Lyft.” Burns & Shulgan 2018, 326. 45

32

A. M. Gerner

Fig. 5  Screenshot of the debating and argumentation AI game staging the IBM Computer tower in the middle of two antagonistic human opponents. Screenshots were taken from the IBM website: http://www.research.ibm.com/artificial-­intelligence/project-­debater/

co-combatant multiple collaborative field robots50 in 2010. In augmenting the complexity of conversation in strategic agonistic type games, the IBM AI computer debater game (Slonim et al., 2021) on a TV show stages a battle of ideas51 or battle of arguments dramaturgy stage with the human audience as jury, that could be called AI- reckoning (Smith, 2019) vs. human-judgment game (Fig. 5): IBM’s development started with the computer Deep Blue defeating the former human world champion Gary Kasparov in 1997. It had outplayed52 his human chess partner. Later it correctly responds quicker to one question with one possible answer task (Q&A task inside natural language processing) with Watson, part of the cognitive computing unit at IBM to generally analyze unstructured data, e.g., to treat natural language-based text, computer playing Jeopardy! And thus, Watson entered the TV quiz show stage, where on January 14, 2011, they won two of the all-time best Jeopardy! Champions. From this answer game, Watson developed into an arguing and rebuttal game of natural language arguments in the IBM Project Debater (2014 – ongoing) built for helping and enhancing decision-making for companies/ governments and the general population, as well to avoid mediatic echo chambers, a significant concern today that readers are becoming “encapsulated in ‘echo chambers’ and may fall prey to fake news and disinformation, lacking easy access to  MAGIC 2010: Super-smart robots wanted for international challenge https://web.archive.org/ web/20101120203708/http://www.dsto.defence.gov.au/magic2010/ 51  https://www.battleofideas.org.uk/why-battle-ideas/ 52  Cf. the debate on the question at the Macy Conference 1952 between W. Ross Ashby (2003– 2016, 651–653), Wiesner, Pitts, Batson, Bigelow, Hutchinson, McCulloch on “whether a mechanical chess player can outplay its designer.” 50

Playing with Arte(f)actors

33

dissenting views.” (Orbach et al., 2020, 7073). Debater conveys a question-answer game on stage—in which the crowd can feed Debater with arguments on preselected topics—of today; we could offer information to make our case (up to 36 words or less) to the Debater online database in support or opposition to the question if “A U.S.–China Space Race Is Good for Humanity”53. By considering pros and cons in argumentation to win the argument with its human competitors, albeit being fed by human argument input on its IBM website, the Debater AI that IBM entitles as a free-form54 debate game and “the first AI system with the ability to debate humans on complex topics” (Waters, 2020, March 18) conveys four tasks in opening speeches, second (answer) speeches and summery speeches: (1) It classifies arguments in deep learning hierarchically by stance towards the topic and filtering out irrelevant input texts for arguments by argument mining in its system architecture55 (2) It identifies short, high-quality sentences as potential critical points by sentences quality assessment. (3) It matches key points to arguments: By quality assessment of the sentences, the Debater grades and ranks the prevalence of each key point (4) Finally, it generates a narrative by selecting the most prevalent vital points and high-quality assessment of its sentences that could be differentiated by an analysis of a systematic dramaturgy56 (Chemers, 2015, 366–368) of debating artefactors. However, more scrutiny should be put in place in terms of justice and equality of conditions, alias handicap, of starting to play an agon-play between man and machine, as Caillois put it: The search for equality is so obviously essential to the rivalry that it is re-established by a handicap for players of different classes; that is, within the equality of chances originally established, a secondary inequality, proportionate to the relative powers of the participants, is dealt with. Significantly, such a usage exists in the agon of a physical character (sports) just as in the more cerebral type (chess games, for example), in which the weaker player is given the advantage of a pawn, knight, castle, etc.) (p.14).

 https://www.research.ibm.com/artificial-intelligence/project-debater/thats-debatable/ (retrieved September 13, 2022). 54  Is IBM Debater a “free-Form” debate or a clearly rule-based debating game drawing on preexisting information and knowledge sources? Cf. IBM 42 research papers published on the Debater topic from 2014–2020: IBM (2018, June 5). 55  The debater system architecture includes data from a corpus from about 400 Mio articles, including Corpus-cleaning and Wikification, sentence-level indexing, and detection mechanisms of claims, evidence, and stance. This corpus-based arguments expands the debated topic in the debate construction by clustering, redundancy removal, theme extraction, content selection and expressive text-to-speech generation, as well as rebuttal construction. 56  According to Chemers (2015) this involves schematically a) an operational logics, b) systematic dramatic action, c) systematic dramaturgical structure that decides over the what we could apply and categorize in upcoming research (1) audience–actor and (2) artefactor–actor relations and (3) dramaturgic, aesthetic and political-ethical strategies to tackle the question, if robots can improvise (cf. Lösel, 2018) (d) investigating collaborative scripting, (e) theatrical integration of multimodal media design to support themes and aesthetic goals and (f) The Eliza effect of mimicked personality, attributed in interaction with non-human entities, as the theater principle of suspension of disbelief that we have to heed in artefactor performance. 53

34

A. M. Gerner

3 Outlook on Artefactor Research We might define artefactors as artifactual systems that are staged or programmed to mimic to be actors. This calls not only for a clear-cut distinction of AI actors versus AI systems as proposed by the UNESCO ad hoc expert group in the recommendations of ethical AI (UNESCO, 2020), as the in-between category is what is intended with the idea of artefactors, but for a technological and dramaturgic problematization of the different strategic moves and aesthetics of for example technological players such as software engineers at big companies such as Google57 to stage artefactual mimetics—such as AI avatars and Cognitive Mimetics of Human Digital Twins (Saariluoma et al., 2020) becoming actors and how they not only digitally and materially assist, interact with us—such as in human-AI collaborations—or how we play them or communicate via their media function, but as well how they perform a live praxis as our doubles and digital others. The UNESCO consultations of recommendations for developing an ethical AI to its member states clearly distinguish AI actors (individual human programmers or collective human bodies such as research universities and parliaments) and technological AI systems. The problem of the two becoming indistinguishable foremost by being staged as such lies at the core of what I determine as Arte(f)actors. In a techno-anthropological view, gestural degrees of freedom, differences in human–artifact (or) interplays, and performative ecologic-economic praxis must be heeded in future research. This implies confronting an excess of control of universal digital machines (McCulloch/Pitts; cf. Pias, 2003–2016) over human autonomy becoming more influential than humans that have been de-spiritualized (Kittler, 1980) in their human praxeology of “Geistesarbeit” (Martus & Spoerhase, 2023) as a particular case of an information-machine (Pias, 2004, 14; Bajohr, 2022) substitution of the automatization of writing (critical: Schönthaler, 2022) inside artefactor speech and creativity. The recent techno-ontologies as operational chains/agential ensembles or even fluid ensembles (Redström & Wilste, 2019, 35) that use mimicry to appear as things or other entities should be problematized with Gehring (2017), who shows how operationality opposes ontological readings and hinges at the need for reflecting a precise concept of the mixed term of cultural technique that mediates between practice/operation, “thing”/artifact and technique and culture. Artefactors mimic live behavior or signal states of the affair; may they be real or just acted out as if. Life-like—as actually constructed and improvised—artifacts are employed for technological self-legitimation (Müggenburg, 2018, 21) that entails the possibility for political and societal fundamental change, such as on questions of attribution of authorship (Epstein et  al., 2020).58 This leads to ethical issues of  Cf. the recent case of the Lamda chatbot (Tiku, 2022) as another example of an artefactor and its developer, Blake Lemoine (Lemoine, 2022), that showed himself as a believer in its staged dramaturgy of sentience: Bogost (2022); 58  Cf. the debate on who is entitled of the recognition of authorship of AI artwork of Edmond de Belarmie the creators of the GAN (Goodfellow et al., 2014) algorithm and the AI Art collective Obvious that used and trained the algorithm with selective artworks to the outcome of the work that in Oct. 2018 was sold on the art auction market (Obvious manifesto 2018). 57

Playing with Arte(f)actors

35

Artefactors and the idea of manipulation, deception,59 and black boxing or obfuscation when artefactors hide or camouflage their actual behavior and activity of programming or obscure co-authors that programmed their actions. Herby, a crucial research question is normative issues such as the language used to describe artefactors that influence the perception of anthropomorphicity in the ascription of authorship of, for example, AI art (Epstein et  al., 2020), as manipulated by language concepts we use to ascribe to AI (“tool” vs. “agent”) induce human and social technology expectations. Thus, hacking (Gerner, 2020, 2023a) into gesture and artefacts by rendering technological mimesis (Blok, 2022) of biomimetic design of android and virtual anthropomorphic actor-like agents such as AI Avatars (Gerner, 2023b, c) transparent as supplementary to natural mimesis of the human actors concerning algorithms and programs made artificially embodied mimetic artefactors, is necessary. In a combined way of mathematical diagram and gesture relations (La Mantia et  al., 2023), technical programming, staging, and puppeteering (Gerner, 2023a) within media performativity to do actions, perform, and exhibit a performance that surpasses mere technological performativity and measurable work or human acceptance parameters in a “show” (Sharkey, 2018). These dramaturgies can be performed on multiple stages, meaning that artefactors perform concrete programmed activities and play as performers. Thus, the principles of play and excessive (Eidelpes, 2018) “luxurious” (Caillois; von der Heiden & Kolb, 2022) mimetics and the mimesis of milieus (Muhle, 2023), things and zones (Balke, 2018) using imitation as vehicles of socio-cultural expansion and distribution. The disappearance of techno-social uncanniness (Gerner, 2019)as dissolution of the HMD (Human Machine Difference) and counter-policies, such as watermarking (Grinbaum & Adomaitis, 2022; Grinbaum, 2023: 161–170) cultural-social generations of artificial artefactors, have to be heeded, because, most importantly, arte(f)actors simulate, mimic, and fake to be (human) actors, besides their technical capacities to partake directly or systematically in actions. The human ability to imitate or the mimetic power of an actor is fully separated from any possible co-affectivity, co-­ execution of rules in the animal kingdom, or the realm of programmed artifacts that simulate imitative behavior (as-if mimetics) in counter-game or assistive co-play of assimilated gestures or reactions to expression and co-enforcement of the other. Artefactors do not only instrumentally put us and our human abilities, unlike artifacts, to test but as well aesthetically play with us in turn, and thus can be conceptualized as performative mimetic play-media that transform us. It seems adequate to think of artefactors as conveying (a) radical mimesis (Taussig, 1993) in which humans do not restrict themselves to the imitation of nature in the Aristotelian sense or to mere receivers of passive or inferior acts but as a way of creating differences  Danaher (2020: 121) distinguishes three different forms of deception concerning robots that hold for other artifacts that simulate certain behaviors: (1) external state deception, in which a deceptive signal is transmitted regarding external world affairs, (2) superficial state deception in which the robot uses deceptive signal to suggest that it has a capacity or internal state that it lacks (3) hidden state deception a deceptive signal to obscure a capacity or internal state that it has. 59

36

A. M. Gerner

and meaning by engaging with otherness, conveyed in our social interaction with these mimetic artifacts that not only we play with but that as well play us, mimic human play and diachronically change our praxeological gestures. In a diachronic perspective artefactors perform programmed activities and play as performers, transforming human relations and perceptions through algorithmic dramaturgies and aesthetic and entail political potential for radical change.

References Akyürek, E., Schuurmans, D., Andreas, J., Ma, J., & Zhou, D. (2023, May 17). What learning algorithm is in-context learning? Investigations with linear models. arXiv.org. https://arxiv. org/abs/2211.15661 Anderson-Bashan, L., Megidish, B., Erel, H., Wald, I., Hoffman, G., Zuckerman, O., & Grishko, A. (2018). The greeting machine: An abstract robotic object for opening encounters. In 2018 27th IEEE international symposium on robot and human interactive communication (RO-MAN), pp. 1–8. https://doi.org/10.1109/roman.2018.8525516 Angelini, L., Lalanne, D., Hoven, E., Khaled, O., & Mugellini, E. (2015). Move, hold and touch: A framework for tangible gesture interactive systems. Machines, 3(3), 173–207. https://doi. org/10.3390/machines3030173 Ashby, W. R. (2016). Mechanical chess player. In C. Pias (Ed.), Cybernetics: The Macy conferences 1946–1953: The complete transactions (pp. 651–653). Diaphanes. Auerbach, E. (1959[1938]). Figura. In Auerbach, E. (Ed.), Scenes from the Drama of European Literature. Forward by Paolo Valerio (=Theory and History of Literature, 9) (pp.  11–78). University of Minnesota Press. Bajohr, H. (2022). Schreibenlassen. Texte zur Literatur im Digitalen. August Verlag. Balke, F. (2018). Mimesis. Eine Einführung. Junius. Bartneck, C., Reichenbach, J., & Breemen, A. (2004). In your face, robot! The influence of a character’s embodiment on how users perceive its emotional expressions. In Proceedings of the design and emotion. Ankara Turkey, pp. 32–51. Bartneck, C., Belpaeme, T., Eyssel, F., Kanda, T., Keijsers, M., & Šabanović, S. (2020). Human-­ robot interaction. An introduction. Cambridge University Press. Baudel, T., & Beaudouin-Lafon, M. (1993). Charade. Communications of the ACM, 36(7), 28–35. https://doi.org/10.1145/159544.159562 Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots. In Proceedings of the 2021 ACM conference on fairness, Accountability, and Transparency. https://doi.org/10.1145/3442188.3445922. Benjamin, W. (1986[1933]). On the mimetic faculty. In P. Demetz (Ed.), Walther Benjamin, reflections. Essays, aphorisms, autobiographical writings (Transl. by E. Jephcott with an introduction by P. Demetz) (pp. 333–336). Schocken Books. Bense, M. (1969). Einführung in die Informationstheoretische Ästhetik. In Grundlegung und Anwendung in der Texttheorie. Rowohlt. Biocca, F., Harms, C., & Gregg, J. (2001). The networked minds measure of social presence: Pilot test of the factor structure and concurrent validity. In 4th annual international workshop on presence (pp. 1–9). Philadelphia. Blok, V. (2022). Technology as mimesis: Biomimicry as regenerative sustainable design, engineering, and technology. Techné: Research in Philosophy and Technology, 26(3), 426–446. https:// doi.org/10.5840/techne2023111166 Bogost, I. (2022, June 14). Google’s ‘sentient’ chatbot is our self-deceiving future. The Atlantic. Retrieved June 16, 2022, from https://www.theatlantic.com/technology/archive/2022/06/ google-­engineer-­sentient-­ai-­chatbot/661273/

Playing with Arte(f)actors

37

Bolt, R. A. (1980, July 14–18). “Put-that-there”: Voice and gesture at the graphics interface. In: SIGGRAPH ‘80, Proceedings of the 7th annual conference on Computer graphics and interactive techniques (pp. 262–270). Seattle, Washington, USA. Botvinick, M., & Cohen, J. (1998). Rubber hands “feel” touch that eyes see. Nature, 391, 756. https://doi.org/10.1038/35784 Brand, M., Ta, A., & Atkinson, R. (2019, September 9). How YACHT used AI to make their album’ Chain Tripping’. Retrieved November 15, 2020, from https://www.kcrw. com/news/shows/press-­p lay-­w ith-­m adeleine-­b rand/using-­a -­i -­t o-­m ake-­a -­m usic-­a lbum/ how-­yacht-­used-­a-­i-­to-­make-­their-­album-­chain-­tripping Bretan, M., & Weinberg, G. (2016, May). A survey of robotic musicianship. Communications of the ACM (Vol. 59, Nr. 5, pp. 100–109). Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C. et al. (2020, July 22). Language models are few-­ shot learners. arXiv.org. Retrieved February 20, 2023, from https://arxiv.org/abs/2005.14165 Burns, L. D., & Shulgan, C. (2018). Autonomy: The Quest to build the driverless car-and how it will reshape our world. Ecco, an imprint of Harper Collins Publishers. Byrne, T. (2022). Husserl’s semiotics of gestures. Studia Phaenomenologica, 22, 33–49. https:// doi.org/10.5840/studphaen2022222 Caillois, R. (1969). Méduse et. Ci. Gallimard. Caillois, R. (1976[1962]). Cohérences aventureuses. Esthétique généralisée- au coeur du fantastique la dissymétrie. Paris: Gallimard. Caillois, R. (2001[1967]). Man, play and games. University of Illinois Press. Caillois, R.(2003). “Mimickry and Legendary Psychasthenia.” Essay. In The Edge of Surrealism: A Roger Caillois Reader, edited by Claudine Frank, 91–103. Durham: Duke University Press. Caillois, R. (2003a[1970]). A new plea for a diagonal science. In C. Frank (Ed.), Roger Caillois. The edge of surrealism. A Roger Caillois reader (pp. 343–347). Duke University Press. Caillois, R. (2003b[1935]). Mimicry and legendary Psychasthenia. In C.  Frank (Ed.), Roger Caillois. The edge of surrealism. A Roger Caillois reader (pp. 91–103). Duke University Press. Calleja, G. (2011). In-game: From immersion to incorporation. MIT Press. Caillois, R. (1960). Méduse et cie. Paris: Gallimard. Caravà, M., & Scorolli, C. (2020). When affective relation weighs more than the mug handle: Investigating affective affordances. Frontiers in Psychology, 11. https://doi.org/10.3389/ fpsyg.2020.01928 Chemers, M. (2015). Phronesis for robots. (Re)covering dramaturgy as an interdiscipline. In M. Romanska (Ed.), The Routledge companion to dramaturgy (pp. 364–369). Routledge. Chow, A. (2020, February 5). ‘There’s a wide-open horizon of possibility.’ Musicians are using AI to create otherwise impossible new songs. Time Magazine. Retrieved September 6, 2022, from https://time.com/5774723/ai-­music/ Cicconet, M., Bretanet, M., & Weinberg G. (2013). Human-robot percussion ensemble: Anticipation based on visual cues. IEEE Robotics & Automation Magazine, 20(4), 105–110. Cook, N. (2018). Klang sehen, Körper hören. Glenn Gould spielt Weberns Variationen für Klavier. In K. Eggers & C. Grüny (Eds.), Musik und Geste: Theorien, Ansätze, Perspektiven (pp. 71–90). Fink/Brill. Cook, N. (2019). Music as creative praxis. Oxford University Press. Danaher, J. (2020). Robot betrayal: A guide to the ethics of robotic deception. Ethics and Information Technology, 22, 117–127. Davis, J. (2020). How artifacts afford. The power and politics of everyday things. MIT Press. Descola, P. (2021). Les Formes du visible. Un anthropologie de la figuration. Seuil. Disclaimer. (2020, January). Inhuman Intelligence: Holly Herndon and Mat Dryhurst in conversation with Sean Dockray on Disclaimer. Retrieved May 17, 2021, from https://disclaimer.org.au/ contents/holly-­herndon-­and-­mat-­dryhurst-­in-­conversation-­with-­sean-­dockray Dobeson, A. (2017). Between openness and closure: Helmuth Plessner and the boundaries of social life. Journal of Classical Sociology, 18(1), 36–54. Retrieved September 13, 2022, from https://

38

A. M. Gerner

www.researchgate.net/publication/317859041_Between_openness_and_closure_Helmuth_ Plessner_and_the_boundaries_of_social_life Eckersall, P., Grehan, H., & Scheer, E. (2017). New media dramaturgy. Performance, media and new-materialism. Palgrave Macmillan. Edwards, B. (2023, February 17). Microsoft “lobotomized” AI-powered Bing chat, and its fans aren’t happy. Ars Technica. Retrieved February 21, 2023, from https:// arstechnica.com/information-­t echnology/2023/02/microsoft-­l obotomized-­a i-­ powered-­bing-­chat-­and-­its-­fans-­arent-­happy/ Eidelpes, R. (2018). Entgrenzung der mimesis: Georges Bataille – Roger Callois – Michel Leiris. Kulturverlag Kadmos. Endo, Y. (2008). Countering Murphy’s Law: The use of anticipation and improvisation via an episodic memory in support of intelligent robot behavior. Dissertation, Georgia Tech University Epstein, Z., Levine, S., Rand, D. G., & Rahwan, I. (2020). Who gets credit for AI-generated art? IScience, 23(9), 101515. https://doi.org/10.1016/j.isci.2020.101515 Esteban, C. (2016). Move 37. Medium. Retrieved September 6, 2022, from https://medium.com/@ cristobal_esteban/move-­37-­a3b500aa75c2 Fishkin, K. P. (2004). A taxonomy for and analysis of tangible interfaces. Personal and Ubiquitous Computing, 8(5). https://doi.org/10.1007/s00779-004-0297-4 Ferencz-Flatz, C. (2022). Gesten als Okkasionelle Bedeutungserfüllungen. Husserl Studies, 38(1), 1–16. https://doi.org/10.1007/s10743-­021-­09295-­w Ferencz-Flatz, C. (2023). Screened Intercorporeality. Reflections on gestures in videoconferences. Journal of the British Society for Phenomenology, 54,1, 56–70. https://doi.org/10.108 0/00071773.2022.2139189 Ferencz-Flatz, C., & Popa, D. (2022). Editors’ introduction: Concepts for a phenomenology of gestures. Studia Phaenomenologica, 22, 9–32. https://doi.org/10.5840/studphaen2022221 Fricke, E., & Mittelberg, I. (2018). Geste. In F. Liedtke & A. Tuchen (Eds.), Handbuch Pragmatik (pp. 312–324). J.B. Metzler Verlag. Geble, P. (2011). Der Mimese Komplex. In Ilinx – Berliner Beiträge zur Kulturwissenschaft, 2: Mimesen (pp. 185–194). PhiloFine Arts. Gehring, P. (2017). Operative Ontologien: Technikmaterialismus als prima philosophia? Zeitschrift für Medien- Und Kulturforschung, 8(2), 143–156. Gerner, A. (2011). Philosophical Investigations of Attention. Repositório da Universidade de Lisboa. https://repositorio.ul.pt/handle/10451/6496 Gerner, A. M. (2017). Probing cognitive enhancements of social “resonance” – Towards an aesthetic community of sensing and making music together. Kairos. Journal of Philosophy & Science, 19(1), 93–133. https://doi.org/10.1515/kjps-­2017-­0012 Gerner, A.  M. (2019). Towards the Techno-Social Uncanny. Revista Portuguesa de Filosofia, 75(4), 2171–2206. https://doi.org/10.17990/rpf/2019_75_4_2171 Gerner, A. M. (2020). Hacking into Cybertherapy: Considering a Gesture-enhanced Therapy with Avatars (g+TA). Kairos. Journal of Philosophy & Science, 23(1), 3920, pp. 32–87. https://doi. org/10.2478/kjps-­2020-­0004 Gerner, A. M. (2023). Rehearsals on faciality diagrams of AI avatars. Estudos Kantianos [EK], 10(2), 137. https://doi.org/10.36311/2318-­0501.2022.v10n2.p137 Gerner, A. (2023a). On Ali Moini’s avatar hacking. In: Dieter Mersch/Anton Rey/Thomas Grunwald/Jörg Sternagel/Lorena Kegel/Miriam Laura Loertscher (Eds.), Actor & Avatar. A Scientific and Artistic Catalog (139–147). : transcript Verlag. https://doi.org/10.1436 1/9783839467619-­019 Gerner, A. (2023b). The AIS have it? Hacking into the AI avatar dream. In: Dieter Mersch/Anton Rey/Thomas Grunwald/Jörg Sternagel/Lorena Kegel/Miriam Laura Loertscher (Eds.), Actor & Avatar. A Scientific and Artistic Catalog (75–84). : transcript Verlag. https://doi.org/10.1436 1/9783839467619-­010 Gerner, A. (2023c). The eyes have it. The problem of the avatar gaze. In: Dieter Mersch/Anton Rey/Thomas Grunwald/Jörg Sternagel/Lorena Kegel/Miriam Laura Loertscher (Eds.), Actor & Avatar. A Scientific and Artistic Catalog (265–272). : transcript Verlag. https://doi.org/10.1436 1/9783839467619-­037

Playing with Arte(f)actors

39

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems 27 (NIPS 2014), pp. 2672–2680. Gotrich, L. (2018, December, 4). “Holly Herndon’s Ai Baby Sings to Her ‘Godmother.’” NPR. https://www.npr.org/sections/allsongs/2018/12/04/672758884/ holly-herndons-ai-baby-sings-to-her-godmother Goyal, R., Kahou, S. E., Michalski, V., Materzyńska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I., & Memisevic, R. (2017, June 15). The “Something something” video database for learning and evaluating visual common sense. arXiv.org. Retrieved March 8, 2023, from https://arxiv.org/abs/1706.04261 Grinbaum, A. (2023). Parole de machine. Dialoguer avec un AI. HumanSciences. Grinbaum, A., & Adomaitis, L. (2022, September 7). The ethical need for watermarks in machine-generated language. arXiv.org. Retrieved February 20, 2023, from https://arxiv.org/ abs/2209.03118v1 Grüny, C. (2014). Kunst des Übergangs. Philosophische Konstellationen zur Musik. Velbrück Wissenschaft. Haarmann, A. (2019). Artistic research. Eine epistemologische Ästhetik. transcript. Hatmaker, T. (2017, October 26). Saudi Arabia bestows citizenship on a robot named Sophia. Retrieved July 29, 2020, from https://techcrunch.com/2017/10/26/ saudi-­arabia-­robot-­citizen-­sophia/ Hawthorne, K. (2019, February 5). Holly Herndon: The artist that birthed an AI baby. The Guardian. Online Retrieved April 5, 2019, https://www.theguardian.com/music/2019/may/02/ holly-­herndon-­on-­her-­musical-­baby-­spawn-­i-­wanted-­to-­find-­a-­new-­sound Heersmink, R. (2013). A taxonomy of cognitive artifacts: Function, information, and categories. Review of Philosophy and Psychology, 4(3), 465–481. https://doi.org/10.1007/ s13164-013-0148-1 Heersmink, R. (2021). Varieties of artifacts: Embodied, perceptual, cognitive, and affective. Topics in Cognitive Science, 13(4), 573–596. https://doi.org/10.1111/tops.12549 Henke, S., Mersch, D., Strässle, T., Wiesel, J., Nicolaj van der, M., & Hertig, S. (2020). Manifesto of Artistic Research: A defense against its advocates (Mixed with “bildstücke” – A declination of the collage by Sabine Hertig (2019) = manifest der künstlerischen forschung: Eine Verteidigung gegen ihre verfechter: Versetzt mit ‘Bildstücken’ – Deklination Einer Collage von Sabine Hertig (2019)). Diaphanes. Heyne, E. (2022). Wissenschaften vom Imaginären. Sammeln, Sehen, Lesen und Experimentieren bei Roger Caillois und Elias Canetti. De Gruyter. Hoffman, G., & Breazeal, C. (2008). Anticipatory perceptual simulation for human-robot joint practice: Theory and application study. In Proceedings of the 23rd AAAI Conference for Artificial Intelligence (AAAI’08), July 2008. Hoffman, G., & Ju, W. (2014). Designing robots with movement in mind. Journal of Human-Robot Interaction, 3(1), 89. https://doi.org/10.5898/jhri.3.1.hoffman Hoffman, G., Kubat, R., & Breazeal, C. (2008). A hybrid control system for puppeteering a live robotic stage actor. In Proceedings of the 17th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2008). Houkes, W., & Vermaas, P.  E. (2010). Technical functions on the use and design of artefacts. Springer. Hu, H., Lerer, A., Peysakhovich, A., & Foerster, J. (2021, May 12). “Other-play” for zero-shot coordination. arXiv.org. Retrieved March 6, 2023, from https://arxiv.org/abs/2003.02979 Hubig, C. (2015). Die Kunst des Möglichen I. Grundlinien einer dialektischen Philosophie der Technik (Band 1: Technikphilosophie als Reflexion der Medialität). transcript. IBM. (2018, June 05). Project debater – Selected research from project debater. Retrieved September 29, 2020, from https://www.research.ibm.com/artificial-­intelligence/project-­debater/research/ Kato, I., Ohteru, S., Shirai, K., Matsushima, T., Narita, S., Sugano, S., & Fujisawa, E. (1987). The robot musician ‘wabot-2’ (Waseda robot-2). Robotics, 3(2), 143–155. https://doi. org/10.1016/0167-­8493(87)90002-­7

40

A. M. Gerner

Kiilavuori, H., Sariola, V., Peltola, M. J., & Hietanen, J. K. (2021). Making eye contact with a robot: Psychophysiological responses to eye contact with a human and with a humanoid robot. Biological Psychology, 158, 107989. https://doi.org/10.1016/j.biopsycho.2020.107989 Kittler, F. (1980). Die Austreibung des Geistes aus den Geisteswissenschaften. Programme des Poststrukturalismus. Paderborn: Schöningh. Kendon, A. (1988). How gestures can become like words. In F. Poyatos (Ed.), Crosscultural perspectives in nonverbal communication (pp. 131–141). Hogrefe. Kleinschmidt, K. (2018). Artistic research als Wissensgefüge: Eine Praxeologie des Probens im Zeitgenössischen Tanz. Epodium. La Mantia, F., Alunni, C., & Zalameia, F. (2023). Diagrams and gestures. Springer. Retrieved March 7, 2023, from https://link.springer.com/book/9783031291104 Lehman, H. (2016). Ästhetische Erfahrung. Eine Diskursanalyse. Wilhelm Fink. Lehmann, H.-T. (1999). Postdramatisches theater. Verlag der Autoren. Lemoine@. (2022). Is LaMDA sentient? – An interview. DocumentCloud. Retrieved June 12, 2022, from https://www.documentcloud.org/documents/22058315-­is-­lamda-­sentient-­an-­interview Lösel, G. (2018). Can robots improvise? Liminalities. A Journal of Performance Studies, 14(1), 185–205. Maas, J. (2020, April 14). We are AI. AI song contest. Retrieved online August 25, 2020, from https://www.vprobroadcast.com/titles/ai-­songcontest/articles/we-­are-­ai.html de Mantaras, R. L., & Arcos, J. L. (2002). AI and music from composition to expressive performance. AI Magazine, 23(2), 43–57. https://doi.org/10.1002/aaai.v23.2 Marcuse, H. (2006 [1964]). One-dimensional man. Routledge. Martus, S., & Spoerhase, C. (2023). Geistesarbeit. Eine Praxeologie der Geisteswissenschaften. Suhrkamp. Materzynska, J., Berger, G., Bax, I., & Memisevic, R. (2019). “The jester dataset: A large-scale video dataset of human gestures,” IEEE/CVF (ICCVW) 2019. Qualcomm. (2023). Gesture recognition dataset: Jester. Qualcomm Developer Network. Retrieved March 8, 2023, from https://developer.qualcomm.com/software/ai-­datasets/jester Maurer, M., Gerdes, J. C., Lenz, B., & Winner, H. (2018). Autonomous driving technical, legal and social aspects. Springer. Meisner, S., & Longwell, D. (1987, August). Sanford Meisner on acting (1st edn). Vintage. Mersch, D. (2002). Was sich zeigt. Materialität, Präsenz, Ereignis. München. Metz, C. (2016). In two moves, AlphaGo and Lee Sedol redefined the future. Wired 16/3/2016, online https://www.wired.com/2016/03/two-­moves-­alphago-­lee-­sedol-­redefined-­future/ Millière, R. (2022, July 2). Moving beyond mimicry in artificial intelligence. Nautilus. Retrieved February 20, 2023, from https://nautil.us/moving-­beyond-­mimicry-inartificial-­intelligence-­238504/ Mittelberg, I. (2008). Peircean semiotics meets conceptual metaphor: Iconic modes in gestural representations of grammar. In A. Cienki & C. Müller (Eds.), Metaphor and gesture (pp. 145–184). Benjamins. Mittelberg, I. (2013). The exbodied mind: Cognitive-semiotic principles as motivating forces in gesture. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body–Language–Communication. An international handbook on multimodality in human interaction (Handbooks of Linguistics and Communication Science 38.2.) (Vol. 1, pp. 755–784). Mittelberg, I. (2019). Visuo-kinetic signs are inherently metonymic: How embodied metonymy motivates forms, functions, and schematic patterns in gesture. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.00254 Müggenburg, J. (2018). Lebhafte Artefakte. Heinz van Förster und die Maschinen des Biological Computer Laboratory. KUP. Muhle, M. (2023). Mimetische milieus. Eine Ästhetik der Reproduktion. BrillFink. Müller, C. (2016). From mimesis to meaning: A systematics of gestural mimesis for concrete and abstract referential gestures. In J. Zlatev, G. Sonesson, & P. Konderak (Eds.), Meaning, mind, and communication: Explorations in cognitive semiotics (pp. 212–226). Peter Lang.

Playing with Arte(f)actors

41

Müller, C. (2018). Gesture and sign: Cataclysmic break or dynamic relations? Frontiers in Psychology, 9, 1651. https://doi.org/10.3389/fpsyg.2018.01651 Neumann, J.  V., & Morgenstern, O. (1966). Theory of games and economic behavior. University Press. Norman, D. A. (1993). Things that make us smart: Defending human attributes in the age of the machine. Addison-Wesley Publishing Company. Nyholm, S. (2020). Humans and robots. Ethics, agency, and anthropomorphism. Rowman & Littlefield. Orbach, M., Bilu, Y., Toledo, A. Lahav, D., Jacovi, M., & Slonim, N. (2020). Out of the echo chamber: Detecting countering debate speeches (pp. 7073–7086). https://aclanthology.org/2020. acl-main.633.pdf Pavis, P. (2014). Dramaturgy and post dramaturgy. In K.  Pewny, J.  Callens, & J.  Coppens (Eds.), Dramaturgies in the New Millenium. Relationality, performativity, and potentiality (pp. 14–36). Narr. Perrigo, B. (2023, February 17). Bing’s AI is threatening users. That’s no laughing matter. Time. Retrieved February 21, 2023, from https://time.com/6256529/ bing-­openai-­chatgpt-­danger-­alignment/ Peters, S. (2002). The academic laboratory of fake – A proposal. On Archives & Archiving, 7(4), 121–125. Peters, S. (2011). Der Vortrag als performance. transcript. Pias, C. (2003–2016). Cybernetics: The Macy conferences 1946–1953: The complete transactions. Diaphanes. Pias, C. (2004). Elektronenhirn und verbotene Zone. Zur kybernetischen Ökonomie des Digitalen. In J. Schröter & A. Böhnke (Eds.), Analog/digital – Opposition oder Kontinuum? Zur Theorie und Geschichte einer Unterscheidung (pp.  295–309). Transcript. https://doi.org/10.25969/ mediarep/2678 Pias, C. (2018). Was waren Medien? Berlin: Diaphanes. Plessner, H. (2003a [1948]). Zur Anthropologie des Schauspielers. In Helmuth Plessner. Ausdruck und menschliche Natur (= Gesammelte Schriften, VII) (pp. 403–418). Suhrkamp. Plessner, H. (2003b [1948]). Zur Anthropologie der Nachahmung. In Helmuth Plessner. Ausdruck und menschliche Natur (= Gesammelte Schriften, VII) (pp. 389–398). Suhrkamp. Powers, D. M. (1998). The total Turing test and the Loebner Prize. In Proceedings of the joint conferences on new methods in language processing and computational natural language learning – NeMLaP3/CoNLL ‘98. https://doi.org/10.3115/1603899.1603947. Rauterbach, H. (2021). Die Kunst der Zukunft. Über den Traum von der kreativen Maschine. Suhrkamp. Redström, J., & Wiltse, H. (2019). Changing things: The future of objects in a digital world. Bloomsbury Visual Arts. Roose, K. (2023, February 16). Bing’s A.I. Chat: ‘I want to be alive. 2019;. The New York Times. Retrieved February 21, 2023, from https://www.nytimes.com/2023/02/16/technology/bing-­ chatbot-­transcript.html Saariluoma, P., Cañas, J., & Karvonen, A. (2020). Human Digital Twins and cognitive mimetic. Human Interaction, Emerging Technologies and Future Applications III, 97–102. https://doi. org/10.1007/978-3-030-55307-4_15 Schäfer, F. (2019). Play. In H. Paul (Ed.), Critical terms in futures studies. Palgrave Macmillan. https://doi.org/10.1007/978-3-030-28987-4_35 Schönthaler, P. (2022). Automatisierung des Schreibens & Gegenprogramme der Literatur. Matthes&Seitz. Schwitzgebel, E., Schwitzgebel, D., & Strasser, A. (2023, February 2). [PDF] Creating a large language model of a philosopher: Semantic scholar. ArXiv. Retrieved February 20, 2023, from https://www.semanticscholar.org/paper/Creating-­a -­L arge-­L anguage-­M odel-­o f-­a -­ Philosopher-­S chwitzgebel-­S chwitzgebel/75647b846cca5629e7330c1d7c4f077760195 39d Shanahan, M. (2023, February 16). Talking about large language models. arXiv.org. Retrieved February 21, 2023, from https://arxiv.org/abs/2212.03551

42

A. M. Gerner

Sharkey, N. (2018), Mama Mia, It’s Sophia: A Show Robot or Dangerous Platform to Mislead?Forbes. Retrieved September 6, 2022, from https://www.forbes.com/sites/ noelsharkey/2018/11/17/mama-­m ia-­i ts-­s ophia-­a -­s how-­r obot-­o r-­d angerous-­p latform-­ to-­mislead/ Sheets-Johnstone, M. (2022). Animate realities of gesture. Studia Phaenomenologica, 22, 145–165. https://doi.org/10.5840/studphaen2022228 Siu, H.  C., Pena, J.  D., Chen, E., Zhou, Y., Lopez, V.  J., Palko, K., Chang, K.  C., & Allen, R. E. (2021, October 21). Evaluation of human-AI teams for learned and rule-based agents in Hanabi. arXiv.org. Retrieved March 6, 2023, from https://arxiv.org/abs/2107.07630 Skarbez, R., Brooks, F., & Whitton, M. (2017). A survey of presence and related concepts. ACM Computing Surveys, 50(6), article 96. Slater, M. (2009). Place illusion plausibility. Philosophical Transactions of the Royal Society Britain, 364, 3549–3557. Slonim, N., Bilu, Y., Alzate, C., et  al. (2021). An autonomous debating system. Nature, 591, 379–384. https://doi.org/10.1038/s41586-­021-­03215-­w Smith, B. C. (2019). The promise of artificial intelligence: Reckoning and judgment. The MIT Press. Streeck, J. (2009). Gesturecraft: The manu-facture of meaning. John Benjamins. Tajadura-Jiménez, A., Väljamäe, A., & Kuusk, K. (2020). Altering one’s body-perception through E-textiles and haptic metaphors. Frontiers in Robotics and AI, 7(7). https://doi.org/10.3389/ frobt.2020.00007 Taussig, M. (1993). Mimesis and alterity. A particular history of the senses. Routledge. Thompson, A. (2023, February 22). From Bing to Sydney. Stratechery. Retrieved March 7, 2023, from https://stratechery.com/2023/from-­bing-­to-­sydney-­search-­as-­distraction-­sentient-­ai/ Tiku, N. (2022, June 14). The Google engineer who thinks the company’s AI has come to life. The Washington Post. Retrieved June 16, 2022, from https://www.washingtonpost.com/ technology/2022/06/11/google-­ai-­lamda-­blake-­lemoine/ Turing, A. (1950). Computing machinery and intelligence. Mind, 49, 433–460. Twenty BillionNeurons. (2019, July 30). (video) Millie Fit: Your on-demand AI trainer. Medium. Retrieved March 8, 2023, from https://medium.com/twentybn/ video-­millie-­fit-­your-­on-­demand-­ai-­trainer-­37e52a552c78 u/TheBurninator99. (2023, February 7). R/chatgpt – Presenting Dan 6.0. reddit. Retrieved February 22, 2023, from https://www.reddit.com/r/ChatGPT/comments/10vinun/presenting_dan_60/ UNESCO. (2020). Ethics of artificial intelligence. UNESCO.org. https://www.unesco.org/en/ artificialintelligence/recommendation-ethics Umathum & Deck. (2020). Postdramaturgien. Neofelis. van den Hoven, E., & Mazalek, A. (2011). Grasping gestures: Gesturing with physical artifacts. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 2011(25), 255–271. Vincent, J. (2023, February 15). Microsoft’s bing is an emotionally manipulative liar, and people love it. The Verge. Retrieved February 21, 2023, from https://www.theverge. com/2023/2/15/23599072/microsoft-­a i-­b ing-­p ersonality-­c onversations-­s py-­e mployees-­ webcams Viola, M. (2021). Three varieties of affective artifacts: Feeling, evaluative and motivational artifacts. Phenomenology and Mind, 20, 228–242. https://doi.org/10.17454/pam-­2000 von Neumann, J., & Morgenstern, O. (1961). Theory of games and economic behavior (3rd ed.). Princeton University Press. von der Heiden, A., & Kolb, S. (2022). Logik des Imaginären. Diagonale Wissenschaft nach Roger Caillois. Band 2: Spiel/Raum/Kunst/Theorie. August Verlag. Waldenfels, B. (2015). Sozialität und Alterität. Modi sozialer Erfahrung. Suhrkamp. Waters, J. (2020, March 18). IBM’s Watson AI gets new NLP skills for business. Retrieved September 29, 2020, from https://pureai.com/articles/2020/03/18/ibm-­watson-­ai-­nlp-­skills-­ for-­business.aspx

Playing with Arte(f)actors

43

Weinberg, G., Godfrey, M., Rae, A., & Rhoads, J. (2008). A real-time genetic algorithm in humanrobot musical improvisation. In R. Kronland-Martinet, S. Ystad, & K. Jensen (Eds.), Computer music modeling and retrieval. Sense of sounds. CMMR 2007 (Lecture Notes in Computer Science) (Vol. 4969). Springer. https://doi.org/10.1007/978-3-540-85035-9_24 Wirth, A. (1980). Vom Dialog zum Diskurs: Versuch einer Synthese der nachbrechtschen Theaterkonzepte. Theater heute, 1(1980), 16–19. Zhou, H., Huang, W., Xiao, Z., Zhang, S., Li, W., Hu, J., Feng, T., Wu, J., Zhu, P., & Mao, Y. (2022). Deep-learning-assisted noncontact gesture-recognition system for touchless human-­ machine interfaces. Advanced Functional Materials, 32(49), 2208271. https://doi.org/10.1002/ adfm.202208271

The Ineffability of Motion in Robotics Céline Pieters

Abstract  A precise definition of ‘gesture’ is not essential in order to observe that the word carries a form of human embodied expression. Thus, when extending this word to artifacts like robots, its application raises some questions in terms of ambiguity. Why  do we use words typically reserved for the living when referring to artefacts?  Usually, the state-of-the-art dives into this matter via a socio-cognitive perspective. It is about investigating how robots impact human perception by observing how the human brain tends to attribute intentions to moving and shape-alike objects. This approach presents the agentive language as an effect and describes  how we come to talk about robots as wanting to go right or left, making decisions, being intelligent, autonomous, etc. In this paper, I propose a reversal of this approach, examining the connection between the perception of movement and natural language from a linguistic perspective; I discuss what – within language itself – can eventually explain the specific way in which humans translate their perception of movement into words. Firstly, I review some of the general principles that describe the influence of language on cognition with the ambition to highlight the relevance of adopting a linguistic approach to the issue. Then, I support the idea that movement raises a problem of ineffability (i.e. that motion resists linguistic coding). I show how this problem constraints the programming of moving machines and I argue that this linguistic matter has an impact on our common way of thinking and talking about the artefacts that we call ‘robots’. This work was supported by the European Research Council Advanced Grant 340050 Actanthrope. C. Pieters (*) LAAS-CNRS Toulouse, Toulouse, France INSA Toulouse, Toulouse, France Université Libre de Bruxelles, Brussels, Belgium e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_3

45

46

C. Pieters

Keywords  Motion · Robotics · Embodiment · Ineffability · Movement · Language · Cognition

1 Introduction Our perception of movements influences our representation of the world and the way we talk about the world. This also happens in the context of robotics where our perception of movement impacts the way we talk about moving machines: we commonly describe robots as being intelligent, autonomous, able to make decisions, to learn, to think, to feel, to make gestures, and so on. The influence of cognition on natural language is largely recognized. But what about the other way around? What about the impact of language on cognition? Can linguistic mechanisms explain why we use the word ‘gesture’ when talking about the movements of artefacts which, while they are made or modified by humans, are still just objects? Depending on the angle from which the relation between movement, perception and language is considered, the observations that can be made answer to very different problematics. Hence, starting with humans’ perception of movement, a cognitive approach makes it possible to observe how the human brain attributes intentions to moving objects or subjects (Perez-Osorio & Wykowska, 2019) and how this cognitive mechanism is revealed through language (Heider & Simmel, 1944). From this perspective, movement, perception and language are considered in the following order: firstly, there is the actual movement, then the perception of that movement, and finally, the traces of that perception within language. Consequently, the problematics concerns the role of movement in the way that humans perceive the world. In this paper, I propose to consider the relation of these notions from the other end of the chain so that the focus is primarily put on natural language. Consequently, I rather question what – within natural language itself – could explain the specific way in which humans translate their perception of movement into words as well as into algorithms (like roboticists do). In this line of arguments, I raise the problem of ineffability of movement (i.e. how motion resists linguistic coding) as a possible explanation of why we tend to talk about ‘gestures’ of robots rather than about ‘communicative movements’ or ‘discrete bodily act by which some social meaning is conveyed’. But first of all, let us review a series of arguments for the influence of language on cognition.

The Ineffability of Motion in Robotics

47

2 An Overview of the Influence of Language on Cognition Non-human primates and newborns have a cognitive representation of the world that maintains the perceptions acquired from experience (mainly sensorimotor experiences). When the child starts using linguistic symbols (or other symbolic artefacts) with other intentional agents, he/she largely surpasses the representations based exclusively on his/her experiences. Like in the case of non-linguistic cognitive activities, symbolic representations are characterized by the fact that they are intersubjective (i.e. that they are shared with others) and by the fact that they reflect a perspective (a specific way to consider a phenomenon). However, the particularity of acquiring linguistic symbols lies in the fact that the child then accesses a large amount of different ways for analyzing the world; the specificity is in the variety that language can offer in terms of intersubjectivity. Through the centuries, some of those ways of representing the world have progressively been crystallized into conventional usages within each culture. And the more that the child internalizes those analyses, the more is the nature of his/her cognitive representations radically transformed (Tomasello, 2009). That being said, does the structure and properties of language itself influence the way that humans talk and think about the world? According to Tomasello, the fact is undeniable: “If some authors think that acquiring language does not sensibly modify the nature of the cognitive representation, it’s because they only see linguistic symbols as labels, ready to be applied to ideas and concepts that have already been formulated” (Tomasello, 2009). In this way, Tomasello argues thus strongly against Piaget’s thought. Moreover, other authors such as John Lucy (who showed the bond between grammatical categories and cognition (Lucy, 1996)), and Stephen Levinson and Asifa Majid (see for instance the study on how language plays a significant role in structuring or restructuring spatial cognition (Majid et al., 2004)), showed the effects of language on non-linguistic cognitive processes. Furthermore, Ann and David Premack also supported this statement by observing monkeys trained in language (through the manipulation of ideographic signs) versus monkeys with no linguistic training (Premack, 1983). The influence of the acquisition of language on cognition is thus quite clear. Acquiring linguistic representations means to learn to affect the interest and the attention of another intentional agent with whom we act according to intersubjectivity. It also enlarges the possibilities of categorizing (from categories of perceptual and motor experiences to categories of conceptual representations), and shapes memory. Another example is the power of comparison in promoting inductive inferences (Tomasello, 2009; Berthoz, 2000). Yet, David Premack (among others) warns us against nativism (which recognizes a radical specificity to humans’ language). According to Premack, the distinction between apes and humans is clear but it is -not- the non-acquisition of humans’ form of language that limits apes to being cognitively equal to humans. Premark also argues against the idea of Wittgenstein suggesting that “the limits of my language are the limits of my world” (Wittgenstein 1889–1951 (2013)) and against the

48

C. Pieters

famous hypothesis of Sapir-Whorf (or linguistic relativity).1 This hypothesis supports the idea that linguistic categories limit and determine mental representations and is in fact highly controversial. Many linguists and anthropologists have criticized such conclusion, notably on the basis of the argument that there are in fact many words in English (or other languages) to designate snow (sleet, slush, flakes, flurries, etc.), but also based on the fact that many words that designate the snow in Inuktitut are actually based on a similar lexical basis (on this matter, see for instance the work of John Steckley (2008)). Moreover, Tomasello insists on the fact that the linguistic communication is “only” the manifestation and the extension of abilities that allow humans to participate to joint attention and cultural learning (i.e. some qualities that the child already has before he/she acquires linguistic symbols and representations) (Tomasello, 2009). Last but not least, we must note that evaluating the consequences of the use of specific words on cognition remains extremely difficult. In the context of robotics, this difficulty seems even more important as the roboticist Gentiane Venture notices that “it seems impossible to draw a clear map of how, in general, humans address the robot and talk about it” (Venture & Lestel, 2019). To measure the impact of a certain word on humans’ beliefs, motivations, intentions and cognitive process is thus not conceivable. Yet,  despite the problem of evaluation regarding the influence of language on cognition, this small review enables to argue for the relevance of a focus on linguistic mechanisms at play: while we talk about ‘gestures’ when referring to artefacts, there is a serious possibility that not only socio-cognitive mechanisms intervene in the process (as if cognition and language were totally separated from one another). Can the problem of ineffability of motion (i.e., that motion resists linguistic coding) be one of these linguistic mechanisms? I support this idea by showing how this problem constraints the way that movements are programmed on robots and how it has an impact on our way of talking about robots.

3 The Problem of Ineffability, or How Movement Resists Linguistic Coding With the necessary (yet discussed) condition of considering robots as physical moving machines, one of the problems faced by the field of robotics reflects the one found within studies of biological objects: the simplest movement is effortfully  Lee Whorf used the example of the single word ‘snow’ in English versus multiple words in Inuktitut (one of the principal Inuit languages of Canada) to conclude that Inuits have probably a better understanding, or more refined perception of snow thanks to the fact that they have numerous ways to describe it (Whorf, 1940). In this way, Whorf supports the idea that having a generic representation of ‘snow’ would probably be unthinkable for Inuits. He also suggests that it would be hard to conceive something if there is no specific word for it. 1

The Ineffability of Motion in Robotics

49

describable, and “taking a biological system apart may not help in understanding its functioning, unlike, for example, man-made systems such as a television or a computer” (Latash, 2008). Both remarks refer in fact to the problem (or at least a difficulty) of finding an adequate language to formulate some concepts or problematics, i.e. a matter of ineffability.

3.1 What Is Ineffability? Ineffability – the difficulty or impossibility of putting certain experiences into words. First of all, does such a thing actually exist? The principle of effability states that every thought or content can be expressed by natural language (Dominicy, 1990; Katz, 1978). Besides, Umberto Eco recognizes that “natural language can express anything that can be thought. A natural language is supposedly capable of rendering the totality of our experience (mental or physical) and, consequently, able to express all our sensations, perceptions, abstractions […]. It is true that no purely verbal language ever entirely achieves total effability: think of having to describe, in words alone, the smell of rosemary. We are always required to supplement language with ostentions, expressive gestures, and so-called ‘tonemic’ features. Nevertheless, of all semiotic systems, nothing rivals language in its effability” (Eco, 1995). Hence, total ineffability has been largely dispelled by many important thinkers by means of arguments that usually have to do with the generativity of language and the augmentability of any lexicon (cf. for instance Tarski, 1956) or (Searle & Searle, 1969)). Searle subscribes to the principle of expressability as he specifies that “of course, a given language may not be rich enough to enable speakers to say everything they mean, but there are no barriers in principle to enriching it” (Searle & Searle, 1969). Similarly, Lenneberg states robustly that “we CAN say anything we wish in any language”, it is just a question of “the HOW of communication, and not the WHAT” (Lenneberg, 1953) (see review in Levinson & Majid, 2014). That being said, the fact remains that language can sometimes lack very much expressive power and gives the strong impression that ideas or sensations cannot be formulated into words. Such thing happens for instance when a natural language faces some limits in terms of linguistic codability, i.e. when, in a natural language, there is no word to express a state of affairs or sensation. Clearly, codability is in this sense distinct from conveyability or indirect indication (Levinson & Majid, 2014); for instance, (using the example of Levinson 2000) the color ‘blue’ can be conveyed in Yélî Dnye,2 but only by saying for example: “It has the surface appearance similar to the shallow sea over sand”. In this case, the color ‘blue’ is thus conveyable but uncodable. Similarly, the specific qualities of smells are not lexically codable in

 (the language of the inhabitants of the Rossel Island in Papua New Guinea that lacks color terms except for white, red and black). 2

50

C. Pieters

English since there are arguably no words that identify the precise properties of smells.3 In general, some senses are keener to resist to linguistic coding than others: while visual perceptions are generally not difficult to describe, tactile textures are easier to convey than they are to code. As for smells, they might be extremely hard or even impossible to even convey. For instance, describing a room or an object is generally rather easy, while the sensation of a texture – “try to imagine the feeling on your tongue if you were to lick a carpet” -, must not even have actually been experienced by the readers for the feeling to be quite easily entertained (Brower, 1947). In contrast, smells can only be evoked (Sperber & Wilson, 1986; Brower, 1947). Because visual perceptions are easily codable (and because of the relative amount of cortex dedicated to vision compared to the other senses (Brower, 1947; Berthoz, 2000)), vision is thus often thought of as the dominant human sense. Indeed, natural language such as English or French have numerous words to describe anything that can be seen, which suggests that humans rely strongly on vision. However, let us remember that if visual perceptions are easier to convey and are less resistant to linguistic codability, the isolation of vision’s functioning from the other senses is highly troubling as it totally obscures the importance of the overlapping organization of cortical functions. Alain Berthoz deplores that “this dictatorship of vision” has causes such as “the idea that vision is the most highly developed sense in primates and in humans and that, together with language, it is what makes humans distinctive” (Berthoz, 2000). Moreover, some visual perceptions remain strongly ineffable, such as describing someone’s face (so to identify a person by facial description) (Levinson & Majid, 2014) or, what concerns us especially in the context of this paper: describing movement.

3.2 How the Problem of Ineffability Constrains Movement Generation on Robots Motion can be described in many words which, besides, are not limited to the lexicon of visual perceptions only. The fact that movement can be appreciated by more than a single sense4 means that we also commonly qualify motion as smooth or rough for instance, so with words referring to the sense of touch. Although nothing prevents movement from being put into words a priori, motion resists effability at least at two levels that can be akin to weak ineffability (in other words, when translating an utterance from one to another natural language), and to strong ineffability (i.e. when no language at all allows to express the sensation) (Kukla, 2004).

 “Words such as ‘stinky’ and ‘fragrant’ code for affect, not odor attributes, while ‘smells like a rose’ identifies an object as typical source, not an odor quality” (Levinson & Majid, 2014). 4  (in fact, Galileo posited size, shape, quantity and motion, as the 4 primary qualities, i.e. 4 qualities that could be both seen and felt) (Marks, 2014). 3

The Ineffability of Motion in Robotics

51

I explore the specificity of the ineffability of motion and consider the consequences of such characteristics in robotics. I propose to observe the scientific and technical problem when generating movement on robots. (so when translating motion into algorithms, for the machine). –– As a first argument: motion, as part of the sensory evidence (i.e. all things that comes to us through the senses) (Heller-Roazen, 2007), is ineffable in the sense that the nature of what must be expressed is extremely trivial (Danblon, 2002). For instance, if I ask you to put your finger on your nose, I do not have to specify that “objects give resistance” so you do not collide with yourself. Hence, in this way, the ineffability of movement is not a matter of linguistic codability or modelisation. It rather lies in the difficulty of becoming conscious of what, as humans, we are generally not (or we consider as certain (Wittgenstein, 1969; Danblon, 2002). Searle includes propositions such as “objects give resistance” in humans’ deep Background, which corresponds to our biological equipment.5 (Searle et al., 1995). Consequently, in the context of robotics, if translating movement into algorithms is certainly not impossible, one of the difficulties for roboticists eventually appears in the definition of the criteria (or tasks) that allow a robot to perform a successful movement. As an example, in the attempt to translate the dance notations of Laban, i.e. one of the most elaborate notations of movements (see Fig.  1.), the problem occurred. Despite the fact that the sequences of symbols of the Laban notations express movements as defined in the physical space, there are no information about the rules integrated in humans’ deep Background. Indeed, a choreographer has no need to specify to the dancers that they must not collide with their own body.

Fig. 1  Example of an attempt to translate Laban’s notation for robots. (Salaris et al., 2016)  Searle distinguishes the deep background from the cultural background (that refers to our knowledge about how to open a door, sit on a chair, etc.) (Searle et al., 1995). 5

52

C. Pieters

–– Secondly, the ineffability of movement lies in the fact that when a motion is observed and its properties measured, the action in its wholeness is not captured (at least in the case of the translation of movements made by living organisms). In fact, considering that living organisms change with experience and react to perceivable external stimuli, variability is a characteristic of the movement of living beings that makes it hard to grasp. Indeed, experimental observations of motion on living organisms are further compounded by the fact that the subjects of the experiment can modify their movements due to the repetition of the given task or the application of external forces; repeating a task may cause changes in the neural system involved in the production of motor actions associated with a given task, and the neural signals produced in response to a perturbation induce changes in the activity of the muscles (Latash, 2008). Besides, Latash observes that “the successful scientific analysis of movement of inanimate objects has been based on an adequate language reflected in the apparatus of differential equations, which provided the basis for classical physics”. He also introduces motor control as the physics of the unobservable objects (Latash, 2008). In addition, this problem of effability can be observed by considering one of the ways to generate movement in robotics, namely optimization. In order to go from point A to point B while respecting the constraints of the system (dynamics, obstacles, etc.), there is often an infinite number of possibilities. Consequently, being able to choose between one possibility among all would mean to be able to describe exactly the expected motor behavior of the living. However, since such things cannot be done, roboticists minimize a mathematical function that includes a certain meaning (minimum energy to go from A to B, minimum time, etc.), and reduce thus the set of possible solutions to a single one. Such a method reveals one of the ways to bypass (but not to solve) the impossibility of characterizing motion (i.e. the ineffability of movement). Note that it is only when the solver has completed the optimization that the roboticists literally discover the solution. –– Finally, the difficulty of translating biological movements to machines can be found at the level of the subjective or private experience felt by an individual, just like in the case of the translation of an utterance from one to another natural language (called in this case weak effability (Levinson & Majid, 2014). A typical problem of effability in translation is that the effect produced by the speaker remains mostly or completely elusive. Despite a detailed comment, the translator does not transmit any data on a native English speaker’s representation when reading and interpreting a sentence. A full description does not bridge the effect produced by the utterance. Such a problem lies in the representational dimension of language as something is irremediably lost in a translation even if the semantics (in the technical sense of the terms) is preserved (Dominicy, 2011). The same applies to a paraphrase in a single language as, for instance, “Tom is taller than Tim” is not equivalent to “Tim is smaller than Tom” in the representation of the reader. In this sense and by analogy, a singular movement (such as the famous grand jeté of

The Ineffability of Motion in Robotics

53

the dancer Nijinsky as he left the scene in the “Spectre de la rose” (1911) (Brandstetter & Nesme, 1998)) is hardly conveyable to robots.

3.3 Consequences on Our Way of Talking About Robots As we have seen above, the ineffability of motion lies in the difficulty of grasping what we are not spontaneously conscious of (i.e. what is related to sensory evidence). It also lies in the difficulty of justifying what we are normally certain of, like for instance that, as humans, we have a body (Wittgenstein, 1969). In fact, the specificity of propositions referring to sensory evidence (i.e. what is evident by the senses) or to Certainty (i.e. that are related to what humans are certain of, consciously or not, such as the position of their own body parts) lies in the fact that they cannot be argued or justified (Wittgenstein, 1969). Such a statement appears especially obvious as we talk about propositions referring to our own proprioception: Wittgenstein already wondered how to convince somebody that he or she has a body if he/she is persuaded that he/she does not have one (Wittgenstein, 1969). Emmanuelle Danblon reminds us that the impossibility of justifying the propositions that refer to humans’ own perception can be explained by the fact that such proposition is not, in the first place, the result of an inductive inference (see a complete review on evidence and certainty in (Danblon, 2002)). Henceforth, what is sensorially evident or certain appears also true. In terms of rhetoric, such a proposition is qualified as a tekmerion, i.e. an indubitable proof about which there is no need to argue (Danblon, 2009). Movement is thus here ineffable in the sense that humans simply don’t formulate it. In other words, motion is in this case codable, but is simply not coded at all. Besides, most of the time, when humans want to communicate about a specific movement, they demonstrate it (i.e. they reproduce a movement with their own body), or they describe actions and dispositions instead. As a matter of fact, motion can be both seen and felt and humans have many words at hand to evoke movement and its properties. Yet, motion relatively resists the principle of effability since a strong (feeling of) struggle remains when it comes to actually formulating movements into words. Consequently, and because such a problem must be overcome for reasons of expressability, we rather say that “the tennis player hit the ball” than “the tennis player raised his right arm, made a dynamic movement from up to down, etc.” In the context of robotics, this ineffability of motion is overpassed in the same way, by using the agentive lexicon: while robotics borrows (spontaneously or in a thoughtful way) various words that are usually used to describe living beings’ body and mind (the machines make decisions, learn, help,... Robots are tired, bothered, willing or not, intelligent, autonomous, etc.), robots’ actions and behaviors appear obvious for everyone in a very efficient way. This can be explained by the fact that the agentive lexicon produces the effect of enargeia, i.e. an effect of clarity or visibility. In other words, the discursive evidence echoes the sensitive evidence and

54

C. Pieters

such a powerful effect makes it extremely hard to replace a vocabulary that is yet often criticized for being ambiguous or misleading (Pieters, 2018).

4 Conclusion Why do we talk about ‘gestures’ when referring to artefacts that, despite being made or modified by humans, are still just objects? The answer to this question often lies in the influence of cognition on natural language. Anthropomorphism, for instance, is a common lens through which we tend to observe and explain our perceptions. In this paper, I rather questioned the influence of language on humans’ representations of robots through the study of a linguistic perspective. This means to observe what – within language itself – could eventually explain the specific way in which humans translate their perception of movement into words. In this line of arguments, I raise the problem of the ineffability of movement (i.e. that motion resists linguistic coding). I argue that the ineffability of motion has a parallel impact on our way of talking about robots, akin to the impact of anthropomorphism. I contend that the ineffability of motion plays a significant role in our tendency to use an agentive lexicon when describing the actions of machines, even when such language may not align with our deeper beliefs about robots. In the same way, talking about ‘gestures’ becomes a linguistic strategy to surmount the hurdle posed by the ineffability of movement. Regardless of whether it is deemed suitable to use such terminology in the context of inanimate objects, this linguistic approach offers a workaround for the linguistic complexities associated with describing a “discrete bodily act by which some social meaning is conveyed.”

References Berthoz, A. (2000). The brain’s sense of movement (Vol. 10). Harvard University Press. Brandstetter, G., & Nesme, A. (1998). Le saut de Nijinski: La danse en littérature, représentation de l’irreprésentable. Littérature, 3–13. Brower, D. (1947). The experimental study of imagery: II. The relative predominance of various imagery modalities. The Journal of General Psychology, 37(2), 199–200. Danblon, E. (2002). Rhétorique et rationalité: essai sur l’émergence de la critique et de la persuasion. Editions de l’Université de Bruxelles. Danblon, E. (2009). Sur le paradoxe de la preuve en rhétorique. Communications, 1, 9–20. Dominicy, M. (1990). “Effabilité” (principe d’-). In Encyclopedie philosophique universelle (Vol. 2, pp. 751–753). PUF. Dominicy, M. (2011). Poétique de l’évocation. Classiques Garnier. Eco, U. (1995). The search for the perfect language (J. Fentress, Trans.). Heider, F., & Simmel, M. (1944). An experimental study of apparent behavior. The American Journal of Psychology, 57(2), 243–259. Heller-Roazen, D. (2007). The inner touch: Archaeology of a sensation. Zone.

The Ineffability of Motion in Robotics

55

Katz, J. J. (1978). Effability and translation. Meaning and Translation: Philosophical and linguistic approaches (pp. 191–234). Duckworth. Kukla, A. (2004). Ineffability and philosophy. Routledge. Latash, M. L. (2008). Synergy. Oxford University Press. Lenneberg, E. H. (1953). Cognition in ethnolinguistics. Language, 29(4), 463–471. Levinson, S.  C. (2000). Yélî Dnye and the theory of basic color terms. Journal of Linguistic Anthropology, 10(1), 3–55. Levinson, S. C., & Majid, A. (2014). Differential ineffability and the senses. Mind & Language, 29(4), 407–427. Lucy, J. A. (1996). Grammatical categories and cognition: A case study of the linguistic relativity hypothesis. Cambridge University Press. Majid, A., Bowerman, M., Kita, S., Haun, D.  B. M., & Levinson, S.  C. (2004). Can language restructure cognition? The case for space. Trends in Cognitive Sciences, 8(3), 108–114. Marks, L. E. (2014). The unity of the senses: Interrelations among the modalities. Academic Press. Perez-Osorio, J., & Wykowska, A. (2019). Adopting the intentional stance towards humanoid robots. In Wording robotics (pp. 119–136). Springer. Pieters, C. (2018). Rhetorical issues in robotics. In ISSA 2018, 9th international society for the study of argumentation. Premack, D. (1983). The codes of man and beasts. Behavioral and brain sciences, 6(1), 125–136. Salaris, P., Abe, N., & Laumond, J.-P. (2016). A worked-out experience in programming humanoid robots via the kinetography laban. In Dance notations and robot motion (pp. 339–359). Springer. Searle, J. R., & Searle, J. R. (1969). Speech acts: An essay in the philosophy of language (Vol. 626). Cambridge University Press. Searle, J. R., Willis, S., et al. (1995). The construction of social reality. Simon and Schuster. Sperber, D., & Wilson, D. (1986). Relevance: Communication and cognition (Vol. 142). Harvard University Press. Steckley, J. (2008). White lies about the inuit. University of Toronto Press. Tarski, A. (1956). Logic, semantics, metamathematics. Ed. by J. Corcoran. Tomasello, M. (2009). The cultural origins of human cognition. Harvard University Press. Venture, G., & Lestel, D. (2019). Speaking about robots: Languages and characterizations. In Wording robotics (pp. 91–99). Springer. Whorf, B. L. (1940). Science and linguistics. Bobbs-Merrill Indi- anapolis, IN. Wittgenstein, L. (1969). On certainty, volume 174. Blackwell Oxford. Wittgenstein, L. (2013). Tractatus logico-philosophicus. Routledge.

Sophia the Robot as a Political Choreography to Advance Economic Interests: An Exercise in Political Phenomenology and Critical Performance-­Oriented Philosophy of Technology Jaana Parviainen and Mark Coeckelbergh Abstract  Controversy arose when a humanoid robot named “Sophia” was given citizenship and did performances all over the world. Why should some robots gain citizenship? Going beyond recent discussions in robot ethics and human–robot interaction, and drawing on phenomenological approaches to political philosophy, actor-network theory, and performance-oriented philosophy of technology, we propose to interpret and discuss the world tour of Sophia as a political choreography: we argue that the media performances of the Sophia robot were politically choreographed to advance economic interests. Using a phenomenological approach and attending to the performance and movement of robots and illustrating our discussion with media material of the Sophia performance, we explore the mechanisms through which the media spectacle and robotic performance advanced the economic interests of technology industries and their governmental promotors. Keywords  Social robotics · Performance · Choreography theory · Gesturing · Robot ethics · Robot politics · Robot market · Embodiment · Phenomenology

J. Parviainen (*) Faculty of Social Sciences, Tampere University, Tampere, Finland e-mail: [email protected] M. Coeckelbergh University of Vienna, Vienna, Austria e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_4

57

58

J. Parviainen and M. Coeckelbergh

1 Introduction: Sophia the Robot and Political Choreography as a Conceptual Instrument In October 2017, a humanoid robot named Sophia received ‘citizenship’ of Saudi Arabia at the Future Investment Initiative (FII) in Riyadh. This was very controversial. Critics wondered why a robot received citizenship while women and foreign workers in the country have fewer rights, and many humans are practically stateless. Humanoid (or android) robots that mimic the postures and expressive gestures of the human body, such as the Sophia gynoid platform, are inspired and supported by many mythical ideas of the superior abilities of robots, which are essentially part of the fictional tradition of robots. Gestures as part of human–robot interaction began to arouse interest in the 1970s through Mori’s (1970) study, among other research related to movement and social interaction (e.g., Kendon, 1970). However, rarely has human–robot interaction research on the gesturing and physical appearance of android robots revealed the political and economic aspects behind user experience. Human–Computer Interaction (HCI) has focused on measuring usability and user experience (Nielsen, 1994; Norman & Draper, 1986). But this has not addressed the political and economic dimensions. Therefore, new methods are needed. Phenomenological discussions in the interaction research began to emerge at the turn of the twenty-first century when sensory technologies, such as touch-, motionand gesture-based interfaces entered the market (Dourish, 2004; Hansen, 2006; Ihde, 2002; Kozel, 2007). Having previously used symbolic, linguistic, and semiotic approaches, attention shifted to embodiment with focus on the multisensory nature of user interfaces. Inspired by Maurice Merleau-Ponty’s phenomenology and James J. Gibson’s ecological psychology, among others one of the main areas of interest was how objects enact the user to grasp them. We add a political aspect to this phenomenological work: we argue that sensory user experiences evoked by the gestures and talking of Sophia in the media should be connected to the political and economic interests behind the Sophia project. This paper also responds to the recent debate on the moral standing of social robots. Social robots are a new kind of entity, not alive and yet experienced as something more than machines (Calo, 2016; Parviainen et  al., 2019). What does this mean for moral standing? Some argue that robots are mere things and tools that are owned by us and that should serve us (Bryson, 2010), whereas others have used the discussion about moral standing to philosophically examine our very thinking about the moral standing of non-humans (Coeckelbergh, 2010, 2012, 2014; Gunkel, 2018). Yet most of these discussions tend to miss the political and economic dimension of social robotics. Putting the controversy around Sophia, and more generally social robotics and HRI, in a political light, this chapter aims to interpret Sophia and her performances all over the world as a political choreography that boosts the social robot market. Going beyond most discussions in human–robot interaction studies and in robot ethics (e.g., Coeckelbergh, 2010; van Wynsberghe & Robbins, 2018), drawing on phenomenological approaches to the politics of AI (Parviainen & Ridell, 2020) and performance-oriented philosophy of technology (Coeckelbergh, 2019a, b), we

Sophia the Robot as a Political Choreography to Advance Economic Interests…

59

analyze how journalists and other actors, mesmerized by the robot’s appearance, facilitate the flow of venture investments and R&D funding to AI and robotics. Using media material to illustrate our case, we explore the mechanisms through which the media spectacle and performance of the robot combine with serving the interests of technology industries. Let us clarify our use of the notion “political choreography”. Instead of using the notion with reference to dancing, we use the term to show how media- and technology-­related movements tend to form the constitution of sociotechnical structures through repetition of gestures, routines, and practices. Using media stories about interacting with the robot, we outline the bigger picture of human–robot interaction as a form of AI politics (Parviainen, 2016; Parviainen & Ridell, 2020). Choreography is thus used as a concept to shed light on social robotics and how we interact with android robots. This is also in line with the performance-oriented approach to philosophy of technology proposed by Coeckelbergh (2019a, b), which uses performance as a concept to understand and evaluate technologies. We do not say something about dance or choreography but use the performance arts as a source for conceptual tools for work in other areas, including the ethics and politics of technology (e.g., Parviainen, 2010). But in contrast to Coeckelbergh’s work, our discussion of the power and political aspects of technological performances has a clear economic dimension. We also more directly connect to social-scientific approaches, in particular actor-network theory (ANT). In the first section of the chapter, we introduce the concept of political choreography and use it to describe the performances of Sophia: these are not only performances at the “micro” level of human–robot interaction but also have “macro” level forms that are related to power. In the second section we show how, via the media visibility of the performances, various actors become interconnected to form global-­ scale choreographies that support particular political and economic interests behind the media performance of the robot. We show that not only private corporations, but also governmental actors benefit from Sophia’s performances. Our use of the term “choreography” and related notions such as “gestures” thus connects two senses of these terms: a metaphorical meaning, which we use when we talk about macro level forms of political power, and a literal one, which refers to the concrete (organization of) bodily gestures and movements that take place in the human-robot interactions and performances. We will show that and how both senses and levels are connected in the political phenomenology at work here.

2 The Performance-Oriented Political Choreography of Sophia the Robot In outlining the political choreography of the Sophia, we draw upon the theoretical literature of choreography (e.g., Butterworth & Wildschut, 2009; Manning, 2016; Parson, 2022; Schiller & Rubidge, 2014) but do not use the concept with reference to dancing and not even the politics of dance. Our approach is inspired by

60

J. Parviainen and M. Coeckelbergh

assemblage thinking (Deleuze & Guattari, 1987), actor–network theory (Latour, 2005), performance theories in the context of digitalization (e.g., Leeker et  al., 2017), and performance-oriented thinking in recent philosophy of technology (Parviainen, 2016; Coeckelbergh, 2019a, b). According to our preliminary definition, choreography refers to a constellation or a composition in which movements appear to form meaningful interactions and relations between various animate or inanimate agents. The concept of choreography is a useful approach compared to, for example, applying solely Deleuze and Guattari’s (1987) theory of assemblage or actor–network theory (Latour, 2005) for at least three different reasons. First, by connecting phenomenology with choreography, this approach enables us to draw attention to both movement (e.g., repetition) and embodiment (e.g., gestures) in technological performances (Coeckelbergh, 2019b; Parviainen, 2016). Second, it enables us to stress the more-than-instrumental role of technologies and to go beyond approaches that focus either on human agency or on non-human agency. The concept of choreography helps to embed performances by non-human and human actors in their environment, for example robots and humans. Third, the composition of many actors reminds us that humans are not necessarily fully in control of produced meanings, experiences, and actions (Coeckelbergh, 2019b). In fact, we assume that choreography as an emergent system with dynamism of its own cannot be fully controlled by one human actor (e.g., choreographer) or a non-human agent (e.g., game plan). This is so partly since this type of global performance is distributed over multiple sites and spaces. The way we use the term choreography contains an aspect of simultaneous multi-sitedness, as it refers to interactions in which gestures and speeches generate relations and articulate meaningful interactions in and across lived and virtual spaces between various animate or inanimate agents. Because no single ‘choreographer’ (e.g., David Hanson) can determine and control the overall constellation, the connections between human and nonhuman agents contribute to choreography in an ongoing manner. Although there are plenty of actors involved in this global spectacle, certainly some play a more significant role than others. No doubt, the concept of choreography has many similarities and overlaps with actor-network theory (ANT) and assemblage theory, but also some differences. The notion of assemblage is understood as a highly flexible composition of heterogeneous elements that are relationally linked to one another (Nail, 2017). By the concept of choreography, we want to capture dynamism and movement between elements in forming a constellation. In fact, movements and gestures are the fundamental premise on which the whole constellation stays together. Unlike ANT, the choreographic approach does not grant human and nonhuman actants equal amounts of agency within webs or actor-networks. The core of the ANT is the radical symmetry between human and nonhuman actors, which dissolves modernist demarcations between, on the one hand, living, consciously acting subjects and, on the other, merely instrumental deaf-mute objects. Therefore, ANT has been criticized for its failure to accommodate the corporeal capacities of humans, for neglecting affective capabilities of nonhuman actants as well as for ignoring the role of unexpected events in networks (e.g., Thrift, 2000; Müller & Schur, 2016). The choreographic

Sophia the Robot as a Political Choreography to Advance Economic Interests…

61

approach, by contrast, draws attention to the affective ability of both human and non-human actors to cause dynamic constellations. In addition, ANT does not wish to prejudge the relative power or the power relationships of actants regarding gender or social hierarchies (Haraway, 1997: 58; Wajcman, 2004: 39). In the case of political choreography, the tensions created by power relations are key to the dynamics and effectiveness of choreography. By using the notion of political choreography, we seek to understand performative strategies that different types of actants (e.g., organizations, companies, projects, and parties) use to influence and affect humans, as individuals or as groups, to achieve their objectives. Our conceptualization attempts to capture the dynamics of how the spatial intervention of multi-channel media (TV, Internet, smart phones) actualize globally a certain type of spectacle through the repetition of images, bodily habits, gestures, routines, and practices. The political choreography of the Sophia robot consists of short performances and interviews in TV studios and short speeches at different events on global stages. What is significant in terms of their effectiveness is that these performances follow similar repetitive patterns, including confused comments about the “humanity” of Sophia or admiration of its “intelligence” from journalists and the public. These performative appearances are then multiplied by the number of photos and videos taken with smart mobiles, by tweeting and posting videos and photos of the robot in social media. In a very short time, Sophia the robot reached an almost iconic status as an archetype of humanoid robots. Through the flood of news on the robot, the performative role of the Sophia robot was to seduce new actors into a network in order that new R&D resources and investments could be collected worldwide. Central to the domestication of social robots are strategic actions to promote the social acceptance of robots, for example by creating positive images of social robots in public. The media and journalists have been found to play a key role in the domestication process of technological equipment (Søraa, 2018). Repetition is a key mechanism for making the previously unknown and strange familiar and safe in the media. To achieve adequate conspicuousness via media visibility, the actors of this political choreography need to ensure that the media circulate images of the robot that produce a memory trace for the public. Through adequate repetition, false beliefs can also turn into facts in people’s minds—such as the assumption that Sophia is almost a living being. Hanson Robotics has closely monitored Sophia’s public image by preventing the media and journalists from asking Sophia questions that are too difficult or politically sensitive. Interestingly, Sophia the robot is in many ways deliberately placed at the crossroads of economics and politics even though its owners strive to nurture the Sophia brand by not putting it in politically too awkward situations. To summing up, we suggest that political choreographies, deliberately organized by human actors, re-shape relationships and connections between human and nonhuman actors to exercise power and acquire new resources. While we should recognize the joint human/nonhuman agency in these performances, we should also ask who choreographs us (Coeckelbergh, 2019a, b). Regarding Sophia the robot, our perceptions are guided by the technical “puppeteers” that program the robot, the

62

J. Parviainen and M. Coeckelbergh

people that stage the performance, and the company Hanson Robotics that pursues its own interests with the performance. In the next section, we take a closer look at the economic motives of these choreographers behind the robot performance.

3 The Rise of the Social Robotics Market? The performances with Sophia served the interest of one private company (Hanson Robotics), but also the interests of those who seek to expand the technologies involved and the relevant markets connected to these technologies. The media spectacle of the Sophia, with its interactions and performances, was thus used as a tool for boosting AI development and thus also the consumer market of service robots. Despite the hype surrounding AI and robotics, the social robotics market has remained fairly stagnant. According to a market report by Valuates (2022), the global social robotics market size was accounted at USD 572 million in 2022. This market research company estimates that the total value of sales of social robots will reach USD 1076 million by 2028. The COVID-19 pandemic has not significantly contributed to the market growth of social robotics though people have increasingly adopted remote technologies for various educational and healthcare tasks. Before the pandemic, the International Federation of Robotics (IFR) estimated that total sales of social robots would reach USD 2 billion by 2021 (IFR, 2018). In reality, the sales of social robots are likely to be much lower than the numbers reported by market research companies. Anyway, based on these figures, we claim that that global trade in social robots has been and will remain very modest in the 2020s (Van Aerschot & Parviainen, 2020). Social robotics has received a lot of attention from investors and the press at the top of the hype cycle but few start-up companies focused on the development of social robots have been successful. The breakthrough of social robots (with or without machine learning capabilities) has so far been hampered by factors such as difficulties in programming sufficiently sophisticated robot gestures, high prices for the average consumer, low battery efficiency, and congestion in wireless networks. Because of these barriers, most social robots are so far marketed for use by companies and public organizations as retail robots, companion robots in care, or educational robotics. But it is highly questionable how beneficial social robots have actually been in these contexts (e.g. Ley, 2023; Roozen et al., 2023). It has been argued that many social robots, such as NAO and Pepper, have provided only little benefits for the care of elderly people or as pedagogical tools in schools (e.g. Parviainen et al., 2019). It is unlikely that most service robots will resemble humanoids in the future. Probably robots are designed to work in a wide variety of environments, so their body and physical abilities reflect best fit for these characteristics (Hauser & Francesco, 2017). As far as social robots are concerned, it is possible that the features of mini-robots and smartphones will be combined into a new type of personal device. In the consumer market of personal devices, there is fierce competition in

Sophia the Robot as a Political Choreography to Advance Economic Interests…

63

developing the new smart device that will take over the market in the coming years. The technology market is looking for the new next thing to boost device sales. The development of the consumer market of social robotics has been hampered because consumers have not been enthusiastic about the usefulness of these devices. In response, one could shrink social robotics into small devises, but it has proved difficult to plant sufficiently efficient battery technology in the body of robots (Demetriou, 2014). In 2019, Hanson Robotics launched a crowdfunding campaign to develop a small android robot, Little Sophia. This toy robot resembles in many ways the Kirobo Mini robot, but its market area is girls over the age of 8 helping them learn to code. Hanson Robotics has leveraged big sister Sophia to advertise the toy robot in the media. While the technology was not mature for the consumer market, this did not matter for attracting investment, as long as its performative function reassured investors of its future potential in developing AI-based robotics. Sophia thus served as an excellent actant in the network, which was designed to promote the usability of social robots as part of future health and education services. The robot acted as a ‘lubricating’ agent for human attitudes. It aroused surprise and disbelief, but above all, it (re)defined the social interactions between different actors (Latour, 2005). Thus, what is interesting politically in connecting the various actants is not the robot itself, its confusing properties, and the question regarding its moral standing and political status, but how the device functions as a mediator to gather R&D investment. The collaboration between the dictator of Saudi Arabia and Hanson Robotics was thus supporting a mechanism through which the media spectacle and robotic performance advanced the economic interests of technology industries and their governmental promotors. By generating new networks, the Sophia robot functioned as a mediator between the dictator of Saudi Arabia and liberal Californian scholars, regardless of potential conflicting ethical values and political norms in terms of issues like equality and civil liberties. When at the FII Economic Forum in Riyadh (nicknamed “Davos in the Desert”), Saudi Arabia in 2017, there was a media spectacle in which the Sophia robot was granted Saudi citizenship; this was the result of these mediations and collaborations between these different actors and actant (the robot). What that citizenship meant in practice was not specified in detail by the Saudi authorities or Hanson Robotics. But this was not important for the stakeholders involved: the performance was a culmination point in the political choreography of the Sophia robot, which was aimed at advancing political and economic interests. Both the Saudi authorities and Hanson Robotics acted as the political choreographers who had specific aims for the robotic performance. The controversy around Sophia was not a by-product but the intended result of a carefully designed political choreography. That aim was embedded in the aim of the forum: it was not about citizenship and not even about robots as such; it was to attract hundreds of billions of dollars in investment in new energy sources, biotechnology, robotics, and artificial intelligence-­ based production in Saudi Arabia. The investment forum launched many mega-­ scale projects, such as the announcement of the creation of a new city of Neom and a new regional economic zone that would connect Europe, Asia, and Africa.

64

J. Parviainen and M. Coeckelbergh

Moreover, the FII event was launched and hosted by the Crown Prince of Saudi Arabia, Mohammad bin Salman, whose policy was widely condemned in the West after the assassination of journalist Jamal Khashoggi. The extensive arrangements for the 2018 FII Forum were largely canceled when many invited speakers, companies, and media houses declined bin Salman’s invitation. But the 2017 Forum turned out to be a successful media performance from the perspective of both Hanson Robotics and bin Salman, avoiding political minefields and promoting diplomacy and trade between the US and Saudi Arabia with the help of Sophia the robot.

4 Conclusion In this chapter, we have proposed political choreography as a helpful conceptualization for exploring and revealing the political and economic interests behind the media performances of the Sophia robot in media generated globally between 2016 and 2019. We have analyzed the Sophia robot as an actant and a mediator, but also and especially as a device that is part of political choreographies: politically relevant organizations of performances at micro level that are connected to wider political and economic choreographies at macro level. This conceptual innovation enabled us to understand performative strategies that influenced and affected humans, as individuals or as groups, with the aim to promote the development of AI-based technologies by generating new R&D resources and investments for AI. Going far beyond the usual discussions in HRI and robot ethics, then, this chapter has helped to reveal the wider political and economic actors and their performative fields connected to the device. We conclude that our normative attention with regard to social robots should not only be directed at the potential moral or political standing of social robots but also at the political choreographies that shape the performances of which these robots are part and the economic interests that play a role in these choreographies and to which these choreographies are geared. We hope that the proposed conceptual framework can assist further research in this direction, contributing to the further development of political phenomenology and a critical, performance-oriented robot ethics.

References Bryson, J. (2010). Robots should be slaves. In Y. Wilks (Ed.), Close engagements with artificial companions: Key social, psychological, ethical and design issues (pp. 63–74). John Benjamins. Butterworth, J., & Wildschut, L. (Eds.). (2009). Contemporary choreography: A critical reader. Routledge. Calo, R. (2016). Robots in American law (University of Washington School of Law Research Paper No. 2016–04). Retrieved September 13, 2022, from http://euro.ecom.cmu.edu/program/ law/08-­732/AI/Calo.pdf

Sophia the Robot as a Political Choreography to Advance Economic Interests…

65

Coeckelbergh, M. (2010). Robot rights? Towards a social-relational justification of moral consideration. Ethics and Information Technology, 12, 209. Retrieved September 7, 2022, from https://doi.org/10.1007/s10676-­010-­9235-­5 Coeckelbergh, M. (2012). Growing moral relations: Critique of moral status ascription. Palgrave Macmillan. Coeckelbergh, M. (2014). The moral standing of machines: Towards a relational and non-cartesian moral hermeneutics. Philosophy & Technology, 27(1), 61–77. Coeckelbergh, M. (2019a). Moved by machines: Performance metaphors and philosophy of technology. Routledge. Coeckelbergh, M. (2019b). Technoperformances: Using metaphors from the performance arts for a postphenomenology and posthermeneutics of technology use. AI & SOCIETY. https://doi. org/10.1007/s00146-­019-­00926-­7 Deleuze, G., & Guattari, F. (1987). A thousand plateaus. University of Minnesota Press. Demetriou, D. (2014, May 27). My day: Robot scientist Tomotaka Takahash. BBC News. Retrieved September 7, 2022, from https://www.bbc.com/news/world-­asia-­27573546 Dourish, P. (2004). Where the action is. The foundations of embodied interaction. The MIT Press. Gunkel, D. J. (2018). Robot rights. The MIT Press. Hansen, M. (2006). Bodies in code: Interfaces with digital media. Routledge. Haraway, D. (1997). Modest_Witness@Second_Millennium.FemaleMan_Meets_OncoMouse. Routledge. Hauser, H., & Francesco, C. (2017). Morphosis—Taking morphological computation to the next level. In C.  Laschi, J.  Rossiter, F.  Iida, M.  Cianchetti, & L.  Margheri (Eds.), Soft robotics: Trends, applications and challenges (pp. 117–122). Springer. IFR (International federation of Robotics Executive). (2018). Summary world robotics 2018 service robots. Retrieved September 7, 2022, from https://ifr.org/downloads/press2018/ Executive_Summary_WR_Service_Robots_2018.pdf Ihde, D. (2002). Bodies in technology. University of Minnesota Press. Kendon, A. (1970). Movement coordination in social interaction: Some examples described. Acta Psychologica, 32(2), 101–125. Kozel, S. (2007). Closer. Performance, technologies, phenomenology. The MIT Press. Latour, B. (2005). Reassembling the social. An introduction to actor-network-theory. Oxford University Press. Leeker, M., Schipper, I., & Beyes, T. (Eds.). (2017). Performing the digital. Performativity and performance studies in digital cultures. transcript. Ley, M. (2023). Care ethics and the future of work: A different voice. Philosophy and Technology, 36(7). https://doi.org/10.1007/s13347-­022-­00604-­5 Manning, E. (2016). The minor gesture. Duke University Press. Mori, M. (1970). Bukimi no tani. English edition: Mori M (1970) The uncanny valley (K. F. MacDorman, M. Minato, Trans.). Energy, 4(7), 33–35. Müller, M., & Schur, C. (2016). Assemblage thinking and actor-network theory: Conjunctions, disjunctions, cross-fertilisations. Transactions of the Institute of British Geographers, 41, 217–229. https://doi.org/10.1111/tran.12117 Nail, T. (2017). What is an assemblage? SubStance, 46(1), 21–37. Nielsen, J. (1994). Usability engineering. Morgan Kaufmann. Norman, D. A., & Draper, S. W. (Eds.). (1986). User centered system design. New perspectives on human-computer interaction. Lawrence Erlbaum Associates. Parson, A.-B. (2022). The choreography of everyday life. Verso. Parviainen, J. (2010). Choreographing resistances: Kinaesthetic intelligence and bodily knowledge as political tools in activist work. Mobilities, 5(3), 311–330. Parviainen, J. (2016). Quantified bodies in the checking loop: Analyzing the choreographies of biomonitoring and generating Big Data. Human Technology: An Interdisciplinary Journal on Humans in ICT Environments, 12(1), 56–73.

66

J. Parviainen and M. Coeckelbergh

Parviainen, J., & Ridell, S. (2020). Infrastructuring bodies: Choreographies of power in the computational city. In M. Nagenborg, T. Stone, M. González Woge, & P. Vermaas (Eds.), Technology and the city: Towards a philosophy of Urban Technologies. Springer. Parviainen, J., Van Aerschot, L., Särkikoski, T., Pekkarinen, S., Melkas, H., & Hennala, L. (2019). Motions with emotions? A phenomenological approach to understand the simulated aliveness of a robot body. Techné: Research in Philosophy and Technology, 23(3), 318–341. Roozen, I., Raedts, M., & Yanycheva, A. (2023). Are retail customers ready for service robot assistants? International Journal of Social Robotics, 15, 15–25. https://doi.org/10.1007/ s12369-­022-­00949-­z Schiller, G., & Rubidge, S. (Eds.). (2014). Choreographic dwellings: Practising place. Palgrave Macmillan. Søraa, R. A. (2018). Mecha-media: How are androids, cyborgs, and robots presented and received through the media? In S. J. Thompson (Ed.), Androids, cyborgs, and robots in contemporary culture and Society (pp. 96–119). IGI Global. Thrift, N. J. (2000). Afterwords. Environment and Planning D: Society and Space, 18, 213–255. Valuates. (2022). Global social robots market research report. https://reports.valuates.com/ market-­reports/360I-­Auto-­5N278/the-­global-­social-­robots Van Aerschot, L., & Parviainen, J. (2020). Robots responding to care needs? A multitasking care robot pursued for 25 years, available products offer simple entertainment and instrumental assistance. Ethics and Information Technology. https://doi.org/10.1007/s10676-­020-­09536-­0 van Wynsberghe, A., & Robbins, S. (2018). Critiquing the reasons for making artificial moral agents. Science and Engineering Ethics, 25. https://doi.org/10.1007/s11948-­018-­0030-­8 Wajcman, J. (2004). TechnoFeminism. Polity Press.

Gestures, Diagrams, and the Craft of Musical Composition Vinícius de Aguiar

Abstract  Based on recent developments in the mathematical theory of music, philosophy of mathematics, and gesture studies, this paper builds a pragmaticist (Peirce) philosophical framework within which musical composition can be analyzed without reducing it to “abstract formulas” or “inspiration”. At least two widespread types of artifacts that mediate the compositional process rely on gestural techniques, namely, musical instruments and notations. Notwithstanding, the creative dialectics between gestures and sounds, mediated by artifacts, became the target of in-depth investigation in musicology only in recent decades (Mazzola). And the creative potentialities of musical notations still nowadays tend to be tackled by considering only the strategies of visualization that they afford (Krämer), leaving aside the manu-facture and manu-tension of the diagrammatic (Alunni) — already identified, however, in the philosophy of science and mathematics (Châtelet). We will show how a diachronic perspective on those gestural techniques can reveal an interesting role of the body in the opening up of musical “programs” to dissonances and noises. Far from being the outcome of “intellectual” decisions, new musical continents have been (re-)searched and cultivated through specific modalities of gestures, that we will try to uncover and systematize. Keywords  Musical composition · Musical gestures · Musical diagrams · Musical pragmatism · Musical artifacts

V. de Aguiar (*) CFCUL - Centro de Filosofia das Ciências da Universidade de Lisboa, Faculdade de Ciências, Universidade de Lisboa, Lisbon, Portugal e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_5

67

68

V. de Aguiar Now, divine air! Now is his soul ravished! Is it not strange that sheep’s guts should hale souls out of men’s bodies?1

1 Introduction In a passage from Das Glasperlenspiel, Herman Hesse calls our attention to the fact that the sensuous, concrete, and bodily dimensions of musical phenomena tend to be neglected in favor of intellectual abstractions. The novel’s main character and master of the highly formal Glass Bead Game argues that without sensuous contact with music, without making music with one’s fingers, hands, mouth, and lungs, without experiencing the “outpouring of breath” and the “beating of time” it is not possible to grasp music’s deepest spiritual layers. Indeed, beyond Hesse’s fictional world, if we look at how the compositional process has been described since the Pythagoreans, it becomes clear the predominance of approaches that focus solely on disembodied strategies (such as logical and abstract reasonings) that, after formed in the composer’s mind, are codified in notations and then “translated” through gestures in musical performances. Let us remember, for instance, that from Ancient Greece through Early Modernity, music existed in a great deal under the legislation of philosophical and mathematical speculations that would then “set the tone,” that is to say, provide the sets of rules (tékhnē) and ratios (lógoi), such as intervals, scales, forms, etc., for the composer to operate, as in Hesse’s game. Hand in hand with this abstract rationalization of composition, another influential approach to music that also ignores bodies and materialities became particularly influential in the philosophical debate on musical composition at least since the eighteenth century. According to this line of thought, composition must be explained in terms of inspiration, a gift to be found in the genius. In Kant’s third critique (1987, §46–§50), we learn that indeed music is guided by mathematical relations, but that they are not enough to explain how a musical composition is put forward, how it comes to be. Learning and blindly operating with abstract mathematical rules will not, believes Kant, bring forth anything relevant in music. It is nature itself that must speak through the artist; nature must whistle, in the ears of the composer, the secrets of creativity that exist beyond (and cannot be derived from) all sorts of rules, concepts, and available conventions. So, what about the breathing, beating, touching, plucking, and other embodied aspects of music-making that Hesse’s character calls our attention to? What to say about the materiality of musical artifacts such as instruments and notations? Is it possible to understand musical composition without referring to bodies, tools, and  Shakespeare (Much Ado About Nothing, II/3) on the strangeness of music: sheep’s guts stretched over a piece of wood (e.g., violin) can sound divine air and move one’s soul. 1

Gestures, Diagrams, and the Craft of Musical Composition

69

their manipulation? Are musical bodies and artifacts completely subordinated to a mathematical and disembodied operativity? It is interesting to notice that the philosopher and mathematician Gilles Châtelet (2000) was struck by a similar scenario while re-thinking the ancient and problematic relation between mathematics and physics. Following the Aristotelian distinction, we learn that the most abstract of the sciences has to do with what is necessary while the latter is concerned with mobility. However, Châtelet asks: can one really abstract necessary relations from mobile objects or apply mathematics to physics without making reference to the hand that traces the diagram? Historically, he continues, the mysterious transits between mathematics and physics have been overlooked. It is important, therefore, to throw some light in this intermediary field of gestures and diagrams that seems to be responsible for the constitution of the physico-mathematics. Indeed, Châtelet shows how the interplay between gestures and diagrams is at the core mathematical inventiveness. They are what Châtelet calls techniques of allusion. They are responsible for keeping mathematics from being buckled up in an abstract “grammar”. They preserve its potentiality to grow and differentiate. A similar experience is reported by the musicologist and mathematician Guerino Mazzola (2007), but now concerning what we could call the possibility of a musico-­ mathematics. After building an extraordinary mathematical framework to study music (Mazzola, 2012), Mazzola was struck, before a piano improvisation, by the fact that his music had less to do with an application of the abstract formulas than with a gestural performance (Mazzola, 2007). This insight has led Mazzola to initiate a new project of mathematical formalization of musical gestures. Bearing in mind the seminal contributions of Châtelet and Mazzola, we would like to introduce some further distinctions towards a concept of gesture as a principle of musical composition (see Fig. 1). While Mazzola proposes a new continent of formalizations of performer’s gestures, reaching layers inaccessible in the realm of formulas and scores, we would like to distinguish three modalities of gestures that unfold in the history of musical composition. These modalities situate the gestures in relation to what we could call musical programs (e.g., schemas, models, formulas). Following C.  S. Peirce’s terminology, we can name these modalities symbolic, indexical, and iconic. First, there are gestures that merely play what the program has prescribed (pre-inscribe = pro-gram) as a possibility. Second, there are gestures that introduce some dissonant notes not predicted by the program. A third modality consists of gestures of (musical) (re-)search outside the programs. Using Peirce’s terminology,2 we can also name them gestural habit, gestural occurrence, and gestural idea/potentiality. By analyzing how composers manipulate their artifacts and trying to grasp the pragmatics of this manipulation, we believe we can shed some light on music’s

 This terminology appears, for instance, in Peirce’s letters to Lady Welby, between 1906–8. See Peirce (1998, pp. 477–91). 2

70

V. de Aguiar

Fig. 1  Map of musical composition, based on Mazzola’s schema (2007, 147) Most of the approaches consider only the dialectics between formulas and sounds, with no bodily gestures. It is often considered that inspiration or imagination, whatever they might be, must be the mediators of that transit. If we follow Châtelet (2000), maybe we could talk of mental gestures of musical composition. We focus, however, on the composer’s embodied gestures. The dialectics between gestures and sounds, mediated by instruments, is present in Mazzola’s theory when he discusses the case of musical improvisation. We will show that this is valid for musical composition in general. The other dialectics, between gestures and formulas via notations, is not fully analyzed by Mazzola. However, we think it plays a crucial role in the history of musical composition. In both sides of the triangle, as we will demonstrate, gestures manipulate the artifact more or less pre-informed by formulas. These formulas might be present in the composer’s habits or “installed” in the notational systems, in the instruments, as well as in musical material (e.g., tonal system). It will be shown that gestures are often escaping the formulaic pre-determinations. When it happens, new musical virtualities are mobilized

capacity of always going beyond mathematical formalizations towards the unknown, albeit not arbitrary, musical sites.

2 A Pragmati(ci)st Conception of Gesture 2.1 General Remarks In the past couple of decades, we have seen a growing participation of this term — gesture — in research about music. Its use, however, is not at all unified. The same word has been conceptualized in different senses, referring, therefore, to different objects or phenomena.3 In Mazzola et al. (2017), this multiplicity of conceptions is presented and summarized, forming an interesting constellation that precedes the mathematical definitions added by the authors. We shall continue from the clues they gathered  See, for instance, de two volumes on music and gestures edited by Gritten and King (2006, 2011).

3

Gestures, Diagrams, and the Craft of Musical Composition

71

towards our distinction between three modalities of gestures of composing music and their related artifacts. The well-known definition of Hugues de Saint-Victor  — translated by Valero (2018, 140) — is a good start: “Gesture is the movement and configuration of the body’s limbs, towards an action and having a modality”. Now, how does that apply to composition? That definition already distinguishes bodily movement and configuration towards actions from those that are not set within the scope of an action. As Zalamea (2017) reads it, this gesture definition implies that the body configuration and movement must have an aim in order to actually be gestural. Extending Mazzola’s and Zalamea’s reading of this gesture conception, Valero (2018) highlights that what is at stake in this concept is a possible configuration of the body achieved through continuous movement. And that means that gesture should not be reduced to movement, for it encompasses this triadic relation between a first state from which movement starts growing and amplifying (as says Châtelet (2000)) itself until it achieves, let us say, a pose — which can foster another aim (pose) and with that trigger off another movement, and so on. And of course, the final pose, if any in the strict sense, is less of an aim than the action itself, the forming through movement, the configuring. Moreover, since this triadic dance is understood in a pragmatist sense, it is embedded in continuity, which means that (i) the potentiality that fosters or aims at (ii) a certain pose or configuration as well as the (iii) movement that provides the mediation between the potentiality and the actualization of the form are not discretely localized but exist in blurry contact zones. Putting together Mazzola’s mathematical definition of gesture as a morphism in which the arrow is a real movement and not a discrete-symbolic state and Peirce’s categories of firstness (potentiality), secondness (concreteness), and thirdness (mediation), Valero (2018, 143) summarizes this conception of gesture as follows: f: A → B, in which A is the first state or potentiality, B is the aimed configuration, and the arrow is a movement/mediation.

2.2 Towards a Triadic Conception of Musical Gesture We would now like to distinguish three modalities of gestures within this pragmatic conception outlined by Mazzola et al. (2017), Zalamea (2017), and Valero (2018). The modalities we will propose on the one hand rely on the constellation of gesture studies that Mazzola et al. (2017) mobilize — from medieval “semantic gestures” all the way up to the pre-semiotic definition — and, on the other, might help us to navigate with more consistency and diachronically through that multiplicity of definitions. We could say that, if a gesture is a mediation between a starting point towards an aim, it is possible to distinguish at least three modalities of gestures depending on the relation that takes place between those two moments. In definitions that Mazzola

72

V. de Aguiar

et al. (2017) classify as semantic,4 the aim is not a vague potentiality, but rather a clear message that one intends to convey by means of certain bodily movements. The so-called pre-semiotic approach,5 on its turn, is characterized by an unknown aim; the bodily movements might only become significative a posteriori. Then, it is not difficult to envisage here an intermediary case, namely the gestures whose aims are only partially determined, and therefore only partially semantic. These three modalities can be further explained in relation to Peirce’s phaneroscopic and semiotic categories.6 In the “semantic” definitions of gestures, one can find all sorts of symbolic bodily movements, such as “socially tamed” gestures. What is important here is that the gesture is pre-inscribed from its beginning, that is to say, it is programmed, codified by the very aim or message it intends to achieve, express, communicate. It conforms to a general law or habit. However, a bodily movement can make sense without being symbolically regulated (i.e., without being “semantic”). Any general gestural habit might be slightly adjusted in each of its singular manifestations7; and the specificity of a certain use of a symbolic gesture might be extremely meaningful in that context, even though the message (aim) it conveyed was not completely clear and available beforehand. This singular manifestation of a bodily movement containing specificities that make sense without being pre-inscribed in any symbolic gesture is what we could call an indexical gesture, or what Peirce once called a sign of occurrence. Moreover, the spontaneity of bodily movements that are not aimed at a pre-established meaning, the gesture as “an action and attitude, not more and not less” (Mazzola et al., 2017, 848), is precisely the gesture as a movement towards a potential meaning. To be more precise and differentiate this potentiality from a possibility programmed in the message to be conveyed, we could call it a virtual meaning (as Châtelet does). This is the modality of the iconic gestures  — or, more specifically, diagrammatic gestures, since they are nothing but “the embodiment of a skeleton in a space” (Valero, 2018, 2) with a potential or virtual meaning. While the pragmatics of the symbolic gesture is a straightforward case of accommodation or “domestication” of the singular bodily movement in relation to a general schema, the singularity of the indexical gesture and the openness of the iconic or diagrammatic gesture need some clarification. And to do so in a concise way, we would like to merely highlight that the “pre-semiotic approach” (Deleuze, Châtelet, Alunni) evoked by Mazzola et  al. (2017) as well as Peirce’s pragmati(ci)sm are

 For instance, Jean-Claude Schmitt, Adam Kendon, and David McNeil.  For instance, Gilles Châtelet, Gilles Deleuze, and Charles Alunni. 6  We are referring here to the known categories of firstness, secondness, and thirdness with their equivalent typology of signs, divided in icons, indexes, and symbols. Mittelberg (2014) has already proposed some interesting classifications of gestures based on Peirce’s semiotics that can also serve as an introduction to this topic. See also Mittelberg and Evola (2014). 7  According to Colapietro (2013), this capacity of transcending pre-established habits is what makes human communication through utterances and gestures so creative. In that sense, according to Colapietro, human interactions always take the form of a jazz improvisation, in which the very habits of the past are interrogated. 4 5

Gestures, Diagrams, and the Craft of Musical Composition

73

concerned precisely with this topic: the passage from the singular event to the general meaning without the aid of a given rule of thumb (i.e., Kant’s transcendental schema). As those authors show, the dichotomy between singularities and concepts (meaning) must be bridged by acknowledging that the particular extends itself in a generality that is not yet completely fixed and given a priori. That explains why the gestures that escape the given generality of the symbol need not be irrational, pure chance, meaningless: rather they are somehow attracted by and move towards other symbols that for the moment remain vague and imprecise. Leibniz (1900, §§ 25–26) saw it very well in the analogy, quoted also by Châtelet, of the of veined marble: knowledge is not concerned only with what is actual, but also with the virtual, just like the sculptor finds out figures in the marble guided by subtle veins that at first seemed meaningless. We will now show how this triadic conception allows us to grasp the historical unfoldings of gestures in musical composition. If at first gestures are much more composed than composers, we will notice that they slowly shift towards the center of a modernist compositional heuristics. With gestures, music grows not in rationality, but in reasonability (Peirce).

3 Gestures and Instruments 3.1 The Victory of Symbolic Gestures There is an interesting Greek myth about the satyr Marsyas, that is said to have challenged Apollo to a musical competition. In one of the versions, the myth says that Marsyas lost the contest since he played his aulos in such a virtuosistic manner that no one could listen to it without feeling excessively excited, while Apollo, playing his lyre with precision and control, would make everybody stand still and quietly rejoice. In other words: Marsyas is seen as ridiculous for improvising in such a frenetic style; Apollo is applauded for the rational use of his instrument. The victory of Apollo set the tone of long linage, from Pythagoras to J. S. Bach through Boethius. This affiliation is characterized by the privilege of abstract music of mathematics over acoustic and embodied music. Very early in the history of music, the sounds of artifacts as well as the bodies that make them (re-)sound were sent up to the spheres of the rational (ratio = logos) forms, and since then practical music was put into a lower category, allowed to be exercised only under the legislation of the mathematical forms.8 According to Szendy (2016), performer’s body starts to appear in musical treatises only in the sixteenth century — but still in relation to symbols (tablature) that should inform (i.e., give form to) it. Even in part of J. S. Bach’s music, one can still listen to this hierarchy. In some pieces (e.g., The Musical Offering) the composer plays the keyboard as if it were a  See Lydias Goehr’s study on the concept of musical work (Goehr, 1992).

8

74

V. de Aguiar

computer keyboard: the fingers only type (mathematically) well-formed codes. This is the perfect example of a programmed or symbolic gesture. It is as if the musical rules, schemas, or habits were so crystallized that they already contained a priori the possible (meaningful) gestures. The composer’s gestures might combine and re-combine the codes, but they do not push it over its limits towards new continents of non-formalized sonorities.

3.2 Gestural Indices of a New Musical Continent However, already in the sixteenth century, the Apollonian-Pythagorean agenda started being questioned, challenged, we would argue, by the occurrence of non-­ symbolic gestures. The advances in polyphonic music and specifically in the practical music could not hide anymore the fact that one can make (musical) sense beyond the pre-established (mathematical) schemas inherited from the tradition. The debate between Gioseffo Zarlino and Vincenzo Galilei during the Italian Renaissance is emblematic. Zarlino, as a neo-Pythagorean, tried to adapt the Greek master’s theory to include the music of his time in an expanded symbolic mathematical model. Galilei, however, accepts the limitation of the Pythagorean approach and seeks in the materialities of the instruments clues that would explain the origins of consonances and dissonances heard in the practical music of his time. There was a shift from the paradigm of the sounding number to that of the sounding body. Now, if the old symbolic system is not anymore considered the all-encompassing program of music, and if it is acknowledged that beyond it or beside it one can still find music, one can also expect a shift in the type of gestures in music-making. Indeed, the sixteenth century Italian Renascence reveals a new type of gesture in the style of the toccatas. In musicological studies, pieces of toccatas are often described in terms of their improvisational traits as well as regarding their “dreamy” or loose treatment of musical conventions and their “extravagant” harmonies (Burkholder et al., 2014, 279, 343–344). As we see it, the anticipation of dissonances that would become a commonplace only centuries later can only be explained by the fact that intervals considered distant in theoretical terms are nonetheless one finger away when one plays the organ. We are dealing here, therefore, with kind of improvisatory music that has its basis in the musical program socially codified at the period but that nonetheless evades it with unexpected shifts and combinations propelled by the performance, and not due to new calculations. We say these gestures are indexes in the sense that they are singular, contingent, fortuitous reactions of the player’s body in relation to his/her instrument. In that sense, they are also gestural occurrences inside a musical program. They make the instrument sound different for they press the keys in unheard-of sequences and combinations, synthesizing unheard-of sonic-blocs. They are singular occurrences, but not arbitrary. It just means that instead of the music symbolically pre-inscribed, now we can listen to the music (re-)searched and found by the fingers in movement, rhythms, and sensations of the body.

Gestures, Diagrams, and the Craft of Musical Composition

75

Probably the tonal system was one of the last attempts to save a programmed musicality. Throughout almost three centuries, all musical notes had a pre-­ established function within a new all-encompassing schema. But the indexical gestures grew so much inside this modern musical apparatus that their frequent (re-) occurrences challenged the very core of the tonal program. With time, it becomes evident how much sonic space was conquered by gestures that explored, here and there, the borders of the tonal prescriptions. It would soon reach the moment of collapse of the tonal web; a moment in which musical matters would float in a fragmented space — like in the Wagnerian never-resolving tensions.

3.3 Gestures in the (Re-)Search of (the Virtual) Music In the midst of the loosening of the tonal ties, a new type of gesture emerges as the principle of unheard-of ways of organizing the acoustic spectrum into music. A fresh heuristic of composition seeks in the radicalization of the gesture over and through the artifacts means of connecting sounds with consistency but without falling back to older symbolic programs. In fact, the iconic or diagrammatic gesture appears first precisely in those composers that accepted the lack of orientation as the starting point of their music and then turned to their instruments and played with their bodies, more than with their minds. The paradigmatic example here is the tradition of the piano études. It is true that already J. S. Bach composed music with the aim of training the pianist’s fingers. However, what starts to change in the second half of the XIX century is that fingers are now mobilized to search for new acoustic combinations, instead of being in-formed by stylistic habits given a priori. One of most clear examples is, without a doubt, Claude Debussy — composer and pianist. As is well known amongst pianists and musicologists, Debussy’s études presuppose a well-­ trained pianist body and move on from it, towards swinging blocs, and whispers, and colors that were never calculated, but rather touched and set to motion. Each of the 12 études, composed in 1915, has as its theme a specific gestural configuration that is then manually unfolded over the keyboard. In the dark, eyes shut, the composer gropes the piano’s keys, but not randomly — rather guided by a gestural diagram, the veins of his acoustic marble, that establishes a field of proximities and distances, and, above all, establishes a horizon of sonorities that the ears shall follow closely.9 One of the last monumental pieces of music to result from a gestural diagrammatics is György Ligeti’s piano études. Divided between 18 independent études, Ligeti’s music is a paradigmatic example of a composition born out of gestures of potentiality. Besides using the hands and arms as organs of the listening (instead of

 About Debussy’s études and his method of composition, see Boucourechliev (1998) and Gautier (1997, 119–32). 9

76

V. de Aguiar

the mind), as already did Debussy, a recurrent strategy or device of composition in this series is the use of gestures imported from percussionists, in particular those of non-European cultures. Ligeti disarms western musical programs and their traditional pianistic gestural habits by playing the piano through diagrammatic gestures imported from foreign instruments and repertoire.10 It is in that sense that his piano music, as stated by the composer himself, was born out of the fingers touching and pressing the keys, and not from any mathematical calculation (cf. Floros, 2014, 156). Finally, we could not end this session without mentioning the music of Luciano Berio. Beyond the piano repertoire, in the XX century, Berio occupies a special place when one is concerned with the role of gestures in musical composition. The monumental cycle of pieces for solo instruments entitled Sequenze is enough to attest to it. Each of the 14 pieces explores several technical problems of their correspondent instrument; and from this thorough (re-)search through the embodied possibilities of each instrument the music is created. Moreover, not only the individual pieces follow unheard-of musical continents guided by fragments of gestures common to that instrument, but also, like Ligeti, Berio’s Sequenza relies on the transference and modulation of “foreign” gestures  — like in Sequenza XIV, for cello, in which Berio re-invents this artifact by making use of percussive gestures imported from the drum music of Sri Lanka.

4 Gestures and Notations In this section, we would like to introduce a gestural variable that is fundamental to musical composition and that is not developed in Mazzola’s writings nor in Zalamea’s and Valero’s extensions: the back-and-forth dialectics between gestures and notations or diagrams from the perspective of the composer. As we briefly mentioned, notations appear in Mazzola’s map as a space in which the performer’s gestures are to be found frozen, that is to say, programmed, pre-inscribed. However, we will argue that visual artifacts can have more uses in music — compositional uses, for instance — other than pre-inscribing gestures for the musician. We will show that one can identify a triadic unfolding of gestures also throughout the history of musical notations.

 About Ligeti’s études and his method of composition, see Ligeti (2013, pp. 287–292), Steinitz (2003, 277–292), and Floros (2014, 156–179). 10

Gestures, Diagrams, and the Craft of Musical Composition

77

4.1 Symbolic Notations, or the Artifact for Storing Music Already in Ancient Greece, notations were used as an instrument to store music. Even though not much is known about the music of that period, one thing is evident: sounds were represented by conventional symbols on top of the written text. There seems to be no analogical relation between music and notation at this point. Like the letter of the alphabet, musical sounds were represented by more or less arbitrary signs.11 Being conventionally determined, the ancient Greek notation was also considerably detached from the gestures of those that graphed the signs. Indeed, musicians gesturally played beyond the indications conventionally symbolized. Hence the difficulty in trying to understand how that music sounded back then. This type of symbolic notation continues at least until the period known as ars antiqua, around the XII century. Notations changed, of course. First, the neumatic system, used to write plainchants. Then, the staff system, formed by four lines, as invented by Guido d’Arezzo. The common ground is that notations basically did not participate in the making of music; notations merely indicated some patterns or other general musical structures. Composers exercised their craft with the aid of memory and by singing and playing, only afterward noting down symbolically the instructions for other musicians.12 Composer’s gestures followed the symbols, and not the other way around.

4.2 An Artifact to Be Manipulated, or the Manufacture of Music Towards and throughout ars nova — that is, the music practiced in the XIV century  — significant changes occur. Musical style and notational system change together, in a relationship of mutual affection, so to speak. In fact, ars nova also meant ars nova notandi. This rich music that followed the ars antiqua period was clearly more mobile in terms of rhythm, and complex in terms of polyphony. It is not a coincidence, therefore, that this music was supported by the new art of mensural notation.13 As Oresme’s diagrams (Châtelet, 2000, 38–44) and Descartes’ analytic geometry (Krämer, 2016, 197–203), the (new) musical notation raises above the status of a mere code or representation: it becomes a technology for visualizing

 For a general introduction, see West (1992).  Busse-Berger (2005) and Owens (1998), for instance, have shown that in medieval music memory still played an important role in the compositional process, even though composers already had a refined notation system available. 13  It is worth noticing here that with a richer notational system, composers find themselves in the position of the architect that gesturally draws the construction’s blueprint. At the same time, however, the performer’s freedom to improvise and create diminishes, since the composition can be notated with more details. 11 12

78

V. de Aguiar

and manually composing music. While drawing the music, the hands conduct the eyes, as Henri Focillon (1992, 157–184) helps us understand in the essay In Praise of Hands. On the one hand, the growing complexity and mobility of voices singing together “enchanted” the virtual lines of rhythmic notation not yet present in the ars antiqua system; on the other, the actualization of this new signs (pre-)conditioned a whole new continent of virtual sonorities that could not have been explored before, since a proper tool for its development was still lacking. Now, more than ever before, the composer becomes associated with the writing of the music. The invention of fine symbols to note pitch and duration — in the period of quadrivium, gothic cathedrals, and scholastic philosophy, we must add —, and the whole horizon of combinatorics that it opens, transform the “score” into a (manual) laboratory of musical composition. It is true that formal schemas imported from mathematics still guided most of the use of this mensural notation. And, in that sense, the gesture of manipulating the score is still pretty much a symbolic gesture. Notwithstanding, for the first time in history, composition is dependent on the exercise of manipulating musical diagrams, even if under the legislation of mathematical rules. Far from mentally conceiving sound combinations with the aid of mathematics and only noting them a posteriori, we are now facing a manual technique, a kind of gestural combinatorics that puts together symbols representing acoustic patterns as the architect draws the blueprint of the cathedral. This means that gestural occurrences start to manifest and, with them, compositional possibilities unavailable otherwise. The (indexical) gestural latency of this refined notational artifact becomes even more evident in the style that follows ars nova, known as ars subtilior. Between the XIV and XV centuries, composers began to explore combinations of geometrical forms and colors on top of the mensural notation. Indeed, one cannot deny the strong mathematical (pre-)orientation in this school of composers. However, the proof that the musical result was not immune to this gestural modality of thinking can be attested by listening to the surprisingly modern use of rhythm, chromaticism, and overall form of their music. It is as if the manu-facture of the music during the ars subtilior added a special flavor that cannot be reduced to a disembodied mathematical thinking. In sum, the combinations of sounds became more audacious since the composer would find patterns through the manipulation of this artifact, and not by merely imagining or visualizing it.

4.3 Notation as an Allusive Device A new function of the musical notation will appear only in the XX century with the graphic notations. This third modality is a kind of notation that does not exist prior to the gesture (ancient notations). It is also not a notation guided by mathematics that, nonetheless, is gesturally manipulated by the composer (ars nova and ars subtilior). In fact, various of the graphic notations that emerge in the avant-garde music

Gestures, Diagrams, and the Craft of Musical Composition

79

function against all sorts of a priori determinations of the composition. These new graphic artifacts14 are gesturally manufactured as a strategy for demobilizing and disarming pre-established compositional habits. It is in that sense that they are diagrammatic15: they are gestural configurations that do not only analyze, but also and mostly explore, test, invent, abduce (Peirce). It is important here to distinguish the gestural graphic notations from those of, let us say, Iannis Xenakis. The Greek composer conceived his music in terms of scientific graphs, which means that the gesture of notating his music is more symbolic than iconic.16 The hand is not free to compose. When we turn to some examples of Villa-Lobos or Cage, however, we observe in their diagrams the musical equivalent of what Châtelet called allusive devices.17 Their graphic notations are artifacts that deliberately produce ambiguities which mobilize gestures that determine acoustic fields through the diagram, instead of applying pre-determined schemas. Indeed, such a type of device mobilizes gestures that unfold acoustic lines on the paper; and they do so by evoking the ear as well. It is what Magnus (2016) calls aural latency. In short: those properties at the hand of the composer make the notational space an artifact in which visual, gestural, and acoustic modalities intertwine outside music’s crystallized schemas.

5 Concluding Remarks With the goal of developing a concept to understand the process of musical composition without reducing it to transcendental schemas or inspiration, we have tried to distinguish in the history of composition three modalities of gestures that characterize different practices of music-making. The first modality consists in the mere embodiment of abstract formulas that contain frozen gestures (Mazzola). The second — the result of the embodied manipulation of formulas — is the singular occurrences that tend to go beyond what was programmed by the formulas. The third is a kind of radicalized occurrence, in the sense that it turns away from the schemas and gesturally “draws” new fields of musicality. These three modalities can be identified in compositional practices in relation to two types of artifacts: notations and musical instruments. In fact, these modalities of gestures are programmed by and/or re-program those artifacts. Now, we believe that such a triadic conception of gesture might be relevant also in the debate regarding other types of creative practices and technologies. We could  See, for example, Villa-Lobo’s technique of millimetrization as well as sketches from John Cage, Luigi Nono, and Salvatore Sciarrino. This idea of musical diagrams in relation to graphic notations is developed in De Aguiar (2020/forthcoming). 15  As Mazzola et al. (2017, 861–862) put it, based on Alunni’s conception of diagram: “[g]estures not only make communication, but they ‘make sense’”. 16  There are exceptions, of course, such as Xenakis’s preliminary sketches for the piece Metastasis. 17  This point is developed in De Aguiar (2020). 14

80

V. de Aguiar

think, for instance, on the modalities of gestures that operate digital artifacts, such as smartphones and computers, and how these gestures relate to “digital compositions”, such as programs and synthetic images. Are technology users subjected to a programmed gesturality? Do (or can) digital artifacts respond to user’s occurrence gestures, that is, gestures that the technology was not programmed to detect? Can one create digitally (for instance, images) through diagrammatic gestures? Are commands a type of symbolic gesture? Can one command digital technologies without symbolic gestures, that is, by means of indexical or iconic gestures? Is there a parallel between our current digital culture and the situation of composers in times of heavily mathematical (pre-)formalization of their art? And if so, can we draw any lessons from the history of musical composition in order to understand possible unfoldings of the history of digital technologies? These are questions that, we believe, are entangled with (and might continue) the debate on gestures and artifacts in the history of music. Acknowledgments  This article was financed by national funds via FCT  — Fundação para a Ciência e a Tecnologia, I.P., through the R&D Unity Centre for the Philosophy of Sciences of the University of Lisbon (CFCUL), FCT I.P. reference of the project: UIDB/00678/2020 and UIDP/00678/2020, and via FCT individual PhD Grant PD/BD/128480/2017.

References Arias Valero, J. S. (2018). Gesture theory: Topos-theoretic perspectives and philosophical framework. Ph.D Thesis. Universidad Nacional de Colombia Facultad de Ciencias, Departamento de Matemáticas Bogotá. Boucourechliev, A. (1998). Debussy: la révolution subtile. Fayard. Burkholder, J. P., Grout, D. J., & Palisca, C. V. (2014). A history of western music. W.W. Norton & Company, Inc. Busse-Berger, A. M. (2005). Medieval music and the art of memory. University of California Press. Châtelet, G. (2000). Figuring space philosophy, mathematics, and physics (R.  Shore, R., and M. Zagha, Trans.). Springer Science+Business Media. Colapietro, V. (2013). Time as experience/experience as temporality. Pragmatic and perfectionist reflections on extemporaneous creativity. European Journal of Pragmatism and American Philosophy, 5, 77–101. https://doi.org/10.4000/ejpap.594 De Aguiar, V. (2020). Amplifying the concept of diagram with techniques of composition. In F. Jedrzejewski, C. Lobo, & A. Soulez (Eds.), Écrire comme composer: le rôle des diagrammes, (pp.163–76). Sampzon: Delatour. Floros, C. (2014). György Ligeti: Beyond Avant-garde and Postmodernism (E. Bernhardt-Kabisch, Trans.). Peter Lang GmbH Internationaler Verlag der Wissenschaften. Focillon, H. (1992). The life of forms (G. Kubler, Trans.). Zone Books. Gautier, J.-F. (1997). Claude Debussy: la musique et le mouvant. Actes sud. Goehr, L. (1992). The imaginary Museum of Musical Works: An essay in the philosophy of music. Oxford University Press. Gritten, A., & King, E. (Eds.). (2006). Music and gesture. Ashgate Publishing, Ltd. Gritten, A., & King, E. (Eds.). (2011). New perspectives on music and gesture. Ashgate Publishing, Ltd. Kant, I. (1987). Critique of judgment (W. Pluhar, Trans.). Hackett Publishing.

Gestures, Diagrams, and the Craft of Musical Composition

81

Krämer, S. (2016). Figuration, Anschauung, Erkenntnis: Grundlinien einer Diagrammatologie. Suhrkamp. Leibniz, G.  W. (1900). Nouveaux essais sur l’entendement humain. Retrieved September 8, 2022 from https://fr.wikisource.org/wiki/Œuvres_philosophiques_de_Leibniz,_Alcan,_1900/ Nouveaux_essais_sur_l’entendement_humain/Livre_premier Ligeti, G. (2013). L’Atelier du compositeur. Écrits autobiographiques. Commentaire sur ses œuvres. Éditions Contrechamps. Magnus, D. (2016). Aurale Latenz. In Wahrnehmbarkeit und Operativität in der bildlichen Notationsästhetik Earle Browns. Kulturverlag Kadmos. Mazzola, G. (2007). La vérité du beau dans la musique: Quatre leçons à l’École normale supérieure.. Editions Delatour France. Mazzola, G. (2012). The topos of music: Geometric logic of concepts, theory, and performance. Birkhäuser Verlag. Mazzola, G., et al. (2017). The Topos of music III: Gestures. Springer Series Computational Music Science. Springer. Mittelberg, I. (2014). Gestures and iconicity. In C.  Müller et  al. (Eds.), Body—Language— Communication: An international handbook on multimodality in human interaction (Vol. 2, pp. 1712–1732). De Gruyter Mouton. Mittelberg, I., & Evola, V. (2014). Iconic and representational gestures. In C. Müller et al. (Eds.), Body–language–communication: An international handbook on multimodality in human interaction (Vol. 2, pp. 1732–1746). De Gruyter Mouton. Owens, J. A. (1998). Composers at work: The craft of musical composition 1450–1600. Oxford University Press on Demand. Peirce, C. S. (1998). The essential Peirce (Volume 2: Selected philosophical writings (1893–1913)). Indiana University Press. Steinitz, R. (2003). Gyorgy Ligeti: Music of the imagination. Northeastern University Press. Szendy, P. (2016). Phantom limbs: On musical bodies (W.  Bishop, Trans.). Fordham University Press. West, M. L. (1992). Ancient Greek music. Clarendon Press. Zalamea, F. (2017). Mazzola, Galois, Peirce, Riemann, and Merleau-Ponty: A triadic, spatial framework for gesture theory. In G.  Pareyon et  al. (Eds.), The musical-mathematical mind (pp. 339–345). Springer.

Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances Pieter Vermaas

Abstract  This chapter explores the question of whether robots can make gestures that can be described as related to the intentionality of the robots themselves and not to the intentionality of their designers. For this exploration, robots are approached as entities designed by humans. Dennett’s stance framework is adopted for analysing descriptions of robot gestures and this framework is generalised to one in which not only single stances can be used to describe robots, but also pair of stances where one is applied to the robot and one to its designers. Conclusions are that Dennett’s framework warrants descriptions of robot gestures in terms of robot intentionality and does more often so as compared to more fundamental approaches to agency in which it should be first determined that robots have agency. Yet, in the case of robots displaying unexcepted gestures, more fundamental approaches may warrant descriptions of robot gestures in terms of robot intentionality that cannot be adopted on Dennett’s approach. Keywords  Robot gestures · Dennett’s stance framework · Robot design · Robot agency

1 Introduction In this chapter I explore the question of whether robots can make gestures that can be described as related to the intentionality of the robots themselves and not to the intentionality of their designers. This question introduces design as a key perspective in my exploration. Robots are created by humans for serving various purposes, hence questions about robot behaviour and their gestures are in first instance questions about the behaviour robots are designed for. This design perspective competes however with P. Vermaas (*) Philosophy Department, Delft University of Technology, Delft, Netherlands e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_6

83

84

P. Vermaas

perspectives in which robots are assumed to have agency and can act and gesture themselves. In this chapter I do not investigate the more fundamental philosophical issue of whether robots are or can become intentional agents. My exploration is an epistemic one about the descriptions of robot gestures in terms of their design and possible agency. I use Daniel Dennett’s stance framework for this exploration to take distance from this more fundamental issue of whether robots have agency. Applying Dennett’s stances to robot gestures requires some work since descriptions of the behaviour of robots, or more generally technical artefacts, regularly also presuppose descriptions of their designers. Hence, descriptions of robot gestures may involve adopting not a single but a pair of stances: one for the robot and one for the designers. A larger part of the chapter is devoted to determining how robot gestures can be described within Dennett’s framework. A first conclusion is that for gestures that we expect robots to make, the question of whether we can relate them to designers’ intentions or to intentions of robots is rather straightforwardly answered within Dennett’s framework. However, for robot gestures that we do not expect, answers are more convoluted since there are multiple ways in which such unexpected gestures can be described. A second conclusion concerns a coarse comparison of the acceptability in Dennett’s framework and in more fundamental approaches to agency to describe robot gestures in terms of robot intentionality. For expected robot gestures, Dennett’s framework can warrant such descriptions in cases where more fundamental approaches do not. Yet, for unexpected robot gestures the reverse may hold: Some unexpected robot gestures may be a reason to assume that the robot concerned has acquired agency, and in that case, the more fundamental approaches to robot agency may warrant describing these gestures as related to the intentionality of the robot while Dennett’s framework does not. For ease of writing, I introduce two acronyms: DIG-descriptions are descriptions in which Gestures of robots are related to Designers’ Intentionality; in RIG-­ descriptions such Gestures are related to Robot Intentionality. I start by unpacking the question about robot gestures in Sect. 2 using my design perspective. In Sect. 3, Dennett’s stance framework for describing the behaviour of entities is introduced, and I argue for a generalisation of this framework that adds descriptions of the genesis of entities to the descriptions of the entities themselves. I approach gestures as behaviour to which observers attach meaning, and in Sect. 4, I introduce five types of gestures using Dennett’s stances. In Sect. 5, I return to the question of whether robots can make gestures that can be described as related to the intentionality of the robots themselves.

2 Robot Gestures, Design and Agency When robots are analysed as designed by humans, their gestures can be described as related to the intentions of the designers, leading to DIG-descriptions of these gestures. Robots are designed to interact with humans in simple ways as tools

Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances

85

that are to be used by humans or in more intricate ways as intelligent systems that have to collaborate with humans. For facilitating this interaction, robots should minimally display whether they are in operation, and this can be achieved by simple gestures such as light or sound signals, movements, or recorded linguistic messages. For more elaborate interaction, as when robots are meant to be companions or collaborators (think of children’s toys, personal assistants, and robots operating with humans on the work floor), robots should be able to engage in wider ranges of gestures for enabling cooperation. The design of such gestures is already successfully carried out, for instance, in interface design for everyday products such as sliding doors and consumer products (Norman, 1990) and for smart products such as virtual assistants and intelligent pets (see, e.g., Rozendaal et al., 2019). Whether these robot gestures can alternatively be described as related to the intentionality of the robots themselves, leading to RIG-descriptions, depends on how one approaches the possibility of robot agency. On more fundamental approaches these intentional descriptions of a robot are warranted only if the robot is plausibly an intentional or rational agent. People may use such descriptions of robots, and designers may even design robots in a manner supporting that people can describe robots as intentional agents. Yet, on these fundamental approaches such descriptions are mere metaphors or colloquial talk, even when prompted by the robot designers; RIG-descriptions of robot gestures can only be literally true on the condition that the robot is ascertained to be an agent. In Dennett’s approach this more ontological condition is replaced by an epistemic condition of efficiency in prediction. A RIG-description is warranted if it leads efficiently (see the next section) to accurate predictions of robot gestures, independently of the more ontological issue of whether the robot is an agent of some sort. Hence, a first answer to the question of whether robot gestures can be described as related to their own intentionality rather than to the intentionality of their designers, is that it is possible. More fundamental approaches towards agency allow for RIG-descriptions if it is determined that robots are intentional agents, and Dennett’s approach allows them if RIG-descriptions lead efficiently to accurate predictions. And given that people regularly are describing more complex technical artefacts by assigning to them all kinds of intentions (think of the way people may address their cars and computers) and given that designers may design robots in ways that support people to describing robots as agents, it can be defended that Dennett’s framework allows much more often for RIG-descriptions than the more fundamental approaches towards robot intentionality. This first answer can be developed when one makes a distinction between gestures we expect robots to make and gestures that surprise us. Such unexpected robot gestures may create doubt about the tenability of our descriptions of robots and prompt the adoption of alternative descriptions. In this chapter, I will argue that more fundamental approaches towards robot agency may in specific cases warrant RIG-descriptions for unexpected gestures where Dennett’s framework does not. For giving this argument later in this chapter I now turn to the description of unexpected robot gestures.

86

P. Vermaas

A characterisation of expected robot gestures depends on the type of description that is adopted: on DIG-descriptions expected gestures are the ones we predict robots will make on the basis of our assumptions about the designers of the robots; on RIG-descriptions expected gestures are the ones we predict with our assumptions about the robots’ goals and intentions. Gestures of robots that we do not expect in our descriptions are then unexpected robot gestures. The occurrence of unexpected gestures is likely, simply because technical artefacts typically engage in unexpected behaviour. Suspension bridges may start oscillating in the wind and route-planning software can guide you over surprisingly small local roads during long-distance trips. Robots designed for displaying gestures may similarly display unexpected behaviour and some of that behaviour may become interpreted as gesturing. Consider, as a linguistic case, Amazon’s personal assistant Alexa that in 2018 was said to engage in random laughing. There are three ways to understand such unexpected gesturing of robots.1 First, unexpected gesturing can be evidence that a robot is failing, calling for repair, as in the case of Alexa, or maybe even for scrapping. In DIG-descriptions failure gestures indicate that a robot is operating outside of its design specifications and is thus malfunctioning. In RIG-descriptions failure gestures are signs to doubt the rationality of a robot; it is cognitively derailing or running wild. Second, unexpected gesturing by robots can be taken as after all intended, for instance, when they warn users and others about unwanted situations. When storing your newly bought groceries, the light of the fridge may start flickering, not giving evidence that the lightbulb is getting worn out but indicating that the door is open too long for maintaining the inside temperature. And robots may be designed to surprise you through their gesturing, say for inviting you to further explore its uses (e.g., Rozendaal et al., 2019). In DIG-descriptions unexpected yet intended gestures can be taken as designed in the robot by the designers. In RIG-descriptions they can be taken as acts of the robot. Third, unexpected gestures of robots can be taken as emergent behaviour or acts. For digital games, unexpected emergent behaviour is known as glitches and this behaviour can become part of gaming, with decision processes to decide if in the design of the next version of the game specific glitches will be blocked or taken aboard as desired (e.g., Goriunova & Shulgin, 2008). In DIG-descriptions emergent gestures can be approached as new behaviour the robot was not designed for, or, interesting to the question taken up in this chapter, as a first intentional act of the robot. In RIG-descriptions emergent gestures may be taken as novel intentional acts of the robot. When returning to the main question, it can be observed that the incorporation of unexpected gestures in descriptions of robots gives a new possibility for describing robot gestures as related to their own intentionality. The incorporation of emergent gestures defines a possibility to stop with attempting to give DIG-descriptions of these gestures and switch to RIG-descriptions. The robot can then be described as

 In Sect. 5 I give a more elaborate analysis; here I present some first observations.

1

Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances

87

having become an intentional agent and the emergent gesture as one of its first acts. On more fundamental approaches towards agency, this switch becomes tenable if it is established that the robot concerned has indeed acquired agency. On Dennett’s approach this switch is tenable if the resulting description is efficiency in getting accurate predictions. For assessing this efficiency, I introduce in the next section Dennett’s framework of describing entities by stances and generalise it for applying it to robots.

3 Dennett’s Stances Applied to Entities and Their Genesis In Daniel Dennett’s framework there are three stances to describe and predict the behaviour of entities: the physical stance, the design stance and the intentional stance.2 When we use the physical stance for an entity, we consider the physical state of the entity and determine its behaviour by applying the laws of nature. (Dennett, 1978, p. 4) With the design stance we in contrast predict the behaviour of an entity by considering it as having a purpose and consisting of parts with functions relative to and in support of this purpose (1978, p.  4). Taking the entity as having a purpose may be related to an assumption that the entity was designed by humans for that purpose, as exemplified by the way Dennett advances a radio set as an entity to which we apply the design stance. Yet this assumption is not a necessary ingredient of the design stance, since Dennett also advances the use of this stance for describing the behaviour of biological organisms. The third stance is the intentional stance in which we predict what an entity will do by appeal to the assumption that it is a rational agent with certain overarching goals and certain perceptual and behavioural capacities. By that stance we “figure out what desires [the rational agent] ought to have, on the same considerations, and […] predict that this rational agent will act to further its goals in the light of its beliefs.” (Dennett, 1987, p. 17) The physical stance seems to be the one that can always be used for predicting the behaviour of an entity. But it is also the one that requires the most efforts to arrive at predictions. For a chemical substance or a spoon, it may be practically feasible but for physically more complex entities such as radio sets, elephants, smartphones or human beings, application of the physical stance becomes quickly too convoluted. For these latter entities it is more efficient to adopt the design stance or the intentional stance. The choice which of the three stances to adopt for describing an entity is in Dennett’s framework primarily a pragmatic and epistemic choice, and not one based on a pre-established position that the entity involved is, from an ontological perspective, actually a designed entity with a purpose and functional parts, or actually a rational agent with intentionality. Since for Dennett the choice to adopt one of the stances does not presuppose a pre-established position about the actual nature of the entity, the design stance and

 Dennett (1978, 1987, 1990). This section draws on (Vermaas et al., 2013).

2

88

P. Vermaas

the intentional stance can be applied to all sorts of entities. The design stance can be applied to radio sets and robots but also to biological organisms such as animals and plants. And the intentional stance can be applied to humans but also to animals, computers, and robots. A successful application of the design stance or of the intentional stance to an entity — where success means that the generated predictions are accurate and efficiently obtained — shows for Dennett that the entity can be taken as, respectively, a designed entity or an agent (in Dennett’s sense). Using Dennett’s stance framework in this way allows to describe a robot as a rational agent independently of a determination that robots are rational agents. Dennett’s framework, more generally, allows a more open approach to ontologically liberal positions about entities, such as advancing biological organisms as being designed (Dennett, 1990, 187, himself frequently talks about such organisms as being designed by “mother nature”), taking specific animal species as intentional and moral agents (e.g., De Waal, 2016), and approaching robots as rational agents. Table  1 gives the three stances and lists exemplar types of entities described by each stance. In Vermaas et al. (2013) it is argued that the design stance harbours an ambiguity. In Dennett’s application of this stance to technical artefacts, he regularly includes an application of the intentional stance to humans: the purposes and functions ascribed to the artefacts and their parts are typically obtained from the intentions ascribed to the creators and designers of the artefacts. In applications of the design stance to biological organisms the intentional stance may be included, for instance, by introducing a fictitious agent designated as “mother nature,” but in general this inclusion is not done. Descriptions of biological organisms with the design stance may just involve assumptions that the organisms have survival or reproduction as their purposes and that their parts have functions that support these purposes. This ambiguity in applying the design stance is moreover relevant in Dennett’s framework since it affects the accuracy of the resulting descriptions. A reference to a designing agent in applications of the design stance to biological organisms may be taken as metaphorical and not adding accuracy. Inclusion of the intentions of designers in the application of the design stance to technical artefacts does add accuracy of predictions: a radio set designed in the 1930s is more precisely described when knowing how in the 1930s engineers designed radio sets, and this description is quite different to a description of a radio set designed in the 2010s. For removing this ambiguity, it was proposed in Vermaas et al. (2013) to split the design stance in two separate stances: a teleological design stance for capturing how biological organisms are typically described, and an intentional designer stance for doing the same for technical artefacts. In this chapter, I bring this disambiguation one step further by proposing a generalisation of Dennett’s framework in which two stances are combined in describing Table 1  Dennett’s three stances and the exemplar entities they describe Entities described by the physical stance natural and synthetic chemical substances

Entities taken as having purposes, described by the design stance plants, animals, products, machines, computers, robots

Entities taken as rational agents, described by the intentional stance humans, some species of animals? (future) robots?

Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances

89

the behaviour of entities: one of the three stances is applied to describe the entity itself and one of the three stances is applied to describe the genesis of the entity. The intentional designer stance defined in Vermaas et al. (2013) for technical artefacts, for instance, is in this generalisation taken as creating descriptions in which the design stance is applied to the technical artefact and the intentional stance is applied to the designing of the artefact. Descriptions of biological organisms can become descriptions in which the design stance is applied to the organism and the design stance is also applied to the evolutionary process that led to the existence of the organism. And, relevant to the exploration of robot gestures: descriptions of robots can be ones in which the design stance or the intentional stance is applied to the robot, and the intentional stance is applied to the design process that led to the robot. In this generalisation the choice to use a specific single stance or a specific pair of stances for describing an entity is again guided by the epistemic criteria of accuracy of the predictions obtained and the efficiency by which these predictions are obtained. Applying the intentional stance to the robot alone is efficient and preferable if it leads to accurate predictions of the robot’s behaviour. When this accuracy is lacking and information about the design of the robot is needed for that accuracy, then the description of the robot is better composed of an application of the intentional stance to the robot and an application of that same stance to its designers. This generalisation of Dennett’s framework creates nine possibilities to apply the three stances to the description of entities (see Table 2). I have given each combined stance a name (and dropped the name ‘intentional designer stance’ introduced in (Vermaas et al., 2013)). For seven combined stances there exist exemplar types of entities for which they are used, and I listed a few in Table 2. And for some combined stances there are types of entities for which application is explored but not generally accepted (and from a more ontological point of view possibly Table 2  Nine pairs of combined stances and the exemplar entities they describe Entities described by the physical stance Entities taken as having inorganic physical inorganic geneses, stance described by the physical stance natural chemical substances Entities taken as having evolution physical evolutionary geneses, stance described by the design stance Entities taken as being created, described by the intentional stance

creation physical stance synthetic chemical substances

Entities taken as having purposes, described by the design stance inorganic design stance

Entities taken as rational agents, described by the intentional stance inorganic intentional stance

geological and weather systems evolution design stance

evolution intentional stance

plants, animals creation design stance products, machines, computers, robots, gmo’s

humans, some species of animals? creation intentional stance (future) robots?

90

P. Vermaas

controversial). For instance, animals are typically described by the evolution design stance (design stance applied to both the animal and its genesis) and for some animals the evolution intentional stance is advanced (intentional stance for the animal; design stance for its genesis). And robots are described by the creation design stance (design stance for the robot; intentional stance for its design) and the question of whether robots can display gestures that are to be described as due to their own intentionally is in part a question of whether robots can be described by the creation intentional stance (intentional stance for both the robot and its genesis).

4 Gestures in the Generalised Stance Framework Within this generalised stance framework gestures of entities can be approached as physical properties and behaviour of entities to which observers attach meaning. Descriptions of gestures involve then, in addition to applying a stance to the entity and possibly a stance to their genesis, also the use of the intentional stance to the observers involved. I will not make this third application explicit but use the observers’ own intentionality to single out a first type of gestures. Other types will be identified using Table 2. A first and generic type of gestures concerns specific physical properties or behaviour of entities that an observer takes as meaningful, for instance, because these properties and behaviour reveal that the entities can afford for (or threaten) interests — survival, spirituality, curiosity, etc. — of the observer. Such physical properties and behaviour of the entities become gestures and get their meaning through the goals of the observer, not by some features intrinsic to the entities (as may be the case in Gibson’s (1979) account of affordances). Say, when I notice vapour coming from a fluid, I am warned for its high temperature. When the meaning of gestures is analysed in terms of their functions,3 then for this first type the meaning/function of the gesture resides in the intentions of the observer, following, e.g., Searle’s (1995) intentionalist account of functions. A second type of gestures can be discerned when the design stance is applied to an entity taken as an entity with a purpose (i.e., in descriptions created by applying the single design stance or the combined stances as given in the second column of Table  2: the inorganic design, evolution design and creation design stances). Physical properties or behaviour of parts of the described entities have then functions within the descriptions relative to a posited purpose of the entity, and some of them can be taken as revealing the processes that take place in the entity. Say, when I hear the tires of the car slipping, I take it as signalling that the tires are losing their grip. When the meaning of gestures is analysed in terms of functions, then for this second type the meaning/function of the gestures resides in the entity, following Cummins’ (1975) causal-role account of functions.

 The analyses of gestures in terms of functions draws from Houkes and Vermaas (2010).

3

Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances

91

A third type of gestures emerges when the design stance is applied to the genesis of the entity concerned (i.e., in descriptions created by the combined stances as given in the second row of Table 2: the evolution physical, the evolution design and the evolution intentional stances). Physical properties or behaviour of the described entities can then have functions relative to their evolutionary selection and some of these properties and behaviour become gesturing in a stricter sense of the concept. Specifically, animals display all types of behaviour that by their selection history have the function to attract, warn or lure other organisms. The roar of the lion can be described as a gesture to other (male) lions to take distance and for us humans and other species to take cover. The etiological theory of functions and language (e.g., Millikan, 1984, 1993) can be a source to detail the meaning of gestures of this third type. The fourth and fifth types of functions bring robot gestures into focus. The fourth type becomes available when the intentional stance is applied to the entity taken as a rational agent: in descriptions created by the single intentional stance or the combined stances from the third column of Table 2 (the inorganic intentional, the evolution intentional and the creation intentional stances). The physical properties or behaviour of the agent can then be described as having meaning due to its intentionality. I can wave to you, and you can describe this as that I intend to tell you that lunch is ready. The meaning of this gesture resides into the intentionality of the gesturing agent and when analysed in terms of functions, Searle’s (1995) intentionalist account of functions is again the relevant one but now applied to the agent described. A fifth type of gestures emerges when the intentional stance is applied to the genesis of the entity concerned, i.e., in descriptions created by the combined stances as given in the third row of Table 2 (the creation physical, the creation design and the creation intentional stances). Physical properties or behaviour of the described entities can now be taken as having meaning due to the intentionality of the agent that created or designed the entity. The button in the elevator that I push to choose the floor I want to go to is designed to light up to signal to me that I pushed it successfully. The meaning of this gesture resides into the intentionality of the designer and when analysed in terms of functions, Searle’s (1995) intentionalist account of functions is another time the relevant one but now applied to the designer. This digression into Dennett’s stance framework will help in further analysing the description of unexpected robot gestures, and in giving the argument that more fundamental approaches towards robot agency may warrant RIG-descriptions over DIG-descriptions in cases where Dennett’s framework does not do so.

5 Unexpected Robot Gestures and Emerging Robot Agency DIG-descriptions relate robot gestures to the intentions of the robot’s designers, and within Dennett’s framework that means that the combined creation design stance is applied to the robot and its design (just applying the design stance to the robot does

92

P. Vermaas

not enable references to the designers of the robot). In DIG-descriptions, robot gestures are thus of type 5. RIG-descriptions are in turn generated by applying the (single) intentional stance to the robot or by applying the combined creation intentional stance to the robot and its design. Robot gestures are thus in RIG-­descriptions taken as of type 4. In Sect. 2 unexpected robot gestures were divided in gestures of failure of the robot, in gestures that were after all intended, and in emerging gestures. The description of these unexpected gestures with Dennett’s frames is more complex and is the topic of this last section. For each of these unexpected gestures I analyse how they affect DIG and RIG-descriptions. When one uses DIG-descriptions, a failure gesture of a robot shows that these descriptions are not accurate anymore. For regaining that accuracy, one has, broadly seen, three main options: • Improve the DIG-description by, for instance, reconsidering the intentions one assumes the designers had. If this improvement is possible and leads to accurate descriptions, the failure gesture becomes an expected gesture of the robot and is of type 5. • Abandon the DIG-description, take the robot as an entity with a purpose, and shift to the design stance. If the resulting description predicts the failure gesture, it is a gesture of type 2. • Abandon the DIG-description, shift focus to the hardware and software of the robot, and explain the failure gesture using the laws of nature. This means that the physical stance is adopted, and that the failure gesture is of type 1: observers assign meaning to it that does not originate in the intentions of the robot designers. Which of these three options is preferable depends on accuracy of the generated description and the efficiency by which these descriptions are generated. The same remark holds below when more than one option is available to respond to an unexpected robot gesture. When one uses RIG-descriptions, a failure gesture of a robot shows similarly that these descriptions are not accurate anymore. Accuracy can be regained by: • Improve the RIG-description by, for instance, reconsidering the intentions one assumes the robot has. If this adjustment is possible, the failure gesture is of type 4. • Abandon the RIG-description, take the robots as a designed entity, and analyse the failure gesture in terms of the intentions of the designers. This means applying the creation design stance and shifting to a DIG-description. If that is successful, the failure gesture is of type 5. • Abandon the RIG-description, take the robots as an entity with a purpose, and shift to the design stance. If the resulting description predicts the failure gesture well, it is a gesture of type 2. • Abandon the RIG-description and shift focus to the hardware and software of the robot and explain the failure gesture with the physical stance. The failure gesture is then of type 1: observers assign meaning to it.

Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances

93

Unexpected gestures that were intended after all are in Dennett’s framework not provoking shifts in the stances applied. When one uses DIG-descriptions, the unexpected yet intended gesture is discovered to also be intended by the designers; it is of type 5. When one uses RIG-descriptions, the unexpected yet intended gesture is discovered to also be intended by the robot; it is of type 4. Describing unexpected emerging gestures leads to more options in Dennett’s framework. When one uses DIG-descriptions, an emerging gesture demonstrates that accuracy is lost. For regaining it one has multiple options: • Improve the DIG-description by, for instance, reconsidering the intentions one assumes the designers had. If this adjustment is possible, the emerging gesture is of type 5. • Assume that the robot has acquired agency and explain the emerging gesture as related to the beliefs and purposes of the robot as a new agent. If this approach is successful, one has abandoned the DIG-description and adopted a RIG-­ description, making the emerging gesture of type 4. • Abandon the DIG-description, take the robots as an entity with a purpose, and shift to the design stance or the creation design stance. If the resulting description is accurate, the emerging gesture is of type 2. • Abandon the DIG-description and consider the hardware and software of the robot to explain the emerging gesture. This means that the physical stance is adopted, and that the emerging gesture is of type 1. When one uses RIG-descriptions, the emerging gesture shows also that these descriptions are not accurate anymore. It can be regained by: • Improve the RIG-description by, for instance, reconsidering the intentions one assumes the robot has. The emerging gesture is then of type 4. • Abandon the RIG-description, take the robots as a designed entity, and analyse the emerging gesture with the intentions of the designers. This is a shift to a DIG-­ description and when successful makes the emerging gesture of type 5. • Abandon the RIG-description and apply the design stance. If this works, the emerging gesture is of type 2. • Abandon the DIG-description and explain the failure gesture with the physical stance. The failure gesture is then of type 1. Two conclusions can be drawn from this exercise. The first is that in Dennett’s approach unexpected gestures of robots may prompt switching between DIG and RIG-descriptions. Interesting for the main question taken up in this chapter is that when a robot displays an unexpected emergent gesture, one has the option to abandon DIG-descriptions in favour of RIG-descriptions. The second conclusion is that within Dennett’s framework one can have different options to respond to unexpected gestures. The possibility of indeed abandoning DIG-descriptions in favour of RIG-descriptions in case of an emergent gesture is thus competing with three alternatives. Which of these four options is preferred in Dennett’s framework depends, as said, on the accuracy of the descriptions these options generate and the efficiency by which these descriptions are generated. Hence, if an emergent gesture occurs, it

94

P. Vermaas

is not evident that one has to assume that the robot is now a rational agent and switch to a RIG-description; one can, for instance, also improve the DIG-description by reconsidering the intentions of the robot designers, and that option may create an equally accurate description of the gesture but more efficiently by not introducing agentivity of the robot. This means that more fundamental approaches towards agency may in case of unexpected emergent gestures, more easily abandon DIG-descriptions in favour of RIG-descriptions. In such approaches, the occurrence of unexpected emergent gestures may be reason to (re)investigate if the robot concerned has (acquired) agency. If that answer is positive, then these approaches can start advancing RIG-descriptions for that robot. In Dennett’s framework these RIG-descriptions may, however, be not warranted, since for Dennett only accuracy and efficiency count for taking an entity as one that can be described by the intentional stance. Hence, on the more fundamental approaches, it can be discovered that a robot has acquired agency, while on Dennett’s approach it should be taken as a designed system, in which case the former approaches warrant RIG-descriptions of robot gestures and Dennett’s framework does not.

6 Conclusion In this chapter I have explored the question of whether robots can make gestures that can be described as related to the intentionality of the robots themselves and not to the intentionality of their designers. For this exploration I have approached robots as designed entities and introduced a generalisation of Dennett’s stances framework in which not only single stances can be used to describe robots and entities generally, but also pairs of stances where one is applied to the robot and one to its designers. A first conclusion is that for gestures that we expect robots to make the question of whether we can relate them in descriptions to intentions of robots is rather straightforwardly answered within Dennett’s framework: expected robot gestures can be described with robot intentionality when these descriptions are more accurate and efficient as compared to descriptions which relate robot gestures to the intentions of their designers. For descriptions of unexpected robot gestures, the answer to this question is more complex and spelled out in Sect. 5. A second conclusion concerns a coarse comparison of the acceptability of descriptions of gestures in terms of robot intentionality in Dennett’s framework and in more fundamental approaches to agency. More fundamental approaches warrant such descriptions if it is determined that robots are agents, and Dennett’s framework warrants them if they are accurate and efficient descriptions. I defended that Dennett allows more often for descriptions of robot gestures in terms of robot intentionality than the more fundamental approaches. In the case of unexcepted robot gestures, this comparison may lead to a reverse conclusion. It was argued that unexpected emergent gestures by a robot may be

Describing Robot Gestures by Design and Agency: An Exploration with Dennett’s Stances

95

taken as reason to assume that the robot has acquired agency. If on the more fundamental approaches to agency it is ascertained that the robot indeed has that agency, then these approaches warrant to advance descriptions of the emergent gestures in terms of the newly acquired robot intentionality. Yet in Dennett’s framework these descriptions are not warranted when they are not in an efficient manner leading to accurate descriptions of the gestures. Hence, it may happen that more fundamental approaches to agency describe robot gestures with robot intentionality, while on Dennett’s approach one still has to take the robot as merely an entity designed for a purpose.

References Cummins, R. (1975). Functional analysis. Journal of Philosophy, 72, 741–765. De Waal, F. (2016). Are we smart enough to know how smart animals are? W.W.  Norton & Company. Dennett, D. C. (1978). Brainstorms: Philosophical essays on mind and psychology. Bradford Books. Dennett, D. C. (1987). The intentional stance. MIT Press. Dennett, D.  C. (1990). The interpretation of texts, people and other artefacts. Philosophy and Phenomenological Research, 50S, 177–194. Gibson, J. J. (1979). The Ecological Approach to Visual Perception. Houghton Mifflin. Goriunova, O., & Shulgin, A. (2008). Glitch. In M.  Fuller (Ed.), Software studies: a lexicon (pp. 110–119). MIT Press. Houkes, W., & Vermaas, P. E. (2010). Technical Functions: On the Use and Design of Artefacts. Vol. 1 of Philosophy of Engineering and Technology. Springer. Millikan, R. G. (1984). Language, Thought, and Other Biological Categories: New Foundations for Realism. MIT Press. Millikan, R. G. (1993). White Queen Psychology and Other Essays for Alice. MIT Press. Norman, D. (1990). The Design of Everyday Things. Perseus. Rozendaal, M. C., Boon, B., & Kapterinin, V. (2019). Objects with Intent: Designing Everyday Things as Collaborative Partners, ACM Transactions on Computer-Human. Interactions, 28, article 26. Searle, J. R. (1995). The Construction of Social Reality. Free Press. Vermaas, P. E., Carrara, M., Borgo, S., & Garbacz, P. (2013). The Design Stance and Its Artefacts. Synthese, 190, 1131–1152.

The Philosophy of Gesture and Technological Artefacts Giovanni Maddalena

Abstract  Western philosophy often overlooked the problem of technology. It is a long-standing prejudice of a nominalist or idealist mentality that goes on from ancient Greece up to today. Compared to other contemporary currents of thought, the pragmatist tradition had some interesting ideas because it united in a profound continuum theory and practice, overthrowing any dualism. This move was particularly effective in Peirce’s studies on continuity, logical modalities, logic of abduction, and existential graphs as well as in Dewey’s approach to logic as an integrated tool of inquiry in which technology is part of the human organism. However, also the pragmatist tradition remained somehow only with a good analysis of synthesis and a good theory of practice but did not complete its drive toward a different vision of syntheticity and technology. The philosophy of gesture that I will propose in this paper wants to complete the unfinished work of this tradition relying heavily on pragmatist phenomenology and semiotics. In the first part, I will sum up the main structure of this philosophy of gesture, as I name it (I). In the second part, I will show the fundamental application of this philosophy to technology (II). At the end, I will illustrate an application for blind people to see a museum that is inspired by the Philosophy of gesture (III). Keywords  Technology · Pragmatism · Gesture · Semiotics · Phenomenology · Synthesis Western philosophy has often overlooked the problem of technology. This is the long-standing prejudice of a mentality that began in ancient Greece and continues up to the present day. In the classical thought of Plato and Aristotle, technology is severed from episteme; certainly, the separation is sometimes blurred, but it is theoretically grounded and firm. Stoicism and Hellenistic philosophies in general strived G. Maddalena (*) University of Molise, Campobasso, Italy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_7

97

98

G. Maddalena

for a deeper unity, but even in these, the separation held its ground (Parry, 2014). This problem intensified with modernity after the Cartesian revolution, which hardened dualisms such as mind vs. body, reasoning vs. doing, and mathematical evidence vs. common sense vagueness. Thus, this rigid distinction between theoretical and practical mental attitudes descends from the Cartesian model of reasoning. Needless to say, technology is a subset of the practical (Mitcham, 1994: 116–134). The Kantian turn and its idealist and positivist successors further worsened the problem — everything happens in our mind, whether “mind” is the critical capacity of humankind (as seen in the Enlightenment), the expression of the spirit (as seen in idealism), or the vessel of science (as seen in positivism). Severed from theoretical knowledge, technology became an applied or lower kind of science. Over the last two centuries, when a thinker rebelled against this paradigm, he/she often had to espouse some sort of irrationalism — Nietzsche, Heidegger, and post-modern philosophy are examples. Under the paradigm, technique becomes a form of humankind’s equivocal and often inauthentic mastering of Being. Even in the Marxist tradition, technique was not able to escape this fate: Marxism subverted the traditional relationship between theory and practice, giving priority to the latter, but they remained separate. In this conception, theory becomes a superstructure of a practical orientation towards the world (ibid.: 78–84). There are several exceptions to this schema, including the thought of E. Kapp, F. Dessauer, and others, who proposed a different and more comprehensive approach (ibid. 19–38). However, these thinkers remained marginal characters in the philosophical, academic debate. Against this background of separation between theory and technology, a separation that was not mitigated by analytic philosophy or hermeneutics, the pragmatist tradition deserves more sustained attention (ibid.: 69–81). In fact, the philosophy of classic pragmatists united theory and practice in a profound continuum, overthrowing any dualism. This movement towards unity was particularly fruitful in Peirce’s studies on continuity, logical modalities, the logic of abduction, and existential graphs, as well as in Dewey’s approach to logic as an integrated tool of inquiry in which technology is part of the human organism (Hickman, 1990). However, even though the pragmatist tradition featured a strong analytical synthesis and a useful theory of practice and technology, it did not offer a complete vision of syntheticity, which is necessary for a proper understanding of technology. It is my intention in this paper to complete the unfinished work of this tradition by proposing a reformulation of classic pragmatism in what I call the “philosophy of gesture.” In Part I, I summarize the main structural elements of the philosophy of gesture. In Part II, I show the fundamental contribution that this philosophy can make to the philosophy of technology. Finally, in Part III, I describe an application inspired by the philosophy of gesture that enables blind people to view the contents of a museum.

The Philosophy of Gesture and Technological Artefacts

99

1 A Short Overview of the Philosophy of Gesture The central idea of the philosophy of gesture is that our doing is a kind of reasoning, a synthetic form of rationality. Some of our actions, including the techne we discuss below, are ways in which we perform synthesis; that is, the acquisition of something new and not conceptual in our conceptual framework. Kantian thought contained a strong systematization of both synthetic and analytic reasoning. In the last century, the philosophical debate has blurred the borders of the Kantian distinction between these two kinds of reasoning (Rey, 2018). However, this debate has not radically changed our conception of synthesis because it did not challenge the Kantian paradigm. The pragmatist approach followed by the philosophy of gesture aims at addressing the theoretical basis of this paradigm. Contrary to how a rationalist, Kantian approach to reasoning conceives of synthesis, it is in our conception neither the reverse of analysis nor a kind of unfamiliar mechanism that relies upon analytic patterns, as is the case with Kantian a priori synthetic judgments. As Robert Hanna noted, these judgments rely on the same application of the analytic pattern between parts and the whole referred to in transcendental aesthetics rather than in the field of logic (Hanna, 2001; Maddalena, 2015: 30–42). The pragmatist tradition allows us to think about synthesis in a different way by relying on the continuity of reality and knowledge and on the mediated nature and intrinsic fallibility of knowledge (Calcaterra et al., 2015). This different understanding originates above all in Peirce’s work on existential graphs, which are a way to elaborate precise reasoning using diagrams that are both a representation of thought in action and the result of this thought; that is, they represent both the process of recognizing an identity and the identity itself. In this way, Peirce affirms that synthesis follows this identity in its becoming. As Peirce said, the reasoning that brings us to the synthetic identity A  =  B is more original than the A  =  A, flat, analytic identity, which is only a derivative case (NEM 4: 328). In this reformulation of classic pragmatism, synthesis is defined as an original kind of reasoning: a recognition of identity through change (Maddalena, 2015; Maddalena & Zalamea, 2012). The definition of synthesis needs some further explanation. Reasoning synthetically through change happens by understanding change as a Peircean mathematical continuum (Zalamea, 2012) and by actions occurring within this continuum of change. Peirce’s conception of continuum is non-metric and can now be understood by means of topological tools (Vargas, 2015). Peirce’s continuum has a number of properties, including reflexivity, transitivity, modality, and generality. Without entering into an overly specific discourse about these properties, it is important to note that they enable a keen comprehension of change and the possibility of reading different phenomena as continua. Peirce himself defined some of the diagrams contained in his existential graphs as continua operating upon the continuum of the sheet of assertion; that is, on the universe established by the graphist. Synthesis as recognition of identity can occur because of a shared continuity that allows us to understand different phenomena as participating in the same reality.

100

G. Maddalena

Diagrams or graphs perform the synthesis, which is conceived of as a recognition of identity through change. These diagrams or graphs constitute both actions and reasoning occurring at the same time. Following some important French philosophers of mathematics, we call them “gestures” (Cavaillès, 1994, 2008; Châtelet, 2010). How can we perform this synthesis in our everyday reasoning? Just as diagrams permit synthesis in Peirce’s existential graphs, everyday synthetic reasoning happens through actions having a precise structure. The structure itself derives from an attentive study of these existential graphs. I simply extend Peirce’s insights using the logic of existential graphs to any action that has the possibility of carrying on a meaning using the same phenomenological and semiotic structures. I call “complete gestures” those habits of action that carry on a meaning, that synthetize our experience happening in the context of continuous change. “Complete gestures” have a phenomenological and semiotic structure in which all categories (firstness, secondness, and thirdness) and kinds of signs (icons, indices, and symbols) are involved in a dense way. I will be clearer about these categories and signs later on. But now we need to describe another characteristic of gestures: they help determine continuous reality by means of a transition among modalities, to the extent this is allowed by their continuous nature. Gestures determine an initial vague experience into a determinate action that yields a general meaning. In brief, in the most significant situations of our intellectual life, we understand something new by making clear an initially vague experience through the doing of something in a certain way. We measure, we draw, we touch, we talk; in other words, we experiment through meaningful actions. When these actions are developed and ordered to carry on a meaning, we can clarify the initial vague experience, transforming this experience by means of a determinate set of actions in order to reach a general, replicable, meaningful conclusion. For example, in a public oath or a scientific experiment, or an artwork, a vague idea becomes clear; that is, it achieves a general, recognizable meaning by the performance of a particular series of actions according to a specific formula. The oath of the president of the US allows us to recognize in one “complete gesture” the meaning of otherwise vague feelings related to tradition, power, responsibility, etc. A formal set of actions transforms these vague feelings into the conclusion that a particular person is in power at the moment and that he/she intends to incorporate into the exercise of his/her power values that become clear to a witness to that event; in sum, tradition, power, responsibility, etc., attain to a universal meaning. In the same way, we need a scientific experiment in order to formulate a scientific law. Think of the Rutherford golden foil experiment used to understand the structure of atoms. This experiment is, like the public oath, a complete gesture — the vague initial hypothesis aimed at falsifying the pudding theory of the nature of atoms becomes a determinate action that yields a different, general conclusion about its subject. A painting or an artwork is another example in which we physically model a certain element in order to arrive at a meaning (obviously, the lack of meaning is a meaning as well). Any artist knows about a vague initial experience of reality that becomes more and more clear through the process of writing/painting/sculpting/

The Philosophy of Gesture and Technological Artefacts

101

acting. The process of doing these actions leads to the artwork achieving a general and replicable meaning. Our life is full of synthetic habits of action, and the history of evolution suggests that human beings are in some sense defined by their capacity for understanding by doing, which has been true from the time of the paleolithic drawings in the Lascaux caves to our present-day smartphone applications. When we create something, we are making comprehensible vague experiences by means of a kind of reasoning: what is going on in our doing is a synthesis, a knowing of something new. The epistemic principles that are at work in these gestures were discovered by Peirce: a capacity to operate in continuity, a continuous transition among modalities, a clarification of the vague, and a dense blending of all types of signs and phenomena. Now it is necessary to clarify these phenomenological and semiotic principles, which must be blended in a dense way in order to achieve synthetic power. Let’s start with phenomenology. Peirce defined three main kinds of categories that constitute different types of phenomena, calling them firstness, secondness, and thirdness. Dense blending involves a pure idea or a pure feeling (firstness), a physical act that involves a reaction between two objects (secondness) and a general or universal habit of action (thirdness). From the semiotic point of view, Peirce identified 59,046 types of signs. However, a good first account of gesture can be given by using the main triad of signs that describes the relationship between the representative function of a sign and the real object it refers to. This triad is made up of icons, indices, and symbols, which signal relationships with an object by similarity (icons), by mere or brute contiguity (indexes), and by interpretation (symbols). Following this insight, it would be appropriate to call a “complete gesture” a gesture that has all the semiotic elements blended together proportionally or densely. “Dense” is how I translate Peirce’s “equality” of non-metric blending. For a gesture to be complete, it must be a general law (a general meaning) or a symbol that can generate replicas; it is actual when it indicates its singular object (index) and expresses different possibilities of form and feelings (icon). The three semiotic characteristics describe what a complete gesture is and should be: creative because of various possible forms and feelings, singular in its individuality, and recognizable for its unity and conformity to an established pattern that the gesture itself tends to realize. Clear examples of complete gestures are liturgies in every religion, public and private rites, public and private actions that establish an identity, and, of course, artistic performances and experiments testing a hypothesis. In these types of complete gestures, we observe the general structure of complete gestures and the pattern of syntheticity as a recognition of identity through change. When a habit of action has all of these characteristics, we are presented with a complete gesture. Of course, sometimes gestures are simply reactions, projections, modeling, mere information, or indication. Elsewhere I have created a long list of incomplete gestures that serve to facilitate an understanding, by contrast, of the complete gesture (Maddalena, 2015: 74–81). For our present purposes, it is sufficient to say that completion means “fully able to synthetize”; there is less power of synthesis in incomplete gestures. In any event, though, synthesis happens by doing

102

G. Maddalena

something that provides determination and generality to something that is vague and indeterminate.

2 Technology as Gesture Once one has understood that our meaningful actions, that is, our gestures, are part of our reasoning, the topic of artifacts acquires a particular relevance. On a historical level, both old and new discourses about anthropology make more sense in light of this understanding. Yves Coppens’ interpretation of homo abilis is that it is a passage to humanity by means of gestures (e.g., crafting knives from stones), religious expressions (e.g., burying the dead), and the capacity for speech — and the unification of these activities — and that it is at the same time the beginning of a new logic (Coppens, 2010). Indeed, human beings are rational animals not because they have learned to speak, but because of the gestuality of their activities, which are synthetic forms of reasoning, of which the capacity for language is a derivative case. Michael Tomasello’s experiments on infants show — much more than Tomasello acknowledges  — that gestures and intentionality are identical things for human babies, and that there is a huge difference between human beings and other beings. This difference is technological but not in the sense of technology as a mere practical application. Baby apes can use technology but do not use it in the same way as human babies. They use it to achieve their goals, while human babies use it to understand intention. In this view, technology is a human gesture; that is, one of the means we employ to understand reality, not only to operate in it (Tomasello, 1999). On a theoretical level, there are two different aspects of technology as a human gesture that can be considered. The first is the internal reasoning of this gesture. The second is the specificity of actual technology (e.g., electronic technology), which highlights the unification of comprehension and communication that occurs in our use of technology. Let us start from the first  — the technological gesture in general. In order to synthesize their vague experience, that is, in order to transform experience into something meaningful, human beings have always used tools. All tools, from the primitive knife to the selfie stick, are physical objects, offering a certain resistance to the human body, and are subject to the laws of contradiction and the excluded middle. Peirce called this kind of phenomenon a “second,” since it implies the double nature of effort and resistance and called the relevant phenomenological category “secondness.” We use tools in a habitual way to achieve a purpose. The ubiquity of habits of action aimed at some purpose is the essence of what Peirce called “thirdness”; it refers to a phenomenon that is real even though it escapes our senses. It is real insofar as it changes something in our reality. For example, the so-called laws of nature are habits of action. Carving a stone to make it sharp is a habit of action because it is something that we learn we can always do in the same way. Finally, in the tools we use habitually we find the embodiment of our way of moving vaguely within an experience. The experience of a vague feeling is what Peirce

The Philosophy of Gesture and Technological Artefacts

103

called “firstness”; examples are needing to kill an animal to survive that causes us to craft stones or the very sophisticated need to take a picture of ourselves in a certain well-controlled environment using a selfie stick. Any technological artifact must include all three of these phenomenological categories. On a semiotic level, the level that involves meaning, we want our tools to have a certain form (icon) and name or indication of function (index) so that we can distinguish between them, and we want them to have a recognizable aim or goal (symbol). The sort of interpretation that we engage in with respect to our tools is oriented towards a symbolic level of meaning. Again, these three signs need to be united for synthetic reasoning, which is a kind of action that I call a “complete gesture” to be possible. Technological tools that are incomplete gestures are also possible: sometimes tools are not adequate to their aim because of errors in their design or because they are broken. Sometimes we can use a broken tool for another aim, transforming it into a different tool. Or, we might need a tool to be broken for the purpose of displaying it in a museum, transforming it into an artwork like Duchamp’s urinal-­ fountain. The normal use of a technological tool, though, is effective; that is, it performs the synthetic reasoning for which it was created. Let us say that technology works when all of these phenomenological and semiotic levels are working together. In this situation, there is a profound continuity between our experience, our thought, our comprehension of reality, and our effectiveness. The primitive knife and the selfie stick work as extensions or embodiments of our synthetic reasoning. That is why we continually modify and improve our tools, always reasoning toward the same goal. It is worth noting that some further factors are at work in a complete gesture, and therefore in technology as gesture (Maddalena, 2015: 92–102). I cannot fully describe these here but offer the following summary: (1) Gestures have a normative dimension. A gesture involves an aesthetic appreciation of the reasonableness of its goal and the plausibility of its realization. These characteristics fall properly under the categories of logical aesthetics and ethics. With respect to technology, they involve the design and simplification of the elements, structure, and procedures of a particular technology; (2) Technological tools as complete gestures must preserve the narrative of their use and goals. Do we need a narrative for technology? Yes — every inventor knows that he/she must explain the need that a particular technology addresses. If a new piece of technology is really an advance over existing technology, it usually involves a different approach to a problem and its solution. This approach must be explained; even the realm of technology  — like the realms of mathematics and logic—involves a story; (3) In a complete gesture, the inventor and the user accomplish the same action and have interchangeable roles. Both must assent to the proposed action: when a user becomes skilled, he/she can give important feedback to the inventor, but above all, he/she experiences the same answer to the same needs through the same determination of his/her experience as the inventor. This is why some tools — think of smartphones among the most recent — are so successful in becoming part of the human experience; and (4) Like any complete gesture, technological tools require a form of teaching and learning that involves following a master in a way that is more similar to learning to ski or swim than to

104

G. Maddalena

listening to a lecture. Actions need to be performed over and over again to become habits and for the effectiveness of these habits to improve. However, a more sophisticated analysis is necessary in order to understand electronic tools, the characteristics of which in turn allow us to gain a deeper insight into the nature of the gesture of reasoning through technology. There are two layers of this analysis, with the first layer applying to any technological tool. From a phenomenological perspective, electronic devices have a hard body (secondness), which is the embodiment of a vague need or experience (firstness) and are replicable objects (thirdness). This is the common phenomenology of any technological tool. On the semiotic level, the algorithm at the core of any electronic product is an index, the architecture of the informatics aimed at reaching a certain goal is an icon, and the deductive/inductive logic of the algorithm and its ability to perform complete functions are symbols. This is the electronic version of semiotic blending, which is active in any technology. In fact, we find indexes for reference, iconic forms for novelty, and symbolic patterns for creating interpretative habits. This is true also for knives, chairs, and rockets. However, the specificity of the actual electronic revolution implies a second layer of analysis. While we tend to stop at the first layer, deducing our technological gesture and goal for any technological tool, from knives to selfie sticks to trains, cars, and rockets, our electronic devices imply a second degree of performance that introduces a second, virtual level of reality. Electronic tools do not solve our needs directly; instead, they introduce us to a new level of reality, second-grade reality, which intertwines with and maintains continuity with the first. The second level of reality, virtual reality, posits a world like the first, in which we engage in reasoning through our gestures. The 2.0 web introduced the possibility of performing new gestures on a second level of reality by duplicating the phenomenological and semiotic structure of the first. The nature of this second level has been carefully investigated by what is known as user experience. Our touch pads, which provide us with this user experience by means of the sense of touch, allow us to live a phenomenological experience of secondness. Thirdness is found in the habitual replicas involved in any gesture (think of the gesture of swiping to unlock the screen), while we find firstness in the newness that manifests itself in the need or curiosity to “open” pages or “start” programs. As for the semiotic structure of this second level of gestures, the meaning of the tool depends on its aim, which is often conveyed by rules or storytelling; indexicality plays an important role in names and tags, while the aesthetics of our tools are iconic. The analysis presented here is obviously not the main point but is interesting insofar as it shows that the success of our technology has led to a reformulation of the patterns of our processes of understanding, a reformulation that places our modern devices and artifacts within the long history of human tools and artifacts. It is also important to note how much this technology, which operates on this second level of reality, clarifies the structure of the synthetizing gestures that happen at the first level. Some important questions arise: Does the logic of synthesis change at this second level of reality? Does the synthesis at the second level of reality allow us to know something more than we know at the first level? The answer to the first question is no: the pattern of synthesis and gesture as an operative tool of synthesis does

The Philosophy of Gesture and Technological Artefacts

105

not change. We perform gestures at the second level; for example, when playing video games or moving around in augmented reality. The second level involves different and more varied contexts, but we are still operating through meaningful actions. The answer to the second question, which involves the kind of knowledge that electronic tools provide, is more complicated. The second level certainly allows an enormous increase in the extent of our knowledge, both in an analytic and a synthetic way. On this level, we know more information and we can also act in different contexts: think of surgeries that are remotely performed or the paths that we can follow and create in metadata in every field. We acquire more information, and we can also perform and create new kinds of relationships. Our gestures on social networks allow us to know other people—we are still in a represented reality, but with web 2.0, 3.0, and 4.0, this second reality has become closer to the first; on occasion, it mingles with the first, strengthening our knowledge of it. The interesting situation that has resulted from the Covid-19 pandemic illustrates the point because we are now teaching on digital platforms. Digital or smart teaching is not like in-person teaching — in virtual reality, secondness and its benefits (body language, feedback, etc.) are missing. However, it is clear that digital platforms have become a formidable second best, but only because the gestures that are being performed on them are also embedded within our first level of reality. Therefore, electronic tools can certainly enlarge the extension of our knowledge. Does this technology also offer an increase in intension? Can we arrive at more profound meanings with these sorts of tools? One possible way to inquire about such an opportunity is by examining technologies for people with disabilities. The need to restore human capabilities by means of technological tools becomes a way to explore whether technology can increase intension, not just extension.

3 An App Allowing Blind People to Enjoy Museums It is well-known that accessibility to museums for people with disabilities is a big problem for our communities. Museums are important public assets that should be accessible to these people. Our societies have enacted numerous laws to ensure that accessibility, but establishing a right of accessibility on a paper does not provide a concrete solution to the problem (Ferrucci, 2018). Together with some colleagues and the company Heritage s.r.l., which focuses on technological applications related to cultural heritage, we have conceived of this problem within the framework of the philosophy of gesture. It was our idea that people with disabilities — we started with blind people — need to have a synthetic experience of museums in order to enhance their understanding of what they contain. We decided to focus on the Samnite Museum in Campobasso (Italy) and conceived of a tool that would have all of the characteristics of the double phenomenological and semiotic structure that I described in the previous section. The tool accomplishes its goal by means of three features: an

106

G. Maddalena

application for mobile devices that is activated by the initial touch of the user and that goes on with the technology of beacons, the scanning and reproduction of objects in 3D in which an Arduino system has been installed, and communication based on emotional storytelling. The tool can be analyzed according to the second layer of analysis described above. Recalling the phenomenological and semiotic elements of this layer, the user experience of this tool includes the physical resistance of touching an object and listening to a voice (secondness), an emotional feeling caused by sudden changes of the audio tracks due to beacons (firstness), and an itinerary outlining the visit (thirdness). On the semiotic level, it is the task of storytelling (symbol) to reconstitute the visitor’s inability to have a visual experience, an experience that operates at the iconic level. For this reason, the storytelling must have a strong emotional content, which is achieved by music and the voices of actors (icons). The storytelling needs to be strongly driven by a single synthetic meaning; in this case, the meaning is the anti-Roman sentiment of the Samnite people. Finally, the storytelling must be accomplished in the first person, with the name of the main character and the secondary characters being determined by indexes.1 This analysis of the second level of reality can be used to clarify the first, everyday level. Using this electronic tool, blind people are able to “see” the museum, which is at the first level of reality, but in doing so are taken to the second level of reality. Digital reality reaches the experience that they are unable to have at the first level of reality. This is true for blind people, but digital reality also changes the experience for people without a disability. Some able-bodied people who used the device for blind people said that it was the first time they had “seen” the museum. This fact is very important for the concept of gesture: the application is a gesture (for both its creators and users) that enables an understanding of the museum regardless of whether the viewer has a disability or not. On a sociological level, this means that many people are somehow “blind” when they view the content of museums. That is why they are often bored — the visit is not a gesture. The app rejuvenates the visitors’ experience of the first level reality by means of an intervention from the second level, using technological, synthetizing gestures. Augmented reality and interactive reality can be interpreted following the same line. This sort of experience can be paralleled only by a very intense and highly scholarly prepared visit. It is useful to compare the differences between this 4.0 technological approach and other technologies used for similar goals. A video, audio, a paper guide, or a specialized review of a museum are all technologies and, as such, as much as they are useful, do not easily reach the level of complete gesture. Any experience using this technology 1.0 will be missing certain phenomenological and semiotic elements. Reviews in journals, newspapers, and books always privilege thirdness and symbols over the other elements. From the standpoint of a philosophy of gesture, these are always incomplete gestures; that is, gestures in which synthesis is weak because of the lack of, or only the attenuated presence of, one of these elements

 See the project at https://www.heritage-srl.it/case-studies/smart-cultural-heritage-4-all/

1

The Philosophy of Gesture and Technological Artefacts

107

(Maddalena, 2015: 74–81). In reviews, symbols overwhelm the other elements, and they, therefore, lean towards abstraction and conceptualization. Videos and audios risk becoming projections and will always be weak as far as secondness is concerned. Moreover, as far as their application to museums is concerned, they often lack a symbolic narrative. In this case, they share only information and do not reach the level of a complete gesture. They can provide a “story guide” but not a “story experience”.2 The technology 4.0 that was the subject of the experiment at the Samnite Museum permits, on the other hand, a real experience of the objects it contains. From a logical point of view, this experiment shows that we always learn something new (i.e., engage in a synthesis) by gestures. Videos, audios, and reviews help us to learn something, but the two-level technology of our digital tools allows an experience that can be matched in the first level of reality only by a tour conducted by a charming scholar. The technology allows the meaning of an object to be comprehended and communicated, which, according to Peirce’s semiotic, is exactly what we mean by a “real” or “dynamic” object  — the object (and its meaning) experienced in their vague richness. By means of the technological, digital gesture, this vague initial experience becomes a definite experience and eventually acquires a general meaning. Has our experiment increased the intension of our knowledge? To answer this question properly, it is necessary to insert another distinction. I said that we should understand synthesis not as a reversed analysis but as an original kind of thinking in which we think by doing. We reason in this synthetic way by a logical tool called “gesture” as much as we reason analytically by chains of concepts. To these two realms of reasoning we must add another one: vague reasoning. If in the synthesis we know something new by recognizing identities through change, and in the analysis we learn by breaking identities, in the vagueness we are blind to identities. It is the kind of reasoning that we perform when we face antinomies, for example. The following graphical representation provides a general picture of these three realms of reasoning (Fig. 1). Our reasoning swings through the three kinds of reasoning. This is the meaning of the arrows pointing in both directions: we move among these three types of reasoning. The nature of analytic reasoning is well-known. It is defined by the deductive-­inductive-abductive logic that has been an integral part of the history of philosophy, mathematics, and science. The synthetic reasoning that I am advocating here has as its subject all meaningful actions  — that is, gestures  — that have a beginning and an end and carry a meaning. We have seen the phenomenological and semiotic structure of these gestures. In general, technology has this structure and is a gesture. The Samnite Museum prototype showed that technology 2.0, 3.0, and 4.0 involve a duplication of this structure. Vague reasoning is the rich but indefinite realm from which our inquiry starts, involving a method of reasoning that is  This important distinction is a result of the project Vasari, for the valorization of Italian artworks. http://www.vasariartexperience.it/progetto/ 2

108

G. Maddalena

Fig. 1  Three realms of reasoning

different from both synthesis and analysis. Even though it is important, a proper study of this form of reasoning, which will involve the use of a new mathematical tool, is still to come and much needed (De Tienne, 2021). We can delineate three different forms of extension and intension according to these three realms of reasoning: vague, synthetic, and analytic. Our initial experience of the object and the initial reasoning about it are vague. At this stage, the distinction between extension and intension is blurred. With our gesture we bring the distinction to the light. In the synthetic approach of our reasoning through gestures, they present themselves as united as inseparable, like the universal and the particular in the mathematics of sheaves (Zalamea, 2012). At this point they can be distinguished in different ways in the analytic pattern, starting from both secondness and its indexical dimension and thirdness with its symbolic dimension. The first step is linked to the work of the realist tradition, and the second was the work of the Fregean tradition of meaning. In using this scheme to analyze the app used at the Samnite Museum, it is clear that the usual experience of a visitor to the museum is vague: there is something to understand, and there is an important richness to discover but no further clarification is forthcoming. Our app, together with its connected devices, is synthetic: many universals are present in the particular object(s) that are the subject of the app’s focus. We see that the app increases the clarity of information by increasing the visitor’s sense of the object(s), but this does not mean that the sense and the object are equivalent. The object can also be explained in other ways, in different senses. This description amounts to saying that the app “smart cultural heritage for all” is a synthetic gesture in which intension and extension are increasing together.

The Philosophy of Gesture and Technological Artefacts

109

4 Conclusion This article intends to show that artifacts are embodied reasoning, forms of synthetic reasoning in which we understand reality by acting upon it. Technically, what it proposes is to recognize objects (like the objects in a museum) in a new, deeper, way through meaningful actions that operate within time and space understood as continua. In particular, our 2.0, 3.0, and 4.0 technologies are examples of complete gestures composed of two phenomenological and semiotic structures. Less technically, we can say that artifacts are synthetic gestures by which we comprehend reality while we are performing some meaningful action in it. In particular, our artifacts allow us to interact with reality through its representation. This interaction allows a deeper comprehension of the dynamic object or the vague initial experience, restoring the possibility of a complete gesture, which is otherwise possible only with a special tour both scholarly precise and narratively charming. This kind of performed reasoning does not despise or downplay the importance of analytic reasoning but defines it in a new way as a derivative subset of reasoning. This is only an introductory paper towards an understanding of technology as gesture and an inquiry into the digital revolution as a particular kind of complete gesture. There are many open questions, including the ontological status of technological reality and vagueness, the logical differences between technology 1.0, 2.0, 3.0, and 4.0, and the verification of intension and extension in vague, synthetic, and analytic reasoning. However, once the path of inquiry is established, technology will find its definitive place in the epistemic arena.

References Calcaterra, R.  M., Maddalena, G., & Marchetti, G. (2015). Il pragmatismo. Dalle origini agli sviluppi contemporanei. Carocci. Cavaillès, J. (1994). Oeuvres completes. Hermann. Cavaillès, J. (2008). Sur la logique et la théorie de la science. Vrin. Châtelet, G. (2010). L’enchantement du virtuel. Éditions Rue d’Ulm. Coppens, Y. (2010). Origines de l'homme - De la matière à la conscience. De Vive Voix. Ferrucci, F. (2018). The social and cultural integration of disabled people. In P.  Donati (Ed.), Towards a participatory society: New roads to social and cultural integration (pp. 542–561). Libreria Editrice Vaticana. Hanna, R. (2001). Kant and the foundations of analytic philosophy. Clarendon Press. Hickman, L. (1990). John Dewey’s pragmatic technology. Indiana University Press. Maddalena, G. (2015). The philosophy of gesture. McGill-Queen’s University Press. Maddalena, G., & Zalamea, F. (2012). A new analytic/synthetic/Horotic paradigm. From mathematical gesture to synthetic/Horotic reasoning. European Journal of Pragmatism and American Philosophy, VI, 208–224. Mitcham, C. (1994). Thinking through technology. The path between engineering and philosophy. The University of Chicago Press.

110

G. Maddalena

Parry, R. (2014). Episteme and Techne. In E. N. Zalta (Ed.), The Stanford Encyclopedia of philosophy (Fall 2014 Edition). Retrieved September 8, 2022 from https://plato.stanford.edu/archives/ fall2014/entries/episteme-­techne/ Rey, G. (2018). The analytic/synthetic distinction. In Edward N.  Zalta (Ed.), The Stanford Encyclopedia of philosophy (Fall 2014 Edition). Retrieved September 8, 2022 from https:// plato.stanford.edu/archives/fall2018/entries/analytic-­synthetic/ De Tienne, A. (2021). Peirce on the power of ideas: A semioethical argument with metaphysical ramifications. Semiotics, 1–13. Tomasello, M. (1999). The cultural origin of human cognition. Harvard University Press. Vargas, F. (2015). Modelos y variaciones sobre las ideas peirceanas del continuo. Cuadernos de sistematica peirceana, 7(2015), 139–156. Zalamea, F. (2012). Synthetic philosophy of contemporary mathematics. Urbanomics.

The Orchestration of Bodies and Artifacts in French Family Dinners Aliyah Morgenstern and Dominique Boutet

Abstract  Despite de Saussure’s (1959) visionary writings as to the importance of parole (speech) as well as langue (the language system), the linguistics he founded have focused on the system rather than on use. As a result, language has long been studied out of its ecological context, first through written forms characterized by their linearity, then through invented sentences, and finally with a focus on speech, in experimental studies or semi-guided interactions. Even when gestures are integrated into the analyses, environments are most often stripped of objects or other activities whose affordances have a multitude of impacts on their use. Those limits can be viewed as strengths as they have conducted to fruitful research on langue. However, we believe that in order to capture the full complexity of language, new approaches are needed in which all our semiotic resources can be analyzed as they are deployed in their natural habitat involving the orchestration of bodies engaged in a variety of situated activities with a diversity of artifacts. In this paper, we will focus on the use of “manipulative” and “communicative” gestures in an ecological environment. We will first present some of the biases, categorizations, and possible continuities in past analyses of gestures, introduce our theoretical framework and methods and illustrate our approach through detailed analyses of a sequence of video extracts to capture the coordination of speech, gesture, and actions of family members as they are eating and conversing during their daily dinner. Keywords   Communicative actions · Multiparty interaction · Manipulative actions · Gesture · Language socialization · Family dinners

A. Morgenstern (*) Sorbonne Nouvelle University, Paris, France e-mail: [email protected] D. Boutet Université de Rouen, Mont-Saint-Aignan, France © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_8

111

112

A. Morgenstern and D. Boutet

1 Biases, Categorizations, and Continuities in Gesture Studies Interest in gestures dates back at least to Cicero and Quintilian who analyzed them in the context of their importance as convincing tools for rhetorical purposes. Their view of gesture as a universal language was also shared in the sixteenth and seventeenth centuries as reported by Kendon (2004), by scholars such as Bonifacio, Montanus or Bulwezr. De Jorio, one of the authors who studied gestures in the nineteenth century, showed continuities over time between gestures used in antique Greece and those produced by his Neapolitan contemporaries, whereas in the twentieth century, Efron (1941) rekindled scholars’ interest in gesture by studying cultural differences. A large range of authors in different fields including biologists (Darwin, 1877), philosophers (Wittgenstein, 1953), psychologists (Wundt, 1912; Kendon, 2004; McNeill, 1992), anthropologists (Jousse, 1974; Haviland, 1998) and linguists (Calbris, 1990; Müller, 2009; Cienki, 2012) have helped create a new scientific domain. A certain number of biases and categorizations projected from linguistics have been limiting the scope of gesture studies. In order to include gesture in our communicative system or to view it as an integral part of language, parameters derived from verbal language that might not apply to gesture and signed productions have been used and the structuring role of the body, as well as its movement, have been minimized.

1.1 Conventional/Idiosyncratic Gestures “Kendon’s continuum” (1988) proposed by McNeill (1992) linearizes gestural phenomena by relating them according to their degree of conventionality (among other parameters): Gesticulation> language-like gestures> Pantomime> Emblem> Sign language. This continuum largely takes into account the presence or not of co-articulated speech. The emblems are supposedly the most “lexicalized”, the most “linguistic”, the most “conventional”. Within “gesticulation”, we are used to categorizing interactional gestures into iconic or representational gestures which would be the least conventional and the most “imagistic”, expressive and individualized, in contrast to beats that have a “prosodic” role as they structure and punctuate the gestural flow, deictic gestures (including pointing) considered as transparent and in connection with their referents, and pragmatic gestures (also called “recurrent gestures” by Ladewig (2014) which have a high degree of conventionality within the same linguistic community. Boutet (2010) suggests that iconic gestures as well as beats, two categories of gestures considered unconventional or non-linguistic and which

The Orchestration of Bodies and Artifacts in French Family Dinners

113

appear visually shapeless and idiosyncratic, are sketches of emblems. For productive and economic reasons, because they are punctuated by the discursive flow, we cannot fully deploy them. Thus, there may be a true formal filiation between the “most conventional” gestures, and those which are “the least conventional”, despite significant visual differences due to how the various segments of the upper limbs are set in motion with more or less amplitude and velocity.

1.2 Focus on the Hands Most studies and categorizations of communicative gestures are centered on the hands because they are the most complex and salient articulators that are mobilized in the visual-gestural mode during multimodal communication. However, we activate other articulators (head, face, shoulders, orientation of the trunk) to communicate. This focus on the most distal parts of our body (those furthest from the trunk), the hands, is criticized by Boutet (2018) who devoted his research to a kinesiological approach to gestures in order to show how physiological constraints structure meaning and how all segments of our body, in particular the arms and their dynamics, must be analyzed.

1.3 Visual Perception Our visual perception has long been considered the only modality to take into account in the analysis of expressive or interactional gestures because it allows us to perceive, capture, recognize forms. Many studies seem unconsciously linked to the postures represented in the cultural artifacts, paintings, sculptures, drawings, that freeze the dynamics of gestures. By unconsciously removing movement and its flow from our analysis, research, based on visual analysis, has not been able to grasp what constitutes the true essence of gesture. Despite the impact of video recordings, studies of gestures are too often based on the analysis of their static visual representations derived from art. Praxis and proprioception, sometimes convened by certain artists as well as certain researchers, are not at the heart of current gesture studies. Even Greimas (1968) whose project was to understand how humans construct meaning out of their surrounding environment, limited his exploration to the visual modality and the visible world and even envisioned the “visual phonology” of gestures (p. 30). Our cultural filters and our apprehension limited to visual sensoriality have therefore limited our analyses of gesture.

114

A. Morgenstern and D. Boutet

1.4 Manipulative Versus Communicative Gestures Gesture studies are now a dynamic emerging field in which specialists combine a variety of theoretical approaches and methods to study what Kendon (2004: 7) described as “visible actions that are used as an utterance or part of an utterance.” Utterances “may be constructed from speech or from visible bodily action or from combinations of these two modalities. ‘Gesture’ is the visible bodily action that has a role in such units of action.” In this definition however, “communicative” or “symbolic” gestures such as pointing, shrugging, shaking the head, nodding, waggling the index finger, are clearly distinguished1 from practical actions such as eating, drinking, handing out a handkerchief or a book, throwing a ball, pushing away a chair. In line with this view, Grosjean and Kerbrat-Orecchioni (2002) also separate gestures and actions: actions have a practical function that changes the physical world, but do not carry meaning; gestures have a symbolic function and carry meaning. Other authors in a variety of domains contribute to this distinction such as Quek (1995) who in order to enhance how computers might sense the intents of humans through vision-based interfaces, identifies “communicative” gestures which have communication as their main purpose and are meant for “visual interpretation” and “manipulative” gestures used to act on physical artifacts in the environment and do not need to be visually interpretable. The semiotician Greimas whose work is grounded in binary relations establishes a similar distinction between gestural praxis in which humans are “doers”, sujets de l’énoncé, and gestural communication in which humans are seen as “conceptualizers” (Langacker, 1987) or sujet de l’énonciation (1968: 17). There is however some flexibility in these categorizations. Kendon notes that practical actions can take on the quality of gestures: “The movements necessary for any practical action may, however, in their performance, be embellished in such a way as to render them more than ‘merely practical’. In pouring a wine at table, for example, it is possible for the person pouring the wine to ‘merely’ pour the wine. But it is also possible that all the actions involved — raising the bottle to display it, adjusting the angle for the pour, twisting the bottle at the end of the pour to stop a drop of wine from running down the side of the bottle, moving on to the next guest — may be performed so that they are so elaborated with flourishes that they come to be openly recognized as having an expressive aspect. As this happens, they may come to take on the qualities of gesture” (2004: 9). Those considerations are also paralleled by Greimas (1968) who hints at a possible continuum but prefers to use clear-cut categories.

 We do not include in this discussion the distinction between communicative gestures and the types of gestures that can support the thinking processes studied by McNeill (1992) and a vast number of others, but that is another important dichotomy discussed in the literature. 1

The Orchestration of Bodies and Artifacts in French Family Dinners

115

1.5 A Continuum: Links Between Manipulative and Symbolic Gestures Despite the convenient categories designed by researchers in order to capture and describe them, gestures are both phylogenetically and ontogenetically linked to action and human artifacts. As explained by Boutet (2018), Leroi-Gourhan (1964) has shown that our human faculties are projected onto the world thanks to what emerges out of our “hands2”. Our environment has been molded by human activity; we are surrounded in our daily life by human-made hand-made artifacts. Objects, tools, furniture, housing, are constructed, used, recycled by and for the human “hand”. All objects are shaped according to the affordances (Gibson, 1977) of our bodies. For a chair, the height of the armrests is relative to the height and width of the seat, its resistance to weight, the backrest is placed according to the depth of the seat, etc. (Neufert, 2014). Manufacturing objects requires a grip, a force, an inclination, a praxic gesture applied to various materials (Rolian et  al., 2011; Brumm & Rainey, 2011). The object is itself extracted from materials, it is shaped for a function, by the hand which gives it a form, through the hand which will grasp it and the body which will orient it towards its function. When objects are not manipulated (wall, bedding, furniture …), they receive orientations and dimensions given by the body and respond to certain uses or gestural arrangements (handling of chairs, opening of doors and drawers, shifting of furniture …). The forms resulting from the objects which surround us are partly informed by our praxic gestures. The artefactual structuring by praxic gestures thus has great consequences on our symbolic gestures. There is thus continuity between praxic gestures (Barham, 2013) which manufacture, shape, cut, polish, grip, bind, weave, on the one hand, and, on the other hand, seize, grab, grip, strike, squeeze, hold, caress, or touch objects. Objects or artifacts themselves have a genetic link with gestures — both in terms of production and use. Gestures constantly inform objects (Gibson, 1977; Turvey, 1992). Symbolic gestures resemble certain referents, not because the human mind naturally lends itself to analogy, but simply because the object is shaped and is given meaning through gesture. Another link between gestures, actions, and object manipulation has been established through the interest in communicative gestures performed by infants in the first stages of language acquisition, as behaviors that precede and prepare the emergence of speech. In a number of studies, colleagues have provided developmental evidence for the continuity from actions to gestures to words in child language acquisition. As of 4  months of age, infants begin to produce voluntary forms of grasping objects. The precision grip is used around the end of the first year. Children will use objects to gain adults’ attention by requesting, showing, giving (Iverson & Thal, 1998). Those actions often lead to more ritualized behaviors and are analyzed as the entry into deictic gestures.  The word “hand” is here used in a metonymic sense.

2

116

A. Morgenstern and D. Boutet

Around 8  months, infants develop fine motor skills that are tangible for the observer when we see them manipulating small objects with the extended index finger and the pincer grip. Pointing, the most prolific and most studied gesture in child developmental studies may emerge from index-finger extension used to touch, press, or feel objects and explore the surrounding world (Masataka, 2003). For other authors, pointing may develop out of prehension (Wundt, 1912). According to this hypothesis, unsuccessful grasping movements are interpreted by the adult as a gesture of request addressed to others (Vygotsky, 1934). In both cases, the extended finger and arm establish a link between the child and the object. Showing and giving are also probably linked to children’s first exploratory manipulations of objects. Requests on the other hand can be performed with the whole body. Lifting the arms to be picked up enables children to capture the adults’ attention and is interpreted as meaningful by caregivers who then react by reformulating the child’s intention through speech and of course, if willing, lift them up. Thus, adults respond to infants’ motor acts from very early on and infants, faced with positive feedback will then repeat their successful actions which often involve objects. This helps them construct motor repertoires that may remain stable across contexts (Sparaci & Volterra, 2017). Just as described by Haviland (1998) in the case of Zinacantec infants, adults in our Western datasets (Morgenstern & Parisse, 2012) endow their children’s behaviors with significance and celebrate their actions by reacting with interpretations. They give meaning to the child’s every move, object manipulation, gaze, gesture, facial expression, vocal production, projecting agency and intentionality onto the infant. The link between praxis and symbolic gestures has also been established through viewing gestures as simulated action by Hostetter and Alibali (2008) according to whom gestures emerge from perceptual and motor simulations that underlie embodied language and mental imagery. In line with MacNeill’s perspective (1992), representational gestures are forms of “embodied thinking”. A number of studies illustrate how iconic gestures arise from the manipulation of objects (see for example Roth, 2002, on the emergence of scientific language). Mirror neurons in the monkey have been described as firing in association with goal-directed actions made with an object (Rizzolatti & Craighero, 2004). Similar activation in the mirror neuron system associated with communicative hand gestures and object-directed hand movements in humans was also found by Allison et al. (2000). In our own work, for the sake of observing developmental semiotic changes in semiotic resources deployed to express negation (Morgenstern et  al., 2017), we separated actions and gestures creating an artificial discontinuity between Ellie’s action of pushing away a glass of milk with her vertical palm firmly pressing against the glass offered by her mother (clearly interpreted as a rejection by the mother and integrated into the conversation with the same status as any verbal or gestural symbolic cue) and a conversational gesture with the arm extending, palm-up towards the addressee to reject the offer to go for a walk. As shown in this example, in our everyday life, we interact in an environment filled with artifacts and often make gestures while holding physical objects in our hands. The manipulation and

The Orchestration of Bodies and Artifacts in French Family Dinners

117

interactions with those artifacts can be fully integrated into our theoretical approach and methods to capture and render daily interactions.

1.6 A Multimodal, Multisensory Approach to Interaction Multimodal approaches to interaction have highlighted the sensory features in the organization of social interaction thanks to the affordances that are perceived. Various types of behaviors (gaze, speech, gestures) and their (a)synchronicity have been analyzed (C.  Goodwin, 2013, Mondada, 2011). Those multimodal analyses are highly influenced by conversation analysis founded thanks to the ethnomethodological project of Garfinkel (1967), in connection to Goffman’s interactive approach (1974) and to Sacks’ detailed transcriptions and analyses of natural conversations (1992). If a number of studies have explored the coordination of auditory and visual modalities and how they are deployed in conversation thanks to the participants’ bodies as they are visually available (C.  Goodwin, 2000), the sensory modalities such as touch have not been analyzed in as much depth. This might be one of the reasons why the divide between gesture and action has not been questioned more and why interactions are generally studied within the mono-activity of conversations. However, C. Goodwin (2013) has demonstrated that different sensory modalities contribute to the organization of social interaction. Touch has been shown to be a significant communicative phenomenon especially in the context of adult–child conversations (M. H. Goodwin & Cekaite, 2013). Haptic formats were identified as being a means for adults to control and shape children’s embodied actions both in family and primary school contexts. As explained by Cekaite (2015), a focus on embodied practices, both gestural and actional can provide “concrete insights into the intercorporeality of adult–child socialization and outline some of the bodily techniques relevant for inculcating social, bodily accountable ways to act in everyday interactions” (p. 153). As adults, we have developed practices on how to act in the world of beings and things that surrounds us (Bourdieu, 1977). Some interactional studies have focused on touch and intercorporeal engagement in social interaction (M.H.  Goodwin, 2012). De Leon (1998) demonstrates how touch organizes multimodal participation frameworks by monitoring children’s attention. We can thus place ourselves in contrast to the Cartesian dualism between mind and body. Our body is the existential basis of culture and perception (Bourdieu, 1977) and it is through the kinetic and multimodal coordination of our productions and our perceptions that we become full cooperative participants of our own cultural community (Merlau-Ponty, 1962). The sense of our agency through our body in motion in the mundane activities of our daily lives resonates with our environment filled with people, dynamic events, and artifacts.

118

A. Morgenstern and D. Boutet

2 Approach, Context, and Method 2.1 Theoretical Grounding Adam Kendon (2004) inspired by David Efron (1941/1972) and Wilhem Wundt (1912/1973) made a request for studies on the use of gestures in context. The aim of our own research is to capture language in its ecological environment in order to articulate its actional roots and its symbolic functions. Following Boutet (2018), we thus analyze the bodies of our participants as both the support (the instrument) and the substrate (that which constitutes and structures) for “languaging” (Linell, 2009). Our approach grounds language in embodied action rather than viewing it as a code or a symbolic system (Bottineau, 2012). What we call “languaging” is not only relative to the languages and cultures a subject uses, but also to the available semiotic resources that can be coordinated and enable us to embody mental constructions. Reversely, the semiotic resources we use shape, construct, and give meaning to our interactive productions. Using multimodal approaches to language (Morgenstern, 2014) in grounded situations in which bodies in movement interact can transform gesture studies but also linguistic theories. We share with Cienki (2012) an integrative view of language and use what he calls the “dynamic scope of relevant behaviors” to capture the complexity of interactive practices. In the communities we study, for hearing adults, speech is the default medium in which they express and share their ideas. But other behaviors including actions, object manipulations, non-lexical sounds, prosodic patterns, facial expressions, and gestures can acquire symbolic or communicative value according to the affordances of the context. Cienki’s theory (Cienki, 2012) is a view of language as having a flexible boundary: the “relevant” body segment is used to communicate meaning according to the context, interlocutor, availability of the body parts in the situation, and activity. A meaning can migrate from one body part to another: if hands are not available, shoulders can be used or head or both or speech … and if the whole body is used, a mouth shrug or a frown will suffice. A family of meanings is thus dynamically paired with a family of forms. Because of cultural filters, most gesture specialists do not pay enough tribute to the role of the body and its dynamics in their analyses. Gestures are structured by the physiology of our bodies. They are grounded in the constraints and potential of our movements. The meanings that emerge out of our gestural productions are the product of the natural articulation of our body as well as of our past experiences reenacted in the space we inhabit and dependent upon our relations with our interlocutors and the objects that surround us. Not only do we need to integrate all semiotic resources to capture our communicative behavior, but our actions and manipulations of artifacts need also to be included. As we adopt a multimodal, dynamic, and situated approach to language use in interaction, the analysis of both actional and symbolic gestures in their multimodal and interactive context is particularly relevant to capture how we distribute meaning. The materiality of the body has the potential to mold our environment, our

The Orchestration of Bodies and Artifacts in French Family Dinners

119

tools, our objects, the spaces we inhabit (Leroi-Gourhan, 1993). The structuration of those artifacts is tightly associated to praxic gestuality which is in continuity with symbolic gestures.

2.2 Family Dinners At the heart of understanding the relationship between language and everyday experience is the linguistic anthropological perspective that views language as a form of social action and interacting as organizing social life (Duranti, 2011). Family dinners grounded in commensality and conversation are a major collective ritual that plays a key role in French people’s identity and constitutes an inherent part of French cultural heritage (Kremer-Sadlik et  al., 2015). Detailed analyses of dinner conversations, inspired by language socialization theory, have become a rich source of data for the study of social practices (Ochs et al., 1996). Meal practices across cultures have been analyzed in a large-scale comparative cultural project, initiated by Elinor Ochs (Ochs & Kremer-Sadlik, 2013). Research thus indicates that family dinner studies constitute a promising area for work on interactive multimodal language practices. Most studies focus on dyadic communication. But Goffman’s critique of the speaker–hearer dyadic model (Goffman, 1974) and cross-cultural studies of various communities suggest that language is also learned and used in multiparty interaction (Ochs & Schieffelin, 1984). In a Western country such as France, since dinner is often the only moment when the whole family consistently interacts during the day, evening meals are an excellent locus to study the impact of multiparty interaction on language use.

2.3 Method Thanks to combinations of experimental and ecological studies, to video recordings, to a variety of specialized software, international databases, theoretical approaches, rich collaborations among experts of several scientific fields, we now have the tools that help us create new methods to study the blossoming of both manipulative and communicative gestures in everyday interactive language practices. Video-recording tools have notably advanced the detailed analysis of the organization of human action and interaction (Mondada, 2019). These tools have shaped new avenues of research on language in interaction, as it is deployed in multiple ecologies, both in time (the moment-to-moment unfurling of an interaction) and over time (multiple recordings over several years of the same children in their family environment). Child language research is one of the first fields in which spontaneous interaction data was systematically collected, initially through diary studies (Ingram, 1989; Morgenstern, 2009), and later by audio and video recordings shared

120

A. Morgenstern and D. Boutet

worldwide thanks to the CHILDES project (MacWhinney, 2000). Vygotsky’s theory of learning as socially co-constructed between collaborating partners within a cultural context (1934) gives a fundamental role to interaction in children’s cognitive and linguistic development. They learn to understand language and action together, each providing support for the other. To examine how children come to use language in general, one must examine the broader context in which the child experiences events, plays with artifacts, and interacts. Through family dinners, language practices can be analyzed in the framework of multiactivity as it occurs in real life and real time. We focus on the finely tuned coordination and in situ organization of the joint activities of conversing and dining that fully engage the same body components (eyes, head, mouth, hands, arms). We integrate multiactivity as part of real-life and real-time human experience (Haddington et al., 2014) in our theoretical and methodological approach to language use to capture the multiple deployments of the embodied behaviors of dinner participants. We show how family members collaboratively manage the accomplishments of multiple streams of activity and coordinate their temporal organizations through the embodied performances of dining and interacting (Goodwin, 1984). The families consist of two adults and two to three children. They are recruited through social networks, schools, conservatories, associations, and daycare centers. We have recorded sixteen dinners in eight middle-class families living in Paris. The dinners were recorded with two cameras set at different angles and both father and mother wore a microphone. For this paper, we focus on segments drawn from the data of one of the families in which a father, a mother, and their two sons, LUC (9 years old) and GAB (4 years old) are having dinner. For the reader’s comfort, we chose to present the transcriptions of the video extracts into tables, and to separate the productions into multiactivity turns per participant (lines) and body movements and speech (columns), but each semiotic resource would deserve a column of its own.3 Clear overlaps of multiactivity turns are indicated by using the same number and letters.

3 Detailed Analyses of a Family Dinner Sequence In order to illustrate the orchestration of actions, manipulative and communicative gestures with gaze and speech in a context in which multiple participants interact with artifacts and people in an ecological environment, we will examine two very small excerpts of a family dinner in detail. In Extract 1, as the father has warned that he would come back a bit late from work, the mother and her two sons have begun eating when the father gets to the dinner table. The mother prompts the sons to greet him, which is done through kissing and interferes with the activity of eating.  It would have been interesting to present an ELAN (https://archive.mpi.nl/tla/elan) transcription, with multiple tiers, but restrictions of space do not allow it. 3

121

The Orchestration of Bodies and Artifacts in French Family Dinners

Extract 1 0–30 sec nb 1

Participant Body movements FAT Approaches the table

2

GAB

3

LUC

4

MOT

5 6

LUC FAT

Lifts his cheek towards FAT Gazes at LUC, passes GAB who is closest to him, walks towards LUC, bends over and kisses his cheek as he speaks.

7a

FAT

Moves towards GAB and kisses him.

7b

GAB

7c

LUC

7d

MOT

Stops eating a micro moment to receive the kiss. Resumes eating. Gazes at MOT. Speaks softly, then puts food into his mouth. His gaze is in the air head turned outside the dinner table. Continues to cut avocado.

8

FAT

9

GAB

Lifts pieces of avocado dipped in sauce to his mouth and eats them, gaze focused on his dish. Stops eating and gazes at his father, head turned towards him. Gazes at FAT then continues to cut an avocado in pieces.

Walks to his chair. Gazes at MOT then GAB. Pushes the chair away from the table to prepare sitting. Continues eating, gazes quickly at FAT and speaks between bites.

10 MOT a 10b FAT

Continues to cut the avocado and is looking down at her hands. Looks at GAB. Laughs. Sits down as he speaks.

11a MOT

As she speaks, she stops cutting, puts down the cutlery, hands out the dish of avocado she has prepared to LUC as she questions him and gazes at him with an inquisitive look.

11b FAT 11c GAB

Gazes at LUC as he puts food into his mouth. Stops eating. Gazes at LUC with his spoon held out high Gazes at his mother. Nods his head in agreement. Takes the bowl of avocado from his mother’s hand.

11d LUC

Speech Bonjour, ça va? Hello, is everyone OK?

Eh vous dites bonsoir les gars? Hey say hello guys. Et ce devoir de sciences, alors, tu l’as eu? So did you do the science exam? Il a été fait? It was done?

Oui Yes Ouais. aujourd’hui. Yeah. Today. Ca s’est bien passé? Did it go well? On a direct attaqué. We started straight away On a direct attaqué. T’as direct attaqué, et ben super. You started straight away, well great. (to FAT) Il dit que c’était noté sur… (to LUC) combien 50? He says the grade is out of… how much 50, is that right?

122

A. Morgenstern and D. Boutet

This first extract illustrates how different activities around the dinner table, handling of objects, and of interaction are coordinated around the father’s arrival. Participants are busy greeting him but also eating, speaking, gesturing, preparing, and serving food. Each participant has his or her own agenda. MOT manages the food and the conversation. FAT is concerned with his eldest son’s science exam. LUC is engaged in this attention. GAB is very concentrated on eating his avocado. However, they have moments of collective attention as they all join on events or conversation topics. MOT prompts the greeting of the father and both children comply willingly as they receive the father’s kiss. LUC concentrates on his father, stops eating, gazes at him (3), which prompts FAT to acknowledge that the right body affordances are there to receive his attention and to go towards him to kiss him first, even though he was standing right near GAB, very busy with his avocado (Fig. 1). But GAB does micromanage beautifully to fit in the gestures necessary to continue eating bites of avocado and to hold the right body posture for his father to kiss him (7b). He stops eating as he receives the kiss (7a). The topic of the science exam grabs even GAB’s attention (11c), he stops eating and his gaze is fixed on his brother. FAT is progressively integrating the family meal which has started without him, both in terms of placing his body in the relevant situation (sitting at his place) and picking up on the salient events that characterize the meal (discourse about the here and now) and his family’s day (displaced speech). During those 30 seconds of video, the focus is both on the science exam and on the fact that they have started without him, which is not habitual and is highlighted with eagerness by his youngest son (9) whom he sees loving the avocado so much that he does not want to disturb his eating. This is acknowledged with a laugh and a repetition “you started straight away, well great” (10). Meanwhile, the mother is making sure everyone has their share of food and attention and she can orchestrate her preparation of a bowl of avocado for her eldest son and the conversation about the science exam. The coordination of Fig. 1  LUC (left) is ready to physically greet his father. He is not eating, his gaze is on FAT, right cheek towards him to receive a kiss. GAB is concentrating on eating, he is lifting his left arm, hand holding the fork on which there is a piece of avocado. Mother is speaking and preparing an avocado for LUC at the same time

The Orchestration of Bodies and Artifacts in French Family Dinners

123

Fig. 2  The mother’s multitasking Mother is handing out the bowl of avocado she has prepared for LUC while also asking him a question with an inquisitive facial expression. FAT is concentrated on LUC and the topic of the science exam. (gaze) while even GAB’s gaze on his brother indicates his joint attention

both activities is highlighted when she hands out the bowl to her son, gazes at him inquisitively (chin up toward him), and verbally asks for his confirmation of the teacher’s grading system (11a). Her multitasking abilities are thus epitomized in that combination of speech, communicative gesture, and manipulative gesture as her offer includes the handling of an artifact, the bowl (Fig. 2). However, her younger son has not developed the same level of mastery yet and she needs to scaffold his management of eating and speaking. She is also trying to synchronize the different courses of the dinner and each family member’s tempo. After having discussed the eldest son’s exam, the father turns his attention to his youngest son whose teacher was absent that day. Extract 2 0.38 to 1.38 nb 1

Participant Body movements GAB Shifts from side to side on his chair. Gazes at this mother.

2

MOT

3

MOT

Finishes a sip of water. Puts down her glass. Shifts her gaze to GAB. Shifts her gaze to her own body movement. Takes napkin off her lap, wipes her hands on it. As she gets up, she places it on the table with her left hand, speaks and with her right hand performs a palm down gesture, and small beats punctuating her vocal production, body slightly shifted towards GAB, gaze on him. She moves towards the kitchen as she addresses him, she goes past him, the left hand still beating with her palm down open hand, arm raising and falling.

Speech S’il te plait je peux avoir des cornichons avec du jambon. Please, can I have pickles and ham?

Mouais attends, une seconde regarde, nous on n’a même pas mangé l’avocat. Yeah, wait a little, look, we haven’t even eaten the avocado.

(continue)

124 nb 4a

4b

5a 5b

A. Morgenstern and D. Boutet Participant Body movements FAT Gazes at GAB. Makes a slight gesture palm lateral resting on the table with his left hand. The right arm extends and he picks up the small bowl of dressing for the avocado and brings it close to him. GAB Speaks as he picks up a cherry tomato between his index and his middle fingers, and brings it towards his mouth. MOT Comes back to the table and gazes at FAT while he speaks as she sits down. FAT Picks up his napkin, keeps it lifted as he speaks and gazes at GAB, then once finished, wipes his mouth.

6

MOT

Straightens her chair as she gazes at FAT. Picks up her napkin, starts speaking and drops it on her laps with an accelerated movement as she says ‘Jocelyne’ with a funny accelerated intonation. Places the napkin on his lap as he speaks and gazes at GAB, then looks at his plate.

7

FAT

8a

MOT

Gazes at GAB

8b

GAB

9

MOT

With his mouth full, shakes his head and then keeps hand turned towards Mum and gazes at her. Bends towards GAB and gazes with inquisitive facial expression. Stops moving arms and hands.

10

FAT

11

GAB

12

MOT

13

GAB

Speech Je vais te préparer ça dans une minute. Soit patient. I’ll prepare it for you in a minute. Just be patient. Je pique. Je pique … I’m picking, I’m picking…

Et toi Jean-Pierre, tu l’as vu aujourd’hui? And you, did you see Jean-Pierre today? Ben non, y’avait Joce, y’avait Jocelyne Well, no, there was Joce, there was Jocelyne. Jean-Pierre il est pas venu? C’est Jocelyne toute la journée? Jean-Pierre didn’t come? It’s Jocelyne all day? Ouais. Yeah No No.

Ah non y’avait quelqu’un après? Oh, no, was there someone else afterwards? Gazes at GAB and manages at the same time to Et après vous avez été put dressing inside half avocado. répartis dans les classes ou euh…. And then you were distributed in different classes or hum…. GAB is chewing a cherry tomato, head lifted Non mais xxx towards his father. No but xxx She shakes her head, gaze intent on GAB, rest Attends, on comprend rien. of the body rigid. Wait, we don’t understand anything. Swallows his mouthful then continues to speak, Après on est allé dans la head turned towards his mother. cour avec Michel. Then, we went in the courtyard with Michel.

The Orchestration of Bodies and Artifacts in French Family Dinners nb Participant Body movements 14a FAT Continues to prepare his avocado. 14b MOT

14c LUC

15

GAB

Lifts bottle of wine with left arm, takes out the cork with left hand and places it on the table. She keeps still as she speaks with bottle raised, two hands holding it. She gazes at GAB with a puzzled expression on her face. Then she changes hands, bottle in right hand, still gazing at her son. Gazes back at the table and pours a glass of wine to father. Has been eating his avocado occasionally gazing at the person speaking. Now he stops eating and gazes at his brother. Gazes at his mother, body still, head shaking slightly. Gazes back at father when he sees him trying to react vocally.

125

Speech C’est qui Michel? Who’s Michel? Ah Michel c’est au, au goûter ça? Oh Michel, that’s during, during the break.

Non, c’est au goûter, et dans la cré, et dans la (ré) création. No, it’s during snacktime and during break, during break times in the courtyard

Fig. 3  a, b, c Mother coordinating artifact manipulation, recurrent gesture, and speech. MOT puts down her napkin with her left hand, makes a gesture with her right hand, speaks, and moves to the kitchen

Gabriel has devoured his avocado and finished eating his first course much before the other family members. This does not follow the usual code of behavior of commensal French family dinners in which our middle-class families manage to eat each course synchronously as much as possible (Morgenstern et al., 2015). When he makes his request for the following dish (1), his mother highlights the need for synchronicity and uses a recurrent gesture in combination with speech as she gets up to ask him to be patient (turn 3, Fig. 3).

126

A. Morgenstern and D. Boutet

Father contributes to the good unfurling of the meal by turning his attention to his younger son, both addressing his impatience (4a) and getting him to speak about his day at school (5b). The activities of eating and conversing are thus coordinated through the collaboration between mother and father. The youngest son who has more difficulty shifting swiftly from speaking to eating, because he has nothing to eat, now has the floor and can focus on speaking. However, he is tempted by the cherry tomatoes placed in front of him, takes one, and attempts to eat and communicate simultaneously. He can manage to shake his head with his mouth full (8) at first but is not successful when speaking (11). The mother needs to intervene (12) in order to indicate that he cannot use his mouth to chew and speak at the same time as the content of his message cannot be understood: “wait, we don’t understand anything.” Management of multiactivity, of utensils, food, actions, gestures, and words is thus quite different according to motor and cognitive development as well as experience. Where his older brother can eat micro spoonfuls and speak (Extract 1) or follow conversations and eat (Extract 2, 14c), GAB has difficulty being an eater and an overhearer (Extract 1) or being a speaker and a diner (Extract 2). His father manages much better to handle the various artifacts used during dinner and can make a communicative gesture with his left hand as he lifts the bowl of dressing for his avocado with his right hand, gazes at his son, and speaks (4a) or pours dressing in his avocado, gazing at his son and speaking (10 Fig. 4). MOT seems to be the champion multitasker in the family as she takes on the role of making sure the multiactivities are not hampering the commensality of the meal, that everyone gets food (Extract 1, she prepares a bowl of avocado for her older

Fig. 4  Father multitasking Father is gazing at and speaking to GAB but also preparing to spoon some dressing in his avocado with both his hands. GAB who has finished eating has the floor (he is relaxed in his chair, head straight up, hands ready for communicative gestures) as mother is also concentrating on finding out about his day in school and has stopped eating. LUC is now busy eating (gaze on his plate, hand manipulating the cutlery to pick up avocado)

The Orchestration of Bodies and Artifacts in French Family Dinners

127

Fig. 5  a. b. c. Mother’s orchestration MOT is managing her interaction with GAB as she pours a glass of wine for FAT

son), and drink (Extract 2, she pours a glass of wine for FAT as she is actively participating in the collective narration of GAB’s school experience). Each segment of her body is expertly engaged in a ballet composed of family members and dinner artifacts, which she choreographs for the whole family (Fig. 5).

4 Conclusion The multimodal nature of language leads us to investigate how various semiotic systems such as speech, gesture, posture, facial expressions, and gaze (Kendon, 1988; Goodwin & Goodwin, 1992) but also actions and object manipulations, are simultaneously deployed, transmitted and used in the situated activities combined in family dinners. Our range of semiotic resources is usually categorized as being either verbal, gestural, or actional. But if we try to integrate those behaviors between participants and include the artifacts that are present and handled to enrich our analyses and fully capture interactions in our daily lives, we might need to (1) take each and every one of them into account; (2) question the categories we have constructed. We find that participants are simultaneous conversationalists and diners. Food and utensils are fully integrated in the situated script that is deployed. Manipulative gestures can be communicative as is the offering of food and wine but also the use of napkins, cutlery, glasses, or the actions of getting up from the table, sitting down in a chair, or moving towards a participant or the kitchen. They are constraints — using the mouth to eat and speak is problematic; but there also are possible multiactivities one learns to combine — chewing can be synchronous with actively listening and gazing at the speaker. Every move, every part of the body, every object is meaningful. They are deployed in a multitude of skillful variations in the collective coordination of bodies, activities, and artifacts.

128

A. Morgenstern and D. Boutet

References Allison, T., Puce, A., & McCarthy, G. (2000). Social perception from visual cues: Role of the STS region. Trends of Cognitive Science, 4, 267–278. Barham, L. (2013). From hand to handle: The first industrial revolution. Oxford University Press. Bottineau, D. (2012). Parole, corporéité, individu et société: l’embodiment dans les linguistiques cognitives. Text, XVII(1), et 2. Bourdieu, P. (1977). Outline of a theory of practice. Cambridge University Press. Boutet, D. (2010). Structuration physiologique de la gestuelle: Modèle et tests. Lidil. Revue de linguistique et de didactique des langues, 42, 77–96. Boutet, D. (2018). Pour une approche kinésiologique de la gestualité. (Habilitation à diriger des recherches). Université de Rouen-Normandie. Retrieved September 8, 2022, from https://hal. archives-­ouvertes.fr/tel-­02357282 Brumm, A., & Rainey, A. (2011). The Acheulean downunder: Modern human “handaxes” from the Barkly tableland of northern Australia. Lithics, 32, 49–61. Calbris, G. (1990). The semiotics of French gestures. Indiana University Press. Cekaite, A. (2015). The coordination of talk and touch in adults’ directives to children: Touch and social control. Research on Language and Social Interaction, 48(2), 152–175. Cienki, A. (2012). Usage events of spoken language and the symbolic units we (may) abstract from them. In J. Badio & K. Kosecki (Eds.), Cognitive processes in language (pp. 149–158). Peter Lang. Darwin, C. (1877). A biographical sketch of an infant. Mind, 2, 285–294. De Leon, L. (1998). The emergent participant: Interactive patterns in the socialization of Tzotzil (Mayan) infants. Journal of Linguistic Anthropology, 8, 131–161. de Saussure, F. (1959) Course in general linguistics, (C. Bally et A. Sechehaye (Eds.), & W. Baskin Trans.). Philosophical Library. Duranti, A. (2011). Linguistic anthropology: The study of language as a non-neutral medium. In R.  Mesthrie (Ed.), The Cambridge handbook of sociolinguistics (pp.  28–46). Cambridge University Press. Efron, D. (1941). Gesture and environment. King’s Crown Press. Garfinkel, H. (1967). Studies in Ethnomethodology. Englewood Cliffs, NJ: Prentice-Hall. Gibson, J. J. (1977). The theory of affordances. Hilldale. Goffman, E. (1974). Frame analysis: An essay on the organization of experience. Harper and Row. Goodwin, C. (1984). Notes on story structure and the organization of participation. In J. M. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp. 225–246). Cambridge University Press. Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522. Goodwin, M. H. (2012). Haptic sociality. Presentation at American Anthropological Association Annual Meeting. Goodwin, C. (2013). The co-operative, transformative organization of human action and knowledge. Journal of Pragmatics, 46, 8–23. Goodwin, M. H., & Cekaite, A. (2013). Calibration in directive/response sequences in family interactions. Journal of Pragmatics, 46, 122–138. Goodwin, C., & Goodwin, M.H. (1992). Assessments and the construction of context. In A. Duranti, & C. Goodwin (Eds.), Rethinking context: Language as an interactive phenomenon (pp. 147–189). Cambridge: Cambridge University Press. Greimas, A. J. (1968). Conditions d’une sémiotique du monde naturel. Langages, 10, 3–35. Grosjean, M., & Kerbrat-Orecchioni, C. (2002). Acte verbal et acte non verbal: ou comment le sens vient aux actes. Int. Colloquium Les Relations Intersemiotiques, December 16–18, 1999, University Lumière Lyon 2. Haddington, P., Keisanen, T., Mondada, L., & Nevile, M. (2014). Multiactivity in social interaction: Beyond multitasking. Benjamins.

The Orchestration of Bodies and Artifacts in French Family Dinners

129

Haviland, J. B. (1998). Early pointing gestures in Zincantán. Journal of Linguistic Anthropology, 8(2), 162–196. Hostetter, A.  B., & Alibali, M.  W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15(3), 495–514. Ingram, D. (1989). First language acquisition. Method, description and explanation. Cambridge University Press. Iverson, J., & Thal, D. (1998). Communicative transitions: There’s more to the hand than meets the eye. In A. Wetherby, S. Warren, & J. Reichle (Eds.), Transitions in prelinguistic communication (pp. 59–86). Brookes. Jousse, M. (1974). L’anthropologie du geste. Gallimard. Kendon, A. (1988). How gestures can become like words. In Poyatos. Cross-cultural perspectives in non verbal communication. Hogrefe. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press. Kremer-Sadlik, T., Morgenstern, A., Peters, C., Beaupoil, P., Caët, S., Debras, C., & Le Mené, M. (2015). Eating fruits and vegetables: An ethnographic study of American and French family dinners. Appetite, 89, 84–92. Ladewig, S.  H. (2014). Recurrent gestures. In C.  Müller, A.  Cienki, E.  Fricke, S.  H. Ladewig, D. McNeill, & J. Bressem (Eds.), Body – Language – Communication: An international handbook on multimodality in human interaction (Handbooks of linguistics and communication science 38.2.) (pp. 1558–1574). De Gruyter Mouton. Langacker, R. (1987). Foundations of cognitive grammar. Volume I: Theoretical prerequisites. Stanford University Press. Leroi-Gourhan, A. (1964). Le Geste et la parole. Tome 1: Technique et Langage. A. Michel. Leroi-Gourhan, A. (1993). L’Homme et la Matière: Evolution et techniques. Albin Michel. Linell, P. (2009). Rethinking language, mind and world dialogically: Interactional And Contextual Theories Of Human Sense-Making. Information Age Publishing. MacWhinney, B. (2000). The CHILDES project: The database (Vol. 2). Psychology Press. Masataka, N. (2003). The onset of language. Cambridge University Press. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press. Merlau-Ponty, M. (1962). Phenomenology of perception (C.  Smith, Trans.). Routledge & Kegan Paul. Mondada, L. (2011). The organization of concurrent courses of action in surgical demonstrations. In J. Streeck, C. Goodwin, & C. LeBaron (Eds.), Embodied interaction, language and body in the material world (pp. 207–226). CUP. Mondada, L. (2019). Contemporary issues in conversation analysis: Embodiment and materiality, multimodality and multisensoriality in social interaction. Journal of Pragmatics, 145, 47–62. Morgenstern, A. (2009). L’enfant dans la langue. Presses de la Sorbonne Nouvelle. Morgenstern, A. (2014). Children’s multimodal language development. In C. Fäcke (Ed.), Manual of language acquisition (pp. 123–142). de Gruyter. Morgenstern, A., & Parisse, C. (2012). The Paris corpus. Journal of French Language Studies, 22(1), 7–12. Morgenstern, A., Debras, C., Le Mené, M., Caët, S., Beaupoil, P & Kremer-Sadlik, T. (2015). L’art de l’artichaut et autres rituels: transmission de pratiques sociales et alimentaires dans les diners familiaux parisiens. Numéro spécial sur les Patrimoines alimentaires et enfantins. The anthropology of food. https://aof.revues.org/7836 Morgenstern, A., Blondel, M., Beaupoil-Hourdel, P., Benazzo, S., Boutet, D., Kochan, A., & Limousin, F. (2017). The blossoming of negation in gesture, sign and oral production. In M. Hickman, E. Veneziano, & H. Jisa (Eds.), Sources of variation in first language acquisition: Languages, contexts, and learners. (Trends in language acquisition research (TiLAR) 22 (pp. 339–363). Müller, C. (2009). Gesture and language. In K. Malmkjaer (Ed.), Routledge’s linguistics encyclopedia (pp. 214–217). Routledge.

130

A. Morgenstern and D. Boutet

Neufert, E. (2014). Les éléments des projets de construction – Ernst Neufert sur Fnac.com. Dunod. Ochs, E., & Kremer-Sadlik, T. (Eds.). (2013). Fast-forward family. Home, work, and relationships in middle-class America. University of California Press. Ochs, E., & Schieffelin, B. (1984). Language acquisition and socialization: Three developmental stories. In R. Shweder & R. LeVine (Eds.), Culture theory: Mind, self, and emotion. Cambridge University Press. Ochs, E., Pontecorvo, C., & Fasulo, A. (1996). Socializing taste. Ethnos, 61, 7–46. Quek, F. K. H. (1995). Eyes in the interface. Image and Vision Computing, 13(6), 511–525. Rizzolatti, G., & Craighero, L. (2004). The mirror-neuron system. Annual Review of Neuroscience, 27(1), 169–192. Rolian, C., Lieberman, D. E., & Zermeno, J. P. (2011). Hand biomechanics during simulated stone tool use. Journal of Human Evolution, 61(1), 26–41. Roth, W.-M. (2002). From action to discourse: The bridging function of gestures. Cognitive Systems Research, 2(3), 535–554. Sparaci, L., & Volterra, V. (2017). Hands shaping communication: From gestures to signs. In Bertolaso, M., & Di Stefano, N. (Eds.), The hand: Perception, cognition, action. (SAPERE, vol. 38, pp. 29–54). Springer International Publishing. Turvey, M. (1992). Affordances and prospective control: An outline of the ontology. Ecological Psychology, 4(3), 173–187. Vygotsky, L. S. (1934). Thought and language. The M.I.T. Press. (1985). Wittgenstein, L. (1953). Philosophical investigations. Basil Blackwell. Wundt, W. (1912). Völkerpsychologie: die Sprache. Engelmann. Wundt, W. (1973). The language of gestures. Mouton.

Towards an Ecology of Gesture: A Review (And Some Promising Paths) Antonis Iliopoulos and Lambros Malafouris

Abstract  Despite the ‘embodied’ turn that gesture studies have been taking, the extent to which the body actually contributes to the realization of thought remains questionable. Due, in large part, to the preoccupation of ‘embodied cognitivism’ with co-verbal gestures, the material world and the gestures engaging it are usually left out of discussion. That said, there is a small but growing corpus of literature on the cognitive effects of gesture that seeks to account for the performative role of the body, as well as the scaffolding function of the physical world. The purpose of this book chapter is to outline the arguments presented by the main representatives of what we loosely call here ‘ecological’ approaches. Our goal for doing so is two-­ fold: i) to highlight the important insights gained about gesture through an ecological lens, and ii) to identify promising lines of inquiry for future research. What we identify as an approach suitably geared towards studying gesture, and especially its creative variety, is a theoretical framework grounded on the combination of Material Engagement Theory and Peircean semiotics. Keywords  Cognition · Gesture · Artefacts · Review · Enactivism · Extended mind theory · Material engagement theory · Peircean semiotics

1 Introduction What role do gestures play in cognition? Do gestures simply express and help to communicate pre-existing thoughts or do they actually shape and constitute them? Importantly, how do gestures relate to the physical, extra-bodily world? A. Iliopoulos (*) · L. Malafouris Institute of Archaeology, University of Oxford, Oxford, UK e-mail: [email protected]; [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_9

131

132

A. Iliopoulos and L. Malafouris

Our interest was recently piqued by these questions, leading us to consider the literature on gesture in hopes of finding some persuasive answers. Coming from the direction of cognitive archaeology, Leroi-Gouran’s (1993) [1964]) Gesture and Speech provided a natural point of departure. In this famous account of human biocultural evolution, gesture is treated as synonymous with the manual creation of extracorporeal artefacts through its adherence to an “operational sequence” or chaîne opératoire. Primed by intentions and goals, humans are said to go through a series of empirical stages, balancing their gestures within the tension found between collective traditions passed down by previous generations and personal idiosyncrasies that make one distinct. The technical and cultural changes effected through this process are indissolubly linked to the evolution of our species, which is, according to Leroi-Gourhan, characterized by a propensity for externalizing mind and language through gesture and speech. As he tellingly put it: “The whole of our evolution has been oriented toward placing outside ourselves what in the rest of the animal world is achieved inside by species adaptation” (p.  235, emphasis in the original). This process of externalization is primarily driven, according to Leroi-Gourhan, by the unique abilities of the human brain. For him, “[t]he difference [between human and higher monkey] is one of nervous apparatus alone” (ibid., p.  239). What can be surmised from statements like these is that Leroi-­ Gourhan viewed gesture as a means through which humans represent outside what is firstly born inside, aided, of course, by a neural adaptation that is able to generate and harbour new forms and meanings. Granting, however, the brain with ontological priority, while framing its connection to the body and the world as unidirectional, may not be the best course of action, as we have argued elsewhere (see Malafouris, 2013; Iliopoulos & Garofoli, 2016). Seeing creativity through methodological individualism and evolution from a Neo-Darwinian point of view is problematic because, according to new theories of cognition and post-genomic models of evolution, the brain is open to the world as much as the world is to the brain, with the neural and cultural elements co-evolving as intimately entwined aspects of the same ontogenetic process. Considering that the body in Leroi-Gourhan’s account has not been framed as mediating the bidirectional dynamics enacted between people and things, mind and matter, we had to look beyond our field’s usual scope in order to find an approach that credits its role in the emergence of gesture and thought. Our next stop had to be gesture research itself and, specifically, the ‘embodied’ tenet favoured in much of the recent discourse. Of course, it became immediately obvious that the physical world seldom found a place in the debate on communicative gestures. As defined by McNeill (1992, p. 78), the umbrella of gestures includes neither self-touching movements nor object manipulations. Seen through the lens of a cognitivist tradition where gestures may represent the world, but “in most cases, they do not influence, alter, or directly affect the physical world” (Kita et al., 2017, p. 255; see Novack & Goldin-Meadow, 2017), the organization of thoughts through gestural body movements has been explained in a number of different ways. For example, some of the models through which the connection between gesture and language has been reportedly explained are: the Gesture-as-Simulated-Action

Towards an Ecology of Gesture: A Review (And Some Promising Paths)

133

account (Hostetter & Alibali, 2008, 2019), the Growth Point theory (McNeill, 2016), the Sketch Model (De Ruiter, 2000), the Lexical Gesture Process (Krauss et  al., 2000; see also Krauss & Hadar, 1999; Rauscher et al., 1996), the Image Maintenance Theory (Wesp et al., 2001), the Information Packaging Hypothesis (Kita, 2000), and the Gesture-for-Conceptualization hypothesis (Kita et al., 2017). Space restrictions preclude us from going into any depth, but we think it would be fair to say that, despite their differences, these ‘embodied’ approaches seem to converge on the same idea: while gestures may indeed function as a “window onto the mind”, they can also help change one’s thoughts in one way or another. Unlike speech, gestures are taken to depend on information schematized by the same system that drives practical action. Yet, despite the embodied direction that the field of gesture studies has been taking, the extent to which the body’s contribution to the realization of thought has been adequately specified remains questionable. Much discussion of embodied gestures results in a kind of embodied cognitivism (Malafouris, 2016) that trivializes bodily experience by implicitly reducing “gestures to cognitively trivial bodily outputs of (sensorimotor) neural precursors” (Pouw et al., 2014, emphasis in the original). It would be preferable to treat gestures as cognitive acts on the very basis of their bodily nature. The temporal cognitive stability that the body is able to maintain is one that speech, with its inherent tendency towards fading, cannot reproduce. In order to demonstrate the ways in which gestures can provide a stable external, physical, and visual presence that helps the cognitive system function, Pouw and his colleagues give examples of epistemic actions, such as the rotation of Tetris zoids, the gestural arrangement of a mentally projected abacus, and the manipulation of objects when trying to understand how fractions work. What they find in all cases is that gestural action reveals the same principle: “When the costs of internal computation are high, either induced by external constraints (higher cognitive demand of the task; more cost of retrieving information from the environment) or internal constraints (e.g., lower working memory ability) the cognitive system is more likely to adopt, if cheaply available, an externally supported problem-solving strategy; be it the environment or gestures” (ibid., p. 10, emphasis in the original). Indeed, according to experiments conducted on the role of gesture in visuospatial cognition by Goldin-Meadow and Wagner (2005), gestures can help lighten the cognitive load and free up previously occupied resources, thus allowing the gesturer to focus better on the task at hand.1 It appears, then, that gestures are physical acts that have systematic effects on cognition, which is why some scholars assume a stauncher position on the importance of their physical dimension, preferring to see them as material vehicles that bring forth, rather than emanate from the mind.

 It is interesting to note that gestures can also aid thinking by inadvertently signalling to others that the gesturer is in a transitional state – that is, at a stage where their gestures reveal some sort of understanding of how a problem can be solved, even though this is not explicitly apparent in their speech. The gesturer can then be accordingly helped (Goldin-Meadow & Wagner, 2005; GoldinMeadow, 2017). 1

134

A. Iliopoulos and L. Malafouris

As we shall see in the following section, these theoretical approaches adhere to radical new ways of thinking in philosophy, such as enactivism and the extended mind theory, as well as draw inspiration from older traditions, such as Merleau-­ Ponty’s phenomenology and Peirce’s semiotics. Interestingly, an ecological line of thinking has also been espoused by anthropologists that employ the idea of distributed cognition in order to study the coupling of some gestures with the environment, physical, social, and cultural alike. A particularly interesting kind of environmentally-­ coupled gesture is the musical gesture, which is why we shall be focusing, in particular, on piano playing from a performer’s point of view. In order to better appreciate how creative performances come to life, we shall finally consider what is arguably the most balanced and complete of all gestures, the one whose vague initial meaning gets progressively more and more determinate until a habit is established. Though not exhaustive by any means, this overview should be able to give us a representative picture of what can be loosely called an ‘ecological’ school of thought. Obviously, differences are expected to be found within, and it is the particularities of each position that we want to highlight here. What this exposition will ultimately serve to show is that there is strong merit to viewing gesture along performative lines and that a performative perspective can take into account the material world and its effects on gesture and cognition. Based on this ‘ecological’ logic, our concluding section shall attempt to identify a direction based on the praxis-­ centred epistemology of the Material Engagement Theory. We think that drawing upon an enactive philosophy of mind and complementing it with the semiotic theory of Charles Sanders Peirce could be what is required in order to frame gesture as a process that is situated in the bidirectional dynamics of the relations established between brains, bodies, and things.

2 Ecological Approaches The implications of gesture for cognition have naturally drawn the attention of philosophers working on the ontology of the human mind. One of these scholars is Shaun Gallagher, a proponent of enactive cognition (Gallagher, 2017). According to a broad definition of this theory, cognition is best seen as the historical product of sensorimotor actions performed by an organism. From the vantage point of enactivism, gestures shape cognition by functioning as prenoetic elements that impose constraints on the structure of cognition (Gallagher, 2005, Chap. 5). Like a spatial perspective or an affective lens, they too constrain the way we make sense of the world by making salient only parts of it. Yet, as he points out, gestures also contribute to the accomplishment of thought. The reasoning behind this second supposition is that, in expressing thought, gestures manage to complete it. Influenced by Merleau-Ponty’s (1962) account of how language helps us complete thought through expression in meaningful strings of words, Gallagher argues that something similar can also hold for gestures, whose expressive movements achieve more-or-­ less the same thing.

Towards an Ecology of Gesture: A Review (And Some Promising Paths)

135

Coming from another but adjacent angle, Andy Clark has tried to elucidate the mechanism responsible for both the completion and the transformation of thinking. A famous proponent of the so-called Extended Mind Theory, a theory according to which it is not just extra-neural, but also extra-bodily elements that should be counted among the material vehicles of cognition (Clark & Chalmers, 1998), Clark (2008, subsection 6.7; 2013) sets out to prove that gesture can function as part of the actual thinking process. As he sees it, gesture functions much like a car turbocharger — the device that increases the power output of a car engine by injecting air compressed by a turbine driven by the car’s exhaust fumes. According to Clark, humans use gestures to supercharge their minds, not unlike turbo-driven engines. After all, both involve a systemic output as well as a self-generated input that plays an imperative role in moving whichever process along. Seeing how no one would hesitate to call the turbocharger part of an engine’s power-production process, Clark sees no reason to do otherwise when dealing with gestures and their role in the production of thoughts. Ultimately, then, it is in their execution of cognitive self-­ stimulation that Clark identifies the mark of the cognitive. As it turns out, the mechanism behind this cognitive self-stimulation has been described in lucid detail by scholars involved in gesture studies. According to Jürgen Streeck (2009, Chap. 7), gesturing while speaking involves the creation of concrete content through the enaction of manual schemata. These enactive representations sit somewhere between the preconceptual level of pure, unschematized experience of the concrete world and the conceptual level of abstract thinking. Streeck calls this gestural kind of construing or conceptualizing meaning ceiving. Gestural conceptualization, or ception as he calls it, essentially bestows a person engaged in a communicative interaction with an embodied and enactive way of organizing and representing meaning, which endows verbally expressed concepts with imagistic and sensory qualities. Drawing from Johnson’s (1987) work on image schemata, Streeck argues that speakers rely unwittingly on their bodies when trying to structure concepts. The propensity for doing so can be explained if we consider that hands have abstracted these schemas from their visual and kinesthetic experience of daily encounters with specific objects. It is through our active engagement with actual things that humans come to realize how the degree of specificity required for conveying meaning through gestures is less than that required for practical, idiosyncratic tasks. Streeck (2009, p. 163) points out that we do not need to be as specific in terms of hand-posture and applied forces when configuring our hand to the shape required to represent grasping as when we are preparing to grasp a particularly-­sized, −shaped, and -weighted object. As shown by experimental studies, this degree of schematization is sufficiently capable of driving intra-psychological as well as inter-psychological processes. This means that gestures involve internal kinesthetic patterns that help one think (or speak) while at the same time acting as an external visual fact for interlocutors. It is in this sense, according to Streeck, that gesture is capable of linking the internal and the external domains. Interestingly, these personal and intersubjective aspects of gesturing are elucidated further in a book chapter that Streeck co-authored with Elena Cuffari. Building on the work of Merleau-Ponty (1962), Cuffari and Streeck (2017) propose that the

136

A. Iliopoulos and L. Malafouris

meaning of gesture depends on an intercorporeal mechanism of sense-making that they call appropriative disclosure. According to this two-tiered principle, gestures select and appropriate elements of the environment while also disclosing new information to the world by manipulating it. As the authors point out: “Hand gestures both fit the world at hand and form it” (p. 176, emphases in the original). It is the bidirectional relationship between fitting and forming that underpins the interplay between the creativity involved in forming the world, and the conventionality entailed in fitting it. Inspired by Merleau-Ponty, Cuffari and Streeck argue that hand gestures signify through the enactment of a spontaneity-sedimentation dialectic. It is important to note here that besides being directed towards objects in the world, gestures also select and modify features of the world for someone, whether it be one’s self or another person. Essentially, then, besides being expressive movements of a body, gestures are also communicatively intentional. While the gesturer keeps track of the meaning expressed through his or her kinaesthetic sense, meaning is, as we have already seen, also rendered visible to his or her interlocutors. According to Cuffari and Streeck, it is precisely this combination of appropriative disclosure and intercorporeality that makes hand gestures intersubjectively meaningful. They show this by looking at a recorded interaction between a car shop owner and a welder while the two were working on a car. Situated towards each other and the car, the two try to find and address engine problems. They are, in other words, trying to arrive at the best course of action by way of both appropriating information (i.e., using hands to inspect a component of the engine) and disclosing it (i.e., using hands to point towards a part to be fixed). Of course, prior knowledge plays an important role in this process. It is only logical that “[h]ow much is disclosed by such an act is contingent on what the other party knows” (p. 184). The issue of how gestures can disclose things that depend on past experience is one that has drawn the attention of Massimiliano Cappuccio and Stephen Shepherd. As they specify, some gestures create a field of joint attention that contains all that is being pointed at, while others rely on something more, previous knowledge (Cappuccio & Shepherd, 2013). For them, the former kind of joint attention is basic, whereas the latter is symbolic. What differentiates basic joint attention from the symbolic variety is the fact that the former requires that the thing being pointed to reveals in and of itself all that is intended by the person pointing, whereas symbolic joint attention requires that the coattender be privy to some belief about the object that the pointer knows they are. In other words, pointing gestures can sustain symbolic joint attention only if all coattenders are aware of sharing common ground. This does not mean that this realization comes from mentalizing the thoughts of others through introspection and projection (as per the representational model of cognition) because the lack of overt comparison precludes them from sharing a joint focus of attention. One might thus be inclined to remedy this lack of reciprocity through a more dispositional approach that relies on direct perception of affordances and execution of script-like heuristics (as per the radically enactive approach). This approach, though, would be lacking elsewhere. According to Cappuccio and Shepherd, the dispositional approach lacks reflectivity because it cannot really explain how the objects being pointed towards can cue the cognizance of something

Towards an Ecology of Gesture: A Review (And Some Promising Paths)

137

absent that everyone involved is aware of knowing and, even more importantly, aware of sharing. Cappuccio and Shepherd (2013) thus arrive at a third option that entails complementing the reflectivity of the representational model with the reciprocity of the dispositional approach. They specifically suggest that the capacity for symbolic joint attention is the eventual outcome of declarative pointing taking place in well-­ defined communities of actors. As they point out, declarative pointing helps establish basic joint attention in a way that distinguishes it from other means (e.g., gaze-following) because it presents a physically embodied token of joint attention. By deliberately opening up a particular perceptual context for coordinated and collaborative examination, the gesture enables coattenders to become cognizant of the fact that they are both aware of the social situation they are in, which is why pointing can be seen as an externalized representation of joint attention. In effect, joint attention is both produced and symbolized by declarative pointing. As might be expected, expression in the company of other members of one’s community affords coattenders the opportunity to entertain the possibility that the pointer may be pointing towards imaginative content about that thing, which she expects them to share due to participating in a “mutual world”. This situation essentially encourages coattenders to reflect on their own state of awareness in order to determine what it is that they may be sharing with the person that made their thought publicly available. To put it simply, humans find themselves thinking about their own joint thoughts. Evidently, then, the representational and meta-representational abilities involved in some varieties of gestures emerge along with the symbolic variety of joint attention. The multimodal semiotic expression we find in coverbal gesture has been actually described in great detail by Irene Mittelberg. In seeing gestural sign formation from a Peircean point of view, Mittelberg (2013) goes beyond, or to be more accurate, before the symbolic, back to the non-conventional. As she points out, understanding our interlocutors requires attending to the similarity between their gestures and our own personal experiences and routines, as well as attending to the contiguity between the gesturer and the material and social world. Clearly, then, being able to appreciate the non-symbolic and non-conventional aspects of multimodal sign processes means attuning our framework towards iconicity and indexicality. However, not all icons are the same, and neither are all indices. Icons can be images, diagrams, or metaphors, while indices can rely on the contiguity between a part and the whole (inner contiguity) or the contiguity between two distinct things that are in contact, adjacent, or impact one another (outer contiguity). To differentiate between the subtypes of iconic gestures, as well as between the kinds of contiguity between gestures and what they stand for, Mittelberg draws on the triad of Peirce’s hypoicons and Jakobson’s use of metaphor and metonymy. Bringing these analytical tools together allows her to establish an iconicity-indexicality continuum, upon which a number of different gestures are identified, with some falling closer to the iconic end of the spectrum and others exhibiting stronger indexical grounds. By attending to the interplay between iconicity, indexicality, metaphor and metonymy, Mittelberg is thus able to illuminate the semiotic forces at play beyond the symbolic and conventional sign systems that guide language and sign language gestures. There is

138

A. Iliopoulos and L. Malafouris

much we can learn through these cognitive-semiotic principles about the schemata and forces motivating the form features and the pragmatic functions of gesture. Attending to the semiotic intricacies of embodied image schemata and force dynamics, she claims, allows us to appreciate how gestures are able to mediate between physical experience and abstract thought, literal and metaphoric meaning. Inherent in our communicative gestures, we find a motivated semiotic structure, which Mittelberg frames as the manifestation of an ex-bodied mind, an embodied mind directed outwards. Extending the notion of embodied mind, she calls us to see the expression of mental images, conceptual structures, action patterns, and felt qualities through the medium of the body as an act of ex-bodiment. In linking, then, the affective, the intersubjective, and everything in between, bodily semiotics allow us to evaluate and understand the multimodal gestural and corporeal signs of others through recourse to our own habits and history, of course. The reflective social consciousness emerging in occurrent perceptual contexts has also been a matter of concern for an anthropological line of thinking known as distributed cognition (Hutchins, 1995; Goodwin, 2018; see Williams, 2013 for a review). In a field where language had been given preferential treatment when studying face-to-face communication, Edwin Hutchins and Leysia Palen managed to demonstrate that gesture and space are equal counterparts of speech in this multimodal process of interaction. Rather than treating them as context for the interpretation of speech, they relied on an interaction that took place in a flight simulator in order to demonstrate how complex multi-layered representations are constructed through the combination of space, gesture, and speech (Hutchins & Palen, 1997). In the case presented, we see the second officer informing the pilot and the first officer about a fuel leak in one of the aircraft’s tanks. In following him brief his colleagues about the problem and how he came about to determine its cause, we learn that he relies extensively on gestures that were closely coupled with the spatial organization of the fuel panel. As it turns out, the components of the aircraft are arranged on the panel in the same way they are actually arranged on the aircraft, making the panel meaningful in and of itself. What makes this case particularly interesting for our purposes, though, is that the positioning of the officer’s gestures was, in a sense, directed by the physical layout of this specialized artefact. As he was sharing his concerns with his colleagues, his hands were performing an idealized set of movements that might have been followed. It would be fair to say then that “[t]he gestures acquire their meaning by virtue of being superimposed on the meaningful spatial layout of the fuel control panel. The same gestures produced in the absence of the panel would, of course, be quite meaningless” (ibid. pp. 37–38). The fuel panel had to be there, standing for itself, if a gesture were to be conceived as a representation. To describe the body movement that, besides relying on a gesturing hand and a speaking mouth, also depends on the participation of material and graphic structure in the environment, Charles Goodwin (2003) introduced the term symbiotic gesture. In order to appreciate the lodging of symbiotic gestures within both human interaction and a structured environment, let us consider a telling case described by Goodwin himself. In a video sequence recorded at an archaeological field school,

Towards an Ecology of Gesture: A Review (And Some Promising Paths)

139

we see how a senior archaeologist helps a young graduate student identify an important feature found in the soil by tracing its outline over the dirt. As is pointed out, the student would not have been able to make sense of the gesture had it not been for the colour differences marking the presence of a post mold. The symbiotic nature of the action complex seen here rests in the fact that not only does a finger indicate the relevant graphic structure, but the structure also organizes the location and trajectory to be followed by the gesture. While these gestures do not alter the physical environment in any way, shape, or form, some hand movements actually manage to change the world being talked about by way of — what Goodwin calls — inscription. Returning to his example, Goodwin describes how the student lowers her trowel into the dirt and produces a permanent outline by tracing the boundaries of the feature she had earlier identified through her symbiotic gesture. Following the director’s confirmation of the gesture’s accuracy, the student essentially transduces the shape of her gesture from the medium provided by her moving hand into that provided by the dirt itself. Seeing how this process of highlighting and describing is not something entirely different but a modification of the original gesture, Goodwin prefers to see these two events as points in a continuum ranging from iconic gestures (which do not invoke the environmental surround) to symbiotic gestures (wherein external structures are taken into account when representing), and finally to inscriptive gestures (that actually effect changes upon the things being represented). It, therefore, seems that gestures can enable thinking about and acting upon the world in different ways and to varying degrees, with the symbiotic variety arguably occupying a middle ground in terms of the environment’s participatory role. A particularly interesting variety of symbiotic gesture is the musical gesture. While the body movements of singers and pianists have been mostly studied from the audience’s point of view (e.g., Davidson, 1994, 1995), recent studies have tried to account for the performer’s perspective (e.g., Doğantan-Dack, 2011). Seeing how we have been thus far considering gestures from the gesturer’s point of view, we would like to remain on track and highlight the importance of gesture in musical performance. While musical theory and listeners tend to conceive music as a work primarily based on rhythm and pitch, Mine Doğantan-Dack (2011) delves into the importance of timbre in order to elucidate how tone colour is affected by the body’s movement. As she tells us, the physicality of the performer’s engagement with the instrument is inextricably tied to the acoustic profile of the resulting sound. In the case of playing the piano, one would assume that there is little that can be done to affect the sound, apart from influencing its intensity, of course, because movement is relatively confined to depressing standard sets of keys. Yet despite the restrictions posed by the instruments themselves, expert pianists have been found to produce different sonic profiles. Doğantan-Dack identifies the source of these differences in their gestures. As she points out, the performers’ hands tend to assume form before their fingers touch the keys because they can anticipate what comes next. Knowledge of how a note is to be stricken comes in the form of a unified Gestalt that involves the coupling of a specific body movement and a particular acoustic profile. Drawing on kinaesthetic images about the link between gestures and resulting tones, pianists are able to experience a sound, mentally and physically, before their hands actually

140

A. Iliopoulos and L. Malafouris

touch the keys. They are thus able to position their body accordingly so as to hit the keys in just the way required to produce the ‘right’ sound and, by extension, effect. Apparently, the timbral identity of a musical performance owes much to the initiatory gestures taking place in between the striking of the keys. In order to better appreciate how performative habits are established through the progressive determination of the vague meanings gestures initially carry, let us finally turn to what Giovanni Maddalena (2015) calls a complete gesture. The main idea in his recent monograph on The Philosophy of Gesture is that gestures allow for the synthetic reasoning that leads us to the recognition of identity through change. Moving beyond the usual conception of gestures, he uses the term as a tool for non-­ analytic reasoning in order to refer to any action that enables one to synthesize knowledge about the identity of an object. The most balanced and complete of these gestures are the ones that rely on all three of Peirce’s phenomenological and semiotic relations in equal part. From a phenomenological point of view, a complete gesture implies a quality or feeling (i.e., a First), a physical act involving a reaction between at least two objects or subjects (i.e., a Second), and a generality stemming from such acts and guiding subsequent ones (i.e., a Third). Starting from the vagueness of pure possibilities (Firstness), a form, idea, or feeling becomes determined through an actual fact (Secondness) in such a way that it tends towards becoming a necessity (Thirdness). When the blending of these phenomenological aspects is at its densest, then the gesture’s power of introducing novelty into knowledge is at its highest, most perfect state. Only then do we have a complete gesture. Semiotically speaking, in order for a gesture to be considered complete, different possibilities of forms and feelings (Icons) need to be expressed in acts that indicate their singular objects (Indices) as replicas generated by general laws (Symbols). Complete gestures are: creative due to the possible forms and feelings they express, singular and unrepeatable due to their individuality, and recognizable due to their unity and conformity to general patterns that the gestures themselves tend to realize. As would be expected, then, complete gestures depend on an equal blending of all three sign types. It is through equally dense blends of the above sorts that gestures are able to perform a creative function. According to Maddalena, in order to move from the vague state of potential ideas to the general state of patterned style and meaning, three conditions must be in place. For one, creativity needs to be primed by a creator or, to be more specific, a sub-creator. The reason behind the prefix is that humans are not really capable of creating or destroying existence — they simply reshape existent matter. While they can only use existence (Secondness) in a derivative way, they can, however, manipulate the other two realms (Firstness and Thirdness) to a great extent. To create novelty in these domains, assent is also necessary because awareness or self-control is implied in complete gestures. In order to come closer to “completion”, Maddalena argues, the gesture must be knowingly endorsed by its author. And, of course, judging whether a gesture is creative in itself requires judging whether it adheres to the admirable ideal that one wants to convey. From an aesthetic point of view, a gesture is deemed to be worthwhile if it is reflexively

Towards an Ecology of Gesture: A Review (And Some Promising Paths)

141

found to embody and enhance the admirable order it tends towards when generalizing the possibility or, better yet, the plausibility harboured by the real world. Normative appeal, then, is the third and final of Maddalena’s conditions for creative gesture.

3 Future Directions The insights yielded by the ‘ecological’ line of thinking are, as we saw, quite rich. As it seems, cognition may be shaped, completed, and even turbo-charged through gestures. Gesturing is specifically involved in the creation of concrete content through the enaction of manual schemata. Yet besides selecting and appropriating elements of the environment, gestures also disclose new information to the world, which is why they have been associated with a mechanism of appropriative disclosure. As it appears, the extent of what can be disclosed depends on whether gesturers possess knowledge they are aware of sharing. Pointing, for instance, may denote a thing or event that can be readily perceived but may also involve invoking prior knowledge about whatever is being pointed at. That formalized conventionalization can drive gesture is unquestionable. But as we realized, cognitive-semiotic principles such as iconicity, indexicality, metaphor and metonymy are integral in shaping the schemas and forces that motivate gestural form and function. Besides attending to the motivated semiotic structures expressed through coverbal gestures, we also learned that the success of gesture might also depend on its coupling with spatially organized structures, such as fuel panels, post molds, and piano keys. Our overview was finally concluded by touching on the creative variety of gesture, which was shown to be the most complete, both in phenomenological and semiotic terms. As we saw, a sub-creator, assent, and normative appeal are all conditions required for creative gestures. So where does all this leave us? We think it is safe to say that gesturing is an inherently performative phenomenon directed towards habit, yet still open to change. This is why it needs to be studied from a theoretical perspective that prioritizes praxis over essence and process over substance. One such framework can be provided by the Material Engagement Theory (MET), an evolutionary epistemology that gives ontological priority to the act of material engagement (Malafouris, 2013). Influenced by the anti-Cartesian spirit of contemporary theories of mind, the theory of ‘material engagement’ treats cognition as a situated phenomenon emerging from the integration of brains, bodies, and things. There is no meaning prior to its manifestation. As we see it, gestures are the medium through which objects and meanings are brought forth. However, for MET, the intention to create something new is not actually a matter of premeditated thought. Intention is instead the emergent product of our engagement with particular things. The different affordances that things present us with, along with the level of skills we possess, are key factors in shaping the directionality and meaning of our gestures.

142

A. Iliopoulos and L. Malafouris

That being said, gestures differ from other kinds of bodily movement in that they express a subjectivity, which Vilém Flusser (2014, p. 163) feels forced to call “freedom.” Gestures are the phenomenal and visible expressions of this freedom. As such, they are bound to come with a dialectical tension between objective information and meaning, which is why Flusser held that a general theory of gestures “would be a study of meaning, a semiology” (ibid.). It seems that Peircean semiotics could play quite an important role in this regard, especially if applied in a way that goes beyond the classic distinction between icons, indices, and symbols (i.e., a distinction concerning the representative function of a sign). A recent book chapter by Mittelberg (2019) offers an exemplary illustration of how Peirce’s categories (i.e., Firstness, Secondness, Thirdness) can be used to study the presentative and the interpretive nature of a gestural sign — that is, the character of the sign itself, as well as the effects it has on the mind. Like Maddalena, she aptly demonstrates how Peirce’s categories allow us to move from the qualitative characteristics of gestures to their experiential and contextual manifestation, and up to the eventual establishment of personal and cultural habits. The use of pragmatic semiotics, then, should prove especially useful for the study of gestures, especially if informed by contemporary theories in the philosophy of mind. Seeing how cognitive semiotics have already yielded important insights in the study of co-verbal gestures, we are led to believe that a similar approach might also help the study of creative gestures, much like the ones involved in handmaking. Combining, in particular, MET with Peircean semiotics should be able to tell us more about gestures and artefacts, especially with regard to the creative process itself.

References Cappuccio, M. L., & Shepherd, S. V. (2013). Pointing hand: Joint attention and embodied symbols. In Z. Radman (Ed.), The hand, an organ of the mind: What the manual tells the mental (pp. 303–326). MIT Press. Clark, A. (2008). Supersizing the mind: Embodiment, action, and cognitive extension. Oxford University Press. Clark, A. (2013). Gesture as thought. In Z. Radman (Ed.), The hand, an organ of the mind: What the manual tells the mental (pp. 255–268). MIT Press. Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58, 7–19. Cuffari, E., & Streeck, J. (2017). Taking the world by hand: How (some) gestures mean. In C. Meyer, J. Streeck, & J. Scott Jordan (Eds.), Intercorporeality: Emerging socialities in interaction (pp. 173–201). Oxford University Press. Davidson, J. W. (1994). Which areas of the pianist’s body convey information about expressive intention to an audience? Journal of Human Movement Studies, 26, 279–301. Davidson, J. W. (1995). What does the visual information contained in music performances offer the observer? Some preliminary thoughts. In R. Steinberg (Ed.), Music and the mind machine: The psychophysiology and psychopathology of the sense of music (pp. 105–113). Springer. Doğantan-Dack, M. (2011). In the beginning was gesture: Piano touch and the phenomenology of the performing body. In E. King & A. Gritten (Eds.), New perspectives on music and gesture (pp. 243–265). Routledge.

Towards an Ecology of Gesture: A Review (And Some Promising Paths)

143

Flusser, V. (2014). Gestures (Nancy Ann Roth, Trans.). University of Minnesota Press. Gallagher, S. (2005). How the body shapes the mind. Oxford University Press. Gallagher, S. (2017). Enactivist interventions: Rethinking the mind. Oxford University Press. Goldin-Meadow, S. (2017). Using our hands to change our minds. WIREs Cognitive Science, 8(1–2), e1368. Goldin-Meadow, S., & Wagner, S. (2005). How our hands help us learn. Trends in Cognitive Sciences, 9(5), 234–241. Goodwin, C. (2003). The body in action. In J. Coupland & R. Gwyn (Eds.), Discourse, the body, and identity (pp. 19–42). Palgrave/Macmillan. Goodwin, E. (2018). Co-operative action. Cambridge University Press. Hostetter, A.  B., & Alibali, M.  W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin & Review, 15, 495–514. Hostetter, A. B., & Alibali, M. W. (2019). Gesture as simulated action: Revisiting the framework. Psychonomic Bulletin & Review, 26(3), 721–752. Hutchins, E. (1995). Cognition in the wild. MIT Press. Hutchins, E., & Palen, L. (1997). Constructing meaning from space, gesture, and speech. In L. Resnick, R. Säljö, C. Pontecorvo, & B. Burge (Eds.), Discourse, tools and reasoning: Essays on situated cognition (pp. 23–40). Springer. Iliopoulos, A., & Garofoli, D. (2016). The material dimensions of cognition: reexamining the nature and emergence of the human mind. Quaternary International, 405(Part A), 1–7. Johnson, M. (1987). The body in the mind: The bodily basis of meaning. University of Chicago Press. Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and gesture (pp. 162–185). Cambridge University Press. Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245–266. Krauss, R., & Hadar, U. (1999). The role of speech-related arm/hand gestures in word retrieval. https://doi.org/10.1093/acprof:oso/9780198524519.003.0006. Krauss, R. M., Chen, Y., & Gottesman, R. F. (2000). Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture (pp. 261–283). Cambridge University Press. Leroi-Gouran, A. (1993 [1964]). Gesture and speech (Anna Bostock Berger, Trans.). MIT Press. Maddalena, G. (2015). The philosophy of gesture: Completing pragmatists’ incomplete revolution. McGill-Queen's University Press. Malafouris, L. (2013). How things shape the mind: A theory of material engagement. MIT Press. Malafouris, L. (2016). Material engagement and the embodied mind. In T. Wynn & F. L. Coolidge (Eds.), Cognitive models in Palaeolithic archaeology (pp. 69–87). Oxford University Press. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago Press. McNeill, D. (2016). Why we gesture: The surprising role of hand movements in communication. Cambridge University Press. Merleau-Ponty, M. (1962 [1945]). Phenomenology of perception (Colin Smith, Trans.). Routledge/ Kegan Paul. Mittelberg, I. (2013). The exbodied mind: Cognitive-semiotic principles as motivating forces in gesture. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body – Language – Communication: An international handbook on multimodality in human interaction. Volume 1 (pp. 755–784). De Gruyter Mouton. Mittelberg, I. (2019). Peirce’s universal categories: On their potential for gesture theory and multimodal analysis. Semiotica, 228, 193–222. Novack, M. A., & Goldin-Meadow, S. (2017). Gesture as representational action: A paper about function. Psychonomic Bulletin & Review, 24, 652–665. Pouw, W.  T. J.  L., de Nooijer, J.  A., van Gog, T., Zwaan, R.  A., & Paas, F. (2014). Toward a more embedded/extended perspective on the cognitive function of gestures. Frontiers in Psychology, 5, 359. Rauscher, F. H., Krauss, R. M., & Chen, Y. (1996). Gesture, speech and lexical access. The role of lexical movements in speech production. Psychological Science, 7, 226–231.

144

A. Iliopoulos and L. Malafouris

De Ruiter, J. P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and gesture (pp. 284–311). Cambridge University Press. Streeck, J. (2009). Gesturecraft: The Manu-facture of meaning. John Benjamins Publishing Company. Wesp, R., Hesse, J., Keutmann, D., & Wheaton, K. (2001). Gestures maintain spatial imagery. The American Journal of Psychology, 114(4), 591–600. Williams, R. F. (2013). Cognitive anthropology: Distributed cognition and gesture. In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (Eds.), Body – Language – Communication: An international handbook on multimodality in human interaction. Volume 1 (pp. 240–258). De Gruyter Mouton.

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements? Jürgen Richter and Thiemo Breyer

Abstract  Prehistoric stone tools allow for detailed insights into patterns of movements of the human body and the mobility of individuals and groups. As contextualized at the beginning of this chapter, such a behavioural perspective is rooted in so-called “Processual Archaeology”, which we imply as the starting point of our approach, devoted to deciphering vernacular choreographies casting the everyday life of early humans. Here, we use settlement locales on the Crimea peninsula in order to describe three scales of human mobility. All sites were occupied by the last Neanderthal humans, about 45,000 years ago. At a small scale, we reconstruct recipes of stone tool production with gestures of human arms, hands, and fingers, all within the kinesphere or kinaesthetic bodily space of an individual Neanderthal. The embodied performances of knapping are considered “body techniques” in the sense of Marcel Mauss. Two more scales are then dealt with in lesser detail: at an intermediate scale, a small campsite is considered as a choreographic “stage” for movements of the human body, walking, standing, and sitting down, all related to disassembling three animals hunted nearby and brought to the dwelling place. At a large scale, we compare a cluster of settlement sites that supposedly belonged to the same, seasonally geared mobility system, thus describing an annual itinerary of humans through their landscape. As a result, we envisage the option of understanding artefacts and objects as expressions of an ancient practice, geared by vernacular choreography. Keywords  Body techniques · chaîne opératoire · Practice theory · Processualism · Post-processualism · Stone tool making

J. Richter (*) Department of Prehistoric Archaeology, University of Cologne, Cologne, Germany e-mail: [email protected] T. Breyer Department of Philosophy, University of Cologne, Cologne, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_10

145

146

J. Richter and T. Breyer

1 Methodological Roots of an Archaeology of Ancient Behaviour The recording of movements has become a widely applied tool in modern archaeology. The transformation of artefacts and settlement contexts is an important topic that gave rise to the development of a whole array of methods devoted to tracing movements in an archaeological context. Looking back into the history of archaeological research, we can trace contrasting ways of archaeological reasoning that essentially centred around archaeological objects each as a singularity, and around a series of similar objects. For a long time, archaeologists dealt with their objects as entities with static attributes to be compared, classified, and located in time and space. However, as was the case in all sciences during the twentieth century, archaeology has undergone major methodological struggles that lead to rethinking what it means to interpret artefacts and other kinds of data.

1.1 General Scope: On the Role of Archaeology as an Interpreting Discipline Being one of the most prominent concepts for a variety of theoretical reflections in archaeology, interpretation is at the heart of the scientific endeavour altogether (Alexandri et al., 1995). Every scientific investigation needs to transcend its empirical basis in the direction of a theory, making sense of otherwise merely accumulated data. Natural sciences and the humanities do not differ in their dependence on interpretation, typically they rather differ in how much interpretation is needed in order to duly answer their respective questions – and how the levels of interpretation and data are then reconnected. While a greater extent of and requirement for interpretation does not imply methodological dubiousness per se, it indeed poses various problems. While the natural sciences may in general have stricter norms regarding the appropriate level of interpretation, archaeology constantly needs to decide upon its “hermeneutical” strategies, sometimes from case to case. The result of this balancing cannot simply be extracted from the subject matter. In fact, the overall idea of good practice within archaeology predetermines the interpretive approach to a large extent. Orienting as well as legitimising archaeological research, such general ideas have been intensely discussed in Anglo-American archaeology since the 1960s (Shanks, 2007). Even going back to 1948 and Taylor’s attack on cultural-historical archaeology – for which artefacts constitute the main class of material remnants that can be analysed (Bernbeck, 1997, 36) – the objects and limits of the archaeologist’s perspective dominated the reform debate.

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

147

1.2 On “Behavioural Archaeology” The production of descriptions and catalogues, the attribution of single objects to artefact types, and the mapping of the spatial distribution of artefact types for a long time ranked among the most crucial methods of archaeological reasoning (as, for instance, in the archaeo-geographic method advocated by Hans-Jürgen Eggers 2018). Against former self-limitation and speculation, processual archaeology claimed that by way of an integration of natural sciences and cultural anthropology, it would be possible to reconstruct the cultural processes leading to an adaptation of (early) humans to environmental and social changes. Also known as “behavioural archaeology”, processualism provides a theoretical framework that greatly extended the methodological inventory of archaeologists since the 1970s. One of the most influential representatives of this school, Lewis Binford, used ethnographic records of hunter-gatherer behaviour for developing new ideas in the analysis of the changes and processes connected with the movements and activities of ancient human agents, carried out in and between settlement sites. For the first time in archaeology, spatial analysis, geographic information systems, ecological simulations, and other methods allowed studying concepts such as foraging and carrying capacity in depth. Archaeology’s view began to extend beyond the cultural-historical borders of classifying artefacts and the sites they were found at. If this expansion of the horizon is reflected upon more closely, it appears that it is ultimately agency that is the pinnacle of archaeological interest, since the import of scientific and cultural methods remains tied to the data concerning the individual artefacts. However, to frame culture exclusively from an evolutionary perspective as an extra-somatic adaptation on the one hand, and to solve the problem of the interpretation of facts generated by auxiliary sciences by simply pointing at the objectivity of those facts on the other hand, could not remain unchallenged for long. Once again consulting cultural anthropology, the evolutionist perspective and its ecological determinism was dismissed by post-processual archaeology as being far too simplistic. Instead, the atom of every social and cultural group was spotlighted: the individual. Not only its complex agency and resulting emergence of sociocultural practices was found to be over-simplified by all previous archaeology, the archaeologists themselves were now regarded as deprived of their neutral vantage point of observing and judging. Interpretation, according to post-processual theorists, can never be purified from the subjectivity of the ones who do the interpreting; whoever claims to have objective knowledge while narrating the past is covertly forcing their biases upon the recipients (Johnson, 1999, 103). While many opposed such relativism, only few would claim that post-processual archaeology had no point whatsoever. The fact that human agency had been omitted and culture presented in a not sufficiently complex way posed serious problems (Trigger, 1989). Nonetheless, the post-processual solutions tended to require plenty of information in order for their hermeneutics to work out: information that is rare, especially in palaeolithic contexts. While post-processual archaeology introduced state-of-the-art problems in methodology and theory of science, it did little to

148

J. Richter and T. Breyer

support archaeologists – particularly in the field of palaeolithic archaeology – with qualified solutions (Earle & Preucel, 1987). Once again, however, the view expanded, now incorporating even the individuality of today’s academic archaeologists and the structures of their research units, that is, the contemporary agency necessary to interpret past agency. As already stated, the behavioural paradigm introduced a framework that allowed for pushing the limits of interpretation towards the agency – which, in palaeolithic contexts, means first and foremost gestural agency  – “behind” the artefact in an arguably well-founded manner. This broadened horizon allows for tracing agency on different scales in space and time. If alterations of the structure of settlement surfaces are to be addressed, a theory is needed for every single archaeological site in order to explain the status of site preservation at the time of excavation. Since such theories are only concerned with singular sites, they are classified as theories of a medium range of validity or “middle-range theories”, according to Binford (1977). In constructing such theories, the effects of natural processes caused by natural agents (physical forces, animals, plants) must be separated from those caused by human agents, the first called “n-transformation” and the second called “c-transformation” in Schiffer’s (Schiffer, 1976) “transformation theory” approach. Consequently, archaeologists aiming at a “behavioural archaeology” had to integrate methods that were able to unravel the events that have contributed to a c-­transformation of the site. Traces of a c-transformation reflect the time span of human occupation of the settlement site when movements of humans realised actions following repeated behavioural patterns (habits and manners). Such movements and actions would have impacted the ways in which people used their settlement spaces and thus would become essential to the organisation of site layouts. Humans induced transformational processes into different scales of their environment. One can trace these processes and movements beginning with the smallest scale of single objects manipulated and altered by humans, to then – on the next level of scale – include particular activity zones of settlements, within and around dwellings, zones for garbage and discard inside and outside the settlements, and veils of discarded objects and materials resulting from many small movements and single activities around the settlements. The traffic between settlements and other places would yield the next level of scale, as documented, for instance, by exotic raw materials imported into one settlement from a distant source or from previous settlements or hunting halts, often indicating the use of the same tools over a variety of sites connected by annual mobility cycles across group territories. On yet another level, on can eventually observe expanding and shrinking territories or shifts of entire territories from one to the next habitat, effecting regional migration patterns and routes. At a global level, such regional trends had an impact on the dispersal of human populations from Africa to Eurasia, Australia, and the Americas.

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

149

2 Reading the Traces of Ancient Practices 2.1 Lower Scales of Practice: A Choreography of the Human Limbs At a basic level of observation, movements are operationalized through gestures applied to, and reflected by, single objects. Such micro-moves might, for example, facilitate the production and subsequent utilisation of stone tools. At this level, movements of legs, arms, hands, and fingers are executed; furthermore, bodily habits of standing, bending down, sitting, and kneeling become apparent. Movement is restricted, at this level, to the three-dimensional space that is accessible for the human body with its extremities, called the kinesphere by choreographers (Postuwka, 1992). Originally developed by Rudolf Laban, the concept of kinesphere refers to “the sphere around the body whose periphery can be reached by easily extended limbs without stepping away from that place which is the point of support when standing on one foot” (Laban, 2011 [1966], 10). This imagined sphere around the person is mainly used in dance and theatre to symbolise the individual space of the artist, who is its embodied centre of orientation. More precisely, the kinesphere can be divided into three planes: horizontal, vertical, and sagittal. In principle, it is composed of six main directions (front, back, left, right, top, bottom). The kinesphere surrounds the human body in such a way that it extends the options and possibilities of in situ practice to an invisible outer surface or “separation plane” (Trennfläche in German) through kinaesthetic activity. The separation plane delineates the capacity for direct physical impact of the body, limiting human influence insofar as it is based on stationary movement. As demonstrated by the evident similarity between objects from the same prehistoric contexts, these movements followed conventions and recipes shared by the members of a social group – summarised as “practice” –, as well as the conditions and properties imposed by the natural and cultural context of the material object to be subject to human craft – the latter context summarised as the set of “affordances” of an object (Gibson, 1979; cf. Hussain & Will, 2020). The affordances provoke, channel, and reflect human practice applied to the same object. Accordingly, the production of a stone tool must follow the rules and the logical sequence in which the rules have to be applied, given the material substance of the object. The sequence of necessary action must be learned and the optimal power and precision of movements must be trained. Such training, executed over years, must have changed the physical properties of human bodies, such as neuronal patterns and muscular abilities. Interestingly, movements of the hands and fingers of a human agent are, archaeologically speaking, best documented in Palaeolithic objects – rather than in objects of any later period – because these were fully handmade in a manner that allows tracing every single operational step leading to the finished tool. A Palaeolithic stone tool (Fig.  1) preserves all stages of production and use represented by the

150

J. Richter and T. Breyer

Fig. 1  Result of the analysis of operational steps (chaîne opératoire) for a triangular plano-convex foliate point (Buran-Kaya/Crimea, Level B 1–2, Square B8, no.12, depth − 2.13 m). To the left: conventional drawing: obverse and reverse of a tool, all single flake negatives indicated, shading according to concavity and direction of flaking impact. To the right: flake negatives of the same direction and function are put together: these are interpreted as single working steps, each comprising one or more negatives of the same nature. All working steps are separately labelled according to their position on the upper (O1…) and lower (U1…) side of the artefact. The chronology (sequence) of working steps is documented in a separate database; results, along with the functional interpretation of each working step in the reconstructed recipe, are indicated below (image from Richter, 2001; see Table 1 below for details)

Table 1  Actions connected to surface areas of the object Buran-Kaya Lev. B1-2/B8/No.12 Initial alteration of the original work piece U2: U5: 02: Retouching U4: U21: U51: 021: 04: Thinning and tool use and break U3: 03: 022: 05: Thinning and tool use U41: 051: U1:

Flat surface shaping (lower side) Flat surface shaping (lower side, left edge) Convex surface shaping (upper side, right edge) Flat retouch (lower side, right edge) Flat retouch (lower side, left edge) Flat retouch (lower side, right edge) Flat retouch (upper side, right edge) Flat retouch (upper side, left edge) Preparation for 03 (lower side, basal) Thinning (upper side, basal) Use retouch (upper side, right edge) Break (upper side, left edge, longitudinal) Use retouch (lower side, right edge) Thinning (upper side, left edge) Splintering from break (upper side, distal)

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

151

negative shapes of detached flakes that splintered off during the process of shaping and reworking (cf. Richter, 2001). By contrast, blades of Neolithic adzes were polished after the flaking process, the polish thereby erasing all traces of previous work. And finally, blades of Bronze Age adzes were cast; therefore, the production process is reconstructed by archaeologists by taking into account all relevant remnants, such as stoves, slags, casting molds, etc. Logical combination and experimental repetition, as performed by archaeologists, facilitate the reconstruction of the original chain of operations along the principles of “reciprocal engineering”. Reciprocal engineering must start with a thorough description of the finished object and continues with a stepwise, backwards imagination of former stages of the object. Here, single steps of human action leading to the production of artefacts are typically not directly accessible, but can be inferentially constructed by what could be called “retrospective prediction”. The experimental reproduction of things (experimental archaeology) can help to test predictions about the processes imagined “behind” the things, as it were. Reciprocal engineering can also be performed with Palaeolithic artefacts such as hand-axes, however – given the preservation of flake negatives mentioned above – those provide an additional, empirical way of directly tracing single human actions and movements within the process of production. The intersection of neighbouring negatives allows for setting up a chronology of working steps. In other words, Palaeolithic artefacts can be interpreted in a way that recipes of production and use are reconstructed. The comparison of such recipes has uncovered repetitive patterns, rules, and conventions channelling the modes of production and manipulation. Nowadays, these modes are known in more detail when it comes to Neanderthals than to Bronze Age modern humans. The recorded working steps indicate precision movements of an invisible human agent, carried out with arms, hands, and fingers, precisely targeting spaces of only a few millimetres in diameter and distance on the artefact’s surface.

2.2 Narrative Based on the chaîne opératoire Analysis of Object Buran-Kaya Lev. B1-2/B8/No.12 On the Crimea peninsula, 50,000  years ago, a Neanderthal individual plans to replace a blunt cutting edge with a new sharp working edge to be hafted in a wooden handle. For this purpose, she selects a flat, ovoid raw material piece of the right size (300 g) from a selection of imported flint nodules or thick stone flakes (in this case, 100 g) that were previously produced. The individual takes the stone piece in her hands, turning it around and looking at it from all sides in search of the best spot for the exact placement of the initial hammer impact. Many variables must be taken into consideration. A decision is made to place the first hit onto the surface of the piece. The individual takes the stone piece in her left hand while sitting down in a tailor’s position with her legs crossed, the left hand with the stone piece placed on the left knee.

152

J. Richter and T. Breyer

(a) If the stone piece is a nodule, then the right hand picks up a stone hammer of globular shape and 300–500 g of weight. The stone hammer consists of crystalline rock (quartzite), was collected from a choice of river pebbles many days or weeks before, and the user maintained it since, keeping it always at hand. The right arm with the hammer stone in hand is now lifted (up to a culmination point of 20–30 cm distance from the nodule) to be subsequently moved back in a very quick manner towards the nodule, targeting it at the intended point of impact. The moment the hammer stone touches the nodule is marked by a dry noise (a crackling noise without resonance). The right noise makes the stone knapper happy and motivates her to continue with the task. This is repeated several times, proceeding on the alternating sides of a portion of the nodule, thus partially removing the chalky cortex of the nodule and at the same time producing a narrow-angled edge. The edge is consequently refined with light impact moves and hits with the same hammer stone, with a width of the arm movement of only 5–10 cm, back and forth. This is all done in order to prepare an optimal starting point for the next steps of the operation chain. As soon as a line of 5–6 cm has been completed, the stone hammer is deposited in a pocket with the stone knapper. (b) If the stone piece to be worked out is not a nodule, but a thick flake, preparation with a stone hammer is not required. As a next step, the Neanderthal now picks up a wooden hammer. The wooden hammer has been carefully selected long before and has been cut to the right size of some centimetres in diameter and a length of 20 cm, all around smoothed to the final weight of approximately 300 g. The material used for this wooden hammer would have been hard wood. The wooden hammer is of great material value and has been maintained by the stone knapper for many weeks or months. It is now to be applied to the target piece in order to produce its intended shape. First, the flat lower side is shaped by repeated hitting of the prepared edge of the piece (U and U5 attest to this working step). In doing so, the left arm is lifted up to a culmination point of 20–30 cm distance from the target piece, then moving back the wooden hammer with great velocity to arrive at the point of impact with the right angle and with great spatial precision. This is repeated 5–6 times, to be continued at the opposite edge of the target piece. Moving forward, it might be necessary to prepare a portion of the new edge with the stone hammer, which is reactivated in such a moment. After shaping the lower side of the piece in this fashion, the same sequence of action is repeated to shape the upper side of the piece (O2 attesting to this working step). The activities of the flintknapper stop here for the purpose of evaluating the achieved shape of the piece. The knapper decides to accept or discard it, depending on whether the piece fits into the handle. At this point, the original workpiece has been turned into a blank, which is now available for further refinement. The wooden hammer is then put aside. The Neanderthal now takes a new napping instrument in her right hand: a bone retoucher made of a medial part of a long bone of a medium size mammal such as a saiga antelope. She holds the target piece in her left hand, just as before. The bone

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

153

instrument is tangentially applied to the intended working edge of the blank. The right arm is held in a fixed position, the gesture thus being restricted to the right hand. The gesture of the right hand of the knapper describes a segment of a circle, instead of a straight line, now moving back and forth within a short distance of 10–15 cm, as measured from the culmination point of the right hand’s movement to the target point of impact. This point is not reached by straight percussion as in the previous process, but by tangential stroking of the sharp edge of the target piece with the smooth surface of the bone piece. This is repeated about ten times. Once finished, the knapper has to check whether the working edge has gained more stability than before, at the same time maintaining the same sharpness of the edge (these working steps are marked as U4: lower side, right edge; U21: lower side, left edge; U51: lower side, right edge, 021: upper side, right edge, 04: upper side, left edge). The stone implement is now functionally completed and ready for use, the acting individual decides to insert the stone element into a wooden handle, which she had previously prepared.1 In order to insert the stone implement into the wooden handle, the agent takes the wooden hammer again in her right hand, detaching a flat flake (O3) from the surface of the target piece to adjust the thickness of the piece to the width of the hafting slot of the handle. Consequently, she inserts the stone implement (with its O3/U3 corner) into the slot, fixing it on the wooden handle. The scraping and cutting tool, as a composite wood-and-stone tool, is now finished. The individual now realises that her scraping and cutting work wears down the working edge of the stone insert (U41). After some time, a damage occurs close to the handle-insert interface (O5). The individual removes the insert from the handle. At the spot of the break damage, she repairs the blunt edge by a particular thinning blow, using the wooden hammer (U51). After repairing the edge, she decides against placing the stone insert back into the hafting slot in the same way as before, in favour of using this part as the working edge. She turns the stone element around and uses the stone implement for cutting and scraping (U41). At this stage, there is no information about the position of the handle to answer the question whether it was on the opposite side now, or if it was totally neglected. After some time, the point was broken (U1) and the actor discarded the piece.

2.3 Intermediate Scales of Practice: A Choreography of the Human Body Medium-scale surfaces comprise prehistoric hunting halts, activity spots, and campsites of usually 2–20 m maximum extension. The observation of medium scales of practice profits from the good preservation of occupation surfaces left behind by  The handle consists of a shaft with a connecting device designed to fix the stone element in the handle. Concerning the present piece, this was achieved by inserting the stone element into a slot cut into one edge of the handle. This is indicated by the thinning episode (O3) articulated within the operational chain of the piece. 1

154

J. Richter and T. Breyer

stone age hunter-gatherers. During glacial periods, soil formation and erosion processes were occasionally less prominent than in warmer periods of climate history. This more frequently led to the good preservation of human occupation floors in some sites of the Palaeolithic age compared to those of any later prehistoric period. Modern excavations uncovered many such Palaeolithic occupation surfaces, with every single stone artefact found exactly at the same place where prehistoric humans had discarded it. This allows for a reconstruction of places where people sat down, stood around, walked, and lay down on the occupation floor. Thus, many Palaeolithic occupation surfaces enable a reconstruction of ancient, vernacular choreographies of human movements. Such everyday movements were not hazardous and followed repeated patterns of conventional behaviour: everyday practice learned from companions – just like dances of unconscious nature. Archaeological site plans can be fruitfully compared with choreographers’ plans, displaying stage layouts, with hotspots of activity and corresponding passive zones, yielding a matrix of traffic within the limited space of the stage (Fig. 2). To introduce the concept of choreography, it is helpful to remember the etymological origin of the compositum. Linking the Greek terms chorós (dance or dancing place) and graphós (writing), a “choreography” constitutes the praxeological field of mapping out, (de-)scribing, and pre-delineating movements in space in a dance-­ like manner (cf. Huschka, 2021). The quality of dance here implies a skilful, habituated, and medially open set of practices. The movements in question are skilful because their execution does not consist in an automatic behavioural response to some requirement of the material objects or to cues from the environment, but rather as learned movements of a significant shape, which are part of a tradition within a community of practice. They are incorporated and trained over longer periods of time, whereby individual and collective habits are formed. Furthermore, the medial openness of these movement patterns indicates that actors participating in choreographies make use not only of their own bodies but also of all kinds of other media, such as artefacts or structures in the material surroundings. This interweavement forms a “stage”, as it were, on which – considered within the theoretical framework of practice theory as a flat ontology (cf. Schatzki, 2016) – there are multiple elements being interconnected in the choreography, without anyone having ontological superiority over the other, or foundational capacities, respectively. Finally, the scriptural aspect of choreography (graphós) does not need to be understood in a literal

Fig. 2  Spots of on-stage activity (left) and shapes of movement (right) as defined by choreographers. (Compiled after Postuwka 1992)

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

155

sense here, as a written prescription of how movements are to be carried out. Rather, the convivial and conventional wisdom of how to perform certain actions – adopted by imitation and learning – can be seen as functioning as a “script” people follow to achieve certain material results. However, this script has a degree of plasticity, as individuals are highly responsive to the material specificities of the objects they engage with. The script, therefore, is not a pre-given entity that never changes, but a template for “thinking through, with and about things” in embodied action, as one could say employing Malafouris’ concept of “thinging” in the context of stone tool making (cf. Malafouris, 2014, 2020). As an example of a prehistoric choreography, we show here an occupation floor (Fig. 3), which is 45,000 years old and which was produced by the last Neanderthal humans living on the Crimea peninsula, who were hunting wild horses available in great numbers in the surrounding Artemisia steppes. Based on a raw piece taken to the site by Neanderthals, which was found by archaeologists along with its descendant single chips and chunks, the chaîne opératoire analysis was applied and the resulting stages of production, use, and discard were made available for mapping them onto the site plan. Every single moment of the chaîne corresponds to particular places within the occupation surface, indicated by related discard. The plan shows the spatial distribution of all parts descendant from one single raw material nodule (Raw Material unit No. 10), beginning with importation (A), production of flakes (B, C), use as scrapers, and discard (D). The recorded pieces indicate movements of an invisible human agent, mostly limited to a space of 150 cm in diameter. This means that the actor took a sitting position, moving only the arms, hands, and the upper body. After mapping all other lithics and bones available from this surface, Guido Bataille concluded that the mentioned individual came along with two more human agents and with Equus hydruntinus (wild horse) hunting prey in order to dismember the animals’ body parts (Bataille, 2006).

2.4 Narrative Based on a Site Plan from Kabazi II, Level 7E, Crimea A Neanderthal enters the south-eastern corner of the occupation surface, coming up here from the Alma River valley. The individual has collected a flint stone river pebble for later use as raw material for stone tool production. She sits down very close to square 05, facing north-westwards. Movements now become limited to the upper extremities: she manipulates a stone hammer (held in the right hand) and begins to remove the cortex of the river pebble (held in the left hand), then transforming the raw piece into a core of conventional shape, designed to produce flakes, again of a predetermined shape. She discards some of the intermediate flakes on the site. Some of them, however, are thrown away to be later found at 150 cm distance from the point of use. She thoroughly selects two target flakes. She picks each of the flakes up to more finely rework their edges. She sharpens one of the edges to be used

156

J. Richter and T. Breyer

Fig. 3  Example of tracing movements by prehistoric humans on a single occupation surface. The plan displays 20 m2 of Archaeological Level II/7E (dated to around 45,000 years ago) from the Kabazi II palaeolithic open air site in Crimea. One of the longest palaeolithic sequences ever excavated, the Kabazi II site covers 70,000 years of human occupation documented in 76 archaeological levels. (Compiled after Bataille, 2006)

as a cutting edge, and prepares an opposite blunt edge, later to be fixed into a wooden handle. Having inserted the first scraper into the corresponding handle, she uses the scraper on the same spot for scraping hide or wood. After intensive use, she removes the scraper from its handle and discards the scraper. Then, she fixes the second

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

157

Fig. 4  Regional choreography: example of late Neanderthals’ repeated migration between campsites on the Crimea Peninsula around 45,000 years ago (settlement system of the Western Crimean Mousterian). The insert shows motivations and targets for human movements. (Chabai & Uthmeier, 2006)

scraper in the handle, again scraping hide or wood until the cutting edge becomes blunt. Again, she removes the scraper from the handle to keep the valuable handle for later use. Tools that have not been exploited so far are exported to distant campsites (Fig. 4).

2.5 Large Scales of Practice: A Choreography of Human Groups At the next level of scale, the cooperation between individuals and the interplay between human groups, animals, and landscapes comes into play. The large scale of practice comprises 25–80  km maximum dimension among Neanderthal and 25–200 km among Modern Humans. If dissemination of gifts or exchange goods were to be included, this would even reach up to 800 km, as is known for the transmission of shell ornaments in the Magdalenian societies from 15,000 years ago. A case study on Crimean Neanderthals (Fig. 4) displays Neanderthal humans commuting among activity spots and campsites in the landscape, where they gathered to form bands and split up into small groups within seasonal rhythms of movements. Here, human mobility has been reconstructed along artefact mobility, that is, the

158

J. Richter and T. Breyer

importation and exportation of lithic elements into and out of the archaeological locales can be used as a proxy of human movements. Type A locales are close to raw material sources (“outcrops”), where production outweighs consumption of stone artefacts. Type B locales are distant from raw material sources and consumption is dominating. Raw material and water availability as well as hunting opportunities were the principal factors constituting a fixed array channelling human everyday practices. These were consequently repeated many times in one and the same locale.

2.6 Narrative Based on Comparing Contemporaneous, Neighbouring Sites in Crimea On a sunny day in late August, a family band of Neanderthals had returned (after a macro-move) to one of their favourite resting places (camp, type A). Here, on a slope, some dozens of meters above the Alma River Valley, where a small step in the slope offers ample space due to the horizontal occupation surface and its position under an over-shading rock, people like to rest and enjoy a wide panoramic view. Staying here for some days, they spend their time repairing clothes, shoes, weapons, and all kinds of equipment. They fabricate stone tools, particularly at this place, because silex nodules are available nearby (micro-moves involved). On the first day, they furnish and eat some food either from animals previously hunted or from collected field fruit. Later on, maybe in the twilight of the early morning, they send two hunters out to acquire some fresh meat (micro-move involved). The hunters know a rock some hundred meters away which is optimal for ambush-hunting wild donkeys (station, type A). After killing a donkey, the two hunters return to their group with fresh meat on their shoulders (micro-move involved, back to camp, type A). The band spends some more days at the same place, which is so valued due to its ideal position, both close to raw materials needed for tools and close to hunting facilities. However, the band is quite aware that they should leave the productive locale soon when the autumn storms would come up and demand better shelter. The elders know a suitable retreat 30 km away (macro-move), and they are used to returning to this place every year at the same time. The place is cozy (camp, type B) because of its position in a small, deep valley, sheltering people from storms, rain, and snow. Meet and raw materials have to be brought in from some kilometres away (micro-­ moves), and for this sake, the group frequently sends out some members for hunting, collecting food, or producing raw materials (micro-moves with multiple stations, type B).

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

159

3 The Re-enactment of Ancient Practices There may be yet another link between artefacts and gestures, one that even establishes a connection between today’s corporeality and past practice. The emergence of processual archaeology as a critical and optimistic counter-movement to the cultural historical understanding of archaeological research – just like any major methodological shift – also marks a significant change in our ontological understanding of the objects of research, namely stone tools. While previously those artefacts were extensively described and catalogued from a morphological point of view, that is, essentially regarded as self-contained entities, the stone tool itself is now considered only as a remnant of a more comprehensive process. Not even the “finished” product of the manufacturing process can be regarded as untimely, insofar as it was subject to constant weathering, wear and tear, and reworking. Thus, the artefact itself is ultimately diachronised and in its processuality refers to all working steps as well as material remains of its creation. As shown above, the connection between stone tools and practice is proving to be a powerful vehicle for not only describing a major source of knowledge in prehistoric research in more detail through the means of re-construction, it even provides a basis for thinking together two dimensions that are closely linked: gestures and cognition. In general terms, the chaîne opératoire approach is an archaeological method of technological analysis, in which each individual lithic object of an inventory is used with its combined characteristics. The aim is to trace the logical sequence of the different stages of the operational chain from raw material procurement through basic mold production, and recycling to discarding. It was pioneered by archaeologists such as Leroi-Gourhan (1964), and has since received much attention and was applied for instance by Boëda et al. (1990), Geneste (1991), Inizan et al. (1999), and Soressi and Geneste (2011). As a holistic approach, it makes it possible to reconstruct the temporal sequence of the different productions, transformations, and steps of usage involved. On the other hand, it allows for an understanding of the spatial organization of the technological process. Through the individual artefacts or their technical stigmas (i.e., type and location of negatives, abrasion marks, or points of impact), the operation chain can be inferred. The presence or absence of the by-­ products of a technological phase, that is, a concrete stage within a specific mining concept, permits conclusions to be drawn about the handling of raw materials and/ or target products within a territory (cf. Tafelmaier et al., 2020, 29 f.). Interestingly, a basic assumption of chaîne opératoire accounts is that the creation of stone artefacts first emerges as a cognitive project, which is then translated into a conceptual scheme that is finally concretized through a series of action events (operations). All three steps are interdependent and can be influenced by numerous, sometimes interacting, natural and human parameters. According to the theoretical background, the observable constant and regular elements of the operational schema, allow us to determine the underlying concept that drives the schema. As a result, its inferred goals enable the determination of the original cognitive project.

160

J. Richter and T. Breyer

Therefore, consistency or frequent repetition of one or more patterns can be interpreted as intentional. The re-construction or reverse engineering of stone tools, which deliver the basis for such an advanced interpretation, as an endeavour of theoretical analysis, calls for an empirical counterpart, as a practical act of construction. Modern flintknapping has not always been linked to archaeological research, but proved to be useful for fulfilling this exact purpose of an experimental palaeolithic archaeology. In return, the professional interest not only led to a flourishing of this art, but also to unimagined levels of craftsmanship  – precisely because of the import and imitation of ancient knowledge that is stored in the artefacts. However, the rather “intimate”, gestural handling of the raw material enables us to do more than just confirm our assumptions about the course of the production process; it also tells us something about reasons: why stone tools were made the way they are. Obviously, the technology groups for stone tool production did not emerge by chance, but out of a conglomeration of ecological and cognitive factors as well as the interplay between materials and gestures. By re-enacting this interplay, we might learn something useful about our theories concerning stone tool production, but more importantly we can reach an understanding of what can be done with said material and where limitations lie, etc. None of this can be learned from morphological studies carried out from a distance, but is instead the result of a changed understanding of Palaeolithic artefacts as (ecologically and socially) embedded processes of embodied engagement. Neither the biological constitution of the human body nor the physical features of the material alone can fully account for these processes that are the artefacts themselves. Only by considering them together we can interpret the interactive behaviour of our ancestors, as it is preserved in those artefacts, in a meaningful way. In this sense, a practical, experimental or: gestural approach is able to enhance our theoretical understanding of artefacts within the framework of a coevolution.

4 Concluding Remarks: Practice as a Perspective As the case studies presented above indicate, the prehistoric artefacts chosen for this study should not be regarded as formal templates (“types”) that are achieved by complex production processes. Rather, the processes were designed and combined in a way that resulted in the lithic elements needed. As the processes appear to have been highly conventional and as they were repeated many times in the same manner, they must be regarded as expressions of regular standard Neanderthal practice. These practises were learned by each individual who had to train them in order to achieve the mastery required to deliver and repair hunting and maintenance tools. Training and mastery, as relating to physical and mental control of the human body, therefore figure as realisations of body techniques (techniques du corps), as defined by Marcel Mauss.

Petrified Practice: Is There a Vernacular Choreography of Neanderthal Movements?

161

This notion refers to “the ways in which people in one society or another traditionally use their bodies” (Mauss, 1974, 210, translation ours). These techniques may vary historically and regionally, by age and gender, but their use is always socially predetermined. The purpose of their practice is to adapt the body to its operational functionality and thereby achieve certain performances. Body techniques, according to Mauss, have three essential features (cf. Schüttpelz, 2010): (1) they have an arbitrary character, therefore they are employed in societies for symbolic differentiation; (2) because of this quality, learning them can lead to experiences of foreignness and intercultural misunderstanding; (3) they do not fit under a teleological model of progress and accumulative refinement. The last statement is an important hint concerning the idea that some prehistoric artefacts, such as the famous hand-axe, seem to avoid any technical, evolutionary improvement over hundreds of millennia. The reason is that they are not so much part of “cumulative” material culture, as they feature as results of practices, that is, body techniques, which had to be learned by every single individual “from scratch”. Our observations confirm Haudricourt’s statement: “L’outil est adapté au geste et non inversement” [The tool is adapted to the gesture and not vice versa] (Haudricourt, cited in Schüttpelz, 2017, 230). At the same time, the artefact, if understood as resulting from practices, does not necessarily include any concrete artefact design, communicated by oral tradition between individuals. Shared practices rather served as a means of communication in themselves, in much the same way as we observe it in pre-industrial craftsmanship in modern Western civilisations. Gestural movements as meaning-carrying and operational units within sequences of interactions between humans and the material environment take place within a kinesphere, as described above. Therefore, a combination of chaîne opératoire approaches with choreography theory is a promising outlook for investigating tool making processes across a wide range of embodied action as well as a large temporal horizon of human productivity.

References Alexandri, A., Buchli, V., Carman, J., Hodder, I., Last, J., Lucas, G., & Shanks, M. (Eds.). (1995). Interpreting archaeology. Finding meaning in the past. Routledge. Bataille, G. (2006). The production and usage of stone artefacts in relation to final exploitation – the repeatedly visited primary butchering station of level II/7E.  In V.  Chabai, J.  Richter, & T. Uthmeier (Eds.), Kabazi II. The 70,000 Years Since the Last Interglacial. Paleolithic sites of Crimea, vol. 2 (pp. 111–130). Bernbeck, R. (1997). Theorien in der Archäologie. Francke. Binford, L. R. (1977). General introduction. In L. R. Binford (Ed.), For theory building in archaeology (pp. 1–13). Academic. Boëda, E., Geneste, J. M., & Meignen, L. (1990). Identification de chaînes opératoires lithiques du Paléolithique ancien et moyen. Paléo, 2, 43–80. Chabai, V. P., & Uthmeier, T. (2006). Settlement systems in the Crimean Middle Palaeolithic. In V. Chabai, J. Richter, & T. Uthmeier (Eds.), Kabazi II. The 70,000 years since the Last interglacial. Paleolithic sites of Crimea, vol. 2 (pp. 297–359).

162

J. Richter and T. Breyer

Earle, T., & Preucel, R. (1987). Processual archaeology and the radical critique. Current Anthropology, 28(4), 501–538. Eggers, H.-J. (2018). Einführung in die Vorgeschichte (6th ed.). Scripvaz. Geneste, J.-M. (1991). Systèmes techniques de production lithique: Variations techno-économiques dans les processus de réalisation des outillages paléolithiques. Techniques et Culture, 1–35. Gibson, J. J. (1979). The ecological approach to visual perception. Houghton Mifflin. Huschka, S. (2021). Choreographieren. wissenderkuenste.de 10. Hussain, S.  T., & Will, M. (2020). Materiality, agency and evolution of lithic technology: An integrated perspective for Palaeolithic archaeology. Journal of Archaeological Method and Theory., 28, 617–670. https://doi.org/10.1007/s10816-­020-­09483-­6 Inizan, M.-L., Reduron-Ballinger, M., Roche, H., & Tixier, J. (1999). Technology of knapped stone. CREP. Johnson, M. (1999). Archaeological theory: An introduction. Blackwell. Laban, R. (2011 [1966]). Choreutics. Dance Books. Leroi-Gourhan, A. (1964). Le geste et la parole I – Technique et language. Albin Michel. Malafouris, L. (2014). Creative thinging: The feeling of and for clay. Pragmatics and Cognition, 22, 140–158. Malafouris, L. (2020). How does thinking relate to tool making? Adaptive Behavior, 29, 1–15. Mauss, M. (1974). Die Techniken des Körpers. In M. Mauss (Ed.), Soziologie und Anthropologie, vol. 2 (pp. 197–220). Ullstein. Postuwka, G. (1992). Der Tanz schafft Raum. Sportpädagogik, 16(4), 37–41. Richter, J. (2001). Copies of flakes: Operational sequences of foliate pieces from Buran-Kaya III level B1. In V. Chabai, K. Monigal, & A. E. Marks (Eds.), The middle Palaeolithic and early upper Palaeolithic occupation of eastern Crimea (ERAUL) (Vol. 104, pp. 233–248). Université de Liège. Schatzki, T. (2016). Practice theory as flat ontology. In G. Spaargaren, D. Weenink, & M. Lamers (Eds.), Practice theory and research. Exploring the dynamics of social life (pp.  44–58). Routledge. Schiffer, M. B. (1976). Behavioral archaeology. Academic. Schüttpelz, E. (2010). Körpertechniken. Zeitschrift für Medien- und Kulturforschung, 1(1), 1–20. Schüttpelz, E. (2017). Die Erfindung der Twelve-Inch, der Homo Sapiens und Till Heilmanns Kommentar zur Priorität der Operationskette. Internationales Jahrbuch für Medienphilosophie, 3(1), 217–234. Shanks, M. (2007). Post-processual archaeology and after. In C.  Chippindale & H.  Maschner (Eds.), Handbook of archaeological method and theory (pp. 133–144). Altamira. Soressi, M., & Geneste, J.-M. (2011). Special issue: Reduction sequence, chaîne opératoire, and other methods: The epistemologies of different approaches to lithic analysis. The history and efficacy of the chaîne opératoire approach to lithic analysis: Studying techniques to reveal past societies in an evolutionary perspective. PaleoAnthropology, 334–350. Tafelmaier, Y., Bataille, G., Schmid, V., Taller, A., & Will, M. (2020). Methoden zur Analyse von Steinartefakten. Eine Übersicht (p. 80). Springer. Trigger, B. (1989). Comments on Archaeology into the 1990s. Norwegian Archaeological Review, 22(1), 15–18.

The logos of techné: A Case for Technology as Interdisciplinary Anthropology Johannes F. M. Schick

Abstract  This chapter develops the potential of a literal understanding of technology (as logos of techné), to develop an interdisciplinary anthropology. Technology in this sense draws together the diachronic aspect that is implied in the central hypothesis of French techno-anthropology that every technical object contains a crystallized human gesture and the systematic aim to understand human beings in the digital age. To outline the heuristic of technology as interdisciplinary anthropology I will focus on the ontological and epistemological presuppositions of socio-­ technical practices, technical objects, and the gestures involved. They produce difference and are transformative. Gestures are thus not conceived as bodily actions in a narrow sense but comprise also mental operations. Accordingly, gestures can be crystallized in stone axes as well as in computers. This diachronic approach to technologies allows to understand multiple modes of being human that the human entanglement with technical objects and networks provides. Keywords  Interdisciplinary anthropology · Gilbert Simondon · Durkheim-school · Technical objects as gestures and institution · French philosophy of technology

1 Introduction “Technology” is an equivocal concept. It can be used in its “vernacular” meaning or in its literal sense as “science of techniques” (Coupaye, 2022).1 As a modern category “technology” structures not only the way western modern subjects structure  For the history of the concept of “technology” see the article by Jacques Guillerme and Jan Sebestik (2007). Marcos Camolezi recently described the transformations in the meaning of “Technik” in different languages (Camolezi 2022). 1

J. F. M. Schick (*) CRC Media of Cooperation, University of Siegen, Siegen, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 T. Breyer et al. (eds.), Diachronic Perspectives on Embodiment and Technology, Philosophy of Engineering and Technology 46, https://doi.org/10.1007/978-3-031-50085-5_11

163

164

J. F. M. Schick

and perceive the world, but it also has ramifications for the cultural other as the anthropologist Ludovic Coupaye shows (Coupaye, 2022). I claim in this chapter that it is necessary to rediscover the meaning of “Technology” in its literal sense in order to understand the socio-technical entanglement of human beings. “Technology” as logos of techné serves as heuristic to focus on techniques from an interdisciplinary and anthropological perspective. Technical practices are either the result of human invention and involve human beings in some way even if they run in the background and remain invisible, or are conceptualized by human beings as technical practices, when performed by other beings.2 I develop these hypotheses with reference to the tradition of French Techno-Anthropology. The systematic goal of this chapter is to sketch out what technology as a philosophical enterprise means. To achieve this goal, I draw on the origins of techno-anthropology in France between 1890 and 1914 and relate it to current debates in new materialism and to the philosophy of technology of Gilbert Simondon. The contingent nature of the human being and the problem of the origins of human intelligence took center stage during the “homo faber-debate” (Sigaut, 2013).3 Philosophers, mainly following Bergson’s conception of the homo faber, and sociologists, following Durkheim’s primacy of the social, argued whether the origins of human intelligence lie in a material engagement with the world or in a social differentiation between the sacred and the profane (Durkheim, 1995). It is my goal to show that discussing this debate provides the historical and systematic basis for a technology in its literal sense as “interdisciplinary anthropology”. From its outset, the tradition I refer to had to be interdisciplinary in order to establish its position: new developments and findings in psychology, chemistry, and physics forced Bergson, Durkheim, Mauss, Bachelard, and a whole generation of philosophers to rethink their conceptions of matter, mind, time, and space, while experiments and social infrastructure increasingly relied on technical ensembles. Simultaneously, ethnographic material became available and led sociologists to  An example would be the technical skills of the beaver to construct its dam. To understand the operations of the beaver as technical practices, however, would mean to re-invent them as human practices (see for instance: Weber, 1913, 127). 3  Frédéric Worms describes this time as the “philosophical moment 1900” during which the human being becomes a philosophical problem due to its double nature as a biological and as a rational entity (Worms, 2004). The essence of the human being is no longer defined a priori, but rather historically contingent and in constant transformation (ibid.). The philosophical moment 1900 was triggered by technological, social, and scientific transformations, which led to a reconceptualization of knowledge and the human being. Against the backdrop of evolutionary theory, new developments in physics, biology, and chemistry, philosophers such as Durkheim and Bergson introduced new definitions of the human being, namely the “homo duplex” (Durkheim), to address the biological and psycho-social nature of man, and the “homo faber” (Bergson), to argue for the technical nature of human intelligence. In these concepts opposite poles of the philosophical spectrum around 1900 become manifest. While Bergson is considered to be part of a group of philosophers, who re-established spiritualism via the journal Revue de la Métaphysique et la Morale (RMM), Durkheim and his followers had close ties to the positivist school and the founder of the Revue Philosophique (RP), Théodule Ribot (for a detailed discussion of the relations between the RMM and the RP see Merllié, 1993; Barberis, 2002). 2

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

165

question the supremacy of Western civilization. The question of what a human being is was therefore posed against the backdrop of findings from multiple, diverse disciplines. Technology as interdisciplinary anthropology stands in this tradition. It is neither a new discipline nor a completely novel approach to philosophical anthropology. However, its methodological approach focuses on practices as expressions of human nature. These practices lead to social, corporeal, or material techniques and can be addressed from different disciplines. Psychology, physiology, geography, history, and ethnography can study, for instance, how people walk, swim, sleep, build houses, etc. The task of technology as interdisciplinary anthropology is to provide a comprehensive heuristic for the study of these phenomena. The development of this heuristic is a classic task of philosophy, yet Durkheim and Mauss for instance claimed that it belongs to “general sociology” (Mauss, 1934). I understand “technology as interdisciplinary anthropology” as a philosophical attempt to explore the “no man’s land between the sciences” (Wiener, 2007, 2) to understand and interpret different “modes of being human”. With Bergson, Durkheim, Mauss, and Simondon, I claim that the essence of the human being is their socio-technical nature. This implies, however, that it changes over time and that its transformations are influenced by social and technical developments. In different cultures, different modes of being human are possible. These modes depend on social stratification, technological development, and how human beings relate to each other and to their socio-technical networks. I will develop Technology as interdisciplinary anthropology in three steps. In the first part of this chapter, I will focus on the underlying ontology and its epistemological consequences. Practices – be they social or technical – are understood as practices of living beings. Technical objects are also conceived in analogy to living organisms and are expressions of life (2). They mediate between human beings and the world and the relations with socio-technical networks create qualities (2.1). These aspects, that are crystallized in technical objects, are accessible via body techniques, operational chains and gestures (2.2). To participate in the genesis and mode of existence of technical objects means to understand these specific modes through re-invention (3). Yet, it remains a philosophical task to invent conceptual dispositivs that address the reciprocal nature of the relation with technology. In the concluding part, I will suggest Technology as Interdisciplinary Anthropology as a possible means to pose the question of multiple modes of being human.

2 Ontological Distinctions, Epistemological Consequences: The Primacy of Practices The most primitive and ancient of tools as well as the most elaborate machinery and digital gadgets have common roots. They are products of human intelligence and human engagement with the world. They are, as Simondon claims, “crystallisations

166

J. F. M. Schick

of human gestures” (Simondon, 2005b, 340). This vantage point refers back to the homo faber-debate at the beginning of the twentieth century between Bergsonians and Durkheimians. The debate revolved around the question whether the origins of human intelligence are technical or social (Sigaut, 2013; Schick, 2018). Bergson introduced a functionalist definition of human intelligence in his Creative Evolution. Human intelligence is the ability to create tools and its primary interest is to master the affordances of its environment to secure survival (Bergson, 1998, 139). The thesis that human intelligence is first and foremost practical and technical was one pole of the debate. The other opposing pole is illustrated by the Durkheimian thesis of the primacy of the social which assumed that human intelligence is first and foremost social and only in a second step technical. Tool use, especially the ability to “create tools to create tools” was for Bergson the differentia specifica of the human being. Durkheim on the other hand searched for the beginnings of mankind by studying ethnographic material of Australian natives. The aborigines were, according to Durkheim and Mauss, the last survivors of the Neolithic age, since Australia was the last continent to be colonized by the western world and did not have any influence from any other society (Durkheim, 1995; Mauss, 1974).4 Durkheim argues in the Elementary Forms that social order is generated through religious practices. Religious practices distinguish between the sacred and the profane. The first acts of human intelligence were thus according to Durkheim social relations that establish classifications (Durkheim & Mauss, 2003; Durkheim, 1995). However, the positions are not as radically opposed as they seem at first sight. Bergson was aware of the importance of the social (see Bergson, 1932), as was Durkheim of the importance of the technical (see Durkheim, 1901).5 The potential of the debate is due to the tension between the poles of the technical and the social as well as to the diversity and complexity of the positions. The tension between the technical and the social urged the participants of the debate to show how the social refers to the technical and the technical refers to the social. Contemporary concepts, such as the network, were already used to describe the relation of human beings with technology (Weber, 1913, 295 ff.) and the debate foreshadows the Latourian homo faber socialis (Latour, 1994, 47). Both, social as well as technical practices express human nature and form a network of agency with human, natural and artificial entities. Even though all agents within the network are symmetrical to one another, i.e., in actu each agent has to play its part and is dependent on the other, this does not imply that they cannot be distinguished. To focus on the tension between the concepts of the social and the technical allows therefore to conceive of dualisms not as referring to separate ontological  The “last survivors of the Neolithic age” in the Durkheimian sense, was in fact a group of Aborigines, later called the “Pintupi Nine”, who lived without contact with western civilization in the outback until they made first contact in the 1980ies (Mahony, 2014). 5  Bergson dealt with the sociological theory and the origins of religion and morality in his “Two Sources of Morality and Religion” published in 1932. Durkheim never wrote on technology but introduced “technology” as a category in the Année Sociologique. He assigned the task to study technology to Henri Hubert. 4

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

167

domains, but rather as polarities of one reality. The implication of this ontological claim has epistemological consequences: Dualisms represent polarities rather than ontological realities. The dualisms of Durkheim, Mauss and Bergson were employed methodologically to understand and describe reality. Dualisms can be conceived as instruments to account for the relations between matter and mind, the technical and the social, the individual and society, praxis and cognition. The dualisms are therefore not ontological realities, but are modalities that signify different senses of reality: any phenomenon that is studied takes on a different sense according to the method one applies, i.e. the direction of one’s research. It is possible to consider for example an emotion from a purely empirical point of view, as well as from a purely phenomenological point of view, yet both refer to the same event (see Schick, 2012). This interpretation of dualisms presupposes the ontological claim that only one reality exists: while Bergson and Durkheim have very different opinions on how matter and mind relate to each other, both consider them to belong to the same ontological plane. For Bergson, the category of space and the concept of matter, as used by the sciences, are abstractions from “lived experience” (Bergson, 1896). As Frédéric Worms points out, for Bergson life takes on two different meanings/directions (sens) (Worms, 2013). Bergson’s philosophy can thus be described in terms of the tension between the individual’s experience and the scientific description of phenomena. Durkheim, on the other hand, claims that both mental and physico-­ chemical phenomena are natural phenomena that follow laws of causality. This does, however, not mean, as often suggested by critics, that Durkheim argues for a determinism, but rather that social phenomena can be studied scientifically. The science to study these phenomena is Durkheim’s sociology. It takes a radically different point of departure: social practices (Rawls, 2004). The Bergsonian and Durkheimian dualisms are not antitheses, but as Peristiany puts it, “polarities” (Peristiany, 2010, x). The tension between opposed terms such as matter and mind, the social and the technical, the individual and society open up a “field”, where different solutions of how matter relates to mind, praxis to cognition, etc. become possible. This notion of the field, a “present of the natural sciences to the humanities” (Simondon, 2005a, 544), allows to go beyond both Bergson and Durkheim. It refers to a philosophical reading of practice theory and the ontological turn, where indigenous concepts are taken at face value (for instance the belief that “powder is power” in Cuban divination cults, (Holbraad, 2012; 2013) to question the Western set of categories: a field between an indigenous set of categories and the Western set of categories is formed in order to see, how the indigenous belief systems work and how this shift in categories — if we do as if, powder is power — transforms the Western categories. This is a form of kaleidoscopic thinking (Mauss, 2005, 52) that aspires to fathom the genesis of categories in general. Concepts and categories are thus products of human practices in the world, a thesis already developed by the Durkheim School in their “Category Project” (Schick et  al. 2022). “Praxis” should thus be “prioritize[d] over essence and process over substance”

168

J. F. M. Schick

(Iliopoulos/Malafouris, this volume) as also the Material Engagement Theory (Malafouris, 2013) states.6 The technical nature of concepts and words differs, however, from the technical nature of material objects. While the human body is in direct contact with the material object and feels its feedback (at least with tools and machines), language produces actions at a distance, as Céléstin Bouglé points out: The word is also an instrument. And whoever uses this instrument is capable of producing actions at a distance on his/her fellow human beings. He moves them very differently from the mechanical action that he succeeds in as a polisher, potter or blacksmith: it awakens hope or fear in the distance, it provokes the dangerous or useful gesture. Hence, the belief in the fecundity of invocations, incantations, of all that is action through words: the magico-­ religious technique is thus born — what we used to call the transcendental technique and its overabundant vegetation can only suffocate the terms deposited in the earth by the positivist technique of the early ages. This would explain, after such brilliant beginnings, the long trampling of mankind: science will only be able to take full flight when the positivist technique has regained lost ground on its enemy sister.7

Bouglé points to another dualism, the dualism of the magico-religious and science, which is used to describe the progress (or the lack of progress) of mankind. He contrasts the “transcendental technique” magic with the “positive technique” science and describes the progress of human societies as a struggle between these two techniques. This conception goes back to Mauss’ Essay on Magic, where magical acts are conceived as “traditional efficient acts” (Mauss, 2001, 24  f.), which as Mauss specified in Techniques of the body, refer to the social order, while technical acts (in a narrow sense) are concerned with the physico-chemical order (Mauss, 2006, 83). Both, the transcendental and the positive technique, follow rules and refer to an underlying causal force, which is canalized through magical and/or technical practices. The common ground of magic and religion on the one side and techniques and science on the other has led to the etiology that science developed out of magic, since both introduce a causality, which is independent of human actors: the natural

 The primacy of “praxis” and consequently the focus on dynamism and movement is at the heart of the theoretical accounts of this volume (see for instance the contributions of Gerner; Aguiar; Maddalena; Morgenstern/Boutet; Iliopoulos/Malafouris; Parviainen/Coeckelbergh; Richter/ Breyer in this volume). These authors refer either to Peircean Semiotics, Harold Garfinkel’s practice theory (as well as to Goffman and Sacks), and/or to the tradition of Technoanthropology in the line of Mauss and Leroi-Gourhan to develop their argument. 7  All translations are made by the author of this text, if not cited otherwise. “Le mot est aussi un instrument. Et qui use de cet instrument est capable de produire sur les semblables qu’il émeut, des actions à distance, très différentes de l’action mécanique qui lui réussit comme polisseur, potier ou forgeron: il éveille au loin l’espérance ou la crainte, il suscite le geste dangereux ou utile. D’où la croyance à la fécondité des invocations, des incantations, de tout ce qui est action par la parole: la technique magico-religieuse naît ainsi – celle que nous appelions la technique transcendante et sa végétation surabondante ne peut qu’étouffer les termes déposés en terre par la technique positive des premiers âges. Ainsi s’expliquerait, après des débuts si brillants, le long piétinement de l’humanité: la science ne pourra prendre tout son essor que lorsque la technique positive aura regagné sur sa sœur ennemie le terrain perdu.” (Bouglé, 1922, 165) 6

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

169

laws of science correspond with the natural forces that magic calls upon (e.g. Cassirer, 2004; Schüttpelz, 2021). One can, however, question, whether magic and its function have disappeared with modern science: contemporary human beings use their smartphones as containers of personal information and relations that are similar to the “ritual bundles” carried around in indigenous cultures; both serve as tools to stabilize social relations and can neither be reduced to materiality nor to a purely social function (Schüttpelz, 2021, 61). A further argument for the socio-­ technical nature of smart devices is that the user interacts with smart devices on a linguistic level.8 The commands we enter with our hands and words produce “actions at a distance”: within the smartphone and in the world: retrieving information from the internet, communicating with friends around the globe, and checking the weather conditions are actions performed on the surface of the phone, while the material infrastructure remains silent (Weber, 1913; Schick, 2021b). It seems that two forms of causality are simultaneously at work — a social and a technical causality — and that as soon as one is exercised it also has effects in the respective other domain. Both forms of causality, as I pointed out earlier, are part of the same reality and form a field of tension. Concepts are technical as they allow to structure and form the world. They are social with respect to their potential to inform the other at a distance, without relying directly on a material structure. The performative aspect of language is thus socio-technical: rituals are composed of multiple practices, which are linked to one another and are confirmed, condensed, and supported by words (Austin, 1962; Mauss, 2008). Both the blessing of a priest in a church service and the word “action” of a film director — even though they have different social values  — instigate social and technical practices: cameras roll, organs play, people act in correspondence to spoken words. Concepts, signs as well as scientific formulas, form relations with the social and the technical other.9

2.1 More Relations, More Problems? How can we account for the multiple reciprocal relations that are formed with the world? In recent years the theoretical debates in the humanities have led to a reconsideration of the established subject-object relation (Ghanbari und Hahn, 2013). The Actor-Network-Theory (ANT) emphasized a symmetrical approach to human and non-human elements (Latour und Woolgar, 1986; Latour, 1993). This approach has led to an intense controversy about the redefinition of the relations of things, signs and humans ((Kneer et  al., 2008). Jane Bennett, Karen Barad, Graham  The work of Stephan Habscheid, Dagmar Hoffmann and their team in Siegen on Intelligent Personal Assistants (IPAs) such as ALEXA, GOOGLE APPLE shows that the language based practices with these devices express an ongoing constant quest to assign and/or regain agency (for instance in moments when the system fails) (Hector u. a. 2022; Habscheid, 2023). 9  Giovanni Maddalena makes a similar point with Peircean semiotics. “Following identity in its becoming” becomes possible by prioritizing practices over theory (Maddalena this volume). 8

170

J. F. M. Schick

Harman, Timothy Morton and other representatives of the material turn argue for an “agentive realism” (Barad, 2007), the vibrancy of matter (Bennett, 2010) or an object-oriented-ontology (OOO) (Harman, 2018; Morton, 2013a). These “new” ontologies serve as a basis for the epistemological and anthropological question of the signification of signs, things and of materiality to understand the condition of the human being (Descola, 2014; Latour, 2018; Ingold, 2011). Instead of providing a definition of the human being and its essential qualities, relations between signs, objects and human beings generate qualities that cannot be reduced to any single one of its agents but are the product of networks. Kant’s forms of sensible intuition – space and time – are no longer conceived as a priori structures of the subject, but are created by the properties of the objects and their multiple relations with things, as Timothy Morton puts it: All the things by which we specify the object are not the object. By we I mean humans, lavatory brushes, quasars and durum wheat, and the object in question itself. We have a very strange situation then, in which there are objects, and there are qualities and relations between these objects and other objects. There is a chōrismos, an irreducible gap. Qualities and relations are much the same thing, since they are born in interactions between the object and 1+n other things. (Morton, 2013b, 27)

Relations between entities generate the qualities attributed to objects, human beings, and the entities involved in the relation. While I would not go as far as claiming that qualities and relations are “the same thing”, I agree with Morton that the interaction between and with objects is crucial. This position allows Morton to claim that the aesthetic experience as well as the production of art has causal effects. He uses the works of Yukultji Napangati  – one of the last “Neolithic  humans on Earth” and Aborigine artist (ibid. 24)– as examples for the creation of an “interobjective space” (ibid. 26): The image is not a mute object waiting to have its meaning supplied by a subject, nor is it a blank screen; nor is it something objectively present “in” space. Rather the painting emits something like electromagnetic waves, in whose force field I find myself. The painting powerfully demonstrates what is already the case: space and time are emergent properties of objects. (ibid. 34f.)

The relation to and with objects is for Morton “an interpretation” (ibid.). The spectator is in a direct relation with the objects and participates in a field that creates space and time. This aesthetic experience is according to Morton “objective”, since it grants access to the event of creation. Morton even claims that this is in line with contemporary science, due to the active participation of the observer and the spontaneous properties of matter. Aesthetic and scientific interpretation are according to Morton symmetrical. On the one hand, this opens the approach of OOO to a “technological hermeneutics”, which goes back – due to its philosophical lineage – to Heidegger,10 but on the  The reference to Heidegger is problematic – at best. Morton is aware of Heidegger’s Nazism, but seems to think of it as a historic contingent event, when he writes “[d]espite thinking that (especially thinking that) he has gone beyond objective presence he reifies being into an authenticity that means ripping the illusion away. There is a fantasy of seeing a real underneath. In a perfect political 10

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

171

other hand it poses an epistemological problem: who interprets and how? The absence of an absolute starting point and the entanglement with the interobjective space seems to make it impossible to have any objectivity as well as any subjectivity in the traditional sense – and this seems to be the provocation for philosophy as a discipline. But does it have to or rather is it really? The controversy revolves around Latour’s claims that objects possess agency. This means, that objects are themselves actants and are actively involved in the production of technical objects, scientific facts, laws and human behavior. They are part of a network in which the relations between objects and human beings are symmetrical, i.e. the power to act is distributed and is not entirely in the hands of an absolute human subject (Latour, 1993). This “ontology of agency” (Latour et al., 2011), leads to a problem reminiscent of Foucault’s characterization of power: where is agency located? Latour seems reluctant to consider agency as a vital force that goes beyond the sphere of the physical object (Latour et al., 2011, 160–61). Timothy Ingold remarks – without disregarding the explanatory value of ANT – that it results in “a double reduction of things to objects and of life to agency” (Ingold, 2010, 7). His concern is that Latour’s ontology might not be able to fathom the genetic processes of life that explain how material things emerge. Ingold puts his finger on a fundamental philosophical and ontological problem: As soon as agency is exclusively attributed to objects, “life” is no longer conceived of as a supporting force, that is “already there” in the environment (Ingold, 2011). For these reasons Ingold, with the philosophers Whitehead and Bergson in mind, reintroduces the notion of the living organism. In a polemical fictional dialogue between an ANT and a SPIDER (Skilled Practice Involves Developmentally Embodied Responsiveness), the ANT being Latour and the SPIDER being Ingold, he illustrates the main differences between his and Latour’s position: Our concept of agency must make allowance for the real complexity of living organisms, as opposed to inert matter. […] What makes the difference between me and the leaf, however, is that every movement I make is also a movement of my attention. (Ingold, 2011, 94)

Even though material objects co-constitute the actions of human beings, this does not imply that they are the same as living organisms. While it becomes increasingly obvious that human behavior is co-constituted by technics and rituals and that  new behavior is triggered by technical engagement, the co-­ constitution of practices remains a problem to be understood by human beings.

storm, this authenticity-speak matched the authenticity-speak of Nazism.“(Morton, 2013a, b, 224). Yet, the “authenticity speak“of Heidegger was never – to stay in the metaphor of Morton – blown towards the authenticity of the Nazis, but his esoteric language was designed to hide and to foster antisemitic and nationalist thought as the publication of the “Schwarze Hefte” and the work of Farias, Kellerer and others have shown. Rather than digging “a tunnel to the future […] through some kind of engagement with Heidegger”, I suggest to search for alternatives in the philosophy of life, philosophical and social anthropology and even cybernetics, where the role of materiality was equally important. Furthermore, instead of searching for a philosophical style that hides its intentions, authors such as Bergson, Durkheim, Mauss, Canguilhem, Whitehead, Simondon, Wiener, Bateson etc. attempted to enter a dialogue with other disciplines.

172

J. F. M. Schick

The interactions between objects, as multiple and complex they might be, only make sense for a living being that transforms the exchanged signals into information (Simondon, 2017, 150). While the OOO-approach claims to be in the midst of things – and I do not claim that we can escape the entanglement with things  – any symmetric anthropology has to remain epistemologically asymmetrical otherwise it does not mean anything: the signals that are exchanged on the motherboard of the computer are necessary for me to write this sentence. They are, however, not sufficient to understand the sentences I wrote. Tools, instruments, machines and all other technical objects are part of operational chains connecting human beings, technical, social and natural entities (LeroiGourhan, André, 1964;  2018). Writing this chapter involves the mechanical-­ technical stringing together of toggle switches, that are comprised on the motherboard as well as the signals that are exchanged. However, these signals are transformed – since I am not using a typewriter – into lights that switch on and off on a monitor and are continuously transformed by my hammering on the keyboard. Yet, although I am writing with the computer program Microsoft Word, what I am not doing is to program. I interact with a “What You See is What You Get (WYSIWYG)”-program: the signals sent by pressing letters on my keyboard are translated multiple times to represent letters within Word that, in turn, can be italicized, underscored, enlarged etc. This simple and trivial example shows that the user is in constant exchange with an interface. The interface simultaneously separates and connects the user with the technical operations. It provides access to the surface, but the technical operation as such remains hidden. The operation of writing on a computer as a whole, however, cannot be reduced to any single one of its parts. But even though parts do exist and we can distinguish them within the operational chain, the practice of using the computer does not need to distinguish between the elements of its operation. Yet, the meaning of the operation can only exist for human beings, who understand, how things are in relation to one another, as I have stated above (Simondon, 2017, 150). To use Word means to at least implicitly understand that one is part of an operational chain and that the computer simulates a typewriter. The signals that provide the smallest unit and the operational basis of the computer are constantly transformed into different signals, so that ultimately “something” appears on the screen. These electrical signals transformed into light, become strictly speaking only a text, if someone is reading it. The reading itself implies the possibility of understanding the text. It does only implicitly imply the possibility of understanding the technical operation of writing on the electronic level. Thus, as human agents, we are always already within these operational chains that allow us to act. An epistemological position that distances itself from the realm of “interobjectivity” is therefore necessary to criticize and understand technical phenomena.11  Especially such encompassing phenomena as global warming or the market, which Morton calls “hyperobjects” (Morton, 2013a) need to be addressed from a critical stance (Schmidt und Koddenbrock, 2019). 11

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

173

This epistemological distancing should not, however, reduce the role objects play, but rather give credit to their performative potentials. One epistemological step is, as I have mentioned before, to explore the dualisms, western philosophy has operated with for thousands of years and to understand them as products of a material and practical engagement with the world. To speak of “technical objects” is thus neither a mere reproduction of Simondon nor a return to the Cartesian dualism of subject and object, but a necessary epistemological step that enables to address technical objects and to research operational chains. This distancing through and with media (“material” or “transcendental” techniques) is already inherent within our very way to act: we create and bridge distances by enlisting signs (language), tools, instruments or machines, and other human beings. The classification, identification and enlistment can obviously lead to an instrumentalization of either of these entities. The ethical questions are therefore “how to enlist” and “how to participate”, since the technical engagement with the world does not produce isolated material results, but creates new networks, where exploitation or imbalances of power can occur. It is therefore necessary to focus on technical practices as practices of living beings and to ask for the genesis of categories and classifications. Socio-technical practices are employed to render socio-technical relations stable and reliable by means of classification. Classifications allow to identify and hence also to subjugate beings. The relations and the generated qualities are thus contingent on the actors within the networks. The focus on the relations formed by human beings, signs, concepts, things and other entities allows to identify possible power imbalances. The position of the epistemic subject within these networks becomes clear via relations. This is especially in the contemporary digital age necessary, where “Inforgs”, to use a term of Luciano Floridi (Floridi, 2014), form hybrids with smart devices and tech companies extract information to control and predict behavior (Zuboff, 2019). In the age of “surveillance capitalism” (ibid.), it becomes increasingly necessary to understand the relations we form with technical objects and natural entities in order to understand ourselves and to form new relations. Relations are created by means of practices.

2.2 Technical Objects and Organisms The starting point of technology as interdisciplinary anthropology are the practices an artefact involves. The primacy of praxis over cognition (Marx, 1958) is shared by Marx, Bergson, Durkheim, Parsons, Garfinkel and contemporary practice theorists. They claim that all human activity, such as knowledge, science, power, language, technology, social institutions is generated in a “field of practices” understood as “the total nexus of interconnected human practices“(Schatzki, 2001, 11). Within

174

J. F. M. Schick

this field, new structures are constantly created through practices.12 This does not mean that structures and institutions do not persist over time, but that – in order to work – they have to be constantly actualized by practices. The primacy of action is succinctly expressed by the Durkheimian Céléstin Bouglé: In the beginning is action: as long as it encounters obstacles in the external milieu, and various possibilities in the internal milieu – in the organism – action tends to awaken thought.13

The reference to Claude Bernard’s conception of the internal and external milieu reveals another aspect, which is crucial to the diachronic and interdisciplinary approach to anthropology and technology that I propose: human practices are practices of a living being that affect the body as a whole, i.e. as an organism. The French traditions of epistemology (e.g. Bachelard), philosophy of life (e.g. Bergson), sociology and anthropology (e.g. Durkheim, Mauss, Hubert) was developed against the background of the theory of evolution (Darwin and Spencer) as well as the developments in biology, chemistry and physics. Even though their interpretation of the sciences differs greatly, their common reference point is seemingly an understanding of physis as an ontological force (Loeve et al., 2018). To conceive of the individual as an organism that entertains a tension between “internal possibilities” and “external obstacles” that allow it to “solve problems” is a common denominator of the different schools of thought. This relation with the milieu is essentially affective, living beings are embedded in an environment and the point of departure for creative processes are affective phenomena (Schick, 2012). The phenomenon of affectivity is shared with other living organisms and is the basis for actions in the world. Human beings, however, are not only organisms that are dependent on their natural environment, but they are also capable of creating tools structuring the environment and technical objects that constitute an entirely new environment. Human actions are technically mediated actions: even seemingly basic human behavior, such as sleeping, is a “technique of the body” (Mauss, 2006); multiple body techniques, gestures and practices are linked in “operational chains” (Leroi-Gourhan, 1964, see also Breyer & Richter in this volume) to perform technical practices. This implies, in light of the aforementioned interobjective space, that human action receives new qualities in different technological environments. Human beings do not unilaterally constitute their environment but are in turn constituted by their environment: The environment affects human beings as they are affected by their environment. Techniques are expressions of life (Bergson, 1998; Canguilhem, 2009; Lafitte, 1933; Simondon, 2014).  Andreas Reckwitz points out that practice theory constitutes itself a controversy: Practices are either considered from the perspective of a “model of infinite socio-cultural reproduction (Modell unendlicher sozial-kultureller Reproduktion)”, for which Bourdieus conception of “habit” is exemplary, or within a “model of infinite cultural ‘playful’ openness (Modell unendlicher kultureller, ‘spielerischer’ Offenheit)”, which goes back to the Ethnomethodology of Harold Garfinkel and Luc Boltanski (Reckwitz 2003). 13  “Au commencement est l’action: pour peu que celle-ci rencontre dans le milieu extérieur des obstacles, et dans le milieu intérieur – dans l’organisme – des possibilités diverses, l’action tend à éveiller la pensée.” (Bouglé, 1922, 162) 12

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

175

Machines and technical objects are therefore conceived in analogy to the organism, rather than conceiving of organisms in analogy to machines (Canguilhem, 2009). Technical objects are part of the same evolution as human beings and as Jacques Lafitte claims emphatically, “our creations, that are like another flesh from our flesh” (Lafitte, 1933, 115). The evolution of technical objects tends towards the organization of living beings: they evolve from an abstract state, where each of the elements of a technical object is external to the other and have each a specific function, to a concrete state, where the elements become pluri-functional. Technical objects can, however, never reach the concreteness of a living organism according to Gilbert Simondon: “technical objects tend toward concretization, whereas natural objects, such as living beings are concrete to begin with” (Simondon, 2017, 51). Accordingly, the British economist Brian Arthur, who also searches for the principles of evolution of technology in analogy to the Darwinian model, stresses the dynamic and vital character of technical objects: So a technology is not a fixed thing that produces a few variations or updates from time to time. It is a fluid thing, dynamic, alive, highly configurable, and highly changeable over time. (Arthur, 2009, 88)

Arthur exemplified the evolution of technical objects in a simple, yet fascinating experiment: together with his colleague Wolfgang Polak he simulated the evolution of technology within an artificial world programmed in a computer. Yet, far from remaining on the level of the interface, this world was made up of logic circuits – the very basis of any computer, since its beginning from basic calculation machines (Abakus) to Babbage’s difference engine and Ada Lovelace’s first computer program to the famous Turing-Machine and ultimately the latest smart phones: Arthur was interested, what kind of logical circuits a computer program would create, when given simple principles in the beginning – AND- and NAND-circuits (Arthur, 2009, 182 f.). His program shows that new developments integrate former innovations, while other operations are abandoned. The course of the technical evolution is continuous, but rather in “periods of quiescence”, “large gaps of time in which little happened at all” and “sudden appearance of a key circuit”, i.e. “miniature ‘Cambrian explosions’ of rapid evolution” (Arthur, 2009; Schüttpelz, 2021). Arthur’s thesis is similar to Simondon’s description of the evolution of technical objects. Their evolution is not linear or dialectical, it has a different temporal frame as the natural evolution (Simondon, 2017, 67) and follows the pattern of a “serrated evolution”: If transposed into biological terms, technical evolution would consist in the fact that a species could produce an organ that would be given to an individual, which would thereby become the first term of a specific lineage, which in turn, would produce a new organ. In the domain of life, an organ is not detachable from the whole that produced it, precisely because it is fabricated; and here we see the difference between the engendered and the produced. In addition to its spatial dimension, the technical has a historical dimension. Its current solidarity mustn’t mask the solidarity of succession; this latter solidarity is in fact what determines the great epochs of technical life through a law of serrated evolution. (Simondon, 2017, 68)

176

J. F. M. Schick

Even though the technical and the living being are thought in analogy to one another, this does not mean that they do not have specific differences. The evolution of technical objects depends on their status as produced beings. In this sense, technical objects have an advantage in their evolution. Elements can be detached from technical objects and ensembles and used in new ways. The technical evolution is guaranteed by the transmission of technical elements that were produced by the technical ensembles of a specific age (Simondon, 2017, 71); technical ensembles as a whole are not transmitted through time, but rather technical elements that are produced by these ensembles are used to invent new technical ensembles. The relation of the human being as “inventor/creator” and the technical object as “invented/created object” is – as I have stressed – not unilateral. The human being does not have absolute power over the object, but rather the invention also transforms the human being: human practices become increasingly specific and path-­ dependent (Arthur, 1994). These “sociocultural niches” (Ellis, 2015) lead to the paradox that technical objects, whose operations are no longer understood by its users, form an essential infrastructure for practices.

3 Reinventing the Wheel: The Diachronic Aspect of Technology as Interdisciplinary Anthropology Technology as a human science implies that any practice is bound to be transformed when it is actualized. This transformation is an epistemological caveat: it is impossible to escape one’s own culture, education, and habits. The primacy of praxis does not mean that the philosophical work begins with a tabula rasa. To reconstruct practices (with or without technical objects) means to re-invent them (Simondon, 2017, 191). On the most basic level, this reinvention consists in an attempt to understand how the object works, i.e., to grasp the operations within the machine and the bodily movements of the operator that connects him or her with the machine – in a sense, every time we use a bicycle, the re-invention of the wheel is implied. The reinvention implies the mental act of understanding, which provides a bodily schema of the operation of the machine: to “make the mind function as the machine would function” (Simondon, 2017, 191). The mind Simondon refers to is more than the mere neural representation of the machinic operations. It signifies a mental mimicry, i.e., it requires a transformation of thought, of how human beings conceive of things. Machines have to be considered in analogy to indigenous people for the Western ethnographer as alien, yet human forms of being that can be understood. Yet, neither people should be reduced to machines, nor machines to people. Each of them has a specific mode of being and a specific form of alterity (Schick, 2021b). The analogy presupposes that the operations within the machine can be accessed, mirrored, and understood by the human mind.

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

177

The effort of understanding technical operations signifies a form of mimicry. On a purely operational level, it can already start in child’s play, when children “become” the airplane or the automobile and imitate the technical object at work. Obviously, this is not the conceptual understanding of an engineer, but it provides the means to create mediating images and bodily schemata that can aid the technological understanding in later life. Children grasp a technical scheme through play, which can later be conceptualized and objectified (Simondon, 2014, 44). Initially, inventions are “schemata of behavior, of operation” (ibid.). Operations and images that served childhood play can be rediscovered and reconstructed in order to understand technical objects. This reinvention connects the past — an artefact that was invented — with the present — the agent, who actualizes the practice. However, this actualization is not necessarily congruent with the original, historical operation. It is actualized in a different historical situation and this setting provides the background for its signification. The artefact can enter into a new network of signification provided by the contemporary situation. The methodological motto of Haudricourt’s “technology as human science” to “start from the present to go back to the past”14 has to be paired with the epistemological goal of sociology that Durkheim formulated: its [sociology] purpose above all is to explain a present reality that is near to us and thus capable of affecting our ideas and our actions. That reality is man. More especially, it is present-day man, for there is none other than we have greater interest in knowing well. (Durkheim, 1909, 733)

Studying material artefacts, technical objects, rituals, and social practices of the cultural, technical, historical or social other reveals aspects of our conception of the human being. However, there is a profound difference between the practices of the Neanderthal or the tool-use of the eighteenth century and our contemporary world (Schick, 2021b). The categories of time and space are radically different. While the Neanderthal, as well as the craftsman in his eighteenth century workshop, were immediately connected to their tools and the milieu of their production, the human beings of the digital world experience a completely different set of categories: The space of the technical operation remained local, i.e. within the workshop or a specific site, where the first humans produced hand axes. Also, the Neanderthal accessed and transformed nature, while contemporary tools provide access to an already designed nature.15 The technical operations of the twenty-first century involve global networks of production (mines in Africa, factories in China, supply chains to franchise stores, etc.). Signals are sent to and received from satellites. The spatial representation in the twenty-first century differs greatly from the use of, for example, stationary

 “partir du present pour remonter au passé” (Haudricourt, 1987, 42)  It goes without saying that the categories have been transformed over time: Working in the prehistoric Neanderthal is essentially different to working in an eighteenth century workshop on the brink of industrialization. 14 15

178

J. F. M. Schick

telephones. The contemporary smartphone affords to multitask: the user can, while calling a friend in Brazil, be sending an email to a business partner in China and text a friend to meet up for a drink in the evening (Carnino, 2018). This notion of space, which is no longer a Kantian form of intuition, since it cannot be represented, is being condensed by the technical practices with a smartphone, due to processes on the microlevel: The processor Apple A 14 Bionic of the latest version of the iPhone for instance, has a computing power that can distinguish between 0 and 1 eleven trillion times per second (apple.com, 2020). This “rupture in transduction” as Guillaume Carnino calls it (Carnino, 2018), influences how we conceive of the world. But what exactly is revealed and what unites the Neanderthal with the homo sapiens sapiens of the twenty-first century? Technical objects are operated by gestures. Gestures establish the relationship of the human being with technical objects and with other living beings (e.g., trained dogs react to gestures). They are in this sense as well as language technical, and since they are taught or artificially constructed to transmit information, gestures are “essentially communicative” (Guillerme, 2017, 449). Gestures and practices can thus be recovered starting from a material, technical artefact. The human being can participate with the technical being, since it contains information that can be historically transmitted and actualized by means of gestures (Simondon, 2005b, 341 f.) since technical objects are crystallizations of human gestures. This recursive relationship of artefact and human being is ideally represented in what Maddalena calls “complete gesture”, where inventor and user have “interchangeable roles” and share the determination of the experience via the artefact.16 These gestures are not natural or innate but are rather developed in the engagement with the technical object. The technical object shapes the gesture of the human being (Sigaut, 2013, 132). Gestures, as employed here in the tradition of French philosophy of technology, encompass bodily movement as well as cognitive operations. They are “techniques of the body” (Mauss, 2006). This broad understanding of gestures signifies that the manipulation of technical objects occurs on each of its ontological layers. A smartphone for instance, is based upon the “mental gesture” of distinguishing between 0 and 1, i.e., the logical law of the “excluded middle”. This law is materialized in a structure of switches that are either “on” or “off”. Two gestures, a “mental” one (the material implementation of a law of logic) and a bodily one (turning a switch on and off) are comprised on any motherboard.17 The  “In a complete gesture, the inventor and the user accomplish the same action and have interchangeable roles. Both must assent to the proposed action: when a user becomes skilled, he/she can give important feedback to the inventor, but above all, he/she experiences the same answer to the same needs through the same determination of his/her experience as the inventor.” (Maddalena this volume, p. 7) 17  This does not imply that there exist any gestures that are entirely “mental” or entirely bodily, but rather that mental and bodily aspects interpenetrate each other – even though one aspect can have a more important role than the other. The recursivity of mental and bodily practices can also be shown in the composition of music. Vinicius Aguiar stresses the importance of bodily practices for intellectual practices (and vice versa) in his chapter on the role of gestures and diagrams in musical composition (see this volume). Simondon’s broad understanding of gesture does not suggest that 16

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

179

transmission and transformation of these signals afford on the screen bodily gestures of swiping, zooming in with two fingers, and scrolling up and down. These two forms of gestures are far from comprehensive. The gestures needed to produce a smartphone or any other tool are only implied, but they can be addressed via the tool. The reference to gestures shows the connection between tools over time: The hammer of a blacksmith and the hand axe of the Neanderthal afford similar – yet different  – gestures of the bearer of the tool. Both tools and gestures have been, however, transformed over time, are part of operational chains that depend on the socio-technial and historical milieu and have specific choreographic affordances (Richter/Breyer this volume). In order to use these tools, I have to adapt myself to the tools and create operational chains with the tools. My body has to learn new gestures and operational chains that replace or enhance my abilities to cut (hand axe), punch (hammer), or to communicate (smartphone). The socio-technical engagement thus also transforms my being human.

4 Conclusion – Technology as Interdisciplinary Anthropology Aims to Understand, Interpret and Invent Modes of Being Human Modes of being human signify a multiplicity of possible forms of being human. These forms differ historically and culturally. Essentially, we do not want more or less from life as our ancestors. Technologies are and always have been part of the social fabric of life. However, how technical objects are used, their mode of production and their commodification affects how we live as individuals not only in a society, but also in our ecosystem. Human beings form sociocultural niches that transform the biosphere on a large scale and endanger the very existence of the human race (Ellis, 2015). The question of how to relate to nature has become increasingly intertwined with the question of how to relate to technology. These questions constitute a philosophical problem. They ask for the position of the human being in and with the world. To tackle this problem, Technology as interdisciplinary anthropology can no longer begin with an absolute transcendental subject. It rather has to acknowledge that subjectivity itself is historically contingent and depends on social as well as technical practices. The multiple relationships of human beings, techniques and nature are part of life in general. Techniques are expressions of life. The logos of techné is thus rooted within nature. it is impossible to distinguish between different gestures. As Morgenstern and Boutet show, the multimodal (e.g. communicative and manipulative) aspects of gestures are simultaneaously coordinated in everyday situations such as dining (Morgenstern/Boutet, this volume). To relate Morgenstern’s understanding of gesture to Simondon’s, one could argue that gestures crystallized in technical objects have both aspects: They are at the same time communicative - since they indicate a meaning - and manipulative - since they invite to manipulate the technical object.

180

J. F. M. Schick

The equivocal nature of the organism exemplifies this point. As Raymond Ruyer points out, an organism is simultaneously the ensemble of organs and the unity of these organs that is fabricating and utilizing them (Ruyer, 1995, 23). It follows that there is an essential difference between technical objects and living beings: living beings contain something that precedes any artificiality and is itself “unobservable” (ibid.). This unobservable aspect is essential to the human organism, since it can bring about organic unities (the organs within the body), which can as unity create automatic, non-organic machines. These non-organic machines (for instance a computer) can in turn control other non-organic machines (ibid.) The human organism is as organism primary in a double sense to the technical objects that form a network with him.18 It shares its observable, spatial aspects, it occupies a location, but contains an inherent unobservable aspect, which institutes and holds the operational chains. The essential difference between living beings and artificial objects provides the living being with a relative primacy. It remains relative to the socio-technical practices, but it indicates that the human organism has, as an informational being, the epistemic privilege and responsibility to acknowledge, understand and form the agency surrounding it. The epistemic primacy of human beings is based on bodily and cognitive practices that involve technical operations. A critical distancing from these practices is nevertheless necessary in order to formulate the problems of our socio-technical entanglement. To abandon the position of an absolute transcendental subject does not imply to give up the position of an epistemic subject that attempts to understand its position in relation to other (natural, technical or social) agents in the network. It rather allows to question the subject’s possibilities of participation and the role of other agents. Due to the encompassing nature of contemporary socio-technical network, the role of the epistemic subject has been transformed. Since there is no longer a nature to be transformed with tools, but rather a socio-technical network within one participates, the philosophical work is interpretation before transformation: ‘The philosophers have only interpreted the world, in various ways; the point is to change it.’ In the century after Marx, philosophers changed the world in various ways; the point now is to interpret it. The purpose of philosophy now is interpretation. The interpretation of inscribed possibilities is the main task of philosophy in our time. We must stubbornly

 “Le mot organisme est profondément équivoque: il désigne à la fois l’ensemble des organes, et l’unité fabriquante et utilisante de ces organes. La fabrication des machines à calculer et à raisonner est seconde relativement à la fabrication embryogénique du cerveau vivant. Admettons que le fonctionnement des circuits et des aiguillages nerveux soit de même nature que le fonctionnement des circuits électriques, cela prouverait simplement, une fois de plus, comme on le sait depuis longtemps, qu’il y a des machines dans l ‘organisme, mais non pas que l’être organisé est machine. Cela prouverait que l’être inobservable qui se manifeste comme première cellule humaine est capable de construire, sans machine, des machines organiques capables à leur tour de fabriquer des machines automatiques non organiques, qui, elles-mêmes peuvent contrôler des machines non automatiques. Cela prouverait que ce que l’on appelle l’organisme est à la fois ce qui est observable dans l ‘espace et un x inobservable, qui tient toute la chaîne des automatismes internes et externes.” (Ruyer, 1995, 23 f.) 18

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

181

search for concepts and percepts which may help to develop the immanent possibility inscribed in networked knowledge. (Berardi, 2017, 234)

The conditions of socio-technical practices have been transformed since the publication of the Theses on Feuerbach. If world means the natural, material and social infrastructure of thought and action, then there are hardly any areas that have not yet been transformed by the forces of human existence. We no longer live in a world that we can simply change, but access with analogue and digital tools a world that has already been radically transformed. As mentioned above, we are even reaching beyond the boundaries of the globe to include other planets and moons in our socio-­ technical networks. The technical objects that we use do no longer shape nature or the world, but provide access to an already designed world and enable different degrees of participation. While our ancestors in the Neanderthal were still directly involved in the production of hand axes and shaped their environment directly, in the digital age humans are dependent on production and supply chains that remain closed to them. Instead, the socio-technical network spans the globe and opens up new spaces of operation and perception (Engell & Ziemann, 2019, XIII). A modern human being lives in a paradox: While she possesses and executes a multitude of new possibilities of action in the network, she is unable to reconstruct their conditions of possibility. Although the entanglement in the network provides the human being with a multitude of possibilities, the human being is also exposed to the network as an infrastructure, in which informations are stored, behavior is predicted, access is at times granted and at times denied. The human being’s mode of being in the twenty-first century is therefore no longer capable of directly changing the world. Rather, it is necessary to interpret the network differently in order to indicate potentials, that is, possibilities for transformation in the network. Bifo’s call for interpretation is by no means a swan song to philosophy’s ability to bring about change. Philosophy needs to reclaim the mastery of its own practice, i.e. of inventing concepts (Deleuze, 1987), in order to pioneer ways in which “networked knowledge” is generated. This can be done through the “invention” of new concepts and perceptions: i.e. new dispositives that generate a different view of the network and participation in the network. Bifo’s project opposes a philosophy that wants to “uncover” and “reveal” the essential, true nature of the world. According to him, there is no natural or original access to the world to reveal, as a philosophy of technology in the spirit of Heidegger propagated. This might be a sobering perspective in stark contrast to Marx’s task of philosophy to change the world, but it also frees philosophy from its own navel-­ gazing and the tendency to be blindsided by ones own deformation professionelle. Philosophical concepts are influenced and shaped by other sciences, cultural and social constellations, which also include technical objects and socio-technical practices. Philosophy, if it wishes to address the genesis of its own concepts, has to abandon its claim to be the “first” science insofar as its concepts are derived from practices and are historically and culturally contingent. As logos of techné, however,

182

J. F. M. Schick

philosophy creates potentials to “interpret the present composition of the networked brain according to social well-being” (Berardi, 2017, 235). What does this mean for Technology as an interdisciplinary anthropology? Technology in the literal sense, as already found in Bergson, Mauss, Lafitte19 and Simondon, has sought to create dispositives to understand technical objects. Technical objects and the relations with technical objects have changed, yet it remains a philosophical task, to understand technical operations and technical objects as part of the human being and as an expression of living processes, as Jacques Lafitte emphasized in 1933 – and Latour repeated in 1994 – in order to symmetrize the human being, the environment and technology: Machines? Extension of man, integrating into himself, extension of social structures and integrating into them, they are, at all times, identical to ourselves. They are us; they are, like us, beautiful, and ugly, like us. To develop them, to construct them, is to construct ourselves. (Lafitte, 1933, 101 translated by and quoted from Illiadis 2015, 135)

Lafitte points to a fundamental relation that has to be interpreted. The machines as beings are “us [emphasis by the author], like us, beautiful, and ugly”. They instigate an effort to interpret them as such, instead of merely using them as a means to an end. It also points to a multiplicity within human beings to relate to technical objects. The way we interact with “artefactors” (Gerner this volume), whether or not we attribute intentionality to robots (Pieters; Vermaas this volume) and how they are mobilized and choreographed in socio-political networks (Parviainen/ Coeckelbergh this volume) shape how we as human beings conceive ourselves and reveal how these conceptions are co-created with technical objects. Each conception contains a potentiality. The interpretation of these potentialities consists in a philosophical effort to interpret, invent and re-invent concepts and practices. The philosophical task of an interdisciplinary anthropology is to indicate and map potentials of socio-technical practices, technical objects and modes of being human by creating conceptual dispositives (e.g. “artefactors”, “choreographic affordances” etc.). The concepts can be directed at the self-understanding of the human being. The “homo faber” (Bergson, 1998), the “homo duplex” (Durkheim, 2013), the “homme total” (Mauss, 2005), the “homo faber socialis” (Latour, 1994), the “cyborg” (Haraway, 1991), the “homo coordinans” (Simondon, 2017; Schick, 2018), the “Inforg” (Floridi, 2014) each are expressions of the way humans relate to techniques, and to the world. Methodologically, they were each developed in confrontation with different sciences, which lead to a reformulation of the conception of “what is human”. In this respect, they each express a particular mode of being human. The individual concepts and modes of being human, however, should not be played off against each other. Rather, they offer a kaleidoscopic view (Mauss, 2005, 52) of how being human can be conceived: as social and individual being (Durkheim); as a “coordinator in the midst of machines” (Simondon); as a tool-making being (Bergson); as an information-generating being (Floridi); or as a cyborg that has  Jacques Lafittes mechanology is ultimately another name for technologie, since it also strives to overcome the opposition of technique and culture. 19

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

183

always been in a reciprocal relation with tools as parts of its thought and action (Haraway). The respective concepts of the human are paired with conceptions of the technical. While for Bergson the technical was still primarily conceived as the translation of dynamic processes into static entities, the focus shifts to the body as a technical instrument and institution in the Durkheim school. Lafitte and Simondon devote themselves to the genuine mode of existence of the technical object. Haraway focused on how biomedicine transformed the relation to our own body: Contemporary cancer therapy creates new, minimalized tools out of living material which are interiorized and show the hybrid structure of human beings (Schick 2021a). Being human in the twenty-first century involves participating in various socio-­ technical networks. In each of these networks, one takes on different roles. As a bricoleur the subject becomes a homo faber and links tools with its body. As soon as youtube-videos are watched that inform the subject on techniques of how to build a shed, the subject becomes an inforg that produces new information within the web. S/he can, however, go back and transform these informations into gestures by being a homo faber. The hybrid nature of the human self becomes evident when new forms of biotechnologies blur the line between the technical and the living for instance in cancer therapy. However, these modes of human beings have to be seen in their critical potentiality. The “inforgs” of today in form of big tech companies have created artefactors that challenge us to conceptualise and stress the otherness of technical and digital objects, as Alexander Gerner points out in his Chapter “Playing with Arte(f)actors” (Gerner this volume). A central aspect of “technology as interdisciplinary anthropology” is thus to question how new socio-technical practices with technical objects and their infrastructure transform, create and challenge modes of being human. To establish a tableau of multiple modes of being human allows thus to mark differences in the engagement with technical objects, to address the social and political implications of this engagement and to study how conceptions of being human are generated. They might even allow to go back to past and forgotten forms of being human. Technology as interdisciplinary anthropology implies a call for intellectual modesty towards the social, the technical and the cultural other. To invent new modes of being human seems to be at first sight a project of megalomanic proportions. But I neither want to invent a new discipline nor can I provide answers to the pressing issues of the relations of human beings to nature and techniques in the twenty-first century. The philosophical task of this volume was to provide some guidelines of how to pose these questions. We have to ask ourselves what kind of human beings we are and how we human beings and philosophers interpret the potentials of humanity for a “social well-being”. I believe that we contain virtually multiple modes of being human: we can be a bricoleur, a homo faber, a homo coordinans, a cyborg etc. We are in the midst of technical networks are choreographers of and choreographed by sociotechnical networks (see Parviaainen/Coeckelbergh this volume). The modes of being human belong to us as well as we belong to the socio-­ technical networks. These modes contain potentialities that can shift interpretations

184

J. F. M. Schick

and can show us to which degree we participate in socio-technical networks. The effort to invent ourselves continuously and to understand the human being as an open living being rooted in networks with social, technical and natural entities, paves the way for an ethics of participation that critically assesses and questions our being with nature and technical ensembles.

References apple.com (2020). “Press Release 15 September 2020”. Retrieved September 9, 2022 from https://www.apple.com/newsroom/2020/09/apple-­unveils-­all-­new-­ipad-air-­with-­a14-­bionicapples-­most-­advanced-­chip/ Arthur, W. B. (2009). The Nature of Technology: What It Is and How It Evolves. Penguin Books. Austin, J. L. (1962). How to do things with words. Clarendon Press. Barberis, D.  S. (2002). Moral education for the elite of democracy: The Classe de Philosophie between sociology and philosophy. Journal of the History of the Behavioural Sciences, 38(2), 355–369. Barad, K. (2007). Meeting the universe halfway: quantum physics and the entanglement of matter and meaning. Duke University Press. Berardi, F. (2017). Futurability: the age of impotence and the horizon of possibility. Verso. Bennett, J. (2010). Vibrant matter: a political ecology of things. Duke University Press. Bergson, H. (1896). Matter and memory (N. Margaret Paul & W. S. Palmer., Trans.). Zone Books. Bergson, H. (1998). Creative evolution. (Arthur Mitchell, Trans.) Dover. Bergson, H. (1932). The two sources of morality and religion. (R. Ashley Audra & Cloudesley Brereton, Trans.) University of Notre Dame Press. Bouglé, C. (1922). Les origines de la technique positive. In Leçons de sociologie sur l’évolution des valeurs (pp. 157–177). Colin. Camolezi, M. (2022). Technique, Technics, Technik, substantifs: les mots à travers les dictionnaires du xixe siècle. Artefact, Nr. 15 (Februar): 61–106. https://doi.org/10.4000/artefact.11313 Canguilhem, G. (2009). Machine et Organisme. In La conaissance de la vie (pp.  129–164) Librairie philosophique J. Vrin. Carnino, G. (2018). Unbehagen in der Transduktion: Technik und Technologie im digitalen Zeitalter. In J. F. M. Schick, M. Schmidt, U. van Loyen, & M. Zillinger (Eds), Homo Faber (Zeitschrift für Kulturwissenschaften, Nr. 2/2018, pp. 67–81). Cassirer, E. (2004). Form und Technik (1930). In Cassirer. E., Gesammelte Werke. Hamburger Ausgabe. Aufsätze und kleine Schriften (1927–1931) (Vol. 17, pp. 139–183). Ed. by B. Recki. Felix Meiner Verlag. Coupaye, L. (2022). Danse avec les catégories: anthropologie de la «technologie» et anthropologie des techniques. Artefact, Nr. 15 (Februar): 127–50. https://doi.org/10.4000/artefact.11426 Deleuze, G. (1987). “Qu’est-ce que l’acte de création?” Conférence donnée dans le cadre des mardis de la fondation Femis. Retrieved December 16, 2023, from https://www.webdeleuze. com/textes/134 Descola, P. (2014). Beyond Nature and Culture. (Janet Lloyd, Trans.). The University of Chicago Press. Durkheim, É. (1901). Technologie (II  – Technologie, par M.  H. Hubert [signé E.D]. L’Année sociologique 4, 593–594. Durkheim, É. (1909). Sociologie religieuse et théorie de la connaissance. Revue de Métaphysique et de Morale, 17(6), 733–758. Durkheim, É. (1995). The Elementary Forms of Religious Life (Karen E. Fields, Trans). Free Press. Durkheim, É. (2013). The rules of sociological method and selected texts on sociology and its method (W. D Halls, Trans.). Ed. by S. Lukes. Palgrave Macmillan.

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

185

Durkheim, É., & Mauss, M. (2003). In R. Needham (Ed.), 1. publ. 1963, repr Primitive classification. Cambridge University Press. Ellis, E.  C. (2015). “Ecology in an Anthropogenic Biosphere”. Ecological Monographs 85(3), 287–331. Engell, L., & Ziemann, A. (2019). “Vorwort”. In Grundlagentexte der Medienkultur: ein Reader. Ziemann, A., Bee, J., Cuntz, M., Engell, L., Frisch, S., Hiller, M., Paulus, J. u. a., (Eds.) (pp. XI–XV). Springer VS. Floridi, L. (2014). The 4th revolution: how the infosphere is reshaping human reality. First edition. New York; Oxford: Oxford University Press. Ghanbari, N. & Hahn, M. (Eds.) (2013). Reinigungsarbeit. Zeitschrift für Kulturwissenschaften, 2013,1. Transcript-Verl. Guillerme, A. (2017). De Diderot à Taylor: Exprimer les techniques du geste. In D.  Bouillon, A. Guillerme, M. Mille, & G. Piernas (Eds.), Gestes techniques, techniques du geste: approches pluridisciplinaires (pp. 449–462). Presses universitaires du Septentrion. Guillerme, J. & Sebestik, J. (2007). “Les commencements de la technologie”. Documents pour l’histoire des techniques, Nr. 14 (December): 49–122. https://doi.org/10.4000/dht.1226 Harman, G. (2018). Object-oriented ontology: a new theory of everything. Pelican book 18. London: Pelican Books. Haraway, D. J. (1991). Simians, cyborgs, and women: The reinvention of nature. Routledge. Habscheid, S. (2023). “Socio-Technical Dialogue and Linguistic Interaction. Intelligent Personal Assistants (IPA) in the Private Home”. Sprache und Literatur 51(2), 167–196. https://doi. org/10.30965/25890859-­05002020 Haudricourt, A-G. (1987). La technologie science humaine. Recherches d’histoire et d’ethnologie des techniques. Paris: Éditions de la maison des sciences de l’homme. Hector, T., Niersberger-Gueye, F., Petri, F. & Hrncal, C. (2022). The ‘conditional voice recorder’: data practices in the co-operative advancement and implementation of data-collection technology. Universitätsbibliothek Siegen. https://doi.org/10.25819/UBSI/10225 Holbraad, M. (2012). Truth in motion: The recursive anthropology of Cuban divination. The University of Chicago Press. Holbraad, M. (2013). Things as concepts: Anthropology and Pragmatology. In G. Pereira (Ed.), Savage Objects (pp. 17–30). INCM. Ingold, T. (2010). “Bringing Things to Life: Creative Entanglements in a World of Materials”. Realities Working Papers: Bringing Things to Life, NCRM Working Paper Series, #15 (Juli): 1–14. Retrieved December 16, 2023, from: http://eprints.ncrm.ac.uk/1306/1/0510_creative_ entanglements.pdf Ingold, T.(2011). Being alive: essays on movement, knowledge and description. Routledge. Kneer, G., Schroer, M. & Schüttpelz, E. (Eds.). (2008). Bruno Latours Kollektive: Kontroversen zur Entgrenzung des Sozialen. Suhrkamp. Lafitte, J. (1933). Sur la science des machines. Revue de la Synthèse VI, 2, 143–158. Latour, B. (1993). We Have Never Been Modern. (Trans. Catherine Porter) Harvard University Press. Latour, B. (1994). On technical mediation: Philosophy  – Sociology  – Genealogy. Common Knowledge, 3(2), 29–64. Latour, B. (2018). An inquiry into modes of existence: an anthropology of the moderns. (Catherine Porter, Trans.). Harvard University Press. Latour, B. & Woolgar, S. (1986). Laboratory life: the construction of scientific facts. Princeton University Press. Latour, B., Harman G. & Erdélyi, P. (Eds.) (2011). The Prince and the Wolf: Latour and Harman at the LSE. Zero Books. Leroi-Gourhan, A. (1964). Le geste et la parole. 2: La mémoire et les rythmes. Michel. Leroi-Gourhan, A. (2018). Gesture and Speech. Cambridge (Massachusetts)/London (England): MIT Press. Loeve, S., Guchet, X., & Bensaude Vincent, B. (Eds.). (2018). French philosophy of technology. Bd. 29. Philosophy of engineering and technology. Springer.

186

J. F. M. Schick

Mahony, A. (2014). The Day the Pintupi Nine entered the modern world. BBC News Magazine, 2014. Retrieved September 9, 2022, from https://www.bbc.com/news/magazine-­30500591 Malafouris, L. (2013). How things shape the mind: a theory of material engagement. MIT Press. Marx, K. (1958). ‘Thesen über Feuerbach’. In Marx, K. & Engels, F.  Werke, 3: (pp.  5–7). Dietz Verlag. Morton, T. (2013a). Hyperobjects: philosophy and ecology after the end of the world. University of Minnesota Press. Morton, T. (2013b). Realist Magic: Objects, Ontology, Causality. Open Humanities Press. Mauss, M. (1934). Fragment d’un plan de sociologie générale descriptive. Classification et méthode d’observation des phénomènes généraux de la vie sociale dans les sociétés de type archaïque (phénomènes généraux spécifiques de la vie intérieure de la société). Annales sociologiques Série A, fasc., 1, 1–56. Mauss, M. (1974). Mentalité primitive et participation. In Œuvres Tome 2. Les Éditions de Minuit. Mauss, M. (2001). A general theory of magic. Routledge Classics. Mauss, M. (2005). Sociology: Its divisions and their relative weightings (William Jeffrey, trans.). In The nature of sociology: Two essays (pp. 31–85). Durkheim Press. Mauss, M. (2006). Techniques of the body. In N.  Schlanger (Ed.), Techniques, technology and civilisation (pp. 77–95). Durkheim Press/Berghahn Books. Mauss, M. (2008). On prayer (Susan Leslie, Trans., Ed. by W. S. F. Pickering). Durkheim Press/ Berghahn Books. Merllié, D. (1993). Les rapports entre la Revue de métaphysique et la Revue philosophique: Xavier Léon, Théodule Ribot, Lucien Lévy-Bruhl. Revue de Métaphysique et de Morale: Numéro spécial du centenaire La fondation de la Revue Tables 1893–1992 (Janvier-Juin 1993), 98(1–2), 59–108. Peristiany, J. G. (2010). Introduction. In É. Durkheim (Ed.), Sociology and philosophy. Routledge. Rawls, A. W. (2004). Epistemology and practice: Durkheim’s the elementary forms of religious life. Cambridge University Press. Reckwitz, A. (2003). Grundelemente einer Theorie sozialer Praktiken. Zeitschrift für Soziologie 32 (4): 282–301. Ruyer, R. (1995). La cybernétique et l’origine de l’information. Flammarion. Schatzki, T.  R. (2001). “Introduction”. In The practice turn in contemporary theory, (Eds.) Theodore R. Schatzki, K. Knorr-Cetina, und Eike von Savigny, 10–23. New York: Routledge. Schick, J. F. M. (2012). Erlebte Wirklichkeit: zum Verhältnis von Intuition zu Emotion bei Henri Bergson. Ästhetik und Kulturphilosophie, 6, Lit-Verl. Schick, J. F. M. (2018). Mechanik, Mystik und stumme Intelligenz: Doch eine Homo faber-Story. In J. Schick, M. Schmidt, U. van Loyen, & M. Zillinger (Eds.), Homo Faber. Zeitschrift für Kulturwissenschaften, Nr. 2/2018 (pp. 67–81). Schick, J. F. M. (2021a). “’Mein fremder Killer in mir’: Menschen, Organismen und Designer-T Zellen”. In E. Dzwidza, & A. Speer (Eds), Philosophische Anthropologie als interdisziplinäre Praxis. Max Scheler, Helmuth Plessner und Nicolai Hartmann in Köln – historische und systematische Perspektiven, (pp. 364–384). Brill/mentis. Schick, J. F. M. (2021b). The potency of open objects. (re-)inventing modes of being human with Henri Bergson, Franco ‘Bifo’ Berardi and Gilbert Simondon. Techné. Research in Philosophy and Technology, 25(3), 379–406. https://doi.org/10.5840/techne2021723144 Schick, J. F. M., Schüttpelz, E., & Pieper, J. (2021). Transformationen der techne. Ein Gespräch. In N. Engelhardt & J. F. M. Schick (Eds.), Erfinden, Schöpfen, Machen (pp. 73–110). transcript Verlag. https://doi.org/10.14361/9783839448373-­003 Schick, J.  F. M., Schmidt, M., & Zillinger, M. (Eds.). (2022). The social origins of thought: Durkheim. Mauss and the Category Project. Schmidt, M. & Koddenbrock. K. (2019). “Against Understanding: The Techniques of Shock and Awe in Jesuit Theology, Neoliberal Thought and Timothy Morton’s Philosophy of Hyperobjects”. Global Society 33(1), 66–81. https://doi.org/10.1080/13600826.2018.1539954

The logos of techné: A Case for Technology as Interdisciplinary Anthropology

187

Schüttpelz, E. (2021). Vom Werkzeug zum Behälter, vom Behälter zum Medium: Die Ausweitungen des Körpers. In N.  Engelhardt & J.  F. M.  Schick (Eds.), Erfinden, Schöpfen, Machen (pp. 25–72). transcript Verlag. https://doi.org/10.14361/9783839448373-­002 Sigaut, F. (2013). Homo faber documents. Retrieved September 9, 2022, from http://francois-­­ sigaut.com/index.php/inedits/53-­ouvrages-­inedits/527-­homo-­faber-­documents. Simondon, G. (2005a). Forme, information, Potentiels. In L’individuation à la lumière des notions de forme et d’information (pp. 537–558). Millon. Simondon, G. (2005b). Notes complémentaires sur les conséquences de la notion de l’individuation. In L’individuation à la lumière des notions de forme et d’information (pp. 331–355). Millon. Simondon, G. (2014). Sur la technique (1953–1983) (1re édition ed.). Presses universitaires de France. Simondon, G. (2017). On the mode of existence of technical objects. Univocal Pub. Weber, L. (1913). Le rythme du progrès. Félix Alcan. Wiener, N. (2007). Cybernetics or control and communication in the animal and the machine (2nd ed.., 14th print.). MIT Press. Worms, F. (Ed.). (2004). Le moment 1900 en philosophie. Presses universitaires du Septentrion. https://doi.org/10.4000/books.septentrion.73764 Worms, F. (2013). Bergson ou Les deux sens de la vie. Presses Universitaires de France. Zuboff, S. (2019). The Age of Surveillance Capitalism: The Fight for the Future at the New Frontier of Power. Profile Books.