Sounds from Within: Phenomenology and Practice (Numanities - Arts and Humanities in Progress, 18) 3030725065, 9783030725068

Citation preview

Numanities - Arts and Humanities in Progress 18

Paulo C. Chagas Jiayue Cecilia Wu   Editors

Sounds from Within: Phenomenology and Practice

Numanities - Arts and Humanities in Progress Volume 18

Series Editor Dario Martinelli, Kaunas University of Technology, Kaunas, Lithuania

The series originates from the need to create a more proactive platform in the form of monographs and edited volumes in thematic collections, to contribute to the new emerging fields within art and humanistic research, and also to discuss the ongoing crisis of the humanities and its possible solutions, in a spirit that should be both critical and self-critical. “Numanities” (New Humanities) aim at unifying the various approaches and potentials of arts and humanities in the context, dynamics and problems of current societies. The series, indexed in Scopus, is intended to target an academic audience interested in the following areas: – Traditional fields of humanities whose research paths are focused on issues of current concern; – New fields of humanities emerged to meet the demands of societal changes; – Multi/Inter/Cross/Transdisciplinary dialogues between humanities and social and/or natural sciences; – Humanities “in disguise”, that is, those fields (currently belonging to other spheres), that remain rooted in a humanistic vision of the world; – Forms of investigations and reflections, in which the humanities monitor and critically assess their scientific status and social condition; – Forms of research animated by creative and innovative humanities-based approaches; – Applied humanities.

More information about this series at http://www.springer.com/series/14105

Paulo C. Chagas · Jiayue Cecilia Wu Editors

Sounds from Within: Phenomenology and Practice

Editors Paulo C. Chagas Department of Music—Composition University of California, Riverside Riverside, CA, USA

Jiayue Cecilia Wu Department of Recording Arts University of Colorado Denver Arvada, CO, USA

ISSN 2510-442X ISSN 2510-4438 (electronic) Numanities - Arts and Humanities in Progress ISBN 978-3-030-72506-8 ISBN 978-3-030-72507-5 (eBook) https://doi.org/10.1007/978-3-030-72507-5 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Introduction: Transcending from Within

What is sound and how do we express ourselves in and through sonic worlds? This anthology combines phenomenological and pragmatic approaches to sound and music and is written by authors who are both scholars and artists. As an interdisciplinary field at the crossroads of science and art, music is attached to sound and embodies sound as physical matter and form. Music shapes sound just as sound shapes music. Through sound and noise, music evolves—with sound and music connecting and disconnecting to each other constantly. These essays are reflections of the human experience through sound and music. The authors’ contributions articulate a dialogue between various fields of knowledge and inquiry from philosophical accounts to descriptive experiences of artistic and musical creativity. These different views all focus on the theory and practice of three main genres: classical, popular, and digital music. The essays provide a pluralistic account of both sound phenomena and the technology of sound underlying sonic consciousness. The collection unfolds multiple views for engaging with sound and music and for investigating them in multi-relational, artistic, and research contexts. The authors are both subjects and objects of the observations—observers and self-observers. The nine essays contained here elaborate on a broad range of topics including sound, truth, and the notion of paradigm in art and science (Chagas); the system of existential semiotics and the zemic model (Tarasti); the qualities and flow of the imagined sound (Chafe); the decoding of the imaged sound (May/Casey); the inquiry on the extramusical experience of listening (Varankait˙e); the quest of the sonic self (Navickait˙e-Martinelli); the notion of sound in popular music (Martinelli), the spiritual and digital experience of embodied sonic meditation (Wu); and the nomadic experience of sonic imagination (Forcucci). A phenomenological approach to sound has its roots in Husserl’s phenomenology of time consciousness, which provides us with an introspective and robust method for investigating the temporal characteristics of physical objects. In Husserl’s account, listening to a melody is an emblematic metaphor of how the present is experienced as an extended frame of simultaneity. Studying sound in relation to both the lived human experience and the possibilities of transforming the living experience requires a multilayered approach that includes the memory of the past and the prospection of the v

vi

Introduction: Transcending from Within

future. This anthology offers diverse philosophical ideas and theories to illuminate semiotic and transcendental dimensions of experience, opening a broad scope of analytical approaches for further investigation. Emerging from this broad phenomenological field, this work collectively proposes observations on the shift from aesthetic to post-aesthetic, from the reality of the subject/object relation of our own subjectivity to the primordial level of a practically engaged “hands-on” existence—a mode of being in which self and world are unified. The aesthetic approach, as Heidegger claimed, has resulted in Western humanity attempting to understand and experience the work of art in a way that occludes its true historical significance. This collection offers an alternative, as it seeks to engage with the richness of sound and music practices—from classical to electroacoustic music to contemporary sound art and multimedia—while conceptually accessing their complexity. Moving into new and rich areas of experience necessitates novel approaches and methodologies. Wittgenstein suggests that sound is merely the surface of music and that the musical work conceals something more profound that can hardly be described by philosophical models or scientific theories. Therefore, the infinite complexity of music can only be understood on a more fundamental level of experience that creates meaning beyond what is expressed on the surface. As Heidegger has reminded us, the best way to get beyond aesthetic experience is to transcend it from within, as the title of this book references. In the poetic of art, we see ourselves, we learn to hear into the nothing, and from there, disclose meaningful possibilities for understanding and enjoying sound and music. Paulo C. Chagas Jiayue Cecilia Wu

Contents

1 Sound, Truth, and Paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Paulo C. Chagas 2 Existential Semiotics and Its Application to Music: The Zemic Theory and Its Birth from the Spirit of Music . . . . . . . . . . . . . . . . . . . . . Eero Tarasti

1

29

3 The Qualities and Flow of Imagined Sound and Music . . . . . . . . . . . . . Chris Chafe

57

4 Decoding Imagined Sound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lloyd May and Michael Casey

75

5 Music, Listener and Extramusical Experience . . . . . . . . . . . . . . . . . . . . . 107 Ulrika Varankait˙e 6 In Quest of the Sonic Self: Sound as Expression of the Performer’s Individuality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Lina Navickait˙e-Martinelli 7 The Notion of “Sound” in Popular Music . . . . . . . . . . . . . . . . . . . . . . . . . 143 Dario Martinelli 8 Experiencing Embodied Sonic Meditation Through Body, Voice, and Multimedia Arts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Jiayue Cecilia Wu 9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Luca Forcucci Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

vii

Editors and Contributors

About the Editors Paulo C. Chagas is a Professor of Composition at the University of California, Riverside. A highly versatile and prolific composer, Chagas, has written over 160 works for orchestra, chamber music, electroacoustic music, audiovisual, and multimedia compositions. His music unfolds a pluralistic aesthetic, combining diverse musical materials from different cultures with acoustic and digital media, dance, video, and audiovisual installations. His award-winning and ambitious productions have been applauded throughout the Americas, Europe, and Asia. Chagas’ scholarly activity has organically evolved alongside his artistic career. He has written a significant number of book chapters, journal articles, and proceedings in his four languages of English, Portuguese, German, and French. Unsayable Music (Leuven University Press, 2014) develops in great detail the main themes of Chagas’ research, which include electroacoustic and digital music, musical semiotics, and philosophy. Jiayue Cecilia Wu (AKA: 武小慈), originally from Beijing, is a scholar, composer, vocalist, multimedia technologist, and audio engineer. Cecilia earned her Bachelor of Science degree in Design and Engineering in 2000. She then worked as a professional musician at EMI Records and Universal Music Group for ten years. In 2010, Cecilia produced her original album of spiritual electronic music, Clean Your Heart. In 2013, Cecilia obtained her Master of Arts degree in Music, Science, and Technology at the Center of Computer Research in Music and Acoustics (CCRMA) at Stanford University. In 2018, Cecilia earned her Ph.D. in Media Arts and Technology from the University of California, Santa Barbara, where she studied computer music with Dr. Curtis Roads. As an audio engineer, she received a two-year grant award from the Audio Engineering Society (AES). As a musician, she received an award from the California State Assembly for being a positive role model in sharing Chinese culture. As a multimedia artist, she received the “Young Alumni Arts Project Grant Award” from Stanford University. Her work has been exhibited at museums and international arts and engineering society such as the National Museum of China, Denver Art Museum, and IEEE VIS. Her piece Virtual Mandala has been selected ix

x

Editors and Contributors

by the Denver Art Museum for its permanent collection of Asian Art. Cecilia is also a U.S. National Academy of Sciences Sackler Fellow. Currently, she is Assistant Professor at the University of Colorado’s College of Arts and Media.

Contributors Michael Casey Dartmouth College Hanover, Hanover, NH, USA Jiayue Cecilia Wu University of Colorado, Denver, USA Chris Chafe Center for Computer Research and Acoustics, Stanford University, Stanford, USA Paulo C. Chagas University of California, Riverside, CA, USA Luca Forcucci Berlin, Germany Dario Martinelli Kaunas University of Technology, Kaunas, Lithuania Lloyd May Dartmouth College Hanover, Hanover, NH, USA Lina Navickait˙e-Martinelli Lithuanian Academy of Music and Theater, Vilnius, Lithuania Eero Tarasti University of Helsinki, Helsinki, Finland Ulrika Varankait˙e Kaunas University of Technology, Kaunas, Lithuania; Vilnius Gediminas Technical University, Vilnius, Lithuania

Chapter 1

Sound, Truth, and Paradigm Paulo C. Chagas

Abstract This essay proposes a phenomenology of sound and music focused on the notions of truth and paradigm. It begins with Heidegger’s view of art as a tool for achieving truth, followed by Kuhn’s ideas on advances in science by way of paradigms and revolutions. The notion of paradigm as a disciplinary matrix leads to a comparative approach between science and art, aiming to observe similarities and differences. With a brief look at Wittgenstein’s concept of the language game, we will examine the notions of paradigm and signature in Agamben’s philosophical archeology. Three cultural paradigms within the history of Western music are explored: vocal, instrumental, and electroacoustic. Delving into the notion of apparatus as a process of subjectivation, we will investigate how the electroacoustic paradigm builds capacity for apparatuses to expand consciousness of sound phenomena. A paradigm shift has occurred from a subjective hermeneutic view of the world, to a thinking shaped by cybernetics and information theory. The automaticity of music apparatuses is viewed through several lenses, including the way that the electroacoustic paradigm inaugurated the era of music as a torture apparatus. Flusser’s notion of the telematic paradigm highlights the communicative complexity of modern society and the idea of chamber music as a model for dialogic telematic communication. A final question is posed on how creativity continues to shape contemporary music. Keywords Sound · Truth · Paradigm · Electroacoustic music · Heidegger · Kuhn · Wittgenstein · Agamben · Flusser

1.1 Art and Truth In The Origin of the Work of Art, Heidegger (1993) advances the view of truth as the essence of art. The truth of art is not a matter of beauty or accuracy of representation, but rather, it provides a view of the world to show how things really are. By moving from an aesthetic experience of art to a post-modern approach, art helps us develop a P. C. Chagas (B) University of California, Riverside, CA 92521, USA e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 P. C. Chagas and J. Cecilia Wu (eds.), Sounds from Within: Phenomenology and Practice, Numanities - Arts and Humanities in Progress 18, https://doi.org/10.1007/978-3-030-72507-5_1

1

2

P. C. Chagas

better understanding of the being of entities. For Heidegger, true means disclosing; the essence of truth lies in the unconcealment of beings. Heidegger’s philosophical method consists of providing descriptions that disclose the deep roots of this phenomenon. Works of art operate like phenomenological descriptions to illuminate a world while providing a view from within.1 Borrowing from Husserl’s terminology, we might say that works of art are objects, events, and streams of consciousness. Heidegger takes this a step further. The work of art provides a coherent way of being, or style—that conjures a world and shines a light on the struggle through which things appear in that world. Each work of art functions to “remove ourselves from our commonplace routine and move into what is disclosed by the work, so as to bring our essence itself to take a stand in the truth of beings” (Heidegger 1993, 199). Art is a setting-into-work of truth—a distinctive way for truth to come into being and become historical. The creative preserving of truth in a work of art is what allows it to articulate a style within a culture. Heidegger’s notion of style can be compared to what Wittgenstein calls a “form of life.” By sharing resources, roles, or practices of a specific culture, art becomes a form of life. Music has a privileged status in Wittgenstein’s philosophy, as he considered it the most sophisticated art for providing a simple surface (the sound) that conceals an infinite complexity we try to understand by using language. For him, “understanding” is not a process of going inwards but of looking at the surface of our practice, something that is connected to the complexity of patterns that characterize our form of life. Wittgenstein’s philosophy uses music as a tool for reflecting on the understanding of language and understanding in general. As a language onto itself, music is not an abstract system of signs conveying some kind of meaning, but a particular form of life that displays structures shaped by the practices of music. Wittgenstein invites us to look at the way we determine musical concepts, constitute regular patterns by following some rules while disregarding others, and apply words like “correct” or “incorrect” to make aesthetic judgments about music and in general (Chagas 2015, 301). In addition to Heidegger and Wittgenstein, we must also consider Kuhn’s thoughts on art and science and examine his notion that the development of science is driven by what he calls paradigm. More than a system of beliefs and rules, a paradigm is best understood as a disciplinary matrix. The function of the paradigm is to provide scientists with a vast reservoir of puzzles along with the tools to solve them. Science advances through the shifting of paradigms, as new ones replace older ones. The search for a replacement paradigm is driven by a crisis, in which science loses confidence in an existing paradigm to overcome an important puzzle or anomaly, at which 1 Heidegger’s

involvement with Nazism and his anti-Semitism remains a highly controversial issue that has been widely discussed and divides distinguished philosophers in a broad front. Some readers may realize that this is a problematic subject, particularly from the ethical point of view. Nevertheless, Heidegger’s perspective on art, technology, and the relationship between these two domains remains relevant because it instigates the reflection on ontological concepts, which are little explored in the field of sound and music studies. In accordance with Heidegger, I believe that “the more questioningly we ponder the essence of technology, the more mysterious the essence of art becomes” (Heidegger 1977, 35).

1 Sound, Truth, and Paradigm

3

point a new paradigm takes shape. The awareness and acknowledgement that a crisis exists loosens theoretical stereotypes and provides the incremental data necessary for a fundamental paradigm shift. As examples, the Copernican revolution proposed a solution for the crisis of Ptolemy’s system of astronomy, and the Newtonian paradigm was replaced by the Einsteinian as it made electrodynamics compatible with a system of motion.

1.2 Paradigm and Disciplinary Matrix Primarily interested in the way science progresses, Kuhn studied the patterns of innovation in the history of science. In the concluding remarks of The Structure of Scientific Revolutions, Kuhn (1970) asked why science moves steadily ahead in ways that are different from art, political theory, and philosophy. His theory was based on the definition of science as it related to progress. Given that progress is an attribute of technology, it can be difficult to see the profound differences between science and technology since progress is also an attribute of science. To address this, Kuhn distinguished between what he called normal science and scientific revolutions, although both operate on the basis of progress. In normal science, the members of a scientific community work from a single agreed upon paradigm or closely related set of ideas as they investigate the same problems. From the point of view of the scientists, progress results from successful creative work that adds to the collective achievement of the group. The scientific community legitimizes both paradigm and progress. Doubts that arise about progress occur during periods of competing tendencies in a transitional period between an old and a new paradigm. It is between the post-paradigm and pre-paradigm time that a scientific revolution produces a new paradigm, which is then adopted by the scientific community and becomes normal science. When Kuhn suggested a new term for paradigm known as the disciplinary matrix, there were two implications. On the one hand, “disciplinary” refers to the collective practices of a particular discipline. On the other hand, “matrix” indicates an ordering of elements. As a form that functions as a whole, a disciplinary matrix is distinguished by four components: Kuhn (1970, 181–6). 1. 2. 3. 4.

Symbolic generations—definition of laws. Science has the power to increase the number of symbolic generations available to practitioners. Metaphysical paradigms—shared commitments to beliefs, analogies, metaphors, and models used to evaluate and legitimate puzzle-solutions. Values—More widely shared among different communities than symbolic generalizations or models. Their application is affected by individual features. Exemplars—Includes concrete problem-solutions for scientific education and technical problem-solutions for post-educational research. The differences between sets of exemplars provide the community with the fine structure of science.

4

P. C. Chagas

Kuhn’s theory of scientific revolution had a great impact on twentieth century philosophy of science. His account of the development of science challenged the traditional view of knowledge as the application of a set of rules and the idea of scientific neutrality. The standards of assessment of the paradigmatic theory—the puzzles—are not neutral but depend on the particular disciplinary matrix within which the observers operate. This point of view is expressed by the thesis of the incommensurability of paradigms, according to which “the normal-scientific tradition that emerges from a scientific revolution is not only incompatible but often actually incommensurable with that which has gone before” (Kuhn 1970, 102). The notion of incommensurability emphasizes the impossibility of comparing, contrasting, and discussing different observational languages, theories, and standards when different paradigms are involved. In fact, there is widespread consensus on the limitation of the paradigm notion. As Bird points out, “paradigm puzzle-solution, will not solve all problems. Indeed, it will probably raise new puzzles” (2018, 12). Kuhn questioned the idea of progress as evolution toward a goal. The abolition of the teleological idea of evolution, he claimed, was the most important and disruptive contribution of Darwin’s theory: The belief that natural selection, resulting from mere competition between organisms for survival, could have produced man together with the higher animals and plants was the most difficult and disturbing aspect of Darwin’s theory (Kuhn 1970, 171).

The idea that, in the evolution of organisms, development and progress can occur “in the absence of a specified goal” seems contradictory. It shakes the fundamentals of scientific beliefs. As Kuhn noted, a scientific theory is expected “to be better than its predecessors not only in the sense that it is a better instrument for discovering and solving puzzles but also because it is somehow a better representation of what nature is really like” (1970, 205).

Kuhn denied the vision of scientific evolution, according to which scientific theories come closer and closer to truth. There is no coherent correlation between the ontology of a theory and the reality of nature, no coherent direction of ontological development in the succession of scientific revolutions. On the contrary, new paradigms can sometimes conceal things and prevent developments: “Einstein’s general theory of relativity is closer to Aristotle’s than either of them is to Newton’s” (1970, 205). It is for this reason we can say that scientific theories are “forms of life” in the sense of Wittgenstein, because they consist of ensembles of rules and interactions describing a specific culture.

1.3 Relations of Science and Art Triggered by a problem with avantgarde art, Kuhn expressed his views on the relations of science and art in an article published in 1969 (Kuhn 1978), many years after The Structure of Scientific Revolutions. He admitted that the discovery of parallels

1 Sound, Truth, and Paradigm

5

between science and art motivated some of his theories in Structure; for example, the idea of patterns of innovation. Kuhn believed the innovation of normal science occurs when scientists work on shared patterns of innovation, digging in the interdisciplinary matrix, trying to solve puzzles, specializing in esoteric puzzles.2 Secondly, Kuhn believed that innovations in scientific revolutions occur in times of revisionary changes, when different scientific schools compete with each other and different traditions, standards, values, and modes of perception give birth to parallel developments. In addition to the idea of patterns, Kuhn borrowed the idea of multiplicity from art. All elements belonging to both normal science and scientific revolutions are multiple; even a paradigm is a structure of multiplicity. This becomes clear when Kuhn reframes the notion of paradigm as disciplinary matrix. Two other likely contributions from art to Kuhn’s account of paradigm are the notions of autonomy and self-reference. Both art and science share the concept of a unity detaching itself from the environment. They constitute autonomous, autopoietic systems of society. Autopoeitic social systems have the ability to reproduce themselves on the basis of communications.3 Kuhn’s theory focuses on the ability of science to reproduce itself on the basis of its own paradigms—puzzle-solving elements that constitute the domain of interactions of its organization. Kuhn described himself as a science historian, but detached science from its historical, political, and social environment. As science operates according to its own rules and values legitimated by the scientific community, Kuhn made clear there is no neutral place, as everything acquires a meaning in relation to the organic structure governed by the principles of autonomy and self-reference. He opposed the classic dichotomy distinguishing art as subjective and intuitive and science as objective and inductive (Kuhn 1978, 340). Regardless of the purpose “to change the image of science by bringing it closer to the image of art” (Pinto de Oliveira 2017, 747), Kuhn considered science and art very different enterprises and addressed the differences in three ways. First was the idea that the product and goal of any artistic activity is a work of art—an aesthetic object able to be perceived. This is in opposition to the objects of science, which act merely as tools for scientific research in which aesthetics is not a primary goal. For example, Pythagoras’ discovery of elliptical orbits deduced mathematical harmonies in nature—the vibration of strings shows these elliptical orbits. Music took advantage of the Pythagorean discovery to build intonation systems and the Renaissance period developed a new spatial perception exploring the ellipse. However, ancient and medieval astronomers were attached to the aesthetic perfection of the circle and it wasn’t until Copernicus placed the sun at the center of the solar system that the ellipse was able to contribute to the solving of astronomical problems. Kuhn emphasized that parallels between art and science might be beneficial for understanding how these fields articulate creativity, but excessive emphasis could obscure vital differences. We can summarize his analysis of the products of art and 2 The

fundamental meaning of “esoteric” is “specialized,” likely to be understood only by a small number of people. 3 For an account of the theory of autopoiesis, see Chagas (2014a, 65–102).

6

P. C. Chagas aesthec objects

technical puzzles science

art

producing

solving

Fig. 1.1 Semiotic square of art and science

science by a semiotic square with two horizontal semantic axes.4 On the top horizontal axe would be the opposition between two kinds of objects: “aesthetic objects” and “technical puzzles.” On the bottom horizontal axe would be the opposition between two activities: “producing” and “solving.” Art and science operate symmetrically in this semiotic square by connecting the four categories along their diagonals. As Kuhn claimed: Whatever the term “aesthetic” may mean, the artist’s goal is the production of aesthetic objects; technical puzzles are what he must resolve in order to produce such objects. For the scientist, on the other hand, the solved technical puzzle is the goal, and the aesthetics is a tool for its attainment (Kuhn 1978, 243) (Fig. 1.1).

The relationship to the public is fundamentally different in art than it is in science, despite both fields depending on public support. The public is a consumer of art and technological objects, while there is no audience per se for science. For Kuhn, art is intrinsically destined to communicate with an audience in a way science is not. This difference is particularly visible in the way contemporary art addresses and incorporates the past, such as with music where the past continues to be a vital part of contemporary music. In contrast, science destroys its past; scientists don’t practice based on past scientific theories. Even so, Kuhn wondered why the artist who admires the past is not interested in reproducing it. Why do we recognize works by Bach and Beethoven as masterpieces of aesthetic achievement and at the same time reject as forgeries attempts to reproduce similar works? In science, the past is judged according to categories such as correct and incorrect. Whatever science considered Newton a genius, his discoveries were considered wrong from Einstein’s point of view. Scientists are committed to solving puzzles to find the best solution. Kuhn reframed the problem of the public audience as a puzzle-solving event: “Both disciplines present puzzles to their practitioners, and in both cases the solutions to these puzzles are technical and esoteric” (1978, 347). But for the artist, the puzzlesolving is the work of art itself, whereas for the scientist, the theory is the primary tool. Kuhn recognized similarities along evolutionary lines in the way art and science developed over time. He considered his original idea to have applied the developmental pattern of art on science theory. However, he emphasized a significant number of differences in the developmental line structure of art and science. The most important of which is that art allows a higher degree of multiplicity than science. Art can 4 For

an application of the semiotic square to the post-human, see Hayles (1999, 247–282).

1 Sound, Truth, and Paradigm

7

support a large number of incompatible traditions and schools and the acceptance of a new tradition doesn’t hasten the end of the old. For example, contemporary classical music embraces various music traditions reaching back as far as the Middle Ages right up to the current myriad music styles, genres, and forms of today. All of that is available and compatible to both artists and the public. In science, on the other hand, a new tradition emerges as a consequence of the scientific revolution and necessarily eliminates the previous one, banishing those who lose the revolution. Conversely, artists can significantly change styles many times during their lives, something that scientists don’t do. Kuhn sees the function of internal crisis in the sciences as a built-in signaling system prompting innovation, even though innovation itself is not a primary value—there is no scientific avantgarde as there is in art. Art makes innovation a primary value, whereas science has no innovation strictly for innovation’s sake. Kuhn considers innovation a component for the development of art in the way internal crises promote revolutions in science. The term “style” in art and “theory” in science describe a group of similar works. How can we distinguish one style or theory from another? This is where the notion of paradigm comes in, as it deals with processes like abstractions of elements that simultaneously recognize the paradox of integrating the observer in the observation. The research subjects are observed from ethical and social points of view. As Kuhn affirmed, “science and art are both products of human behavior (1978, 351).

1.4 Language Games A parallel can be drawn between Kuhn’s notion of paradigm and Wittgenstein’s concept of language game. The language game is an object of comparison for exploring similarities and differences in order to clarify certain aspects of language. It is considered a tool for accessing meaning and one of the most powerful arguments of modern and postmodern pluralism. Wittgenstein argued that the meaning of a word or any observational term, is determined by its use. He mentioned, for example, the analogy between melody and sentence (phrase), which is a language game that underlies the understanding of music across different cultures: “Understanding a sentence is much more akin to understanding a theme in music than one may think” (2001, Sect. 527). What does it mean to understand a spoken phrase? What does it mean to understand a melody? How do we justify playing the melody in this manner and not differently? Wittgenstein emphasized the idea of melody as a tautology, a phrase that expresses itself and nothing more. He used an approach not unlike the notion of paradigm to explore the connections between a linguistic and a musical phrase: “What is it like to know the tempo in which a piece of music should be played?” And the idea suggests itself that there must be a paradigm somewhere in our mind, and that we have adjusted the tempo to conform to that paradigm. But in most cases if someone asks me, “How do you think this melody should be played?”, I will, as an answer, just whistle it in a

8

P. C. Chagas particular way, and nothing will have been present to my mind but the tune actually whistled (not an image of that) (Wittgenstein 1978, 166).

Wittgenstein doesn’t transform the phrase into a paradigm of musical understanding; rather, he uses the analogy between melody and phrase as a bilateral tool, going back and forth between music, language, and other kinds of representation such as pictures, faces, and gestures—in order to explore any internal similarities. The analogy serves to illuminate both things by way of comparison. Comparing the melody to the phrase helps us understand both music and language. Instead of theories to explain everything, Wittgenstein’s philosophy operates with partial models that are instruments of making analogies to function as objects of comparison. For example, we make gestures and gesticulations when we talk about music. We use a gesture to say that music conveys that or that, but the gesture is not the music. Gestures can be considered language games with aesthetic values and become associated with the way we react to music. For Wittgenstein, the gesture physicalizes the impossibility of describing what we feel when listening to music and highlights the inadequacy of logical and scientific theories used to clarify music.5

1.5 Philosophical Archeology The key idea of the archeological method of philosophy is that systems of thought and knowledge are governed by rules that operate beyond rational logic and individual consciousness. This method, essential to Foucault, allows one to operate on the unconscious level of history, digging for analogies and paradigms. Agamben (2009a) recognized a crossroads between Foucault’s philosophical archeology and Kuhn’s concept of paradigm—with the “disciplinary matrix” that designates the common possessions of members of a certain community and with a “single element” within the set that serves as a common example replacing explicit rules. Foucault’s inquiry on the relation of power and knowledge is an example of his approach of the paradigmatical method—instead of analyzing juridical and institutional models, he focused “on the concrete mechanisms through which power penetrates the very bodies of subjects and thereby governs their forms of life” (Agamben 2009a, 12). At the core of Foucault’s philosophical archeology is the paradigm of modern “disciplinary society.” Rather than torturing and killing criminals, modern society has found ways of imprisoning them. The disciplinary paradigm is not restricted to specialized institutional contexts such as prison and mass incarceration but has become a model for control of the entire society—perfected through modern techniques of observation, control, and punishment. In the lineage of Foucault, Agamben explains how the notion of paradigm is used as a tool for illuminating a broader historical context: In the course of my research, I have written on certain figures such as homo sacer, the Muselmann, the state of exception, and the concentration camp. While these are all actual 5 For

an account of Wittgenstein and musical understanding, see Chagas (2015).

1 Sound, Truth, and Paradigm

9

historical phenomena, I nonetheless treated them as paradigms whose role was to constitute and make intelligible a broader historical-problematic context (2009a, 9).

The two main paradigms mentioned are the state of exception and the concentration camp. Agamben noted that in the first half of the twentieth century, there was a great acceleration in the history of human sciences with linguistics and comparative grammar studies. The problem with the linguistic paradigm is that it doesn’t provide insight on the more archaic stages of human history. As Agamben explains, philosophical “archeology moves backward through the course of history” (2009a, 107) not looking for an indestructible truth, but making accessible an individual or collective history for the first time. Philosophical archeology is not to be understood as an ontological regression to a given location in chronological time, but instead as an investigation of the operative forces within history. It doesn’t search for substances, but for streams of events emerging and becoming. The paradigm shift in human science, according to Agamben, moves from comparative grammar, considered a historical discipline—to generative grammar, ultimately a biological discipline. As a consequence of this shift of the epistemological paradigm, there is a predominance of models from cognitive science. For Agamben, the point is not to go back in history looking for an ontological anchoring as something irreducible which we can hold onto, like the big bang theoretically giving birth to the universe. By giving up the very idea of an ontological anchoring, we “envisage being as a field of essentially historical tensions” (2009a, 111). As models of knowledge, paradigms are figures that are able to detach themselves from a historical background. They are singular objects that constitute a broader problematic context. Scientific paradigms like those of Kuhn contain common knowledge of a particular community—tools, models, techniques—and individual elements that replace explicit rules. It is not the rule that determines the paradigm; the rules are taken from paradigms. As an ersatz of rules, the paradigm is a synthetical category that makes science operate through analogies instead of logic, while eliminating the empire of rules. It replaces the universal logic of the world with the singular logic of the example. The analogy is opposed to the binary logic of Western thinking, offering an alternative to overcoming oppositions of A or B, universal or particular, form or content, and so on. Analogy also creates indiscernibility, the inability to make things detach from each other—the particular from the universal, the form from the content. Agamben emphasizes that paradigm does not act as metaphor, by which a signifier is extended to heterogeneous phenomena, but instead, as an analogy—occurring between the singularity of itself as a specific example, and its making intelligible “a new ensemble, whose homogeneity it itself constitutes”(2009a, 18). The paradigmatic method provides a solution for the paradox of the hermeneutic circle of human sciences, according to which understanding as a whole is established by referencing individual parts and vice-versa. The solution is to consider the hermeneutic circle itself a paradigm so there is no duality between parts and the whole; there are only individual cases. A paradigm is the place where analogy lives in perfect equilibrium beyond the opposition of generality and particularity. The single phenomena is

10

P. C. Chagas

capable of constituting itself as a paradigm and can constitute the whole of which it is the paradigm. Agamben summarizes it this way: 1. 2. 3.

4. 5. 6.

A paradigm is a form of knowledge that is neither inductive nor deductive but analogical. It moves from singularity to singularity. By neutralizing the dichotomy between the general and the particular, it displaces a dichotomous logic with a bipolar analogical model. The paradigmatic case becomes such by suspending and, at the same time, exposing its belonging to the group, so that it is never possible to separate its exemplarity from its singularity. The paradigmatic group is never presupposed by the paradigms; rather, it is immanent in them. In the paradigm, there is no origin or arche; every phenomenon is the origin, every image archaic. The historicity of the paradigm lies neither in diachrony nor in synchrony but in a crossing of the two (Agamben 2009a, 31).

As Agamben notes, “the intelligibility of the paradigm has an ontological character” (2009a, 32) and deals with the nature of being. Paradigms illuminate broader historical contexts. They make intelligible phenomena we overlook when working on history through logical categories such as induction or deduction. Agamben considers the concentration camp and Nazi death camps in which people were treated as if they had no value, examples of how the state is constructed as an ontological paradigm of sovereignty within the political space of modernity. The problem of sovereignty is ontological because it is closely related to the question of life. Agamben articulates sovereignty in terms of exclusion of exception. The state decides who is to be incorporated into it and who must remain outside of it. Politics is dependent on making people feel hopeless and imposes vulnerability as a condition of participation in public and political life. From the paradigm of the concentration camps emerges contemporary states of exception, characterized by the suspension of normal legal protections and denial of basic rights. States of exception involve exclusion from the fields of the law and political value. Prisons like Guantanamo or Abu Ghraib, where abuses are committed under the cover of military impunity are examples. The paradigm of states of exception can also be applied to prisons and policing regimes, detention camps, and refugee camps. Moreover, the paradigm state of exception applies to the current pandemic of COVID-19. Agamben argues that the epidemic is a discursive phenomenon that functions like a conspiracy and considers the current lockdowns as extensions of the ‘state of exception.’

1.6 The Paradigm of Signature How do we learn to think about language? How do we build sentences that actually carry meaning? Agamben has suggested we learn to think of language as a sort of

1 Sound, Truth, and Paradigm

11

“magic.” The signs are not neutral, but they carry a kind of “magical” meaning, which is called signature. As he claims, a signature does not merely express a semiotic relation between a signans and a signatum; rather, it is what—insisting on this relation without coindicant with it—displaces and moves into another domain, thus positioning it in a new network of pragmatic and hermeneutic relations (Agamben 2009a, 40).

In other words, with language being the most archaic signature of all, a signature is what shifts the semiotic relation between a signifier and the signified to other domains and adds a surplus of meaning. The signature is the creating act of naming things in the world, whatever the world is. As a synthesis of faith, knowledge, and ethics, it points to our inability to grasp the world exclusively through objectivity consistent with logical systems—this is precisely what gives the signature a “magical” meaning. The signature has a relation of similarity with the sign, which is not to be understood as “something physical, but according to an analogical and immaterial model” (2009a, 36). The art system reveals an analogical structure of works of art to function as signatures of subjects and objects. The function of art, as Luhmann argued, is to make the external world appear within the internal world of the artwork. Art gives visibility to an ambivalent situation in which every time something is shown in the world, something else is concealed; art makes observable the unobservable. This is the paradox of art (cf. Luhmann 2000, 149). Over the course of history in the arts, we see a similarity between the figure of author and the belief of the Creator, who gives his own proper name to everything. When we observe a painting in a museum, listen to a piece of music, read a novel, hear somebody speak, or listen to our own thoughts, “we do not pay attention to the operation implicit in the signature, an operation that is anything but trivial” (2009a, 39). The signature places the work of music in relation to the name of a man. This is the key of Western music understanding. If this information were missing, the music would remain unchanged in its materiality and quality, but it would be something completely different. Agamben shows how the paradigm of signature has exerted a decisive influence on disciplines of the Renaissance and the Baroque such as medical science, magic, astrology, and theology. The signature is to be understood as a decisive operator of knowledge that compensates the inadequacy of the sign to address immaterial issues. For example, the sign sacrament, which functions not only as the sign of a sacred thing, is also a sign of efficacy; the sacrament needs an operator, an active principle in order to be animated, and a signator that animates and makes it effective. However, it would be wrong to consider the signature only from a mystical perspective, as it is not simply what manifests the occult virtue of things. Indeed, it is the decisive operator that establishes relations between different domains of knowledge. The key to the concept of signature is that the world itself is mute and silent; it has no reason in itself. The signature is what animates the silence, the sound that makes the world intelligible. The analogy between signature and sound is implicated in this quote from Jakob Böhme’s The Signature of All Things [De Signatura Rerum] (1621): The signature stands in the Essence, and is as a Lute that lies still, and is indeed a dumb Thing that is neither heard or understood; but if it be played upon, then its form is understood

12

P. C. Chagas …. Thus likewise the sign [Bezeichnung] of Nature in its Form is a dumb Essence …. In the human Mind, the Signature lies most artificially composed, according to the Essence of all Essences, and Man wants nothing but the wise Master that can strike his instrument (Agamben 2009a, 42).

In other words, the signified is like an untouched musical instrument and the signature is what makes it produce sound. But who is the master that plays the instrument? What gives birth to the sound? What makes the mute matter vibrate? What is this essence that allows us to listen to the sound of the instrument? That we listen to everything? This fundamental idea that a mute matter could resonate and then become musical belongs to the mythological roots of Western music extending back to Antiquity. The signature conveys not only the idea that everything can resonate, but expresses our desire to listen to sounds everywhere, resonating in different spaces and temporalities, from molecular vibrations of the planet and other cosmic entities of the universe. Sound is pervasive, Kahn (2001) claims, “all space becomes indelibly, inaudibly, or pervasively filled with voices and sounds awaiting to be heard” (2001, 200). Between silence and cacophony, music makes tangible the active principles of sound and listening as fundamental signatures. Sound is embodied in everything, all the time, even if we cannot listen. Music embodies the desire to comprehend the ubiquity of all sounds. For example, in Pythagoras’ mythic music of the spheres, the sounds of the orbital motion of the planets are analogous to the sounds of musical harmony, although we cannot listen them. The ratios of the movement of the planets corresponded to tonal musical intervals in the Pythagorean scale. Kepler calculated that the motion of the planets is not circular, but elliptical, and thus refined the signature of the harmony of the universe. The contemporary cosmological view of quantum mechanics explains features of the universe as behavior of subatomic particles that can be visualized as vibrations, like that of a musical instrument. A tone can determine its physical form. It can remain invisible and inaudible. There is a music that is not audible, and, at the same time, there is a space that is always sounding. As Agamben notes, the concept of signature disappears from Western science with the advent of the Enlightenment but reemerges in the second half of the nineteenth century in different concepts. He claims, all research in the human sciences, particularly in a historical context, necessarily has to do with signatures. Saussure’s attempt to conceive language solely as a system of signs is insufficient to provide an explanation of the passage from sign to speech. The interpretation of the language as a semiotic sign was blocked by the sign itself. The necessity to supply the sign with a supplementary symbolic content characterizes a significant part of twentieth century thought. This approach favors the signature as the constitutive primary of signification over signification. Derrida’s deconstruction and Foucault’s archeology are two examples of the sciences of signatures. But they did not exhaust the possibilities of signatorial inquiry. Agamben says that it’s possible, for example, to imagine a practice that without infinitely dwelling in pure signatures or simply inquiring into their vital relations with signs and events of discourse reaches back beyond the split between signature and sign and between the semiotic and the semantic in order to lead signatures to their historical fulfillment (2009a, 80).

1 Sound, Truth, and Paradigm

13

The emerging discipline of sound studies investigates sound signatures. It is an example of how sound became an interdisciplinary paradigm of knowledge encompassing both science and art. Viewed through the lens of Kuhn, sound studies can be considered a disciplinary matrix established to deal with discourses of sounds, different methods and traditions, and both practical and theoretical forms of knowledge. Sterne’s notion of “sonic imaginations” denotes an opening to the network of signatures who might deal with concepts “with a “sonic dimension like hearing, listening, voice, space and transduction (to name just a few)—and sound itself” (Sterne 2012, 9).

1.7 Apparatus and Music Paradigms The notion of “cultural paradigm” emerges in Dreyfus (2005) as an analytical tool for relating different kinds of experiences. He argues that “a cultural paradigm collects the scattered practices of a group, unifies them into coherent possibilities for action, and holds the resulting style up to the people concerned, who then act and relate to each other in terms of it” (2005, 410). An artwork is a cultural paradigm in the sense that it articulates an understanding of shared practices of a culture. In Dreyfus’ account of Heidegger , artworks can function as revolutionary paradigms as they open the process of setting-into-work of truth, to make possible a new beginning. The experience of radical transformation is not limited to the future but grounded in the historical understanding of human experience. A major focus of studying artwork as cultural paradigm is the relationship between art and technology. Art provides objects for observation, which unconceals the essence of technology. The observation of art includes both the formal distinctions the artist makes in the process of producing artwork, which determines its form as well and the different ways it is perceived and communicated in the society. From a historical perspective, the technology of mechanical reproduction, which emerged in the second half of the nineteenth century, introduced a revolutionary change in art. As prototypes of delivering sound and image, phonography and photography apparatuses in the age of mechanical reproduction challenged the notion of authenticity of the work of art—the artistic aura as Benjamin (1977) claimed. The aura is a signature that activates the truth in a work of art. When art is part of a ritual, like a medieval Gregorian chant resonating in the interior of the Gothic cathedral, it celebrates a ritual. Music performance activates the embodied presence of the sound in a particular space and time. The mechanical reproduction introduces a revolutionary change: it disengages the artwork from the physical performance and its aura of authenticity. The artwork is transformed into an autonomous object that becomes available for reproduction. The paradigm of mechanical reproduction goes together with the increasing proletarization and development of a mass culture in modern society. The consequences of this process, observed by Marx, Nietzsche, and others can be approached from two opposite poles. On the one hand, the technical apparatus is pressed into the production of ritual, including political values serving fascistic

14

P. C. Chagas

purposes. On the other hand, it expands the field of creative experience and promotes new models of creativity and artistic forms. Apparatus is a key concept for understanding contemporary artistic creativity. A broad concept not restricted to the technicality of objects, Agamben (2009b, 2–3) approaches the notion of apparatus through the following three definitions: a.

b. c.

It is a heterogeneous set that includes vitally anything, linguistic and nonlinguistic, under the same heading discourses, institutions, buildings, laws, police measures, philosophical propositions, and so on. The apparatus is the network established between these elements. The apparatus has a concrete strategic function and is always located in a power relation. As such, it appears at the intersection of power relations and relations of knowledge.

Agamben reframes the concept of being by partitioning it into two large classes— living beings (or substances) and apparatuses—mediated by a third class, subjects. This partition means that apparatuses imply a process of subjectivation. They have “the capacity to capture, orient, determine, intercept, model, control, or secure the gestures, behaviors, opinions, or discourses of living beings” (2009b, 14). All kinds of social instances such as prisons, schools, factories, disciplines, juridical measures; but also literature, philosophy, art, entertainment, computers, telephones, racism, and language itself are apparatuses. The growth of apparatuses in capitalism can be traced through the multiplication of processes of subjectification. Agamben defines the current extreme phase of capitalism as a massive accumulation and proliferation of apparatuses. Apparatuses are rooted in the very essence of humanization and represent the division that separates the living being from itself and its environment: “At the root of each apparatus lies an all-too-human desire for happiness. The capture and subjectification of this desire in a separate sphere constitutes the specific power of the apparatus” (2009b, 17). Considering this, what are the paradigms of music? From an evolutionary point of view, there are three cultural paradigms in the history of Western music: vocal, instrumental, and electroacoustic. This segmentation is based on the physicality of the source that produces sound: voice, instrument, or apparatus. The musical paradigms can be investigated linearly, as successive periods in musical history—or as a network of parallels and interacting paradigms that have influenced each other. As Kuhn notes, art has a structure of multiplicity that allows for a parallel and interdisciplinary matrix of paradigms. In other words, vocal music continued developing during the period of the instrumental paradigm while vocal and instrumental music continues to develop during the current electroacoustic paradigm. The puzzle-solving activities of the paradigms are grounded in the materiality of the apparatuses, but the paradigm is not restricted to the material world. The technology of the voice paradigm points to the bodily origin of human produced music. Ihde (2007) compares vocal music to language and suggests that singing may precede language:

1 Sound, Truth, and Paradigm

15

Language itself is, after all, musical in some sense too. There is cadence, tone, pitch, and the like, although usually highly conventionalized. Singing can be at the least a sort of exaggeration of the sound patterns that are more limited in ordinary language (2007, 253).

Musical production of the vocal paradigm is directly bodily expressive, exploring different kinds of singing, speech, and variations of bodily sound production. The voice indicates human individuality directly connected to the self, a living presence. As the Sufi mystic Khan claims, “sound can be louder than the voice, but sound cannot be more living than the voice” (1996, 88). Voice is therefore a signature, its sound connecting the eternal, universal, and individual. It manifests consciousness and the duality of creation and destruction in many ways: masculine and feminine, prayers and confessions, joy and suffering, loud and soft, authoritarian and submission, tender, broken, emphatic, light and deep, and ultimately—the voice of silence. Medieval music is historically associated with the voice paradigm. The puzzlesolving activity of this era includes Gregorian chant, choral music, music for voices accompanied by instruments, and other practices. One of its most obvious developments is the invention of music notation systems, which enabled composers to write music for multiple voices performing simultaneously. What is the difference between learning a melody by “ear” and learning to read a score? Music notation is an apparatus with a double function, internal and external. Internally it provides advances for measuring and synchronizing music objects and events such as rhythms and melodies. Externally, it functions as a reproduction apparatus that disseminates music works. The development of polyphony is a significant technological achievement of the Middle Ages. Polyphony creates a multi-layered auditive spatiality that unfolds not one singular source but several sound objects and events. Polyphony is a dialogical paradigm activated by multiple voices that appear, disappear, and connect to other voices in the polyphonic network. Voices activate sound with the living experience of the present, like embodied figures emerging from a mute background. When we listen to a musical work, we are necessarily recognizing figures that are formed on a background, which, at the same time, might not be openly manifested. The paradigm of polyphony makes clear that, by listening to a musical work, one cannot simultaneously realize all layers of expression. There is no fixed-point functioning as a stable state of affairs for interpreting music. There are states of affairs, but these are intentional, i.e., they are objectifications of subjects. The listener is an integral part of the process of musical meaning. Listening and performing are viscerally connected. Performing music is primarily listening to oneself, allowing the self to appear as sound embodiment. The instrumental paradigm encompasses the variety of musical technologies known as instruments. As an old technology that emerged when man began to use tools for purposes such as hunting and cultivating land, instruments have in common what Ihde (2007) calls embodiment relations. Instruments that emerged thousands of years ago such as flutes, rudimentary string instruments, or simple percussion instruments have been a distinctive human-technology since then. As Ihde notes,

16

P. C. Chagas

musical instruments require a practice-based learning and the development of techniques and skills of music making including the understanding of the music tradition. The puzzle-solving activity of the instrumental paradigm is directed toward technological innovations that remain consistently attached to the body: the capacities of the breath, hand, and fingers. Musical nuances are attained though learned skills. A second aspect of the instrumental paradigm is the mechanization of playing, which include improvements in the design of instruments such as the use of keys with mechanical plucks (harpsichord), valves (brass), key mechanisms (woodwinds), and more complex instruments like piano. The instrumental paradigm progresses toward the mechanization of music. Wittgenstein expressed his thoughts on the limits of music driven by the instrumental paradigm in a comment to a friend: “Music came to a full stop with Brahms; and even in Brahms I can begin to hear the sound of machinery (Rhees 1981, 127). The instrumental paradigm transforms the vocal phrase into an instrumental gesture. Chopin’s writing for piano, for example, is charged with the affect of the human voice, as his melodies and dramatic gestures are introduced to express the transformation of the vocal apparatus into the mechanical medium of piano. We can say, in the sense of Tarasti’s existential theory (2000), that Chopin’s instrumental music accomplishes a semiotic operation of negation and affirmation, as if the piano could embody the self and bring human existence to a level of transcendental realization. The piano music first negates the biological foundation of the human by transforming the body into machine, and then reaffirms the organism by transmuting the machine into a kind of living system (cf. Chagas 2014a, 47). The instrumental paradigm developed a significant collection of puzzle-solving gestures functioning as analogies for musical understanding and appreciation, triggering affect that can be associated with the way we perform or listen to music. Although gesture doesn’t constitute musical understanding, it can accompany or manifest our feelings about music. Hatten developed a comprehensive theory of emergence and generalization of musical gesture. Defining gesture as “significant energetic shaping of sound through time” (Hatten 2004, 95), he demonstrates the evolution of stylistic gestures in the works of classical and romantic composers such as Beethoven and Schubert. Through the vast repertoire of gestures, music became a very particular form of life, and can therefore act as language. The instrument paradigm opens the space for various achievements such as orchestration, opera, programmatic music, and later cinematography. The paradigm of orchestration constructs a music surface that can be connected or detached to the musical subject. Similar to polyphony, the concept of multimedia is associated with simultaneity of objects and events. But while polyphony focuses on auditory perception—how we sonically distinguish multiple and independent objects and events—multimedia is a structural coupling involving different sensory and perception domains such as vision and language.

1 Sound, Truth, and Paradigm

17

1.8 The Electroacoustic Paradigm As an extension of the acoustic paradigm, the electroacoustic paradigm combines “electronics” with “acoustics.” Understanding the acoustic paradigm calls upon the notions of sound space and auditive spatiality. We assume a stable acoustic space populated with sound-producing objects and sound producers. Listening is the primary mode of accessing the sound space. The most common mode of listening is to gather information about the source of sound—what has produced the sound and how it was produced. For example, we recognize voices of persons, mechanical and electrical sounds, and sounds of nature. We also follow the evolution of the sound in time and space. A car driving in front of us activates a variety of sounds including the motor, the rolling of the tires on the ground, voices of people inside the car, and sounds coming out of the car radio. The car is a signature of sound events emerging from a background and disappearing back into it. The background can be approached from two opposite perspectives: silence or noise. Either the sound fades in from silence and fades out into silence or arises from noise and collapses back into noise. Our imagination is populated with images of silence and noise backgrounds from which the creation emerges. As the current paradigm of physics, the Big Bang theory posits the idea that the universe was formed from sound waves that triggered quantic streams of sound waves originating matter and life. With the electroacoustic paradigm, we build the capacity to use apparatuses to produce and move sounds around spaces. For instance, the sounds of a car can be recorded and transformed in various different ways. If we are watching a movie, we probably hear car sounds in synchrony with the images. The sounds can be combined with diegetic voice and music, be in the background, or be the signature of a meaningful event that interprets the meaning of the image. A piece of electroacoustic music can either feature the sound of a car in a recognizable way or in a transformed way in which we are no longer able to recognize its origin. This is what has been called reduced listening in the tradition of musique concrète (cf. Chion 1994, 25–34). With the electroacoustic paradigm it becomes clear that the acoustic space, which is experienced as a relationship between subjectivity and the world, can become itself a signature for further elaboration. The electroacoustic paradigm displaces the previously imagined living spaces and reconstructs the meaning of the lived experience. The paradigm shift, according to Kittler (1993), affects the way we see the world and how we define ourselves as human beings; we have moved from interpreting meaning through philosophy, poetry, and hermeneutics to thinking in terms of cybernetics, information theory, and stochastic systems. We have removed ourselves from the central position of human subjectivity to become a sort of machineobject, operating independently from human agency. The phenomenon of noise is crucial for understanding the paradigmatic shift in terms of information theory and cybernetics. The impulse for developing cybernetics came from the effort to develop an anti-aircraft control apparatus in World War II in order to extract messages from a disturbing background on the basis of simultaneous statistics of noise and message—with the goal of better predicting the future, i.e.,

18

P. C. Chagas

the position of an enemy aircraft in order to shoot it down. Both the irregular motion of an airplane in flight and the unpredictable reactions of the pilots and gunners are considered noise. Therefore, the human links must be included in the control chain of the system. “The anti-aircraft control apparatus was in essence a feedback loop and contained in its construction many subsidiary feedback loops, we had to find out something of the characteristics of these loops” (Wiener 1956, 254). The revolutionary idea was thus to connect the fire-control machine to the human nervous system. The term feedback was an appropriate way of describing phenomena in the living organism as well as in the machine. This is the birth of the idea of the new science of cybernetics, which is the foundation of the electroacoustic paradigm. But what is the music of the electroacoustic paradigm? How should it be considered in relation to the vocal and instrumental paradigms? What is the “setting-intowork of truth” (Heidegger) of electroacoustics? How do we best approach the network of language-games (Wittgenstein) of the electroacoustic paradigm? The search for the answers to these questions is driven by the fact that electroacoustic music has become a fertile field of artistic and technological exploration and innovation affecting practically everything that is understood as music or sound art—including the collective perceptions of listeners, creators, and investigators. The electroacoustic paradigm is actually our very particular form of life as it concerns the whole of communication with sound and music in the society. It has exponentially increased the music information available to us and contains a universe of similarities, analogies, and correspondences that transform sound perception and consciousness. In this sense, it is a propitious field for exercising cultural criticism of the present. From an evolutionary perspective, the paradigms of modality and tonality have been the foundation of Western music composition since the Middle Ages. The neutralization of tonal functionality through the atonal, serial, and other types of organization of sound material in the end of 19th and beginning of the twentieth century, established a harmonic discontinuity through myriad “anomalies” that disrupted and eroded the fundamental role of tonal harmony as disciplinary matrix. This crisis triggered responses that led to non-tonal textures in the music of composers such as Schoenberg, Webern, Debussy, Stravinsky, Bartok, Messiaen and many others. It pushed composers to explore other constructive principles of musical organization focused on the physical reality of sound phenomena and emphasizing sound qualities such as timbre and noise. Within the crisis of the paradigm of tonality as foundation of musical composition, electroacoustic music met the demands of the aesthetic sensibility focused on this expanded consciousness of sound phenomena. It emerged from three different directions: musique concrète, elektronische Musik and computer music. The musique concrète that appeared in Paris after World War II around Pierre Schaeffer began with experiments involving recording techniques for capturing sounds of the acoustic environment. This approach engaged the persistent myth that the world is the primary acoustic space of music extending from the earth to the whole universe. The acoustic myth allows sound phenomenon to be isolated from the physical environment, to be heard as unique object and event, and eventually be disconnected from its material source and origin. Released from its cultural references, sound become a

1 Sound, Truth, and Paradigm

19

self-referential paradigm for composing new audible forms. The composition takes advantage of new technology for recording, manipulating, and reproducing sound. Drawing ideas from Husserl phenomenology, the aesthetics of musique concrète developed notions such as sound object and reduced listening. These categories emerged through the interaction of sound material with technical apparatuses, especially the tape recorder. Musique concrète provided electroacoustic composition with analytical and synthetic approaches to sound perception and composition. The elektronische Musik, associated with the electronic music studio of Cologne Radio, pioneered the creation of sounds whose models are neither found in nature, nor possess the qualities of instrumental or vocal sounds. The methods adopted by Karlheinz Stockhausen and other composers of the elektronische Musik were used to invent new sounds building from simple elements of technical apparatuses. For example, the signal generator and the noise generator became the prototypes of electronic sound devices despite being designed to test equipment and not for making music. These apparatuses are both mathematical constructs; the signal generator explores the simplicity of a single harmonic motion such as the sine wave, while the noise generator explores the statistical model of all possible vibrations occurring randomly in the auditive space. The aesthetics of elektronische Musik took advantage of electroacoustic technologies during the Third Reich, which radically transformed the experience of listening. Radio broadcasting and sound amplification were interconnected technologies for acoustic landscape control and organic synchronization of masses. They created new logics to frame political activity. Radio in particular activated the sonic experience of private intimacy. The transformation of radio preceded the universe of telematic paradigm. However, radio preserves the ancient magic of mythical worlds. As McLuhan notes, “The subliminal depths of radio are charged with the resonating echoes of tribal horns and antique drums” (McLuhan 1964, 299). The historical opposition between musique concrète and elektronische Musik is emblematic for the diversity of the electroacoustic paradigm. After World War II, the activity of cultural apparatuses such as the radio studios of Paris and Cologne, promoted a shift of consciousness in electroacoustic music composition. On the one hand, musique concrète developed a poetics of detachment from the previous vocal and instrumental paradigms and attachment to the sound phenomenon; it disengaged sound consciousness from the models of traditional vocal and instrumental music, and at the same time, moved toward interactions with sound that revealed cultural values and identities. The musique concrète achieved the myth of listening to all kinds of sounds. On the other hand, elektronische Musik developed a poetics of detachment from the sound and attachment to the paradigm of music composition. By carrying on the compositional path of the previous vocal and instrumental paradigm, it disentangled consciousness from the representative background of sound as a meaningful artifact and focused on the musical relevance of sound phenomenon. Elektronische Musik explores differentiations of acoustical agency in the vibrationcentered model of sensitivity. The heterogeneity of sound material is an aesthetic foundation of electroacoustic composition. The opposition between recorded sounds (musique concrete) and synthetic sounds (elektronische Musik), quickly dissipated as any kind of sound

20

P. C. Chagas

could become the object of musical composition. The electroacoustic paradigm not only integrated the musical puzzles of the previous vocal and instrumental paradigms but provided new ways for representing and manipulating sound. As the prototype of a reproduction apparatus, the tape machine was able to radically transform and manipulate recorded sound despite the fact that electromagnetic tape symbolizes linear thinking. On the other hand, digital systems of audio recording introduced non-linear representation in which the sound is broken down into an atomic dot-like structures that disintegrates into a mosaic of numbers as the bond with temporal sound tissue dissolves. The fragmented granular structure of the sound, which can be manipulated by computers and artificial intelligences, replaces linear thinking and promotes the consciousness of the microstructure of the sound. Digital systems such as samplers, programs, and storage devices stand as prototypes with the potential to project sound artificially, and thus, reinvent it. As Pousseur observed, electroacoustic music articulates a continuous interaction between different levels of sound organization, so that it becomes difficult “to draw a precise boundary between internal composition of sound and higher levels of composition” (1970, 82). A myriad of sound poetics emerged within the electroacoustic paradigm such as soundscape composition, deep listening, live-electronics, and other musical distinctions involving vocal, instrumental, or electronic sounds. The electroacoustic paradigm extended sound perception and consciousness, especially in the way it relates to microscopic and macroscopic levels of sonic composition. The opposition of macro/micro sound, along with the methodic use of music apparatuses, is a signature of the electroacoustic paradigm symbolizing a desire for intensification of the living experience. Automaticity is also a signature of the electroacoustic paradigm. The automation of processes is the mode of operation of apparatuses. Repetition is a music signature that benefits from automation. Electroacoustic music composition deals with the repetitive nature of apparatuses in many different ways. On the one hand, it works against automaticity by seeking to intensify the affective experience with sound. Electroacoustic music is able to access the vibratory experience and directly address the body; for example, through low frequencies, noise, and spatial control of the acoustic environment. On the other hand, electroacoustic music composition embraces repetition and automaticity at the base level of sound composition. For example, the technique of the loop in the analog era was introduced as a tool for creating sounds with tape machines, used to repeat and build sound textures. A great deal of experimental and popular electronic music explores the loop as a signature of composition and the technique was further developed with the drum machine and the digital sequencer. The automaticity of music apparatuses plays a role in the processing of music information in society as seen with digital playlists found on streaming services. The electroacoustic paradigm promotes the accumulation of sound and music objects such as sound archives, song collections, and repertoire of musical styles. Techno is an example of how electroacoustic music can match the automated nature of the apparatuses by activating repeating events that account for the ritualization of models. These models are basically variations of the apparatuses’ own

1 Sound, Truth, and Paradigm

21

programs that can affect the society in contradictory ways—either to liberate creative forces or to reinforce authoritarian tendencies. For example, electroacoustic dance music elevates the DJ from a bureaucrat with the function of selecting songs, to performance artist. On the other hand, it cultivates a magical fascination that eliminates critical thinking and homogenizes behavior, encouraging individuals to replicate the same movements and thoughts. The darker side of the electroacoustic paradigm is that it inaugurated the era of music as a torture apparatus. Cusick (2013) examines how loud music has been systematically used as instrument of violence in United States-operated prisons around the world, especially in the realm of the so-called global war on terror. Loud music when played through loudspeakers in a prison setting disrupts prisoners’ ability to use the acoustical behaviors of hearing and vocalizing to maintain relationships within their environment. The prisoner is immersed in a vibrating world that takes control over the body, mind, and senses. The extreme acoustical energy of loud music becomes a physical force that eliminates any illusion of privacy or agency. Music becomes not a metaphor for power, but power itself, literally—a vibrating presence of power that can deliver a miraculously ubiquitous battering to the sympathetically vibrating bones and skin of a man, beating him from within and without, while leaving no marks. This, too, is a way of destroying a person’s subjectivity, his sense of interior and exterior, private and public, a way of trying to reduce him to a vibrating object with its fingers in its ears (Cusick 2013, 288).

Specialized apparatuses for perpetrating this kind of violation have been developed for use in individual prison cells. A former prisoner of the CIA accounted how he was kept in isolation in a so-called “dark prison,” an individual cell with near absolute darkness and very low temperatures while being bombarded with “very aggressive, intrusive, annoying sounds” (Cusick 2013, 286). In 1971 in Rio de Janeiro during the Brazilian dictatorship, there was an instrument of torture referred to as the “refrigerator.” It was there I was held in a military prison as a teenage political prisoner: I was arrested for having collaborated with opposition groups. Arriving in the military prison I was put in the “refrigerator.” It was a small room, acoustically isolated, completely dark and cold. Different kinds of noises and sounds—such as hauling oscillators, rumbling generators, distorted radio signals, motorcycle-like sounds, etc.—shot from loudspeakers, invisible behind the walls. The electronic sounds filled the dark space and overwhelmed my body for three long days, uninterrupted: raw sounds, loud sounds, piercing sounds, disturbing noises. After a certain time, I lost consciousness. The auditory and acoustic torture I had been exposed to was then, a recent development. It partially replaced traditional methods of physical coercion that killed thousands of people in Latin American prisons between the 1960s and 1990s. The sounds injure the body without leaving any visible trace of damage. The immersive space of the torture cell, soundproofed and deprived of light, resonates in my memory as the perfect environment for experiencing the power of sound embodiment (Chagas 2006, 120–1).

In 2014, I composed the digital oratorio The Refrigerator for two singers, instrumental ensemble, electronic sounds and visual projection, in which I elaborated a multi-layered narrative for observing my personal experience of torture and the

22

P. C. Chagas

reality of torture more generally.6 Associated with the darkness of ignorance, the torture apparatus stands for the “logistics of violence and cruelty in society” (Chagas 2017, 335). The practice of sound torture, as Cusick claims, constitutes evidence of the importance of the acoustic environment for the construction of human subjectivity. We live immersed in a vibrating world that keeps us in constant touch with other vibrating entities, such as human bodies. A silent acoustic is what allows hearing and speaking, which are the fundamentals of the relationship between self and other, individual and collective, private and public. The practice of torture using sound and music shows that within the electroacoustic paradigm exists a vibrationcentered model that produces the presence of a ubiquitous, invisible vibrating power, eliminating subjectivity.

1.9 The Telematic Paradigm Technical apparatuses are black boxes that activate a ritualization of programs by drawing attention to the surface of things and keeping what is inside hidden—for example, with a cell phone. As Heidegger argues, modern technology has changed our sense of the world as it tends to reduce everything into mere resources, including human beings. The programmatic magic of technical apparatuses, including artistic apparatuses that produce synthetic sounds and images, tends to eliminate critical thinking, replacing historical consciousness with a second-order magical consciousness that reduces culture to its lowest denominator. With the technical apparatus, relations of power move from physical objects to a symbolic level of programs and operators. In the era of global networks, algorithms and systems dealing with large amounts of data and complex probability problems take control of events and make decisions. Automatic programming is the mode of operation of “intelligent” apparatuses such as cybersecurity, cybercrime, cyberattack, cyberespionage, and cyberwar. Having emerged during World War II, the science of cybernetics was born out of the need to intercept enemy signals, not to interpret the world. The operations of mathematical, probabilistic communication technology usurped the responsibility of decision maker and the result is that we no longer act as independent subjects but as pieces of war machinery. However, Kittler (1993) detected a new kind of freedom, symbolized by the existence of noise beyond any possible event. This may sound like an apocalyptic apology for military technology, but in fact, it conveys the acceptance of life as impermanent and unpredictable. How can humans construct a space of creativity that withstands automaticity and repetition? According to Luhmann (2000), art can factually exclude the world and the observer and temporally generate a randomness that transcends form. Works of art are basically self-descriptions, or signatures. Music engages with self-descriptions that operate with codes different than in the art system. In the system of education, for example, musical self-descriptions may be labeled 6 Video

documentation of the first performance of The Refrigerator: https://youtu.be/KH_EnK IttHM. Accessed June 31, 2020.

1 Sound, Truth, and Paradigm

23

as research or scholarly production. In the system of economy, music functions as commodity for monetary exchange. The overabundance of communicative possibilities presents the problem of distinguishing between art and non-art (Luhmann 2000, 314). But the art system is able to resolve the paradoxical unity between art and non-art in the art system itself, as long as it keeps its autonomy. The telematic paradigm embraces the communicative complexity that emerges from the convergence of telecommunications and information processing in today’s society. Flusser believes that telematic communication has the potential to radically transform the way we communicate. Telematics can reverse the natural tendency of entropy—the state of randomness in which information is unpredictable and therefore impossible—by converting historical and discursive thinking into dialog. In Flusser’s telematic dialog, man and apparatuses act as partners devoting themselves to the systematic generation of information through a playful game. The telematic dialog embodies Flusser’s utopia of freedom as a struggle against entropy, which emancipates man from the controlling functionality of the machine. Flusser suggested chamber music “as a model for dialogic communication in general, and for telematic communication in particular” (2011, 162). Chamber music is a paradigm of creativity in telematic society. Traditional chamber music, he claims, is a pre-industrial form of communication that anticipates the technical apparatus. The performance of chamber music requires a multi-layered interaction between bodies and instruments that shapes the communication between the musicians and ultimately the production of sound. Inspired by the chamber music paradigm, Flusser describes the telematic performance as a dialog between “musicians” and “intelligent memories,” which are at the same time transmitters and receivers of information, with the goal of synthesizing new information. But unlike traditional chamber music, which unfolds a linear flux of events, the telematic dialog opens the space to simultaneity, multiplicity, and ubiquitous communication.7 Among his many metaphors for the telematic paradigm, Flusser proposed: The scenario, the fable, I propose here is this: people will sit in separate cells, playing with their fingertips on keyboards, staring at tiny screens, receiving, changing, and sending images. Behind their backs, robots will bring them things to maintain and reproduce their derelict bodies. People will be in contact with one another through their fingertips and so form a dialogical net, a global superbrain, whose function will be to calculate and compute improbable situations into pictures, to bring information catastrophes about. Artificial intelligences will also be in dialogue with human beings, connected through cables and similar nerve strands. In terms of function, then, it will be meaningless to try to distinguish between natural and artificial intelligences (between private brains and secondary brains). The whole thing will function as a cybernetically controlled system that cannot be divided into constituent elements: a black box (Flusser 2011, 161).

The scenario evokes a scene from a science fiction movie, perhaps even the plot of a B-movie. Despite the rudimentary situation and stereotypical characters—human beings, robots, computer terminals, and a superbrain—it conveys a quite familiar account of the present that can be interpreted, for instance, as a futuristic post-human version of Plato’s cave, in which man shares the existential space with robots. Humans 7 For

an account of the model of telematic dialog, see Chagas (2014b).

24

P. C. Chagas

are not alone and yet we sense the loneliness described in this scene; a prevailing theme in science fiction literature and cinema. Certain details are familiar; the decay of the body, the relationship between humans and robots, and the inference that robots are helpers of human beings—almost care-takers working to alleviate human suffering caused by the extensive use of technology. Flusser’s metaphor can be seen as a daunting prospect of the future infused by a general skepticism concerning our bodies—or, as an optimistic belief in our capacity to reshape the world. With the COVID-19 pandemic gripping the world, many people have been working online, sitting in front of computers and mobile devices, receiving and sending messages, streaming audio and video, socializing with friends and family, and meeting new people on social media. While using digital devices, we continue to communicate with our eyes, ears, voice, gestures, smiles and laughs, conveying feelings of happiness, anger and so on. Moreover, we have begun to accept that we are increasingly more dependent on apparatuses to connect us and provide social interaction when social distance becomes a strategy for protecting ourselves from invisible living organisms. Therefore, we are using apparatuses for isolating ourselves, while at the same time continuing to communicate with the world despite experiencing this as an improbable situation and something unexpected rather than an active choice.

1.10 Conclusion The word “compose” from the Latin verb “componere,” means “to put together.” This idea of composition characterizes the music produced since the Renaissance. Several quantitative and qualitative processes of acoustic “machinery” have been developed in the course of music history such as melodies, harmonies, timbres, sound events, structures, and complete music works. Traditional sounds of vocals and instruments represent the alienation of analogical, logical states of affairs. The complexity of concrete phenomena is reduced and simplified in musical codes that represent abstractions of these phenomena. The abstract encodings, which include, among other things, musical notation and oral tradition, have been the foundation of composition for centuries. With the emergence of technical apparatuses, the symbolic function of musical codes was extended to electronic and digital apparatuses. Electronic sounds, from a symbolic point of view, are decoded as conceptual abstractions of the apparatuses. The meaning of musical apparatuses is linked to cultural paradigms. The term “electronic music,” in the semiotic sense, must be attributed to music that provides an understanding of the musical apparatus. Analog and digital devices break down traditional sounds and accelerate their alienation. The process of decomposing traditional sounds characterizes twentieth century music. The term “decomposition” would be more appropriate than “composition” to designate current musical creativity. Acoustics and electroacoustics are complementary intertwined fields of creativity. The dissolution of the analogical

1 Sound, Truth, and Paradigm

25

foundation drives the analytical approach to “natural” sound. The musical information of the electroacoustic paradigm is generated primarily through the analysis and synthesis of sound phenomena. Sound analysis serves as a model for new apparatuses and sound constructions. But the “new music,” in the emphatic sense of the term, requires a dialogue in order to be elaborated—a dialogue symbolized by a networked structure that allows the exchange of information between humans and intelligent apparatuses acting as equal partners. The exchange of information takes place in the form of a directed game in which new models of creativity are continuously developed. In the face of the diversity of tasks and activities involved in contemporary music production, the figure of the individual “creator” loses its traditional meaning and gives way to the intertwining of subjects and knowledge interacting within the scope of the dialogical network. With digital apparatuses, the concept of “game” takes on a social dimension. Only in this way is it possible to criticize the ritualization of apparatus’ programs, which encourage repetition of human behavior leading to their oppression and redirect them for creative purposes. The music cultural paradigm of the twentieth century has failed to provide a plausible answer to the tendency towards automation. Instead of searching for the new, media apparatuses promote models of the past. How do we free contemporary music from this repetitive dasein at the service of the bureaucratic function of the apparatus? How do we direct the musical dialogue towards the creation of new information? I propose the following thoughts: 1.

2.

3.

4.

5.

The electroacoustic paradigm encompasses a complexity of meta-apparatuses such as music theory, performance, music industry, audio technology, information, and communication technology. Electroacoustic music can now be produced anywhere by anyone with current technology. However, as long as the music is not integrated into a meta-apparatus of distribution it does not yet exist as information. The term “electroacoustic music” encompasses a variety of cultural paradigms, including those of “contemporary music,” which can only be decoded in the context of criticism of devices and their programs. The problem of information generation is that it must be freed from the mythical context of individual creation and replaced by the strategy of the directed game, of a collective character, which is able to unleash the creative forces of change. The game concept includes production methods and performance situations. Chamber music is a prototype of creative dialogue of the electroacoustic paradigm. The practice of electroacoustic music encourages the combination of traditional voices and instruments with musical apparatuses. Mastering a music apparatus is a time-consuming practice, comparable to learning to play an instrument. Gesture and execution must be discovered, invented, and experienced (Chagas 2014b). The new technologies of the twenty-first century—for example, bioinformatics, nanotechnology, and robotics—will soon be in a position to create intelligent machines superior to human beings. Future apparatuses might be able

26

P. C. Chagas

to reproduce not only their programs, but themselves. This perspective raises the following fear: if man allows these increasingly complex devices to assume his decisions, it may cause the programs of the devices to take control of human beings. Will we become superfluous as a species and doomed to disappear? Within this controversial context, the question of freedom arises. How are we to deal with technology? How are we to deal with the perspective of a “disciplinary” society, in which intelligent machines take charge of the functions of observing and controlling human behavior? The manipulation of the apparatuses of the telematic society gives us the opportunity to shape freedom through a dialogical game that involves a methodical search for new information and, at the same time, the engagement against the automation of apparatuses and the entropic tendency of the excess information. The question here is not the technology itself, but the social consensus that is required to change the way we think: from the current predominantly discursive communication structure towards a society forged by dialogue. The convergence of information, communication, and audiovisual technology creates new categories of artistic, social, and political practices. Automated media programs—including contemporary social media—act as a steamroller pushing us into the void of entropy. The creation of new paradigms requires dialogical processes of production of perceptions and affects, which also manage to articulate diversity and heterogeneity. As Wittgenstein argues, artistic creation must be guided by an ethical-aesthetic paradigm (Chagas 2015) of an almost therapeutic nature, in order to encompass complexity without reducing it. On the other hand, art cannot give up its autonomy as a social system in order to be able to observe the outside world. The main function of art, like ethics, is to make sense of the world.

References Agamben Giorgio. 2009a. The Signature of All Things. Trans. L. D’Isanto, and K. Attel, New York: Zone Books. Agamben Giorgio. 2009b. What is an Apparatus? Trans. D. Kishik, and S. Pedatella. Stanford, CA: Stanford University Press. Benjamin, Walter. 1977. Das Kunstwerk im Zeitalter seiner technischen Reproduzierbarkeit. In Walter Benjamin: Illuminationen, ed. S. Unseld, 136–169. Frankfurt am Main: Suhrkamp. Bird, Alexander. 2018. Thomas Kuhn. In The Stanford Encyclopedia of Philosophy (Winter 2018 Edition), edited by E. N. Zalta. https://plato.stanford.edu/archives/win2018/entries/thomaskuhn/. Chagas, Paulo C. 2006. The blindness paradigm: the invisibility and visibility of the body. Contemporary Music Review 25 (1/2): 119–130. Chagas, Paulo C. 2014. Unsayable Music: Six Essays on Musical Semiotics, Electroacoustic and Digital Music. Leuven: Leuven University Press. Chagas, Paulo C. 2014b. Creativity with Apparatuses: from Chamber Music to Telematic Dialog.” Flusser Studies 17: 2–15. https://www.flusserstudies.net/sites/www.flusserstudies.net/ files/media/attachments/paulo-chagas-creativity-with-apparatuses.pdf. Accessed July 31, 2020.

1 Sound, Truth, and Paradigm

27

Chagas, Paulo C. 2015. Musical Understanding: Wittgenstein, Ethics, and Aesthetics. In Music, Analysis, Experience: New Perspectives in Musical Semiotics, ed. C. Maeder and M. Reybrouck, 115–133. Leuven: Leuven University Press. Chagas, Paulo C. 2017. Revolt and Ambivalence: Music, Torture and Absurdity in the Digital Oratorio The Refrigerator. In Bridging People and Sound. CMMR 2016. Lecture Notes in Computer Science, vol. 10525, ed. M. Aramaki, R. Kronland-Martinet, and S. Ystad, 331–46. New York: Springer International Publishing. Chion, Michel. 1994. Audio-Vision. Sound on Screen. New York: Columbia University Press. Cusick, Suzanne G. 2013. Towards an Acoustemology of Detention in the ‘Global War on Terror’. In Music, Sound and Space: Transformations of Public and Private Experience, ed. G. Born, 275–291. Cambridge: Cambridge University Press. Dreyfus, Hubert. 2005. Heidegger’s Ontology of Art. In A Companion to Heidegger, ed. H. Dreyfus and M. Wrathall, 407–419. London: Blackwell. Flusser, Vilém. 2011. Into the Universe of Technical Images. Trans. N. A. Roth. Minneapolis: University of Minnesota Press. Hayles, Katherine. 1999. How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. Chicago: University of Chicago Press. Hatten, Robert S. 2004. Interpreting Musical Gestures, Topics, and Tropes. Mozart, Beethoven, Schubert. Bloomington and Indianapolis, IN: Indiana University Press. Heidegger, Martin. 1977. The Question Concerning Technology. In The Question Concerning Technology and Other Essays, trans. W Lovitt, 3–35. New York: Harper and Row. Heidegger, Martin. 1993. Basic Writings. Ed. D. F. Krell: San Francisco: HarperPerennial. Ihde, Don. 2007. Listening and Voice: Phenomenologies of Sound, 2nd ed. Albany: State University of New York Press. Kittler, Friedrich. 1993. Signal-Rausch-Abstand. In Draculas Vermächtnis: Technische Schriften, 161–81. Leipzig: Reclam. Khan, Douglas. 2001. Noise, Water, Meat: A History of Sound in the Arts. Cambridge, MA: The MIT Press. Khan, Hazrat Inayat. 1996. The Mysticism of Sound and Music. Boston and London: Shambhala. Kuhn, Thomas. 1970. The Structure of Scientific Revolutions, 2nd ed. Chicago: University of Chicago Press. Kuhn, Thomas. 1978. Comments on the Relations of Science and Art. In The Essential Tension: Selected Studies in Scientific Traditions and Change, 340–351. Chicago: University of Chicago Press. Luhman, Niklas. 2000. Art as Social System. Trans. E. M. Knodt. Stanford, CA: Stanford University Press. McLuhan, Marshall. 1994. Understanding Media: Th e Extensions of Man. Cambridge, MA: The MIT Press. Pinto de Oliveira, J.C. 2017. Thomas Kuhn, the Image of Science and the Image of Art: The First Manuscript of Structure. Perspectives on Science 25 (6): 746–765. https://doi.org/10.1162/ POSC_a_00264 Pousseur, Henri. 1970. Fragments théoriques I: Sur la musique expérimentale. Brussels: Editions de l’Institut de Sociologie de l’Université Libre de Bruxelles. Rhees, Rush. 1981. Ludwig Wittgenstein: Personal Recollections. Totowa, NJ: Rowman and Littlefi eld. Sterne, Jonathan, ed. 2012. The Sound Studies Reader. London and New York: Routledge. Tarasti, Eero. 2000. Existential Semiotics. Bloomington: Indiana University Press. Wiener, Norbert. 1956. I am a Mathematician: The Later Life of a Prodigy. Cambridge, MA: The MIT Press. Wittgenstein, Ludwig. 1978. The Blue and Brown Book, 2nd ed. Ed. R. Rhees. Oxford: Basil Blackwell.

28

P. C. Chagas

Wittgenstein, Ludwig. 2001. Philosophical Investigations [Philosophische Untersuchungen]. The German text with a revised English translation. 3rd ed. Trans. G. E. M. Anscombe. Oxford: Blackwell Publishing. Wrathall, Mark. How to Read Heidegger. Granta Publications. Kindle Edition.

Paulo C. Chagas is a Professor of Composition at the University of California, Riverside. A highly versatile and prolific composer, Chagas has written over 180 works for orchestra, chamber music, electroacoustic music, audiovisual, and multimedia compositions. His music unfolds a pluralistic aesthetic, combining diverse musical materials from different cultures with acoustic and digital media, dance, video, and audiovisual installations. His award-winning and ambitious productions have been applauded throughout the Americas, Europe, and Asia. Chagas’ scholarly activity has organically evolved alongside his artistic career. He has written a significant number of book chapters, journal articles, and proceedings in his four languages of English, Portuguese, German, and French. Unsayable Music (Leuven University Press, 2014), develops in great detail the main themes of Chagas’ research, which include electroacoustic and digital music, musical semiotics, and philosophy. Chagas received numerous prizes for his work including recently a Fulbright Scholar award.

Chapter 2

Existential Semiotics and Its Application to Music: The Zemic Theory and Its Birth from the Spirit of Music Eero Tarasti

Abstract Existential semiotics, a new theory stemming from combination of classical semiotics (Paris school) and continental philosophy, has led us to two notions of transcendence and—zemic. By zemic we understand four modes of being (Hegel) interpreted via the categories of Moi and Soi, inserted first to a semiotic square and then dynamized by two movements of sublimation and embodiment. A new model for music analysis grows from this starting point. In the background loom the two ways of transcending: transascendence and transdescendence (Jean Wahl), which appear as gestures of musical discourse. Ultimately this is close to the’breathing of the hand’ at the piano playing method by the Parisian teacher Jules Gentil. Moreover, historically, the idea occurs also in some notes in the Brown Book by Richard Wagner referring to Indian Hindu philosophy. This has its consequences in the whole of theory of existential semiotics as a metaphysical system. Keywords Existentiality · Moi and Soi · Gestures · Transcendence · Semiotics · Performance We have arrived in our theoretical efforts at the most important section of the inquiry, namely, so-called zemic theory and closer definition of this concept than in any of my earlier essays. Yet, let us recall a little our earlier achievements: the emergence of the zemic model from four modes of being and their ‘semiotic square’ according to the Paris school, of the dynamization of this achronic model, when it became Z-model, in which four modes Moi1, Moi2, Soi2, Soi1 (later often abridged into M1M2S2S1) are in constant movement; moreover, as the background of the model has served, since the beginning, the notion of transcendence, in the model (Tarasti 1994, 2000, 2012, 2015) which portrayed the journey of the subject to the emptiness of transcendence or to its plenitude depending on whether one arrived there by negation or affirmation. Yet, the zemic structure was determined as M1M2S2S1; however, because zemic is a part of human reality it strives for being represented as objective sign or by E. Tarasti (B) University of Helsinki, Helsinki, Finland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 P. C. Chagas and J. Cecilia Wu (eds.), Sounds from Within: Phenomenology and Practice, Numanities - Arts and Humanities in Progress 18, https://doi.org/10.1007/978-3-030-72507-5_2

29

30

E. Tarasti

reaching the level of sig-zemic. Inside it function in addition the categories of logzemic as particular function of those signs which describe how they function. Furthermore, subject has ability to continue its transcendental journey towards the level of reflection when he/she attains the conceptual level or what I call suprazemic, or what Hegel called Wesen, essence. However, even behind this level there is one level more, higher which has been called radical transcendence or trans-zemic. Radical because it differs from the abstract spiritual world of values of the mode Soi1, which constitutes a kind of immanent zemic transcendence. In this case, therefore, our subect aims for the highest transcendence which is the eternal world of concepts—concepts which exist definitely absolutely. This would be Hegel’s level of the absolute spirit to which the spirit aspires through trial and error in his/her worldly wandering. Transcendence is thus on one hand the creation and construction of our Sartrean subject, reaction to our unbearable situation that the real empirical world of our Dasein is so imperfect. Yet, if we do other epistemic choice it is contrarily so that the transcendence is the only real and it only announce itself via suprazemic to us, our concrete zemic reality where we stay. It is thus like theological annunciation. From this point of view, the speeches of those philosophers of classical centuries of God are understandable. Now we are ready to look what the reality of zemic contains after all. Accordingly, if we have found the following levels: zemic (M1M2S2S1) suprazemic, concept, Wesen or essence, transzemic or transcendence, sig-zemic or the representation of zemic in signs, utterances, texts (many consider only this level semiotical!) and log-zemic or the categories functioning in the model. Then, what problems do still remain? First, insofar what is involved is a far-reaching world explanation model as many have come to say to me when I have lectured my theories, like, in St. Petersburg or in Moscow, in which it has been understood (surprisingly!), then two questions rise: what after all makes the zemic hold together and guarantees its inner coherence? Although many postmodern philosophers do not even try to attain this. For them the faultless, flawless, unified subject is already non-existent. Yet, in many models of semiotics, such a unifying principle has been postulated: The Prague school had semantic gesture, Russian formalists dominant, Greimas isotopy, Peirce synechism, Lotman culture, Eco encyclopedia of knowledge. Could we dare to call this unifying and assembling aspect the soul of our zemic subject? Another problem is the sociality of zemic. Of course, one can say that the social is already embedded in our model into its Soi and the idea of the whole model was to clarify just the human mind as a synthesis of Moi, ego and Soi, the social. However, if every subject—or the substance of the world if one wants to think like Schelling, Leibniz, Spinoza and McTaggart (1988)—constitutes a ‘zemic’, then how our zemic1 can live together with zemi2? How the zemics 1, 2, 3 etc. can live in an interaction without hurting all the time each other, and without falling to the trap of Empedoklean principle of quarrel and aiming instead for his principle of friendship—which was also the most important to the composer Nietzsche (see his composition Hymnus an die Freundschaft). In other words: upon what the social is based which would prevent the zemics from being a kind of solipsistic monads?

2 Existential Semiotics and Its Application to Music …

31

Fig. 2.1 Semiotic processes of embodiment and sublimation

It is best to start to analyze our zemic model and its implications. First, one has to correct one ‘mistake’ appearing in the earlier versions or we have to make a small improvement. The model situates the four modes M1M2S2S1 or body, person, praxis and value to a square in which Moi is up and Soi is down. However, if we think that Z portrays the inner movement of the model, then what is involved is the shift from the material to immaterial, from sensible to intelligible, from ‘heavy’ to ‘light’, so to say. It would be intuitively more logical if Moi as ‘heavier’ were down and Soi as a spiritual entity up. The movements under question were sublimation or the change of the corporeal into spiritual and embodiment or the change of the spiritual to concrete and bodily. These two processes I indicate by signs (Fig. 2.1). If we look in practice what these zemic modes keep inside or how the zemic is eine Tatsache and substance (see McTaggart and other metaphysician since Aristotle), then that would be crystallized: Moi1 = my body breathes, heart is beating, blood is circulating—these are facts (Wittgenstein describes this level of being: “Comparison of bodily processes and states, like digestion, breathing etc. with mental ones, like thinking, feeling, wanting, etc. What I want to stress is precisely the incomparability. Rather I should like to say, the comparable bodily states would be quickness of breath, irregularity of heartbeat, soundness of digestion, and the like. And, of course, all these things could be said to characterize the behavior of the body.” (Wittgenstein 1980: 122c).; Moi2: cogito ergo sum, I exist, Erlebnisstrom; Soi2: being is always social, it is fait total social (Marcel Mauss), Qui dit l’homme, dit société (Rousseau), being is cultural; Soi1: values exist but they cannot be concretized. Again if we want to concretize them like for instance: S1 = patriotism, Finland; S2 = orchestra, ballet group, ice hockey team, seminar in the university; M2 = personality (see Eino Kaila, Goethe), sewing club, family, amateur group; M1 = body, sportsmen, sick, musicians, gourmets, sybarites, vegans, faqirs. Wittgenstein distinguished psychology = Moi1, and aesthetics = Soi1 as to the arts. He makes observations of composing techniques when he spoke about Brahms and Joachim: Joachim suggested to Brahms to begin his IV symphony not with first subject but with 2 introductory chords. Brahms refused…. That European music has certain laws does tell you something about psychology of Europeans, just as that they have this sort of mathematics or physics (music as Soi2!)… you may ask: what is common to all music from Palestrina to Brahms? And one might answer: they start from tonic, go to dominant, and return to tonic… Suppose I hear a waltz played. I am old enough to know how a waltz used to be danced 30 years ago & therefore how one ought to be played. I know a young man who can’t play a waltz, because he’s never seen one danced… How here I’ve given a cause of my dissatisfaction with his playing… psychology can give the cause why I have this idea rather than another. And when he speaks about the excessive speed or heaviness of the bass line (Wittgenstein 2016, 350–1).

32

E. Tarasti

Fig. 2.2 Indications of proper transcendence

Here is connected still another pair of notion: transascendence and transdescendence (distinction by Jean Wahl). In a way Soi1 is already transcendence i.e. not-visible, that which is absent but present in the mind, yet to be thought; for instance, ‘imagined nations’. However, this empirical transcendence has to be distinguished from the proper transcendence which is indicated by arrows up and down (Fig. 2.2). ‘Can Soi change into Moi’ is a fundamental question. Can Soi for instance compensate Moi? Therefore, something is lacking in Moi and Soi replaces it? When I wrote my memories I thought: “My whole life has been a struggle between two forces, I have tried to sublimate Moi into Soi by attempting to become a professor, scientist, musician, chairman of societies etc. and altogether all kinds of activities of Soi. And at the same time amidst Soi the Moi living in me has demanded to be heard of or I have tried to embody the orders of Soi in my mind, those so-called ‘objective requirements of life’. Soi had not yet intruded me completely and developed in me as demands set from the outside, but as internalized mastership (Marcuse). But it cannot compensate Moi. Once, the rector of Indiana University, Myles Brand, in his speech of the Day of Commencement in Bloomington, was able to formulate a balance between Moi and Soi: “Follow your passion but furnish your mind!” Thus Moi and Soi fight with each other. This was the constant theme at the philosophy of Theodor Adorno. Moi can be like the Finnish poet Uuno Kailas: “….so I once started my way and so I end it - barefooted”. The last movement of Moi is not the return of Jed but return of Moi, Hafiz’s second youth: “The young looked at mirror and did not recognize himself, only the second youth is right one”. How Moi has influence upon one’s life as Soi, or the Soi world? Is it not rather so that by the time getting older, the Moi should weaken, shrink and at the end disappear like in the teachings of Sufi mystics and Hindus. Now one should gather the themes of one’s life from the viewpoint of Moi like in a symphony in which Moi has been transformed into symphonism. In my original theory, Moi is always only to great extent in the side of Soi. And this is in fact a Proustian theme, the social surface of the reality keeps the narrator as its prisoner at the party of Mme Verdurin and only when fleeing from salon the narrator experiences that all that did not move his deep ego or Moi. The examples I shall deal with will be from now on mostly from music. Often in music there are two interlinked motifs; 1. embodiment which is close to the transdescendence and 2. sublimation which is akin to transascendence. For instance,

2 Existential Semiotics and Its Application to Music …

33

the Tristan chord; ↓ indicates dissonance, dysphory and ↑its resolution upwards, euphory. In the Pilgrims’ choir of Tannhäuser, the tension between the wave-like sublimation and embodiment, their alternation with plain linear movement connected by chromaticism (see the meaning of chromaticism at Gilbert Rouget in his study La Transe). The Wolfram song has oscillating motion, does the motif or the theme M2 emerge from its sound background? The beginning of the Rhinegold is an example of how a long field of Moi1 grows and expands and resolves at the end into M2 or song of the Rhinedaughters. The Lohengrin overture is an illustration of the movement from ethereal sound of Moi1 and its gradual transdescendence or sinking down on earth and climax with the dissonant chord of IV# and its resolution…and then again raising to the immaterial Klang universe of Moi1. Or is it so that M2 creates convenient ground for itself, does M2 emerge from some topics? Beethoven’s Sonata op. 2 nr. 3 C major. Beginning is completely S2, conventional style, M2 is a canonic four-bar unit, symmetrical, galant style with trill as a motif, always repeated… until from this breaks out the field of Moi1 as arpeggiated triads, which sweep over claviature (although even this is also topical: virtuoso concertante style! Also S2, in fact). What is involved is damming of S2 which explodes into Moi1 or it is a contrary method to Rheingold and Lohengrin overtures. In Wagnerian leitmotifs, which one might imagine to be typical M2 actors, very frequently S2, the topics is present, so they are not only iconic signs representing something: Valkyries = riding of galloping horses, the return of the Valkyries in III act = fugue, Siegfried’s horn signal = hunting topos, Loge’s M2 emerges from flippering fire as M1. Winterstürme… = Volkslied topics, redemption, aesthetic idea of S1 = Italian bel canto, vocal style, it also evokes the value of hope closing Götterdämmerung (note like Christian Thielemann: no Wagner opera ends with a minor chord!). Does M2 ever emerge from S1 directly or from an aesthetic value? The finale of Mastersingers: Gino Stefani’s prenatal style of rotating, rolling style, the ecstasy of the end of the Preislied, making one drunk. Entsagung motif. Its effect is based on minor harmony, the tonal passage behind the M2 is Es:I-VI: it is M2 which creates itself; Fafner who appears unprepared clumsily—something frightening like the Flying Dutchman with his fifth-fourth motif. In any case, the above offered series of music examples shows how zemic is carried to symbolic meaning or how zemic is represented in and by its own signs or sig-zemics, as we call them. What Wittgenstein says justifies to us the comparisons between zemic and leitmotifs The way music speaks…might one not imagine someone who had never known music, and who came to us and heard someone playing a reflective piece of Chopin, being convinced that this was a language and people were merely keeping the sense secret from him? Verbal language contains a strong musical element —A sigh, the modulation of tone for a question, for an announcement, for longing: all the countless gestures in the vocal cadences (Wittgenstein 1980, 156–7c).

It is right he speaks of gestures. Namely the relationship between the proper transcendences and sublimation/embodiment movement is just this one: they are representation of sinking and rising transcendences, as gestures. Music is just language of such gestures.

34

E. Tarasti

Now I introduce a method of piano playing elaborated by my Parisian teacher Jules Gentil, which was just based upon such gestures of which he found two basic species: pousser, push and tirer, draw or shown by signs ↓ and ↑. He drew these signs all over a score, upon notes, pousser signified the attack of the hand down and tirer its rising up, relaxation and the so-called la main morte, ‘dead hand’ as he called it. It is enacted by waving the hand in the air letting it hang totally freely without any tension from the arm. The alternation of these movements Gentil portrayed as ‘breathing of the hand’. They can be now identified with the embodiment and sublimation of the zemic. So, we get a new and complete system of musical interpretation. I quote some examples. For instance, Beethoven’s last piano sonata op. 111 begins very dramatically with a dotted rhythm and diminished seventh chords. They are on one hand an example of how on the level of topics or S2 a composer in the classicoromantic age composes with topics and not only blindly copy them. The rhythm refers to Baroque French overture and the dissonant chords to Sturm und Drang. Texture is dense but it starts so to say animate and breathe, when it is provided with arrows↓↑ (Fig. 2.3). It is particularly worth noticing that music lives and continues to breath although one would not hear any sound attack: hand rises with breathing out even in pauses i.e. suspended notes (see bar 6), And when it descends to the gaps of transdescendence (see bars 11–13), there the movement pousser/tirer is quite essential to yield the right character. Another example is the E flat major piano concerto by Liszt; it opens with a motif with a chromatically descending M2 motif from E flat major tonic, which is a direct evocation of the beginning of Eroica of Beethoven with altered notes e flat-d-c sharp—which was according to Wagner the beginning of the modern music (Cosima: Tagebücher). Yet, in this music which is fully romantic and where the content determines everything, is the aforementioned breathing interpretation a and o (Gentil told his method was in fact based upon the notes of Liszt’s pupil Mme Boissier, from his lessons see Boissier 1997: 40 et passim: “Oggi prima di tutto ha fatto studiare alcuni passaggi ai quali tiene molto per rendere la mano più elastica ed energica…”; as such she does not portray the method but it is rather based upon oral tradition in Paris). A.o. the heavy C major parallel chords assumed their correct elastic Klang just from this technic (Fig. 2.4). Furthermore, Debussy’s l’Isle joyeuse is an example of Gentil’s method as applied to impressionistic music. This work elucidates—again following the Parisian traditional knowledge—Watteau’s painting L’embarquement à - or’de’! - l’Isle de Cythère; music has movement towards ethereal air, but also electricity and even oriental flavor of broken major triads without a third, the 18th century exoticism as a reference to gamelan if one so wants (Fig. 2.5). In fact, the interpretation of this breathing music can be already juxtaposed with oriental philosophy, the Hindu doctrine which Wagner in his time diligently studied, not only adopting them from Schopenhauer but examining a.o. the history of India by Burnouf: Introduction à l’histoire du bouddhisme (see Millington 1992:147). Wagner once said to Cosima (in her diaries) that his leitmotifs which always return are like reincarnations of souls.

2 Existential Semiotics and Its Application to Music …

Fig. 2.3 Beethoven Opus 111

35

36

E. Tarasti

Fig. 2.4 Liszt, Piano Concerto No. 1 beginning and cadential passage

Nevertheless, the central concepts in India are Brahma and atman. Atman is a linguistic form of reflexive pronoum in sanskrit, it can mean the body, and whatsoever what one can adopt as one’s ego or belonging to ‘me’, the meaning of which leads to the question: what is this ‘self’ or ‘I’, the subject of all emotions, thoughts and wishes, of which elements it consists. The term Brahman has many meanings, it has been derived from the root “brh” or to grow, to become great. In the Vedas Brahman means a sacred utterance, the one

2 Existential Semiotics and Its Application to Music …

37

Fig. 2.5 Debussy L’Isle Joyeuse

whereby the beings become great. Later it started to mean a ritual and those who took care of them or brahmans, Upanishads finally furnished it with meaning which signifies extreme reality, and whereby one must understand the breathing of the life of the universe and everything included. Yet, it is also used later analogically to describe word, eye, ear, heart, sun, space, what all are called brahmas. However, every identification with whatsoever concrete object was forbidden and brahma became synonymous to that which cannot be thought imagine, to a mystic entity which does not have a name. Specialists, however, want to identify atma with soul and brahma the supreme being, since those terms have a Western tinge and they refer to the problem which remains unresolved in Upanishads, namely the question whether reality is a unity or multitude (Klostermaier 1994, 205–6). In atma one distinguishes also five levels corresponding to five brahma realities, each reality having another one as its core, until one reaches the center of being which is bliss. Many brahma myths portray the striving for a brahma state, in which there is no evil, no ageing, no death, no worry, no hunger, no thirst and whose desire is truth and reality. Indra, the king of heavens, aspires for, but he has to wait for 101 years before he reaches atma, in which are all the worlds and fulfilments of his wishes. These ideas were brought to music by Richard Wagner. In his Das Braune Buch, the diary that he kept after writing his Mein Leben (1869) and before Cosima started her diary, Wagner quotes these ideas directly from the Indian philosophy. He sketched in 1868 a Buddhist drama the Winners (Die Sieger) and tells about it in his letter to Ludwig II; he writes: Truth = Nirwana = Night

38

E. Tarasti Music = Brahma = Twilight (Dämmerung) Poetry = Sansara = Day

In this connection, he also wrote a musical theme which later became the socalled Welterbschaft motive in Siegfried’s 3rd act and under these lines Wagner prepared a larger text (Fig. 2.6): New worldview: In Dhyana the rising essences ascend to the world, which were allowed to enjoy a reward for their previous virtuous acts, and now they step again to the sphere of the newborn, in order to obtain there even greater perfection. Sweet juice emerges from the earth; it ends the longing for sucking the desire for a new life: then the juice gets dry, rice bursts out infertile, there is satisfaction but then it descends. Now all has to be cultivated by themselves, plough and seed, The complaint of the life starts. Paradise is lost. At Brahman music calls it back from the memory, it leads to truth, Who understands it? Milk which has not flowed from any cow? It becomes Brahma as a nostalgia, it aspires the music; yet the music of samsara is an applied music, or poetry (zugewandte Musik), what is the other side of the one turned away from samsara? Nirwana = careless pure harmony? (Wagner 1975, 176–8)

This text is very significant also for our metaphysic zemic system. For the first, it was stated above that in musical interpretation the principle of inhalation and exhilation is realized, pousser and tirer movements, and from this we have only one step to interpretation in Brahma’s sense. Yet, we can also pose another question: what is the soul of a zemic? Shall we dare to claim that every zemic has a core, atma or self? The task of this soul is to constitute a unity of modes of zemic, in which there are four characteristics or qualifications to use McTaggart’s terms. In Wagner, the core of every leitmotif is atma, but it gets different qualification along the zemic. For instance, the basic meaning of the Waberlobe motif is to get freed, replaced, à l’aise, catharsis—but it assumes different varieties to its contents when it leads to the truncated version of Siegfried’s hero motif, it is by its mode melancholic, a look at the past, to something lost, it is a farewell motif, Wotan’s farewell at the end of Walküre (Fig. 2.7). The series of leitmotifs illustrate splendidly McTaggart’s notion of determining correspondence i.e. the chains of zemic, somehow like in email messages! They continue the same plot, and this can be also interpreted as Wittgenstein’s family resemblance (Wittgenstein 2008) or Carnap’s Ähnlichkeitserinnerung. Atma determines which zemic mode is foregrounded. In Waberlobe of course M2 but its M2 develops from the flickering fire motif of Loge, which is a sonorous Klang motif and therefore M1. Yet, also aesthetic idea S1 emanates there i.e. the tragical katharsis and here the S2 is enacted or style rules and their praxis following Aristotle. This zemic is based upon freedom, carelessness and is thus different at the end of Götterdämmerung, where Wagner does not want its return perhaps as too light, but rather combines the hope motif and hero motif in the quite last measures of the tetralogy. The metaphysical breathing of the music or Wagnerian atma—not asthma as Wagner once joked with Cosima!—is dominated by those two forces in the zemic process: one striving for lightness, airiness, spirituality, idea…and the other for heaviness, earthliness, gravity, depth. They are the breathing of musical works, just like in

2 Existential Semiotics and Its Application to Music …

Fig. 2.6 Wagner’s manuscript

39

40

E. Tarasti

Fig. 2.7 Wagner Waberlobe motif in Die Walküre, Act 3

the aforedescribed method by Gentil we found principles of pousser (like professor Tapani Valsta adviced in piano playing. Press, press!) and draw, tirer; they constitute not only breathing of the hand but also the pulsation of the musical organism. However, can we expand our metaphysical principle to our whole existential system? Could we jubilate: bravo, now the whole theory of semiotics has been interpreted, in fact, existentially, it has been transcended! Summa totius semioticae. All the theories can be hence seen in this light: • • • • •

Saussure: langue↑ parole↓, signifiant ↓ signifié ↑ Peirce: representamen ↑ object ↓ interpretant = Lotman: culture ↑ text ↓ dominant ↓ biosemiotics: organism and Umwelt, merken ↑ wirken ↓ (Ich-Ton ↑) music: (in general: tonic ↑ dominant ↓, (Schenkerian) Ausfaltung ↑ and Fokusieren ↓ • painting: Attraction point (theory by Altti Kuusamo)↓ looking elsewhere↑ • Ortega y Gasset: the detachment of art from the human ↑

2 Existential Semiotics and Its Application to Music …

41

Fig. 2.8 Metaphysical forces in Wagner’s Tristan chord passage

• Schiller: Spiel, elevation lightness ↑ • Hegel: the development of the spirit towards the absolute ↑; errors of the spirit in this journey ↓ • Greimas: modalities = atmosphere where all happened (Heidegger’s Befindlichkeit): isotopy ↑, modalities (corporeal) ↓, • generation portrays all in relation either to ↑ from phonemes to symbols or ↓ from semantic gesture of phemes. In any case, such an analysis goes always in two directions. Therefore when we study, let us say, Wagner by zemic i.e. look at which zemic mode is looming behind sig-zemic, it always occurs following these two metaphysical forces in two directions which alternate. As early as Umberto Eco spoke about two approaches: one starting from the global meaning of text to its parts and the other starting from smallest units to major entities (which were like two trains leaving at the same time from the West and East costs and meeting at one point in the middle); so ↓ and ↑ (Fig. 2.8). In melody, there is the ↑ movement and ↓ and stress is on dissonance, but its resolution demands continuous rise. The whole Tristan prelude as a whole is continuous sublimation, inhalation ↑ and there is only one exhalation↓, pousser, namely the famous Tristan chord (of course the shift from E flat major in Isolde’s Liebestod to B major at the end is a huge exhalation, liberation ↓↑. If music begins with whatsoever later mode of zemic, it is always possible that the earlier mode draws it down, embodies it, and makes it heavy; so happens in the Beethoven sonata in C major Op. 2 Nr. 3 in the first part: it launches in complete M2/S2 mode world, i.e. theme and recurrence, dialogue, Doppelschlag, learned style etc. but all these dissolves in bar 13 beginning in the energetic Moi1 world which rushes up and down as a virtuoso texture (Fig. 2.9). which makes it heavy and substantial and correspondingly the earlier one starts i.e. by Moi1 then it has three possibilities to elevate, spiritualize and sublimate. Yet, the dissolution to a descending figure towards Moi1 and the disappearance of Soi2

42

E. Tarasti

Fig. 2.9 Beethoven, Piano Sonata in C major, Op. 2 No. 3

and Moi2, dissolution, evaporation, pulverization, particularization, occurs there and what remains is only the perfume of previous elements in the air. Now one has to take into account that zemic as such is not the same as the whole universe of the musical work. All those signs and symbols whereby zemic appears are called sig-zemic. As to music, there is the further problem whether music can be reduced to any zemic, since music is always a world with its own laws. All the efforts to reduce music to some specific intonations determined by social class or culture are doomed to failure. The saddest case perhaps is socialist realism, but also in other domains the idea to take zemic as the starting point of musical sig-zemics leads astray or argumentum ad hominem reasoning (“… if a composer, performer has a bad ideology and identity then automatically the music he produces is also bad and is to be rejected”, there are many examples in new musicology movement of such a reasoning in gender analysis a.o.). Between zemic and sig-zemic a huge network of influence is formed. Essential is that when pTq (see v. Wright’s logic of change in his Norm and Action (1963) or MoiTSoi and MoiTMoi (circled), then is the change T an act or an event? Act or choice of the zemic mode and invention or organic following or action and event without a special agent? We get a chart: Moi1TMoi2, Moi1TS2, Moi1TS1; Moi2TMoi1, moi2TSoi2, Moi2TSoi1; Soi2TMoi1,S2TMoi2,S2TSoi1;and Soi1TMoi1, Soi1TMoi2 and SoitTSoi2; what about if mode influences itself and a.o. Moi1TMoi1? And if MoiTMoi i.e. Moi keeps the same? And Moi1TMoi1, MoixT or Moi changes by its own desire and will and power?

2 Existential Semiotics and Its Application to Music …

43

Fig. 2.10 Peirce’s triangle applied to the zemic

Separation of zemic and sig-zemic functions well a.o. in the theory of art education. The aesthetic experience of the subject, consciousness, and enlightening is zemic but a text, sign or artwork is sig-zemic. What happens in fact between zemic and sig-zemic? Is that ‘something’ between them T or transformation or R i.e. representation? If R-relation is an act, is it zemic which always yields sig-zemic? Yet, sig-zemic can strongly influence upon zemic but not directly. There is needed always mediation of suprazemic or signifying operation. If sig-zemic is for instance flag of a country, it can exercise its effect upon zemic’s behavior only if it is recognized to carry certain meaning. For instance, blue and white cross flag had an impact upon conduct of a Finnish soldier as S2. Is then suprazemic a kind of Peirce’s interpretant or secondary sign which connects sig-zemic or representamen to its object, or could we here apply Peirce’s triangle (Fig. 2.10). Moreover, we get the diagram of various combinations of representation between Moi (zemic) and Moi (circled) i.e. sigzemic. Such a combination evokes the chart of modalities in Paris school. They are possible but did any one after all use them in practical semiotic analysis? If we think again as an exercise Wagner and Tannhäuser has two kinds of sigzemics (Fig. 2.11). Behind them is transzemic which is lucky experience, embodiment is realized on earth, happiness can occur on earth as its manifestation. In the background, there is the unisono chromaticism of strings as a reference to the Venusberg or theme M2 plus its background M1, which are totally opposed. The prelude already contains like a hologram just as in Tristan and Meistersinger the whole opera. In Ring the sig-zemics of the leitmotifs get old. They appear so many times that one gets tired with them. It is not enough that we dissolve the sig-zemic into its parts, ‘sigs’ i.e. signs, one has to look after what happens inside them and in the whole. Cosima always relates in her diaries sig-zemic always to zemic. Every day in Richard’s life is like a complete manifestation of the zemic model, we could say!

44

E. Tarasti

Fig. 2.11 Sig-zemics of Wagner’s Tannhäuser

Fig. 2.12 Zemic lines

Next we have to talk about the temporality or so called zemic line. Is it like Schenker’s Urlinie i.e. one might postulate M1M2S2S1 as superimposed levels, from which one gradually abstracts concrete Moi1 (where all the notes are included) selecting M2 and S2 and finally aesthetic ideas of S1? (Fig. 2.12) Does the musical process take place as a metamorphosis and as a transformation from M1 to M2 and therefrom to S2 and S1? How should we think of cases in which, for instance, M1 directly conveys the S1 and its aesthetics? Or praxis S2 (like Klangfarbenmelodie) and M2 in general? We encounter a special problem which is how can we join temporality to zemic. We approach “music as heard” (see a.o. Thomas Clifton, Gisele Brelet etc.), i.e. the surface of music. If Schenker’s levels Hinter, Mittel and Vordergrund are superimposed, then in zemic they are also like that, but at the same time also consecutive, but it can as well be entropy. In principle, speech about zemic lines of course refers to Schenker. Moreover, here we approach the essential question which is the notation of zemic and zemiclines. Can it happen with traditional notes like in Schenker?

2 Existential Semiotics and Its Application to Music …

45

Is there any leading line in Middle and Background? What does it mean that zemic line M1M2S2S1 is fore- middle- or background? Does it mean hearing as something i.e. something phenomenological à la Wittgenstein: “Only of course if I say to someone ‘Hear it like this’ he must now be able to say: ‘Yes, now I understand it;: now it really makes sense! (Something must click).” He proposes that a composer can write an advice how to hear: “Would not such a direction be comparable to a title to Programme music (Dance of the Peasants)”, so Wittgenstein makes reference to Beethoven’s Pastoral symphony (Wittgenstein 1980, 102c). Just like the example of Tom Pankhurst (2008) from his excellent guide for Schenker analysis i.e. example from the beginning of the Fifth symphony (Fig. 2.13). (What if Kandinsky would intervene here and say: please hear it instead as dots, lines and levels!) We have to go through also the analyses by Marianne Kielian-Gilbert (in music theory at School of Music in Bloomington, Indiana University): the zemic can put hanging upon a normal form analysis like at David Neuemeyer or Malcolm Brown or Robert S. Hatten (see my notes of their courses) (Fig. 2.14). In the above scheme, one notices a new notation: we have started to indicate modes by four different symbols: ◯ = Moi1 ∧ = Moi2 v = Soi2  = Soi1 or by colours: red, green, blue and yellow, in the same order. The musical piece in the sceme 10 is completely fictive. Yet, the problem is how to describe the alternating stresses and pertinences in the zemic line? Particularly their impact upon each other, can the boxes be replaced by notes? It is very important since our previous Greimassian music semiotic method remained in the phase in which the symbols of formal logics could not be replaced by signs which would have been comprehensible to a broader audience, for instance musicians themselves. Here we should not hurt because of the same problem. Zemic levels burst out from each other. They also follow categories of log-zemic. Now we got a new notion, which has not yet been defined more clearlu. Log-zemic are categories or functions which explain the course and nature of zemic process. Thus they concern all modes of zemic. Question is: do log-zemics appear only in certain limited places in sig-zemics or rather in the border moment and whole sections? As applied to Wagner, for instance, does the log-zemic analysis explain the charm of his music (charme in the sense of Jankélévitch) or qualification (McTaggart) or does it reveal only its functioning? I have distinguished the following categories of log-zemic: (1)

Z model in two directions: it was already remarked that this model of movement can be taken as a metaphor of transcendence, its gesturalization to the zemic reality; moreover there are movements to and from transcendence: transascendence and transdescendence = ↑ and ↓

46

E. Tarasti

Fig. 2.13 Beethoven Symphony No. 5. Pankhurst’s Analysis

(2) (3)

(4)

relation of dialogue: two linear units in zemic are put in dialogical contact with each other and it is indicated  moreover two segments or zemic utterances are completely detached from each other, they are either similar or different and what is involved is fragmentarization (fragment = metaphor favored by romantics see Friedrich Schlegel’s Athenäum Fragmente, in music for instance middle sections in Schumann C major Phantasy; the beloved animal of the romantics was hedgehog, entity separated from the rest of the world by its spines)staccato fourteenth notes there is rushing forwards or teleological process, towards (Morris):→

2 Existential Semiotics and Its Application to Music …

47

Fig. 2.14 The zemic applied to form analysis

Fig. 2.15 Log-zemic categories in music

(5)

there is retardation, slowing down process of resistance, away from,←

Hence, they are called sublimation/embodiment, transascendence up and transdescendence down (cf above Gentil’s breathing of the hand), connection to Indian philosophy, dialogue, different or fragmentarity and teleologicity and resistance. These perhaps correspond to Greimas’ so-called aspectual semes, but these functions are, however, philosophically more loaded. One may ask: to which place in music log-zemics are situated? What if several log-zemics are valid simultaneously? For instance, Isolde’s Liebestod: the tonality B major represents transascendence ↑ and it is aimed for → and it is sublimation, B = C flat enharmonically. But it is also embodiment or heaviness ↓ in chords A flat—C flat—As early as Beethoven anticipated this with the false cadence of the opening motif of Les Adieux sonata. Yet in Isolde’s theme, transcendence is as a gesture an embodiment but the end with B major is sublimation (see the harmonic analysis by Tuttle). Or transcendence (Fig. 2.15). In practice, the modes of the zemic level burst out from each other.

48

E. Tarasti

Fig. 2.16 The modes of the zemic

They follow also the categories of log-zemic i.e. if what is involved is sublimation one starts ‘heavily’, from material attack, inhalation, pousser…and then one is shifted to lighter, more spiritual tirer, exhalation. Log-zemic explains the hierarchy of zemic. Yet, moreover one has to take into account the Erlebnisstrom or whether it is so that listening experience arranges the zemic lines as foreground and middleground. And that there is also Aufführungstrom thinking performer of music, énonciation. If basically Moi1 is Klang then from it actors outburst or Moi2:s. They again arrange according to the form or S2 and it in turn follows the ideas of aesthetics or S1 (Fig. 2.16). Sometimes it is hard to distinguish the kinetic field of Moi1 and Moi2, the motifs glimmering there, like in the last movement of Beethoven E flat major sonata op. 31 nr. 3. There finally Moi1 field raises into an expressive utterance i.e. the gesturality is ennobled into ‘symphonic’, as Adorno said it, when one modulates I.VI.IV.V (the same techniques at the end of finale of Pastoral symphony in bars 219–237. F:I-VI-V7IV„IV… I) In some cases, Moi1 pulsates all the time in the background, like in the same sonata op. 31, nr. 3, scherzo staccato fourteenth notes figurations. Modes do not disappear anywhere during the process when new modes in music burst out, Klang exists always. Themes dive up and remain hidden, the form is always present, and aesthetics determine all. See for instance the prelude to Rheingold (Fig. 2.17). Can any music do with only one or even two zemic lines? Nothing can be without Moi1 Klang …or no, in Bach’s Kunst der Fuge we have a music for whatsoever instrument and basically for thought only. In the extreme, we find something like Dieter Schnebel’s Music zum Lesen. How is serial music in its utmost forms? Is it not merely S2 or constructed, conventional system? (Fig. 2.18) In fact, one could call foreground overground i.e. Obergrund (the level is aesthetics) and the genres and forms Middlegorund together with M2:s thematicity, and then as the lowest would remain Hintergrund or in this case Moi1 Klang. And

2 Existential Semiotics and Its Application to Music …

49

Fig. 2.17 The zemic on Wagner’s Das Rheingold Ouverture

Fig. 2.18 The zemic and structural levels

this is not far afield from Schenker who believed that Ursatz or Uurlinie plus Klang were nature’s biological principles in Goethean sense. Zemic line does not consist only of Z movement but of the synthesis of all logical relations and categories, consecution and simultaneity; one might analyse Finlandia by Sibelius as follows (Fig. 2.19). What is then the soul or atma of this Moi2? It namely determines how those log-zemics are stressed and organized. It is noteworthy that for instance, the gesture rising at the end of the Finlandia motif is not any longer mere sublimation but really transascendence or a prayer! (even if the symbols subl./embod. are never directly same as transascendence and transdescendence).

50

E. Tarasti

Fig. 2.19 The zemic applied to Sibelius’ Finlandia

The surface of music hence consists of themes and forms, M2 and S2, and they emerge vertically between two forces. Soi1 or the aesthetic idea strives for concretization in embodiment and Moi1 or body or khora strives for upwards. The log-zemics are either vertical or horizontal: Z↑ and Z↓. ↑,↓ are vertical and , , →← are horizontal. Difference with Schenker lies in fact that Schenker is merely linear or horizontal whereas zemic analysis is also vertical or takes place according to this scheme (Fig. 2.20). or the superimposed categories from up to down are (1) aestheticality Soi1, (2) structurality Soi2, (3) thematicity = Moi2 generalized as intonativity, and (4) Moi1 or kinetic energy. Yet, how this functions in the notation of analysis? Since the modes are continuous, is it sufficient that only their beginnings are indicated and the rest with some of the aforementioned symbols i.e. Z↑ and Z↓ contain all four modes which can be notated by circles, triangles down and up and square. Among them one describes which qualification each one has? They are numbered and listed down as a paradigmatic chart:

2 Existential Semiotics and Its Application to Music …

51

Fig. 2.20 Vertical log-zemics

circle 2 triangle 3 triangle 4 square 2 which means: 1. gesture, 2. kinetic quality, 3. rollling, staccato etc. (Stefanis’ prenatal styles) and 4. idea, value 1 theme ABA 2 theme ABB 3 Fugue, etc. 1 sonata 2 rondo, 3 fugue, 4 anaphora, 5 ellipse, etc. 1 balance, 2 symbolism, 3 Erhabene (the sublime), grotesque, comic, tragic, etc. In this analysis, shall we pick up in notation the key note i.e. those in which the aforementioned modes and qualifications appear, like Schenker for instance enlarges or italicizes the notes of Urlinie and Ursatz? Does zemic analysis at the end come to similar type of scheme as in Pankhurst’s book and his example of the beginning of Beethoven’s V symphony, in which behind the motivic surface texture a broader neighbour-note passage is revealed? Is it also so in zemic analysis that for instance behind M2 and S2 we can find M1 and S1? How all is notated? By different colors as proposed already? Let us go further! We have hitherto dealt with in fact only one zemic at a time, but like in McTaggart’s philosophy, their series form zemic time series T1, T2 and T3. Or music is essentially time art (Fig. 2.21). In this picture, the numbers inside the levels 4,3,2,1 portray the pertinence of each level i.e. importance and centrality in music…. At the background is the idea that M1 = background, M2 + S2 = middle ground and S1 = foreground. Namely the modes or pertinences vary according to numbers continuously according to their attractivity. We get another scheme in which we replace the contents of the levels by symbols of modes when the lines between T1, T2 and T3 describe how the importance levels vary along the work. At the end these signs— circle, triangles, square can indicate either one note or their group, and finally the whole texture. Can one thus pick up from the sig-zemic its hidden attraction points and determining zemic moods like in Schenker analysis? The size of the sign—big triangle, small triangle, small circle, etc. (Fig. 2.22) could thus represent their digitalization into numbers 4,3,2,1; in other words, inside those modes and symbols one could insert another symbol from the series of seven signs of logzemic (Fig. 2.23).

52

E. Tarasti

Fig. 2.21 Paradigmatic categories

Fig. 2.22 Zemic modes and symbols

Furthermore, do the log-zemic symbols hold true in all modes and not only for instance in Moi1? Do they stem in depth dimension foreground—middle ground— background? Or are fore (or over), middle and back (under)ground observations phenomenological categories? In Schenker, background, which is inferred as such in the analysis and is a deduced concept, can change or shift into foreground. For instance the triple motif of Schubert’s Erlkönig is the Grundbrechung of the whole work in Ursatz… but at the same time a motif of surface level which is heard right at beginning in piano and which pulsates all the time as a flickering figure on musical surface. Which notation takes into account the aforementioned alteration?

2 Existential Semiotics and Its Application to Music …

53

Fig. 2.23 Log-zemic symbols inside modes

Now, what remains is still that how musical works influence each other and how finally the zemics change in the time series. This has to be studied inside one work or its part and as a series T1 T2 T3 as well as in the entire artistic culture or what we call music history. Or this is a question of the internal versus external sociability and influences of music. It is claimed that behind each work by Brahms behind its sig-zemic looms some work of Beethoven. In other words: behind Brahmsian sigzemic there is Beethovenian sig-zemic (see example for instance in the Brahms C minor piano quartet!). Or if one thinks of the beginning of Liszt’s piano concerto E flat major as it was said earlier (Fig. 2.24). Or if one wants to go to intertextual level: the relationship of Liszt’s Dante sonata to Dante’s Divina Commedia! McTaggart’s concept of determining correspondence explains this phenomenon excellently. The aforementioned cases Brahms/Beethoven, Liszt/Beethoven, Liszt/Dante are described by McTagggart’s extrinsic determination. Good example is the fifth/fourth tremolando at the beginning of Beethoven’s Ninth symphony, its Klang field, Moi1, and its borrowings at the beginnings of Bruckner’s III, Wagner’s Flying Dutchman and Sibelius’s En Saga. By this theory, one may finally explain the whole citation technique of music, references and evocations which Zofia Lissa once studied. And at the end, is the ekphrasis phenomenon, after all, nothing but determining correspondence among different art forms and in the intertextual field? Yet, there is also intrinsic correspondence among the qualities of modes (Fig. 2.25). or if zemic A has a quality M1a then the zemic B must also have it M1a’ However, what about the determination inside zemic, difference between modes? Do they determine each other? A certain body implies a certain person and it implies a certain profession and it again a certain value. Or the quality of each substance is determined by all other qualities in modes inside that substance or zemic. Or it is

54

E. Tarasti

Fig. 2.24 Zemic correspondence in music history

Fig. 2.25 Zemic correspondence among the modes

hence possible that modes determine each other in different order than that ordinary ‘organic’ model of growth and development from M1 to M2 and to S1. In other words, M2 influences M1 and S2 influences S1 or M1 directly to S1 etc. In any case, if we have the zemics A,B and C and they share same value or S1:s then the determination takes place in this framework. This explains the social i.e. that A, B and C are in principle different substances, different zemic subjects but they hold together via that principle whereby we so to say smuggle into one zemic the influences or effects from another. In music, a whole musical culture is explained by this—or this specifies Asafiev’s idea of intonation culture a.o.

2 Existential Semiotics and Its Application to Music …

55

Fig. 2.26 Beethoven Op. 111

I want to conclude this essay by a music example which crystallizes many essential things in zemic analysis. The end of Beethoven’s sonata op. 111. As such the theme of this variation movement already illustrates impressively the theory of Jules Gentil of the breathing of the hand and music, pousser/tirer. Variation form is as such one wherein M2 is kept the same whereas M1 changes all the time i.e. the theme falls in different textures and enlightening; sometimes even S2 joins i.e. some topic serves as a variation (for instance, in Brahms’s Händel variations, there is Hungarian topic, siciliano topic, and reference to baroque, not to mention the finale which is a fugue or displays the learned style—which at the end, however, is in fact against the strict rules of fugue and ends at melodious style due to the brilliance of M2 or the theme itself and its intervallic turn; yet, Händel did the same himself, see suite F minor, second movement, the end of the fugue!). Nevertheless, in this end of Beethoven sonata, the parts are detached from each other either extremely low—embodiment—or extremely high—sublimation, which also as special gestures evoke tranascendence and transdescendence. What is involved is also a dialogue between the motifs, the theme of the beginning M2 has turned around by its intervals, descending fourth into raising. Teleologically, the work aims forward but, at the same time, retardates the coda until its ultimate according to the log-zemic of resistance. Sonata is now ready to close with this breathing of universe if one can say so, when the music really assumes a metaphysical meaning (Fig. 2.26).

References Boissier, Caroline Butini. 1997. Liszt Maestro di piano. Palermo: Sellerio Editore. Herman. A.L. 1976. An Introduction to Indian Thought. New Jersey: Prentice Hall. Klostermaier, Klaus K. 1994. A Survey of Hinduism. New York: State University of New York Press. McTaggart, John. 1988. The Nature of Existence I–II. Cambridge: Cambridge University Press. Millington, Barry 1992. The Wagner Compendium. New York: Schirmer Books.

56

E. Tarasti

Pankhurst, Tom. 2008. Schenkerguide: A Brief Handbook and Website for Schenkerian Analysis. New York: Routledge. Soulez, Antonia. 2012. Au fil du motif: Autour de Wittgenstein et la musique. Paris: Editions Delatour France. Tarasti, Eero. 1994. A Theory of Musical Semiotics. Bloomington: Indiana University Press. Tarasti, Eero. 2000. Existential Semiotics. Bloomington: Indiana University Press. Tarasti, Eero. 2012. Semiotics of Classical Music. How Mozart, Brahms and Wagner Talk To Us. Semiotics, Communication and Cognition. Edited by Paul Cobley and Kalevi Kull. Berlin: Mouton de Gruyter. Tarasti, Eero. 2015. Sein und Schein. Explorations in Existential Semiotics. Berlin: Mouton de Gruyter. Wagner, Richard. 1975. Das Braune Buch. Tagebuchaufzeichnungen 1865 bis 1882. Zürich: Atlantis Verlag. Wittgenstein, Ludwig. 1980. Remarks on the Philosophy of Psychology, vol. 1. Edited by G.E.M. Anscombe, G.H. von Wright. Translated by G.E.M. Anscombe. Oxford: Basil Plackwell. Wittgenstein, Ludwig. 2008. Philosophical Investigations. Translated by G.E.M. Anscombe. Oxford: Blackwell Publishing. Wittgenstein, Ludwig. 2016. Lectures. Cambridge 1930–1933. From the Notes of G.E. Moore. Edited by David G. Stern, Brian Rogers, and Gabriel Citron. Cambridge: Cambridge University Press. Wright, von, Georg Henrik. 1963. Norm and Action. A Logical Enquiry. London: Routledge and Kegan Paul.

Eero Tarasti is a professor of Musicology at the University of Helsinki and was chair from 1984 to 2017. As President of the International Association for Semiotic Studies from 2004 to 2014, he remains its Honorary President. In 2016 he founded the Academy of Cultural Heritages. He has studied music at the Sibelius Academy, Helsinki, and in Vienna, Paris, Rio de Janeiro, and Bloomington and earned his Ph.D. from Helsinki University (1978) after studying in Paris with Claude Lévi-Strauss and A.J. Greimas. He is a co-founder and current director of the international research group Musical Signification since 1984. Tarasti has become Honorary Doctor at Estonian Music Academy, New Bulgarian University (Sofia), Indiana University (Bloomington), University of Aix-Marseille and Georghe Dima Music Academy in Cluj-Napoca, Romania. He has published approximately 400 articles, edited 50 anthologies, and written 30 monographs, including Myth and Music (1979), A Theory of Musical Semiotics (1994), Heitor Villa-Lobos (1996), Existential Semiotics (2000), Signs of Music (2003), Fondéments de la sémiotique existentielle (2009), Fondamenti di semiotica esistenziale (2010), Semiotics of Classical Music (2012, in French 2016), and Sein und Schein, Explorations in Existential Semiotics (2015); two novels: Le secret du professeur Amfortas ( 2002) and Retour à la Villa Nevski (2014, in Italian L’heredità di Villa Nevski, 2014 in Finnish Eurooppa/Ehkä 2017). He has supervised 150 Ph.D.s in Finland and abroad.

Chapter 3

The Qualities and Flow of Imagined Sound and Music Chris Chafe

Abstract A new lens on experience is proposed. It is a way of understanding how we produce rhythmic syncronization between each other. Our musical ability for fine ensemble precision is exquisite in the way it adapts, predicts and communicates. Studying this empirically takes at least two participants but understanding its mechanisms requires entering into the minds of individuals. There, in the realm of imagination, is where sound lives on a different kind of stage. Its churning temporal flows are laced with memory and anticipation, consuming the present and producing it, as well. Keywords Imaginary sound · Music · Phenomenology · Psychology of time

3.1 Sound in the Imagination When we imagine sounds and music our “tonal thoughts” have qualities like loudness or timbre which resemble the qualities which we ascribe to perceived sound. Or are these dimensions of imagined sound analogies, somehow borrowed, weaker, or imposed? Imagined sounds are not accessible for measurement like their counterpart physical phenomena. Qualities of perceived sound are graspable in part because we can link them to measurable attributes of external sounds. That which a microphone (or earlier pre-electronic apparatus) registers can be compared to that which we report via listening. The physical signal can be repeated, analyzed and described in physical dimensions. Links between physical properties of sound and related perceptual attributes are the product of centuries-long work in acoustics and psychoacoustics (Hui 2012). I’m certain that the greater fraction of my auditory experience is, in fact, internal. Modes of hearing and listening vary continually—perceived, imagined, unconscious, conscious, reflex-triggering, enacting—and it’s those which take place C. Chafe (B) Center for Computer Research and Acoustics, Stanford University, Stanford, USA e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 P. C. Chagas and J. Cecilia Wu (eds.), Sounds from Within: Phenomenology and Practice, Numanities - Arts and Humanities in Progress 18, https://doi.org/10.1007/978-3-030-72507-5_3

57

58

C. Chafe

on the proscenium of the imagination which are the least studied. Alexandra Hui found it “curious this skill was not of psychophysical interest” in the time of Helmholtz, a time of the “geistige” ear and the theory of unconscious inference and when “The listener’s ability to generate or at least reproduce sounds absent the stimulation of actual sound waves.” was certainly apparent at the time (Hui 2012). Most people are able to willfully imagine sounds. For example, to conjure the sound of a familiar musical instrument or to attend to their own inner voice while reading silently. Conscious manipulation of auditory imagery is also commonplace. Given phrases of text, participants can imagine them spoken by friends or relatives and will usually report that it’s easy to do. The first of two pilot studies described below gave me a sense of just how ubiquitous this capability is. Of 100 subjects, 98 reported they could imagine the sound of “a hard ball dropped from waist height onto pavement.” The Bucknell Auditory Imagery Survey (Halpern 2015) tests the vividness with which a participant can imagine sounds and change them. The survey presents a series of imagery tasks by textual description. Subjects are asked to imagine, for example “The sound of an all-children’s choir singing the first verse of a song” and then “An all-adults’ choir now sings the second verse of the song.” Why does this matter to me as someone mostly working in computer music? The answer begins with an entirely different line of inquiry. Using the Internet for highquality “teleconcerts” and rehearsing Fig. 3.1 has been a research project which I’ve been involved with for two decades (Chafe 2000). The work has become increasingly important during the present COVID-19 pandemic. There is great interest on the part of musicians worldwide who are directly impacted by the impossibility of gathering in person for group music making. As soon as network music performance became technically feasible and first experiences showed promise for rhythmic syncronization, we tested pairs of subjects clapping together to learn about the effect of very short time delays on their ensemble tempo coordination (Chafe 2004, 2010). Questions of interest in these studies included, “What is the latency limit (in msec) beyond which maintaining a shared pulse becomes difficult?” And, “If tempo degrades with increasing lag, are there naturally-occuring coping mechanisms which arise in ensemble playing?” The range of delays tested covered a large portion of the range experienced in network music performance situations (from 1ms to 78ms one-way). Our early study, which was run between adjacent studios in our center, featured the simple clapping pattern shown in Fig. 3.2. The experiment was conducted using headphones with very little ambient reverberation and only a sparse set of starting tempi. Two of the recorded trials can be heard in these sound examples, Fig. 3.3 and Fig. 3.4. You can hear the sounds by using a mobile phone with a barcode reader and aiming the camera at the each figure. Open the link it provides in a browser. Any modern browser should be able to play the sounds. 1 The two recordings illustrate the effect of manipulating temporal separation when a pair of clappers are separated with minimal latency ( 15 ms) and very long latency ( 78 ms). 1I

am indebted to Diana Deutsch for introducing this practical way to include sound examples in a book.

3 The Qualities and Flow of Imagined Sound and Music

59

Fig. 3.1 A split ensemble with members playing together located in California, Michigan, New York, Belgium and Germany Fig. 3.2 The test rhythm. Pairs of people clapping the rhythm together were situated in separate rooms (Chafe 2010)

Those results led to modeling work in which two interacting “clapper algorithms” interact to approximate the same behavior. Perhaps it’s no surprise that the early clapping models (Gurevich 2004; Caceres 2013) under-performed relative to our clapping humans in terms of tempo stability as shown in Fig. 3.5. Our ability to synchronize and to adapt our music making to different conditions is superb. We live in and participate in event flows and the intriguing challenge now is to tease apart the how’s and why’s of this very human skill. Applying the latest in synchronization theory and innovating in the mathematical description of coupled synchronous processes has increased the competence of modeling algorithms (Roman 2019) but performers cope with these short time lags in ways we have yet to understand. This is where the interest in auditory imagery ties in. Can we experiment directly with rhythmic flows at the level of protention and retention? These are names for the mental near-time in which events are planned to happen and have just happened.

60

C. Chafe

Fig. 3.3 Recordings of three trials of the clapping rhythm shown in Fig. 3.2 when participants were 15 ms apart. You can hear the sounds by using a mobile phone with a barcode reader and aiming the camera at the figure. Open the link it provides in a browser. Any modern browser should be able to play the sound

Fig. 3.4 Recording of three trials of clapping rhythm shown in Fig. 3.2 when participants were 78 ms apart

3.2 The Specious Present As listeners and performers, we are good examples of creatures who live in the “specious present.” The ways in which we think ahead in time, for example reading a bar ahead while performing from a score or anticipating upcoming chord changes in a tune or vaulting through the arch of a phrase we’re building, these are all examples of planning and thinking in tones. William James (following Kelly, and followed by Husserl and others) contributed descriptions of temporal flow in the mind. The practically cognized present is no knife-edge, but a saddle-back, with a certain breadth of its own on which we sit perched, and from which we look in two directions into time.

3 The Qualities and Flow of Imagined Sound and Music

61

Fig. 3.5 All trials of all pairs of people clapping the rhythm together shown above in Fig. 3.2. Averages of tempo deacceleration vs. temporal separation (black) with linear regression (green). The same task performed by a computer model with human syncronization traits (red) (Caceres 2013)

The unit of composition of our perception of time is a duration, with a bow and a stern, as it were” a rearward- and a forward-looking end (James 1890)

It seems likely that some of the difference between a mechanistic model using two interacting clapper algorithms and two human clappers will reside in how such flows are engaged in performance. In summarizing his modeling attempt, Juan-Pablo Caceres wrote: The model presented performs better than previous attempts, but humans are still better at this task. Second-order adaptation in a finer inter-beat granularity that are not accounted in the presented model may explain this. Our model includes an anticipation parameter that is static, and a better understanding of this prediction mechanism is a desirable goal to improve the performance of the model. (Caceres 2013)

The next step in creating a lens for examining attributes of human clappers’ performance is to combine our increasingly sophisticated models with humans. Improvements in modeling are bringing us closer to that goal. In his work incorporating “strong anticipation” Iran Roman found that there may be a simpler approach.

62

C. Chafe We have demonstrated that one can use a dynamical system with delayed feedback to simulate human anticipation during perception-action coordination. We only needed to add delayed feedback to an oscillator in order to explain human anticipation. A more complex mechanism of anticipation, like statistical inference, was not necessary (Roman 2019) 2

If we’re nearly there and the algorithm/human subject combination can provide a means for rapid experimentation then we can broaden the experimental conditions. The ability to test, for example, a wider set of musical behaviors and different acoustical conditions, and then evaluate the results more rapidly is key to learning more about possible factors involved. The present chapter presents preliminary aspects and aims at gaps which I see in our understanding of “thinking in sound and music.” What is particularly relevant to performance is the our lack of understanding of “planning sounds” internally. The present discussion now moves from this narrower starting point of modeling to ask more widely about the content of auditory imagery. Temporally-related objects, but what objects? The arguments to be made will set the stage to move full circle back to modeling if, as seems likely, simple structures emerge as they have in related fields. Temporal “chunks” of sound have explanatory value in linguistic discourse (Chafe 1994, 2018). Auditory stream formation of perceived sound has well-known importance in parsing complex sound worlds (Bregman 1990). “Socially-endowed internal models” have been proposed in which the performer internally simulates their co-performers’ immediate tendencies (Wolpert 2003; Keller 2008, 2012). Marc Leman suggests structures related to “imagery and object concepts” in his discussion of sensorimotor prediction (Leman 2016). Brain studies using imaging techniques (Halpern 2004; Herholz 2012) and EEG observations (Schaefer 2011, 2013) are advancing correlations with auditory imagery. If we call that a kind of “outside-in” experimentation, what I propose here is on a parallel front, from the inside-out. It involves gathering introspective evidence (Casey 2000; Ihde 2007) to describe aspects of temporal flow (Schmicking 2005; Varela 1999) and tonal flows (Hodges 2011; Keller 2006). The following pilot studies demonstrate how self-reports can provide such evidence with the advantage that today such surveys can reach an internet-enabled, scaled-up population of “imagers.” Eventually, these same techniques can be used with musician subjects. Talks which I’ve given on the subject have titles which develope the flavor of the approach: “The Acoustics of Imagined Sound” and “The Sound Stage of the Mind: Imagined Sounds and Inner Voices, Probing inner sound via crowd-sourced self reports.” The seeds are in the ground as far as motivation and proof-of-concept and what remains now is to adapt the techniques to questions related to timescape and flow.

2 “Strong

anticipation” is explained by Roman. “Specifically, ‘anticipatory synchronization’ in strong anticipation emerges from the coupling between a ‘response’ system and a ‘driver’ system (e.g., stimulus input) wherein the response system also receives delayed feedback about its own activity. One of the major strengths of the strong anticipation approach is that it accounts for anticipatory phenomena beyond human behavior, and that collectively all such phenomena can be modeled as coupled dynamical systems.”

3 The Qualities and Flow of Imagined Sound and Music

63

3.2.1 Pilot Studies (2013-14) The pilot studies asked questions aimed at describing qualities of mainly static mental objects. Study 1 surveyed 100 online “workers” for clarity ratings of their own inner voice, friends’ voices, musical instruments and environmental sounds. The rating scale was adopted from the Vividness of Visual Imagery Questionnaire (VVIQ Scale) (Marks 1973). As an example, Fig. 3.6 shows the results for 100 subjects rating the clarity of the imagined sound of “a hard ball dropped from waist height onto pavement.” In addition to gathering ratings of clarity of certain imagined sounds, the survey probed imaginary acoustical dimensions of location and relative loudness. Figure 3.7 shows the results when subjects were asked to silently read the words “mechanical turk” and judge whether the sound of their inner voice was located inside or outside their head. Figure 3.8 shows which imagined sound is louder when comparing a few suggested sounds. These examples were only intended to test the efficacy of crowd-sourcing judgements about imaginary sound made through self-reports. More methodical manipulations followed in the next experiment. Study 2 (50 subjects) involved listening to sounds as well as imagining them. It began by introducing a set of novel synthesized sounds played from the browser and then experimented with imaginary recall of those sounds later in the survey. You can hear the sounds using the same technique as above. As will be heard, the set of sounds increases in loudness Fig. 3.9. They were generated with a novel physical model which synthesized something resembling a bowed metal plate and their intensities were calibrated to increase in 6 dB steps. Subjects were presented

Fig. 3.6 Pilot study 1: Vividness self-reports of 100 subjects rating the clarity of the imagined sound when imagining the sound of a hard ball dropped from waist height onto pavement

64

C. Chafe

Fig. 3.7 Pilot study 1: Self-reports of 100 subjects judging whether the sound of their inner voice was located inside or outside their head when subjects asked to silently read the words “mechanical turk.”

Fig. 3.8 Pilot study 1: Self-reports of 100 subjects judging which of three imagined sound was loudest (their inner voice, static, or a bee buzzing)

with them in random order and were asked to rank whether they were louder or softer than a recorded voice. The unfamiliar sound and the voice probe could be repeated as needed by pushing buttons on the browser interface. Later in the survey, subjects were asked to recall the voice and recall the unfamiliar sounds across their range from softest to loudest. The recalled sounds were again compared for loudness and these answers provided rankings which allowed for comparison of the two sets, namely perceived and imagined. Two loudness curves were produced which are shown superimposed in Fig. 3.10. The imaginary one, which is the one of interest, appears to be a compressed version of the perceived one. This finding is consistent with the theory of imaginary extension (Casey 2000). Briefly

3 The Qualities and Flow of Imagined Sound and Music

65

Fig. 3.9 Pilot Study2: The five synthesized sounds which subjects played from a browser for experimentation with later imaginary recall during the survey. You can hear the sounds using the same technique as above. As will be heard, the set of sounds increases in loudness

Fig. 3.10 Pilot Study2: Compared to the perceived scale (blue), the imaginary quasi-loudness scale (red) is compressed and shifted

stated, it’s the phenomenon that imagined horizons are limited in their extent and that the distances between in which imaginary scenes are constructed have a relatively smaller scale than the same thing perceived. In terms of loudness, imaginary soft is not as soft as perceived soft and loud is not as loud.

66

C. Chafe

3.3 About Imaginary Loudness Daniel Schmicking poses the question “Is there imaginary loudness?” in an essay on phenomenological method (Schmicking 2005). The article has merit for its critique of approaches and because its conclusions suggest openings for further investigation. He proposes that “agreement procedures” could be developed to confront the former. Introspective evidence is difficult to validate, otherwise. “Cooperating phenomenologists then should be able to decide whether there is always quasi-loudness in auditory imagery, even if they could not claim apodicticity.” He makes a strong assertion that there is indeed imaginary loudness (quasi-loudness). If you doubt the quasi-loudness of your own inner speech or voice just ask yourself: does the voice whisper or does it roar? Does it sound like calling from a great distance, from the room next door, or rather from within your head? You are very likely to (implicitly) imagine accents along with the melody. This is the reason why the accentuated notes are quasi-louder; otherwise your melody, lacking accents, will be rhythmically ambiguous.

These examples ultimately require greater precision about loudness itself. From our perceived impressions of the physical world we know that confounding cues like timbre and attack, and underlying concomitants like force need to be accounted for. Impressions are not unidimensional and illusions abound. Can the whisper itself can be made louder? If it can, what’s the"quasi" mechanism producing the change? Does it get louder by whispering more (quasi-)forcefully or by (quasi-)amplification of sound pressure level? Many other ways exist to make a whisper louder, such as cupping your hands or moving closer. Informally, in discussions of self-reports of imaginary loudness, I’ve noted a prevalence for force being strongly coupled with impressions of loudness and less so that sound pressure level is coupled with imaginary loudness. That’s because it seems to be more difficult to perform strictly intensity-based manipulation in your mind. It’s apparently easier to imagine hitting an object harder than to imagine the sound of a hit emitted via a device with a variable volume control. But some subjects who report they can accomplish this often report that the task evokes an image of a volume knob on an amplifier or volume slider in a sound playback application. Imaginary context helps. As Schmicking himself points out, it’s impossible to know whether one is instead imagining a difference based on distance. But in like manner an audio-imaginary (quasi-auditory) scene is related to the implicit imaginary body. When this perspectival organization of perception is transmitted to imagery then there might as well be a quasi-distance of imagined sounds and hence a correlative quasi-loudness. The quasi-loudness of imaginary events varies with imagined changes of quasi-distance.

Imaginary accent as a proof of the existence of imaginary loudness also requires closer examination since accent can depend on relative pitch height, timbre, attack and timing. My first pilot study (which compared only static mental objects, not melodies) found (quasi-)timbral manipulation was easily accomplished across a large number of subjects.

3 The Qualities and Flow of Imagined Sound and Music

67

In conclusion, Schmicking leaves the question open. Perhaps this might still be contested since it may be arguable that not every subject is able to focus on imagined loudness; or maybe there ‘is’ loudness of imaginary sounds only when one’s attention is shifted to it by external factors (e.g., an experimenter’s instructions), or there may even be subjects who cannot imagine loudness at all.

3.4 Constructing the New Lens Schmicking’s challenge here is apt and the pilot studies were an initial attempt to use crowd-sourced surveys to gain statistical significance. The results were encouraging enough that I can propose extending the technique toward questions of temporal flow. I intend to further engage crowd-sourcing services (like Amazon’s Mechanical Turk which was used in the pilot studies) in order to reach large numbers of dedicated survey takers. The goal is to test “phenomenological agreement” of a statistical kind with little constraint on the size of subject pool. In these studies as in the pilot experiments, browser-based presentation of sounds and music, and inner voice will figure in the designs. One possibility is to engage mental counting as a way of accessing and “time-stamping” events and objects in the “near now.” These are some experimental design criteria for going forward: 1. 2. 3. 4.

accounting for subject variability, eliminating experimenter bias, methodically entering into the realm of the “specious present” and enhancing the validity of online surveys.

There are prior studies that inform strategies for each. 1. The BAIS will be useful for characterizing participants’ individual aptitudes. 2. Gelding, et al. have devised an experiment for isolating imaginary pitch with a novel degree of purity (Gelding 2015). 3. Pecenka (2013), Keller (2006) and others have shown a relationship between musician’s auditory imagery abilities, temporal prediction and sensorimotor synchronization. In one case, participants were asked to to mentally continue a tempo change across a short auditory sequence with a gap, and then to judge whether a probe tone occurred early or late relative to the imagined continuation. 4. Validity of online surveys has been a concern in the social sciences and methods for reinforcing their significance are available to be put into practice (Berinsky 2014). The designs will attempt to characterize mental sound objects, object “chunking” and temporal flow with a particular goal of examining the near-time “retention / protention” qualities described by Husserl. Husserl’s attempts to diagram this mental timescape are fascinating, Fig. 3.11 and Fig. 3.12 (Dodd 2005). They depict event objects in procession as they recede from now into the near-past and as they set up

68

C. Chafe

Fig. 3.11 Husserl’s time chart shows it as an assemblage of experienced objects

Fig. 3.12 Husserl’s representation of objects with independent dimensions

expectations in the near-future. If one were to diagram echoic memory in which the persisitence of the sound of events gives way from veridical to schematic in about a half-dozen seconds, the result might resemble these sketches of Husserl’s. The physical timescape of musical syncronization at the micro-time level was studied in our work with pairs of subjects clapping together (Chafe 2010). A prototype system has been tested in which an adaptive and listening rhythm tapping algorithm is coupled with a human tapping on a keyboard, Fig. 3.13. Leading and lagging in very small proportions is visible. Keeping a steady pulse is a dynamic process with alternating give and take. The braided appearance is typical of duos being analyzed at this level of temporal resolution. Combining the ideas present in Husserl’s timescapes with micro-time measurements will give us a new lens on imaginary flows of sound and music.

3 The Qualities and Flow of Imagined Sound and Music

69

Fig. 3.13 Automatic tapping algorithm (red) and human tapper (blue). The inter-onset interval time series is plotted as a function of tempo (above), time of occurence (middle) and histogram (below)

3.5 Aspects of the Proposed Research Imaginary sound as a field of research is expanding. Scientific and artistic projects are motivated by a curiosity to understand more about this hidden realm and manipulate its qualities. It’s clearly a central element of human experience. For my part, my interest begins with my seemingly tangential research and practice involving online music performance. It was there that I discovered behaviors and paradoxes that led me to consider related studies of time consciousness. Three components make up the further study proposed by the present contribution: motivation, literature and method. The first is to give an account of the fundamental inverse relationship between increasing temporal separation and tempo. Surprisingly, our data showed that there was a range of extremely short delays with tempo acceleration. The finding supported the notion of a natural delay range for best tempo stability on the order of delays in familiar acousical settings. The “sweet spot” is similar to the propogation delays in air between musicians in rooms and on stages. For tight rhythmic syncronization

70

C. Chafe

in network music performance it’s especially important to replicate this range in engineering the online systems. What of these accelerating flows? These are manifest in a context requiring two clappers. Are human-human coupled flows one flow or two? Will algorithm-human coupled flows be a way into manipulating that? Methodically observing qualities of such flows with self-reports has not yet been tried. Further study will include more deeply tracing the topic historically across domains. Literature on temporality spans philosophy, linguistics, psychophysics and music. The data include everything from solo arm-chair introspection through how temporal concepts are expressed in the world’s languages to neuroimaging-based approaches. On the music side, there are a growing number of composers who manipulate the “sound thoughts” of their audience. Their compositions ask their listener participants to actively engage in auditory imagination. For example, the works of Pauline Oliveros, Vanessa Tomlinson and Amnon Wolman literally are played on the internal proscenium and are not static scenes. How do they engage temporal flow? There may be methods in these works which can inform experimental designs. Lastly, I suggest that further studies take a cue from Schmicking (above) and examine quantitative methods in phenomenology. A useful product will be the establishment of a protocol which can be shared across experiments. Reproducibility is essential, and providing a practical “howto” for crowd-sourced methods will allow replication of the findings. Acknowledgements With thanks to Juan Pablo Caceres, ´ Hongchan Choi, Rob Hamilton, John Granzow, June Holtz, Pauline Oliveros, Andrea Halpern, David Huron, Chryssie Nanou, Jonathan Berger, Jieun Oh, Andy Stuhl, my Music 220 classes and many others who have contributed to discussions of imaginary sound, network music performance and techniques for crowd-sourced studies.

Appendix As a part of both of the two pilot studies, survey-takers were asked for their own observations about the subject of imaginary sound. Their answers were revealing and a sampling of them is included here. Some encountered their inner voice for the first time, others experienced difficulties. The term “hit” refers to a task which Amazon Turk workers choose to complete. 1. This was a very interesting hit. It is one that really makes you listen and concentrate, thanks. 2. Is there any way to find out what this study is for? This was interesting to do. 3. This was an interesting task. I’ve never been asked to do anything quite like it before. This was fun to complete. 4. I found it interesting to not be able to imagine my voice coming from a spot outside myself or any voice saying mechanical turk including that of my mother. I could hear her voice in my mind but could not imagine or hear her saying mechanical turk. Odd and I wonder why that is.

3 The Qualities and Flow of Imagined Sound and Music

71

5. very interesting study... a subject I’ve never thought about but makes complete sense. 6. Definitely a different type survey. It was fun. Thanks for being creative! 7. This was a weird but interesting hit. I had never thought about this aspect of hearing or inner voices. I will work with it a bit and see what happens. I realized while doing it than depending on where my focus is, I can hear the sound/voice in different manners- in my head- my voice or a different voice, outside my head like someone else is speaking, in my head like someone else is speaking, in my head with the voice I usually hear speaking to me- whatever I think I can make physically happen. 8. As I began the task, I was able to imagine hearing my voice. Then as the task went on, it became more difficult to hear anything, because there was no sound. 9. This was interesting. I have very good hearing and often hear things other people don’t. However, I wasn’t very good at imagining a friend’s voice or a sound coming from outside my head. 10. This was an unusual but very interesting study. It is almost like separating body and soul.

References Berinsky, Adam J., Michele F. Margolis, and Michael W. Sances. 2014. Separating the shirkers from the workers?. Making sure respondents pay attention on self-administered surveys. American Journal of Political Science. https://doi.org/10.1111/ajps.12081. Bregman, Albert. 1990. Auditory Scene Analysis. Cambridge, MA: MIT Press. Caceres, ´ Juan-Pablo. 2013. Synchronization in Rhythmic Performance with Delay. Stanford, CA: thesis. Casey, Edward. 2000. Imagining: A Phenomenological Study. Bloomington, IN: Indiana University Press. Chafe, Chris, Scott Wilson, Randal Leistikow, Dave Chisholm and Gary Scavone. 2000. A simplified approach to high quality music and sound over IP. COST-G6 Conference on Digital Audio Effects. Available via DAFX Conference Web Page. https://www.dafx.de/paperarchive/2000/pdf/chafe.pdf. Cited 19 Sep 2020 Chafe, Chris, Michael Gurevich, Grace Leslie and Sean Tyan. 2004. Effect of time delay on ensemble accuracy. Proc. of the International Symposium on Musical Acoustics 31 Available via CCRMA Stanford University. https://ccrma.stanford.edu/ cc/pub/pdf/ensAcc.pdf. Cited 19 Sep 2020 Chafe, Chris, and Juan-Pablo Caceres, ´ and Michael Gurevich. 2010. Effect of temporal separation on synchronization in rhythmic performance. Perception. https://doi.org/10.1068/p6465. Chafe, Wallace. 1994. Discourse, Consciousness and Time. Chicago: University of Chicago Press. Chafe, Wallace. 2018. Thought-Based Linguistics: How Languages Turn Thoughts into Sounds. Cambridge: Cambridge University Press. Dodd, James. 2005. Reading Husserl’s time-diagrams from 1917/18. Husserl Studies. https://doi. org/10.1007/s10743-005-6403-2. Gelding, Rebecca W., William F. Thompson, and Blake W. Johnson. 2015. The pitch imagery arrow task: effects of musical training, vividness, and mental control. PLoS One. https://doi.org/ 10.1371/journal.pone.0121809.

72

C. Chafe

Gurevich, Michael, Chris Chafe, Grace Leslie and Sean Tyan. 2004. Simulation of networked ensemble performance with varying time delays: Characterization of ensemble accuracy. Available via CCRMA Stanford University. https://ccrma.stanford.edu/ cc/pub/pdf/simNetEnsPerf.pdf. Cited 21 Sep 2020 Halpern, Andrea. 2015. Differences in auditory imagery self-report predict neural and behavioral outcomes. Psychomusicology: Music, Mind, and Brain. https://doi.org/10.1037/pmu0000081. Halpern, Andrea R., Robert J. Zatorre, Marc Bouffard, and Jennifer A. Johnson. 2004. Behavioral and neural correlates of perceived and imagined musical timbre. Neuropsychologia. https://doi. org/10.1016/j.neuropsychologia.2003.12.017. Herholz, Sibylle C., Andrea R. Halpern, and Robert J. Zatorre. 2012. Neuronal correlates of perception, imagery, and memory for familiar tunes. Journal of Cognitive Neuroscience. https://doi. org/10.1162/jocn_a_00216. Hodges, Donald A., and David C. Sebald. 2011. Music in the Human Experience: An Introduction to Music Psychology. New York: Routledge. Hui, Alexandra. 2012. The Psychophysical Ear: Musical Experiments, Experimental Sounds, 1840– 1910. Cambridge, MA: MIT Press. Husserl, Edmund. 1928. Vorlesungen zur Pha¨ nomenologie des inneren Zeitbewusstseins. Tubingen: ¨ Max Niemeyer Verlag. Ihde, Don. 2007. Listening and Voice: Phenomenologies of Sound. Albany: State University of New York Press. James, William. 1890. The Principles of Pschology. New York: Holt. Keller, Peter. 2008. Joint action in music performance. In Enacting Intersubjectivity: A Cognitive and Social Perspective to the Study of Interactions, ed. Francesca Morganti, Antonella Carassa, and Giuseppe Riva, 205–221. Amsterdam: IOS Press. Keller, Peter E. 2012. Mental imagery in music performance: underlying mechanisms and potential benefits. Annals of the New York Academy of Sciences. https://doi.org/10.1111/j.1749-6632.2011. 06439.x. Keller, Peter, and Iring Koch. 2006. The planning and execution of short auditory sequences. Psychonomic Bulletin & Review. https://doi.org/10.3758/BF03193985. Leman, Marc. 2016. The Expressive Moment: How Interaction (with music) Shapes Human Empowerment. Cambridge, MA: MIT Press. Marks, D.F. 1973. Visual imagery differences and eye movements in the recall of pictures. Perception & Psychophysics. https://doi.org/10.3758/BF03211175. Percenka, Nadine, Annerose Engel and Peter E. Keller. 2013. Neural correlates of auditory temporal predictions during sensorimotor synchronization. Frontiers in Human Neuroscience. https://doi. org/10.3389/fnhum.2013.00380 Roman, Iran R., Auriel Washburn, Edward W. Large, Chris Chafe, and Takako Fujioka. 2019. Delayed feedback embedded in perception-action coordination cycles results in anticipation behavior during synchronized rhythmic action: A Dynamical systems approach. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1007371. Ross, Stewart L. 1985. The effectiveness of mental practice in improving the performance of college trombonists. J. of Research in Music Education. https://doi.org/10.2307/3345249. Schaefer, Rebecca S., Rutger J. Vlek, and Peter Desain. 2011. Music perception and imagery in EEG: Alpha band effects of task and stimulus. International J. of Psychophysiology. https://doi. org/10.1016/j.ijpsycho.2011.09.007. Schaefer, Rebecca S., Peter Desain, and Jason Farquhar. 2013. Shared processing of perception and imagery of music in decomposed EEG. NeuroImage. https://doi.org/10.1016/j.neuroimage. 2012.12.064. Schmicking, Daniel. 2005. Is there imaginary loudness?. Reconsidering phenomenological method: Phenomenology and the Cognitive Sciences. https://doi.org/10.1007/s11097-005-7597-7.

3 The Qualities and Flow of Imagined Sound and Music

73

Varela, Francisco. 1999. The specious present: The neurophenomenology of time consciousness. In Naturalizing phenomenology, eds. Jean Petitot, Francisco. J. Varela, Bernard Pachoud and Jean-Michel Roy 266–314. Stanford, CA: Stanford University Press. Wolpert, Daniel M., Kenji Doya and Mitsuo Kawato. A unifying computational framework for motor control and social interaction. 2003. Philosophical Transactions of the Royal Society B. https://doi.org/10.1098/rstb.2002.1238

Chapter 4

Decoding Imagined Sound Lloyd May and Michael Casey

Abstract The experience of hearing sounds not present in the air around us has been explored by composers, artists, and researchers for decades. Understanding and communicating the experience of these imagined sounds is often at the core of these explorations. This chapter presents guidelines and best practices for designing frameworks to decode imagined sound. The guidelines explore various design considerations and provide a detailed taxonomy of imagined sound for additional clarity. The second-half of the chapter details the design and results of a contemporary example of the decoding of imagined sound through a brain imaging study where decoding involves machine learning of the pattern of brain activity associated with specific properties imagining of sound, such as pitch-class and timbre, and then estimating those properties from novel data. The discussion of the study outlines the successful decoding of pitch-class and timbral information from brain scans of trained musicians imagining individual musical notes.

4.1 Introduction Sound is a complex phenomenon that encompasses far more than vibrations in air. Imagined sounds, which do not occur as vibrations in air but solely in the minds of the listener, have been encountered and studied in many interconnected forms. Common examples include internal reading voices, auditory hallucinations, the sound worlds inhabiting a composer’s mind, and countless more. These imagined sounds are an indication of the incredible aspects of sound that do not exist as vibrations, but rather as neurological events. Throughout history, we have communicated these sounds through writing, compositions, visual art, and many other media. Understanding the physiological processes involved in the creation and experience of imagined sounds is of great interest to neuroscientists as they shed additional light on other L. May (B) · M. Casey Dartmouth College Hanover, Hanover, NH, USA e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 P. C. Chagas and J. Cecilia Wu (eds.), Sounds from Within: Phenomenology and Practice, Numanities - Arts and Humanities in Progress 18, https://doi.org/10.1007/978-3-030-72507-5_4

75

76

L. May and M. Casey

high-level cognitive processes such as idea inception and sensory perception. Decoding these sounds is positioned in the middle of efforts to understand the underlying neural mechanisms involved as well as communicate these inner sounds to others. In this chapter, we will propose a framework for developing methods of decoding imagined sound, followed by an introduction to a taxonomy developed to aid in describing specific characteristics of imagined sound experiences. After a review of recent literature in the field, we will discuss the process and findings of a recent study in which pitch and timbral information were successfully decoded from the brain images of participants actively imagining individual notes. This chapter is intended for a general audience and no specific knowledge regarding neuroscience or music is required.

4.1.1 What Is Imagined Sound? Our inner sound worlds are varied, brilliant, and banal. Certain psychoacoustic phenomena allude to the power and importance of the human mind and body in the detection, perception, and understanding of sound. Binaural beating is an example of such phenomena, where two pure tones of slightly different frequencies are presented individually and simultaneously to both ears of a listener wearing headphones and an additional frequency, the difference tone, which is not present in the sounds made by the headphones, can be heard by most listeners with stereo hearing abilities (Deutsch 2013). This is a clear indication that our perception of the sounds we hear can differ greatly from the sounds that vibrate our inner ear. The field of music cognition is full of examples of studies that show that context, expectation, and other factors impact our perception of sound. Psychologist Diana Deutsch has studied and cataloged many of these phenomena (Deutsch 2016). These experiences of sounds not present in the air around us are far more than simple idiosyncrasies and have deepened our understanding of fields as disparate as pitch perception and seizure suppression (Lin et al. 2014). They have also been used to great effect by artists and composers, such as Hans Zimmer’s use of the Shepard Tone, an auditory illusion where the pitch of a note seems to increase indefinitely, in his score for Dunkirk (Shepard 1964; Nolan 2017). Non-acoustic sounds have also been used in ways aside from compositional perceptual manipulation. Popular culture is rife with depictions of experiences of non-acoustic sound. From the closedmouth inner-monologues of film to thought bubbles in comic books, the idea that many humans, and possibly ginger cats, experience imagined sounds is not novel or controversial. However, one specific category of imagined sound seems to dominate conversation and coverage: psychosis. The killer who is compelled by inner voices. The indistinct chatter signaling the protagonist’s weakening mental state. These often problematic and cliché plot devices fetishize experiences of imagined sound to be a sign of societal deviance. This is exacerbated by potentially narrow understandings of the experiences of those with a mental health diagnosis. Unlike other experiences distorted by popular culture, like romance, beauty, or success, we are generally not

4 Decoding Imagined Sound

77

exposed to many other conversations or depictions of imagined sound. It seems that a few experiences of imagined sound, such as inner-reading voices, are societally allowable and most others are deemed worrying. This is not to undermine the experiences of those who do experience unwanted and personally troubling imagined sounds, but to allow these less common occurrences to dictate public perception of a natural phenomenon is potentially problematic. This stigma associated with certain experiences of imagined sound has produced additional difficulties to discussing and understanding these experiences. Therefore, a framework that both normalizes this potentially stigmatizing experience and provides sufficient levels of clarity and specificity is needed. The frame of science and technology, either fictional or real, often offers an alternative method to discuss issues distorted by popular culture. The concepts of devices and mechanisms that decode, communicate or steal imagined sounds and images are also not new to popular culture. Composer Edgard Varese even dreamt of an “instrument obedient to thought” which he could use to express the “exigencies of [his] inner rhythm,” and often attempted to express this through his music (Fisk et al. 1997). The film director Anne Sewitsky introduced the Temporal Interceptor, an imagined sound retrieval device used in Black Mirror: Rachel, Jack and Ashley Too (Sewitsky 2019). The device is used to steal songs from the mind of Ashley O, a pop-star held captive and exploited for capital gain. The device extracts fragments of melodies and lyrics from the anesthetized singer which are then censored and reconfigured into commercial pop songs. Another nuanced depiction of imagined sound in popular media can be found in the award-winning video game Disco Elysium (ZA 2019). In this game, Kurvitz turned imagined sounds and internal monologues into a skill-tree and morality meter in the Thought Cabinet. These more detailed depictions of vivid imagined sound experiences, largely divorced from the stigma of a psychosis label, were possible because of the lens of science and technology. Because these experiences could be communicated clearly and legibly to another person, they were able to be explored with less doubt of the mental stability of the person experiencing the sound. By decoding the experience through a clear and legible framework, imagined sound can be explored with increased reliability, safety, and participant comfort. The lens of technology, an external force which measures and normalizes, could provide a more nuanced and inviting discussion of the experience of imagined sound for those who might be uncomfortable discussing it in other frames. Experiences of imagined sound are some of the most personal encounters with sound we have. The formal study of these sounds is relatively new, but it is growing quite rapidly. These sounds have a great deal of untapped potential to deepen our understanding of what sound is and how it functions as well as to create new experiences of sound. The practice of decoding imagined sound draws from neuroscience, experiment-based data science, as well as music theory, musicology, and creative practices. As trans-, inter-, and anti-disciplinary studies are growing increasingly popular, imagined sound also provides an exciting opportunity to include diverse perspectives throughout the process.

78

L. May and M. Casey

4.1.2 What Is Decoding Before analytical models are made and neural correlates are parsed, it is important to ask not only what the process of decoding entails, but what it hopes to achieve. Decoding can be thought of as a translation of representational space in which specific information is made more interpretable while other information is filtered out. While it may seem that decoding is the simple unlocking of certain hidden information, it is instead the product of a series of filtering decisions that rely on a strong conception of what information should be prioritized. The process of decoding asks technologists and designers to prioritize certain information and discard the remaining “noise” from the signal. The information loss in Morse code is an interesting example to illustrate this. Morse code is a system in which individual letters from the Roman alphabet are transmitted as a series of long and short sounds; most often electronic beeps (Gleick 2011). This system decodes a series of sounds and translates them into letter(s). In this system, the classification of the individual letters being transmitted is prioritized over all other information, such as the timbre or sound characteristics of the beeps or the subtleties of the rhythms used. While this may seem logical, even preferable, it is important to acknowledge that this was a design decision that impacted the scope in which the code would be used. In designing systems, products, games, or experiments, certain information will be inherently prioritized through its design. While this processing of information filtering and labeling of unwanted information may seem daunting, it is often an incredibly beneficial process as it forces the design to prioritize. However, if design goals and priorities are not established, the decoding system may discard information that may have been valuable. Therefore it is important that decoding frameworks are designed with the appropriate information prioritization and removal in mind. The design and execution of a decoding process not only impacts the quality of its results and how that technology will be used but fundamentally hinges on information prioritization and a notion of what aspects of music will be decoded. Therefore, briefly reviewing this design theory is incredibly important when attempting a decoding of something as complex as sound or music. The breadth of models that describe aspects of music, such as Western tonality, rhythmic swing, and even taxonomies for harsh noise, is an indication of music’s complexity. Some nuances that will inevitably be lost in any decoding procedure. This complexity motivates the need to distill the goals of decoding into definite, detailed statements as upon closer examination, the phrase “decoding music” can quickly become too vague to be meaningful. This vagueness also presents opportunities for the implementation of a variety of decoding techniques. Most imagined sound decoding techniques that one might think of are generally based on naturally occurring bio-signals within the human body, such as brain activity or heart rate. However, this does not have to be the case. This chapter focuses primarily on decoding sound from human brain activity; however a host of techniques, such as interviews, creative practices, comic book art, and more may be employed with varying degrees of technology incorporated to decode experiences of something as complex as imagined sound and/or music.

4 Decoding Imagined Sound

79

These various techniques do not have to exist as separate entities, but can instead be used to create a co-productive network of decoding models. For example, we conducted a variety of interviews with musicians, composers, and Foley artists to gain a deeper understanding of the phenomenological experience of various imagined sounds. The insights from these interviews aided the experiment design of the fMRI decoding experiment by allowing us to tweak the language of prompts to align more closely with the language people might naturally use to describe these phenomena. Interestingly, some of the people we interviewed, our interlocutors, were far more open about their experiences with imagined sound once the fMRI experiment was mentioned. This not only helped focus the interviews but upon post-interview reflection, some interlocutors mentioned that some additional neuroscientific knowledge around imagined sound helped them to feel comfortable discussing these experiences as it slightly normalized them. This illustrates one small instance where decoding processes of different modalities were used in a mutually beneficial framework. The interviews also informed the creation of a taxonomy of imagined sound characteristics. This taxonomy was designed to provide additional focus to conversations about imagined sound where more general terms with ambiguous interpretations might be used.

4.2 Background 4.2.1 What Is Imagined Sound? Imagination is a complex neurological and phenomenological concept that overlaps with studies of creativity, empathy, and other high-level cognitive function. While many models describe imagination, Coleridge’s relatively broad proposition is productive in framing the discussion and highlighting some possible pitfalls. Samuel Taylor Coleridge, a 19th century English poet and philosopher, drew a distinction between fancy and imagination (Willey 1947). Fancy, as Coleridge modeled, is the logical manner in which sensory material is organized that does not include the synthesizing of any new information or material. In this model, the basic semantic representations of raw sensory information that fancy are learned. Imagination was an expansion of fancy into spontaneous and original acts of creation where the raw sensory information provided by fancy was processed and abstracted into relatively novel concepts. Coleridge further distinguishes the primary imagination as “the living power and prime agent of all human perception,” from the secondary imagination, a superior conceptualization of the primary imagination and the foundation of artistic genius (Coleridge 1817). Coleridge links practices typically associated with Western high art, such as instrumental music and painting, as not only realizations of the secondary imagination, but the only ways in which it could be realized. This leads to the dubious conclusion that the secondary imagination can only be experienced and realized by those who are able to dedicate long periods of time to creative prac-

80

L. May and M. Casey

tices. While Coleridge’s stratification of imagination is problematic, the distinction between fancy and imagination highlights the impossibility of stratifying imagination into separate distinct concepts. Imagination is an expansive idea that captures the multi-modal experiences of what occurs in the human mind, and attempting to draw boundaries around each discrete component is, at this stage of neuroscientific understanding, impossible. Murray Hunter, a contemporary philosopher, instead proposes 8 overlapping types of imagination that we commonly experience (Hunter et al. 2013). Models that are example-driven and explicitly allow the for co-occurrence of its constituent parts seem most productive in attempting to adequately frame imagination. Hunter’s 8 types of imagination are: 1. Effectuative Imagination: The formation of a new concept created by synergizing previous information. (E.g. Imagining a new Pokémon that is a combination of existing creatures) 2. Constructive Imagination: The consideration of various hypotheses that are informed from multiple pieces of information. (E.g. Deciding which party to vote for in an election) 3. Imaginative Fantasy: The creation of fictitious objects such as stories, pictures, etc. (E.g. Writing a sci-fi comic book) 4. Empathy Imagination: The act of applying another’s frame of reference to oneself in an attempt to understand what another is experiencing emotionally. (E.g. Attempting to understand the pain of the loss of a friend’s loved one by imagining the hypothetical loss of a beloved family member) 5. Strategic Imagination: The construction of mental scenarios with the aim of modeling or forecasting. (E.g. Imagining what one’s college experience might be like as an engineering or a philosophy major) 6. Emotional Imagination: Similar to strategic imagination, but with a focus on the emotional component of the imagined scenarios. (E.g. Imagining the mix of emotions one might feel after separating with a romantic partner) 7. Dreams: The unconscious formation of sensory experiences that can occur during sleep. (E.g. A nightmare) 8. Memory Reconstruction: The process of actively reconstructing events or concepts from memory. (E.g. Imagining the face of a past elementary school teacher in great detail) Hunter’s model views imagination as a neurological tool that we draw upon in many ways as opposed to a fixed entity we experience. The lines between imagination, conscious thought, memory recall, and creative thought are blurry at best. Therefore, decoding procedures that avoid the trapping of hard stratification of imagination and other neurological processes would be most productive.

4 Decoding Imagined Sound

81

4.2.2 Imagination Versus the Real Another common trap in models of imagination is the hard distinction often made between perception and imagination, where imagination and perception are often described as opposed conditions rather than two states which mutually inform each other. One might assume that perception is the sensing and/or processing of a real event while imagination is reserved for phantasmal events. However, upon closer inspection, it becomes quite challenging to separate the two. There are many cases where one impacts the other to such a degree that it becomes impossible to parse the two. For example, the study of creative and deep listening practices, which often emphasizes attentive listening to external as well as internal sounds, shows how mental imagery can alter human perception (Dunn 1997; Oliveros 2005). Philosopher Bence Nanay posits that when we perceive (i.e. see/touch/hear/taste) an object, we construct and hold non-perceptual beliefs about the object as a mental image1 (Nanay 2010), such as when we see an orange, we often have a mental image of the entire fruit being roughly spherical even though we have not seen the backside of the fruit. Additionally, perception is ultimately involved in the process of training the human mind to imagine by giving it a database of experiences and phenomena to learn from. The two processes even recruit many of the same areas of the human brain in both visual and auditory contexts (Podgorny et al. 1978; Zatorre et al. 2005). Therefore, it quickly becomes impossible to draw distinctions between concepts as broad and vague as perception and imagination. It becomes apparent that precise and descriptive language is an essential tool in creating a productive framework for discussions involving the decoding of imagined information. For example, a productive distinction might be made between perception and volitional imagination, or mental images deliberately created by the person experiencing them. Using specific language allows for not only a greater degree of clarity in communication but may provide useful focus during the designing of a decoding procedure. Additionally, a taxonomy of terms that described a phenomenon’s immediate origin might be employed to meaningfully distinguish between the two.

4.2.3 Taxonomy of Imagined Sound The study of imagined sound intersects with many fields of study and the resulting terms used to describe the characteristics of imagined sound experiences are either closely tied to specific musical practices or quite general. This sub-section will introduce a taxonomy of sound that defines terms that generalize tendencies we observed throughout interviews with interlocutors about imagined sound. We attempted to use terms that carry little to no inherent value judgement of the sounds themselves, but rather focus on the experience of the sound. However, this taxonomy does not 1 Image

here refers to the mental representation of the object which can include multiple senses simultaneously and does not necessitate an imagined visual component.

82

L. May and M. Casey

Fig. 4.1 The sub-divisions of the taxonomy of experiences of sound

exist in a vacuum. Biases, especially those surrounding mental illness and psychosis, undoubtedly impact value judgements of certain imagined sound experiences. It is best practice to be aware of these biases and language that carries strong inherent value judgements when discussing imagined sound. Possible terms to omit from these conversations include: real, fake, made-up, delusion, authentic. Similarly, the ability to deliberately incite and control imagined sound occurs on a spectrum and is not indicative of other abilities. Therefore, the language used in the recruitment of participants or the discussion of imagined sound should reflect this (Fig. 4.1). The taxonomy introduced here will define acoustic sound, and the related concepts of bodily-auditory and cochlear-auditory, as well as imagined sound and several of its constituent characteristics. In this taxonomy, acoustic sounds are defined as vibrations that occur in air or other liquid media. These vibrations are then detected by a receiver, which may be mechanical in the case of a microphone, or organic, as in the case of the human body. This bodily reception of sound occurs when the human body, strictly excluding the cochlea (inner-ear), vibrates in response to an acoustic sound. The resultant vibration of the body can often be consciously detected, such as the thumping sensation one might feel in their chest while a bass-heavy dance song plays at a nightclub. This is most clearly felt with infra-sound and tactile vibration but occurs to varying degrees with all acoustic sound. We will refer to this mode of sound reception as bodily-auditory. Cochlear detection occurs when acoustic sound is transmitted through the eardrum and/or via bone conduction and stimulates the cilia of the cochlea. Artificial cochlear implants were invented in 1957 and, through electro-mechanical means, emulate the function of a traditional cochlea (Clark 2004). The phenomenological experience and context of hearing through a cochlear implant is potentially vastly different than hearing through a traditional cochlea. However, the physical mechanisms of traditional and artificial cochleas are similar in that complex vibrations in a medium are converted to electrical signals that are then sent to the brain and are experienced as sound. We will use the phrase cochlear-auditory to refer to this mode of sound reception. In people with traditionally functioning cochleas and nervous systems, these two modes of reception co-occur, to varying degrees, in response to acoustic sound. Imagined sounds are non-acoustic sounds perceived by a human that do not occur in a medium as pressure waves but only as electro-chemical signals in the brain and body. Whereas acoustic sound exists regardless of a human presence, imagined sound explicitly requires a human mind-body for it to exist. These sounds are not present

4 Decoding Imagined Sound

83

in any physical media outside of the human that is experiencing them. Therefore, imagined sound cannot be detected by acoustic sensors such as microphones and can only be detected through measurements made on the human, including the selfreporting of the sound. We propose three characteristics of imagined sound that all exist on a fluid spectrum that may be useful to describe experiences of imagined sound. Each characteristic has multiple features associated with it. However, a feature associated with all three characteristics is control: i.e. how easily that characteristic may be manipulated. These characteristics are not intended to be a complete model of imagined phenomena, but provide points of comparison and conversation about these often ethereal and personal experiences. These characteristics are: • Inception The event, or lack thereof, after which the imagined sound became perceivable. This is not the origin of the sound. For example, the inception characteristic of an imagined folk song is not when the song was first heard, but rather what event triggered the imagined sound currently being experienced, such as experiencing an event closely linked to the memory of the song or the deliberate imagining of the song. • Presence The perceived vividness or loudness of the imagined sound. This can be described in relation to the loudness of acoustic sound (quasi-loudness) and/or by the actions or events that significantly decrease the vividness of the sound. For example, one might imagine the sound of a waterfall that has an approximate quasiloudness of a whispering voice and whose vividness decreases severely when an acoustic sound of white noise is heard. • Quality This is an umbrella category that encompasses traditional characteristics of acoustic sound such as pitch, timbre, rhythm, etc. However, volume is strictly not included in this characteristic. A system of symbolic representation does not currently exist to discuss imagined phenomena without reference to acoustic descriptor. For example, the use of phrases like bright or boomy, to describe the spectral qualities of imagined sound. This could hinder the depth of description of imagined sound phenomena as these descriptors inherently reference acoustic phenomena. Developing a system of symbolic representation that navigates this issue is outside the scope of this chapter, but it is a notable limitation. As we have seen in the previous section, the study of the perception of acoustic sound has uncovered fairly common experiences of imagined sound that are caused by specific acoustic sounds. These include famous phenomena such as the Shepard tone, the octave illusion, and the tritone paradox (Shepard 1964; Deutsch 1981, 1991). All of these perceptional phenomena are not simply quirks of the human anatomy, but instead, an indication of the role imagined sound plays in human experiences of sound. Therefore, a more complete modeling of experiences of sound should include an acoustic as well as an imagined component wherever possible. The acoustic component of this model would encapsulate detectable acoustic sound as well as individual physiological differences, such as differences in vibrational sensitivity of the body, the frequency response of one’s cochlea, etc. The imagined component consists of all of the imagined sounds including perceptual phenomena,

84

L. May and M. Casey

deliberately imagined sounds, as well as imagined sounds with a spontaneous, nondeliberate inception. This model, like any framework, is not helpful for every inquiry and other taxonomies may be more productive in specific tasks or contexts. However, it does highlight and provide a structure to discuss an often-neglected component of the phenomenological experience of sound. This taxonomy could also be applied, with minor modifications, to other imagined sensing phenomena such as sight or taste.

4.3 Relevant Literature 4.3.1 Audiation and Correlates of Imagined Sound Audiation typically refers to the deliberate creation of an auditory image and has been used as a pedagogical tool by many music educators, most famously in the Gordon method (Gordon 1989). The most common form of audiation occurs as notational audiation where an auditory image (or imagined sound) is formed when reading a written musical score. Notational audiation is a phenomenon that has been confirmed to occur in a large percentage of surveyed expert musicians trained in the Western classical style (Brodsky et al. 2003). This is similar to how many may create an experience of imagined sound in their own voice when reading a written text in their home language. Further studies of audiation have found a variety of behavioral and self-reported correlates to expressive timings in imagined music in pianists (Repp 2001). Most notably, the pianist’s relationship to the metronome, either strong or weak, was roughly equivalent if they were playing or imagining a piece of solo piano music. Additionally, the spectral qualities of imagined sound, such as timbre and pitch, were shown to be more present and easily changed when compared to the dynamic qualities (Pitt 1992). An influential behavioral study by memory psychologists Baddeley and Logie found that echoic memory, the brain’s temporary storage of auditory material, operates only in the presence of auditory stimuli and, therefore, cannot be the seat of auditory imagery and verifying that imagined sound is more than the recollection of sound (Baddeley et al. 1992).

4.3.2 Creative and Deep Listening There are a variety of creative and meditative practices that focus on the mindful listening of sound, often with a focus on the context of sound. Empirical experiments, involving both self-identified musicians and non-musicians, have illustrated that the act of music listening and perception can be an actively creative process through the self-modulation of attention to different sound characteristics (Dunn 1997). This creativity of listening and heightened awareness of context has been leveraged by

4 Decoding Imagined Sound

85

composers, most notably in composer John Cage’s silent piece 4 33 . While Cage’s, and other silent pieces, generally highlight the fuzzy lines between silence and noise, with a focus on the mind’s wanderings (Cage 1962), there exists a tradition of mute musical pieces which, while also containing little to no acoustic sound themselves, generally highlight other areas of non-cochlear sound. For example, Christine Sun Kim’s face opera ii depicts the fluidity and communal characteristics of Deafness as well as highlight the spatial and performative aspects of American Sign Language (ASL) (Holmes 2016). It is important to note that there has been little research into the experiences of imagined sounds in deaf and hard of hearing populations. This is something we hope to address in future work. Pauline Oliveros, a composer and deep-listening pioneer, created work that approached the non-acoustic qualities of sound from a different perspective. Her deep listening and sonic meditation practices highlighted the healing and communal nature of sound, often viewing acoustic sound as a bi-product of a larger healing gesture (Oliveros 2005). She viewed composing as a “trans[mission] of inner sounds” whose central concern was to “provide attention strategies for participants.” That attention can be directed outward to various acoustic sounds, or inward to the breath, heartbeat, or imagined sounds. In Deep Listening Meditations—Egypt (1999), Oliveros explicitly prompted participants to imagine sounds while phrasing additional prompts as questions. These prompts ask participants to imagine new sound sources or contexts, or remember old sounds and invite participants to speculate about what others might hear or what it would be to hear with an extended hearing range. Oliveros also calls the process of imaging sound into question by frequently beginning prompts with the phrase “can you imagine.” The subtle shift from active commands telling participants what to imagine to an open question inviting insights into the process of imagination aids in providing additional depth and accessibility to the piece. Imagining sound, as with all other human phenomena, exists on a spectrum of ability and intention. By wording prompts in such a way as to examine the process, Oliveros invites participants who may not be able and/or willing to complete other imagined prompts to think critically about the generation of imagined sound. This is an exceptional example of how carefully considered word choice can be used to actively include a broader group of people into an experience or experiment.

4.3.3 Measurement and Representations of Neural Signals The human brain has been the participant of scientific inquiry for hundreds of years. These studies have ranged from autopsy and early surgeries to behavioral studies and analysis. However, the technology to rigorously and reliably analyze the human brain was only developed in the past several decades and is still a burgeoning field. This technology has been the bedrock of a variety of scientific, philosophical, and artistic practices. This subsection offers an overview of the relevant insights and

86

L. May and M. Casey

findings that various neural signal measurement and representation techniques have yielded.

4.3.3.1

Electro-Conductive Techniques

The presence of electric potentials in the brain was first measured by Hans Berger in 1924 (Berger 1933). Adrian and Matthews later confirmed Berger’s findings by sonifying the potential of brain-waves (Adrian et al. 1934). The possible uses (some might say, the potential) of these signals were soon realized across disciplines. This discovery allowed technologists, researchers, and artists to noninvasively measure neural activity. This inspired the creation of technology that interfaced directly with these neural signals, often referred to as a Brain Computer Interface (BCI). BCI is a term coined by Jacques Vidal in 1973 and has since been expanded upon in recent years (Vidal 1973). Johnathan Wolpaw provided a more complete definition of BCIs in 2012, defining them as “a system that measures central nervous system (CNS) activity and converts it into artificial outputs that replaces, restores, enhances, supplements, or improves natural CNS output, and thereby changes the ongoing interactions between the CNS and its external or internal environment” (Wolpaw 2012). Composer and sound artist Alvin Lucier’s Music for Solo Performer (1965) utilized the same phenomenon to excite percussion instruments and is widely considered the first piece of BCI art (Straebel et al. 2014). David Rosenboom, a contemporary of Lucier, implemented these technologies in an ensemble setting during a performance of Ecology of the Skin (1971) (Rosenboom 1977). Many artists and technologists have leveraged these electro-conductive technologies. For a full review of BCI art, see (Prpa et al. 2019). These systems use electroencephalography (EEG), a technique used to measure the electric signals produced by neural activity by placing conductive electrodes on the surface of a participant’s scalp. EEG allows for thousands of readings to be taken every second, providing fine temporal resolution to the readings. Unfortunately, EEG is somewhat limited as it can often only measure neural activity in regions of the brain close to the scalp and the measured signals are often quite noisy. However, EEG has been a useful tool in understanding music cognition and has been used to gain insights into working memory as well as expectation in music (Beisteiner et al. 1994; Janata 2001) To further this line of inquiry, an open-source dataset of 64 channel EEG readings of individuals perceiving and imagining sounds was created (Stober et al 2015). This data proved useful for tasks such as determining the beats per minute (BPM) of perceived and imagined Western music (Stober et al 2016). However, due to the high noise floor of EEG data, it is extremely challenging to decode pitch or timbral characteristics of cochlear-auditory or imagined sound from the data. These limitations motivated researchers to employ new techniques in the study of imagined sound.

4 Decoding Imagined Sound

4.3.3.2

87

Imaging Techniques

While electro-conductive techniques allow for relatively fine temporal detail, the spatial resolution of these techniques is heavily constrained by the sizes of sensors as well as the depth of activity they can measure. Magnetic Resonance Imaging (MRI) uses an extremely large electromagnet to align the protons in the material being scanned. A radio-frequency is then pulsed through the scanned object, disturbing the equilibrium of the protons. The extent and duration of these proton disturbances can then be measured and rendered into images with 0.1mm isotropic resolution (Helthuis et al. 2018). This means that the blood flow in a participant’s brain, which is used as a proxy measurement for activation, can be measured with extremely precise spatial resolution. While MRI offers far greater spatial resolution than other techniques, its temporal resolution is hindered by both the technique of scanning and the Blood-oxygen-level-dependent (BOLD) response. The scanning technique limits the scanning rate to between 0.1 and 1 Hz and typical BOLD responses occur with 5–7 s lag (McKeown et al. 2003). While MRIs are often invaluable in clinical settings, Functional MRI (fMRI) are generally more common in neuroscientific research. fMRI studies involve a participant performing a known action or receiving a known stimulus, such as looking at a picture or hearing a song, while being scanned. fMRI studies typically employ univariate techniques, which correlate the activation in tiny sections of the brain with a stimulus. By understanding which areas of the brain are particularly active while doing specific tasks, researchers have begun to understand the general function of specific parts of the brain as well as an idea of the flow of information. These tiny sections of the brain, called voxels, are typically cubes with a dimension of 1–3. Even these extremely small cubes can contain hundreds of thousands of neurons, the basic working unit of the brain. An example of a univariate study would be measuring which voxels show a higher level of activity when a participant sees a picture of a familiar face versus an unfamiliar one (Eger et al. 2005). A variety of univariate studies of neural imaging data have been previously conducted to identify candidate regions of interest (ROI) that are recruited when a human participant is actively perceiving or imagining sound. Imagined complex sounds not involving music or speech have been shown to involve the secondary auditory cortex, which deals with higher-level abstractions of sound, with no significant activity in the primary auditory cortex, the part of the brain responsible for parsing low-level data directly from the cochlea via the auditory nerve (Bunzeck et al. 2005). Studies examining imagined musical sound using positron emission tomography (PET) scans, a technique similar to MRI that instead uses radioactive substances instead of radio-waves, found activity in brain regions responsible for perception: the superior and inferior right temporal cortex. The frontal lobe, an area associated with higherlevel cognition, as well as the hippocampus, the brain’s long-term memory area, were implicated in tasks that involved recollection of a melody from memory (Zatorre et al. 1996). This highlights a possible important distinction between imagined sound that is more novel to the imaginer, as in the case of jazz improvisers or freestyle rappers (Liu et al. 2012; Limb et al. 2008; Donnay et al. 2014), and sounds that would be

88

L. May and M. Casey

recollected, such as a familiar melody. fMRI studies in non-musicians (Herholz et al. 2012) as well as trained classical singers (Kleber et al. 2007) have found that activity in the secondary auditory cortex is correlated with experiences of imagined sound. Activity in the Supplementary Motor Area (SMA), associated with motor planning, has been shown as an area of interest in a variety of studies exploring imagined music (Zatorre et al. 1996; Kleber et al. 2007; Herholz et al. 2012). However, the SMA was not cited as an area of interest in the imagination of non-musical complex sounds (Bunzeck et al. 2005). This could imply that covert vocalization, or silent singing, could be an important component of musical imagery. Univariate techniques are powerful and incredibly useful in answering certain questions, however, these methods do not detect complex patterns in the neural activity. While the brain does have regions associated with specific functions, complex stimuli, such as music, often cause activity in multiple different regions. Additionally, this activity is not always as simple as a linear increase or decrease. Therefore, new techniques that leveraged machine learning techniques and additional computational power emerged. These multivariate and machine learning techniques have allowed for more complex patterns to be detected from imaging data (Norman et al. 2006). Multi-Voxel Pattern Analysis (MVPA) is a collection of techniques that analyses the relationship between BOLD responses of different voxels. A team led by computational neuroscientist Jack Gallant successfully decoded low-level features from the visual cortex in an image perception and imagination task using these multivariate techniques (Naselaris et al. 2015). This is an incredible example of the effective decoding of an imagined phenomenon. In the domain of imagined sound, foundational work by psychologist Andrea Halpern has shown the many correlates between perceived and imagined sound. Halpern’s 2004 fMRI study illustrated the possible differences in perceived and imagined timbral space (Halpern et al. 2004), confirming previous findings that indicated the presence of differences in perceived and imagined timbres (Crowder 1989). In addition to this, Halpern developed the Bucknell Auditory Imagery Scale (BAIS). This behavioral scale measures the individual differences in the ability to deliberately incite and control imagined sound (Halpern 2015). Computational scientist (and chapter co-author), Michael Casey created an accurate genre classifier through a regression comparison of audio features and activity in the superior temporal sulcus (STS) (Casey et al. 2012). The fMRI study described in the next section builds off of this body of knowledge.

4.4 fMRI Decoding of Imagined Sound This section will describe the design and methods used in a contemporary fMRI decoding experiment as well as a discussion of both the univariate and multivariate results. The research team consisted of Lloyd May, Prof. Andrea Halpern, Sean Paulson, and the principal investigator, Prof. Michael Casey. Scans were performed by members of the research team at the Dartmouth Brain Imaging Center. This section

4 Decoding Imagined Sound

89

aims to provide a high-level overview of the experiment, its design considerations, and results. It is intended for a general audience. For additional technical details and neuroscientific interpretations, please see the related journal article (May et al. 2020).

4.4.1 Experiment Overview The goal of this experiment was to decode components of imagined sound through a combination of computational neuroscience and imaging techniques. Specifically, pitch and timbral information were decoded from fMRI scans of trained musicians imagining individual musical notes from the major scale. The accuracy and distribution of errors of this decoding were used to inform a discussion regarding the pathways and neural mechanisms recruited when imagining sound. While being scanned in an MRI, musicians were played the first few notes of the major scale followed by a pause and then the next note in the scale. This was used to establish a baseline of a musician’s neural activity while hearing different notes of the major scale. The participants were then asked to imagine the next note in the major scale. They then rated the vividness of the note they had just imagined. After every rating, participants were played a random chromatic note and asked to rate the goodness-of-fit of that note in the major scale they had just heard. The motivation behind these choices as well as additional experimental procedures will be explained in the following subsections.

4.4.2 Methods As discussed in the previous section, one of the most important decisions that we as designers had to make was what information we would prioritize in the decoding process. Given the unique opportunities and constraints of working with fMRI data, we elected to decode pitch and timbral information from the fMRI data. The MRI runs on a fixed scanning rate, which in our experiments was once every 2 s (0.5 Hz). This meant that conventional notions of rhythm and timing would most likely be extremely challenging to decode. Additionally, there are multiple fMRI experiments that studied imagined pitch and timbre. These previous experiments were helpful references as we designed the finer points of the experiment. The answer to “what was excluded as noise?” is a bit more complicated and was largely influenced by standard fMRI conventions. Measuring the movement of small volumes of blood using a large magnet generates many unique forms of noise. The exact noise reduction strategy will be discussed in the Preprocessing subsection.

90

L. May and M. Casey

Table 4.1 Inclusion criteria of fMRI study. An asterisk (*) denotes a safety constraint unique to using an MRI Inclusion Criteria Age Handedness Absolute Pitch Affiliation Musical training or performance experience Musical background Ability to see color Hearing impairments Oral corrective devices* Claustrophobic Pacemaker, aneurysm clip, or neurostimulator

4.4.2.1

At least 18 years old Right-handed only Cannot identify pitches absolutely Must be affiliated with the college At least 8 years Grown-up listening to mainly Western tonal music Able to see full-spectrum color No significant hearing loss or impairments No braces or a non-removable upper retainer Not claustrophobic Not have any of these devices

Participant Recruitment

One of first steps of the fMRI experiment was ensuring there were enough willing participants and that we were able to scan them safely. Participants were recruited and screened through a multi-stage process to ensure both the safety and experimental validity of all participants. To accurately compare participants’ results, variables known to affect fMRI results had to be controlled for. The first-round of participant screening determined experimental validity and screened for these variables, such as the participant’s years of musical training, ability to hear in a medically traditional manner, handedness, ability to see color, and whether or not the participant has the ability to identify pitches absolutely. The second-line screening was focused primarily on the safety of the participant and ensured that they could be safely scanned. To summarize, participants were excluded if they did not meet the inclusion criteria outlined in Table 4.1. If the participant cleared all screening, they were asked to complete additional forms containing the Bucknell Auditory Imagery Scale (BAIS) (Pfordresher et al. 2013) and the Bregman Musical Ability Rating Survey (BMARS) (Casey et al. 2012). The BAIS is used as an approximate indicator of a participant’s ability to deliberately create and control imagined sounds. It is divided into two sections: a vividness and a control section. BMARS is a detailed account of a participant’s formal and informal musical education, listening habits, and performance experience. The participant’s BAIS score and years of musical training were collected here as a variable of interest that can be used to answer questions regarding individual participant differences. This data would be used to answer questions like: “Does a participant’s musical training affect the decoding algorithm’s performance?”

4 Decoding Imagined Sound

91

When arriving for the fMRI scanning session, participants completed an informed consent form, an additional safety screening form, and the Edinburgh Handedness Inventory to verify their handedness before scanning. Unfortunately, left-handed participants had to be excluded as handedness greatly impacts the side of the brain certain neural activity is found. If any participant did not consent or was deemed unsafe to scan, they did not take part in the experiment. In total 69 people completed the interest form, 43 completed the second-line screening and scheduling form, and 23 participants were scanned. As fMRI studies are both time and resource-intensive, study populations often range from 5 to 50 participants.

4.4.2.2

Timbre Selection

One of the goals of the experiment was to decode timbral information as well as pitch. Given the time constraints of an fMRI study, we were forced to prioritize either pitch or timbre. If we attempted to vary too many variables in a single experiment, the statistical confidence with which we are able to construct insights drops dramatically. Building off of the relatively established field of pitch perception and cognition, we elected to prioritize pitch and ensure that each participant imagined all 7 degrees of the major scale over multiple runs. Therefore only two distinct timbres could be selected for the experiment. Imagined timbre groupings, as outlined by Halpern (2004), are similar to groupings participants make of acoustic timbres. Three possible timbre pairs were identified as being sufficiently separable and having significant overlap in range. These pairs were: • Trumpet and Clarinet • Tenor saxophone and Violin • Oboe and Flute A survey was created to determine the experimental efficacy of each instrument. Participants were asked to wear headphones and listen to clips of an instrument playing in multiple octaves while MRI scanner noise was also played. Instrument samples were taken from the EastWest Hollywood Series sample library (East West 2019b) The scanner noise was recorded using an optical microphone placed where a participant’s head would be during scanning, in the center of the MRI. After listening to the mixture of instrument sounds and scanner noise, participants were asked to identify the instrument and rate its familiarity, pleasantness, and the degree to which the sound represented an acoustic instrument (similarity) on a 7-point rating scale. The survey was completed by 13 participants, 30.7% of whom were female and 69.23% had 8 or more years of musical training. The results of the survey are summarized in Table 4.2. The oboe, and therefore the paired flute, was eliminated from selection due to its low identification rate of 31%. Trumpet and clarinet were selected as they performed relatively well on all metrics and had significantly more overlap in range, requiring

92

L. May and M. Casey

Table 4.2 Results of timbre differentiation survey (n = 13). All non-percentage ratings are on a 1–7 scale. Similarity refers to a rating of the sample’s similarity to an acoustic version of the same instrument. (SD = sample standard deviation) Familiarity Pleasantness Similarity Instrument identification (%) Mean SD Mean SD Mean SD Saxophone Trumpet Violin Flute Oboe Clarinet

5.69 4.92 5.15 4.31 4.69 5.00

1.11 1.26 1.46 1.70 1.49 1.63

5.46 3.77 5.54 4.92 4.77 5.08

0.97 1.42 1.33 1.19 1.48 1.38

5.31 4.00 5.00 4.92 4.92 4.77

1.44 1.41 1.53 1.50 1.19 1.88

54 69 62 69 31 77

less digital manipulation of the acoustic samples. The experiment stimuli were constructed using the East West sample library and player.2 All samples were pooled and root mean square (RMS) normalized, ensuring all samples were of approximately equal volume and none of the samples distorted on playback.

4.4.2.3

Experimental Procedure

To better understand the motivation behind experimental design choices, this subsection will provide a detailed overview of the experiment from beginning to end. Upon entering the scanning suite, the participants were greeted and the details of the tasks they would be asked to perform were explained. After a final safety procedures, participants entered the scanning room and were asked to wear MRI-safe earphones and were shown the four-button box they would use to answer the questions during the experiment. Once inside the scanner, a short scanning sequence was run to precisely measure the location of participant’s head inside the scanner. This initial scan also allowed the scan operator to set the three-dimensional slab to be scanned. As the experiment used a fixed repetition time (TR), the volume and dimensions of the scanning slab could not be adjusted, only the relative position of the slab. The slab was slightly smaller than most participants’ brain which forced us to optimize the slab position to exclude equal proportions of the motor cortex and cerebellum, essentially splitting the scanning deficit between the top and bottom regions of the brain. A detailed anatomical scan was then conducted which allowed for the creation of a precise scan of the physical anatomy of the participant’s brain. This scan can be referenced later on to determine which voxels are in which anatomical area of the brain. Finally 2 Specifically, the front-mic’d legato samples from the Hollywood Brass collection was used for the trumpet (East West 2019a) and the sustained, non-vibrato samples from the Hollywood Orchestral Woodwinds collection was used to generate the clarinet sound (East West 2019b).

4 Decoding Imagined Sound

93

Fig. 4.2 Illustration of the ordering of one complete cycle in the fMRI experiment

Fig. 4.3 Illustration of the events and their durations for a single run of the fMRI experiment

a button-box and audio volume test was conducted to ensure the stimuli was both audible and comfortable for the participant. fMRI experiments are traditionally divided into cycles, the smallest complete unit of the experiment, and runs, a sequence cycles which make up a complete run of the experiment. Experiments often have multiple cycles to test different variables, such as the musical note being imagined, but have multiple runs to gather additional data for the statistical analysis. Participants were scanned while completing the first of eight runs as outlined in Fig. 4.3. Each run consists of 21 cycles, with the sequence of a cycle illustrated in Fig. 4.2. Two variable pauses, labeled as Jitter 1 and Jitter 2 respectively in Fig. 4.3, were used to remove temporal correlation between scale degrees and stimulus. Each run consisted of 21 stimuli, namely the second to the eighth degree of the major scale in three octaves, resulting in 21 stimuli per run. The experiment involved two primary conditions: a perceived3 and an imagined condition. A triad in the appropriate octave and key was played at the beginning of each cycle to sufficiently prime participants to imagine the pitches and timbres accurately. This context triad was also used in the perceived condition to balance the experiment. All the preceding scale degrees were played in the appropriate octave from the tonic for 0.5 s per note and the key for the experiment was fixed, with participants completing the entire experiment in either E or F major. The key was alternated so that all even participants completed the experiment in E and all odd participants in F. In the perceived condition, participants were played the stimuli 3 We

use the term perceived in this section strictly as it pertains to this line of inquiry. While other terms, such as primarily-acoustic or cochlear-listening, may have been used here, we chose to use perceived to maintain a clear connection with the preceding literature.

94

L. May and M. Casey

note for four seconds. Whereas in the imagined conditions, participants are asked to imagine the stimuli, the next pitch in the major scale, in the current timbre. Notes from the major scale were chosen as all our participants had extensive training in Western tonal music and the major scale is an extremely popular pitch sequence. This choice also provided a reasonable degree of certainty that the participant would be imagining the correct pitch. Notably this choice only includes a very specific population and these results cannot be interpreted outside of that population. The ordering of both the perceived and imagined conditions and timbre were fixed during the experiment to allow participants to familiarize themselves with the perceived timbres before being asked to imagine them. The order was always: perceived timbre A, perceived timbre B, imagined timbre A, imagined timbre B. This order was then repeated an additional time to create sufficient data points for analysis. Timbre A is either clarinet or trumpet, rotating every two participants to ensure a mixture of key and timbre orders in the dataset. Timbre B is whichever instrument timbre A is not. The probe tone was selected from all 12 equal-tempered chromatic notes and was always in the appropriate octave and timbre as the run it was played in. Once the probe tone was played, participants were asked to rate its fit within the major scale. This was included in the task to determine if the participant was attending to the task and if they were imagining the required pitches accurately. If responses were either inconsistent or did not seem to represent the tonal hierarchy of the major scale (Krumhansl et al. 2010), the accuracy and attention of the participant were then brought into question. The orderings of stimuli and probe tone presentation were randomized. Millions of random orderings of both stimuli and probe presentation were created, and the orderings which maximized entropy were selected. Between each run, participants were asked if they had any questions and if they felt fit to continue. If a participant expressed a desire to stop the scanning session, they were immediately removed from the scanner and the session canceled. In addition to the auditory stimuli participants heard, they were also looking at a screen that indicated which task they should be performing. The sequence of visual stimuli can be seen in the Visual Display section of 4.2. Colors were chosen to maximize neural contrast when participants should attend to stimuli and all visual stimuli were set against a low contrast grey background. A black fixation cross was used to indicate no stimuli, while white indicated the context arpeggio or scale. The cross turned magenta to prime suspects and cyan while the main stimulus was presented or imagined. All questions were displayed in large, white Arial typeface. While these small details of the visual stimuli may seem insignificant, if they are not controlled for, the algorithm may detect patterns with unwanted variables. For example, if the fixation cross was solid in the heard but flashed during the imagined condition, we would detect many spurious correlations and unknowingly answer a completely different question.

4 Decoding Imagined Sound

95

4.4.3 Participant Ratings While the fMRI experiment is, in some aspects, a very detailed measurement of the neuro-physiology of imagined sound, it is a coarse measure of the overall phenomenological experience of imagined sound. Therefore it was important to collect additional data from the participants during scanning. The experiment collected two main types of data, namely neural activation and self-reported behavioral data. A snapshot of brain activity can be taken by measuring the BOLD response in the brain using the MRI scanner. This technique was used to create a high dimensional dataset that can provide insight into the process of deliberately imagining sound and be used to train a decoding and classification algorithm. The behavioral data are self-reported answers to the following questions: • “Vividness of previous note?” 1—Not at all vivid, 2—Somewhat vivid, 3—Moderately vivid, 4—Very vivid • Clarity of previous note? 1—Not at all clear, 2—Somewhat clear, 3—Moderately clear, 4—Very clear • Goodness of fit of this note? 1—Very poor fit, 2—Somewhat good fit, 3—Moderately good fit, 4—Very good fit The questions were all answered on a 1–4 scale using a button box with four pushbuttons. The timings of the questions were the same as those outlined in 4. The vividness question, used in the imagined condition, and clarity question, for the perception condition, were used as a proxy measurement for data noise. For example, if a participant rated a run as a “1—Not at all vivid” that indicated that the participant did not complete the imagined sound task and the associated scan contained a high level of noise. These self-reported measures were extremely important as we could not directly observe if the participant was performing the task accurately. The response to the goodness-of-fit of the probe tone was compared to the expected tonal hierarchy of Western tonal music to measure if participants were attending to the task. If a participant did not answer, did not reflect the tonal hierarchy, or did not have consistent ratings of fit for the same pitch classes, they were excluded from further analysis. These exclusions were done to avoid adding additional noise to the data.

96

L. May and M. Casey

4.5 Results It is widely understood that activity in certain lobes of the brain is highly correlated to specific functions (Gazzaniga et al. 2006). An overview of those functions are as follows: • Cingulate cortex: Associated with awareness management, as well as interfacing sensory information with action and emotion regulation. • Frontal lobe: Correlated with task-specific actions as well as working memory and other high-level functions such as problem solving, language, and judgement. • Prefrontal cortex: An active component in the planning and execution of speech as well as projecting future consequences of current actions. • Hippocampus: Primarily responsible for the retrieval and recollection of longterm memories. • Parietal lobe: Correlated with the integration of sensory information as well as action planning. • Temporal lobe: The temporal areas are correlated with auditory, language, and semantic processing, generally involving automatic or non-volitional responses to stimuli. Notably the temporal lobe contains the primary and secondary auditory cortices which are largely responsible for the perception of acoustic sound through the ear in people with traditionally functioning cochleas.

4.5.1 Exclusions Based on Behavioral Data This experiment examined both state and trait variables. State variables are those that were actively changed by the participant during the experiment, such as what note they are imagining and their answers to the probe tone tasks. Whereas trait variables are those participants possess outside of the experiment, such as their listening and practicing habits, years of musical training, and handedness. State variables are useful in measuring performance during an activity while trait variables often provide insight into the impact of individual differences. Comparing these trait variables against a standardized scale allows for the degree of individual difference to be accounted for. The tonal hierarchy is a pattern found in certain traditions of Western tonal music where certain equal-temperament notes are rated as fitting better or worse within the context of a scale. For example, within the context of a major scale, the tonic would be rated a better fit in the sequence than the tritone of the scale. This pattern extends deeper as four distinct tonal groups emerge. In a major context, the tonic (T1) is rated as best fitting followed by the triad notes (M3, P5), the non-triad diatonic notes (M2, P4, M6, M7), and lastly the non-diatonic notes (m2, m3, d5, m6, m7). This pattern has been found in many studies (Krumhansl et al. 2010), and was therefore used as a proxy for task attention. Specifically, if a participant’s probe tone ratings differed

4 Decoding Imagined Sound

97

substantially from the tonal hierarchy, it was suspected that they were not attending to the task and were excluded from further analysis. Other exclusion criteria, such as self-consistency and the number of missing values were also explored, but none proved as reliable as measuring the probe tone ratings against the tonal hierarchy. 23 participants completed the study, 6 were excluded, resulting in 17 participants who were further analyzed. It is extremely important to recognize the biases that this choice exacerbates, as it actively prioritizes the Western tonal system and cannot be generalized past this. While this does create a reliable frame of reference for our participants, who all selfidentified as having at least 8 years of training in Western tonal music, it is always important to acknowledge who is excluded by design and interpret the experimental results accordingly.

4.5.2 Preprocessing and Univariate Results The data were preprocessed using the fMRIprep pipeline (Esteban et al. 2019). This series of algorithms, or pipeline, is widely used in fMRI research. They have been developed and tweaked by the community over many years to prioritize information regarding neural activation and remove the remaining noise. This pipeline was used to perform anatomical as well as functional preprocessing. The anatomical preprocessing normalized the detailed anatomical (T1) scan, removed all voxels that did not contain brain segments, and segmented the white and grey matter. Functional preprocessing, which used all scans except the T1, was implemented using the same pipeline. This included data smoothing via interpolation between individual 2D slices, removing head movement noise, as well as aligning the scans in reference to both the anatomical T1 and the common Montreal Neurological Institute (MNI) space. By referencing the scans to the MNI common space, the data from different participants can be meaningfully compared to each other as well as to studies conducted by teams. To determine which areas of the brain show differential activity when imagining sound versus perceiving it acoustically, a univariate contrast analysis was conducted. This univariate analysis was completed as part of the data validation process. If the contrast analysis found similar ROIs as previous studies, it would increase our confidence in the data and preprocessing methodology. A general linear model (GLM) was fitted to each participant’s BOLD response, correcting for overall signal drift, white matter activity, and frame-wise displacement. The GLM also performed a contrast subtraction where the average activity in all scans where a participant was listening to acoustic sound was subtracted from those where the participant was imagining sound. This contrast subtraction results in a heat map of differential activity as well as a correlated T-map which can be used to determine the statistical significance of the differential activity. A three-dimensional T-test was then performed across all participants to generate a heat and T-map, averaged across all participants. After

98

L. May and M. Casey

Fig. 4.4 Mean BOLD response of all voxels in all significant clusters. Color gradient shows only relative activation, with red being highest. (N = 17)

clustering4 nearby voxels and ignoring outliers, this map was used to determine the ROIs that show differentiated activity between heard and imagined sound. These ROIs are shown in Fig. 4.4. The ROIs in found in the univariate analysis largely agree with previous findings in the literature (Halpern et al. 2004; Halpern 2015; Zatorre et al. 2005). This agreement provides evidence that the experiment did measure experiences of both acoustic sound perception and deliberate imagining of sound. All ROIs shown in Fig. 4.4 are regions that have differentially higher activity when participants were imagining sound rather than perceiving it. Lateralized activity refers to patterns of activation that do not occur in both the right and left hemispheres. For example, in this analysis, activity in the Middle Temporal Gyrus is left-lateralized, meaning no corresponding activity was found in the right hemisphere. Non-lateralized activity, present in both hemispheres, was more common in this analysis and were as follows: Large clusters of activity in the insula, a small region in the cerebral cortex, were found in both hemispheres. It is hypothesized that activity in the insula is correlated with awareness of the present moment and the self (Damasio 1996; Uddin et al. 2017). Activity in the Inferior Parietal Lobe is associated with language and attention (Bzdok et al. 2016), while a response in the Middle Frontal Gyrus (MFG) is associated with attention and task arbitration, potentially acting as a switchboard for the frontal gyrus (Japee et al. 2015; Koyama et al. 2017). Activity in the Supplementary Motor Area (SMA) is perhaps one of the most unexpected areas that has shown repeated activity in the imagined sound literature (Zatorre et al. 2005). This area is hypothesized to be involved in the planning of movement and activity in this area is thought to indicate the importance of covert sub-vocalizations as a part of imagining pitched sound, but not in imagining complex, non-pitched sounds (Bunzeck et al. 2005). The right STG displayed lateralized activity. This area has been correlated with auditory and language processing as well as emotion recognition (Bigler et al. 2007; Gharabaghi et al. 2006; Radua et al. 2010). Finally, the left middle temporal gyrus is associated with various cognitive processes, including language and memory pro4 The

face.

minimum cluster size cut-off was 40 voxels and only included voxels that shared a common

4 Decoding Imagined Sound

99

cessing as well as sense information integration (Wei et al. 2012; Whitney et al. 2011). Aside from the SMA, the ROIs highlighted in this analysis are what one might expect as they are generally involved with attention, memory, and high-level processing of sensory information.

4.5.3 Multivariate Results and Behavioral Correlates Multivariate techniques, such as MVPA, allow for advanced pattern recognition between voxels. While univariate techniques average activation levels and form clusters based on this mean activation, multivariate methods allow for the detection of patterns of activation between voxels as opposed to the binary difference between them. In this study, specific ROIs were selected to reduce the chance of spurious correlations, as fMRI data contains thousands of voxels over hundreds of scans. ROIs were selected before the analysis from relevant literature. For simplicity, these ROIs will be grouped by their corresponding brain region: cingulate, frontal, prefrontal, parietal, or temporal. In the analysis, all octaves and timbres were combined, resulting in a seven class classification task in which the classifier analyzed a scan and determined which scale degree of the major scale the participant was imagining. Various classifiers were trained on data from the selected ROIs of each of the 17 participants. These classifiers look at the data from an individual scan and predict which pitch class the participant was imagining. The classifier accuracies were averaged across participants to create a mean accuracy for each ROI. This mean accuracy is used as a measure of the relative importance of that ROI in the imagined pitch task. The statistical significance of the accuracy gains were calculated by comparing the model performance to a baseline as well as a null model. The baseline accuracy is the expected accuracy from random predictions, namely 14% or a 1 in 7 chance. The null model is a classifier trained on the same data, but with the data labels scrambled. It is a helpful point of comparison as it indicates the complexity of the task and data. Additional cross decoding classifiers were constructed by training the classifier on only one type of data (the heard pitch classes) and having it decode the other (the imagined classes). For information and analysis of the cross decoding, please see (May et al. 2020). The regions highlighted in Fig. 4.5 show the statistically significant ROIs. These findings show the first statistically significant direct decoding of imagined sound. These regions are similar to those in the univariate analysis. The hippocampus was not differentially active in either timbre task in the heard condition, likely due to longterm memory not being recruited to process timbre. Similarly, the prefrontal areas were only recruited for pitch-class classification and not for timbre discrimination. The findings in both the univariate and multivariate analyses show that there is a large amount of representational overlap for heard and imagined sound. These analyses show that areas associated with sensory processing, executive planning, and awareness are recruited when sound is deliberately imagined.

100

L. May and M. Casey

Fig. 4.5 ROIs correlated with statistically significant pitch class decoding. Medial and lateral view of left hemisphere for clarity

The question that inevitably follows is: Can I now go into an MRI and have this melody I’m imagining transcribed? Perhaps unsurprisingly, the answer is a complicated “not yet.” The individual ROI accuracies, which range from 15 to 22%, are not explicitly reported here as they were not the optimized for in this mapping study. Much like the difference between creating a map of a continent and one of a town, the goal of this study was to determine which regions should be focused on and not the fine details that would result in raw accuracy. An additional complication is that these mappings differ between participants. In some cases these differences can be substantial. A GLM was used to analyze which of the measured state and trait variables had an impact on the classifiers accuracies. The coefficient of this analysis are shown in Table 4.3. As this analysis potentially non-linear results, interpreting these results absolutely is slightly more complicated than a standard regression as they represent the relative change in odds. For example, the odds of a correct pitch class prediction by the classifier trained on frontal areas increased by 21.23% if the stimuli was a trumpet compared to the clarinet. That is, if the classifier was 16% accurate in the clarinet timbre, it was 19.36% (16% + [16% × 21.23%]) in the trumpet timbre. The octave of the note had no significant impact on classification performance. Decoding in the parietal areas improved by 10.4% for each point in the participants’ 1–4 vividness rating. A 1 point increase in the participants’ BAIS score, a 7 point measure of their ability to vividly imagine sound, improved classification in the prefrontal areas by 11.5%. Each additional year of musical training improved classification odds by 9.7% in the prefrontal ROIs while the participant’s training in musical improvization had no measurable influence on the accuracy of pitch classification. Finally, the specific pitch class being decoded impacted classifier performance in both the cingulate and temporal ROIs. As the pitch classes were normalized, the classification in the cingulate was 2.3% less likely to be correct when the pitch class was a P5, M6, or M7 whereas a 4.1% improvement was seen in the temporal ROIs for these same pitch classes. The GLM analysis is not only a measure of the effect of experimental variables, such as timbre. It also illustrates the impact of individual differences in these models

4 Decoding Imagined Sound

101

Table 4.3 Relative change in odds of correct pitch class classification, (p 0.06 behavior becomes x = 1 or x < 0.94. What was previously a jump between stopped and spinning now became a gap in which the parameters never smoothly approach the ceiling but rather clip at the x < 0.94 maximum until jumping discontinuously to x = 1. In addition, the performer

8 Experiencing Embodied Sonic Meditation Through Body …

187

kept the TSPW in motion throughout all 6 pieces except for only a few complete stops. The normal TSPW usage pattern skews towards higher spinning speeds, with limited stopped or zero values, while the reversed TSPW parameters rarely hit the capped speed and contains longer periods at zero value. Reversing the parameters thus created a highly reverberant processed sound that could not be produced with a real TSPW. To keep the reversed version from sounding like a qualitatively different instrument, we experimented with several offsets before selecting a 0.09 offset added to the reversed parameters. Figure 8.7b compares the original and reversed pulse-width parameter for vocal piece #2. For each song in each evaluation, there was an original video and an altered video, both using the same visual component and the same performance, but with a different audio track. For experiment 1, the altered video was the desynchronized version, and for experiment 2, the altered video was the ‘inverted’ version. Each participant then randomly watched either the original or altered for each of the performances, so he or she never viewed both versions of a piece. This design was chosen so that, in the aggregate, there was an approximately equal number of viewings of each version of each performance, and allowed for participants to comment on differences in processing, if the participants detected any, while avoiding the potentially confusing situation of the same subject seeing both versions of the same performance. The experiment was administered via a custom interactive interface written in Processing (Reas and Fry 2007) and was presented in a quiet room on a 15-in. Apple laptop using high-quality headphones. Figure 8.8 shows the stimuli presentation interactive user interface. At the beginning of the experiment, participants were told that there were different vocal processing methods being implemented with the evaluated instrument during various performances. Instead of focusing on the performer’s ability to perform, participants were instructed to rate the performances and to pick their favorite ones based on the vocal processing method and how the instrument was used during the performances. Each participant watched a total of 6 performances and answered 6

Fig. 8.8 Interactive user interface for presenting stimuli and record quantitative and qualitative answers from the study’s participants

188

J. C. Wu

questions after watching each video. The questions were 5-point Likert-scale questions with the descriptors “extremely,” “very,” “moderately,” “slightly,” or “not at all,” and were presented without an associated number. The six questions were: “How expressive was the performer?”, “How engaged were you in the performance?”, “How effectively did the performer use the instrument to control the voice processing?”, “How convincingly was the instrument responding to the performer’s vocal expression?”, “How closely aligned was the movement of the instrument and the voice processing?”, “How much do you like the way the instrument was changing the vocals?”. After all videos were presented, a demographic questionnaire was administered, and a final question was asked: “Please recall your favorite performance. What did you like about it? Do you think the instrument did or did not change the performer’s ability to be expressive? Please elaborate if possible. If you recall which video it was in order of presentation (e.g., first, second…), please state so.” An entire experimental session took 20–30 min.

8.3.5 Results Since the Likert-scale questions resulted in ordinal data, I chose to analyze the responses from those who saw the original or altered versions of the songs using Fisher’s exact test, which tests if there was a difference in distributions using contingency tables (i.e., does the distribution of responses to the questions differ depending on seeing an original or altered video?). This test is comparable to the chi-squared test but was chosen instead since the sample size was small enough that the chi-squared test’s dependence on a chi-square distribution would not hold. For all questions in the synchronicity evaluation, the responses to the original version were significantly different from those of the altered (p < 0.001 always). For each question, those who saw the original version found it more frequently to be more expressive, frequently were more engaged, and both preferred and found more convincing the interaction between performer and instrument. In the mapping intuitiveness evaluation, the responses to Q2 – Q6 of the original version were significantly different to those of the altered (p < 0.05 always), while the response to Q1 was nearly so (p = 0.07). Figure 8.9 shows the count of responses based on movie version for each condition of the two evaluations. Group means are presented as dashed vertical lines. The red band above each “5” indicates that subjects viewing the original version of the mapping were always more likely to answer “extremely.” The final question of the study asked participants to reflect in writing on which performance was their favorite and why. I categorized their responses into 5 general types, presented in Table 8.1 along with their frequency in response to the original and altered videos. Some participants noted several videos in their responses, as well as more than one reason for selecting one video–I have included each reason discussed for each video mentioned in the table.

8 Experiencing Embodied Sonic Meditation Through Body …

189

Fig. 8.9 Count of responses based on movie version viewed for each evaluation

Table 8.1 Counts per movie type of various categories of free-response answers to why a particular performance was preferred by the participant Synchronization

Mapping

Response type

Original

Altered

Original

Altered

Instrument added to performance, instead of distracted from it

8

2

1

2

Very expressive by performer (no mention of instrument)

4

4

2

1

The song itself is beautiful and the processing match the song

3

3

4

3

Think our mapping and processed vocal are more intuitive, thus making the piece most expressive than other mapping

7

0

0

0

The physical/visible intensity of 5 the gesture is aligned (synchronized) with intensity in the effects processing

0

9

0

Notably, in the synchronization evaluation, the intuitive matching of the gestures, and the vocals present in the original version, was a frequent reason given for why a video was preferred, and in no instance did a person refer to the altered videos as being more intuitive. Interestingly, there was an equal number of participants who justified their favorite performance in terms of the performer’s particular expressiveness (without mentioning the instrument or its mappings at all) and those who chose an altered video as their favorite.

190

J. C. Wu

In the mapping evaluation, the most frequent reason to prefer a video was that the gesture and processing were closely aligned, which matches with the experimental manipulation of these videos. In comparison to Experiment 1, this emerged as a more frequently reported factor in a video being a favorite, rather than the intuitive nature of the matching of the gestures and processing. It is also worth mentioning that some of the respondents specifically mentioned that the instrument did not distract them from the performance.

8.3.6 Discussion Designers and researchers’ general assumption about using synchronous control and mapping strategies in designing DMI seems reasonable. The tight synchronicity between movement and sound is implicit in the design of NIME-style instruments (Wessel and Wright 2002). The “straw man” non-synchronous mapping is a situation where the movement does not affect the sound processing. It (not surprisingly) tended to be less favored by audience members. In order to further examine this assumption, I asked six questions of the participants relating to perception of performer expression, participant engagement, and perception of the relationship between body movement and vocal processing. Non-synchronicity seemed to have widespread effects on participants’ responses. When watching the original videos, participants found themselves more engaged, the performer more expressive, and liked the processed voice and performances more than when they watched the altered videos. When using the desynchronized processing and mapping, some audiences perceived the prayer wheel as processing the vocals in a “forceful” way. For example, one participant said: “It is not intuitive when a delay with a particular length lingers on; where the motion already changed to something else.” Most of the participants noted the difference between gesturevocal synchronization and desynchronization, as one participant described: “There is a natural relationship between the spinning motion and tension. Some effects work better than others.” One song seemed to stand out as participants’ favorite, especially in the synchronized version. The vocalist went into particularly high ranges and sang through several different octaves, and participants stated such things as “the different octaves presented interesting ways for the instrument to manipulate sound,” and “the way it elongated the impressive high notes was what I enjoyed the most.” This again highlights the perhaps complex relationship between how the instrument can accentuate the expressiveness of the performance, provided the performance is already detected to be expressive. In the synchronicity experiment, people more frequently preferred the original performances and found them more engaging and expressive than the altered ones. However, in the intuitive mapping experiment, while people still tended to prefer the original and find it more engaging, people generally did not find the performance to be more expressive in the original than in the altered versions (p = 0.07). For

8 Experiencing Embodied Sonic Meditation Through Body …

191

example, several participants stated that the instrument added “something” to the performance; but it “just amplified what was already there.” Two participants stated that the expressiveness of the performer remained independent. One said: “I think the instrument did not change the performer’s ability to be expressive inherently but did change the overall expressiveness of the music itself.” Conversely, some participants clearly noticed the relationship between voice processing and gesture intensity. For example, one participant said: “I liked when the rate of rotation of the instrument matched the speed (how short the chopped-up audio bits were) of the effect. The relationship between the gesture and the effect seemed natural when intensity in the vocal performance aligned with intensity in the effects processing.” When using the alternative mapping where the gestures and vocal processing intensity are in the reverse relationship, one participant said: “The singer’s long notes also sound longer in contrast to the quick moving murmuring effects.” This reflects the data showing that a more intuitive mapping between spinning gestures and vocal processing intensity results in a more highly engaged audience and a performance perceived to be more expressive. Some improvements could be made to this experimental design. First, the performer in the videos was the person who conducted the experiment. This might have led to an overall bias of the ratings. However, since the purpose of the experiment was not to study absolute ratings to the questionnaire, but rather the difference between one version and another, the potential for inflation does not seem wholly problematic. Further, even if participants did detect that the manipulation was that some of the movies was synchronized with the gestures, while others were not; it was unlikely that they guessed it every time, and additionally, they did not explicitly know which version the study’s authors were expecting to be preferred. Thus, the fact that the original version was the one that was more frequently preferred should not be invalidated by any contamination of the data or subjects attempting to “guess” the “preferred” answer. Second, in the mapping experiment, the single ‘non-intuitive’ mapping for comparison was subjectively chosen in my own way. The results showed that this was more frequently not preferred to the original. However, if I had chosen a different mapping (from among infinitely many possible mappings) to compare, the results might be different. While a valid concern, testing more than one “non-intuitive” mapping was out of the scope of this study. Third, since the question about the performance expressiveness asked the subject to focus only on the performance itself, which was in fact completely identical across all experimental conditions, the participants are “correct” not to perceive a difference. However, there is still a perceived difference of the performance’s expressiveness across the synchronized/unsynchronized conditions. It appears that some participants cannot clearly comprehend the questions in the way that I expect them to. Therefore, how to precisely articulate and phrase the questions is crucial if I expect the audience to provide the most useful and accurate feedback. Last but not least, regarding the method of how to use this instrument in general, audiences suggest that the instrument does not need to be used all the way through the performance. For example, one participant said: “I would consider not using the

192

J. C. Wu

instrument the ENTIRE time you sing, maybe just do it at certain parts, I feel that will also add to dramatic effect.” When not “overused,” another participant felt “it was expanding the space and brought depth to her [the performer’s] voice,” and “it augmented the expression of the performance, providing additional dimensions of expression through the use of an instrument that looks analog and traditional, but that results in a very modern sound.” Indeed, this insight provides a unique perspective urging DMI designers, musicians, and performers to rethink the role of technology in novel musical expressions. The future work could be user centered design and usability studies for both expert vocalist and non-singers. For example, asking multiple vocalists to perform westernmusic pieces with it, and studying their perspectives. Alternatively, providing the TSPW to Tibetan monks or chanting practitioners and letting them use the TSPW to practice their traditional chanting, and studying their feedback, could also be interesting in the cultural context and to better understand the possibilities that the TSPW could bring to both music performance and cultural exchange.

8.4 Conclusion In this chapter, I have outlined what embodied sonic meditation entails and my methodology of practicing and evaluating it through DMI and user experience design, composition, interactive art installations, electroacoustic vocal performance, and teaching in higher education. This practice situates its roots in embodied cognition, Tibetan Buddhist philosophy and meditation practices, as well as a series of Western music influences, musical and social theories, and DMI design principles. To explicate how body and mind can be unified through a multisensory experience using music mediation technology to integrate our physical body with real-time sonic events. I provided three case studies of designing and sharing embodied sonic meditation experiences with a broader audience as well as an evaluation framework to evaluate the effectiveness of the embodied sonic meditation technological system design. This evaluation methodology eliminated possible confounds associated with comparing different performances, and it had clear evaluation goals, stakeholders, criteria, methodology, and duration. I compared alternate gesture mappings for audio processing by processing sound from videoed performances in three different ways. The data analysis shows subjects preferred the original mapping (unaltered performance) across many metrics in both experiments 1 and 2 and found it to be more engaging. This finding strongly supports my hypotheses in body-sound intuitive mapping strategies as well as Embodied Sonic Meditation DMI design. The data indicates that intuitive gestural mapping strategies and synchronization mapping between body movement and sounds are crucial to provide users a smooth and successful embodied sonic meditation experience. It offers a framework for audience evaluation of Embodied Sonic Meditation system design in the context of gesture-controlled audio processing DMI.

8 Experiencing Embodied Sonic Meditation Through Body …

193

The experimental data, case studies investigation, and effective teaching practice suggest that using technology-mediated gesture-controlled DMIs, multimedia arts, and interactive installation can enhance both the audience and the user’s Embodied Sonic Meditation experience, increase audience engagement, deepen people’s understanding of sonic art, and serve as a tool to connect people from diverse backgrounds and improve cross-cultural communication. This series of studies support my argument that 21st-century media artists should not only promote traditional underrepresented cultures but also project these cultures’ values and identities into the cutting-edge art and technology context of contemporary society. In conclusion, this first-hand evidence suggests a series of successful applications of Embodied Sonic Meditation DMI design and the possibility of using sensorimotor coupling through auditory feedback to create novel artistic expressions that deepen our sonic awareness and engagement in the world around us. Through a number of large scales interactive art installations, performances, and pedagogical practices, I have discovered that Embodied Sonic Meditation has been creating positive user experiences, calm, and happiness among hundreds and thousands of people and thus has the potential to benefit laypersons’ wellbeing and help people become more connected with themselves and ripple out with a larger intent to connect in an open way to others, building bridges across the world.

References Abel, Jonathan S., Sean Coffin, and Kyle Spratt. 2014. A modal architecture for artificial reverberation with application to room acoustics modeling. In Audio Engineering Society Convention 137. Audio Engineering Society. Barbosa, Jeronimo, Joseph Malloch, Marcelo M. Wanderley, and Stéphane Huot. 2015. What does” Evaluation” mean for the NIME community? In International Conference on New Interfaces for Musical Expression. Barbosa, Jerônimo, Filipe Calegario, Veronica Teichrieb, Geber Ramalho, and Giordano Cabral. 2013. Illusio: A drawing-based digital music instrument. In International Conference on New Interfaces for Musical Expression, 499–502. Behne, Klaus-Ernst, and Clemens Wöllner. 2011. Seeing or hearing the pianists? A synopsis of an early audiovisual perception experiment and a replication. Musicae Scientiae 15 (3): 324–342. Broughton, Mary, and Catherine Stevens. 2009. Music, movement and marimba: An investigation of the role of movement and gesture in communicating musical expression to an audience. Psychology of Music 37 (2): 137–153. Cáceres, Juan-Pablo, and Chris Chafe. 2010. JackTrip: Under the hood of an engine for network audio. Journal of New Music Research 39 (3): 183–187. Chagas, Paulo C. 2014. Unsayable Music: Six Reflections on Musical Semiotics. Electroacoustic and Digital Music: Leuven University Press. Crossley-Holland, Peter. 1967. The state of research in Tibetan folk music. Ethnomusicology 11 (2): 70–187. Conti, Francois, and Oussama Khatib. 2005. Spanning large workspaces using small haptic devices. In First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference, 183–188. Davidson, Jane W. 1994. What type of information is conveyed in the body movements of solo musician performers. Journal of Human Movement Studies 6: 279–301.

194

J. C. Wu

De Silva, Padmasiri, 1995. Theoretical perspectives on emotions in early Buddhism. Emotions in East Asian thought: A dialogue in comparative philosophy, 109–122. Eitan, Zohar, and Roni Y. Granot. 2006. How music moves: Musical parameters and listeners images of motion. Music perception 23 (3): 221–248. Hege, Anne Katherine. 2014. Spirit in the flesh: Essays on the relationship between the body and meaning in performance. PhD dissertation, Princeton University. Johnston, A.J. 2011. Beyond evaluation: Linking practice and theory in new musical interface design. New Interfaces for Musical Expression. University of Oslo Norwegian Academy of Music. Jordà, Sergi, and Sebastián Mealla. 2014. A methodological framework for teaching, evaluating and informing NIME design with a focus on expressiveness and mapping. In International Conference on New Interfaces for Musical Expression, vol. 30, 233–238. Kiefer, Chris, Nick Collins, and Geraldine Fitzpatrick. 2008. HCI Methodology for evaluating musical controllers: A CASE STUdy. In International Conference on New Interfaces for Musical Expression, 87–90. Kolas, Ashild, and Monika P. Thowsen. 2011. On the margins of Tibet: Cultural survival on the Sino-Tibetan frontier. Seattle, WA: University of Washington Press. Lakoff, George, and Mark Johnson. 1999. Philosophy in the Flesh, vol. 4. New York: Basic Books. Leman, Marc. 2008. Embodied Music Cognition and Mediation Technology. Cambridge, MA: MIT Press. Oliveros, Pauline. 2005. Deep Listening: A Composer’s Sound Practice. IUniverse. O’Modhrain, Sile. 2011. A framework for the evaluation of digital musical instruments. Computer Music Journal 35 (1): 28–42. Pratt, Carroll C. 1930. The spatial character of high and low tones. Journal of Experimental Psychology 13 (3): 278. Reas, Casey, and Ben Fry. 2007. Processing: A Programming Handbook for Visual Designers and artists. Cambridge, MA: MIT Press. Roffler, Suzanne K., and Robert A. Butler. 1968. Localization of tonal stimuli in the vertical plane. The Journal of the Acoustical Society of America 43 (6): 1260–1266. Schloss, W. Andrew. 2003. Using contemporary technology in live performance: The dilemma of the performer. Journal of New Music Research 32 (3): 239–242. Schutz, Michael, and Scott Lipscomb. 2007. Hearing gestures, seeing music: Vision influences perceived tone duration. Perception 36 (6): 888–897. Snellgrove, David L., Hugh Edward Richardson, Hugh Richardson, and David Snellgrove. 1968. A Cultural History of Tibet. London: Weidenfeld and Nicolson. Trimble, Otis C. 1934. Localization of sound in the anterior-posterior and vertical dimensions of auditory space. British Journal of Psychology 24 (3): 320. Varela, Francisco J., Eleanor Rosch, and Evan Thompson. 1993. The Embodied Mind. Cambridge, MA: MIT Press. Vines, Bradley W., Carol L. Krumhansl, Marcelo M. Wanderley, and Daniel J. Levitin. 2006. Crossmodal interactions in the perception of musical performance. Cognition 101 (1): 80–113. Vuoskoski, Jonna K., Marc R. Thompson, Eric F. Clarke, and Charles Spence. 2014. Crossmodal interactions in the perception of expressivity in musical performance. Attention, Perception, & Psychophysics 76 (2): 591–604. Wanderley, Marcelo Mortensen, and Nicola Orio. 2002. Evaluation of input devices for musical expression: Borrowing tools from HCI. Computer Music Journal 26 (3): 62–76. Wessel, David, and Matthew Wright. 2002. Problems and prospects for intimate musical control of computers. Computer Music Journal 26 (3): 11–22. Wu, J.Cecilia. 2015. The Expressionist: a gestural mapping instrument for voice and multimedia enrichment. The International Journal of New Media, Technology and the Arts 10 (1): 11–19. Wu, J.Cecilia, Julius O. Smith, Yijun Zhou, and M. Wright. 2017. Embodied Sonic Meditation and Its Proof-of-Concept: “Resonance of the Heart”. Ann Arbor, MI: Michigan Publishing, University of Michigan Library.

8 Experiencing Embodied Sonic Meditation Through Body …

195

Wu, J. Cecilia, and Ren Donghao. 2019. Resonance of the heart: A direct experience of embodied sonic meditation. In 25th International Symposium on Electronic Art (ISEA), 247–253. Wu, J. Cecilia, Yoo-Hsiu Yeh, Romain Michon, Nathan Weitzner, Jonathan S. Abel, and Matthew Wright. 2015. Tibetan singing prayer wheel: a hybrid musical-spiritual instrument using gestural control. International Conference on New Interfaces for Musical Expression, 91–94. Yang, Qi, and Georg Essl. 2014. Evaluating gesture-augmented keyboard performance. Computer Music Journal 38 (4): 68–79.

Chapter 9

Sonic Imagination: Body, Visual Mental Imagery, and Nomadism Luca Forcucci

Abstract The science and technology needed to achieve the finesse of the visual mental image stimulated by a sonic experience are still in their infancy. Visual mental imagery relies on the subjective, internal experience and environment of one individual. Why, then, investigate it? The ‘inner’ first person experience in relation to the ‘outer’ sonic environment may illuminate the mechanisms involved in listening—which differ from the mechanisms of hearing. The relationship between internal bodily experiences triggered by external ones, and vice versa, forms a whole. Subjective and first-person experience, dedicated listening, embodiment, affect, sitespecificity, and practice-led research into sonic arts are, in this chapter, contextualized through two works exploring the listening act in the Amazon rainforest and through EEG as applied to sound and architectural space. The investigation of an audience’s perception of visual mental imagery observes whether—and if so, which—common patterns emerge from their listening to five sonic artworks. Two artworks are inspired by, and are the result of, the outcomes of such investigation. An analysis of the experience of an artwork necessarily includes the creator and focuses not only on the ‘beholder’. Here, the experience is contextualized within a project by a nomadic lab, which transforms ideas and works according to site-specific contexts, alongside the perceptions of different cultural audiences. Such a journey stimulates questions about memory and consciousness when exploring perception in the sonic arts. It includes visual mental imagery as a first-person experience and considers cultural contexts of perception. Keywords Sonic arts · Listening · Visual mental imagery · Oliveros · Massumi · Sonic imagination · Body · Perception · Nomadism

L. Forcucci (B) Berlin, Germany e-mail: [email protected] URL: https://www.lucaforcucci.com; https://www.ubqtlab.org © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 P. C. Chagas and J. Cecilia Wu (eds.), Sounds from Within: Phenomenology and Practice, Numanities - Arts and Humanities in Progress 18, https://doi.org/10.1007/978-3-030-72507-5_9

197

198

L. Forcucci

9.1 Introduction While these lines are being written, Humanity finds itself in a peculiar situation, given the lockdowns of the COVID-19 pandemic. It seems paradoxical in such an interconnected and globalized world, and even more so when writing a chapter about a nomadic project. Movement is certainly more restricted; online interconnections became more essential. The issues the world will experience when life goes back to ‘normal’, whatever that may be, and to which kind of normality, are as yet unknown and unknowable. The nomadic human is now a recluse. Globality will probably be reconsidered and reshaped. The disease appeared to come all of a sudden, putting an abrupt end to the cycles and rhythms of contemporary life, yet Nature had already sent strong signals that were not taken seriously enough. Nomadism is now mostly digitalized and happening online. Moving without moving, as does a body without organs, perhaps it is the realization of the metaverse? Nevertheless, it seems that we need to continue to dream and reinvent a new future, one that, it is hoped, will avoid dystopian realities. Wim Wender’s movie Until the End of the World starts with an orbiting Indian nuclear satellite out of control and predicted to re-enter the atmosphere, threatening unknown populated areas of Earth. Also, a stolen device, wanted by governmental agencies, translates brain activity and records dreams. Wenders wanted it to be the ultimate road movie. Ideas about the exploration of brain activity as visual mental images and the road trip are certainly central to the current project. One’s subjective experience emerging from an artwork is per se difficult to translate, even into words. It might be as difficult to translate as visual images triggered by sound within one’s mind. Federico Fellini worked with the content of his dreams to create movies. He drew his own dreams then collaborated with a Jungian psychoanalyst to analyze and translate them into the foundation material for his movies (Bondanella 2002, 10; Fellini in Laudadio 2020). This process might explain the oneiric aesthetic of his movies. He was building bridges between worlds, between the real world and dreams. Although personal and invisible, dreams necessarily influence our lives: they are the connections with our subconscious. Contextualized in the current project, the two directors’ works suggest the possible exploration of the subjectivity of our imaginary and mental worlds through artworks as being research processes. For Wim Wenders, it is through a fiction, where he proposes a direct visualization of such an imaginary world with a futuristic device. Fellini, with drawing and Jungian psychoanalysis, has really explored his dreams as material for his movies. What is indeed fascinating is the utopian wish to reach the subconscious and to explore it as the starting point for artworks. For the current project, the main idea resides in exploring whether patterns exist between listeners to sonic artworks within the visual mental imaginary.

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

199

9.1.1 Context Around 2006, as a performative artwork experience, I took the decision to include all of my belongings within a 23 kg quantity of luggage, one piece of hand luggage, and a laptop bag. The idea came after having carefully observed what one really needs on a daily basis. I noticed that many things have no more purpose than collecting dust, and thus burden one’s mind. The major challenge was to dispose of approximatively two thousand vinyl records that had been carefully collected over two decades, and quite a few books. This is how the nomadic laboratory for creation-research was established. Tim Ingold mentions that “movements of all kinds are profoundly social activities that are both perceptive of the world and generative and transformative of it” (Ingold cited Vannini 2015, 3). Accordingly, those movements, through the entire world, would (re)shape my future artworks and self. The main interest here is found in exploring the internal worlds and architectural spaces that the sonic arts might trigger within the audience’s mind, as these relate to the intention of the author. What kind of, if any, spatial patterns could emerge among various participants from the observation of their own visual mental imagery? This nomadic project paved the way for almost a decade of research work based on visual mental imagery triggered by the sonic. The research was approached from a qualitative rather than a quantitative methodology: it aimed to collect through a survey the audience’s answers on their perceptions of these works, and to establish a research-creation methodology for the development of the artworks for the process of their composition. The background, motivations, and main concepts related to the research about sonic architecture, listening, sound-walking, site-specificity, embodiment, affect, space, and research-creation will be introduced. Then, two pieces are presented; they are emblematic of the context of the research. De Rerum Natura is an electroacoustic composition, which was recorded and written in the Amazon Rainforest. Music for Brainwaves is a live electronic composition based on brain activity, performed at the ex-NSA listening station in Berlin. The results of a survey conducted over five pieces, including De Rerum Natura and Music for Brainwaves, explores the typology of perceived spaces within the mental imagery of the audience. Next, the outcomes of the survey are recontextualized within two pieces: Bodyscape is inspired by the sound of the body of a dancer as initial material, and Carnets de Routes is based on the Brazilian travels of the Swiss/French poet Blaise Cendrars. The piece explores the possibility of embodiment through the writings of the poet. Finally, I shall provide a conclusion for this nomadic project and possible future research.

9.1.2 Background The seeds for this project were probably planted during the Notting Hill Carnival in London in the early 1990s. Little did I know about the presence of the sound systems

200

L. Forcucci

at the Carnival (Fig. 9.1), which were playing Jamaican dub, reggae or futuristic versions of this music. The sensation, while walking from one sound system to another, was literally like crossing walls of sounds. The experience was unique because, as invisible these walls were, the materiality of the sound was felt by my body. The materiality of sound here is related to physical perception of the pulsing bass of the sound systems. This first impression triggered the idea of an invisible (material) sonic architecture. It was not only the physicality of the sound as related to the space where it was perceived, but also the (sonic) imagery it triggered at the same time and how the perceptive experience of the sound was embodied.

Fig. 9.1 Sound System during Carnival in Notting Hill, London, UK, early 1990s (Photo: Luca Forcucci)

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

201

However, the focus here is not on Jamaican sound systems, although a good deal of literature on the topic exists, but on the sonic imagination that the sonic triggers within the audience’s mind. The sonic resonates with the body of the audience or the composer; consequently, it may generate visual mental imagery, depending on the degree of attention and listening dedicated to the sound. Goodman proposes that “this differential ecology of vibrational effects directs us toward a non-anthropocentric ontology of ubiquitous media, a topology in which every resonant surface is potentially a host for contagious concepts, percepts, and affects” (Goodman 2010, 79). The focus on the current project is, however, related to the perception of the human body through the sonic, thus will develop the idea of the tactility of sound and its perception beyond the ears in order to investigate the resonance of the body to sound and how it affects it. For Henriques, “thinking through sound thus evolves into a philosophy of resonance (…) it includes embodied practice and subjective sensory experience as well as the manipulation of mental images or cognitive process” (Henriques 2011, xxx). Mental images or cognitive processes are paramount points when addressing the subjective experience. However, there is no manipulation, but an exploration of a possible common pattern of visual mental image when an audience and the composer are exposed to a sonic experience and thus the relationships between intention and perception. Jasen suggests that a lack of communication between disciplines leaves “a vast and still largely unexplored region between the interests of the humanities and the sciences (…) we generally lack theoretical language suited to dealing with questions of (sonorous) materiality and sensory experience, not to mention affect and the incorporeal” (Jasen 2016, 5; 19). For the current project, expertise from the cognitive sciences informs perception and practice-led research is the main methodology. Moreover, the relationship between the author’s intention and the perception of the audience is equally a significant part of the project. Through the experience of the visual mental imagery as related to the sonic, an awareness develops about dynamic relationships between interior and exterior perceptions of one’s bodily experience with sound and space. Knowles and Cole suggest that “the central purposes of arts-informed research are to enhance understanding of the human condition through alternative (to conventional) processes and representational forms of inquiry and to reach multiple audiences by making scholarship more accessible” (Knowles and Cole 2008, 35). Porcello et al. posit that “transdisciplinary ethnographic studies of the senses are ideal sites in which to question the relationship among artefacts, technologies, personhood, and the body (…) as an ontological object of anthropological study” (Porcello et al. 2010, 61). Massumi states that “research-creation as embodying techniques of emergence takes it seriously that a creative art or design practice launches concepts in-the-making” (Manning and Massumi 2014, 89). Sheller underlines that “many researchers (…) are rethinking the form their work takes (how can text capture performance?), the linear temporality of traditional data-collection/data-analysis/output models (how can “outcome” capture process?)” (Sheller 2015, 143). The writing alone does not provide full understanding of the bodily experience of the sonic phenomenon. What kind of knowledge do we get when blurring the distinction between listening, experiencing, and knowing? The science and technology needed to achieve the finesse of

202

L. Forcucci

the visual mental image stimulated by a sonic experience are still in their infancy. Visual mental imagery relies on the subjective, internal experience and environment of one individual. Why investigate it, then? The ‘inner’ first person experience in relation to the ‘outer’ sonic environment may shed light on the mechanisms involved in listening—which are different from the mechanisms of hearing. The relationship emerging between internal bodily experiences triggered by external ones, and vice versa, forms a whole. Dedicated listening plays a major role as discussed, for example, by the late American avant-garde composer Pauline Oliveros, through deep listening (Oliveros 2005). She proposes an expansion to embrace all to which it is humanly possible to listen. Doing so leads to the phenomenal world that lies inside the auditory cortex and contains one’s space perception; Oliveros specifically proposed sound imagining in her sonic meditations (Oliveros 1974). Heightened listening observes each detail of sound: “his enables acute voice recognition, echo detection, spatial location, etc. Such heightened listening substitutes auralization for visualization (or seeing) by creating sonic pictures” (Oliveros 2010, 79). The sonic pictures relate to the idea of sonic architecture (or ‘auralized’ architecture) defined in the next section. She developed “practice that is intended to heighten and expand consciousness of sound in as many dimensions of awareness and attentional dynamics as humanly possible” (Oliveros 2005, xxiii). The notions of sonic pictures and sound imagining contribute towards the development of a new dimension in listening, composing, and perceiving sound. The heightened and deep listening modes predominate in the exploration of the relation of sound and space, and for the current study of visual mental imagery induced by sound. If the listener experiences the space and environment in her/his/their visual mental imagery while deep listening, then it is expected that s/he/they will also perceive her/his/their body as part of the environment that can be acted upon and thus impart a sense of agency. An ecological approach in sound perception (Gaver 1993) may provide information on the nature and content of what is perceived and imagined. However, the term ‘imagination’ relates to the visual, thus a new vocabulary needs to be developed in the future for the sonic. Oliveros proposes the term ‘auralization’, demonstrating that “[y]ou might begin to notice how your attention changes when you use auditory terms instead of visual terms to speak about sound” (2011, 167). Feld proposes the term ‘acoustemology’, which is “about the experience and agency of listening histories (…) a deeper engagement with the phenomenology of perception, body, place, and voice” (Feld 2015, 15–16), and that is “what I came to call ‘acoustemology’, one’s sonic way of knowing and being in the world” (Feld and Brenneis 2004, 462). The mediated experience creates spatial perception, where room exists for novel philosophy of auditory perception and sound imagination (Erlmann 2005; Cobussen et al. 2017; O’Callaghan 2017, 2010; O’Callaghan and Nudds 2010). In the upcoming artworks of this chapter, I shall propose perceived environments within visual mental imagery while deep listening, which is the internal experience as resonance of the body with sound. Varela proposes an embodied experience where, “according to the enactive approach, the human mind is embodied in our entire organism and embedded in the world, and hence is not reductible to structures inside the head” (Thompson and Varela 2001

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

203

cited Thompson 2005, 408). The idea of enaction is addressed here in the ecological interrelationships of the composer, firstly by engaging, exploring, and recording physically the sonic environment as intention in context; then, the resulting perception of the audience is enacted while (deep) listening to the content of the sound thus explored within visual mental imagery. The analysis of the experience of an artwork should include the author, too, and not only focus on the ‘beholder’ (Blanke et al. 2009). Here, the experience is contextualized by the drifting author into the environment, enveloping ten years of his life within twenty-three kilos, one item of hand luggage and one laptop folder; this led to the development of a series of artworks. The nomadic life certainly transformed the works according to site-specific contexts, as well as alongside the perceptions of different cultural audiences. From such a journey emerge questions about memory and consciousness when exploring perception in the sonic arts, and includes visual mental imagery as a first-person experience, from the author and from the audience. Sonic arts are understood as music that goes beyond notes, including composition as well as sound installations and performances in which sound plays a fundamental role and promotes a specific dedication to listening. The following terms are key in the development of the artworks introduced in the next sections: 1.

2.

3.

Listening is approached through deep listening, ‘a practice that is intended to heighten and expand consciousness of sound in as many dimensions of awareness and attentional dynamics as humanly possible’ (Oliveros 2005, xxiii). However, it also includes the modes of listening derived from the Oliveros’ approach (e.g. reduced listening, heightened listening). In addition, the German philosopher, Gernot Böhme underlines that “the best existing model for describing listening proposes that one inwardly re-enacts that which is heard” (Böhme 2000, 18). In this way, the acoustic space expands, by re-enactment, from outside to inside the body and back. Affect: Once one has listened, s/he/they are necessary affected, meaning that the sound reaches the ears and body of the listener such that before it perceives and receives, the body already experiences the sonic vibration. It is not only a physical phenomenon, but a social one. It is a relational action between one’s body (the sound) and another, the listener. For Massumi affect is “a way of talking about that margin of maneuverability, the ‘where we might be able to go and what we might be able to do’ in every present situation (…) The way I use it comes primarily from Spinoza. He talks of the body in terms of its capacity for affecting or being affected” (Massumi 2015, 22–23). Embodiment: The centrality of the world to embodiment has been a common theme of phenomenological thinking. Husserl’s notion of the ‘life-world’, for instance, highlights ”the intersubjective, mundane world of background understandings and experiences of the world” (Dourish cited Boellstorff 2011, 513). For Varela embodiment “encompasses both the body as a lived, experiential structure and the body as the context or milieu of cognitive mechanisms” (Varela et al. 1991, 236). Zavahi proposes that

204

L. Forcucci

the mind–body problem does not take the form of a metaphysical theory of mental causation, nor does it consist in an explanation of how the body interacts with the mind; rather, it seeks to understand to what extent our experience of the world, our experience of self and our experience of others are formed by and influenced by our embodiment (Zahavi 2012, 154).

4.

5.

Soundwalking in the urban and natural environment allows a comprehension of the site (and of its sound) and the listener/perceiver will become the author through listening, while being affected by sound. Site-specificity: The works presented here relate to the environment through a set of defined conditions, which are framed in order to deliver the aesthetic experience. The aesthetic experience develops through embodiment, which is the felt atmosphere as a relation between the viewer/auditor and the work of art, and Rebentisch proposes that: Installations are context-sensitive with regard not only to the interior or exterior space in which they are exhibited but also to the social frameworks that influence the reception of art in general (…) the aesthetic concept of installation must be thought of – in its ambiguity – as based on the concept of Ge-stell (Enframing) the process of aesthetic experience that ‘set up and set forth’ (…) The autonomous logic of the aesthetic can be understood only with reference to the structure of aesthetic experience (Rebentisch 2012, 221; 239).

The embodiment of artworks is linked to the performer’s and audiences’ experience of their own body as it is subsumed into the visual mental imagery. The artworks proposed in the following sections focus on the exploration of perception through the observation of visual mental imagery of spaces triggered by the audiences’ experience. It leads to the perceptions of the phenomenal world that lies inside the auditory cortex and relates to one’s personal architectural and environmental spaces. In this sense such experience is the artwork. Space is approached from an architectural angle, meaning that it is not only a container defined by borders and geometrical values, “the Receptacle not a Void” (Casey 1998, 33); rather, it is a sum of relations between architectural elements and sites (of sounds) “providing a situation [hedran] for all things that come into being” (ibid., 33). Casey further observes that “for if it is true that space is determined entirely by relations, then what matters most is not the size or shape of space, its capacity or volume, but the exact positions of the items related to each other in a given spatial nexus” (ibid., 182). The architectural approach refers to spatial relationships between sound and space, and how the resonance, reverberation, and vibrational properties influence sound in space and vice versa. Such relationships define a sonic architecture, which is an immaterial architecture, an architecture of atmosphere. The movement and the perception of the listener’s body in visual mental imageries are important components that allow an embodiment of the architectural sonic space through sound, which leads to its embodiment in visual mental imagery. In the current context, the idea of atmosphere relies on the relationships engaging sound, space (architectural and/or environmental) and the body. The sum of the relationships is felt and is actually the atmosphere that consequently affects and leads to perception within the body of the auditor. For Böhme, “perception is basically the manner in which one is bodily present for something or someone or one’s bodily state in

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

205

an environment. The primary ‘object’ of perception is atmospheres” (Böhme 1993, 125). Therefore, atmospheric architectures are felt, and the relationship is established between the architecture and the body of the viewer/auditor. One’s body consequently becomes part of the work. As examples of atmospheric architectures, the New Yorkers architects Diller and Scofidio + Renfro proposed their Blur Building at the Swiss Expo in 2002, which “is an architecture of atmosphere – a fog mass resulting from natural and manmade forces” (Diller et al. 2002). Dissimilarly, the Swiss architect Peter Zumthor designed the Swiss Sound Box for the Hannover Expo 2000, a pavilion made of wood stacks, and described by Kanekar as “more than a mere amplified and scaled instrument to encompass the human body – the instrument as building, rather in this case, it is the building that is an instrument – a device and an apparatus that orchestrates movement, sound, smells, vision” (Kanekar 2015, 83). The role of the movement of the body is paramount in Zumthor’s Swiss Sound Box architecture, and the related embodiment of space is made through sound, smells, and vision. In the work of the artist Olafur Eliasson there is the inclusion, in addition to the idea of architecture of atmosphere, of the notion of affect, described by Frichot as “the movement between emotional registers rather than the emotion itself once it can be named. Likewise, the percept is less about the named perception than what happens in the encounter that causes a pure percept to emerge” (Frichot 2008, 34). In the current project, ‘affect’ includes the body as mental image. Moreover, Massumi relies “on the irreducibly bodily and autonomic nature of affect” (Massumi 2002, 28). Massumi underlines that “affect is synesthetic, implying a participation of the senses in each other: the measure of a living thing’s potential interactions is its ability to transform the effects of one sensory mode into those of another” (ibid., 35). The emergence of synesthetic experiences in the works is the relation between the sonic part produced by the author and the mental imagery perceived by the auditor as architectural and environmental spaces; however, it is also the perception of one’s own body. Although this research focuses on sonic works, the relationship with light is interesting in order to apprehend the notion of immaterial architecture. In Eliasson’s The Weather Project, housed in the Turbine Hall of Tate Modern in London, “visitors were sprawled across the ground, transfixed by the looming interior sun and the subtle shifts in light and humidity, as well as their own images reflected back to them from the mirrored ceiling high above” (Frichot 2008, 34). The work not only transfigures the space (Tate Modern), but also immerses and affects the visitors. The sonic field of the actual project engages the auditor in order possibly to trigger the movement of her/his body into a sonic architecture through deep listening and perceived it as mental imagery. However, in reference to Douglas Kahn essays “Let Me Hear My Body Talk, My Body Talk”, the artist and critic Seth Kim-Cohen underlines Kahn’s thoughts about the role of the body, percepts and ambience: What the body means; the body’s role as a producer and/or receiver of signals; the body’s status as a component of the subject, as a discrete object, or as an entity that complicates this divide. Following from this, I want to think about how Cage, Turrell, and so many contemporary artists working with sound direct attention toward percepts, toward the sensory conditions of a given time and space. I want to think about how this turn toward a situation’s

206

L. Forcucci

ambience downplays other situational relations: issues of interiority and exteriority, real versus mediated experience, and how these relations instantiate power in one location, one actor, or another (Kim-Cohen 2013, 18–19).

In relation to Kim-Cohen’s proposal, sound is not approached here as a creator of ambience; rather, it is investigated towards the relationships between sound, space, body, perception, and issues about interiority, exteriority, and interstices. The term space derives from the French espace and Latin spatium. The latter, from the point of view of this project, refers to the Greek chôra as being the distance between objects, sites, and places. Casey underlines that “Chôra’ is ‘room’ that is filled, not vacant space (kenon) (…) Heidegger remarks that the Greeks experienced the spatial on the basis [of] chôra, which signifies (…) that which is occupied by what stand there, the place belongs to the thing itself. Each of all the various things has its place. That which becomes is placed in this local ‘space’ and emerges from it” (Casey 1998, 353). Chôra in the current project relates to the idea of space filled with sound and how it leads to i.) a definition of space by (moving) sound and ii.) a merger defined by the relation of sound and space. Therefore, the space is not a vacant space, but it exists because sound defines the space and vice versa. Sallis in Boellstorff cites Plato about chôra as “a mass of wax or other soft material on which the imprint of a seal can be made” (Sallis cited Boellstorff 2011, 515). In this sense, the project investigates the space occupied by sound and the kind of imprint that emerges from such relationship. Extensive philosophical questions defining the space and how it is perceived are proposed in The Poetics Of Space by Gaston Bachelard (Bachelard 1992). The development of Bachelard is based on the poetic representation of space as internally produced by the imagination. Bachelard interprets metaphorically intimate spaces by proposing the house as a symbolic view of the body. Therefore, the study of space within the humanistic perspective is the study of ideas and spatial feelings linked to sensation, perception and conception (Sanguin 1981, 568). Such ideas of sensation, perception, and conception are explored in the actual study by presenting five works to an audience through a survey. The answers are consecutively analyzed through methods based on phenomenology as developed at sect. 9.4. That is, the development of the artworks and the related answers from the participants form the methodology of the thesis as a perceptual analysis of an artistic practice.

9.2 Deep Listening to the Amazon Rainforest Through Sonic Architectures Every man should pull a boat over a mountain once in his life. Werner Herzog

De Rerum Natura is an electroacoustic composition based on field recordings from the Brazilian Amazon rainforest. The section focuses on the process of development

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

207

of De Rerum Natura. The approach is based on deep listening, a philosophy and practice developed by the composer, musician, and scholar Pauline Oliveros. De Rerum Natura also examines the merging of the composition with the acoustics of the performance space. De Rerum Natura explores the unique sonic nature of the Brazilian Amazon rainforest. The natural environment is dense in terms of sonorities; it therefore absorbs and influences the composers while soundwalking and seeking a place to record, or while recording. The ears and the mind are fully dedicated to listening. This leads to vivid visual mental imageries; these are later included in the composition as the first stratum of intention and transmitted to the audience during the performance. Thus, “to listen is to decode; it is to make sense of a sensory input” (Chare 2009, 254). The field is discussed through Guattari’s ideas on ecosophy as “an ethicopolitical articulation—that I call ecosophy—between the three ecological registers, those of the environment, of the social relations and of human subjectivity” (Guattari 1989, 12). These ecological registers appear in De Rerum Natura (Forcucci 2009) as follows: 1.

The (sonic) environment of the Amazon rainforest merges to create a temporary location that engages the listener in a sonic experience, where several spaces

Fig. 9.2 Location Example for Field Recordings, Mamori Lake, Amazon Rainforest, near Manaus, Brazil (Photo: Luca Forcucci)

208

2.

3.

L. Forcucci

and places from the Amazon rainforest are layered and recombined within the performance space. The social relations are those of the fauna from the Amazon rainforest and their sonic activity, since “animals are Deep Listeners. When you enter an environment where there are birds, insects or animals, they are listening to you completely. You are received. Your presence may be the difference between life and death for the creatures of the environment. Listening is survival!” (Oliveros 2005, xxv). Careful observation during listening and attention develop a tension as an experience while recording. Human subjectivity relies here on interpretation by the audience. The memories triggered of the Amazon rainforest by the author/composer transfer her/his/their impression of the tension in the sound, then translate this within the visual mental imagery of the audience.

Barry Truax underlines that “the real goal of the soundscape composition is the reintegration of the listener with the environment in a balanced ecological relationship” (Truax 1996, 63). The link between listener’s inner perception and the environment is indeed paramount.

9.2.1 Mapping of Virtual and Real Spaces De Rerum Natura emphasizes the listening experience with a dynamic link between the Amazon rainforest (Real spaces) and the perception of the audience during the performance (Virtual spaces). Virtual spaces rely here on the relationship between sound and space as a merger perceived within the mind of the listener (composer and audience). According to Dixon, visionary French theatre theorist Antonin Artaud was perhaps the first person to coin the term ‘virtual’, describing in The Theatre and Its Double (Artaud 1958) how “theatre’s virtual reality develops (…) [on the] dreamlike level on which alchemist signs are evolved” (Dixon 2006, 24). Artaud establishes a relationship between the action of the theatre and the subjective perception of the spectator. In that sense, sound does not actually carry its reality with itself. Instead, sound is comparable to a symbol that triggers virtual reality in the mind of the listener. The mind transforms those symbols into visual mental imagery, which is the subjective perception of the audience. Georgina Born stresses that “social mediation makes it possible to distinguish between the different degrees and kinds of co-present and virtual sociality, as well as of individuation and aggregation, privatization and public-ization, afforded by today’s ramifying musico- or sonic-social-technological assemblages” (Born 2013, 32). De Rerum Natura assumes Artaud’s signs more as triggering virtual reality and subjectivity, and less towards a virtuality associated with technological apparatus. The perception of the body includes important cues on the production of space as proposed by Henri Lefebvre, emphasizing that “it is from the body that one perceives

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

209

and lives the space and that it happens” (Lefebvre 2000, 190). The production of space develops in three ways within De Rerum Natura: 1.

2.

3.

During field recordings, the author/composer perceives the space while soundwalking, defining his compositional space by her/his/their own movements while deep listening. This is the first production of space, which is subjective and resides in the mind of the composer, leading to the initial layers of the composition; During the compositional process, the layers of sounds perceived while field recording are classified (into families of sound that ‘work’ together) and are composed in the studio, alongside that which his (the author’s) memory has released during the process of composition, and consequently what the audience perceives; The performance space, which already contains a quantity of spaces that I name internal spaces, receives the composition. Those internal spaces include an addition of reverberations, the sonic characteristics and sonic identities of spaces, places and locations. Thus, the addition of multiple spaces (by layering) of De Rerum Natura produces a metaphoric sonic moiré pattern, a polyphony of spaces, that also integrates the performance space with its own resonant properties.

9.2.2 Strategies of Recording in the Amazon Rainforest Field recordings took place during two weeks, between 6 and 19 December 2008 at the Mamori Art Lab, a sound lab near Manaus in the Amazon rainforest of Brazil. The recordings were made in the early morning, during the day or at night, during daily expeditions into the deep jungle on foot or by boat (Fig. 9.3). The Mamori Art Lab was led by the composer Francisco López. De Rerum Natura’s emphasis through deep listening during the recording process allows for concentration on the perception of the environment. The positioning of the recording equipment in the Amazon rainforest includes a sonic appreciation of the location. Regrettably, the recordings of wildlife suffer from our human presence. Therefore, three main recording strategies are defined: 1.

2. 3. 4.

Set up the equipment and wait for thirty minutes at five hundred meters’ distance—at least. While waiting, avoid making any noise, to allow nature to absorb the human presence, then deep listening among those waiting happens; Arrive by boat (pirogue) on location and set up the equipment; Wait for thirty minutes; deep listening also takes place; Set up the recording equipment, leave the equipment on location for several hours, return to the base camp for two hours, and deep listen while walking back to camp.

The waiting period while recording and careful listening increase the visual and sonic memories of the composer; those memories are the initial elements of the

210

L. Forcucci

Fig. 9.3 Boat Expedition for Field Recording on Mamori Lake, Brazil (Photo: Luca Forcucci)

future composition. The recording method follows precise steps in order to find the optimum locations, where no obstacles interfere, carefully listening to them and avoiding human presence. ‘Carefully listening’ to the locations means: 1. 2.

3.

4. 5.

To observe the environment’s acoustics (e.g. leaves, forest, open space, water, dry versus wet environment); To observe the diversity of the sound of the fauna with specific recording devices such as hydrophones to record pink dolphins or ultrasonic microphones to record bats; Typical birds, for example the Screaming Piha, are part of the sonic identity of the sonic environment of the Amazon Rainforest, as well as the spatial distribution of the howler monkeys’ call and response. Howler monkeys are active at different times, depending on many factors. In Mamori they tended to be more active typically very early in the morning (4–6 a.m., approximately). The recordings take place at specific times of day or night, according to the activity of the animals; The tension emerges while walking into the rainforest to avoid snakes, or when multiple eyes in the dark illuminated by the torch are those of alligators. The composers immerse themselves in deep listening.

The Amazon rainforest is so remote that one concentrates only on the recording task, doing so with absolute dedication. The dedication is so intense that the sounds

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

211

are deeply engraved into the mind. The powerful images created by the sonic environment while recording create the first visual mental images for the composer. It is a memory of space related to a mental imprint acquired during the deep listening sessions in the Amazonian rainforest. The imprint is the story told by the Amazonian space.

9.2.3 Background For Pauline Oliveros “sound impacts my body and resonates within. Sounds keep returning to me as I listen. Our vocabulary limits discussing inner or mental sound and sounding or listening in dreams and daydreams. We need words that highlight the auditory cortex” (Oliveros 2011, 163). The interpretation of (visual) mental imagery triggered by sound needs such vocabulary. Her practice of deep listening is central here. It originates in a highly reverberating underground tank where she practiced with her Deep Listening Band (Fig. 9.4), “[t]he cistern at Ft. Worden, Port Townsend 70 miles northwest of Seattle WA is a two-million gallon, 186-foot diameter water tank made of reinforced concrete” (Oliveros 2007). During a deep listening session, Listening for Peace, in Berlin (CTM 2016), she asked us, approximatively thirty persons, to sit on mattresses in the most comfortable position, and ‘don’t discriminate any sound, just let it happen’. At the note of a small

Fig. 9.4 Pauline Oliveros recording in the cistern at Fort Worden, Port Townsend, Washington State for The Readymade Boomerang (New Albion) (Photo: Gisela Gamper)

212

L. Forcucci

gong the session began. With closed eyes, and deep listening in the HAU 1 Theatre, I daydreamed. Time was no longer relevant. Tsabary proposes that “the practice of deep listening is concerned primarily with sound as a means for heightened environmental and awareness, devoid of analytical thought processes” (Tsabary 2009, 301). De Rerum Natura is influenced by composers’ integrating the performance space into the composition, by transforming the space itself with sound projection. Edgar Varèse anticipated sound projection in the performance space as early as in 1936, mentioning that “the entire work will be a melodic totality. The entire work will flow as a river flows” (Varèse and Chou 1966, 11) and “for the ear as for the eye, this phenomenon gives a sensation of extension, of travel within space” (Castanet 2007, 53). Maryanne Amacher integrated the architecture of buildings, specifically with her idea of ‘structure-borne’ sound, where the sound enters directly the structure and the architecture becomes an instrument. For De Benedictis “the whole of (Luigi) Nono’s output is based on the pursuit of new sonorities, requiring not only a different manner of experiencing sound (by performers and listeners), but also new configurations for concert venues” (De Benedictis 2013). The performance space is ‘augmented’. It is no longer the space per se; rather, it is illuminated by the sonic in such a way that the composition interacts and merges with the performance space.

9.2.4 On the Influence of the Performance Space De Rerum Natura exists once it is diffused and when the sounds dynamically incorporate the performance space during each concert. For Windsor “the motivation for adopting an ecological approach in this context is to redress the balance between abstract approach to musical structures and those that take into account the connections between sounds and the environment that produces them” (Emmerson 2003, 13). The combination of field recordings and the live manipulation of rich material from the rainforest in the performance space, as well as the technology that allows the live manipulation of extensive sections of material, lead to an evolution of a language and form of soundscape composition. The greater the reverberation into a space, the greater the play with silences and resonances. The resonance of the performance space with the sound (as input) is interpreted by the composer: firstly, by re-injecting the sound live captured with microphones (as output) and/or secondly, by playing with the diffusion of sound and the resonance. The perception therefore resonates with the performance space. The perception of the performance space’s architecture changes according to the perception of the sound in the minds of the audience, who are listening with eyes closed to focus on sound and visual mental imagery. It is a movie for the mind. The world premiere of De Rerum Natura on 21 August 2009 at Performance Festival, Belo Horizonte, Brazil was at Galpão Cine Orto, an old cinema from the 1920s transformed into a theatre. The audience sits in front of the proscenium. The set-up includes two microphones positioned on the sides of the venue to capture the room’s sound live and to send it back to the P.A. system (controlled by the

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

213

composer), creating a feedback loop. The PA is a quadraphonic system: two loudspeakers are at the front, and two rear loudspeakers sit behind the audience. While the side microphones capture the sound, the audience experiences a double perception of the performance space: the first is the illumination of the space by the diffusion of the composition, which the side microphones capture then subsequently re-inject into the performance space, with a small delay, through the rear loudspeakers, for the second perception.

9.2.5 Development of the Composition De Rerum Natura evokes a voyage through an unknown imaginary territory. The movements of the piece are balanced between real and unreal sounds, giving rise to powerful subjective images. The expedition was a particular and unique experience: it showed the contrast between a person’s initially arriving in the jungle full of the usual urban stress then quickly relaxing and forgetting everything except what was immediately there. The experience was of floating between reality and a dreamy state, a kind of unplugged or disconnected impression. De Rerum Natura includes an abstracted syntax, and a combination of aural and mimetic discourse following Emmerson’s grid of the language of electroacoustic music (Emmerson 1986, 33; Truax 2002, 7), where the syntax derives from the acoustic properties of the sonic materials themselves. Field recordings and manipulation of rich material from the rainforest (mimetic) are sometimes abstracted electronically (aural). The electronic manipulation includes convolution, a mathematical way of combining two signals to form a third signal, as well as bit depth and sample rate reduction. Seven movements structure the piece for a duration of 37’40”. As a live construction the length may vary, yet the movements stay. The piece progresses as a sonic continuum, close to what “Murail thinks of each composition as a continuous sound slowly metamorphosing (…) the composer thinks of sound masses and not of events” (Alla 2008, 254). The performance space is progressively transformed by the diffusion of sound masses. The greater the reverberation into a space, the greater is the play with silences and resonances. Luc Ferrari proposed that “to recognize the sonic sources equalled in my conception to a surrealist collage, meaning to make a succession of recognizable noises and not recognizable invented noises, as a surrealist poem, where one can see juxtaposed a sentence sourced as a citation and an invented word” (Gayou 2001, 29). The movements balance between two entities, both specific identifiable and subjective realities in the mind of the listener.

214

L. Forcucci

9.3 Music for Brainwaves: Embodiment of Sound, Space and EEG Data The essence and inspiration of Music for Brainwaves came from Alvin Lucier’s iconic piece Music for Solo Performer. Neurofeedback and physiological data (EEG) are explored as components in the relationship among sound, space, and the body of the performer. Although EEG registrations have been well explored for sound installations, music performances, and music cognition, relatively little is as yet known of the application of physiological data (EEG) within a system that includes the projection of sound into an architectural space, and the consequent embodiment, as inner experience, by a performer. Music for Brainwaves focuses on sound, space, physiological data (EEG), and perception. From a compositional point of view and in contrast to Lucier’s Music for Solo Performer, where EEG triggers a physical movement on percussion instrument, Music for Brainwaves explores a sound continuum based indeed on EEG data, yet generated by an algorithm. The sound continuum is based on Xenakis’s algorithm Gendy3 originally adapted by Nick Collins for the SuperCollider software and now ported to the Max/Msp software by Stephen Lumenta (Lumenta 2020). The algorithm produces a chaotic continuous sound that is, metaphorically and aesthetically, intended to reflect what the sound of the firing neurons might be.

Fig. 9.5 Ex-NSA Teufelsberg Listening Station, Berlin (Photo: Luca Forcucci)

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

215

Music for Brainwaves is based on an interface employed in collecting physiological data from a performer. Then, the algorithm processes the collected data, and the consequent generated sound is projected in a (resonant) performance space; the result is heard by the performer, and thus included in a neurofeedback loop. In Sect. 9.3.1, the definition of hyperbiological space will be introduced, and also how the performer primarily perceives it. Section 9.3.2 moves towards the definition of EEG, Brain Computer Interfaces (BCI), Brain Computer Musical Interfaces (BCMI), and neurofeedback. Section 9.3.3 introduces the origin of the piece influenced by Alvin Lucier’s Music for Solo Performer. It shows how, from the 1960s, such devices triggered curiosity among other composers, until today’s composers. The section ends on the raison d’être of the piece. From Sect. 9.3.4 to 9.3.6, the compositional process and the Gendy3 algorithm are analyzed. The section 9.3.7 introduces the ex-NSA Teufelsberg listening station in Berlin (Fig. 9.5), where the reception of the hyperbiological space was influenced through neurofeedback, and how this may relate to ancient sacred architectures in terms of resonance and perception.

9.3.1 Hyperbiological Space The hyperbiological space is an augmented peripersonal space, which is defined as “the space immediately surrounding our bodies” (Rizzolatti et al. cited Holmes and Spence 2004, 94) and proposed as a conceptual relationship of space, sound, body, and physiological data (EEG). The relationship between the performer/machine computer network delineates the hyperbiological space, where physiological data processed by computer are determined and sent as sonic information to the performance space where it is, in turn, received and processed by the performer. The performer’s reception of the sound closes the loop, causing the corresponding modulation of physiology known as neurofeedback. Vernon proposes neurofeedback as “A sophisticated form of biofeedback based on specific aspects of cortical activity. It requires the individual to learn to modify some aspect of his/her/their cortical activity (…) the amplitude, frequency and/or coherence of distinct electrophysiological components of one’s own brain” (Vernon 2005, 347). Accordingly, considerations in higher-dimensional spaces are not discussed here, nor are mathematical models proposed. However, research done in such areas is useful in understanding abstracted dimensions related to a space where biological information is included. In particular Lewis Carroll’s ideas about higher-dimensional spaces emerges from the great nineteenth-century German mathematician Georg Bernhard Riemann, who demonstrated that these universes obey their own inner logic (Kiku 1995, 22–23). The space sensed through neurofeedback by the performer includes the idea of the ‘higher-dimensional space’. Such sensed space appears when the performer hears, in a very resonant space, the sonic result created by his own EEG. The hyperbiological space is a complex dynamic subjective internal and external space in a closed loop provided by neurofeedback, and thus the cognitive-architectural space. Roy Ascott proposes the term ‘cyberception’ in explaining the relationship between our selves

216

L. Forcucci

and the mediated world, and how we are augmented and “involves a convergence of conceptual and perceptual processes in which the connectivity of telematic networks plays a formative role (…) It redefines our individual body, just as it connects all our bodies into a planetary whole” (Ascott 2003, 320–321). Although the experience of the augmentation of the self in Music for Brainwaves does not involve the telematic mediation through the Internet as proposed by Ascott, a network indeed exists in the relationship among body, physiological data, sound, and space through neurofeedback.

9.3.2 EEG, BCI, BCMI and Neurofeedback EEG The discovery of human EEG (Electroencephalogram) measurements, and in particular the alpha wave, were first discovered by the German neurologist Hans Berger in 1924 (Millet 2001, 522). The procedure of the measurement of the electrical activity of the firing neurons in the brain is known as EEG. The data are then filtered to obtain different frequencies, each of which has a different function. Without proper processing and filtering, EEG data are essentially random data. BCI (Brain-Computer Interfaces) Brain Computer Interfaces (BCI) monitor the activity of the brain measurements (EEG). In the last decade this technology has improved as an extension of the physical body in order to control computers, prostheses or wheelchairs. BCI has appeared in many other domains than the neuroscience and medical fields, such as video games, media art and music, among others. The next decade could unlock other opportunities, according to Lance et al.: Based on advances in sensor technologies, analysis algorithms, artificial intelligence, multiaspect sensing of the brain, behaviour, and environment through pervasive technologies, and computing algorithms will be capable of collecting and analysing brain data for extended time periods and are expected to become prevalent in many aspects of daily life (Lance et al. 2012, 13).

BCMI (Brain-Computer Music Interfacing) As an extension of the BCI, the BCMI is an interface that is specifically directed toward the production of music. The idea is directly inspired by Alvin Lucier’s piece Music for Solo Performer and has since then evolved according to the power of the computers and the availability of affordable devices (Miranda 2014, 1–27). For the actual work, the IBVA (Interactive Brainwaves Analyser System) BCI was used. The IBVA, developed by Masahiro Kahata, was one of the earliest affordable systems. It filters the raw data from the EEG into four frequencies, named alpha, beta, delta and theta, ranging from 2 Hz to 45 Hz (ibid., 202). Music for Brainwaves focuses on the alpha frequency, as in Alvin Lucier work’s Music for Solo Performer.

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

217

Neurofeedback By watching and listening to real-time multimedia representations of its own electrical activity (EEG), the brain can improve its functionality and even its structure (Budzynski et al. 2009, xxi).

9.3.3 Background The major influence of Music for Brainwaves came from Alvin Lucier’s piece, Music for Solo Performer. The points of convergence rely on the idea of control and energy data (EEG), present as a main component in Music for Brainwaves. The control of the body through inducing a meditative state includes the energy data transferred to an algorithm, as proposed by Lucier: Dewan described to me this phenomenon that had to do with visualization, that by putting yourself in a non-visual state, it would be called a meditative state now, you could release the potential of the alpha that is in your head. It’s a very small amount, but it would become perceptible, at least to an amplifier (…) Actually, it doesn’t sound like anything because it’s ten hertz and below audibility; it isn’t a sound idea, it’s a control of energy idea (Lucier 1995, 48; 50).

During the same period, composers and artists were collaborating with engineers to integrate new technology and thus discover new tools for new forms of music and art. A noteworthy example is a series of performances in New York in 1966 called 9 Evenings: Art, Theatre and Engineering. These events were developed by artists and engineers and “endeavoured to reassess a legendary series of ten experimental performances that were presented at New York’s 69th Regiment Armory on East 25th Street in October, 1966” (Garwood 2007, 36). Among others were the performances of Cage’s Variations VII: Cage was performing an unscored work for the first time, attempting a live broadcast of all the sounds in the world at once. Variations VII, like other Cage compositions, departed from art-making as a purely pictorial process and moved it toward the spatial, experiential, and conceptual. This particular work highlighted the soup of invisible frequencies in the realm of immediate experience (ibid., 40).

Music for Brainwaves relates to Cage’s work Variations VII through the “soup of invisible frequencies in the realm of immediate experience” described by Garwood, since the performance includes invisible frequencies as EEG from the body performer into the performance space. Consequently, Music for Brainwaves attempts to bring brainwaves into the musical realm, which includes the notion of making an invisible activity perceptible. David Rosenboom, one of the closest allied with brainwave music, and Richard Teitelbaum with his pieces Organ Music and In Tune from 1968 (Branden 2011, 132) and, later, Pauline Oliveros explored EEG for musical purposes around the same period. Rosenboom describes the first experiments from the 1930s with EEG technology:

218

L. Forcucci

While listening to his own alpha rhythm presented through a loudspeaker, Adrian tried to correlate the subjective impression of hearing the alpha come and go with the activity of his eyes (Adrian and Mathews, 1934). Inevitably, artists with an experimental bent would come to apply this – and subsequent developments in brain science – to both artistic production and research in artistic perception (Rosenboom 1990, 48).

Improvements in technology, and subsequent developments in brain science, lead to more affordable devices; advances in neuroscience research from the last decade provide a growing interest in and rediscovering of music produced with EEG. Multiple directions emerged. Among them, notable examples may be found in the work of: ‘Wellenfeld’ quartet The quartet includes Rudolf Eb.er, Joke Lanz, GX Jupitter-Larsen and Mike Dando. The sound and the procedures developed by this performance are close to those of Music for Brainwaves in the sense that “the performance took place without prior tests or rehearsals. The performer develops and gains control over his own brainwave patterns during the performance, by listening to the sonic results of his mental activity” (Eb.Er et al. 2014).

Stelarc In our correspondence, Stelarc responded thus about his use of EEG: My use of amplified body signals and sounds, including EEG – both for sound and control purposes – occurred primarily from 1972 to approximately 1986. These were amplified live as part of body installation performances (and from 1980 with my Third Hand). I do not have a music background although I’ve always used sound in my performances. So there are no scores as such. The performances began when the body was switched on and they ended when it was switched off. The sounds varied through partly physiological control (control of breathing, state of relaxation and tension and muscle tension) and partly through body fatigue. So a cacophony of sound was generated that varied in complexity and density depending on what sounds were switched on and off and were happening separately or synchronously (Personal communication).

Furthermore, and according to an interview in 2001 with Linz, Stelarc mentioned: In the late ’60s there was a lot of interest in biofeedback mechanisms (…) For me there was a desire to make sound as part of a body’s motion. It wasn’t a case of making the piece more dramatic through the use of sound, but rather that I’d always taken a multisensory approach to art. The premise of amplifying the body sounds was to articulate what’s happening inside the body, and the possibility of monitoring these signals enabled a kind of structural relationship. (Linz 2001)

Music for Brainwaves relies, in terms of sonic aesthetics and improvisation, on the performance piece Wellenfeld from GX Jupitter Larsen. Music for Brainwaves also relates to Stelarc’s views with its aim “to articulate what’s happening inside the body, and the possibility of monitoring these signals enabled a kind of structural relationship”. Music for Brainwaves is influenced structurally by action music, where “the composer prioritizes exploration of performative actions as opposed to investigation of particular sonic parameters in the creation of this music” (Kojs 2009, 286). As such, the work addresses the process of making decisions about artistic systems and

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

219

therefore how the body replaces the art object in determined environments, proposed by LaBelle as follows: Cage addressed the very act of making decisions, the artist being understood not so much as the maker of objects but as an individual in the act of making decisions as to what, how, and where art take place and the systems by which to initiate its production (…) The body literally comes to replace the art object, for it pushes up into the realm of form to such a degree as to explode definition and the literal lines of material presence (Labelle 2006, 54–55).

Following LaBelle’s claims, Music for Brainwaves includes not only the body as a work of art, but also the decisions from the composer articulated within the development of the performance through the following procedures and decisions: 1. 2. 3. 4. 5. 6. 7.

Find a very resonant space; Wear the EEG device and start the algorithm; Sit for as long as you feel it necessary; Then lie on the floor, for as long as you feel it necessary; Sit again for as long as you feel it necessary; Take a text and read it mentally; End the performance or start again from the beginning if you feel it necessary.

The dramaturgy of the piece, its raison d’être, resides in the notion of investigating the potential of a form of music generated directly by the inner experience of the body. How, in this context, can the audience perceive the movement and the influence of the brainwaves on the sound without relying on visual information? A direct relation between actions and reactions and their visualisation adds a too-predictable flavour to the composition—and probably a distraction from it. Instead, creating a focus on listening, in particular the relation between sound and (very resonant) space instead of the visual aspect of the work, demands a greater participation from the audience; when it is approached as a change in the sonic cloud with artefacts in the sound, it could lead to dedication rather than distraction, because the attention of the audience is directed towards few movements and events. During the development phase of the project, trials included other performers, such as a cellist and a dancer. The former was very convincing in terms of musical interactions and gestural presence towards the audience; the latter formally provided an interesting perspective, given the contrast between a moving dancer and an immobile EEG performer. Both ideas, and their visual contributions, were, however, abandoned in order to concentrate on a purely (hyperbiological) spatial relation among an architectural resonant space, sound, and physiological data (EEG) from the performer.

220

L. Forcucci

9.3.4 Composition Music for Brainwaves is based on the Gendy3 algorithm from Iannis Xenakis and implanted into the software Max/Msp. The performance relies on four different predefined parameters of Gendy3, which structure the movements of the composition. Each movement lasts three minutes.

9.3.5 Performance Actions for the performance are pre-determined yet improvised in length, according to the performance space and the audience. Improvisation gives a more flexible range of possibilities; the actions (i.e. sitting, lying on the floor, and reading a text) provide: 1. 2.

Different sonic gestures; Different states of consciousness, however limited by the duration of the performance.

The algorithm is based on aleatory procedures and thus the results are not always predictable; they are mainly derived from changes in the frequency, amplitude, and timbre of sound. The differences in brainwave activity modulate the sonic cloud from the Gendy3 algorithm in the resonating space and the resulting sonic artefacts alter the sonic continuum. No action provides a direct effect on the sound, yet there is always a delay. In the same order of ideas, Birringer proposes the sensation of mediations in Prehn’s work, as follows: Signals generated through electro-physiological monitoring of vital data (…) Prehn strives to concentrate not on semiotic processes of sense-making but on the immediate physical and emotional experience of the endo-movement, so to speak, the movements inside the body. For Prehn, such experiences are transcendental, ecstatic. They even resemble the hypnagogic trance states one might experience in a ‘ritualistic or liturgical’ context (…) The performer’s own immediate experience of multiple, simultaneous, fluid ‘phantoms’ of self or of the body’s signs in motion: the immediate sensations of mediations (Birringer 2008, 31–33).

The points claimed by Birringer and relevant to Music for Brainwaves are: 1. 2. 3. 4.

The interaction with the mediated environment ‘with signalsgenerated through electro-physiological monitoring of vital data’; ‘The immediate physical and emotional experience of the endo- movement, so to speak, the movements inside the body’; ‘The hypnagogic trance states one might experience in a “ritualistic or liturgical” context’; ‘The performer’s own immediate experience of multiple, simultaneous, fluid “phantoms” of self or of the body’s signs in motion: the immediate sensations of mediations.’

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

221

9.3.6 Xenakis’s Gendy3 Algorithm The choice of the Gendy3 algorithm is based on the quest for a sonic aesthetic that develops a chaotic continuum, metaphorically intended to reflect the activity of firing neurons. Also, the idea of the performance relies mostly on the improvisation of bodily pre-defined gestures (i.e. put on the EEG device and start the algorithm; sit for as long as you feel it necessary; lie on the floor, for as long as you feel it necessary; sit again for as long as you feel it necessary; take a text and read it silently; end the performance or start again from the beginning if you feel it necessary), which move according to pre-defined parameters derived from Gendy3. In these predefined parameters, the responsibility for generating numbers inside the algorithm is left to the computer. Roads shows that “Gendy makes sound by repeating an initial waveform and then distorting that waveform in time and amplitude. Thus, the synthesis algorithm computes each new waveform by applying stochastic variations to the previous waveform” (Roads 1996, 342). Gendy3 is based on probabilities and on stochastics. According to Xenakis: Stochastics studies and formulates the law of large numbers (…) the laws of rare events, the different aleatory procedures, etc. As a result of the impasse in serial music, as well as other causes, I originated in 1954 a music constructed from the principle of indeterminism; two years later I named it ‘Stochastic Music’. The laws of the calculus of probabilities entered composition through music necessity (Xenakis 1992, 8).

The term used by most people is ‘the study of probability along the time dimension’. Stochastic music is when “Xenakis started to use the computer to automate and accelerate the many stochastic operations that were needed, entrusting the computer with important compositional decisions that are usually left to the composer” (Serra 1993, 237). Serra defines the algorithm itself as follows: The program is based on an extensive use of stochastic laws. This creates a homogeneous composition in which the microstructure and macrostructure are conceived through the same perspective, i.e. filling sonic space with sound material and structuring this space are accomplished with similar means (ibid., 255).

In this sense, “filling sonic space with sound material and structuring this space are accomplished with similar means” has a strong metaphorical connotation with Music for Brainwaves: filling the physical performance space by using the sounds emerging from Gendy3 as a strong sonic impulse into the resonant space, to trigger the hyperbiological space, which in turn leads to neurofeedback.

9.3.7 Ex-NSA Teufelsberg Listening Radome, Berlin During the research process, different architectural configurations (e.g. an apartment, artist’s atelier, theatre, and university laboratories) were tested, but none of them was sufficiently resonant to sense the hyperbiological space, thus could not perceive the

222

L. Forcucci

relation between energy data (EEG), sound, and space through the whole body as neurofeedback. They sounded ‘dry’; a specific element was lacking from the neurofeedback, meaning that the sensation of the embodiment of the relation between sound and space was absent. A church or a cathedral was the first logical choice because both of the reverberant acoustic properties of such architecture and the way in which a musical past linked to the idea of a composer’s creating a piece with a specific building in mind. However, an unexpected solution (in terms of acoustic properties) appeared in Berlin in May 2014: the ex-NSA Teufelsberg listening station in Berlin. The first impression during the performance in this space was the embodiment of the relation of sound and space through the physical sensing of the neurofeedback. Here, the Teufelsberg listening radome is approached for its exceptional properties of resonance. For Merill and Schmidt “the field station was then used till 1990 by the U.S. Army & U.S. Air Force Intelligence together with the NSA for tapping and interfering with the radio communication of the Eastern Bloc” (Merill and Schmidt 2009, 23). Music for Brainwaves was recorded in the almost spherical radome, the name given to spheres used to protect and conceal radar antennas, on top of the highest derelict tower. Cox and Ings define the sonic properties of the Teufelsberg radome: Teufelsberg, on the outskirts of Berlin (…) a disused military facility contains ‘radomes’ – spheres used to protect and conceal radar antennas. The highest radome is on the sixth storey of a derelict tower. Jump onto the concrete plinth in the centre of the room, and any sound you make is focused back towards you. Sway to the right so the focal point is at your left ear, and the amplification afforded by the curved walls lets you whisper into your own ear (Cox and Ings 2014).

The acoustics of the place, the highest radome on the sixth storey mentioned by Cox and Ings, are so reverberant that it makes it difficult even to communicate with one another. When the sound of Music for Brainwaves was sent into the room, the neurofeedback became something unique that I experienced only in that particular location: the timbre of the sounds had changed, in contrast to all of the other places where it has been played, not only because of the resonant acoustics of the place, but also how it affected the amplification of my ‘reaction’ in the neurofeedback loop. It means that the unusual resonance to the sound created by the relation between the energy data (EEG), the transformation by the Gendy3 algorithm and its propagation in the highly resonant space, augmented my bodily perception, as an embodiment. However, to explain the embodiment of the relation of sound, space, and EEG data, in order to be fully understood, it must be experienced, as an inner experience. The process emerging from Music for Brainwaves is obtained through physiological data captured and processed by modern technology. The field of archaeoacoustics investigates the sonic properties of ancient architecture and provides a good deal of insights (Kolar 2013). Possible similar resonant issues as the ones experienced at the ex-NSA Teufelsberg listening station in Berlin seem already to have been discovered in ancient caves from the Neolithic period, for example, on Malta, as mentioned by Eneix:

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

223

Researchers detected the presence of a strong double resonance frequency at 70 Hz and 114 Hz inside a 5,000-year-old mortuary temple on the Mediterranean island of Malta. The − Hal Saflieni Hypogeum is an underground complex created in the Neolithic (New Stone Age) period as a depository for bones and a shrine for ritual use. A chamber known as ‘The Oracle Room’ has a fabled reputation for exceptional sound behaviour (…) resonant frequencies can have a physical effect on human brain activity (Eneix 2014).

In addition, those particular resonant frequencies are found also in other sacred locations around the world: Special sound is associated with the sacred: from prehistoric caves in France and Spain to musical stone temples in India; from protected Aztec codexes in Mexico to Eleusinian Mysteries and sanctuaries in Greece to sacred Elamite valleys in Iran. It was human nature to isolate these hyper-acoustic places from mundane daily life and to place high importance to them because abnormal sound behavior implied a divine presence (Eneix 2014).

Cook, Pajot and Leuchter suggest that further research should be conducted in order better to understand the links between resonance and emotional processing: Previous archaeoacoustic investigations of prehistoric, megalithic structures have identified acoustic resonances at frequencies of 95–120 Hz, particularly near 110–12 Hz, all representing pitches in the human vocal range (…) We evaluated the possibility that tones at these frequencies might specifically affect regional brain activity (…) These intriguing pilot findings suggest that the acoustic properties of ancient structures may influence human brain function, and suggest that a wider study of these interactions should be undertaken (Cook et al. 2008, 95).

Such pilot studies are of interest for future investigation in order to explore potential relationships between resonances, frequencies, and neurofeedback.

9.4 Proprioception in Visual Mental Imagery of Spaces While Deep Listening This section explores the sense of proprioception within visual mental imagery. The research is based on an experiment named In/Pe (Intention/Perception), developed by the author. The analysis of the data investigates the perception by an audience of architectural spaces, as well as natural environments, in visual mental imageries, emerging from a focused listening process of three fixed-medium pieces, one sound installation, and one performance. This highlights the idea of the experience of the artwork as a virtual constructed perception within one’s mind, an embodied experience that triggers the phenomenal world of sensation. The study examines what kind of spaces are visualized by the ‘beholder’. The overall agreement in how participants imagine spaces suggests that the perception is linked to their own bodies. This article discusses the results of a larger project developed by the author, exploring aesthetic and perceptual issues of the dynamic relations in sound and space perception (Forcucci 2015; 2017b). Participants in the Intention/Perception (In/Pe) project were invited to attend sonic art works from the portfolio of the author

224

L. Forcucci

in order to explore their own subjective visual mental imagery of architectural spaces (imagined three-dimensional spaces) by a focused listening, which leads to a sense of embodiment as inner perception of body image. If the listener develops a visual mental imagery of architectural spaces and environments, then it may be anticipated that the resulting representation includes, too, the beholder. If there is a common perceived space among the listeners, what does it represent? Nancy (Nancy 2007, 37) proposes that ‘it is always in the belly that we—man or woman—end up listening or start listening. The ear opens into the sonorous cave that we then become.’ Therefore, the first common memory of space among human beings might possibly be the womb. The representation of the body within a visual mental image is linked to its motor system (Gallagher 2005; Berlucchi and Aglioti 2009; Tsakiris 2009; De Vignemont 2010; Blanke 2012; Seth 2013; Dieguez and Lopez 2016). In addition, ‘[mental] imagery not only engages the motor system, but also affects the body, much as can actual perceptual experience’ (Kosslyn, Ganis and Thompson 2001, 641). Proprioception invigorates the idea that if deep listening induces spatial and visual mental imagery, it may include the whole body. Smetacek and Mechsner define proprioception as follows: Proprioception provides information on the physics of the body, the momentary distribution and dynamics of masses, forces acting on the limbs and their highly nonlinear interactions. The maps derived from these complex calculations not only guide body movement, they also (together with touch) sense the size and shape of objects and measure the geometry of external space (Smetacek and Mechsner 2004, 21).

Bringing the concept of proprioception into focused listening and visual mental imagery opens fields of possibilities where the audience is sensually involved with sound, which generates an embodied experience. Proprioception in the current study relies on the perception of one’s own body within the visual mental image. More fully explored in dance, proprioception could enhance the perception of sonic arts in a potential rethinking of the notion of spatialization. The artistic and theoretical background is introduced in Sect. 9.4.1. The section moves towards deep and heightened modes of listening. Next, space is defined, along with notions of affect and atmosphere. Section 9.4.2 continues with the introduction to the Intention/Perception (In/Pe) project, where five electroacoustic pieces were presented to an audience for focused listening in order to stimulate the experience of a spatial representations in visual mental imagery. Section 9.4.3 introduces the portfolio of works for the research. An interpretation of the results is introduced at Sect. 9.4.4.

9.4.1 Background The In/Pe project mapped visual mental imagery in order to foster the link between intention (composer) and perception (audience). The inquiry relies on the process of the development of the artworks not as a sole project, but in relation to and in

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

225

osmosis with the audience perception. Those ideas are central to the current study. Sound triggers purely fictitious and illusory worlds, as a constructed virtual reality in the mind of the listener. The mind transforms sound into visual mental imagery, which is the subjective perception of the audience. Consequently, as an inner experience, the body becomes the theatre where the work appears. The Brazilian neo-concretists Lygia Clark and Hélio Oiticica included the body of the audience in their works. With her relational objects, “Lygia] Clark’s experiences tend to merge the body’s interior and exterior spaces, stressing the direct connection between the body’s physical and psychological dimensions” (Osthoff 2004). The connection between external and inner spaces of the body explored by Clark emphasises the active participation of the audience’s body. Moreover, Hélio Oiticica pioneered works such as Parangolé, where the body of the participant becomes the theatre of the experience. In this sense, Oiticica in Notes on the Parangolé (Anotações sobre o Parangolé) mentions that “the spectator ‘wears’ [veste] the cape, which is made of layers of coloured cloth that appear to the extent that he moves, running or dancing. The work requires direct corporal participation” (Oiticica 1965, 93). The participant embodies the work in Parangolé and becomes part of it. The composer Maryanne Amacher’s work included pieces, such as Music for Sound Joined Rooms and Mini-Sound Series, directly relating to architecture (as structure-born sound), which allows the audience to embody the sound through space by exploring it during her performances. She composed using physiological properties, where the body is the sonic emitter and receptor. She claims that “when played at the right sound level, which is quite high and exciting, the tones in this music will cause your ears to act as neurophonic instruments that will seem to be issuing directly from your head” (Amacher in Ouzounian 2006, 74). In an interview with Oteri she discusses the phenomenon of otoacoustic.1 They are directly produced by the listener, who ‘actually has vivid experiences of contributing this other sonic dimension to the music that their ears are making (…) it has to be done in such a way that the sonic shapes are lingering in your mind afterwards’ (Amacher cited Oteri 2004). The embodiment of sound she proposes is a very sensual one. It is an experience that the listener will hear through her/his/their own body, a very personal experience, a response of her/his/their ears to a composition; the response is the work produced by her/his/their own bodily reactions. The artist Max Neuhaus, who developed mainly sound installations, also created situations in which the participants embody the works. For example, in his Water 1 ‘In 1992 I read a remarkable short article: Ear’s Own Sounds May Underline Its Precision – A Tiny

Loudspeaker Inside the Ear (New York Times) by Dr. William E. Brownell, of the Johns Hopkins University School of Medicine. A leading researcher in “otoacoustic emissions,” or OAE, sound which is generated from within the inner ear, Brownell reported dramatically: “Physiologists are still marvel- ing at the discovery that ears produce sound. It is almost as astonishing as if the eye could produce light or the nose produce odors.” And further: “A person who fails to emit sounds from his or her ears in response to a test tone generally turns out to be deaf, or suffering from disease or the influence of certain drugs. Significantly this response disappears a few minutes after death. This, many scientists believe, implies that the otoacoustic response is the result of ACTIVE SOUND PRODUCTION, NOT JUST A PASSIVE ECHO OF EXTERNAL SOUND’ (Amacher in Zorn 2008: 11; emphasis by Amacher).

226

L. Forcucci

Whistle series developed between 1971 and 1974, the sound is transmitted to the whole body, which is immersed in water. Neuhaus explains that “a network of hoses fed water through a configuration of whistle-like devices, each enclosed in a reflector (…) This formed a shifting sound texture, which varied, according to the listener’s position in the pool” (Neuhaus n.d.). In this piece, Neuhaus considered the body not only as the interface between the sonic piece and its perception, but also as the medium. Again, the artwork is then solely one’s own personal corporeal sensual experience.

9.4.2 In/Pe Project The Intention/Perception (In/Pe) project is an empirical survey informed by phenomenology. The qualitative phenomenological methodology (Forcucci 2015, 38; 194), however, not presented here, relies on a questionnaire investigating the experience of spatial visual mental imagery of the participants exposed to the portfolio of sonic art works. In Husserlian phenomenology it is claimed that “our experience is directed toward – represents or “intends”—things only through particular concepts, thoughts, ideas, images, etc. These make up the meaning or content of a given experience, and are distinct from the things they present or mean” (Stanford Encyclopedia of Philosophy 2013, 2). The In/Pe project investigated the experience of the audience’s ‘ideas, images’ (as perceptions) of architectural and environmental spaces, which make up the meaning, and it is thus distinct from the compositions per se (things they present or mean). Merleau-Ponty “focused on the ‘body image’, our experience of our own body and its significance in our activities (…) In short, consciousness is embodied (in the world), and equally body is infused with consciousness (with cognition of the world)” (ibid., 11). The aim here is to observe how the participants perceive their body (image) in the constructed virtual architectural and environmental spaces enacted by the listening experience. Husserl and MerleauPonty “spoke of pure description of lived experience” (ibid., 4), which is the lived experience, through listening, of the spaces currently being investigated. In In/Pe, the compositions are played only once, and thus the first impression is observed, in line with the proposal about the affective corporeal involvement of the philosopher Tonino Griffero: The first impression is an affective corporeal involvement that, interrupting the habitual observational and pragmatic flux, can, for this very immediacy, represent for the subject an identity certificate much better than the cogito – and, even more so, better than objective facts, which as such are ours as much as others’, in principle (Griffero 2014, 88).

The investigation explores first-person experience and how such experience affects the audience. Mental imagery visualization of space through heightened listening needs consistency of attention from the participants. Such consistency is

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

227

fragile and subject to shifts of attention. Following Griffero’s claim, the first impression is “an affective corporeal involvement” and relies here on the auditor’s perception of inner space. A majority of the participants in the survey were chosen mainly from among students in music, sound studies, media art, and fine art, as well as established artists, musicians, composers, and curators; only a small fraction were not professionally involved in the arts. The reason for the selection lay in their inclination and their specific training in spatial representation. Artists in these disciplines usually have a clearer mental representation of space, because their practice requires constant and intense visualization resulting in visual mental imagery (e.g. imagining playing a particular section of a piece or imagining a person or an object being drawn or sculpted). Thus, the choice relies primarily on their potential respective abilities “as artists to represent objects in space, and for musicians, because they perform well in visuo-spatial tasks” (Brochard, Dufour and Després 2004, 103–4; 106–8). While the relation between their professional backgrounds as regards their perception is not analyzed, it nevertheless provides information about their familiarity with experimental forms of art, music, and spatial perception. The investigation was conducted without a control group; rather, it was based on observations through individual experiences with a questionnaire. In addition, most of the answers were provided during fieldwork, where organizing any contributory features of control groups is problematic. Between 23 and 35 candidates per piece, not always the same individuals, were invited to listen, for no longer than 35 min per session, in darkened spaces, to each of the five pieces of the portfolio (the pieces were never played in the same order). They were recruited while in art residency in China and at universities in the UK, Germany, and Brazil. The participants were introduced to the aims of the research before the listening sessions. They were asked to concentrate on their listening and on the visualization of architectural spaces while listening. No prior information about the pieces and the questions in the questionnaire was provided; this was omitted deliberately in order to avoid influencing their answers. The participants themselves filled out the questionnaires in writing. All participants gave written consent for the study, according to the protocols of the human research ethics policy of De Montfort University, Leicester, UK. DMU’s relevant committee approved their written consents. None of the participants was remunerated.

9.4.3 Portfolio The author, who is also the researcher, developed five pieces. Although this may appear to have problematic aspects, it acknowledges the importance of including the author when studying others’ perception of art. The reason lies within the investigation of the subjective nature of the phenomena as inner experience, which necessitates first-person subjective experience. In this case, the researcher must have experienced for himself the nature of the relationships between inner and outer spaces that are

228

L. Forcucci

proposed to the audience. Moreover, the process, as practice-led research, observes not only the development of the artworks, but also the inherent audience perception, and the quality and typology of the inner spaces; hence the non-orthodox nature of the experimental design and procedure proposed here. Ideas about deterritorialization and re-territorialization are addressed, since “Reterritorialization must not be confused with a return to a primitive or older territoriality: it necessarily implies a set of artifices by which one element, itself deterritorialized, serves as a new territoriality for another, which has lost its territoriality as well” (Deleuze et al. 1987, 174). The study of spatial visual mental imagery is indeed a new territoriality within the mind of the participating audience, which has abandoned its external territoriality to become a constructed virtual inner space. The focal points of the portfolio were the following: 1. 2. 3. 4.

deep listening to sounds and spaces; Plasticity of sound as a relation to visual arts; Embodiment of sound and space; Visual mental imageries of sound and space.

Number of participants and short description of the portfolio’s pieces: The Fall—35 participants attended the listening sessions. The Fall is an eight-channel piece exploring the plasticity of sound, as a sculpture, and conceived to resonate with a physical sculpture as a sonic counterpoint. Moreover, the piece integrates the sounds of the site where the sculpture is located. De Rerum Natura—30 participants attended the listening sessions. De Rerum Natura is an electroacoustic composition based on field recordings from the Brazilian Amazon forest. My Extra Personal Space (Forcucci 2017a)—33 participants attended the listening sessions. The soundscapes of the city of Paris and the coastal town of Etretat on the Normandy coast in France are recorded during an exploratory soundwalk blending urban and natural environments. Kinetism—23 participants attended the listening sessions. Kinetism invites the audience to take a sonic walk, where people might realise the dialectic between the internal and the external perceptions of space through sound: one’s own body’s sound (internal auditory space), mostly unconscious, is balanced against the sounds of the city (external auditory space). Music for Brainwaves—30 participants attended the listening sessions. The piece is based on an interface employed in collecting physiological data from a performer. Then, the algorithm (based on Xenakis’s own Gendy3) processes the collected data, and the consequent generated sound is projected in a (resonant) performance space; the result is heard by the performer, and thus included in a neurofeedback loop. This work was inspired by Alvin Lucier’s composition Music for Solo Performer.

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

229

9.4.4 Interpretation of the Results When comparing the results of the survey among all of the pieces in eliciting perceptions of architectural spaces, similar patterns appeared: 1. 2. 3. 4. 5.

The Fall: smaller spaces such as a cave, tunnel and pipe were triggered by the sound of the dripping/flow of the water in them. De Rerum Natura: a dark, confined space (such as a cave and underground). My Extra Personal Space: an internal space (a building, studio, beach house, or shelter). Kinetism: an enclosed space (inside the human body, a box or a room). Music for Brainwaves: a room (apartment, closed room or dark room).

Similar typologies of small architectural spaces appear in visual mental imagery of the candidates when cross-comparing the answers. The mental representation of space through perceptions of the body links to what Woelert calls ‘the body as the generative interface’ and ‘human conceptual thought’ (Leroi-Gourhan in Woelert 2011, 115–16). The body acts as the interface between the physical movement into the external environment and the inner mental perception of the world. That is, exterior environment and ‘human conceptual thought’ (here, visual mental imagery) are not two separate processes; they are closely related and form a whole.

9.5 Bodyscape Bodyscape is a composition and a performance exploring the body as a sonic landscape. A sensor collects and measures the data generated by a dancer’s movements. In this ecosystem, the dancer produces sounds, too, mainly inaudible because they are mostly frictions of the body with soft and non-conductive surfaces or delicate movements inducing almost no sound; these are amplified and live-processed electronically, and sent back to the performance space, where the dancer interacts with them as biofeedback. Biofeedback is feedback about one’s physiology. The sitespecificity of the work relates to the spatial acoustic properties and typology of the performance space in relation to the dancer’s movements. The creation of the piece was a work in progress that changed during its three-day installation at The Lab gallery in San Francisco, US (The Lab 2015), where it was performed by musician Cheryl Leonard, dancer Crystal Sepúlveda, and myself at the beginning of August 2015 (Fig. 9.6). Bodyscape is the first piece inspired by the survey’s responses introduced in Sect. 9.4.4. The survey provided insights into the embodiment of sonic artworks and visual mental imagery characteristics as the audience’s perception. In fact, the interest lies not in exploring one’s mind content per se, because it constitutes a metaphorical uninvited intrusion into someone’s house and therefore causes obvious ethical problems. Instead, the information gained during the processes of the development

230

L. Forcucci

Fig. 9.6 Crystal Sepúlveda, The Lab Gallery, San Francisco, US (Photo: Swissnex San Francisco)

of the previous works and the outcomes of the survey are compositional information for my next generation of works. The idea resides in establishing relationships between the sonic, on the one hand, and the audience’s perception of spatial typologies within their very own visual mental imagery, on the other, as sonic imagination. That which was learned from the investigation into the perception of sonic arts and the resulting visual mental imagery is found in the centrality of the role of the body image. Gallagher proposes the body image as “a mental construct or representation, or a set of beliefs about the body” (Gallagher in Bermudez et al. 2001, 226). I must immediately clarify that the present study relies on subjective perceptions and qualitative methods as structured interviews, which are conceptually different from cold, scientific quantitative approaches. The answers are not measurable outcomes of body image; instead, they are interpretations based on the answers given in the survey. In addition, casual discussions with the audience after concerts, or after their having experienced my sonic installations, often report consistent sensations of travel, where some sonic frequencies were ‘moving’ them. The investigation and progression of the visual mental image as related to the sonic coupled with practice-led research is in its infancy, yet I am confident, on the basis of the research conducted so far, that the body image and embodiment are indeed central to further understanding. I can only speculate on this at the present juncture, with this very first work emerging straight after my research into visual mental imagery.

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

231

9.5.1 Processes of Development The work was developed during a residency at the Djerassi Foundation in Woodside, close to San Francisco in the US to which I was honoured to be invited to participate. The one-month experience was a unique combination of artists and scientists, the Scientific Delirium Madness, a collaboration with Leonardo/ISATS (Knight 2015, 219). We (thirteen artists and scientists) were asked just to ‘be’ during the month of July 2015, where this beautiful location offered a daily view of the Pacific Ocean (fog permitting). My idea was to work on a performance project and electroacoustic music based on the body of a dancer as initial source. However, at the end of that month, I had worked on a permanent site sculpture entitled Silence Capsule within two redwood trees, inspired by Michael Jackson’s deprivation tank, in collaboration with the artist Christine Lee. (A deprivation tank is a soundproof pod where one is floating in saltwater and in the dark.) Composed of the shared space between two redwood trees, the sculpture becomes a natural deprivation chamber, where sounds experienced by the participants may be partially perceived and/or imagined. A manifesto related to the piece can be found online by accessing the etched 2D barcode displayed inside the sculpture. Also, I proposed a soundwalk in the trails of the property during the open studio event. I had the idea of a nomadic lab for art and science, which today has already achieved seven editions and eleven podcasts on three continents (ubqtlab.org 2020), and the performance named Bodyscape. During the residency and before the meeting with Crystal Sepúlveda, choreography and dance performance, and Cheryl E. Leonard and her natural-object instruments, I worked mainly on the composition and on the calibration of the sensor. The sensor, an EMG (electromyogram), is made by the Portuguese brand Bitalino (https://bit alino.com/en/) to measure the electric activity of the muscles. Also, I worked on the sonification of the data received from the sensors through a Max/Msp patch, and on field recordings that would ‘work’ well together sonically in order to develop a coherent whole between the dynamic of the gestures of the dancer, the interaction with the musician, and the sonic outcomes. However, my development needed to be restricted so as to provide adequate space in the composition for both performers. The idea was to work on a composition with enough space for improvisation. It would however contain certain predefined actions and take place in the space of the gallery. In early August 2015, the performance was developed on site at The Lab gallery of San Francisco. Over three days, we worked collaboratively with the dancer and the musician in the gallery. We defined the following actions with the dancer, which would in turn define the interaction between us three: 1. 2.

3.

Move from this location of the gallery to this one and play with a piece of black textile. When you think you are done, stand still; While Cheryl brings stones and plays with them in the centre of the gallery, interact with her, the stones, and the architectural space of the gallery. When you think you are done, stand still; Let me attach a piece of plastic and a contact microphone to your forearm, and play with it;

232

4. 5. 6. 7. 8. 9.

L. Forcucci

Let me attach a sensor to your leg, and move with it; Grab the hydrophone and take it with you; Lie on the floor and put the hydrophone on your heart; Breathe, deeply breathe; Slowly stand, and play with a sand bag; Come back to the initial position.

Bodyscape was then proposed to the Centre Dürrenmatt in Neuchâtel, Switzerland (Centre Dürrenmatt 2016). Friedrich Dürrenmatt was a Swiss author and painter, with a great interest in science. He wrote several important theatre pieces such as La Visite de la Vieille Dame. The Centre Dürrenmatt museum was built next to his house in Neuchâtel by the Swiss architect Mario Botta. As a commissioned work, the director of the Centre Dürrenmatt asked me to integrate a text from Dürrenmatt within Bodyscape. In November 2015, I was planning to travel to South Africa for three months. The project included fifteen days field recordings on the border with Botswana in the Limpopo Region, accompanied by the biologist and composer Francisco Lopez. I conducted research further into other regions of South Africa, and L’épidémie virale en Afrique du Sud (Dürrenmatt 2007) by Dürrenmatt informed the journey. The story is situated during the apartheid period, although Dürrenmatt never went to South Africa. His novel was published originally in the column of a Swiss newspaper. It is the story of a virus that transforms the bodies of white persons into black ones thus is a text about privilege and how this applies within a specific context. I explored the country with the book in mind, observing the division twenty years after the end of the apartheid. I collected picture, video, and field recordings. When I travelled back to Switzerland in January 2016, we met up with the dancer Crystal Sepúlveda at the Centre Dürrenmatt of Neuchâtel and began working on the same procedures as we had in San Francisco: defining a number of actions within the building of the museum. The procedures would define the performance. The composition includes layers of cut-up text, pictures, video, and sound composed into an electroacoustic piece, as in a road movie (Fig. 9.7). We worked in different parts of the museum, using an overhead projector on wheels. Crystal danced through the architectural space and placed the various images from the journey in South Africa onto it. The difference between the performance in San Francisco and that in Neuchâtel lies mainly in the definition of the actions, and the architecture of the performance space. In San Francisco, we defined movements and actions in sequence. In Neuchâtel we deliberately worked in the appreciation of the architecture of the building, which defined situations and actions. The sensor was the same model used in San Francisco, as was the sonic composition. The greatest difference was that we asked a cellist to join us, because of financial constraints thus the impossibility of flying Cheryl over. The sound recordings of the performances in San Francisco and Neuchâtel were then composed at NOTAM in Oslo Norway in May 2017. NOTAM is the Norwegian center for technology, art, and music (NOTAM 2016). The idea for the composition was to develop a live electronic eight-channel electroacoustic piece to be played in the dark, images and videos appearing sporadically. The visual material came from

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

233

Fig. 9.7 Crystal Sepúlveda and over projector at Centre Dürrenmat, Neuchâtel (Photo: Olivier Chételat)

the performance in Neuchâtel, and were edited as a film. For the composition, I took the two recordings of the performances in San Francisco and Neuchâtel and layered the recordings on top of each other inside the DAW (Digital Audio Workstation). The envelopes, the equalizations, and the volumes of the two tracks were reworked, leading to new sonorities emerging from this new combination. Also, I processed samples of the recordings with the GRM tools’ plug-ins from INA/GRM, and thus added a third layer with the processed material. The concept behind this new version of Bodyscape is an architectural journey between the two locations, the dancer moving between them. The first concert with this new version of Bodyscape, without the performers although with sound diffusion for eight loudspeakers, was premiered at the ISEA (International Symposium of Electronic Arts) in Manizales in Colombia in June 2017 (ISEA 2017, 5). The piece was performed at CCC Teatro Los Fundadores, a theatre with a capacity of 1,232 seats. Bodyscape was then played in several other contexts around the world until the last performance, so far, in December 2018 in Salvador de Bahia in the School of Fine Arts of the Federal University of Bahia (Forcucci 2018).

234

L. Forcucci

9.5.2 Outcomes Because the performance is site-specific and relates to the movements of the dancer within each performance space, I struggled with how to develop an album from it. I finally decided to publish it as digital and in ten limited copies of vinyl (Forcucci 2020). The release includes three tracks named Bodyscape, Bone, and Cell. The project was recorded at The Lab Gallery in San Francisco, USA by Cheryl E. Leonard. The live recordings from Centre Dürrenmatt in Neuchâtel, Switzerland, the field recordings made at the Mmabolela Reserve, Limpopo, South Africa, the mix and composition at NOTAM, in Oslo, Norway were made by me. Bodyscape was a finalist of the Luigi Russolo Prize 2019.

9.6 Carnet de Routes: Following Blaise Cendrars in Brazil It is from the body that one perceives and lives the space and that it happens Henri Lefebvre A kind of aesthetic osmosis between the artist and the viewer via the artwork Marcel Duchamp

Carnet de Routes is a research work based on the travels of the Swiss/French writer and poet Blaise Cendrars, whose real name was Frédéric Louis Sauser. The work is the second piece inspired by the outcomes of the survey into visual mental imagery. Carnet de Routes explores through his texts and the idea of transferring such impressions to the audience the embodiment of someone else’s experience of a journey. The aim is to develop a sonic practice by exploring the multimodal possibilities of sound and the plasticity of the sonic in relation to territory, architectural space, body, and perception. Many forms are combined: an electroacoustic composition, a sound installation, a performance, photography, poetry, writings, and most importantly the interrelationships between them all and their perception by the audience. The proposal relates to i.) the visual possibilities of the sonic and ii.) to leave to the audience my responsibility as composer, which in turn recomposes the sonic away from the loudspeaker. In this sense, Roy Ascott proposed a series of pictures named Change Painting, where the composition of these interactive constructions changed over time as viewers-participants altered the works by sliding Plexiglas panels along a horizontal axis, bringing the image painted on each panel into myriad configurations with regard to the images painted on the others. The work itself entailed a durational aspect comprised of process, behavior, and change (Shanken 1997, 66).

The audience is invited to move the sliding panels on which are the paintings, and this translates the idea of Carnet de Routes, where the work is left to disposal of the

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

235

audience, who interact with it and thus create novel possibilities with the proposed elements. Another approach is that of Marcel Duchamp and The Large Glass, from which different versions of the same piece are obtained. According to Shanken, “in Duchamp’s Large Glass, the marriage of technology and intuition is characterized by a constant shifting between transparency and reflection, ritual union and possessive obsession, unconsummated desire and metaphysical impregnation” (ibid.). Carnet de Routes consolidates the composition of soundscapes, the texts, and the pictures by including the perception of the audience. This means the artwork is not a final object per se, it is the result of the addition of all or some elements of the work, depending on the locations where the piece is held. This leads eventually to a perception within the audience’s mind of the final work. Carnet de Routes explores issues of serendipity, psychogeography, affect, and embodiment. The sonic magnifies situations and fields of possibilities beyond phenomenological considerations and the creation of ambience; rather, it is directed towards philosophical and anthropological issues of interiority and exteriority. For example, in May 2016, I arrived in Brazil to develop the work; two days later, the then President Dilma Rousseff was impeached. Over the next three months protests arose, and the words and writings from the streets slowly entered my mind and works. These, I may in turn have transmitted to the audience through my works. The point of departure of the project Carnet de Routes starts with texts about the Brazilian travels of the French-Swiss writer Blaise Cendrars by poets (e.g. Oswald de Andrade), researchers, and his own texts. The travels took place in the cities of Rio de Janeiro and São Paulo, and in the region of Minas Gerais. In his book Moravigne, there is mention of a fictional journey into the Amazon rainforest, where his character felt ill and was cured by Indigenous communities because they saw him as a deity (Cendrars 1926, 190). I had my own experiences in each of those locations and some of his experiences coincided with mine. Furthermore, the memory of Blaise Cendrars includes the city of La Chaux-de-Fonds in Switzerland, where we were each born at different times. Brazilian friends commented to me on the Brazilian travels of Cendrars, of which I was not previously aware, nor of the notoriety of the writer in Brazil. Cendrars was sympathetic to anarchism and it can be perceived in his book Moravigne (ibid.) for example, where chaos spread from Berlin to the Amazon. These political positions may have existed when he was travelling with the Modernists, who were close to the aristocratic ‘milieu’ and related to the monoculture of coffee, a ‘milieu’ in which he had little interest per se. He was invited to Brazil by his friend Paulo Prado, a descendant of coffee planters, and also a writer, art patron and collector, which explains part of his connection with Cendrars. Cendrars first went to Brazil in 1924, and travelled with a group of Brazilian Modernists (Mário de Andrade, Tarsila do Amaral and Oswald de Andrade, among others). He would call the country ‘Utopialand’ and describe it as his spiritual home. It seems to me important to underline that the Brazilian Modernists were linked to the aristocracy and bourgeoisie of the coffee industry. The “descendents of the earliest coffee planters, such as the Prados, sought to secure social status by taking possession of colonial history. They sensed that rapid socioeconomic transformation would sooner or later

236

L. Forcucci

cause the economic and political downfall of the ruling oligarchies” (Gouveia 2009, 102). The coffee industry is significant of the problems faced by Brazil, namely, the promotion of a monoculture encouraged by Western corporations among other societal and systemic problems such as the “mass destruction of the Amazon jungle, the repression of an economic underclass, the solidification of a police state, and the eradication of previously isolated Amerindians and their diverse cultures” (Noland 2000, 406). Although this section is about the artwork, it is vital to mention the context existing then and up until today. Cendrars’ invitation to Brazil by Prado was linked to Cendrars position as a leading figure in the Parisian avant-garde. Cendrars, instead of discussing Parisian views on his work, was more interested in the Sertão region of North East Brazil, the indigenous natives and their cultures, than the aristocracy and bourgeoisie of São Paulo and Rio de Janeiro. It may be asked how Cendrars’ experience of his travels is to be embodied on the basis of his writings from the past as a conceptual work. How can such experience be transmitted to the audience? The research work includes soundscapes, texts, video and photos collected between December 2008 and June 2016, and presented between January and August 2016 in Switzerland and Brazil.

9.6.1 Processes of Development During the months of September and October 2015, I began research into the travels of Blaise Cendrars in Brazil, including the famous journey made with the Brazilian Modernists in Minas Gerais in 1924. The research was carried out at the Biblioteca Parque Estadual in Rio de Janeiro; it allowed me to locate and map the places he went in order later to record them. The research and my own travels in Brazil in his tracks were serendipitous, arising as they did from spontaneous thoughts and fortuitous occurrences. Discovering, causality, accidents, improvements, and knowledge combine towards describing serendipity, a term coined by Horace Walpole in 1754 (Van Andel 1994, 633). The origin seems to be found in a Persian fairy tale in which an ancient king sent his sons to discover and experience the world (the three princes of Serendip) (Merton and Barber 2004). In following their quest, the princes had experiences that were not those originally planned but accidental and coincidental— and which took them to new horizons. Real or not, the story behind serendipity suggests a field of possibility for the soundwalker. The title Carnet de Routes (Book of Roads/Ways) includes a metaphor for the audience’s travel experiences on the basis of my own journeys in Brazil. Carnet de Routes are roads crossing as a kaleidoscope of my experiences, perceived by the audience as a mental space in relation to the territory I explored and from which visual mental imagery may be generated. How would serendipitous encounters determine my routes? For example, I have visiting Brazil since 2008, when the choreographer and Professor Johannes Birringer invited me to participate in his yearly International Interaction Lab workshop (Interaktion Labor 2008) for the International Theatre Festival of Belo Horizonte. The workshop was initially

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

237

planned to be held in Germany; Birringer proposed holding the event in Brazil as a new opportunity. At that moment, I was unable financially to go, and shared my concerns with him. He proposed thinking about it for a week, yet in fact within ten minutes I confirmed my intention without knowing how I would finance it. This would serendipitously begin ten years of discoveries in the country with which I would fall in love. Since then and up to the end of 2018, I worked on field recordings, developed workshops, and was invited as guest lecturer to universities and independent spaces in the Amazon Region, Nordeste (Recife, João Pessoa, and Salvador de Bahia), Belo Horizonte, Rio de Janeiro and São Paulo. This period had a huge influence on my work, in particular by studying the Brazilian Neo-concretist movement with artists such as Lygia Clark, Lygia Pape, and Hélio Oiticica. Moreover, my fascination for concretist poetry led to the great opportunities to meeting artists from the Brazilian scene, such as Paulo Brusky and Eduardo Kac.

9.6.2 Outcomes Museum of Fine Arts, Le Locle, Switzerland 23 January 2016/Performance The performance with sound, cut-up text, video, and dancer Crystal Sepúlveda took place within an exhibition of the Brazilian artist Vic Muniz’s paintings (Fig. 9.8) at

Fig. 9.8 Crystal Sepúlveda’s rehearsal among Vic Muniz’s Paintings, Museum of Fine Arts, Le Locle, Switzerland (Photo: Luca Forcucci)

238

L. Forcucci

the museum of Fine Arts of Le Locle in Switzerland. On the base of the writings and the locations where Blaise Cendrars had travelled, I went and recorded the sonic material of these locations and photographed them. The material forms the basis of the sonic performance. The images and texts emerging from the research are projected onto the walls from a wheel-mounted projector. The dancer moves with the projector as the images move around the walls. Composition: Luca Forcucci Dance and choreography: Crystal Sepùlveda Centro Municipal de Arte Helio Oitica, Rio de Janeiro, Brazil May-June 2016/Exhibition 1. 2. 3. 4.

12 C Print 70 × 100 with text; 6 C Print 30 × 40 with text; Video 10’; Sound Performance 30’.

The official premiere of Carnet de Routes with all of its elements was held on the top floor of the Centro Municipal de Arte Helio Oitica (Fig. 9.9). We installed the C-Prints on wood panels and on tables used by clandestine street vendors to sell goods around the Centro Municipal de Arte Helio Oitica. The system allows them to move quickly if the police are monitoring the area. On the C-Print, I added cut-ups of texts by Blaise Cendrars. The video included historical images from the research into the travels of Blaise Cendrars (e.g. his arrival in São Paulo or in the company of

Fig. 9.9 Installation at Centro Municipal de Arte Helio Oitica, Rio de Janeiro, Brazil, May-June 2016/Exhibition (Photo: Pedro Victor Brandão)

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

239

his fellow travellers). For the opening night I presented a soundscape performance of my own Brazilian journeys. Red Bull Station, São Paulo, Brazil May–July 2016/Residency and Exhibition 1. 2. 3.

Digital print on voile fabric/1 (4 × 2.80 m) 2 (4.25 × 2.90 m) 3 (4.25 × 2.90 m); Spatial occupation 3 × 4 × 4.25 m; Container Steel 0.60 × 1 × 3 m/Black Ink & Urban Activity/Site-Specific; Amplifier Ampeg/Contact microphone/Urban Activity/Site-Specific.

During the residency, I pushed forward the reflection on Blaise Cendrars and dedicated myself to the final development of the project. On the first floor of the Red Bull Station building, the digital prints on voile fabric included the first layer with the word Utopialand, and the next two layers were extracts from a piece by Oswald de Andrade on Blaise Cendrars (Fig. 9.10). One can walk into the text, and the words are perceived differently depending on the position of the beholder within the work. One’s reading voice is the sonic part, which appear only within oneself. On the second floor, a container with black ink invited the audience to look carefully at the liquid, in which were visible the vibrations of the city as transmitted into the building. While thus observing the liquid, the audience entered into deep listening. On the third floor, a contact microphone attached to the building was capturing the vibrations of the city, consequently amplified and sent back to São Paulo.

9.7 Conclusions The research project began in November 2008 with an expedition in the Brazilian Amazon rainforest and terminated with the presentation of the last project in June 2016 in São Paulo, Brazil. Over these years the project explored many geographical territories in the form of a nomadic laboratory to research into the perception of sonic arts. My quest was driven by the aim to link sonic arts, architecture, and cognitive science in order to investigate visual mental imagery. These disciplines have been present during my whole professional life, to different degrees. For each field, I had the privilege of developing skills with exceptional professionals and mentors, who showed me possibilities I could not have known existed or even possible. Within this conclusion, I wish to demonstrate certain major aspects I have discovered and hope that new knowledge has been produced for these fields. First, the piece De Rerum Natura’s key achievement was an experience of deep listening during field recording in the Amazon rainforest, and how this powerful experience created the elements for the composition. The composition began to emerge while listening to the sounds and experiencing the different spaces of the forest; also, a link is established between the deep listening experience, on the one hand, and on the other the sonic material perceived by the audience. The experience during the performance is a feedback loop between the space, listening, and memory. The perception of the composition is the result of an addition of rich sounds and spaces from the Amazon rainforest projected into spaces of performance with different

240

L. Forcucci

Fig. 9.10 Digital Print on Voile Fabric, Red Bull Station São Paulo, June-July 2016/Exhibition (Photo: Ignacio Aronovich)

specific acoustic characteristics. The conceptual idea of the sonic moiré is an auditory perception of multiple layers leading to an illusional pattern, comparable to that which emerges visually from the Rotoreliefs of Marcel Duchamp, ones he calls ‘non-retinal art’, as something happening between the work and the viewer; in the current example, it takes place between the work and the listener. The sonic moiré

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

241

pattern changes according to the respective natures of each performance space and leads to a film for the mind. Second, the development of Music for Brainwaves as a performance explored issues emerging from a performer’s body emitting physiological data as EEG, transformed into sound, sent into a resonant space and received back as neurofeedback by the same performer. The process leads to the term ‘hyperbiological’ space, which relates to an augmented peripersonal space (by physiological data through sound in space) and the impression of sensing the relation of sound and space by the performer in the first instance, since ‘the composer, wired-up in various ways, would become the performer of and primary listener to the sounds produced’ (Branden 2011, 132). The sensed space appeared in particular when sounds were sent into the resonant space of the ex-NSA Teufelsberg listening station; there was a clear impression of the embodiment of the sensed space. It has been shown that such experiences with resonating space existed from at least the Neolithic period in different parts of the world and lead ‘to isolat[ing] these hyper-acoustic places from mundane daily life and to attribute high importance to them because abnormal sound behaviour implied a divine presence’ (Eneix 2014). Further research must be pursued into very resonant spaces, since the most active sensation of neurofeedback, at least for the performer, is experienced in such spaces. How to explore the inner experience of sensing and to transfer it to the audience? Third, the survey explored an experiment, named In/Pe, where the fully consenting audience was required to listen carefully to the portfolio of sonic artworks, and later asked, through a questionnaire, to describe the experience as visual mental imagery. The proposal of this research relies on the relationship between the visual mental representation of architectural and environmental space as demonstrating their embodiment through sound instead of through sight. Accordingly, the areas investigated focused on the production of space in relation to the body (Lefebvre 2000, 190). The idea includes the ecological context in which the sonic event is perceived (Gaver 1993; Gibson 1979). The intention/perception link between the composer and the participants pushes further the idea that ‘there is no basic difference in this respect between what happens when a person looks at the world directly and when s/he sits with his eyes closed and thinks’ (Arnheim cited Biggs and Karlsson 2011, 149). The claims proposed by Arnheim differ from this study, because the same mental phenomenon or perception of space proposed by Arnheim is not solely for one and the same person; in this research, it includes both the composer and the participants. The concept of deep listening was a paramount component since, for the composer, it allowed for emphasis on the perception of the environment when recording the sounds in the field for the production of space (intention), and for the audience when asked to concentrate on its own visual mental imagery (perception). Therefore, the perception of visual mental imageries through a focused mode of listening by the composer and by the participants leads to an essential link between intention and perception. Moreover, proprioception in the current study relied on the perception of one’s own body within the visual mental image while deep listening. The common patterns of small architectural and environmental spaces perceived

242

L. Forcucci

among participants are proposed here as the possible representation of the body of the beholder. Fourth, Bodyscape was the first work to emerge from the outcomes of the survey. The body of a dancer was the sonic source for a performance, where the inaudible movements of the body were amplified, live processed, together with the physiological data as EMG (electromyogram) which were sonified and sent back to the dancer, resulting in a biofeedback. We, Cheryl E. Leonard and I, were working live together on the sonic aspect, and the dancer Crystal Sepúlveda interacted with her movements. The first performance was based on actions as trajectories in space we had pre-defined at The Lab gallery in San Francisco. The second performance was held at the Centre Dürrenmatt of Neuchâtel in Switzerland, where the actions of the dancer were defined according to the architectural space. The second performance further included L’Epidémie Virale (translated as the Viral Epidemic), a piece of writing by the Swiss author Friedrich Dürrenmatt about South Africa during the Apartheid period: a text where a virus would transform white bodies into black ones, and how privileges are retained in such a context. The final version of the performance was recomposed on the basis of the recordings of the first two performances at NOTAM in Oslo, as an audiovisual version meant to be diffused in the dark and for eight channels, and intermittent projections of text, photography and video. The movements of the body can nevertheless be perceived in this version. Fifth, Carnet de Routes is the second piece based on the outcomes of the survey. The piece explored the writings and the travels to Brazil of the Swiss-French poet Blaise Cendrars, a country Cendrars called Utopialand. My own travels to Brazil crossed those of Cendrars. I did the famous journey he accomplished with the Brazilian Modernists in Minas Gerais in 1924, where I collected sounds and images from the locations of and writings by him and Oswald de Andrade. The project’s aim was to explore the embodiment of someone else through writings. It led to several performances and exhibitions in Switzerland and Brazil. This work also concluded the current research project begun in 2008 in the Amazon rainforest, when Brazil was experiencing an economic boom and social progress, filled with hope for the future. However, what started as a conceptual work on embodiment through the writings of someone else quickly became contaminated by the demonstrations happening in the streets of São Paulo as a result of the impeachment of the former president Dilma Rousseff. Even now, the situation of the Amazon rainforest continues to disintegrate, as does the country itself, where the combination of COVID-19 and the politics have become a threat to the Indigenous communities and the rest of the country. Finally, the main discovery was the possible representation of the body of the beholder in visual mental imagery while experiencing and deep listening to sonic artworks. It was also a practice-led research or research-creation endeavour as an important methodology for the investigation of the sonic phenomenon, which could not be completely translated into words. Therefore, artworks are paramount for research into the sensorium, and in this case the sonic. This is an outcome based on subjectivity and thus subject to interpretation. Future research in this field will be a combination of sonic arts as research-creation, phenomenology, and cognitive

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

243

science. New tools and methodologies will be developed further to investigate mental imagery in order to shed more light on the mechanisms of listening. Acknowledgements Lucrezia Forcucci, Ana Hupe, Paddy Long, Ricardo Garcia, Paulo C. Chagas, Johannes Birringer, Yuri and Paulo Bruscky, Paula Barretto and her team, Valerio Fiel da Costa and his team, Ciane Fernandes and her team, Ines Linke, Jorge Antunes, Jocy de Oliveira, Marisa Mello, Caroline Valansi, Pedro Victor Brandão, Eduardo Kac, Francisco López and all of the people I met in Brazil over a decade and who helped me to understand and discover this beautiful and complex country. Jill Scott and Irene Heidigger at Zürich University of the Arts, Pierre Magistretti and Olaf Blanke and their teams at the Brain Mind Institute at EPFL Lausanne, NOTAM in Oslo, Leigh Landy, Simon Emmerson and John Richards at the Music, Technology & Innovation Research at De Montfort University, René Laurenceau and the Swatch Art Peace Hotel in Shanghai, Djerassi Foundation in San Francisco, Margot Haliday Knight, Tami Spector, Crystal Sepúlveda and Cheryl E. Leonard, Leonardo/ISATS, Pro Helvetia, Nicati de Luze Foundation, Swissnex in Shanghai, San Francisco, Rio de Janeiro and Bengalore, Canton de Neuchâtel, Ville de Neuchâtel, Centre Dürrenmatt Neuchâtel, The Lab gallery in San Francisco, Musée des Beaux Arts Le Locle.

References Alla, T. 2008. Tristan Murail, La Couleur Sonore. Paris: Michel de Maule. Artaud, A. 1958. The Theatre and its Double, trans. from French by M. C. Richards. New York: Grove Press. Ascott, R. 2003. Telematic Embrace, Visionary Theories of Art, Technology, and Consciousness. Berkley and Los Angeles: University of California Press. Bachelard, G. 1992. The Poetics of Space, trans. M. Jolas. Boston: Beacon Press. Berlucchi, G., and S.M. Aglioti. 2009. The Body in the Brain Revisited. Experimental Brain Research 200 (1): 25–35. Bermudez, J.L., A. Marcel, and N. Eilan (eds.). 2001. The Body and the Self . Cambridge, MA: MIT Press. Biggs, M., and H. Karlsson. 2011. The Routledge Companion to Research in the Arts. London: Routledge. Birringer, J. 2008. Performance, Technology and Science. New York: PAJ Publications. Blanke, O. 2012. Multisensory Brain Mechanisms of Bodily Self-Consciousness. Nature Reviews Neuroscience 13: 556–571. Blanke, O., L. Forcucci, and S. Dieguez. 2009. Don’t Forget the Artists when Studying Perception of Art. Nature 462, December: 1984. Boellstorff, T., 2011. VIRTUALITY, Placing the Virtual Body: Avatar, Chora, Cypherg. In A Companion to the Anthropology of the Body and Embodiment, ed. F.E Mascia-Lee, 504–20. Chichester: Blackwell. Böhme, G. 2000. Acoustic Atmospheres: A Contribution to the Study of Ecological Aesthetics. Soundscape, The Journal of Acoustic Ecology. 1 (1): 14–18. Böhme, G. 1993. Atmosphere as the Fundamental Concept of a New Aesthetics. Thesis Eleven 36: 113–126. Bondanella, P. 2002. The Films of Federico Fellini. Cambridge: Cambridge University Press. Born, G. (ed.). 2013. Music, Sound and Space: Transformations of Public and Private Experience. Cambridge: Cambridge University Press. Branden, W.J. 2011. Biomusic. Grey Room 45, Fall Issue: 128–150. Brochard, R., A. Dufour, and O. Després. 2004. Effect of musical expertise on visuospatial abilities: Evidence from reaction times and mental imagery. Brain and Cognition 54: 103–109.

244

L. Forcucci

Budzynski, T., H.K. Budzynski, J.R. Evans, and A. Abarbanel. 2009. Introduction to Quantitative EEG and Neurofeedback, 2nd edn. New York: Academic Press. Casey, E. 1998. The Fate of Place: A Philosophical History. Los Angeles: University of California Press. Castanet, P.A. 2007. Edgard Varèse et la “philosophie du bruit” - Ionisation en questions. In Edgar Varèse: Du Son Organisé aux Arts Audio, ed. T. Horodyski, P. Lalitte, 77–131. Collectif. Paris: L’Harmattan. Cendrars, B. 1926. Moravigne. Paris: Éditions Grasset & Fasquell. Centre Dürrenmatt, 2016. https://www.cdn.ch/cdn/en/home/visiting/events/events-2016/-bodysc ape–composition-electroacoustique-pour-paysages-sonores-.html. Accessed 31 Aug 2020. Chare, N. 2009. Upon the Scents of Paint: Bacon and Synaesthesia. Visual Culture in Britain 10 (3): 253–270. Cobussen, M., V. Meelberg, and B. Truax. 2017. The Routledge Companion to Sounding Art. Oxford: Routledge. Cox, T., and S. Ings. 2014. Acoustic Art and Industrial Architecture Make Music. http://www.newsci entist.com/article/dn25808-acoustic-art-and-industrial-architecture-make-music.html. Accessed 31 Aug 2020. Cook, I.A., S.K. Pajot, and A.F. Leuchter. 2008. Ancient Architectural Acoustic Resonance Patterns and Regional Brain Activity. Time and Mind 1(1): 95–104. CTM Festival. 2016, Listening for Peace, https://archive2013-2020.ctm-festival.de/archive/festivaleditions/ctm-2016-new-geographies/specials/listening-for-peace/?L=0. Accessed 31 Aug 2020. Deleuze, G., F. Guattari, and B. Massumi. 1987. A Thousand Plateaus: Capitalism and Schizophrenia. Minneapolis, MN: University of Minnesota Press. De Benedictis, A.I. 2013. Biographia, trans. from Italian by M. Weir. http://www.luiginono.it/en/ luigi-nono/biography. Accessed 31 Aug 2020. De Vignemont, F. 2010. Body schema and body image—Pros and cons. Neuropsychologia 48 (3): 669–680. Dieguez, S., and C. Lopez. 2016. The bodily self: Insights from clinical and experimental research. Annals of Physical and Rehabilitation Medicine 60 (3): 198–207. Diller, E., Scofidio, R., and Renfro, C. 2002. Blur Building. https://dsrny.com/project/blur-building. Accessed 31 Aug 2020. Dixon, S. 2006. A history of virtual reality in performance. International Journal of Performance Arts and Digital Media 2 (1): 23–54. Dürrenmatt, F. 2007. L’épidémie virale en Afrique du Sud. ethnographiques.org, 13. https://www. ethnographiques.org/2007/Durrenmatt. Accessed 31 Aug 2020. Emmerson, S. (ed.). 2003. Music, Electronic Media and Culture. Hampshire: Ashgate Publishing. Emmerson, S. 1986. The Language of Electroacoustic Music. London: Macmillan Press. Eneix, L. 2014. Ancient Man Used ‘Super-Acoustics’ to Alter Consciousness (… and Speak with the Dead?). https://sciencex.com/wire-news/164386603/ancient-man-used-super-acoustics-to-alterconsciousness-and-spe.html. Accessed 31 Aug 2020. Erlmann, V. 2005. Hearing Cultures. Essays on Sound, Listening, and Modernity. Oxford: Bloomsbury. Feld, S. 2015. Acoustemology. In Keyword in Sound, ed. D. Novak and M. Sakakeeny, 12–21. Durham, NC: Duke University Press. Feld, S., and D. Brenneis. 2004. Doing anthropology in sound. American Ethnologist 31 (4): 461– 474. Forcucci, L. 2015. Mapping Dynamic Relations in Sound and Space Perception. PhD thesis, De Montfort University. http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.679035. Accessed 31 Aug 2020. Forcucci, L. 2017b. Mechanisms of Listening and Spatial Mental Imagery. Bio-Creation and Peace Proceedings. Manizales, Colombia: ISEA (International Symposium of Electronic Arts). Forcucci, L. 2018. LASER NOMAD in Brazil. https://coincidencia.net/en/supported-projects/lasernomad-in-brazil-luca-forcucci/. Accessed 31 Aug 2020.

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

245

Frichot, H. 2008. Olafur Eliasson and the Circulation of Affects and Percepts. Conversation. Architectural Design 78 (3): 30–35. Gallagher, S. 2005. How the Body Shapes the Mind. Oxford: Oxford University Press. Garwood, D. 2007. The Future of an Idea, 9 Evenings, Forty Years Later. PAJ 29(1): 36-48. Gaver, W.W. 1993. What in the World do we Hear? An Ecological Approach to Auditory Source Perception. Ecological Psychology 5 (1): 1–29. Gayou, E. 2001. Luc Ferrari, Portraits Polychromes. Paris: Ina-GRM, Cdmc. Gibson, J.J. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin Harcour. Griffero, T. 2014. Atmospheres: Aesthetics of Emotional Spaces. Surrey: Ashgate. Goodman, S. 2010. Sonic Warfare: Sound, Affect, and the Ecology of Fear. Cambridge, MA: MIT Press. Gouveia, S. 2009. Private Patronage in Early Brazilian Modernism, Xenophobia and Internal Colonization Coded in Mário de Andrade’s ‘Noturno de Belo Horizonte’. Luso-Brazilian Review 46 (2): 90–112. Guattari, F. 1989. Les Trois Ecologies. Paris: Galilée. Henrique, J. 2011. Sonic Bodies: Reggae Sound Systems, Performance Techniques, and Ways of Knowing. London: Continuum. Holmes, N.P., and C. Spence. 2004. The Body Schema and Multisensory Representation(s) of Peripersonal Space. Cognitive Processing 5(2), June Issue: 94–105. Interaktion Labor. 2008. 6th International Interaction Lab. https://interaktionslabor.de/lab08/index. htm. Accessed 31 Aug 2020. ISEA, 2017. News. Computer Music Journal 41(4): 5–7. Jasen, P.C. 2016. Low End Theory: Bass, Bodies and the Materiality of Sonic Experience. London: Bloomsbury. Kanekar, A. 2015. Architecture’s Pretexts: Spaces of Translation. Oxon: Routledge. Kiku, M. 1995. Hyperspace: A Scientific Odyssey Through Parallel Universes, Time Warps and the 10th Dimension. New York: Oxford Press. Kojs, J. 2009. The language of action and cyberaction-based music: Theory and practice. Journal of New Music Research 38 (3): 285–294. Kolar, M.A. 2013. Archaeological Psychoacoustics at Chavín de Huántar, Perú. https://ccrma.sta nford.edu/groups/chavin/kolardissertation.html. Accessed 31 Aug 2020. Kim-Cohen, S. 2013. Against Ambience. New York, NY: Bloomsbury. Kosslyn, S.M., G. Ganis, and W.L. Thompson. 2001. Neural Foundations of Imagery. Nature Reviews Neuroscience 2 (9): 635–642. Knight, M. 2015. Scientific Delirium Madness. Leonardo 48 (3): 219–222. Knowles, G.J., and A.L. Cole. 2008. Arts-informed research. In The Sage Encyclopedia of Qualitative Research Methods, ed. L.M. Given, vols. 1&2, 55–70. London: SAGE Publications. LaBelle, B. 2006. Background Noise, Perspectives on Sound Art. New York: Continuum. Lance, B.J., S.E. Kerick, A.J. Ries, K.S. Oie, and K. McDowell. 2012. Brain-computer interface technologies in the coming decades. Proceedings of the IEEE, 100, May 13, Special Centennial Issue: 1585–99. Laudadio, E., G.L. Farinelli, and S. Toffett (eds.). 2020. Federico Fellini: The Book of Dream. New York, NY: Rizzoli International Publications. Lefebvre, H. 2000. La Production de l’Espace, 4th ed. Paris: Editions Antropos. Linz, R. 2001. An Interview with Stelarc. http://www.rainerlinz.net/NMA/repr/Stelarc_interview. html. Accessed 31 Aug 2020. Lucier, A. 1995. Alvin Lucier, Reflections, Interviews, Scores, Writings 1965–1994/Reflexionen, Interviews, Notationen, Texte 1965–1994. Köln: Musik Texte. Lumenta. S. 2020. Resources for Gendy3 by Iannis Xenakis. https://github.com/sbl/sc-max. Accessed 31 Aug 2020. Manning, E., and B. Massumi. 2014. Thought in the Act: Passages in the Ecology of Experience. Minneapolis, MN: University of Minnesota Press.

246

L. Forcucci

Massumi, B. 2002. Parables for the Virtual. Durham, NC: Duke University Press. Massumi, B. 2015. Politics of Affect. Cambridge: Polity Press. Merill, S., and L. Schmidt. 2009. A Reader in Uncomfortable Heritage and Dark Tourism. https://www-docs.b-tu.de/fg-denkmalpflege/public/downloads/UHDT_Reader.pdf. Accessed 31 Aug 2020. Merton, R.K., and E. Berber. 2004. The Travels and Adventures of Serendipity. Princeton: Princeton University Press. Millet, D. 2001. Hans Berger: From psychic energy to the EEG. Perspectives in Biology and Medicine 44(4), Autumn: 522–42. Miranda, E. 2014. Composing Music with Computers. London: Routledge. Nancy, J.L. 2007. Listening, trans. Charlotte Mandell. New York: Fordham University Press. Neuhaus, M. n.d. https://sculpture-synchronisee.villa-arson.org/2013/11/08/max-neuhaus-waterwhistle-series-70s/. Accessed 31 Aug 2020. Noland, C. 2000. The metaphysics of coffee: Blaise Cendrars, modernist standardization, and Brazil. Modernism/modernity 7 (3): 401–422. NOTAM. 2016. Luca Forcucci. https://notam.no/residency-luca-forcucci/. Accessed 31 Aug 2020. O’Callaghan, C. 2010. Sounds: A Philosophical Theory. Oxford: Oxford University Press. O’Callaghan, C. 2017. Beyond Vision: Philosophical Essays. Oxford: Oxford University Press. O’Callaghan, C., and M. Nudds. 2010. Sounds and Perception: New Philosophical Essays. Oxford: Oxford University Press. Oiticica, H. 1965. Notes on the Parangolé – Anotaçoes Söbre o Parangolé. Exhibition Opinião 65. Published by the Artist, Rio de Janeiro: Museu de Arte Moderna. Oliveros, P. 1974. Sonic Meditations. Sharon, VT: Smith. Oliveros, P. 2005. Deep Listening: A Composer’s Sound Practice. Bloomington: iUniverse.com. Oliveros, P. 2007. Improvising with spaces. In Proceedings of the 13th International Conference on Auditory Display, Montréal, Canada, June 26–29. Oliveros, P. 2010. Sounding the Margins: Collected Writings 1992–2009. Kingston, NY: Deep Listening. Oliveros, P. 2011. Auralizing in the sonosphere: Vocabulary for inner sound and sounding. The Journal of Visual Culture 11 (2): 163–168. Osthoff, S. 2004. Lygia Clark and Hélio Oiticica: A Legacy of Interactivity and Participation for a Telematic Future. Leonardo on-line. https://www.leonardo.info/isast/spec.projects/osthoff/ost hoff.html. Accessed 31 Aug 2020. Oteri, F.J. 2004. Extremities: Maryanne Amacher. NewMusicBox. https://nmbx.newmusicusa.org/ extremities-maryanne-amacher-in-conversation-with-frank-j-oteri/. Accessed 31 Aug 2020. Ouzounian, G. 2006. Embodied sound: Aural architectures and the body. Contemporary Music Review 25(1–2): 69–79 https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1045.5247& rep=rep1&type=pdf. Accessed 31 Aug 2020. Porcello, T., L. Meintjes, A.M. Ochoa, and D.W. Samuels. 2010. The reorganization of the sensory world. Annual Review of Anthropology 39: 51–66. Rebentisch, J. 2012. Aesthetic of Installation Art. Berlin: Sternberg Press. Roads, C. 1996. The Computer Music Tutorial. Cambridge, MA: MIT Press. Rosenboom, D. 1990. The performing brain. Computer Music Journal 14(1), Spring Issue: 48–66. Sanguin, A.L. 1981. La Géographie Humaniste ou l’Approche Phénoménologique des Lieux, des Paysages et des Espaces. Annales de Géographie 90 (501): 560–587. Serra, M.H. 1993. Stochastic composition and stochastic timbre: GENDY3. Perspectives of New Music 31(1), Winter: 236–57. Shanken, A.E. 1997. Technology and intuition: A love story? Roy Ascott’s Telematic Embrace? Leonardo 30(1), February. Sheller, M. 2015. Vital methodologies: Live methods, mobile art, and research-creation. In NonRepresentational Methodologies Re-Envisioning Research, ed. by P. Vannini, 130–145. New York, NY: Routledge.

9 Sonic Imagination: Body, Visual Mental Imagery, and Nomadism

247

Seth, A.K. 2013. Interoceptive inference, emotion, and the embodied self. Trends in Cognitive Sciences 17 (11): 565–573. Smetacek, V., and F. Mechsner. 2004. Making sense. Nature 432: 21. Stanford Encyclopedia of Philosophy. 2013. https://plato.stanford.edu/entries/phenomenology/. Accessed July 2015. The Lab. 2015. Bodyscape. https://www.thelab.org/projects/2015/8/4/bodyscape-by-luca-forcucciwith-cheryl-leonard-and-crystal-seplveda. Accessed 31 Aug 2020. Tsabary, E. 2009. Which aural skills are necessary for composing, performing and understanding electroacoustic music, and to what extent are they teachable by traditional aural training? Organised Sound 14 (3): 299–309. Thompson, E. 2005. Sensorimotor subjectivity and the enactive approach to experience. Phenomenology and the Cognitive Sciences 4 (4): 407–427. Truax, B. 2002. Genres and techniques of soundscape composition as developed at Simon Fraser University. Organised Sound 7 (1): 5–14. Truax, B. 1996. Soundscape, acoustic communication and environmental sound composition. Contemporary Music Review 15 (1–2): 49–65. Tsakiris, M. 2009. My body in the brain: A Neurocognitive model of body-ownership. Neuropsychologia 48 (3): 703–712. Ubqtlab.org. 2020. www.ubqtlab.org. Accessed 31 Aug 2020. Van Andel, P. 1994. Anatomy of the unsought finding. Serendipity: Origin, history, domains, traditions, appearances, patterns and programmability. The British Journal for the Philosophy of Science 45(2): 631–48. Vannini, P. 2015. Non-Representational Methodologies: Re-envisioning Research. New York, NY: Routledge. Varela, F., E. Thompson, and E. Rosch. 1991. The Embodied Mind, Cognitive Science and Human Experience. Cambridge, MA: MIT Press. Varèse, E., and W.C. Chou. 1966. The liberation of sound. Perspectives of New Music 5 (1): 11–19. Vernon, D.J. 2005. Can neurofeedback enhance performance? An evaluation of the evidence with implications for future research. Applied Psychophysiology and Biofeedback 30(4), December Issue: 347–64. Woelert, P. 2011. Human cognition, space, and the sedimentation of meaning. Phenomenology and the Cognitive Sciences 10 (1): 113–137. Xenakis, I. 1992. Formalized Music: Thought and mathematics in composition. Hillsdale, NY: Pendragon Press. Zahavi, D. 2012. The Phenomenological Mind, 2nd ed. London: Routledge. Zorn, J. 2008. Arcana III: Musicians on Music. New York: Hips Road.

Discography Eb.Er, R., J. Lanz, G. Jupitter-Larsen, and M. Dando. 2014. Wellenfeld, (CD), Hamburg: Fragment Factory. Forcucci, L. 2020. Bodyscape. https://lucaforcucci.bandcamp.com/album/bodyscape-2. Accessed 31 Aug 2020. Forcucci, L. 2017a. The Waste Land (cassette), Porto: Crónica Electrónica, Crónica 128 ~ 2017. Forcucci, L. 2009. De Rerum Natura. https://lucaforcucci.com/works-2/performances/de-rerum-natura/. Accessed 29 Jan 2020.

248

L. Forcucci

Luca Forcucci is a distinguished artist, composer, and researcher, who has a background in sonic arts, architecture, and cognitive science. His research observes the perceptive properties of sound, space, and memory. The field of possibilities of the first-person experience is explored as the artwork. In this context, Forcucci is interested in perception, subjectivity, and consciousness. He collaborates with scientists in the fields of cognition, perception, and biology and is particularly fascinated by visual mental imagery as sonic imagination. As chair of Laser Nomad, a lab for art and science research in collaboration with Leonardo/ISATS, Forcucci aims to decolonize knowledge by presenting talks and podcasts around the world. Forcucci’s artworks are presented internationally and his research conducted at the University of the Arts of Berlin, INA/GRM Radio France in Paris, Brain Mind Institute EPFL in Switzerland, has explored cognitive neuroscience of out-of-body experiences. An international lecturer, Forcucci achieved a PhD in Music Technology and Innovation in UK, and a M.A. in Sonic Arts from Queens University of Belfast. He has been awarded numerous prizes, including the Swiss Artists in Lab Residency, Swiss Digital Art Award for research in the Brazilian Amazon Rainforest, Cité Internationale des Arts in Paris, NOTAM in Oslo, and Djerassi Foundation Scientific Delirium Madness residency in San Francisco. Forcucci was nominated in the Arts at the World Technology Summit in New York and was a finalist for the Luigi Russolo Prize.

Index

A Acoustic sound, 82, 83, 85, 96–98 Active (attentive) music listening, 107, 108 Affect, 197, 199, 201, 203–205, 223, 224, 226, 235 Agamben, Giorgio, 1, 8–12, 14 A kind of magic, 156 All That You Can’t Leave Behind, 155 Amazon rainforest, 197, 199, 206–210, 235, 239, 242 Another one bites the dust, 156 Anticipation, 57, 61, 62 Apparatus, 1, 13–26 Art and science, 2, 5, 6 Audiation, 84

B Barthes, Roland, 155, 159 Beatles Anthology, The, 163 Beatles, The, 156, 157, 162 Beauchamp, George, 153 Beethoven, Ludwig van, 151, 154, 155 Believe, 154 Berio, Luciano, 146 Berlioz, Hector, 153, 154 Blacking, John, 150 Blank space, 148 Bohemian rhapsody, 155 Bowie, David, 156 Brainwaves, 199, 214–222, 228, 229, 241 Broja, Jan Krzysztof, 133–139

C Cher, 154 Chôra, 206 Clapton, Eric, 157, 162 Classical style, 29 Cochlea, 82, 83, 87, 96 Contemplative practices, 171, 173 Cultural arts, 173 Cumming, Naomi, 124–126, 128, 129, 131, 139 Cure, The, 146

D Davies, Rick, 157 Davis, Miles, 146, 155 Debussy, Claude, 154, 155 Decoding, 75–81, 88–90, 95, 99–101 Deep listening, 81, 84, 85 Delalande, François, 145 Digital music instrument evaluation, 172, 183 Di Marcoberardino, Matteo, 145 Disciplinary matrix, 1–5, 8, 13, 14, 18

E Earth, Wind and Fire, 156 Ecosophy, 207 Eco, Umberto, 150 Edge, The, 154 EEG, 86, 197, 214–219, 221, 222, 241 Electroacoustic music, 17–20, 25 Electronic music, 19, 20, 24 Elektronische musik, 18, 19

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 P. C. Chagas and J. Cecilia Wu (eds.), Sounds from Within: Phenomenology and Practice, Numanities - Arts and Humanities in Progress 18, https://doi.org/10.1007/978-3-030-72507-5

249

250 Embodied cognition, 171, 172, 192 Embodiment, 197, 199, 203–205, 214, 222, 224, 225, 228–230, 234, 235, 241, 242 Emotion, 107, 109, 110, 116–118 Existential semiotics, 29 Extramusical, 107–115, 118–121

F Familiarity factor, 110, 111 Ferrario, Roberto, 145, 147 Flusser, Vilém, 1, 23, 24 FMRI, 79, 87–91, 93, 95, 97, 99, 101 From experience to theory, 143, 145, 157, 158, 161, 167

G Game, The, 156 General codes, 150, 151, 161 Gesture, 123, 125–131, 133, 136–139 Greimassian modalities, 127, 137 Guerra Lisi, Stefania, 148

H Harrison, George, 157, 162, 163, 166 Heidegger, Martin, 1, 2, 13, 18, 22, 147 Hendrix, Jimi, 153 Holm-Hudson, Kevin, 145, 147 Hot Space, 155 Hyperbiologicals, 215, 219, 221, 241

I Imagination, 57, 58, 70 Inception, 76, 83, 84 Incommensurability, 4 Individuals and works, 155 Interactive arts, 173, 181, 192, 193 I want to break free, 155, 156

K Kinks, The, 154 Kravitz, Lenny, 152 Kreisleriana, 155 Kuhn, Thomas, 1–9, 13, 14 Kulikauskas, Justas, 149

L Language game, 1, 7, 8

Index Latency, 58 Lennon, John, 153 Lietaus kambarys, 148 Ligeti, Gyorgy, 147 Listening, 197, 199, 201–212, 214, 215, 217–219, 221–224, 226–228, 239, 241–243 Loudness, 57, 63–67 Lyrics, 107–110, 115, 119–121

M Madonna, 156 Mamontovas, Andrius, 149 Manzarek, Ray, 157 Martinelli, Dario, 143, 144 May, Brian, 155, 156 McCartney, Paul, 157 Media arts and technology, 173, 193 Meditation, 171–174, 178, 180–182, 192, 193 Mercury, Freddie, 155, 156 Metaphysics, 29, 38, 40, 41, 55 Modeling, 59, 61, 62 Moi and Soi, 29, 32 MRI, 87, 89–92, 95, 100 Multi-Voxel Pattern Analysis, 88 Musical competence, 143, 149–151, 158, 161 Musical grips (\prese di musica\), 157, 158, 161 Musical techniques, 150, 153, 160, 161 Music analysis, 29 Music-induced/music-evoked, 107, 109, 110, 116, 121 Music listening, 107, 108, 111, 116, 120, 121 Musique concrète, 17–19 My generation, 154

N Nattiez, Jean-Jacques, 144 Neurofeedback, 214–217, 221–223, 228, 241 Nirvana (band), 152 Nomadism, 197, 198 Notation, 44, 45, 50–52

O Oasis (band), 152 Odenall Pi4, 148

Index P Paradigm –electroacoustic paradigm, 1, 14, 17–22, 25 –instrumental paradigm, 14–16, 18–20 –telematic paradigm, 1, 19, 22, 23 –vocal paradigm, 15 Peirce, Charles Sanders, 149 Perception, 197, 199–206, 208, 209, 212– 215, 218, 222–230, 234, 235, 239– 241 Performance, 123–126, 128–137, 139 Performer‘s expression, 136 Performer’s body, 125, 128, 132, 139 Phenomenological agreement, 67 Phenomenology, 1, 19 Philosophical archeology, 1, 8, 9 Philosophy, 29, 32, 34, 37, 47, 51 Pictures at an Exhibition, 163, 164, 166 Pop (U2 album), 155 Popular music, 143–145, 147, 150, 152–155, 157, 161, 162 Posthumus, E.S., 148 Presence, 82–84, 86, 88 Prince, 156 Proprioception, 223, 224, 241 Puzzles, 2–6, 20 Puzzle-solving, 5, 6, 14–16

Q Quality, 78, 83–85 Queen, The, 155, 156, 164, 166

R Radio Ga-Ga, 155, 156 Rattle and Hum, 155 Razauskas, Domantas, 148 Romanticism, 34, 46 Ruwet, Nicolas, 150

S Scarlatti, Domenico, 134 Schumann, Robert, 155 Scientific revolution, 3–5, 7 Semiotic self, 125, 126 Serendipity, 235, 236 Show must go on, The, 156 Signature, 1, 11–13, 15, 17, 20, 22 Signifier-signified, 118 Site-specificity, 197, 199, 204, 229 Smith, Robert, 146

251 Social practices, 150, 152, 161 (socio)cultural influence, 111, 113 Sonic architecture, 199, 200, 202, 204–206 Sonic art, 193 Sonic awareness, 172, 193 Sonic Imagination, 197, 201, 230 Sonic self, 123, 125, 129, 131 Sound, 143–149, 151, 154–156, 158–160, 163–168 Soundwalking, 204, 207, 209 Specious present, 60, 67 Spector, Phil, 147 Stefani, Gino, 143–155, 157–163, 167 Styles, 146, 147, 149, 150, 152–156, 161 Supertramp, 156 Swift, Taylor, 148, 149 Synchronization, 59, 62, 67 Synesthesia, 145, 148

T Teleconcerts, 58 Telematic –telematic communication, 1, 23 –telematic dialog, 23 –telematic society, 23, 26 Tibetan Buddhism, 172–175, 177, 192 Tonal beauty, 124, 128, 130–132, 139 Tone quality, 123, 128 Tool (band), 146 Topics, 33, 34, 55 Touch, 123, 124, 130, 134, 136–139 Transcendence, 29, 30, 32, 33, 45, 47 Treatise on instrumentation, 153 Trunova, Victoria, 145–147, 160 Truth, 1, 2, 4, 9, 13, 18

U U2, 154, 155 Unconcealment, 2 Under pressure, 156

V Varankait˙e, Ulrika, 148, 149 Virtual, 208, 223, 225, 226, 228 Visual imagery, 107, 110, 113–116, 118, 119, 121 Visual mental imagery, 197, 199, 201–204, 207, 208, 212, 223–230, 234, 236, 239, 241, 242 Vox, Bono, 154

252 W We are the champions, 155 Where the streets have no name, 155 While my guitar gently weeps, 162, 164, 168 White Album (conventional name for The Beatles), The, 162 Who, The, 154 Who wants to live forever, 156 Wittgenstein, Ludwig, 1, 2, 4, 7, 8, 16, 18, 26

Index Y You really got me, 154

Z Zemic model, 29, 31, 43