Multimodality and Identity 9780815349044, 9780815349051, 9781003186625

Citation preview

MULTIMODALITY AND IDENTITY

This book brings together the work of leading theorist, Theo van Leeuwen, on typography, colour, texture, sound and movement, and shows how they are used to communicate identity, both corporate and individual. The book provides a detailed approach to analysing the key elements of multimodal style, and shows how these can be applied to a wide range of domains, including typography, product design, architecture, and animation films. Combining sociological insights into contemporary forms of identity with multimodal approaches to analysing how these identities are expressed, the text is richly illustrated with examples from fashion, the built environment, logos, modern art and more. With sample analyses, this user-friendly text provides clear methods for analysis and creative strategies for the practice of multimodal communication. Providing an invaluable toolkit to analysing the key elements of multimodal design and the way they work together, this book is essential reading for students, teachers and researchers in the field of multimodal communication, whether in communication studies, linguistics, design studies, media studies or the arts. Theo van Leeuwen is currently Professor of Language and Communication at the University of Southern Denmark and Honorary Professor at the University of New South Wales, Australia. His many influential publications include Multimodal Discourse: The Modes and Media of Contemporary Communication (with Gunther Kress); Speech, Music, Sound; Introducing Social Semiotics, Discourse and Practice: New Tools for Critical Discourse Analysis, and Reading Images (with Gunther Kress).

MULTIMODALITY AND IDENTITY

Theo van Leeuwen

First published 2022 by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN and by Routledge 605 Third Avenue, New York, NY 10158 Routledge is an imprint of the Taylor & Francis Group, an informa business © 2022 Theo van Leeuwen The right of Theo van Leeuwen to be identified as author of this work has been asserted by him in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library Library of Congress Cataloging-in-Publication Data Names: Van Leeuwen, Theo, 1947- author. Title: Multimodality and identity / Theo van Leeuwen. Description: London ; New York : Routledge, 2021. | Includes bibliographical references and index. Identifiers: LCCN 2021002182 | Subjects: LCSH: Modality (Linguistics) | Semiotics‐‐Social aspects. | Synesthesia. Classification: LCC P99.4.M6 V36 2021 | DDC 302.2‐‐dc23 LC record available at https://lccn.loc.gov/2021002182 ISBN: 978-0-8153-4904-4 (hbk) ISBN: 978-0-8153-4905-1 (pbk) ISBN: 978-1-003-18662-5 (ebk) Typeset in Bembo by MPS Limited, Dehradun Access the Support Material: https://www.routledge.com/9780815349051

CONTENTS

List of figures Acknowledgements

vi viii

Introduction

1

1 The social semiotics of identity

5

2 Functionality and identity

24

3 Analysing style

40

4 Shape

55

5 Colour

77

6 Texture

97

7 Movement

117

8 A social semiotic theory of synaesthesia

139

References Index

164 174

FIGURES

1.1 Maori Moko 1.2 The Mary of Warmun (George Mung Mung, late 1970s) 1.3 Russolo, Carà, Marinetti, Boccioni and Severini (unknown photographer, 1912) 2.1 Hot tips genre (after Cosmopolitan magazine, November 2002, US edition) 2.2 Microsoft Word template 2.3 SmartArt “Half-Circle Organization Chart” 2.4 SmartArt “Continuous Block Process” 3.1 Persian and modern carpet 3.2 “This is a car” – drawing by a three-year-old child 3.3 A system network (top) and a parametric system (bottom) 4.1 Communication model (Riley and Riley, 1959) 4.2 Communication model (Shannon and Weaver, 1949) 4.3 Apple organization chart 4.4 A likeness of the Nokia logo 4.5 Pattern with sloping, disconnected rectangles (Sarah Angold) 4.6 Breton top 4.7 The distinctive features of shape 4.8 Likeness of three pharmaceutical logos 4.9 City living (Carlene Edwards) 5.1 Historic colours (House Beautiful, September 1998: 21) 5.2 The distinctive features of colour 5.3 “Landscape” colour scheme (Le Corbusier, 1931) 5.4 City living (Carlene Edwards) 6.1 Opened by customs (Kurt Schwitters, 1937) 6.2 The Hundertwasserhaus

7 10 14 26 30 33 33 42 45 47 56 56 63 67 69 71 72 74 75 89 91 94 95 101 110

Figures

6.3 6.4 6.5 7.1 7.2 7.3 7.4 7.5 7.6 7.7 8.1 8.2 8.3

Self-portrait (Jean Tinguely, 1988) Björkudden table (left) and Grimle table (right) The distinctive features of texture A likeness of the Intel logo Sindy and Action Man Tizio light (left) and Anglepoise light (right) A still from Help stop the spread (Australian Government video, 2020) The distinctive features of movement Still from The Girl Effect The system of mobility design Likeness of a Google doodle The distinctive features of timbre Stone carvings (Jonathan Barnbrook, 1998)

vii

111 112 115 120 121 122 124 131 132 136 143 151 160

ACKNOWLEDGEMENTS

Many of the ideas presented in this book originated in ever-inspiring conversations and joint publications with Gunther Kress, whose vision and intellectual generosity I still miss every day. Collaborations with Emilia Djonov continue to be exciting. Her sharp eye for detail has saved me from many inaccuracies and other wrongdoings. Discussions with Christian Johannessen on graphic design, experiential meaning potential and materiality, often over a beer in the unsurpassable Smagløs pub in Odense, have been equally important, as have regular discussions with Bob Hodge, and his comments on drafts of several chapters. Discussions on many aspects of visual communication with Morten Boeriis, and on music, sound and technology with Johannes Mulder have also made an important contribution. The work on texture draws on collaborative work with Emilia Djonov. Two of my PhD Students, Gisela Leão and Joshua Han, developed the work on movement on which I draw in Chapter 7, and Yufei He, during a period as my research assistant, made further contributions. In the second semester of 2019, I tried out my ideas for this book with a class of PhD students at the University of Sydney. Our lively discussions, and the project work the students did as part of the course, have been invaluable in bringing the content and structure of the book into clearer focus, and in convincing me that it will be interesting and useful to the next generation of multimodality scholars. I want to thank them all – Lilian Ariztimuño, Georgia Carr, Jiani Chen, Anna Crane, Awni Etaywe, Eirik Foss, David Goldman, Joshua Han, Yufei He, Olivia Inwood, Nataliia Laba, Jun Li, Lorenzo Logi, Giselle Newton, Jude Page, Chengfang Song, Chavalin Svetanant, Annette Turney, Alice Wen, Xiaoqin Wu, Junjun Xing, Qingkin Xu, Zhigang Yu and Mus Zhang. This work would not exist without my early discovery of the writings of Roland Barthes, which ignited my interest in semiotics, my later discovery of the work of Michael Halliday, whose social semiotic approach to linguistics changed

Acknowledgements

ix

my view of what semiotics could and should be, and my still later discovery of the metaphor theory of Lakoff and Johnson, which opened my eyes for the role of embodied experience in making meaning. I would like to thank Toby van Leeuwen for his skilful and clever drawings of examples which the ever more prohibitive restrictions on reproducing logos and other corporate visuals would otherwise have prevented me from using. I would also like to thank Louise Semlyen and Eleni Steck for their continuing encouragement and support in publishing my work. And, above all, I would like to thank my wonderful partner Deborah – with whom I have had many enjoyable and sometimes heated discussions about identity. Every effort has been made to trace all the copyright holders but if any have been inadvertently overlooked the publishers will be pleased to make the necessary arrangements at the first opportunity.

INTRODUCTION

The “grammar of visual design” Gunther Kress and I first published 25 years ago (Kress and Van Leeuwen, 1996) focused on the way composition enables the representational and interactive functions of visual communication. Earlier work on images had mostly dealt with the denotative and connotative meanings of the people, places and things shown in images and treated composition in formalistic and aesthetic ways, or in terms of the quite specific conventions of genres like the Nativity, the Pietà, the Last Supper, etc. We wanted to show that composition realizes meaningful and systematic relations between those represented people, places and things, just as grammar realized meaningful and systematic relations between the words in sentences. To do so we combined concepts from literature on art and design, such as Arnheim’s (1982) theory of visual composition, with concepts from Halliday’s functional grammar, to show that images and words can make quite similar kinds of meaning, albeit in different ways. This work had some success. A third edition of our book Reading Images – The Grammar of Visual Design (1996) has recently been published, and, during the 25 years of its existence, the book has stood model for other work on nonlinguistic modes of communication and been of use to a range of research projects. Yet we soon realized its limitations. Two are particularly important. First, in Reading Images we had focused mainly on images and diagrams. The book was called “Reading Images.” But visual design manifests itself in other ways as well. It also includes, for instance, typography and the decorative patterns that are so ubiquitous in our everyday environment. Although most people will never “read” such designs consciously, they will nevertheless appreciate them and choose them with care, as expressing their taste, and hence their identity. Second, we had developed a grammar, which could be applied to materially very different images, to oil paintings as well as photographs, to still as well as moving images, and to two-dimensional as well three-dimensional images such as sculptures

2 Introduction

and children’s toys. But perhaps oil paintings, photographs and other media, even when drawing on the same compositional resources, do not make meaning in quite the same way. Perhaps they bring their own unique semiotic resources into play as well. This realization resulted in the distinction between “modes” and “media”, which we made in our book Multimodal Discourse (Kress and Van Leeuwen, 2001). As I will discuss in more detail in Chapter 3, the semiotic resources we had described in Reading Images we now called modes, and semiotic resources that are grounded in specific materials we now called media, using the term the way artists do, as an indication of the materials used in a work. It also resulted in work on colour which we first published in 2001 and later included in the 2nd edition of Reading Images. In this work on colour, we took our inspiration, not from grammar, but from phonology, the theory of the sound, hence the materiality, of language. In Jakobson and Halle’s (1956) “distinctive feature” theory, speech sounds are bundles of simultaneously present qualities. A [p] for instance, is unvoiced and labial and plosive, among other things. Colours, we thought, can, similarly, be conceived of as bundles of features, features such as hue, value, saturation, luminosity, and so on. All of these are simultaneously present, but to different degrees, and it is through the way they combine, in their specific proportions, that colours acquire their specific character. For Jakobson and Halle, distinctive features have no meaning in themselves. They function only to distinguish phonemes, and hence words, from each other. For us, distinctive features are also semantic features. To explain how they make meaning, we drew on the cognitive metaphor theory of Lakoff and Johnson (1980), for whom meaning derives from metaphors based on concrete, embodied experience. In the case of colour, this would include, for instance, our experience of day and night, of using lights and lighting in different contexts, and so on. We also realized that there exist standardized configurations of features, cliché’s people can readily associate with specific provenances, with the times or places or other contexts from which they come. Meaning then derives from that provenance, from the associations which people have with those times, places or other contexts. In this, we built on the way Barthes (1977) described how “mythologies” such as “italianicity” can be triggered by the use of the colours of the Italian flag in an advertisement for a brand of pasta. We referred to resources such as colour as parameters, because they co-occur with other resources in specific media. Oil paintings involve shape and colour, and perhaps also texture, for instance, and animation films involve shape, colour, timbre and movement, to mention just two examples. Parameters behave differently from the systems we described in Reading Images. Rather than offering more or less binary choices (placement on the right or the left, looking at the viewer or not) that lead to other more “delicate” choices, they offer bundles of features which are always all present, but graded. The choice is not whether a colour is either dark or light, but how dark or light, how luminous or dull, how modulated or plain and so on, a colour is, and it is this that gives colours their complex, composite meanings. The two colours in Rothko’s Red on Maroon (1959), for

Introduction

3

instance, are dark, a little desaturated, warm, mixed, modulated and luminous. This makes them dark and brooding (value), complex and subtle (modulation), ambivalent (red mixed with blue), yet also deeply felt (warm) and illuminating (luminosity) – not for nothing did Rothko call his paintings “mood pictures” (Moszynska, 1990: 167). Over the past 20 years, I have developed these ideas further, together with key collaborators and friends, and applied them to work on vocal timbre (van Leeuwen, 1999, 2014; Mulder and Van Leeuwen, 2019), typography (Van Leeuwen, 2006; Johannessen and Van Leeuwen, 2018), decoration (Van Leeuwen, 2011) and texture (Djonov and Van Leeuwen, 2011), while two of my PhD students developed the approach to movement I will draw on in Chapter 7 of this book (Leão, 2012; Han, 2021, see also He and Van Leeuwen, 2019). One of the aims of the book is to update this work and bring it together in a single volume. The book therefore offers a comprehensive approach to analysing graphic shape, colour, texture, timbre and movement, and the way they interact in multimodal texts, artefacts and performances. But the book has another aim as well. Parametric systems not only make meaning in a different way, they also make different kinds of meaning. Whereas the systems we outlined in Reading Images described the functional design of images, the resources I describe in this book realize the style in which this is done – and style is, and always has been, a resource for expressing identity. This book therefore complements Reading Images, because in contemporary visual design, functional design and identity design always go together. Identity is a complex issue. It has been, and still is, differently understood and differently experienced in different times and places. In today’s complex and rapidly changing society, it is often something people search for, uncertain where and to whom they belong, and what styles of life they should adopt. The other aim of this book, then, is to study identity from a social semiotic point of view, showing how, not only stable but also hybrid and conflicted or confused identities manifest themselves through different uses of shape, colour, texture, timbre and movement, and how these uses are socially and culturally valued and regulated. The first chapter will set the stage by presenting different conceptualizations of identity in their cultural, social and historical contexts, with particular emphasis on the contemporary “lifestyle” identities that form the main topic of this book, and the way in which they transform other, older concepts of identity. The second chapter explains the differences between functional design and identity design, stressing that both always co-exist, even though identity design will form the main topic of this book. The third chapter discusses the semiotic principles that underlie how style realizes identity design. The remaining chapters describe and exemplify methods of analysing shape, colour, texture, movement and timbre, and, in the final chapter, a new take on analysing how they interact in multimodal texts, artefacts and performances. As digital technologies play a key role in providing every computer user with resources and platforms for identity design, technology will play a key role in all of these chapters. Finally, throughout the book, I will make a point of showing how 20th-century experimental artists have paved the

4 Introduction

way for contemporary identity design. The book therefore not only provides methods for analysing the values that are expressed in contemporary identity design but also seeks to understand the history of identity design, asking how and why identity design, and the values it expresses, has become such a dominant aspect of contemporary culture.

1 THE SOCIAL SEMIOTICS OF IDENTITY

Introduction The term “identity” plays a key role in today’s social and cultural life. While it may seem to suggest something fixed and definite, as often as not it refers to something searched for, something lost that must be rediscovered, or something to be created anew from a bewildering range of possibilities. As Bauman has put it (2011: 19): One thinks of identity whenever one is not sure where one belongs; that is, one is not sure how to place oneself amongst the evident variety of behavioural styles and patterns, and how to make sure that people around would accept this placement, so that both sides would know how to go on in each other’s presence. ‘Identity’ is the name given to the escape sought from that uncertainty. In seeking to understand identity, we will need to draw on two kinds of resources, sociological (and sometimes philosophical) resources for understanding what identity is, and semiotic resources for understanding how identity is expressed. There are many different ways of thinking about identity. Some focus on the unique individual characteristics that distinguish the “self” from others, whether in psychological terms such as “personality” and “temperament,” or in terms such as “character,” which carry moral overtones. Others focus on dimensions of our social selves, for instance gender, skin colour, religion, nationality and culture. Appiah (2018) has stressed how complex such identities can be – he himself was born in England with an English mother and a Ghanaian father, grew up in Ghana, is black, gay, and lives in New York. While Appiah accepts and even appreciates this complexity, others may turn a single strand of their complex identities into the

6 The social semiotics of identity

core of how they see themselves and want to be seen by others. And that is not all. The social self also includes the roles we have to play in life, as partners in different kinds of relationships, as parents or children, as workers of one kind or another and so on, and the attitudes and lifestyles of the real or virtual communities we belong to or would like to belong to. Again, some think of identity as something that can change, or even be chosen, for others identity is an inescapable label that constrains the rights and opportunities they have in life. All of these are discursive resources for understanding and for making identity. The second set of identity resources is semiotic – resources we use to express our identity in ways that others can recognize and accept as evidence of our identity. As already foreshadowed in the introduction, these are stylistic resources – styles of embodied performance (the way we speak, the facial expressions we use, the way we hold and move our bodies) and the styles of the material artefacts we use to express identity – styles of dress and grooming, the styles of the objects we use, and the styles of the settings we create for our lives. This chapter will describe four concepts of identity, together with the way they manifest themselves semiotically – social identity, in which our identity essentially derives from our place in the social order; individual identity, in which our identity is a set of consistent, individual and inner characteristics; role identity, in which we have as many identities as the roles we have to play in life; and contemporary “lifestyle” identity, which focuses on leisure time activities and consumer pre­ ferences, but also on attitudes and worldviews. I will try to show that these dif­ ferent concepts of identity have different semiotic realizations which, however, often co-exist in complex combinations.

Social identity Social identity, as I use the term here, stems from people’s place in a pre-existing social order. It has no place for a deeper self, a “real me,” separate from society. It has two key characteristics. First, it defines identity in terms of people’s relations to each other, for example in often complex kinship systems. Von Sturmer (1981:13) has described how Aboriginal Australians, when they first meet, introduce themselves in terms of their relation to each other before a conversation can be properly started: MAREEBA MAN:

Where you from? I’m Edward River man. Where you from? MAREEBA MAN: I’m Lama Lama man …do you know X? MICKEY: No, do you know Y? MAREEBA MAN: No, do you know Z? MICKEY: Yes, she’s my auntie. MAREEBA MAN: That old lady is my granny, I must call you daddy. MICKEY: I must call you boy. You give me a cigarette. MICKEY:

The social semiotics of identity

7

This differs from contemporary Western introductions, where, in meetings with strangers, people’s names (hence their unique identities) and professions (roles) are usually the first questions asked, together, of course, with nationality or ethnicity if you happen to have a foreign accent or “look different.” Second, in this kind of social identity, the social order mirrors the order of the known world. Social identity has meaning, through stories that describe this known world and its creation by real or mythical founders or original ancestors. It relates people, not only to other people but also to things, places, animals and their spiritual values. Social identity is crucially expressed on the body. Durkheim (1976 [1915]: 232) already noted that people “are led by an instinctive tendency, as it were, to paint or cut upon the body images that bear witness to their common experience.” The face is a specially important signifier of identity, and in the case of social identity, the unique face we are born with must become a social face. Maori face tattooings or mokos (traditionally they involved chiselling the skin to form furrows) are a case in point. The face shown in Figure 1.1 is a kind of identity card in which each area conveys specific identity markers – the centre forehead (ngakaipikirau) indicates rank; the area above the brows (ngunga) position; the temples (uma) marital status; the eyes and the nose kinship group (uirere); the cheek the man’s kind of work (taiohou); the chin prestige and sacred power (wairua); the jaw birth status (taitoto)

FIGURE 1.1

Maori moko. Wikipedia

8 The social semiotics of identity

and the area under the nose (raurau) bears a “signature” used, for instance, in buying property or signing contracts. But the patterns do not just signify the elements of social identity. They are also, as Lévi-Strauss noted (1963: 257) “messages fraught with spiritual and moral significance. The purpose of Maori tattooings is not only to imprint a drawing onto the flesh, but also to stamp onto the mind all the traditions and philosophy of the group.” Today, many people again seek to imprint a durable identity on their faces, or on other body parts. But, as I will discuss in more detail later on, the identities they convey are now individual identities. A New Zealand tattoo studio (http:// www.metadigital.co.nz) explains “how to tell your story in kirihuto (skin art) tattoo otherwise known as ta moko” and provides potential customers with a catalogue of motifs, whose meanings now convey individual character traits, a dog skin cloak (hikuaua) representing courage and strength, for example, and fish scales (unaunhi) abundance and health. As the catalogue explains, you can then “add the important people in your life journey” as well. In short, tattoos are now seen as giving meaning and value to people’s individual identities and life stories. To give another example, Lévi-Strauss (1962: 171) has described how the hair of boys from the Native American Osage and Omaha people was traditionally cut to indicate their clan. These clans were named after the totems (objects, animals or parts of animals, etc.) which provided the main motifs, not only for the boys’ hair styles but also for other body decorations and for objects and dwellings, and which represented the clan’s key values and ideas and its rules for what the members of each clan could eat, who they could marry, and so on. All this, and more, was expressed quite precisely and quite specifically by hair style. Like Maori tattoo motifs, such Native American “scalp lock” hairstyles have also come back into fashion, for men, women and children alike, sometimes even representing animals (the “Lizard Mohawk”). But again, rather than specific social identities, they now embody generalized values, often based on an association with warriors. In specific contexts, this can then come to signify, for instance, aggressive vigilantism (e.g. Robert de Niro in Taxi Driver (1979), rebellion (e.g. punks’ “Mohawks”), non-conformity (e.g. David Beckham, in one of his many guises) and more. Dress is another key signifier of social identity. The Prague School semiotician Bogatyrev (1971 [1935]) has described how traditional dress and hair style signified social identity in Moravian Slovakia. It could tell you where the wearer came from – there were 28 costume districts, and you could, for instance, recognize a man from Pozlovice because he would wear two velvet bands round his hat and two carmine ribbons with a green one in between, while a man from Biskupice would wear one velvet band and a red ribbon. It could tell you the wearer’s occupation – magistrates wore boots, workers the rough leather krpce, a kind of moccasin. It could tell you the social class of the wearer – squires wore bright blue breeches, peasants black or coarse white ones. It could tell you the age and marital status of the wearer – in the Mutinece-Novorany district, for instance, unmarried men wore hats with narrow rims and red and white ribbons, while married men would widen the rim and wear

The social semiotics of identity

9

a broad gold band. It could even tell you the wearer’s religion – Protestant girls would twist their hair around lacing, while Catholic girls would wear “horned” head pads. In short, costume very visibly indicated one’s place in those small, rural communities, a place which would have been experienced as necessary and meaningful, and grounded in as yet unquestioned divine authority. Bogatyrev wrote in a time of accelerating industrialization and urbanization, where such traditional communities began to be affected by the emigration of the younger generation. Earlier, Tönnies (2001 [1887]) had influentially written about the difference between Gemeinschaft and Gesellschaft – “community” and “society” – the former based on durable relations of kinship, neighbourliness and friendship, the latter on ever-changing, purely functional, contractual roles in which people act from individual rather than communal interest and values. This had created a nostalgia for (and perhaps idealization of) the idea of “community” and an interest in dialects and folkloristic customs which were in fact gradually changing or even disappearing. But few things disappear entirely. The costumes could still be revived in national celebrations, or, for instance, in the work of fashion designers such as Edwina Hörl (quoted in Mora, 2009: 75): My collection celebrates the rediscovery of my Austrian culture and multicultural traditions. It embodies a humorous mix of elements from Central and Eastern Europe: artistic handicraft, like handmade straw shoes from Austria and embroidered fabric from Czech and Switzerland. I picked up clothing themes and silhouettes that are frequently inspired by what ordinary people wear in these regions, like aprons or a Sunday suit. Traditional societies can also themselves recontextualize their identity signifiers. Anna Crane (2019) has described how the Aboriginal artist George Mung Mung created the sculpture shown in Figure 1.2. The sculpture is hand-carved from wood and painted with charcoal, white and red paint and natural ochre, and was made for a school which the Sisters of St Joseph had established for the Gija people of North-Western Australia at a time when cattle barons had evicted them from their native lands to avoid complying with a new law that obliged them to pay Aboriginal workers the same wages as white workers. George made the “Warmun Mary” after the school’s plaster statue of the Virgin Mary had broken. To mention just two examples from Crane’s detailed analysis, the dotted design traditionally identifies unmarried girls, but now also came to stand for Mary’s status as a virgin, while charcoal traditionally refers to Gija birthing practices, but now also came to stand for Mary’s role as the mother of Christ. Thus the sculpture could be used in the school both to tell the story of the Virgin Mary and to explain traditional Gija female identities and spiritual and social practices. In the words of George’s friend Hector Jandany (cited by Crane, 2019: 16), George “was looking forward to a blackfella way and a kartiya (white person) way, he was a two-way man for the Dreaming.”

10

The social semiotics of identity

The Mary of Warmun (George Mung Mung, late 1970s). Reproduced with kind permission from the Warmun Art Centre

FIGURE 1.2

To give one other example, in the Pacific kingdom of Tonga, the fakaleiti, who dress like women, have sex with men and work in helping professions, have long been an accepted and appreciated identity, as are, for instance, the fa’afafine in Samoa, and the mahu in Hawaii. Traditionally, if a Tongan woman had too many sons and not enough daughters, one of the sons would be brought up as a daughter and assist with women’s work such as cooking and house cleaning. “Mothers are just tickled by having a boy who becomes a leiti and will put him in frilly clothes and make him dance” (Besnier, cited in Munro, 2015). These days fakaleitis may experience their gender both in traditional Tongan and Western ways. They may say “My mother wished I was a girl and she prepared everything for me to be a girl and when I was born I was not” as well as “I am a woman trapped in a man’s body (…) People say we choose to be what we are but I didn’t choose to be a leiti – I

The social semiotics of identity

11

was born like this.” (Munro, 2015). But they have no doubt that being a leiti is a socially accepted identity: “I find life here for leitis is very normal. I don’t get much discrimination here, people respect me so much. They say it is very open in Australia, but I find it hard when I go over there because they don’t respect you, they look you up and down with a bad look” (ibid).

Individual identity The kind of social identity’ described in the previous section has been the norm in human societies. But from the Renaissance on, a new way of thinking about identity gradually began to emerge. Here the “real self ” was no longer a social self, but an individual self which saw itself as separate from society, and therefore as needing to come to terms with it – conform to it, maybe, or struggle to fit in, or even rebel. In Berger’s words, society became an “external reality that exerts pressure and coercion upon the individual” (Berger, 1963: 110). The anthro­ pologist Clifford Geertz found it “a rather peculiar idea within the context of the world’s cultures” (Geertz, 1983: 59) McLuhan saw it as “the supreme artefact of Western man” (quoted in Erikson, 1976: 25) Of course there were predecessors. Already in Ancient Athens, Theophrastus, a disciple of Aristotle, wrote a book called The Characters, in which he described a number of character types, always introducing them with the phrase “He is the sort of person who…” and then describing them in terms of their habits, e.g. (as quoted in Frow, 2014: 108) The Chatterer is the sort of person who sits down beside someone he doesn’t know and begins by delivering a panegyric on his own wife, continues with an account of his dream of the night before; then describes in detail what he had for supper … (etc) Theophrastus was rediscovered in the 17th century, a time when the modern idea of “character” began to take root. However, in Ancient Athens identity was still social. To be alone, isolated from society, meant losing your identity. In one of Sophocles’ tragedies, Philoctetes, it is, according to MacIntyre (1981: 135) essential to the action that Philoctetes, by being left on a desert island for ten years, has not been merely exiled from the company of mankind; but also from the status of a human being: ‘You left me friendless, solitary, without a city, a corpse among the living.’ Christianity, on the other hand, taught that we have, besides our earthly social identity, an immortal soul, an identity before God in which we are all equal. Such an identity could persist even when it was not socially recognized. Many early Christians were slaves, and slaves, in Ancient Rome as in Ancient Greece, were not recognized as having any form of identity. Christianity gave them their

12

The social semiotics of identity

identity back, albeit that this identity was not of this world. The same idea still resonates in secular discourses, for instance in a report issued by the “California Task Force to Promote Self-Esteem and Personal Social Responsibility” which stated (as quoted in Fukuyama, 2018: 94) that every person has unique significance, simply because the precious and mysterious gift of life as a human has been given. This is an inherent value which no adversary or adversity can take away. The key characteristics of individual identity, as it emerged from the Renaissance onwards, are uniqueness and inwardness.

Uniqueness In the Renaissance, the particular and unique, rather than the general and the social, became the hallmark of what is real. As can be seen in the rise of the realistic portrait, the face now was no longer a social face, “a semiotic instrument for the display of messages, and thus, in principle, akin to the mask” (Frow, 2014: x), but a particular face, “that part of the body that is most expressive of inner character” (ibid) and “an acute manifestation of the individualized ‘I,’ the homeless private self, astray in an overwhelming world” (Sontag, 1977: 119). The face now expressed those inner characteristics, read, not semiotically, as signs, but psycho­ logically, as symptoms. And so did the body. The German psychiatrist Ernst Kretschmer (1921), for instance, described body types as revealing character traits – the “asthenic type” was thin and lanky, signifying introversion and ti­ midity, the “pyknic type” was stocky and fat, signifying friendliness, interpersonal dependence and gregariousness, and so on. More recently, Jerome Kagan (e.g. 1997) distinguished between “high reactive” (inhibited) and “low reactive” (uninhibited) personalities and related these personality traits to physical char­ acteristics – high reactivity to blue eyes, a thin body and a narrow face, for in­ stance. Such symptoms can become signs again in theatrical and movie casting and in the design of cartoon characters, something which well deserves further study. Disney, for instance, gave Cinderella and Pinocchio blue eyes, and more extroverted characters such as Cinderella’s stepsisters or Peter Pan darker eyes. The idea of inborn, physiologically based characteristics has a long history, starting with the theory of the “four humours” (internal secretions) of Hippocrates, around 400 BC – blood, yellow bile, black bile and phlegm – which, respectively, relate to sanguine (cheerful, vigorous, optimistic), choleric (hot tempered), mel­ ancholic (depressed, moronic) and phlegmatic (slow moving, calm, unexcitable) temperaments. Many modern psychologists use the term “temperament” for such inborn patterns and “personality” for the result of the influence of “nurture.” But they nevertheless believe that inborn characteristics will always re-emerge, for in­ stance in moments of crisis. One well-known theory of personality is that of the British psychologist Hans Eysenck (e.g. 1967), whose still widely used personality

The social semiotics of identity

13

test is based on a range of factors that can combine in different ways, factors such as introversion/extraversion, neuroticism/stability and psychoticism/socialization. Each of these is then associated with certain personality traits e.g. “psychotic” with “aggressive,” “assertive,” “egocentric,” “unsympathetic,” “manipulative,” etc., and extraversion with “sociable,” “irresponsible,” “dominant,” “sensation seeking,” “impulsive,” “active,” etc. There is an important difference between such psychological accounts of “personality” and the notion of “character.” Character traits have moral implications, whether they are positive virtues such as humility, courage or generosity, or negative traits such as jealousy, greed or rudeness. Such traits can be developed (“character building”), or, in the case of negative traits, combatted and vanquished. Personality traits, on the other hand, are understood as genetically given, even if they can be moderated under the influence of environmental factors.

Inwardness A second key characteristic of individual identity is “inwardness.” Virtues that would formerly manifest themselves in humble, or brave, or generous deeds, are now thought of as individualized, psychologized, inner characteristics and “no longer deployed in great deeds of military valour in public space, but rather in the inner domain of passion by thought” (Taylor, 1989: 153). This inner life is then revealed and celebrated in literature, especially in the novel, “the affective and moral technology of self-shaping inwardness…that conforms much of our contemporary understanding of what it means to be a person” (Frow, 1989: x). The stories novels began to tell from the 18th century onward, were no longer about past kings and heroes, but about contemporary ordinary people (and more inclusive of women) and the inner lives of these characters were often in conflict with the rules and customs of society. This also meant that the semiotic expression of individual identity by styles of dress and grooming became less important. From the 19th century on, the dress of, especially, “bourgeois” men became homogenous, “fashion-exempted (…), simple, coarse, unchangeable and sombre” (Davis: 1995: 39). Even the Futurist artists of the early 20th century, who in their writings advocated “daring clothes with brilliant colours and dynamic lines” (Balla, quoted in Appolonio, 2009: 132), still dressed quite homogeneously, as can be seen in Figure 1.3. Women’s dress, however, was more varied. In his book on colour, Goethe (1970 [1810]) asso­ ciated colour with character, and this idea was soon further developed, first by the painters of the German Lucasbund, who devised colour codes for the expression of character (but, again, only in women, because they thought that the colour of male dress was determined by their profession, as indeed it often was in the age of industrialization) and later in colour psychology, which integrated the idea of colour preferences in personality tests (cf. Van Leeuwen, 2011: 22–26).

14

The social semiotics of identity

Russolo, Carà, Marinetti, Boccioni and Severini (unknown photo­ grapher, 1912). Wikipedia

FIGURE 1.3

But precisely because dress was now more homogeneous, it created room for personal accents that were, again, understood as symptoms rather than signs of identity. Floch (2000: 19–21) gives an example, a photo of two children, twin brothers, who wear “the same tie, the same blazer, the same sweater, the same socks,” a uniform that expresses the institutional identity of an English public school. But (ibid: 23): One of the children wears his socks impeccably pulled up straight, while the other has let his right sock slip down his leg. Also, this boy’s shirt sticks out between his shorts and his sweater – and he too is the one who has not bothered to button up his jacket and who has allowed his collar to stick out over his sweater. Floch then characterizes this boy as “mischievous” and “untidy,” and his twin brother as “serious and dignified.” Similar differences can be seen in Figure 1.3. Individual identity is also meant to be consistent, throughout life, whatever it brings: “Someone who genuinely possesses a virtue can be expected to manifest it in very different types of situation,” (MacIntyre, 1981: 205). Identity thus man­ ifests itself as a coherent narrative, a story with a beginning, a middle and an end: “I am what I may justifiably be taken by others to be in the course of living out a story that runs from my birth to my death“(MacIntyre, 1981: 217). The beginning of the story is particularly important, because childhood is now seen as the foundation of the “unique, ingrained and enduring inner self” (Lynch, 1998: 62),

The social semiotics of identity

15

as shown, for instance in the Bildungsroman, which deals with the development of character in the early stages of life, and in the importance of early experiences in psychoanalysis. Like novels, biographies and autobiographies can be seen as “technologies” for modelling character. Here is an extract from a biography of Primo Levi (Angier, 2002: xiii) Primo Levi’s most characteristic and unvarying trait was his reserve. It is in his books: in that just, detached narrative voice, whose most unbuttoned mood is quiet irony; in the narrator himself, on the rare occasions he appears, as confidant, adviser, assistant, peacemaker. We feel we know him – and love him – because we know every movement of his gentle, rigorous, open mind. But it is only his mind we know. He very rarely betrays his feelings, and almost never his negative feelings This excerpt displays all the characteristics I have discussed. There is a set of inner characteristics – “reserved,” “detached,” “gentle,” “open-minded,” but also “rigorous.” These traits are consistent – “characteristic” and “unvarying,” with an exemplary consistency between the man (“the narrator himself”) and his work (“the book”). And they are inner traits, traits that characterize his “mind” rather than his actions. Individual identity continues to play a role in popular media, especially in relation to stories about the “characters” created by authors or played by movie actors and actresses. In television talk shows, guests are asked to reveal their true selves, and hosts seek to link their work to their life. An author has invented a character who rebels against his father – did he, himself, in his own life, rebel against his father? An actress plays the role of a working class woman – did she, herself, ever work in a factory or, acceptable substitute, spend arduous months “researching” the role in an actual factory? Bell and van Leeuwen (1994: 195) have described the appearance of actress Cheri Lunghi in such a talk show, Aspel and Co. In a recent starring role in the television series The Manageress, she had played the manageress of a football club in stories that turned around her character’s difficulty in being accepted in a man’s world, despite her obvious competence. So Michael Aspel first asked her whether she grew up amongst men. Alas, this turned out not to be the case – she grew up amongst women. Aspel then asked about her ex­ periences during the shooting of the series. Was it difficult to be the only woman in the cast? One way or another, her role in the series had to be anchored in her own, real experience, her “true self.” But with postmodern artists such as Madonna, who deliberately change their persona from time to time, identity is no longer consistent. As Aspel put it (quoted in Bell and Van Leeuwen, 1994: 221) Non-celebrity humanity that is really I think what we’re after. The mask slipping, or hoping that it will. I think Madonna would be anathema to this sort of thing, I think if we had Madonna on the show it would be a lot more

16

The social semiotics of identity

like a political interview, we would actually have every excuse for being firm and direct and challenging, because with a politician, you don’t know whether there is any kind of truth in there at all, and so you have every right to demand to know what it is.

The individual and the social Society, in the age of “character,” is no longer the traditional Gemeinschaft com­ munity, held together by ties of kinship and locality. It centres on the one hand on the micro society of the couple and the family, and on the other hand on the macro society of the nation state. From the 19th century onwards each nation state began to develop a “national character,” a “homo nationalis” with characteristic traits, dispositions and behaviours, with shared cultures and traditions (often in­ vented, cf. Hobsbawn and Ranger, 1983), and with shared stories, histories and songs (Wodak et al., 2009). In my Dutch primary school in the 1950s, patriotic songs such as this one, dating from 1815, were still sung in class: Whoever has Dutch blood flowing in their veins Free of foreign blemishes Whose heart glows for king and country Rejoice in song as we do Or this one from 1910: Dutch flag, you are my glory Dutch flag you are my desire From pure joy I shout ‘Victory’ When I see you on foreign shores Today such songs continue to be celebrated on websites as “old school songs,” meant to be nostalgically remembered. Like individual characters, national characters were understood in terms of “character traits.” During WWII the US Office of War commissioned American anthropologists such as Ruth Benedict, Gregory Bateson and Margaret Mead to study the national character of the countries America was about to liberate, so as to ensure effective propaganda. Ruth Benedict characterized the Dutch as “morally righteous,” “individualistic,” “tolerant,” “freedom-loving,” “selfconfident,” “proud,” “ironic,” “puritan,” “tidy,” “cautious,” “thrifty,” “con­ servative,” “domesticated,” “serious” and “somewhat melancholy” (Van Ginkel, 1991: 41). As Van Ginkel (ibid) has shown, her analysis corresponded by and large to what Dutch historians and sociologists had also written. Such characterizations have by no means died out. The Australian national character, as described for instance in the work of historian Russell Ward (1958), was based on shearers, stockmen and drovers (yes, all men), seen as exemplifying

The social semiotics of identity

17

anti-authoritarianism and masculine solidarity. This may now seem a caricature, but it still resonated in the 2020 Australia Day speech of Prime Minister Scott Morrison – Australians as “gritty,” “open-hearted and generous” and “supporting each other when the going gets tough”: Our willingness to generously support each other is a reflection of the gritty practicality that has always been part of our national character (…) we celebrate an Australian spirit that is open-hearted and generous, and a people that support each other when the going gets tough. The biography of Primo Levi I discussed above seeks to reconcile individual and social (in his case religious rather than national) character. The “reserve” which biographer Angier had previously described as Primo Levi’s “most characteristic and unvarying trait” was also a trait of the environment he grew up in – Turin, in Catholic Italy (Angier, 2002: xiii–xiv). There is a high concept of personal and family honour: confessions are for the confessional, and it is always a moral duty (…) to cover up, to keep up appearances. To this cultural and Catholic base the Turinese added their own Puritan, almost Victorian reserve (…) Primo Levi was a Jew; so were – naturally – all his relatives and many of his friends. But as one of them (a Jew) said to me, when Jews assimilate they become 110 per cent like their neighbours (…) in that great reserve, therefore – as in several other things – Primo Levi was a typical, a conventional Turinese. But Angier soon reaffirms the autonomy and independence of Primo Levi’s “reserve” – he chose it (ibid: xv): The rational and the irrational were his two sides (…) but he chose to live on only the rational half of himself and closed the door on the other. This was his armour, but also the gap in it. He repressed half of himself, and it is that repressed half which I try to reach in my own repressed chapters. Semiotically, this results in what Floch (2000: 23) called “a double principle of difference and repetition.” His two brothers wearing the same school uniform in different ways, are different in their individual characteristics, but homogeneous in their public, institutional character. The same pattern can be seen in handwriting. From the 19th century on, each European nation taught its own style of handwriting, in disciplined and uniform ways. However, on this basis, and at the same time, people could develop their own unique, instantly recognizable and consistent hand, as expressive of their character as their face. Handwriting thus reveals national character (and period) as well as individuality, although graphologists have mainly focused on individual

18

The social semiotics of identity

character traits, as in this website “graphology chart,” which lists 57 different character traits of the kind illustrated below: Aggressiveness Altruism Ambition

heavy pressure, angles, upstrokes straight and diagonal, speed, exag­ gerated right thrown tending t bars Right tending movement in the middle zone Uneven pressure, speed, large capitals rising t bars, rising lines, extensions into upper and lower zones

Role identity American role theory of the 1950s and 1960s radically broke with the idea of identity as unique, inward, consistent and autonomous. Identity once again became social, but in a different way now, based, not on durable social rela­ tions, but on functional roles which are more or less given, “scripted,” before we get to play them. As a result identity is no longer solid and unified, but multiple and changeable: “if you want to ask who someone really is, then it is the sum total of all the situations in which the individual is one thing or another, held together by the slender thread of memory” (Berger, 1963: 124). According to Park (1950: 249–50) It is probably no mere historical accident that the word person, in its first meaning, is a mask. It is rather a recognition of the fact that everyone is always and everywhere, more or less consciously, playing a role […] It is in these roles that we know each other; it is in these roles that we know ourselves. Behind these roles, there is, therefore, no longer a “true self” – Goffman (1959: 30): In a sense, and insofar as the mask represents the conception we have formed of ourselves – the role we are striving to live up to – this mask is our truer self, the self we would like to be. In the end, our conception of our role becomes second nature and an integral part of our personality. This new conception of identity originated in the sociology of G.H. Mead (e.g. 1967), for whom acquiring an identity and discovering society were one and the same thing. According to Mead, the child learns “to take the role of the other,” starting with the “significant others” in the child’s environment, then extending this to the “generalized” other, that is, society at large. I will discuss three key characteristics of role identity – emotive investment, social recognition and multiplicity. To start with emotive investment, playing a role is not just a matter of acting out a “script,” it has to be done “with feeling”: “roles carry with them both certain actions and the emotions and attitudes that

The social semiotics of identity

19

belong to them” (Berger: 113). They may not be “inner” but they are inter­ nalized. It is of course possible to play roles insincerely, to pretend, but in reality this rarely happens. It is difficult to do and it is easily spotted. Normally the role becomes an identity: “one becomes what one plays at” (ibid: 115). Rafaeli and Pratt, in an article about dress codes (1993: 44) quote a prison guard: Once you put on that uniform then you are not the same person. You really become that role. You put on that khaki uniform. You put on the glasses. You take the night stick. You act the part. A second characteristic is the need for social recognition. As with social identity, identity must be socially ratified, and when it is not, identity is lost, or transformed into another identity. This differs from individual identity, which can in principle be maintained, even in extreme circumstances, for example in concentration camps, as attested by Angier in the case of Primo Levi who remained true to himself, even in Auschwitz (Angier, 202: 298): He had, above all, fortitude. Consciously he needed to believe in human goodness; but emotionally he expected little for himself and was used to doing without it. Like Vanda he suffered very much from fear, because secretly he always expected the worst; unlike Vanda, when the worst happened it roused in him a stubborn combative side, which refused to be defeated by the forces of darkness According to role theory, on the other hand, someone “turned overnight from a free citizen into a convict” would “with frightening speed discover that he is acting as a convict is supposed to, and feeling all the things that a convict is supposed to feel” (Berger, 1963: 118). As we have already seen, role theory also implies that identity is multiple, “the sum total of all the situations in which the individual is one thing or another” (Berger, 1963: 124). The question then becomes how we change from one identity to another, which mostly involves keeping the different roles strictly apart and, when playing a particular role, forgetting about other identities. Thus mur­ derous gangsters can also be loving husbands and fathers, as is frequently portrayed in movies and television series. In The Lawyer, a current Swedish television series, a crime boss with many murders on his conscience dotes on his daughter, but is in the end unable to keep the two roles separate. The examples role theorists such as Goffman (e.g. 1959) use are mostly from work roles, for instance in the service industry and the health sector, although role theorists were early in recognizing that “even identities we consider to be our essential selves have been socially assigned” (Berger, 1963: 115), and this already included identities such as race and gender. In role identity, the semiotic expression of identity (the sign rather than the symptom) becomes important again. As Goffman (1959: 32) explains it, “impression

20

The social semiotics of identity

management” crucially involves three kinds of “expressive equipment” – setting ,“furniture, décor, physical layout, and other background items which supply the scenery and stage props for the space of human action” (ibid: 33); appearance, “in­ signia of office or rank, clothing, sex, age and racial characteristics, size and looks, etc” (ibid:” 34); and manner, “the interaction role the performer is expected to play” (ibid: 35), e.g. “aggressive” or “meek,” as expressed by speech patterns, facial ex­ pressions, bodily gestures, etc. Such “equipment” then expresses the values that characterize the role and give meaning to it, but these values differ from those expressed by social identities and individual identities. Identity performances in service encounters, for instance, may express role-related values such as “cleanliness, modernity, competence, integrity” (ibid: 35). “Impression management” of this kind is, today, also increasingly important in politics, as in this assessment of the performance of Australian Prime Minister Scott Morrison in interviews: “the PM’s polite yet rigid approach proves he’s an elite athlete in the sport of maintaining a façade” (Mathieson, 2020: 3).

Lifestyle identity Lifestyle identity has much in common with role identity. It, too, is a form of social identity. It, too, is multiple and changeable. It, too, is semiotically expressed, through setting, appearance and manner. But there are also differences. Where role identity focuses mostly on the functional roles we play in institutional settings, lifestyle identities focus on leisure time activities and on consumption patterns. They seek to give meaning to our personal life and connect us with like-minded communities. They are for “people for whom occupational and economic roles no longer provide a coherent set of values and for whom identity has come to be generated in the consumption rather than in the production realms” (Zablocki and Kanter, 1976: 270). In the case of lifestyle identities, social recognition is therefore not a matter of successful “impression management,” of skilfully and sincerely performing a role for audiences or clients, but a matter of seeking belonging, affiliation with com­ munities that share their understandings of the world, their affiliation with certain values and attitudes, and express it through similar ways of dressing and grooming, and through the interests and activities they share. On this basis they can recognize others and be recognized by others. Yet lifestyle identity differs from traditional social identity. It is not based on a pre-existing social order, but individually and freely chosen, and thought of as something that can be changed (“adjusted” and “reinvented”) as “trends” change. According to social researcher Mark McCrindle (quoted in Schipp, 1920) Traditionally we think about tribes based on demographics and age (…) This is a move to ‘psychographics’ – more social trends and attitudes and lifestyles. We are who we are not because of our age or family situation, but because of our attitude to life and how we view it, adjust and reinvent.

The social semiotics of identity

21

Styles of dress, consumer goods, interior decoration, and so on, are therefore also individual and creative choices, notwithstanding the importance of “sharing” them with online or offline communities. In the words of style icon David Beckham (2000: 94), who has experimented with many styles, now dressing as a family man, now as a rebel: I don’t think we should be sheep and follow anyone else. We’re individuals and should be prepared to show that in our behaviour. Clothes are just one way of expressing your individuality, but it’s an important one for me. I also think of dressing as a way of being artistic and art is something I’m quite into. But the resources for doing so are no longer individual. Lifestyle is not expressed through idiosyncratic personal accents imposed on homogeneous practices, as in the age of individual identity. It is expressed by choices from goods and services made available, across the globe, by powerful corporations, and the values these goods and services express are compatible with the interests of these corpora­ tions, even in the case of writing. If, in the age of individual identity, hand­ writing was the personalized version of a uniform national style, in the age of lifestyle, it is a choice from the fonts made available by Microsoft to many millions of users, and that choice expresses quite different kind of values. Rather than personality traits such as “aggressive,” “altruistic,” and so on, we may now encounter social values such as those described in an article in the Dutch version of Cosmopolitan (October 2001: 75): According to psychologist Aric Sigman the font you choose on your computer for writing letters reveals a great deal about your personality: Courier New Georgia Helvetica Times New Roman Verdana

a little oldfashioned you have flair modern confident professional

Today, lifestyle identities and corporate identities are becoming increasingly si­ milar. The self can come to be seen as a “brand,” displayed online to seek re­ cognition in the form of “likes” and “shares,” and corporate logos increasingly use cursive and irregular fonts to suggest the authenticity and individuality of hand­ writing (Johannessen and Van Leeuwen, 2018). Role style and lifestyle begin to intermingle. The cabin crew of an Austrian airline I travelled with, wore not only red waistcoats, white shirts and blouses, red and white dotted scarves and ties but also blue jeans, combining a traditional “service profession” uniform with casual,

22

The social semiotics of identity

individually chosen jeans. The furniture in office spaces and the living rooms of private apartments increasingly resembles each other. The idea of lifestyle emerged in the 1970s in the field of marketing, for instance in the work of Arnold Mitchell (1978), who introduced the term “psycho­ graphics,” defining it as clusters of “behaviours,” “attitudes” and consumption patterns. Lifestyle is therefore not only social because it seeks community but also because it is closely linked with the new marketing strategies and the new ways of influencing and controlling people’s behaviour that are now everywhere facilitated by digital technologies. There is, on the one hand, Zuckerberg’s vision of Facebook as enabling and enhancing lifestyle, as “freeing us up to spend more time on the things we all care about, like enjoying and interacting with each other and expressing ourselves in new ways” (quoted in Zuboff, 2019: 403), and on the other hand, Facebook as secretly assessing its users across “twelve categories of ‘needs’ including excitement, harmony, curiosity, ideal, closeness, self-expression, liberty, love, practicality, stability, challenge and structure” so as to identify “motivating factors which influence a person’s decision making across five di­ mension – self-transcendence/helping others, conservation/tradition, hedonism/ taking pleasure in life, self enhancement/achieving success, and open to change/ excitement” (Zuboff, 2019: 273, 276). These two ways of talking about technology – technology as liberating and democratizing communication, and technology as increasing surveillance and control – can also be found in academic literature. Many social and cultural theorists have critiqued essentialist conceptions of identity and described con­ temporary identity as context-dependent, fluid, complex and multiple. This has of course been crucially important in work on race and gender, for instance in Homi Bhabha’s (1990) critique of the West’s essentialization of other cultures, in Stuart Hall’s (1989) work on the Caribbean diaspora and its rediscovery of Africa, and, with respect to gender, for instance in the work of Butler (1990) and Connell (1990), who have argued against essentializing masculinity and femininity as stable identities and inner essences. But it also resonates with the lifestyle discourses of the marketers Others, such as, recently, Turkle (2011), Zuboff (2019) and Davies (e.g. 2015, 2020) have lamented the disappearance of true intimacy and privacy, and of the “autonomous individuals whose inner resources and capacity for moral judgement rise to the challenge of self-authorship that history demands and act as a bulwark against the predation of power” (Zuboff, 2019: 469). Zuboff quotes a student (2019: 472) as saying: The backstage is shrinking. There is almost no place left where I can be my true self. Even when I am walking by myself and I think I am backstage, something happens – an ad appears on my phone or someone takes a photo, and I discover that I am on stage and everything changes.

The social semiotics of identity

23

Summary The four types of identity described in this chapter manifest themselves in four different kinds of style – which, in today’s complex, multicultural societies, often mix or merge in different ways, e.g. in the way lifestyles can draw on traditional social styles of dress and decoration. Social style uses relatively stable and semiotically rich ways of bodymodification, dress and grooming and of decorating artefacts, dwellings and other structures. It not only indexes people’s place in the social order (what, today, we would call demographics) but also expresses people’s relation to the world they live in, and its meanings. Individual style expresses people’s unique identity, understood in terms of inner character traits that manifest themselves in consistent, individual ways of wearing functional forms of dress, speaking and writing socially prescribed national lan­ guages, and, in the arts, performing fully scripted plays and music, to mention just some examples. It may be more or less unconsciously acquired, in which case it will be seen as “symptomatic” and psychological rather than as semiotic and cultural, or deliberately developed, for instance by modernist artists. National style is based on the same principles. Role styles are ways of performing roles, usually in more or less institutionalized settings. Like theatrical performances, they come with specific ways of dressing and grooming, and specific props and settings. They are often institutionally regulated, or even scripted, but must nevertheless be “owned” and performed sincerely, “with feeling.” Lifestyles are ways of performing leisure time activities, which, like role styles, come with ways of dressing and grooming, accessories and settings (and, today, also body modifications such as tattoos, cosmetic surgery and gender reassignment surgery), and which are understood as conveying attitudes and worldviews. Though individually chosen, they are shared with communities, whether on- or offline. There are many such communities but the resources they use are often provided by powerful global industries, even if they may then be used in con­ textually specific and often creative ways. Today, role styles and lifestyles (the public and the private) increasingly intermingle, perhaps accelerated by an increase in working from home caused by the COVID-19 pandemic. Industrialization, with its migration from country to city, heralded a welldocumented shift from social identity to individual identity and role identity. It created a world in which functionality, what we do, became detached from identity, from a stable sense of who we are and what the meaning of our lives is. This often resulted in alienation and a search for meaningful identities and meaningful ways of expressing these identities – a search for styles with substance, styles that can again express values worth living by. Functionality continues to be the cement of society, but it is now newly reunited with style, and hence with identity. It is this new relation between functionality and identity I will discuss in the next chapter.

2 FUNCTIONALITY AND IDENTITY

Functional design and identity design If we think of the functional design of a pair of spectacles we think of its con­ struction in terms of what it does – making us see better – and its ergonomic design, how easy and comfortable it is to wear. If we think of its identity design, we think of what it means in terms of identity, how it looks and how it makes us look. During the 18th and 19th centuries, there was a focus on functional design. Many functionally different types of glasses were invented – the pince-nez, which pinches the bridge of the nose; the lorgnette, which must be held in front of the eyes with a handle; the monocle, which covers only one eye and is said to have been developed by an antiquarian to closely inspect antique gems, and so on. New frames can still be inspired by functional considerations such as protecting the eyes of pilots or people involved in extreme sports. But today the functional design of spectacles is increasingly uniform: two lenses, plus or minus rims, the bridge that sits on your nose, the nose pads that make the bridge comfortable, and the hinged “temples” that fit the glasses over your ears. However, this basic functional design comes in many different styles, and choosing a style is no small matter. It is part of expressing your identity. It can make you appear bold or modest, adventurous or studious, fun-loving or serious, and so on. The identity design of glasses is a relatively recent phenomenon. Brand names such as Zeiss and Rodenstock used to index standards of functional, rather than identity design. When “designer glasses” began to be mass marketed in the late 1980s, such names were replaced by fashion brands – Prada, Gucci, Chanel, and so on. This soon became a global phenomenon. While charities had long distributed pairs of second-hand glasses from richer to poorer countries, in 2001 the World Health Organization advised them to stop this practice – the people in poorer countries did not want to wear outdated glasses. “Being poor doesn’t mean we

Functionality and identity

25

want to look stupid you know” according to one NGO spokesman (Knight, 2018). In short, we have moved from a focus on the functionality to a focus on the semiotics of objects. Identity design works with the style of frames, with their concrete, observable qualities – their shape, colour, texture, materiality. Lenses and rims can be round, oval, rectangular and many shapes in between. The colours of the frames (and, in the case of sunglasses, of the lenses themselves) can vary widely, as can their textures, which may be mottled, marbled, speckled, striated, and so on. And the materials the frames are made of can be light or heavy, rigid or flexible, hard or somewhat softer, and so on. Such concrete qualities have a semiotic potential. Physical lightness may signify “cheerfulness,” actual flexibility can become me­ taphorical flexibility, and so on. Identity meaning can also stem from provenance, from contextual and cultural references – Ray-Ban’s aviator glasses can evoke the real or imagined qualities of pilots, frames associated with iconic celebrities the qualities associated with those celebrities – John Lennon’s round glasses, the squared shades and thick plastic frames of Bob Dylan’s Wayfarer sunglasses; Marilyn Monroe’s Cat Eye glasses or the heart shaped lenses famous from Stanley Kubrick’s Lolita. Contemporary identity design is deliberately created, in the case of spectacles by the eyewear industry. The Italian frame manufacturer Luxicotta produces more than 27,000 different frames. It has bought up most of the well-known brands such as Ray-Ban, Vogue, Prada, Persol, Sunglass Hut, etc. and works closely with Armani, Ralph Lauren, Chanel and others to, quite deliberately, “transform a physical device into a statement of style” (Knight, 2018). Since its merger with Essilor, the main manufacturer of lenses, it sells close to a billion pairs of lenses and frames each year, charging consumers 10–20 times more for a frame than it costs to make. This move from functional to identity design underlies not only product de­ sign, fashion and architecture, but also many other contemporary practices, in­ cluding text production and social interaction. The text reproduced in Figure 2.1 (slightly adapted for copyright reasons) comes from a research project on Cosmopolitan magazine which David Machin and I conducted in the early 2000s (Machin and Van Leeuwen, 2003, 2004, 2005). It is clearly a multimodal text, using, not only language and image (in the original, the image of the young man is a photo) but also colour, typography, layout and visual framing (in the original the shirt of the young man is pale blue, the background behind him a blue-tinged grey, and the background of the textbox is lilac, with white bullet points and “headlines”). These modes of expression realize the functional as well as the identity design of the text. Its functional design demarcates the functional elements of the text, the ele­ ments which, in their particular order, make the text into what Machin and I called a “hot tips genre”: (1) the reader’s imagined Dilemma, expressed both textually (“Does he want to be your new dude?”) and visually, by the young man’s unsmiling, enigmatic look at the viewer; (2) the “Hinge“ (“Here are the hints that

26

Functionality and identity

THE GUY A focused stare means he’s keen.

DILEMMA

DOES HE WANT TO BE YOUR NEW DUDE? Here are the hints that your best guy friend would prefer some romance.

He acts disinterested. Once he feels he has a chance with you, his dude instincts kick in. So if you’re feeling neglected, ask him why he’s been ignoring you lately. If he can’t answer, he’s into you. He always looks worried. When a best guy friend wants to become your dude, he cannot relax. Tight facial muscles are a sign that he’s worried, as is a furrowed brow, and sweaty hands.

HINGE TIP 1

TIP 2

He asks your friends for advice. you might think that by asking your friends for advice, that he’s starting to ignore you. Far from this, this means he’s into you big time.

TIP 3

He checks out the competition. Guys are territorial animals. When he sees you around other dudes, he’s looking out for signs of a spark. By keeping his eye on other dudes, he’s showing he’s keen.

TIP 4

FIGURE 2.1

Hot tips genre (after Cosmopolitan magazine, November 2002, US edition)

your best guy friend would prefer some romance”) that connects the image to the text box; (3) four “Tips,” each bullet-pointed and headlined in white lettering: “He acts disinterested,” “He always looks worried,” “He asks your friends for advice,” and “He checks out the competition.” Each of these “Tips” is then elaborated further. “He asks your friends for advice,” for example, is followed by

Functionality and identity

27

“You might think that by asking your friends for advice, that he’s starting to ignore you. Far from this, this means he’s into you big time.” As a whole, this functional structure, with its specific sequence of elements or “stages,” aims to provide the reader with guidance in the tricky area of dating, where women, so the magazine suggests, always run the risk of being either too forward or too cautious. The structure is summarized in Figure 2.1, alongside the text itself. Three things are worth noting. Firstly, at the time we conducted our study, Cosmopolitan applied this structure to many types of content, not only relation­ ships, but also health, beauty, career, etc. In other words, functional structures of this kind are relatively content-free. They are templates that can be used over and over and fit many different contents. But their style can be varied, along with their content. Secondly, the magazine applied the format globally, in all of the 42 different versions it published at the time, with the exception of the Japanese version. The format is therefore not only content-free but also context-free. And although it has been argued that “hortatory texts,” texts that tell us what to do and how, have a universal “deep structure” (Longacre, 1974), the “hot tips” genre provides a particular kind of guidance, quite different, for instance, from the traditional problem-solution magazine genres in which well-credentialled experts author­ itatively solve readers’ problems. Cosmopolitan, steered from New York by the Hearst Corporation, addresses its readers as independent “fun, fearless women” who solve their own problems, with the help of a set of options provided by the magazine, rather than, as would still be more common in many of the countries in which the magazine is published, with the help of, for instance, family, com­ munity or religious leaders. This underlines the continuing importance for social semiotics of Bakhtin’s call for the development of “a special history of speech genres […] that reflects more directly, clearly and flexibly all the changes taking places in social life […] speech genres are the drive belts from the history of society to the history of language” (Bakhtin, 1987 [1952]: 65). Third, the template is multimodal, and it is through their use of visual design that functional and identity design are brought together. The text’s functional design visually demarcates its functional elements and makes its overall structure instantly recognizable – the white line that separates the “Dilemma“ and the “Tips” marks them as distinct parts of the text, while the “cool” colour scheme and the use of white lettering in both the “Dilemma” and the text box binds them together. The four “Tips,” similarly, are spatially separated and made distinct by having their own bullet points and white titles, yet also form a coherent, symmetrically ar­ ranged classification syntagm, indicating their common identity as “Tips.” This structure will be familiar to Cosmopolitan readers from other uses of the same template, and therefore also fulfils an ergonomic function, making the text easy to read and use. From the point of view of identity design, colour is here particularly important, especially the colour lilac. In Victorian times, people had a more explicit under­ standing of the meanings of flowers and their colours, which they used when

28

Functionality and identity

choosing which flowers to give as a present. In this context, lilac signified the first bloom of youthful love, as lilacs are the first flowers to bloom and last only a few weeks. But even disregarding that, lilac is a paler, more diluted version of the sensuality of violet, and hence conveys a “cool” romantic sensibility, mixed in with the purity and innocence of white, and the organic roundness and generous spacing of the typography. Thus functional design and identity design, function and meaning, fuse, and at the same time add an aesthetic dimension (Van Leeuwen, 2015), hence a certain kind of pleasure in the text. To sum up: • •

•

•

Many contemporary texts and semiotic artefacts combine functional design and identity design. In functional design a set of functional elements combines in a particular procedural (temporal) or constructional (spatial) order that realizes an overall practical goal in an ergonomically optimal way. In identity design, physical qualities of the functional elements – colour, texture, shape and materiality – realize the styles which express the values that make up identities. Today, functional design is increasingly homogeneous across different con­ texts, and across the globe, whereas identity design is increasingly diverse, as it has to produce both individual self-expression and affinity between the members of like-minded communities.

The rise of genericity and the rule of the template Genericity is the key principle of contemporary functional design. In the case of texts, it provides the socially standardized ways of combining functional elements in par­ ticular procedural or constructional orders which, today, we call “templates.” It is such templates, many of them built into semiotic technologies, which allow the increasing homogenization of functional design and diversification of identity design. The history of the grids used in graphic design can serve as an example (cf Djonov and Van Leeuwen, 2013). Grids are orthogonal structures of horizontal and vertical lines whose intersections define regions of different sizes for the placement of written text and images on printed or electronic pages. Any given content (e.g. a paragraph or an image) can occupy one or more of the resulting modules, and the more fine-grained or granular the grid, the greater the potential for forming spatial zones with various sizes and orientations. During most of the 20th century, grids were used for “proportional regulation,” that is, for aligning the size and relations of the elements to each other in ever new ways, and in accordance with what a page or spread is meant to convey: “The underlining principle behind any grid is that it is most successful as an expression of content” (Kane, 2002: 53). From approximately the 1960s, however, graphic designers moved away from this approach and began to design re-usable grids with “se­ mantically distinct zones” (Samara, 2002: 20), layouts that could accommodate

Functionality and identity

29

different contents. In other words, the grid became generic, a template. According to Italian designer Massimo Vignelli, one of the pioneers of this approach, “design should reject the individual impulse for expression in favour of developing overall systems.” Such “systems” could then become vehicles for combined functional and identity design, as in Vignelli’s 1964 posters for the Piccolo Teatro in Milan, which had distinct semantic zones for the name and address of the company, the dates of the particular play advertised, the title and director of the play, the cast, the creative team, and the ticket prices, while its style, its colours and fonts created a distinctive and immediately recognizable brand for the company. The “hot tips genre” shown in Figure 2.1 works in the same way – as do most web pages, with their standardized format of clearly framed and predictably placed functional elements (logo top left, global navigation bar on top, local navigation bar on the left, main content in the centre, sub-content and links on the right, and legal and privacy matters in small print at the bottom). Such templates restrict designers’ freedom to experiment with the many choices which layout makes in principle available, but it does allow for the creative use of images, fonts and colour schemes. Another area in which genericity has been on the rise is literature. As discussed in the previous chapter, novels, often serialized in magazines and newspapers, emerged as a highly popular art form in the 18th century. Moving away from the stock characters, settings and plots of earlier forms of literature, they portrayed the unique, individual life stories of ordinary people, with characters understood as motivated by an enduring inner self. As a result, genre literature, with its stan­ dardized characters, settings and themes, became less highly valued. This is now changing. Already in the late 1950s, the critics of the French film magazine Cahiers du Cinéma began to celebrate Hollywood directors of genre films such as Westerns and melodramas as “authors,” and Stuart Hall and Paddy Whannel’s The Popular Arts (1964) argued that the Westerns of John Ford were works of distinction, well worth critical attention and studying in schools in the way in which, thus far, only “high literature” had been studied. Today, it is genre fiction, especially crime fiction, which produces the bestsellers that fill the shelves of airport bookshops, with stock characters such as the grumpy, divorced, middle-aged police inspector who prefers a particular brand of whisky and adores his daughter and the deranged serial killer who mutilates his victims in gruesome ways. Academic writing, too, is characterized by increasing genericity, with models originating in science, and an emphasis on “research questions” and highly technical, often computer-aided, methodologies rather than theoretical arguments and historical narratives. All this has been greatly aided by digital resources for text production such as Word and PowerPoint. Word, for instance, provides, and constantly updates, thousands of templates such as the one in Figure 2.2 – templates for brochures, invitations, reports, business letters, invoices, order forms, thank you cards, etc. At the time of writing, it offered no less than 154 different templates for business letters. Such templates authoritatively impose the synergy of functional design and identity design I have discussed in this chapter. On the one hand, they constrain

30

Functionality and identity

FIGURE 2.2

Microsoft Word template

what users can and cannot do. In the case of Figure 2.2, for instance, users cannot decide which functional elements the design should contain, how much space these elements should take up, and where they should be placed. There must be a photo top left, with the title and date of the event below it. There must be an “event description” below that, and a logo below that. And there must be several points of “key event info” in the righthand bar, as well as information about “your company” and dates. On the other hand, they facilitate diversity in identity design.

Functionality and identity

31

What users can change is the picture itself, the colours and the fonts, in short the style that will not only make the design “look good” but also convey something about “your company” and its values, though always on the basis of an ultimately limited range of predetermined options. PowerPoint is based on the same principle. On the one hand, it compels users to make each slide a self-contained unit, an aspect which was part of PowerPoint’s design from the start (Gaskin, 2012: 61), and to choose whether or not to use the default “title plus list of bullet points” format. Changing this takes a little time and effort, and most people do not bother. In a study of 34 corporate and academic PowerPoint presentations (Djonov and Van Leeuwen, 2013: 23), all but two used the standardized bullet point format. On the other hand, PowerPoint provides many style options, many options for varying the background, the colour scheme, the typography and page furniture such as the dividing lines between the title and the bullet points – and hence many options for expressing identity. The Head of Finance at a University where I worked used PowerPoint to present the University’s annual budget, alternating slides with budget figures and charts with slides quoting business gurus, and using a calm blue seascape with wispy clouds as his background. In this way, he sought to present himself as a visionary leader rather than a bean counter. But, as in the case of Word, such choices are selected from a pre-determined range of “themes” and favour particular kinds of identity. You can choose “modern” and “future-oriented” or “traditional” and “retro”; “business-like” and “technical” or “organic” and “eco-conscious”; “concise” and “focused” or “elaborated with flourishes and aesthetic elements,” and so on – all values that matter in the global corporate culture which has brought PowerPoint into the world. PowerPoint was originally designed by engineers from the Bell Laboratories (Gaskin, 2012) to pitch ideas to management for funding. Such pitching of ideas, as memorably depicted in Robert Altman’s film The Player (1992) in the case of Hollywood, has to be bold and concise. Since then PowerPoint has been used for many other purposes, including of course education. According to Microsoft, some 35 million PowerPoint presentations are created every day. However, rather than PowerPoint adapting to these contexts, they had to adapt to it and learn to provide information in the form of limited lists, rather than extended arguments or historical narratives. This risks the omission of detail, qualification, evidence and comment. Edward Tufte, in The Cognitive style of PowerPoint (2003: 9), described how a PowerPoint presentation by Boeing engineers at NASA assessed the risk posed by debris which had hit the Columbia shuttle just after take-off: These reports provided mixed readings of the threat to the Columbia; the lower level bullets often mentioned doubts and uncertainties, but the highlighted executive summaries and big-bullet conclusions were quite optimistic. Convinced that the reports indicated no problem rather than uncertain knowledge, high level NASA officials decided that the Columbia was safe and that no additional investigations were necessary.

32

Functionality and identity

On re-entering the earth’s atmosphere, however, the shuttle disintegrated, and all seven crew members died. Tufte’s conclusion included many of the points I have discussed in this chapter, including PowerPoint’s “preoccupation with format rather than content” and its use of a “single-path model for organizing every type of content”: PowerPoint … is costly to the content and the audience. These costs arise from the cognitive style of the standard default PP presentation, foreshortening of evidence and thought, low spatial resolution, an intensely hierarchical single-path structure, a model for organizing every type of content, breaking up of narratives and data into slides and minimal fragments, rapid temporal sequencing of thin information rather than focused spatial analysis, conspicuous chartjunk and PP phluff, branding of slides with logotypes, a preoccupation with format rather than content, incompetent design or data graphics and tables, and smirky commercialism that turns information into a sales pitch and presenters into marketeers. To give one more example, Microsoft SmartArt offers 233 different versions of just nine basic diagram templates such as “lists,” “processes,” “cycles,” “hier­ archies,” etc. SmartArt describes these in functional terms. “Lists” should be used “to show non-sequential or grouped blocks of information,” “processes” “to show progression or sequential steps in a task, process or workflow,” “cycles” “to show a continuous sequence of stages, tasks, events, in a circular flow,” “hierarchies” “to show hierarchical information or reporting relation­ ships.” The style of the different versions of these basic types then makes it possible to express values such as “creativity,” “innovation,” etc. Shape is a key style element here – text boxes, for instance, may take the form of closed rectangles, closed circles, open circles, chevrons, etc., and such shapes, as we will see in Chapter 4, have a symbolic meaning potential. Literal openness, as in Figure 2.3 can come to express the idea of “openness” or “transparency,” for example. Other diagrams add background shapes – a background arrow (Figure 2.4). for instance, may suggest “progress,” “growth” or “sense of di­ rection.” In this way, companies can represent themselves as “open,” “pro­ gressive,” “personable,” and so on. Choices of colour, typography and texture then offer further potential for identity design. Like PowerPoint, SmartArt started with a very specific aim, providing just six diagrams, which were all explicitly related to office management – workflow charts and org charts. In 2007, Microsoft broadened this to “relations” and in­ troduced an element of aesthetics (the “art” in SmartArt). Since then SmartArt, though still modelled on management practices and values, has come to be used in many other practices including education and academic research in the humanities and social sciences. As Kvåle has concluded (2016: 269)

Functionality and identity

FIGURE 2.3

SmartArt “Half-Circle Organization Chart”

FIGURE 2.4

SmartArt “Continuous Block Process”

33

34

Functionality and identity

Ideologies from organization management are thereby infused into higher education – not explicitly by verbal instruction, but by being buried in the templatized formats for multimodal representation as part of the technolo­ gization of discourse.

Design and society The relation between functional design and identity design, which I have here dis­ cussed in semiotic terms, has roots in sociological theory. Weber (e.g. 1947, 1976 [1930]) distinguished between Zweckrationalität (goal-oriented rationality) and Wertrationalität (value-oriented rationality). In the case of goal-oriented rationality, functionally rational individuals or institutions set goals and then design the means for achieving these goals. Weber associated this with magic – deterministic, repetitive and unreflective procedures which nevertheless seek to achieve particular goals. These goals are “given,” not open for discussion, and the means, whether prayer, research or managerial processes, are “techniques,” valued in terms of their efficiency and ef­ fectiveness. Bureaucratic rationality was Weber’s key example. Value-oriented rationality, on the other hand, views the world in terms of meanings and moral categories. This Weber (1976 [1930]) associated with re­ ligion, especially puritan Calvinism, in which traditional orientations became selfconscious, and in which instrumental action was grounded in moral responsibility (e.g. honesty is useful because it assures you credit), and the values that come with it, e.g. self-control, conscientiousness, frugality. Later sociologists elaborated these ideas. Mannheim (1936) renamed Weber’s two types of rationality “functional” and “substantial.” His “functional rationality” was based on action, his “substantial rationality” on meaning, on understanding the relations between apparently separate phenomena. Habermas (1984) re­ cognized four types of action, based on different “validity criteria” and realized by different kinds of speech act. Two of these corresponded to Weber’s distinction: teleological action, which focuses on the means needed to achieve a goal, and normconformative action, which, like Weber’s “value-oriented rationality,” focuses on morally grounded action. But Habermas added two further types of action, dra­ maturgical action, which focuses on self-expression and is expressed by style, thus related to the notion of identity design developed in this chapter, and communicative action, in which participants make their different understandings and values explicit in dialogue, so as to allow consensual, co-ordinated action. In Habermas, the first three forms of action, together with their ways of reasoning and understanding the world, belong to distinct domains of practice – teleological action to domains in which knowledge is principally stored in the form of technologies and strategies; norm-conformative action to legal-moral institutions and dramaturgical action to the arts and other institutions which foreground subjectivity over objectification and morality, for instance therapy. I have argued that instrumental action and what Habermas called “dramaturgical action” have become merged, not only in

Functionality and identity

35

marketing practices but also in many other domains of social life, and in the public as well as the private sphere. The “objective” values of “true” and “right” have made place for the instrumental and subjective values of “it works” and “I like it,” which fuse in forms of communication (and forms of interpretation, under­ standing) that combine functional and identity design. The social semiotics of Halliday (1978), took another approach. It recognized the difference between ideational meaning-making, which seeks to understand the world, and interpersonal meaning-making, which centres on doing things for or with or to other people and on the relations this creates. But here both were in­ terpreted in functional terms. Language, and, by extension, other semiotic modes, were thought of as grounded in instrumentality. According to Halliday, children learn the functions of language before they begin to learn to speak, using their vocal resources to explore what can be done with language and, to that end, developing a “proto-language” of 50-odd vocal noises. Only the mother and perhaps one or two others will understand these, but the child will get her message across and learn that language is a means to achieve your personal goals, and to regulate, or even ma­ nipulate, others’ behaviour, rather than, for instance a set of rules, or a system of understanding the world that must be “acquired.” The child is therefore zweckra­ tional from the start. At about the age of 18 months she then “moves into the adult language,” generalizing the seven “protofunctions of language” into three “meta­ functions”: understanding the environment (ideational metafunction), acting on the others in it (interpersonal metafunction), and “breathing relevance into the other two” (textual metafunction) (Halliday, 1985: xviii). This functional approach was later also applied to the analysis of other semiotic resources, for instance in Reading Images (Kress and Van Leeuwen, 2006), in which we assumed that all semiotic systems will be able to fulfil all three of Halliday’s metafunctions, and then in­ vestigated how they can be realized in visual communication. In that work we therefore focused primarily on functional design, although in the second edition we added a chapter on colour. The current book seeks to complement Reading Images by focusing on identity design. But it should be remembered that, in many con­ temporary semiotic practices, the two go together. In functional theories of language, and in semiotics generally, style was ne­ glected, or seen as aesthetic, with aesthetics seen as pleasurable, poetic or beautiful, but ultimately meaningless, an end in itself. The Prague School semiotician Mukařovský (1964: 19) saw the “poetic” as a use of language which is “not in the service of communication,” but “foregrounds the act of expression, the act of speech itself,” and this idea was taken up widely in literary stylistics, which often focuses on parallelism, on formal style features such as alliteration, rhyme, vowel harmony and assonance, without reference to meaning. But foregrounding is not without meaning. Mukařovský, for instance, saw the use of archaism, foreign expressions, dialect words, and so on, as “poetic” because it disrupts the “auto­ matization” of the “standard language” and draws attention to language itself. But it also introduces the ideas and values readers may associate with the period from which the archaisms come, or the countries from which the foreign expressions

36

Functionality and identity

are drawn, or the groups which speak the dialects. And playing with words is always also playing with meaning, as demonstrated, for instance, by Geoffrey Leech (1969: 69): In ‘I kissed thee ere I killed thee’ (…) the parallelism urges a connection between kissed and killed (…) which combines contrast with similarity. Kissing and killing have opposite connotations, the former being associated with love, the latter with hate and aggression. On the other hand, the sentence as a whole suggests that they are similar: that kissing and killing are compatible actions. On a wider scale, therefore, this parallelism summarizes with great concentration the paradox of Othello’s jealousy, and the irony of his final tragedy. The post-structuralist French semioticians also saw aesthetics as meaningless. For the later Barthes, the stylistic parameters I discuss in this book provide a kind of pleasure, jouissance, which is individual, pour moi, and escapes “the laws of history, culture, psychology” (Barthes, 1973a: 25–26) Kristeva saw colour as, on the one hand “situated within the formal system of painting,” but on the other hand as “in­ stinctual” and “destroying normativity.” “It is through colour, she said, “that the subject escapes its alienation within a code, representational, ideological, symbolic, and so forth” (Kristeva, 1980: 221). But arguably they fought a rearguard battle, defending individual identity at the very moment it began to give way to new forms of social identity – defending the individuality of the voice at the very moment companies such as Acapela began to create synthesized voices, offering clients “your own exclusive voice” and a wide range of “voices that surprise you with their naturalness” (http://www.acapela-group.com), and defending the subjectivity of colour at the very moment organizations such as the Color Marketing Group began to determine, each year, which colours will dominate the market, “appear in retail, health and wellness, multifamily homes and spas, among others” and “emotionally and tactilely touch us”(Thomson, 2019). Others sensed the new unity of aesthetics, functionality and meaning early on. The American semiotician Charles Morris defined the meaning of aesthetic signs as “value” and “interest” (1938: 418), illustrating this idea with a simple example: while ordinary signs might refer to objects (eg food), aesthetic signs refer to values and interests connected to objects and in that way “release a reassessment of the content.” Even Mukařovský saw that aesthetics plays a role in “almost all acts of man,” including “sexual selection, fashion, social amenities, and the culinary arts” (1964a: 19), and Jakobson (1960), argued that the “poetic” functions alongside all other communicative functions in all uses of language. Sociological and philosophical ideas relate closely to real world developments, sometimes reflecting them, sometimes anticipating them in what Mumford (1934) called “cultural preparation”; sometimes critiquing them, sometimes legitimating or even celebrating them. Utilitarianism, and American pragmatism anticipated, and went hand in hand with, new practices in, for instance, architecture and

Functionality and identity

37

product design. By the 19th-century industrialization had begun to require new types of buildings such as railway stations and factories. While houses and tradi­ tional public buildings were still built in heavily ornamented styles, for these buildings new, purely functional materials and construction methods (e.g. steelframe building) were developed. It was an architect, Louis Sullivan, who, in the late 19th century, introduced the idea of “form follows function,” which would become gospel for generations of designers across the globe. In the words of the Czech architect Karel Teige (1929: 40): Architecture creates instruments. Aesthetic intervention in utilitarian design inevitably leads to an imperfect object. It obscures the utilitarian aspect and considers practical values such as comfort, warmth, stability, to be necessary sacrifices which people should make for cultural tradition…Where the idea of style prevails, form precedes function. The same applied to product design. Once the industrial revolution had started, people began to reflect on the difference between handmade and machine-made objects, and about the nature and purpose of what had come to be called “dec­ oration” or “ornament,” but which had, in the age of traditional social identity, always been part of everyday objects, and always meaningful. An influential mid19th century textbook, The Grammar of Ornament, argued that”ornament (…) has no business beyond appealing to the eye, in order to entertain the imagination in free play with ideas, and engage actively in the aesthetic judgment independently of any end” (Owen Jones, cited in Brett, 2005: 109). William Morris, on the other hand, in Some Hints on Pattern Design (1895: 177) insisted on meaning: “You may be sure that any decoration is futile when it does not remind you of something of which it is a visible symbol (…) I must still insist on plenty of meaning in your patterns.” But soon decoration would be banned altogether. In a famous tract, Ornament and Crime, written over a hundred years ago, the Austrian architect Adolf Loos wrote “Herein lies the greatness of our age, that it is incapable of producing new ornament. Ornament is no longer organically linked with culture. The evolution of culture is synonymous with the removal of ornament from utilitarian objects” (Loos, quoted in Brett, 2005: 195) and Le Corbusier (1987 [1925]: 188) followed suit, declaring that “Every citizen must replace his hangings, his damasks, his wallpapers, his stencils with a plain coat of white Ripolin.” And from the early 20th century onwards utilitarian types of design were actively propagated and produced by the influential artist-designers of the Bauhaus in Germany, the Werkbund in Austria and so on. Language, too, came to be functionalized, for instance in journalism. In a 1915 brochure, Reuters already wrote of “compressing news into minute glo­ bules” that would eliminate the stylistic flourishes which had been common in 19th-century journalism and focus on facts and facts only (Palmer, 1998: 184). A study of an English language Vietnamese newspaper (Machin and Van Leeuwen,

38

Functionality and identity

2007) showed how the Western subeditors who introduced this style to local journalists discouraged the wordplay and poetic flourishes that were prized by Vietnamese journalists. As for linguistics, Halliday (1985), however much he was influenced by the Prague School functionalists, did away with their “poetic function” altogether. But history is never simple. As functionalism triumphed, advertising and branding had already started to sow the seeds for the reintroduction of aesthetics (and hence also of pleasure and affect) in formerly purely functional forms of communication, in effect functionalizing aesthetics and aestheticizing function­ alism. In advertising, style always comes with “substance.” Even Mukařovský recognized this when he said that advertising uses “a euphonious sequence, an unusual pattern… to attract attention first to the wording and then to the thing advertised” (1964a: 39). In the age of the “marketization of discourse” (Fairclough, 1993), the “functional aesthetics” of advertising has gradually per­ meated other areas of social communication, leading to the synergy of functional and identity design I am describing in this chapter. Just as medieval communication integrated aesthetics with the communication of the theological and chivalrous concepts and values of the time, so contemporary communication integrates aesthetic pleasure with the concepts and values of global corporate culture, whether in the form of personal lifestyle identities or in the form of corporate branding. Architecture, too, has rejected functionalism. Not the relation between the functional elements of a building, but the façade or the overall shape became central, for example through keeping the facades of older buildings and building new structures behind them, or by covering buildings in decorative cladding. But, hidden from the eye, the buildings behind these façades are built from prefabricated concrete modules, still functional and increasingly homogeneous. The return of style has also manifested itself in the increasing importance of performance. In the age of individual identity, performance was understood as the realization of pre-existing scripts, scores and so on, without adding meaning of its own, just as, in language, the shape of letter forms and the sound of speech sounds, was thought to realize pre-existing content, without adding further meaning. This has now changed, both in practices of musical and theatrical performance, and in linguistics. The sociolinguist Hymes, for instance (1972: 62), defined “key” as “the tone, manner or spirit in which an act is done,” as expressing values such as “perfunctory” vs “painstaking” or “quiet” vs “exu­ berant,” and as multimodal, able to be realized verbally as well as non-verbally, often through aspects of speech that were traditionally disregarded in linguistic analysis, such as aspiration and vowel length. As we have seen in Chapter 1, Goffman (e.g. 1959) also foregrounded performance, with social actors per­ forming roles to enact values such as “being up to date,” “competent,” “sin­ cere,” “friendly” and so on, through their appearance and their style of speech and non-verbal communication. More recently Coupland (2007) introduced the concept of “high performance,” in which professional performers, often through

Functionality and identity

39

the media, produce “identity stylizations” to construct “social personas” which then influence people’s “self-styling,” their “projection of an attractive in­ dividuated self” (ibid: 188), or, in terms of this book, their identity design. All of this has made style a key concept for social semiotics and created the need for developing ways of analysing how it works in contemporary semiotic practices. It is this I will address in the next chapter.

3 ANALYSING STYLE

Introduction In Chapter 1, we saw how different kinds of identity are expressed through dif­ ferent kinds of style. Traditional social styles express socially given, durable iden­ tities such as clan, gender, age, marital status and rank through customary forms of bodily hexis and performance (including music and dance), body modification, grooming and dress, and customary forms of decorating artefacts, dwellings and other structures. Individual style expresses individual characteristics, through subtle differences in the otherwise homogeneous styles of, for instance, standardized forms of handwriting, “received pronunciations” of national languages, and more or less fashion-exempt styles of dress, for instance the width of stripes on standard business shirts or suits – “both the banker and the gangster wear striped suits and shirts (…) narrow and discrete in the first case, wide and garish in the second” (Pastoureau, 1991: 68). Role styles are ways of performing more of less institutionalized roles, which come with specific ways of dressing and grooming and specific props and settings that are often institutionally regulated, whether formally or informally. Lifestyles express the common attitudes, beliefs, interests and tastes of like-minded communities, through leisure time activities and, again, ways of dressing and grooming (e.g. hair styles), objects (e.g. cars) and settings (e.g. interior decoration) and through leisure time activities. But they also retain elements of individual style, as they are no longer durable and unavoidable, but freely chosen and changeable. Here individuality is an active process of what Coupland (2007) called “self-styling.” Just as individual style interacted with, and formed part of national character, lifestyle interacts with, and forms part of, the new, market-driven, global con­ sumer society. In this, lifestyle communities differ from traditional social groups. Marketing gurus such as Seth Godin (2008) may refer to them as tribes’, speaking

Analysing style

41

of “consumer tribes,” “tribal marketing,” and so on, and arguing that the market takes its cues from lifestyle communities (Godin’s book is called Tribes We Need You to Lead Us), but lifestyle communities must work with what the market provides, and advertisements, lifestyle magazines, television makeover pro­ grammes and, above all, the internet, constantly elaborate the meanings and values on offer. Finally, while the styles of traditional societies are more or less fixed, even if gradually developing over time and able to adapt to new circumstances, this is not the case with lifestyles. Rather than offering fixed forms of expression for more or less stable, given meaning systems, designers now create meaning potentials that can be understood and used differently in different contexts, in ways even designers may not always be able to foresee. Taking dress again as an example, a white business shirt combined with brown brogue shoes may mean one thing when combined with a staid business suit, another when combined with skinny ankle jeans, with the shirt not tucked in and the shoes worn without socks. There is always an element of individual or group creativity and originality, both in the process of design and the process of its take up by users. But lifestyle also comes with an element of anxiety: in a world where nothing is stable and where fashions change all the time, it is difficult to securely settle down in an identity, difficult not to keep searching, especially, perhaps, for young people. The psychoanalyst Erik Erikson (1963: 271), reflecting on the evolution of American identity, noted that “The patient of today suffers most under the problem of what he should believe and who he should – or indeed might – be or become; while the patient of early psychoanalysis suffered most under inhibitions which prevented him from being what and who he thought he knew he was.” Figure 3.1 further illustrates the difference between traditional style and life­ style. Persian rugs use simplified motifs representing animals and plants that have particular spiritual meanings in the life of nomadic and herding people – a camel might stand for wealth and prosperity, a snake for wisdom, and so on. The meanings of such motifs were clear and well-known to all members of the culture, even though today most users of Persian rugs will not be aware of them and attach quite different meanings and values to Persian carpets. But what is the meaning of the other carpet? Here many interpretations are possible. What to make of the irregular and ragged-edged pentagons and hexagons, with their fleecy texture, and their more or less concentric arrangement? What to make of the green and white colours, reminiscent of forests and snow? Different attributes of the carpet might be picked up, and different associations conjured up, by different interpreters, depending on their interests and values. As we saw in Chapter 2, people have, for well over a 100 years, discussed whether patterns such as these are merely dec­ orative or meaningful. Yet people (or companies) choosing this particular pattern for their carpets or curtains will somehow find it meaningful and expressive of the values they stand for, whether in the realm of personal or in the realm of corporate choices, just as in the case of the PowerPoint backgrounds discussed in the pre­ vious chapter.

42

Analysing style

FIGURE 3.1

Persian and modern carpet. iStock

In the domain of lifestyles, meaning-making is to some extent always a creative process, both on the part of designers and on the part of interpreters and users. It is not possible to describe style as if it were a language, with a lexicon that fixes his­ torically established relations between signs and their meanings, and a grammar that regulates how signs may combine in sign-complexes. All we have are creative principles, operating both in the process of creation and the process of interpretation and use. It is this that allows lifestyle communities to “own” lifestyle meanings, even though their signifiers are often provided by powerful global industries, in the same way that people develop their own way of using writing software such as Word and PowerPoint, while at the same time working within systems developed by Microsoft. Can lifestyle then be analysed at all, or is any analysis doomed to be subjective? Undoubtedly and inevitably, analysing contemporary identity design is to some extent also a creative act. It is of course possible to represent identity design as a rigid system, to turn the analogue into the binary, but such an analysis would no longer be faithful to the way practices of identity design and interpretation actually work, and should perhaps be avoided. This does not mean that style analysis will be entirely subjective. It can still accurately observe what there is to be observed and analyse and interpret it in principled and plausible ways. This chapter seeks to explain how this might be done, focusing on three main principles of style analysis – experiential meaning potential, provenance and contextuality.

Experiental meaning potential Metaphor is a key principle of semiotic creativity. Aristotle knew this more than 2000 years ago: “Ordinary words convey only what we already know, it is from metaphor that we can best get hold of something new” (Aristotle, 1954: 1410b). The essence of metaphor is transference – transferring something from one “place” to another, on the basis of a perceived similarity between the two “places.” Most often that something is a word, transferred from one meaning to another on the basis of a partial similarity between the two meanings. The word metaphor itself is an example – it originally meant “transport” (in Greece you can see it written on

Analysing style

43

every lorry). Clearly the ancient Greeks perceived a similarity between trans­ porting goods between places and transporting words between meanings. The idea of experiential metaphor focuses on a particular kind of metaphoric transference, the transference of our experience of concrete material qualities to more abstract ideas relating to these qualities. This idea derives from Lakoff and Johnson (1980), who argued that the understanding (and we might add, the creation) of metaphors is based on concrete experience: “No metaphor can ever be comprehended or even adequately represented independently of its experiential basis.” This includes bodily experiences shared by all humans, such as walking upright, the physical experience of handling objects and our experience of human interaction (Lakoff and Johnson, 1981: 117), hence universal as well as socially specific experiences. Take the example of speech, the materiality of language itself. We all know how in some circumstances our voice may become tense, and hence higher, sharper and brighter, because in their tensed state the walls of the throat cavity dampen the sound less than they would in their relaxed state. The resulting sound not only is tense, it also means tense, and this meaning derives from our experience of the circumstances in which our voice becomes tense – when we feel threatened, for instance, or when we have to restrain strong emotions, whether anxiety or excitement, to mention just some of the possibilities. This range of experiences therefore creates a meaning potential. It can come to mean a range of things – “anxiety,” “repression,” “fear,” “excitement,” and so on, and how that potential will be actualized and narrowed down then depends on the context – the specific situational context as well as the broader cultural context. Alan Lomax, in an overview of singing styles across the world, has observed that in cultures where the sexual repression of women is severe, a high degree of tension in female singing is considered beautiful (Lomax, 1968: 194): It is as if one of the assignments of the favoured singer is to act out the level of sexual tension which the customs of the society establish as normal. The content of this message may be painful and anxiety-producing, but the effect upon the culture member may be stimulating, erotic and pleasurable since the song reminds him of familiar sexual emotions and experiences. The lyrics of a song can also provide context, as in this analysis of Madonna’s performance of “Like a Virgin,” in which she tenses her voice on lines like “I made it through the wilderness,” and “I was beat, incomplete/I’ve been had, I was sad and blue” (Van Leeuwen, 1999: 153): In ‘Like a Virgin’ she addresses a ‘you’ who saved her from perdition and made her feel ‘new and shiny', ‘like a virgin touched for the very first time’. She addresses this ‘you’ in a high, ‘feminine’ voice. But she combines this ‘little girl’ voice pitch with a more hardened sound, quite tense and strident, bearing the scars of abuse and betrayal

44

Analysing style

The same approach can be applied to analysing material objects rather than bodily performances. Here, too, meaning can derive from our experience of perceiving the material qualities of the object or our experience of what we do when we produce or use the object. Take the example of irregular written or printed words. The teaching of handwriting in schools has always emphasized regularity and neatness, control and discipline. Experience tells us, however, that writing can become ir­ regular for a range of reasons (Johannessen and Van Leeuwen, 2018: 186) – because we lack the skills needed to produce regular writing, as is the case, for instance, with young children, because we refuse to produce neat, regular writing for one reason or another, because the tools and materials we use make it difficult to produce reg­ ularity, as when we try to write with a springy brush or with watercolour, or on a paper serviette, or because of infirmity or intoxication. We also know that even neat handwriting is always more irregular and less mechanical than printing. When ir­ regularity is deliberately produced, as it very often is, we understand what it means on the basis of these experiences and of the context. On an invitation to a children’s party it may mean playfulness, on the cover of a heavy metal album rebellion, on the menu of an expensive restaurant unique quality and personal service. Although the idea of experiential meaning potential was inspired by Lakoff and Johnson’s metaphor theory, it differs from their account in a number of ways. First, Lakoff and Johnson work primarily with linguistic evidence, although they briefly mention that the conceptual structure which metaphors set up “is not merely a matter of the intellect – it involves all the material dimensions of our experience, including aspects of our sense experiences: colour, shape, texture, sound, etc.” (Lakoff and Johnson, 1980: 235). Second, my focus is not on conceptual structures, but on the creative semiotic processes that realize style, and not on specific con­ ceptualizations, but on meaning potentials and the way they are narrowed down in context. Some of the “orientational metaphors” Lakoff and Johnson discuss also have a very wide meaning potential, but their discussion of the concepts these metaphors can stand for does not involve context, except when they explain that conceptual frameworks, taken as a whole, are not universal, and that understanding metaphors always occurs in a cultural context. There is less emphasis on the social here, and less emphasis on the role of human agency in design and interpretation. The idea of experiential meaning potential is also indebted to Gibson’s theory of affordance (1979) and the way it has been taken up in semiotics, especially by Kress. The drawing in Figure 3.2 is one of Kress’ crucial examples, used throughout his work of the 1990s and after. The three-year-old child, he explains, had just learnt to draw circles, and this had opened up the multiple representational affordances of circles. From these the child selected the affordance of circles to represent wheels, and hence, metonymically, cars, something which he had, at that moment, an in­ terest in drawing. Generalizing from this and similar examples, Kress concludes that material signifiers “carry a set of affordances from which sign-makers and inter­ preters select according to their communicative needs and interests in a given context” (Kress and Van Leeuwen, 2006: 232), and that the result of that process, “like all signs and sign-complexes, is a metaphor, newly made” (2011: 55).

Analysing style

FIGURE 3.2

45

“This is a car” – drawing by a three-year-old child

This clearly and plausibly describes how young children explore meaning-making, but it cannot be a general theory of meaning-making. It would not apply, for instance, to contexts in which individuals have to follow contextually specific rules. But it does very much apply to the way meaning is made in contemporary identity design, where innovation is fundamental and where meaning, like consumer goods, must be cus­ tomized, individualized, leading to a fragmentation of identity meanings, which is then counterbalanced by the homogenization of functional design I discussed in the previous chapter. It is therefore not surprising that Gibson’s work has been widely taken up, not only by semioticians but also by designers. It aptly describes the new way of meaning-making that has emerged in the age of lifestyle. The idea of experiential metaphor underlines that materiality is fundamental in meaning-making. This was not recognized in the linguistics and semiotics of the age of individual identity. In phonology, the linguistic approach to the study of sound, speech sounds were defined in functional terms, as serving to express meanings made at the more abstract level of lexis and grammar, distinguishing words from each other without themselves contributing to meaning. The “p” and the “b,” for instance, expressed the meaning difference between the words “pet” and “bet.” That p’s and b’s can be pronounced in many different ways (styles) was ignored. Yet, as poets have long understood and as will be discussed in detail in Chapter 8, spoken language makes meaning at two levels – the level of word meaning and the level of sound meaning. Here is Murray Schafer (1986: 180–1), using the materiality of the “p” and the “b” and the physicality of their articulation to describing their meaning potential: the [b] “has bite. Combustive. Aggressive. The lips bang over it,” and the [p] is “Combustive, comical. Pip pop pout. Listen to the soft popping of the pipe smoker.” Written language also has these two levels of meaning. According to Bellantoni and Woolman (1999: 6): The totality of the word produces two meanings. One is related to the idea represented by the word itself, constructed from a string of letters – the word-image – and the other from its holistic visual manifestation – the typographic image.

46

Analysing style

The same applies to music and to visual communication. In Reading Images (Kress and Van Leeuwen, 2006) we abstracted away from the materiality of the signifier and described functional systems that can be applied to materially different visual media – photography, drawing, paintings, cinema, websites, and so on. But we also began to realize that this was not sufficient (ibid: 215): In music, the performance of a composition contributes a great deal to its meaning, and in many cases it is difficult, if not impossible to separate composition and performance. In visual communication, similarly, the material production of a design is not just the execution of something already complete, but a vital part of meaning-making. At the functional level, meaning making can be described systematically, as a more or less binary system of choices, as we did in Reading Images. The system network in Figure 3.3, for instance, describes the different ways in which classifications, “kind of” relations between items, can be visually realized – through a tree diagram (“covert taxonomy”), which can be single-levelled, having only one branching level, or multi-levelled, or by distributing items symmetrically across the visual space and making them equal in size, so that identical size and placement comes to stand for identical class membership (covert taxonomy). But making meaning with materiality works differently. It combines, first of all, a set of parameters, for instance colour and texture and graphic shape, or melody and rhythm and timbre, welding them into a multimodal unity, and, second, a set of what, following Jakobson and Halle (1956) Kress and I called the distinctive features of these parameters – colour for instance combines hue and value and saturation (and several other features, as will be discussed in Chapter 5), and timbre combines tension and pitch and loudness and roughness (and other features, as will be dis­ cussed in Chapter 8). Moreover, these distinctive features are always all at play – and they not binary, not a matter of either/or choices, but gradable, “more or less” choices, scales which run, for instance, from maximally dark to maximally light, or from maximally tense to maximally lax. The relevant systems, if indeed we can them “systems,” look like the second diagram in Figure 3.3, with the curly bracket representing simultaneous choices, choices which must all always be made, and the double-headed arrow gradation (it should be noted that the drawing does not represent all the distinctive features of colour. All distinctive features carry experiential meaning potential. The meaning potential of “value,” for instance (the scale from maximally dark to maximally light) is based on our experience of light and dark, a universal experience which, in different contexts, can evoke quite different meanings – divine revelation in a Rembrandt painting, for instance, or upbeat “brightness” in a Hollywood co­ medy. And the meaning of what we see, hear and feel derives from the meaning potential of all these features, as contextually narrowed down. A given colour is never just red or green, it is always, for instance, red and light and pale (pastel) and translucent, and so on. Similarly, a voice is never just high or low, but always, for

Analysing style

47

covert taxonomy classification

single-levelled overt taxonomy multi-levelled

dark value light saturated saturation desaturated pure purity impure transparent transparency opaque warm temperature cold ......

FIGURE 3.3

A system network (top) and a parametric system (bottom)

instance, high and lax and soft and breathy. The total impression then blends all these things together, just as the taste of a dish blends the taste of all its ingredients in their different proportions. Clearly, this makes for an endless number of pos­ sibilities. It is a key principle of creative design. But describing the blended meanings that result remains elusive, a problem I will address in Chapter 8.

Provenance Experiential meaning potential is only one of the resources of identity design. Another key resource is “provenance,” the “importing” of signifiers from one

48

Analysing style

context (for instance, one era, one social group, one culture) into another, in order to signify ideas and values associated with that other context in the context that does the importing. Thus an advertisement for a Citroen car may import the opening of the overture from Mozart’s The Marriage of Figaro from the domain of the concert hall into the domain of advertising to imbue the Citroen with the timeless high art values associated with classical music (the example comes from Cook (1998: 6), who analyses it in detail) and, as we saw in the previous chapter, literary styles can import archaic expressions, foreign words and dialects into the “standard language” (Mukarovsky, 1964: 338–339) to evoke ideas and values associated with the periods, countries and regions they come from. The concept of “provenance” was inspired by Roland Barthes’ groundbreaking analysis of the role of “connotation” and “myth” in popular culture. In “Rhetoric of the Image” (1977: 32–51), Barthes described how “mythical” signifieds may be “imported” from a specific country to signify a complex of ideas and values which another country (in Barthes’ case France) associates with that country. His key example was an advertisement for Panzani pasta, sauce and parmesan. The name Panzani and the colour scheme of the advertisement (based on the Italian flag) signify, said Barthes, not “Italy,” but “Italianicity,” a specifically “French” knowledge of things associated with Italy, because “an Italian would barely per­ ceive the connotation of the name, no more probably than he would the Italianicity of tomato and pepper” (1977: 34). Meanings such as “Italianicity” are, as Barthes put it, “a kind of nebula, the condensation, more or less hazy, of a certain knowledge” (Barthes, 1973b: 122), for instance “the condensed essence of everything that could be Italian from spaghetti to painting” (Barthes, 1977: 48). Elsewhere, in his Mythologies (1973b:121) he used sinité (“Chinese-ness”) as an example (“A certain melange of bells, rikshas and opium dens”). John Berger, in discussing advertising images, followed a similar line of thought when he described how “myths” use history and geography to lend meaning to advertised products so that “cigars can be sold in the name of a King, underwear in connection with the Sphinx, a new car by reference to the status of a country house.” Such meanings, he said, “are imprecise and ultimately meaningless (…) they should not be un­ derstandable, they should merely be reminiscent of cultural lessons half-learnt” (1972: 140). Contemporary popular culture remains a storehouse of such “myths.” Associating China with “bells, rikshas and opium dens” may be outdated, but in advertisements, comic strips, tourist brochures, and so on, such “exotic” and orientalist references are alive and well, mostly in images, and rarely formulated in so many words – they are “recognized” unreflectively rather than made explicit. Clearly, provenance often relies on cliché’s, and clichés are an important mechanism for communicating dominant cultural meanings (Zijderveld, 1979), although it is of course also possible to find new signifiers to “import.” Analysing provenance is therefore a matter of recognizing where the signifiers come from. We may be ignorant of the traditional symbolic language used by the Persian carpet in Figure 3.1, but we will immediately recognize it as a Persian carpet with all the associations clinging to that in the West (rather than in its region of origin).

Analysing style

49

Analysing provenance therefore requires an engagement with cultural history. People seeing a bonbon box or a hotel lobby using a purple and gold colour scheme will unreflectively recognize its royal provenance, but as semioticians we cannot be unreflective and need to excavate the cultural history of that colour palette, the way it has come to be associated with “royalty.” The modern carpet from Figure 3.2, however, cannot be recognized in this way, whether reflectively or otherwise. Here we need to analyse the experiential meaning potential of the carpet and the context in which we have encountered it. Provenance has been a key design resource ever since Europe began to import semiotic artefacts from other cultures (chinoiserie, arabesques, and so on). In today’s multicultural societies we again import (and adapt, or fuse with other provenances) the cuisines, musics and fashions of other cultures into global consumer culture in ways which often have little to do with what they mean or meant in the cultures from which they came. Fashion designers, for instance, constantly import ideas from other periods and other cultures (or subcultures) to connote ideas and values attached to these periods and cultures. When seeing the designs of Edwina Hörl I discussed in Chapter 1, people might not know where they actually came from, but they will in all likelihood recognize them as drawing on traditional Eastern European rural styles, with all the semiotic baggage that brings.

Contextualization The iconography of Panofsky (e.g. 1970) and other scholars of the so-called Warburg School was another inspiration for my approach to the analysis of identity design because of its emphasis on contextualization. Iconographers need contextualization because they deal with the meanings of art works from the past. This means that they cannot recognize what these art works represent and sym­ bolize from personal experience. Five hundred years ago everybody would have been able to recognize a male figure with a knife as St Bartholomew and a female figure with a peach in her hand as the personification of Veracity, just as today we can instantly recognize images of Donald Trump or Boris Johnson. Today that language has been lost, and iconographers must therefore “find out as much as they possibly can of the circumstances under which the objects of their studies were created,” “collect and verify all the available factual information,” and “read books on theology an mythology in order to identify the subject matter” (Panofsky, 1970: 41). All this is relevant also to analysing the provenances of contemporary images and other semiotic artefacts, as I have argued in detail elsewhere (Van Leeuwen, 2001). In researching provenances we should, like iconographers, pay attention to titles, captions and other verbal descriptions, compare images or other semiotic artefacts with other images or semiotic artefacts, and do various kinds of culturalhistorical background research. Nederveen Pieterse (1992: 42), for instance, stu­ died the cultural history of the physiognomic stereotyping of Africans and African Americans. Such stereotypes were still absent in paintings of black people by artists

50

Analysing style

such as Rubens and Rembrandt, but became common in the age of colonialism, spearheaded by the writings of 18th- and 19th-century scientists such as Georges Cuvier (1769–1832) who wrote that “the projection of the lower parts of the face, and the thick lips, evidently approximate it [‘the Negro race’] to the monkey tribes, the hordes of which it consists have always remained in the most complete state of utter barbarism” (quoted in Nederveen Pieterse, 1992: 42). Contextual research also includes ethnographic research, for instance interviews with designers and/or users. In her study of the Mary of Warmun (see Figure 1.2), Crane combined ethnographic research about the meanings Gija culture tradi­ tionally attaches to ochre and charcoal with her analysis of how these materials were used in the case of the Mary of Warmun. As for image research, by collecting and comparing many pictures of black servants, Nederveen Pieterse (1992: 131) found that century-old stereotypes continue in contemporary advertisements, with black servants still depicted as smaller and slightly stooped in posture, as smiling to express availability, as watching the eyes of the person being served without that look being reciprocated by the person served, and so on. Today’s image banks can be an important re­ source for comparative image studies. In a study of advertisements for contra­ ception (Van Leeuwen et al., 2016), we noted that the colour blue dominated in advertisements for LARCs (long acting contraceptive methods), while pinks, magentas, reds and maroons dominated in advertisements for oral contraception. The former targeted (and depicted) somewhat older women who already had children and focused on health benefits. The latter targeted (and often depicted) younger women and hinted at romance or sexual exploration free from worries about pregnancy. Blue can mean many things besides “health,” depending on the context. Nevertheless, a search of Getty Creative Images, an image bank for de­ signers and editors, brought up 43,453 images for the combined search terms “health” and “blue” as compared to 429 for “health” and “red,” 616 for “health” and “green,” and just 220 for “health” and “yellow.” Such searches can sig­ nificantly enhance the credibility of semiotic interpretations. Finally, it is important to distinguish between the analysis and the interpretation of identity design. Analysis concerns the signifier, interpretation the signified. Analysis deals with material evidence different analysts should be able to agree about. In analysing the logos of oil companies (Johannessen, 2017), for instance, it should be possible to agree that the BP logo uses the colours green, yellow and white, while the Shell logo uses the colours yellow and red. Although such an agreement is based on a culturally specific system of colour names (cf. Van Leeuwen, 2011, for a discussion of different ways of classifying colours), this is not likely to lead to significant differences of opinion and can therefore ground semiotic analysis firmly in empirical evidence. Interpretation, on the other hand, deals with meaning, and this creates more space for difference. But that does not mean that interpretation is entirely subjective. The key is providing plausible arguments, plausible grounds for the interpretation – experiential metaphor, grounded in common experience; provenance, grounded in

Analysing style

51

visually and verbally documented cultural history; contextuality, grounded in eth­ nographic and documentary evidence. The colour green in the BP logo, for in­ stance, can be interpreted as seeking to communicate the company’s “concern for the environment” on the basis of documentary evidence, for instance intertextual comparison with texts in which this meaning is linguistically anchored, on the basis of a comparative image search combining the search term “environment” with the search terms for “green” and other colours, and on the basis of documentary evi­ dence of BP’s widely reported culpability for a massive oil spill in the Gulf of Mexico in 2010, and its subsequent attempts to repair its environmental reputation. All this supports and makes plausible the meaning of green in this context. The colour red in the Shell logo can be understood on the basis of our shared experience of red, which, despite its wide range of possible meanings, always involves a sense of energy or energetic action, whether it is the red of passion, the red of danger, the red of warmth, or any other kind of red. When this colour becomes part of the identity of an energy company, interpreting it as signifying “energy” becomes at the very least plausible and can then be supplemented with documentary research and comparative image research. Finally, the wider significance of logos, what iconographers call their “ico­ nological symbolism,” can be grounded in broader theoretical considerations. In the case of the BP logo’s use of the colour green, we could for instance think of Habermas’ account of legitimation (1976) in which he argues that contemporary legitimation disengages “generalized motives” from the discourses they derive from. BP green, as interpreted here, does seek to legitimate BP as an en­ vironmentally aware company, but in a way that does not even begin to specify what they might mean by this, and therefore warrants critical attention. To sum up, social semiotic interpretation needs to build on three kinds of knowledge: a knowledge of language and other semiotic modes; a knowledge of cultural history, and a knowledge of sociological and philosophical theories that can help us understand the role of semiosis in social life. This modern “trivium” will allow social semiotics to renew and reinvigorate the ancient art of inter­ pretation in a way suited to our times. The remaining chapters of this book will apply the principles of identity design – experiential meaning potential, provenance and contextualization – to the para­ meters of shape, colour, texture, movement and timbre, and the way they combine in multimodal styles. But first two key relevant theoretical concepts will need to be discussed and related to the topic of this book, “modes” and “media.”

Modes and media In this book, I use the term “mode” to refer to resources for the functional design of performances, texts and artefacts and the term “medium” for resources for their identity design. Despite its central importance in multimodality studies, the term “mode” has been somewhat elusive. Often it is simply defined through examples (“modes, e.g.

52

Analysing style

language, image, music”) and even when defined more closely, the definitions may not include an indication of the way modes are organized, e.g. “Mode is a socially shaped and culturally give semiotic resource for making meaning” (Kress, 2011: 79) and “Mode is that which a community, a group of people who work in similar ways around similar issues, has decided to treat as a mode” (Kress, in Andersen et al., 2015: 77). The stress on social “shaping” in these definitions is important, but as “mode” has generally been taken to include language, we can ask whether definitions should also specify in which respects other modes are like language, or what aspects of semiotic organization all modes have in common. This is done, for instance, by Lemke (in Andersen et al., 2015: 126), who defines mode as “a system of meaningful contrasts between forms in a community that has conventions for the interpretation of those forms and contrasts, as paradigms, as syntagms, and this can be done through multiple expression planes.” An equally complex issue arises with the term “media.” It can be used in the way artists use it, not only as a term for the material resources they use in their art works, e.g. “oil,” “tempera on paper,” “bronze mounted on a marble base“, etc. but also to refer to technical media that serve to preserve, distribute and/or exchange information – print, radio, television, the internet, and so on. The way I use “mode” here was first developed in Kress and Van Leeuwen (2001) and relates closely to the issues of functional design and identity design introduced in Chapter 2. A mode is a resource for functional design, which can be realized in different materialities. Language, for instance, is a mode because it can be realized by means of sounds (as speech) or by means of graphic traces (as writing). The resources for visual communication we described in Reading Images (Kress and van Leeuwen, 2006) form a mode, because they can be realized in materially different ways – as drawings, photographs, paintings, and so on. The genres and templates discussed in Chapter 2 are also modes, because they too can be materialized in different ways. Media are the resources that materialize functional designs but themselves also produce meaning. They can be of two kinds – production media and distribution media. Production media are defined in the same way that artists define them, as the material resources that realize designs. Distribution media are technical media which may have been invented merely for the purpose of preserving and dis­ tributing already completed designs, in other words, for re-production, but usually soon acquire semiotic potentials of their own. The microphone and the loud­ speaker, for instance, not only allow speakers to be heard across greater distances, “distributing” the speech to more listeners but also open up new semiotic choices, such as close miking, which can signify intimacy even in contexts where formerly only formal, public modes of address would have been possible (Mulder, 2012). The term “resource,” finally is the overarching category, encompassing both modes, non-material resources for functional design, such as knowing how to construct a narrative or a pair of glasses, and media, material resources for identity design, such as the body, pen and paper, or the computer. In the age of artificial intelligence, the distinction between material and immaterial resources can

Analysing style

53

become blurred as semiotic knowledge may be built into material tools, ex­ ternalizing, for instance grammatical knowledge or schemas for producing texts which formerly would have been internalized. Sometimes the term mode is used to refer to practices such as typography, architecture, fashion, and so on, but I prefer to call these practices rather than modes – they use semiotic resources but do not constitute resources themselves, and specific modes and media may be used across many practices. Colour, for instance, is used in typography, graphic design, image making, dress, interior decoration, and so on, as do the other identity design parameters I will discuss in this book. It follows that modes and media are resources for different stages of semiotic practices, modes for the stage of conception, media for the stage of material realization. Goffman (1981: 44), in his description of the “production format” of talk, describes four such stages as a division of semiotic labour. The principal is the person (or institution) “whose beliefs are told,” “whose position is established”; the author is the person who “selects the sentiments that are being expressed and the words in which they are encoded,” and the animator is “the sounding box in use.” When discussing the “animator” Goffman adds that animators sometimes “share this physical function with a loudspeaker system or telephone.” By ex­ tension this could lead to another role, that of the technician, the recordist. This analysis can be applied to other practices as well. In my terms, Goffman’s author is the designer. There is a “content” to be communicated or an object to be created to fulfil a practical function, in a way that will be suitable for specific users, and the designer works out how this will be done. The design stage is therefore essentially functional. But it does not yet result in the text users will eventually see or hear, or the building that will eventually be built, although it may have an intermediate realization as a sketch, a blueprint, a script, a score or a prototype. Goffman’s “sounding box” is, in my terms, the producer, the person or persons creating the material text or object, drawing on technical skills, skills of the hand and the eye – skills of of bodily articulation (voice, facial expression gesture, posture) or skills or working with tools and materials such as oil or water colour, cellos or trumpets. Sometimes design and production, mode and medium, merge. Improvising musicians, for instance, are both the designers and the producers of their music, and so are we all when we speak spontaneously, without scripts or notes. At other times, there will be a division of labour. Composers design the music and performers execute it. Playwrights write the play and actors perform it. In such cases the work of the producers will often be seen as adding little meaning, as merely realizing, fluently and skilfully, the intentions of composers or play­ wrights or other authors whose designs may seek to exercise as much control as possible over the way they will be performed (sometimes mediated by coaches, conductors, directors, and so on, who are also “designers,” because they, too, do not, or not usually, themselves perform their designs). Design then becomes prescription, pro-gramme. Yet, in many domains production, making meaning di­ rectly with materiality, has become increasingly important, and the meanings

54

Analysing style

produced in this way are identity meanings that work with the affordances and cultural histories of the materials used. Finally, there is “distribution,” only briefly hinted at by Goffman – the tech­ nician, who records the music or produces the prototype (for instance of a pair of spectacles). Like the distinction between design and production, the distinction between production and distribution is not always clearcut. Media designed for distribution may in time become production media, and the contribution of the sound engineer may become equal to that of the musician, with, for instance, reverb used not to (re)create what you would have heard from the best seat in the concert hall, but to make sounds either “interior” and subjective or “exterior” and objective, as in many contemporary dance music mixes, where the drum and bass are so “close up” that they seem to be played, not in actual space, but inside the listener’s head or body, a space where all sound is absorbed instantly. For this reason, Brian Eno has called the sound studio “a compositional tool” (1983: 56). These three stages or aspects of semiotic practices – design, production and distribution – apply not only to the creation but also to the interpretation and use of texts and semiotic artefacts. Interpreters and users need to draw on semiotic resources at all of these levels, and they may do so in ways that differ from what designers and producers might have intended. Music designed to dance on, for instance, may be listened to in a concert hall or played softly in the background during breakfast. Such interventions have become even more important as “dis­ tribution” media now allow users to modify what is being distributed – to amplify it, filter it, edit it, and so on. Increasingly, interpretation also becomes creation.

4 SHAPE

The semiotics of shape Several earlier examples illustrated the role of shape in identity design – the circles on the carpet in Figure 3.1 and the drawing of the 3-year-old in Figure 3.2, for instance, and the meaning potential of irregularity in handwriting and drawing. In Reading Images (Kress and Van Leeuwen, 2006) we had already started exploring the semiotics of shape in a discussion of the shapes of the boxes, ovals and circles in communication models, a kind of diagram which, for a time, played a key role in communication theory, with many scientists and social scientists debating the pros and cons of different models – McQuail and Windahl (1993) devoted a whole book to it. We noticed that the descriptions of these models, like the glosses of the SmartArt diagram templates I discussed in Chapter 2, focused on their functional design, on the way they linked elements such as “sender,” “message,” “receiver” etc., rather than on their choice of specific shapes for representing these elements, and although our book was about functional design, inspired by Halliday’s functional theory of language, we nevertheless realized that shape also makes meaning, – a different kind of meaning which, at the time, we referred to as “ideological” and “mythical” (Kress and Van Leeuwen, 2006: 53–54): There can be no doubt that such choices [e.g. between rectangular and circular shapes] are charged with meaning … As we are here primarily concerned with the relation between participants, this subject falls somewhat outside our main concern; it deserves a separate study. But given the semiotic and ideological (mythical) significance of these aspects, we will at least indicate the issues with which such a study might be concerned

56

Shape

So I will start this chapter by revisiting some of these communication models. The functional structure of the model in Figure 4.1 expresses the relation between “overall social systems,” “larger social structures” and “primary groups.” Here the relation is one of inclusion: the “overall social system” includes “larger social structures.” These larger social structures, in turn, include “primary groups.” The diagram also shows two individuals – “C” (“Communicator”) and “R” (“Receiver”). They are involved in a process of exchanging information, as in­ dicated by the arrows and the word “messages,” and portrayed as half inside and half outside of their different “larger social structures” and as connected to, but not inside of, “primary groups.” Figure 4.2 shows the famous Shannon and Weaver communication model. Its functional design focuses, not on social structures, but on the efficiency of pro­ cesses of telecommunication, and it does not contain inclusive structures, but is made up out of separate boxes that act on each other in linear fashion, as re­ presented by the arrows. Two different functionalities – the one showing the relation between in­ dividual acts of communication and the social structure in which they happen, the other mapping the process of long distance information transmission. But why in

PRIMARY GROUP

PRIMARY GROUP

PRIMARY GROUP

MESSAGES

C

LARGER SOCIAL STRUCTURE

MESSAGES

R

MESSAGES

PRIMARY GROUP LARGER SOCIAL STRUCTURE

OVER-ALL SOCIAL SYSTEM

FIGURE 4.1

information source

Communication model ( Riley and Riley, 1959)

transmitter

signal

received signal

receiver

noise source

FIGURE 4.2

Communication model ( Shannon and Weaver, 1949)

destination

Shape

57

Figure 4.1, is the “primary group” a rectangle and the “larger social group” a circle? Why do Shannon and Weaver prefer angularity and Riley and Riley curvature? And why do such choices remain implicit, even in the work of sci­ entists and social scientists? Basic geometric shapes have long been a source of meaning, and the difference between angularity (the straight line, the rectangle, the triangle) and curvature (the curved line, the circle, the ellipse) has played a key part in this. For the Ancient Greeks terrestrial things moved in straight lines, heavenly bodies in circles. When, much later, the 17th-century astronomer Johannes Kepler found that the planets move in ellipses, Galileo refused to accept this finding. For him, “the circle was still the only natural movement, whereas rectilinear movement came about through the interference of some foreign agent” (Arnheim, 1969: 277). To give another example, in the Middle Ages, the circle represented the crucial theological concept of the Trinity, because of the way it unites three parts – the centre, the circumference and the space in between. The centre represented God the Father, the space in between the Son, spreading God’s power in all directions, and the circumference the Holy Ghost, holding the two together. In this way, the concept of the Trinity was naturalized, incorporated in the geometry of the time. According to Kepler, “The image of the triune God is in the spherical surface. The Father is in the centre, the Son is in the outer space, and the Holy Ghost is the equality of relation between point and circumference” (quoted in Arnheim, 1969: 281). Today squares and rectangles are still associated with the “terrestrial” – with the mechanical, technological order, the world of human construction. Unlike circles, which are self-contained, complete in themselves, rectangular shapes can be stacked and aligned in geometrical patterns, to form the modules, the building blocks with which we construct our world. They therefore represent the values of builders and engineers, and of those who think like builders and engineers, like Shannon and Weaver in Figure 4.2 – or like the 1920s art movements which embraced technology, such as Constructivism and De Stijl. As Mondrian wrote at the time (quoted in Jaffé, 1967: 64) More and more the machine displaces natural power. In fashion we see a characteristic tensing of form and intensification of colour, signifying the departure from the natural. In modern dance steps (boston, tango, etc) the same tensing is seen: the curved line of the old dance (waltz, etc) has yielded to the straight line. The glosses in “dictionaries of visual symbols” point in the same direction. According to Thompson and Davenport (1982: 110), the square “represents the world and denotes order.” According to Dondis (1973: 44) it represents “honesty, straightness and workmanlike meaning.” This combines associations with human work and an experiential metaphor in which literal “straightness” becomes fig­ urative “straightness” – principledness, integrity, honesty, and so on.

58

Shape

Circles are glossed very differently in such dictionaries, as denoting “end­ lessness, warmth, protection” (Dondis, 1973: 44) or as “the traditional symbol of eternity and the heavens” (Thompson and Davenport, 1982: 110). This again draws on what circles literally are – without beginning or end (“eternal,” “end­ less”), and enclosing things (“warmth,” “protection”), as well as on provenance (“traditional symbols”). Circles evoke the natural world and are often endowed with spiritual meaning. In Figure 4.1, society is represented as a natural order, organically evolved rather than humanly constructed. But the “primary groups” are depicted as rectangles. Perhaps this betrays an unconscious bias in favour of modern urban identity, a view in which traditional, close-knit communities in which everyone knows everything about everyone are seen as oppressive. “C” and “R” are then represented as hailing from primary groups, still connected to them by a thin line but having left them behind to become modern, autonomous in­ dividuals, partially inside the “larger social structure” (the big city? the nation?) and partially outside of it. Diagrams, rational and scientific as they may seem, can convey identity meanings. And these identity meanings tend to be expressed vi­ sually rather than verbally. Like the rectangle, the triangle is angular, invoking the mechanical, techno­ logical order. But, unlike the square, the triangle is dynamic. It can point at something, convey directionality, and hence a sense of action. This can play a role in functional design, with triangles combining the function of the box and the arrow. But it can also carry more abstract meanings and values such as “generative power” (Thompson and Davenport, 1982: 110), “action, conflict, tension” (Dondis, 1993: 44). In El Lissitzky’s famous poster “Beat the Whites with the Red Wedge,” dating from the early days of the Russian revolution, the red wedge is an elongated triangle representing the revolution as wedging itself into the selfcontained inert circle that represents the traditional order of White Russia. But triangles can also, and at the same time, carry abstract meanings and values. Pyramid triangles can also express dynamism. For Kandinsky they represented “striving,” in particular the striving of the artist “towards the abstract, the nonmaterial” (Kandinsky, 1977 [1914]: 19), which for him had spiritual value. But pyramid triangles can also represent hierarchy, for instance in one of the SmartArt diagram templates provided by Microsoft SmartArt, recommended as useful for showing “proportional, interconnected or hierarchical relationships.” From these basic shapes other geometrical shapes can be derived: square, circle and triangle can all be horizontally or vertically stretched, tilted towards the right or the left, and so on, creating many other shapes – ovals, rhomboids, polygons, etc. Microsoft Word provides only nine functional types of shape, but many different variants of each – 9 different rectangles, 44 “basic shapes,” 28 shapes for the ele­ ments of flowcharts, 27 “block arrows,” and so on. From the point of view of semiotics, however, such lists are not very helpful. It better to focus on the dis­ tinctive features that create the differences and similarities between the meaning potentials of different shapes. This will be discussed in detail below, in Section 3.

Shape

59

Function and identity in typography and graphic design – a short history Like the teaching of handwriting I briefly discussed in Chapter 1, typography has long focused on functional design, on making letter forms distinctive and legible. Its practitioners saw their work as a self-effacing craft that should not attract attention to itself, not get in the way of the words. Yet they also appreciated typography aesthetically, based on their detailed knowledge and appreciation of pre-industrial designers, as can be seen in this quote, where the 20th-century American designer W.A. Dwiggins describes how he designed the Caledonia typeface (quoted in McLean, 2000: 60): We turned to one of the types that Bulmer used, cut for him by William Martin around 1790…an attempt was made to add weight to the characters and still keep some of the Martin swing…that quality in the curves, the way they get away from the straight stem with a calligraphic flick, and in the nervous angle in the underside of the arches as they descend to the right. But subtleties such as these “calligraphic flicks” and “nervous angles” were not understood as meaningful. Although it was acknowledged that “to a very limited extent, lettering may help to express a feeling or a mood that is in harmony with the meaning of the words” (McLean, 2000: 56), for the most part “lettering and calligraphy are abstract arts … what moves us is something formal, and, in the last resort, inexplicable” (ibid: 54). Such classifications of typefaces as existed were formal and historical, not semiotic: “Humanist typefaces are closely connected to calligraphy and the movement of the hand, transitional and modern typefaces are more abstract and less organic. These three main groups correspond roughly to the Renaissance, Baroque and Enlightenment” (Lupton, 2004: 42) In the heyday of functionalist design, typographers such as Jan Tschichold (1998 [1928]) sought to create entirely rational and functional typefaces, doing away, not only with “calligraphic flicks” but also with serifs, “entasis” (difference in thickness and thinness within letters), and so on, and using a minimal set of interchangeable components, making, for instance, the “bowls” of “a,” “b,” “p,” “d,” “g” and “q” identical, which they usually were not. This was rejected by many typographers. McLean (ibid: 67) compared it unfavourably with traditional typographic practices, both on functional (legibility) and aesthetic grounds: This seductive theory had to be paid for in loss of legibility, since the effect was to reduce the differences … Eric Gill’s sans was different in that it was drawn by an artist and designer who was deeply involved with the classical roman alphabet … His letters contained subtleties and refinements which the German designers, preferring the logic (or dictatorship) of rulers and compasses could not admit.

60

Shape

More recently, however, many typographers have come to recognize that letter forms are not just functional and can also express identity meanings. The reason for this development is that today’s corporate culture makes new demands on typeface design. It is no longer only about legibility and elegance, it is also about expressing the identities of companies and other organizations, for instance in logos, those “20th century battle standards in the fight for profit” (Neuenschwander, 1993: 80), and about expressing the identities of everyday computer users who can now choose from some 500 typefaces – psychologists have already begun to elaborate their potential for identity design: “Courier New – a little old-fashioned”; “Georgia – you have flair,” “Times New Roman – confident,” and so on (cf. Kapica, 2018). To give an example, designer Nancy Nowacek (2005: 160) describes her DIN typeface both in terms of its intended functionality (“a typeface that is designed to help people, particularly the elderly”) and in terms of identity meanings. The ascender and the crossbar of her “f,” for instance, are equal in length, so that they “seem to be seeking, leaning forward, enthusiastic, earnest, curious, engaged”; her “h” has a tall x-height and a short descender, making it “a hard worker,” “solid and muscular, but not athletic,” and her “o” “prizes decorum, but has moments of yearning to be bigger, bolder, rounder.” This is very much in line with the way I will describe the meaning potentials of the distinctive features of shape later in this chapter, with the exception that (in contrast to what I wrote in Van Leeuwen, 2006), I now believe they apply not only to typography but to all semiotic practices that draw on the resources of shape, whether still or moving, twodimensional or three-dimensional. These developments happened, not only in typography, but also in the visual arts of the early 20th century. Just as typographers such as Tschichold had tried to create letter forms with a minimal set of stems and bowls, so European painters tried to represent the world with a minimal set of geometric shapes. Mondrian, for instance, progressively abstracted trees, and then complained that it was difficult to represent trees as arrangements of rectangular shapes: “In painting a tree, I pro­ gressively abstracted the curve; you can understand that very little ‘tree’ remained” (quoted in Jaffé, 1967: 120). Soon these painters went a step further and aban­ doned their attempts to reconcile the visible appearance of things with models of their inner structure. Their work now became fully abstract. From there it was only one step to moving from art to design, from representing the world to constructing it, from making objects for people to look at to designing environ­ ments for people to live in. In the Netherlands, Rietveld used Van Doesburg and Mondrian’s rectangular building blocks to make his famous Rietveld chair and his equally famous Rietveld house, the Schröder Residence in Utrecht. In the Bauhaus, students were encouraged to apply Kandinsky’s “grammar of forms” to the design of fabrics and furniture (Fielder, 2006). Such practices centred on functionality. As we saw in Chapter 2, in the early years of the 20th century, Adolf Loos had declared that “the evolution of culture is synonymous with the removal of ornament from utilitarian objects,” even in

Shape

61

fashion: “so immensely strong is the individuality of modern man that it can no longer be expressed in items of clothing” (quoted in Brett, 2005: 195) and Karel Teige (1929: 40) had said that “aesthetic intervention in utilitarian design in­ evitably leads to an imperfect object.” Paradoxically, functionality could itself become “aesthetic” in this context. For Le Corbusier the ocean liner represented the essence of functional design and so he used features of the ocean liner to make buildings look rather than function like ocean liners, for instance by making the edge of a building pointed like the bow of a ship, or by using the grey steel of battleships and the red, yellow and blue of semaphore flags to decorate buildings. And Gropius, the first director of the Bauhaus pleaded for a “poetic” elaboration of technological forms, focusing on their non-functional aspects (quoted in Fielder, 2006: 17): All inessential details are subordinated to a great, simple representational form which finally, when its definitive shape has been found, must constitute the symbolic expression of the inner meaning of the modern artefact But contrary to what Loos had predicted, identity is, today, once again expressed, not only in typography, but also in styles of dress, product design, interior dec­ oration, building, and so on. In contemporary identity design, aesthetics and social semiotics merge (Brett, 2005: 105): Decoration, even of the most humble and everyday kind, acts as an integrator of individual pleasure with social life and its ideological commitments

The experiential meaning potential of shape In this section, I focus on the distinctive features of shape. Their meaning potential is common to typography and other design practices, and applies to abstract as well as figurative visuals. As discussed in Chapter 3, distinctive features are simultaneous and graded, rather than binary and hierarchically ordered. The meanings they make are therefore also simultaneous, merging with contextual factors in a complex whole. This is not to say that it is impossible to reconfigure them as a binary system. Semiotic resources are structured the way they are because people have structured them that way, and when semiotic choices are digitized, they inevitably become binary. Non-digital brush strokes, for instance, are endlessly variable, created by the interplay of the painter’s gestures, the qualities of the brush, the kind of paint used and the surface the paint is applied to. But Brushes Redux, an app for digital painting offers just 12 discrete brush strokes, although it does allow parametric variation through a set of sliders that can modify the density, intensity and spacing of the brush strokes. However, even though clicking and dragging and moving a

62

Shape

stylus over glass is a very different physical experience from working with brushes, oils and canvas or board, I believe that the resulting visuals will still be understood on the basis of our material experiences with brushes, paints and surfaces, ex­ periences which, for most of us, come early in life, for instance in preschool, where, if all is well, we are still allowed to mess with paints and papers, clay and cardboard, sand and water. I should finally mention that the distinctive features of shape apply not only to boxes, circles, triangles, and so on, but also to lines. When not used to delineate shapes, lines, too, can be curved or angular, regular or irregular, bold and large or small and light, and so on, regardless of whether they are relatively abstract ele­ ments in diagrams, graphic designs and art works, or figurative, as the wrinkles of a face, the folds of a curtain, or the rays of the sun - this is a fundamental aspect of style in drawing.

Curvature and angularity Letter forms and abstract or figurative shapes can, to different degrees, stress angularity or curvature. Our experience of trace-making tells us that producing straight, angular forms requires brisk, decisive movements, producing round forms more gradual, fluid movements. Our experience of our natural and cultural environment tells us that curved forms and curved movements dominate the natural world, angular forms and rectilinear movements the world created by humans, and we also know that mod­ ernity has favoured the values of angularity, while postmodernity has brought back curved forms, for instance in car design and architecture. The experiential meaning potential of curvature and angularity is broad, but will be narrowed down by other, co-present features of shape, by colour and texture and by the context. The Apple ‘org chart’ shown in Figure 4.3 is round. Instead of placing the CEO at the top of a hierarchical tree, it places him at the heart of a “centre-margin” structure, in which the centre “is presented as the nucleus of the information to which all the other elements are in some sense subservient” (Kress and Van Leeuwen, 2006: 196). As we saw, in medieval images of the Trinity, God the Father was the central source of power, but here it is Steve Jobs, the CEO, and he exudes a dazzling white aura which shades into an intense yellow and becomes more desaturated as it gets close to the outer ring. Only the members of his executive team are represented by rectangles, surrounding the charismatic CEO like a wall of solid bricks. Lines, too, may be angular or curved, spiralling, and so on. Microsoft provides a ready-made vocabulary of 12 kinds of line which incorporates all three of these options, and more. In typography, curvature can be realized, not only by general roundness, as for instance in Century Gothic, but also by rounded ascenders and descenders as for instance in Script MT Bold or curved loops at the end of ascenders and/or descenders as in Pristina. Angularity, similarly, can be realized, not only by general angularity, as in Copperplate, but also by predominantly straight ascenders

Shape

FIGURE 4.3

63

Apple organization chart

and descenders, as in Agency FB. “Black letters,” like Old English Text MC also have pronounced angularity. In figurative visuals, shapes can characterize depicted objects as well as com­ positional structures. Kandinsky not only taught his Bauhaus students to represent the world with circles, rectangles and triangles, but also to detect such shapes in figurative art works, for instance in the composition of Cezanne’s “Bathing Women” (Kandinsky, 1977 [1914]: 31).

Regularity Graphic shapes and letter forms may have deliberate irregularities – ragged edges, distressed textures, irregularly sized and placed letters, etc. We know from ex­ perience that it is near impossible to draw perfect circles, rectangles and triangles

64

Shape

by hand, so we can usually tell whether letters, lines and shapes are hand-drawn or produced with a compass, a ruler or a computer. The first, and fundamental, meaning potential of irregularity, therefore, is the difference between the handmade and the mechanically produced, with all the associations and values this brings. The mechanical may not only convey “perfection,” “technological achievement,” for instance, but also inauthenticity, disembodiment and a lack of aura. The hand-drawn may convey its opposite – the “human touch,” authenti­ city, character. Increasingly, many computer-generated letters use irregular fonts, though regularity usually lurks beneath the surface. The breakfast menu of a hotel I stayed in suggested that each breakfast can be different (“Your way as much as you like”) and it did so with irregularly aligned and differently sized lettering. Yet all the ‘E’s and all the ‘R’s were identical, to the slightest irregular detail, and the same applied to the other letters. The type family “Kiln,” designed in 2016 by Ryan Martinson for Yellow Design Studio, is described as a “timeworn handcrafted type family” (MyFonts, 2016), and looks as if the print is worn off. But even though each character comes with two “distress options” (MyFonts, 2016) so that no two adjacent letters have the same texture ( Johannessen and Van Leeuwen, 2018), close inspection eventually detects the regularity that is inevitable in computergenerated shapes. As discussed in Chapter 3, experience also tells us that certain conditions can produce irregularity – lacking the skills to produce neat writing or drawing, re­ fusing to produce neat writing or drawing, using tools and/or materials that make it difficult to produce neat writing or drawing, intoxication, infirmity. All of these are more or less involuntary symptoms of certain conditions, but they can also be used deliberately, made into signs. Transgressing the discipline of neat, regular writing, for instance, can, more abstractly, come to signify the idea of “trans­ gression,” as in a government-issued brochure warning young drivers against driving under influence of alcohol or drugs in which the word “BUSTED” has distressed edges. Irregularity can also evoke ideas about childhood (playfulness, innocence, unruliness, and so on) or about forms of deprivation that make ap­ propriate tools and materials unavailable (carvings on the walls of prison cells, for instance). Because it contrasts with the regularity of mechanically or digitally produced traces, irregularity can also mean “personal,” “individual,” whether in art works or everyday handwriting. It is exactly because contemporary marketing, assisted by the extraction of private data from social media, must turn massproduced goods into seemingly personalized, individual choices, and because life styles must combine individuality with group affinity, that computer-generated design must hide its regularity beneath a surface of irregularity.

Repetition Repetition is a feature, not only of individual lines and shapes but also of the way they combine in patterns. Patterns are usually regular and repetitive, though the

Shape

65

unpredictable and irregular patterns of nature can also become a source of in­ spiration for pattern design (Cole, 2007). Experience tells us that repetition is a fundamental feature of human action. It makes music danceable. It is fundamental in acquiring skills. It creates the habits that give a sense of order to everyday life. But there is another important aspect to repetition. When patterns are repeated, it is no longer necessary to see the whole. You can be secure in the knowledge that the pattern will continue in the same vein. You know that you are in a stable world. Sigmund Freud understood this and expressed it memorably in his account of the fort … da game (“gone … there”) in which the child enacts the absence and reappearance of the mother by means of the repetitive manipulation of a wooden reel, with a great yield of pleasure and excitement (Freud, 1984 [1920]: 283–287). But repetition also opens up the possibility of non-repetition, not only in the sense of more random patterns but also of sudden interruptions of regularity, or of gradual transformations of what might at first sight seem repetitive patterns. People – and societies – need continuity and predictability as well as change, and change may be gradual, perhaps almost unnoticeable, or sudden and disruptive. This is at the heart of the meaning potential of repetition. Many semiotic phenomena do not invite the focused and concentrated attention that art works do in a gallery, or images in a book or magazine. They are part of our environment, noted peripherally – background music, the patterns of carpets, or of the sculpted pilasters and architraves of doors or the wrought iron balustrades of balconies. Such patterns almost always involve a good deal of repetition, and in the age of lifestyle identity they have become increasingly important. Musicians like Brian Eno deliberately compose ambient music, and artists like Olafur Eliasson and many others create environments for people to be immersed in, as opposed to works for people to look at in a more detached, observational manner. The ambient meanings of such art works are often referred to as “elemental, like atmosphere” (Kim-Cohen, 2016: 54). According to McCormack (2008: 413) Atmospheres are a quality of environmental immersion that registers in and through sensing bodies whilst also remaining diffuse, in the air, ‘ethereal’. Ambient meaning has also become important in marketing and branding – the ambient music in restaurants and shopping centres (cf. e.g. Graakjaer, 2012), or the branding of entire environments by companies like Starbucks (Aiello and Dickinson, 2013), which convey identity, not only through the names of the store design styles (“heritage,” “artisan,” etc.) but also by using recycled local materials to signify authenticity and environmental awareness – “a coffee bar covered in scrap leather obtained from shoe and automobile factories,” a “long community table salvaged from a local restaurant,” “old wood theatre seats,” “coffee pots ‘reconverted’ into containers for sugar, napkins, water and flowers” (ibid: 312). All these things are not meant to be “read,” but to be “sensed,” to seep into our unconscious by being repeated over and over, with local variations, across Starbuck’s 20,000 stores.

66

Shape

Repetition also can have an aesthetic effect. Poetry uses regularized rhythmic patterns (metre), rhyme and assonance (repetition of sound) to aestheticize lan­ guage. In music, repetition is fundamental, though always balanced by variation. In visuals, shapes and colours can “rhyme,” as seen, for instance, in Figure 4.5 below (cf. Van Leeuwen, 2005: 12).

Weight and size In typography, weight differentiates between the bold and the light versions of a typeface, and size between large or small versions. As with all distinctive features these are graded contrasts. There is a continuum of size, even though it is regulated by being measured in “points,” and there is a continuum of boldness, even if technologies like the word processor can make it binary choice, for example by providing only two degrees of thickness for underlining. Increased weight and/or size plays a key role in functional design as it can increase salience, and so draw readers’ or viewers’ attention more strongly to some parts of a message (e.g. headlines) than others, as we saw in Figure 1.1, where the key message of each of the dot points used a larger and bolder font than the elaborations of these messages printed below them. But they can also be used for identity design. “Large” can mean “powerful” and “impressive,” as opposed to “second rate” and “insignificant.” But it can also mean being “overbearing” or “domineering,” as opposed to “modest” or “delicate.” “Bold” can mean “daring” and “assertive,” or “solid” and “substantial,” as opposed to “timid” and “in­ substantial.” But it can also mean “exaggerated,” “hyped,” “overblown,” etc. Adjectives can never pinpoint the whole reach of the meaning potential of these features, and analysis will always have to take the context into account. When television commercials for large hardware stores advertise the latest specials, boldness and size signify an aggressive “hard sale” approach. In the Nokia logo, of which a likeness is shown in Figure 4.4, the brand name is large and assertive, and also more angular, indicating something like “technical precision.” The slogan “Connecting People” contrasts with this by being less bold and less angular, so providing a more subtle “human touch.” This combination of “technical per­ fection” and the “human touch” characterizes the identity design of many tech companies, and is always multimodally expressed. Van Leeuwen (2017) has de­ scribed how the functional design of the Sonic logo of AT&T uses a “heraldic” melody, an unresolved four-note ascending melody which forms the kind of “rallying cry” that has long been used in many different contexts, from national anthems to news signature tunes and advertisements. Its identity design then combines an old piano, a glockenspiel and a Wurlitzer with and electronic edge. Thus sounds evoking technological perfection and innovation blend with sounds that seek to make AT&T warmer, more human. Lines, too, may be bold and/or large or light and/or small. Functionally this may indicate the salience of their connective function. Very light lines in a flowchart diagram may be so insignificant that the diagram at first sight looks like a

Shape

FIGURE 4.4

67

A likeness of the Nokia logo

structure rather than a flowchart. This was the case in a diagram showing the “funding model” of an Australian University. Hairline arrows connected bold, coloured boxes, so that the diagram seemed to represent the funding level of the various Faculties and Divisions as a fait accompli, rather than as a process of allo­ cating funding to these Faculties and Divisions. The colours of this diagram also realized significant identity meanings. The elements of abstract paintings and decorative patterns, too, can be small or large and light or bold – and so of course can objects such as spectacles. Making meaning with line and shape is part and parcel of a wide range of semiotic practices, from handwriting to architecture, and from the design of wrapping paper to the creation of art works. In each case the same kinds of experiences underlie the meaning potential of the distinctive features of shape, even though we might reach for quite different adjectives to describe it in words.

Connectivity and sloping In running script, looped feet connect the letters within words. In “print script,” letters are disconnected, separate and self-contained. Connection and disconnection can be external, between letters, or internal, within letters. The curves of the bowls of ‘a’s or ‘p’s may then not reach the stem towards which they curve, as in the Bauhaus 93 font. Abstract visual motifs, too, may be connected to each other, or disconnected, separated by frame lines or space. Sloping refers to the scale from maximally left-leaning to maximally right-leaning lines and shapes. Traditional forms of handwriting such as copperplate combined sloping and connectivity. 18th-century copperplate, with its sinuous curves and curlicues was deliberately decorative, and expressive of the values of the period, in harmony with “the convoluted courtesies of the age, the gentleman’s extravagant bow, the intricate yet controlled movement of the gavotte, the music of Handel and the rococo tastes of the time” (Sassoon, 1999: 9). 19th-century handwriting continued to be connected and sloping, for instance in what in Britain was called the “civil service” hand, but functionality, the ability of handwriting to achieve speed and legibility, became increasingly important.

68

Shape

Inspired by mechanical writing, early 20th-century educationalists began to replace connected, sloping hands with “semi-upright hands,” “vertical hands” and “print script” hands, again for functional reasons: “An angle of 15 degrees is the natural slope for bold, clear and rapid writing” (Philip’s Semi Upright System of Writing, quoted in Sassoon, 1999: 44), or, in the case of print script: “Owing to the simplicity of the forms of print script characters, consisting entirely of straight lines and circles or portions of circles, it was expected that they would be more readily learned than the ordinary cursive forms” (1923 UK Board of Education report, quoted in Sassoon, 1999: 62). But as typewriting became more widespread, handwriting became increasingly associated with personal writing, and already in the 1930s educationalists began to argue for the individuality and authenticity of running writing, promoting “a swift and running hand which a child need never have to unlearn, but which will grow as he grows into something that is increasingly his own” (Marion Richardson, quoted in Sassoon, 1999: 78). Today, typographic connectivity and sloping can still signify the organic rather than the mechanical, the personal rather than the formal and impersonal, the aesthetic rather than the purely functional, and so on. The potential of connectivity for expressing identity meanings follows from what connection and disconnection are and do. Disconnection separates and fragments, connection integrates and fuses. But, depending on the context, dis­ connection may also be valued for articulating the individuality of shapes, and connection devalued for slurring them together. Internal disconnection, in which, for instance, the outline of a dove in a logo does not come full circle, may be seen as “unfinished,” and negatively valued as “sloppy,” or positively as “informal,” “easy going,” “not buttoned up.” The meaning potential of sloping relates to gravity. When lines or shapes are not aligned with the vertical and/or horizontal axis, leaning forward or backward, we sense a downwards pull. For this reason oblique lines and shapes suggest dy­ namism and action, rather than inertia and stasis. The wrapping paper pattern in Figure 4.5 shows three shapes against a green background: two longitudinal rectangular shapes, one white, one black, and one plumper yellow shape. In alternate rows, the black and the white elements are either unconnected and left-leaning or connected by the yellow shapes and right-leaning. Abstract designs such as this can be interpreted in many different ways. But whatever the interpretation, these observable facts must be taken into account. We could think of racism, with black and white segregated if we go back in time, but united by a “sunny” shape if we move forward. However, the designer, Sarah Angold, told a different story “This was inspired by the evening sunlight reflecting off the sea, combined with the shadows cast by Brighton’s turquoise beach railings” (quoted in Cole, 207: 139). If the context is the same for designer and users, their interpretations may converge but if the context of design and the context of use is different, interpretations may diverge, even if based on well-grounded interpretations of observable features. It is the power of abstract designs and abstract art works to enable creative interpretation.

Shape

69

Pattern with sloping, disconnected rectangles (Sarah Angold). Reproduced with kind permission from Sarah Angold

FIGURE 4.5

Most of the time we do not consciously interpret patterns of this kind. But we still choose them from among many others as, for instance, an apt fabric pattern for a shirt or dress, or as suitable for wrapping a present for a particular person, to convey, in that way, a particular message to that person. As semioticians we need to try and understand this kind of meaning-making, and remember what William Morris (1895: 177) said in a lecture on pattern design, well over a century ago: “You may be sure that any decoration is futile when it does not remind you of something of which it is a visible symbol.”

Expansion and density Shapes may be densely packed together, or widely spaced, “expanded,” on a scale which runs, one could say, from the extreme of claustrophobia to the extreme of agoraphobia, or, in Maree Stenglin’s terms (2009: 44ff ), from maximally “bounded” to maximally “unbounded” spacing. The meaning potential of this feature derives from our experience of space. Maximally condensed patterns or fonts make maximal use of limited space. They are precise, economical, packed with content. Or we might say that the shapes in such patterns huddle closely and intimately together. Widely spaced elements or letter forms spread themselves around, using space as if it is in unlimited supply, claiming large amounts of

70

Shape

territory for themselves, perhaps even wasting space. In different contexts, ex­ pansion and density may be valued differently. Wide spacing could also be in­ terpreted positively, as room to breathe, room to move, and condensed spacing might be seen as cramped, overcrowded, and restrictive of movement. It all de­ pends on the context. In an exercise with my students in Odense we compared the covers of two Danish food magazines. One portrayed a celebrity chef holding a sprig of thyme in his hand, against a spacious and glossy black background. The other showed a tightly packed collage of several colourful plates of food. Identity values were at stake here. Wide spacing suggests the taste of the elite, which, as Bourdieu (1979) has explained, prefers aesthetic presentation of food and refined rules of table arrangement and eating. Density suggests the popular taste, which prefers sub­ stance and nourishment, and cares little for aesthetic presentation.

Orientation Shapes may be flattened, horizontally stretched, or elongated, vertically stretched. The meaning potential of horizontal and vertical orientation is again based on our experience of gravity. Horizontal orientation is like lying down, giving in to gravity. It can therefore suggest, for instance, “heaviness” or “inertia.” Vertical orientation is like standing up, defying or overcoming gravity. It can therefore suggest, for instance, “lightness” or “upwards aspiration” but also “instability,” the risk of falling. In typography, orientation also relates to the differences between, on the one hand, typefaces with short ascenders and descenders and typefaces with long descenders and ascenders, as for instance in the appropriately named High Tower Text, and, on the other hand, the differences between a downwards orientation in which the descenders are longer than the ascenders and an upwards orientation in which the ascenders are longer than the descenders, as in the Poor Richard font. The meaning potential of this derives from our fundamental experience of the dimensions of “up” and “down” – shapes with downward projections seek roots, as it were, those with upward projections some kind of metaphorical elevation. All this applies to abstract as well as figurative shapes, and to a wide range of design practices. We already came across the difference between vertical and horizontal stripes in fashion (cf. Pastoureau, 2001). Vertical stripes make people seem slimmer, taller, more powerful. Hence they are used, for instance, in business shirts and business suits. Vertical stripes (“Breton tops”) are more down to earth. Traditionally they were part of the outfit of sailors. In the 1920s, they became popular in the then new beach culture, in the fabrics of deck chairs, parasols, and so on (see Figure 4.6). From there they came to stand for uninhibited outdoor summer fun and, for men, for the un­ conventional, rugged masculinity of the lone yachtsman or skipper, as embodied by Humphrey Bogart in To Have and Have Not. Brigitte Bardot was often photographed wearing Breton tops, and so was Pablo Picasso.

Shape

FIGURE 4.6

71

Breton top. iStock

Figure 4.7 summarizes the distinctive features discussed in his section. The curly bracket indicates simultaneous choices – any shape is determined by all of these choices. The square brackets indicate “either-or” choices, but the doubleheaded arrow means that these choices are not binary but graded.

Some examples Anyone who is in tune with contemporary popular culture will recognize what the “Breton top” stands for, even though not everyone may be able to put this into words. While the pattern reproduced in Figure 4.5 cannot be easily associated with particular periods, activities, celebrities, and so on, the “Breton top” can. It immediately evokes the pleasures of the beach or the values associated with celebrities such as Bardot and

72

Shape

angular angularity curved regular regularity irregular predictable repetition unpredictable bold weight light large size small connected internal

connection disconnected upright slope

external left-leaning

sloping wide

right-leaning

expansion condensed horizontally stretched orientation vertically stretched FIGURE 4.7

The distinctive features of shape

Shape

73

Picasso. As semioticians, it is important to document such provenances, to research, for instance, how stripes came to mean what they mean today, as has been done by Pastoureau (2001) in his excellent history of stripes and striped fabrics. Analysing provenance is for the most part analysing the provenances of styles. But shapes can of course also be associated with real world objects. Circles can be wheels or peaches, balls or heavenly bodies. Decorative patterns can hover on the borderline between recognizability and abstraction. Is it a vine or just a wavy line? Such associations with objects may be valid, but they risk becoming too subjective and idiosyncratic, and, from a semiotic point of view, they should be grounded in contextual evidence of the kind outlined in Chapter 3. In analysing the use of shape in specific examples, all three of the methods outlined in Chapter 3 should be used, and abstract as well as figurative shapes should be included. Figure 4.8 shows a likeness of some of the logos from a sample of 30 contraceptive advertisements found in medical journals, slightly altered for copyright reasons (cf. Van Leeuwen et al., 2016) – Yasmin and Diane are oral contraceptives, Mirena is a long acting contraceptive. Starting with the figurative elements, the jumping woman in the Yasmin ad­ vertisement, her shape rhyming with the “y” of Yasmin, expresses the joys of freedom, in this context presumably freedom from worry about becoming pregnant (the search term “freedom” in the Getty image bank, yields many images of jumping women, cf. Machin, 2004), and the tender butterfly is a traditional symbol of femininity and female beauty (as well as of other things, such as tran­ sience). The elongated leaf shapes in the Mirena and Diane logos sit on the edge between the figurative and the abstract – a Google image search reveals that such “lanceolate” leaf shapes often serve as abstract representations of the vulva. Although these advertisements are aimed at medical practitioners, they still allude to sex and sexuality, they still assume that sex sells. As for the distinctive features, angularity/curvature and bold/light clearly play a key role in these examples. The graphic elements surrounding the Diane brand name, for instance, combine curvature and solid, bold angularity, and this combi­ nation can also be observed in the typography of many of the other brand names in the sample. Along with femininity and sexuality, the logos also seek to express the idea of strength and reliability – an emphasis which is reflected in the verbal text of the ads. The Mirena ad, for instance, carries the slogan “Confidence that lasts.” In short, the typography and the abstract and figurative shapes of these logos lend distinctive values to the brands – values such as health, sexual freedom, sexuality, femininity, strength and reliability. These then combine in different ways to create the distinct identities of specific products and specific brands. Advertisements for long acting contraception, for instance, feature images of more mature women and focus on health and effectiveness, though they contain allu­ sions to sexuality as well. Advertisements for oral contraception, on the other hand, feature images of younger women and focus more fully on femininity, sexuality and sexual freedom.

74

Shape

FIGURE 4.8

Likeness of three pharmaceutical logos

Shape

75

To give one further example, the pattern in Figure 4.9 comes from an an­ thology of pattern designs (Cole, 2007). The designer, Carlene Edwards, says she was “inspired by the visual and physical surroundings of my London borough of Hackney. I used a vibrant colour scheme to lift the whole piece and produce a bright pattern around everyday subject matter” (ibid:25). Beginning with the figurative elements (the “everyday subject matter”), the newspaper fragment and the black and white photo of a housing estate are both repeated four times, and in the newspaper fragment we can recognize the words “table” (truncated to “tab”) and “School” and the phrase “A to C.” Clearly it was part of a league table of schools, based on the grades achieved by their students. But because we do not see the whole of the newspaper article, and because of the way the fragment is repeated, it is, in this context, meant, not to convey specific information about specific schools, but to evoke an impression of dull numbers, of statistics rather than people. The photograph may represent a housing estate, but of this we cannot be certain. It could be a school, or a hospital, or some other dreary all-purpose concrete box.

City living (Carlene Edwards). Reproduced with kind permission from Carlene Edwards

FIGURE 4.9

76

Shape

The abstract shapes (the “bright patternings” around the “everyday subject matter”) are angular, representing the city as a warren of criss-crossing elements. These shapes are densely packed together, closely connected and elongated, and they include some salient triangular right-leaning shapes, which adds a dynamic feel. They are also drawn irregularly, adding a sense of disorder and haphazard development which a more regular grid would not convey. Finally, they are re­ petitive. Despite the apparent disorder, the city is built up out of a small number of elements which differ only in surface irregularities. The outlines of the newspaper fragments and the black and white photos align with those of the abstract shapes, showing what is behind the disorderly yet vibrantly coloured abstract shapes – a monotonous and grim reality. The compiler of the anthology, a lecturer in pattern design at the London University of the Arts, classifies this design as a “conversational pattern” (Cole, 2007: 9): Conversational patterns are sometimes referred to as novelty prints and contain images of objects or situations. In these designs the artists’ inspirations are not always immediately apparent … Some patterns in this section tell a story without words or promote a point of view. If so, it is important, not only to start paying attention to such apparently ephemeral but omnipresent designs and to learn to appreciate them aesthetically but also to learn to read them critically, as points of view, and to remember that, as semioticians, as students of meaning in society, we should heed William Morris’ insistence on “plenty of meaning in your patterns” (1895: 177).

5 COLOUR

Colour in functional design and identity design Colour is a semiotic resource for functional design as well as identity design. Functionally, it serves two main purposes: creating colour codes to aid the re­ cognition of different items in specific, limited domains of meaning and creating coherence. Colour codes are mini colour “vocabularies” serving to distinguish, for instance, the different routes of public transport systems, the different varieties of a brand on supermarket shelves, the different degrees of safety on drug labels, and so on. Different codes will give different meanings to the same colours, but that does not necessarily mean that the colours are chosen arbitrarily – they may have been chosen for a reason, even if that reason will not always be evident to users. However, most colour codes will initially need a verbal legend to be understood, although people who use specific colour codes regularly will soon remember them and no longer need verbal explanation. The role of colour in creating textual coherence is threefold. Firstly, it can segment texts into their generic stages, in the way I described in Chapter 2, where colour, together with layout and typography, visually demarcated the stages of the “hot tips genre.” On the home page of the website of House Industries, a US-type foundry and design studio (cf. Lupton, 2004: 161), colour was similarly used to demarcate the key elements of the page – local navigation (a column on the left of the page) was black, branding and global navigation (a band on the top of the page) a slightly greenish grey, and the specific content space in the centre white. Second, colour can create salience, drawing attention to some elements over others. On the same website the red and black font kit in the centre of the white space, stood out as the most eye-catching element be­ cause of the salience of the colour red and the contrast between black and white.

78

Colour

Finally, colour provides overall cohesion, a function which, today, is built into many text-making technologies. In website design, for instance, cascading style sheets were introduced to give sites an integrated “uniform appearance” (Smith, 2005: 158), allowing a single external file to dictate the colour and typography of every page of the site. The background of PowerPoint presentations, similarly, remains constant across the whole of specific presentations, thus creating a sense of cohesion. All this applies not only to texts but also, for instance, to institu­ tional buildings, where the doors and walls of different departments may have different colours, to provide internal cohesion and external segmentation. But colour always also conveys identity meanings, whether the role identities of company uniforms and dress codes, or personal and lifestyle identities. The uni­ forms of the stewardesses of Singapore airlines, for instance, are blue for flight stewardesses, green for “leading stewardesses,” red for “chief stewardesses” and purple for “in-flight supervisors.” And as already mentioned in Chapter 1, the chief financial officer of a University gave a PowerPoint presentation in which the University’s annual budget figures appeared against the background of a calm ocean and a clear blue sky. This not only provided cohesion for his presentation as a whole but it also identified him as confidently in control and (combined with the quotes from business gurus with which he interlarded the budget figures) as a man with strategic, long range vision. Colour meanings are not universal. In an article documenting the redesign of the English-language South China Morning Post, designer James de Vries (2008) first described how the colour scheme he designed for the paper visually distinguished the different sections of the newspaper – local news, world news, business, life­ style, sport, etc. But designing this new palette also involved identity meanings, some of which de Vries had not expected, such as the traditional Chinese meanings of yellow as an imperial colour representing power, royalty and pros­ perity. As De Vries recounted (De Vries, 2008: 21): We proposed a crimson red for their life-style sections (which had a lot of material about shopping, beauty etc). But this met with resistance from the management committee. They thought it was too feminine and a bit “tarty” – there was a specific Chinese word for that. In the end we prevailed, though it was in fact our second approach. We had proposed an eggy yellowy colour which they had absolutely rejected. Modern softwares such as Word and PowerPoint combine a single functional colour system with multiple options for identity design. All PowerPoint palettes (“themes”), for instance, use colour to index the same specific functions: headline, copy, “fills” (e.g. the bars of bar graphs), “accents” (e.g. bullet points or hy­ perlinks) and background. This system applies regardless of whether a pre-set palette is used or whether users construct their own palettes. A colour scheme might then look like this:

Colour

Background: Text and lines: Title: Shadows: Fills: Accent 1: Accent 2:

79

relatively dark, blue-grey white yellow black orange red turquoise

The names of PowerPoint’s pre-set colour schemes give a sense of the cor­ porate and personal identity meanings on offer – “Urban monochrome,” “Metropolitan,” “Future Forward,” “Office,” “Dividend,” “Boardroom” but also “Savon,” “Feathered,” “Earthy inspiration,” “Floral Flourish,” “Droplet,” to give just some examples Thus, PowerPoint’s colour system pre-designs both what can be done with colour and what can be meant with it. And what can be meant with colour accords with the values that underlie contemporary corporate and personal identities, in which, as Machin has put it (2004: 330–331) “connotations of ser­ enity, escape and freedom bring a sense of ‘philosophy’ or even morality into the corporate world of branding and consumerism” and in which “positive thinking is a crucial moral value.”

Colour and identity – a short history Colour has always expressed identity. The colour schemes of premodern societies focused on gender, status, rank, class, ethnicity and religion. In ancient Rome, foreigners, slaves and freed slaves wore beige tunics, citizens clean white togas over their tunics, Senators togas with a purple border, while women wore more colourful clothes – red, indigo and yellow were particularly popular (Angela, 2015: 45–53). The Middle Ages introduced laws to specify who could wear which colours, “creating a form of segregation by dress, a system in which all members of society had to wear garments proper to gender, estate, dignity and rank” (Pastoureau, 2000: 87). Blue and scarlet were reserved for the aristocracy – merchants who had ac­ cumulated wealth were not allowed to wear these aristocratic colours and gradually took to wearing black. Particular colours were assigned to “men and women who practice dangerous, dishonest or dubious activities: doctors, surgeons, execu­ tioners, prostitutes, usurers, minstrels, musicians, beggars, vagabonds and all sorts of outcasts” (ibid: 91), and to the Jewish and Muslim communities which had settled in Europe, especially in southern Europe. Such dress codes differed between cities. In Venice, in the early 15th century, prostitutes wore yellow scarves, in Bologna green scarves, and in Milan white cloaks (ibid: 94). Such colours not only functioned as identity badges but also conveyed values. Medieval Benedictine monks wore black to express their core values of humility and penitence. Cistercians wore white to express innocence, purity and eternal

80

Colour

life. Such meanings were hotly debated. In 1124, Peter the Venerable, the Benedictine abbot of Cluny, accused his colleague Bernard, the Cistercian abbot of Clairvaux, of vanity and pride for using white, the colour of glory and resur­ rection, rather than black, the colour of humility and renouncement, which he thought befitted monks who had turned their back on the world. To which Bernard replied that black was the colour of death and sin (Pastoureau, 2008: 66). The same applied to knights, whose colour codes not only indicated their in­ dividual and social identity but also their values and beliefs, “[defining] their position within the socio-ethical code of chivalry and [providing] the means of public accountability for their behaviour” (Huxtable, 2006: 211). The 13thcentury Icelandic Didrecks Saga, for instance, explains that Heine the Proud carried a blue shield to signify his cold breast and grim heart, while Fasold and his brother Ecke had red shields to signify their love of fighting, and Hornbogi of Wendland and his son Amelung brown shields to express worth and courtesy (Gage, 1999: 84). But as individual identity became more important, Europe entered a blackand-white age that would last till at least the end of the 19th century (with a short intermezzo in the age of Enlightenment, from about 1720 to 1780, when espe­ cially pastel colours were in vogue). Protestant Reformers condemned multicoloured clothes and explicitly instructed believers to wear black (or at least dark) clothes to express humility and contrition, as well as simplicity and functionality, though white (“innocence”) was recommended for children and sometimes for women, and blue was tolerated so long as it remained dark and subdued (Pastoureau, 2008: 132). Artists began to favour chiaroscuro, the play of light and dark over colour, and while medieval manuscripts had been bright and colourful, the printed word and the engraved image became increasingly monochrome. And in Northern European churches, stained glass windows were replaced with clear glass, murals were whitewashed, and paintings, statues and colourful liturgical vestments were removed, as can be seen in Dutch church interior paintings of the time. The last vestiges of this black-and-white world are still with us. Most people in “Western” countries still wear black, dark blue or grey clothes to work. Black still “conveys elegance, sophistication and a touch of mystery” (fashion designer Ana Sekularac, quoted in Mora, 2009: 169). And although Hollywood movies are now increasingly “colourized,” black and white can still be a mark of artistic distinction in “art house” cinema. In the Romantic era, colour became a marker, not only of social identity but also of “personality,” although class differences in dress persisted for a long time. In his remarkable Theory of Colours (1970[1810]), Goethe first formulated the idea that colour was not about symbolic meanings, but about affect and character. “Colour,” he said, “is immediately associated with the emotions of the mind” (ibid: 304) and “in dress … we associate the character of the colour with the character of the person” (ibid: 326). This applied especially to women, who were thought to have stronger colour preferences than men (ibid: 328). Inspired by Goethe, the German “Lucasbund” painters devised colour codes for expressing

Colour

81

character, again only for women – black hair combined with the use of black and violet, black and blue, or white and violet, signified a “proud and cool” yet “cheerful and happy” personality. Blonde hair combined with blue and grey, or grey and crimson, signified “solitariness, modesty, good-heartedness and calm,” and blonde hair could also combine with grey and crimson to express “feminine amiability,” while reddish-brown hair combined with violet-grey and black and crimson to express “happiness and good temper, innocent roguishness, naiveté and cheerfulness” (Gage, 1999: 188–90). Such practices were the forebears of later psychological theories in which colour preferences were seen as symptoms of individual personality rather than as signs of social identity, and strongly related to gender. An overview of the use of colour in personality testing (Pickford, 1972), for instance, cites studies establishing that men like cool colours and women warmer colours, and that yellow is asso­ ciated with “joy, love and sexuality” and blue with “control and mother-centred emotions and attitudes,” while red is either positively associated with “love and the need to be loved” or negatively with “aggression.” This approach was widely taken up by colour consultants, for instance in relation to interior design. According to colour consultant Lacy (1996: 29), for instance, “A warm pink entrance hall indicates a home which is warm and loving. People using this colour like to care for others.” In short, colour preferences are here seen as symptoms of individual personalities, often driven by unconscious motives, and as strongly related to gender, with women more directly expressing their colour preferences, for instance in the way they dress, and men’s personalities manifested in different ways – by small details of the way they wear otherwise relatively homogeneous or professionally prescribed clothes (loose or tight, buttoned or unbuttoned, wellironed or crinkled, and so on) or by the colours of their cravats or neck ties. More recently, colour has made a comeback in many domains of life. Heralding the renaissance of multimodality, early 20th-century Futurists preached the virtues of multi-coloured clothes (Balla, 1914, quoted in Koolhaas et al., 2001: 308): Today we want to abolish … all neutral, ‘pretty’, faded, semi-dark and humiliating shades … Futurist clothes shall therefore be … joyful. Clothes in exciting colours and iridescenses. Using muscular, ultra violet, ultra red, ultra turquoise, ultra green, bright yellow and vivid oranges, and rich vermilions… If the Government does not drop its staid traditional dress of fear and indecisiveness, we will double, we will centuplicate the red of the tricolour we wear. Gradually colour returned in everyday life. The dreary regulation colours of public transport and the black of motor cars (Henry Ford for long time produced only black cars) were replaced by brighter colours. The grey suits of businessmen were enlivened by colourful ties, the faded browns of domestic interiors modernized with brighter shades. Magazines, photographs, films, television, newspapers one by one abandoned black and white and moved to colour. Desktop publishing and

82

Colour

presentation software replaced the austerity of type-written and densely printed documents with a new emphasis on visuality – on layout, colour and typography. Business presentations and annual reports, invoices and forms, diagrams and charts, began to be produced in vivid colour, and the use of colour became available to anyone with access to a computer. As a result, colour now plays a key role in communicating the lifestyle identities of individuals and communities and the branding of corporations and other or­ ganizations. In the process, identity design began to shift from a psychological to a semiotic approach, an approach based on cultural provenances and the creative use of the experiential meaning potential of colour and other parameters of style in typography, graphic design, fashion, product design, interior design and archi­ tecture. Rather than describing particular colours in terms of personality traits such as “courage” and “energy” or in terms of emotions such as “happiness, passion and love,” designers now began to base their choice of colours on cultural references, as in this quote from the Color Marketing Group, an international association of 1,700 colour designers (Colour Marketing Group, 2020), which announces “Crown” (a “new kind of violet”) as the “hot” colour for March 2020. Taking its provenance from the “Chakras” of Indian religion, in which violet is the colour of the Crown, situated just above the head, and related to spirituality, they describe Crown as being about wisdom being at one with the world, and … [offering] balance, instilled with both strength and dignity … Home appliances will take on a new vibe with Crown with everything from water kettles to vacuums benefitting from its spiritual, yet fun colour. To give another example, the model in a Vogue fashion spread (American Vogue, January 2010, p. 135) wore a minidress in black and white “houndstooth” tweed with flak vest type pockets in military brown. From her bag hung a fox tail, like a kind of hunting trophy. Consider some of the provenancs of this outfit. Tweed, originating from Ireland and Scotland, was in Edwardian times associated with country pursuits of the upper middle classes, such as hunting and golfing, and with the coats and suits of conservative professors and country women. But here it is combined with a mini-skirt and with military connotations. The outfit as a whole says, as it were: if you wear this, you are sexy, yet conservative, and yet also tough – a fighter and a hunter. Its signifiers have been taken out of the environments in which they once signified age, gender and class (the “houndstooth” tweed, hunting) and a profession (the military) to signify values typically associated with, but not necessarily confined to these environments (“conservativeness” and “fighting spirit”). And the accompanying text confirms it with a pun: the model is dressed for a “tactile manoeuvre.” While this may seem to present a powerful model for feminine identity, the un-serious, “over the top” style of the fashion spread and the model’s childish “body clowning” pose (Goffman, 1976) of course immediately undermine this.

Colour

83

People are not necessarily aware of the provenances of the colours they wear. They may, for instance, wear military colours or carry bags with camouflage motifs without intending to convey an affinity with military values. Yet the military connotation is objectively there. The military is objectively the place where these colours come from, as any image bank search would verify. Appeals to fashion and taste can disavow meaning and allow people to take distance from the very cultural trends they are participating in. Designers, however, are well aware of these provenances – and semioticians should be too. Corporate logos also use experiential metaphor and provenance to express the values they seek to embrace. The logo, the lettering on aircrafts and the cabin crew uniforms of THAI airlines, for instance, use gold, magenta and purple. The shape of the logo can be interpreted on the basis of experiential metaphor – it is a rather bold arrow, which, as the company’s website confirms (THAI, 2010), suggests “solidity” as well as “dynamism.” But it also has the curves of traditional Thai design, in­ dicating “elegance,” and the colours, too, are chosen for their provenance, as they “reflect the culture and the country” and “recall the gold of temples, the intensity of Thailand’s famous shimmering silks and the brilliant hues of the orchids” (Ibid).

The distinctive features of colour The psychological approach to colour and identity sees colour preferences as personality traits that are deep-seated, involuntary, and difficult to change. Contemporary identity design, on the other hand sees colour preferences as ex­ pressive of the values of brands and shared lifestyles, and as fluid and constantly changing. As Davis (1992: 17) has put it, “the designer-artists who initiate fashion intuit somehow the currents of identity instability pervading a people and seek through the artful manipulation of conventional visual and tactile symbols of clothing presentation to lend expression to them.” The understanding and creative use of these “visual and tactile symbols,” whether by designers or fashion-conscious users, requires not only an awareness of cultural and historical references but also reintegrates the material characteristics of colour with its role in contemporary cultural expression and social communication. These material characteristics include not only hue, value and saturation but also a range of other distinctive features.

Value The scale of value is the grey scale, the scale from maximally light (white) to maximally dark (black). As Wierzbicka (1996) has stressed, the difference between light and dark is a universal experience, and there is probably no culture which has not used it to express symbolic meanings and values, even though different cul­ tures may do so in different ways. We already discussed the rejection of multicoloured clothes by the early 16th-century Protestant Reformers. The same tendency can be observed in 17th-century Dutch painting. Pastoureau (2008: 125) explains this as follows:

84

Colour

For many Calvinist painters we can even speak of a true Puritanism of colour … That is the case with Rembrandt, for example, who often practices a kind of colour asceticism, relying on dark tones, restrained and limited in number (to the point that he is sometimes accused of ‘monochromy’), to give precedence to the powerful effects of light and resonance. From this particular palette emerges … an undeniable spiritual intensity. In other words, by refraining from multi-colourism Rembrandt unlocked meta­ phors such as “asceticism” and “austerity,” and, given his subject matter, made light express an “undeniable spiritual intensity.” But lightness is not only “spiritual,” it can also be “light-hearted” as in the lighting of Hollywood comedies, or “summery,” as when lighter clothes are worn in summer or in the tropics, or “sterile” as in hospitals. Darkness can engender fear, but also intimacy, as when people are gathered around candlelight or a fire in the evening. Rothko often used dark colours and wanted his works to be shown in dimly lit rooms, because he wanted them to be “mood pictures.” The psycho­ analyst Goldman (personal communication) changed the colour of the walls in his consultation room from a “sterile” white to a warm and rather dark blue-grey and reported that his patients experienced this as restful and conducive to the state of “reverie” psychoanalysis requires. These are just some of the ways in which the meaning potential of value can be exploited.

Saturation Saturation is the scale from the most intense, pure manifestation of a colour to “chromatic grey,” a grey with just a tinge of that colour, and ultimately to complete desaturation, achromatic grey. If colour expresses emotion, then sa­ turation is the fulness of that emotion, and the saturation scale is a scale that runs from maximum emotive intensity to maximally restrained, maximally toned-down emotion. In context, this can then acquire more precise meanings and values. High saturation might be positive, exuberant, adventurous – or vulgar and garish (for instance from the point of view of “puritan” values). Low saturation might be subtle and tender, or cold and repressed. Van Gogh used saturated colour to maximize the intensity of colours, their “passion,” so as to express the “character” of the people he portrayed (quoted in Riley, 1995: 101): I should like to paint portraits which would appear after a century to the people living then as apparitions. By which I mean I do not endeavour to achieve this by a photographic resemblance, but by means of impassioned expressions that is to say, using our … modern taste for colour as a means of arriving at the expression and the intensification of the character.

Colour

85

The Futurist “anti-neutral dress” manifesto I quoted earlier called for highly saturated (“exciting,” “ultra,” “rich”) colours such as “muscular, ultra violet, ultra red, ultra turquoise, ultra green, bright yellows and vivid oranges and rich vermilions” (quoted in Koolhaas et al., 2001: 308) so as to express “joy,” “deci­ siveness” and “daring.” In the same period, fashion designer Paul Poiret not only liberated women from the corset but also introduced bright primary colours into fashion, thereby “disturbing the ultra-conservative retina of the bourgeoisie” (Mora, 2009: 13).

Purity Purity is a scale that runs from the maximum “purity” of undiluted colour to maximum “hybridity” or “mixedness.” The question of colour purity emerged as soon as people realized that dyes and paints could be mixed. In Antiquity, mixed colours were thought to be “defiled,” inferior to unmixed colours. In con­ temporary colour theory, the “primary” colours, red, blue and yellow, are unmixed and can be mixed to create secondary and tertiary colours. But others define “primary” on linguistic grounds, regarding colours with common names such as “brown” and “green” as primary, and colours with compound names such as “blue-green” or “yellow-green” as mixed. Although the search for primaries has never resulted in a generally accepted system and has “proved to be remarkably inconsequential and … freighted with a heavy burden of ideology” (Gage, 1999: 107), people do perceive colours either as irreducible or as mixing, for instance, visibly “blueish” and visibly “reddish” characteristics. Sometimes purity is pre­ ferred, sometimes hybridity. Fashion designer Karen Walker uses “colouring book colours” because “they’re strong and there’s no complexity to them – they’re totally up-front” (quoted in Mora, 2009: 81). Fashion designer Bora Aksu, on the other hand, prefers “in between colours” (ibid: 93): I like colours that belie description. When you see a colour but cannot label it instantly such as ‘this is red or green’ this draws me to it … The colours in between other colours are also attractive.

Transparency Transparency is the scale that runs from transparent to opaque, via translucency. A colour is transparent when light can pass through it, so that things in the background can be seen behind the coloured foreground. It is translucent when the light is partially blocked, making the background hazy, or altogether undecipherable. Water colours show the paper behind the colour, glazing the colours behind the glaze, coloured stain the grain of the wood. Through such media colour becomes light and ethereal, lying like a film over the material it covers, and the materiality of the paper (or the canvas or the wall of a mural) becomes part of the work, transforming the

86

Colour

colour and being transformed by it. Coloured glass, similarly, both conceals and transforms what lies behind it, “embedding colour within a translucent overlayer” (architect Shawn Mulligan in Mora, 2009: 107). Matisse used it in a fresco of dancers above the door to a garden, so as to place the dancers “away from the detail of what people are doing on the earth toward the ethereal areas of sky” and he also used it in a fresco in Moscow (Matisse, 1972: 117): When I undertook the Moscow Dance and Music, I had decided to put colours on flat and without shading. What seemed essential to me was the surface quantity of the colours. It seemed that these colours, applied by no matter what medium, fresco, gouache, watercolour, coloured material would give the spirit of my composition. I was quite astonished, when I saw the decorations in Moscow, to see that I had, in applying my colours, played a little game with the brush in varying the thickness of the colour, so that the white of the canvas acted more less transparently and threw off a quite precious effect of moire silk.

Luminosity The luminosity of a colour lies in its ability to glow from within. Luminous colours are “self-illuminating colours that place an aura around things,” as designer Alessandro Mendini put it (quoted in Koolhaas et al., 2001: 243). Lighter and more strongly saturated colours are more luminous than darker and less saturated ones, and projected colour is more luminous than surface colour. In many cultures and periods, luminosity has been one of the most highly valued characteristics of colour. While Renaissance and 17th-century Dutch painters depicted the light falling on people and objects, medieval painters wanted light to radiate directly from the colours and used contrasting blues and greens to bring out the luminosity of gold paint as strongly as possible. Its quality of radiating from within suggests the meaning potential of luminosity – splendour and glamour as well as the super­ natural and the divine. In a letter to his brother Theo, Van Gogh (1978: 25) explains his use of luminous colours to signify the “eternal”: I try to paint men and women with that something of the eternal which the halo used to symbolize and which we seek to envy by the actual radiance and vibration of our colouring.

Luminescence Luminescent colour is emitted directly by a light source, for instance a TV screen or neon lights. Its quality of emitting light is the key to its meaning potential. Light has long been experienced as supernatural, and the sun has been worshipped in many religions. Dionysius the Areopagite, a 1st-century judge at the Areopagus

Colour

87

Court in Athens, described God as “light,” “fire” and “fountain of light” (Eco, 2002: 102): Light derives from Good and is the image of Goodness, hence Good is celebrated with the name of Light … [Goodness] illuminates all those things able to partake of it, and its light is diffused over all things while it spreads the splendour of its rays over all the visible world. Today’s cities shine with the immaterial glow of humanly produced radiant objects – the light emitted by neon advertising, the brightly lit skyscrapers and the radiance of television and computer screens. Architect Rem Koolhaas (Koolhaas et al., 2001: 12) stresses both the glamour and the magic of luminescence: We have increasingly been exposed to luminescent colour, as the virtual rapidly invaded our conscious experience – colour on TV, computers, movies – all potentially ‘enhanced’ and therefore more intense, more fantastic, more glamorous than any real colour on real surfaces. Colour, paint, coatings in comparison, became matt and dull

Lustre Lustre results from the reflection, rather than the emission or transmission of light. The key to its meaning potential lies in its brilliance, its ability to light up. In the Middle Ages, lustre came from purple gowns, from the shiny tesserae of mosaics, and from the gold overlays on paintings, sculptures and other objects. Today it comes from the gloss of expensive magazines, the metallic sheen of motorcars, the shine of marble. A Vogue Italia New Trends Report of 1995 hailed it as follows (quoted in Koolhaas et al., 2001: 273): Lustres: midnight satin, coloured paint, mock-crocodile and Cadillac metallic effect, high-tech fabrics with an iced luminosity, magic embroid­ eries. Plenty of deep white.

Temperature Temperature is the scale from blue to red. In a parametric approach to colour it becomes only one of the distinctive features constituting the complex and composite meanings of colour, and not necessarily always the most important one. Nevertheless, although “the” meaning of red cannot be reliably established, the red end of the colour temperature scale remains associated with warmth, energy, salience, foregrounding, and the blue end with cold, calm, distance, backgrounding. The cold-warm continuum has a wide-ranging meaning potential. Itten (1970) listed sedative/stimulant, rare/dense, airy/earthy, far/near, light/heavy and wet/dry.

88

Colour

In actual reds, warmth combines with other distinctive features. Red is never just red, it might for instance also be very warm, medium dark, highly saturated, and quite pure and modulated. Its meaning potential will then flow from all these features as they are proportionally applied and configured in specific contexts.

Modulation The modulation scale runs from fully modulated colour, for example from a blue that is richly textured with different tints and shades, as in paintings by Cézanne, to flat colour, as in many comic strips or in paintings by Matisse. It was already recognized as a feature of colour by Goethe, and Itten (1970: 19) notes that Titian already described it as “one principal hue and many variant tints and shades.” He adds that variation in saturation also plays a role in modulation. Modulated colour can not only be used for purposes of naturalistic depiction but it might also be used as source of meaning on its own, suggesting qualities such as “nuanced,” “subtle,” “variegated” or, more negatively, “fussy” and “overly detailed.” Matisse used flat colours as a reaction against the “jumpy surface” and “disruptive vibrato” of the modulated, textured effects other painters used (1972: 143). Flat colour can not only be used to show what are thought to be the essential colours of things (“grass is green,” “water is blue,” and so on) but it might also be a source of meaning on its own and used for its “boldness” or “simplicity,” Matisse saw flat colours as “calm” and used them to escape naturalism and stress the in­ dependent expressiveness of colour. Alessandro Mendini (in Koolhaas et al., 2001: 238) values modulation in architecture, but argues that it should come from the influence of the environment on in themselves flat colours: As I work increasingly as an architect, I try to be careful about the relation between colour and material: in the transparency of glass, for instance, in its translucency, its being matt or lustrous or wrinkled, its systems of veining. As a rule I tend to use flat colour, which later acquires nuance in relation to light and shade or through the effect of curves. I have described modulation not only as a quality of a single colour but also as a quality of colour schemes. A Rothko painting, for instance, might have subtly shaded reds and maroons. Are these single colours or a colour scheme of different reds and maroons? The question becomes easier to answer if we move from interpreting the meaning of individual colours to interpreting the meaning of colour schemes, as I will advocate in the next section of this chapter. Modulation is also closely related to texture. Its use might be motivated by naturalistic considerations, such as the shape and depth of shadows, but modulations may also be distributed randomly, or display more abstract patterns – striations, mottling, veining, and so on. These can then be interpreted on the basis of the distinctive features of shape I discussed in the previous chapter.

Colour

89

Differentiation Differentiation, too, is a property of colour schemes rather than individual colours. It is the scale that runs from monochrome to the use of a maximally varied palette. Its meaning potential again flows from what it literally is: varied. High differ­ entiation could therefore mean, for instance, “diversity” or “exuberance,” low differentiation “restraint.” In an article from a home decoration magazine, a couple used “nearly the whole spectrum” in their house and commented that “it’s great that there are so many bright shades in the house. It’s a shame people aren’t more adventurous. It’s when you start being timid that things go wrong.” (House Beautiful, September 1998: 21). High differentiation here expressed “adventur­ ousness” and low differentiation “timidity,” but it is clear that in other contexts restraint might be more positively valued. A closer look at the couple’s “adventurous” colours (Figure 5.1) reveals that they are not quite as bright as the two made out. They seem relatively dark, “impure” and somewhat desaturated. Such colours were very much in fashion in the 1990s and they had a specific provenance in the historical television series of the period. They are in fact an exact match to some of the historic colours on the colour chart of paint manufacturer Farrow & Ball, a specialist firm which produced authentic colours for the television adaptations of Pride and Prejudice and Middlemarch and subsequently saw the sales of these paints to domestic customers rise by 40% each year over a ten-year period (Guardian Weekend Magazine,

FIGURE 5.1 Historic colours (House Beautiful, September 1998: 21). Reproduced with kind permission from Colin Poole

90

Colour

19 January 2002: 67). Perhaps the couple’s interior was not just a creative ex­ pression of their personalities (it was that as well) but also based on cultural re­ ferences to the period where these colours came from. As can be seen in Figure 5.1, the couple’s furniture is not exactly modern, and their lounge room is decorated with black and white photos and vintage wooden figures of a balding English butler and a black jazz band. By decorating their home in this way, the couple might be said to identify with the nostalgia for that “lost Englishness,” which John Major characterized at the time as “long shadows on county grounds, warm beer and invincible green suburbs.” But this not explicitly expressed in the Home Beautiful article. Figure 5.2 summarizes the distinctive features discussed in this section. As before, the curly bracket indicates simultaneous choices (any colour is determined by all these choice) and the square brackets indicate “either-or” choices, with the double-headed arrows indicating that these choices are not binary but graded.

Colour schemes Colours rarely come separately. Even the “little black dress” combines with the tones of the skin and the colour of the hair of its wearer; even colour field paintings hang on white gallery walls. Artists and theorists alike have always seen colours as interactive, brightening or dulling each other, harmonizing or clashing. In a letter to his brother Theo, Van Gogh (1978: 28) explained how, in The Night Cafe (1888) he had expressed “the terrible passions of humanity” by means of “a clash and contrast of the most disparate reds and greens”: I have tried to express the terrible passions of humanity by means of red and green. The room is blood red and dark yellow with a green billiard table in the middle; there are four citron-yellow lamps with a glow of orange and green, everywhere there is clash and contrast of the most disparate reds and green … the blood red and the yellow-green of the billiard table, for instance, contrast with the soft tender Louis XV green of the counter in which there is a pink nosegay. The white coat of the landlord, awake in a corner of that furnace, turns citron-yellow, or pale luminous green Matisse advocated designing a scheme of three to six colours before starting a painting “to find the equations that make up the life of a picture.” As he explained this to his students (1995: 267) he pointed at a house: “Do you see the colour of the foundation, of the molding, of the wall and the shutters? That forms a unity, a similar unity to that which a picture needs.” Some colour schemes define an era, a culture or an institution. The traditional Indian colour scheme is marked by red-orange, yellow-orange, rust colour, as well as blue-purple, blue-green and yellow-green. The traditional Japanese scheme has more blues, purples and greys, as well as browns and red-orange colours. The Mondrian colour scheme of pure red, yellow and blue (plus black and white) came

Colour

light value dark strong saturation weak pure purity impure transparent transparency opaque luminous luminosity dull luminescent luminescence dim lustrous lustre matt red temperature blue modulated modulation flat multi-chrome differentiated monochrome FIGURE 5.2

The distinctive features of colour

91

92

Colour

to characterize Modernity, while a more “impure,” hybrid colour scheme of pale, anaemic cyans and mauves became, for a time, a key signifier of Postmodernity. Functional colour schemes use distinct strong colours for purposes such as dis­ tinguishing between protection devices (red), switch control boxes (blue), first aid equipment and medicines (green), and so designers such as fashion designer Bora Aksu also work with colour schemes (quoted in Mora, 2009: 93): I love the effect of splashing outstanding colours into a muted colour palette. My AW 2007–2008 collection was a collection of warm eggplants and fuchsias with contrasting silver grey. Traditional ideas of colour harmony are based on the “colour wheel,” which arranges the primary colours (red, blue and yellow) in a circle, with, in between, the secondary colours that result from mixing them – orange between yellow and red; green between yellow and blue;purple between red and blue. Secondary colours are then said to harmonize well with their “complementary colour,” i.e. the primary colour that was not involved in their mix. Thus blue harmonizes with orange, red with green, and yellow with purple. Variations include the “Double Complementary Colour Scheme,” which has two sets of complementary hues, e.g. blue and orange as well as yellow and purple, the “Split Complementary Scheme,” which combines any hue with two hues on each side of its com­ plementary, for instance red with yellow-green and blue-green, the “Harmony of Adjacents,” which combines three or more colours that are next to each other on the colour wheel, for instance yellow-orange, orange and red, and the Triad Colour Scheme which combines three colours equidistant from each other on the colour wheel, e.g. yellow, orange and red, or yellow, red and blue (this will always lead to a bias for either “warm” or “cool” colours). William Ostwald was the first to suggest that colour harmony may also be based on value or saturation, for instance in “monochromatic harmonies,” harmonies of different values of the same colour (Scott Taylor, 1935). He favoured colour schemes of four different values of two complementary colours. His ideas influ­ enced Itten’s take on modulation (“one principal hue and many variants and tints and shades,” Itten, 1970: 19). More recent theorists have pointed at the value of disharmony, making comparisons with music, where the to and fro between dissonance and harmony is fundamental to the interest and “drama” of much European music. As Arnheim has said (1974: 348), colour harmony based on similarity and concordance is …the most primitive kind of harmony, suitable at best for the colour schemes of nurseries and baby clothing … A colour composition based on nothing but such a common denominator could describe only a world of absolute peace, devoid of action, static in mood.

Colour

93

The influential colour theorist Albers also criticized the traditional approach to colour harmony (Albers, 1975 [1963]: 42): We may forget for a while those rules of thumb of complementaries whether complete or split and of triads and tetrads as well. They are worn out … No mechanical colour system is flexible enough to precalculate the manifold changing factors in a single recipe. Good painting, good colouring is comparable to good cooking. Even a good cooking recipe demands tasting and repeated tasting while it is being followed … By giving up preference for harmony, we accept dissonance to be as desirable as consonance. Colour schemes may also be inspired by naturally occurring colour combinations, such as the colours of a seashell or an autumn leaf, or by the colour schemes of Pompeian frescoes or Baroque paintings. Many contemporary designers use colour in this way, often combining different sources, as in the case of Bora Aksu, who likes “a pinch of punk, a pinch of Edwardian and a pinch of dream” while at the same time taking inspiration from “nomadic warriors” (quoted in Mora, 2009: 93). Like van Gogh, such designers think semiotically, contrasting or harmonizing colours on the basis of the meanings they want to express, rather than on the basis of formal schemes. The relatively proportionality of the different colours is an important part of colour scheme design. Often one colour is “anomalous,” contrasting with the others, but used sparingly, to stand out as an “accent.” The architect Norman Foster uses this in a functional way. His designs for airports and metro stations tend to use neutral colours, to “offer some calm, tranquillity and reassurance amid the stresses of 20thcentury travel” (quoted in Koolhaas et al., 2001: 123) but strong colours, for instance reds, for direction signs and announcement. But accents of this kind might also just add a touch of excitement to an otherwise placid, calm scheme. Thus fashion designer Allegra Hicks “teams saturated colours with muted organic ones,” loving “the effect of splashing outstanding colours into a muted colour palette” (quoted in Mora, 2009: 93). Figure 5.3 shows a colour scheme designed by Le Corbusier. Although Le Corbusier had earlier advocated entirely white interiors (“Every citizen must re­ place his hangings, his damasks, his wallpapers, his stencils, with a plain coat of white Ripolin,” Le Corbusier, 1987: 188), in the early 1930s he accepted a commission from wallpaper manufacturer Salubra to design a colour scheme for wallpapers. He selected 43 plain colours and presented them in 12 “keyboards” which had names such as “Universe,” “Sand,” “Landscape” and “Verve.” Each keyboard had a base colour in three different values (see the green bands in Figure 5.3) and 14 other tones. In terms of distinctive features, most of the colours shown were relatively light. In the 1920s, progressive people favoured light, airy interiors. They are also relatively desaturated and impure – earth colours in several values dominate. Differentiation is reduced to natural greens and earth colours, with just a touch of red. Finally, they are unmodulated and matt – texture and

94

Colour

“Landscape” colour scheme (Le Corbusier, 1931). FoundationLeCorbusier/ ADAGP, Copyright Agency 2020

FIGURE 5.3

lustre will derive from the play of light and shade in the environment in which the wallpapers will be used, rather than from the colours themselves. Other keyboards follow the same pattern. The “Sky” keyboard for instance combined pale blues with earth colours and warm greys in different values, adding just one red. As a whole, the keyboards were therefore structured along an opposition between such natural colours (“the major scale,” as Le Corbusier said) and “disturbing colours” that provide accents, or a bit of spice, a touch of dissonance. The analysis of colour schemes should focus first of all on each of the individual features discussed in this chapter, and then try to synthesize them in an inter­ pretation that takes the co-text and the context into account. Although an in­ terpretation of this kind emerges from a methodical and detailed analysis of the way colour is actually used, there is no simple formula for measuring what each feature contributes to our integrated perception of the whole. Interpretation re­ mains a creative act, and several equally valid interpretations may be possible, especially when the object of interpretation is decontextualized, when we do not know whether, for instance, an abstract pattern is used for a gift card, as wrapping paper, or on the cover of a book, and to whom and on what occasion the gift is given, or for the cover of which book the pattern is used.

Colour

95

I will end this chapter by returning to an example from Chapter 4, “City Living” (shown again in Figure 5.4). In Chapter 4, I interpreted it as expressing “city living” as chaotic yet lively on the surface, but repetitive, monotonous and grim underneath. What would an analysis of the distinctive features of colour add to this? First of all, we can observe that the background is not only somewhat darker than the foreground but also less differentiated – while the foreground uses a range of colours, the background is monochrome and hence maximally desaturated,

City Living (Carlene Edwards). Reproduced with kind permission from Carlene Edwards

FIGURE 5.4

96

Colour

which accords with my earlier analysis. But while the foreground uses flat, un­ modulated colours, the background includes black and white photographs, and hence the fine modulations of the grayscale. This makes it less abstract, suggesting that it reveals the “reality” beneath the surface. And although the colours of the surface are varied, they are (with the exception of the red lines) not as warm as they might be and they are also somewhat unsaturated and impure – the blue is a somewhat greyish blue, the green is a somewhat greyish and bluish green, and the lightest colour is not a pure white but a light cream. In short, the “vibrancy” of the city is there, but it is tempered, as if by a layer of polluted air. Yet vibrantly red lines crisscross through it, as if providing the city with its lifeblood. Clearly, all this adds further nuances to the analysis of graphic shape and brings out how, in multimodal texts, all stylistic parameters affect each other and integrate in our perception of the whole.

6 TEXTURE

The semiotics of materiality Today, texture plays an increasingly important role in identity design. Fashion trend advisers tell their clients that “the right material can be more important to your consumer than patterns or shape” (Trendstop, 2020), and the “nine trends” Elle (2020) announces for 2020 stress the meanings of the textures of fabrics and of their enhancements with embroidery and other embellishments – “silk, linen and crisp cotton” are “Victorian-inspired” and “modest,” “delicate embroidery and eyelets” provide “a touch of craft and folklore” and suggest “rustic romance,” for instance. The same applies to interior designers and architects: “When the viewer imagines how their surfaces feel to the touch it creates an emotional connection” (Wolf Architects, 2020). Digital technologies also incorporate texture. PowerPoint, for instance, offers, not just white or solid-colour backgrounds but also background textures such as “parchment,” “white marble,” “pink tissue paper,” and “woven mat.” The meanings of texture derive from our direct, embodied experience of the materiality of objects, from touch and taste. In the Western tradition of semiotics, however, materiality has long been regarded as not-yet-meaningful, “presemiotic.” Saussure (1974: 12) described the materiality of speech, the “phonic substance,” as “a vague plane of sounds,” and Hjelmslev (1961), while recognizing that signs have substance, did not see this as playing a role in the process of meaning-making. To talk about language was to talk about form, not substance, about the shape of the mould, not about the stuff that is being moulded by it. As a result, the handwritten and the printed sentence, the sentence written in the sand and the sentence carved in stone are treated as identical for the purposes of linguistic analysis. And linguistically inspired approaches to other semiotic modes followed suit. In Reading Images (Kress and Van Leeuwen, 2006), Kress and I applied the same analytic criteria to paintings, etchings, relief sculptures,

98

Texture

photographs, and so on, disregarding their material differences. This works at the level of the functional design principles of language and other semiotic modes, but in the age of identity design we need a semiotics that can explain why and how a glass bottle means something different from a plastic bottle, a velvet jacket something different from a denim jacket, a shiny surface something different from a matt surface or a crusty hard loaf of bread something different from a soft loaf of white bread. Such a semiotics needs to be embedded in the still broader context of a semiotics of action, as it is through our embodied and active experience of materiality that the qualities of materials can be perceived and understood as meaning-making. A semiotics of action should not oppose touch too strongly to the other senses. Looking at things can also be a form of action. All sensations can be “passive” or “active.” They are passive when impinging themselves on us unbidden, so that we become aware, suddenly or gradually, of a darkening of the visual field, a foul or agreeable smell, a glowing of heat or a cold draught, the touch of something wet in our skin. They are active when they result from an active scanning of our en­ vironment – looking around, pricking up our ears, sniffing to find the source of a smell, exploring an object with our fingertips. Yet there remains a difference between, on the one hand, sight, sound and smell, and on the other hand, touch and taste. With our hands and our mouths, we can perceive material objects as well as transform them, in the case of touch with or without the aid of instruments. Looks can kill, they say, and it is true, looks can acknowledge, appreciate, un­ nerve, terrify, and much more. But the looks that kill, the sounds that alarm, the smells that mark the territory, are interactions with fellow creatures, not with material objects. Material objects we can see, but not change with our eyes; hear, but not change with our voice; smell, but not change with our chemical excre­ tions. For this we have two organs, our hands and our mouth, and their actions play a fundamental role in the exploration of material objects. Perceiving texture is always an interaction between particular actions and particular material qualities. We perceive the roughness or smoothness of a surface by running a finger over it; its temperature by a tentative touching; its thinness or thickness by pinching or rubbing; its rigidity or pliability and its ability to be bent, kneaded or stretched by palpitating, squeezing, and bending, etc. Material qualities may be naturally given – the rough texture of tree bark, weatherbeaten rocks, rusted iron, for instance. But textures can also be artificially enhanced or even synthetically produced to provide culturally preferred material qualities. Milk, for instance, is (or was?) a dairy product, but it is transformed in multiple ways before it reaches the supermarket shelves – not only pasteurized and vitaminized, but also homogenized, made sweeter or sharper, flavoured, and available in liquid as well as hydrated form. Today it can even be synthesized, by companies such as Edlong: “There is a science and an art to creating authentic dairy taste…only Edlong delivers the subtle flavour nuances that creativity inspires and experience ensures” (Edlong, 2020). The point is, material qualities derive, as often as not from social practices of transforming materials. They are socially

Texture

99

produced, in some cases by enhancing natural materials, in other cases by being synthesised from materials that have little to do with the material qualities pro­ duced. In materials like plastic, the social production of material qualities reached a new stage. As I will discuss in more detail later in this chapter, almost every kind of plastic (as defined chemically) can be given almost any material quality. It can be hard or soft, thin or thick, rough or smooth, breakable or unbreakable, tearable or untearable, flexible or rigid, transparent or opaque, and it can be made available in the form of fibres, foams, pellets, sheets, and more. Here an abstract system of material qualities precedes the materials, so that materiality is no longer a medium, in the sense in which I defined that term in Chapter 3, but a mode. And as plastic is made from crude oil, raw plastic would indeed be “pre-semiotic” and “amor­ phous,” looking like a shapeless oil slick. Colour, too, has moved from “medium” to “mode,” from concrete materiality to an abstract system that can be realized in different materials. In Chapter 2, we saw how an Aboriginal artist used natural materials such as charcoal and natural ochre to express Gija values based on, and embedded in, traditional Gija practices. In the Europe of the Middle Ages, too, the materiality of colour was still important. Red, for instance, was made from kermes insects, but when the Spanish conquistadores returned from the Americas they brought not only gold and silver and new food­ stuffs but also a new red, made from cochineal beetles (Finlay, 2002: 124). Such colours were expensive and valuable in their own right. Patrons stipulated the use of expensive pigments to add value to the paintings they commissioned. But in the Renaissance, new types of paint were developed that made mixing much easier and much more effective. As a result, individual colours were no longer unique materials and no longer precious: “Beauty does not mean ultramarine at sixty scudi the ounce,” said Paolo Pino, a 16th-century Venetian painter (quoted in Gage, 1993: 131). This, step by step, led to colour systems such as the still widely used “colour wheel” of Albert Munsell (1858–1918) which has five principal colours (Yellow, Red, Green, Blue, Purple), and their complementaries (Blue-Purple, Blue-Green, Red-Purple, Orange, Yellow-Green), ten steps from light to dark and ten steps from neutral grey to greatest saturation. Colour now became a system that can be realized in different materials – pigments and crayons, but also the chemistry of photographic materials and the red, blue and green phosphor dots of electronic screens. But, as I argued in Chapter 5, in the age of identity design, material qualities of colour that were not included in such systems, such as transparency, lustre and luminosity, have become important again. To sum up, •

•

In a semiotics of materiality, touch and taste are key sources of experiential meaning potential, creating active and embodied relationships between spe­ cific actions and specific material qualities. Visual and aural perception should also be seen as active and embodied, rather than only as mental processes.

100

•

•

Texture

As often as not, material qualities are socially produced, either by enhancing and modifying natural materials or by forms of synthesis that use materials which bear no resemblance to the materials synthesized. Synthetic materials operate as modes. A model, an abstract system of qualities, precedes their material realization, and the same system can then, at least in principle, be materially realized in different ways, for instance physically as well as electronically.

The revival of texture in identity design As in the case of colour, artists played a key role in the revival of texture as a resource for meaning-making. After the Middle Ages, when texture had been highly valued, the materials used by artists became more homogeneous. Painters used the same kinds of canvas and oils, sculptors the same bronzes and marbles, composers the same combinations of instruments, whether in the symphony or­ chestra or the string quartet. Meaning came, not from materiality, but from the design of the work, the drawing, the script, the score. As a result a work could be realized in different materials and still regarded as the same – reproductions of well-known paintings, miniature versions of famous sculptures, piano versions of orchestral works, and so on. In the early 20th century, all this began to change. In a 1912 manifesto titled “Technical Manifesto of Futurist Sculpture,” Umberto Boccioni wrote: Destroy the literary and traditional ‘dignity’ of marble and bronze statues. Refuse to accept the exclusive nature of a single material in the construction of a sculptural whole. Insist that even twenty different types of materials can be used in a single work of art in order to achieve plastic movement. To mention a few examples: glass, wood, cardboard, iron, cement, hair, leather, cloth, mirrors, electric lights, etc. Many artists began to put this into practice. The collages of Kurt Schwitters (1887–1948), his so-called “Merz Bilder,” for instance, “start with the material, every possible material, and use it as paint” (Schwitters, quoted in Leaper, 2016). In his “Opened by Customs” (1937) (Figure 6.1), he daubed oil paint and crayon marks on a collage of different materials – wrapping paper, a fragment from a German newspaper containing travel-related words such as “boarding pass,” “baggage insurance” and “sleeper car,” a scrap of paper with the word “Norway” written in Norwegian, a blue label for Spanish oranges (in reverse and upside down), labels with the words “Opened by Customs” and Hanover post stamps. In this way, he told the story of his life as a refugee from Nazi Germany – dis­ placement and fragmentation, a confused melee of languages, the marks of sus­ picious authorities. At the same time, composers began to be interested in the texture of a wide range of sounds. Satie added a typewriter to the orchestra, Gershwin a taxi-horn,

Texture 101

FIGURE 6.1

Opened by Customs (Kurt Schwitters, 1937). 2020, Tate. Photo Scala,

Florence

Antheil a propeller. The Futurist composer Russolo created a noise orchestra that included buzzers, bursters, a thunderer, whistlers, rustlers, gurglers, a shatterer, a shriller and a snorter, and the Dada poets of the 1920s began to treat words like music (“concrete poetry”), as in this poem, also by Kurt Schwitters, who was a truly multimodal artist:

102

Texture

priimiitittiii tesch priimiitittiii tusch priimiitittiiii tescho (and so on)

tisch tesch tischa

In the early 1920s, students of the Bauhaus, were told to explore the tactile qualities of a range of materials before they could enrol in specific workshops such as furniture making, ceramics, graphic design and weaving. As documented by teacher Johannes Itten (quoted in Fiedler, 2006: 365) As an introduction, long lists of different materials such as wood, glass, textiles, bark, pellets, metal and stone were recorded. Then I had the optical and tactile qualities of these materials assessed. It was not enough simply to recognize the qualities. The characteristics of the materials had to be experienced and described. Contrasts such as smooth-rough, hard-soft and light-heavy not only had to be seen but also felt. Itten made them close their eyes as they felt these materials with their fingertips. For a later version of the course, László Moholy-Nagy devised elementary ex­ ercises in relating gestures, tools and materials to each other, asking students for instance to produce works with paper and needles or tongs using gestures such as piercing, pressing, rubbing and boring. The same interest in materiality existed in the short-lived but influential Constructivist art that blossomed in the “art la­ boratories” of the early days of the Russian Revolution. Here students of Tatlin and Rodchenko also conducted “immediate studies of materials in the material itself, in order to discover the aesthetic, physical and functional capacities of such materials” (Stepanova, quoted in Gray, 1962: 251), all of this in the service of moving from creating art works to designing practical objects such as teapots, chairs, furniture, clothing and of course architecture, precisely the kind of objects which play a key role in identity design: “Let us cease our speculative activity [painting pictures] and take over the healthy basis of art – colour, line, materials and forms – into the field of reality or practical construction” (Alexei Gan quoted in Grey, 1962: 256). Texture had also come to play a role in early childhood education. Inspired by the Romantic educational ideas of Jean-Jacques Rousseau and Pestalozzi, Friedrich Fröbel had, in 1837, opened the first “kindergarten.” He gave young children blocks in the shape of spheres, cubes and cylinders and wooden pins to make holes in paper, to develop their senses and create “a sensitive, inquisitive child with an uninhibited curiosity” (quoted in Elkind, 1997). Just as the semiotics of identity design I am proposing moves from embodied experience to meaningmaking, so Fröbel’s approach had children move from the concrete to the abstract,

Texture 103

for instance from actually manipulating objects by moving them around to the first principles of mathematics. Later, Maria Montessori, like Itten in the Bauhaus, would give young children rough and smooth pieces of cloth and sandpaper to feel while being blindfolded – Brosterman (1997) has pointed out that many of the innovative early 20th-century artists had in fact enjoyed this kind of early child­ hood education. The practices which pioneers like Fröbel and Montessori introduced are still central in early childhood education. Children are given “mystery boxes” with tactile and smell experiences, encouraged to make creative use of recycled ma­ terials such as egg cartons, cardboard boxes and bottle tops, and allowed to play with water and sand and mess with finger paint. Montessori Australia provides nursery schools with materials such as boards of sandpaper mounted on wood that are explicitly designed for exploring textural qualities such as “rough and smooth” and “fine and coarse,” and for developing “sensory discrimination,” “fine motor skills” and “mathematical thinking” (Montessori Australia, 2020): The first board introduces the contrasts of rough and smooth. The second board helps to coordinate finger movement and builds dexterity. The third board introduces gradation of texture from fine to coarse. Ormerod and Ivanic (2002: 69–72) have given many examples of children’s creative use of material resources in their school projects, testifying to what Freud once described as “die strahlende Intelligenz des Kindes” (“the radiant intelligence of the child”): In her project on birds, Denise attached feathers and broken egg shells to the pages […] in his project on aggressive blading, Robbie carefully smeared some rollerblade wax on one of the pages […] and Ray includes some thick card tickets in his railway project […] Kyrah’s butterfly collage contains wood shavings crated from sharpening coloured crayons [and] Lucy created her beautifully embroidered picture of an owl out of silk fabric and thread using the different stitching devices on her mother’s electric sewing machine to create the impression of the delicate intricacy of the bird’s feathers. No wonder that some educationalists worry about the diminished textural affor­ dances of tablets and mobile phones, for instance Armstrong (2006): Television and computer screens are not the sensory-rich environment that young children need in order to exercise their multimodal brains […] young children need hands on interaction with the content of the real world. From the point of view of a semiotics of materiality, the “touch screen” is indeed a very unresponsive surface. The different gestures it allows us to make – tapping, pinching, dragging, and so on – produce a visual but not a tactile response. The

104

Texture

digital age also diminishes opportunities for touch in many other ways. Skyping or zooming with distant loved ones does not allow us to sense each other’s bodily presence, to touch, to share food and drinks. Buying clothes online does not allow us to feel or physically fit clothes. Buying groceries online does not allow us to test the ripeness of a peach or avocado. Perhaps as a result of this there is a renewed hankering for touch, a new interest in making meaning with touch, in art as well as in many identity design practices, including branding. To give some examples, Mercedes-Benz foregrounds the metallic texture of its silver paint (“It’s like spraying liquid metal,” Mercedes-Benz, 2020). Dulux foregrounds the materiality of its colours, blurring the boundaries between colour and texture, between the visual and the tactile (Completecraft, 2020): Popularity in texture continues to grow […] Suede [creates] a stylish finish for feature walls or throughout the room […] Metallic creates the subtle polished sheen or precious metals on your walls, doors, trim or furniture. River Rock [is] a subtle texture inspired by weathered rocks from nature’s riverbeds that adds a natural ambience to your room, and Pearlustre [is] an intriguing and luxurious lustre which adds vibrancy and radiance to your home. I already mentioned the work of Aiello and Dickinson (2014) on the use of texture by Starbucks. As they tell the story, in 2009 architect Arthur Rubinfeld designed a “Kit of Parts” store design model for the company. It specified specific ranges of colours, furniture, light fixtures, murals and artwork which local designers were to use in “creating seemingly distinctive looks” to express Starbuck values such as environmental sustainability by using local, recycled materials, communityorientation by using “community tables” and authenticity by using “purposefully grainy, rough and uneven” materials and the textures and provenances of “woods, slates, metals, leathers and textiles” (ibid: 305). Here branding is no longer only a matter of logos, distinctive visual emblems, but of the whole environment, and of all the materials used in the interior decoration of the Starbucks coffeeshops (ibid: 308), which should be: filled with knotty and discoloured wood panelling, live-edge granite countertops, scratched slate boards, cracked leather armchairs, clotty concrete ceilings, unpolished metal fixtures, stools and sinks, and rustic canvas ropes and wall tapestry. Fashion designers also increasingly work with the material qualities of fabrics, with yarn, weight and stitch size, with the firmness and roughness of denim and the lightness and smoothness of chiffon, with changing fabrics by pleating or exposing them to shrinking, creasing, metalizing, singeing, and so on, and by adding stitching, trimming and embellishment or even by creating whole new textures with 3D printing and materials such as wood, paper and metal. Alexander

Texture 105

McQueen, for instance, uses embroidery to create texture and embellishes ma­ terials with shells, feathers and other elements. All of this yields a rich meaning potential. As fashion designer Jyoti Sachdev (2014) wrote, fabrics can be “loopy,” “fuzzy,” “furry,” “soft,” “shiny,” “bulky,” “rough,” “crisp,” “smooth,” “sheer,” and more. Graphic designers, too, have begun to explore materiality, by printing their designs on cotton organza or hemp fabric or polyamide fabrics (Cole, 2007: 129) or, as in the case of the Swiss avant-garde art magazine Parkett, by photo­ graphically reproducing letters that were hand embroidered by the designer’s mother to celebrate the authenticity of traditional crafts.

The distinctive features of texture Like shape and colour, texture makes meaning on the basis of provenance as well as experiential meaning potential. Materials may be typically associated with certain objects – marble with imposing temples and luxurious palaces, leather with the accessories of cowboys, motorcyclists and military pilots (saddles, gun holsters, gloves), to give just two examples. When Apple uses faux marble and leather for the cover of laptops, these materials become detached from their provenances and acquire more abstract values – a taste for power and opulence, perhaps, or for tough masculinity. This kind of meaning rests on the cultural histories of these materials. The texture of denim, for instance, was originally associated with the heavy-duty trousers of miners. As a young immigrant in New York, Levi-Strauss was unable to sell a supply of canvas for tents and wagons he had brought with him and decided to put the material to other uses (later replacing canvas with denim). Once taken out of their original (highly functional) context and introduced into the wardrobes of people who were not workers, blue jeans began to acquire identity meaning – an identification with cowboys and pioneers, the frontrunners of the American dream, and, from there, a preference for simplicity and func­ tionality, for the egalitarianism of the “self-made man” over the old class society. As a result, denim could even become chic. In choosing for denim, people may not consciously reflect on such histories. But they will be attracted to them, recognize them as part of “their world,” and so participate in the values of that world. To give another example, cashmere ori­ ginates from Kashmir, a mountainous region in Mongolia, and was first imported to Europe during Britain’s colonial rule in that area. As a labour-intensive material, produced in a distant land, it was exotic, expensive and luxurious. But the meaning of such materials is also, and at the same time, based on embodied ex­ perience. Denim is rough and coarse, cashmere smooth, soft and warm. Both semiotic principles, provenance and experiential meaning potential, are drawn on in the following excerpt from the fashion section of the Australian newspaper The Age (Horton, 2005): Mongolia is a place where the wind cuts you in two. No sane man would venture there if it were not for the lucrative inhabitants – Kashmir goats –

106

Texture

whose winter coats provide the raw material for the finest yarn in the world. It takes five months and 80 people for cashmere to arrive as a garment in a store near you. But it’s worth it. No other substance combines warmth and softness to such perfection that you want to stroke it just for pleasure. A oneply cashmere sweater is two or three times as warm as a wool jumper of similar weight and Mongolian cashmere is the finest. As in the case of shape and colour, mapping the experiential meaning potential of texture involves identifying the distinctive qualities that allow textures to be de­ scribed and compared with each other. Although any given material will always display all these qualities (always being soft or hard or something in between and warm or cold or something in between and smooth or rough or something in between, not to mention various other qualities), to understand their experiential meaning potential, they must be understood separately before we can begin to try and understand how they blend or fuse together, as I will try and do in Chapter 8. In describing the distinctive features of texture, I will draw on collaborative work with Emilia Djonov (cf Djonov and Van Leeuwen, 2011).

Liquidity and viscosity All surface textures have a value on a scale that runs from wet to dry, with various degrees of moistness in between. As water is essential to life, liquidity is a fun­ damental experience and a fundamental quality of living bodies, fresh food and many other things. But in some contexts, high liquidity will evoke negative ex­ periences of rot and decay. Dryness, similarly, can not only connote aging and desiccation, withering and shrivelling but also cleanliness and comfort, as in freshly laundered towels. Liquidity is closely associated with viscosity, the thickness of liquids, and hence with the ability of liquids to flow and stick to surfaces (to some more easily than to others). This can be an important aspect of the functional design of substances. Toothpaste, for instance, needs to flow when squeezed from the tube, but without dripping from the toothbrush. But viscosity can also create identity meaning, for instance in food and beauty products – an emphasis on sensualism and the pleasure of rich, smooth creaminess, or the freshness of “light liquidity.” Edlong, a com­ pany which produces synthetic dairy products, uses “mouthfeel flavours” to “replace the richness and creaminess of dairy commodities, resulting in products with the same deliciousness as the full commodity versions” (Edlong, 2020) and an advertisement for “Nude by Nature Flawless Foundation” stresses the product’s liquidity and hydrating qualities: “It provides a smooth finish with its highly pigmented but light liquid base.” (Australian Cosmopolitan, May 2018: 54). Solid objects may be made to stick with grip glue, often for functional reasons, as on steering wheels and bicycle handlebars, where it assures “grip,” “support,” and hence “safety.” But, as the term “sticky” already suggests, in other contexts, for instanced unclean kitchens or bathrooms, stickiness can repel. In nature,

Texture 107

“carnivorous” plants have sticky leaves to lure and digest insects, something which has been explored in horror films such as The Day of the Triffids (Sekely, 1962) and Venus Fly Trap (Thomson, 1970).

Temperature As already discussed in the chapter on colour, temperature is a rich source of metaphors. “Cool” can be calm and serene, but also cold and lacking in affect. “Warm” can suggest affective engagement and move up the scale to “intimacy” and the “heat” of passion. In a feature on “autumn decorating,” Good Homes magazine (Nov 2002) not only recommended painting interior walls in a burn­ ished orange shade but also using a range of materials – “cherry wood” and “copper accessories” to “keep the glow going throughout your home,” and “a squashy sofa stacked high with sensually soft satin cushions in faded gold” to “snuggle up in.” Surface temperature does not necessarily result from heating or cooling. Under the same room temperature, different substances (say glass or metal as opposed to wood) may have different temperatures, as can be readily felt by the kind of comparative touching Itten recommended to his Bauhaus students.

Relief Surfaces also vary in the degree to which parts of them extend below and above the horizontal plane. In the absence of such extensions, a surface is flat. The greater their number and frequency across space, the more relief the surface has. Relief may be caused by protuberating elements, ridges, pimples, bulges, bumps, em­ bossments, etc., or by indentations, grooves, pits, pockmarks and other depres­ sions. The presence or absence of relief may be functional. The raised ridges of chargrill pans, for instance, facilitate direct, radiant heat and allow meat to be cooked quickly. But culinary aesthetics can also play a role here: chargrilled meat acquires a distinct roast aroma and flavour, with “nice char marks,” as one cooking book has it (McKinnon, 2019: 67). Flat surfaces offer no resistance to the touch, and, depending on the kind of surface, can suggest pristine purity, or polish and sophistication, or impeccable perfection. Relief offers the finger something to explore. Eggers, an interior design company, markets “feelwood” with “softly brushed pores” and “soft hollows” (Eggers, 2010). But as already suggested by qualifications such as “softly brushed” and “soft,” the experiential meaning po­ tential of relief depends on other, co-present features – the smoothness of roughness of the protuberations or indentations, their size, their density, their regularity or irregularity. Irregularly pockmarked surfaces may not please the finger in the way that “softly brushed pores” do. Relief is an important aspect of ceramics and sculpture. Giacometti’s Man Pointing, for instance, has a charred, craggy black surface, so suggesting the suf­ fering of an emaciated and scarred body.

108

Texture

Density Density is determined by the distribution of textural elements in space (and, when touched, in time). Egyptian cotton sheets have high density, with many threads woven tightly together. A muslin wrap has a relatively sparse texture, with fewer threads woven loosely together. The functional advantages of density or sparsity relate closely to their meaning potential. Dense fabrics are strong, solid, compact. Loosely woven fabrics are open, hence “breathable” – “easy,” “breezy,” “light­ weight,” and therefore suitable for summer. But they can also be gauze-like, partially concealing and partially revealing what is behind them, which may create a sense of mystery, or of exposure and vulnerability. Or they can suggest a lack – a lack of strength and solidity, perhaps stemming from poverty or neglect. Other materials, too, can vary in density or sparsity and this may interact with other features. The density of wood, for instance, affects its hardness and strength.

Rigidity Surfaces may resist the pressure of touch to various degrees: the more resistant the surface, the harder the texture. Rigidity is a rich source of experiential me­ taphor. Softness may be weak and submissive or sensitive and accommodating, for instance; hardness strong, stable and durable, or unforgiving and harsh. The way rigidity is valued varies historically and culturally. The 1960s sought to revalue softness. Hippies wore soft clothes, and long, soft flowing hair. Designers in­ troduced soft chairs. Claes Oldenburgh began to make his soft sculptures, to great acclaim. But it did not last: to be a “softie” or “a soft touch” is still no praise, and much of today’s fashion is harder – short, bristly hair, denim and leather, hard muscles, steel and glass, more aggressive “heavy metal” styles of popular music.

Roughness Rough textures are abrasive to the touch and can even hurt the finger if too much pressure is applied. Roughness is the texture of sandpaper, rough plaster walls, the bark of some trees, unshaven cheeks, coarse burlap coats. Smooth, by contrast, is the texture of unblemished young skin, polished surfaces, immaculate tuxedos. The two terms are widely used to describe texture, sometimes referring to the quality of “unevenness” (which I here refer to as “irregularity”) or to “relief” generally, rather than to roughness as I define it here. But roughness necessarily combines with relief, and often with irregularity. As I will discuss in more detail in Chapter 8, roughness is also a multimodal concept. It applies, for instance, to the sound of the voice. We know what makes our voice hoarse or harsh – a temporary or permanent change in the very texture of the walls of our throat cavity, whether because of a cold, or because of the wear and tear of heavy smoking and drinking. In some contexts, this is positively valued, for instance as a sign of masculine toughness or of having lived a hard life. In Afro-American

Texture 109

singing, hoarse voices are highly appreciated – think of the voice of Louis Armstrong. And as mentioned earlier, Starbucks values “grainy, rough and uneven” materials as signs of authenticity. In other contexts, smoothness is more highly valued, for example in certain foods. Edlong’s synthetic dairy products seek to capture “not only […] the authentic taste of dairy, but its smooth, indulgent and creamy texture” (Edlong, 2020). A company called “Smooth Fashion” advertises its “luscious silks and linens” and in advertisements for women’s beauty products, skin has to be “beautifully soft and smooth,” and “smooth, radiant and youthful” (Gumtree, 2020).

Regularity Regularity and irregularity have already been discussed in the chapter on shape. But irregularity can be a feature of surface texture as well. Experientially, it resonates with our expectation of surfaces. Which surfaces do we expect to have regular, predictable and repetitive tactile patterns of protuberations and indentations? Which irregular patterns, varying in size and contours and unevenly distributed across the surface? Nature is for the most part fundamentally irregular, hence irregularity can be ap­ preciated as “natural,” “wild,” or “untamed,” whether in Romantic or en­ vironmentalist terms – the gnarled bark of mature trees, the weathered surface of rocks, the intricate textures of moss and lichens. But in other contexts, we may positively value the taming of nature – trees planted in neat rows, symmetrically designed gardens with neatly trimmed trees and hedges. Romantic nostalgia for the past may make us value the weathered ruins of ancient buildings, the rusty remains of the industrial age, crumbled walls overgrown with ivy. Elsewhere we expect and value regularity, as in the facades of classic buildings, or the mosaics of tiled floors. Such expectations may be deliberately subverted, whether out of playfulness, capri­ ciousness or rebelliousness. The Austrian artist Friedensreich Hundertwasser, on a crusade against the “straight line,” created an apartment building in Vienna on which the exteriors of the different apartments not only had different colours but also dif­ ferent textures, using recycled bricks as well as remnants of tombstones, and con­ trasting regularly aligned stone blocks with irregularly applied plaster (Figure 6.2). In art, Willem De Kooning was admired for the irregular textures resulting from his unique use of unconventional tools such as knives and scrapers, and unconventional materials – he would, for example, mix granular materials into commercial paints to get a roughly textured finish, and use smooth fibre boards for a smooth finish and rough fibre boards for a rougher finish (Lake, 2010).

Consistency Textures may be consistent across a surface or heterogeneous, with some areas drier, stickier, colder, flatter, denser or softer than others. Such qualities may vary within a single material, as with the bronze of Giacometti’s Man Pointing, or of Miró’s Little Girl with Honeycomb Hat (1967), in which the girl’s body is a kind of knobbly cone, from which a long, thin neck leads to a flat, half-moon-shaped head

110

Texture

FIGURE 6.2

The Hundertwasserhaus. iStock

with regularly spaced holes in it and a hat with a regularly shaped honeycomb pattern and sharp, ragged edges. Other sculptures are no longer made of a single material but combine different materials. Jean Tinguely co-created works with his partner Niki de Saint Phalle (The Fantastic Paradise, 1966–67) in which the forklike tentacles of his black metal mechanical creatures caressed the Saint Phalle’s colourful and amply proportioned polyester women. In later works he used, not only metal machine parts but also plastic toys, feathers, pieces of fabric and animal skulls. His Self Portrait (1988), for instance, was a kind of life-size ragdoll with an animal skull, dressed in clothes which he had recently worn at a carnival (see Figure 6.3). This figure was then chained to a large heavy wheel that pulled it now in one direction, now in another. Thus he represented himself as a somewhat macabre clown unable to control the forces that pulled him in different directions. In another sculptural self-portrait the American artist Jimmie Durham used canvas, cedar, synthetic hair, scrap fur, dyed chicken feathers, human rib bones, sheep bones, seashell and thread to represent himself. Such sculptures, he said, “bring materials together to create something new” (Durham, 2017): The starting point of my work is almost always material. That can be an object, bought, found, or given by somebody; material by itself such as wood, glass, stone, plastic; or something else, even just a word. I am interested in it because of its specificity and specific materiality, I enjoy playing with materials, bringing them together so that they create something new.

Texture 111

Self-portrait (Jean Tinguely, 1988). Jean Tinguely/ADAGP. Copyright Agency 2020

FIGURE 6.3

Jewellery regularly combines different materials. Naum Slutzky (1894–1968), who for some time taught jewellery at the Bauhaus, would set pebbles in ornate golden settings, or combine wood with silver and ivory to celebrate the beauty of simple non-precious things. Finally, surface structures may have qualities that stem from changes they undergo over time. Metals rust, papers yellow, pines get a grey sheen as they age and weather. This changes the meaning of these materials and blurs the distinction between provenance and experiential meaning potential. It can be positively va­ lued – rust as the patina of history – or negatively, as a sign of neglect and disrepair. Björkvall (2009) studied functional and identity design in IKEA tables, com­ bining descriptions of the tables’ design and style with observations of their use and focus group discussions with IKEA customers from different cultural backgrounds.

112

Texture

While their functional design allows tables to be more or less suitable for work, or for gathering people around them or placing objects on them, their identity design creates meaning potentials which, in a multicultural society like Australia, may be taken up differently by people coming from different backgrounds. A key dis­ tinction was the division between “modern” and “country” styles. Modern tables were light, dark brown or black wooden tables, glass tables and white tables. Country tables were made of solid wood, medium brown in colour, often antique-stained (ibid: 250). Australian buyers preferred the latter. A 24-year-old Australian student who bought a wooden Björkudden table said she would never consider buying a dining table that was not made of solid wood. Her choice of the Björkudden table resulted from a compromise with her flatmate, who wanted a more modern table. She herself would have preferred an antique-stained wooden table. That colour, and the solidity of the wood, signified “family values” to her, something “genuine” and “sturdy.” But customers from non-Australian back­ grounds valued wooden tables differently. A buyer who had just moved to Australia from Indonesia found them “too common” and “boring,” preferring tables that are more obviously industrially produced. He chose a Grimle table, which has a white melamine plastic-covered top and steel legs. Two Swedish focus group participants found the Grimle table a good-looking “formal” table, but Australian focus group participants thought its steel legs not solid enough to re­ present “formality.” People from an Asian background liked tables with glass tops, because they were “tidy” and “clean” and evoked “luxury.” Different cultural values found their expression in the same materials, and similar values could be expressed by different materials (Figure 6.4).

Björkudden table (left) and Grimle table (right). Reproduced with kind permission from Anders Björkvall

FIGURE 6.4

Texture 113

Meanings of this kind nevertheless stem directly from the unique qualities of specific materials, from materials as “medium,” in the sense in which that term was defined in chapter 3. But, as briefly discussed in the introduction of this chapter, media can transform into modes, into abstract systems that can be realized with different materials. The distinctive features I have described in this chapter are in fact such a system, as they can be applied to the description of different materials, which, however, are still characterized by unique config­ urations of distinctive features. Plastic, on the other hand, is an abstract system made concrete, a mode rather than a medium. It originally developed as a surrogate for luxury materials which were in short supply, such as ivory, shellac and mother-of-pearl. As such it still related to specific materials, even though it could only represent their visually perceivable qualities. But soon plastic would become a mode, a material able to represent almost all configurations of material qualities without having any of its own. Plastic can be made available as sheets fibres, foams, pellets, and more, and almost every kind of plastic can be made hard or soft, heavy or light, thin or thick, red or green, and so on, even though some tend towards some qualities more than others. In short, plastic is a “language,” a system to signify material qualities, a material that “adapts itself to the syntax of the design in the same way that the words of a language adapt themselves to the syntax of a text” (Manzini, 1990: 138). In the words of Baudrillard (1996: 112) The manufacture of synthetics signifies for materials a stepping back from their natural symbolism towards a polymorphism, towards a superior level of abstraction which enables a game of universal associations to take place. As such plastic has been both admired and dismissed. In the 1930s, designer Paul Frankl thought plastic was the dream of alchemists come true: “it transmutes base materials into marvels of beauty” (quoted in Meikl, 1990: 44). Roland Barthes (1972: 54–5 and 97–8) was in two minds. On the one hand he called it “a mir­ aculous substance” and “a spectacle to be deciphered.” On the other hand, he called it “graceless” and “destroying all the pleasure, the sweetness, the humanity of touch,” while its colours were “mere names,” “able to display only concepts of colour.” To give some examples of the use of plastic in identity design, squeezing the heads of two plastic dolls in the same price class, Sindy and Action Man, shows that Action Man’s head is harder than that of Sindy (which is relatively soft and can be squeezed) and Action Man’s synthetic hair is rougher to the touch than Sindy’s. Plastic here articulates gender meanings: men are hard and rough; women soft and smooth. And these meanings are perceived, not visually, but haptically, with the hands, as the dolls are posed and played with. In an earlier publication (Kress and Van Leeuwen, 2001), we compared two plastic tooth­ brush containers:

114

Texture

toothbrush container 1

toothbrush container 2

flesh-coloured plain slightly translucent shiny comparatively light slightly bendable smooth

white plain slightly translucent shiny comparatively heavy rigid smooth with ribbed grooving

The colours of these two objects can be interpreted on the basis of provenance – the one has the colour of flesh, the other the white of sterility and hygiene. But their tactile qualities have experiential meaning potential – the one is light, flexible and smooth, speaking of a pleasurable pampering of the body beautiful, the other is heavier and sturdier, speaking of practicality and durability. Even such simple ev­ eryday objects can use texture to make identity meanings. As Italian historian of technology Manzini (1990: 133) has said: “Plastics have exploited their formidable qualities not so much in the direction of fulfilling technical and constructional needs as in the expression of different images.” Figure 6.5 summarizes the distinctive features of texture – the curly brackets again indicate simultaneous choices, the square brackets “either-or” choices. Where they come with double-headed arrows, however, they are graded (“more or less”) rather than binary choices.

A note on the visual representation of texture The visual representation of texture has a long history going back to the rich still lifes of 17th-century Dutch painters who discovered how the play of light on surfaces can represent the textures of glass and metal, or fruit and pastry, in “a new and concrete knowledge of our common world” (Alpers, 1983: 91). It continues in today’s glossy photographs which can make food, fashion, or skin and hair more beautiful and luscious than they are in real life. Texture is also a visual and an “imaginary” ex­ perience. As expressed in the website of an architectural firm (Wolf Architects, 2020): Anyone who’s ever watched an interior design TV program or flipped through the pages of a magazine can tell you that you don’t need to have physical contact with a room to feel the power of texture. When the viewer imagines how these surfaces feel to the touch it creates an emotional connection. But only some tactile qualities can be rendered visually, primary those which throw shadows – relief, density and roughness, together with their (in)con­ sistencies and (ir)regularities. Line and colour may also play a role. Relief can be achieved with curved or jagged lines and variations in colour value; density with

Texture 115

wet liquidity dry adhesive viscosity non-adhesive hot temperature cold relief relief flat dense density sparse hard rigid soft rough roughness smooth regular regularity irregular homogeneous consistency heterogeneous FIGURE 6.5

The distinctive features of texture

116

Texture

the distribution of lines and colours across a surface or with variations in colour saturation; roughness or smoothness with abrupt or more subtle gradations in colour value, as indicated in textbooks for art and design courses: “An implied rough textured surface should impart the feeling that the surface is all the same colour but that the texture is in some way irregular. The greater the contrast between the values used, the rougher the surface will appear” (Feisner, 2006: 89) and “a smooth, shiny surface is usually rendered in a cool temperature, while warm colour usually imparts a smooth, napped texture such as velvet” (ibid). Even when visually represented, however, the meaning potential of such qualities will be based on embodied experience rather than on the provenance of specific objects. Other qualities – liquidity, viscosity, rigidity and in some cases temperature – can only be depicted in relation to objects they are qualities of. Dryness, for in­ stance, can be suggested by images of cracked soil or old wood, wetness by droplets on a bunch of grapes or sparkling water in a glass, viscosity by sticky subjects such as honey or resin exuding from a tree, the softness of velvet by images of velvet dresses, or theatre curtains. A visually represented ice-cream will evoke coldness, regardless of its colours. Here meaning does not only stem from the material qualities represented but also from the objects that carry them, blurring the boundary between provenance and experiential meaning potential. To complicate matters, some visual textures do not have a tactile equivalent. These include textural patterns deriving from the interaction of an object with its environment such as transparency and luminosity as well as non-relief abstract patterns intrinsic to a given material, for instance the veining of marble or the grain of well-sanded wood. These can be natural, as in the case of marble, or artificially produced, for instance abstract patterns carved in pottery, which may be inspired by natural and usually irregular patterns (striated, mottled, veined, grained, dotted, and so on) or regular and geometric. Transparency can be achieved by varying the distribution and value of colours and may or may not represent tactile qualities such as liquidity or density. Luminosity can be visually rendered by manipulating the colour variables of hue, saturation and value. Meaning potentials may draw on the physical actions required to produce the patterns, or on the provenance of patterns in, for instance, Islamic or Oriental art. In sum, while some textural qualities are applicable across media, others may be purely tactile or purely visual.

7 MOVEMENT

Movement and mobility as semiotic resources From the beginning of the 20th century, movement has been actively developed as a semiotic resource in its own right. It first manifested itself in the work of avantgarde artists but was soon picked up, first in the mass media, and, today, in video and animation software available, with different degrees of sophistication, to any computer user. In his Technical Manifesto of Futurist Sculpture of 1912, the sculptor and writer Umberto Boccioni described machine movement as a potential new form of ar­ tistic expression (quoted in Kepes, 1965: 810): We cannot forget that the tick-tock and the moving hands of the clock, the in-and-out of a piston in a cylinder, the opening and closing of two cogwheels with the continual appearance and disappearance of their square cogs, the fury of a flywheel or the turbine of a propeller, are all plastic and pictorial elements of which a Futurist work in sculpture should take account. In 1920, the brothers Naum Gabo and Antoine Pevsner, in their Realist Manifesto, introduced the term “kinetic art” (Gabo & Pevsner, quoted in Kepes, 1965: 81) We renounce the thousand-year-old delusion in art that held that static rhythms are the only elements of the plastic and pictorial arts. We affirm in these arts a new element, the kinetic rhythms, as the basic form of our perception of real time. In the same period, Duchamp mounted a bicycle wheel on a stool and called it Mobile, Meyerhold developed his biomechanics, a method of acting based on

118

Movement

movement and contrasting with Stanislavski’s focus on “inner” inspiration, and avantgarde artists everywhere embraced the moving image, playing a key role in developing its semiotic potential. Entirely new this was not, of course. For several hundreds of years, people had been fascinated with movement, using clockworks to create walking dolls, animated chess players, dolls playing a miniature piano or writing a few words, and more, and developing the forerunners of the cinema, from flip charts to the more complex “kinetoscopes,” “praxinoscopes,” etc., which were popular throughout the 19th century, as toys and as tools for studies of movement by Muybridge and others. But it was only in the 20th century, the age of the motor car and the movies, that movement integrated, not only with art but also with the design of everyday objects, and even of texts, where the screen, the medium of the moving picture, began to merge with the page, the medium of the static word. Historian of typography Beatrice Warde recounted her amazement at seeing an animation by Norman McLaren projected onto the gigantic Animated Electric Screen in Times Square, New York (quoted in Bellantoni and Woolman, 2000: 5) I saw two Egyptian A’s walking off arm in arm with the unmistakable swagger of a music-hall comedy team. I saw base serifs pulled together as if by ballet shoes, so that the letters tripped off literally sur les pointes. I saw words change their mind about how they should look even more swiftly than a woman before her milliner’s mirror. After forty centuries of the necessarily static alphabet, I saw what its members could do in the fourth dimension of time, ‘flux’, movement. Today, “kinetic typography,” first developed in animation films by Len Lye and Norman McLaren and now accessible to every computer user, is hailed as a new form of writing, able to convey “a speaker’s tone of voice as well as qualities of character, and affective (emotional) qualities of texts” (Forlizzi et al., 2003: 377). Lewis Mumford (1939) has described the long incubation periods of inventions which eventually changed people’s lives as periods of “cultural preparation” in which they already existed as ideas that powerfully attracted philosophers, scientists and artists, as apparently trivial yet very popular playthings, and as tools for rela­ tively marginal specialist endeavours. These “cultural preparations,” and the crucial inventions to which they eventually led, were driven by broader social and cultural quests, in this case the pursuit of dynamization, of ever greater mobility and ever greater speed, which continues to be a major source of further social and cultural change. In this chapter, I will discuss two aspects of movement design, starting with styles of movement in everyday and artistic performances, and in films, videos, and computer animations, and then moving to mobility, that is, to the design principles underpinning the resources for creating meaning with movement which, today, are built into many everyday objects and even into ubiquitous text-production software such as PowerPoint. Below I will briefly introduce each.

Movement

119

The meaning of movement The approach to the semiotics of movement I introduce here can be applied to live action (recorded or otherwise) as well as to animated films and videos. The main differences between live action film or video and animation are, first, that in animation movements are not recorded but created by artists or software designers, second, that animation can make inert objects come to life, and third, that ani­ mation not only represents actions and events but also dynamizes identities and attributes, making objects or letter forms change colour or shape in front of our eyes, or, for instance, morphing letter forms into images or images into letter forms. Live film and video can do this only to a limited extent, for instance through time lapse photography. My approach is again based on the semiosic principles of provenance and experiential meaning potential. When represented movements are naturalistic, whether in live action or animation, we readily recognize that they represent, for instance, walking or running or swimming. But styles of human movements (and, in animation, also the movement of things) can convey identity meanings. There are many ways of walking, for instance – strolling, striding, strutting, lumbering, shuffling, to name just a few. Some of these have recognizable provenances, military marching steps for instance, or the characteristic walks of star performers, such as Mick Jagger’s strut or Michael Jackson’s “moonwalk,” others do not. But even movements with specific provenances can also be understood on the basis of embodied experience. Our body knows how much energy they would require and with what situations, activities or moods they can be associated. This is so even in the case of the movement of objects we have never seen. As I write this chapter, the corona virus pandemic dominates the news, and the cyclorama style back­ grounds of many TV news studios show large corona viruses, sometimes still, sometimes moving. In the news of the Dutch national broadcaster NOS, large corona viruses float in a blue space, seemingly weightless and slowly moving, mostly towards the right and slightly upwards. Multiple iterations of a single word, indicating the topic of the item (e.g. “education” or “the economy”) float along in the same space and at the same speed, but in a straight line and in both directions. Direction of movement is here a key source of meaning – the corona viruses slowly and steadily progress, and the trend is upwards. In the news of the Australian ABC, the viruses are large and pink and slowly and menacingly hover in place, while little bubbles float upwards. Here we are kept in suspense as to what the viruses will do, whether they will move, and if so, where to. Scholars of dance discuss the relation between everyday movements and the seemingly abstract movements of dance. Sheets-Johnstone (2013), for instance, sees movement in everyday life as symptomatically expressive of feeling, and movement in dance as symbolically expressive. “Linear design” and “linear pattern” are then among the qualities of movement that carry symbolic meanings (and the term “qualities” corresponds to what, in this book, I call “distinctive features”). As Joshua Han explains (Han, 2021), in a dissertation about the relation

120

Movement

between movement and music: “Linear design refers to the line created by the body as a whole or by any individual body parts alone or in combination, e.g. twisted, angular, diagonal, vertical, etc. Linear pattern refers to the linear paths created by the body as the result of the direction of movement, diagonal, zig-zag, circular.” There are parallels to be explored with the movement of melodies in music, where “direction of movement” is also a key feature. Tagg (2000: 5) describes and exemplifies ten common “contours” – rising, falling, “tumbling strain,” V-shaped, centric, terraced (falling), terraced (rising), oscillatory, arched and wavy. All of these could easily be translated into body movements or the movements of animated objects. Tagg also notes that some melodic contours make meaning on the basis of provenance, for instance the semiquaver triplets of Irish traditional music, or certain “onbeat figures” in “popular notions of Hispanic music.” He describes these as formal features, but, as the corona virus example shows, and as we will see in more detail in the next section, they all carry meaning potentials. To give a further example, in the well-known Intel “vector logo,” a likeness of which is reproduced in Figure 7.1, a beam of flashing white light makes a swooshing orbit and then disappears into the distance of a dark blue background which dissolves into the logo itself – the word “Intel,” surrounded by the light beam, now in blue against a white background, and morphed into two static curves which increase in size and thickness towards the front. At the point where they almost meet, the point where a spark might jump across them, the words “leap ahead” appear, letter by letter. Meanwhile the soundtrack accompanies the initial light flash with an “audio sparkle” which first becomes louder, then softer again, as if describing an orbit and, as the words “leap ahead” appear, we hear a four-note melody that spells “Intel Inside.” The logo thus creates a dynamic, energetic identity, entering with a burst of energy, then “leaping ahead.” The short melody blends electronic sounds with a xylophone, a marimba and a

FIGURE 7.1

A likeness of the Intel logo

Movement

121

“secret recipe of instruments” (Jackson, 2003: 128), so combining the sounds of technological perfection with sounds that seek to evoke affective resonance, in order to create “a logo with emotional content, in the same way a film score works” (ibid).

The meaning of mobility The meaning potential of mobility derives from resources for meaning making that are built into the design of objects, whether these are everyday objects or works of art, and whether they are physical objects or computer software. In earlier work (Van Leeuwen and Caldas-Coulthard, 2004), Caldas-Coulthard and I compared a Sindy doll and an Action Man in the same price class (see Figure 7.2) and found some striking differences, not only in their use of plastic, as already discussed in the previous chapter, but also in their kinetic design.

FIGURE 7.2

Sindy and Action Man

122

Movement

Action Man’s head could only move sideways, and not up and down. Sindy’s head, on the other hand, could move freely in every direction, allowing “head cants,” the coy poses with the head cocked sideways which Goffman had com­ mented on in his study of gendered poses in advertisements (1979: 47) and which, 40 years later, can still be seen in many fashion photographs. Again, Action Man’s legs could spread, allowing him to sit the way many men sit, with the legs spread wide, but Sindy’s legs could not. She could however bend her knees (in fact she had three degrees of knee bend) so that she could adopt the “body cants” and “bashful knee bends” also described by Goffman (ibid: 45–46). This Action Man could not do. In addition, Action Man could be made to stand on his feet while Sindy could not, and Action Man’s hands could hold objects while Sindy’s could not. In short, kinetic design expressed the gender identities Goffman had so ably analysed : in contrast to men, women are kinetically represented as literally and figuratively unable to support themselves, which can then “be read as an accep­ tance of subordination, and an expression of ingratiation, submissiveness and appeasement” (ibid: 46). Men, meanwhile, are kinetically represented as oriented towards action, seeking to occupy the maximum amount of space, and standing erect, in no need of support. These meanings are never made explicit but com­ municated as children (or adults) actively explore the movements which the dolls’ kinetic design affords. The kinetic design of everyday objects also involves identity design. The Tizio light designed by Richard Sappor (see Figure 7.3) can move only vertically, only tilt up and down. It is designed to illuminate a single, small object, the book read in bed for instance, and to assist single-minded concentration. The now ubiquitous Anglepoise light (see Figure 7.3) designed by George Cawardine in 1932 is still a

FIGURE 7.3

Tizio light (left) and Anglepoise light (right)

Movement

123

desk light – “Focus brings learnings, experience and knowledge,” the “Anglepoise manifesto” declares (Anglepoise, 2020). But its spring-loaded arm and its shade can move in any direction, so bringing different objects and activities in focus, and directing the shade upwards can cause light to be reflected from the ceiling so as to illuminate a larger area. Although the design of these lights is functionally motivated, they may not be used for the functions they have been designed for. They may just signify “single-mindedness of concentration” or “multiplicity and versatility.” The same applies to the use of spotlights on rails fixed to the ceiling in domestic settings. In film and video studios such lighting is functional, as lighting set ups must be changed frequently. In home settings their kinetic capacity may be underused or not used at all. It then becomes symbolic, either on the basis of provenance, signifying some kind of allegiance to, or alignment with, the world of film and television, and/or on the basis of experiential metaphor, signifying a taste for change, flexibility, versatility. In short, while kinetic design is usually functionally motivated, whether for the purpose of recognizably re­ presenting specific movements or effectively facilitating specific activities, it also leaves room for style, and hence for identity design. In addition, the activities represented or facilitated may themselves be associated with lifestyle identities.

The distinctive features of movement In this section, I discuss the representation of movement in multimodal texts of various kinds, drawing first on pioneering work by Gisela Leão (2012), who studied movement as representation and then on equally pioneering work by Joshua Han (2021), who studied movement style. Movements in texts are of course always carried by the elements that move, whether these are images, words, or abstract shapes (or parts of images, words or abstract shapes). They must be the movements of something, or someone. To exemplify the distinctive features of movement I will use a short video issued by the Australian Government, (https://www.health.gov. au/resources/coronavirus-video-help-stop-the-spread), and transcribed below – in this video corona viruses are the primary moving elements. Figure 7.4 gives an impression of the style of the video. Note that the transitions between the shots are not cuts or dissolves – people and things appear and disappear as if being drawn or erased in front of our eyes by an invisible hand. 1. CLOSE SHOT of boy coughing. Pink viruses escape from his mouth, growing, and coalescing into a large cloud which eventually fills the whole screen. VOICE OVER Viruses spread when you cough and sneeze. The cloud moves up, releasing small pink droplets which fall on a table. A hand moves into the shot and touches the table, which now is covered in pink dots.

124

Movement

A still from Help stop the spread (Australian Government video, 2020). Reproduced by kind permission from the Department of Health, Commonwealth of Australia

FIGURE 7.4

VOICE OVER And the tiny droplets land on surfaces others may touch. As the hand moves left, out of shot, the drawing of the table is erased. 2. LONG SHOT. A drawing appears of a group of five people, one of them holding hands with a young child. Pink viruses fly in and hover around the people. VOICE OVER You can help reduce… The viruses turn into white crystals as the drawing is erased. 3. MEDIUM CLOSE SHOT. The drawing of a boy appears. He coughs up purple viruses, this time in his arm. VOICE OVER …the risk by coughing or sneezing into your arm… The drawing is erased 4. MEDIUM CLOSE SHOT. The drawing of a girl appears. She coughs into a white tissue which then colours pink.

Movement

125

VOICE OVER …or a tissue. The drawing of the girl is erased 5. FULL SHOT of the drawing of a bin, with purple tissue falling in, dropped by an invisible hand. VOICE OVER Bin the tissue. The drawing of the bin is erased. 6. FULL SHOT. The drawing of a tap appears. Camera tilts down with the water that comes out of the tap, to end up in shot of hands washing, with pink viruses disappearing as a result. VOICE OVER Wash your hands with soap and water. 7. MEDIUM CLOSE SHOT. The drawing of a boy with a thermometer in his mouth appears. VOICE OVER And if… 8. FULL SHOT thermometer. Camera zooms in. A dialogue balloon appears at end of thermometer (“37.5+”) VOICE OVER …you’re sick… 9. CLOSE SHOT. A hand holding a mobile phone, moves up, then down again. VOICE OVER …seek medical advice. Together… 10. LONG SHOT (AS SHOT 2) The drawing of group of five people appears, viruses and white spots hovering around them. VOICE OVER …we can help stop the spread… The drawing is erased

126

Movement

11. CLOSE SHOT. The drawing of six hands making ‘thumbs up’ gesture ap­ pears. As the drawing appears, the viruses from the previous shot change into pink ‘plus’ signs, the dots into smaller, white ‘plus’ signs VOICE OVER …and stay healthy. 12. TITLE SHOT reading “Visit Health Gov Au”, with the Australian Government’s coat of arms in the centre and some pink viruses at the bottom of the shot. A hand moves in, wiping out the viruses with a cloth. To represent people or things (including abstract things and letterforms) as engaged in action, they must be displaced, i.e. change their position on the page or screen, as in the Dutch news programme where corona viruses slowly move to the right and upwards. Such movements can also be “in place” (He and Van Leeuwen, 2019), as in the Australian news programme, where the viruses mostly hover in place. Such “in place” movements are increasingly used in the minimal, looped animations of still images and words that can now be so easily produced with various softwares. Actions can be represented as transactional, in which case they are shown to somehow affect another element, or as non-transactional, in which case they do not affect another element. In shot 1 of the above transcript, the droplets affect the table (they “infect” the table, or rather, they make it into a source of infection) and the hand of the boy affects the table by touching it. The movements in shot 9 are non-transactional. Here the boy’s gesture is not directed at anyone or anything. The fact that he is contacting a doctor is only conveyed verbally, by the word “Doctor” on the mobile phone and the words “medical advice” in the voice over. The distinctions between displacement and movement-in-place, and between transactional and non-transactional, are essentially functional distinctions, systemic distinctions which fulfil more or less the same representational functions as the system of transitivity in language and the systems of narrative and conceptual visual representation described by Kress and Van Leeuwen (2006). Identity design, on the other hand, relies on the how of these basic movements, on their style, and hence on the gradable distinctive features necessarily present in any movement. Drawing specifically on the work of Han (2021), the most important of these are direction, directedness, expansiveness, velocity, force, angularity, fluidity and regularity.

Direction Directionality involves the horizontal, left-right dimension as well as the vertical, up-down dimension. Our experience tells us that moving upwards involves effort and energy, while moving downwards allows a decrease in effort, a relaxation. This is again a matter of degree, with, in between the two extremes, horizontality,

Movement

127

flatness. In context this meaning potential is activated in specific ways. In the Dutch news programme, for instance, many of the corona viruses move up, and in shot 1 of the transcript they also rise, becoming a menacing cloud. In both cases the pandemic is, literally and figuratively, “on the rise.” The movement in shot 10 is also upwards, showing the boy as actively taking control of his health. As Lakoff and Johnson (1982) have shown, “up” and “down” are key sources of metaphors, and the same applies to their dynamic counterparts. “Rising” can be “aspiring,” “making one’s way up,” “climbing the ladder,” for instance; “going down” can be “lowering oneself” or “sinking” – and these are only some of the many possibilities. Horizontal movements on pages or screens relate to the direction of writing. In spoken language, sentences (and longer stretches of speech) begin with the “Given,” with something that has already been mentioned or is assumed to be known by the listener and then move to the “New,” the information the speaker wants to impart (Halliday, 1985: 277). When we move from speech to writing, “before” and “after” become “left” and “right,” at least in Western writing. This is so also in musical phrases and in the performance units of acting (Van Leeuwen, 2005: 202). Moving from left to right can therefore suggest some form of goal-directedness or futureorientation, moving from right to left its opposite.

Directedness Displacement necessarily occurs in specific directions, but not all displacements take the shortest route. Movements may turn and twist, zigzag, move stepwise, and more. Embodied experience can tell us what kinds of things can cause in­ direction. We may zigzag to avoid obstacles, for instance. But indirection may also be symbolic – and aesthetically pleasing. Like the trills, mordents and turns of baroque music, flourishes in dancing intersperse displacement with “movementsin-place.” In the minuet, as described by Sachs (1937: 407), dancers moved “with little dainty steps and glides, to the right and to the left, forward and backward, in quarter turns, approaching and retreating hand in hand, searching and evading, now side by side, now facing, now gliding past one another.” Such movements were aesthetically pleasing as well as meaningful: the minuet was a couple dance, enacting a stylized courtship ritual, but the couples acknowledged the other dancers and the spectators in the room with little bows in their direction. This would no longer happen in the waltz, the minuet’s 19th-century successor, in which couples only had eye only for each other, as if there were no other dancers in the ballroom at all – the 19th century was a time of increasing separation between private and public life.

Expansiveness The same kinds of action can extend over a larger or smaller amount of space. We can walk with large strides or measured steps, jump with energetic leaps or skip

128

Movement

from one foot to the other, wave our arms around or restrain our gestures. And such degrees of expansiveness can be associated with different kinds of activity and different moods and emotions. A comparison with music can again be made. As described by Cooke (1959: 109) Medieval and early Renaissance music tend to move in stepwise progression at a normal medium pitch, befitting man’s humble subjection to the deity, but with the growth of human self-realization, music drama […] began to introduce more and more liberty of pitch movement to express the rhetoric of human passion In this example, expansiveness characterizes the style of an era but it can also characterize the movement style of an individual, a social group or a nation, and both expansiveness and constraint can have positive as well as negative overtones. Exuberance may be seen as domineering or as impressive, for instance, constraint as showing admirable moderation or as shy and timid.

Velocity The meaning potential of velocity derives from our physical knowledge of what slows us down and what speeds us up (age or fatigue, for instance) and our cultural knowledge of occasions which require slow movement – solemn processions, for instance, or funerals. But slowness can also be pleasurable and relaxing – leisurely strolling through a park, for instance, or taking time over a job that needs care and precision. Members of the “Slow Movement” see slowness as a “life affirming” way of “connecting to spirituality” (Slow Living, 2020). In the corona virus video, the slow movement of the viruses becomes menacing and suspenseful, as it delays the feared impact of the viruses. Speed, on the other hand, suggests energy, as in the animated Intel Logo (Figure 7.1). It is highly valued in many circumstances, but too much speed can overwhelm and confuse, making it impossible to keep up with things.

Force Movements with the same directionality, expansiveness and/or speed may have dif­ ferent degrees of force. Like loudness in sound, force can suggest vigour or power as well as anger. The opening of the Intel Logo (Figure 7.1) has force, as do the stamping rituals of military drills, or forceful blows in a fight. But force can also express a positive intensity of belief and commitment. Softness, too, can have positive or negative overtones – it can be weak and timorous, or gentle and tender, for instance.

Angularity and fluidity Like graphic shapes, movements can be angular or curved, regardless of whether they are, for instance, movements of the whole of a body or parts of it. As

Movement

129

discussed in Chapter 4, our experience of our natural and cultural environment tells us that curved forms and curved movements dominate the natural world and rectilinear forms and movements the world created by humans. The curved trajectory of the flash of light in the Intel Logo (Figure 7.1) therefore also suggests a natural movement, akin to the orbits of planets and moons. Closely related is the issue of fluidity, the contrast between “connection” and “disconnection,” between long, unbroken, smooth movements and movements that consist of distinct short steps. Like staccato in music, disconnection can be lively, energetic and determined or disjoined and mechanical, with movement becoming a succession of still frames, as in the experiments of Muybridge. And like legato in music, connection can not only be smooth and sensual but also imprecise, slurring things together. It all depends on the context. Some well-known ballets of the late 19th century and early 20th century, such as Coppélia (1870) and Petrouchka (1911) revived the age-old story of Pygmalion, in which a sculptor’s creation comes to life (Austin, 2016). Such ballets contrast the angular, mechanical movements of automata with more fluid and natural human movement. Today, machine-like forms of dance and music have entered popular culture, e.g. the “Dancing Machine,” which Michael Jackson performed with the Jackson Five in 1973, and the jerky, mechanical movements of the “robot dancing” of the 1980s.

Regularity Movements can be regular or irregular, rhythmically organized and periodically patterned, or meandering, wavering, teetering, oscillating irregularly. The meaning potential of irregular movement stems from the same experiences as the meaning potential of irregular shapes – physical conditions such as intoxication or infirmity, uncertainty and confusion, or a refusal of control and discipline. But irregularity may also celebrate human spontaneity and unpredictability, for in­ stance in forms of dancing and music that contrast the regularity of the mechanical (or, today, the electronic) with a human touch. In such cases, regular movements may be seen as repetitive and mechanical, but in other cases they may be considered well-proportioned and elegant. On the ABC news, the movements of the quivering corona virus are irregular, and in shot 2 of the corona virus video, the viruses hover irregularly around the depicted people. This makes the viruses unpredictable and hence dangerous – we do not know where they may move next. Complex movements of this kind could of course also be interpreted with reference to creatures or objects that we know to move this way, for instance mosquitoes. In a 2003 New York exhibition, David Byrne, who is not only a musician but also an author and artist, used PowerPoint as an art medium in a number of videos (cf. Van Leeuwen et al., 2013). In one these, “Architectures of Comparison,” arrows wander slowly and aimlessly across the screen, accompanied by the sound of a dreamy waltz (Byrne’s interpretation of the love duet of Un di Felice, from Verdi’s La Traviata). Arrows, in PowerPoint, function to point at important

130

Movement

content elements or make meaningful connections between content elements. But here the arrows meander without a sense of direction and without affecting anything. As Byrne has commented: “Goal-oriented behaviour is like sleepwalking. It is easy and purposeful, but what is its purpose? Its purpose is itself” (Byrne, 2003: 7). Figure 7.5 summarizes the distinctive features of movement styles. They all play a role in every instance of movement, and they are all graded features.

Change of identity and attributes So far, I have discussed the distinctive features of movements that realize actions and events. But animation can also dynamize the attributes and identities of the elements seen on the page or screen. Elements, be they images, texts (or parts of images or texts) or abstract graphic shapes, can, in front of our eyes, change shape, colour, or texture (at least insofar as texture can be visually represented). Change of shape can either mean a change in attributes such as size or boldness, or it can change the identity of an element entirely, changing an image into a letter, for instance, or a frog into a prince, or, as can now be easily done with a programme called “Monsterfly,” changing yourself into a zombie, werewolf or vampire. Changes of shape can dynamize all the distinctive features discussed in Chapter 4, separately or in various combinations: curvature/angularity, regularity, weight, size, connectivity, sloping, expansion, density and orientation. Changes of colour, similarly, can dynamize all the distinctive features discussed in Chapter 5 – value, saturation, purity, transparency, luminosity, luminescence, lustre and temperature. In shot 1 of the corona virus video, the “cloud” of viruses grows in size. In shot 12, the viruses change identity and transform into plus signs, which, along with the thumbs up gesture of the hands in that shot, suggests that the preventive measures covered by the video will have a positive outcome, although it is somewhat strange that this transformation happens both to the still active pink viruses and to the white dots that represent suppressed viruses. The following excerpt comes from a three-minute animation titled The Girl Effect – The Clock is Ticking (Girl Effect, 2020). It is almost entirely based on changes of shape. As a clock shows the years of her life ticking away, we see a girl growing up in poverty, marrying at age 14, having her first child at age 15, and ending up selling herself and risking HIV to feed her family. The clock then goes back and restarts with a positive story in which the girl has an education that allows her to earn a good living and have children when she is ready. Figure 7.6 gives an impression of the style of the video. FULL SHOT DETAIL OF CLOCK. At ‘1’ the clock stops moving and fades out. The number one remains in the centre of the image. A dot appears to the right of the ‘1’ and grows in size. The number ‘1’ then changes into ‘2’ and the dot continues to grow in size, taking the shape of a small baby, sprouting a single hair and issuing tears. As ‘2’ changes into ‘3’,

Movement

Vertical Direction Horizontal Direct Directedness Indirect Expanded Expansion Constrained Fast Velocity Slow Strong Force Weak Straight Angularity Curved Connected Fluidity Disconnected Regular Regularity Irregular

FIGURE 7.5

The distinctive features of movement

131

132

Movement

FIGURE 7.6

Still from The Girl Effect. Reproduced with kind permission of Girl Effect

and ‘3’ into ‘4’, the dot grows further and becomes a young girl with pigtails. As ‘4’ changes into ‘5’ and ‘5’ into ‘6’, she grows still further, acquiring a skirt and legs. As ‘6’ changes into ‘7’, she acquires arms. As ‘7’ changes into ‘8’, she makes a few dance steps. As ‘8’ changes into ‘9’ and ‘9’ into ‘10’, she grows further. Then, as ‘10’ changes into ‘11’, she moves towards the left, merges with the number 11, and becomes a black pillar. The pillar then changes into the number ‘12’, the girl emerges from it and moves to a position at the left of the number. Two white lines appear on her body, outlining budding breasts. This short segment is entirely told in terms of changes of shape. Although we briefly see the hands of the clock move, after that only the numbers remain, morphing into each other, and although the girl makes a few brief dance steps and later moves to the other side of the screen, her story, too, is told only by her changing shape. When she becomes 12 years old, her identity changes: she merges with the number that indicates her age (“11”) to momentarily turn into a black monolith, which perhaps symbolizes the fatal transition to a damaged life.

Existential processes Elements can also appear or disappear, whether gradually or instantaneously. The cinema early on developed dissolves and fades (and many variants that have not stood the test of time) to provide gradual transitions between shots or scenes. In animation films and videos, not only whole shots can appear or disappear, also parts of shots. In The Girl Effect, for instance, the dot that will grow into a girl appears out of nowhere in the middle of a continuing shot, and in the corona virus video people and things appear and disappear as if created by an invisible hand. Such transitions can be compared to Halliday’s “existential processes” (1985: 130).

Movement

133

They signify “coming into being” or “moving out of being.” They dynamize the idea of “being.” The “animation schemes” offered by PowerPoint are for the most part tran­ sitions of this kind (“entrances” and “exits”), and they can also provide salience, as movement attracts the viewer’s attention more than any of the other factors that can create salience. The PowerPoint animation system is therefore in the first place functional. It provides the textual functions of framing and salience (Kress and Van Leeuwen, 2006). But the different “entrance,” “exit” and “emphasis” options PowerPoint offers are subcategorized as “basic,” “subtle,” “moderate” and “ex­ citing,” and these kinds of meaning are realized by the distinctive features dis­ cussed in this chapter-direction, velocity, angularity and directedness, and so on, as also indicated by the names of the animation effects – exciting entrance effects, for instance, include “boomerang,” “float,” “spiral in” and “swivel,” exciting em­ phasis effects “wave,” “blast,” “blink,” and more.

Mobility and identity design Moving now to resources for creating mobility, these can also realize identity meanings, albeit in a more interactive manner. The wheeled desk chair designs its user as someone who is “mobile” in some way, perhaps by being busy, moving from one activity to another, attending to a range of different things. In one University where I worked, the chairs in new lecture theatres were made to swivel, so that students could turn away from the lecturer, and interact with each other. Such differences facilitate preferred identities – the multitasker, the active learner, who learns through team projects rather than from a teacher. In the case of toys, such subject positions become imaginary. lf, in a toy car, only the driver’s door and the steering wheel can be moved, the user becomes an imaginary driver. If the bonnet can also be opened, revealing an engine, the user can also become an imaginary mechanic. Such toys have built-in assumptions about the user’s needs and interests and may provide obstacles for other uses. An important part of learning consists precisely in encountering these obstacles and their implicit messages. What you can not, or not easily, do with objects is as important as what you can do with it. “Rigid” objects can of course also be made to move. An inert wooden figurine can be made to walk, jump, dance, fly and much more. I can unproblematically balance my kitchen chair on its back legs to lean back or move it from one position to another. In fact, inert objects may allow more uses than kinetically designed objects, which inevitably favour certain uses or positions. In the world of toys, many objects allow mobility. Toy telephones, toy computers, even books for young children may all have wheels. In such cases, mobility becomes a metaphor. It teaches children that telephones, computers and books “get you somewhere,” help you to “be on the move.”

134

Movement

Articulation and flexibility There are two basic ways in which objects can be mobile (“non-rigid’) They can be articulated, in which case they will have joints or hinges, as with the limbs of Sindy and Action Man, or they can be flexible, made out of materials that can be bent into different positions or moulded into different configurations, as with Sindy’s hair, which, unlike Action Man’s rigid cap of short hair, can be styled in different ways. Flexibility is a of course matter of degree. Sindy’s face is also flexible. Unlike Action Man’s face, it is made of soft plastic, whereas her body is made of hard plastic. But this does not make it as flexible as the face of a ragdoll or a muppet-style soft glove puppet. Articulation and flexibility define what we can do with an object or what we can make an object do. In the case of representational toys, they represent how objects, animals or people can move. As we have seen in the case of Sindy and Action Man, such representations are not necessarily faithful to physical reality. In one plastic toy for very young children, a number of objects can rotate around a plastic pipe – a kind of paddle-wheel, something resembling a heavily profiled tyre, and, in the centre, a yellow plastic dog, The “technical” objects, the “pad­ dlewheel” and the “tyre,” rotate easily, without friction. The dog, on the other hand, rotates stiffly, by means of a noisy ratchet mechanism. An abstract quality (and a technological bias) is demonstrated here: technology is more easily controlled and moves more freely than living beings. The difference between articulation and flexibility is important. Articulation is necessarily based on a principled approach to what should and what should not be able to move and how, as in the case of the “body language” underlying the articulation of Sindy and Action Man, or the kinds of work which the design of a chair can support. Flexibility not only provides users with more tactile satisfaction, as it necessarily in­ volves a degree of softness, it also allows more creative freedom. Soft toys such as teddy bears, can be cuddled. They have affective value. But most importantly, they have a relatively open-ended kinetic potential, although the transformations flexibility allows will often be fleeting, as the materials will bounce back in their original shape as soon as the user withdraws his or her hands. This provides another lesson: tactile, affective and creative experiences are fleeting, leaving no traces, while the more deliberate transformation of objects through a system of articulation does. There are of course flexible materials which do allow more permanent transformations, for instance rubber, but their use in toys has remained relatively marginal.

(De)constructability Deconstructability provides a “hands on” lesson in what things are made of and how they fit together. Even when toys are not designed for deconstruction, children may want to take them apart, not from some kind of destructive urge, but from a desire to learn. Deconstructable toys teach analysis – though, as Ito (2009: 162ff) has shown in a study of children playing the SimCity game in an after school

Movement

135

workshop, children may also “deconstruct” (destroy) in an act of silent rebellion against adult attempts to control their playing. Any object can be analysed in different ways and according to different princi­ ples. In the past, and to a degree still today, construction toys such as Meccano or Lego taught children that the world is built up out of a small number of in themselves meaningless units (“atoms” or “molecules”, one might say). Larger, meaningful units could be constructed from these basic units. The basic units themselves connote the dominant construction methods of the time, steel girders in the case of Meccano, bricks, in modernist colours, in the case of Lego. Such toys position children as imaginary engineers or designers. More recently, construction toys often come in do-it-yourself kits which contain just the right amount and kind of blocks for one and only one larger item, for instance a Ninjago warship, or a Star Wars spacecraft with helmeted occupants. This makes users consumers rather than builders or designers and withholds a whole dimension of analysis and construction from them. Children can of course work against such restrictions. In one of the videos we made as part of a project on toys, a seven-year-old mounted the rotor blade of a Lego helicopter on a kind of pedestal and put it in the garden of a house, a garden otherwise filled with plastic miniature ready-made trees, parasols and garden furniture, etc. It would keep the garden cool, he explained.

Sources of kinetic energy Mobile and articulated toys may either be manually driven or powered by some other form of artificial energy, a clockwork mechanism, for instance, or an electric motor. Toys may also be driven by natural power, by wind, water or gravity. Few contemporary toys make use of this, so that children perhaps do not get enough chances to learn how to work with these forms of power, despite the fact that many traditional toys such as kites and spin tops owe their enduring fascination precisely to the way in which they allow the exploration of natural sources of energy. The powering of mobile objects can be “instigatory” or “continuous.” In the former case the user sets the object in motion or exposes it to a natural source of energy, but does not control its movements throughout the duration of the movement, as in the case of clockwork-driven cars or paper aeroplanes thrown and then left to the currents of the air. In the case of continuous control, control is exercised throughout the duration of the movement. Hand-driven instigatory control requires skill and co-ordination, especially when the effects of natural forces such as the wind are to be predicted and inculcated in the movement. The powering of objects can also be “direct” or “remote.” The continuous control of artificial energy is always remote, leading to different forms of humanmachine interfaces: rotating knobs, steering wheels, joysticks, the computer mouse, the touch screen, the drawing pad – and to the subtle skills of fingertip control which, today, are of increasing importance in contemporary society as more and more activities are powered by remote control, from street sweeping to brain surgery. Figure 7.7 summarizes the types of mobility discussed in this section.

FIGURE 7.7

The system of mobility design

136 Movement

Movement

137

Kinetic sculpture as semiotic research I started this chapter by stressing the role of avantgarde and modernist artists in developing the resources of mobility design, and will end it with the work of the Swiss sculptor Jean Tinguely (cf. Van Leeuwen, 2016), who, during his long career, has explored all the dimensions of kinetic design I have discussed in this chapter. His earliest works were abstract paintings inspired by Malevich, Kandinsky and others (e.g. “Meta-Malevich,” 1956) in which the abstract shapes were articulated by means of pulleys linked by a rubber belt to a 78 rpm gramophone motor hidden behind the painting. Later he experimented with other energy sources, using electromotors and even petrol engines to drive the gears, cams, cranks and levers that produced the movements. In many of these works the articulated movements were repetitive. In Inferno (1984), for instance, a metal beam repeatedly tried to lift itself into an upright position, but never managed to do so, in an endless Sisyphuslike toil, while other, smaller parts of the same sculpture energetically repeated the same pointless movements. Tinguely also devised many techniques for producing irregularity, unpredict­ ability of movement - differently sized wheels, elastic rubber belts, the use of several motors, etc. His Jealousy 1 (1960), for instance, represented jealousy by the nervous jiggling of wooden beads suspended form a moving metal bar. In later work, he introduced flexible materials such as feathers, dusters, rags and clothes. As described by Hulten (1975: 204), Maranar (1961) was a hanging sculpture in which: nine levers hung from camshafts which were hidden by an intermediate ceiling. When the machine was switched on, the levers tore at a tangled heap of junk and rags, a dirty nightdress, an artificial leg wearing a red sock, a washing-up bowl, coffee-tins and lengths of film. After the machine had been running for a certain time, this collection of junk suddenly exploded into violent, spastic jerks. Although many of Tinguely’s sculptures are immobile, he used mobility almost from the beginning, mostly by putting his sculptures on wheels, but also by using rails, conveyor belts and so on. Auto-Mobile (1954) was a small sculpture which could move through the room with a small electromotor. In Klamauk (1979), Tinguely used a tractor to support a contraption of cogwheels, the motions of which activated hammers and metal sticks banging on bells and cymbals in a happy fairground cacophony. While many of Tinguely’s sculptures were driven by motors, he also explored manually and naturally powered motion. His Prayer Mills (1954) were wire sculptures with interlocking wheels, some vertical, some horizontal, which viewers could move with a crank handle. His Cyclograveur (1960) was a stationary bicycle with a saddle for the viewer to sit on and pedals to activate a rusty and

138

Movement

complex set of interlocking cogwheels, which ultimately drove a drum, a cymbal and a rusty toy car. In many of these sculptures, viewers could instigate the action manually (or with their feet), as in Prayer Mills and Cyclograveur. In other cases, they could switch on a motor. In Dissecting Machine (1965), viewers activated drills and saws which at­ tacked the dismembered parts of a life-like window dummy, so diminishing, as one reviewer put it, the “gap between the vicarious enjoyment of cruelty and the act itself” (Herald Tribune, 28 May 1971). In other cases, movement was “auto­ kinetic,” so that the machine appeared to have a life of its own. Velocity also played a role – animal skulls rocked gently, discs rotated furiously, expressing anger and frustration. Throughout his career, Tinguely developed an arsenal of resources of mobility design and explored their meaning potentials, initially primarily in relation to the machine itself – the machine’s mindless repetition, its wastefulness (one of his works endlessly smashed bottles). In later work, he broadened his vocabulary, using not only machine parts but also plastic toys, clothes, feathers, animal skulls, etc., animating them with the movement resources he had developed, to create a wide range of genres and contexts from altar pieces and portraits (cf. his selfportrait in Figure 6.3) to children’s playgrounds and theatrical sets. In short, Tinguely was a semiotician. Along with Calder, he developed mo­ bility as a semiotic resource, albeit expressed in the form of art works rather than academic prose. And, I might add, he did so a long time before I woke up to the importance of mobility design in my research on toys (Van Leeuwen and CaldasCoulthard, 2004) and kinetic typography (Van Leeuwen and Djonov, 2015).

8 A SOCIAL SEMIOTIC THEORY OF SYNAESTHESIA

Introduction So far I have discussed the distinctive features of shape, colour, texture and movement separately. But in actual multimodal designs these parameters work together to create a unified impression. In earlier work (Van Leeuwen, 2005), I introduced four methods of analysing how more or less abstract semantic and interactive relations connect elements of composition and units of rhythm – spatial composition, temporal composition, information linking and dialogue. All of these are modes, functional systems that can be realized in different media. Showing how the concrete, material and embodied characteristics of individual expression media combine is more difficult. To give an example, the regular alternation of “stressed” and “unstressed” moments is a key characteristic of rhythm in all temporal ex­ pression media, and in all these expression media it functions to highlight com­ municatively important moments in the flow of information. But materially, it can be realized by different configurations of distinctive features, both within a given medium (for instance speech) and between different media (for instance between speech and gesture). In speech, the stress of syllables is realized by four distinctive features – pitch, loudness, duration and vowel quality. All of these are always present in every stressed syllable, though in different proportions, opening up different meaning potentials, as one or other feature may dominate and as each feature is tempered or qualified by all the other features. Dominant pitch can convey engagement and excitement; dominant loudness a sense of urgency or an upsurge of anger or some other strong emotion; increased duration a didactic, or even patronizing form of emphasis. But stress can also, and at the same time, be realized gesturally, by blinking, hand movements, head nods, and so on. In their multimodal combination (and together with tempo), speech and gesture can create many different speech styles, all capable of expressing identity – just as happens in

140

A social semiotic theory of synaesthesia

music, where such combinations of tempo and style are traditionally labelled with Italian words – adagio (“very slow, leisurely”), grazioso (“gracefully”), vivace (“li­ vely”), agitato (“agitated“, “restless”), and so on. To understand how the distinctive features of different parameters work to­ gether to form complex, composite meanings in a given design practice or instance, four questions need to be asked: •

What individual parameters are involved, what are their distinctive features, and what are the meaning potentials of these features?

Fashion design, for instance, combines shape, colour and texture (fabric). Film making combines shape, colour, texture and movement (as does product design) – the distinctive features of these parameters and their meaning potentials have been discussed in previous chapters. But film making (and, increasingly often, product design) also includes sound and music. For this reason, I will, later in this chapter, add one further parameter, timbre, the materiality of sound. •

What are the affinities and differences between these parameters and their features?

Chapter 5 established brightness (also known as “value”) as a key distinctive feature of colour. But the term “brightness” can also be used to describe the sound of voices and musical instruments, so suggesting an affinity between these two parameters. Again, Chapter 4 established (ir)regularity as a feature of shape, but, in later chapters, it turned out to be a feature of texture and movement as well, as did angularity. Such affinities or correspondences between different parameters have long been recognized, for instance by John Locke, in his Essay Concerning Human Understanding (1972 [1706]: 30) A studious blind man who had mightily beat his head about a visible object, and made use of the explications of his books and friends, to understand the names of light and colours, which often came his way, betrayed one day that he now understood what scarlet signified. Upon which his friend demanded what scarlet was? The blind man answered, it was like the sound of a trumpet. In the same period in which artists and designers began to work multimodally, and in which the new science of psychology began to explore colour preferences and colour effects, neurologists and psychologists became interested in synaesthesia, the study of how “the stimulation of one sensory modality automatically triggers a perception in a second modality” (Harrison and Baron-Cohen, 1997: 3). For artists and designers (and, later, semioticians), this raised the question of the cor­ respondences between parameters. Just as composers and arrangers must under­ stand the “acoustic affinity” of different instruments, (Adler, 2016: 627, 634), so artists, designers and semioticians must understand the affinities (as well as the

A social semiotic theory of synaesthesia 141

differences) between the meaning potentials of the different parameters that are “orchestrated” together in multimodal designs. The question of synaesthesia will be discussed in more detail below. •

What are the principles by which the parameters and their distinctive features combine?

In Chapter 5, I discussed some of the ways in which colours combine into colour schemes. Some of these are based on affinity, as when the colours of a scheme all have the same degree of brightness or saturation. Others are based on contrast, as when a bright red stands out amidst a range of subdued earth colours. Such principles also apply to combinations of parameters. In a dance performance, the movements of the dancers and the music may both be connected and flowing. The set design, too, may then use flowing, wavy forms, but it may also contrast with the movements and the music by using disconnected, fragmentary elements. To discuss how parameters combine, I will draw inspiration from the practice and study of orchestration. Semiotics studies of non-linguistic semiotic modes have long drawn inspiration from the study of language and this has proved productive. But non-linguistic modes can also provide inspiration for the study of other non-linguistic modes – and for the study of language. Not only the “lan­ guage of music” should be studied but also the music of language (Van Leeuwen, 1999), and the practice and study of music, in turn, could learn much from the practice and the study of movement, as Han has shown (2021). The questions I asked in the introduction of this chapter are also asked by orchestrators. They must, first of all, understand the expressive potential of the individual musical instruments in the orchestra. Adler (2016), in a monumental treatise of the art of orchestration, describes musical instruments in terms of three features, “range and registral characteristics” (“the most effective range on the oboe is from F4 to C6, for a thin, softly piercing effect, the oboe is most beautifully scored in the range between this high C and the F above it”), their “articulative” potential (do they, for instance, lend themselves to “quick single-tongue staccato passages” or “fairly large skips”?) and their ability to perform “coloristic effects” such as “pitch bending” or producing “non-pitched passages”(e.g. in the case of the oboe, by removing the reed and blowing air through the tube only, ibid: 205). Second, they must know how to combine instruments in terms of affinities and differences between their distinctive features. If instruments are to “double” the same melody without “loss of individual colour characteristics”, for instance, they must be “acoustically compatible” (ibid: 633) – in the first movement of Brahms’ 3rd Symphony, Adler explains, only the flutes explore the highest register and all others play in their middle and low registers, preventing the flutes from being “neutralized by other instruments” and giving “clear separation between high and low and beautifully luminous spacing” (ibid: 630). Third, they must understand how instruments can be combined. They can all play the same notes (unison), for instance, with or without separation between high and low, and with or without one or more instruments providing “accents” to

142

A social semiotic theory of synaesthesia

“add subtle flavour” (ibid: 612). Proportion is important here. Trombones may have to be marked “pianissimo” to prevent them from dominating other instru­ ments. They can also play different notes, e.g. homophonically (“melody plus accompaniment”), with the melody, played by a clarinet for instance, in the foreground, and the strings providing accompaniment in the background. This accompaniment may be harmonious and supportive, or contrast in some respect with the melody. The clarinet may, for instance, play a flowing melody while the strings play “pizzicato” (ibid: 325). Principles such as “unison” and “homophony” of course describe functional designs and can apply to different instrumentations. But the timbres, the actual sounds of the actual instruments used, can play a significant role in identity design, for instance in branding, as the example below will show. •

What is the relevant context?

The meanings of timbre, like that of other parameters, can only be understood in context. To give an example, the logo of AT&T, the American international telecommunications corporation, uses a four-note ascending melody which re­ mains unresolved. Ascending melodies of this kind can be found in many contexts, in classical music (e.g. Beethoven’s Leonore Overture and Strauss’ Also Sprach Zarathustra), in patriotic songs and military hymns, and in news signature tunes (Van Leeuwen, 2017), to mention just some examples. In all these cases, they have a heraldic function, enacting a call to attention, or a call to action, a “rallying cry” (Jurgensen, 2012). But it is through their timbres that such “rallying cries” can embody the identities of nations, of news programmes, of companies, and so on. The timbre of the AT&T logo is not that of a loud, military brass band, as would be the case in a patriotic song, but a blend of an old piano, a glockenspiel and a Wurlitzer, which is then given an electronic edge. Joel Beckermann, the designer of the logo, has described the intended meaning (Man Made Music, 2016): “AT&T tasked us with creating a Sonic Identity System that would embody 'Rethinking Possible' in a more human and relatable way.” To achieve this, he blended electronic sounds, evoking technological perfection, with the human touch of nostalgic instruments, all this to make AT&T “more human and ex­ pressive” (Kessler, 2014) and to combine “warm and forward thinking” (Man Made Music, 2016). The AT&T example describes and interprets the distinctive feature config­ uration of a single parameter, timbre. But the same approach can be applied to combinations of parameters. The “Google doodles” Google puts out on the birthdays of famous people or on the occasion of special events (Britten, 2020), play an important role in the way the company communicates its identity. Table 8.1 lists the distinctive features of each of the three relevant parameters (shape, colour and movement) in one relatively simple doodle. As the shapes and placement of the letters is not consistent throughout the logo, the different letters are analysed separately. Figure 8.1 shows a likeness of the doodle, of course

A social semiotic theory of synaesthesia 143 TABLE 8.1 Analysis of a Google doodle

LETTERS

Shape

Colour

Movement

G, o, o, g

Rounded Bold Horiz. Orientation Expanded Disconnected

n.a.

l

Angular Bold Vertical. orientation Expanded Irregularly placed Rounded Bold Sideways orientation Expanded Irregularly placed

Bright Saturated Pure Plain Differentiated: all three primary colours Bright Saturated Pure Plain Green Bright Saturated Pure Plain Differentiated: Red (+ rainbow on drums)

e

Forceful Up & down movement Regular Medium fast Forceful Sideways movement Regular Medium fast

without the movement, the up and down, hip hop style dance movements of the “l,” and the energetic drumming of the “e.” Several features of shape are consistent across all the letters. They are all bold and rounded, and they all sit confidently and solidly on the baseline, with plenty of space between them. But the “l” and the “e” are irregular – the “l” is angular rather than rounded (which is accentuated by its up and down movement), while the “e” is rotated, has a sideways movement, and both are “sunk” below the baseline. The colours are also consistent across all letters – bright, saturated, plain and pure. But the “l” is, again, irregular, because green is not a primary colour. All movements, finally, are forceful and energetic. There is therefore an affinity

FIGURE 8.1

Likeness of a Google doodle

144

A social semiotic theory of synaesthesia

between the boldness of the letterforms, the strong colours, and the forceful movements. But there are also contrasts, deriving both from the placement and the shapes of the letters. What does this configuration of features say about Google? Ruth Kedar, the logo’s original designer, saw Google as a new kind of encyclopedia and drew inspiration from the way encyclopedias juxtapose items in unpredictable, quirky, and sometimes even humorous ways (e.g. “parakeet” next to “paralegal,” Paget, 2019). The company itself described its latest version (which is the one Figure 8.1 is based on) as “combining the mathematical purity of geometric forms with the childish simplicity of schoolbook letter printing and maintaining the multicoloured playfulness and rotated ‘e’ of our previous mark” (Jennings, 2015). Irregularity here becomes childlike and playful as well as irreverent, flouting the rules of typography and the laws of colour. Over the years, the logo had become bolder, losing details such as serifs and drop shadows. The company thus presents itself as young, playful, irreverent, brightly positive, energetic and progressive in the issues its doodles support (here the environment, through the transformation of the green “l” into a tree). Although Google has, today, become a tech titan, working closely with the American government in expanding its power of sur­ veillance (Zuboff, 2019), its logo still portrays it as a young, friendly, fun-loving, irreverent, innovative and progressive company.

Synaesthesia At the same time as early 20th-century artists became interested in multimodality, psychologists and neurologists became interested in synaesthesia. The term derives from the Greek words for “together” (syn) and “perceiving” (aisthano), and sy­ naesthesia was originally conceived of as a neurological condition, an involuntary perception – as when people see, for instance, the letter “a” and simultaneously see the colour blue, as vividly as they see the shape of the “a,” even though in reality the “a” is not blue but black. Artists and writers, on the other hand, deliberately produced combinations of “sensory stimulations.” The artists who, in 1911, formed the Blaue Reiter (“Blue Rider”) art movement, for instance, sought to create Gesamtkunstwerke (“total artworks”) that combined music, dance and theatrical production. Kandinsky was a member and contributed a theatrical piece called Der Gelbe Klang (“The Yellow Sound”) and in his teaching at the Bauhaus encouraged students to experiment with the correspondences between shape, colour, touch, temperature, sound and energy (Van Campen, 2006: 56). Contemporary designers do much the same. Haverkamp (2013: 17), for instance, encourages product designers to “optimize the alignment of the auditory perception and the visual appearance of products with one another,” and to ask questions like “Which colour scale best represents the timbre in engine noise?”, or “Which indicator sound harmonizes ideally with the visual features of the indicator switch lever?”

A social semiotic theory of synaesthesia 145

Neurologists have traditionally rejected such “artistic aspirations to sensory fu­ sion” and “cross-modal associations” as “pseudo synaesthesia” (Cytowic, 1997: 20) and Huisman (quoted in Van Campen, 2008: 146) has suggested the term “syn­ chronesthesia” to distinguish them from “genuine synaesthesia”. Nevertheless, scientific and artistic interest in synaesthesia grew at the same time, inspired by the same changes in the semiotic landscape – and not all neurologists agree with the strict separation between “genuine” and “pseudo” synaesthesia. Some have revitalized a line of thought which has existed since Antiquity when Aristotle (2008: 425–27) asked a question that is still of central interest to multimodality researchers: how can humans perceive a unity in the multitude of sense impressions? His answer was that there is a “koine aisthesis”, a “common sense”, which perceives common qualities in the input from different senses. This insight, which was indeed “common sense” in the Middle Ages, returned, in a new way, in the Romantic era, for instance in the work of Herder (2002 [1772]), who proposed that those common qualities are feelings – it is feelings, emotions, Herder thought, which create the affective, synaesthetic bond between the senses. In the work of late 19th-century scientists such as Féré, Bleuler and Lehman and Flournoy (cf. Marks, 1997) and, more recently, in the work of neuroscientists such as Cytowic and Marks, these ideas re-emerge: synaesthesia might be ex­ plained by “elemental perceptual qualities,” “form constants,” which are “rather abstract,” yet “provoke reactions” (Cytowic, 1997: 29). Not much progress was made, however, in establishing just what these elemental qualities are, although “brightness” (corresponding to high pitch in sound and lightness in colour) has been mentioned as a candidate since the late 1920s (Von Hornbostel, 1931). In this context, the idea that synaesthesia is a unique condition affecting only a few people is beginning to be watered down. According to Marks (1997: 870-88) Synaesthesia is not an isolated phenomenon, separated from non-synaesthetic perception and thought. Rather, synaesthesia is a cross-modal manifestation of meaning in its purely sensory, and in one sense its strongest form. As a result, assessments of the prevalence of synaesthesia have changed radically. Originally stating that it affects 1 in 100,000 people, Cytowic later reduced the ratio to 1 in 25,000 (1997: 33) and more recently synaesthesia researchers have proposed ratios of 1 in 500, 1 in 200 and even 1 in 100 (Van Campen, 2008: 128). Incidence of synaesthesia in children had long been reported to be higher than in adults, with estimates in the range of 40–50% (Marks, 1997: 88). We are all sy­ naesthetes, Marks concludes, but it may be that in adulthood “it becomes valuable, indeed necessary for the child to transfer the meanings from the perceptualsynaesthetic to the verbal realm” so that for most adults, synaesthesia becomes “vestigial” (ibid: 89). However, neurologists continue to link synaesthesia with emotion, situating it in the limbic brain, which handles emotion, memory and attention, and “easily overwhelms thinking” (Cytowic, 1997: 32). They also continue to regard

146

A social semiotic theory of synaesthesia

synaesthesia as idiosyncratic, with different synaesthetes linking letters and colours, or sounds and colours, in different ways. From the point of view of social semiotics, one cannot help noticing how closely these new scientific developments fit, not only with the simultaneous explorations of artists but also with the way public communication has developed since the 1920s – its increasing multi­ modality, its appeal to emotion rather than reason, and its emphasis on aesthetic qualities, even in everyday documents like invoices and business reports (cf. Van Leeuwen, 2015). From the point of view of social semiotics, synaesthesia is a social phenomenon, however, much it is grounded in physiological principles of mul­ timodal perception, and however, strongly the identity values it expresses are linked to aesthetic appreciation (“taste”) and emotive investment. As Freud knew, emotions do not exist in a vacuum. Both negative and positive emotions are always, though with different degrees of strength, connected to what he called “representations” and what in social semiotics, we see as meanings – both “common,” social meanings and the individual meanings that stem from people’s unique life stories.

The distinctive features of timbre Before moving to a detailed discussion of the correspondences between shape, colour, texture and movement, I will add another parameter to these four, timbre. In this section I will therefore introduce the key distinctive features of timbre, drawing on earlier work (Van Leeuwen, 1999, 2014). Their meaning potentials rest on our basic physical experience as speakers and singers, as creatures with a voice. But, as we will see, they can be extended to the production and under­ standing of non-vocal sounds – instrumental music, sound effects and ambient sounds.

Pitch range Experience tells us that our pitch range flattens not only when we feel bored and listless but also when we adopt a soft, intimate voice – some of those sensuous Brazilian bossa nova songs have melodies with a very limited pitch range. When we are excited and energized, on the other hand, our pitch range increases – songs with a wide pitch range are the staple fare of patriotic hymns and other songs that seek to energize people into collective action. Our experience of pitch range tells us something else as well: men’s voices are, on average, lower than those of women and children, and smaller resonating chambers (e.g. violins) are sympathetic to higher sounds than larger resonating chambers (e.g. double basses). But paradoxically men tend to use the higher re­ gions of their pitch range to assert and dominate, while women tend to use their lower register for that purpose. As it is difficult to be low and loud at the same time, women face a dilemma in expressing their identity through vocal timbre.

A social semiotic theory of synaesthesia 147

They must either speak low (which is assertive) and soft (which is intimate), risking to evoke the “dangerous woman” stereotype, or high (“belittling them­ selves”) and loud (“being assertive”) – risking to be thought “shrill.” Iconic female voices express different female identities in this way – think of the high, childish voice of Marilyn Monroe, or the low sensuous voice of Lauren Bacall.

Loudness Loudness exudes power. The more powerful people or institutions are, the more noise they are allowed to make. The loudest noises, Schafer writes, are always “sacred noises”: “Thunder, the voice of God, migrated first to the cathedral, then to the factory and the rock band” (1977: 79). Schafer also recounted how a group of Hare Krishnas in Vancouver were fined for causing excessive noise while, nearby, the noise of a building site exceeded 90 decibels (1977: 201) - without the builders being fined. Loudness also relates to social distance, both literally and figuratively. In in­ timate situations we are not only physically close to other people but also use an intimate, soft whisper. In informal meetings with friends and acquaintances, our voice will be fuller, but still soft enough not to be overheard by strangers. At what Edward Hall (1966: 154) called “public distance” (5–8 feet), our voice will be “full with slight over-loudness.” Beyond that, we project our voice to address larger groups of people, though amplification has now made it possible to whisper to a large gathering, and, more generally, to use loudness as a means of expressing social proximity or distance.

Tension We all know what happens when we tense the muscles of our throat. Our voice becomes higher (lower overtones are reduced, higher overtones increased), sharper, brighter, and above all more tense, because in their tensed state the walls of the throat cavity dampen the sound less than they would in a relaxed state. The sound that results from this not only is tense but it also means “tense” and makes tense. This can then be coloured in by the context in which the sound is used, to become “aggression”, “repression”, “excitement“ and a host of other meanings which can be said to include the idea of “tension”. When, on the other hand, we open our throat and relax our muscles, our voice becomes smooth and mellow. Non-vocal sounds can also be tense. The crucial factor here is the rigidity of the objects that cause the sound and/or the resonating environment. Public life in the city is full of tense sounds – high heels on marble stairs, the clatter of forks, knives and plates in restaurant kitchens, the metal doors of cars and trains slamming shut. Compare this to soft shoes on grass, the dull thud of a rubber-lined fridge door, the muffle of heavily carpeted and cushioned rooms.

148

A social semiotic theory of synaesthesia

Rough, breathy and trembling voices In the rough voice, we hear other things beside the tone of voice itself – hoar­ seness, harshness, rasp, grit. Its opposite is the smooth voice, from which all noisiness is eliminated. Much of the effect of roughness comes from the aperiodic vibration of the vocal cords, which causes noise in the spectrum (Laver, 1980: 128). Again, the meaning of roughness lies in what it is: rough. Our experience tells us that roughness comes from wear and tear, whether as a result of smoking and drinking, hardship and adversity or old age. But roughness is valued differently in different contexts. In African American music, roughness has a strong emotive effect. In the words of Harold Courlander (quoted in Williams-Jones, 1975: 377): Some outstanding blues, gospel and jazz singers have voices that can be described as foggy, hoarse, rough or sandy. Not only is this kind of voice not derogated, it often seems to be valued. Sermons preached in this voice appear to create a special emotional tension The contrast exists in popular music as well – compare the lived-through voice of Tom Waits with Leonard Cohen’s pleasant croon, or the nasal trademark of Marianne Faithfull to Celine Dion’s warm head voice. In the breathy voice another sound mixes in with the tone of the voice itself – breath. Its meaning potential derives from our experience of what can make our voice breathy – exertion of some kind, excitement, or sexual arousal. It often combines with a soft voice, suggesting intimacy. Advertisers use it to give their message a sensual, erotic appeal, and singers may use it for the same reason. The meaning potential of vibration, the trembling voice, derives from what we know makes our voice tremble – emotion. It plays a key role in the musical expression of emotion. Strings, for instance, are particularly good at producing vibrato sounds and hence universally used to “pull the heartstrings,” to present and represent love and romance. But other emotions, too, can make us tremble, and vibrato, or alternatively tremolo, is also used, for instance, in horror movie music, to express and instil fear and uncertainty.

Articulation Some vowels are “frontal,” articulated with the tongue in front of the mouth (e.g. the [i] of heed), others are articulated in the back of the oral cavity (e.g. the [a] of hard). This has often been related to sound symbolism. In many languages, words meaning “close” use frontal vowels and words meaning “far” back vowels (e.g. near and far in English, hier and daar in Dutch, ici and là-bas in French, hier and dort in German). But frontality and its opposite can also be overall articulatory settings expressing a quality of being “upfront,” “confronting,” or of “holding back,” “not coming out with it.”

A social semiotic theory of synaesthesia 149

The same can be said for “aperture.” Some vowels are produced with the mouth comparatively closed and the oral cavity therefore comparatively small, others with the mouth comparatively open and the oral cavity therefore larger. The [i] of heed and the [u] of hood are less open, for instance, than the [a] of hard. This too has mostly been discussed in terms of sound symbolism: words with an open [a] have been said to be “heavy, big and round,” for instance, and words with an [i] “small, light and pointed” (Hildum and Brown, 1956). But aperture can also characterize vocal settings. In Star Wars, The Phantom Menace (2000), the treacherous Viceroy of Naboo, a character with an inscrutable fish-like physiog­ nomy, not only has a vague Chinese accent but also speaks with a stiff jaw and an almost-closed mouth, using a breathy, hollow-sounding, faucalized voice. Such articulatory settings are often used by puppeteers and by the actors who create the voices of cartoon characters. As mentioned, earlier, all these parameters combine, in different proportions, in actual voices. The voice of Marlon Brando in The Godfather (1972), for instance, is comparatively high, and hence assertive and dominating. It is also hoarse and rough, signalling the Godfather’s harsh and unforgiving side. And it is articulated with a stiff jaw and an almost closed mouth, suggesting an unwillingness to “give” that keeps us guessing as to what he might be keeping from us. Yet it is also soft and breathy, at times almost a whisper, making his menacing presence disturbingly intimate and attractive. The meanings of vocal timbre thus derive from specific configurations of all these qualities, and all play a role in expressing identity, whether in everyday life or in acting or singing. But they can also derive from provenance. The use of am­ plification in recorded music and the cinema has allowed iconic actors such as Lauren Bacall, Marilyn Monroe, and Marlon Brando to develop specific styles of speech, and iconic singers such as Ray Charles, Bob Dylan, Nina Simone and others to develop specific styles of singing. This has created a lexicon of possible voices which draw identity meaning from the kinds of roles these actors have played, the kinds of songs these singers have sung, as well as from the stories that circulate about their private lives. Marlon Brando’s hoarse whisper, for instance, is now part of the repertoire of many actors, to be deployed whenever a sense of brooding, yet sensuously attractive menace is required. Madonna was perhaps the first singer to use quite different voices for different songs (cf. Van Leeuwen, 1999: 150–152). In Like A Virgin, for instance, she ad­ dressed a “you” who saved her from perdition and made her feel “new” and “shiny,” “like a virgin touched for the first time.” This “you” she addresses in a voice which is high and feminine, yet has a tense and strident edge, signalling scars of abuse and betrayal. In Live to Tell, on the other hand, her voice is lower, more relaxed, as well as softer, breathier and warmer. Only later in the song does an occasional vibrato and increased tension add the strength of revived emotion to some of the key moments. So vocal styles have begun to be understood on the basis of set configurations of values of the distinctive features I have discussed in this section, and on the basis of

150

A social semiotic theory of synaesthesia

listeners’ familiarity with the movie roles and songs in which these styles have been used and the public profile of the actors and singers who developed them. Everything I have said here of the voice can also be applied to musical instru­ ments. Key instrumentalists, for instance jazz players such as Miles Davis and Stan Getz, have also developed instrumental styles that have been taken up by many other players. Just as voices can be soft, smooth and well oiled, or rough, raspy and cracked, so saxophones and trumpets can be soft and mellow, or tense and strident, sound like a hoarse whisper or a foghorn in the mist – and both now include a large repertoire of howls, wails, groans and other vocalizations that can be captured neither by alphabetic writing nor by musical notation. Experience here includes not only the distinctive features discussed in this section but also our experience of the gestures, tools and materials of musical sound production. Schafer has given an excellent example of the social and cultural significance of such physical modes of sound production (Schafer, 1977: 109): The substitution of the punched string piano for the plucked harpsichord typifies the greater aggressiveness of a time in which objects were punched and beaten into existence by means industrial processes where once they had been stroked, carved and kneaded into shape. Finally, the distinctive features of timbre can also be applied to the analysis of sound effects and ambient sounds. Their meaning not only derives from their source, from what they are the sound of, but also from their timbral qualities. The closing of a door, too, can sound loud or soft, tense or lax, rough or smooth, and so on. Contemporary digital technology has generated many new timbres, but for the most part we still interpret their meaning on the basis of our experience of the features of non-electronic sound, even when these are digitally enhanced. Breathiness, for instance, can often be suggested by added hiss, roughness by distortion, and tremolos can also be digitally programmed. (See Mulder and Van Leeuwen, 2019, for a detailed discussion.) But some electronic sounds will be “extra-experiential,” with qualities that cannot be produced by the human voice, very fast tremolos, or on­ going drones, for instance (humans have to take a breath every now and again). Their meaning may lie in their technological origin. The voice of the robot, for instance, may be deprived of vocal features that express human emotion, such as variations in pitch and loudness. But it may also derive from their “non-humanness,” which can then, in context, become, for instance, the “divine” or the “alien,” as shown by this quote from the 1959 review of a radio play that pioneered the use of electronic sound (quoted in Niebur, 2010: 67): The three-note phrase was something like the slow swell of the sea but also like a groaning dirge from some inhuman voice … and as affective as an audible equivalent to nightmare. Figure 8.2 summarizes the distinctive features discussed in this section.

A social semiotic theory of synaesthesia 151

high low pitch range extended monotone loud loudness soft tense tension lax rough roughness smooth breathy breathiness non-breathy vibrato vibrato plain frontal back articulation open closed

FIGURE 8.2

The distinctive features of timbre

152

A social semiotic theory of synaesthesia

Affinities Comparing the distinctive features of shape, colour, texture, movement and timbre suggests a number of common, or closely related, qualities.

Intensity and energy At least two of the distinctive features of colour can be associated with the idea of “intensity” – temperature and saturation. In the case of shape, I associated bold shapes with intensity, and oblique, slanted shapes with energy. In the case of sound, energy and intensity can be expressed by timbral features which we know involve increased vocal effort – pitch range, loudness and tension. In the chapter on movement I introduced “force,” which could also be seen as ex­ pressing intensity. Some textural features, finally, require more effort of touch than others, for instance rigidity (the harder the material, the more energy is needed to squeeze it) and weight (the heavier an object the more energy is needed to lift it). Some of these features foreground the idea of energy, relating to latent or actual movement, others foreground a more static sense of intensity. The famous Nike logo combines the two, expressing intensity through the boldness, and energy through the slanting of the letter forms. In this case, the same parameter expresses both meanings, in other cases energy and force may be expressed cross-modally. Many television commercials present “special offers” in strong, saturated colours, and with bold letters and numbers moving energetically, almost explosively, towards the viewer.

Brightness As has long been recognized in the synaesthesia literature, two parameters can realize “brightness” – colour and timbre. Colour realizes it through the dis­ tinctive feature of value (but also through the features of luminosity and lu­ minescence), timbre through the feature of pitch – the higher the pitch, the brighter the sound. “Frontality,” too, can evoke a sense of brightness, both in specific speech sounds (the frontal [i] for instance appears brighter than the back [a]) and in the use of frontality as a vocal setting. And musical instruments can also be more or less bright as result of their pitch register and through the formant structures resulting from the shape of their resonators and the way they are played. How brightness might be realized in graphic shapes, textures and movements is less obvious. This has been acknowledged in the synaesthesia literature: “Certain combinations of synaesthesia almost never occur (for example touch to hearing)” (Harrison and Baron-Cohen, 1997: 3). It is important, not only to look at the affinities between parameters but also at their unique qualities and meaning potentials.

A social semiotic theory of synaesthesia 153

Roughness and regularity All parameters can, in their own ways, express roughness. Graphic shapes can have ragged edges or irregularly textured areas. The textures of surfaces can feel smooth or rough to the touch. Movements can be purposeful and directed, or irregular – jerky, or wavering, for instance. Sounds can be smooth or “noisy,” adding hoarseness, breath, or rasp to the actual tone, and aural roughness can also result from a lack of blending between the different instruments or voices that play or sing in unison. Colour modulation, finally, can also be rough, as when colour is applied with rough brush strokes. Comparing roughness across different parameters can lead to a rethinking of the features of individual parameters. In discussing the irregularities of graphic shapes, for instance, I originally focused only on roughness (irregularity of the contours of shapes and of the visual texture of the area they enclose), not on irregularities between recurring letter forms or other shapes, or of irregularities in the distribution of shapes across a surface. And in discussing colour modulation I did originally not consider the question of whether nuances of the same colour are applied more or less randomly or in neat patterns, something which has been explored extensively in modern art. Regularity can therefore be redefined as the degree to which different instances of a given colour, shape, sound, texture or movement within the same text, ar­ tefact or event are identical or varying in one or more aspects, and the way in which colours, shapes, sounds, textures and movements are distributed across space and time. At its core, therefore, is the idea of predictability, which, in turn, links regularity to patterns of attention. In the case of graphic shape, regularity refers to the degree to which repeated letterforms or other graphic shapes are identical or varying in form, and the degree to which letter forms and shapes are distributed regularly across the surface. In the case of texture, regularity refers to the degree to which relief patterns create regular or irregular touch impressions. In the case of colour, it refers to the regularity of colour modulations and differentiations. In the case of sound, it pertains to the degree to which a sustained tone or recurrent phrase varies over time in terms of features such as loudness, roughness, frontality and aperture. In the case of movement, finally, it concerns the degree to which a gesture or a repeated movement of the whole body varies in terms of features such as expansiveness, velocity, force or fluidity and connectedness. Regularity is therefore a complex feature, a feature that applies to the organi­ zation of other features across the whole of a design (cf. Djonov and Van Leeuwen, 2013).

Differentiation Differentiation is also a complex feature. In the case of colour, it refers to the scale from monochrome to a maximally wide variety of colour. That I did not at first

154

A social semiotic theory of synaesthesia

recognize the possibility of graphic shape differentiation perhaps stemmed from the fact that my ideas about graphic shape were originally developed in relation to typography. Only later I realized that the distinctive features of typography could be applied to other kinds of graphic shape as well. Such shapes can clearly be differentiated. Abstract paintings and decorative patterns, for instance, could use only rectangles or only circles, or a variety of shapes. Other parameters also allow differentiation. Relief textures can consist of a single type of indentation or protuberation, or a variety of relief patterns, as e.g. in Braille. As for movement, some forms of marching combine different steps while others use a single step throughout, and differentiation is of course also an important resource in choreography. Timbre, finally, can be simultaneously differentiated, as when a given sound blends, for instance, the different pitch registers of a choir or the different instruments in an orchestra, or different sound effects, as is now often the case in movie sound design. There are also sequential forms of timbral differentiation, for instance in the Klangfarben (“sound colour”) orchestrations pioneered by composers such as Anton Webern, in which each of the notes of a single melody is played by a different instrument, often also with different playing techniques, e.g. pizzicato or bowed violins, muted or unmuted horns, etc.

Connection Connection is another complex feature. It can be applied to graphic shapes as well as to timbre (Van Leeuwen, 1999), where it is realized by the contrast between “legato” (in which the notes of a melody or the syllables in an utterance form a fluent, connected line) and “staccato” (in which the notes of a melody or the syllables in an utterance are articulated as separate, short stabs). Movements, too, can be connected or disconnected to different degrees. In the case of colour, the boundaries between different colours may be fluid or sudden and abrupt. In the case of texture, areas of different density or roughness may flow into each other or be separated by more or less distinct boundaries.

Expansion “Expansion” can be found, not only in graphic shapes and movement but also in other parameters – decreased density in the case of texture, for instance, and in­ creased aperture in the case of timbre. In each case, there is a sense of “opening out” or “closing in,” which, in context can come to mean “freedom” or “constraint,” “outward-ness,” “introversion,” and so on.

Angularity I identified the contrast between angularity and roundness as a feature of graphic shape and movement. But related contrasts can be recognized in other parameters

A social semiotic theory of synaesthesia 155

as well. Textures, for instance can feel sharp or blunt to the touch, and sounds can be either mild and soothing or sharp and stabbing. However, colour can perhaps not realize this contrast. Table 8.2 provides an overview of the common qualities explored in this section. It shows that for some of the features of specific parameters no plausible common qualities could be found – purity and lustre in the case of colour, for instance; vibrato in the case of sound; liquidity and viscosity in the case of texture; velocity and direction in the case of movement. It may well be possible to “translate” these features into other parameters, using, for example, wavy lines to indicate vibrato, or fluid and repetitive melodies to indicate liquidity. But such translations would be iconic, rather than based on the visible, audible, touchable and kinetic affordances of the parameters themselves and on the tactile experiences of articulating or perceiving them.

Principles of combination So far, I have explored the affinities between the distinctive features of colour, shape, texture, timbre and movement. But this still does not answer the question Aristotle asked nearly two and a half thousand years ago: how can we perceive a unity in the multitude of sensory impressions? How do the different parameters blend or fuse in actual texts, artefacts and performances? The discussion of colour schemes in Chapter 5 suggested one possible an­ swer. It posited three principles which can be applied, not only to combining different colours but also to combining different parameters. The first is that of colour harmony, where the colours in a scheme are equally light or dark, or equally saturated or pale, and so on. This then implied the possibility of disharmony, as in the case of Van Gogh, who wanted to “express the terrible passions of humanity” by means of “a clash and contrast of the most disparate reds and greens” (Van Gogh, 1978: 28), and in the case of the fashion designer I quoted in Chapter 5, who loved “the effect of splashing outstanding colours into a muted colour palette” (Mora, 2009: 93). It also included the idea of colour accents, a single colour which contrasts with the others, but is used sparingly. The table of the common qualities of different parameters (Table 8.2) can help explore cross-modal harmony, disharmony and accentuation. The sculptures of Jean Tinguely, discussed in Chapter 7, use all three principles extensively. His Fairy Tale Relief (1978) is dominated by a large iron frame housing the cogwheels of a complex machine, all in cheerless grey and black. But the foreground shows a small yellow toy duck whose behind is tickled by a pink duster, an accent of colour as well as texture, bringing a tender and also humorous touch of childhood into a mechanical industrialized environment. But examples can also be found in ev­ eryday objects, where, for instance, a black velvet jacket may have yellow metal buttons and combine with faded blue jeans, or where, as on my dinner table, rattan placemats in natural colours can be decorated with colourful beads and tassels.

Temperature Purity Lustre

Degree of merging transitions

Warm/cool

Connected/ disconnected

Expanded/dense Angular/round

Rough/smooth

Value Luminosity Luminescence Modulation

Saturation Temperature

Intense/weak

Energetic/lax Bright/dull

Colour

Common quality

TABLE 8.2 Common qualities

Dis/connection

Expansion Angularity

Irregularity (of contours & visual texture)

Orientation

Weight Size

Shape

Degree of merging transitions

Liquidity Viscosity

Temperature

Sharpness

Relief

Weight Rigidity

Texture

Legato/staccato

Vibrato

Aperture Staccato Friction

Roughness Breathiness Blending

Pitch range Loudness Tension Pitch movement Pitch level Frontality

Timbre

(Continued)

Velocity Degree of fluidity

Expansion Angularity Fluidity

Irregularity (of individual movements

Direction Directedness

Force

Movement

156 A social semiotic theory of synaesthesia

Modulation (distribution of features) Differentiation

Regular/irregular

Differentiated/ homogeneous

Colour

Common quality

TABLE 8.2 (Continued)

Regularity (distribution of features) Differentiation

Shape

Timbre

Relief Regularity (distribution (distribution of features) of features) Orchestration/ Relief blending (distribution of types of unevenness)

Texture Regularity (distribution of features) Differentiation of movements

Movement

A social semiotic theory of synaesthesia 157

158

A social semiotic theory of synaesthesia

Such combinations can say something about the people who wear the clothes or choose to make the objects part of their world, for instance whether they like the “natural” and the “organic” with or without a touch of exuberance and artifice, or whether they prefer egalitarian blue jeans with or without a touch of refined elegance. Such “touches” can be expressed in and through all parameters. In musical orchestration, a high flute may “pierce through the rest of the orchestral texture,” a xylophone may add a “ping” to the woodwinds (Adler, 2016: 619), and a tremolo of violas may create “a subtle flavour to an otherwise singleminded melodic statement” (ibid: 612). Just what that accent is, what meaning it contributes to the whole, will then depend on the timbre and the way of playing, for instance the tense sound of the xylophone, or the vibrato of the violas. It is here that treatises on orchestration, however technical in every other respect, often rely on synaesthetic references such as “flavour,” “colour,” “brilliance,” “warmth,” and “luminosity” to express precisely what such accents achieve. The concept of perspective, of foreground, middle-ground and background suggests a second possible answer to Aristotle’s question, not least because they can be applied to visual as well as to aural media. In visuals, the foregroundbackground relation can take many forms – the landscape behind the people in an image, the plain paper on which a drawing is inked, the abstract or figurative backgrounds of printed words on book covers or PowerPoint slides, or the settings in which newsreaders read the news or in which we live our daily lives. In music (cf. Adler, 2016: 126), the foreground is often a melody. The middle ground then involves countermelodies or contrapuntal material, and the background a chordal accompaniment or an ostinato of short, repeated phrases. In contemporary dance music, on the other hand, it is often the rhythm section, rather than the melody, which is foregrounded. The middle ground may then be a keyboard playing sustained chords or repetitive patterns that alternate rather than progress, and the background faint snippets of melody or sound effects – for instance the phrase “Give me some love,” sung very softly and plaintively, in Love Corporation’s eponymous track. Such music positions us as dancers, participants rather than listeners. It foregrounds embodied movement rather than melodic “messages.” Film sound designers design soundtracks according to the same principles. According to Walter Murch, who was the sound designer on many of George Lucas’ and Francis Ford Coppola’s films (Weis and Belton, 1985: 357): The thing is to think of the sound in layers… separate out the backgrounds from the foregrounds, and the foregrounds from the mid-grounds, then you go out and record. Since each of these layers is separate, you can still control them, and you can emphasize certain elements and de-emphasize others the way an orchestrator might emphasize the strings versus the trombones or the tympani versus the woodwinds.

A social semiotic theory of synaesthesia 159

In music, it is of course also possible that all instruments play (or all voices sing) the same notes. Individual instruments or voices may then blend to different degrees, ranging from still being discernible in the whole to being fully blended and sounding like a single voice. Adler (ibid: 630) uses a section from the first movement of Brahms’ 3rd symphony as an example of the former: “Only the flutes explore the highest register and all others play in their middle and low registers which provides clear separation between high and low and beautifully luminous spacing.” The meaning potential of blending derives from what it lit­ erally is – a unification of voices. Depending on the context, this can then come to mean solidarity, consensus, a positive sense of joint experience and belonging – or conformity, strict disciplining, lack of individuality, and so on. All this can be applied, not only to combinations of the distinctive features of a single parameter, for instance timbre but also to the way different parameters are orchestrated in multimodal texts, artefacts and performances. So let me end with one further example, which will also serve as a more or less explicit demonstration of how such an analysis can be carried out. Figure 8.3 shows one of the three panels of “Stone Carvings,” a work by the typographic artist Jonathan Barnbrook. A first step in any multimodal analysis is to look for provenances, and there are provenances here: the image is readily recognized as a stone carving, and stone carvings evoke ancient monuments and tombstones – texts that are meant to last through the ages. That some of the carved words are crossed out may seem to contradict this, but early stone carvings in fact often crossed out mistakes (Barnbrook, personal communication). The text itself, however, is not such a monument. It is a work of experimental typography which imports stone carving, together with the associations that cling to it, as a new material resource for contemporary typography. The letter forms themselves, on the other hand, and the use of yellow on a stone carving, do not have clear provenances, and should therefore be analysed on the basis of the way the artist has creatively exploited their experiential meaning potential. The next step is to ascertain which parameters are involved (graphic shape, texture and colour, as it happens) and to analyse these one by one. The graphic shapes in Figure 8.3 include the letter forms, the arrows between the letters of the word “technology” (which come from the way letter spacing is indicated in books on stone carving, and here serve to increase “expansion”), and the lines that radiate from the “o” of the word “not,” which are formed by a piece of string that was part of the packaging of the stones when the artist received them (Barnbrook, personal communication). The values of the distinctive features are summarized in the table below (Table 8.3). Overall, the chiselled letter forms are strong and solid. This derives, not so much from boldness as from size and expansion. Their angularity and horizontal orientation results from the fact that they had to be chiselled in stone, rather than drawn. But they are also irregular, in part because some are crossed out, and in part

160

A social semiotic theory of synaesthesia

Stone carvings (Jonathan Barnbrook, 1998). Reproduced with kind permission from Jonathan Barnbrook

FIGURE 8.3

because mistakes (the upside down “2,” for instance), tests of the stone’s depth (at the top of the work) and chalk marks have been retained. All this suggests an attempt to avoid perfection, and to contrast the immediacy and improvisation of creation with the solidity and durability of stone. Size is another source of irre­ gularity, with small “grammatical words” like “a,” “is” and “not” especially large, which almost makes them messages in their own right, quite separate from the sentence “technology is nothing more than a process, not an end in itself.” The “a” could then signify that this is just a thought, not the definitive thought. The “is,” almost Zen-like, could signify that things “just are,” and the “not”

A social semiotic theory of synaesthesia 161 TABLE 8.3 The distinctive features of graphic shape in Barnbrook’s “Stone Carving”

Distinctive features

Value

Curvature Regularity

Angular Irregular: letters vary in size, weight, and are irregularly spaced, with some crossed out Medium, but some letters drawn with chalk Varying, with the words “a,” “not” and “is” especially large Disconnected, but with lines emanating from the “o” of “not” Horizontal Upright, except for the lines that emanate from the “o” Expanded

Weight Size Connection Orientation Slanting Expansion

makes “nothingness” a core from which what look like rays of light radiate, or alternatively, the point in which the lines of what looks like one-point perspective converge. The texture includes three elements, the stone, on which I have already commented, the paper, and the string. The table below summarizes the values of the various textural features, insofar as they can be ascertained from an image (Table 8.4): Clearly, the way Barnbrook uses the resources of texture harmonize with the way he uses the resources of typography, contrasting the solidity and rigidity of stone with the flimsiness of paper, and referring to the rays of light and the lines of one-point Renaissance perspective with the discardable material of packaging string. Colour is used in a limited, but nevertheless significant way, with the yellow standing out against the natural colours of the stone and the paper. The table below summarizes the distinctive features in play (none of the colours display transparency, luminosity or luminescence) (Table 8.5) Overall, these are the natural colours of stone and paper, and a good re­ production will show their modulated and impure surfaces. The yellow then accents the spacing between the letters of the word “technology,” so literally “de-constructing” it, and at the same time provides warmth and luminosity to the words “a,” “is,” and “not.” TABLE 8.4 The distinctive features of texture in Barnbrook’s “Stone Carving”

Distinctive features

Value: stone

Value: paper

Value: string

Relief Density Rigidity Roughness Regularity

Relief (letters carved) Dense Rigid Smooth Regular

No relief Dense Flexible Smooth Irregular edges

Some relief Dense Flexible Somewhat rough Regular

162

A social semiotic theory of synaesthesia

TABLE 8.5 The distinctive features of colour in Barnbrook’s “Stone Carving”

Distinctive features

Value grey (the stone)

Value: white (the paper)

Value: yellow

Value Saturation Purity Luminosity

dull unsaturated impure none

bright unsaturated impure none

Modulation

modulated

Somewhat modulated (incl. contrast between front and back)

bright saturated pure highlights in the word “is” not modulated

A final table summarizes the orchestration of the parameters (Table 8.6): The “principles of combination” I have introduced in this section may well work somewhat differently in different cases, but in this case an analysis based on foreground, background and accent quite clearly brings out the role and con­ tribution of the different parameters. “Stone Carvings” is a written text in which the graphic shapes of the letter forms, together with other (typo)graphical marks,

TABLE 8.6 Relations between the parameters in Barnbrook’s “Stone Carving”

Graphic shape Foreground: the message Includes a “linguistic message” (about technology) and a “typographic message,” namely the deconstruction of technology and the separate messages of the words “a,” “is” and “not” (the latter including the radiating lines) Harmony: The size of the typographic message harmonizes with the intensity of the colour Disharmony: The size of the linguistic message contrasts with size of the typographic message The radiation of the lines formed by the string contrasts with the linear directionality of the carvings Texture Background: materiality The background makes the messages at once eternal and ephemeral, improvised and carefully executed Disharmony: The materiality of the stone contrasts with that of the paper in terms of rigidity, regularity and relief Colour Accent: the typographic message The yellow colour makes the typographic message warm and luminous. Harmony: The colour of the typographic message harmonizes with its size (intensity) Disharmony: The colour of the background clashes with the colour of the accents in saturation, purity and modulation

A social semiotic theory of synaesthesia 163

form the foreground, the main message, the melody. The material on which the message is inscribed then forms the background for this message, giving it its unique and contradictory status as being at once eternal and ephemeral. Central to the design is the contrast between what, adapting Barthes’ “three messages” (1977: 33), I will call the linguistic message (the sentence “technology is nothing more than a process, not an end in itself”) and the typographic message, realized by the way the word “technology” is “deconstructed,” the typographic treatment of “a,” “is,” and “not,” and the marks of imperfection. This contrast involves cross-modal features such as intensity as well as the intra-parameter fea­ tures of colour saturation, purity and modulation, which, like Adler’s “piercing flute” and xylophone “ping,” provide the typographic message with its warmth and luminosity. The contrast between the linguistic message and the typographic message fi­ nally needs to be placed in context. Made public through exhibitions and books on experimental typography (e.g. Triggs, 2003), “Stone Carvings” is the work of a typographic artist who no longer sees typography as a self-effacing craft in the service of the word, but extends typography’s range of gestures, tools and mate­ rials, and hence the range of what it can express, so increasing its ability to play the role it is called upon to play in contemporary semiosis – identity design.

REFERENCES

Adler, S. (2016) The Study of Orchestration. 4th Edition. New York: W.W. Norton & Co. Aiello, G. and Dickinson, G. (2013) Beyond authenticity: A visual-material analysis of locality in the global redesign of Starbucks stores. Visual Communication 13(3): 303–322. Albers, J. (1975 [1963]) Interaction of Colour. New Haven, CT: Yale University Press. Alpers, S. (1983) The Art of Describing – Dutch Art in the Seventeenth Century. Harmondsworth: Penguin. Andersen. T.H., Boeriis, M., Maagerø, E., and Seip Tønnessen, E. (2015) Social Semiotics – Key Figures, New Directions. London: Routledge. Angelo, A. (2009) A Day in the Life of Ancient Rome. New York: Europa Editions. Angier, C. (2002) The Double Bond – Primo Levi. A Biography. Harmondsworth: Penguin. Anglepoise (2020) https://www.anglepoise.com/manifesto/the-long-view Appiah, K.A. (2018) The Lies that Bind – Rethinking Identity. London: Profile Books. Appolonio, U., ed. (2009) Futurist Manifestos. London: Tate Publishing. Aristotle (1954) The Rhetoric and the Poetics. New York: Random House. Aristotle (2008) De Anima. New York: Cosimo Classics. Armstrong, T. (2006) The Best Schools – How Human Development Research Should Inform Educational Practice. Alexandria, VA: ASCD Books. Arnheim, R. (1969) Visual Thinking. Berkeley and Los Angeles: University of California Press. Arnheim, R. (1974) Art and Visual Perception – A Psychology of the Creative Eye. Berkeley and Los Angeles: University of California Press. Arnheim, R. (1982) The Power of the Centre. Berkeley and Los Angeles: University of California Press. Austin, L. (2016) Elaborations of the Machine: The Automata Ballets. https:// modernismmodernity.org/articles/elaborations-machine-automata-ballets Bakhtin, M.M. (1987) Speech Genres & Other Late Essays. Austin: University of Texas Press. Barthes, A. (1973a) Le plaisir du texte. Paris: Éditions du Seuil. Barthes, R. (1973b) Mythologies. London: Paladin. Barthes, R. (1977) Image-Music-Text. London: Fontana. Baudrillard, J. (1996) The System of Objects. London: Verso.

References

165

Bauman, Z. (2011) From pilgrim to tourist – Or a short history of identity. in S. Hall and P. Du Gay, eds. Questions of Cultural Identity. London: Sage, pp. 18–36. Beckham, D. (2000) Beckham: My World. London: Hodder & Stoughton. Bell, P. and Van Leeuwen, T. (1994) The Media Interview – Confession, Contest, Conversation. Sydney: University of New South Wales Press. Bellantoni, J. and Woolman, M. (2000) Type in Motion – Innovations in Digital Graphics. London: Thames and Hudson. Benjamin, W. (1980) Illuminations. London: Fontana. Berger, J. (1972) Ways of Seeing. Harmondsworth: Penguin. Berger, P. (1963) Invitation to Sociology – A Humanistic Perspective. Harmondsworth: Pelican. Besnier, N. (2004) The social production of abjection: Desire and silencing among transgender tongans. Social Anthropology 12: 301–323. Bhabha, H. (1990) The third space in identity. In J. Rutherford, ed. Community Culture, Difference. London: Lawrence and Wishart, pp. 222–237. Billig, M. (1995) Banal Nationalism. London: Sage. Björkvall, A. (2014) Practical function and meaning: A case study of IKEA tables. In C. Jewitt, ed. The Routledge Handbook of Multimodal Analysis. London: Routledge, pp. 342–353. Bogatyrev, P. (1971 [1937]) The Function of Folk Costume in Moravian Slovakia. The Hague: Mouton. Bourdieu, P. (1979) Distinction: A Social Critique of the Judgement of Taste. London: Routledge and Kegan Paul. Brett, D. (2005) Rethinking Decoration: Pleasure and Ideology in the Visual Arts. London: Laurence King. Britten, B. (2020) Google doodles and collective memory-making. In S. Josephson, L. Kelly and K. Smith, eds. Handbook of Visual Communication: Theory, Methods and Media. New York: Routledge, pp. 334–339. Brosterman, N. (2014) Inventing Kindergarten. New York: Kaleidograph Design. Butler, J. (1990) Gender Trouble: Feminism and the Subversion of Identity. London: Routledge. Byrne, D. (2003) Envisioning Emotional Epistemological Information. New York: Steidl Pace-MacGill. Chaney, D. (1996) Lifestyle. London: Routledge. Coca Cola (2010) http://questions.coca-cola.com/NSREExtended.asp?WhatUserSaid= coca-cola+red&VRepTalk.x=12&VRepTalk.y=15 Cole, D. (2007) Patterns – New Surface Design. London: Laurence King Publishers. Color Marketing Group (2020) https://colormarketing.org/2020/03/01/color-alert-crown Completecraft (2010) https://www.completecaft.com/au/articles Connell, R. and Pearse, R. (2015) Gender in World Perspective. 3rd Edition. Cambridge: Polity. Cook, N. (1998) Analysing Musical Multimedia. Oxford: Oxford University Press. Cooke, D. (1959) The Language of Music. Oxford: Clarendon. Coupland, N. (2007) Style: Language Variation and Identity. Cambridge: Cambridge University Press. Crane, A.R. (2019) Mary of Warmun: A new Gija identity, a new Catholic identity. Unpublished paper. Cytowic, R.E. (1989) Synaesthesia: A Union of the Senses. New York: Springer Verlag. Cytowic, R.E. (1997) Synaesthesia: Phenomenology and neuropsychology. In S. BaronCohen and R. Harrison, eds. Synaesthesia – Classic and Contemporary Readings. Oxford: Blackwell, pp. 17–42.

166

References

Davies, W. (2015) The Happiness Industry – How the Government and Big Business Sold Us Well-Being. London: Verso. Davies, W. (2020) Nervous States: Democracy and the Decline of Reason. New York: W.W. Norton & Co. Davis, F. (1992) Fashion, Culture and Identity. Chicago: University of Chicago Press. De Vries, J. (2008) Newspaper design as cultural change. Visual Communication 7(1): 5–27. Djonov, E. and Van Leeuwen, T. (2011) The semiotics of texture: From tactile to visual. Visual Communication 10(4): 541–564. Djonov, E. and Van Leeuwen, T. (2013) Between the grid and composition: Layout in PowerPoint’s design and use. Semiotica 197: 1–34. Dondis, D.A. (1973) A Primer of Visual Literacy. Cambridge, MA: MIT Press. Durham, J. (2017) https://www.foundationforcontemporary.arts.org/recipients/jimmiedurham Durkheim, E. (1976 [1915]) The Elementary Forms of Religious Life. London: Allen and Unwin. Eco, U. ed. (2002) History of Beauty. New York: Rizzoli. Edlong (2020) The scientific art of authentic taste. https://www.edlong.com Egger (2020) https://www.egger.com/shop/en_AU/products/furniture-interior-design/ decorative-collection/feelwood Elkind, D. (1997) Play’s the thing. New York Times 7 September 1997. Elle (2020) https://www.elle.com/fashion/trend-reports/131006915/fashion-trends-2020 Eno, B. (1983) The studio as a compositional tool. Downbeat July 1983 (part 1) and August 1983 (part 2), unpaginated. http://music.hyperreal.org/artists/brian_eno/interviews/ downbeat79.htm Erikson, E. (1963) Childhood and Society. Harmondsworth: Penguin. Erikson, E. (1976) Toys and Reason. New York: W.W. Norton & Co. Eysenck, H.J. (1967) The Biological Basis of Personality. Springfield, IL: Thomas. Fairclough, N. (1993) Critical discourse analysis and the marketization of public discourse. Discourse and Society 4(2): 133–169. Feisner, E.A. (2006) Colour – How to Use Colour in Art and Design. London: Laurence King Publishing. Fielder, J. ed. (2006) Bauhaus. Königswinter: Tandem Verlag. Finlay, V. (2002) Colour – Travels Through the Paintbox. London: Hodder & Stoughton. Fischer-Jorgenson, E. (1966) Form and substance in glossematics. Acta Linguistica Hafniensia 10: 1–3. Floch, J.-M. (2000) Visual Identities. London: Continuum. Flügel, J.C. (1930) The Psychology of Clothes. London: Hogarth Press. Forlizzi, J., Lee, J., and Hudson, S. (1997) The kinedit system: Affective messages using dynamic texts. Proceedings of the CHI 2003 Conference on Human Factors in Computer Systems, pp. 377–384. Freud, S. (1984 [1920]) On Metapsychology. Harmondsworth: Penguin. Frow, J. (2014) Character & Person. Oxford: Oxford University Press. Fukuyama, F. (2019) Identity – Contemporary identity Politics and the Struggle for Recognition. London: Profile Books. Gage, J. (1993) Colour and Culture – Practice and Meaning from Antiquity to Abstraction. London: Thames and Hudson. Gage, J. (1999) Colour and Meaning – Art, Science and Symbolism. London: Thames and Hudson.

References

167

Gaskin, R. (2012) Sweating Bullets – Notes about Inventing PowerPoint. San Francisco: Vinland Books. Geertz, C. (1983) Local Knowledge – Further Essays in Interpretive Anthropology. New York: Basic Books. Gibson, J.J. (1986) The Ecological Approach to Visual Perception. Hillsdale, NJ: Lawrence Erlbaum. Godin, S. (2008) Tribes We Need You to Lead Us. New York: Portfolio. Goethe, J. W. von (1970 [1810]) Theory of Colours. Cambridge MA: MIT Press. Goffman, E. (1959) The Presentation of Self in Everyday Life. Harmondsworth: Penguin. Goffman, E. (1976) Gender Advertisements. London: Macmillan. Goffman, E. (1981) Forms of Talk. Oxford: Basil Blackwell. Graakjaer, N. J. (2012) Dance in the store: On the use and production of music in Abercrombie & Fitch. Critical Discourse Studies 9(4): 393–406. Gray, C. (1962) The Russian Experiment in Art 1863–1922. London: Thames and Hudson. Gumtree (2020) https://www.gumtree.com.au/s-melbourne/smooth+skin+beauty/k0i3001317 Habermas, J. (1976) Legitimation Crisis. London: Heinemann. Habermas, J. (1984) The Theory of Communicative Action. Vol. I. Boston: Beacon Press. Hall, E.T. (1966) The Hidden Dimension. New York: Doubleday. Hall, S. and Whannel. P. (1964) The Popular Arts. London: Hutchinson Educational. Halliday, M.A.K. (1978) Language as Social Semiotic. London: Arnold. Halliday, M.A.K. (1985) An Introduction to Functional Grammar. London: Arnold. Han, J. (2021) A social semiotic account of music-movement correspondences. Unpublished PhD thesis, University of New South Wales. Harrison, J.E. and Baron-Cohen, S. (1997) Synaesthesia: An introduction. In S. Baron-Cohen and J. Harrison, eds. Synaesthesia – Classic and Contemporary Readings. Oxford: Blackwell, pp. 3–16. Haverkamp, M. (2013) Synesthetic Design. Berlin: Birkhäuser. He, Y. and Van Leeuwen, T. (2019) Animation and the remediation of school physics – A social semiotic approach. Social Semiotics 30(5): 665–684. Herder, J.G. von (2002 [1772]) Treatise on the Origin of Language. Philosophical Writings. Cambridge: Cambridge University Press. Hildum, D.C. and Brown, R.W. (1956) Verbal Reinforcement and Interviewer Bias. Journal of Abnormal and Social Psychology 53(1): 108. Hjelmslev, L. (1961) Prolegomena to a Theory of Language. Madison: University of Wisconsin Press. Hobsbawn, E. and Ranger, T. eds. (1983) The Invention of Tradition. Cambridge: Cambridge University Press. Höllerer, M., Van Leeuwen, T., Jancsary, D., Meyer, R.E., Andersen, T.H., and Vaara, E. (2019) Visual and Multimodal Research in Organization and Management Studies. London: Routledge. Horton, O. (2005) Absolute luxury with a twist. The Age 30 April 2005. Hulten, P. (1975) Jean Tinguely: Meta. Boston, MA: New York Graphic Society. Huxtable, M.J. (2006) The medieval gaze at grips with the medieval world. In C.P. Biggam and C.J. Kay, eds. Progress in Colour Studies. Vol I. Language and Culture. Amsterdam: John Benjamins, pp. 199–217. Hymes, D. (1972) Models of the interaction of language and social life. In J.J. Gumperz and D. Hymes, eds. Directions in Sociolinguistics: The Ethnography of Communication. New York: Holt, Rinehart and Winston, pp. 35–71.

168

References

Ito, M. (2009) Engineering Play – A Cultural History of Children’s Software. Cambridge, MA: MIT Press. Itten, J. (1970) The Elements of Color. New York: Van Nostrand Reinhold. Jackson, D.M. (2003) Sonic Branding – An Introduction. London: Palgrave Macmillan. Jaffé, H.L.C. (1967) De Stijl 1917–1931 – Visions of Utopia. Oxford: Phaidon. Jakobson, R. (1960) Closing statement: Linguistic and poetics. In T.A. Sebeok, ed. Style in Language. Cambridge, MA: MIT Press, pp. 3350–3377. Jakobson, R. and Halle, M. (1956) Fundamentals of Language. Mouton: The Hague. Jennings, R. (2015) Google alphabetized new logo is childish (who moved my cheese?) https://www.computerworld.com/article/2979573/google-logo-itbwcw.html Johannessen, C.M. (2017) The challenge of simple graphics for multimodal studies: Articulation and time scales. Visual Communication 17(2): 163–185. Johannessen, C.M. and Van Leeuwen, T. (2018) (Ir)Regularity. In C.M. Johannessen and T. van Leeuwen, eds. The Materiality of Writing – A Trace-Making Perspective. London: Routledge, pp. 175–192. Johannessen, C.M. and Van Leeuwen, T., eds. (2018) The Materiality of Writing – A TraceMaking Perspective. London: Routledge. Jones, O. (2006 [1856]) The Grammar of Ornament. Princeton, NJ: Princeton University Press. Jurgensen, J. (2012) Making an Impression in Just Four Notes. Wall Street Journal, 28 January. Kagan, J. (1997) Galen’s Prophecy: Temperament in Human Nature. Boulder, CO: Westfield Press. Kandinsky, W. (1977 [1914]) Concerning the Spiritual in Art. New York: Dover Publications. Kane, J. (2002) A Type Primer. London: Laurence King. Kapica, J. (2018) The secret language of fonts. The Globe and Mail. www.theglobeandmail.com/technology/the-secret-language-of-fonts/article25705684 Kepes, G. (1965) The Nature and Art of Motion. New York: George Braziller. Kessler, S. (2014) The next (sound) wave in branding: How AT&T created its new sonic identity. http:///www.fastcodedesign.com/1671164/the-next-sound-wave-in-brandinghow-att-created-its-new-sonic-identity Kim-Cohen, S. (2016) Against Ambience and Other Essays. London: Bloomsbury. Knight, S. (2018) The spectacular power of big lens. The Guardian May 10, 2018. https:// www.theguardian.com/news/2018/may/10/the-invisible-power-of-big-glasses-eyewearindustry-essilor-luxottica Koller, V. (2008) ‘Not just a colour’: Pink as a gender and sexuality marker in visual communication. Visual Communication 7(4): 395–423. Koolhaas, R., Foster, N., and Mendini, A. (2001) Colours. Blaricum: V&K Publishing. Kowalski, K. (2020) Slow living 101P: What is slow living? https://www.sloww.co/slowliving-101/ Kress, G. (2011) Multimodality – A social semiotic approach to contemporary communication. London: Routledge. Kress, G. and van Leeuwen, T. (2001) Multimodal Discourse - The Modes and Media of Contemporary Communication. London: Arnold. Kress, G. and Van Leeuwen, T. (1996) Reading Images – The Grammar of Visual Design, London: Routledge. Kress, G. and Van Leeuwen, T. (2006) Reading Images – The Grammar of Visual Design. 2nd Edition. London: Routledge. Kretschmer, E. (1921) Körperbau und Charakter. Berlin: Springer.

References

169

Kristeva, J. (1980) Desire in Language: A Semiotic Approach to Language and Art. Oxford: Blackwell. Kvåle, G. (2016) Software as ideology: A multimodal critical discourse analysis of Microsoft Word and SmartArt. Journal of Language and Politics 15(3): 259–273. Lacy, M.L. (1996) The Power of Colour to Heal the Environment. London: Rainbow Bridge Publications. Lakoff, G. and Johnson, M. (1980) Metaphors We Live By. Chicago: Chicago University Press. Laver, J. (1980) The Phonetic Description of Voice Quality. Cambridge: Cambridge University Press. Leão, G. (2012) Movement in film titles: An analytical approach. Unpublished PhD, University of Technology, Sydney. Leaper, H. (2016) Opened by Customs 1937–8. https://www.tate.org.uk/schwittersopened-by-customs-t00214 Leech, G. (1969) A Linguistic Guide to English Poetry. London: Longman. Le Corbusier. (1987 [1925]) The Decorative Art of Today. Cambridge, MA: MIT Press. Lévi-Strauss, C. (1962) The Savage Mind. London: Weidenfeld and Nicolson. Lévi-Strauss, C. (1963) Structural Anthropology. Harmondsworth: Penguin. Locke, J. (1972 [1706]) An essay concerning human understanding. Vol II. New York: Dutton. Lomax, A. (1968) Folk Song Style and Culture. New Brunswick NJ: Transaction Books. Longacre, R.E. (1974) Narrative and other discourse genre. In R.M. Brend, ed. Advances in Tagmemics. Amsterdam: North Holland Publishing Co., pp. 357–376. Lupton, E. (2004) Thinking with type – A critical guide for designers, writers and students. New York: Princeton Architectural Press. Lynch, D.S. (1998) The Economy of Character: Novels, Market Culture and the Business of Inner Meaning. Chicago: University of Chicago Press. Machin, D. (2004) Building the world’s visual language: The increasing global importance of image banks in corporate media’ Visual Communication 3(3): 316–336. Machin, D. and Van Leeuwen, T. (2003) Global schemas and local discourses in Cosmopolitan. Journal of Sociolinguistics 7(4): 493–513. Machin, D. and Van Leeuwen, T. (2004) Global media: Generic homogeneity and discursive diversity. Continuum 18(1): 99–120. Machin, D. and Van Leeuwen, T. (2005) Language style and lifestyle: The case of a global magazine. Media, Culture and Society 27(4): 577–600. Machin, D. and Van Leeuwen, T. (2007) Global Media Discourse – A critical introduction. London: Routledge. MacIntyre, A. (1981) After Virtue – A Study of Moral Theory. 2nd Edition. London: Duckworth. Man Made Music (2016) www.manmademusic.com Mannheim, K. (1960 [1940]) Man and Society in an Age of Reconstruction. London: Routledge and Kegan Paul. Manzini, E. (1990) And of plastic? In P. Sparke, ed. The Plastics Age: From Modernity to Post-Modernity. London: Victoria and Albert Museum. Marks, L.E. (1997) On colored-hearing synaesthesia: Cross-modal translations of sensory dimensions. In S. Baron-Cohen and J.E. Harrison, eds. Synaesthesia – Classic and Contemporary Readings. Oxford: Blackwell, pp. 49–9. Mathieson, C. (2020) Scott Morrison’s mastery of the TV interview means something has to change. Sydney Morning Herald 22 January 2020. Matisse, H. (1972) Matisse on Art. London: Phaidon.

170

References

McCormack, D. (2008) Engineering effective atmospheres on the moving geographies of the 1897 Andre Expedition. Cultural Geographies 25: 413–430. McKinnon, H. (2019) Community – Salad Recipes from Arthur Street Kitchen. Sydney: Pan Macmillan Australia. McLean, R. (2000) The Thames and Hudson Manual of Typography. London: Thames and Hudson. McQuail, D. and Windahl, S. (1993) Communication Models. London: Longman. Mead, G.H. (1967) Mind, self and society – From the standpoint of a social behaviorist. Chicago: University of Chicago Press. Meikl, J.L. (1990) Plastics in the American Machine Age. In P. Sparke, ed. The Plastics Age: From Modernity to Postmodernity. London: Victoria and Albert Museum. Mercedes-Benz (2020) https://www.mirru.com/alubeam_chrome_paint_mrercedes_benz_ cl-65.htm Mitchell, A. (1978) Consumer Values: A Typology. Menlo Park, CA: Stanford Research Institute. Montessori Australia (2020) https://montessori.org.au/product/rough-and-smooth-boards-set Mora, C. (2009) Colour in Fashion. Singapore: Maomao Publications. Morris, C. (1938) Writings on the General Theory of Signs. The Hague: Mouton. Morris, W. (1895) Some Hints on Pattern Design. London: Longman. Moszynska, A. (1990) Abstract Art. London: Thames and Hudson. Mukařovský, J. (1964) Standard language and poetic language. In P.L. Garvin, ed. A Prague School Reader in Esthetics, Literary Structure and Style. Washington DC: Georgetown University Press, pp. 17–30. Mulder, J. (2013) Making things louder: Amplified music and multimodality. Unpublished PhD thesis, University of Technology Sydney. Mulder, J. and Van Leeuwen, T. (2019) Speech, sound, technology. In M. GrimshawAagaard, M. Walther-Hansen and M. Knakkergaard, eds. The Oxford Handbook of Sound and Imagination Vol I, New York: Oxford University Press, pp. 473–492. Mumford, L. (1934) Technics and Civilization. New York: Harcourt, Brace and World. Munro, P. (2015) Transgendered Tongan ‘leitis’ finding their way in the conservative country. Sydney Morning Herald 25 August 2015. MyFonts (2016) Kiln – Webfont and Desktop font. www.myfonts.com/fonts/yellowdesign/kiln/ Nederveen Pieterse, J. (1992) White on Black: Images of Africa and Blacks in Western Popular Culture. New Haven: Yale University Press. Neuenschwander, B. (1993) Letterwork – Creative Letterforms in Graphic Design. London: Phaidon. Niebur, L. (2010) Special Sound: The Creation and Legacy of the BBC Radiophonic Workshop. Oxford: Oxford University Press. Nowacek, N. (2005) Character to Character. Visual Communication 4(2): 158–165 Ormerod, F. and Ivanič, R. (2002) Materiality in children’s meaning-making practices. Visual Communication 1(1): 65–91. Paget, I. (2019) Designing the Google Logo – An interview with Ruth Kedar. https:// logogeek.uk/podcast/ruth-kedar Palmer, M. (1998) What makes news. In O. Boyd-Barrett, and T. Rantanen, eds. The Globalization of News. London: Sage. Panofsky, E. (1970) Meaning in the Visual Arts. Harmondsworth: Penguin. Park, R.E. (1950) Race and Culture. Glencoe, IL: The Free Press.

References

171

Pastoureau. M. (1991) The Devil’s Cloth – A History of Stripes and Striped Fabric. New York: Columbia University Press. Pastoureau, M. (2001) Blue – The History of a Colour. Princeton, NJ: Princeton University Press. Pastoureau, M. (2008) Black – The History of a Colour. Princeton NJ: Princeton University Press. Pickford, R.W. (1972) Psychology and Visual Aesthetics. London: Hutchinson Rafaeli, A. and Pratt, M.G. (1993) Tailored meanings: On the meaning and impact of organizational dress. Academy of Management Review 18(1): 32–55. Ravelli, L. and McMurtrie, R.J. (2016) Multimodality in the Built Environment – Spatial Discourse Analysis. London: Routledge. Riley, C.A. (1995) Color codes – Modern theories of color in philosophy, painting and architecture, literature, music and psychology. Hanover: University Press of New England. Riley, J.W. and Riley, M.W. (1959) Mass communication and the social system. In R.K. Merton, L. Broom and S. Cottrell eds. Sociology Today. New York: Basic Books, pp. 537–578. Sachdev, I. (2020) Understanding texture and its importance in fashion. https:// iyoyisahcdeviyer.wordpress.com/2014/02/06/understanding-texture-and-itsimportance-in-fashion Sachs, C. (1937) World History of the Dance. New York: W.W. Norton & Co. Samara, T. (2002) Making and Breaking the Grid: A Graphic Design Layout Workshop. Beverly, MA: Rockport Publishers. Sassoon, R. (1999) Handwriting of the Twentieth Century. London: Routledge. Saussure, F. de (1974) Course in General Linguistics. London: Peter Owen. Schafer, R.M. (1977) The Tuning of the World. Toronto: McClelland and Stewart. Schafer, R.M. (1986) The Thinking Ear. Toronto: Arcana Editions. Schipp, D. (2016) Are you a Peter Pan? The new tribes of the property market. http:// Propertywomen.com.au/new-tribes. Scott-Taylor, J. (1935) A Simple Explanation of the Ostwald Colour System. London: Winsor and Newton. Shannon, C. and Weaver, W. (1949) The Mathematical Theory of Communication. Urbana: University of Illinois Press. Slow Living (2020) https://vanillapapers.net/2019/02/20/slow-living-blog-quotes-trends Smith, C. (2005) Photoshop and Dreamweaver Interaction – Creating High-Impact Web Pages. New York: McGraw-Hilll/Osborn. Sontag, S. (1977) On Photography. Harmondsworth: Penguin. Stenglin, M. (2009) Space odyssey: Towards a social semiotic model of three-dimensional space. Visual Communication 8(3): 35–64. Tagg, P. (2000) Melody and Accompaniment. https://tagg.org/articles/melodaccUS.pdf Taylor, C. (1989) Sources of the Self – The Making of Modern Identity. Cambridge: Cambridge University Press. Teige, K. (1974 [1929]) The Mundaneum. In P. Serenyi, ed. Le Corbusier in Perspective. Englewood Cliffs, New Jersey: Prentice-Hall. THAI (2010) Public Information Centre: Thai Logo. https://www.thaiair.info/tips/h1 013e.htm Thompson, P. and Davenport, P. (1982) The Dictionary of Visual Language. Harmondsworth: Penguin. Thomson, A. (2019) Color Marketing Group Reveals 4 Global Key Colors for 2020. https://www.interiorsandsources.com.article-details/articleid/22771/title/4-globalkey-colors-2020

172

References

Tönnies, F. (2001 [1887]) Community and Civil Society. Cambridge: Cambridge University Press. Trendstop (2020) https://www.trendstop.com/portfolio/materials-trend Triggs, T. (2003) Type Design: Radical Innovations and Experimentation. New York: Harper Collins. Tschichold, J. (1998) The New Typography. Berkeley and Los Angeles: University of California Press. Tufte, E.R. (2006) The Cognitive Style of PowerPoint: Pitching Out Corrupts Within. Cheshire, Connecticut: Graphics Press LLC. Turkle, S. (2011) Alone Together – Why We Expect More from Technology and Less from Each Other. New York: Basic Books. Van Campen, C. (2008) The Hidden Sense – Synaesthesia in Art and Science. Cambridge, MA: MIT Press. Van Ginkel, R. (1991) Typisch Nederlands … Ruth Benedict over het ‘nationaal karakter’ van Nederlanders. Amsterdams Sociologisch Tijdschrift 18(2): 37–62. Van Gogh, V. (1978) The Complete Letters of Vincent van Gogh. Vol III. London: Thames and Hudson. Van Leeuwen, T. (1999) Speech, Music, Sound. Basingstoke: Palgrave Macmillan. Van Leeuwen, T. (2005) Introducing Social Semiotics. London: Routledge. Van Leeuwen, T. (2006) Towards a semiotics of typography. Information Design Journal 14(2): 139–155. Van Leeuwen, T. (2011a) The semiotics of decoration. In K. O’Halloran and B. Smith, eds. Multimodal Studies – Exploring Issues and Domains. London: Routledge, pp. 115–130. Van Leeuwen, T. (2011b) The Language of Colour – An Introduction. London: Routledge. Van Leeuwen, T. (2014) Parametric systems: The case of voice quality. In C. Jewitt, ed. The Routledge Handbook of Multimodal Analysis. 2nd Edition. London: Routledge, pp. 76–84. Van Leeuwen, T. (2015) Looking good: Aesthetics, multimodality and literacy studies. In J. Rowsell and K. Pahl, eds. The Routledge Handbook of Literacy Studies. London: Routledge, pp. 426–439. Van Leeuwen, T. (2016) Creativity in the fourth dimension – The grammar of movement according to Jean Tinguely. In R. Jones, ed. The Routledge Handbook of Language and Creativity. London: Routledge, pp. 336–352. Van Leeuwen, T. (2017) Sonic logos. In Way, L.C.S. and McKerrell, S. eds. Music as Multimodal Discourse. London: Bloomsbury, pp. 119–134. Van Leeuwen, T. and Caldas-Coulthard, C.R. (2004) The Semiotics of Kinetic Design. In D. Banks, ed. Text and Texture: Systemic-Functional Viewpoints on the Nature and Structure of Text. Paris: L’Harmattan, pp. 355–380. Van Leeuwen, T., Bateson, D., Le Hunte, B., Barratt, A., Black, K.I., Kelly, M., Inoue, K., Rutherford, A., Stuart, M., and Richters, J. (2016) Contraceptive Advertising – A critical multimodal analysis. Journal of Applied Linguistics and Professional Practice 13(1-3): 321–342. Van Leeuwen, T., Djonov, E., and O’Halloran, K. (2013) “David Byrne really does love PowerPoint”: Art as research on semiotics and semiotic technology. Social Semiotics 3(3): 409–423. Van Leeuwen, T. and Djonov, E. (2015) Notes towards a semiotics of kinetic typography. Social Semiotics 25(2): 244–253. Van Leeuwen, T. and Jewitt, C. (2001) Handbook of Visual Analysis. London: Sage.

References

173

Von Hornbostel, E.M. (1931) Über Geruchtshelligkeit. Pflügers Archiv für die Gesamte Physiologie 227: 517–538. Von Sturmer, J. (1981) Talking with Aborigines. Australian Institute of Aboriginal Studies Newsletter 15: 13–30. Ward, R. (1958) The Australian Legend. Melbourne: Oxford University Press. Weber, M. (1947) The Theory of Social and Economic Organization. New York: The Free Press. Weber, M. (1976 [1930]) The Protestant Ethic and the Spirit of Capitalism. London: Allen and Unwin. Weis, E. and Belton, J., eds. (1985) Film Sound: Theory and Practice. New York: Columbia University Press. Wierzbicka, A. (1996) Semantics – Primes and Universals. Oxford: Oxford University Press. Williams-Jones, P. (1975) Afro-American Gospel Music: A Crystallization of the Black Aesthetic. Ethnomusicology 1975: 373–385. Wodak, R., de Cillia, R., Reisigl, M., and Liebhart, K. (2009) The Discursive Construction of National Identity. 2nd Edition. Edinburgh: Edinburgh University Press. Wolf Architecture (2020) https://wolfarchitectscom.au/texture-in-architecture Zablocki, B.D. and Kanter, R.M. (1976) The differentiation of lifestyles. Annual Review of Sociology 7:269–298. Zijderveld, A.C. (1979) On Clichés – The Supersedure of Meaning by Function in Modernity. London: Routledge and Kegan Paul. Zuboff, S. (2019) The Age of Surveillance Capitalism. London: Profile Books.

INDEX

Aboriginal identity 6, 9–10, 99 Acapela 36 accentuation 93, 94, 141, 155–158 Adler, S. 141, 159, 163 advertising 36, 48, 50, 75, 109, 152 aesthetics 28, 35, 36–38, 59, 61, 66, 70, 127 affinities of distinctive features 140, 141, 152–155 affordance 44–45 Aiello, G. 65, 104 Aksu, B. 85, 92, 93 Albers, J. 93 Alpers, S. 114 Also Sprach Zarathustra 142 Altman, R. 31 ambient meaning 65 ambient sound 150 Ancient Greek identity 11 Andersen, T.H. 52 Angela, A. 79 Angier, C. 15, 17, 19 Anglepoise 122 Angold, S. 68, 69 angularity 57–58, 62–63, 128–129, 154–155 animation 119, 123–126, 129, 130–133, 142–144 Antheil, G. 101 appearance (in impression management) 20 Appiah, K.A. 5 Appollonio, U. 13 architecture 37, 38, 93–94, 97, 109 Aristotle 42, 145 Armstrong, T. 103 Arnheim, R. 1, 57, 92 articulation 134, 148–149 artificial intelligence 52–53 Aspel, M. 15 Austin, L. 129 Auto-Mobile 137

Bacall, L. 147, 149 background 158, 162–163 Bakhtin, M.M. 27 Bardot, B. 70 Barnbrook. J. 159–163 Baron Cohen, S. 140, 152 Barthes, R. 2, 36, 113, 163 Baudrillard, J. 113 Bauhaus 37, 61, 63, 102, 107, 111, 144 Bauman, Z. 5 Beckermann, J. 142 Beckham, D. 21 Beethoven, L. von 142 Bell, P. 15 Bellantoni, J. 45, 118 Belton, J. 158 Benedict, R. 16 Berger, J. 48 Berger, P. 11, 18, 19 Besnier, N. 10 Bhabha, H. 22 biographies 15 biomechanics 117 Björkvall, A. 111–112 Blaue Reiter blending 159 Boccioni, U. 100, 117 Bogart, H. 70 Bogatyrev, P. 8 Bourdieu, P. 70 Brahms, J. 159 Brando, M. 149 breathiness 148 Brett, D. 37, 61 brightness 152 Britten, B. 142 Brosterman, N. 103 Brown, R.W. 149 Brushes Redux 61

Index

Butler, J. 22 Byrne, D. 129–130 Caldas-Coulthard, C.R. 121, 138 Calder, A. 138 Cawardine, G. 122 character 11, 13–15, 80, 81 Charles, R. 149 childhood and identity 14–15 Christian identity 11–12 circles 58–59 cliché 50 Cohen, L. 148 Cole, D. 65, 68, 75, 76, 105 colour and culture 78, 91–92 colour and identity 13 colour codes 77 colour cohesion 78 colour harmony 92–93 Colour Marketing Group 36, 82 colour schemes 88, 90–94 communication models 56–57, 58 comparative image analysis 50 connection 67–69, 154 Connell, R. 22 consistency: of individual identity 14; of texture 109–111 constructivism 57, 102 contextualization 49–51, 66, 70, 142, 144 Cook, N. 48 Cooke, D. 128 Coppélia 129 Cosmopolitan magazine 25, 26–28 Coupland, N. 38–39, 40 Courlander, H. 148 Crane, A. 9, 50 curvature 57–58, 62–63, 128–129, 154–155 Cuvier, G. 50 Cyclograveur 137, 138 Cytowic, R.E. 145 Dada 101 dance 119–120, 127, 129 Davenport, P. 57, 58 Davies, W. 22 Davis, F. 13, 83 Davis, M. 150 The Day of the Triffids 107 (de)constructability 134–135 De Kooning, W. 109 density 69–70, 108 De Niro, R. 8 Der Gelbe Klang 144 De Saint Phalle, N. 110

175

De Stijl 57 De Vries, J. 78 Dickinson, G. 65, 104 differentiation 89–90, 153–154 Dion, C. 148 Dionysius the Areopagite 87 directedness 126–127 direction of movement 126–127 disconnection; see connection Disney, W. 12 Dissecting Machine 138 distinctive features 2, 45, 61–62, 73, 83–90, 105–114, 123–130, 140, 146–151; see also affinities; dynamization; principles of combination distribution media. see media Djonov, E. 3, 28, 31, 106, 138, 154 Dondis, D.A. 57, 58 dress 8–9, 13, 14, 19, 21–22, 40–41, 78, 79–81, 82, 85, 87, 104, 105–106, 108, 109 Duchamp, M. 117 Durham, J. 110 Durkheim, E. 7 Dwiggins, W.A. 59 Dylan, B. 149 dynamization 118, 119, 130, 133 early childhood education 102–103 Eco, U. 87 Edlong 98, 106, 109 Edwards, C. 75–76, 95–96 Eggers 107 Eliasson, O. 65 Elkind, D. 102 El Lissitzky 58 energy 152 Eno, B. 54, 65 ergonomic design 24 Erikson, E. 11, 41 ethnography 50 expansion 70–71, 127–128, 154 experiential meaning potential 42–47, 61–73, 82, 83, 98, 119, 123 Eysenck, H. 12 Facebook 22 facial identity markings 7–8 Fairclough, N. 38 Fairy Tale Relief 155 Faithfull, M. 148 The Fantastic Paradise 110 fashion design. see dress Feisner, E.A. 116

176

Index

Fielder, J. 60, 61, 102 Finlay, V. 99 flexibility 134 Floch, J.-M. 14, 17 fluidity 129 Ford, J. 29 Ford-Coppola, F. 158 foreground 158, 162–163 Forlizzi, J. 118 force 128 Foster, N. 93 Frankl, P. 113 Freud, S. 65, 103, 146 Fröbel, F. 102 Frow, J. 11, 12, 13 Fukuyama, F. 12 functional design 3, 24–28, 36–38, 59, 77–79 functionalism 34–39 furniture design 111–112 futurism 13, 81, 85, 100, 101, 117 Gabo, N. 117 Gage, J. 80, 81, 85, 99 Gan, A. 102 Gaskin, R. 31 Geertz, C. 11 gender identity 10–11, 113, 121–122 genericity 28–34 geometric figures. see shape Gershwin, G. 100 Getz, S. 150 Giacometti, A. 107, 109 Gibson, J.J. 44–45 The Girl Effect 130, 132 The Godfather 149 Godin, S. 40 Goethe, J.W. von 13, 81, 88 Goffman, E. 18, 19–20, 38, 53, 54, 82, 122 Goldman, D. 84 Google 142–144 Graakjaer, N. 65 gradation 2, 46, 66 Gray, C. 102 grids 28 Gropius, W. 61 Habermas, J. 34, 51 hair styles 8, 9 Hall, E.T. 147 Hall, S. 22, 29 Halle, M. 2, 46 Halliday, M.A.K. 1, 35, 38, 55, 127, 132 Han, J. 3, 119, 123, 126, 140, 141

handwriting 17–18, 44, 67, 68; see also typography harmony 142, 155 Harrison, J.E. 140, 152 Haverkamp, M. 144 He, Y. 3, 126 Herder, J.G. von 145 Hicks, A. 93 Hildum, D.C. 149 Hippocrates 12 Hjelmslev, L. 97 Hobsbawn, E. 16 Hörl, E. 9, 49 Horton, O. 105 Hulten, P. 137 Hundertwasser, F. 109 Huxtable, M.J. 80 Hymes, D. 38 iconography 49–51 identity 1, 3, 5–6, 23, 41. see also individual identity; lifestyle identity; role identity; social identity identity design 24–28, 79–83 impression management 20 individual identity and its semiotic expression 6, 11–18, 23, 41 Inferno 137 intensity 152 interior design 81, 89–90, 93–94, 97, 104, 107 interpretation 50–51 irregularity. see regularity Ito, M. 134 Itten J. 87, 92, 102, 107 Ivanic, R. 103 Jackson, D.M. 121 Jackson, M. 119, 129 Jaffé, H.L.C. 57, 60 Jagger, M. 119 Jakobson, R. 2, 36, 46 jealousy 1 137 Jennings, R. 144 jewellery 111 Jobs, S. 62 Johannessen, C. 3, 21, 44, 50, 64 Johnson, M. 2, 43, 44, 127 Jones, O. 37 Jurgensen, J. 142 Kagan, J. 12 Kandinsky, W. 58, 60, 63, 144 Kane, J. 28

Index

Kapica, J. 60 Kedar, R. 144 Kepes, G. 117 Kepler, J. 57 Kessler, S. 142 Kim-Cohen, S. 65 kinetic typography 118 Klamauk 137 Knight, S. 25 Koolhaas, R. 81, 85, 87, 88, 93 Kress, G. 1, 2, 35, 44, 46, 52, 55, 97, 113, 126, 133 Kretschmer, E. 12 Kristeva, J. 36 Kubrick, S. 25 Kvåle, G. 31 Lacy, M.L. 81 Lakoff, G. 2, 43, 44, 127 Laver, J. 148 The Lawyer 19 Leão, G. 3, 120 Leaper, H. 100 Le Corbusier 37, 61, 93–94 Leech, G. 36 Lego 135 leitis 10–11 Lemke, J. 52 Lennon, J. 25 Leonore Overture 142 Lévi-Strauss, C. 8 lifestyle identity and its semiotic expression 3, 6, 20–22, 23, 24–25, 36, 40, 42, 78, 82–83 “Like a Virgin” 43 lines 62, 67, 119, 120, 123 liquidity 106 Little Girl with Honeycomb Hat 109 Locke, J. 140 logos 50–51, 66, 72, 74, 75, 83, 120–121, 128, 142, 152 Lomax, A. 43 Longacre, R.E. 75 Loos, A. 37, 60 loudness 139, 147 Love Corporation 158 luminescence 86–87 luminosity 86 Lunghi, C. 15 Lupton, E. 59, 77 lustre 87 Luxicotta 25 Lye, L. 18 Lynch, D.S. 14

177

Machin, D. 25, 37, 79 MacIntyre, A. 11, 14 Madonna 15, 149 Malevich, K. 37 manner 20 Mannheim, K. 34 Man Pointing 107, 109 Manzini, E. 113, 114 Maori identity 7–8 Maranar 137 marketing and lifestyle 21, 22, 40–41, 65 Marks, L.E. 145 The Marriage of Figaro 48 materiality 2, 43, 45–46, 97–100, 113–114, 139,140, 150 Mathieson, C. 20 Matisse, H. 86, 88, 90 McCormack, D. 65 McKinnon, H. 107 McLaren, M. 118 McLean, R. 59 McLuhan, M. 11 McQuail, D. 55 McQueen, A. 105 Mead, G.H. 18 meaning potential 41, 43, 44, 46, 49, 61, 66, 105, 139, 140 medium 2, 52–54, 98, 99, 113, 139 Meikl, J.L. 113 Mendini, A. 86, 88 Meta-Malevich 137 metaphor 42–43, 44, 57, 127 Meyerhold, V. 117 Middlemarch 89 Miró, J. 109 Mitchell, A. 22 mobility 117, 121–123 mode 2, 51–52, 98, 99, 100, 113, 139 modulation 88 Moholo-Nagy, L. 102 Mondrian, P. 57, 60, 90 Monroe, M. 25, 147, 149 Montessori, M. 103 Mora, C. 9, 80, 85, 86, 92, 93, 155 Morris, C. 36 Morris, W. 37, 69, 76 Morrison, S. 17, 20 Moszynska, A. 3 movement 117–121 Mozart, A. 48 Mukařovský, J. 35, 36, 38, 48 Mulder, J. 3, 52, 150 Mumford, L. 36, 118 Mung Mung, G. 9

178

Index

Munro, P. 10, 11 Munsell, A. 99 music 46, 48, 54, 66, 108–109, 120, 128, 140, 142, 146, 158. see also orchestration Murch, W. 158 Muybridge, E. 129 NASA 31 national identity 16–18, 23 Native American identity 8 Nederveen Pieterse, J. 49–50 Neuenschwander, B. 60 newspaper design 78 new typography 163 Niebur, L. 150 The Night Café 90 novels and identity 13, 14–15, 29 Nowacek, N. 60 Oldenburgh, C. 108 orchestration 141–142, 155–159 org chart 32, 62 orientation 70–72 Ormerod, F. 103 Ostwald, W. 92 Paget, I. 144 Palmer, M. 37 Panofsky, E. 49 parameters 2, 3, 46, 140. see also principles of combination Park, R.E. 18 Pastoureau, M. 40, 70, 72, 79, 80, 83 pattern design 68–69, 75–76, 95–96 performance 20, 23, 38–39, 46 personality 12–13, 80, 81, 83 perspective 158. see also background; foreground Pestalozzi, J.H. 102 Petrouchka 129 Pevsner, A. 117 phonology 2, 45 Picasso, P. 70 Pickford, R.W. 81 Pino, P. 99 pitch range 11, 146–147 plastic 99, 113 The Player 31 Poiret, P. 85 portraits and individuality 12 PowerPoint 29, 31–32, 42, 78–79, 97, 129–130 Prague School 35, 38 Prayer Mills 137, 138 Pride and Prejudice 89

Primo Levi 15, 17, 19 principles of combination 141–142, 155–158, 159 product design 24–25, 37, 111–112, 121–123, 133, 134, 135, 136 production format of talk 53 production media. see medium provenance 2, 25, 47–49, 72, 82, 83, 89–90, 105, 114, 116, 119, 120, 130, 149, 159 psychographics 20, 22 psychological conceptions of identity 12–13, 81, 83 purity (of colour) 85 Rafaeli, A. 19 Ray-Ban 25 rectangles. see angularity resource (as a semiotic term) 52–53 regularity 44, 63–64, 109, 128–12, 153 relief 107 Rembrandt 46, 50, 84 repetition 64–66 Reuters 37 Rietveld, G. 60 rigidity 108 Riley, C.A. 84 Riley, J.W. 56 Rodchenko, A. 102 role identity and its semiotic expression 6, 18–20, 23, 40, 78 Rothko, M. 2–3, 84, 88 roughness 108–109, 148, 153 Rousseau, J-J. 102 Rubens 50 Rubinfeld, A. 104 Russolo, L. 101 Sachdev, J. 105 Sachs, C. 127 Samara, T. 28 Sappor, R. 122 Sassoon, R. 67, 68 Satie, E. 100 saturation 84–85 Saussure, F. de 97 Schafer, R. Murray 45, 147, 150 Schipp, D. 20 Schwitters, K. 100, 101 sculpture 100, 109, 110, 111, 117, 137–138, 155 Sekularac, A. 80 Self Portrait (Tinguely) 110, 138 setting (in impression management) 20

Index

Shannon, C. 56, 57 shape and identity design 32, 55–70 Sheets-Johnstone, M. 119 Simone, N. 149 size 66–67 sloping 67–69 Slow Living 128 Slutzky, N. 111 SmartArt 32–33 Smith, C. 78 social identity and its semiotic expression 6–11, 23, 40 social recognition of identity 19, 20 Sontag, S. 12 Sophocles 11 sound effects 120, 150 sound symbolism 45 sources of kinetic energy 135 Stanislavski, L. 118 Starbucks 65, 104, 109 Star Wars 149 Stenglin, M. 69 Strauss, R. 142 style 3, 6, 23, 24, 35–36, 38–39, 41–54, 72, 119 Sullivan, L. 37 synaesthesia 144–146 Tagg, P. 120 taste 98 Tatlin, V. 102 tattoos and identity 7–8 Taxidriver 8 Taylor, C. 13 Teige, K. 37, 61 television 15, 19 temperament 12–13 temperature 87–88, 107 templatization 27, 28–34 tension 147 texture and identity 100–105 texture, visual representation of 114–116 Theophrastus 11 Thompson, P. 57, 58 Thomson, A. 36 Tinguely, J. 110, 111, 137–138, 155 Titian 88 To Have and Have Not 70 Tönnies, F. 9

179

touch 98, 102, 103 toys 121–122, 133, 134, 13, 136 transparency 85–86 triangles 58 Triggs, T. 163 Tschichold, J. 59 Tufte, E. 31–32 Turkle, C. 22 typography 21, 45, 59–60, 118, 163 Van Doesburg, T. 60 value 83–84 Van Campen, C. 144, 145 Van Ginkel, R. 16 Van Gogh, V. 81, 86, 90, 93, 155 Van Leeuwen, T. 13, 28, 31, 37, 43, 44, 49, 50, 52, 60, 62, 66, 70, 97, 106, 113, 121, 126, 127, 133, 137, 138, 139, 141, 146, 149, 150, 154 velocity 128 Venus Fly Trap 107 vibrato 148 Vignelli. M. 29 viscosity 106–107 Vogue magazine 82, 87 Von Sturmer, J. 6 Waits, T. 148 Walker, K. 85 Ward, R. 16 Warde, B. 118 Weaver, W. 56, 57 Weber, M. 34 Webern, A. 154 website design 29, 77, 78 weight 66–67 Weiss, E. 158 Westerns 29 Wierzbicka, A. 83 Windahl, S. 55 Wodak, R. 16 Woolman, M. 45, 118 Word 29–31, 42, 58 World Health Organization 24 Zablocki, B.D. 20 Zijderveld, A. 48 Zuboff, S. 20, 22, 144 Zuckerberg, M. 22